# Automatic Image Content Extraction:
Operationalizing Machine Learning in Humanistic
Photographic Studies of Large Visual Archives

Anssi Männistö1, Mert Seker2, Alexandros Iosifidis3, Jenni Raitoharju2,4

(1Unit of Communication Sciences, Tampere University, Finland
2Unit of Computing Sciences, Tampere University, Finland
3DIGIT, Department of Electrical and Computer Engineering, Aarhus University,
Denmark
4Programme for Environmental Information, Finnish Environment Institute,
Finland
)

###### Abstract

Applying machine learning tools to digitized image archives has a potential to
revolutionize quantitative research of visual studies in humanities and social
sciences. The ability to process a hundredfold greater number of photos than
has been traditionally possible and to analyze them with an extensive set of
variables will contribute to deeper insight into the material. Overall, these
changes will help to shift the workflow from simple manual tasks to more
demanding stages.

In this paper, we introduce Automatic Image Content Extraction (AICE)
framework for machine learning-based search and analysis of large image
archives. We developed the framework in a multidisciplinary research project
as framework for future photographic studies by reformulating and expanding
the traditional visual content analysis methodologies to be compatible with
the current and emerging state-of-the-art machine learning tools and to cover
the novel machine learning opportunities for automatic content analysis. The
proposed framework can be applied in several domains in humanities and social
sciences, and it can be adjusted and scaled into various research settings. We
also provide information on the current state of different machine learning
techniques and show that there are already various publicly available methods
that are suitable to a wide-scale of visual content analysis tasks.

##  1 Introduction

The so called _visual turn_ has been one of the major paradigmatic changes in
the fields of humanities and social sciences in the past few decades. Usually
_visual turn_ refers to a shift in emphasis in these fields towards an
increasing importance of the visible. It gained prominence in the 1990s and
succeeded the _linguistic turn_ ([28, 267-278]; [52, 7]). At the beginning of
the 21st century, the visual turn in humanities can be seen as an aspect of
another major change, called _digital turn_ ([50]; [72, 194]). These two
_turns_ \- visual and digital - go on at the same time when media companies,
museums, archives, and other traditional holders of vast image reservoirs have
increasingly brought their collections into public. This opening has provided
completely new possibilities for exploring and analyzing the historical images
and visual heritage.

One of the most important prerequisites for building a more visual social
science is, as [23, 211] defines, demonstrating that visual data provide
answers to research questions, which are not addressed satisfactorily by the
use of more conventional, non-visual, methods. According to [13, 237],
researchers should be aware of the importance of historical photographs when
examining culture, because of their potential to reveal social, economic, and
political life of the past. [18, 2] argue that photographs have been
misunderstood as constituting forms of data in their own right when they
“should be considered in the first instance as means of preserving, storing,
or representing information.” In this sense, photographs should, in their
view, be seen as analogous to code-sheets, tape recordings, or any one of the
numerous ways in which the social researchers seek to capture data. [15, 6]
states that strong visual research collections present rich potential because
they may be productively analyzed in many different ways, both direct and
indirect. The ideal situation would be a carefully made visual research
collection with comprehensive temporal, spatial and other contextual
recording, good annotations, and collection of associated information in an
organized data file. However, as [15] points out, such systematic collections
are relatively rare, which leads many scholars to underestimate the true
potential of photography, film, and video as reliable sources of cultural
information.

Efficient use of visual data in humanities has been hindered by the lack of
suitable methodologies for analyzing and accessing large digital image
collections. Traditional research methods are based on manual and mostly text-
based tools developed in the era of analogue photographs. This limits the
amount of considered photographs usually somewhere between 400-1000 (see
examples in Section 4). These numbers are difficult to scale up as each photo
must be analyzed individually and manually by a human coder. It is time-
consuming to search typical or certain kinds of photos, special patterns or
features in photos from a large image collection as the images usually lack
content-aware annotations, or this information is incomplete or inadequate.

[4, 1] point out that while visual collections may include extensive metadata
about the provenance of a digital image, there is often little to no
structured data pertaining to the content of the image itself. Similarly, [72,
195] see that textual analysis in all shapes and forms, has come to dominate
digital humanities and, as text-based querying provides mainly textual view of
digital sources, “ _as a field, digital humanities has grossly neglected
visual content, causing a lopsided representation of all sorts of digitized
archives._ ” Another problem is that different fields in social sciences,
humanities, and behavioral sciences have been developing and using their
methods distinct from each other. As [53, 2] states, “ _while visual practices
and approaches are being proposed and advocated from myriad disciplinary and
theoretical positions, it seems that often more effort is expended in trying
to ‘appropriate’ a domain, method, or technique than in developing a more
cumulative and integrative stance._ ”

Many scholars are nevertheless enthusiastic about the possibilities of using
machine learning111Note that we mainly use the term _machine learning_ , while
_computer vision_ could have been an appropriate term as it covers also non-
learning-based automatic methods to analyze visual content. However, the main
advances discussed in this paper have been made in learning-based techniques,
especially in _deep learning_. Thus, also deep learning could be an
appropriate term to be used in many places, but our analysis covers a wider
variety of machine learning techniques, making machine learning the term of
choice. methods in various fields of social sciences and humanities (e.g.,
[8]; [11]; [19]; [63]). [51, 18] ponders in her book chapter named
Quantitative Content Analysis of the Visual in The Handbook of Visual Research
Methods as follows: “ _Traditional media research techniques have been
designed with limited media outlets in mind, manageable via human coding
alone. But the ‘big data’ generated by online media practices requires
appropriate digital tools and methods, including automated coding or sorting._
” In a same manner, [53, 9], notes that while “ _the current hype around ‘big
data’ contributes to the idea that empirical research is becoming ‘more
visual’, (…) it does not take advantage of the many visual dimensions of
culture and society as a prime source of data._ ”

Currently, there are only few examples of the actual use of machine learning
tools and methods in these fields and many researchers see machine learning
more as a far-future direction than a realistic option for their studies.
Rose, who has written one of the most comprehensive books on visual
methodologies refers to image content recognition software only briefly in one
paragraph ([58, 292]). She mentions that researchers are working on programs
that will be able to undertake searches based on images’ visual content rather
than on the tags that have been attached to them. As Rose found such software
commercial and very expensive, she chose not to discuss them further in her
book. She ([58, 86-87]) mentions that practical challenges of accessing and
storing images at a large scale are significant. This is why she assumes that
most readers of her book will be able to work with a few hundreds of images at
most. Surprisingly, machine learning themes are not covered even in a recent
comprehensive volume concentrating entirely on methodology, The Handbook of
Visual Research Methods ([54]).

Under these conditions, although more and more historical image collections
are made public, their utilization does not advance equally fast, unless new
methodological tools are put into operation. An obvious reason explaining the
rare use of machine learning tools in humanities is the extremely rapid
development of the required machine learning techniques during the last
decade. In particular, _deep learning_ with convolutional neural networks
(CNNs) along with the improved computing capabilities have relatively recently
opened completely new opportunities in image content analysis ([36]). With the
CNNs it is possible to explore the content (what is represented) and the style
(how is represented) of images in large archives ([72, 195]) as well as to do
distant viewing222[3, i6] describe distant viewing of large image collections
as “framework for the automatic extraction of semantic elements of visual
materials followed by the aggregation and visualization of these elements via
techniques from exploratory data analysis.” They call for establishing code
systems for specific computational interpretations through metadata extraction
algorithms. ([3, i3,i4,i13]) of large collections of visual material before
actually studying them ([3, i3]).

Until recently, as [72] point out, the computational analysis of visual
material has been limited to practical considerations of simple operations,
such as adding metadata ([35]) or looking at some basic features, such as
size, color, or saturation of the visual material ([43]). [72, 195,203-204]
call for new techniques for exploring visual material in archives using non-
textual search methods which enable accessing the contents of images directly
without referring to textual content, such as titles and captions, and without
having to browse through archives manually. They ([72, 198]) state that new
machine learning-based techniques contribute also to the construction of a
radically new overview of visual culture, which allows for the analysis of
trends and changes over extended periods.

During the last few years, pilot studies applying state-of-the-art algorithms
for photographic studies have started emerging. [72] demonstrated the use of a
pretrained CNN model for three different automatic visual content analysis
tasks over a collection of 450,000 historical images: 1) determining whether
an image is an illustration or photograph and defining the historical
watershed moment when the number of photographs overtook the number of
illustrations, 2) querying images based on abstract visual aspects (clustering
visually similar advertisements), and 3) retraining the last layer of the
network to classify images into nine relevant visual categories defined by
domain experts. A recent study by [14], co-authored by most of the authors of
this papers, demonstrated the use of CNNs for conducting a novel kind of
analysis of prominent Finnish World War II photographers. A CNN was trained to
recognize the photographer from the photo, which allowed learning some
abstract similarities and differences between the photographers. Furthermore,
object detection was applied to analyze most typical topics for each
photographer. [4] used CNN-based semantic image segmentation to produce
structured data on the content of historical photos. [68] applied state-of-
the-art semantic clustering, scene classification, and object detection
methods to analyze human-nature interactions in Finnish national parks using
visitors’ geotagged social media photos as source data.

[2] created a protocol for using commercial Application Programming Interfaces
(APIs) for Automatic Visual Content Analysis (AVCA) in large-scale image
datasets. Traditional visual content analysis (see Section 2) starts from
particular research questions and hypothesis, which are approached by
examining structural content of individual images in a sample. The protocol
introduced by [2] does not focus on these structural features and related
variables and values. Rather, it creates a more qualitative level of analysis:
“Using our proposed framework, researchers can evaluate the extent to which
computer vision labels may be able to assist in the classification of concepts
that are theoretically relevant to Communication Science, yet in many cases,
the level of complexity of these concepts may fall short.” ([2, 259]) As [2]
focused on existing APIs instead of the underlying computer vision and machine
learning methods, they point out that “the black-box approach of commercial
API’s also means that these models may change over time, which may pose
challenges to replicability”. ([2, 258])

In this paper, we take a more holistic view on the use of machine learning
methods for visual studies in humanities and social sciences. We aim to
provide a comprehensive picture of how adopting new methods and techniques
will enhance traditional settings of photographic studies in various academic
fields and of what new approaches these tools enable. We summarize our views
into a novel Automatic Image Content Extraction (AICE) framework for large
image archives that can replace the previous models when automatic tools are
used. In AICE, we reformulate and expand the traditional visual content
analysis framework to be compatible with the current and emerging state-of-
the-art machine learning tools and to cover the novel machine learning
opportunities for automatic content analysis. We also provide information on
the current state of different machine learning techniques and show that there
are already various publicly available methods that are suitable to a wide-
scale of visual content analysis tasks. Our goal with AICE is to provide a
general framework for analyzing and searching large image collections. It can
be applied in several domains in humanities and social sciences, and it can be
adjusted and scaled into various research settings.

The rest of this paper is structured as follows. Section 2 focuses on the
basics of traditional visual content analysis. Section 3 introduces our novel
Automatic Image Content Extraction (AICE) framework. The following sections
illustrate the use of AICE framework by explaining how machine learning can
enhance traditional visual content analysis tasks (Section 4) and what kind of
novel opportunities it will open (Section 5). Section 6 discusses the
challenges and drawbacks associated with the novel tools. Finally, Section 7
concludes the paper.

##  2 Basics of Visual Content Analysis

Content analysis is a method of analysing visual images that was originally
developed to interpret written and spoken texts. It was first used in the
interwar period by social scientists wanting to analyse journalism of the
emerging mass media. It was given boost during the Second World War in order
to detect implicit messages in German radio broadcasts. Hence, it is an
explicitly quantitative methodology and it was developed to address the need
of tackling the sheer scale of the mass media. ([58, 85-86], see also [7,
21-25])

According to the classic definition of [7, 18]: “Content analysis is a
research technique for the objective, systematic, and quantitative description
of the manifest content of communication.” While content analysis initially
focused mainly on texts, it was also considered in the context of non-verbal
information. Nevertheless, it is notable that [7] in his classic book Content
analysis in Communication research does not even include photographs in his
list of “interesting applications” of quantitative content analysis. Instead,
he lists with examples following non-verbal forms of communication: paintings,
sculptures, drawings, cartoons, maps, gestures, tone and voice patterns, and
music. ([7, 108-109])

Reading Images: The Grammar of Visual Design, a book by [32], can be
considered as turning point enhancing the use of quantitative methods in
studying the contents of photographs in wide variety of fields in humanities
and social sciences. In their book, [32, 1] provided “inventories of major
compositional structures which have become established as conventions in the
course of the history of visual semiotics, and to analyze how they are used to
produce meaning by contemporary image-makers.” They called their approach _the
grammar of visual design_ , which is a large, theory-like concept covering all
sorts of images, such as photographs, movies, drawings, paintings, and
advertisements. Many scholars have adjusted the theory and its tools to more
narrow research settings. [5] was maybe the first to use the term _Visual
Content Analysis_ (VCA) leaning mostly on tools, variables, and values created
by Kress and van Leeuwen. In his analysis, Bell studied the differences of
cover images of Cleo Magazine published 25 years apart. Bell’s version of VCA
is a readily operable adaptation of the original grammar and its particularly
suitable for photographic analysis. Because of its practicality we have chosen
to use it as one of the major sources for the AICE framework we introduce in
this paper.

[5, 8] describes VCA as “a systematic, observational method used for testing
hypotheses about the ways in which the media represent people, events,
situations, and so on. It allows quantification of samples of observable
content classified into distinct categories.” In a same manner, [56, 3] define
quantitative content analysis “as the systematic assignment of communication
content to categories according to rules, and analysis of relationships
involving those categories using statistical methods.” Although Bell and Riffe
speak in the context of media or communication content, there is nothing that
restricts the use of VCA only to those research areas.

Lutz and Collins (in [58, 87]) separate three factors which favor the use of
content analysis in dealing with relatively large numbers of images. First,
they suggest that content analysis can reveal empirical results that might
otherwise be overwhelmed by the sheer bulk of material under analysis. Second,
they insist that content analysis and qualitative methods are not mutually
exclusive and content analysis can include qualitative interpretation.
Finally, they also suggest that content analysis prevents a certain sort of
‘bias’, because it is a method for analysing images that does not rely on pre-
existing interpretative categories.

The factor separating quantitative analysis, like VCA, from many other main
research techniques, such as qualitative analysis or discourse analysis, is
its focus on the ‘manifest content’. [56, 22] state that important in this
specification is focusing on content’s manifest (or denotative or shared)
meaning as opposed to connotative (or latent) ‘between-the-lines’ meaning”.
They stress that quantitative content analysis is the “systematic and
replicable examination of symbols, which have been assigned numeric values
according to valid measurement rules.” The analysis, then, involves comparing
relationships of those values using statistical methods. According to [5,
7-8], typical research questions which may be addressed using content analysis
include questions of priority/salience or of ‘bias’ of the content and
historical changes in modes of representation of, for example, gender,
occupational, class, or ethically codified images. As stated by [18, 58], one
of the principal advantages of quantitative visual research is the historical
depth, which can be obtained in an inquiry.

One of the main challenge of content analysis has been that, if a study
focuses on human behavior or visibility in photographs (as the typical cases
are), even selecting the photos depicting people is laborious. The process is
most burdensome, understandably, with photo archives that are not digitized.
With such archives collecting and selecting material and annotating it
manually is a very time-consuming practice as researchers must examine each
photo individually. This is why it is usually not possible to consider a whole
large-scale dataset with the traditional manual methods. With digitized
archives, the process is mitigated as researchers can use dedicated software
to browse and store images and to use search tools. As search tools usually
are text-based, it typically means that only the original caption is used to
describe or tell about the content. In such cases, it is not possible to make
true content-based searches of the collection. Otherwise, the workflow mode
with the digitized archives is quite similar to the traditional process where
a researcher actually picks printed photos from a drawer or filing cabinet or
uses a microfilm reader. With these traditional workflow modes, a researcher
observes features or elements of a photo that are relevant for particular
research questions. In quantitative analysis, a researcher or coder writes
down certain aspects of a photo to a spreadsheet or database. Typically, these
aspects are the amount or qualities of humans or of various object classes. As
this observation process is time consuming, these object classes form usually
just a fraction of the total amount of features in the photos.

A fundamental difficulty with visual media is, as [44] see it, that they do
not have a standard vocabulary or grammar. This makes the adaptation of
automatic visual analysis challenging. The solution [44] suggest is to focus
on “visual form”, such as size, color, or saturation, which is easy for
computers to analyze, and not semantics, which are harder to automate.
However, we believe that systematic semantic analysis of the visual content
has potential to provide much more interesting research outcomes and we see
the current state-of-the-art machine learning techniques adequate to start
using them also for practical semantic analysis purposes. In the next section,
we describe our novel AICE framework designed for systematic large-scale
visual content analysis using automatic computer vision techniques.

##  3 Automatic Image Content Extraction (AICE) framework

Our main purpose is to transform and expand the traditional visual content
analysis framework in a way that is compatible with the current and future
automatic image analysis methods. We formulate our framework in a similar
manner as the version of VCA by Bell (2004) discussed in the previous section,
but we significantly enrich it with new variables, values, and connections to
existing machine learning techniques. The guiding light for Bell, which we
also follow in our framework, was that in order to choose variables, visual
content “must be explicitly and unambiguously defined and employed
consistently (‘reliably’) to yield meaningful evidence relevant to an
hypothesis.” ([5, 9]). To do that it is “first necessary to define relevant
variables of representation and/or salience. Then, on each variable, values
can be distinguished to yield the categories of content which are to be
observed and quantified.” ([5])

Our AICE framework is summarized in Tables 1-3. The first two columns form the
core of framework. They represent the variables and corresponding values for
content analysis considering both the needs of photographic studies and
feasibility of automatic analysis. The variables are divided into six main
topics: Technical variables (1) share some common, basic features of every
image and they are useful in many subsequent higher level image analysis tasks
(e.g., social distance estimation ([62])). With modern photographs, variables
1.2-1.4 can usually be obtained from the metadata, but for historical photos
this is typically not the case. When the information is not included, it may
also be predicted from the photo content and, therefore, is included in our
AICE framework. Especially for the historical photos, the camera technical
variables may be also a study question as such to analyze the development of
photography. Predicting the photographer from a general photo is naturally
almost impossible, but may be possible to some extent when the possible pool
of photographers is limited ([14]). The variables in 1 and 1 correspond to the
“visual form” that is mostly easy to evaluate from the images. The semantic
image content and participants, i.e., topics, people, objects, settings, and
their properties are included in 1, whereas the interaction between the
participants and between the participants and the camera are evaluated in 3.
The variables in 3, on the other hand, bring into our framework many important
applications of visual analysis based on appearance-based similarity, which
may be evaluated focusing on different variables defined in AICE. The
practical use of these variables will be illustrated in different examples
below.

Table 1: Automatic Image Content Extraction (AICE) framework, part 1/3 Variable | Values | Difficultya | Tasks
---|---|---|---
1 Technical |  |  |
1.1 Image type | Photograph/illustration/map/… | Easy | Classification
1.2 Camerab | Camera model | Difficult | -
1.3 Focal lengthb |  $<$17mm/17-34mm/34-70mm/ | Easy | -
71-300mm/$>$300mmc
1.4 Sensor/film sizeb | Numerical | Medium | -
1.5 Photographerb | Photographer name | Difficult | Photographer recognition
2 Composition |  |  |
2.1 Photo shape | Horizontal/vertical/square | Trivial | -
2.2 Photo framing | Close-up/medium shot/overview | Easy | Framing classification
2.3 Camera angle | Numerical | Medium | Camera pose estimation
3 Modality |  |  |
3.3 Color type | Grayscale/color | Trivial | -
3.4 Tonality | Dark/medium/light | Easy | -
3.5 Dominant colors | Blue/red/green/yellow… | Easy | -
3.6 Color distribution | Histogram | Easy | -
4 Content/Participants |  |  |
4.1 General |  |  |
4.1.1 Main topic | Person/animal/object/ | Medium | Classification
environment/event/…
4.1.2 Salience | Salience map | Easy | Salience estimation
4.2 Persons/actors |  |  |
4.2.1 Characteristics |  |  |
of single persons |  |  |
4.2.1.1 Status | Main character (MC)/ | Easy | Main character recognition
side character (SC)
4.2.1.2 Age | Baby/child/young/adult/old | Easy | Age estimation
Exact age | Medium
4.2.1.3 Gender | Male/female/other | Easy | Gender estimation
4.2.1.4 Identity | Name | Easy | Face recognition
4.2.1.5 Ethnicity | European/Asian/African/… | Medium |
4.2.1.6 Height | Short/average/tall | Medium | Height estimation
Exact height | Difficult
4.2.1.7 Weight | Thin/average/fat | Medium | Weight estimation
Exact weight | Difficult
a) Evaluated by machine learning experts without experimental testing.
b) This information may be available as metadata, but it can be also estimated
from the visual content.
c) This typology corresponds to superwide/wide/normal angle / medium telephoto
/ supertelephoto categories
with cameras that have ’full size’ (35mm) digital sensor or use 35mm film.
Table 2: Automatic Image Content Extraction (AICE) framework, part 2/3 Variable | Values | Difficulty | Tasks
---|---|---|---
4.2.1.8 Occupation | Doctor/police/cook/pilot/… | Medium |
4.2.1.9 Role | Child/mother/friend/neighbor/… | Difficult |
4.2.1.10 Nudity | Nude/partially nude/clothed | Medium | Nudity detection
4.2.1.11 Condition | Healthy/sick/wounded/dead/… | Medium |
4.2.1.12 Clothes | Skirt/trousers/jacket/coat/… | Medium | Clothes recognition
4.2.1.13 Clothing style | Representative style | Medium | Style clustering
4.2.2 Characteristics |  |  |
of multiple persons |  |  |
4.2.2.1 Number of people | 0/1(single)/2(couple)/ | Easy | Person detection
3-6(small group)/7-12(medium group)/
13-30(large group)/31-(crowd)d
4.2.2.2 Number of groups | 1/2/3+ | Medium |
4.2.2.3 Group typology | Unfocused/common focused/ | Medium | Gaze estimation,
jointly focusedd |  social distance est.e
4.2.2.4 Group type | Family/friends/sport team/… | Medium |
4.2.2.5 Atmosphere | Casual/formal/intimate/festive/… | Difficult |
4.3 Objects/goals |  |  |
4.3.1 Status | Main motif (MM)/side motif (SM) | Medium |
4.3.2 Text in image | Textual | Easy |  Optical character rec.e
4.3.3 Categories |  |  |
4.3.3.1 Animals | Cat/dog/cow/horse/fish/… | Easy | Object detection
4.3.3.2 Common objects | Book/ball/car/boat/… | Easy | Object detection
4.3.3.3 Rare object |  | Medium | Object detection
4.4 Settings/events |  |  |
4.4.1 Status | Main motif (MM)/background | Medium |
4.4.2 Indoor/outdoor | Indoor/outdoor | Easy | Scene recognition
4.4.3 Privacy | Private/semi-public/public | Medium | Classification
4.4.4 Scene | Urban/rural/forest/hospital/school/… | Medium | Scene recognition
4.4.5 Event | Sport game/wedding/concert/… | Medium | Event recognition
4.4.6 Locationb | Location | Medium | -
4.4.7 Time of dayb | Morning/day/evening/night | Medium | Classification
4.4.8 Time of yearb | Winter/spring/summer/autumn | Medium | Classification
4.4.9 Weather | Sunny/cloudy/raining/snowing/… | Medium | Weather recognition
b) This information may be available as metadata, but it can be also estimated
from the visual content.
d) Categories adopted from ([16])
e) est. = estimation, rec. = recognition
Table 3: Automatic Image Content Extraction (AICE) framework, part 3/3 Variable | Values | Difficulty | Tasks
---|---|---|---
5 Interaction |  |  |
5.1 Spatiality |  |  |
5.1.1 Of MC in image plane | Left/right/up/down/middle | Easy |  Main character rec.f
5.1.2 Of MC in depth | Front/middle ground/background | Easy |  Main character rec.f
5.1.3 Of MM in image plane | Left/right/up/down/middle | Medium | Object detection
5.1.4 Of MM in depth | Front/middle ground/background | Medium | Object detection
5.1.5 Of SC/SM in image plane | Left/right/up/down/middle | Medium | Person/object detection
5.1.6 Of SC/SM in depth | Front/middle ground/background | Medium | Person/object detection
5.1.7 Spatial relationships | ‘X on Y’/‘X behind Y’/… | Medium | Visual relationship det.
5.1.8 Spatial distribution | Image segmented in classes | Medium | Scene segmentation
5.1.9 Spatial coverage | Percentage covered by classes | Medium | Scene segmentation
5.2 Distances |  |  |
5.2.1 Of MC to camera | Intimate(-0.5m)/close personal | Medium | Social distance est.f
(0.5-0.8m)/far personal(0.8-1.2m)/
close social(1.2-2m)/far social(2-3.5m)/
public(3.5m- )g
5.2.2 Of MC to others | As above | Medium |  Social distance est.f
5.2.3 Between groups | As above | Medium |  Social distance est.f
5.3 Activity |  |  |
5.3.1 Gaze direction of MC | Towards/away/right/left | Easy | Gaze estimation
5.3.3 Expressions of MC/SC | Happy/sad/afraid/surprised/… | Medium | Expression recognition
5.3.4 Pose of MC/SC | Standing/sitting/laying | Easy | Pose estimation
5.3.5 Action of MC/MM | Jumping/running/falling/… | Medium | Action recognition
5.4 Interaction |  |  |
5.4.1 Gaze target of MC | Object/person/other | Medium | Gaze estimation
5.4.2 Human interaction | ‘X hugs Z’/‘X talks to Y’/… | Medium |  Human iact. rec.f
5.4.3 Human-object interaction | ‘X sits on Z’/‘X reads Y’/… | Medium |  Human-object iact. det.f
6 Visual similarityh |  |  |
6.1 Similar images | Images ranked by similarity | Medium |  Content-based image ret.f
6.2 Appearance-based grouping | Most similar group | Medium | Clustering
f) rec. = recognition, est. = estimation, iact. = interaction, det. =
detection, ret. = retrieval
g) The original distances by [26] were in feet. We converted to the metric
system and rounded to the nearest 0.1m for
closer categories and to the nearest 0.5m for further categories.
h) Similarity may be based on any of the above defined variables, e.g., visual
form, face, or objects.

It should be taken into account that the variables are interconnected. Machine
learning methods for more complex tasks can need other variables, e.g., to
perform age estimation, the people in the images must be detected first. In
some cases, the same machine learning method (e.g., a trained CNN) may
evaluate several variables (e.g., height and and weight estimation) or partial
combinations of different variables (e.g., a classifier can categorize both
scenes and events). The values suggested in the second column are only
advisory. After selecting the variables of interest, the target values must be
decided based on the research goals (and constrained by available machine
learning techniques).

The third column (‘Difficulty’) aims at giving an impression on the difficulty
of the tasks for the current automatic image analysis and machine learning
algorithms. ‘Trivial’ stands for variables that can be easily and without any
errors determined from the image, ‘Easy’ tasks can be performed using the
existing algorithms with a high accuracy, but there may be some errors or
ambiguities. ‘Medium’ tasks can be currently performed to some extent, but
errors are expected. Finally, ‘Difficult’ tasks may remain partially
unsolvable also in the future. For example, professions may be recognized in
some special cases where distinctive clothing or equipment is used, while
defining the profession of a random person walking on a street during his/her
free time is almost impossible. It should be noted that the evaluation is
based on subjective views of the co-authors having a background in machine
learning research. No experimental comparisons of the difficulty level were
conducted. The difficulty levels depend significantly on the source images
(e.g., the time period and environment where the photos were taken) and are
subject to change as new algorithms for different tasks are being developed.
Furthermore, it should be noted that we evaluated the difficulty levels
considering only the difficulty of the task itself assuming that large-scale
annotated training datasets are available. However, the more specialized the
analysis task at hand, the less likely it is to find suitable training dataset
or the training sets may contain only a small subset of topics of interest. We
return to the challenges of the training data in Section 6.

Finally, the last column (‘Tasks’) links the visual content analysis variables
with the available methods and terminology used in the computer vision field.
The description of the specific methods available for different tasks is
beyond the scope of this paper, but interested readers may refer to articles
in computer vision and machine learning fields to look for more information. A
list of state-of-the-art methodologies for the different problems is:
classification ([34]), photographer recognition and framing classification
([14]), camera pose estimation ([30]), basic color analysis ([22]), salience
estimation ([37]), main character recognition ([61]), age and gender
estimation ([57]), face recognition ([71]), height and weight estimation
([1]), nudity detection ([27]), clothes recognition ([41]), clothing style
clustering ([45]), person detection ([10]), gaze estimation ([29]), social
distance estimation ([62]), optical character recognition ([47]), object
detection ([40]), scene recognition ([76]), event recognition ([70]), weather
recognition ([75]), scene segmentation ([20]), expression recognition ([38]),
pose estimation ([12]), action recognition ([17]), visual relationship
detection ([74]), human interaction recognition ([64]), human-object
interaction detection ([39]), content-based image retrieval ([67]), and
clustering ([48]).

We can now return to the recent pilot works demonstrating the use of CNN
models for photographic studies described in Section 1. The first task
considered by [72] was to classify images into illustrations and photographs,
which corresponds directly to the variable 1 of the AICE framework. In the
second task, they clustered visually similar advertisements, which corresponds
to the AICE variable 3. In the third task, they classified images into nine
classes: buildings, cartoons, chess, crowds, logos, maps, schematics, sheet
music, and weather reports. This task can be seen as a combination of the
variables 1, 1, and 2. [14] aimed at evaluating the photographer based on the
photo, i.e., the variable 1. Furthermore, they applied object detection to
locate common objects and animals in the photos according to the variables 2
and 2 and evaluated the variable 1 by categorizing the photographs into close-
ups, mediums shots and overview. [4] segmented historical photos into regions
containing elements such as sky, water, trees, grass, and roads according to
variable 3 to produce structured data on the content of historical images.
[68] used scene classification (2), object detection (variables 2 and 2), and
semantic clustering (3) to analyze human-nature interaction in Finnish
national parks. They analyzed typical photo content and differences between
national and international national park visitors. In the next two sections,
we provide more examples of potential uses of our AICE framework by discussing
how automatic image analysis could have enhanced some traditional visual
content analysis studies and what novel opportunities it can open.

##  4 How machine learning can enhance traditional visual content analyses

In this section, we have collected some pioneering studies that have used
relatively large collections of images - photos the researchers themselves
have taken, advertisements and media images, photos stored in various
photographic archives - in many fields of humanities and social sciences. We
describe the material collection and the research methodology used and
elaborate how applying machine learning tools according to the proposed AICE
framework could have advanced the process of collecting material or
determining variables and their values, and, thus, possibly enhanced the
results. List of studies selected here is, of course, by no means
comprehensive.

In traditional photographic research settings of media images using visual
content analysis, the number of photographs is typically between 400-1000. As
examples of war or crisis related topics, [25] constructed a systematic
inventory of the types, range, and frequency of photographic images presented
in American news magazines during Persian Gulf War with 1104 images (also
[24]), [42] compared the topics of photographs of Afghanistan War (2001, 204
photos) and Iraq War (2003, 343 photos) in Finnish newspapers, [49] studied
death images in Israeli newspapers with 506 photos, and [73] explored how
Syrian refugees have been visually portrayed in the U.K. online media by
employing a visual content analysis of 299 photographs. These studies share
some basic variables that are core in our AICE framework, such as 2, 1, 1, 1,
1, and 1. In all these studies, similar to all the other studies discussed
later in this section, one of the main benefits of using automatic image
analysis would be the possibility to conduct the study on a much larger
dataset, possibly collected from additional sources and time periods.

One of the pioneering projects using photographic information in the field of
visual anthropology was ‘Balinese Character: a Photographic Analysis’ by
Bateson and Mead (1942). The study was carried out in the late 1930s with the
aim of inquiring Balinese culture. In the center of the book, there are 759
photographs organized in different themes. The original amount of photos was
25,000. The analysis focused on gestures and bodily expressions ([18]).
Machine learning tools for AICE variables 3, 3, 3, and 3 could be used to
automatically repeat a similar analysis on the full dataset.

[21] analyzed stereotyped portrayals of males and females in magazine
advertisements in a seminal work ‘Gender Advertisements’ including 508 images.
After Goffman’s work, several researchers have extended the inquiry into
gender stereotypes using replications of Goffman’s content analysis
methodology, as stated by [6, 205] (See also [18]). For our paper, it is
beneficial to take a closer look on the way [6] revisit Goffman’s work. Their
goal was to demonstrate usefulness of combining quantitative analysis with
semiotic approach. In order to do this, they developed further Goffman’s
approach utilizing dimensions of visual structure, derived from and
operationalized using Kress and van Leeuwen’s system of analysis (see Section
2). They used this apparatus to analyze 827 advertisements collected from
popular Australian magazines published during 1997–8. ([6])

Many of the hypotheses [6] include are relatively easy to analyze
automatically. Examples of such hypotheses are: “ _In terms of social
distance, females will be framed more intimately and be less likely to be
represented at a ‘public’ distance than men_ ”, “ _Women, more frequently than
men, will be shown on the right section of a layout that is structured along
the horizontal axis_ ”, and “ _Women will less frequently be depicted gazing
at the camera/viewer than men_ ”. AICE variables 1, distance of 3, 3, and 3
directly correspond to and Bell and Milic’s interests, whereas group formation
of genders could be examined by combining methods for 2, 2, and 2 with gender
recognition. One dimension [6] give a lot of attention is the formal relations
between depicted elements which can be analyzed in terms of position in the
frame. The AICE framework is particularly capable acknowledging the spatial
distribution of persons and objects inside the frame both in image plane and
in depth. Several AICE variables, such as the spatiality of the main character
(3) and the main motif (3) in image plane or in depth (3 and 3), will
contribute to this. With machine learning tools it would we easy to add also
other related variables such as 1, 1, 1, or 3.

[31] began in 1980s to research Life magazine, which is considered as the most
influential publication of visual news in the post Second World War period.
She wanted to analyze whether the traditional “family values” were referenced
as an idealized portrait of the white, middle-class nuclear family consisting
of female housewives and male breadwinners. While collecting the material,
Kozol faced several challenges. The first and foremost problem was the amount
of material. There were simply too many issues to examine. In the end, Kozol
selected to her study every issue in the months of October and May from
fifteen years of the post war period looking for news photo-essays that
included pictures of families. Kozol’s experience is a typical example where a
researcher cannot study the large amount of material as a whole. It is also a
good example where machine learning tools could enhance the process. Machine
learning could both filter out the potential family photos by selecting photos
containing a woman, man and children (AICE variables 2, 1, 1) and be used to
carry out also further analysis including 1 and 2. While occupation may be
difficult to define automatically (also manually), housewife detection could
be eased by detecting actions (3) such as cooking, cleaning, or childcare.

Lutz and Collins (in [58, 87-93]), made content analysis of nearly 600 of the
photographs published in National Geographic between 1950 and 1986. They
collected the sample choosing one photo at random from each of the 594
articles on non-Western people. This procedure produced a manageable number of
photos that Lutz and Collins and their research assistants could analyze
manually. In a close reading of those photos, they examined issues of race,
gender, privilege, progress, and modernity through an analysis of factors such
as color, pose, framing, and vantage point used in representations of non-
Western peoples. ([55]). Lutz and Collins (in [58, 93]) coded each photo with
22 categories (instead of term ‘category’ we would use here the term
‘variable’). Most of the variables they used are directly included in or are
combination of variables in AICE. At least the following variables are
included: smiling in a photograph (3), gender of adults depicted (1, 1), age
of those depicted (1), activity type and level of main foreground figures
(1,3, 3), camera gaze of main person (1,3), surroundings of people and urban
versus rural setting (2-2), group size (2), male/female nudity (1, 2), and
vantage point (1). If Lutz and Collins had been able to utilize machine
learning based automatic method for analysing the content, it would have
enabled them to use a much larger sample, e.g., all the several thousands of
photos that were in the original 594 articles. While some of the included
tasks cannot be currently performed automatically at human accuracy, the
larger sample would still give statistically significant results and more
variables could be easily added.

[23, 211] made a close examination of depictions of black persons in
advertisements published in Life magazine, 1936-2000. His study is a good
example of a very demanding research setting in which a carefully developed
quantitative data forms a ground for making interpretations of some sensitive
questions, such as how the commitment of white population to racial
integration has changed. Grady’s sample consisted of nine chronologically
stratified periods from 1936-2000 and contained a total of 590 advertisements,
which were scanned and entered into a database. The coding was done manually
on a spreadsheet. Two among the most central observations in Grady’s study
([23, 225,234]) are very interesting as they are combinations of variables in
the proposed AICE framework. The first observation was on the percentage of
black persons depicted in personal or private settings. This could be
operationalized in AICE using variables of 1 and 2. The other observation was
on the percentage of black persons in the sample where there is eye contact
between black and white persons. The values in this variable were “Eye contact
between races” and “One race looking at another”. Grady noticed that his
analysis shows that in the iconography of segregation black persons tended to
look at whites admiringly, while whites looked at blacks in a more patronizing
and condescending fashion. Today, Grady concludes, there is not much
difference in how these two ethnic groups look each other; it is just that
they hardly do so in advertisements. A part of a similar analysis could be
conducted by combining AICE variables of 1 and 3. However, evaluating whether
a look is admiring or patronizing would be very challenging. Even though the
task can be seen as evaluating the variable 3, it is an example where the
selected target values make the task very difficult. Furthermore, annotated
large-scale datasets for expression recognition do not contain these
categories meaning that a lot of data should be first manually annotated to be
able to even try training a machine learning model.

Of the above-mentioned studies, we take Wilmott’s paper for closer
examination. It was published in 2017 and was thus made in a time when many
machine learning tools would have been already available but were not used.
The paper is noticeable and rare as it brings together a detailed VCA setting
of a very serious theme and the use of Securitization Theory, which is a
special branch of study of international relations. In that context, [73, 68]
focused on visual communication acts, or what she refers to as picture acts.
Two of the three research questions Wilmott sought to answer were such that
they could benefit on the use of AICE: 1) How does gender figure into the
visual depictions of Syrian refugees? 2) How do the visual depictions of
Syrian refugees differ across U.K. online newspapers?

The U.K. online quality news websites in Wilmott’s study were The Guardian,
The Telegraph, and The Independent. Thus, the sample reflects three different
affiliations in the political spectrum. Every newspaper article in a certain
time frame was scanned, and all the photographs of Syrian refugees were
collected and included in the study according to several criteria, finally
yielding inclusion of 299 photographs. Several coders annotated the material
utilizing a codebook created originally by the Media Department of London
School of Economics. ([73, 71])

The research was built on several different technical considerations to
evaluate whether photographs humanize or dehumanize refugees ([73, 72]). Most
of these considerations could be easily automated following the AICE
framework. Wilmott’s first interest was to examine whether refugees are
portrayed as individuals. This was done because previous studies assert that
photographs of individual victims evoke more empathetic emotions in viewers
than groups of victims, and further that large groups overshadow individuality
and attribute common characteristics to all members of a given social group.
Following preceding examples, Wilmott coded images of refugees into four
categories, from (1) individuals to (4) large groups (AICE variable 2). The
content analysis was then further refined through an experiment that explored
a second factor: camera distance, using traditional categories of frame of
view, starting from various types of ‘close-ups’ and ending to various
‘overview shots’ (1). Moreover, coding also looked at whether the refugees
look directly into the camera (3). These decisions were based on literature
that suggests that closer photos, especially those in which subjects look
directly into the camera, produce a more intimate association with refugees.
([73, 72]) Furthermore, as [73, 72-73] describes, men are more often
associated with violent behavior than women, and a group of female migrants
more effectively visualizes the feeling of threat. Therefore, photographs have
been coded by ‘male and female’, ‘female’, ‘male’, ‘only children depicted’,
and ‘unclear’ (1 and 1). Wilmott’s findings indicate that the image of the
refugees has become mainstreamed and that there were no major differences on
representing refugees between the compared newspapers. [73, 77-78]

As the number of the photos in the study is rather low, only 299, it is
obvious that the analysis of such a small sample would not be ideal or even
proper use of machine learning tools at all. However, we note that currently
the results have not been compared to the average images in the investigated
newspapers and, therefore, the conclusion of the image of the refugees
becoming mainstreamed lacks some evidence that could be obtained by comparing
with the actual mainstream images. Furthermore, while no significant
differences between the newspapers were found, comparisons against the average
photos in the respective papers could show whether this differs from the
typical images and whether there are differences between the newspapers from
this point of view. We will get back to this example soon in the following
section.

##  5 Potential new opportunities in photographic studies brought by machine
learning

###  5.1 Large-scale multi-factor comparisons

While we have mentioned already several times that machine learning methods
are especially suitable for very large data-sets, it should be emphasized that
this especially concerns comparisons. Machine learning methods tend to do some
errors. For example, it could be that a distance estimation algorithm tends to
somewhat underestimate the distance of the main character to the camera or a
person detection algorithm might find only 95% of the people in the photos.
However, these sort of errors can be considered systematic as long as the
considered sample is large enough and the photos to be compared have a similar
quality. Therefore, claims such as “Photo collection A contains exactly 4521
persons” or “In photo collection B, the average distance of the main character
to the camera is 2.3m” may not be feasible, but the differences between
datasets can be more accurately estimated, e.g., as “In photo collection A,
photos have on average 30% more people than photos in collection B.”

We now return to Wilmott’s study ([73]) on depicting Syrian refugees in
newspapers, which we described in the previous section. AICE could bring a
totally new layer of analysis as it could annotate all the photographs in a
certain period in the three newspapers, not just the 299 belonging to articles
concerning Syrian refugees. By annotating the whole material of thousands of
images, the machine learning enhanced analysis could create a reliable
reference material. Wilmott’s study did show that the differences between the
newspapers were small, but it did not tell how the samples compared in
relation to the typical ways each of the newspapers represent the qualities of
persons in photographs. Consider, for example, The Independent. What if it
turned out that it does in general have a particularly low number of portrays
with eye contact, while at the same time that number would be very high in the
other two newspapers in question. In this imaginary case, the relative
difference of depicting the eye contacts of Syrian refugees compared to the
general way of showing eye contact in person photos would be considerably
different between the newspapers. This is of course just a hypothetical
example, but nevertheless illustrates how the ability to use the entirety –
not just a fragment – of photos in the analysis would significantly broaden
the analysis.

Another important aspect is that AICE allows to easily include multiple
variables into the research setting and look for any interesting correlations
between them. It is understandable that using the traditional methods no one
wants to invest a large amount time to manually annotate some variables that
are unlikely to be interesting for the study at hand. However, with machine
learning-based methods such analysis can be carried with little extra effort.
While in many cases this may indeed confirm that the variable is not
interesting, it is also likely that at some point such trials lead to
interesting observations and novel insights of some phenomena that would have
never been revealed by the traditional research methodology.

###  5.2 Spatiality and use of space

In our view, two interrelating dimensions can be separated in spatiality and
use of space in photographic studies: (1) positions of persons and objects
inside the frame both in the image plane and in depth and (2) social distance
of participants in an image. Analyzing spatiality has been an important goal
in visual research ever since [26] published his theory of _proxemics_ and
social distances. Spatiality is an important aspect also for [18], who are the
authors of the book Researching the visual, considered as one of the most
comprehensive introductions to visual inquiry. According to [18, 4], objects,
people, and events which constitute the raw materials for visual analysis, are
not encountered in isolation but rather in specific context: “It is this
spatial existence which serves as the means whereby much of their
sociocultural significance is imparted. Visual data, in short, must be
understood as having more than just the two-dimensional component.” They ([18,
191]) describe that everyday public behaviour has a spatial and territorial
component to it. Their thinking leans on some classical texts in sociology,
such as Simmel’s ‘The Sociology of senses’ from early 20th century. Simmel
drew attention to one of the key features of modern life, that interaction is
based on sight rather than sound. ([18, 192])

The study of the ways humans utilize or orient to space described by Hall is
essential also for the theory of [32]. According to Hall, in everyday
interaction, social relations determine the distance (literally and
figuratively) we keep from one another. We carry with us a set of invisible
boundaries beyond which we allow only certain kinds of people to come. The
location of these invisible boundaries is determined by configurations of
sensory potentialities – by whether or not a certain distance allows us to
smell or touch the other person, for instance, and by how much of the other
person we can see with our peripheral (60 degree) vision. ([5, 24]; [32,
130-131])

At the core of Hall’s theory is a typology with categories corresponding
different fields of vision. At intimate distance, we see the face or head
only. At close personal distance we take in the head and the shoulders. At far
personal distance we see other person from the waist up. At close social
distance we see the whole figure. At far social distance we see the whole
figure ‘with space around it’. And at public distance we can see the torso of
at least four or five people. ([26, 114-129]. See also [5, 24]; [32, 129-31])
For Kress and van Leeuwen, equally important as the social distance is the use
of space in terms of positions of persons or objects within the frame. They
show, for example, that the value of an element in the top half of a portrait-
shaped frame is different from its value if located in the bottom half. A
similar difference in information value exists also between the dimensions of
center and margin in an image structure. ([32, 193-208]. See also [6, 211]).

While exploring the changes and trends in the ways humans utilize space in
everyday interaction is at the core of many historical and sociological
studies using photographic archives as source material, proper tools for
finding and measuring spatial qualities effectively have been lacking.
Acknowledging and operationalizing them with machine learning tools would mean
a major leap in photographic studies. Basically, this could enable to consider
2D photos as representations of a certain 3D situation or scene. By doing
that, machine learning has potential to revolutionize many traditional
research settings, especially those interested in the interaction of people.

As shown in Table 3, we have adopted the above-described proximity typology as
suggested target values for distance-related variables under 3. This typology
can be utilized at least in three ways in photographic studies, all of which
could be operationalized with different machine learning tools. First, it will
tell also about the physical distance between the photographer and the main
character or theme (3). This distance unavoidably affects the interaction
between the photographer and the main character. Second, determining the
mutual distances of persons in photos (3) will give information of what kind
of relations the persons have with each other, are they close or far away from
each other, how are they interacting, and so on. Third, the typology can be
used to evaluate group formation (2) and distances between groups (3).

Furthermore, the fields of vision linked with the proximity typology closely
correspond to the traditional definitions of photo framing in film and
television. Framing is also a basic property and feature of every photograph,
telling if it is a close-up or an overview (or something in between) of the
main character or theme. When this information is automatically determined
(AICE 1) and included in the metadata of a photo, it will serve as one of the
basic variables in searching images from large image archives in many
professional use cases. Photo framing information can be accompanied by
information on camera angle (1). To bring out more aspects of the three-
dimensional situation that took place in the front of the camera, the variable
3 focuses on how the participant locations are linked to each other (e.g.,
under/next to/behind/in front of).

To give a more practical example, we turn our attention to a study of [65] on
daily street life in Prague. Their basic questions were “Who are the users of
the space?” and “Are there spatial and temporal differences in the manner
various social groups use the public space?”. In order to demonstrate
dimensions of differentiation, they explored the heterogeneity of users based
on differences in people’s wealth, age, ethnicity, and everyday practices.
Methodologically, their study relied primarily on the fieldwork and direct
observation of incidents and events which characterise the outdoor life of the
area. Photographs were used only to illustrate typical and special ways of
utilizing public space and peoples mutual distances and also to identify
people belonging to various categories (age, professions, etc.). The kind of
research setting Temelová and Novák carried out would be possible to create,
e.g., by using large amount of photographs taken from a fixed position with
certain intervals of time. Along with the above-discussed variables related to
spatiality, it would be easy to automatically analyze other variables, such as
1, 1, 3, and 3, Temelová and Novák were interested in. Others, such as whether
a person belongs to the category of “managers and professionals”, estimated in
terms of his/her clothing, of course, are more challenging, but may be
approximated using variables 2 and 2. This kind of approximation is present
also in the original research setting using direct observation. Moreover, if
specific machine learning tools are developed and finetuned to the settings
like this, a similar research could be then easily repeated in many cities or
places as a comparative approach.

It should be noted that the differences and trends related to proxemics may
naturally be quite subtle and not possible to extract from a small set of
images. Instead, very large datasets are needed to reliably evaluate average
behavioral traits. Such research has not been feasible using traditional
manual research methods and, thus, machine learning methods may open a novel
unexplored research direction. Analyzing photographic contents of the news
media during the COVID-19 crisis, for example, would provide valuable material
for observing the changes in peoples behavior in terms of their use of
personal space. During this “corona era”, the public speech was filled with
notions concerning how far away people should be from each other. Reporting of
the crisis filled or affected practically the whole coverage of the news media
and the time span was rather long. It would form, thus, a good entity against
which to compare findings in different countries or of a reference period few
years earlier.

###  5.3 Analysing interaction, gestures, and gaze

Analysing interaction, gestures, postures, touching, or gazes between persons
in photos is a typical task in many photographic research settings. As [18,
218-220] put it, humans use body language consciously or unconsciously to
convey information to others, e.g., about emotions, social status, openness to
interaction, and sexual arousal. Much of the most interesting theory is on the
role of gaze and eye contact in contemporary social life. However, as the
authors note, such work is difficult to conduct, because of the
unobservability of the gaze. As Simmel and others have observed, the modern
city has brought people into proximity. This has caused problems for the use
of gaze and also, on occasion, for touch ([18, 225]). With traditional visual
inquiry methods, analysing these kinds of variables from the photographs is
one of the most time-consuming phases of any research process as it needs a
lots of interpretation.

Machine learning offers several tools for automating this kind of analysis
shown in AICE under variable categories 3 and 3. Here, it should be noted that
while evaluating a person’s pose (standing/sitting/laying) is relatively easy,
it is challenging to evaluate actions with movement as a single shot of, e.g.,
walking, running, or jumping person, may look very similar and more
information (video or several shots) is needed for reliable estimation. On the
other hand, tools for 3 can be used to find out whether an interesting gesture
or pose appears repeatedly, and tools for 3 can be used to divide action and
poses automatically into groups containing similar gestures (see Section 5.5).

Combining this data (what he/she is doing and to where he/she is looking at)
with scene recognition (e.g., 2 and 2) and with the spatial information as
discussed in 5.2 will produce an unprecedented and rich compilation of content
properties of an image. These data are useful as such when making automatic
annotations of photographic contents for various needs of image archives. It
enables also constructing new kinds of search tools and methods and it will,
thus, also contribute to improving accessibility of the image archives. The
tools would be useful in several fields of social sciences and humanities,
studying, e.g., possible changes of external forms of, or interaction between,
populations in certain time span.

Figure 1: An example of a skeleton model ([12]) that can be used for automatic
pose analysis. (Image source: https://www.pexels.com)

As a practical example, pose estimation, such as the OpenPose model ([12])
illustrated in Fig. 1, could be used in a sociobiological study similar to the
study Klein carried out searching for differences in sitting postures of males
and females. In Klein’s study, postures were recorded in 600 men and women and
the study focused on typical ways of sitting, depending on gender, age, and
place, in one city in 1984. ([18, 220-226]) With machine learning tools it
would be possible to classify not just the original material, but reference
material from other cities and in other times, depending of course on the
available archives. Furthermore, automatically clustering sitting poses
according to the AICE variable 3 could reveal typical sitting pose variants
that were not observed in the manual examination.

###  5.4 Salience

Defining what is the most salient element is present in almost every research
setting in humanities and social sciences using photographs as source
material. The most salient elements are usually also those which at minimum
need to be acknowledged and annotated when describing the content of a
photograph. As shown earlier in this paper, human behavior or visibility is
typically at the core of study in many research settings. In such cases, the
researcher usually needs to define who is the main character, which in most
cases is also the most salient participant in the photograph. Defining salient
elements can, thus, be considered as one central functions in visual content
analysis.

As [5, 21] describes, content analysis can show what is given priority or
salience and what is not. It can show how images are connected, who is given
publicity and how, as well as which agendas are run by particular media. [33,
179-220] discuss salience in detail with examples of many types of images
(movies, classical art, advertisements, and magazine page layout). For them,
salience indicates that  “some image elements are made more conspicuous than
others elements (…) and this makes them more important of information in the
whole” ([33, 181]). Salience is also one of the three key compositional
principles of images alongside framing and information value of different
parts of an image ([33, 204]).

Traditionally, detecting salience from images has required a lot of attention
– and interpretative work from the researcher – as salience can be a result or
a combination of several different variables. This becomes obvious when
looking how [33, 211] define the term: “Visual salience results from a complex
interaction (…) between a number of factors. These include size, sharpness of
focus, (…) areas of high tonal contrasts (…), color contrasts, placement in
the visual field, perspective (foreground objects are more salient than
background objects, and elements that overlap other elements are more salient
than elements they overlap)”. Also, as [33, 216] point out, it is not only
what is made salient, but also how it is made salient that contributes to the
meaning.

Machine learning methods offer a way to carry out large-scale salience
analysis (AICE 1) in an objective manner and to combine the salience analysis
with other AICE variables or with the textual content to analyze how certain
topics are illustrated. Recognizing and analysing the main character can be
seen as a subtask of salience analysis and it is traditionally one of the
basic tasks - and variables - in wide variety of photographic studies in the
fields of social sciences and humanities. A recent work ([61]) demonstrated
that it is possible to automatically recognize the main character with a high
accuracy in different kinds of photos as illustrated in Fig. 2.

Figure 2: An example of automatic main character recognition ([61]).(Image
source: https://www.pexels.com)

###  5.5 Appearance-based search and analysis

It is worth emphasizing that the new opportunities machine learning-enabled
analysis provide extend beyond the existing predefined categories. When deep
neural networks are trained, they internally learn to form generalized
representations of the images that allow them to classify also previously
unseen images. These representations can be also extracted and used for
searching or grouping images based on their visual similarity (AICE 3) as also
discussed by [3] with respect to _distant viewing_ of the databases.

Maybe the most straightforward way to exploit these internal representations
is _content-based image retrieval_ ([67]) via _query by example_. This means
that instead of searching for a particular category, it is possible to search
for images that are most similar to a particular query image or a group of
query images (AICE 3). Here, it should be noted that the similarity always
depends on the task the training was performed for. For example, if a network
was trained for recognizing faces, it will learn to extract representations
that help it in this particular task. These representations can be then used
for _query by face_ , where the query image can depict any (unknown) face and
the network can find the most similar faces. In the same manner, the
representations learned by networks trained for the corresponding tasks can be
used, for example, for querying by objects, environments, or poses. An
illustrative example of _query by pose_ is provided in Fig. 3, where the top
image is used for querying and possible query results with similar poses are
shown below. As the similarity of the photos is measured by the similarity of
poses in this example, other image content can be very different, while the
photo is still considered as a close match to the query image. A more
practical example of using such an approach in photographic studies was
provided by [72] in their second task for querying the advertisements based on
their abstract visual aspects.

Figure 3: An illustration of content-based image retrieval in the case of
query by pose.(Image source: https://www.pexels.com)

Furthermore, the network representation can be used for grouping the images
based on their similarity (AICE 3). This can allow revealing such similarities
and groupings of data that the researcher would not be able to look for. There
are also tools for visualizing such found similarities and groupings as
demonstrated in a recent paper ([14]), where the similarities between
different photographers were illustrated based on clustering their photos. A
simplified illustration of clustering images into groups based on the
similarity of poses is provided in Fig. 4, where the clustering algorithm have
identified three distinct pose groups. It should be noted that in real photo
collections the clusters are rarely so clear and distinct, but the images
represent a more continuous distribution of poses (or any other feature used
for clustering) making the clusters fuzzier by nature.

Figure 4: An illustration of clustering images by the similarity of
poses.(Image source: https://www.pexels.com)

As mentioned in Section 5.3, tools for 3, i.e. querying by example, could be
used to find out whether an interesting gesture or pose appears repeatedly in
a photo collection, and tools for 3, i.e. clustering based on visual
similarity, could be used to divide action and poses automatically into groups
containing similar gestures. These techniques are most suitable for studying
such concepts where the text-based descriptions are cumbersome, but the visual
similarity of the studied categories is high. The appearance-based techniques
can also help avoid the difficulty of predefining the pose/action categories
of interest and may also reveal similarities that could be very difficult to
observe manually.

##  6 Challenges and risks of the automatic approach

In this paper, we have extensively discussed the new opportunities and
directions machine learning-based methods open in photographic studies.
However, it is also good to keep in mind that there are also some challenges
and risks related to the new tools. In this section, we review some of the
challenges, and also give our suggestions for mitigating them.

Maybe the most practical problem is that for many researchers in humanities
having non-technical background, it may be difficult to find and apply
suitable machine learning tools. Even though most of the tools discussed in
this paper are publicly available, it requires some expertise to understand
which methods are the most suitable ones, how to apply pretrained models, and
how to adjust them for specific tasks. On the other hand, there are still
interesting novel challenges also from the machine learning point of view.
Therefore, we believe that interdisciplinary research between humanistic and
machine learning researchers is the most fruitful solution to the practical
issues and novel technical challenges. Furthermore, we believe that publishing
the used algorithms in an easy-to-use format (e.g., as in [61]) can help other
researchers in their work and gradually lead towards easier application of the
tools as well as easier comparison with earlier works.

It is important to keep in mind in the analysis of the results that the
machine learning models are not error free, and the harder the task, the lower
the expected accuracy. Also different machine learning models can have very
different performance in the same task, and it is important to keep in mind
that a low performance may result from an unsuitable or poorly employed model,
while other machine learning models might perform better. For the same reason,
it is important to be aware of the employed models in any comparative studies
and not directly compare the results of studies conducted using different
models for the same task, because this might lead to biased or erroneous
conclusions.

Some of the errors made by machine learning models may appear unexplainable or
even absurd due to the _black box nature_ of the algorithms, which refers to
the difficulty of explaining why a model gives a certain output, e.g., a
classification decision ([60]). The black box nature is an enormous problem in
the fields such as medicine or autonomous driving, where a single critical
error may lead to life-threatening situations. In the humanistic studies, such
individual errors are much less severe, but the possibility of errors should
be still considered in the analysis. For this reason, we see machine learning
algorithms most suitable for searching suitable research material among very
large photo collecting and for conducting large-scale comparative studies. As
long as errors can be expected to occur at approximately the same rate on each
dataset, the results can indicate true findings. At the same time, explainable
machine learning ([66]) is expected to develop rapidly due to the pressure in
the fields with huge economic value, and these developments can then benefit
also the future humanistic photographic studies.

A more challenging problem than individual errors are biased results or
analysis. In their survey on biases and fairness in machine learning, [46]
define fairness in the context of decision making as “ _absence of any
prejudice or favoritism toward an individual or group based on their inherent
or acquired characteristics_ ” and list several different sources of bias in
machine learning applications. Most of the machine learning algorithms are
neutral by nature, but they learn based on the training data shown to them and
any bias in the data can be adopted or even emphasized by the model ([69]).
For example, if a facial expression recognition algorithm is trained using
mainly photos of young white people, it is natural that it will be less
accurate with old and/or people of colour. Many of the data-related biases
have a humanistic or societal nature, and more collaboration between
humanistic and machine learning researchers will be needed to better
understand how these biases are reflected in the machine learning-based
analysis, not only in humanistic studies but in various applications across
the society. Besides the training data, also the design of the machine
learning models may introduce some biases. For example, in our recent social
distance estimation algorithm ([62]), we assume that people’s torsos are
upright and underestimates the distances if this assumption is violated. This
happens more commonly, when people are sitting, which, in turn, most likely
leads to more underestimation in indoor images. To tackle different biases, we
further emphasize the importance of close collaboration between humanistic and
machine learning researcher as well as open publication of the applied models
and data whenever possible without privacy violations ([9]). This will allow
critical evaluation of the results and biases in the future studies and
gradually lead to better understanding of possible pitfalls.

When machine learning becomes more widely-used in humanistic studies, it is
expected that the availability of high-quality datasets will encourage
research in the related themes. While the opportunities can open exciting new
directions in humanistic studies, it may become necessary to consider, whether
this will dictate the topics and research question setting too much leading to
increased inequality, e.g., by favoring majority cultures instead of minority
cultures as research topics. In a similar way, if a certain category of
interest is missing from the available training data, it may be tempting to
just omit the category from the research, leading to altered research
directions. To avoid such problems, humanistic field should not see machine
learning as a ready-to-use tool, but also take responsibility in annotating
large-scale image collections with the concepts relevant for the subsequent
humanistic studies. To this end, the development of both faster and easier
annotation tools and methods for incremental learning that allow adding some
categories to already trained algorithms (e.g., [59]) are important for future
photographic studies.

##  7 Conclusions

Introduction of novel machine learning methods has potential to both
significantly reduce the time needed for selecting the suitable material and
at the same time to give a more detailed picture of the quantitative features
of very large image collections. The data-driven analysis and ability to
automatically search and annotate content of images can remove the most
laborious tasks in visual content analysis. This may also lead to decreasing
the amount of errors in the observation or note taking process. These changes
will help to shift the workflow from simple manual tasks to more demanding
stages. This includes analyzing and examining the databases, performing a
qualitative phase where the researcher seeks to answer ‘what the data mean’
([5, 22]). It will also enable the use of much larger sets of photos than has
been traditionally possible. As each photo can be annotated with these new
tools with detailed information of its content and structural patterns and
locations, it will enable creating much more sophisticated research settings.
Overall, the novel machine learning techniques will allow to renew at least
the quantitative part of photographic studies in the near future.

In this paper, we proposed a holistic framework for machine learning-based
image analysis called Automatic Image Content Extraction (AICE). To make the
traditional visual content analysis methodologies compatible with the novel
machine learning techniques, we reformulated and expanded the traditional
framework by adding several variables and suggesting suitable values for each
variable. We also considered the difficulty of automatic extraction of each
variable with the current state-of-the-art machine learning models and linked
the variables to existing machine learning techniques.

The proposed framework can be applied in several domains in humanities and
social sciences. We provided multiple examples on how machine learning
techniques applied according to the proposed AICE framework could have
enhanced earlier photographic studies and illustrated the novel opportunities
they open for future studies. We also discussed the main challenges in
adopting the automatic approach and suggested solutions to these challenges.
As the main solution, we encourage expanding collaboration between humanistic
and machine learning researchers. In addition, we believe that humanistic
field should not see machine learning as a ready-to-use tool, but also take
responsibility in annotating large-scale image collections with the concepts
relevant for the subsequent humanistic studies.

While this paper and our proposed AICE framework focus solely on images, it is
worth noting that the novel opportunities opened by the advancement of machine
learning techniques are wider. In particular, different _multi-modal_
techniques that simultaneously analyze different sources of information, such
as news texts, graphical elements, or page layout along with the images or
videos consisting of both visual and audio data, can open additional
interesting opportunities for the future humanistic and social science
studies.

## Acknowledgment

We would like to acknowledge support from Intelligent Society Platform funded
by Academy of Finland, project “Improving Public Accessibility of Large Image
Archives”.

## References

  * [1] Can Yilmaz Altinigne, Dorina Thanou and Radhakrishna Achanta  “Height and Weight Estimation from Unconstrained Images”  In _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 2298–2302
  * [2] Theo Araujo, Irina Lock and Bob Velde  “Automated visual content analysis (AVCA) in communication research: A protocol for large scale image classification with pre-trained computer vision models”  In _Communication Methods and Measures_ 14.4 Taylor & Francis, 2020, pp. 239–265
  * [3] Taylor Arnold and Lauren Tilton  “Distant viewing: analyzing large visual corpora”  In _Digital Scholarship in the Humanities_ 34, Supplement 1 Oxford University Press, 2019, pp. i3–i16
  * [4] Taylor Arnold and Lauren Tilton  “Enriching historic photography with structured data using image region segmentation”  In _Proceedings of the 1st International Workshop on Artificial Intelligence for Historical Image Enrichment and Access_ , 2020, pp. 1–10
  * [5] Philip Bell  “Content analysis of visual images”  In _Handbook of visual analysis_ Thousand Oaks, CA, 2004
  * [6] Philip Bell and Marko Milic  “Goffman’s gender advertisements revisited: Combining content analysis with semiotic analysis”  In _Visual communication_ 1.2 Sage Publications Sage CA: Thousand Oaks, CA, 2002, pp. 203–222
  * [7] Bernard Berelson  “Content analysis in communication research.”  Free press, 1952
  * [8] Henrik Bødker  “Journalism history and digital archives”  In _Digital Journalism_ 6.9 Taylor & Francis, 2018, pp. 1113–1120
  * [9] Frederik Zuiderveen Borgesius, Jonathan Gray and Mireille Eechoud  “Open data, privacy, and fair information principles: Towards a balancing framework”  In _Berkeley Technology Law Journal_ 30.3 JSTOR, 2015, pp. 2073–2131
  * [10] Markus Braun, Sebastian Krebs, Fabian Flohr and Dariu M Gavrila  “Eurocity persons: A novel benchmark for person detection in traffic scenes”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 41.8 IEEE, 2019, pp. 1844–1861
  * [11] Marcel Broersma and Frank Harbers  “Exploring Machine Learning to Study the Long-Term Transformation of News: Digital newspaper archives, journalism history, and algorithmic transparency”  In _Digital Journalism_ 6.9 Taylor & Francis, 2018, pp. 1150–1164
  * [12] Zhe Cao et al.  “OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 43.1, 2019, pp. 172–186
  * [13] Ashley Cerku  “Applied visual anthropology in the progressive era: The influence of Lewis Hine’s child labor photographs”  In _Visual Anthropology_ 32.3-4 Taylor & Francis, 2019, pp. 221–239
  * [14] Kateryna Chumachenko, Anssi Männistö, Alexandros Iosifidis and Jenni Raitoharju  “Machine learning based analysis of Finnish World War II photographers”  In _IEEE Access_ 8, 2020, pp. 144184–144196
  * [15] Malcolm Collier  “Approaches to analysis in visual anthropology”  In _Handbook of visual analysis_ London, 2004, pp. 35–60
  * [16] M. Cristani et al.  “The Visual Social Distancing Problem”  In _IEEE Access_ 8, 2020, pp. 126876–126886
  * [17] Jian Dong, Wankou Yang, Yazhou Yao and Fatih Porikli  “Knowledge memorization and generation for action recognition in still images”  In _Pattern Recognition_ 120, 2021, pp. 108188
  * [18] Michael Emmison and Philip Smith  “Researching the Visual: Images, Objects, Contexts and Interactions in Social and Cultural Inquiry”  London: SAGE, 2000
  * [19] Marco Fiorucci et al.  “Machine learning for cultural heritage: A survey”  In _Pattern Recognition Letters_ 133, 2020, pp. 102–108
  * [20] Jun Fu et al.  “Dual attention network for scene segmentation”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 3146–3154
  * [21] Erving Goffman  “Gender advertisements”  London: Macmillan Press, 1979
  * [22] Rafael C Gonzalez and Richard E Woods  “Digital image processing”  Pearson, 2018
  * [23] John Grady  “Advertising images as social indicators: depictions of blacks in LIFE magazine, 1936–2000”  In _Visual studies_ 22.3 Taylor & Francis, 2007, pp. 211–239
  * [24] Michael Griffin  “Picturing America’s ‘War on Terrorism’in Afghanistan and Iraq: Photographic motifs as news frames”  In _Journalism_ 5.4 Sage Publications Sage CA: Thousand Oaks, CA, 2004, pp. 381–402
  * [25] Michael Griffin and Jongsoo Lee  “Picturing the Gulf War: constructing an image of war in Time, Newsweek, and US News & World Report”  In _Journalism & Mass Communication Quarterly_ 72.4 SAGE Publications Sage CA: Los Angeles, CA, 1995, pp. 813–825
  * [26] Edward T. Hall  “The hidden dimension”  Garden City, NY: Doubleday, 1966
  * [27] Cristian Ion and Cristian Minea  “Application of Image Classification for Fine-Grained Nudity Detection”  In _International Symposium on Visual Computing_ , 2019, pp. 3–15
  * [28] Martin Jay  “Cultural relativism and the visual turn”  In _Journal of visual culture_ 1.3 Sage Publications Sage CA: Thousand Oaks, CA, 2002, pp. 267–278
  * [29] Petr Kellnhofer et al.  “Gaze360: Physically Unconstrained Gaze Estimation in the Wild”  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019
  * [30] Alex Kendall and Roberto Cipolla  “Geometric loss functions for camera pose regression with deep learning”  In _IEEE conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 5974–5983
  * [31] Wendy Kozol  “Life Magazine Photographs”  In _Visual Communication Research Designs_ NY: Routledge, 2009
  * [32] Gunther R. Kress and Theo Leeuwen  “Reading images: The grammar of visual design. 1st edition.”  New York: Roudledge, 1996
  * [33] Gunther R. Kress and Theo Leeuwen  “Reading images: The grammar of visual design. 4th edition.”  London: Roudledge, 2021
  * [34] Alex Krizhevsky, Ilya Sutskever and Geoffrey E Hinton  “Imagenet classification with deep convolutional neural networks”  In _Advances in Neural Information Processing Systems (NIPS)_ 25, 2012, pp. 1097–1105
  * [35] Virginia Kuhn  “Images on the Move: Analytics for a Mixed Methods Approach”  In _The Routledge Companion to Media Studies and Digital Humanities_ Routledge, 2018, pp. 300–309
  * [36] Yann LeCun, Yoshua Bengio and Geoffrey Hinton  “Deep learning”  In _Nature_ 521.7553 Nature Publishing Group, 2015, pp. 436–444
  * [37] Guanbin Li and Yizhou Yu  “Visual Saliency Based on Multiscale Deep Features”  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015
  * [38] Shan Li and Weihong Deng  “Deep facial expression recognition: A survey”  In _IEEE Transactions on Affective Computing_ , 2020
  * [39] Yong-Lu Li et al.  “Transferable interactiveness knowledge for human-object interaction detection”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 3585–3594
  * [40] Li Liu et al.  “Deep learning for generic object detection: A survey”  In _International Journal of Computer Vision_ 128.2, 2020, pp. 261–318
  * [41] Ziwei Liu et al.  “Deepfashion: Powering robust clothes recognition and retrieval with rich annotations”  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 1096–1104
  * [42] Anssi Männistö  “Luottamus katosi savuna ilmaan. Journalismikritiikin vuosikirja 2004.”  In _Media & Viestintä_ 27.1, 2004
  * [43] Lev Manovich  “How to compare one million images?”  In _Understanding digital humanities_ , 2012, pp. 249–278
  * [44] Lev Manovich and Jeremy Douglass  “Visualizing change: Computer graphics as a research method”  In _Imagery in the 21st Century_ MIT Press London, 2013, pp. 315–38
  * [45] Kevin Matzen, Kavita Bala and Noah Snavely  “StreetStyle: Exploring world-wide clothing styles from millions of photos”  In _arXiv 1706.01869_ , 2017
  * [46] Ninareh Mehrabi et al.  “A survey on bias and fairness in machine learning”  In _ACM Computing Surveys (CSUR)_ 54.6 ACM New York, NY, USA, 2021, pp. 1–35
  * [47] Jamshed Memon, Maira Sami, Rizwan Ahmed Khan and Mueen Uddin  “Handwritten optical character recognition (OCR): A comprehensive systematic literature review (SLR)”  In _IEEE Access_ 8, 2020, pp. 142642–142668
  * [48] Erxue Min et al.  “A survey of clustering with deep learning: From the perspective of network architecture”  In _IEEE Access_ 6, 2018, pp. 39501–39514
  * [49] Tal Morse  “Covering the Dead: Death images in Israeli newspapers—ethics and praxis”  In _Journalism Studies_ 15.1 Taylor & Francis, 2014, pp. 98–113
  * [50] Bob Nicholson  “The Digital Turn: Exploring the methodological possibilities of digital newspaper archives”  In _Media History_ 19.1, 2013, pp. 59–73
  * [51] Katy Parry  “Quantitative content analysis of the visual”  In _The SAGE Handbook of Visual Research Methods_ , 2020, pp. 352–366
  * [52] Luc Pauwels  “Taking the visual turn in research and scholarly communication key issues in developing a more visually literate (social) science”  In _Visual studies_ 15.1 Taylor & Francis, 2000, pp. 7–14
  * [53] Luc Pauwels  “An integrated conceptual framework for visual social”  In _The SAGE Handbook of Visual Research Methods_ , 2020
  * [54] Luc Pauwels and Dawn Mannay  “The SAGE handbook of visual research methods”  Sage, 2020
  * [55] “Reading National Geographic” Book brochure, 1993  URL: https://press.uchicago.edu/ucp/books/book/chicago/R/bo3697068.html
  * [56] Daniel Riffe, Stephen Lacy, Brendan R Watson and Frederick Fico  “Analyzing media messages: Using quantitative content analysis in research”  NY: Routledge, 2019
  * [57] Pau Rodriguez et al.  “Age and gender recognition in the wild with deep attention”  In _Pattern Recognition_ 72, 2017, pp. 563–571
  * [58] Gillian Rose  “Visual methodologies: An introduction to researching with visual materials”  London: SAGE, 2016
  * [59] Deboleena Roy, Priyadarshini Panda and Kaushik Roy  “Tree-CNN: a hierarchical deep convolutional neural network for incremental learning”  In _Neural Networks_ 121 Elsevier, 2020, pp. 148–160
  * [60] Cynthia Rudin  “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead”  In _Nature Machine Intelligence_ 1.5 Nature Publishing Group, 2019, pp. 206–215
  * [61] Mert Seker, Anssi Männistö, Alexandros Iosifidis and Jenni Raitoharju  “Automatic Main Character Recognition for Photographic Studies”  In _IEEE International Workshop on Multimedia Signal Processing (MMSP)_ , 2021
  * [62] Mert Seker, Anssi Männistö, Alexandros Iosifidis and Jenni Raitoharju  “Automatic Social Distance Estimation From Images: Performance Evaluation, Test Benchmark, and Algorithm”  In _arXiv 2103.06759_ , 2021
  * [63] Kate Sherren et al.  “Digital archives, big data and image-based culturomics for social impact assessment: opportunities and challenges”  In _Environmental Impact Assessment Review_ 67, 2017, pp. 23–30
  * [64] Alexandros Stergiou and Ronald Poppe  “Analyzing human–human interactions: A survey”  In _Computer Vision and Image Understanding_ 188, 2019, pp. 102799
  * [65] Jana Temelová and Jakub Novák  “Daily street life in the inner city of Prague under transformation: the visual experience of socio-spatial differentiation and temporal rhythms”  In _Visual Studies_ 26.3 Taylor & Francis, 2011, pp. 213–228
  * [66] Erico Tjoa and Cuntai Guan  “A survey on explainable artificial intelligence (xai): Toward medical xai”  In _IEEE Transactions on Neural Networks and Learning Systems_ IEEE, 2020
  * [67] Maria Tzelepi and Anastasios Tefas  “Deep convolutional learning for content based image retrieval”  In _Neurocomputing_ 275 Elsevier, 2018, pp. 2467–2478
  * [68] Tuomas Väisänen, Vuokko Heikinheimo, Tuomo Hiippala and Tuuli Toivonen  “Exploring human–nature interactions in national parks with social media photographs and computer vision”  In _Conservation Biology_ 35.2 Wiley Online Library, 2021, pp. 424–436
  * [69] Angelina Wang, Arvind Narayanan and Olga Russakovsky  “REVISE: A tool for measuring and mitigating bias in visual datasets”  In _European Conference on Computer Vision_ , 2020, pp. 733–751  Springer
  * [70] Limin Wang, Zhe Wang, Yu Qiao and Luc Van Gool  “Transferring deep object and scene representations for event recognition in still images”  In _International Journal of Computer Vision_ 126.2, 2018, pp. 390–409
  * [71] Mei Wang and Weihong Deng  “Deep face recognition: A survey”  In _Neurocomputing_ 429 Elsevier, 2021, pp. 215–244
  * [72] Melvin Wevers and Thomas Smits  “The visual digital turn: Using neural networks to study historical images”  In _Digital Scholarship in the Humanities_ 35.1, 2020, pp. 194–207
  * [73] Annabelle Cathryn Wilmott  “The politics of photography: Visual depictions of Syrian refugees in UK online media”  In _Visual Communication Quarterly_ 24.2, 2017, pp. 67–82
  * [74] Ruichi Yu, Ang Li, Vlad I Morariu and Larry S Davis  “Visual relationship detection with internal and external linguistic knowledge distillation”  In _IEEE International Conference on Computer Vision (CVPR)_ , 2017, pp. 1974–1982
  * [75] Bin Zhao, Xuelong Li, Xiaoqiang Lu and Zhigang Wang  “A CNN–RNN architecture for multi-label weather recognition”  In _Neurocomputing_ 322, 2018, pp. 47–57
  * [76] Bolei Zhou et al.  “Places: A 10 million image database for scene recognition”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 40.6, 2017, pp. 1452–1464