11institutetext:  Christian Morsbach 22institutetext: Institute of Propulsion
Technology, German Aerospace Center (DLR), Cologne, Germany, 22email:
<EMAIL_ADDRESS>33institutetext: Bjoern F. Klose 44institutetext:
Institute of Test and Simulation for Gas Turbines, German Aerospace Center
(DLR), Cologne, Germany, 44email<EMAIL_ADDRESS>55institutetext: Michael
Bergmann 66institutetext: Institute of Propulsion Technology, German Aerospace
Center (DLR), Cologne, Germany, 66email<EMAIL_ADDRESS>77institutetext: Felix Möller 88institutetext: Institute of Test and
Simulation for Gas Turbines, German Aerospace Center (DLR), Cologne, Germany,
88email<EMAIL_ADDRESS>
# Modal analysis of high-fidelity simulations in turbomachinery

Christian Morsbach  Bjoern F. Klose  Michael Bergmann  Felix M. Möller

###### Abstract

We revisit recently published high-fidelity implicit large eddy simulation
datasets obtained with a high-order discontinuous Galerkin spectral element
method and analyse them using Proper Orthogonal Decomposition (POD) as well as
Spectral Proper Orthogonal Decomposition (SPOD). The first configuration is
the MTU T161 low-pressure turbine cascade with resolved end wall boundary
layers in a clean version and one with incoming turbulent wakes. We focus on
the behaviour of the laminar separation bubble and the secondary flow system
and how these phenomena are affected by incoming wakes. The second
configuration is a transonic compressor cascade in which we analyse the
unsteady behaviour of the shock wave boundary layer interaction. Throughout
the paper, we try to discuss not only the flow physics but also insights into
how the modal decomposition techniques can help facilitate understanding and
where their limitations are.

##  1 Introduction

Computational Fluid Dynamics (CFD) simulations have revolutionized our ability
to predict and understand complex flow phenomena. With increasing computing
resources and the development of high-order methods, simulations produce more
and more high-fidelity datasets, which are difficult to analyse. While these
datasets have the potential to improve our understanding of flow physics, the
sheer amount of data, and its temporal and spatial resolution, make gaining
insights a challenging task. Hence, most Large Eddy Simulation (LES) studies
only discuss time-averaged quantities or features based on snapshots.

To facilitate the analysis of these datasets, researchers have recently
developed data analysis techniques, such as Proper Orthogonal Decomposition
(POD) Berkooz1993 ; Weiss2019 , Spectral Proper Orthogonal Decomposition
(SPOD) Schmidt2020 , and Dynamic Mode Decomposition (DMD) Schmid2010 , which
have gained significant attention in the computational fluid dynamics
community. These methods allow for the extraction of the most energetic modes
(POD) and sorting by frequency content (SPOD).

Traditionally, these modal decomposition techniques have been applied to
experimental data. Some recent examples include POD of Particle Image
Velocimetry (PIV) data from a centrifugal compressor inlet Banerjee2022 , POD
and DMD applied to analyse PIV data of a laminar separation bubble undergoing
transition to turbulence Lengani2016 ; Alessandri2019 ; Dotto2021 , POD used
to investigate the interaction of high- and low-pressure turbine based on
unsteady total pressure data in planes between the blade rows
Dellacasagrande2019 , or an investigation of transonic buffet using DMD of PIV
data FeldhusenHoffmann2021 . Very early numerical studies were based on
unsteady Reynolds-averaged Navier-Stokes (RANS) data, e.g. Cizmas2003 , but
more recently, turbulence resolving approaches such as hybrid RANS/LES or LES
have become feasible. Fiore et al.Fiore2023  recently applied the SPOD method
to a low-pressure turbine flow with laminar inflow, as well as with wakes
generated by a row of upstream cylinders. They found that the cylinder wakes
amplify certain modes over the blade and increases turbulence downstream of
the trailing edge over a reduced frequency bandwidth. Mechanisms of trailing
edge (TE) cutback film cooling flows at different blowing rations have been
investigated using SPOD on Delayed Detached Eddy Simulation (DDES) data by
Wang et al. Wang2022 . Here, the authors analysed how the structure of vortex
shedding changes with blowing ratio. He et al. He2021  recently presented an
extensive analysis of the compressor tip leakage flow using SPOD on data
obtained with a DDES. They used the complete 3D flow field in the tip region
of the blade as input for the SPOD and identified low-rank behaviour over a
range of frequencies associated with an oscillation of the tip leakage vortex
and vortex shedding after its breakdown.

At the German Aerospace Center (DLR), we generate LES data with a numerical
test rig for turbomachinery based on the high-order Discontinuous Galerkin
Spectral Element Method (DGSEM). In this study, we will reconsider previously
published high-fidelity datasets of turbomachinery flows Klose2023 ;
Morsbach2023  generated with our DGSEM solver TRACE Bergmann2023  and extract
the dominant dynamic features using POD and SPOD. The datasets include the
interaction of wakes with downstream low-pressure turbine blades and the shock
boundary layer interaction in a compressor cascade. The study will discuss the
advantages and limitations of each method and provide insights into how these
techniques can be used to gain a deeper understanding of complex flow
phenomena.

##  2 Modal decomposition techniques

In this section, we will very briefly introduce POD and SPOD to clarify the
terminology.

###  2.1 Proper Orthogonal Decomposition (POD)

POD by itself is not a new method. Its theoretical background and formulation
have been reviewed decades ago Berkooz1993 . The formulation here is inspired
by the review of Schmidt and Colonius Schmidt2020  as well as the tutorial by
Weiss Weiss2019 . For a consistent derivation of how a snapshot POD can be
conveniently computed on non-equidistant grids using a Singular Value
Decomposition (SVD), we refer the interested reader to the appendix.

We define the matrix of $M$ snapshots of fluctuating quantities (mean
subtracted) where each snapshot is a row and spatial dimensions as well as
multiple variables are flattened to size $N$:

$\mathbf{X}=\begin{pmatrix}\text{\\----}&\mathbf{x}_{1}^{H}&\text{\\----}\\\
&\vdots\\\ \text{\\----}&\mathbf{x}_{M}^{H}&\text{\\----}\\\
\end{pmatrix}\in\mathbb{C}^{M\times N}.$ (1)

The POD is based on the sample covariance matrix

$\mathbf{C}=\frac{1}{M-1}\mathbf{X}^{H}\mathbf{X}\in\mathbb{C}^{N\times N}$
(2)

through an eigenvalue problem with a weight matrix $\mathbf{W}$ and inner
product
$\langle\mathbf{x}_{1},\mathbf{x}_{2}\rangle=\mathbf{x}_{1}^{H}\mathbf{W}\mathbf{x}_{2}$
Schmidt2020 :

$\mathbf{C}\mathbf{W}\mathbf{\Phi}=\mathbf{\Phi}\mathbf{\Lambda},\quad\mathbf{W}=\operatorname{diag}\left(w_{1},\dots,w_{N}\right).$
(3)

Here, $\mathbf{\Phi}\in\mathbb{C}^{N\times M}$ is the matrix of POD modes or
eigenvectors and
$\mathbf{\Lambda}=\operatorname{diag}\left(\lambda_{1},\dots,\lambda_{M}\right)$
is the matrix of eigenvalues. The modes are orthonormal under the scalar
product defined above:

$\mathbf{\Phi}^{H}\mathbf{W}\mathbf{\Phi}=\mathbf{1}.$ (4)

A linear combination of the modes to recover the original snapshots:

$\mathbf{X}=\mathbf{A}\mathbf{\Phi}^{T}=\sum_{j}a_{ij}\Phi_{jk}=\sum_{j}\mathbf{\tilde{X}}_{j}\in\mathbb{R}^{M\times
N}.$ (5)

This can be interpreted as sum of modal contributions with the time
coefficients determined by

$\mathbf{A}=\mathbf{X}\mathbf{W}\mathbf{\Phi}\in\mathbb{C}^{M\times N}.$ (6)

###  2.2 Spectral Proper Orthogonal Decomposition (SPOD)

In this section, we briefly review the implementation of the SPOD algorithm
based on Welch’s periodogram method, which is used in this work. For a
detailed description of the theoretical background, we refer the reader to
Towne, Schmidt and collaborators Towne2018 ; Schmidt2020 , as well as to
Mengaldo & Maulik Mengaldo2021  for the implementation.

Just as done for the POD, we assemble the matrix of snapshots $\mathbf{X}$
(1). Next, we apply Welch’s spectral estimation method by decomposing the
matrix along its time axis into $N_{\mathrm{blk}}$ overlapping blocks and,
assuming that each realisation in the snapshot matrix occurs periodically in
time, apply a discrete Fourier transform on each block after multiplication of
the data by a Hamming window. The result is a set of $N_{\mathrm{blk}}$
Fourier transformed blocks $\widetilde{\mathbf{X}}_{k}$ with
$N_{\mathrm{fft}}$ frequencies each.

For each frequency, we assemble the matrix $\widetilde{\mathbf{X}}$

$\widetilde{\mathbf{X}}=\begin{pmatrix}\text{\\----}&\widetilde{\mathbf{x}}_{1}^{H}&\text{\\----}\\\
&\vdots\\\
\text{\\----}&\widetilde{\mathbf{x}}_{N_{\mathrm{blk}}}^{H}&\text{\\----}\\\
\end{pmatrix}\in\mathbb{C}^{N_{\mathrm{blk}}\times N},$ (7)

where, again, $N$ is the number of spatial points times the number of
variables and $N_{\mathrm{blk}}$ is the number of blocks. For each frequency,
we compute the covariance matrix

$\widetilde{\mathbf{C}}=\frac{1}{N_{\mathrm{blk}}}\widetilde{\mathbf{X}}^{H}\widetilde{\mathbf{X}}\in\mathbb{C}^{N\times
N},$ (8)

and obtain the SPOD modes by solving the eigenvalue decomposition of the
covariance matrix for each frequency separately Schmidt2020 :

$\widetilde{\mathbf{C}}\mathbf{W}\mathbf{\Phi}=\mathbf{\Lambda}\mathbf{\Phi},\quad\mathbf{W}=\operatorname{diag}\left(w_{1},\dots,w_{N}\right).$
(9)

Here, $\mathbf{W}$ is a weight matrix, $\mathbf{\Phi}$ are the eigenvectors
and $\mathbf{\Lambda}$ the eigenvalues. Because the weight matrix is problem
specific, we will discuss its form for each application separately.

##  3 Application to turbomachinery flows

In the following, we will apply both POD and SPOD to three different datasets
with different flow phenomena. We will begin by investigating the transitional
flow through the MTU T161 low-pressure turbine (LPT) with focus on separation-
induced transition around midspan and large-scale unsteadiness of the
secondary flow system. Additionally, we will analyse the influence of wakes
from an upstream row. Finally, we will study shock wave boundary layer
interaction in a linear compressor cascade.

###  3.1 Separation-induced transition and secondary flow effects

Figure 1: Illustration of MTU T161 simulation domain with instantaneous vortex
structures and mean in-plane wall shear stress $\tau_{xy}$ on the blade
suction side with $\tau_{xy}=0$ as black contour line

We analyse the turbulent flow through an LPT cascade at an off-design Reynolds
number of $90\,000$ and a Mach number of 0.65 computed as implicit LES with a
high-order DGSEM scheme Morsbach2023 . We use a dataset containing the time-
resolved wall shear stress vector on the blade suction side spanning 100
convective time units $t_{\mathrm{c}}$ based on chord length and outlet
velocity with 3668 samples. The original sampling was 366 samples per
$t_{\mathrm{c}}$ but we use only a tenth of that for the modal analysis, since
that already covers the frequency range of interest. Fig. 1 shows the
configuration with a view of the blade suction side of the configuration with
instantaneous vortex structures visualised by the Q-criterion to illustrate
the separation induced transition around midspan and the secondary flow
regions close to the hub and tip walls. On the right-hand side, the mean in-
plane wall shear stress $\tau_{xy}$ on the blade suction side surface over the
arc length from the leading edge (LE) at $s=0$ to the TE at
$s/s_{\mathrm{max}}=1$ is presented in the way we will display the mode shapes
throughout this section. The $z$ coordinate is normalised by the upstream
channel height $H$ allowing to see the diverging end walls. We will employ
classical Fourier analysis, space-only POD and SPOD to investigate a potential
connection between the Kelvin-Helmholtz (KH) based transition mechanism around
midspan and oscillations of the corner flow.

Figure 2: Midspan premultiplied spectrum of $u^{\prime}_{\|}$ for different
distances $\eta$ from the solid wall in boundary layer cuts of the MTU T161
blade suction side at $x/C_{\mathrm{ax}}=0.83$ (pre-transitional),
$x/C_{\mathrm{ax}}=0.87$ (transition onset) and $x/C_{\mathrm{ax}}=0.90$ (half
way to fully turbulent)

In addition to the surface dataset, we use three wall-normal cuts through the
suction side boundary layer at midspan to be able to relate frequencies to
physical mechanisms in the flow. Fig. 2 shows premultiplied frequency spectra
of the wall-parallel fluctuating velocity $u^{\prime}_{\|}$ for these cuts
varying with the distance from the solid wall. For orientation, the mean
velocity profiles are plotted as black dashed lines. In the cut at
$x/C_{\mathrm{ax}}=0.83$, where the separated shear layer is still laminar,
only non-dimensional frequencies lower than 1 (time scales larger than a
convective time unit) show noticeable content in the region of strongest
shearing. Further downstream at $x/C_{\mathrm{ax}}=0.87$, where the flow
begins its transition to turbulence, non-dimensional frequencies in the range
3-5 become excited due to the KH instability and have already surpassed the
intensity found in the lower frequencies. These instabilities can be found to
grow even further at $x/C_{\mathrm{ax}}=0.90$, where the maximum turbulent
kinetic energy (TKE) in the boundary layer is at 50% of the maximum, reached
slightly downstream. At this position, they dominate the whole spectrum and
peaks at these frequencies can not only be found in the shear layer but also
close to the solid wall. Observe the changed scale of the colourbar. We will
find both the low frequency and high frequency effects in the modal analysis
below.

t]

Figure 3: POD eigenvalues of MTU T161 suction side wall shear stress (top) and
cumulative eigenvalues (bottom); values for Gaussian random data given as
indication of noise threshold

Since our dataset has points clustered towards the leading and trailing edge,
as well as towards the end walls, we use the element area as weights to
perform a POD using an SVD with (14), (17) and (18). The input variables are
the two independent components of the fluctuating wall shear stress vector
$\tau_{xy}^{\prime}$ and $\tau_{z}^{\prime}$, equally weighted using the
weighted 2-norm Schmidt2020 :

$\mathbf{X}=\left[\tau_{xy}^{\prime},\;\tau_{z}^{\prime}\right]^{\mathrm{T}},\quad\mathbf{W}=\int_{V}\operatorname{diag}\left(\mathrm{d}V,\;\mathrm{d}V\right).$
(10)

The resulting eigenvalues (top) and the cumulative energy contained in the
first $r$ modes (bottom) are shown in Fig. 3. It can be instantly noted that,
in contrast to many textbook examples of POD, there is no large-scale
organisation in the flow field responsible for a great portion the fluctuating
energy. On the contrary, the most energetic mode only accounts for 0.14% while
the tenth mode still accounts for 0.11%. Yet, the modes stand out above the
noise threshold and the mode shapes offer some insights into the flow physics.

Figure 4: MTU T161: First eight POD modes for in-plane (top) and spanwise
(bottom) wall shear stress $\tau_{\mathrm{xy}},\tau_{\mathrm{z}}$

Fig. 4 shows the first eight POD modes. Note that the modes are orthonormal by
definition, so the corresponding eigenvalues are given in the title to
indicate their relative strengths. The first two modes exhibit structures
close to the end walls while the area around midspan shows relatively low
amplitudes and rather noisy data. Mode 1 has larger amplitudes at the tip wall
(positive $z$) in both components of the wall shear stress while mode 2 shows
a very similar structure basically mirrored at the $z=0$ plane. This
corresponds to a periodic displacement of flow from the end walls into the
channel and can be associated with an oscillation of the passage vortex (PV).
The third mode is the first one to be dominated by structures associated with
the KH instability in the separated shear layer around midspan. It is notable
that the large amplitudes are only found close to midspan and are perfectly
aligned with the $z$ axis. Larger structures close to the end walls can still
be seen here, yet at a lower amplitude compared to modes 1 and 2. A spanwise
variation of the KH structures can be found in the following modes be it
either in the form of a tilt (mode 4, 5, 8) or an inversion of the phase at
some spanwise location (mode 6 and 7). All the modes featuring KH structures
still show some amplitude at the end walls suggesting that these phenomena
might be connected.

(a) Premultiplied PSD of POD time coefficients (b) Premultiplied SPOD
eigenvalues

Figure 5: Spectra for the MTU T161 without incoming wakes

A look at the POD time coefficients, especially at their spectral content, can
reveal additional information on the modes. Fig. 5 (a) shows the premultiplied
Power Spectral Density (PSD) of the time coefficients for the first eight
modes, obtained via (6), computed with Welch’s method with 256 samples per
segment. Two distinct features can be identified: a low-frequency peak around
0.5 and a high frequency peak around 3. The first two modes, which do not show
any sign of KH structures, clearly peak at the low frequency but show no
contribution in the high frequency range. On the other hand, all modes with KH
structures have the most significant contribution to the spectrum in this high
frequency peak. While most of these modes do not show contributions above the
noise level in the low frequency range, mode 3 is an exception: it features a
low frequency peak in the same order of magnitude as modes 1 and 2. This
suggests some connection of the oscillation of the secondary flow system with
the KH shedding through this mode shape.

Figure 6: MTU T161: First 4 SPOD modes for in-plane (top) and spanwise
(bottom) wall shear stress $\tau_{\mathrm{xy}},\tau_{\mathrm{z}}$ at
$ft_{\mathrm{c}}=0.57$ and $3.71$

At this point, it can be helpful to resort to SPOD to be able to obtain
optimal spatial modes corresponding to a single frequency. We use exactly the
same dataset with weights chosen to represent a weighted 2-norm Schmidt2020 ,
a bin size of 256 and an overlap of 50%. Fig. 5 (b) shows the premultiplied
SPOD spectrum with the area between mode 1 and 4 shaded in red to indicate the
energy contained within the first three modes. The spectrum peaks in the same
high frequency range as the POD time coefficient spectra discussed above. As
expected from the above discussion of the POD, the separation between the
first and second mode is not as clear as in many textbook examples.
Nevertheless, the first three to four modes stand out in terms of separation
between the modes and suggest that some kind of low-rank behaviour can
possibly be identified He2021 . We, therefore, examine the first four modes at
the frequencies 0.57 and 3.72 in Fig. 6. The choice of former is due to the
subtle low-frequency peak while the latter is roughly the maximum
premultiplied eigenvalue over all frequencies. Note, that slight changes to
the investigated frequency do not qualitatively change the mode shapes in this
case.

The first four columns show the first four modes of $ft_{\mathrm{c}}=0.57$. To
present an indication for the energy of the respective mode, the mode
eigenvalue $\lambda_{\mathrm{rel}}$ normalised with the sum over all
eigenvalues at that frequency is given in the column header. The first two
SPOD modes at $ft_{\mathrm{c}}=0.57$ show a great resemblance with the first
two POD modes, which is a consistent picture since the frequency analysis of
the latter revealed that most of their energy is found at this low frequency.
Looking at the higher modes, there is again some indication that the
oscillation of the secondary flow system has an effect on the midspan flow,
although the data in the separation bubble is rather noisy. In contrast to our
argument above using the POD analysis, however, we see no direct connection
with KH structures but a larger scale oscillation in the third mode. KH
structures in the laminar separation bubble (LSB), on the other hand can be
clearly identified in the in-plane wall shear stress for all modes at
$ft_{\mathrm{c}}=3.71$. Similar to what was found above in the higher POD
modes, the first mode at this frequency shows a vertical alignment of the KH
structures while the higher modes introduce some tilting. The spanwise wall
shear stress shows mainly turbulent noise. An interesting difference to the
POD concerns the connection of the KH structures and the oscillation of the
secondary flow system: while the POD suggested some combined mechanism through
mode 3, the SPOD suggests that the two phenomena are, indeed, not directly
coupled as no significant structures in the secondary flow region can be found
at $ft_{\mathrm{c}}=3.71$. This underlines the importance of SPOD to avoid
drawing false conclusions due to the missing frequency resolution of POD. It
has to be mentioned, though, that the SPOD modes are rather noisy and for a
more quantitative analysis, more samples in excess of the available 100
convective time units would be beneficial. Very recently, a sub-sampling
method was suggested to overcome the problems associated with short time
signals by sacrificing the high-frequency part of the spectrum to be able to
obtain more realisations of low-frequency phenomena Schneider2023 .

###  3.2 Blade row interaction

Besides the clean inflow configuration with freestream turbulence only, a
dataset with incoming periodic disturbances generated by cylindrical wake
generators was presented in Morsbach2023 . POD was only used to obtain less
noisy phase averages in the wake plane in this paper. In the following, we
perform an SPOD analysis consistent with the one above. The dataset consists
of 3009 samples, spanning 82 convective time units corresponding to 50 bar
passes. Again, we use only every tenth sample of the original dataset sampled
at 368 per $t_{\mathrm{c}}$. The periodic wakes lead to an intermittently
separated flow around midspan, which does not show any backflow on average
Morsbach2023 .

t]

Figure 7: MTU T161: SPOD spectrum for the case with incoming wakes;
frequencies for mode analysis marked with dashed lines

The SPOD spectrum in Fig. 7 is dominated by the wake passing frequency
$ft_{\mathrm{c}}=0.61$ and its harmonics. Coincidentally, this frequency is
very close to the one of the large-scale secondary flow oscillations found in
the case without wakes. The higher harmonics are in a frequency range
corresponding to the KH shedding. In the high frequency range
$ft_{\mathrm{c}}>5$, another set of peaks standing out of the turbulent noise
can be found which does not occur in the case without wakes. In the following,
we will analyse the first modes at the frequencies marked with dashed lines.
An inspection of the spectrum and the second modes revealed that these do not
generally show larger scale structures with a significant signal to noise
ratio.

Figure 8: MTU T161: First SPOD modes for in-plane (top) and spanwise (bottom)
wall shear stress $\tau_{\mathrm{xy}},\tau_{\mathrm{z}}$ at dominant
frequencies

Fig. 8 shows these modes, again along with their relative eigenvalue
normalised with the sum of all eigenvalues at that frequency. Note that the
ratio of the eigenvalues between different frequencies cannot be derived from
these but must be obtained from Fig. 7. The first mode at
$ft_{\mathrm{c}}=0.58$ is clearly associated with the large-scale wake passing
and is responsible for most of the energy at that frequency with mode 2 at
$\lambda_{\mathrm{rel}}=1.26\%$ only (not shown in the figure). Note that due
to the choice of blocks and the time step size, the available frequencies in
the SPOD only approximately match the bar passing frequency and its harmonics.
The large structures in the in-plane wall shear stress describe the periodic
separation and reattachment, although this conclusion can only be drawn in
combination with knowledge about the local mean values. In contrast to the
case without wakes, the oscillation of the corner separation appears to be
dominated by the wake passing as seen in large coherent structures in the both
wall shear stress components. The first three harmonics quickly decrease in
eigenvalue and show refined structures corresponding to the same phenomenon.
At frequencies in the range of the KH instability of the case without wakes,
i.e. $ft_{\mathrm{c}}$ around 3, the characteristic vertical bands appear in
the aft section of the blade. Again, from this frequency on, at best, a very
subtle interaction with the secondary flow region can be identified.

When the frequency is further increased, another set of vertical structures
with shorter wavelength can be found upstream, starting near the LE at
$ft_{\mathrm{c}}=12.66$. The instantaneous flow field in a midspan plane (not
shown) revealed, that these are the footprints of upstream traveling acoustic
waves originating from the strong vortex shedding of the moving cylinder and
their reflections from the adjacent blade pressure side. The high frequency
structures near the LE could only be identified using the SPOD and cannot be
observed in any of the first 30 POD modes. This, again, highlights the
usefulness of the SPOD in distinguishing between different features of the
flow which occur at different frequencies and are possibly hidden within high-
energy low-frequency behaviour.

###  3.3 Shock wave boundary layer interaction

In this section, we analyse the modal content of the flow over a compressor
cascade at transonic operating condition. The flow is simulated using a 4
order accurate LES based on a high-order DGSEM scheme. For a detailed
description of the scheme, we refer to Bergmann et al.Bergmann2023  and
references therein. The linear cascade is computed based on a spanwise domain
extrusion of 5% chord length, where periodic boundary conditions are applied
along the pitchwise and the spanwise faces to approximate the flow over an
infinite cascade of infinite blades. The computational domain consists of
$2\,276\,829$ hexahedral elements clustered around the blade and the shock
location, adding to a total 145.7 million degrees of freedom per equation. The
simulation is evaluated over 21 convective time units ($t_{\mathrm{c}}$) based
on the chord length $c$ and inflow velocity $U_{0}$ and sampled at a non-
dimensional frequency of 28 samples per time unit. We refer the interested
reader to Klose et al. Klose2023  for a more detailed description of the setup
and discussion of the flow field for this testcase.

Given that the compressor cascade is at transonic operating conditions, the
SPOD is computed based on the compressible energy norm Schmidt2020  with input
vector

$\mathbf{X}=\left[\rho^{\prime},\;u_{\parallel}^{\prime},\;u_{\perp}^{\prime},\;u_{z}^{\prime},\;T^{\prime}\right]^{\mathrm{T}},$
(11)

and weights

$\mathbf{W}=\int_{V}\operatorname{diag}\left(\frac{\overline{T}}{\gamma\overline{\rho}M_{0}^{2}},\;\overline{\rho},\;\overline{\rho},\;\overline{\rho},\;\frac{\overline{\rho}}{\gamma(\gamma-1)\overline{T}M_{0}^{2}}\right)\mathrm{d}V,$
(12)

where $\rho$ is the density, $T$ the temperature, $u_{i}$ the velocity
components and $M$ the Mach number. All physical quantities here are non-
dimensionalised with respect to reference quantities at the inlet measurement
plane (indicated by subscript $0$). A total of 568 snapshots are evaluated for
the modal analysis for this testcase, where the bin size for the SPOD is 142
snapshots with 50% overlap.

(a) Density contours (b) PSD along boundary layer probe

Figure 9: DLR compressor cascade: (a) instantaneous density contours with
boundary layer profile indicator in black; (b) contours of the frequency
spectrum of $u_{\parallel}$ along the boundary layer with mean velocity and
TKE profiles

The flow is characterized by an upstream laminar boundary layer, shock-induced
separation and transition to turbulence. Features of the instantaneous flow
field are shown in Fig. 9 (a), where contours of the density are shown at the
mid-chord section of the blade around the shockwave and the premultiplied
frequency spectrum of the wall-parallel velocity component $u_{\parallel}$
along a wall-normal probe is given in Fig. 9 (b). Additionally, the mean
velocity and TKE profiles are indicated, as well as the peak in the spectrum
at $ft_{\mathrm{c}}$ = 0.75. As shown by Klose et al. Klose2023 , low-
amplitude oscillations of the normal shock occur in the range
$0.6<ft_{\mathrm{c}}<0.8$, as well as high-frequency fluctuations at
$ft_{\mathrm{c}}$ = 2.6, which were associated with a wake mode.

(a) Eigenvalue spectrum of $q$ (b) Relative contributions

Figure 10: DLR compressor cascade: (a) eigenvalue spectrum of
$q=[\rho,u_{\parallel},u_{\perp},u_{z},T]$; (b) relative contributions to the
first mode of the individual components of $q$

The eigenvalue spectrum in Fig. 10(a) is obtained by computing the SPOD of the
input vector (11) and the weights (12) as a whole, where the shaded area
highlights the separation between the first and second SPOD modes. A
distinctly separated peak occurs at $ft_{\mathrm{c}}$ = 0.77 and a series of
separated peaks at around $ft_{\mathrm{c}}$ = 2.52, closely matching the
frequency peak along the boundary layer probe. To extract the contributions of
each term to the overall fluctuating energy, the SPOD eigenvalues of the first
mode for the individual components (with their respective weights) are shown
in Fig. 10(b), together with the result of the _full_ input vector. While the
wall-parallel velocity component $u_{\parallel}^{\prime}$ dominates the energy
in the lower frequency regime, the wall-normal component $u_{\perp}^{\prime}$
is the main contributor for the higher frequency peaks. The components
measuring the compressibility effects ($\rho^{\prime}$, $T^{\prime}$) also
predominantly contribute energy to the low-frequency modes, where they still
only make up around 15% of the total energy.

Figure 11: DLR compressor cascade: First SPOD modes of the density (left),
blade-parallel velocity (middle) and blade-normal velocity (right)

The shape of the first SPOD mode for $ft_{\mathrm{c}}$ = 0.77 and
$ft_{\mathrm{c}}$ = 2.52 for $\rho^{\prime}$, $u_{\parallel}^{\prime}$ and
$u_{\perp}^{\prime}$ is given in Fig. 11. At $ft_{\mathrm{c}}$ = 0.77, the
compression area upstream of the normal shock is distinctly highlighted by the
density, where the thickening of the boundary layer is accompanied by a mild
oblique compression wave. The movement of the normal shock itself is also
tracked by the $u_{\parallel}^{\prime}$ velocity component and shows as a
distinct, sharp mode structure normal to the blade surface. At the higher
frequency of $ft_{\mathrm{c}}$ = 2.52, the shock wave is no longer discernible
in the velocity components. Here, $u_{\perp}^{\prime}$ is dominant and its
mode shape proves that the oscillating energy is almost exclusively contained
in the wake.

##  4 Conclusion

We have presented the application of two modal decomposition techniques,
namely POD and SPOD, to previously published LES datasets of turbomachinery
flows computed with our high-order DGSEM solver TRACE. While POD could be used
to draw first conclusions even when large-scale effects did not dominate the
turbulent spectrum in energy, the frequency resolution capabilities of SPOD
allowed for a clearer analysis und separation of effects. In the LPT flow
without incoming wakes, the weak but dominant feature was a low-frequency
oscillation of the secondary flow system at time scales longer than a
convective time unit. The SPOD showed that these oscillations were coupled
with a large-scale motion of the LSB. We found, that the KH instability,
occurring at higher frequencies was easy to detect and did not directly
interact with the secondary flow system. An SPOD of the flow through the LPT
with incoming wakes revealed the wakes’ influence on the blade including the
secondary flow system while the method was also able to detect the local KH
mechanism in the intermittent separation bubble. It also revealed the
footprint of very high frequency acoustic perturbations, which originate from
the vortex shedding of the upstream moving cylinder. In this case, ”the
frequency decomposition benefitted the analysis to reveal low-energy effects
otherwise obscured by the dominant large-scale motion. Applied to the flow
around a transonic compressor cascade, SPOD was able to discern two effects:
first, low-frequency oscillations are associated with shock-wave boundary
layer interaction and show higher contributions of compressible components to
the total energy. Second, high-frequency oscillations are associated with
vortex shedding in the wake in which compressibility does not play a
significant role. It has to be noted, that many statements made in this paper
depended on a combined analysis of modal decomposition techniques with other
methods such as averages, frequency analysis, space-time diagrams or
visualisation of the instantaneous flow field. In this respect, modal
decomposition techniques are a valuable addition in the toolbox of unsteady
flow analysis in turbomachinery flows.

## References

  * (1) Alessandri, A., Bagnerini, P., Gaggero, M., Lengani, D., Simoni, D.: Dynamic mode decomposition for the inspection of three-regime separated transitional boundary layers using a least squares method.  Physics of Fluids 31(4), 044103 (2019).  DOI 10.1063/1.5086225
  * (2) The American Society of Mechanical Engineers: Unsteady Flow Interactions Between a High- and Low-Pressure Turbine: Part 2 — Rotor-Synchronic Averaging and Proper Orthogonal Decomposition of the Unsteady Flow Fields, _Turbo Expo: Power for Land, Sea, and Air_ , vol. Volume 2A: Turbomachinery (2019).  DOI 10.1115/GT2019-90824
  * (3) The American Society of Mechanical Engineers: Proper Orthogonal Decomposition Analysis of Particle Image Velocimetry Data at the Inlet of a Centrifugal Compressor, _ASME International Mechanical Engineering Congress and Exposition_ , vol. Volume 8: Fluids Engineering; Heat Transfer and Thermal Engineering (2022).  DOI 10.1115/IMECE2022-94308
  * (4) Bergmann, M., Morsbach, C., Klose, B.F., Ashcroft, G., Kügeler, E.: A numerical test rig for turbomachinery flows based on large eddy simulations with a high-order discontinuous Galerkin scheme - part 1: Sliding interfaces and unsteady row interactions.  In: Proceedings of ASME Turbo Expo 2023: Turbomachinery Technical Conference and Exposition, GT2023-101623. Boston, MA, USA (2023)
  * (5) Berkooz, G., Holmes, P., Lumley, J.L.: The proper orthogonal decomposition in the analysis of turbulent flows.  Annual Review of Fluid Mechanics 25(1), 539–575 (1993).  DOI 10.1146/annurev.fl.25.010193.002543
  * (6) Cizmas, P.G., Palacios, A.: Proper orthogonal decomposition of turbine rotor-stator interaction.  Journal of propulsion and power 19(2), 268–281 (2003)
  * (7) Dotto, A., Lengani, D., Simoni, D., Tacchella, A.: Dynamic mode decomposition and Koopman spectral analysis of boundary layer separation-induced transition.  Physics of Fluids 33(10), 104104 (2021).  DOI 10.1063/5.0065554
  * (8) Feldhusen-Hoffmann, A., Lagemann, C., Loosen, S., Meysonnat, P., Klaas, M., Schröder, W.: Analysis of transonic buffet using dynamic mode decomposition.  Experiments in Fluids 62(4), 66 (2021).  DOI 10.1007/s00348-020-03111-5
  * (9) Fiore, M., Gojon, R., Sáez-Mischlich, G., Gressier, J.: Les of the t106 low-pressure turbine: Spectral proper orthogonal decomposition of the flow based on a fluctuating energy norm.  Computers & Fluids 252, 105761 (2023).  DOI 10.1016/j.compfluid.2022.105761
  * (10) He, X., Fang, Z., Rigas, G., Vahdati, M.: Spectral proper orthogonal decomposition of compressor tip leakage flow.  Physics of Fluids 33(10), 105105 (2021).  DOI 10.1063/5.0065929
  * (11) Klose, B.F., Morsbach, C., Bergmann, M., Hergt, A., Klinner, J., Grund, S., Kügeler, E.: A numerical test rig for turbomachinery flows based on large eddy simulations with a high-order discontinuous Galerkin scheme - part 2: Shock-capturing and transonic flows.  In: Proceedings of ASME Turbo Expo 2023: Turbomachinery Technical Conference and Exposition, GT2023-101374. Boston, MA, USA (2023)
  * (12) Lengani, D., Simoni, D., Ubaldi, M., Zunino, P., Bertini, F.: Experimental Investigation on the Time–Space Evolution of a Laminar Separation Bubble by Proper Orthogonal Decomposition and Dynamic Mode Decomposition.  Journal of Turbomachinery 139(3), 031006 (2016).  DOI 10.1115/1.4034917
  * (13) Mengaldo, G., Maulik, R.: PySPOD: A Python package for spectral proper orthogonal decomposition (SPOD).  Journal of Open Source Software 6(60), 2862 (2021).  DOI 10.21105/joss.02862
  * (14) Morsbach, C., Bergmann, M., Tosun, A., Klose, B.F., Bechlars, P., Kügeler, E.: A numerical test rig for turbomachinery flows based on large eddy simulations with a high-order discontinuous Galerkin scheme - part 3: Secondary flow effects.  In: Proceedings of ASME Turbo Expo 2023: Turbomachinery Technical Conference and Exposition, GT2023-101374. Boston, MA, USA (2023)
  * (15) Schmid, P.J.: Dynamic mode decomposition of numerical and experimental data.  Journal of Fluid Mechanics 656, 5–28 (2010).  DOI 10.1017/S0022112010001217
  * (16) Schmidt, O.T., Colonius, T.: Guide to spectral proper orthogonal decomposition.  AIAA Journal 58(3), 1023–1033 (2020).  DOI 10.2514/1.J058809
  * (17) Schneider, N., Köhler, S., von Wolfersdorf, J.: Experimental detection of organised motion in complex flows with modified spectral proper orthogonal decomposition.  Fluids 8(6), 184 (2023).  DOI 10.3390/fluids8060184
  * (18) Towne, A., Schmidt, O.T., Colonius, T.: Spectral proper orthogonal decomposition and its relationship to dynamic mode decomposition and resolvent analysis.  Journal of Fluid Mechanics 847, 821–867 (2018).  DOI 10.1017/jfm.2018.283
  * (19) Wang, R., He, X., Yan, X.: Spectral proper orthogonal decomposition analysis of trailing edge cutback film cooling flow.  Physics of Fluids 34(10), 105106 (2022).  DOI 10.1063/5.0098796
  * (20) Weiss, J.: A tutorial on the proper orthogonal decomposition.  In: AIAA Aviation 2019 Forum. Dallas, Texas, USA (2019).  DOI 10.2514/6.2019-3333

## Appendix

In the following, we will re-arrange (3) so that we can determine the POD
modes and eigenvalues using an SVD of a weighted snapshot matrix. Inserting
(2) into (3) yields

$\left(\frac{\mathbf{X}\sqrt{\mathbf{W}}}{\sqrt{M-1}}\right)^{H}\frac{\mathbf{X}\sqrt{\mathbf{W}}}{\sqrt{M-1}}\mathbf{\Phi}=\mathbf{\Phi}\mathbf{\Lambda}$
(13)

with
$\sqrt{\mathbf{W}}=\operatorname{diag}\left(\sqrt{w_{1}},\dots,\sqrt{w_{N}}\right)$.
The weighted snapshot matrix can be written as compact SVD

$\frac{\mathbf{X}\sqrt{\mathbf{W}}}{\sqrt{M-1}}=\mathbf{U}\mathbf{\Sigma}\mathbf{V}^{H},\quad\mathbf{U}\in\mathbb{C}^{R\times
R},\,\mathbf{\Sigma}\in\mathbb{C}^{R\times
R},\,\mathbf{V}\in\mathbb{C}^{N\times R}$ (14)

with $R=\min(M,N)$ and
$\mathbf{U}^{H}\mathbf{U}=\mathbf{V}^{H}\mathbf{V}=\mathbf{1}$. This yields

$\mathbf{V}\mathbf{\Sigma}^{H}\mathbf{\Sigma}\mathbf{V}^{H}\mathbf{\Phi}=\mathbf{\Phi}\mathbf{\Lambda}.$
(15)

With (4), it can be written as

$\mathbf{V}\mathbf{\Sigma}^{H}\mathbf{\Sigma}\mathbf{V}^{H}=\sqrt{\mathbf{W}}\mathbf{\Phi}\mathbf{\Lambda}^{H}\left(\sqrt{\mathbf{W}}\mathbf{\Phi}\right)^{H}.$
(16)

So the eigenvalues $\mathbf{\Lambda}$ and modes $\mathbf{\Phi}$ are obtained
from the weighted snapshot SVD by

$\displaystyle\mathbf{\Lambda}=\mathbf{\Lambda}^{H}$
$\displaystyle=\mathbf{\Sigma}^{H}\mathbf{\Sigma}$ (17)
$\displaystyle\mathbf{\Phi}$ $\displaystyle=\sqrt{\mathbf{W}^{-1}}\mathbf{V}$
(18)

with
$\sqrt{\mathbf{W}^{-1}}=\operatorname{diag}\left(\sqrt{w_{1}^{-1}},\dots,\sqrt{w_{N}^{-1}}\right)$.

  *[CFD]: Computational Fluid Dynamics
  *[LES]: Large Eddy Simulation
  *[POD]: Proper Orthogonal Decomposition
  *[SPOD]: Spectral Proper Orthogonal Decomposition
  *[DMD]: Dynamic Mode Decomposition
  *[PIV]: Particle Image Velocimetry
  *[RANS]: Reynolds-averaged Navier-Stokes
  *[TE]: trailing edge
  *[DDES]: Delayed Detached Eddy Simulation
  *[DGSEM]: Discontinuous Galerkin Spectral Element Method
  *[SVD]: Singular Value Decomposition
  *[LPT]: low-pressure turbine
  *[LE]: leading edge
  *[KH]: Kelvin-Helmholtz
  *[TKE]: turbulent kinetic energy
  *[PV]: passage vortex
  *[PSD]: Power Spectral Density
  *[LSB]: laminar separation bubble


Department of Computer Science and Engineering (DISI), University of Bologna,
Italy and https://www.unibo.it/sitoweb/delfinasol.martinez2/en
delfinasol.martinez2@unibo.ithttps://orcid.org/0000-0003-2392-6300Department
of Modern Languages, Literatures and Cultures (LILEC), University of Bologna,
Italyvalentina.presutti@unibo.it[http://orcid.org/0000-0002-9380-5160] Delfina
Sol Martinez Pandiani and Valentina Presutti [500]Information systems Content
analysis and feature selection [500]Information systems Ontologies
[500]Information systems Digital libraries and archives [500]Information
systems Multimedia information systems This project has received funding from
the European Union’s Horizon 2020 research and innovation programme under
grant agreement No 101004746. 2 First International Workshop on Multisensory
Data and Knowledge at the 3rd Conference on Language, Data and Knowledge
(2021) MDK 2021 MDK 2021 September 1–3, 2021 Zaragoza, Spain

# Automatic Modeling of Social Concepts Evoked by Art Images as Multimodal
Frames

Delfina Sol Martinez Pandiani111Corresponding author. Valentina Presutti

###### Abstract

Social concepts referring to non-physical objects–such as _revolution_ ,
_violence_ , or _friendship_ –are powerful tools to describe, index, and query
the content of visual data, including ever-growing collections of art images
from the Cultural Heritage (CH) field. While much progress has been made
towards complete image understanding in computer vision, automatic detection
of social concepts evoked by images is still a challenge. This is partly due
to the well-known semantic gap problem, worsened for social concepts given
their lack of unique physical features, and reliance on more unspecific
features than concrete concepts. In this paper, we propose the translation of
recent cognitive theories about social concept representation into a software
approach to represent them as multimodal frames, by integrating multisensory
data. Our method focuses on the extraction, analysis, and integration of
multimodal features from visual art material tagged with the concepts of
interest. We define a conceptual model and present a novel ontology for
formally representing social concepts as multimodal frames. Taking the Tate
Gallery’s collection as an empirical basis, we experiment our method on a
corpus of art images to provide a proof of concept of its potential. We
discuss further directions of research, and provide all software, data
sources, and results.

###### keywords:

Knowledge Engineering, Multisensory Data, Abstract Concepts, Image
Understanding

###### category:

##  1 Introduction and Problem Statement

Rapidly expanding image collections, including ever-growing art catalogues
from the Cultural Heritage (CH) field, have attracted significant research
efforts for automatic retrieval and management of visual data. Social
concepts–such as _horror_ or _consumerism_ –are a powerful resource for
performing these tasks, especially of art images. This is because visual forms
such as paintings and photographs are thought to illustrate, and circulate,
concepts both by providing links to depicted objects through raw features–such
as lines, color, shape, and size– as well as through what Barthes called an
image’s ‘connotation’: a second layer of meaning made from culturally coded
elements [6]. For example, seeing Artemisia Gentileschi’s _Judith Beheading
Holofernes_ (1620 ca.), a human observer is able to detect objects such as a
sword, but a comprehensive understanding of the painting would generally
include a social concept such as _violence_.

In this work, we aim to automatically model social concepts as multimodal
frames based on features of art images that evoke them. We are interested in
_social concepts_ in the ontological sense expounded by [16]–immaterial
products whose conventional constitution involves a network of relations among
social agents–which refer to _non-physical_ objects. The concepts of interest
overlap with those which the field of cognitive neuroscience has labeled as
‘abstract’ concepts, complex situations that do not possess a single and
perceptually bounded object as referent [7]. For the rest of this paper, the
term ‘social concepts’ is used to allude specifically to this category of
social concepts lacking unique, clearly perceivable referents.

The automatic modeling of social concepts based on perceptual features of art
images could lead to breakthroughs in a wide range of applications, including
Web image search, online picture-sharing communities, and scientific and
academic endeavors, by serving as input to models that improve multimedia
querying of digital libraries. Furthermore, it could improve automatic
detection of the potential of images to communicate social concepts, a useful
resource for CH institutions to enhance their visual collections’
documentation, descriptions, and mediation. Additionally, it could aid public
institutions in building narratives about their visual objects, companies of
the creative sector in exploiting their already existing catalogues, and
companies building products or services related to specific social concepts.

While much progress has been made towards the general goal of complete image
understanding in computer vision, including outstanding performance for tasks
like image classification, object detection, and image generation [5], image
understanding methods are not yet successful at detecting social concepts.
This is partly because a main focus of computer vision has been image
segmentation, while these concepts lack unique and distinctive physical
features, such as defined form, color and texture. Instead, they tend to be
more schematic in nature, with a critical parasitic relation between sensory-
perceptual and distributional linguistic data given their _acquired
embodiment_ [10]. While there are multiple works focused on integrating
knowledge in image understanding [2], no frameworks have been developed to
automatically model social concepts based on extraction and integration of
sensory-perceptual data, such as pervasive visual features of images which
evoke them, with distributional linguistic patterns of social concept usage.

As such, our ultimate goal is to translate recent cognitive theories about
concept representation into a software architecture that can automatically
model social concepts based on multimodal features. We hypothesize that a
formal representation of social concepts as multimodal frames can be
automatically produced with a pipeline combining knowledge engineering,
computer vision, and computational linguistics methods. In this paper, we
introduce our approach to generate these representations through the
extraction and integration of features from images to Knowledge Graphs (KG).
Our approach focuses on the extraction, analysis, and integration of
multimodal features (including depicted concrete objects, depicted actions,
and color features) from images tagged with social concepts. Taking the Tate
Gallery collection as an empirical basis, we present a study of extraction and
integration of multimodal data. The contributions of this paper can be
summarised as follows:

  * •

We define a conceptual model and present a novel, pattern-based ontology for
social concept representation, which allows for semantic characterization of
multimodal features.

  * •

We propose a novel approach for the extraction and integration of multimodal
features of images that evoke certain social concepts.

  * •

We implement the proposed method on an corpus of art images from a well-known
catalogue of art images (the Tate Gallery collection).

  * •

We discuss the results and provide all software, data sources, and results to
allow the reproducibility of our experiment.

After discussing related work in Section 2, Section 3 describes the Tate
Gallery Collection’s dataset, which was used as our experiments’ input source.
Section 4 illustrates our approach and its implementation. Section 5 reports
our experiments and results, while Section 6 focuses on the discussion of the
results and their limitations. We conclude with further directions in Section
7.

##  2 Related Work

Social concepts, in the scope of this work, refer to the immaterial products
defined in [16], where social concepts are ‘reified’ so as to be able to
predicate on them, and concept definitions are explicitly introduced as
descriptions that deal with the social, relational, and contextual nature of
social concepts. Cognitive neuroscience has labeled such concepts ‘abstract’:
with content more variable both within and across individuals, lacking a
single and perceptually bounded object as referent, and classifiable in
subtypes such as numerical, emotional, aesthetic, ethical, and moral concepts
[7]. These concepts have previously been characterized as linguistic frames
[17], a view mirrored in lexical resources such as
FrameNet.222https://framenet.icsi.berkeley.edu/fndrupal/ Recent cognitive
theories pose that, in addition to linguistic patterns, knowledge of abstract
words is acquired through _acquired embodiment_ , whereby they become
indirectly associated with the perceptual features of co-occurring concrete
words [13].

In the CH field, social concepts have been recognised as a resource to manage
visual collections, through the ample use of controlled thesauri such as the
Getty vocabularies333https://www.getty.edu/research/tools/vocabularies/ and
classifications systems like Iconclass444http://iconclass.org/ containing
ready-made social concepts to be associated as subject matters of visual
materials [18]. Given how challenging it can be to computationally interpret
culturally coded elements of visual materials, multidisciplinary frameworks
such as _distant viewing_ [4], are calling for the construction of explicit
code systems to automatically extract and interpret semantic elements of
visual materials via techniques from exploratory data analysis.

Automatic detection of social concepts, even in natural images, is scarce,
partly because almost all trainable classes in popular datasets used for image
classification, such as ImageNet555http://www.image-net.org/update-
mar-11-2021.php and Google Open
Images,666https://opensource.google/projects/open-images-dataset refer to
_concrete_ objects. A few works have attempted to train convolutional neural
networks to detect more abstract classes in natural images: in [3], one was
trained to learn and detect abstract concepts and emotions in images using
Flickr images and their tags, and in [20], another was trained to
automatically classify images of chessboards by the compositional elements
_identity_ and _symmetry_. Most related efforts have focused on classifying
images by evoked emotions through visual sentiment analysis [8]. The
multidisciplinary research field of affective computing sees the usage of
multimodal big data in the form of text, audio, visual and physiological
signals as an avenue for more robust systems [19]. Most recently, Google Arts
& Culture and the University of California, Berkeley, released an interactive
map of twenty-five emotions evoked by artworks based on the results of a user-
based study.777https://artsexperiments.withgoogle.com/art-emotions-map/

Computer-vision based tools have greatly benefited the CH sector by improving
access to implicit cultural knowledge contained in CH images, with novel
methods and technologies being used in the last few years to perform image
classification, object detection, caption generation, and more in art images
[11]. Various feature extraction techniques and classification algorithms are
available for 2D object recognition, including ones that use domain transfer
to perform object recognition in paintings and art images [9]. Furthermore,
there is an increasing number of techniques to train neural networks
specifically to process art images, such as using style transfer to generate
novel art image datasets from non artistic images, which are then used to
finetune image processing algorithms (eg., [14]).

Influenced by the idea that a cognitive understanding of our visual world
requires that we complement computers’ ability to detect objects with
abilities to understand relationships within a scene, graphs are increasingly
being employed to structure image information, with a growing number of
datasets of high quality image _scene graphs_ , such as Visual Genome [15],
regarding images’ depicted objects, their attributes, and their relationships
as first-class citizens of the annotation space. Additionally, because
associating a meaning with complex scenes may require an explicit and symbolic
representation of the domain of knowledge, ontologies have been shown to be a
powerful framework to address image analysis and interpretation [2], allowing
queries to be formulated in terms of concepts and their relationships.

##  3 Input Source: The Tate Gallery Collection

As an empirical basis for our study, we use the Tate Gallery’s collection
metadata of 70,000 artworks made available as a Github repository.888
https://github.com/tategallery/collection The Tate is an institution that
houses the United Kingdom’s national collection of British art, and
international modern and contemporary art. Most of the collection is from the
1800s, and a considerable part of it is from after 1960. In 2013, the
institution made their collection metadata available for about 70,000 artworks
that it owns or jointly owns with the National Galleries of Scotland, through
a Github repository. While the repository is no longer actively maintained,
the Tate keeps it available as a useful tool for researchers and developers,
and looks positively on creative remixing, visualization, and analysis of
their collection metadata.999See Eric Drass’ ”Tate Explorer” at
http://shardcore.org/tatedata/ and Florian Kräutli’s ”The Tate Collection on
Github” at http://research.kraeutli.com/index.php/2013/11/the-tate-collection-
on-github/ The dataset contains complete records of most artists and artworks
in the collection. It also includes image and thumbnail urls, but it does not
directly provide images, which still need to be accessed online. The dataset
can be accessed in two ways: either through CSV files containing information
about the artists and artworks, or in a series of thousands of text files
containing all the records in JSON format. The JSON data is much richer than
the CSV, storing a list of subjects associated with the record organised in a
subject taxonomy.

It was precisely because of its subject taxonomy that we selected the Tate
Gallery Collection as a first dataset to test our approach; the rich taxonomy
includes both concrete concepts and social concepts referring to non-physical
objects (“vacuum cleaner,” “shoe”, “consumerism”, “horror”) as subject tags.
As documented in an issue of their Github
repository,101010https://github.com/tategallery/collection/issues/27 the
Tate’s subject taxonomy is a “bespoke taxonomy”, originally developed
alongside the digitisation of Tate’s collection as a means of enabling
visitors to search artworks via subject. The design of the hierarchical
structure and initial tagging of the bulk of the collection was expert-led,
carried out by indexers with art history backgrounds and with the support of
Tate’s curatorial team, and in consultation with pre-existing systems such as
Iconclass and COLLAGE–the now unavailable public-access system for the
Guildhall Library and Art Gallery.

##  4 Approach

Our approach is based on the idea that a social concept is a complex object
whose definition can be formalized as a _description_ [16]. Specifically, our
assumption is that a social concept can be described with a multimodal frame,
whose meaning arises from an integration of linguistic and sensory-perceptual
features of content, such as images, that evoke that social concept. For
example, the meaning of the social concept “death” may be described via a
multimodal frame integrating properties of linguistically co-occurring terms
(e.g., “coffin”, “burial”, “accident”) and sensory-perceptual properties of
images depicting scenes that evoke that social concept (e.g., the color black
as a sensory-perceptual feature of a funeral scene, which evokes the social
concept “death”) [10]. We developed a framework (summarized in Figure 1) to
integrate multimodal features related to social concepts into a scalable
ontology-based Knowledge Graph (KG).

Figure 1: The pipeline under development aims to populate a large scale,
ontology-based Social Concepts Knowledge Graph that describes social concepts
with multimodal frames.

Social Concepts Candidate List Creation. Determining a starting list of
candidate concepts that can be reliably classified as referring to non-
physical objects is necessary to begin this research. In our approach, the
initial list is based on the conceptual taxonomy already in use by the Tate
Collection to tag the content of visual artworks, which explicitly includes
categories such as “universal concepts” and “social comment”.

Social Concept Definition by Multimodal Frames. Our approach is based on the
idea that a social concept can be defined in a multimodal frame which
describes and integrates complex linguistic and sensory-perceptual features.
In order to represent this model formally, we designed the Multimodal
Descriptions of Social Concepts (MUSCO) ontology, based on the Descriptions
and Situations (DnS) ontology [12], which supports a first-order manipulation
of theories and models. DnS was chosen as a core design pattern because it
allows for the modeling of non-physical objects, such as social concepts,
whose intended meaning results from statements, i.e. they arise in combination
with other entities. Specifically, DnS axioms capture the notion of
_situation_ as a unitarian entity out of a State of Affairs (SoA), that is
constituted by the entities and the relations among them, and a _description_
as an entity that partly represents a (possibly formalized) theory that can be
conceived by an agent.

Influenced by the work in [21], in the MUSCO ontology we consider that the
image annotation process is a situation (i.e. a context reified in the class
ImageAnnotationSituation) that needs to be described via a
ImageAnnotationDescription, and that represents the state of affairs of all
related, annotated data: actual multimedia data as well as metadata (see
Figure 2).

Figure 2: The MUSCO Ontology is aligned to and reuses patterns from DOLCE+DnS
Ultralite (DUL) foundational ontology [1] in order to represent and give
meaning to all data created during an image annotation process. All un-
prefixed classes belong to the namespace of the MUSCO ontology.

MUSCO’s architecture is deliberately modular, such that an
ImageAnnotationDescription can be modularized into sub-descriptions that
capture complex structures to be annotated. At this stage, we have identified
three complex structures to be annotated in art images evoking certain social
concepts: dominant colors, depicted physical objects, and depicted actions. As
such, in the MUSCO ontology we define the general description
ImageAnnotationDescription, which is satisfied by the general
ImageAnnotationSituation, and which is composed by three more specific
descriptions (DominantColorDescription, PORecognitionDescription, and
ActionRecognitionDescription) that define concepts and give meaning to data
extracted in the context of each of the complex structures (see Figure 3).
Finally, the MUSCO Ontology defines the description class SCMultiModalFrame,
which (a) defines a SocialConcept, (b) is used by a ImageAnnotationSituation,
and (c) is evoked by an ImageObject. The ontology also already allows for the
conjunct expression of data coming from linguistic and lexical resources
through classes such as WnSynsetFrame and ConceptualFrame. Even though the
current version of the MUSCO ontology is still under a process of revision and
editing, in its current state, it already includes the entities, resources,
and relationships necessary to integrate the experimental outputs presented in
this paper.

Figure 3: The MUSCO Ontology’s reuse of the DnS pattern is modular: defining a
general description for the image annotation situation, composed of simpler,
more specific descriptions which give meaning to specific annotation
structures and results. All classes in the figure belong to the namespace of
the MUSCO ontology defined in this work. Figure 4: Example use of the MUSCO
ontology to formalize multimodal features extracted from one image (dominant
color _bright yellow_ , depiction of the physical object _oven_ , and
depiction of the action _baking_), the image’s evocation of the social concept
“consumerism”, and the concept’s description as a multimodal frame. All arrows
with white arrowheads stand for the relation rdf:type.

Social Concept Visual Dataset Creation. For each social concept, a corpus of
(art) images that have previously been explicitly tagged with that social
concept is created. This corpus is produced by performing surveys of existing
image catalogues which include that social concept in their tagging scheme.

Sensory-Perceptual Data Collection. Our approach is based on the idea that
sensory-perceptual features of social concepts can be extracted from images
that evoke those social concepts. For each social concept, based on its corpus
of images, we extract a set of features including: labels for concrete
objects, actions depicted, and dominant color palettes. The specific technique
for extracting concrete objects labels from the images depends on the
previously provided information from that image. If the images have already
been tagged with labels of concrete objects, these are collected for further
analysis. If they have not, an object recognition task especially attuned to
art images is performed based on [9] in order to recognize physical objects in
the image and gather labels for such objects. A similar approach is taken for
extracting labels for depicted actions (e.g., “sitting”, “standing”) and/or
relationships (e.g., “holding hands”, “hugging”). If tags for actions or
relationships have already been attached to the images in the corpus, these
are used. If not, we perform relation and action detection on the images
following the most recent relation modeling techniques after they have been
trained specifically to process art images, following the approach by [14].
Finally, color analysis is performed on the images following the method
provided by the extcolors Python
package.111111https://pypi.org/project/extcolors/ This technique groups colors
based on visual similarities using the CIE76 formula, and outputs both an
image (for visual representation) and a text result with the usage (in number
and percentage of pixels) of the top five dominant (most used) colors in that
image.

##  5 Experiments and Results

Input Data Preprocessing. In order to apply our method to Tate Gallery
Collection dataset described in Section 2, it was first necessary to
reconstruct the hierarchy of the Tate’s subject taxonomy in a way that allowed
its eventual integration into a Social Concepts KG using the MUSCO ontology.
For this task, we performed a survey of the taxonomy based on metadata
accessed in March of 2021, finding that (1) it has three hierarchical levels,
going from broadest to narrowest categories, and that (2) it includes
hierarchical relationships between 2409 subject tags. These 2409 subject tags
were the ones used to reconstruct and then formalize the Tate’s tagging
hierarchy. In order to integrate the taxonomy and its subjects into our KG, we
extended the MUSCO ontology by reusing concepts defined in the Simple
Knowledge Organization System (SKOS) data
model121212https://www.w3.org/TR/skos-reference/#schemes (see Figure 5). With
this extension we were able to represent hierarchical relations between
subject tags, specifically with the property skos:broader. We implemented a
Python script to transform and serialise the taxonomy from JSON to Turtle
(.ttl)
format131313https://github.com/tategallery/collection/tree/master/processed/subjects.
All resources (MUSCO ontology, .ttl file, Python functions) are available in
the Github repository.141414https://github.com/delfimpandiani/musco

Figure 5: Addition to the MUSCO ontology model to formalize the use of concept
schemes coming from collections’ or other datasets’ taxonomies, such as the
Tate’s subject taxonomy. All classes with no explicit namespace belong to the
namespace of the MUSCO ontology.

Social Concepts Candidate List Creation. Visualizations of the Tate’s subject
taxonomy as graphs (also available in the GitHub repository) were performed
with the Graphviz151515https://pypi.org/project/graphviz/ package, in order to
ease manual investigation of its coverage and identification of areas where
social concepts may be more pervasive. Three areas of interest emerged: first,
the level 0 category “emotions, concepts and ideas” (specifically its level 1
children “universal concepts” and “emotions and human qualities”); second, the
level 1 category “social comment” (child of level 0 category “society”), and
third, the level 0 category “religion and belief” (see Figure 6). A total of
166 “narrow” [level 2] social concepts were manually selected from these
categories (80 from “emotions, concepts, and ideas”, 67 from “society”, and 19
from “religion and belief”). These concepts’ parent [level 1] and grandparent
[level 0] tags were excluded from subsequent analysis for two reasons. First,
compared to their narrow children tags (e.g., “fear”, “education”), many of
the broader tags actually refer to multiple social concepts at once (e.g.,
“emotions, concepts and ideas”, “education, science and learning”). Secondly,
a manual investigation of the Tate artworks’ metadata showed that artworks
seem to be explicitly tagged with level 2 tags, and level 1 and level 0 tags
are only included by virtue of being higher in the hierarchy. That is, there
seem to be no Tate artworks that are tagged only with the broader level 1 or 0
tags. The complete list of social concepts and their parents is available in
Appendix 1.

Figure 6: Three main areas of interest for the identification of social
concepts within the Tate subject taxonomy. Social concepts such as
“destruction”, “peace”, “wealth”, “courage” (surrounding “emotions, concepts
and ideas”), “consumerism”, “freedom”, “slavery”, “nationalism”, “ecology”
(surrounding “society”), “magic”, “enchantment”, “worship”, “blessing”
(surrounding “religion and belief”), among others, were identified in these
areas.

Concept-Artwork Matching. For each of the 166 social concepts, the number of
artworks tagged with that social concept was extracted and recorded.
Additionally, data about each artwork, including its bibliographic information
(name, id, artist, date, etc.) was stored.

Co-occurring Subject Tags. For each social concept, an investigation of the
metadata of the artworks tagged with it was performed, collecting other
subject tags used to index the content of those artworks. In this way, for
each social concept, we created a dictionary holding all co-occurring subject
tags and the frequency of co-occurrence. This allowed for the collection of
object and action tags without having to resort to object or human-object
interaction recognition techniques.

Co-occurring Physical Objects. Symbol grounding and _acquired embodiment_ is
expected to occur with labels referring to physical referents. Therefore, we
wanted to identify all co-occurring tags that specifically refer to physical
objects. The Tate’s subject taxonomy allowed for a quick solution to this
goal, as its hierarchical organization includes the top [level 0] concept
“objects”, which based on a manual investigation seem to refer to physical
objects. We extracted all tags under this category. A manual inspection of the
initial results showed that certain physical entities, such as “woman”, or
“tree” were not being extracted from artworks that clearly depicted these
entities (and were tagged with them). From this observation, we noted that
additional labels referring to physical objects were under the categories
“people” and “nature”, so we also extracted all tags whose parent tag was
“children”, “adults”, or “nude”, as well as those tags whose parent or
grandparent tag was “nature”. While it was not possible to confirm that
absolutely all terms referring to physical objects were extracted, after a
manual inspection of the Tate’s subject taxonomy, it was concluded that a
large majority of them was indeed extracted. We then performed statistical
analyses to obtain the number and frequency of co-occurring physical objects
for each social concept, as well as the top ten most frequent physical objects
co-occurring with each social concept. We also calculated the averages and
medians of these measures by taking into account all social concepts and their
co-occurring physical objects.

Co-occurring Physical Actions. A similar process to the one just described for
labels referring to physical objects was performed for what, within the Tate’s
subject taxonomy, are “action” labels. The identification of tags that refer
to actions was completed by identifying tags under the categories
“actions:postures and motions”, “actions:processes and functions”, or
“actions:expressive”, and “animals:actions”. We extracted all action tags co-
occurring with each of the social concepts, and then performed statistical
analyses as described in the previous section.

Dominant Color Visual Analyses. We then performed color analyses related to
two social concepts selected as case studies: “consumerism” and “horror”. The
hypothesis supporting this choice was that they should show distinctive color
profiles. For each concept, the color analysis was performed with the
extcolors Python package mentioned above, on 30 randomly selected images of
artworks (specifically, of paintings and prints) that had been explicitly
tagged with the concept. For each image, a color palette was created by
extracting the RGB coordinates and occurrence rates (in both number and
percentage of pixels) of the five colors with the highest occurrence rate in
that image. The criterion for this choice was that identifying the colors with
highest occurrence in an image is a proxy for identifying the most
pervasive/visible colors in an image, which may be a relevant feature humans
use to judge whether images evoke certain concepts. In order to represent this
idea more intuitively, a further analysis was completed to generate images of
the _proportional_ palettes, so as to represent the most common colors present
in each of the artworks in a more intuitive way. This final task was completed
with the MulticolorEngine by TinEye,161616https://labs.tineye.com/color/ but
we are developing code for the automation of this procedure.

Data Integration. We are developing software to automatically incorporate the
extracted data, including the co-occurrence patterns and visual features–into
a Social Concepts KG.171717The KG production is a work in progress that can be
monitored on the GitHub repository.

###  5.1 Results

Based on the metadata accessed in March of 2021, the Tate’s subject taxonomy
was found to be divided in three levels: [level 0] top concepts, representing
the most general categories; [level 1] slightly narrower concepts, children of
level 0 concepts; and [level 2] narrowest concepts, children of level 1
concepts and grandchildren of level 0 concepts. Out of the 2409 subject tags
in the taxonomy, 16 are level 0 concepts, 142 are level 1 concepts, and 2251
are level 3 concepts. The 166 social concepts selected as initial candidates
are all grandchildren of 3 out of the 16 top concepts, and children of 15 out
of the 142 middle categories available (see Table 1).

Concept-Artwork Matching. The number of Tate artworks explicitly tagged with
each of the 166 chosen social concepts ranged from 368 (“death”) to 1
(“paranoia”), with an average of 48 matches and a median of 27 matches. The
two case studies were in the top 20% of social concepts ranked on artwork
matches (“consumerism” with 71 artworks, and “horror” with 146 artworks) (see
Table 2).

Co-occurring Physical Objects and Actions. The number of co-occurring tags for
each of the 166 chosen social concepts ranged widely, from 1506 (“death”) to 7
(“paranoia”), with an average of 311 co-occurring subjects and a median of
262. Further analyses were performed to identify physical objects and action
tags from the co-occurring subject tags for each social concept. The number of
co-occurring _physical object_ tags for each of the chosen social concepts
ranged from 288 (“death”) to 6 (“infinity”), with an average of 69 co-
occurring physical objects and a median of 55 physical objects. Table 3 shows
the top ten most frequent co-occurring physical objects for four social
concepts. The number of co-occurring _action_ tags for the 166 social concepts
was decisively smaller, ranging from 38 (“death”) to 0 (“void”), with an
average of 11 co-occurring actions and a median of 8 actions. Table 4 shows
the top ten most frequent co-occurring actions for four social concepts. The
average frequencies of co-occurrence for physical objects and for actions with
each social concept are also presented in Table 2. Finally, Figure 7 includes
more intuitive visual representations for most co-occurring physical objects
(top) and actions (bottom) for the two case study social concepts.

Level | # Tags | Concrete Tags | # Soc. Concepts | Soc. Concept Tags
---|---|---|---|---
0 | 16 | “objects”, “nature”, “people” | 3 | “society”
1 | 142 | “weapons”, “trees”, “adults” | 15 | “social comment”
2 | 2251 | “missile”, “oak”, “old man” | 166 | “consumerism”

Table 1: Distribution of Tate subject tags based on its three hierarchical
levels, going from broadest (0) to narrowest (2). Each level includes concrete
concepts referring to physical objects and social concepts referring to non-
physical objects.

Social Concept | # Matches | # CO Objects | Freq Top Objects | # CO Actions | Freq Top Actions
---|---|---|---|---|---
death | 368 | 288 | 71.1 | 38 | 14.5
horror | 146 | 138 | 33.5 | 30 | 6.0
consumerism | 71 | 129 | 16.4 | 7 | 2.4
paranoia | 1 | 15 | 1.7 | 5 | 1.2
Average | 48 | 69 | 9.7 | 11 | 2.8
Median | 27 | 55 | 5.3 | 8 | 1.6

Table 2: Number of artwork matches, co-occurring physical objects, and co-
occurring actions, along with the average frequency of co-occurrence for the
top ten most frequently co-occurring physical objects and top ten actions for
four social concepts: “death” (with the highest number of matches), “paranoia”
(lowest number of matches), “consumerism” and “horror” (case studies). Average
and median values calculated from all 166 social concepts are also provided.

Social Concept |  Top 10 Physical Objects
---|---
death |  ‘man’, ‘woman’, ‘religious and ceremonial’, ‘clothing’, ‘furnishings’, ‘male’, ‘weapons’, ‘female’, ‘fine arts and music’, ‘sea’
horror |  ‘man’, ‘clothing’, ‘woman’, ‘uniform’, ‘animal/human’, ‘reading, writing, printed matter’, ‘male’, ‘fine arts and music’, ‘monster’, ‘medical’
consumerism |  ‘woman’, ‘reading, writing, printed matter’, ‘clothing’, ‘furnishings’, ‘food and drink’, ‘domestic’, ‘electrical appliances’, ‘kitchen’, ‘tools and machinery’, ‘product packaging’
paranoia |  ‘man’, ‘clothing’, ‘woman’, ‘figure’, ‘male’, ‘furnishings’, ‘curtain’, ‘jacket’, ‘jumper’, ‘suit’

Table 3: Top ten most frequent co-occurring physical objects with the social
concept with the most matched artworks (“death”), for the social concept with
the least matched artworks (“paranoia”), and for the two case studies
(“consumerism” and “horror”).

Social Concept |  Top 10 Actions
---|---
death |  ‘standing’, ‘lying down’, ‘reclining’, ‘supporting’, ‘embracing’, ‘sitting’, ‘flying’, ‘kneeling’, ‘carrying’, ‘sleeping’
horror |  ‘standing’, ‘sitting’, ‘recoiling’, ‘watching’, ‘lying down’, ‘screaming’, ‘carrying’, ‘hand/hands raised’, ‘embracing’, ‘fleeing’
consumerism |  ‘smiling’, ‘sitting’, ‘crouching’, ‘standing’, ‘reclining’, ‘lying down’, ‘talking’
paranoia |  ‘sitting’, ‘crouching’, ‘reclining’, ‘standing’, ‘walking’

Table 4: Top ten most frequent co-occurring actions with the social concept
with the most matched artworks (“death”), for the social concept with the
least matched artworks (“paranoia”), and for the two case studies
(“consumerism” and “horror”). Figure 7: Wordclouds for the top 50 co-occurring
object (top) and action (bottom) tags for all artworks tagged with
“consumerism” (left) and with “horror” (right). Larger words more frequently
co-occurred with the social concept of interest.

Dominant Color Visual Analyses. Figure 8 presents visual representations (as
proportional color palettes) of the color palette analyses performed on 30
randomly chosen images of paintings and prints for each of the two case study
social concepts (“consumerism” and “horror”).

(a)

(b)

Figure 8: Proportional palettes of 30 paintings and print images tagged with
“consumerism” (left) and with “horror” (right).

##  6 Discussion

In our reconstruction and examination of the Tate’s subject taxonomy, we found
that social concepts referring to _non-physical_ objects were concentrated in
three major areas of the taxonomy (“emotions, concepts, and ideas”, “society”,
and “religion and belief”), categories which resemble cognitive subtypes of
“abstract concepts”: emotional concepts, social concepts, and moral/ethical
concepts. This suggests that our manual investigation may have covered a good
amount of social concepts considered ‘abstract’ in the cognitive sense.
Gathered from the tagging taxonomy of a well-known collection of art images,
the initial list of social concepts we present stands as empirical proof of
the use of social concepts for tagging the content of visual material. With
this work, we begin the creation of a corpus of (art) images tagged with
social concepts, currently available in the GitHub repository as a dictionary
with social concepts as keys, and lists of image urls as values. Additionally,
our results show that the chosen social concepts were evoked by an average of
48 artworks, providing further evidence of the availability of data to pursue
subsequent analyses.

Our experiments on the co-occurrence of physical objects and actions further
show that it is possible to develop computational techniques that mirror the
idea of symbol grounding and _acquired embodiment_ , i.e., it is possible to
identify perceptual features of concrete objects and actions that co-occur
with social concepts, at least in the context of art images. While there seems
to be some regularity in the results, the low frequencies of co-occurrence
suggest that further research is needed to understand which of these objects
and actions, if any, have a substantial effect on the evocation of a social
concept. A manual examination of the top ten most frequent concepts co-
occurring with social concepts (some of them presented in Table 3 and Table 4)
suggest some regularity in the physical objects and actions that most
frequently co-occur with certain social concepts in these art images. Most of
these co-occurrences seem to agree with intuition (i.e., “consumerism” co-
occurring with “clothing”, “food and drink”, and “product packaging”; “horror”
co-occurring with “monster”, “recoiling” and “screaming”). The results of the
color analyses, visible in the proportional color palettes presented in Figure
8, also strongly suggest a certain degree of regularity in the color features
of Tate art images that evoke certain social concepts. As with the top co-
occurrence results, at first sight the color palettes seem to agree with
intuition (i.e, “consumerism” showing a greater luminosity and variety of
bright color, as in the aisles of a supermarket; “horror” showing dark and
less varied colors and tones, as in typical scenes from horror movies).

Some limitations of this work should be noted, such as the fact that the
images used in the experiments were not distinguished by type of artwork
medium (painting, print, drawing), which may have an effect on the type and
frequency of features that were extracted from the images. It is also
important to note that the regularity observed from the experimental results
may be limited only to the Tate’s collection and not generalisable to other
art image catalogues or collections. That is, while the Tate dataset provided
us with a clean, annotated corpus in which social concepts are explicitly used
as tags, as a curated dataset it encompasses a limited geographic, historical
and cultural perspective. Additionally, the fact that it is expert-tagged
might result in a biased interpretation and classification of the artworks,
which might differ from the interpretations of a larger, more diverse group of
viewers. Given social concepts’ ambiguous meanings, further work is needed
which takes into account various possible interpretations of the same artwork
by different observers.

##  7 Conclusion and Further Directions

Our approach models social concepts as complex objects, which can be described
with multimodal frames that integrate multisensory information. To represent
this model formally, we designed the pattern-based ontology Multimodal
Descriptions of Social Concepts (MUSCO), which allows for the conjunct
semantic characterization of multimodal features. A crucial next step is to
analyse the suitability of the proposed ontology for practical usage on
datasets from different sources. Our approach also proposes a pipeline for
automatically extracting and integrating features of images that evoke social
concepts. We show its potential by testing it on a corpora of art images from
a well-known collection. Our experimental results point towards some
regularity in certain sensory features of images tagged with specific social
concepts, and open space for further research to evaluate the proposed
approach focusing on datasets with different characteristics. More than
anything, our results serve as a proof of concept and open up new lines of
future research. The automatic population of a Knowledge Graph (KG) with the
extracted data is the natural next step of this work, potentially through
mapping-based knowledge extraction techniques. The generated KG may be
additionally populated with other methods, for example by automatically
generating descriptive paragraphs from the art images, and then performing
knowledge extraction on the natural language paragraphs, as well as by
including additional sensory data, such as sound or smell data, that may evoke
certain social concepts. With this work, we also begin creating a corpus of
(art) images that evoke certain social concepts. Moving forward, it ought to
be enlarged and diversified after a survey of additional catalogues and
collections.

To further our overall goal, we can improve our approach by refining the
initial social concepts list through alignment with the latest cognitive
science research, as well as through user-based studies. Additionally,
disambiguating the terms, expanding the terminology by leveraging lexical
resources such as WordNet, VerbNet, and FrameNet, and studying the terms’
distributional linguistic features in a textual corpus are next steps that we
expect will lead to substantial improvements. The visual analyses can be
further refined by distinguishing artwork medium types, and by extracting
contrast measures, common shapes, repetition, and other visual patterns. A
particularly interesting research direction would be to include facial
recognition analyses in our pipeline, which may allow identification of
emotions expressed by depicted subjects.

Our method is impulsed by the intuition that a KG containing multimodal data
for social concept description can eventually serve as input to a learning
model to automatically detect social concepts in images. Overall, the
experiments performed and results presented in this paper serve as a proof of
concept that extracting, integrating, and coincidentally exploiting multimodal
data related to social concepts is a promising direction for future research.

## References

  * [1] Dolce+dns ultralite (dul) ontology.  Last accessed: 2021-08-05.  URL: http://www.loa.istc.cnr.it/ontologies/DUL.owl.
  * [2] Somak Aditya, Yezhou Yang, and Chitta Baral.  Integrating knowledge and reasoning in image understanding.  In International Joint Conference on Artificial Intelligence (IJCAI), 2019.
  * [3] Youssef Ahres and Nikolaus Volk.  Abstract concept & emotion detection in tagged images with cnns.  2016\.  URL: http://cs231n.stanford.edu/reports/2016/pdfs/008_Report.pdf.
  * [4] Taylor Arnold and Lauren Tilton.  Distant viewing: analyzing large visual corpora.  Digital Scholarship in the Humanities, 34(Supplement 1):i3–i16, 2019.
  * [5] Randheer Bagi, Tanima Dutta, and Hari Prabhat Gupta.  Deep learning architectures for computer vision applications: a study.  In Advances in data and information sciences, pages 601–612. Springer, 2020.
  * [6] Roland Barthes.  Camera Lucida: Reflections on Photography.  New York: Hill and Wang, 1981.
  * [7] Anna M Borghi, Laura Barca, Ferdinand Binkofski, and Luca Tummolini.  Varieties of abstract concepts: development, use and representation in the brain.  Philosophical Transactions of the Royal Society, 2018.
  * [8] Tao Chen, Damian Borth, Trevor Darrell, and Shih-Fu Chang.  Deepsentibank: Visual sentiment concept classification with deep convolutional neural networks.  ArXiv preprint :1410.8586, 2014.
  * [9] Elliot J Crowley and Andrew Zisserman.  The art of detection.  In European conference on computer vision, pages 721–737. Springer, 2016.
  * [10] Charles P. Davis and Eiling Yee.  Building semantic memory from embodied and distributional language experience.  WIREs Cognitive Science, e1555, 2021.
  * [11] José Luis Preza Díaz, Amelie Dorn, Gerda Koch, and Yalemisew Abgaz.  A comparative approach between different computer vision tools, including commercial and open-source, for improving cultural image access and analysis.  In 2020 10th International Conference on Advanced Computer Information Technologies (ACIT), pages 815–819. IEEE, 2020.
  * [12] Aldo Gangemi and Peter Mika.  Understanding the semantic web through descriptions and situations.  In OTM Confederated International Conferences “On the Move to Meaningful Internet Systems”, pages 689–706. Springer, 2003.
  * [13] Paul Hoffman, James L McClelland, and Matthew A Lambon Ralph.  Concepts, control, and context: A connectionist account of normal and disordered semantic cognition.  Psychological review, 125(3):293, 2018.
  * [14] David Kadish, Sebastian Risi, and Anders Sundnes Løvlie.  Improving object detection in art images using only style transfer.  ArXiv preprint :2102.06529, 2021.
  * [15] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International journal of computer vision, 123(1):32–73, 2017.
  * [16] Claudio Masolo, Laure Vieu, Emanuele Bottazzi, Carola Catenacci, Roberta Ferrario, Aldo Gangemi, and Nicola Guarino.  Social roles and their descriptions.  In KR, pages 267–277, 2004.
  * [17] Wiebke Petersen.  Representation of concepts as frames.  Düsseldorf University, 2015.  Reprint.
  * [18] Pauline Rafferty and Rob Hidderley.  Indexing multimedia and creative works: the problems of meaning and interpretation.  Routledge, 2017.
  * [19] Nusrat J Shoumy, Li-Minn Ang, Kah Phooi Seng, DM Motiur Rahaman, and Tanveer Zia.  Multimodal big data affective analytics: A comprehensive survey using text, audio, visual and physiological signals.  Journal of Network and Computer Applications, 149:102447, 2020.
  * [20] Sebastian Stabinger and Antonio Rodriguez-Sanchez.  Evaluation of deep learning on an abstract image classification dataset.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 2767–2772, 2017.
  * [21] Miroslav Vacura, Vojtech Svátek, Carsten Saathoff, Thomas Franz, and Raphaël Troncy.  Describing low-level image features using the comm ontology.  In 2008 15th IEEE International Conference on Image Processing, pages 49–52. IEEE, 2008.