11institutetext: Max Planck Institute for Solar System Research, Justus-von-
Liebig-Weg 3, 37077 Göttingen, Germany 22institutetext: Solar-Terrestrial
Centre of Excellence – SIDC, Royal Observatory of Belgium, Ringlaan -3- Av.
Circulaire, 1180 Brussels, Belgium 33institutetext: Institut d’Astrophysique
Spatiale, CNRS, Univ. Paris-Sud, Université Paris-Saclay, Bât. 121, 91405
Orsay, France

# Beyond small-scale transients: a closer look at the diffuse quiet solar
corona

J. Gorman Beyond small-scale transients: a closer look at the diffuse quiet
solar coronaBeyond small-scale transients: a closer look at the diffuse quiet
solar corona L.P. Chitta Beyond small-scale transients: a closer look at the
diffuse quiet solar coronaBeyond small-scale transients: a closer look at the
diffuse quiet solar corona H. Peter Beyond small-scale transients: a closer
look at the diffuse quiet solar coronaBeyond small-scale transients: a closer
look at the diffuse quiet solar corona D. Berghmans Beyond small-scale
transients: a closer look at the diffuse quiet solar coronaBeyond small-scale
transients: a closer look at the diffuse quiet solar corona F. Auchère Beyond
small-scale transients: a closer look at the diffuse quiet solar coronaBeyond
small-scale transients: a closer look at the diffuse quiet solar corona R.
Aznar Cuadrado Beyond small-scale transients: a closer look at the diffuse
quiet solar coronaBeyond small-scale transients: a closer look at the diffuse
quiet solar corona L. Teriaca Beyond small-scale transients: a closer look at
the diffuse quiet solar coronaBeyond small-scale transients: a closer look at
the diffuse quiet solar corona S.K. Solanki Beyond small-scale transients: a
closer look at the diffuse quiet solar coronaBeyond small-scale transients: a
closer look at the diffuse quiet solar corona C. Verbeeck Beyond small-scale
transients: a closer look at the diffuse quiet solar coronaBeyond small-scale
transients: a closer look at the diffuse quiet solar corona E. Kraaikamp
Beyond small-scale transients: a closer look at the diffuse quiet solar
coronaBeyond small-scale transients: a closer look at the diffuse quiet solar
corona K. Stegen Beyond small-scale transients: a closer look at the diffuse
quiet solar coronaBeyond small-scale transients: a closer look at the diffuse
quiet solar corona S. Gissot Beyond small-scale transients: a closer look at
the diffuse quiet solar coronaBeyond small-scale transients: a closer look at
the diffuse quiet solar corona

(Received 11 January 2023 / Accepted 10 July 2023)

###### Abstract

Aims. Within the quiet Sun corona imaged at 1 MK, much of the field of view
consists of diffuse emission that appears to lack the spatial structuring that
is so evident in coronal loops or bright points. We seek to determine if these
diffuse regions are categorically different in terms of their intensity
fluctuations and spatial configuration from the more well-studied dynamic
coronal features.

Methods. We analyze a time series of observations from Solar Orbiter’s High
Resolution Imager in the Extreme Ultraviolet to quantify the characterization
of the diffuse corona at high spatial and temporal resolutions. We then
compare this to the dynamic features within the field of view, mainly a
coronal bright point.

Results. We find that the diffuse corona lacks visible structuring, such as
small embedded loops, and that this is persistent over the 25 min duration of
the observation. The intensity fluctuations of the diffuse corona, which are
within $\pm 5\%$, are significantly smaller in comparison to the coronal
bright point. Yet, the total intensity observed in the diffuse corona is of
the same order as the bright point.

Conclusions. It seems inconsistent with our data that the diffuse corona is a
composition of small loops or jets or that it is driven by discrete small
heating events that follow a power-law-like distribution. We speculate that
small-scale processes like MHD turbulence might be energizing the diffuse
regions, but at this point we cannot offer a conclusive explanation for the
nature of this feature.

###### Key Words.:

Sun: atmosphere — Sun: corona — Sun: UV radiation

Figure 1: Observation summary. Panel a: the full field of view (FOV) covered
in the 17.4 nm band of HRIEUV. The red box outlines the zoomed-in FOV that is
shown in panel b. Both panels a–b show the time-averaged, normalized intensity
(to the minimum and maximum in the area in panel b). Panel b: the vertical
black line shows the location of the cut that is used in the time-distance
plot displayed in Fig. 2 and the intensity-distance plot shown in Fig. 3. The
cyan box (i) and the orange box (ii) outline the diffuse region and the bright
point used in Fig. 4. The green box (iii) outlines the loop-like features
zoomed into in Fig. 7. See Sect. 3.1.

##  1 Introduction

In studying the coronal heating problem, three main regions are acknowledged
to be observed within the corona. These are active regions (ARs), coronal
holes (CHs), and the quiet Sun (QS). Active regions are the brightest and most
dynamic portions of the corona, often associated with underlying strong
magnetic field patches including sunspots, and seen to consist of coronal
loops as observed in the extreme ultraviolet (EUV) and X-rays. In contrast,
CHs are the darkest portions in the solar corona in EUV and are attributed to
open magnetic fields that connect the solar surface to the heliosphere.
Outside of ARs and CHs, there remains the QS. As the QS makes up the largest
proportion of the solar surface, understanding the processes at work within
this portion of the corona is crucial to understanding coronal heating.

Based on observations, a variety of small-scale, dynamic features, such as
nanoflares, coronal bright points, and jets, have all been considered to be of
high importance to balance the overall energy losses from the QS corona (e.g.,
Aschwanden et al., 2000; Hosseini Rad et al., 2021; Shen, 2021; Chitta et al.,
2021a). More recently, based on high spatial resolution and high cadence EUV
observations from the Extreme Ultraviolet Imager (EUI; Rochus et al., 2020) on
board the Solar Orbiter (Müller et al., 2020), Berghmans et al. (2021)
observed compact isolated coronal brightenings termed campfires. A main
characteristic of all these nanoflare-type heating events, including
campfires, is that they are all clearly distinguishable from the local
background coronal emission. Whether these observable discrete heating events
are sufficient to explain the energy losses from the quiet Sun corona is still
an open question (Aschwanden et al., 2000; Chitta et al., 2021a). While much
focus has been placed on these more distinguishable elements or events in the
past, there remains much to be learned from the quieter portions of plasma
that are devoid of these obvious localized transient brightenings. In
particular, we are referring to the areas of seemingly stable and featureless
EUV emission in the QS that we label the diffuse corona.

Diffuse emission associated with ARs has been studied in the past. Viall &
Klimchuk (2011) looked at the diffuse portions of AR emission (i.e., areas not
associated with any distinguishable loops or loop footpoints) and determined
that these regions encompass a majority of the emitting portion of ARs, are
only marginally less bright (10–35%) compared to AR loops, and are dynamically
heated, as opposed to being energized by some steady process. Whether or not a
similar significance and heating mechanism can be attributed to the diffuse
corona in the QS remains to be seen.

In most studies, however, this diffuse emission is considered as a background
only. Usually, no particular consideration is given to it, apart from the urge
to correct for (i.e., subtract) it when looking at features resolved in space
and time embedded in this background.

In this work, we analyze the evolution of the diffuse quiescent corona
observed with the high spatial and temporal resolution allowed by the EUI
instrument. We compare the diffuse emission and its fluctuations to those seen
in the more dynamic features within the observational field of view (FOV). We
find that not only is the diffuse corona an enduring contributor of seemingly
stable and unstructured emission, but that it also is more widespread and is
therefore considered to be an important factor in the overall energy balance
of the solar corona.

##  2 Observations

On March 26, 2021, Solar Orbiter (Müller et al., 2020) was located at a
distance of 0.72 AU from the Sun on the far-side with respect to Earth. The
EUV High Resolution Imager (HRIEUV) on the Extreme Ultraviolet Imager was
pointed towards the limb at a latitude of about 30∘ and recorded images with
very high cadence of 2 s (1.65 s exposure) between 23:32:20 UT and 23:57:18 UT
(25 min).111Data release 4.0 2021-12. DOI: https://doi.org/10.24414/s5da-7e78

HRIEUV has a plate scale of 0.492″ pixel-1, which amounts to roughly 260 km
pixel-1 on the Sun (as seen from Solar Orbiter) for this data set. The pass-
band of HRIEUV is centered at 17.4 nm and its response peaks at temperatures
of about 1 MK due to the presence of spectral lines of Fe ix (at 17.11 nm) and
Fe x (at 17.45 nm and 17.72 nm). Before conducting our analysis, we aligned
the level 2 (L2) data to remove the jitter in the image sequence as described
in Chitta et al. (2022).

The region observed by EUI is a quiet Sun region outside coronal holes. While
the target area was not visible from Earth-based telescopes, the Full Sun
Imager (FSI) of EUI acquired images in the 304 Å and the 174 Å channels. These
show a north polar coronal hole at latitudes well above 60∘ North, while the
field of view of HRIEUV is below 55∘ North, i.e., located far from the coronal
hole.

##  3 Results

We focus on the on-disk portion of these observations that also covered the
limb and regions off the disk. An overview of the observational FOV, including
the primary areas of interest for this study, is shown in Fig. 1. Panel a
depicts the full-FOV of HRIEUV, and panel b shows a zoom into the on-disk
portion of the observation that is of interest for this study. For both a and
b, the intensity shown has been averaged over the entire 25 min observing time
of this EUI sequence, and then it has been normalized by the minimum and
maximum intensities within the zoomed FOV shown in b. Thus, the image color
scale in the full FOV and the zoom in Fig. 1 is different. With the above
normalization for panel b it is limited between 0 and 1 and for panel a, which
contains pixels with both greater and weaker intensities, it ranges from
$-$0.7 to 1.1.

###  3.1 Diffuse region, loop-like features, and coronal bright points

We find several distinct regions in the corona at $\sim$1 MK. There are (1)
dark patches (with normalized intensities of 0.25 or less), (2) bright patches
associated with loop-like structures and coronal bright points containing
intensities at or near the saturation limit of the detector, and, finally, (3)
there appear to be what we refer to as “diffuse” regions.

These diffuse regions are areas that seem to lack structuring in both time and
space. They are hazy portions seen throughout the FOV with time-averaged,
normalized intensities mostly between 0.25–0.6. An example patch of diffuse
corona is outlined by the cyan box (panel i) in Fig. 1b. The diffuse corona
appears to cover a significant area when compared to the brighter regions in
the FOV. However, the diffuse corona lacks any of the obvious structuring that
is associated with the well-studied structures of the quiet solar corona, such
as coronal bright points that are composed of $\sim$10 Mm long loops (e.g.,
see the review article on coronal bright points by Madjarska, 2019).

Our analysis aims to compare these differing regions in both a qualitative and
quantitative manner to determine the extent of this apparent discrepancy in
terms of spatial structuring. For this we investigate a cut through the FOV
that runs through these different features (black line in Fig. 1b): diffuse
regions, loop-like features, and a coronal bright point. (While many bright
features are saturated in the exposure, in particular at the limb, this bright
point is not). The intensity along this cut as a function of time is presented
in a time-distance plot (see Fig. 2). It is evident from this plot that the
bright point stretching from $y{=}$106 Mm to 117 Mm shows intensity
structuring in both space and time. The same also applies to the clear loop-
like features situated around $y{=}$65 Mm to 100 Mm. The bright point and the
loop-like features appear to be continually evolving on spatial scales of
roughly a few Mm and timescales on the order of a few minutes, although the
loop-like features are comparatively less variable than the bright point.

Figure 2: Temporal evolution of cut through diffuse region and coronal bright
point. This time-distance plot shows the intensity in HRIEUV from along the
cut (1 pixel in $x$) outlined by the black vertical line in Fig. 1b as it
evolves over the entire 25 min observing time. The intensity is scaled
linearly from 915 DN s-1 (black) to 2500 DN s-1 (white). Above the panel the
location in the $y$ direction of three types of regions are marked. The two
arrows mark transient brightenings in the diffuse region. See Sects. 3.1, and
4.2.  Figure 3: Spatial variability along a cut through a diffuse region,
loop-like features and a coronal bright point at four different times. This
intensity-distance plot shows the intensity along the cut outlined by the
black vertical line in Fig. 1b for several intervals of time during the
observation. For each 30 s interval (15 time-consecutive images), the
intensity at each pixel along the cut is averaged over that time period and
plotted as a function of distance. Four of these light curves are shown as a
stack, with the bottom curve being the intensity-distance 30 s average ending
at 1 min (and, therefore, starting at 30 s), the curve directly above it is
the 30 s average ending at 10 min, and so on. The black vertical bars with
each curve show the maximum error for that curve (see Appendix A). Above the
panel the location in the $y$ direction of three types of regions are marked.
See Sects. 3.2 and 4.1.

The bright point is a typical small coronal bright point in appearance and
size. It consists of a number of short coronal loops with the core region
(Fig. 4 ii) having a size of just over 5 Mm (cf. Madjarska, 2019). Such
compact small-loop-type bright points are found in abundance in the quiet Sun
and have been used, e.g., to determine the coronal rotation Brajša et al.
(2001, 2002), even though the spatial resolution of the older data was
inadequate to sufficiently resolve the internal structure of the bright
points. The region with loop-like features does not show clearly
distinguishable loops, but (in the time-averaged image) elongated features
reminiscent of loops (Fig. 7 iii). Because we concentrate in this study on the
diffuse regions, we do not follow this up further (see Appendix B for a
further discussion of the loop-like features).

In contrast to these more dynamic features, diffuse regions seen in Fig. 2 are
rather invariable. The time-distance plot reveals the overall temporal
stability of the diffuse corona. The emission is diffuse in the $y$-direction
in a range of 15–65 Mm, with two seemingly distinct regions of differing
intensity levels: a brighter portion between 15–42 Mm and a darker portion
from 42–65 Mm. For both portions, there is a hazy quality that remains stable
throughout the 25 min observing time on spatial scales comparable to that of a
supergranule. Both the brighter and darker sections show no obvious jumps in
intensity across their width. There is a small brightening at about $y{=}$48
Mm occurring periodically over intervals of time limited to a few minutes
(highlighted by arrows in Fig. 2). This could be categorized as an HRIEUV
campfire (see Berghmans et al., 2021). However, the brightening is only a few
pixels in width and makes up only a small fraction of the overall emission in
that region.

We emphasize that the data shown in the space-time plot in Fig. 2 have a time
cadence of only 2 s and a spatial sampling of 260 km on the Sun. This implies
that transient small-scale features should be visible if they were present
(down to those temporal and spatial scales).

Figure 4: Intensity fluctuations in diffuse and bright point regions. The two
left panels show the zoomed-in FOV covering the diffuse region (panel i) and
bright point (panel ii) outlined by the cyan and orange boxes in Fig. 1b,
respectively. Each image is the same time-averaged and normalized figure as
shown in Fig. 1b, except the intensity range shown is further limited to the
minimum and maximum for each zoomed FOV. Within the regions of interest, four
smaller sub-fields a to d are outlined. Each of these boxes has the same area.
The spatially-averaged relative (left y-axis) and absolute (right y-axis)
intensities for the respective sub-fields are plotted as a function of time on
the right in panels a–d. The vertical bars shown in each of these panels
represent the maximum error for each time-series (see Appendix A).

###  3.2 Spatial and temporal variability

We now dissect the time-distance plot for a closer look at the difference
between the coronal bright points and the diffuse regions. To this end, the
intensity along the cut from Fig. 1b was averaged over each subsequent 30 s
interval and displayed as line plots of time-averaged intensity versus
distance, several of which are shown stacked in Fig. 3. Essentially these
stacked plots are horizontal cuts through Fig. 2 for a given time span.

Across the aforementioned diffuse regions (i.e., $y{=}$15–65 Mm), spatial
changes in the intensity are gradual. As will be discussed below, the small
fluctuations seen on top of this gradual variation are at the level of the
calculated maximum error. The shape of the curve also remains similar for each
plot, despite there being 5–10 min of separation between them. This implies
that the diffuse corona behaves in a spatially coherent fashion over its
entire extent corresponding to the scale of a supergranule of about 20 Mm.

The peak intensity in the brighter diffuse area (at $y{\approx}35$ Mm) is only
slightly less than, or even equal to the peak intensity seen in the bright
point and loop-like features (in particular before $t{=}15$ min). Conversely,
the morphology of the loop-like features and coronal bright point changes
markedly over time, and the intensity fluctuations within these features at
each point in time varies significantly, i.e., well-above the maximum error.
This further differentiates the characteristics of the diffuse regions from
those of the bright points and loop-like features.

To quantitatively characterize the temporal stability of the diffuse corona,
its relative intensity fluctuations provide clearer insight (Fig. 4). Here, we
compared the absolute and relative intensity variations of the diffuse corona
(box i in Fig. 1b) to that within the coronal bright point (box ii). The
absolute fluctuations are derived by simply spatially averaging the intensity
within each of the sub-fields a to d for each snapshot. The relative
fluctuations were then calculated by subtracting the overall time-averaged
intensity within each sub-field from its spatially-averaged intensity and then
dividing this difference by the time-averaged intensity. Figure 4i and ii show
which sub-fields of the observation are sampled, either from a diffuse region
(a, b) or from a bright point (c, d).

The relative fluctuations in the sub-fields of the diffuse region (a, b)
remain within a few percentage points (less than $\pm$5%) over the entire 25
min (Fig. 4a, b). In contrast, in the sub-fields of the coronal bright point
(c, d), the intensity changes by over 10% on shorter timescales of about 5 min
and can sway by up to almost 40% over the entire 25 min (Fig. 4c, d).

To judge the significance of these fluctuations, one has to compare the
observed variability to the measurement errors. In Appendix A we discuss how
we calculate the maximum estimated error. As a very rough estimate of this,
for our current data set in regions that are moderately bright, the typical
error of the intensity is of the order of 3%.

In general, the sub-fields of the bright point show trends that rise well
above the level of the maximum estimated error, both on shorter and longer
timescales. In contrast, in the diffuse regions the variability we see would
essentially be consistent with measurement errors. There might be some
dynamics within the diffuse regions with amplitudes just above the noise that
persist for more than a few time steps, but these fluctuations are still
smaller than the overall level of fluctuations over the observing period (of
25 min). While the fluctuations shown here are only for two sub-sections of
each feature, the behavior is similar when more sub-sections are analyzed.

This relative temporal stability of the diffuse corona and the larger
variability seen in the coronal bright point are further demonstrated using
the respective snapshots, at three instances, without time-averaging (see Fig.
5). It is clear from these images that the diffuse region is not only
temporally stable compared to the bright point, but it also lacks any
distinguishable spatial structuring.

Figure 5: Snapshots of diffuse region and bright point without time-averaging.
The snapshots at three times as indicated by the time stamps are shown in the
three columns. The top row shows the the diffuse region (marked i) and the
bottom row the bright point (marked ii). Images are commonly scaled to the
same minimum and maximum values of intensity. Boxes a–d have the same meaning
as in Fig. 4. See Sects. 3.2 and 4.1.

This time series analysis underlines that the diffuse corona, often considered
as a mere background, contributes a significant amount of the coronal emission
in the quiet Sun. For a quantitative estimate of this contribution, see Sect.
4.1.

##  4 Discussion

We note the general pervasiveness of the diffuse corona seen at temperatures
of 1 MK with HRIEUV. These regions remain featureless in both time and space,
yet are relatively bright compared to the more dynamic features within the
FOV. This naturally leads to questions about the significance of these regions
in terms of their contribution to the emission of the 1 MK corona and on the
possible heating mechanisms operating in such regions. Similarly, questions
arise regarding the reasoning behind the lack of noticeable structuring and
how this is related to the overall energy balance within the solar atmosphere.

###  4.1 Contribution of diffuse regions to radiative losses

The intensity observed in the diffuse regions is relatively strong, even when
compared with bright points. This is illustrated in Fig. 4a-d where we show
the relative and absolute average intensity values [DN] within sub-fields of
the diffuse and bright point regions. We see that, over the entire observing
period, the time-averaged diffuse intensity (i.e., radiative flux per area) is
about 80% that of the bright point, i.e., of the same order of magnitude.
Also, from Fig. 3, the peak intensities along the cut from the brighter
diffuse portion (around $y=\,$35 Mm) are very similar to those from the bright
point (around $y=\,$110 Mm) and also about equal to those from the loop-like
features sampled between $y=\,$65 Mm and 100 Mm. All of this points to the
conclusion that the emission coming from the diffuse regions is non-negligible
and should be an important consideration in any coronal heating model, in
particular when considering that the diffuse corona can cover a large fraction
of the quiet Sun.

Figure 6: Grouping of diffuse coronal regions and bright points in quiet Sun.
The intensity image is the same as Fig. 1b. The diffuse coronal regions (dc1
to dc4) are outlined in blue, the bright points (bp1 to bp6) are highlighted
in yellow. See Sect. 4.1.

We further investigate the significance of diffuse emission by conducting a
rough estimate of the total emission contribution from the diffuse areas
compared to the bright points within the FOV. For this we classify the regions
in the FOV in Fig. 1b into areas covered by coronal bright points and diffuse
regions by a simple by-eye estimate. These regions are marked and labeled in
Fig. 6. Some of the bright points host a few pixels that are saturated on the
detector. Hence our estimate for the bright point emission will be a lower
limit only. However, this effect should be well below a factor of two and thus
our calculation should be just fine for the order-of-magnitude estimate we aim
for here.

Based on the integration of the emission from the respective diffuse and
bright point areas, we find that the seemingly quiet diffuse areas provide
almost 2.7 times the emission as the more dynamic features. Thus overall, the
diffuse emission would dominate the quiet Sun radiative losses at around 1 MK,
while discernible bright features would contribute a minor fraction, maybe
half at best. This is not surprising since these by-eye defined diffuse
regions also have almost three times the area as the bright points (see Fig.
6).

###  4.2 Diffuse quiet-Sun corona and small loops

The magnetic field is space-filling and is the driver behind the energetics of
the corona. The work of Dowdy et al. (1986) described the solar magnetic scene
to be comprised of both (locally) open field lines expanding into funnels with
height and small-scale, closed loops that connect back to the surface. Based
on studies of magnetic field extrapolations, such loops are expected to be
rooted not only in the network regions at the edges of supergranules, but also
in the internetwork within a few Mm of the network boundaries (Schrijver &
Title, 2003; Wiegelmann et al., 2010).

Quiet Sun observations reveal the presence of very short loops clearly
distinguishable in EUV observations, only a few Mm long. They come at
(probably) high coronal temperatures Peter et al. (2013); Barczynski et al.
(2017) as well as at lower transition region temperatures Hansteen et al.
(2014). These short loops have lifetimes of only a (few) minute(s). More
recent observations also reveal the dynamic nature and substructure of such
small EUV loops including propagating features Mandal et al. (2021) and small
jets associated with them Chitta et al. (2021b). Also, transition region loops
crossing a super-granular cell in the quiet Sun (i.e., with a length of ca. 20
Mm), have been reported (e.g., Fig. 1 of Teriaca et al., 2004). However, all
these types of loops are discrete units and far from space-filling.

Still, the presence of loops in the quiet Sun would mean that we should expect
the presence of a few Mm long, small-scale coronal loops in our observation
even within the so-called diffuse areas. As discussed above, these smaller
loops are expected to be dynamic, showing brightness fluctuations on
timescales on the order of minutes (see also Reale, 2010). Similar lifetimes
of (magnetic) loops of a few minutes are also found based on magnetic field
extrapolations from time series of high resolution quiet Sun magnetograms
(Wiegelmann et al., 2010). Case in point, our observations do show such
brightness fluctuations in the coronal bright points, which are indeed on
shorter timescales compared to the 25 min of observed stability for our
diffuse areas.

If the diffuse areas are composed of such smaller loops, then we would expect
loop-like brightenings at some point during our analysis unless a different
loop emission behavior is at play. We do see that there are a limited number
of small-scale brightenings occurring within some diffuse portions of Fig. 2
(see white arrows) that last for a few minutes at a time and are only a few
pixels in length (i.e., some 500 km in $y$-distance). These brightenings,
however, are not ubiquitous as would be expected for the typical picture of
low-lying loops crisscrossing everywhere in the FOV. Also, these small
intensity enhancements barely rise above the level of the local diffuse region
(or background). When compared to the intensity level local to the area of the
brightening in the minutes before and after its enhancement, the brightening
only rises to a level of 5% above this background.

The coherent spatial intensity variations on supergranular scales in these
diffuse regions indicate that all the constituting smaller loops, if present,
must evolve in unison, which is unlikely. Based on this, we think that the
small (coronal) loops reported before may not be the source of emission from
the diffuse regions. In that case we should (occasionally) see a transient
brightening caused by one of the transient Mm-scale loops that have been
reported before, but we do not see these.

###  4.3 Diffuse quiet-Sun corona and jets

Chromospheric spicules (de Pontieu et al., 2007) and transition region network
jets (Tian et al., 2014) are other common, small-scale jet features whose
imprint should be seen everywhere on the quiet Sun. Spicules are chromospheric
jets and are categorized into two types (de Pontieu et al., 2007). Type-I
spicules are longer-lived and seen to both rise and fall at the limb,
remaining at chromospheric temperatures throughout their lifetime. Type-II
spicules are more impetuous and often appear to shoot up before disappearing
from chromospheric imaging channels, sometimes then appearing in the hotter
channels imaging transition region (TR) plasma (Pereira et al., 2014), which
can also include the return flows when cooling back down from hotter
temperatures Bose et al. (2021). Spicules have typical speeds ranging from
tens of km s-1 (type-I) up to 100 km s-1 (type-II), lifetimes of several
minutes, and characteristic lengths ranging from a few hundred to several
thousand km (e.g., Tsiropoula et al., 2012, and included references).

While spicular material is easiest to detect at the cooler temperatures found
in the chromosphere and TR (T $\leq 10^{5}$ K), recent investigations show
that there are exceptions to this where spicule signatures can be observed as
corrugations in the EUV emission. For example, signatures have been found that
the on-disk counterparts of spicules, namely dynamic fibrils, show EUV
emission Mandal et al. (2023). This might indicate higher temperatures of more
than $10^{5}$ K, although this is not conclusive, yet (Henriques et al., 2016;
Martínez-Sykora et al., 2018; Samanta et al., 2019). Similarly, coronal
counterparts of network jets remain elusive (Kayshap et al., 2018; Gorman et
al., 2022).

Small coronal jets are, in general, a common phenomenon in quiet Sun regions.
In particular, recent EUI observations have shown an abundance of small-scale
jets (e.g., Chitta et al., 2021b; Mandal et al., 2022). Small jets and their
substructure, e.g., plasmoids, have recently also been observed in radio
emission (e.g. Shimojo et al., 2017; Rodger et al., 2019). Still, the question
remains if highly dynamic small-scale features such as these jets or spicules
can provide an explanation of the diffuse corona.

At 2 s cadence and about 500 km resolution, the observation from HRIEUV that
we analyze in this study certainly could pick up any spicules or network jets
that would show counterparts in EUV. Yet, we do not detect any such signatures
in the far-reaching diffuse areas. This either means that these jets only very
rarely reach coronal temperatures, or at least do not get heated to 1 MK in
the magnetic regime that is responsible for the diffuse emission. Another
possibility could be that those small-scale jets that do reach coronal
temperatures might lose their identity and structuring as their energy gets
dissipated.

###  4.4 Diffuse quiet-Sun corona and small-scale heating events

The simple presence of a diffuse, seemingly featureless structure has
implications for the heating mechanism that has to sustain the hot corona. The
energization has to be either continuous or be concentrated in a very large
number of small-scale heating events so that this is not leaving an imprint at
the resolution of our observations. Hence, we explore the applicability of
several proposed heating mechanisms for the diffuse corona. Clearly, we cannot
relate our findings to all possible heating scenarios, instead we picked those
we considered most relevant for our study. The reader is referred to recent
reviews that discuss the heating problem in general (e.g., Klimchuk, 2006),
with respect to 3D models (e.g., Peter, 2015), in terms of waves (e.g., Van
Doorsselaere et al., 2020), or in the light of field-line braiding (e.g.,
Pontin & Hornig, 2020).

One heating scenario involves a power-law distribution of the number of
brightening events with energy characterized by a slope of $-$2, at least for
the smaller flare-like events with less than $10^{26}$ ergs (Hudson, 1991).
This requires there to be an increasingly higher number of distinct events at
ever smaller spatial and energy scales. In fact, these events would be so
small that they have not yet been resolvable by Sun-observing instruments
(Parnell & Jupp, 2000; Chitta et al., 2021a). Up until now, however, results
are mixed regarding whether or not the energy distribution for impulsive
heating events has the necessary slope to account for coronal heating (e.g.,
Berghmans et al., 1998; Krucker & Benz, 1998; Aschwanden et al., 2000; Parnell
& Jupp, 2000; Pauluhn & Solanki, 2007; Aschwanden & Shimizu, 2013). It has
also been pointed out that even if the power-index is greater than 2, this is
not necessarily sufficient to heat the quiet Sun (Berghmans, 2002). Joulin et
al. (2016) argue that, regardless of the actual energy distribution slope, it
is inherently unlikely to be able to detect high-frequency, small-scale
brightenings against the background coronal emission. The authors state that
their study combined with the works of others before them shows that heating
cannot be guaranteed to be observed as broken up into discrete events.

Perhaps such unresolved, impulsive brightenings are what is the cause behind
the diffuse corona. If there were a sufficiently high number of small-scale
events resulting in the diffuse emission that we see, they would still have to
be driven in a spatially coherent fashion to create a diffuse region about 10
to 20 Mm large in size, i.e., a region the size of a supergranule. Considering
the significant structuring found at the base below the corona, in the
chromosphere, it is not very plausible that the diffuse corona is driven by
small-scale events.

Should small-scale events drive and heat the diffuse coronal patches, one
would also expect larger events in these regions. This is based on the finding
that even on the smallest scales resolved so far power-law-like distributions
prevail Berghmans et al. (2021). However, with two exceptions of events barely
resolved by HRIEUV (arrows in Fig. 2), we do not see any distinguishable
transient brightenings in the diffuse regions (above the noise level). A more
rigorous statistical analysis is required to draw a final conclusion, but at
this point we consider it unlikely that individual events distributed through
a power law could create a diffuse corona as we observe here.

One might speculate that some form of MHD turbulence might lead to energy
dissipation far below the scales resolvable by current observations and by
this create a quasi-continuous heating in these regions. Of course, this would
raise the question why this should be operating in the diffuse regions while
other areas show much higher contrast in the coronal quiet Sun features. As
such a discussion would go far beyond this observational study, we refrain
from any further speculation on this.

###  4.5 Diffuse quiet-Sun corona and wave heating

If the energization of the hot, diffuse quiet-Sun corona has to be quasi-
continuous, then wave heating could also be an option. In the first
theoretical attempts to explain the hot outer atmosphere, heating through
waves was already suggested, at that time through the dissipation of sound
waves Biermann (1946); Schwarzschild (1948). On average, an energy flux of
about 100 W m-2 is required to balance the energy losses of the corona in the
quiet Sun (e.g., Withbroe & Noyes, 1977), and in general upward propagating
magneto-acoustic waves have the potential to heat the plasma in magnetically
closed structures (e.g., review by Arregui, 2015). Such waves have been
observed (e.g., Tomczyk et al., 2007) and they carry an energy flux sufficient
to heat the quiet Sun corona (e.g., McIntosh et al., 2011).

Assuming that the upward-propagating waves are generated by p-mode leakage
(e.g., de Moortel, 2009), one might expect quite a homogeneous distribution of
the energy flux into the upper atmosphere. While in the photosphere we can
expect a spatial structure on the scale of granulation, the rapid expansion of
the magnetic field with height guiding the wave flux might quickly even out
spatial inhomogeneities. However, a detailed investigation of the expansion of
the underlying magnetic field in the diffuse coronal regions would be required
before drawing any final conclusions on this.

In the time series of small sub-fields, some indications for 3-minute
oscillations might be found. A by-eye inspection of Fig. 4a-d suggests the
presence of fluctuations on a time scale of a few minutes in the diffuse
corona (panels a,b) and the bright point (c,d). If such fluctuations would be
present, that would indeed be supportive of the leakage of wave power from the
photosphere into the higher atmospheric regions seen in the diffuse corona.
Just as with the expansion of the magnetic field, a detailed analysis of a
possible presence of 3-minute oscillations in the diffuse regions will have to
be conducted in the future.

##  5 Conclusions

We analyze the diffuse quiescent corona observed by HRIEUV on board Solar
Orbiter. Here we find large patches of diffuse corona lacking resolvable
physical structuring on scales below those of a supergranule, i.e., about 20
Mm. Still, the coronal emission from these diffuse regions is of comparable
brightness to the more dynamic features like loop-like features or coronal
bright points. The spatial variability in the diffuse corona is below about
5%. The diffuse corona remains temporally stable throughout the observing
period, i.e., for at least 25 minutes.

This diffuse regime is rather commonplace within the coronal makeup,
contributing a large proportion of the emission seen at 1 MK, yet its
underlying nature is still unclear. We consider it unlikely to be connected to
features such as spicules or small loops. A power-law-like distribution of
discrete heating events seems inconsistent with our observations. We speculate
that small-scale processes like MHD turbulence or upward-propagating waves
might be energizing the diffuse regions, but at this point we cannot offer a
conclusive explanation for the nature of the diffuse regions.

The lower atmosphere in the quiet Sun shows a high degree of temporal and
spatial complexity. This is illustrated, e.g., by the cartoon picture
suggested by Wedemeyer-Böhm et al. (2009) in their Fig. 16. The diffuse region
we investigated in this study might well be related to the mixture of
structures in the inter-network, above and below the canopy domain. This would
also fit into the interpretation of Milanović et al. (2023) of a diffuse
region between network patches of the same polarity on opposite sides of a
super-granular cell.

A more extensive investigation of this diffuse component of the quiet Sun’s
corona, using more and more diverse datasets, would help to better estimate
how common this component is, including observations of plasma at other
temperatures. Also, the brightness distribution in the diffuse patches, the
lifetime and size distributions of such diffuse patches, and if wave patterns
(of whatever nature) are commonly seen in them needs further consideration.
Just as important will be studies combining EUV data with magnetic field
measurements, e.g., provided by the SO/PHI instrument Solanki et al. (2020).
Finally, there is also a clear need for studies of possible heating mechanisms
leading to such diffuse parts of the corona.

This work was supported by the International Max-Planck Research School
(IMPRS) for Solar System Science at the University of Göttingen. L.P.C.
gratefully acknowledges funding by the European Union (ERC, ORIGIN,
101039844). Views and opinions expressed are however those of the author(s)
only and do not necessarily reflect those of the European Union or the
European Research Council. Neither the European Union nor the granting
authority can be held responsible for them. Solar Orbiter is a mission of
international cooperation between ESA and NASA, operated by ESA. The EUI
instrument was built by CSL, IAS, MPS, MSSL/UCL, PMOD/WRC, ROB, LCF/IO with
funding from the Belgian Federal Science Policy Office (BELSPO/PRODEX PEA
4000134088, 4000112292, 4000117262, and 4000134474); the Centre National
d’Etudes Spatiales (CNES); the UK Space Agency (UKSA); the Bundesministerium
für Wirtschaft und Energie (BMWi) through the Deutsches Zentrum für Luft- und
Raumfahrt (DLR); and the Swiss Space Office (SSO).

## References

  * Arregui (2015) Arregui, I. 2015, Philosophical Transactions of the Royal Society of London Series A, 373, 20140261
  * Aschwanden & Shimizu (2013) Aschwanden, M. J. & Shimizu, T. 2013, ApJ, 776, 132
  * Aschwanden et al. (2000) Aschwanden, M. J., Tarbell, T. D., Nightingale, R. W., et al. 2000, ApJ, 535, 1047
  * Barczynski et al. (2017) Barczynski, K., Peter, H., & Savage, S. L. 2017, A&A, 599, A137
  * Berghmans (2002) Berghmans, D. 2002, in ESA Special Publication, Vol. 2, Solar Variability: From Core to Outer Frontiers, ed. A. Wilson, 501–508
  * Berghmans et al. (2021) Berghmans, D., Auchère, F., Long, D. M., et al. 2021, A&A, 656, L4
  * Berghmans et al. (1998) Berghmans, D., Clette, F., & Moses, D. 1998, A&A, 336, 1039
  * Biermann (1946) Biermann, L. 1946, Naturwiss., 33, 118
  * Bose et al. (2021) Bose, S., Rouppe van der Voort, L., Joshi, J., et al. 2021, A&A, 654, A51
  * Brajša et al. (2001) Brajša, R., Wöhl, H., Vršnak, B., et al. 2001, A&A, 374, 309
  * Brajša et al. (2002) Brajša, R., Wöhl, H., Vršnak, B., et al. 2002, A&A, 392, 329
  * Chitta et al. (2022) Chitta, L. P., Peter, H., Parenti, S., et al. 2022, A&A, 667, A166
  * Chitta et al. (2021a) Chitta, L. P., Peter, H., & Young, P. R. 2021a, A&A, 647, A159
  * Chitta et al. (2021b) Chitta, L. P., Solanki, S. K., Peter, H., et al. 2021b, A&A, 656, L13
  * de Moortel (2009) de Moortel, I. 2009, Space Sci. Rev., 149, 65
  * de Pontieu et al. (2007) de Pontieu, B., McIntosh, S., Hansteen, V. H., et al. 2007, PASJ, 59, S655
  * Dowdy et al. (1986) Dowdy, J. F., J., Rabin, D., & Moore, R. L. 1986, Sol. Phys., 105, 35
  * Gorman et al. (2022) Gorman, J., Chitta, L. P., & Peter, H. 2022, A&A, 660, A116
  * Hansteen et al. (2014) Hansteen, V., De Pontieu, B., Carlsson, M., et al. 2014, Science, 346, 1255757
  * Henriques et al. (2016) Henriques, V. M. J., Kuridze, D., Mathioudakis, M., & Keenan, F. P. 2016, ApJ, 820, 124
  * Hosseini Rad et al. (2021) Hosseini Rad, S., Alipour, N., & Safari, H. 2021, ApJ, 906, 59
  * Hudson (1991) Hudson, H. S. 1991, Sol. Phys., 133, 357
  * Joulin et al. (2016) Joulin, V., Buchlin, E., Solomon, J., & Guennou, C. 2016, A&A, 591, A148
  * Kayshap et al. (2018) Kayshap, P., Murawski, K., Srivastava, A. K., & Dwivedi, B. N. 2018, A&A, 616, A99
  * Klimchuk (2006) Klimchuk, J. A. 2006, Sol. Phys., 234, 41
  * Krucker & Benz (1998) Krucker, S. & Benz, A. O. 1998, ApJ, 501, L213
  * Madjarska (2019) Madjarska, M. S. 2019, Living Reviews in Solar Physics, 16, 2
  * Mandal et al. (2022) Mandal, S., Chitta, L. P., Peter, H., et al. 2022, A&A, 664, A28
  * Mandal et al. (2023) Mandal, S., Peter, H., Chitta, L. P., et al. 2023, A&A, 670, L3
  * Mandal et al. (2021) Mandal, S., Peter, H., Chitta, L. P., et al. 2021, A&A, 656, L16
  * Martínez-Sykora et al. (2018) Martínez-Sykora, J., De Pontieu, B., De Moortel, I., Hansteen, V. H., & Carlsson, M. 2018, ApJ, 860, 116
  * McIntosh et al. (2011) McIntosh, S. W., de Pontieu, B., Carlsson, M., et al. 2011, Nature, 475, 477
  * Milanović et al. (2023) Milanović, N., Chitta, L. P., & Peter, H. 2023, A&A, 673, A81
  * Müller et al. (2020) Müller, D., St. Cyr, O. C., Zouganelis, I., et al. 2020, A&A, 642, A1
  * Parnell & Jupp (2000) Parnell, C. E. & Jupp, P. E. 2000, ApJ, 529, 554
  * Pauluhn & Solanki (2007) Pauluhn, A. & Solanki, S. K. 2007, A&A, 462, 311
  * Pereira et al. (2014) Pereira, T. M. D., De Pontieu, B., Carlsson, M., et al. 2014, ApJ, 792, L15
  * Peter (2015) Peter, H. 2015, Philosophical Transactions of the Royal Society of London Series A, 373, 20150055
  * Peter et al. (2013) Peter, H., Bingert, S., Klimchuk, J. A., et al. 2013, A&A, 556, A104
  * Pontin & Hornig (2020) Pontin, D. I. & Hornig, G. 2020, Living Reviews in Solar Physics, 17, 5
  * Reale (2010) Reale, F. 2010, Living Reviews in Solar Physics, 7, 5
  * Rochus et al. (2020) Rochus, P., Auchère, F., Berghmans, D., et al. 2020, A&A, 642, A8
  * Rodger et al. (2019) Rodger, A. S., Labrosse, N., Wedemeyer, S., et al. 2019, ApJ, 875, 163
  * Samanta et al. (2019) Samanta, T., Tian, H., Yurchyshyn, V., et al. 2019, Science, 366, 890
  * Schrijver & Title (2003) Schrijver, C. J. & Title, A. M. 2003, ApJ, 597, L165
  * Schwarzschild (1948) Schwarzschild, M. 1948, ApJ, 107, 1
  * Shen (2021) Shen, Y. 2021, Proceedings of the Royal Society of London Series A, 477, 217
  * Shimojo et al. (2017) Shimojo, M., Hudson, H. S., White, S. M., Bastian, T. S., & Iwai, K. 2017, ApJ, 841, L5
  * Solanki et al. (2020) Solanki, S. K., del Toro Iniesta, J. C., Woch, J., et al. 2020, A&A, 642, A11
  * Teriaca et al. (2004) Teriaca, L., Banerjee, D., Falchi, A., Doyle, J. G., & Madjarska, M. S. 2004, A&A, 427, 1065
  * Tian et al. (2014) Tian, H., DeLuca, E. E., Cranmer, S. R., et al. 2014, Science, 346, 1255711
  * Tomczyk et al. (2007) Tomczyk, S., McIntosh, S. W., Keil, S. L., et al. 2007, Science, 317, 1192
  * Tsiropoula et al. (2012) Tsiropoula, G., Tziotziou, K., Kontogiannis, I., et al. 2012, Space Sci. Rev., 169, 181
  * Van Doorsselaere et al. (2020) Van Doorsselaere, T., Srivastava, A. K., Antolin, P., et al. 2020, Space Sci. Rev., 216, 140
  * Viall & Klimchuk (2011) Viall, N. M. & Klimchuk, J. A. 2011, ApJ, 738, 24
  * Wedemeyer-Böhm et al. (2009) Wedemeyer-Böhm, S., Lagg, A., & Nordlund, Å. 2009, Space Sci. Rev., 144, 317
  * Wiegelmann et al. (2010) Wiegelmann, T., Solanki, S. K., Borrero, J. M., et al. 2010, ApJ, 723, L185
  * Withbroe & Noyes (1977) Withbroe, G. L. & Noyes, R. W. 1977, ARA&A, 15, 363

##  Appendix A Preliminary error estimates

To better quantify and compare intensity fluctuations, we calculated the total
error, $\sigma_{\rm{total}}$, associated with the observed emission in the
following manner. First, we converted the observed, spatially averaged
intensities, $I$, from DN s-1 to DN by multiplying them by exposure time,
$t_{\rm{exp}}$, and then converted this to photon count by considering the
current high gain estimate, $a$, for the HRIEUV channel. Next, we used this to
calculate the photon noise, $\sigma_{\rm{ph}}$, by taking the square root of
the total number of photons. This photon noise was then divided by the square
root of the number of pixel values, $N$, averaged over to obtain the initial
average intensity. After converting the photon noise back into DN with the
conversion factor, $a$, the square of this amount was summed with the square
of the read-out noise, $R$, and then the square root of this sum gave the
final $\sigma_{\rm{total}}$. A final, reduced equation for this is:

$\sigma_{\rm{total}}=\sqrt{R^{2}+\frac{I\,t_{\rm{exp}}\,a}{N}}\,,$ (1)

Here, $\sigma_{\rm{total}}$ is in DN, $R=\,$2 DN, $I$ is in DN s-1,
$t_{\rm{exp}}$ is in s, and $a=\,$6.85 DN photon-1. For a typical bright point
with an average intensity of 3450 DN, we find a maximum error of 27 DN. For
the diffuse regions, a typical average intensity of 2765 DN has a maximum
error of 23 DN. Due to the current early stage of the Solar Orbiter mission,
the error estimation calculation is not yet fully clarified. So, we would
expect that any reasonable calculated errors should be comparable to the root-
mean-square (RMS) fluctuations, as these are expected as an upper limit. For
this observation, the maximum errors estimated in the manner detailed above
are comparable to the high-frequency (1 min) RMS fluctuations which are of a
typical order of a few percent. Thus, we consider our error estimation to be
reasonable.

##  Appendix B Region of loop-like features

In the main text we concentrated on the properties of the diffuse region and
compared these to the coronal bright point. Here we concentrate on an area
with loop-like features (green box iii in Fig. 1). This region shows a spatio-
temporal structuring of a higher degree than the diffuse region but much less
variability than the bright point (see Fig. 2). In this appendix we briefly
compare the properties of this region with loop-like features to the diffuse
region and the bright point.

The time averaged intensity map of this region shows distinct structures, even
though it is not simple to attribute a particular nature to these features.
The left panel of Fig. 7 shows a zoomed view of this area. In contrast to the
diffuse region (Fig. 4i) it shows clear elongated structures. For simplicity
we call these loop-like features.

The fluctuations seen in this region are comparable or larger than in the
diffuse region but much less pronounced than in the bright point. This is
evident when comparing the time variability near the center of the loop-like
features (Fig. 7e) to the corresponding variability seen in the other regions
(Fig. 4a-d).

The individual snapshots in this region of loop-like features are noisy, but
still show distinct structures. For example, the first snapshot in Fig. 8
shows some short-lived individual compact brightenings and the last snapshot
some elongated structure (at low contrast) from the bottom left to the top
right. This presence of spatial structures is pointedly different from the
snapshots of the diffuse region (top row of Fig. 5). The levels of the count
rates are comparable in the diffuse region and the loop-like features, hence
noise cannot explain the absence of structures in the diffuse region.

From this discussion we conclude that the region with loop-like features (box
iii in Fig. 1) hosts small-scale distinct features that cannot be identified
in the diffuse region, despite the count rates being comparable. We can
speculate if this region with loop-like features is in a transition phase from
diffuse to more active, like the bright point or vice versa. To draw solid
conclusions on this a more comprehensive study of loop-like features would be
required, in particular including data on the magnetic field configuration in
the photosphere below.

Figure 7: Intensity fluctuations in the region with loop-like features. The
layout of this figure corresponds to Fig. 4. The left panel shows the zoom
into the green box in Fig. 1b (region iii). Within this region a smaller sub-
field e is outlined. This has the same area as the sub-fields in Fig. 4 i and
ii. The spatially-averaged relative (left y-axis) and absolute (right y-axis)
intensity for sub-field e is plotted as a function of time in the right panel.
The vertical bar represents the maximum error for the time-series. See
Appendix B. Figure 8: Snapshots of region with loop-like features without
time-averaging. This figure corresponds to Fig. 5. The snapshots are shown at
three times as indicated by the time stamps. Images are commonly scaled to the
same minimum and maximum values of intensity. Box e has the same meaning as in
Fig. 7. See Appendix B.


# Detection of Propaganda Techniques in Visuo-Lingual Metaphor in Memes

Sunil Gundapu
Language Technologies Research Centre
KCIS, IIIT Hyderabad
Telangana, India
<EMAIL_ADDRESS>
&Radhika Mamidi
Language Technologies Research Centre
KCIS, IIIT Hyderabad
Telangana, India
<EMAIL_ADDRESS>

###### Abstract

The exponential rise of social media networks has allowed the production,
distribution, and consumption of data at a phenomenal rate. Moreover, the
social media revolution has brought a unique phenomenon to social media
platforms called Internet memes. Internet memes are one of the most popular
contents used on social media, and they can be in the form of images with a
witty, catchy, or satirical text description. In this paper, we are dealing
with propaganda that is often seen in Internet memes in recent times.
Propaganda is communication, which frequently includes psychological and
rhetorical techniques to manipulate or influence an audience to act or respond
as the propagandist wants. To detect propaganda in Internet memes, we propose
a multimodal deep learning fusion system that fuses the text and image feature
representations and outperforms individual models based solely on either text
or image modalities.

##  1 Introduction

Memes are usually an image superimposed with a text description and are used
to express a range of ideas such as humour, embarrassment, hate, propaganda,
and even more emotions. The word meme was introduced by Dawkins [1976], and
initially, it refers to an idea, behaviour, or style that disseminates from
person to person within a culture. With the growth of the social network
community, the participation of memes on online social networks has increased
tremendously in recent years. In the age of the internet, it has been adopted
to refer to a part of the culture, typically a joke, rumour, or catchphrase,
which gains influence through online transmission [Davison, 2012]. They spread
among people using social media platforms, blogs, instant messaging
applications, emails, and forums in their simplest form. Content-wise, they
usually include unconventional news, dialogues, catchlines, images, or videos.

According to Nieubuurt [2021], Internet memes are one of the more popular
contents used in online disinformation campaigns. An essential aspect of a
problem that is often overlooked in social media platforms is the mechanism
through which false information is being communicated, which is utilising
propaganda techniques. These techniques are deliberately designed
communication that invites people to respond emotionally, immediately, and in
one way or another. These were initially high on advertising and public
relations, news and journalism, politics, entertainment but are now appearing
on all aspects of daily life, and more importantly on social media platforms.

Propaganda techniques comprise of psychological and rhetorical strategies,
ranging from logical fallacies —[such as black-and-white fallacy (present only
two possibilities among many), straw man (misconstrue person’s actions or
opinion), red herring (introduce irrelevant data to distract), and
whataboutism] —to influencing audience emotions —[such as using slogans,
loaded language, appeal to authority, flag-waving, and clichés]. Identifying
logical fallacies is challenging because, at first glance, the argument may
seem correct and objective. However, we can spot them through careful
observation and analysis. Another set of techniques use emotional language to
motivate the viewer to accept the speaker only based on the emotional bond
being created, which will stop any rational analysis of the argument.

In recent times, propagandists are generating Internet memes by using
propaganda techniques to influence viewers or readers to believe in someone or
something. American scientist Robert Peter W. Singer describes in his book
"LikesWar" that social media propaganda and misinformation are emerging as a
new weapon in the modern
warfare111https://knowledge.wharton.upenn.edu/article/singer-weaponization-
social-media/. Therefore, it is essential to find out propaganda campaigns in
Internet memes to stop spreading them. So, in this paper we aims to build a
multimodal fusion system that can identify the Internet memes in which
propaganda techniques are used. The system takes pair (text, image) of an
Internet meme as an input to determine which of the propaganda techniques is
used in the text and visual content of the meme.

##  2 Related Work

Evolution of Propaganda: Propaganda techniques are not new. The term
propaganda was used in the early 17th century and was first used to propagate
Catholic beliefs and practices in the New World and later to manipulate people
in public gatherings such as festivals, games, and theaters [ASamuel C Woolley
and Philip N Howard, 2018]. But in the present technological world, this
propaganda has progressed to computational propaganda [Bolsover and Howard,
2017b], where information is dispensed through technology such as social media
platforms so that it is possible to reach well-targeted communities at high
speeds. This propaganda shared on these platforms can be text, visual or text-
vision combinations. Internet memes are critical in spreading multimodal
propaganda on social media platforms [DiRESTA, 2018]. The present social media
ecosystem and virality bots allow memes to spread effortlessly, switching from
one target group to another. Currently, efforts to curb the spread of such
memes are focused on analyzing social media networks and searching for fake
accounts and bots to lessen the spread of such content[Cresci et al., 2017,
Yang et al., 2019b].

Propaganda in Text Modality:  In the natural language processing community,
much research is done on propaganda by analyzing textual content [Barrón-
Cedeño et al., 2019, Rashkin et al., 2017, Martino et al., 2019]. Rashkin et
al. [2017] studied the propaganda at documet level by creating TSHP-17
dataset. This dataset developed with the help of English Gigaword corpus and
labelled with four classes: satire, trusted, hoax, and propaganda. On this
dataset, they trained a logistic regression model with n-gram level word
representations. To analyze the propaganda at sentence level Barrón-Cedeño et
al. [2019] developed a new QProp dataset with two labels: propaganda, and non-
propaganda. On this binary labeled corpora they trained various machine
learning models like logistic regression, SVMs to discriminate propaganda from
non-propaganda datapoints.

In a similar fashion, Habernal et al. [2017] created a dataset with 1300 data
points with five propaganda beliefs, including irrelevant authority, ad
hominem, and red herring, which are directly associated with propaganda
techniques. Martino et al. [2019] examines the propaganda at the fragment
level. For this, they created a PTC dataset by annotating the news articles
with 18 propaganda techniques. There are two types of experiments done on this
dataset. The first one is a two-class classification: Whether the given input
news article using any of 18 techniques or not. Another task is multi-label
classification and span detection: For the input text, find the span of text
fragments where the propaganda techniques are used and identify the type of
propaganda technique. Recently, Martino et al. [2020] on detection of
computational propaganda from the perspective of NLP and Network Analysis
mentioned the need for collective efforts between these communities. There is
also a dedicated Big Data Journal on Political Big Data and Computational
Propaganda [Bolsover and Howard, 2017a].

Propaganda in Multimodality:  Originally, propaganda campaigns have appeared
in text modality, but nowadays, they appear in every possible modality.
Propaganda techniques are more accessible to spot in text modality than
multimodalities such as memes because contextual information related to the
propaganda can be included in more than one of the multiple modalities.

To understand aspects of visual propaganda, Seo [2014] analyzes the social
media tweet images posted by the Hamas’ Alqassam Brigades and Israel Defense
Forces during the 2012 Gaza conflict. By selecting 10 Youtube videos, Abd
Kadir et al. [2016] studied the comprehend relationships between these videos
and propaganda techniques and people’s emotions. Simultaneously they analyze
that how these videos are influencing people’s emotions with the help of
propaganda techniques.

Volkova et al. [2019] prepared a dataset with 50K twitter posts consisting of
memes annotated with six labels: propaganda, disinformation, hoaxes,
clickbait, conspiracies, and satire. They developed a multimodal approach for
this problem by considering the textual, linguistic characteristics, and
visual features. Glenski et al. [2019] proposed two classification tasks to
explore the multilingual content for deception detection. Both tasks are
intended to identify the category of a social media post, but the first task
has four output categories (propaganda, conspiracy, hoax, or clickbait), and
the second task has five output categories with an extra category of
disinformation.

Some quality works done on multimodality content before this propaganda
problem , such as the spread of false information [Dupuis and Williams, 2019],
hateful memes identification [Kiela et al., 2020, Lippe et al., 2020, Das et
al., 2020, Gundapu and Mamidi, 2020], antisemitism [Chandra et al., 2021].

Multimodal Models and Fustion Techniques: Facebook Hateful Memes Challenge has
been very helpful in developing different types of multimodal models and to
fine-tune the state-of-art multimodal transformer models such as ViLBERT [Lu
et al., 2019], Multimodal Bitransformers [Kiela et al., 2020], and VisualBERT
[Li et al., 2019]. And also Vidgen et al. [2019] pointed that memes make
perfect sense when both text and image content are taken into account. By
considering this point, many authors [Baltrusaitis et al., 2019][Yang et al.,
2019a][Gallo et al., 2018] have explored different multimodal fusion
strategies to combine modalities.

##  3 Dataset

In this paper, we predominately focused on the task of propaganda techniques
identification in Internet memes. We formulated this task as a multi-label
classification problem because each input is labeled with more than one
propaganda technique. We took the dataset from the 15th International Workshop
on Semantic Evaluation (SemEval-2021) for this task [Dimitrov et al., 2021].
The dataset comprises of 950 pair (text, image) of social media posts, and
each pair labeled with propaganda techniques. In the input pair (text and
image), the text is extracted from the meme using OCR, and the image is a
meme. We used 22 propaganda techniques for our task and put them in Table 1. A
more detailed list of propaganda techniques with definitions and examples is
available
online222https://propaganda.math.unipd.it/semeval2021task6/definitions22.html
and in appendix, we are adding few meme examples.

This dataset contains 687 pairs of data points for model training, 63 data
points for model hyper-parameter tuning, and 200 data points for model
testing. Table 1 shows the proportion of propaganda techniques in the train,
valid, and test sets.

|  Appeal to Emotions |  Appeal to authority |  Appeal to fear |  Bandwagon |  Black-and-white Fallacy |  Causal Oversimplification |  Doubt |  Exaggeration |  Flag-waving |  Glittering generalities |  Loaded Language |  Straw Man |  Name Calling |  Obfuscation |  Red Herring |  Reductio ad hitlerum |  Repetition |  Slogans |  Smears |  Thought-endig cliché |  Transfer |  Whataboutism
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Train | 68 | 19 | 66 | 2 | 19 | 31 | 61 | 60 | 36 | 84 | 360 | 32 | 252 | 5 | 2 | 15 | 10 | 45 | 450 | 20 | 61 | 47
Dev | 3 | 3 | 7 | 2 | 0 | 1 | 9 | 8 | 7 | 4 | 32 | 5 | 30 | 0 | 0 | 1 | 3 | 3 | 47 | 1 | 11 | 6
Test | 19 | 13 | 18 | 1 | 7 | 4 | 41 | 31 | 12 | 24 | 100 | 3 | 65 | 2 | 5 | 7 | 1 | 22 | 105 | 6 | 23 | 14
Table 1: List of propaganda techniques and their count in train, development,
and test sets.

##  4 Proposed Approaches

Individual Modality Models: In the beginning, we started to examine our task,
“Identification of propaganda techniques in Internet memes," with individual
modalities (text or image). For text data, we have developed various Machine
Learning (ML) and Deep Learning (DL) models with different word embeddings.
Nevertheless, BERT and RoBERTa gave superior results for this textual modality
than the ML and DL models with the Glove [Pennington et al., 2014] and
FastText [Bojanowski et al., 2016] embeddings. On the image data, we
experimented with CNN and pre-trained image classification models. Pre-trained
models gave significantly better results than traditional CNN models.And When
comparing the results of individual modalities, text modality models gave more
effective results than image modality models. This is because there is more
information about the propaganda in the textual data than in the visual data
in the memes.

Cross-Modality Multimodal Models:  After analyzing the results of individual
modalities, we started to fine-tune cross-modality by multimodal (Vision-and-
Language) pretraining models like UNITER [Chen et al., 2019], VilBERT [Lu et
al., 2019], and VisualBERT [Li et al., 2019], and these pre-trained models
performed better than individual modality models. Following individual and
cross-modality models, we explored multimodal fusion approaches to combine
information from text and vision input modalities in a principled way.

###  4.1 Multimodal Fusion System

In a multimodal fusion setting, ML/DL models are trained on separate
modalities and integrated simultaneously. When individual modality features
are merged into output layers, this is called Late Fusion. On the other hand,
there is an inversion of late fusion called Early Fusion to fuse modalities.
In this fusion, features are incorporated at the input level before being
given in the model. We experimented on our task by fusing text encoder
(encodes text data) and image encoder (encodes the image data) using early
fusion and late fusion methods. After observing the results of these methods,
merging modalities at their individual deepest (or early stage) attributes is
not mandatorily the most appropriate way to solve our propaganda
identification multimodal task.

In our work, we used an idea “MFAS: Multimodal Fusion Architecture Search" of
considering features collected from the hidden layers of individual modalities
that could effectively enhance performance with respect to only utilising a
single fusion of late (or early) features. Figure 1 describes the structure of
our proposed multimodal fusion system for propaganda techniques identification
in Internet memes with RoBERTa text encoder, pre-trained VGG-19 image encoder,
and multimodal fusion architecture search [Pérez-Rúa et al., 2019] module.

Figure 1: Multimodal Fusion Architecture

Given a pair (text and image) for a social media post, we make use of
Transformer based models to encode the text extracted from the Internet meme.
We use the pre-trained image classification models to encode the image.
Further, the MFAS module fuses the text and image encodings. Next, the fused
encodings are transformed using a fully connected network to identify the
propaganda techniques. The complete model is trained end-to-end utilising
back-propagation. The principal motive of fine-tuning the pre-trained models
on the datasets for our multimodal classification task is that our datasets
are small. Below we describe each module in the architecture in detail.

###  4.2 Text Preprocessing

  1. 1.

Conversion of chat contractions: Chat words/phrases are widely used on social
networks to express emotions and are very helpful in identifying context. We
have created a word contractions dictionary with 250 chat words to convert
these chat words to their complete form. Examples: YOLO → you only live once,
ASAP → as soon as possible.

  2. 2.

URLs removal: The OCR extracted text consists of URLs like memegenerator.net,
etc. We removed those links since they do not give any vital information.

  3. 3.

Conversion of elongated words: With the help of regular expressions,
transformed the elongated words to their original form. Examples: Nooooo → No,
suuuppperrr → super.

  4. 4.

Using ekphrasis [Baziotis et al., 2017] library, normalizes the time, date,
and numbers into a standard format. This library does hashtag splitting and
spelling correction. We removed the non-alpha-numeric characters, punctuation
marks, and non-ASCII glyphs from the text data.

###  4.3 Text Encoder

The preprocessed text data tokenized and then forwarded to the text encoder
module. We tried the BERT [Devlin et al., 2019], XLNet [Yang et al., 2019c],
and RoBERTa [Liu et al., 2019] pre-trained transformer models in the text
encoder module to encode the text because they have been shown to give
incredible results in multiple NLP classification tasks.

BERT (Bidirectional Encoder Representations from Transformers) employs a
multi-layer bidirectional transformer encoder to learn deep hidden
bidirectional representations. It has self-attention layers that perform self-
attention on input text from both directions. This technique allows the BERT
model to know the context of a word in the sentence based on the words around
it. We used the BERT base case model for our task, pre-trained on the large
unlabeled Book corpus and the entire Wikipedia.

RoBERTa makes use of Transformer [Vaswani et al., 2017], and it is a robustly
optimized approach for pretraining NLP models that improve on BERT. RoBERTa
was trained with more data, more iterations, larger batch sizes, and learning
rates than the BERT. And it eliminates the next-sentence pretraining objective
task from the BERT model to boost the training procedure and introduces a new
idea in its architecture called dynamic masking. In this idea, masked tokens
will change during the model training. For our task, we used the RoBERTa base
model.

XLNet is a bidirectional transformer autoregressive model that uses a better
training methodology and a more extensive dataset to achieve better results
than BERT. In pretraining, it integrates the following two techniques: (1)
State-of-the-art Autoregressive model and (2) Transformer XL. Furthermore,
XLNet presents a permutation language modeling technique to predict all tokens
in random order instead of sequential order. This idea helps the XLNet learn
bidirectional relationships among words and handle the dependencies.

For better contextual encodings, we experimented on text data with the above
explained three transformer models. However, the RoBERTa gave slightly better
results than the BERT and XLNet.

###  4.4 Image Encoder

In our dataset, input images (memes) are in different shapes and sizes. So
before forwarding to any CNN model, we resize all the images to $224\times
224$ dimensions. After this step, applied few image transformation techniques
like rotation, flipping, and cropping on the scaled dataset. We extracted
image representations twice from the image encoder. The first step collected
the representations from the second last output layer of the CNN model and
then fused them with text encodings. Next, image encoder final layer outputs
concatenated with text encodings and previous step fused encodings.

An image encoder experimented the various pre-trained CNN architectures like:
VGG-19 [Simonyan and Zisserman, 2015], InceptionV3 [Szegedy et al., 2016],
Resnet-152 [He et al., 2016], DenseNet-161 [Huang et al., 2017], and
InceptionResNetV2 [Szegedy et al., 2017]. Among all these pre-trained models
VGG-19 and Resnet-152 gave considerable results for our task.

###  4.5 Fusion Module

To join the representations procured from the text encoder and image encoder
modules, T, F’, and I, we examine with various techniques: (i) Concatenation
of both modalities output representations, (ii) Early and Late Fusion, and
(iii) MFAS.

#### Early Fusion:

Initially started with a widely used state-of-the-art early fusion technique
to fuse both modalities. Early fusion technique (See figure 2) is also known
as feature-level fusion technique, which concatenates the embeddings from text
and image modalities as input representations for classifiers. This technique
can be conveyed as follows:

$\begin{split}X_{early}=f(U_{1},.....,U_{m},V_{1},......,V_{n})\end{split}$
(1)

Here an aggregated representation of the $X_{early}$ attributes is calculated
by the function f that connects the individual attributes.

#### Late Fusion:

After early fusion, we experimented with the decision-level late fusion
technique (See figure 2). This technique fuses the output level
representations of text and image encoder and computes a late fusion score for
classifiers. Late fusion technique can be conveyed as follows:

$\begin{split}X_{late}=g(f_{1}(U_{1}),...,f_{m}(U_{m}),\\\
f_{m+1}(V_{1})....,f_{m+n}(V_{n}))\end{split}$ (2)

Here functions $f_{1},...,f_{m+n}$ are applied to each individual attribute
and function g is used to integrate every individual decision by
$f_{1},..,f_{m+n}$.

(a) Early Fusion Technique

(b) Late Fusion Technique

Figure 2: Fusion Techniques

#### Multimodal Fusion Architecture Search (MFAS):

Next examined a model-level fusion technique called MFAS, which is a
compromise between the two modalities. It concatenates the hidden layer
representations from different modalities. As shown in model architecture
(figure 1), on our task, this technique initially concatenates the text
encoder output representations (T) with image encoder intermediate hidden
layer representations (H) then applies a non-linearity sigmoid function.

$\begin{split}F^{{}^{\prime}}=\sigma(T\oplus H)\end{split}$ (3)

Next, it fuses the output (F’) with text predictions (T) and image predictions
(I) along with a non-linearity sigmoid function (F).

$\begin{split}F=\sigma(T\oplus F^{\prime}\oplus I)\end{split}$ (4)

###  4.6 Classifier

Followed by the fusion model, we have constructed a fully connected network
that takes the input from the fusion model. The fully connected network
consists of two dense layers with hidden unit sizes of 768 and 384 and finally
the output layer with a size of 22 output units has a sigmoid function.

##  5 Experiments and Results

This experiments section explains model implementation details, hyper-
parameter tuning, results of individual text and image modalities classifiers
and multimodal + fusion classifiers.

###  5.1 Hyper-parameter Settings and Implementation Details for
Reproducibility

The following experimental settings are used for our work. We trained all the
models using the train set and used the validation dataset to find the right
set of hyper-parameters. We experimented with different image encoder
intermediate layer outputs to get better results. However, Block-2 output (H)
gave the best results for our work as compared to other blocks’ output.

For all the experiments, we used the Adam optimization algorithm. To increase
the model speed and performance, used dropout layers with the probability of
0.2. In the dense layers used the ReLU activation function. And, we used
PyTorch [Paszke et al., 2019] and Keras [Chollet et al., 2015] frameworks for
building deep learning models, scikit-learn [Pedregosa et al., 2011] for ML
models. To fine-tune the transformer models used the PyTorch-based
HuggingFace333https://huggingface.co/ transformer library.

Modality Approach Metrics Precision Recall F1-Score FastText with LSTM 0.5809
0.4679 0.5183 Glove + BiLSTM + Attention 0.5910 0.4702 0.5237 Only Text BERT
with Dense 0.5958 0.4780 0.5295 XLNet with Dense 0.5936 0.4738 0.5270 RoBERTa
with Dense 0.6004 0.4901 0.5363 Inception-V3 0.4868 0.4571 0.4841 Only Image
Resnet-152 0.5062 0.4618 0.4925 VGG-19 0.4989 0.4656 0.5038 UNITER 0.5841
0.4986 0.5288 Cross-Modality VilBERT 0.5892 0.5002 0.5348 VisualBERT 0.5965
0.5034 0.5404 Concatenation 0.5684 0.4576 0.4983 Text and Image Early Fusion
0.5703 0.4693 0.5058 Late Fusion 0.6016 0.5076 0.5407 MFAS 0.6214 0.5077
0.5698

Table 2:  Comparison of various modalities classifiers results

###  5.2 Results and Analysis of Individual Modalities Classifiers

We started experiments for both (text and image) modalities with ML algorithms
and TF-IDF word vectors for baseline model results. After that, we
experimented on text modality with five well-known pre-trained text embedding-
based classifiers and three pre-trained CNN-based image classifiers. We used
Glove, FastText, BERT, XLNet, and RoBERTa for the individual text
classification. And we used the VGG-19, InceptionV3, Resnet-152, Densenet-161,
InceptionResNetV2 for image classification.

The comparative f1-scores of individual modalities are presented in Table 2.
After observing the classification results of individual modalities, we made
the following observations:

  * •

Text modality classifiers gave encouraging results, but the image modality
classifiers provide lesser f1-scores. We suspect this because images related
to propaganda techniques are usually news articles, memes, or screenshots
(sometimes) that generally do not convey any spatial information.

  * •

Among text modality classifiers, Transformer based approaches worked better
than the shallow word embedding methods like Glove and FastText. Transformer-
based BERT, XLNet, and RoBERTa gave superior results on our task. However,
RoBERTa model score surpassed BERT and XLNet models. Among the image modality
classifiers, pre-trained VGG-19 gave better results than other pre-trained
models.

###  5.3 Results and Analysis of Multimodal Fusion Classifiers

In this section, we presented results and analysis of various fusion
approaches for our proposed multimodal classifier. Based on the results of
individual modalities (in Table 2), we noticed that transformer-based RoBERTa
is the best text encoder and VGG-19 is the best image encoder for our dataset.
Therefore, we did multimodal fusion experiments with these two encoders only.
We presented the results of this multimodal classifier with different fusion
techniques in Table 2. Moreover, we compared the results of our proposed
multimodal system with individual and cross-modality models.

  * •

All the proposed multimodal fusion models for our task performed better than
other models by a considerable margin. Cross-modality by pre-training
multimodal models are much better than individual modality models. And these
models are working best when the visual information in the memes is not
covered by textual information.

  * •

In the multimodal late fusion architecture, the Image encoder gives only
prediction-level features that do not capture complex information, such as
semantic concepts of faces, animals, trees, etc. Due to this, when we
concatenate image encodings with text encodings, it sometimes misleads the
fusion model outputs.

  * •

Concatenation of encoders features technique and early fusion technique does
not meet the expectations. They only give the approximate individual image
modality model results.

  * •

Decision level late fusion and MFAS approaches efficiently fused the spatial
information from images and semantic information from text and performed
better than all other multimodal systems. However, MFAS multimodal system
outperforms the late fusion approach.

##  6 Error Analysis

For our task, we mostly lean towards pre-trained models because our dataset is
tiny. Transformer-based models performed very well on our task and gave
considerable results than we expected. However, in some cases, transformer
models are incorrectly predicted when the input text length is short.
Furthermore, pre-trained CNN classification models were mainly confused for
our task in the following contexts to extract vital spatial image information.
(i) Few memes are designed with the clubbing of multiple images. In this case,
pre-trained models are stumbled to gather information from them. (ii) Some
users take screenshots of memes posted by other users and post them again on
social media. Lack of additional contextual information in such posts systems
making false predictions. (iii) Sometimes, the entire meme is covered with
text, making it difficult for CNN models to recognize spatial features in
images.

We have analyzed that combining encoder representations from multiple
modalities can help identify propaganda techniques in memes. Though, in some
cases, we realize that noise in one modality leads to a total
misclassification. While doing experiments, we observed that the dataset used
for our task has severe class imbalance due to the heterogeneous frequency of
various types of propaganda techniques in real life. “Bandwagon, Reduction ad
Hitlerum”, “Appeal to Authority”, “Black and White Fallacy”, “Whataboutism,
StrawMen, RedHerring”, and “Thought Terminating Cliche” classes samples are
significantly less in the dataset compare to other classes samples. To tackle
this class imbalance problem, we explored with different data under/over
sampling techniques like SMOTE [Chawla et al., 2002], Tomek Links [Tomek,
1976], Near Miss [Zhang and Mani, 2003] for textual data, tried the position
and colour augmentation techniques [Perez and Wang, 2017] for image data. Also
we used the class weight [Guo et al., 2008] approach which give different
weights to both the majority and minority classes. Experimented with an
enhanced version of Cross-Entropy Loss that is focal loss [Lin et al., 2017]
that seeks to manage the problem of class imbalance by assigning more weight
to rough or easily misclassified samples and to low-weight easy samples.

##  7 Conclusion and Future Work

In this work, we first provided a systematic study on the problem of
identification of propaganda techniques in Internet memes. After that, we
explored this problem with a multimodal fusion architecture. In this
architecture, with the help of the MFAS technique, we fused the text features
extracted using RoBERTa model and image features using pre-trained VGG-19. We
examined our problem with single modality as well as multimodality classifiers
and noticed that fusing features from various modalities enhance the
performance. Our multimodal fusion approach achieves 0.5698 micro f1-score on
the test data, which is a considerable result. We hope these results will
accelerate further research works in this direction.

## References

  * Abd Kadir et al. [2016] S. Abd Kadir, A. Lokman, and T. Tsuchiya.  Emotion and techniques of propaganda in youtube videos.  _Indian Journal of Science and Technology_ , Vol (9):1–8, 12 2016.  doi: 10.17485/ijst/2016/v9iS1/106841.
  * ASamuel C Woolley and Philip N Howard [2018] ASamuel C Woolley and Philip N Howard.  _Computational propaganda: political parties, politicians, and political manipulation on social media_.  Oxford University Press, 2018.
  * Baltrusaitis et al. [2019] T. Baltrusaitis, C. Ahuja, and L.-P. Morency.  Multimodal machine learning: A survey and taxonomy.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 41:423–443, 2019.
  * Barrón-Cedeño et al. [2019] A. Barrón-Cedeño, I. Jaradat, G. Martino, and P. Nakov.  Proppy: Organizing the news based on their propagandistic content.  _Information Processing & Management_, 56, 05 2019.  doi: 10.1016/j.ipm.2019.03.005.
  * Baziotis et al. [2017] C. Baziotis, N. Pelekis, and C. Doulkeridis.  Datastories at semeval-2017 task 4: Deep lstm with attention for message-level and topic-based sentiment analysis.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 747–754, Vancouver, Canada, August 2017. Association for Computational Linguistics.
  * Bojanowski et al. [2016] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov.  Enriching word vectors with subword information.  _arXiv preprint arXiv:1607.04606_ , 2016.
  * Bolsover and Howard [2017a] G. Bolsover and P. Howard.  Computational propaganda and political big data: Moving toward a more critical research agenda.  _Big Data_ , 5:273–276, 12 2017a.  doi: 10.1089/big.2017.29024.cpr.
  * Bolsover and Howard [2017b] G. Bolsover and P. N. Howard.  Computational propaganda and political big data: Moving toward a more critical research agenda.  _Big data_ , 5 4:273–276, 2017b.
  * Chandra et al. [2021] M. Chandra, D. Pailla, H. Bhatia, A. Sanchawala, M. Gupta, M. Shrivastava, and P. Kumaraguru.  “subverting the jewtocracy”: Online antisemitism detection using multimodal deep learning.  pages 148–157, 06 2021.  doi: 10.1145/3447535.3462502.
  * Chawla et al. [2002] N. Chawla, K. Bowyer, L. Hall, and W. P. Kegelmeyer.  Smote: Synthetic minority over-sampling technique.  _J. Artif. Intell. Res._ , 16:321–357, 2002.
  * Chen et al. [2019] Y.-C. Chen, L. Li, L. Yu, A. E. Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu.  Uniter: Learning universal image-text representations.  _ArXiv_ , abs/1909.11740, 2019.
  * Chollet et al. [2015] F. Chollet et al.  Keras, 2015.  URL https://github.com/fchollet/keras.
  * Cresci et al. [2017] S. Cresci, R. Pietro, M. Petrocchi, A. Spognardi, and M. Tesconi.  The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race.  04 2017.  doi: 10.1145/3041021.3055135.
  * Das et al. [2020] A. Das, J. S. Wahi, and S. Li.  Detecting hate speech in multi-modal memes.  _ArXiv_ , abs/2012.14891, 2020.
  * Davison [2012] P. Davison.  _The Language of Internet Memes_ , pages 120–134.  01 2012.  ISBN 0814764061.
  * Dawkins [1976] R. Dawkins.  _The Selfish Gene_.  Oxford University Press, 1976.
  * Devlin et al. [2019] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT_ , 2019.
  * Dimitrov et al. [2021] D. Dimitrov, B. B. Ali, S. Shaar, F. Alam, F. Silvestri, H. Firooz, P. Nakov, and G. D. S. Martino.  Semeval-2021 task 6: Detection of persuasion techniques in texts and images.  _arXiv preprint arXiv:2105.09284_ , 2021.
  * DiRESTA [2018] R. DiRESTA.  Computational propaganda: If you make it trend, you make it true.  _The Yale Review_ , 106(4):12–29, 2018.  doi: https://doi.org/10.1111/yrev.13402.  URL https://onlinelibrary.wiley.com/doi/abs/10.1111/yrev.13402.
  * Dupuis and Williams [2019] M. Dupuis and A. Williams.  The spread of disinformation on the web: An examination of memes on social networking.  _2019 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI)_, pages 1412–1418, 2019.
  * Gallo et al. [2018] I. Gallo, A. Calefati, S. Nawaz, and M. K. Janjua.  Image and encoded text fusion for multi-modal classification.  _2018 Digital Image Computing: Techniques and Applications (DICTA)_ , pages 1–7, 2018.
  * Glenski et al. [2019] M. Glenski, E. Ayton, J. Mendoza, and S. Volkova.  Multilingual multimodal digital deception detection and disinformation spread across social platforms.  _ArXiv_ , abs/1909.05838, 2019.
  * Gundapu and Mamidi [2020] S. Gundapu and R. Mamidi.  Gundapusunil at SemEval-2020 task 8: Multimodal memotion analysis.  In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_ , pages 1112–1119, Barcelona (online), Dec. 2020. International Committee for Computational Linguistics.  URL https://www.aclweb.org/anthology/2020.semeval-1.147.
  * Guo et al. [2008] X. Guo, Y. Yin, C. Dong, G. Yang, and G. Zhou.  On the class imbalance problem.  In _2008 Fourth International Conference on Natural Computation_ , volume 4, pages 192–201, 2008.  doi: 10.1109/ICNC.2008.871.
  * Habernal et al. [2017] I. Habernal, R. Hannemann, C. Pollak, C. Klamm, P. Pauli, and I. Gurevych.  Argotario: Computational argumentation meets serious games.  07 2017.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778, 2016.
  * Huang et al. [2017] G. Huang, Z. Liu, and K. Q. Weinberger.  Densely connected convolutional networks.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2261–2269, 2017.
  * Kiela et al. [2020] D. Kiela, H. Firooz, A. Mohan, V. Goswami, A. Singh, P. Ringshia, and D. Testuggine.  The hateful memes challenge: Detecting hate speech in multimodal memes.  _ArXiv_ , abs/2005.04790, 2020.
  * Li et al. [2019] L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang.  Visualbert: A simple and performant baseline for vision and language.  _ArXiv_ , abs/1908.03557, 2019.
  * Lin et al. [2017] T.-Y. Lin, P. Goyal, R. B. Girshick, K. He, and P. Dollár.  Focal loss for dense object detection.  _2017 IEEE International Conference on Computer Vision (ICCV)_ , pages 2999–3007, 2017.
  * Lippe et al. [2020] P. Lippe, N. Holla, S. Chandra, S. Rajamanickam, G. Antoniou, E. Shutova, and H. Yannakoudakis.  A multimodal framework for the detection of hateful memes, 12 2020.
  * Liu et al. [2019] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv_ , abs/1907.11692, 2019.
  * Lu et al. [2019] J. Lu, D. Batra, D. Parikh, and S. Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In _NeurIPS_ , 2019.
  * Martino et al. [2019] G. D. S. Martino, S. Yu, A. Barrón-Cedeño, R. Petrov, and P. Nakov.  Fine-grained analysis of propaganda in news articles.  In _EMNLP/IJCNLP_ , 2019.
  * Martino et al. [2020] G. D. S. Martino, S. Cresci, A. Barrón-Cedeño, S. Yu, R. D. Pietro, and P. Nakov.  A survey on computational propaganda detection.  In _IJCAI_ , 2020.
  * Nieubuurt [2021] J. T. Nieubuurt.  Internet memes: Leaflet propaganda of the digital age.  _Frontiers in Communication_ , 5:116, 2021.  ISSN 2297-900X.  doi: 10.3389/fcomm.2020.547065.  URL https://www.frontiersin.org/article/10.3389/fcomm.2020.547065.
  * Paszke et al. [2019] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc., 2019.  URL http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf.
  * Pedregosa et al. [2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011\.
  * Pennington et al. [2014] J. Pennington, R. Socher, and C. D. Manning.  Glove: Global vectors for word representation.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, 2014.  URL http://www.aclweb.org/anthology/D14-1162.
  * Perez and Wang [2017] L. Perez and J. Wang.  The effectiveness of data augmentation in image classification using deep learning.  _ArXiv_ , abs/1712.04621, 2017.
  * Pérez-Rúa et al. [2019] J.-M. Pérez-Rúa, V. Vielzeuf, S. Pateux, M. Baccouche, and F. Jurie.  Mfas: Multimodal fusion architecture search.  _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6959–6968, 2019.
  * Rashkin et al. [2017] H. Rashkin, E. Choi, J. Jang, S. Volkova, and Y. Choi.  Truth of varying shades: Analyzing language in fake news and political fact-checking.  pages 2931–2937, 01 2017.  doi: 10.18653/v1/D17-1317.
  * Seo [2014] H. Seo.  Visual propaganda in the age of social media: An empirical analysis of twitter images during the 2012 israeli–hamas conflict.  _Visual Communication Quarterly_ , 21:150–161, 07 2014\.  doi: 10.1080/15551393.2014.955501.
  * Simonyan and Zisserman [2015] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  _CoRR_ , abs/1409.1556, 2015.
  * Szegedy et al. [2016] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna.  Rethinking the inception architecture for computer vision.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2818–2826, 2016.
  * Szegedy et al. [2017] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi.  Inception-v4, inception-resnet and the impact of residual connections on learning.  In _AAAI_ , 2017.
  * Tomek [1976] I. Tomek.  Two modifications of cnn.  1976\.
  * Vaswani et al. [2017] A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.  Attention is all you need.  _ArXiv_ , abs/1706.03762, 2017.
  * Vidgen et al. [2019] B. Vidgen, A. Harris, D. Nguyen, R. Tromble, S. Hale, and H. Margetts.  Challenges and frontiers in abusive content detection.  In _Proceedings of the Third Workshop on Abusive Language Online_ , pages 80–93, Florence, Italy, Aug. 2019. Association for Computational Linguistics.  doi: 10.18653/v1/W19-3509.  URL https://www.aclweb.org/anthology/W19-3509.
  * Volkova et al. [2019] S. Volkova, E. Ayton, D. L. Arendt, Z. Huang, and B. Hutchinson.  Explaining multimodal deceptive news prediction models.  13:659–662, Jul. 2019.  URL https://ojs.aaai.org/index.php/ICWSM/article/view/3266.
  * Yang et al. [2019a] F. Yang, X. Peng, G. Ghosh, R. Shilon, H. Ma, E. Moore, and G. Predovic.  Exploring deep multimodal fusion of text and photo for hate speech classification.  In _Proceedings of the Third Workshop on Abusive Language Online_ , pages 11–18, Florence, Italy, Aug. 2019a. Association for Computational Linguistics.  doi: 10.18653/v1/W19-3502.  URL https://www.aclweb.org/anthology/W19-3502.
  * Yang et al. [2019b] K.-C. Yang, O. Varol, C. Davis, E. Ferrara, A. Flammini, and F. Menczer.  Arming the public with artificial intelligence to counter social bots.  _Human Behavior and Emerging Technologies_ , 1:e115, 02 2019b.  doi: 10.1002/hbe2.115.
  * Yang et al. [2019c] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _NeurIPS_ , 2019c.
  * Zhang and Mani [2003] J. Zhang and I. Mani.  KNN Approach to Unbalanced Data Distributions: A Case Study Involving Information Extraction.  In _Proceedings of the ICML’2003 Workshop on Learning from Imbalanced Datasets_ , 2003.

##  Appendix A Appendix

###  A.1 Sample Memes

(a)

(b)

(c)

Figure 3: Example Memes

In above figure 3, we can see few example memes from the dataset which we used
in this paper. An example (a) is using the Name calling/Labeling propaganda
technique by labeling the Russian vaccine with smirnoff vodka. Next example
(b) applies Exaggeration technique (hyperbole the simple statement of Trump
recovered from corona), there is also a Name calling technique (referring
corona with ‘RONA’), and uses Glittering generalities technique. The example
(c) expresses Doubt and creating a confusion in audience.