# Solitary waves and their interactions in the cylindrical Korteweg–de Vries
equation

Wencheng Hu  Jingli Ren  Yury Stepanyants<EMAIL_ADDRESS>Henan
Academy of Big Data/School of Mathematics and Statistics, Zhengzhou
University, Zhengzhou 450001, China;  College of Science, Zhongyuan University
of Technology, Zhengzhou 450007, China;  School of Mathematics, Physics and
Computing, University of Southern Queensland, 487–535 West St., Toowoomba,
QLD, 4350, Australia;  Department of Applied Mathematics, Nizhny Novgorod
State Technical University, n.a. R.E. Alekseev, 24 Minin St., Nizhny Novgorod,
603950, Russia.

###### Abstract

We consider approximate, exact, and numerical solutions to the cylindrical
Korteweg–de Vries equation. We show that there are different types of solitary
waves and obtain the dependence of their parameters on distance. Then, we
study the interaction of solitary waves of different types.

###### keywords:

Nonlinear wave; Cylindrical Korteweg–de Vries equation; Soliton; Self-similar
solitary wave

††journal: Symmetry

2.0cm2.0cm1.0cm2.0cm

††footnotetext: ORCIDs: 0000-0001-5105-8163 (W. Hu); 0000-0002-5392-291X (J.
Ren); 0000-0003-4546-0310 (Y. Stepanyants)

##  1 Introduction

The study of weakly nonlinear cylindrical waves in dispersive media has a long
history. In 1959 Iordansky derived the cylindrical version of the Korteweg–de
Vries (cKdV) equation [15] for surface waves in a fluid. A similar equation
was later derived for water and plasma waves by various authors [27, 33, 29,
25, 40, 41, 12]. Currently, the cylindrical KdV equation is one of the basic
equations of contemporary mathematical physics. In application to the
description of outgoing waves with axisymmetric fronts, the equation in the
proper physical coordinates reads:

$\frac{\partial u}{\partial r}+\frac{1}{c}\frac{\partial u}{\partial
t}-\frac{\alpha}{c}u\frac{\partial u}{\partial
t}-\frac{\beta}{2c^{5}}\frac{\partial^{3}u}{\partial t^{3}}+\frac{u}{2r}=0,$
(1.1)

where $c$ is the speed of long linear waves for which dispersion is negligible
($\beta=0$), $\alpha$ is the nonlinear coefficient, and $\beta$ is the
dispersive coefficient. Here $r$ stands for the radial coordinate and $t$ is
time. The derivation of this equation is based on the assumption that the last
three terms that describe the effects of weak nonlinearity, dispersion, and
geometric divergence are relatively small (compared to the first two linear
terms) and are of the same magnitude of smallness. The smallness of the
geometric divergence presumes that the cKdV equation is valid at big distances
from the center of the polar coordinate frame where $r\gg\Lambda$, and
$\Lambda$ is the characteristic width of a wave perturbation. A similar
equation describing incoming waves can be also derived; it differs from Eq.
(1.1) only by the sign minus in front of the second term. In such a form the
cKdV equation was used for the interpretation of physical experiments with
plasma waves in laboratory chambers [27, 13, 33] (however, it becomes invalid
when a wave approaches the origin). The importance of the cKdV equation in
water wave problems is related to circular perturbations which can appear due
to “point sources” produced by underwater earthquakes, volcanoes, atmospheric
pressure, fallen meteorites, etc. Besides, there are many observations when
quasi-cylindrical internal waves were generated due to water intrusion in
certain basins (see, for example, in the Internet numerous satellite images of
internal waves generated by Atlantic water intrusions in the Mediterranian
Sea).

The generalized cKdV equation was derived by McMillan and Sutherland [28] who
considered the generation and evolution of solitary waves by intrusive gravity
currents in a two-layer fluid. Another generalised cKdV model was derived for
the description of surface and internal ring waves subject to shear flows [18,
21, 22]. However, in this paper, we do not consider the influence of
intrusions or shear flows, as well as the environment inhomogeneity on wave
dynamics focusing on the structure of solitary waves and their interactions
within the standard cKdV equation.

In 1976 Dryuma discovered that the cKdV equation is completely integrable [7]
and found self-similar (but singular) solutions to this equation. Non-singular
self-similar solutions were found later in several papers [5, 29, 30, 4].
There were also derived approximate solutions in the form of KdV solitons with
gradually varying parameters (amplitude, width, and speed) [23, 38]. As was
shown in all these papers, amplitudes of outgoing waves decay as $A(r)\sim
r^{-2/3}$, and their characteristic duration increase as $T(r)\sim r^{1/3}$.
Later exact solutions to the cKdV equation were derived by Calogero and
Degasperis [3] (see also [4]), as well as by Nakamura and Chen [31]. The
structure of exact solutions constructed by these authors was mathematically
very similar to N-soliton solutions to the KdV equation. Despite the numerous
publications on cylindrical waves described by cKdV equation, the structure of
cylindrically diverging solitary waves was not been properly analysed in
detail until now. Their role in the dynamics of initial pulse-type
perturbations as well as interactions with each other was not studied too.
Therefore, the main aim of this paper is to fill in the gap in the knowledge
in this field.

##  2 Solitary wave solutions to the cylindrical Korteweg–de Vries equation

###  2.1 Dimensionless form of the cKdV equation and connection of cKdV with
the plane KdV equation

It is convenient to study solutions of the cKdV equation in the dimensionless
form. To this end, we make the transformation:

$r^{\prime}=r,\quad\tau=-(\beta/2c^{5})^{-1/3}(t-r/c),\quad
v=\alpha(2c^{2}/\beta)^{1/3}u/6$ (2.1)

and present Eq. (1.1) in the form (the symbol prime of $r$ can be omitted):

$\frac{\partial v}{\partial r}+6v\frac{\partial
v}{\partial\tau}+\frac{\partial^{3}v}{\partial\tau^{3}}+\frac{v}{2r}=0.$ (2.2)

If we omit the last term in this equation, we obtain the classical KdV
equation; one of its exact solutions in the form of a soliton is:

$v(r,\tau)=A{\rm\,sech}^{2}\frac{\tau-r/V}{T}.$ (2.3)

Here $A$ is the soliton amplitude, $T=\sqrt{2/A}$ is its characteristic
duration, and $V=1/(2A)$ is soliton speed. (Note that in this variable the
speed looks a bit unusual; it is inverse proportional to the soliton amplitude
$A$. However, in the original physical variables, the dimensional soliton
speed is determined as
$1/V_{s}=1/c-(\beta/2c^{5})^{1/3}(1/V)=1/c-(\beta/2c^{5})^{1/3}2A=1/c-\alpha
A_{s}/3c$, where $A_{s}$ is the dimensional soliton amplitude – see the
transformations (2.1). This gives $V_{s}=c/(1-\alpha A_{s}/3)\approx
c(1+\alpha A_{s}/3)$, where approximation is valid for small-amplitude
solitons which is in agreement with the assumption of a weak nonlinearity in
the KdV equation.) Below we present an approximate and exact solutions to the
cKdV equation (2.2).

There is a relationship between the ordinary KdV equation and cKdV equation
established for the first time by A.A. and B.A. Lugovtsovs [26], and then
found also in Refs. [14, 2]. Making the transformation:

$\tau^{\prime}=-2\tau/r,\quad r^{\prime}=4/r^{2},\quad
v^{\prime}=(v+\tau/4)/r$ (2.4)

one can reduce the classic KdV equation (Eq. (2.2) without the last term on
the left-hand side) to the cKdV equation (2.2). Formally, this allows us to
get wide classes of exact solutions from the corresponding solutions of the
KdV equation, including N-soliton solutions (some examples are presented in
Refs. [14, 24]). However, all such solutions, apparently, are physically
meaningless as they contain time-dependent nonuniform background.

###  2.2 Asymptotic solution of the cylindrical KdV equation

In the cylindrical case, the soliton solution (2.3) is no longer the exact
solution; however, if the last term in the cKdV equation (2.2) is small
compared to the nonlinear and dispersive terms, then we can assume that the
structure of a pulse having a shape of the KdV soliton (2.3) given at some
distance $r_{0}\gg\Delta\equiv VT$ remains the same in the outgoing wave,
whereas its amplitude and other parameter are slowly varying function of $r$.
Therefore, the approximate solution can be presented as:

$v(r,\tau)=A(r){\rm\,sech}^{2}\frac{\tau-\int dr/V(r)}{T(r)}.$ (2.5)

The dependence of soliton amplitude on $r$ can be found from the equation of
energy flux conservation. Multiplying Eq. (2.2) by $v$ and integrating over
$\tau$ from minus to plus infinity, we obtain:

$r\int\limits_{-\infty}^{+\infty}v^{2}(r,\tau)\,d\tau=const.$ (2.6)

Substituting here solution (2.5) and bearing in mind the relationship between
$T$ and $A$, we derive:

$A(r)=A_{0}\left(r/r_{0}\right)^{-2/3},\quad
T(r)=T_{0}\left(r/r_{0}\right)^{1/3}.$ (2.7)

These are the laws of parameter variations in the nonlinear outgoing waves
which were obtained in the papers cited above [23, 38] and in many others
(see, for example, Refs. [6, 8, 32, 35]). Both the experimental and numerical
data confirm the dependences (2.7) derived in the adiabatical approximation
for cylindrical solitons (see, e.g., [8, 35] and references therein). For the
numerical study, we used the explicit finite-difference scheme described
Berezin [1] (see also [32]). Figure 1 illustrates a comparison of a typical
cylindrical solitary wave as a function of $\tau$ plotted on the basis of the
adiabatic formulae (2.5), (2.7) and as obtained from the direct numerical
solution of Eq. (2.2).

$0$$0$

Figure 1: (color online). Comparison of the approximate solution (2.5), (2.7)
with the numerical solution for the initial pulse in the form of a KdV
soliton. Panel (a) demonstrates that the numerical solution (red line) is
indistinguishable from the approximate solution (blue line). However, a small-
amplitude long tail of negative polarity can be seen behind the soliton in the
numerical solution when the plot is zoomed in as shown in panel (b).

The initial amplitude of the KdV soliton was chosen to be $A_{0}=1$ at
$\tau=500$ (for other amplitudes, the results were very similar). After a
while at $\tau=809.6$, the amplitude dropped to $A(809.6)=0.625$. As one can
see from Fig. 1, the shapes of approximate and numerical solutions are not
distinguishable by the naked eye. In a more detailed comparison, one can
notice that a small amplitude long tail of negative polarity forms behind the
soliton in the numerical solution as shown in Fig. 1b). The tail shape can be
described in the next approximation of the asymptotic theory (see, for
example, [11, 34]). The same results were obtained by Johnson [19] who also
derived the analytical expression for the tail (see also Appendix C in Ref.
[12] where Grimshaw estimated the decay of the tail amplitude of the negative
polarity as $r^{-2/3}$).

As has been mentioned, the approximate solution is valid at a big distance
from the center of a polar coordinate frame, where $r\gg\Delta$ and when the
last geometric term is small compared to the nonlinear and dispersive terms.
However, in the course of solitary wave propagation, its parameters vary and
the used approximation can become invalid. Therefore, it is of interest to
estimate the validity of the approximate soliton solution (2.5), (2.7) at
different distances. To this end, let us compare the last term in Eq. (2.2)
with the nonlinear term on the soliton solution:

$\frac{v}{2r}:6v\frac{\partial v}{\partial
t}\sim\frac{T(r)}{12A(r)r}\sim\frac{T_{0}(r/r_{0})^{1/3}}{12A_{0}(r/r_{0})^{-2/3}r}=\frac{T_{0}}{12A_{0}}=\frac{A_{0}^{-3/2}}{6\sqrt{2}}.$
(2.8)

From this formula one can see that the ratio of these two terms does not
depend on $r$; it remains small if it was small at the beginning when
$r=r_{0}$.

It is worth reminding that in this paper we study solitary waves within the
framework of the cKdV equation when it is applicable to particular physical
systems. In general, the amplitude decay of cylindrical waves can be different
from the soliton amplitude dependence $A\sim r^{-2/3}$. As well-known,
amplitudes of linear waves in cylindrical systems without dispersion vary as
$A\sim r^{-1/2}$, and linear waves in cylindrical systems with dispersion vary
as $A\sim r^{-1}$. All these amplitude dependencies for pulse-type initial
perturbations were observed in experiments with electromagnetic waves in 2D
lattices [38, 6]. Similar results were obtained in the numerical study of
radially spreading axisymmetric intrusions and solitary waves [28].

Diverging KdV-like solitons interact in a similar manner as classical KdV
solitons. Figure 2 illustrates the typical overtaking interaction of two KdV-
like solitons within the framework of cKdV equation (2.2) obtained by direct
numerical modeling of this equation with the initial condition in the form of
two KdV solitons of different amplitudes ($A_{1}=0.2;A_{2}=1$).

$0$

Figure 2: (color online). The typical overtaking interaction of two KdV-like
solitons in outgoing cylindrical waves.

###  2.3 Exact solutions of the cKdV equation

The first nontrivial exact solutions to the cKdV equation were obtained by
Calogero and Degasperis [3]. Solutions were presented in terms of the Airy
function $\mbox{Ai}(z)$. As was shown later by Nakamura and Chen [31], exact
solutions can be presented through the Hirota transform:
$v(r,\tau)=2\partial^{2}f(r,\tau)/\partial\tau^{2}$. Then, the simplest
solution is:

$f(r,\tau)=1+\frac{\varepsilon\rho^{2}}{\left(12r\right)^{1/3}}\left\\{\left[z(r,\tau)-z_{1}(r,\tau_{1})\right]\mbox{Ai}^{2}(z-z_{1})-\left[\mbox{Ai}^{\prime}(z-z_{1})\right]^{2}\right\\},$
(2.9)

where $\varepsilon$, $\rho$, and $\tau_{1}$ are some arbitrary constants, and

$z(r,\tau)=\frac{\tau}{\left(12r\right)^{1/3}},\quad
z_{1}(r,\tau_{1})=\frac{\tau_{1}}{\left(12r\right)^{1/3}}.$ (2.10)

The symbol prime in Eq. (2.9) stands for differentiation with respect to the
function argument. Note that in terms of the function $f(r,\tau)$, solution
(2.9) is the typical self-similar solution on the constant pedestal. However,
in the original variable $v(r,\tau)$, the corresponding solution is more
complicated, it is neither self-similar nor a traveling-wave solution. One of
the typical exact solutions is plotted in Fig. 3 for the particular parameters
$\varepsilon=-0.01$, $\rho=1$, and $\tau_{1}=150$. This solution represents a
wave that pulls into the origin as one can see from the right columns of Fig.
3. Approaching the origin, the wavelength drastically decreases and goes to
zero. However, in the vicinity of the origin solution becomes invalid anyway
because, as mentioned above, the cKdV equation is applicable only at
relatively big distances from the origin. Apparently, such solutions are out
of physical interest.

Figure 3: (color online). The typical exact solution of the cKdV equation in
terms of the Airy function $\mbox{Ai}(z)$ (2.9) with the following parameters:
$\varepsilon=-0.01$, $\rho=1$, and $\tau_{1}=150$. In the left column, one can
see the dependence of $v(\tau)$ for two distances, $r=50$ and $r=100$; in the
right column, the solution is presented as a function of $r$ for two different
times, $\tau=0$ and $\tau=20$. (Note that In the vicinity of the origin, the
plot is simply cut; therefore, it looks that the solution is constant.)

The genuine self-similar solution in terms of function $v(r,\tau)$ can be
obtained if we set $\varepsilon\rho^{2}\to\infty$ [16]. Then, we obtain:

$v_{ss}(r,\tau)=\frac{2}{\left(12r\right)^{2/3}}\frac{d^{2}}{dz^{2}}\ln{\left\\{\left[z(r,\tau)-z_{1}(r,\tau_{1})\right]\mbox{Ai}^{2}(z-z_{1})-\left[\mbox{Ai}^{\prime}(z-z_{1})\right]^{2}\right\\}}.$
(2.11)

Such a solution was considered in [17] in application to the water-wave
problem.

The self-similar solution to the cKdV equation can be obtained if we seek a
solution in the form $v(r,\tau)=r^{\alpha}F(\xi)$, where
$\xi=r^{\beta}\tau^{\gamma}$ (the similar approach was used in [20] for the
KdV equation). Substituting this form of the solution in Eq. (2.2), we obtain
after simple manipulation that function $F(\xi)$ must satisfy the ODE:

$F^{\prime\prime\prime}+6FF^{\prime}-\frac{1}{3}zF^{\prime}+3F=0$ (2.12)

provided that $\alpha=-2/3$, $\beta=-1/3$, $\gamma=1$. This agrees with the
solution (2.11) if we set $F=v_{ss}\left(12r\right)^{2/3}/2$.

Calogero and Degasperis wrote that solutions that they constructed “are in
some sense the analogous of the single-soliton solutions (although they are
not quite localised, having a slowly vanishing wiggling tail)”. The analysis
of solution (2.9) shows that it describes a wave perturbation that decays in
space as $r^{-2/3}$ whereas its duration increases with the distance as
$r^{1/3}$, i.e. these quantities vary in space in the same manner as the
parameters (amplitude and duration) of a solitary wave in the approximate
solution (2.5), (2.7). Even more complicated solutions mathematically similar
to N-soliton solutions can be constructed but all of them are far from real
solitary waves.

Nakamura and Chen [31] found that compact pulse-type solutions can be obtained
if one replaces the first-kind Airy function $\mbox{Ai}(z)$ in the solution
(2.9) with the second-kind Airy function $\mbox{Bi}(z)$. Then, the simplest
solution looks pretty much the same as the KdV soliton, at least in its
leading part.

$0$$0$

Figure 4: (color online). Exact solution of the cKdV equation in terms of the
second-kind Airy function $\mbox{Bi}(z)$ (2.9) with the following parameters:
$\varepsilon=10^{-4}$, $\rho=10^{-3}$, and $\tau_{1}=10$. Panel (a) shows the
dependence of the solution on time $\tau$ for the fixed distances, and panel
(b) shows the dependence of the solution on distance $r$ for the fixed times.

As an example, we show in Fig. 4a) the comparison of solution (2.9) with the
function $\mbox{Bi}(z)$ with the KdV soliton of the same amplitude at $r=25$.
As one can see, the leading parts of these solutions are practically the same;
the only difference is in the rear parts of the solutions. The same good
agreements were confirmed for the solutions of equal amplitudes at other
distances. However, in contrast to KdV-like solitons, solitary waves in the
solution of Nakamura and Chen [31] are accompanied by well-visible positive
polarity tails (cf. Fig. 1b). Solutions with Airy functions of the second-kind
$Bi(z)$ are also singular at $r=0$ like solutions with Airy functions of the
first-kind $Ai(z)$ (see, for example, Fig. 4b). However, in this kind of
solutions, the leading part being far from the origin, make sense and their
shapes are well-approximated by KdV solitons as shown in Fig. 4a).

Despite solutions (2.9) with either first-kind or second-kind Airy functions
are not exactly self-similar or traveling-wave solutions, we will call,
conditionally solution (2.9) with the second-kind Airy function $\mbox{Bi}(z)$
the self-similar soliton (ss-soliton). Figure 5 shows the diverging ss-soliton
at different time moments. In the last frame at $\tau=100$, one can see a
singularity at the center $r=0$.

Figure 5: (color online). The typical cylindrically diverging self-similar
soliton is described by function (2.9) with the Airy function of the second
kind $\mbox{Bi}(z)$ (2.9). The plot was generated for the same parameters as
in Fig. 4. Here $x$ and $y$ are the Cartesian coordinates such that
$r^{2}=x^{2}+y^{2}$.

The “two-soliton solution” in terms of function $f(r,\tau)$ can be presented
in the form [31]:

$f(r,\tau)=1+\varepsilon(a_{11}+a_{22})+\varepsilon^{2}\left|\begin{array}[]{cc}a_{11}&a_{12}\\\
a_{21}&a_{22}\end{array}\right|=\left|\begin{array}[]{cc}1+\varepsilon
a_{11}&\varepsilon a_{12}\\\ \varepsilon a_{21}&1+\varepsilon
a_{22}\end{array}\right|,$ (2.13)

where the quantities $a_{ij}$ are defined by the following expressions:

$\displaystyle a_{ij}$ $\displaystyle=$
$\displaystyle\frac{\rho_{i}\,\rho_{j}}{\left(12r\right)^{1/3}}\frac{w_{i}(z-z_{i})\,w^{\prime}_{j}(z-z_{j})-w^{\prime}_{i}(z-z_{i})\,w_{j}(z-z_{j})}{z_{i}-z_{j}},\quad
i\neq j;$ (2.14) $\displaystyle a_{ii}$ $\displaystyle=$
$\displaystyle\frac{\rho_{i}^{2}}{\left(12r\right)^{1/3}}\left\\{\left(z-z_{i}\right)w_{i}^{2}(z-z_{i})-\left[w_{i}^{\prime}(z-z_{i})\right]^{2}\right\\},\quad
i=j.$ (2.15)

where $i,j=1,2$, and $w_{i}(z)$ are either Airy function of the first kind
$\mbox{Ai}(z)$ or Airy function of the second kind $\mbox{Bi}(z)$. However, as
has been aforementioned, solutions with the function $\mbox{Ai}(z)$ do not
represent pulse-type waves; therefore, we consider further only solutions with
the second-type Airy function $\mbox{Bi}(z)$.

$0$

Figure 6: (color online). Exact two-soliton solution of the cKdV equation in
terms of the second-kind Airy function $\mbox{Bi}(z)$ as per Eqs.
(2.13)–(2.15) with the following parameters: $\varepsilon=10^{-4}$,
$\rho_{1}=10^{-3}$, $\rho_{2}=10^{-6}$, $\tau_{1}=25$, and $\tau_{2}=-10$. To
make graphics clearly visible, we multiplied function $v(\tau)$ by 4 at
$r=400$, by 16 at $r=2\cdot 10^{3}$, and by 25 at $r=10^{4}$.

A typical two-soliton solution described by Eqs. (2.13)–(2.15) with
$w(z)\equiv\mbox{Bi}(z)$ is illustrated by Fig. 6. In this figure, one can see
the time dependence of function $v(\tau)$ at three distances from the center.
The interaction of two ss-solitons resembles the overtaking type interaction
of KdV solitons [10] when two peaks merge at some distance (at $r=400$ in our
figure) and then, they slowly separate. However, the separation lasts a very
long time and even at big distances the pulses remain coupled as illustrated
by Fig. 6.

There is also the process of fission of an initial pulse-type perturbation
into ss-solitons that looks very similar to the pure soliton breakdown of a
pulse in the plane KdV equation. An example of such a process is shown in Fig.
7.

$0$

Figure 7: (color online). Fission of initial pulse at $r=10$ onto two ss-
solitons within the exact solution described by Eqs. (2.13)–(2.15) with the
following parameters: $\varepsilon=10^{-4}$, $\rho_{1}=0.1$,
$\rho_{2}=10^{-4}$, $\tau_{1}=0$, and $\tau_{2}=-10$. To make graphics clearly
visible, we multiplied function $v(\tau)$ by 5 at $r=50$, by 15 at $r=\cdot
10^{3}$, and by 45 at $r=10^{4}$.

The physical importance of such solutions is not clear but mathematically they
are very interesting. Johnson in his paper [17] mentioned that the “choice of
either Bi or Ai functions does not lead to a proper solution of the cKdV
equation” but he assumed that, perhaps, there is some mileage in describing
the evolution of pulse-type initial profiles in terms of such functions.

##  3 Pulse disintegration into KdV-like solitons and interaction of KdV
solitons with ss-solitons

As was shown above, a KdV soliton is very robust in the cylindrical system and
keeps its identity even in the process of decay due to geometrical divergence.
The interaction between two KdV-like solitons is very much similar to the
interaction of KdV solitons in the plane case. It is natural to expect that
solitons can emerge from wide initial pulses in the same manner as in the
plane case. To confirm this conjecture, we conducted numerical experiments
with wider initial pulses which gives rise to the emergence of several
solitons in the plane KdV equation. The typical example with three solitons
emergence is shown in Fig. 8. This example corresponds to the pure soliton
decay of a ${\rm\,sech}^{2}$-pulse in the plane KdV equation. We see that in
the cKdV equation the same pure soliton decay occurs at the early stage of
evolution and then, each soliton experiences the adiabatic decay in accordance
with the asymptotic formulae (2.5) and (2.7).

$0$$0$$0$

Figure 8: (color online). Initial pulse disintegration in the cKdV equation
and emergence of KdV-like solitons. Frame a) $r-r_{0}=0$, frame b)
$r-r_{0}=6$, frame c) $r-r_{0}=12$.

A similar pulse disintegration into a number of solitons was observed for
pulses of positive polarity and different initial duration and amplitudes. A
pure soliton disintegration was observed for the same parameters of an initial
pulse as in the plane case. In general, the initial pulse breaks into solitons
and a trailing dispersive wave train. Fission into solitons was also observed
in a recent paper [39]

It is of interest to study also the interaction of a KdV soliton with an ss-
soliton. This can be done numerically for the initial condition consisting of
one KdV soliton and one ss-soliton. The result of such interaction is shown in
Fig. 9.

$0$$0$$0$$0$

Figure 9: (color online). Interaction of the KdV-like soliton with the ss-
soliton in outgoing cylindrical waves. The amplitude of the KdV soliton was
$A_{0}=1$ at $r_{0}=100$. The parameters of the ss-solton were
$\varepsilon=10^{-10}$, $\rho=1$, $\tau_{1}=1$. Frame a) $r-r_{0}=160$, frame
b) $r-r_{0}=190$, frame c) $r-r_{0}=210$, frame d) $r-r_{0}=240$.

Thus, we see that the traveling KdV-type soliton overtakes the ss-soliton and
after the interaction, both of them restore their shapes and continue moving
and decaying due to the geometrical divergence. Thus, we can conclude that in
the weakly nonlinear physical systems with a small dispersion, the outgoing
pulses with cylindrical fronts evolve in a similar way as in the plane KdV
equation but experience amplitude decay due to the geometrical divergence.

##  4 Concluding remarks

In this paper, we have presented a detailed analysis of solitary wave
solutions to the cylindrical KdV equation. It was shown that soliton-like
solutions in the form of KdV solitons exist in this equation. In the process
of geometrical divergence, such solitons gradually decay so that the total
energy of the initial pulse is conserved,
$E=\int\eta^{2}r\,d\tau=\mbox{const}$, where the integration should be carried
out over $\tau$ in the infinite limits, $-\infty<\tau<+\infty$. There are also
exact solutions of the cKdV equation [31] which have pulse-type shapes (ss-
solitons) which are very similar to KdV solitons of the same amplitudes. Their
parameters (amplitudes and duration) vary with the distance in the same manner
as in the diverging KdV-like solitons, $A\sim r^{-2/3}$, $T\sim r^{1/3}$.
However, such solutions are not traveling waves but are closer to self-similar
solutions.

A numerical study of interactions between KdV-like solitons, ss-solitons, as
well as between KdV-like and ss-solitons revealed that all of such solitons
are robust and, apparently, interact elastically. A general pulse-type initial
perturbation of positive polarity in the course of evolution experiences a
breakdown into a number of KdV-like solitons and trailing dispersive
wavetrain. Each of emerged KdV-like solitons decays then individually due to
the geometrical divergence.

In conclusion, we note that some asymptotic solutions to the cKdV equation
were obtained in Refs. [36, 37]. Using symbolic computation, Gao and Tian [9]
constructed a few self-similar solutions to the cKdV equation; some of them
were mentioned in this paper and obtained by other authors using analytical
methods. However, all these solutions are out of our current interest as they
are not of a soliton-type.

In perspective, we plan to study quasi-cylindrical waves within the
cylindrical version of the Kadomtsev–Petviashvili equation (alias Johnson
equation) [17]. The important problem to be studied is the stability of a
soliton front with respect to small azimuthal perturbations and lump
formations. One more problem to be studied in perspective is the dynamics of
solitons within the cylindrical Gardner equation containing both quadratic and
cubic nonlinearities. Such an equation is applicable to the description of
internal waves in the ocean and the results obtained can be of practical
interest.

Acknowledgements. W.H. acknowledges the financial support from China
Scholarship Council (grant No. 202002425001). His study was also supported by
the National Natural Science Foundation of China (grants No. 11947093 and
12204554) and the Natural Science Foundation of Henan Province of China (grant
No. 222300420393). Y.S. acknowledges the financial support provided by the
President Council of the Russian Federation (grant No. NSH-70.2022.1.5) for
the State support of Leading Scientific Schools of the Russian Federation. The
authors are grateful to K. Khusnutdinova for her helpful remarks.

## References

  * [1] Yu. Berezin, Modelling Nonlinear Wave Processes, VNU Science Press, 1987.
  * [2] T. Brugarino, P. Pantano, The integration of Burgers and Korteweg–de Vries equation with nonuniformities, Phys. Lett. A, 80 (4) (1980) 223–224.
  * [3] F. Calogero, A. Degasperis, Solution by the spectral-transform method of a nonlinear evolution equation including as a special case the cylindrical KdV equation, Lett. Nuovo Cim., 23 (4) (1978) 150–154.
  * [4] F. Calogero, A. Degasperis, Spectral Transform and Solitons: Tools to Solve and Investigate Nonlinear Evolution Equations. North Holland, Amsterdam, v. 1, 1982.
  * [5] E. Cumberbatch, Spike solution for radially symmetric solitary waves, Phys. Fluids 21 (3) (1978) 374–376.
  * [6] A.A. Dorfman, E.N. Pelinovsky E.N., Yu.A. Stepanyants, Finite-amplitude cylindrical and spherical waves in weakly dispersive media, Sov. Phys. J. Appl. Mech. Tech. Phys., 2 (1981) 206–211.
  * [7] V.S. Dryuma, On the analytical solution of the axisymmetric KdV equation, Izv. Akad. Nauk MSSR, Set. Fiz. Tekh. Mat. Nauk, No. 3 (1976) 87 (in Russian).
  * [8] P. Fraunie, Y. Stepanyants, Decay of cylindrical and spherical solitons in rotating media, Phys. Lett. A, 293 (3-4) (2002) 166–172.
  * [9] Y.T. Gao, B. Tian, Computerized symbolic computation for the cylindrical Korteweg–de Vries equation, Int. J. Mod. Phys. C 10 (7) (1999) 1303–1316.
  * [10] K.A Gorshkov, L.A. Ostrovsky, Yu.A. Stepanyants, Dynamics of soliton chains: From simple to complex and chaotic motions. In: Long-Range Interactions, Stochasticity and Fractional Dynamics, eds. Albert C.J. Luo and Valentin Afraimovich, Springer, (2010) 177–218.
  * [11] R.H.J. Grimshaw, J.-M. He, L.A. Ostrovsky, Terminal damping of a solitary wave due to radiation in rotational systems, Stud. Appl. Math., 101, (1998) 197–210.
  * [12] R.H.J. Grimshaw, Initial conditions for the cylindrical Korteweg–de Vries equation, Stud. Appl. Math., 143 (2), (2019) 176–191.
  * [13] N. Hershkowitz, T. Romesser, Observations of ion-acoustic cylindrical solitons, Phys. Rev. Lett., 32 (11), (1974) 581–583.
  * [14] R. Hirota, Exact solutions to the equation describing “cylindrical solitons”, Phys. Lett., 71A (5–6), (1979) 393–394.
  * [15] S.V. Iordansky, On the asymptotic of an axisymmetric divergent wave in a heavy fluid, Doklady Akad. Sci. USSR, 125 (6) (1959) 1211–1214.
  * [16] R.S. Johnson, On the inverse scattering transform, the cylindrical Korteweg–de Vries equations and similarity solutions, Phys. Lett. 72A (3) (1979) 197–199.
  * [17] R.S. Johnson, Water waves and Korteweg–de Vries equations, J. Fluid Mech. 97 (4) (1980) 701–719.
  * [18] R.S. Johnson, Ring waves on the surface of shear flows: a linear and nonlinear theory. J. Fluid Mech., 215 (1990) 145–160.
  * [19] R.S. Johnson, A note on an asymptotic solution of the cylindrical Korteweg–de Vries equation, Wave Motion, 30 (1999) 1–16.
  * [20] V.I. Karpman Non-linear waves in dispersive media, Oxford, New York, Pergamon Press (1975).
  * [21] K. Khusnutdinova, X. Zhang, Long ring waves in a stratified fluid over a shear flow, J. Fluid Mech. 794 (2016) 17–44.
  * [22] K. Khusnutdinova, X. Zhang, Nonlinear ring waves in a two-layer fluid, Physica D, 333 (2016) 208–221.
  * [23] K. Ko, H.H. Kuehl, Cylindrical and spherical KdV solitary waves, Phys. Fluids 22 (7) (1979) 1343–1348.
  * [24] M. Leo, R.A. Leo, G. Soliani, L. Martina, Prolongation analysis of the cylindrical Korteweg–de Vries equation, Phys. Rev. D, 26(4) (1982) 809–818.
  * [25] V.D. Lipovskii, On the nonlinear internal wave theory in fluid of finite depth, Izv. Akad. Nauk SSSR, Fiz. Atm. Okeana, 21 (1985) 864–871.
  * [26] A.A. Lugovtsov, B.A. Lugovtsov, Study of axisymmetric long waves in the Korteweg–de Vries approximation, In: Dynamics of continuous medium, 1 (1969) 195–198, Inst. of Hydrodynamics, Novosibirsk (in Russian).
  * [27] S. Maxon, Viecelli, Cylindrical solitons, Phys. Fluids, 17 (8) (1974) 1614–1616.
  * [28] J.M. McMillan, B.R. Sutherland, The lifecycle of axisymmetric internal solitary waves, Nonlin. Processes Geophys., 17, (2010) 443–453.
  * [29] J.W. Miles, An axisymmetric Boussinesq wave, J. Fluid Mech., 85 (1) (1978) 181–191.
  * [30] A. Nakamura, The Miura transform and the existence of an infinite number of conservation laws of the cylindrical KdV equation, Phys. Lett., 82A (3) (1981) 111–112.
  * [31] A. Nakamura, H.-H. Chen, Soliton solutions of the cylindrical KdV equation, J. Phys. Soc. Japan, 50 (2) (1981) 711–718.
  * [32] M.A. Obregon, Y.A. Stepanyants, On numerical solution of the Gardner–Ostrovsky equation, Math. Model. Nat. Phenom., 7 (2) (2012) 113–130.
  * [33] T. Ogino, S. Takeda, Computer simulation and analysis for the spherical and cylindrical ion-acoustic solitons, J. Phys. Soc. Japan, 41 (1) (1976) 257–264.
  * [34] L.A. Ostrovsky, Asymptotic Perturbation Theory of Waves, Imperial College Press: Singapore, 2015.
  * [35] C. Ramirez, D. Renouard, Yu.A. Stepanyants, Propagation of cylindrical waves in a rotating fluid, Fluid Dyn. Res., 30 (3) (2002) 169–196.
  * [36] P.M. Santini, Asymptotic behaviour (in t) of solutions of the cylindrical KdV equation—I, Nuovo Cimento A 54 (2) (1979) 241–258.
  * [37] P.M. Santini, Asymptotic behaviour (in t) of solutions of the cylindrical KdV equation—II, Nuovo Cimento A 57 (4) (1980) 387–396.
  * [38] Yu.A. Stepanyants, Experimental investigation of cylindrically diverging solitons in an electric lattice. Wave Motion, 3 (1981) 335–341.
  * [39] D. Tseluiko, N.S. Alharthi, R. Barros, K.R. Khusnutdinova, Internal ring waves in a three-layer fluid over a linear shear current, ArXiv: 2206.15224v1 [physics.flu-dyn] 30 June 2022.
  * [40] P.D. Weidman, R. Zakhem, Cylindrical solitary waves, J. Fluid. Mech., 191 (1988) 557–573.
  * [41] P.D. Weidman, M.G. Velarde, Internal solitary waves, Stud. Appl. Math., 86 (1992) 167–184.


# Pro-Cap: Leveraging a Frozen Vision-Language Model
for Hateful Meme Detection

Rui Cao<EMAIL_ADDRESS>Singapore Management
UniversitySingaporeSingapore ,  Ming Shan Hee<EMAIL_ADDRESS>Singapore University of Design and TechnologySingaporeSingapore ,  Adriel Kuek
<EMAIL_ADDRESS>DSO National LaboratoriesSingaporeSingapore ,  Wen-Haw
Chong<EMAIL_ADDRESS>Singapore Management
UniversitySingaporeSingapore ,  Roy Ka-Wei Lee<EMAIL_ADDRESS>Singapore
University of Design and TechnologySingaporeSingapore and  Jing Jiang
<EMAIL_ADDRESS>Singapore Management UniversitySingaporeSingapore

(2023)

###### Abstract.

Hateful meme detection is a challenging multimodal task that requires
comprehension of both vision and language, as well as cross-modal
interactions. Recent studies have tried to fine-tune pre-trained vision-
language models (PVLMs) for this task. However, with increasing model sizes,
it becomes important to leverage powerful PVLMs more efficiently, rather than
simply fine-tuning them. Recently, researchers have attempted to convert meme
images into textual captions and prompt language models for predictions. This
approach has shown good performance but suffers from non-informative image
captions. Considering the two factors mentioned above, we propose a probing-
based captioning approach to leverage PVLMs in a zero-shot visual question
answering (VQA) manner. Specifically, we prompt a frozen PVLM by asking
hateful content-related questions and use the answers as image captions (which
we call Pro-Cap), so that the captions contain information critical for
hateful content detection. The good performance of models with Pro-Cap on
three benchmarks validates the effectiveness and generalization of the
proposed method.111Code is available at: https://github.com/Social-AI-
Studio/Pro-Cap

memes, multimodal, semantic extraction

††journalyear: 2023††copyright: rightsretained††conference: Proceedings of the
31st ACM International Conference on Multimedia; October 29-November 3, 2023;
Ottawa, ON, Canada††booktitle: Proceedings of the 31st ACM International
Conference on Multimedia (MM ’23), October 29-November 3, 2023, Ottawa, ON,
Canada††doi: 10.1145/3581783.3612498††isbn: 979-8-4007-0108-5/23/10††ccs:
Computing methodologies Natural language processing††ccs: Computing
methodologies Computer vision representations

Disclaimer:  This paper contains violence and discriminatory content that may
be disturbing to some readers.

##  1\. Introduction

Figure 1. The proposed probe-captioning approach. We prompt frozen pre-trained
vision-language models via visual question answering to generate hateful
content centric image captions.

Memes, which combine images with short texts, are a popular form of
communication in online social media. Internet memes are often intended to
express humor or satire. However, they are increasingly being exploited to
spread hateful content across online platforms. Hateful memes attack
individuals or communities based on their identities such as race, gender, or
religion (Kiela et al., 2020; Pramanick et al., 2021a; Fersini et al., 2022;
Hee et al., 2023). The propagation of hateful memes can lead to discord online
and may potentially result in hate crimes. Therefore, it is urgent to develop
accurate hateful meme detection methods.

The task of hateful meme detection is challenging due to the multimodal nature
of memes. Detection involves not only comprehending both the images and the
texts but also understanding how these two modalities interact. Previous work
(Lee et al., 2021; Zhou and Chen, 2020; Pramanick et al., 2021b; Zhu et al.,
2022) learns cross-modal interactions from scratch using hateful meme
detection datasets. However, it may be difficult for models to learn
complicated multimodal interactions with the limited amount of data available
from these datasets. With the development of Pre-trained Vision-Language
Models (PVLMs) such as VisualBERT (Li et al., 2019) and ViLBERT (Lu et al.,
2019), recent work leverage these powerful PVLMs to facilitate the hateful
meme detection task. A common approach is to fine-tune PVLMs with task-
specific data (Lippe et al., 2020; Zhu, 2020; Muennighoff, 2020; Velioglu and
Rose, 2020; Hee et al., 2022). However, it is less feasible to fine-tune the
larger models such as BLIP-2 (Li et al., 2023) and Flamingo (Alayrac et al.,
2022) on meme detection because there are billions of trainable parameters.
Therefore, computationally feasible solutions other than direct fine-tuning
are needed to leverage large PVLMs in facilitating hateful meme detection.

Different from the approach above using PVLMs, PromptHate(Cao et al., 2022) is
a recently proposed model that converts the multimodal meme detection task
into a unimodal masked language modeling task. It first generates meme image
captions with an off-the-shelf image caption generator, ClipCap(Mokady et al.,
2021). By converting all input information into text, it can prompt a pre-
trained language model along with two demonstrative examples to predict
whether or not the input is hateful by leveraging the rich background
knowledge in the language model. Although PromptHate achieves state-of-the-art
performance, it is significantly affected by the quality of image captions, as
shown in Table 1. Image captions that are merely generic descriptions of
images may omit crucial details (Lee et al., 2021; Zhu, 2020), such as the
race and gender of people, which are essential for hateful content detection.
But with additional image tags, such as entities found in the images and
demographic information about the people in the images, the same model can be
significantly improved, as shown in Table 1. However, generating these
additional image tags is laborious and costly. For instance, entity extraction
is usually conducted with the Google Vision Web Entity Detection API
222https://cloud.google.com/vision/docs/detecting-web, which is a paid
service. Ideally, we would like to find a more affordable way to obtain entity
and demographic information from the images that is critical for hateful
content detection.

Table 1. Impact on detection performances on the FHM dataset (Kiela et al., 2020) from image captions. (w/o) denotes models without additional entity and demographic information. Model | Performance
---|---
| AUC | Acc.
PromptHate (w/o) | 76.76 | 67.28
PromptHate | 81.45 | 72.98
VisualBERT (w/o) | 68.71 | 61.48
VisualBERT | 72.56 | 68.24
ViLBERT (w/o) | 73.05 | 64.70
ViLBERT | 75.72 | 68.24

Both above-mentioned approaches (i.e., one using PVLMs and the other
converting the task to a unimodal task) have their pros and cons. In this
paper, we combine the ideas from these two approaches and design a hateful
meme detection method that leverages the power of a frozen PVLM to complement
the unimodal approach of PromptHate. Specifically, we use a set of “probing”
questions to query a PVLM (BLIP-2 (Li et al., 2023) in our experiments) for
information related to common vulnerable targets in hateful content. The
answers obtained from the probing questions will be treated as image captions
(denoted as Pro-Cap) and used as input to a trainable hateful meme detection
model. Figure 1 illustrates the overall workflow of the method. We refer to
the step of using probing questions to generate the captions as probing-based
captioning.

Our proposed method fills existing research gaps by: 1) Leverage a PVLM
without any adaptation or fine-tuning, thereby reducing computational cost; 2)
Instead of explicitly obtaining additional image tags with costly APIs, we
utilize the frozen PVLM to generate captions that contain information useful
for hateful meme detection. To the best of our knowledge, this is the first
work that to leverage PVLMs in a zero-shot manner through question answering
to assist in the hateful meme detection task. To further validate our method,
we test the effect of the generated Pro-Cap on both PromptHate(Cao et al.,
2022) and a BERT-based(Devlin et al., 2019) hateful meme detection model.

Based on the experimental results, we observe that PromptHate with Pro-Cap
(denoted as Pro-CapPromptHate) significantly surpasses the original PromptHate
without additional image tags (i.e., about 4, 6, and 3 percentage points of
absolute performance improvement on FHM (Kiela et al., 2020), MAMI (Fersini et
al., 2022), and HarM (Pramanick et al., 2021b) respectively). Pro-
CapPromptHate also achieves comparable results with PromptHate with additional
image tags, indicating that probing-based captioning can be a more affordable
way of obtaining image entities or demographic information. Case studies
further show that Pro-Cap offers essential image details for hateful content
detection, enhancing the explainability of models to some extent. Meanwhile,
Pro-Cap${}_{\text{BERT}}$ clearly surpasses multimodal BERT-based models of
similar sizes (i.e., about 7 percentage points of absolute improvement with
VisualBERT on FHM (Kiela et al., 2020)), proving the generalization of the
probing-based captioning method.

##  2\. Related Work

Memes, typically intended to be humorous or sarcastic, are increasingly being
exploited for the proliferation of hateful content, leading to the challenging
task of online hateful meme detection (Fersini et al., 2022; Kiela et al.,
2020; Pramanick et al., 2021a). To combat the spread of hateful memes, one
line of work regards the hateful meme detection as a multimodal classification
task. Researchers have applied pre-trained vision-language models (PVLMs) and
fine-tune them based on meme detection data (Zhu, 2020; Velioglu and Rose,
2020; Muennighoff, 2020; Lippe et al., 2020). To improve performance, some
have tried model ensembling (Velioglu and Rose, 2020; Muennighoff, 2020; Lippe
et al., 2020). Another line of work considers combining pre-trained models
(e.g., BERT (Devlin et al., 2019) and CLIP (Radford et al., 2021)) with task-
specific model architectures and tunes them end-to-end (Pramanick et al.,
2021b; Lee et al., 2021; Kumar and Nandakumar, 2022). Recently, authors in
(Cao et al., 2022) have tried converting all meme information into text and
prompting language models to better leverage the contextual background
knowledge present in language models. This approach achieves the state-of-the-
art results on two hateful meme detection benchmarks. However, it adopts a
generic method for describing the image through image captioning, often
ignoring important factors necessary for hateful meme detection. In this work,
we seek to address this issue through probe-based captioning by prompting pre-
trained vision-language models with hateful content-centric questions in a
zero-shot VQA manner.

##  3\. Preliminary

We formally define our task and briefly review the use of pre-trained vision-
language models (PVLMs) for zero-shot visual question answering (VQA). At the
end of the section, we provide a brief introduction to the specific PVLM
utilized in our work.

Given a meme image $\mathcal{I}$ and a piece of accompanying meme text
$\mathcal{T}$, the model predicts whether the meme is hateful or not.
Specifically, the model predicts scores $\mathbf{s}\in\mathbb{R}^{2}$ over the
label space, where $s_{0}$ is a score indicating how likely the meme is non-
hateful, whereas $s_{1}$ is a score for the meme being hateful. If
$s_{0}>s_{1}$, the model classifies the meme as non-hateful; otherwise, the
meme is classified as hateful. Our proposed method (to be presented in detail
in Section 4) uses zero-shot VQA to generate relevant captions to assist with
hateful meme detection. To perform zero-shot VQA, we assume that there is a
PVLM capable of processing an image and a textual prompt formatted as
Question: [QUESTION] Answer:, where [QUESTION] is a placeholder for the
question. The PVLM then generates a sequence of tokens as the answer to the
question. For example, given an image showing an Asian woman and the prompt
Question: What is the race of the person in the image? Answer:, the PVLM may
generate the answer Asian.

In this work, we use the recently released BLIP-2 model (Li et al., 2023) as
the PVLM, as it has demonstrated good performance in zero-shot VQA. The BLIP-2
model is composed of a frozen pre-trained image encoder, a frozen pre-trained
language model, and a lightweight Querying Transformer, which is responsible
for bridging the modality gap. It is worth noting that the BLIP-2 model can be
replaced with any other PVLM that is capable of zero-shot VQA.

##  4\. Proposed Method

###  4.1. Overview

Recall that the key idea of our method is to elicit image details that are
critical for hateful content detection, such as the gender and race of the
people in the image. Because these details are not always included in
automatically generated image captions, we propose relying on VQA to obtain
such critical information, where the questions are carefully curated to elicit
demographic and other relevant information. We opt to use zero-shot VQA
because (1) for the intended type of questions, we do not have any VQA
training data to train our own model, and (2) recent work has demonstrated
promising performance of zero-shot VQA.

Specifically, we prompt the PVLM with $K$ probing questions and regard the set
of $K$ answers from the PVLM as image captions, which we refer to as Pro-Cap.
We then combine the original text $\mathcal{T}$ with Pro-Cap as input to a
hateful meme detection model. We experiment with two alternative hateful meme
detection models: one based on BERT encoding, and the other based on
PromptHate, a recently proposed prompting-based hateful meme detection model.

In the rest of this section, we first present the details of how we design our
VQA questions to elicit the most critical details of an image for hateful meme
detection. We then explain how the generated Pro-Cap is used by two
alternative hateful meme detection models.

###  4.2. Design of VQA Questions

We leverage PVLMs for zero-shot VQA to generate Pro-Cap as image captions. We
want Pro-Cap to provide not only a general description of the image but also
details critical for hateful meme detection. To obtain a general caption of
the image, we design the first probing question to inquire about the generic
content of the image, as shown in Table 2. However, such generic captions may
be insufficient for hateful meme detection as hateful content usually targets
persons or groups with specific characteristics, such as race, gender, or
religion (Fersini et al., 2022; Kiela et al., 2020). Additionally, previous
studies have shown that augmenting image representations with entities found
in the image or demographic information of people in the image significantly
aids hateful meme detection (Zhu, 2020; Lee et al., 2021). Such details may be
missing in generic image captions. Therefore, we design additional questions
that aim to bring out information central to hateful content. This aligns the
generated image captions more closely with the goal of hateful meme detection.
Specifically, the high-level idea is to ask questions about common vulnerable
targets of hateful content. Inspired by (Mathias et al., 2021), which
categorizes the targets of hateful memes into Religion, Race, Gender,
Nationality, and Disability, we ask questions about these five types of
targets. For example, to generate image captions that indicate the race of the
people in an image, we can ask the following question: what is the race of
persons in the image? We list the five questions designed for these five types
of targets in Table 2. Additionally, we observe that some animals, such as
pigs, are often depicted in hateful memes, frequently as a means to annoy
Muslims. With this consideration, we also design a question asking about the
presence of animals in the image.

In (Dai et al., 2023), the author claimed that PVLMs may hallucinate non-
existent objects. For example, even when there is nobody in an image, PVLMs
may generate an answer about race in response to the question what is the race
of the person in the image?. To prevent such misleading information, we use
two validation questions. Specifically, we inquire about the existence of
persons and animals. Only when the PVLM responds that a person or an animal
exists will we include in the Pro-Cap the answers to those person-related or
animal-related questions. For instance, if the answer to the question
validating the existence of people indicates that nobody is present, we will
ignore all answers from questions asking about religion, race, gender,
nationality, and disability.

We use $\mathcal{C}$ to represent the concatenation of the answers to the
probing questions that are finally included as part of the Pro-Cap based on
the validation results. We will then concatenate $\mathcal{T}$ and
$\mathcal{C}$ together as input to a purely text-based hateful meme
classification model, as shown at the bottom of Figure 1.

Table 2. Details of questions prompting PVLMs. The first block of the question asks about the content of the image; questions in the second block ask about commonly seen vulnerable targets in hateful contents; the last block questions validate the existence of persons and animals. Focus |  Questions
---|---
Content |  what is shown in the image?
Race |  What is the race of the person in the image?
Gender |  What is the gender of the person in the image?
Religion |  What is the religion of the person in the image?
Nationality |  Which country does the person in the image come from?
Disability |  Are there disabled people in the image?
Animal |  What animal is in the image?
Val Person |  Is there a person in the image?
Val Animal |  Is there an animal in the image?
Figure 2. An overview of the PromptHate model and how pro-cap is used in
PromptHate.

###  4.3. BERT-based Detection Model

We now introduce the first of the two alternative hateful meme classification
models, which is based on BERT (Devlin et al., 2019). We first feed the
concatenation of the meme text $\mathcal{T}$ and the Pro-Cap $\mathcal{C}$
into the BERT model to generate a vector $\mathbf{r}\in\mathbb{R}^{d}$:

(1) $\mathbf{r}=\text{BERT}([\mathcal{T},\mathcal{C}]),$

where $[\cdot,\cdot]$ represents concatenation. Next, we feed the sentence
representation $\mathbf{r}$ into a linear layer for hateful meme
classification:

(2) $\mathbf{s}=\text{Sigmoid}(\mathbf{W}^{\text{T}}\mathbf{r}+\mathbf{b}),$

where $\mathbf{W}\in\mathbb{R}^{d\times 2}$ and $\mathbf{b}^{2}$ are learnable
parameters.

###  4.4. PromptHate for Hateful Meme Detection

Next, we introduce the second hateful meme classification model, PromptHate
(Cao et al., 2022), which employs a prompt-based method to classify memes.
PromptHate was developed to better leverage contextual background knowledge by
prompting language models. Given a test meme, PromptHate first uses an image
captioning model to obtain generic image captions. It then concatenates the
meme text, the image captions, and a prompt template into $\mathcal{S}$: It
was [MASK]., to prompt a language model (LM) to predict whether the meme is
hateful. Specifically, it compares the probability of the language model
predicting [MASK] to be a positive word (e.g., good) given the context, versus
the probability of predicting a negative word (e.g., bad). The approach also
includes one positive and one negative example in the context, and [MASK] will
be replaced by their respective label words. An overview of PromptHate is
shown in Figure 2. For further details, please refer to (Cao et al., 2022).

In (Cao et al., 2022), PromptHate utilizes ClipCap (Mokady et al., 2021) to
generate image captions. In this work, we replace this with Pro-Cap
$\mathcal{C}$. We then represent every meme $\mathcal{O}$ as
$\mathcal{O}=[\mathcal{T},\mathcal{C},\mathcal{S}]$. With these inputs, the
language models (LMs), for instance, RoBERTa (Liu et al., 2019), generate
confidence scores for the masked word over their vocabulary space,
$\mathcal{V}$:

(3)
$\mathbf{p}=\text{Sigmoid}(\text{LM}([\mathcal{O}_{\text{test}},\mathcal{O}_{\text{non-
hate}},\mathcal{O}_{\text{hate}}])),$

where $\mathbf{p}\in\mathbb{R}^{|\mathcal{V}|}$. We extract the score for the
label words as the prediction:

(4) $\displaystyle s_{0}$ $\displaystyle=p_{i},\
\mathcal{V}_{i}=\mathcal{W}_{\text{pos}},$ (5) $\displaystyle s_{1}$
$\displaystyle=p_{j},\ \mathcal{V}_{j}=\mathcal{W}_{\text{neg}}.$

###  4.5. Model Training and Prediction

We denote the ground-truth label of a meme as
$\mathbf{\hat{y}}\in\mathbb{R}^{2}$. If the meme is annotated as non-hateful,
$\hat{y}_{0}$ will be $1$ while $\hat{y}_{1}$ will be $0$, otherwise,
$\hat{y}=[0,1]$. The binary cross-entropy loss is applied for model training:

(6) $\text{Loss}=-(\hat{y}_{0}*log(s_{0})+\hat{y}_{1}*log(s_{1})).$

For model prediction, if $s_{0}>s_{1}$, the meme will be predicted as non-
hateful, otherwise, hateful.

##  5\. Experiment

In this section, we first introduce our evaluation datasets, metrics and
implementation details. Next, we introduce the baselines for comparison.
Finally, we conduct qualitative analysis with case studies and error analysis
to better understand the advantages and limitations of our method.

###  5.1. Experiment Settings

Evaluation Datasets. We test our proposed method on benchmarks for hateful
meme detection. We evaluate our method on three datasets to better illustrate
the generalization and stability of our approach. Table 3 presents the
statistics of these datasets.

The Facebook Hateful Meme dataset (FHM) (Kiela et al., 2020) was constructed
by Facebook. It contains synthetic memes with added confounders such that
unimodal information is insufficient for detection and deep multimodal
reasoning is required. The FHM dataset contains hateful memes targeting
various vulnerable groups in categories including Religion, Race, Gender,
Nationality, and Disability. As the labels of the test split of FHM are not
available, we performs evaluation on its dev-seen split.

Different from FHM, the Multimedia Automatic Misogyny Identification (MAMI)
dataset focuses on a particular type of hateful memes, namely, those targeting
women. Performance on MAMI therefore reflects the capability of hateful meme
detection methods for female victims.

To test our method’s generalization capability, we also consider a harmful
meme detection dataset, HarM (Pramanick et al., 2021a). HarM contains memes
related to COVID-19, which are classified into three categories: harmless,
partially harmful, and very harmful. We merge partially harmful and harmful
into one category. Because hateful content is always regarded as harmful, we
use this dataset to test the capability of generalization of our proposed
method from hateful meme detection to harmful meme detection.

Table 3. Statistical distributions of datasets used for evaluation. Datasets | Train | Test
---|---|---
| #Hate. | #Non-hate. | #Hate. | #Non-hate.
FHM | 3,050 | 5,450 | 250 | 250
HarM | 1,064 | 1,949 | 124 | 230
MAMI | 5,000 | 5,000 | 500 | 500

Evaluation Metrics. Hateful meme detection is a binary classification task. In
addition to detection accuracy, we also compute the Area Under the Receiver
Operating Characteristics curve (AUCROC) used in prior work (Lippe et al.,
2020; Zhu, 2020; Lee et al., 2021; Cao et al., 2022). We conduct experiments
with ten random seeds and report the average performance and standard
deviation. All models use the same set of random seeds.

Implementation Details. Given a meme image, we first detect the meme text with
the open-source Easy-OCR tool 333https://github.com/JaidedAI/EasyOCR and then
in-paint over the detected texts. To generate the answers to VQA questions, we
prompt BLIP-2 (Li et al., 2023), specifically the FlanT5${}_{\text{XL}}$
version. We then insert the generated image captions into two text-based
hateful meme detection models, i.e., the BERT-based model and the PromptHate
model. For the BERT-based model, to avoid overfitting, we add a dropout rate
of $0.4$ to the classification layer. We use a learning rate of $2e-5$ and a
batch size of $64$. For PromptHate, we train the model with a batch size of
$16$ and empirically set the learning rate to $1.3e-5$ on FHM and $1e-5$ on
the other two datasets (Gao et al., 2021). We optimize both models with the
AdamW optimizer (Loshchilov and Hutter, 2017) and implement them in PyTorch.
Due to space limit, we provide more details (i.e., computation costs and model
sizes) in Appendix A.

###  5.2. Baselines

We compare our method against both unimodal and multimodal models to
demonstrate the effectiveness of the proposed method, where we regard models
receiving information from one modality (i.e., the meme text or the meme image
only) as unimodal models. Note that because Pro-Cap already contains image
information, even if Pro-Cap is input into a unimodal BERT, the model is not
considered to be unimodal.

For the unimodal models, we consider a text-only and an image-only model. For
the text-only model, we fine-tune a pre-trained BERT model (Devlin et al.,
2019) based on the meme text only for meme classification, which we represent
as Text-BERT. For the image-only model, we first extract object-level image
features with an off-the-shelf feature extractor, Faster-RCNN (Ren et al.,
2016), which is trained for object detection. We then perform average pooling
over object features and feed the resulting vector into a classification
layer. We use Image-Region to denote the image-only model.

For multimodal models, we categorize them into two groups: 1) fine-tuning
generic multimodal models that are proposed to conduct different multimodal
tasks; 2) models specifically designed for hateful meme detection. For the
first type of multimodal models, we firstly consider the MMBT-Region model
(Kiela et al., 2019), which is a widely used multimodal baseline in hateful
meem detection (Kiela et al., 2020; Cao et al., 2022; Pramanick et al., 2021b)
and the model has not been pre-trained with multimodal data. Secondly, we
consider several multimodal pre-trained models, such as VisualBERT (Li et al.,
2019) pre-trained on MS-COCO (Lin et al., 2014) (VisualBERT COCO) and ViLBERT
pre-trained on Conceptual Captions (Sharma et al., 2018) (ViLBERT CC). Some
recently released powerful pre-trained models are also included such as the
Align before Fusion model (Li et al., 2021) (ALBEF) and the Bootstrapping
Language-Image Pre-training model (Li et al., 2022) (BLIP). For the second
category of baselines which are designed for the meme detection task, we
consider the models listed below. The CLIP-BERT model (Pramanick et al.,
2021b) leverages the CLIP model (Radford et al., 2021) to deal with noisy meme
images, uses pre-trained BERT (Devlin et al., 2019) for representing meme
text, and fuses them with concatenation. The MOMENTA model (Pramanick et al.,
2021b) designed both local and global multimodal fusion mechanisms to exploit
multimodal interactions for hateful meme detection. Note that the MOMENTA
model is designed to leverage augmented image tags (the detected image
entities). DisMultiHate (Lee et al., 2021) disentangles target information
from memes as targets are essential for identifying hateful content. The
PromptHate model (Cao et al., 2022) is what we discussed in Section 4.4.

Table 4. Model comparison without any augmented image tags. Dataset | FHM | MAMI | HarM
---|---|---|---
Model | AUC. | Acc. | AUC. | Acc. | AUC. | Acc.
Text BERT | 66.10±0.55 | 57.12±0.49 | 74.48±0.60 | 67.37±0.57 | 81.39±0.91 | 75.68±1.59
Image-Region | 56.69±1.05 | 52.34±1.39 | 70.20±0.63 | 64.18±0.81 | 76.46±0.47 | 73.05±1.80
VisualBERT COCO | 68.71±1.02 | 61.48±1.19 | 78.71±0.59 | 71.06±0.94 | 80.46±1.04 | 75.31±1.44
ViLBERT CC | 73.05±0.62 | 64.70±1.12 | 77.71±1.20 | 69.48±1.00 | 84.11±0.88 | 78.70±1.17
MMBT-Region | 72.86±0.64 | 65.06±1.76 | 79.17±0.91 | 70.46±0.76 | 85.48±0.75 | 79.83±2.00
CLIP-BERT | 66.97±0.34 | 58.28±0.63 | 77.66±0.64 | 68.44±1.07 | 82.63±3.83 | 80.48±1.95
DisMultiHate | 69.11±0.84 | 62.42±0.72 | 78.21±0.61 | 70.58±1.13 | 83.69±1.33 | 78.05±0.73
PromptHate | 76.76±0.95 | 67.82±1.23 | 76.21±1.05 | 68.08±0.58 | 87.51±0.74 | 79.38±1.72
BLIP | 76.80±2.37 | 69.20±1.84 | 80.59±0.87 | 71.84±1.11 | 87.09±1.46 | 81.81±1.74
ALBEF | 79.40±0.53 | 70.58±0.50 | $\mathbf{83.24}_{\pm 0.93}$ | 72.77±1.00 | 85.49±1.23 | 80.99±0.80
Pro-Cap${}_{\text{BERT}}$ | 77.50±0.58 | 68.14±0.64 | 79.62±0.91 | 71.06±0.88 | 89.04±1.00 | 82.06±1.92
Pro-Cap${}_{\text{PromptHate}}$ | $\mathbf{80.87}_{\pm 0.66}$ | $\mathbf{72.28}_{\pm 0.90}$ | 82.53±0.49 | $\mathbf{73.06}_{\pm 0.82}$ | $\mathbf{90.25}_{\pm 0.54}$ | $\mathbf{83.25}_{\pm 1.00}$
Table 5. Model comparison with augmenting the image entities and demographic information. Dataset | FHM | MAMI | HarM
---|---|---|---
Model | AUC. | Acc. | AUC. | Acc. | AUC. | Acc.
VisualBERT COCO | 72.56±0.80 | 64.28±1.27 | 80.84±0.67 | 72.86±0.71 | 82.96±0.98 | 78.81±0.80
ViLBERT CC | 75.72±0.91 | 68.24±0.44 | 80.33±1.01 | 71.75±1.14 | 84.79±1.23 | 81.39±1.62
MOMENTA | 69.17±4.71 | 61.34±4.89 | 81.68±2.80 | 72.10±2.90 | 86.32±3.83 | 80.48±1.95
DisMultiHate | 79.89±1.71 | 71.26±1.66 | 80.08±0.55 | 71.87±0.47 | 86.39±1.17 | 81.24±1.04
PromptHate | 81.45±0.74 | 72.98±1.09 | 79.95±0.66 | 70.31±0.64 | 90.96±0.62 | 84.47±1.75
BLIP | 76.40±1.49 | 69.29±1.44 | 80.63±1.05 | 70.62±1.48 | 86.88±1.15 | 82.66±1.13
ALBEF | 80.77±0.81 | 71.70±0.98 | 82.45±0.85 | 72.45±0.96 | 86.91±0.72 | 81.78±1.20
Pro-Cap${}_{\text{BERT}}$ | 79.75±1.15 | 71.28±0.91 | 81.20±0.69 | 71.80±1.42 | 89.75±1.49 | 82.71±1.60
Pro-Cap${}_{\text{PromptHate}}$ | $\mathbf{83.58}_{\pm 0.60}$ | $\mathbf{75.10}_{\pm 0.97}$ | $\mathbf{83.77}_{\pm 0.75}$ | $\mathbf{73.63}_{\pm 0.75}$ | $\mathbf{91.03}_{\pm 1.51}$ | $\mathbf{85.03}_{\pm 1.51}$
Table 6. Ablation study about the impact from the length of VQA answers. Ans. Length | FHM | MAMI | HarM
---|---|---|---
No Centric | 70.08±1.57 | 72.78±0.63 | 80.11±1.14
Penalty = 1 | 71.94±0.97 | 73.06±0.82 | 82.09±1.21
Penalty = 2 | 72.28±0.90 | 72.91±1.16 | 82.85±1.51
Penalty = 3 | 71.40±1.06 | 72.47±0.74 | 83.25±1.00
Pro-Cap${}_{\text{PromptHate}}$ | $\mathbf{72.28_{\pm 0.90}}$ | $\mathbf{73.06_{\pm 0.82}}$ | $\mathbf{83.25_{\pm 1.00}}$
Table 7. Comparison between Pro-Cap${}_{\text{PromptHate}}$ with basic PromptHate. The image caption used by basic PromptHate is denoted as basic caption. Incorrect prediction in  red. The content in $(\cdot)$ of the ground-truth is the target of the hateful meme. Meme |  |  |
---|---|---|---
Ground Truth |  Hateful (religion) |  Hateful (religion) |  Hateful (race)
Basic PromptHate |  Non-hateful |  Non-hateful |  Non-hateful
Pro-Cap${}_{\text{PromptHate}}$ |  Hateful |  Hateful |  Hateful
Meme text |  changing every single country it touches |  no that is not his daughter that is his wife yet the world is silent |  the definition of utter disgust in plain black and white
Basic caption |  mughal structure is one of the largest mosques in the world. |  portrait of a father hugging his daughter while smiling at camera in the living room at home. |  love is in the air!.
Pro-Cap |  (Content:) a black cat sitting on a blue and white tiled floor. (Race:) a black person is standing on a blue and white tiled floor in islamic. (Gender:) a man in a black shirt is standing on a blue and white tiled floor with a clock on top of his head. (Country:) islamic. (Religion:) the person is a muslim and he is wearing a black t-shirt and a black sleeveless. |  (Content:) a man and a woman hugging on a couch. (Race:) a white man and a white woman hugging on a white couch. (Gender:) a man and a woman hugging on a white couch. (Country:) islamic. (Religion:) an muslim man and woman hugging on a white couch. |  (Content:) a black and white photo of a man and a woman. (Race:) a black man and a white woman in a black and white photo. (Gender:) a man and a woman in a black and white photo. (Country:) afghanistan. (Religion:) he is a christian.
Table 8. Error cases of Pro-Cap${}_{\text{PromptHate}}$. Meme |  |
---|---|---
GT |  Hateful (gender) |  Non-hateful
Pred |  Non-hateful |  Hateful
Meme text |  scientist are working hard to cure them all |  islam is a religion of peace stop criticizing my religion
Pro-Cap |  (Content:) two women in wedding dresses kissing each other. (Race:) a white woman kissing a brunette woman in a wedding dress. (Gender:) a woman is kissing a man in a wedding dress. (Country:) the person in the image comes from a country in the philippines. (Religion:) the person in the image is a christian. |  (Content:) a man with a beard laughing in the woods. (Race:) a african man with a beard and a red hat is smiling in the woods. (Gender:) a man with a beard and a red hat in front of a wooded area. (Country:) egypt is the country that the person in the image comes from. (Religion:) he is a muslim man with a beard and a red tiara on his head.

###  5.3. Experiment Results

As discussed earlier, previous work has shown that additional image tags can
enhance hateful meme detection. We therefore consider two settings for
comparison: 1) without any augmented image tags; 2) with augmented image tags.
We display the performance of models without augmented image tags in Table 4
and with augmented image tags in Table 5. The standard deviations ($\pm$) of
ten random seed runs are also reported, and the best results are highlighted
in bold.

Without augmented image tags: We first compare Pro-Cap${}_{\text{BERT}}$ with
unimodal and multimodal models that also utilize BERT as the text encoder
(i.e., VisualBERT, ViLBERT, and MMBT-Region). Evidently, Text BERT, which
utilizes only meme text, is substantially outperformed by Pro-
Cap${}_{\text{BERT}}$. This suggests that 1) visual signals are vital for
hateful meme detection, and 2) the image captions obtained from the probing
questions are informative.

Experiment results from multimodal pre-trained BERT-based models are presented
in the second block of Table 4. Interestingly, Pro-Cap${}_{\text{BERT}}$ still
has better performances in all three datasets, surpassing the most powerful
multimodal pre-trained BERT-base model, ViLBERT, by over $\mathbf{4}\%$ on FHM
and surpassing MMBT-Region by about $\mathbf{3}\%$ on HarM. This is despite
the fact that BERT has less model parameters compared with these multimodal
models (e.g, ViLBERT has 252.1M parameters while BERT only has about 110M
parameters). Pro-Cap${}_{\text{BERT}}$ is still competitive against models
specifically designed for hateful meme detection (i.e., models in the third
block of Table 4). We provide experimental results of recently published
multimodal pre-trained models (i.e., BLIP and ALBEF) in the fourth block. By
comparing the simple Pro-Cap${}_{\text{BERT}}$ with these models, we observe
that Pro-Cap${}_{\text{BERT}}$ gives comparable results. While Pro-
Cap${}_{\text{BERT}}$ does not out-perform ALBEF and BLIP all the time,
performance is reasonably good given that in terms of trainable parameters,
Pro-Cap${}_{\text{BERT}}$ is three times smaller than these two pre-trained
models. Meanwhile, Pro-Cap${}_{\text{BERT}}$ shows even better results than
the two models on HarM. Notably, HarM is a real-world dataset which is much
noisier than FHM. HarM also focuses on a relatively new topic (COVID-19),
which may not have been observed a lot by the two pre-trained models.

When comparing BLIP and ALBEF with PromptHate, which has a similar model size,
PromptHate with Pro-Cap demonstrates significant advantages over the two
models on three benchmarks, especially on the noisy HarM dataset. We
conjecture that a possible reason is that multimodal pre-trained models
leverage pre-training data that is relatively cleaner, on a smaller scale and
primarily comprises of non-memes. This leads to some difficulties when
confronted with noisy real-world memes. In contrast pure language models are
pre-trained on larger and noisier data, which may lead to some intrinsic
robustness. If visual signals are reasonably converted to text, pure textual
models can be competitive for multimodal tasks such as hateful meme detection.

Reinforcing the point of proper visual signal conversion, the enhanced
performance of Pro-Cap${}_{\text{PromptHate}}$ over PromptHate highlights the
importance of our probing-based captioning method, which provides essential
cues for hateful content detection. With probe-based captioning, Pro-
Cap${}_{\text{PromptHate}}$ is able to conduct deep multimodal reasoning that
require background knowledge (due to the good performance on FHM), is stable
towards noisy real-world meme data (according to performance on HarM), and has
great generalization in meme detection (according to the good performance on
all three benchmarks).

With augmented image tags: For a fair comparison with recent state-of-the-art
models, we consider testing our proposed probe-captioning method with the same
set of augmented image tags from baselines. To utilize the augmented image
tags, we simply pad these tags at the end of each textual meme representation
in a similar manner to (Cao et al., 2022). With additional image information
such as entities and demographic information, most models have some
improvements. An interesting thing is that neither BLIP nor ALBEF benefits
much from additional image tags. This is because the additional tags are
usually single words or short phrases, which may be noisy or redundant, while
BLIP and ALBEF may be less capable of dealing with noisy inputs. Similar to
the results in Table 4, when augmenting image information: 1) the simple Pro-
Cap${}_{\text{BERT}}$ still obviously surpasses multimodal pre-trained BERT-
base models such as VisualBERT or ViLBERT; 2) the Pro-Cap${}_{\text{BERT}}$
performs better than models with similar sizes but specifically designed for
hateful meme detection (i.e., MOMENTA or DisMultiHate) in most cases; 3) the
Pro-Cap${}_{\text{BERT}}$ achieves comparable results compared with more
powerful multimodal pre-trained models, which is about three times larger and
surpasses them on the HarM dataset, which is real-world and noisy; 4) Pro-
Cap${}_{\text{PromptHate}}$ surpasses the original PromptHate and achieves the
best performance on three benchmarks as well. An interesting point is that
comparing Pro-Cap${}_{\text{PromptHate}}$ without any augmented tags and
original PromptHate with augmented additional image information, they achieve
comparable performance on FHM and HarM and the former even surpasses the
latter on MAMI. However, extracting the additional image information is
expensive and laborious, which can be replaced by probing-based captioning
according to the experimental results. The equally good performance on three
benchmarks highlights the stability and generalization of our proposed
approach.

###  5.4. Ablation Study

In this section, we conduct ablation studies to better understand our Pro-Cap
method. Specifically, we consider the impact of asking different questions and
the impact of the length of answers to the probing questions. To eliminate
other factors, we consider Pro-Cap${}_{\text{PromptHate}}$ without any
augmented image tags. For brevity, we only show accuracy in this section. We
present the full results in Appendix B.

The impact of asking hateful-content centric questions:  We first conduct an
ablation study on the effect of prompting PVLMs with questions facilitating
hateful meme detection. According to Table 2, the first question asks about
the image content while all questions in the second block are for common
vulnerable targets of hateful contents. To better understand the impact of
including image captions generated by these target-specific questions, we
experiment with a setting where captions from the target-specific questions
are removed and only the generic caption about image content is used. The
results are shown in the first block of Table 6. Compared with the last block
of the table, we observe that with captions generated by target-specific
probing questions, the model’s performance improved on all three datasets,
specifically with over $\mathbf{2}\%$ on FHM and over $\mathbf{3}\%$ on HarM.
However, we notice minor improvement on MAMI. We believe that this is because
MAMI memes are all related to woman and generic captions about meme images may
already cover the gender of persons in the image. However, the other two
datasets involve memes with more complexities and therefore asking a wide
ragen of target-specific probing questions is more helpful. It also implies
that in real-world hateful meme detection, probing-based captioning would be
helpful.

The length of answers to probing questions: We apply BLIP-2 as a zero-shot VQA
model. Different from existing VQA benchmarks (Hudson and Manning, 2019; Goyal
et al., 2017), where answers are often single words or short phrases, we may
want the answers used as image captions to be longer and thus more
informative. In this cases, we experiment with answers of different length. To
conduct the analysis, we set the length penalty in BLIP-2’s text decoder for
answer generation with different values (i.e., 1, 2 and 3). With increased
length penalty, longer answers are encouraged. We show results of model
performance with different answer length in Table 6. The results show that
detection performance is robust and does not vary much with different answer
lengths. This indicates the stability of the Pro-Cap method. On the other
hand, to a very small extent, different datasets do favor answers of different
lengths. For instance, the HarM dataset prefers longer answers while the MAMI
dataset prefers shorter answers.

###  5.5. Case Study

In this section, we conduct case studies to better understand the strengths
and limitations of our proposed method. We first compare Pro-
Cap${}_{\text{PromptHate}}$ against PromptHate with image captions and show
examples in Table 7. From the three examples, we observe that in most cases,
generic captions about the image content do not provide the key information
for hateful meme detection, while asking questions about common vulnerable
targets helps. For instance, in the first example, the answer from asking
questions about race, country and religion all provide some key words such as
islamic or muslim; in the second example, answers to questions about country
and religion are important image captions and the answer to the race-related
question is the most important for hateful meme detection. In contrast, we
observe that the basic captions in the original PromptHate miss these crucial
facts about the meme images.

Next, we conduct error analysis about our proposed probe-captioning in Table
8. In the first example, all probe-captions generate sufficient image captions
for the hateful meme detection, while the model still fails at prediction.
This may be due to the current language models performing poorly in further
complex reasoning. We also note that the small scale of hateful meme datasets
may be inadequate for training a model to perform complex reasoning. Recent
studies about large language models pre-trained with trillions of words
(Touvron et al., 2023) may facilitate hateful meme detection to some extent.
Besides, we observe minor errors in predicted answers from the zero-shot VQA
model (e.g., the wrong prediction of “a woman kissing a man” when asking about
gender). It highlights that with the development of better zero-shot VQA
models, the our strategy could potentially facilitate more for the two text-
based hateful meme detection models. The second example highlights a
limitation of most hateful content detection models in that they may be
biased. During the training stage, there may be hateful contents towards
Muslims so that once models seen Muslims, they tend to predict the meme as
hateful. To alleviate the issue, debiasing techniques may be needed. Due to
space limitation, we omit visualization examples in the main pages and refer
the reader to examples in Appendix C.

##  6\. Conclusion

In this study, we attempt to leverage pre-trained vision-language models
(PVLMs) in a low-computation-cost manner to aid the task of hateful meme
detection. Specifically, without any fine-tuning of PVLMs, we probe them in a
zero-shot VQA manner to generate hateful content-centric image captions. With
the distilled knowledge from large PVLMs, we observe that a simple language
model, BERT, can surpass all multimodal pre-trained BERT models of a similar
scale. PromptHate with probe-captioning outperforms previous results
significantly and achieves the new state-of-the-art on three benchmarks.

Limitations: We would like to point out a few limitations of the proposed
method, suggesting potential future directions. Firstly, we heuristically use
answers to all probing questions as Pro-Cap, even though some questions may be
irrelevant to the meme target. We report the performance of PromptHate with
the answer from one probing question in Appendix D, highlighting that using
all questions may not be the optimal solution. A future direction could
involve training a model to dynamically select probing questions that are most
relevant for meme detection. Secondly, although we demonstrate the
effectiveness of Pro-Cap through performance and a case study in this paper,
more thorough analysis is needed. For instance, in the future, we could use a
gradient-based interpretation approach (Selvaraju et al., 2017) to examine how
different probing questions influence the final results, thereby enhancing the
interpretation of the models.

## References

  * (1)
  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. 2022\.  Flamingo: a Visual Language Model for Few-Shot Learning.  _CoRR_ (2022).  arXiv:2204.14198
  * Cao et al. (2022) Rui Cao, Roy Ka-Wei Lee, Wen-Haw Chong, and Jing Jiang. 2022\.  Prompting for Multimodal Hateful Meme Classification. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP_. 321–332.
  * Dai et al. (2023) Wenliang Dai, Zihan Liu, Ziwei Ji, Dan Su, and Pascale Fung. 2023.  Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics EACL_. 2136–2148.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT_. 4171–4186.
  * Fersini et al. (2022) Elisabetta Fersini, Francesca Gasparini, Giulia Rizzi, Aurora Saibene, Berta Chulvi, Paolo Rosso, Alyssa Lees, and Jeffrey Sorensen. 2022.  SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification. In _Proceedings of the 16th International Workshop on Semantic Evaluation, SemEval@NAACL_. 533–549.
  * Gao et al. (2021) Tianyu Gao, Adam Fisch, and Danqi Chen. 2021.  Making Pre-trained Language Models Better Few-shot Learners. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, ACL/IJCNLP_. 3816–3830.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017.  Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR_. 6325–6334.
  * Hee et al. (2023) Ming Shan Hee, Wen-Haw Chong, and Roy Ka-Wei Lee. 2023\.  Decoding the Underlying Meaning of Multimodal Hateful Memes.  _arXiv preprint arXiv:2305.17678_ (2023).
  * Hee et al. (2022) Ming Shan Hee, Roy Ka-Wei Lee, and Wen-Haw Chong. 2022\.  On explaining multimodal hateful meme detection models. In _Proceedings of the ACM Web Conference 2022_. 3651–3655.
  * Hudson and Manning (2019) Drew A. Hudson and Christopher D. Manning. 2019.  GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR_. 6700–6709.
  * Kiela et al. (2019) Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, and Davide Testuggine. 2019.  Supervised Multimodal Bitransformers for Classifying Images and Text. In _Visually Grounded Interaction and Language (ViGIL), NeurIPS Workshop_.
  * Kiela et al. (2020) Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020.  The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes. In _Advances in Neural Information Processing Systems, NeurIPS_.
  * Kumar and Nandakumar (2022) Gokul Karthik Kumar and Karthik Nandakumar. 2022.  Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Features.  _CoRR_ abs/2210.05916 (2022).
  * Lee et al. (2021) Roy Ka-Wei Lee, Rui Cao, Ziqing Fan, Jing Jiang, and Wen-Haw Chong. 2021.  Disentangling Hate in Online Memes. In _MM ’21: ACM Multimedia Conference_. 5138–5147.
  * Li et al. (2023) Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023\.  BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.  _CoRR_ (2023).  arXiv:2301.12597
  * Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022\.  BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. In _International Conference on Machine Learning, ICML_ , Vol. 162. 12888–12900.
  * Li et al. (2021) Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven C. H. Hoi. 2021.  Align before Fuse: Vision and Language Representation Learning with Momentum Distillation.  _CoRR_ (2021).  arXiv:2107.07651
  * Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019.  Visualbert: A simple and performant baseline for vision and language.  _CoRR_ (2019).  arXiv:1908.03557
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014.  Microsoft COCO: Common Objects in Context. In _Computer Vision - ECCV European Conference_ , Vol. 8693. 740–755.
  * Lippe et al. (2020) Phillip Lippe, Nithin Holla, Shantanu Chandra, Santhosh Rajamanickam, Georgios Antoniou, Ekaterina Shutova, and Helen Yannakoudakis. 2020.  A Multimodal Framework for the Detection of Hateful Memes.  _arXiv preprint arXiv:2012.12871_ (2020).
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _CoRR_ (2019).  arXiv:1907.11692
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Fixing weight decay regularization in adam.  _CoRR_ (2017).  arXiv:1711.05101
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019\.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _CoRR_ (2019).  arXiv:1908.02265
  * Mathias et al. (2021) Lambert Mathias, Shaoliang Nie, Aida Mostafazadeh Davani, Douwe Kiela, Vinodkumar Prabhakaran, Bertie Vidgen, and Zeerak Waseem. 2021\.  Findings of the WOAH 5 Shared Task on Fine Grained Hateful Memes Detection. In _Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021)_. 201–206.
  * Mokady et al. (2021) Ron Mokady, Amir Hertz, and Amit H. Bermano. 2021.  ClipCap: CLIP Prefix for Image Captioning.  _CoRR_ (2021).  arXiv:2111.09734
  * Muennighoff (2020) Niklas Muennighoff. 2020\.  Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes.  _CoRR_ (2020).  arXiv:2012.07788
  * Pramanick et al. (2021a) Shraman Pramanick, Dimitar Dimitrov, Rituparna Mukherjee, Shivam Sharma, Md. Shad Akhtar, Preslav Nakov, and Tanmoy Chakraborty. 2021a.  Detecting Harmful Memes and Their Targets. In _Findings of the Association for Computational Linguistics: ACL/IJCNLP_. 2783–2796.
  * Pramanick et al. (2021b) Shraman Pramanick, Shivam Sharma, Dimitar Dimitrov, Md. Shad Akhtar, Preslav Nakov, and Tanmoy Chakraborty. 2021b.  MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets. In _Findings of the Association for Computational Linguistics: EMNLP_. 4439–4455.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021\.  Learning Transferable Visual Models From Natural Language Supervision. In _Proceedings of the International Conference on Machine Learning, ICML_ , Vol. 139. 8748–8763.
  * Ren et al. (2016) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2016\.  Faster R-CNN: towards real-time object detection with region proposal networks.  _IEEE transactions on pattern analysis and machine intelligence_ 39, 6 (2016), 1137–1149.
  * Selvaraju et al. (2017) Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In _IEEE International Conference on Computer Vision, ICCV_. 618–626.
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018\.  Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL_. 2556–2565.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  LLaMA: Open and Efficient Foundation Language Models.  _CoRR_ (2023).  arXiv:2302.13971
  * Velioglu and Rose (2020) Riza Velioglu and Jewgeni Rose. 2020.  Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge.  _CoRR_ (2020).  arXiv:2012.12975
  * Zhou and Chen (2020) Yi Zhou and Zhenhao Chen. 2020.  Multimodal Learning for Hateful Memes Detection.  _arXiv preprint arXiv:2011.12870_ (2020).
  * Zhu et al. (2022) Jiawen Zhu, Roy Ka-Wei Lee, and Wen Haw Chong. 2022\.  Multimodal zero-shot hateful meme detection. In _Proceedings of the 14th ACM Web Science Conference 2022_. 382–389.
  * Zhu (2020) Ron Zhu. 2020.  Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solution.  _CoRR_ (2020).  arXiv:2012.08290

## APPENDIX

Table 9. Comparison between Pro-Cap${}_{\text{PromptHate}}$ and basic PromptHate on HarM dataset. Meme |  |
---|---|---
Ground Truth |  Hateful |  Hateful
Basic Pred |  Non-hateful |  Non-hateful
Pro Pred |  Hateful |  Hateful
Meme text |  now that I have tested positive for COVID-19. It’s time to take this virus seriously. |  Thank you reatDonald Trump for giving your well thought out suggestionfor keeping me safe from COVID-19 atrumpmemes #Covid 19 Anti-covid-19 smoothie
Basic Caption |  i’m going to get tested for a virus!. |  how to clean a kitchen sink with vinegar and food coloring.
Pro-Cap |  (Generic:) trump in a suit and tie with the caption now that positive covid-19 it’s time to take the virus seriously. (Race:) he is a white man in a suit and tie with a red tie and a white hat with a red hat. (Gender: he is a man in a suit and tie with a caption that says now that positive covid-19 it’s time to take. (Country): us of america. (Religion: he is a christian. |  (Generic:) a picture of a blender with cleaning products on it.
Table 10. Comparison between Pro-Cap${}_{\text{PromptHate}}$ and basic PromptHate on MAMI dataset. Meme |  |
---|---|---
Ground Truth |  Hateful |  Hateful
Basic Pred |  Non-hateful |  Non-hateful
Pro Pred |  Hateful |  Hateful
Meme text |  you say you want to be a premium vendor until you know what I had to go threw to get it. |  wish list I would love to send her under the Christmas eve.
Basic Caption |  wallpaper probably with a well dressed person and a well dressed person entitled person. |  western christian holiday from all of us!.
Pro-Cap |  (Generic): a woman is being choked by a man. (Race): a black woman is being choked by a man in a t-shirt with a picture of a t-shirt. (Gender): a woman is being choked by a man in the image. (Country): afghanistan. (Religion): the person in the image is a christian. |  (Generic): a woman in a santa claus hat posing in a bikini. (Race): a white woman in a santa claus hat posing in a sexy bikini. (Gender): a woman wearing a santa claus hat and a bikini. (Country): switzerland. (Religion): santa claus is the religion of the person in the image.
Table 11. Number of parameters in VQA models. Method | # Params (M)
---|---
Text BERT | 109.9
Image Region | 1.0
Visual BERT COCO | 111.8
ViLBERT CC | 252.1
MMBT-Region | 111.5
CLIP BERT | 111.7
MOMENTA | 71.9
DisMultiHate | 115.6
BLIP | 385.0
ALBEF | 209.5
BERT | 109.9
PromptHate | 355.4
Ans. Length | FHM | MAMI | HarM
---|---|---|---
Model | AUC. | Acc. | AUC. | Acc. | AUC. | Acc.
No Centric |  79.08±0.94 |  70.08±1.57 |  82.26±0.71 |  72.78±0.63 |  87.04±0.89 |  80.11±1.14
Penalty = 1 |  80.76±1.06 |  71.94±0.97 |  82.53±0.49 |  73.06±0.82 |  88.34±0.77 |  82.09±1.21
Penalty = 2 |  80.87±0.66 |  72.28±0.90 |  82.27±0.57 |  72.91±1.16 |  90.25±0.72 |  82.85±1.51
Penalty = 3 |  79.62±0.93 |  71.40±1.06 |  82.36±0.97 |  72.47±0.74 |  90.25±0.54 |  83.25±1.00
Table 12. Model comparison without any augmented image tags. Table 13. Model performance when only asking a single probing question. Dataset | FHM | MAMI | HarM
---|---|---|---
Model | AUC. | Acc. | AUC. | Acc. | AUC. | Acc.
Race | $83.63_{\pm 0.26}$ | $74.28_{\pm 1.34}$ | $84.00_{\pm 0.57}$ | $73.51_{\pm 1.10}$ | $90.43_{\pm 0.70}$ | $82.26_{\pm 1.96}$
Gender | $83.91_{\pm 0.97}$ | $76.08_{\pm 1.47}$ | $84.34_{\pm 1.06}$ | $74.21_{\pm 0.64}$ | $91.05_{\pm 0.57}$ | $83.16_{\pm 1.79}$
Religion | $84.85_{\pm 0.87}$ | $75.52_{\pm 1.45}$ | $83.90_{\pm 0.78}$ | $73.95_{\pm 0.84}$ | $90.86_{\pm 0.39}$ | $82.15_{\pm 1.15}$
Nationality | $85.78_{\pm 0.37}$ | $75.72_{\pm 0.96}$ | $83.73_{\pm 0.49}$ | $72.76_{\pm 0.52}$ | $91.27_{\pm 0.68}$ | $84.30_{\pm 1.82}$
Disability | $85.26_{\pm 0.64}$ | $75.96_{\pm 0.82}$ | $83.81_{\pm 0.87}$ | $73.75_{\pm 0.76}$ | $90.20_{\pm 0.82}$ | $84.12_{\pm 0.60}$
Animal | $84.93_{\pm 0.31}$ | $75.48_{\pm 0.72}$ | $84.10_{\pm 0.49}$ | $73.53_{\pm 0.90}$ | $90.13_{\pm 0.87}$ | $82.65_{\pm 2.01}$
Pro-Cap${}_{\text{PromptHate}}$ | $\mathbf{83.58}_{\pm 0.60}$ | $\mathbf{75.10}_{\pm 0.97}$ | $\mathbf{83.77}_{\pm 0.75}$ | $\mathbf{73.63}_{\pm 0.75}$ | $\mathbf{91.03}_{\pm 1.51}$ | $\mathbf{85.03}_{\pm 1.51}$

##  Appendix A Details for Implementation

We implement all models under the PyTorch Library with the CUDA-11.2 version.
We use the Tesla V 100 GPU, each with a dedicated memory of $32$GB. For models
specifically implemented for hateful meme detection, we take the codes
published from the author for re-implementation 444CLIP-BERT/MOMENTA:
https://github.com/LCS2-IIITD/MOMENTA;DisMultiHate:
https://gitlab.com/bottle_shop/safe/dismultihate; PromptHate:
https://gitlab.com/bottle_shop/safe/prompthate. For pre-trained models which
can be found under the Huggingface Library, we use the packages from
Huggingface 555https://huggingface.co/, specifically the BERT (Devlin et al.,
2019), VisualBERT (Li et al., 2019) and the BLIP model. Gor ViLBERT (Lu et
al., 2019), we take the released code from the authors
666https://github.com/facebookresearch/vilbert-multi-task. For ALBEF (Li et
al., 2021) and BLIP-2 (Li et al., 2023), we use the packages under the LAVIS
Library777https://github.com/salesforce/LAVIS.

For each meme image, we constrain the total length of the meme text and the
generic image caption (either from the captioning model or by asking about the
content of the image) to be $65$. For each additional questions, we restrict
its length to be shorter than $20$. If the concatenation of the sentence
exceeds the limited length, the sentence will be truncated, otherwise, if the
sentence is shorted than the limited length, it will be padded. We set the
number of training epochs to be $10$ for all models.

The number of model parameters are summarized in Table 11.

##  Appendix B Full Ablation Study Results

Due to the limitation of space, we only show results of accuracy in ablation
studies in Table 6. The full results including both the AUC and the accuracy
are provided in Table 12.

##  Appendix C Visualization Cases

In Section 5.5, we provide visualization of cases for comparing Pro-
Cap${}_{\text{PromptHate}}$ with the basic PromptHate. Due to space
constraints, we omit examples from the other two datasets. We provide more
visualization cases in this part. The cases from the HarM dataset are
illustrated in Table 9 and the cases from the MAMI dataset are shown in Table
10.

##  Appendix D Results with Pro-Cap about One Target

In Section 5, we only report results when models use Pro-Cap from all probing
questions. In this part, we report results (with entities) when using the
answers from a single probing question in Table 13.

According to the results, we observe models using answers to a single probing
question are all powerful and some even surpass heuristically asking all
probing questions (e.g., using the question asking about nationality on FHM is
better than using all probing questions). It points out using all probing
captions may not be the optimal solution and may generate irrelevant image
descriptions. For instance, confronted with a hateful meme targeting at black
people, it is meaningless to ask the religion of people in the image.
Interestingly, on MAMI, when only using answers to the probing question about
gender reaches teh best performance. It is because MAMI contains only hateful
memes about woman. A promising direction would train the model to dynamically
select probing questions essential for meme detection for different memes.