# Geometrical optics of first-passage functionals of random acceleration

Baruch Meerson<EMAIL_ADDRESS>Racah Institute of Physics, Hebrew
University of Jerusalem, Jerusalem 91904, Israel

###### Abstract

Random acceleration is a fundamental stochastic process encountered in many
applications. In the one-dimensional version of the process a particle is
randomly accelerated according to the Langevin equation
$\ddot{x}(t)=\sqrt{2D}\xi(t)$, where $x(t)$ is the particle’s coordinate,
$\xi(t)$ is Gaussian white noise with zero mean, and $D$ is the particle
velocity diffusion constant. Here we evaluate the $A\to 0$ tail of the
distribution $P_{n}(A|L)$ of the functional
$I[x(t)]=\int_{0}^{T}x^{n}(t)dt=A$, where $T$ is the first-passage time of the
particle from a specified point $x=L$ to the origin, and $n\geq 0$. We employ
the optimal fluctuation method akin to geometrical optics. Its crucial element
is determination of the optimal path – the most probable realization of the
random acceleration process $x(t)$, conditioned on specified $A$, $n$ and $L$.
The optimal path dominates the $A\to 0$ tail of $P_{n}(A|L)$. We show that
this tail has a universal essential singularity, $P_{n}(A\to
0|L)\sim\exp\left(-\frac{\alpha_{n}L^{3n+2}}{DA^{3}}\right)$, where
$\alpha_{n}$ is an $n$-dependent number which we calculate analytically for
$n=0,1$ and $2$ and numerically for other $n$. For $n=0$ our result agrees
with the asymptotic of the previously found first-passage time distribution.

##  I Introduction

The random acceleration process is governed by the Langevin equation

$\ddot{x}(t)=\sqrt{2D}\xi(t)\,.$ (1)

This equation describes the position of a particle moving along the $x$-axis
and subject to a random force which is modeled as a Gaussian white noise with
zero mean, $\langle\xi(t)\xi(t^{\prime})\rangle=\delta(t-t^{\prime})$.
Alternatively, $x(t)$ can be considered as the integral of a Brownian motion
over time. The random acceleration is a fundamental stochastic process in its
own right. On the one hand, it serves as a simple example of a non-Markovian
process (which becomes Markovian when considered in two dimensions $x$ and
$\dot{x}$, see e.g. Ref. BMS2013 ). On the other hand, its mathematical
equivalents have found a variety of applications in physics: from a simplified
description of free semiflexible polymer chains in narrow channels
Burkhardt1993 ; Burkhardt1997 ; Burkhardt2001 ; Burkhardt2007 ; Burkhardt2014
; polymerexplain  to interface growth in 1+1 dimensions GB1991 ; Majumdar2001
; BS  and to decaying turbulence in the Burgers equation Sinai ; Velageas . In
all these systems it is a spatial coordinate which plays the role of time $t$
in Eq. (1), while the polymer shape, or the interface shape, etc. plays the
role of $x$.

Here we are interested in the statistics of first-passage functionals of the
form $I[x(t)]=\int_{0}^{T}x^{n}(t)\,dt$, defined up to the time of first
passage time $T$ of the process, starting say at $x=L>0$, to a specified point
in space, for example to the origin. The case $n=0$ corresponds to the
statistics of the first-passage time itself. The case $n=1$ corresponds to the
area under the graph of $x(t)$ until the first passage to the origin. In the
context of interface growth, governed by the noisy Mullins-Herring equation
GB1991 ; BS , it describes the area under the stochastic interface until it
crosses a zero level in space for the first time. The case $n=2$ corresponds
to the statistics of the moment of inertia of a semiflexible polymer chain of
a given length in narrow channels. It is natural then to attempt to calculate
the distribution of the values of the first-passage functional
$I[x(t)]=\int_{0}^{T}x^{n}(t)\,dt$ for arbitrary $n$.

For comparison, the statistics of first-passage Brownian functionals Kearney ;
Majumdarreview  – where $x(t)$ is a Brownian motion – is well studied, see
Ref. MM2020  and references therein. For the random acceleration process,
however, the problem has been solved only for $n=0$, that is only for the
statistics of the first-passage time itself McKean ; Marshall ; Burkhardt2014
. In the absence of general results for the complete distribution $P_{n}(A|L)$
of the values $I[x(t)]=\int_{0}^{T}x^{n}(t)\,dt=A$, here we focus on the $A\to
0$ tail of this distribution. We show that this tail exhibits an essential
singularity, see Eq. (18) below. To achieve this goal, we employ the optimal
fluctuation method akin to geometrical optics geometricaloptics . The method
relies on the determination of the optimal path, that is the most likely
realization of the process $x(t)$, conditioned on the specified value of $A\to
0$ at given $n$ and $L$. It is this optimal path that dominates the $A\to 0$
tail of $P_{n}(A|L)$. Previously, the geometrical optics was applied to a
plethora of problems related to statistics of Brownian motion Grosberg ; Ikeda
; Schuss ; SM2019 ; M2019 ; MS2019a ; M2019b ; MM2020 ; Agranovetal ; M2020 .
An extension of the method to the random acceleration is a natural next step.

Here is a plan of the remainder of the paper. We complete the formulation of
the problem, establish the scaling properties of $P_{n}(A\to 0|L)$ and derive
the governing equation of the optimal fluctuation method in Sec. II. Some
analytical and numerical solutions for different $n$ are presented in Sec.
III. Section IV includes a brief summary and an extension of our results. A
technical derivation is delegated to the Appendix.

##  II Formulation of the problem and governing equations

We start by completing the formulation of the problem. The initial and final
positions of the particle are

$x(t=0)=L\,,\quad x(T)=0\,,$ (2)

where $T$ is the first passage time to the origin, and $L$ can be assumed
positive without loss of generality. We assume for simplicity that the
particle starts with zero velocity:

$\dot{x}(t=0)=0\,.$ (3)

We consider first-passage functionals of the form
$I[x(t)]=\int_{0}^{T}x^{n}(t)\,dt$ and study the probability distribution
$P_{n}(A|L)$ of their values $A$:

$\int_{0}^{T}x^{n}(t)\,dt=A\,.$ (4)

Equations (1)-(4) define the stochastic problem completely. Their dimensional
analysis (notice that the units of $A$ depend on $n$) yields scalinganalysis
the following _exact_ scaling behavior of $P_{n}(A|L)$:

$P_{n}(A|L)=\frac{D^{1/3}}{L^{n+\frac{2}{3}}}\,F_{n}\left(\frac{D^{1/3}A}{L^{n+\frac{2}{3}}}\right)\,,$
(5)

with a dimensionless scaling function $F_{n}(z)$ of the dimensionless argument
$z=D^{1/3}AL^{-n-\frac{2}{3}}$. The scaling function $F_{n}(z)$ is presently
unknown. On the physical grounds this function is expected to have a single
maximum, of order $1$, at $z\sim 1$ (assuming that $n$ is not too close to $0$
or not too large). Therefore, the probability distribution $P_{n}(A|L)$ is
expected to have its maximum, of order $D^{1/3}/L^{n+\frac{2}{3}}$, at $A\sim
D^{-1/3}L^{n+\frac{2}{3}}$.

Rather than attempting to determine the entire scaling function $F_{n}(z)$,
here we only find its leading-order $z\to 0$ asymptotic. This asymptotic
corresponds to the $A\ll D^{-1/3}L^{n+\frac{2}{3}}$ tail of the distribution
$P_{n}(A|L)$ tooshorttimes . This large-deviation tail can be obtained by the
optimal fluctuation method, akin to geometrical optics. First we need to
identify the action functional, corresponding to the Langevin equation (1). We
start from the probability distribution of a realization of the white Gaussian
noise $\xi(t)$ of unit magnitude, see e.g. Ref. Majumdarreview :

$\mathcal{P}[\xi(t)]\sim\exp\left[-\frac{1}{2}\int_{0}^{T}\xi^{2}(t)\,dt\right]\,.$

Expressing $\xi(t)$ through the particle acceleration $\ddot{x}(t)$ from Eq.
(1), we can evaluate the probability distribution of a realization of the
random acceleration process in the form of $\sim e^{-S[x(t)]}$ with the action
functional

$S[x(t)]=\frac{1}{4D}\int_{0}^{T}\ddot{x}^{2}(t)\,dt\,.$ (6)

The optimal fluctuation method (or geometrical optics) is aimed at finding the
“optimal path” $x_{*}(t)$ that minimizes functional (6) subject to the
boundary conditions (2) and (3), to the positivity condition $x(t)>0$ for
$0<t<T$, and to the integral constraint

$I[x(t)]=\int_{0}^{T}x^{n}(t)\,dt=A\,.$ (7)

The minimization must be performed not only with respect to different paths
$x(t)$, but also with respect to the first-passage time $T$.

Let us rescale the coordinate, $\tilde{x}=x/L$. The action functional (6)
takes the form

$S[x(t)]=\frac{L^{2}}{2D}\,s(\tilde{x}),\quad\text{where}\quad
s(\tilde{x})=\frac{1}{2}\int_{0}^{T}\ddot{\tilde{x}}^{2}(t)\,dt\,.$ (8)

The constraint (7) becomes

$I[\tilde{x}(t)]=\int_{0}^{T}\tilde{x}^{n}(t)dt=\frac{A}{L^{n}}\,.$ (9)

The minimization of the rescaled functional $s(\tilde{x})$ subject to the
constraint (9) can be achieved by minimizing the modified functional

$s_{\lambda}[\tilde{x}(t)]=s[\tilde{x}(t)]-\lambda I[\tilde{x}(t)]\,.$ (10)

The Lagrange multiplier $\lambda$ turns out to be negative, so we can set
$\lambda=-\Lambda^{4}$, where $\Lambda>0$. Now we also rescale time,
$\tilde{t}=\Lambda t$. The first-passage time $T$ also gets rescaled,
$\tilde{T}=\Lambda T$. The functional (10) becomes

$s_{\lambda}[\tilde{x}(\tilde{t})]=\Lambda^{3}\int_{0}^{\tilde{T}}\left[\frac{\ddot{\tilde{x}}^{2}(\tilde{t})}{2}+\tilde{x}^{n}(\tilde{t})\right]d\tilde{t}\,.$
(11)

and we will drop the tildes everywhere in the following. Since the rescaled
functional $s_{0}[x(t)]$ (recall that the tildes are dropped) involves the
particle acceleration $\ddot{x}(t)$, the Euler-Lagrange equation is of the
fourth order (see the Appendix):

$x^{(4)}(t)+nx^{n-1}(t)=0,$ (12)

where the superscript $(4)$ denotes the fourth derivative with respect to
time. Three boundary conditions for Eq. (12) come with the formulation of the
original stochastic problem, see Eqs. (2) and (3):

$x(0)=1\,,\,\,\dot{x}(0)=0\,,\,\,\text{and}\,\,x(T)=0\,.$ (13)

The fourth boundary condition,

$\ddot{x}(T)=0\,,$ (14)

follows from minimization of the action with respect to all possible
variations of the particle velocity $\dot{x}$ at $t=T$ (see the Appendix).

The general solution of the rescaled Euler-Lagrange equation (12) has four
arbitrary constants. When this equation is supplemented by the four boundary
conditions (13) and (14) [and the inequality $x(0<t<T)>0$], the problem of
finding the $A\to 0$ asymptotic of $P_{n}(A|L)$ is determined completely only
for $n=0$ where $A=T$, and one is looking for the distribution $P_{n}(T|L)$ of
first passage times. For all other $n>0$ one should, in addition, minimize the
action $S(A,T)$ with respect to $T$. The minimization yields the _optimal
value_ of the first-passage time $T=T_{*}(A)$ which dominates the probability
$P_{n}(A|L)$ that we are after. As we show in the Appendix, this additional
minimization brings about a fifth boundary condition

$\dddot{x}(T)=0\,.$ (15)

Once the optimal path $x(t)$ and, for $n\neq 0$, the optimal value
$T=T_{*}(A)$, are found, we can determine $\Lambda$ from the relation

$\Lambda=\frac{L^{n}}{A}\int_{0}^{T_{*}}x^{n}(t)\,dt\,,$ (16)

which follows from the constraint (7) or, equivalently, (9). The original
action (6) can now be written as follows:

$\displaystyle S[x(t)]$ $\displaystyle=$
$\displaystyle\frac{L^{2}\Lambda^{3}}{2D}s_{0}[x(t)]\,,\;\>\text{where}$
$\displaystyle s_{0}[x(t)]$ $\displaystyle=$
$\displaystyle\frac{1}{2}\int_{0}^{T}\ddot{x}^{2}(t)dt\,.$ (17)

Plugging Eq. (16) into the first line of Eq. (17) we obtain, up to a pre-
exponential factor, the $A\to 0$ tail of $P_{n}(A|L)$. It scales as

$-\ln P_{n}(A\to 0|L)\simeq S=\frac{\alpha_{n}L^{3n+2}}{DA^{3}}\,,$ (18)

where

$\alpha_{n}=\frac{1}{4}\left[\int_{0}^{T_{*}}x^{n}(t)dt\right]^{3}\int_{0}^{T_{*}}\ddot{x}^{2}(t)dt\,.$
(19)

Equation (18) describes a universal essential singularity $\sim\exp(-A^{-3})$
of the $A\to 0$ tail of the distribution. It is much steeper than the
essential singularity $\sim\exp(-A^{-1})$ of the first-passage Brownian
functionals MM2020 .

In fact, the large-deviation scaling (18) (with a yet unknown $\alpha_{n}$)
immediately follows from the exact scaling (5) once we realize that the $A\to
0$ asymptotic of the function $F_{n}(\dots)$ in Eq. (5) must exhibit, up to a
pre-exponent, the characteristic weak-noise scaling $F_{n}\sim\exp(-\Phi/D)$,
where $\Phi$ depends on $A$ and $L$ but is independent of $D$. Now let us
proceed to finding the optimal path, that is to solving Eq. (12) subject to
the boundary conditions (13)-(15).

##  III Solution

###  III.1 General

Equation (12) is easily solvable for $n=0$, $1$ and $2$, when the equation is
linear. We will present these solutions shortly. In the general case, there is
conservation law

$\dot{x}(t)\,\dddot{x}(t)-\frac{1}{2}\ddot{x}^{2}(t)+x^{n}(t)=C=\text{const},$
(20)

which is a higher-order analog of energy conservation in classical mechanics.
The conservation law (20) reduces the order of Eq. (12) by one. Using the
boundary conditions (13)-(15) at $t=T$, we find that $C=0$ for all $n>0$ Emden
.

Evaluating the left hand side of the conservation law (20) (where $C=0$) at
$t=0$, we uncover one more universal property of the optimal path:

$\ddot{x}(t=0)=-\sqrt{2}\quad\text{for all}\quad n>0\,.$ (21)

Finally, using the conservation law (20) with $C=0$, integration by parts and
Eqs. (13) and (14), we can rewrite the expression (19) for $\alpha_{n}$ in two
equivalent alternative forms:

$\alpha_{n}=\frac{1}{6}\left[\int_{0}^{T_{*}}x^{n}(t)dt\right]^{3}=\frac{27}{32}\left[\int_{0}^{T_{*}}\ddot{x}^{2}(t)dt\right]^{4}\,.$
(22)

###  III.2 $n=0$: First-passage time

The first-passage time distribution $P(T|L)$ of the random acceleration
process was determined quite some time ago McKean ; Marshall ; Burkhardt2014 .
Its short-time asymptotic coincides, in the leading order, with the short-time
asymptotic of the propagator of the random acceleration. For the zero initial
particle velocity, the exact propagator (see e.g. Ref. Burkhardt2014 )
simplifies to

$\rho(T,v)=\frac{\sqrt{3}}{2\pi
DT^{2}}\,e^{-\frac{3L^{2}+3LTv+T^{2}v^{2}}{DT^{3}}}\,,$ (23)

where $v=\dot{x}(t=T)$ is the particle velocity (in the original units) at
$t=T$. We identify the action, corresponding to this distribution,

$S_{\rho}(T,v)=\frac{3L^{2}+3LTv+T^{2}v^{2}}{DT^{3}}\,,$ (24)

and focus on the large-deviation regime $T\to 0$, where this action is much
larger than unity. Minimizing $S_{\rho}(T,v)$ with respect to $v$, we obtain
the optimal value $v_{*}=-3L/(2T)$. The corresponding minimum of the action,

$S_{\rho}(T,v_{*})=\frac{3L^{2}}{4DT^{3}}\,,$ (25)

determines the small-$A$ asymptotic of $P(T|L)$:

$-\ln P(T|L)\simeq\frac{3L^{2}}{4DT^{3}}\,,$ (26)

which obeys our asymptotic scaling relation (18) with $\alpha_{0}=3/4$. Now we
will rederive the asymptotic (26) by using the optimal fluctuation formalism.

For $n=0$ the Euler-Lagrange equation (12) becomes trivial: $x^{(4)}=0$. Its
solution, satisfying the boundary conditions (13) and (14),

$x(t)=1-\frac{3t^{2}}{2T^{2}}+\frac{t^{3}}{2T^{3}}\,,$ (27)

is a cubic parabola. Equation (16) yields $\Lambda=1$. Then, using Eq. (17),
we arrive at Eqs. (25) and (26) as to be expected.

###  III.3 $n=1$: First-passage area

For $n=1$ the Euler-Lagrange equation (12) is still very simple: $x^{(4)}=-1$.
Its solution is a quartic parabola. Here we have to demand all five boundary
conditions (13)-(15) which determine the four arbitrary constants and the
optimal value of the first-passage time $T_{*}=2^{3/4}$. The resulting
rescaled optimal path,

$x(t)=1-\frac{t^{2}}{\sqrt{2}}+\frac{t^{3}}{3\sqrt[4]{2}}-\frac{t^{4}}{24}\,,$
(28)

is depicted, alongside with the optimal acceleration $\ddot{x}(t)$, in Fig. 1.
The optimal acceleration is nothing but the (rescaled) optimal realization of
the white Gaussian noise $\xi(t)$, see Eq. (1). Needless to say, the optimal
realization of the noise looks very differently from a _typical_ realization
of the noise. Now using Eqs. (18) and (19) for $n=1$, we obtain

$-\ln P(A|L)\simeq\frac{108L^{5}}{625DA^{3}}\,,$ (29)

with $\alpha_{1}=108/625$.

Figure 1: The rescaled optimal path $x(t)$ (a) and optimal acceleration
$\ddot{x}(t)$ (b), dominating the $A\to 0$ asymptotics of $P(A|L)$ for $n=1$
(blue) and $n=2$ (magenta). The optimal first-passage time is $T_{*}=2^{3/4}$
for $n=1$ and $T_{*}=\pi/2^{3/4}$ for $n=2$.

For completeness, we also present $\Lambda$, $T_{*}$ and the optimal path
$x(t)$ in dimensional variables:

$\displaystyle\Lambda$ $\displaystyle=$ $\displaystyle\frac{3\cdot
2^{3/4}L}{5A}\,,\quad T_{*}=\frac{5A}{3L}\,,$ $\displaystyle x(t)$
$\displaystyle=$ $\displaystyle
L\left(1-2\tau^{2}+\frac{4\tau^{3}}{3}-\frac{\tau^{4}}{3}\right),$ (30)

where $\tau=t/T_{*}$.

###  III.4 $n=2$

Here the Euler-Lagrange equation (12) is still linear and elementary:

$x^{(4)}(t)+2x(t)=0\,.$ (31)

The solution, obeying the boundary conditions (13)-(15), yields the rescaled
optimal path:

$x(t)=\frac{\left(1-e^{\frac{t}{T_{*}}}\right)\sin\left(\frac{\pi
t}{2T_{*}}\right)+\left(1+e^{\frac{t}{T_{*}}}\right)\cos\left(\frac{\pi
t}{2T_{*}}\right)}{\left(1+e^{-\pi}\right)e^{\frac{\pi t}{2T_{*}}}}\,,$ (32)

where $T_{*}=2^{-3/4}\pi$ is the optimal first passage time. Figure 1 shows
this optimal path alongside with the optimal acceleration $\ddot{x}(t)$.

Using Eqs. (18) and (19) for $n=2$, we obtain

$-\ln
P(A|L)\simeq\frac{27\tanh^{4}\left(\frac{\pi}{2}\right)\,L^{8}}{256DA^{3}}\,,$
(33)

Here $\alpha_{2}=(27/256)\tanh^{4}\left(\pi/2\right)=0.074625\dots$.

In the dimensional variables we have

$\displaystyle\\!\\!\Lambda$ $\displaystyle=$
$\displaystyle\frac{3L^{2}\tanh\,(\pi/2)}{2^{7/4}A}\,,\quad T_{*}=\frac{2\pi
A\coth(\pi/2)}{3L^{2}}\,,$ $\displaystyle\\!\\!\frac{x(t)}{L}\\!\\!$
$\displaystyle=$
$\displaystyle\\!\\!\frac{\left(e^{\pi}\\!-\\!e^{\pi\tau}\right)\sin\left(\frac{\pi\tau}{2}\right)\\!+\\!\left(e^{\pi\tau}\\!+\\!e^{\pi}\right)\cos\left(\frac{\pi\tau}{2}\right)}{1+e^{\pi}e^{\frac{\pi\tau}{2}}}\,,$
(34)

where $\tau=t/T_{*}$.

###  III.5 Numerics

For arbitrary $n$ the optimal path can be found numerically. We used
artificial relaxation in conjunction with iterations over $T$. Artificial
relaxation was implemented as follows. We introduced artificial time $\tau$
and replaced the Euler-Lagrange equation (12) by the fourth-order partial
differential equation

$\partial_{\tau}X(t,\tau)=-\partial_{t}^{4}X(t,\tau)-nX^{n-1}(t,\tau)\,,$ (35)

where the physical time $t$ plays the role of a coordinate. The sign of the
right-hand-side of Eq. (35) is chosen so as to enforce relaxation to a steady-
state, $x(t)=X(t,\tau\to\infty)$ which satisfies our Eq. (12). The initial
condition $X(t,\tau=0)$ is chosen qualitatively similar to the expected
steady-state solution. Since we do not know the optimal first-passage time $T$
a priori, we use iterations. We first solve Eq. (35) with boundary conditions
(13) and (14) for a fixed $T$ (the first guess of $T_{*}$) until the steady-
state solution $x(t)$ is reached. Then we evaluate the third derivative
$\partial_{t}^{(3)}X(t,\tau\gg 1)$ at $t=T$, and iterate $T$ until the third
derivative vanishes [as Eq. (15) demands] with desired accuracy.
Alternatively, one can iterate until $\partial_{t}^{2}X(t,\tau\gg 1)$ at $t=0$
approaches $-\sqrt{2}$, see Eq. (21). We validated the method by comparing the
numerically found $x(t)$ with the analytical solutions for $n=1$ and $2$. The
accuracy was monitored by checking the conservation law (20) with $C=0$. Once
$T_{*}$ and $x(t)$ are found, we can evaluate $\alpha_{n}$ from any of the
equations (19) or (22). We used a standard PDE solver of “Mathematica” wolfram
.

Figure 2 shows the numerically found optimal paths $x(t)$ and the optimal
accelerations $\ddot{x}(t)$ for $n=3$ and $4$. In these cases $T_{*}\simeq
2.036$ and $2.185$, respectively, whereas the $A\to 0$ asymptotics of $P(A|L)$
are described by Eq. (18) with $\alpha_{3}\simeq 0.041$ and $\alpha_{4}\simeq
0.026$. Overall, we solved the problem numerically and found the optimal first
passage time $T_{*}$ and the factor $\alpha_{n}$ for a range of $n$, see Fig.
3. As one can see, $T_{*}$ increases with $n$, while $\alpha_{n}$ decreases.

Figure 2: Numerically found rescaled optimal path $x(t)$ (a) and optimal
acceleration $\ddot{x}(t)$ (b), dominating the $A\to 0$ asymptotics of
$P(A|L)$ for $n=3$ (blue) and $n=4$ (magenta). The optimal first-passage time
is $\simeq 2.036$ for $n=3$ and $T_{*}\simeq 2.20$ for $n=4$.

Figure 3: The optimal first-passage time $T_{*}$, conditioned on Eq. (4) (a),
and the factor $\alpha_{n}$ which enters Eqs. (18) and (19) (b) are plotted as
functions of $n$. The points $n=1$ and $n=2$ were obtained analytically, the
rest of the points numerically.

##  IV Summary and Discussion

Statistics of first-passage functionals provide a useful characterization of
random processes. Here we evaluated the $A\to 0$ tail of these statistics for
the random acceleration process. We also used this problem to extend the
optimal fluctuation (or geometrical optics) method to a stochastic process of
a higher-order. In addition to the $A\to 0$ asymptotic of the probability
distribution $P_{n}(A|L)$, we calculated analytically and numerically the
optimal paths of the conditioned processes at different $n$. These provide an
interesting insight into the nature of large deviations in this system.

The geometrical-optics calculations can be extended to the case where the
particle velocity at $t=0$ is nonzero: $v_{0}\neq 0$. The more interesting
case here is $v_{0}<0$, when the particle arrives at the origin with
probability 1. Here a new effect appears: the particle can reach the origin
_deterministically_ along the ballistic trajectory $x(t)=L-|v_{0}|t$. The
expected first-passage time (not subject to any additional constraint) is
$\bar{T}=L/|v_{0}|$. Evaluating the functional $I[x(t)]$ in Eq. (7) on the
ballistic trajectory, we obtain

$A\equiv\bar{A}=\frac{L^{n+1}}{(n+1)|v_{0}|}\,.$ (36)

If we condition the process on $A<\bar{A}$ or $A>\bar{A}$, the optimal path
will be different from the deterministic one, and it can be found with the
same formalism we used, except that the second condition in Eq. (13) should be
replaced by $\dot{x}(t=0)=-|v_{0}|$ (in the dimensional variables). The
resulting action (6) vanishes at $A=\bar{A}$. In a small vicinity around
$A=\bar{A}$ the action is quadratic in $A-\bar{A}$. For such small
fluctuations of $A$ the probability distribution is approximately Gaussian.
Applicability of geometrical optics requires that the resulting action be much
larger than unity.

The simplest example of such a calculation is the evaluation of the first-
passage time distribution itself, that is $n=0$. Here instead of Eq. (27) we
obtain

$x(t)=L+v_{0}t-\frac{3(L+v_{0}T)t^{2}}{2T^{2}}+\frac{(L+v_{0}T)t^{3}}{2T^{3}}\,.$
(37)

The resulting probability density $P(T,L,v_{0}<0)$ is given, up to a pre-
exponential factor, by the expression

$-\ln P(T,L,v_{0}<0)\simeq S=\frac{3(L-|v_{0}|T)^{2}}{4DT^{3}}\,,$ (38)

which agrees with the short-time asymptotic of the exact propagator of the
random acceleration for nonzero initial particle velocity Burkhardt2014 . As
to be expected, the action (38) vanishes at $T=\bar{T}=L/|v_{0}|$. For
$v_{0}=0$ Eq. (38) coincides with Eq. (25), again as to be expected.

Finally, the problem of statistics of the first-passage functionals
$I[x(t)]=\int_{0}^{T}x^{n}(t)dt$ can be extended to a whole family of
processes, described by the Langevin equation
$d^{k}x(t)/dt^{k}=\sqrt{2D}\xi(t)$, where $k$ is any positive integer. The
cases of $k=1$ and $k=2$ correspond to the Brownian motion and random
acceleration, respectively. Again, let $x(0)=L$, and again suppose for
simplicity that all the derivatives of $x(t)$ with order less than $k$ vanish
at $t=0$. Then the exact scaling behavior of probability distribution
$P_{n}^{(k)}(A|L)$ of the values $I[x(t)]=A$ follows from dimensional
analysis:

$P^{(k)}_{n}(A|L)=\frac{D^{\nu}}{L^{n+2\nu}}\,F^{(k)}_{n}\left(\frac{D^{\nu}A}{L^{n+2\nu}}\right)\,,$
(39)

where $F^{(k)}_{n}(z)$ is an unknown scaling function, and $\nu=1/(2k-1)$. In
its turn, the leading-order $A\to 0$ asymptotic of $P_{k}(A|L)$ must have the
characteristic weak-noise form

$-\ln P_{n}^{(k)}(A\to
0|L)\simeq\frac{\alpha_{n}^{(k)}L^{\frac{n}{\nu}+2}}{DA^{1/\nu}}\,,$ (40)

where $\alpha_{n}^{(k)}$ is a numerical factor which depends on $k$ and $n$.
As one can see from Eq. (40), for all these models theory predicts am
essential singularity at $A\to 0$, and the singularity becomes stronger as $k$
is increased.

## Acknowledgments

The author is very grateful to Satya N. Majumdar for a useful discussion. This
research was supported by the Israel Science Foundation (Grant No. 1499/20).

##  Appendix A Derivation of Eq. (12) and boundary condition (15).

Here we temporarily switch back to the original variables and consider a
linear variation of the constrained action functional

$s_{\lambda}[x(t),T]=\int_{0}^{T}\left(\frac{\ddot{x}^{2}}{2}-\lambda
x^{n}\right)dt$ (41)

with respect to small variations of both $x(t)$ and $T$: $x(t)\to x(t)+\delta
x(t)$ and $T\to T+\delta T$. We need to linearize the variation

$\delta s_{\lambda}=s[x(t)+\delta x(t),T+\delta T]-s[x(t),T]$ (42)

with respect to $\delta x$ and $\delta T$. The linearization yields, after
simple algebra,

$\displaystyle\delta s_{\lambda}$ $\displaystyle=$
$\displaystyle\int_{0}^{T}\left(\ddot{x}\delta\ddot{x}-\lambda nx^{n-1}\delta
x\right)dt$ (43) $\displaystyle+$ $\displaystyle\int_{T}^{T+\delta
T}\left(\frac{\ddot{x}^{2}}{2}-\lambda x^{n}\right)dt\,.$

Performing two integrations in parts in the first integral, evaluating the
second integral in the limit of $\delta T\to 0$, and taking into account the
boundary conditions (13), we obtain

$\displaystyle\delta s_{\lambda}$ $\displaystyle=$
$\displaystyle\int_{0}^{T}\left(x^{(4)}-\lambda nx^{n-1}\right)\delta
x\,dt+\ddot{x}(T)\delta\dot{x}(T)$ (44) $\displaystyle+$
$\displaystyle\left[\frac{\ddot{x}^{2}(T)}{2}-\lambda
x^{n}(T)-\dddot{x}(T)\dot{x}(T)\right]\delta T\,.$

Each of the three terms in the variation must vanish independently for
arbitrary $\delta x$ and $\delta T$. The first term yields the Euler-Lagrange
equation $x^{(4)}-\lambda nx^{n-1}=0$ which, upon the rescaling $\Lambda t\to
t$ (we recall that $\lambda\equiv-\Lambda^{4}$), coincides with Eq. (12). The
second term yields the boundary condition (14). Using the latter condition and
the condition $x(T)=0$ in the third term, we arrive at the condition
$\dddot{x}(T)\dot{x}(T)=0$. Now we have to choose one of two options:
$\dot{x}(T)=0$ or $\dddot{x}(T)=0$ (they cannot hold simultaneously, as the
problem then would be overdetermined). As one can check minmax , the condition
$\dot{x}(T)=0$ would give a local _maximum_ of the action functional $S[x(t)]$
as a function of $T$, whereas the condition $\dddot{x}(T)=0$ yields the
desired minimum.

## References

  * (1) A. J. Bray, S. N. Majumdar and G. Schehr, Advances in Physics 62, 225 (2013).
  * (2) T. W. Burkhardt, J. Phys. A 26, L1157 (1993).
  * (3) T. W. Burkhardt, J. Phys. A 30, L167 (1997).
  * (4) D. J. Bicout and T. W. Burkhardt, J. Phys. A 34, 5745(2001).
  * (5) Y. Yang, T. W. Burkhardt and G. Gompper, Phys. Rev. E 76, 011804 (2007).
  * (6) T. W. Burkhardt, “First Passage of a Randomly Accelerated Particle”, in “First-Passage Phenomena and Their Applications”, edited by R. Metzler, G. Oshanin, and S. Redner (World Scientific, Singapore, 2014), p. 21; arXiv:1603.07017.
  * (7) Indeed, let us consider a long semi-flexible polymer, fluctuating in equilibrium in a narrow cylindrical channel. If the radius of the channel is much smaller than the persistence length of the polymer, a typical polymer configuration only weakly deviates from a straight line, and it can be described by a single-valued function $r(t)$ Burkhardt2014 .
  * (8) L. Golubović and R. Bruinsma, Phys. Rev. Lett. 66, 321 (1991).
  * (9) A.-L. Barabási and H. E. Stanley, Fractal Concepts in Surface Growth (Cambridge University Press, Cambridge, 1995).
  * (10) S. N. Majumdar and A. J. Bray, Phys. Rev. Lett. 86, 3700 (2001).
  * (11) Ya. G. Sinai, Commun. Math. Phys. 148, 601 (1992).
  * (12) P. Velageas, J. Stat. Phys. 134, 589 (2009).
  * (13) M. J. Kearney and S. N. Majumdar, J. Phys. A: Math. Gen. 38, 4097 (2005).
  * (14) S. N. Majumdar, Curr. Sci. 89, 2076 (2005).
  * (15) S. N. Majumdar and B. Meerson, J. Stat. Mech. (2020) 023202.
  * (16) H. P. McKean, J. Math. Kyoto Univ. 2, 227 (1963).
  * (17) T. W. Marshall and E. J. Watson, J. Phys. A 18, 3531 (1985).
  * (18) Geometrical optics approximately describes light waves in terms of rays, or trajectories Feynman . Analogously, geometrical optics of large deviations of Brownian motion approximately describes the probability density of the position of a Brownian particle in terms of optimal trajectories.
  * (19) A. Grosberg and H. Frisch, J. Phys. A: Math. Gen. 36, 8955 (2003).
  * (20) N. Ikeda and H. Matsumoto, in “Memoriam Marc Yor—Séminaire de Probabilités XLVII (Lecture Notes in Mathematics vol 2137), edited by C. Donati-Martin et al (Springer, Berlin, 2015), p. 497.
  * (21) K. Basnayake, A. Hubl, Z. Schuss and D. Holcman, Phys. Lett. A 382, 3449 (2018).
  * (22) N. R. Smith and B. Meerson, J. Stat. Mech. (2019) 023205.
  * (23) B. Meerson, J. Stat. Mech. (2019) 013210.
  * (24) B. Meerson and N. R. Smith, Phys. A: Math. Theor. 52, 415001 (2019).
  * (25) B. Meerson, Int. J. Mod. Phys. B 33, 1950172 (2019).
  * (26) T. Agranov, P. Zilber, N. R. Smith, T. Admon, Y. Roichman and B. Meerson, Phys. Rev. Res. 2, 013174 (2020).
  * (27) B. Meerson, J. Stat. Mech. (2020) 103208.
  * (28) The probability distribution $P_{n}(A|L)$ depends only on the dimensional parameters $A$, $D$ and $L$ and on the dimensionless parameter $n$. In its turn, the dimensions of parameters $A$, $D$ and $L$ include only two parameters with independent dimensions: length and time. By the Buckingham $\Pi$-theorem (see e.g. Ref. Barenblatt ), there is only one independent dimensionless combination of $A$, $D$ and $L$, and it is natural to choose the combination $D^{1/3}AL^{-n-\frac{2}{3}}$, proportional to $A$. Then, using the normalization condition $\int_{0}^{\infty}P_{n}(A|L)\,dA=1$, one arrives at Eq. (5). See Ref. Barenblatt  for many instructive examples of dimensional analysis in physics.
  * (29) At very short times, the random acceleration model (where a real stochastic force is modelled by white noise) breaks down, and some additional, “microscopic” physics should be taken into account. Therefore, as it often occurs in physics, our small-$A$ results should be considered as an intermediate asymptotic: they are expected to be applicable when $A$ is much smaller than $D^{-1/3}L^{n+2/3}$, but much larger than a (system-specific) microscopic time scale where the random-acceleration model breaks down.
  * (30) By introducing a new function $z(x)=[-\dot{x}(t)]^{3/2}$, Eq. (20) can be reduced to a second-order equation  $z^{\prime\prime}(x)=-\frac{3}{2}x^{n}z^{-5/3}(x)\,,$ which is known as the Emden-Fowler equation Polyanin . Unfortunately, its exactly soluble cases do not seem to extend beyond $n=0,1$ and $2$, where it is much easier to solve Eq. (12) directly.
  * (31) Wolfram Research, Inc., Mathematica, Version 13.2, (Champaign, IL, 2022).
  * (32) We did it analytically for $n=1$ and $2$, and numerically for several other $n$.
  * (33) R. P. Feynman, R. B. Leighton, and M. Sands, The Feynman Lectures on Physics, vol. 1 (Addison-Wesley, Reading, MA, 1963).
  * (34) G. I. Barenblatt, Scaling, Similarity, and Intermediate Asymptotics (Cambridge University Press, Cambridge, UK, 1996).
  * (35) A. D. Polyanin and V. F. Zaitsev, Handbook of Exact Solutions of Ordinary Differential Equations (CRC Press, Boca Raton, FL, 1995).


# TOT: Topology-Aware Optimal Transport for Multimodal Hate Detection

Linhao Zhang1,2,3, Li Jin1,2, Xian Sun1,2, Guangluan Xu1,2, Zequn Zhang1,2,
Xiaoyu Li1,2, Nayu Liu1,2,3, Qing Liu1,2, Shiyao Yan1,2,3 Corresponding
Author.

###### Abstract

> Multimodal hate detection, which aims to identify the harmful content online
> such as memes, is crucial for building a wholesome internet environment.
> Previous work has made enlightening exploration in detecting explicit hate
> remarks. However, most of their approaches neglect the analysis of implicit
> harm, which is particularly challenging as explicit text markers and
> demographic visual cues are often twisted or missing. The leveraged cross-
> modal attention mechanisms also suffer from the distributional modality gap
> and lack logical interpretability. To address these semantic gap issues, we
> propose TOT: a topology-aware optimal transport framework to decipher the
> implicit harm in memes scenario, which formulates the cross-modal aligning
> problem as solutions for optimal transportation plans. Specifically, we
> leverage an optimal transport kernel method to capture complementary
> information from multiple modalities. The kernel embedding provides a non-
> linear transformation ability to reproduce a kernel Hilbert space (RKHS),
> which reflects significance for eliminating the distributional modality gap.
> Moreover, we perceive the topology information based on aligned
> representations to conduct bipartite graph path reasoning. The newly
> achieved state-of-the-art performance on two publicly available benchmark
> datasets, together with further visual analysis, demonstrate the superiority
> of TOT in capturing implicit cross-modal alignment.

## Introduction

The flourishing expansion of social media has facilitated the exchange of
opinions among individuals, different cultural or social communities. However,
under the influence of major events such as the Russian-Ukrainian conflict and
COVID-19, these platforms are flooded with hateful content, disseminating
discriminatory statements toward social groups based on their races, religions
or other characteristics. Such hateful content is sowing seeds of disunity to
exacerbate violent and criminal behaviors in conflict areas. Therefore, the
automatic identification of hateful content on the Internet is a social
research issue with great urgency.

Figure 1: Differences between explicit-aligned and implicit-aligned memes. The
aligning procedure is naturally completed in explicit harmful content, but is
indispensable for implicit cases.

Memes, as a popular kind of social media user-generated content (UGC), have
become a prevalent method for propagating harmful content due to their viral
nature. Typically, a meme is an image embedded with a short piece of text.
Although memes have a puckish sense of humor, some implicit hateful content is
concealed. The critical factor in detecting harmful memes is combining well-
aligned visual-linguistic clues and conducting multimodal reasoning.

Previous works have made remarkable progress in detecting explicit-aligned
hate. (Lee et al. 2021; Pramanick et al. 2021). Their models achieved various
kinds of practical reasoning through external knowledge enhancement (Zhu 2020;
Zhou, Chen, and Yang 2021), sophisticated interaction mechanisms (Lippe et al.
2020) or an ensemble of multiple visual-linguistic models (Das, Wahi, and Li
2020; Velioglu, Rose, and Analytics 2020). However, these approaches neglect
the nuance of the multimodal aligning process, which is necessary for
subsequent reasoning procedures. Take the second meme in Figure 1 as an
example, which dehumanizes Pakistanis as pigs. Although the word ’Pakistanis’
has no semantic correspondence with the ’pigs’ in the picture, a human can
easily capture the implicit cross-modal alignment between ’Pakistanis’ and
’pigs’ to perceive the potent offense. Nevertheless, this implicit harm is
challenging for efficient detection. Previous works neglected the aligning
procedure and conducted the reasoning process directly (Pan et al. 2020),
resulting in suboptimal performance on implicit-aligned memes.

Recently, optimal transport (OT), as one of the research hotspots from
optimization theory, has attracted extensive attention in computer vision,
natural language processing and other fields due to its excellent performance
on sequence alignment and domain adaption problems (Duan et al. 2022). OT has
the ability to reduce distributional bias in a more interpretable way under
low-resource scenarios, which solves the problem of lacking external labels
for aligning multimodal clues in implicit harmful memes (Chen et al. 2020;
Maretic et al. 2022).

In this paper, inspired by the OT theory, we propose a general framework TOT:
topology-aware optimal transport to capture implicit cross-modal alignment in
memes. Specifically, we leverage an optimal transport kernel method to
reformulate the alignment problem across different modalities. The leveraged
Gaussian kernel provides the transformation ability to reproduce kernel
Hilbert space (RKHS), in which the distributional modality gap gets eliminated
to calculate an informative cost matrix for generating transportation plans.
Secondly, based on the sinkhorn algorithm, we acquire optimal transportation
plans, which are used for assigning source values to target distribution at
minimum total cost. In our implementation, the cost matrix is a divergence of
alignment represented by the pair-wise dot product of image and text sequence
features in RHKS. Thus the optimal transportation plan with minimum total cost
represents the assigning weights with maximum total alignment. Then we
initialize topology structures to capture inter-modal correspondence between
aligned feature nodes. The initialized structure is dynamically updated
through topology reasoning for several steps to allow comprehensive and
representative information propagation. Finally, we leverage a residual
connection between learned input representations and reasoning scores based on
cross-modal multi-head attention to realize a complementation. Our
contributions are summarized as follows:

  * •

We notice the challenging problem of detecting implicit harmful memes and
leverage optimal transport kernel method to model the cross-modal aligning
task. To the best of our knowledge, this is the first work to incorporate such
kernel methods with transportation theory in multimodal hate content
detection.

  * •

We implement topology-aware optimal transport to capture inter-modal
correspondence, which establishes credible alignment based on transported
feature nodes, and perform dynamic topology reasoning to allow comprehensive
and representative information propagation.

  * •

Without any external knowledge, our model achieves state-of-the-art
performance by a large margin on two publicly available benchmarks. The
subsequent visualization studies further confirm our model’s superiority in
capturing implicit cross-modal alignment.

## Related Work

### Hate Content Detection

With the prevalence of social media platforms, automated identification of
hate content has received academic attention. Researchers from diverse
communities have explored this challenging work (Fortuna et al. 2018), and
produced a large number of benchmark datasets (Bretschneider and Peters 2017;
Ross et al. 2017; Mandl et al. 2021). Previous feature-engineering methods
(Malmasi and Zampieri 2018; Mehdad and Tetreault 2016) mainly extracted and
organized lower-level features, like $n$-gram and sentiment features.
Nowadays, DNN-based methods have garnered better results by aggregating latent
semantic features (Zhang, Robinson, and Tepper 2018) or fine-tuning large pre-
trained models (Tekiroglu, Chung, and Guerini 2020) like GPT-2. Although the
existing methods of hateful content detection have yielded considerable
experimental progress and commercial application, they just focused on text-
based hateful content, neglecting the abundant multimedia UGC.

### Multimodal Hate Content Detection

Frequent and repetitive exposure to multimodal harmful content will increase
personal prejudice against external groups and further affect the status of
socially disadvantaged groups. Enabling explorations were made for flourishing
hateful meme detection studies by publishing benchmarks. For instance,
Facebook had proposed the Hateful Memes Challenge (Kiela et al. 2020), which
encouraged researchers to identify the targeted harmful categories (e.g., race
and sex). (Zia, Castro, and Tyson 2021) proposed to classify memes beyond
hateful and confirm the type of attack (e.g., contempt and slur). Recent
MOMENTA (Pramanick et al. 2021) further extended the limited harmful
categories and proposed two benchmarks related to COVID-19 and US politics.
Approaches of multimodal hateful memes classification (Kiela et al. 2020;
Suryawanshi et al. 2020) adopted early fusion techniques to incorporate
linguistic and visual signals at input space and yield sub-optimal results.
Advanced systems fine-tuned large scale pre-trained unimodal and multimodal
models such as VisualBERT (Li et al. 2019), UNITER (Chen et al. 2019b), VILLA
(Gan et al. 2020), and ensemble thereof. More recently, DisMultiHate (Lee et
al. 2021) disentangled target entities to certain categories of hate (i.e.,
gender, race, etc.) in multimodal memes to improve classification and
explainability. MOMENTA (Pramanick et al. 2021) systematically analyzed the
local and the global perspective of the input meme and performed cross-modal
attention in general level to detect harmful memes. However, aforementioned
rivaling approaches relies on external demographic knowledge and have limited
effectiveness in detecting harmful content beyond categories of gender or
race, especially memes with implicit modal alignment. This paper aims to
address this gap by proposing a general framework with optimal transported
graph.

Figure 2: The model architecture of our TOT. Firstly, the representations of
images and text can be learned through CLIP. Secondly, Kernel Mapping stage
formulates the transportation cost matrix to optimize. Thirdly, we perform
topology aligning and output two graphs with aligned representation. Finally,
TOT conducts reasoning to calculate classification scores.

### Optimal Transport

Optimal transport is a well-studied topic in optimization theory, the original
aim of which is to find the best transportation plan between two data
distributions with minimum cost. Recently, OT has indeed been widely used in
alignment problems by researchers from various communities including computer
vision and natural language processing (Grave, Joulin, and Berthet 2019). VOLT
(Xu et al. 2021) formulated the quest of vocabularization as an optimal
transport (OT) problem by finding the optimal transport matrix from the
character distribution to the vocabulary token distribution. (Chen et al.
2019a) used the transport plan as an ad-hoc attention score in the context of
network embedding to align data modalities. MuLOT (Pramanick, Roy, and Patel
2022) utilized optimal transport based domain adaptation to learn strong
cross-modal dependencies for sarcasm and humor detection. (Ge et al. 2021)
innovatively revisited the label assignment from a global perspective and
proposed to formulate the assigning procedure as an optimal transport (OT)
problem. However, none of these studies have exploited optimal transport to
learn element-level alignment representation as graph nodes. This paper
contributes to memes analysis by proposing a novel framework TOT, which
performs multimodal aligning for implicit harmful memes and constructs
topology alignment graphs to allow dynamic topology reasoning.

## Methods

### Input Representation Learning

Unlike previous works that leveraged isolated encoders for visual and
linguistic signals (Yu et al. 2021), we choose CLIP (Radford et al. 2021) to
learn the unified representations for input memes samples. CLIP is a visual-
linguistic model pre-trained on 400M image–text pairs from the Internet
through contrastive learning, which offers excellent zero-shot capabilities in
capturing semantics for image-text inputs. Towards proposed TOT framework, we
feed a meme’s image and its OCR-extracted text to CLIP model initialized with
pre-trained weights, then four embeddings
$\boldsymbol{v}_{g},\boldsymbol{t}_{g},\boldsymbol{V},\boldsymbol{T}$ from the
multimodal encoder are extracted, where
$\boldsymbol{v}_{g},\boldsymbol{t}_{g}\in\mathbb{R}^{d_{h}}$ are global
representations for multimodal input, and
$\boldsymbol{V}\in\mathbb{R}^{n_{p}^{2}\times d_{h}}$,
$\boldsymbol{T}\in\mathbb{R}^{n_{s}\times d_{h}}$ are sequence vectors
containing local image and text information respectively. Note that
$n_{p}\times n_{p}$ is the patch numbers per image and $n_{s}$ is the word
numbers per sentence.

### Transportation Problem Formulating

This section exploits the Gaussian kernel method (Mialon et al. 2021) to
formulate the cross-modal alignment problem as solutions for optimal
transportation plans.

Based on the CLIP local visual feature sequence
$\boldsymbol{V}=\\{\boldsymbol{v}_{1},\boldsymbol{v}_{2},\dots,\boldsymbol{v}_{n_{p}^{2}}\\}$
and local linguistic feature sequence
$\boldsymbol{T}=\\{\boldsymbol{t}_{1},\boldsymbol{t}_{2},\dots,\boldsymbol{t}_{n_{s}}\\}$,
we can acquire the cost matrix $\mathbf{C}$ towards transportation problem
through a kernel method. Let $\kappa$ be the Gaussian kernel with reproducing
kernel Hilbert space (RKHS) $\mathcal{H}$ and its associated kernel embedding
$\varphi$ : $\mathbb{R}^{d}\rightarrow\mathcal{H}$. Then the $n_{s}\times
n_{p}^{2}$ cost matrix $\mathbf{C}$ can be obtained by calculating the
comparisons $\kappa(\boldsymbol{v}_{i},\boldsymbol{t}_{j})$ before detailed
alignment.

With the cost matrix $\mathbf{C}$, the transportation plan from image to text,
denoted by the $n_{s}\times n_{p}^{2}$ matrix ${\mathbf{P}(\boldsymbol{V,T})}$
is then defined as the unique solution of:

$\displaystyle\mathop{min}\limits_{\mathbf{P}\in
U}\sum\limits_{ij}{\mathbf{C}_{ij}\mathbf{P}_{ij}-\varepsilon H(\mathbf{P})}$
(1) $\displaystyle
H(\mathbf{P})=-\sum_{ij}{\mathbf{P}_{ij}(log(\mathbf{P}_{ij})-1)}$ (2)

where $\mathbf{C}$ represents the pairwise costs to align the elements of
$\boldsymbol{V}$ and $\boldsymbol{T}$, $H(\mathbf{P})$ refers to the entropy
regularization with parameter $\epsilon$ to control the sparsity of
$\mathbf{P}$, $U$ is the space of admissible couplings.

### Topology-Aware Optimal Transport

This section describes the key module of proposed TOT framework, which is
designed to align multimodal representations based on transportation plans,
and then establish topology structures to capture inter-modal correspondence
as candidate edges among aligned feature nodes.

Based on transportation plan ${\mathbf{P}(\boldsymbol{V,T})}$, the aligned
image nodes
$\mathcal{G}^{v}=\\{\boldsymbol{g}_{1}^{v},\boldsymbol{g}_{2}^{v},\dots\boldsymbol{g}_{n_{p}^{2}}^{v}\\}$
can be acquired by:

$\displaystyle\begin{aligned}
\mathcal{G}^{v}&=\sqrt{n_{p}^{2}}\times\left(\sum\limits_{i=1}^{n_{p}^{2}}{\mathbf{P}_{i1}\varphi(\boldsymbol{v}_{i}),\dots,\mathbf{P}_{ip}\varphi(\boldsymbol{v}_{n_{p}^{2}})}\right)\\\
&=n_{p}\times\mathbf{P(V,T)}^{T}\varphi(\mathbf{V})\end{aligned}$ (3)

where the global visual representation $\boldsymbol{v}_{g}$ is appended as the
first node of the aligned image nodes. To establish a topology structure for
comprehensive alignment reasoning, we compute the candidate edge from
$\boldsymbol{g}_{i}^{v}\in\mathcal{G}^{v}$ to
$\boldsymbol{g}_{j}^{v}\in\mathcal{G}^{v}$ as:

$e(\boldsymbol{g}_{i}^{v},\boldsymbol{g}_{j}^{v},\mathbf{W}_{Q},\mathbf{W}_{K})=\frac{exp((\mathbf{W}_{Q}\boldsymbol{g}_{i}^{v})(\mathbf{W}_{K}\boldsymbol{g}_{j}^{v}))}{\sum_{j}{exp((\mathbf{W}_{Q}\boldsymbol{g}_{i}^{v})(\mathbf{W}_{K}\boldsymbol{g}_{j}^{v}))}}$
(4)

where $\mathbf{W}_{Q}\in\mathbb{R}^{d_{h}\times h}$ and
$\mathbf{W}_{K}\in\mathbb{R}^{d_{h}\times h}$ are linear transformations for
generating query and key vector to compute candidate edges. Note that during
edge calculation process, our topology image graph is directed and thus allows
complex information propagation.

Similarly, the aligned text nodes
$\mathcal{G}^{t}=\\{\boldsymbol{g}_{1}^{t},\boldsymbol{g}_{2}^{t},\dots\boldsymbol{g}_{n_{s}}^{t}\\}$
can be acquired based on the transportation plan $\mathbf{P(T,V)}$ from text
to image:

$\displaystyle\begin{aligned}
\mathcal{G}^{t}&=\sqrt{n_{s}}\times\left(\sum\limits_{i=1}^{n_{s}}{\mathbf{P}_{i1}\varphi(\boldsymbol{t}_{i}),\dots,\mathbf{P}_{ip}\varphi(\boldsymbol{t}_{n_{s}})}\right)\\\
&=\sqrt{n_{s}}\times\mathbf{P(T,V)}^{T}\varphi(\mathbf{T})\end{aligned}$ (5)

where the edge calculation of text nodes is based on equation 4. With the
constructed TOT graphs, we perform dynamically topology reasoning to capture
the inner-modal correspondence by iteratively updating graph nodes and
connected edges. The nodes $\boldsymbol{x}_{i}^{n}$ at $n$-th step are updated
as:

$\displaystyle\overline{\boldsymbol{x}}_{i}^{n+1}=\sum_{j}{e(\boldsymbol{x}_{i}^{n},\boldsymbol{x}_{j}^{n},\mathbf{W}_{Q}^{n},\mathbf{W}_{K}^{n})\cdot\boldsymbol{x}_{j}^{n}}$
(6)
$\displaystyle\boldsymbol{x}_{i}^{n+1}=\textrm{ReLU}(\mathbf{W}_{a}^{n}\overline{\boldsymbol{x}}_{i}^{n+1}+b_{a}^{n})$
(7)

where $\boldsymbol{x}_{i}^{n}$ are initialized by taking the $i$-th node from
graph $\mathcal{G}_{v}$ or $\mathcal{G}_{t}$ at $n$ = 0, and
$\mathbf{W}_{Q}^{n},\mathbf{W}_{K}^{n},\mathbf{W}_{a}^{n},b_{a}^{n}$ are
learnable parameters for $n$-th graph layer. After each step of topology
reasoning, the current node $\boldsymbol{x}_{i}^{n}$ is replaced with
$\boldsymbol{x}_{i}^{n+1}$. When $N$ iterations stop, the global node is fed
to a fully-connected layer to infer the reasoning score
$\boldsymbol{s}_{r}\in\mathbb{R}^{d_{h}}$.

### Residual Global Interaction

To complement the topology reasoning results and infer harmfulness from global
perspective, we add residual connection between CLIP global representation and
classification scores based on multi-head cross model attention.

Specifically, for the $i$-th head attention, the multimodal input is
interacted based on the dot-product attention mechanism:

$\displaystyle\textrm{ATT}_{i}(\boldsymbol{v}_{g},\boldsymbol{t}_{g})=\sigma\left(\frac{[\mathbf{W}_{Q_{i}}\boldsymbol{v}_{g}]^{T}[\mathbf{W}_{K_{i}}\boldsymbol{t}_{g}]}{\sqrt{d_{h}/m}}\right)\mathbf{W}_{V_{i}}\boldsymbol{t}_{g}$
(8)

where
$\\{\mathbf{W}_{Q_{i}},\mathbf{W}_{K_{i}},\mathbf{W}_{V_{i}}\\}\in\mathbb{R}^{d_{h}/m\times
d_{h}}$ are learning parameters corresponding to queries, keys and values
respectively. Then the output of $m$ heads attention is concatenated together,
followed by linear transformation and residual connection to get the
multimodal representation $m_{r}$:

$\boldsymbol{m}_{r}=\boldsymbol{t}_{g}+\mathbf{W}_{m}[\textrm{ATT}_{1},\textrm{ATT}_{2},...,\textrm{ATT}_{m}]$
(9)

With the global multimodal representation, a MLP layer is calculated to
acquire the residual global score $\boldsymbol{s}_{g}\in\mathbb{R}^{d_{h}}$.

### Training Objectives

With the residual global score $\boldsymbol{s}_{g}$ and topology reasoning
score $\boldsymbol{s}_{r}$, we get the final multimodal hidden states
$\boldsymbol{s}_{f}\in\mathbb{R}^{d_{h}}$ by weighted sum of both
$\boldsymbol{s}_{g}$ and $\boldsymbol{s}_{r}$. Then the final hidden states
are fed into a linear function followed by a softmax function for the
classification:

$p(y|\boldsymbol{s}_{f})=\text{softmax}(\mathbf{W_{c}}[(1-\gamma)\cdot
s_{g}+\gamma\cdot s_{r}]+b_{c})$ (10)

where $\mathbf{W}_{c}\in\mathbb{R}^{c\times d_{h}}$, $c$ is the category
number of dataset.

The model parameters are optimized through back propagation with minimizing
the standard cross-entropy loss function as the objective:

$\mathcal{L}=-\frac{1}{\lvert\mathcal{D}\rvert}\sum_{k=1}^{\lvert\mathcal{D}\rvert}\mathbf{log}p(y^{k}|\mathbf{s}_{m}^{k})$
(11)

## Experiment

To verify the superiority of our proposed approach, in this section we firstly
describe the evaluation settings. Then we display the model performance on two
benchmarks of our TOT against other state-of-the-art unimodal and multimodal
approaches. Finally, we conduct qualitative studies to analyze TOT’s
superiority in aligning clues from multiple modalities. We also display the
error cases to discuss limitations of our model.

### Datasets Settings

#### Datasets

We evaluate our model on two publicly available harmful memes detection
datasets, Harm-C and Harm-P, which consist of real-world memes relate to
COVID-19 and US politics, respectively. The statistic information of the two
benchmarks are displayed in Table 1. To make a fair comparison with the
previous work, we report three metrics: Accuracy (Acc), Macro-F1 (F1), and
Macro-Averaged Mean Absolute Error (MMAE).

Dataset | Harmfulness | Train | Test | Valid | Total
---|---|---|---|---|---
Harm-C | Very | 182 | 21 | 10 | 213
Partially | 882 | 103 | 51 | 1036
Non | 1949 | 230 | 116 | 2295
Total | 3013 | 354 | 177 | 3544
Harm-P | Very | 216 | 25 | 17 | 258
Partially | 1270 | 148 | 69 | 1487
Non | 1534 | 182 | 91 | 1807
Total | 3020 | 355 | 177 | 3552
Table 1: Statistics of Harm-C and Harm-P dataset. Modality | Model | 2-class Harmful Meme Detection | 3-class Harmful Meme Detection
---|---|---|---
Harm-C | Harm-P | Harm-C | Harm-P
Acc | F1 | MMAE | Acc | F1 | MMAE | Acc | F1 | MMAE | Acc | F1 | MMAE
Text | BERT | 70.17 | 66.25 | 0.2911 | 80.12 | 78.35 | 0.1660 | 68.93 | 48.72 | 0.5591 | 74.55 | 54.08 | 0.7742
RoBERTa | 70.77 | 66.40 | 0.2923 | 80.74 | 78.64 | 0.1644 | 70.11 | 48.94 | 0.5584 | 74.87 | 54.33 | 0.7771
Bertweet | 71.32 | 67.30 | 0.2842 | 81.88 | 79.34 | 0.1573 | 71.65 | 49.82 | 0.5512 | 75.22 | 54.87 | 0.7652
Image | DenseNet | 68.42 | 62.54 | 0.3125 | 74.05 | 73.68 | 0.1845 | 65.21 | 42.15 | 0.6326 | 71.80 | 50.98 | 0.8388
ResNet | 68.74 | 62.97 | 0.3114 | 73.14 | 72.77 | 0.1800 | 65.29 | 43.02 | 0.6264 | 71.02 | 50.64 | 0.8900
Multimodal | MMBT | 73.48 | 67.12 | 0.3258 | 82.54 | 80.23 | 0.1413 | 68.08 | 50.88 | 0.6474 | 78.14 | 58.03 | 0.7008
ViLBERT | 78.53 | 78.06 | 0.1881 | 87.25 | 86.03 | 0.1276 | 75.71 | 48.82 | 0.5329 | 84.66 | 64.70 | 0.6982
Visual BERT | 81.36 | 80.13 | 0.1857 | 86.80 | 86.07 | 0.1318 | 74.01 | 53.85 | 0.5303 | 84.02 | 63.68 | 0.7020
CLIP | 78.10 | 77.64 | 0.2010 | 84.02 | 83.85 | 0.1508 | 71.05 | 45.55 | 0.5887 | 80.75 | 60.23 | 0.7058
MOMENTA | 83.82 | 82.80 | 0.1743 | 89.84 | 88.26 | 0.1314 | 77.10 | 54.74 | 0.5132 | 87.14 | 66.66 | 0.6805
Ours | TOH | 82.16 | 81.36 | 0.1877 | 88.56 | 86.93 | 0.1428 | 75.44 | 53.08 | 0.5276 | 84.71 | 64.21 | 0.6957
TOA | 83.55 | 82.14 | 0.1813 | 89.17 | 87.44 | 0.1367 | 76.25 | 53.74 | 0.5233 | 85.34 | 65.39 | 0.6864
COT | 86.48 | 85.28 | 0.1697 | 90.24 | 90.88 | 0.1278 | 81.56 | 54.21 | 0.5147 | 86.37 | 69.04 | 0.6742
TOT | 87.01 | 85.93 | 0.1634 | 91.55 | 91.29 | 0.1245 | 82.76 | 55.38 | 0.5027 | 88.61 | 71.54 | 0.6697
| $\triangle_{\text{TOT-best\\_model}}$ | 3.19 | 3.13 | 0.0109 | 1.71 | 3.02 | 0.0031 | 5.66 | 0.64 | 0.0105 | 1.47 | 4.88 | 0.0108
Table 2: Overall performance of proposed TOT model against a set of baselines.
For Acc ($\uparrow$) and F1 ($\uparrow$), the larger values are better , while
for MMAE($\downarrow$), smaller values are better.

#### Baselines

For both Harm-C and Harm-P, we compare the model performance with a set of
strong baselines:

  * •

BERT, RoBERTa, Bertweet: For the language models, we choose influential pre-
trained models: BERT (Kenton and Toutanova 2019), RoBERTa (Liu et al. 2019)
and Bertweet (Nguyen, Vu, and Nguyen 2020) .

  * •

DenseNet, ResNet: For the visual models, we choose two famous models pre-
trained on ImageNet: DenseNet (Huang et al. 2017) and ResNet (He et al. 2016).

  * •

MMBT: This is a multimodal bitransformer (Kiela et al. 2019), which is able to
capture the intra-modal and the inter-modal dynamics of the two modalities.

  * •

ViLBERT: This is a robust model trained on an intermediate multimodal
objective (conceptual captions) (Sharma et al. 2018) with task-agnostic
representations.

  * •

Visual BERT: This is a visual BERT (Li et al. 2019) pre-trained on the COCO
dataset (Lin et al. 2014).

  * •

MOMENTA: This is a novel multimodal deep neural network using background
context to detect hate.

#### Implementation Details

For each meme, we take the $d_{h}=512$ to extract representations. We limit
the max lengths of image and text feature sequence by setting $n_{p}^{2}=49$
and $n_{s}=77$ respectively, which adopts the same configuration as the pre-
training process. For the Kernel Mapping, we take Gaussian kernel with
$\epsilon=0.1$ and set the max numbers of sinkhorn iteration for as 3. For the
topology reasoning, we take a configuration of reason step $N=3$ and a
dimension $h=256$ for node vectors.

### Quatitative Analysis

In this section, we report the 2-class and 3-class detection results on Harm-C
and Harm-P datasets to demonstrate the superiority of proposed approach from
multiple perspectives. Note that we implement three variations (TOH, TOA, COT)
from TOT model to demonstrate superiority. TOH is a variation by replacing the
topology aligning module with heuristic aligning (Diao et al. 2021), and TOA
replaces the topology aligning module with attentional aligning. COT directly
concatenates the multimodal features.

#### 2-class Detection Results

The left part of Table 2 reports the experimental results on Harm-C and Harm-P
as the binary case. Firstly, it can be observed Bertweet achieves relative
outstanding performance in the unimodal approaches. A possible reason for the
phenomenon could be due to the excellent social media capability of Bertweet,
which is pre-trained on Twitter corpus. Secondly, the multimodal approaches
achieve overall superior performance against unimodel ones. And approaches
which finetunes the large-scale pre-trained visual-linguistic models are
better compared Intermidiate Fusion, as the pre-trained models are able to
capture stronger inter-modal dynamics of the two modalities. Thirdly, we can
see that our proposed TOT model outperforms the existing methods by a larger
margin, with 3.13% absolute points of M-F1 improvement on Harm-C and 3.02%
points on Harm-P. The other two metrics of Acc and MMAE also get improved by a
competitive margin, demonstrating the effectiveness of our method.

#### 3-class Detection Results

To further validate models’ ability in fine-grained harmful contents
detection, we also report the 3-class classification results in Table 2.
Similar phenomena can be observed compared to the binary case. Our proposed
TOT outperforms other methods by a larger margin, with 5.66% absolute Accuracy
points on Harm-C and 4.88% M-F1 points on Harm-P. It is worth noted that
competitive performance can be also achieved by COT, which directly
concatenates the alignment representations instead of building transported
graphs, further validating the superiority of our models.

### Ablation Studies

In this section, we have conducted extensive ablation studies to verify the
effectiveness of the three modules implemented in our method, including
optimal transported image/text graphs (OTI/OTT), dynamic topology reasoning
(DTOR) and Residual Global Interaction (REG). All the comparative experiments
are conducted in the binary classification task.

#### Optimal Transported Graphs Ablation

Datasets | OTT | OTI | DTOR | Acc | M-F1
---|---|---|---|---|---
Harm-C | × |  |  | 86.41 | 85.33
| × |  | 86.17 | 85.07
|  | × | 85.78 | 84.46
× |  | × | 84.93 | 83.76
| × | × | 84.82 | 84.64
× | × | × | 83.74 | 82.46
Harm-P | × |  |  | 90.34 | 90.17
| × |  | 90.07 | 89.88
|  | × | 89.71 | 89.43
× |  | × | 87.46 | 87.27
| × | × | 87.39 | 87.14
× | × | × | 86.73 | 86.24
Table 3: Ablation results of transported graphs.

Table 3 shows the binary classification results of different settings for
optimal transported graphs and topology graph reasoning module (DTOR). For the
models without DTOR, we adopt a MLP layer instead to output the topology
reasoning scores. Based on the results we can make observations that the two
transported graphs OTT and OTI are both helpful for classification. OTT and
OTI contribute to the Accuracy score at 1.08% and 1.09% absolute points in
Harm-C. And when the two graphs are exploited together, they make further
contributions by performing additional complementary effect. And the
implementation of DTOR brings performance improvement at 1.2% average absolute
points in accuracy based on OT graphs.

#### Residual Global Interaction Ablation

This subsection explores the complementary effect of classification scores by
ablating topology reasoning score and residual global score. We also report
different training strategies for these two modules: joint learning and
independent learning, which aims to further verify their complementary effect
during learning. The ablation results are shown in Table 4. We can observe
that the scores from these two modules together achieve better performance
compared to any single score (1.57% points in Acc and 1.45% points in M-F1 for
Harm-C dataset), which well confirm the effectiveness of REG module. Besides,
the joint training process contributes more compared to separately training by
an improvement of 0.59% in Acc, as the two modules perform better
complementary effect when training jointly.

Datasets | DTOR | REG | Split | Joint | Acc | M-F1
---|---|---|---|---|---|---
Harm-C | ✓ |  | ✓ |  | 85.44 | 84.48
| ✓ | ✓ |  | 84.37 | 83.27
✓ | ✓ | ✓ |  | 86.42 | 85.67
✓ | ✓ |  | ✓ | 87.01 | 85.93
Harm-P | ✓ |  | ✓ |  | 89.83 | 89.44
| ✓ | ✓ |  | 87.27 | 87.03
✓ | ✓ | ✓ |  | 91.15 | 90.87
✓ | ✓ |  | ✓ | 91.55 | 91.29
Table 4: Abaltion results of reasoning modules.

### Qualitative Results

The ability to formulate and capture cross-modal alignment for deciphering
implicit hate is a core contribution in our TOT model. Toward this research
goal, we design qualitative analysis including case study and visualizations.

Figure 3: The visualizations of edge values in TOT. Raw_Image denotes the
input meme image, Aligned Regions denotes the manually selected image regions
containing alignments with text information for disseminating harm. The two
negative samples are randomly selected sub-image incorporated for comparison.
Figure 4: Case studies of memes from Harm-C and Harm-P datasets.

#### Visualizations

We visualize the edge values after $N$ step iterations of topology reasoning
between global text nodes and other candidate nodes. Figure 3 displays what
information the ToT module will aggregate given different visual features as
input queries. The representations of visual regions are acquired by taking
the average values of all the image patches that contains the target. For the
offensive memes, TOT module captures implicit alignment between the meaningful
instances (’mascot of the democratic party’) and the slovenly donkey in the
picture. Moreover, negative samples are exploited as queries to make
comparison. It is observed that when exploiting negative samples, the final
global node randomly aggregates meaningless information, as the misaligned
representations by transportation plans influence the performance of topology
reasoning.

#### Case Study

As shown in Figure 4, we display several memes from the leveraged datasets,
together with the heat map of edge values between global visual nodes and
other candidate nodes. The text information in first meme is an innocent
description, while its picture shows offensive content, which satirizes ’The
Democrat Party’ in the sentence. The second meme disseminates extreme
multimodal harm reflected by the ’CHINA VIRUS’ in the text and the word ’JOKE’
with double meaning, which is implicitly aligned with the vicious painted
picture of Trump. The third picture has a discrimination harm. The two
sentences seem to describe irrelevant things, but when they are put together
with the image, in which a man is firing a gun, the alignment between ’KILL
THE VIRUS’ and ’A CHINESE BOY COUGHING’ is established, and results in the
extreme harm on race discrimination. These three cases demonstrate the
necessity of aligning multimodal features.

Besides, we are also interested in exploring the limitations of our TOT. The
wrongly classified memes are displayed in Figure 4. The fourth example is a
funny meme, which shows a cute sheepdog watching the sheep in the screen to do
his herding. Note that Wilson is the name of the dog. However, the
anthropomorphized expression in the sentence, together with the dog pictures
misleads our TOT to make wrong predictions, which is quite similar to the
dehumanizing hateful content. The similar situation appears in the fifth meme,
which may indicate that well cross-modal alignment is necessary but not
adequate for detecting harm.

## Conclusion

In this paper, we put forward a novel research problem of detecting implicit-
aligned harmful memes, where this is the first work to incorporate such kernel
methods with transportation theory in multimodal hate content detection.
Specifically, topology-aware optimal transport (TOT) is conducted to formulate
the cross-modal aligning problem as solutions for optimal transportation
plans. With the dynamic topology reasoning module (DTOR), our TOT allows
comprehensive information propagation based on the aligned information to make
accurate reasoning. TOT is evaluated on two publicly available datasets, and
our extensive experiments have shown that TOT outperforms the state-of-the-art
baselines. We have also conducted visualizations and case studies to
empirically demonstrated TOT’s ability to capture implicit cross-modal
alignment. While the error case shows that well cross-modal alignment is
necessary but not adequate for detecting harm. We will incorporate a more
advanced reasoning module for future improvements.

## Acknowledgements

This research was funded by the National Natural Science Foundation of China
(62206267), and the Strategic Priority Research Program of the Chinese Academy
of Sciences grant (Y835120378).

## References

  * Bretschneider and Peters (2017) Bretschneider, U.; and Peters, R. 2017.  Detecting Offensive Statements towards Foreigners in Social Media.  In _HICSS_ , 1–10.
  * Chen et al. (2020) Chen, L.; Gan, Z.; Cheng, Y.; Li, L.; Carin, L.; and Liu, J. 2020.  Graph Optimal Transport for Cross-Domain Alignment.  In _ICML_ , volume 119, 1542–1553.
  * Chen et al. (2019a) Chen, L.; Wang, G.; Tao, C.; Shen, D.; Cheng, P.; Zhang, X.; Wang, W.; Zhang, Y.; and Carin, L. 2019a.  Improving Textual Network Embedding with Global Attention via Optimal Transport.  In _ACL_ , 5193–5202.
  * Chen et al. (2019b) Chen, Y.; Li, L.; Yu, L.; Kholy, A. E.; Ahmed, F.; Gan, Z.; Cheng, Y.; and Liu, J. 2019b.  UNITER: Learning UNiversal Image-TExt Representations.  _CoRR_ , abs/1909.11740.
  * Das, Wahi, and Li (2020) Das, A.; Wahi, J. S.; and Li, S. 2020.  Detecting hate speech in multi-modal memes.  _arXiv preprint arXiv:2012.14891_.
  * Diao et al. (2021) Diao, H.; Zhang, Y.; Ma, L.; and Lu, H. 2021.  Similarity reasoning and filtration for image-text matching.  In _AAAI_ , 1218–1226.
  * Duan et al. (2022) Duan, J.; Chen, L.; Tran, S.; Yang, J.; Xu, Y.; Zeng, B.; and Chilimbi, T. 2022\.  Multi-modal Alignment using Representation Codebook.  In _CVPR_ , 15630–15639. IEEE.
  * Fortuna et al. (2018) Fortuna, P.; Ferreira, J.; Pires, L.; Routar, G.; and Nunes, S. 2018.  Merging Datasets for Aggressive Text Identification.  In _COLING_ , 128–139.
  * Gan et al. (2020) Gan, Z.; Chen, Y.-C.; Li, L.; Zhu, C.; Cheng, Y.; and Liu, J. 2020.  Large-scale adversarial training for vision-and-language representation learning.  _Advances in Neural Information Processing Systems_ , 33: 6616–6628.
  * Ge et al. (2021) Ge, Z.; Liu, S.; Li, Z.; Yoshie, O.; and Sun, J. 2021.  OTA: Optimal Transport Assignment for Object Detection.  In _CVPR_ , 303–312.
  * Grave, Joulin, and Berthet (2019) Grave, E.; Joulin, A.; and Berthet, Q. 2019.  Unsupervised Alignment of Embeddings with Wasserstein Procrustes.  In _AISTATS_ , volume 89, 1880–1890.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _CVPR_ , 770–778.
  * Huang et al. (2017) Huang, G.; Liu, Z.; Van Der Maaten, L.; and Weinberger, K. Q. 2017.  Densely connected convolutional networks.  In _CVPR_ , 4700–4708.
  * Kenton and Toutanova (2019) Kenton, J. D. M.-W. C.; and Toutanova, L. K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of NAACL-HLT_ , 4171–4186.
  * Kiela et al. (2019) Kiela, D.; Bhooshan, S.; Firooz, H.; and Testuggine, D. 2019.  Supervised Multimodal Bitransformers for Classifying Images and Text.  In _NeurIPS_.
  * Kiela et al. (2020) Kiela, D.; Firooz, H.; Mohan, A.; Goswami, V.; Singh, A.; Ringshia, P.; and Testuggine, D. 2020.  The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes.  In _NeurIPS_ , 2611–2624.
  * Lee et al. (2021) Lee, R. K.; Cao, R.; Fan, Z.; Jiang, J.; and Chong, W. 2021.  Disentangling Hate in Online Memes.  In _MM_ , 5138–5147.
  * Li et al. (2019) Li, L. H.; Yatskar, M.; Yin, D.; Hsieh, C.; and Chang, K. 2019.  VisualBERT: A Simple and Performant Baseline for Vision and Language.  _CoRR_ , abs/1908.03557.
  * Lin et al. (2014) Lin, T.; Maire, M.; Belongie, S. J.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014.  Microsoft COCO: Common Objects in Context.  In _ECCV_ , volume 8693, 740–755.
  * Lippe et al. (2020) Lippe, P.; Holla, N.; Chandra, S.; Rajamanickam, S.; Antoniou, G.; Shutova, E.; and Yannakoudakis, H. 2020.  A multimodal framework for the detection of hateful memes.  _arXiv preprint arXiv:2012.12871_.
  * Liu et al. (2019) Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _CoRR_ , abs/1907.11692.
  * Malmasi and Zampieri (2018) Malmasi, S.; and Zampieri, M. 2018.  Challenges in discriminating profanity from hate speech.  _J. Exp. Theor. Artif. Intell._ , 30(2): 187–202.
  * Mandl et al. (2021) Mandl, T.; Modha, S.; Shahi, G. K.; Jaiswal, A. K.; Nandini, D.; Patel, D.; Majumder, P.; and Schäfer, J. 2021.  Overview of the HASOC track at FIRE 2020: Hate Speech and Offensive Content Identification in Indo-European Languages.  _CoRR_ , abs/2108.05927.
  * Maretic et al. (2022) Maretic, H. P.; Gheche, M. E.; Chierchia, G.; and Frossard, P. 2022.  fGOT: Graph Distances Based on Filters and Optimal Transport.  In _AAAI_ , 7710–7718.
  * Mehdad and Tetreault (2016) Mehdad, Y.; and Tetreault, J. R. 2016.  Do Characters Abuse More Than Words?  In _SIGDIAL_ , 299–303.
  * Mialon et al. (2021) Mialon, G.; Chen, D.; d’Aspremont, A.; and Mairal, J. 2021.  A Trainable Optimal Transport Embedding for Feature Aggregation and its Relationship to Attention.  In _ICLR_.
  * Nguyen, Vu, and Nguyen (2020) Nguyen, D. Q.; Vu, T.; and Nguyen, A.-T. 2020.  BERTweet: A pre-trained language model for English Tweets.  In _EMNLP_ , 9–14.
  * Pan et al. (2020) Pan, H.; Lin, Z.; Fu, P.; Qi, Y.; and Wang, W. 2020.  Modeling Intra and Inter-modality Incongruity for Multi-Modal Sarcasm Detection.  In _Findings of EMNLP_ , 1383–1392.
  * Pramanick, Roy, and Patel (2022) Pramanick, S.; Roy, A.; and Patel, V. M. 2022.  Multimodal Learning using Optimal Transport for Sarcasm and Humor Detection.  In _WACV_ , 546–556.
  * Pramanick et al. (2021) Pramanick, S.; Sharma, S.; Dimitrov, D.; Akhtar, M. S.; Nakov, P.; and Chakraborty, T. 2021.  MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets.  In _EMNLP_ , 4439–4455.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _ICML_ , 8748–8763.
  * Ross et al. (2017) Ross, B.; Rist, M.; Carbonell, G.; Cabrera, B.; Kurowsky, N.; and Wojatzki, M. 2017\.  Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis.  _CoRR_ , abs/1701.08118.
  * Sharma et al. (2018) Sharma, P.; Ding, N.; Goodman, S.; and Soricut, R. 2018.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _ACL_ , 2556–2565.
  * Suryawanshi et al. (2020) Suryawanshi, S.; Chakravarthi, B. R.; Arcan, M.; and Buitelaar, P. 2020.  Multimodal meme dataset (MultiOFF) for identifying offensive content in image and text.  In _TRAC@LREC_ , 32–41.
  * Tekiroglu, Chung, and Guerini (2020) Tekiroglu, S. S.; Chung, Y.; and Guerini, M. 2020.  Generating Counter Narratives against Online Hate Speech: Data and Strategies.  In _ACL_ , 1177–1190.
  * Velioglu, Rose, and Analytics (2020) Velioglu, R.; Rose, J.; and Analytics, S. D. 2020.  Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge.  _arXiv preprint arXiv:2012.12975_.
  * Xu et al. (2021) Xu, J.; Zhou, H.; Gan, C.; Zheng, Z.; and Li, L. 2021.  Vocabulary Learning via Optimal Transport for Neural Machine Translation.  In _ACL_ , 7361–7373.
  * Yu et al. (2021) Yu, W.; Xu, H.; Yuan, Z.; and Wu, J. 2021.  Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis.  In _AAAI_ , 10790–10797.
  * Zhang, Robinson, and Tepper (2018) Zhang, Z.; Robinson, D.; and Tepper, J. A. 2018.  Detecting Hate Speech on Twitter Using a Convolution-GRU Based Deep Neural Network.  In _ESWC_ , 745–760.
  * Zhou, Chen, and Yang (2021) Zhou, Y.; Chen, Z.; and Yang, H. 2021.  Multimodal learning for hateful memes detection.  In _2021 IEEE International Conference on Multimedia & Expo Workshops_, 1–6.
  * Zhu (2020) Zhu, R. 2020.  Enhance multimodal transformer with external label and in-domain pretrain: Hateful meme challenge winning solution.  _arXiv preprint arXiv:2012.08290_.
  * Zia, Castro, and Tyson (2021) Zia, H. B.; Castro, I.; and Tyson, G. 2021.  Racist or sexist meme? classifying memes beyond hateful.  In _WOAH_ , 215–219.