somespace rowpostcode =somespace, margins = centering

# A Case for a Binary Black Hole System Revealed via Quasi-Periodic Outflows

Dheeraj R. Pasham1, Francesco Tombesi2,3,4,5,6, Petra Suková7,
Michal Zajaček8, Suvendu Rakshit9,
Eric Coughlin10, Peter Kosec1,23, Vladimír Karas7,
Megan Masterson1, Andrew Mummery11, Thomas W.-S. Holoien12, Muryel Guolo13,
Jason Hinkle14, Bart Ripperda15,16,17, Vojtěch Witzany18,
Ben Shappee14, Erin Kara1, Assaf Horesh19,
Sjoert van Velzen20, Itai Sfaradi19, David L. Kaplan21,
Noam Burger19, Tara Murphy22, Ronald Remillard1,
James F. Steiner23, Thomas Wevers24, Riccardo Arcodia1,
Johannes Buchner25, Andrea Merloni25, Adam Malyali25, Andy Fabian26,
Michael Fausnaugh1, Tansu Daylan27, Diego Altamirano28,
Anna Payne14, E. C. Ferrara5,6,29.
1Kavli Institute for Astrophysics and Space Research,
Massachusetts Institute of Technology, MA, USA
2Physics Department, Tor Vergata University of Rome, Via della Ricerca
Scientifica 1, 00133 Rome, Italy
3INAF – Astronomical Observatory of Rome, Via Frascati 33, 00040 Monte Porzio
Catone, Italy
4INFN - Rome Tor Vergata, Via della Ricerca Scientifica 1, 00133 Rome, Italy
5Department of Astronomy, University of Maryland, College Park, MD 20742, USA
6NASA Goddard Space Flight Center, Code 662, Greenbelt, MD 20771, USA
7Astronomical Institute of the Czech Academy of Sciences, Prague, Czech
Republic
8Department of Theoretical Physics and Astrophysics, Masaryk University, Brno,
Czech Republic
9Aryabhatta Research Institute of Observational Sciences (ARIES), Manora Peak,
Nainital, 263002, India
10Department of Physics, Syracuse University, Syracuse, NY, USA
11Astrophysics, Department of Physics, University of Oxford, UK
12The Observatories of the Carnegie Institution for Science, 813 Santa Barbara
St., Pasadena, CA 91101, USA
13Department of Physics and Astronomy, Johns Hopkins University, MD, USA
14Institute for Astronomy, University of Hawaii, HI, USA
15School of Natural Sciences, Institute for Advanced Study, 1 Einstein Drive,
Princeton, NJ 08540, USA
16NASA Hubble Fellowship Program, Einstein Fellow
17Center for Computational Astrophysics, Flatiron Institute, 162 5th Avenue,
New York, NY 10010, USA
18Charles University, Prague, Czechia
19Racah Institute of Physics, The Hebrew University of Jerusalem, Israel
20Leiden Observatory, Leiden University, The Netherlands
21University of Wisconsin Milwaukee, WI, USA
22University of Sydney, Australia
23Center for Astrophysics | Harvard & Smithsonian, 60 Garden St, Cambridge, MA 02138, USA
24European Southern Observatory, Chile
25Max-Planck Institute for Extraterrestrial Physics, Germany
26University of Cambridge, UK
27Department of Physics and McDonnell Center for the Space Sciences,
Washington University, St. Louis, MO 63130, USA
28University of Southampton, UK
29Center for Research and Exploration in Space Science & Technology II (CRESST
II),
NASA/GSFC, Greenbelt, MD 20771, USA
∗To whom correspondence should be addressed; E-mail<EMAIL_ADDRESS>
> Binaries containing a compact object orbiting a supermassive black hole are
> thought to be precursors of gravitational wave events, but their
> identification has been extremely challenging. Here, we report quasi-
> periodic variability in X-ray absorption which we interpret as quasi-
> periodic outflows (QPOuts) from a previously low-luminosity active galactic
> nucleus after an outburst, likely caused by a stellar tidal disruption. We
> rule out several models based on observed properties and instead show using
> general relativistic magnetohydrodynamic simulations that QPOuts, separated
> by roughly 8.3 days, can be explained with an intermediate-mass black hole
> secondary on a mildly eccentric orbit at a mean distance of about 100
> gravitational radii from the primary. Our work suggests that QPOuts could be
> a new way to identify intermediate/extreme-mass ratio binary candidates.

##  1 Introduction

ASASSN-20qc (?) is an astrophysical flare that originated from the nucleus of
a galaxy at a redshift of 0.056 (luminosity distance of 260 Mpcs). It was
discovered by the All-Sky Automated Survey for SuperNovae (ASAS-SN; (?, ?)) on
20 December 2020. Throughout the paper, we reference times with respect to
this discovery date of Modified Julian Date (MJD) 59203.27. A follow-up
optical spectrum revealed the presence of several Hydrogen and Oxygen emission
lines which facilitated the estimate of the redshift (?) (Methods section
LABEL:sisec:data) and a supermassive black hole (SMBH) mass of
log$(M_{\bullet}/M_{\odot})=7.5^{+0.7}_{-0.3}$ (Methods section
LABEL:subsec:bhmass and LABEL:sec:host; Table LABEL:tab:mbh). A luminosity of
6$\times$1040 erg s-1 from archival eROSITA data (Fig. 1a and Methods section
LABEL:sisec:data) indicates that prior to the outburst it was a low-luminosity
Active Galactic Nucleus (AGN; see Methods section LABEL:subsec:bpt) accreting
at $<$0.002% of its Eddington limit.

Roughly 52 days after ASASSN-20qc’s optical discovery the Neil Gehrels Swift
observatory (Swift) observed it and detected X-rays. Following this detection,
the Neutron star Interior Composition ExploreR (NICER) started a high-cadence
(1-2 visits per day) monitoring program (Fig. 1a and 1c). We analyzed the
NICER soft X-ray (0.3-1.1 keV) energy spectra in the early phases of the
outburst and found that the spectrum was thermal (accretion disk dominated)
and contained systematic residuals reminiscent of a broad absorption trough
(Fig. 1d). We also obtained an XMM-Newton observation on 14 March 2021 (MJD
59287.34), roughly a month after the first NICER exposure, noting the presence
of broad absorption residuals. Subsequent NICER spectra taken at various
epochs of the outburst revealed that this absorption was variable throughout
the outburst. A detailed photo-ionization modeling indicates that the dominant
absorption feature is due to O VIII transitions in the 0.75-1.00 keV observed
energy band blue-shifted with a mildly-relativistic velocity of about 30% of
the speed of light. This evidence is indicative of an ultra-fast outflow (UFO)
(?). See Fig. LABEL:fig:xmmresiduals, Methods section
LABEL:sisec:spectral_model for a detailed discussion on X-ray spectral
modeling.

##  2 Results

To probe the interplay between the variable outflow and the thermal continuum
emission we calculated the ratio of the observed, background-subtracted count-
rates in the energy band dominated by the outflow (0.75-1.00 keV) and the
continuum emission (0.30-0.55 keV) (see shaded regions of Fig. 1d). This
quantity, which we define as the Outflow Deficit Ratio (ODR), quantifies the
amplitude of the outflow variability with respect to the continuum and it is
shown in Fig. 2a. Surprisingly, the ODR curve showed repeating variations with
a $\approx$8.5 day quasi-periodicity, which are not present in the unabsorbed
continuum emission (SFs LABEL:fig:lcs and LABEL:fig:hid).

To quantify the variability and search for quasi-periodic signals in Fig. 2a,
we computed the Lomb-Scargle Periodogram (LSP; (?, ?)) of the ODR curve (Fig.
2b). The highest power in the LSP is at $8.3\pm 0.3$ days in multiple
neighboring bins and is consistent with the time series in Fig. 2a. To
estimate the false alarm probability that takes into account multiple bins,
i.e., the chance probability of generating a signal as strong as the one
observed from noise, we devised a detailed Monte Carlo method (see Fig.
LABEL:fig:whitetests, Methods sections LABEL:sisec:odrtiming and
LABEL:sisec:noisenature). The global false alarm probability of the observed
$\approx$8.5 d quasi-periodicity is $<$2$\times$10-5 ($>$4.2$\sigma$; see Fig.
2c). Global refers to a blind search for signal over all the frequencies
sampled, i.e., $\sim$a day to 100 days (see Fig. 2b).

To further probe the nature of this quasi-periodicity, we extracted and fitted
time-resolved NICER X-ray spectra from individual maxima and minima in the ODR
curve (see Methods section LABEL:sisec:timeresolspecs, Methods section
LABEL:sisec:nicercompspecs, Table LABEL:tab:_nicerxraydata, and Fig.
LABEL:fig:specplots). Notably, the outflow has an order of magnitude higher
column density ($N_{H}$) during the minima phases of the ODR curve with
respect to the maxima: median value of $(12.6\pm 5.5)\times$1021 cm-2 and
$(1.8\pm 0.7)\times$1021 cm-2 for the minima and the maxima, respectively. The
ionization parameter, defined as log$\xi$$=$$L/nr^{2}$, in units of erg s-1
cm, where $L$ is the ionizing luminosity between 1 Ryd and 1000 Ryd (1 Ryd $=$
13.6 eV), $n$ is the number density of the material, and $r$ is the distance
of the gas from the central source, is on average only slightly higher during
the minima than during the maxima. Instead, the outflow bulk velocity is
stable at around 0.35$c$, where $c$ is the speed of light (see Fig.
LABEL:fig:specplots).

Based on the above timing analysis and the time-resolved spectral modeling, we
conclude that ASASSN-20qc exhibits quasi-periodic outflows (QPOuts) about once
every 8.5 days (precisely 8.3$\pm$0.3 days). By the term QPOuts we denote
quasi-periodic variations of the outflowing material (see Fig. 2).

##  3 Discussion

We considered several theoretical models to interpret the above observations
including a precessing inner accretion disk, clumpy or slow outflow, X-ray
reflection, accretion disk instabilities, quasi-periodic eruptions, repeating
partial tidal disruption event (TDE), but disfavor them based on several
independent lines of arguments. See Methods section
LABEL:sisec:spectral_model, LABEL:sisec:singleclump, LABEL:sisec:lateufo and
Methods section LABEL:sisec:othermodels for more details and Table 1 for a
summary of the strengths and the weaknesses of these various models.

Instead, we propose a viable model with an orbiting, inclined perturber that
repeatedly crosses the inner accretion flow. This scenario can explain the
presence of QPOuts if the perturber is characterized by a sufficiently large
influence radius at a given distance (?, ?). To further verify this model, we
performed extensive 2D GRMHD simulations of an object in orbit around a SMBH
using the HARMPI code (?, ?) based on the original HARM code (?, ?) (see
Methods section LABEL:msec:modeldiscussion for details). Regardless of the
specific setup, QPOuts are triggered by the passing perturber once per its
orbit (see Fig. 3 and Table LABEL:tab:GRMHD_runs for an overview). The
simulations predict a persistent magnetized outflow from the inner flow with a
roughly constant radial velocity profile, which is mass-loaded periodically
when the secondary crosses the primary disk. This is consistent with the
observation of a persistent outflow in the maxima which is boosted during the
minima of the ODR. For all the cases, the perturber is highly inclined with
respect to the equatorial plane of the accretion flow, which leads to the
recurrent, periodic mildly-relativistic outflow regardless of the background
accretion-flow state. An ordered and stable poloidal magnetic field in the
funnel region accelerates the ejected matter to mildly-relativistic
velocities. Furthermore, a mildly eccentric orbit with an eccentricity of
$0.5-0.7$ can naturally induce departures from strict periodicity which is
evident from the LSP peak FWHM of $\sim 1$ day as well as from the outflow-
rate temporal profiles in the bottom panels in Fig. 3. One caveat of the 2D
GRMHD simulations is that while magnetorotational instability/MRI–which is
responsible for accretion onto the SMBH–is active at the distance of the
perturber, it decays after a few$\times$10,000 $M$ (or $\sim$100 days) in the
inner regions of the accretion flow ($\sim$a few gravitational radii). Thus,
making direct comparisons of simulations to data beyond 100 days becomes
challenging. However, since the observed QPOuts span about 100 days, our
simulations with active MRI were performed on similar timescales and they show
that such a scenario provides a potential mechanism for producing QPOuts.
Further work using 3D GRMHD simulations where MRI does not decay with time are
needed to track such systems for extended periods (see SI section
LABEL:sisec:caveats for more discussion).

The observed ratio of the outflow to the inflow rate of about 20% during the
ODR minima is consistent with a perturber influence radius of $\mathcal{R}\sim
3$ gravitational radii when compared to the analogous ratio derived from GRMHD
simulations (see SI section LABEL:sisec:naturepert for details). Independent
of the GRMHD simulations, simple analytical reasoning yields a similar
estimate (see the second paragraph of SI section LABEL:sisec:naturepert).
Taking into account that the ejected outflow clumps originate in the
underlying flow, which can be treated as an advection-dominated accretion flow
(ADAF: (?), (?)), and their sizes are comparable to $\mathcal{R}$, such a
length-scale would be in agreement with the inferred column density of 1022
cm-2 of the spectroscopically detected UFO. Considering the tidal (Hill)
length-scale of a massive perturber as well as the radius, within which the
surrounding gas comoves with the perturber, we arrive at a rather broad range
of the perturber masses $\sim 10^{2}-10^{5}\,M_{\odot}$. This broad range
already includes the uncertainty in the primary SMBH mass (see SI section
LABEL:sisec:naturepert for further discussion and EDFs
LABEL:fig_influence_radius, LABEL:fig_Hill_syn_radius).

Distinct from the QPOuts, the optical light curve shown in the Fig.
LABEL:fig:uvotevol exhibits a smooth rise, peak, and decay on a timescale of
$\sim$ 150 days. This timescale is broadly consistent with the canonical
fallback time of the debris from a TDE (e.g., (?)) with a black hole mass of
$M_{\bullet}\sim 10^{7}M_{\odot}$ and a solar-like star. The evolution of the
optical/UV temperature and photosphere radius during the outburst is also very
similar to those of known TDEs (compare the Fig. LABEL:fig:uvotevol with
Figure 8 of (?) and Figure 1 of (?)). The time delay between the X-ray and the
optical outbursts of a few months (see Fig. 1a) has also been seen in several
TDEs (e.g., (?, ?)). Finally, the soft X-ray spectrum is also strikingly
similar to thermal X-ray TDEs. Therefore, a reasonable interpretation is that
the overall outburst in the optical, UV and X-rays was induced by a TDE, which
produces a bright inner accretion disk, i.e., a soft X-ray source, which is
quasi-periodically obscured by the blobs driven by the orbiting perturber (see
Fig. 4).

Attributing the outburst to a TDE we can further constrain the mass of the
secondary based on the argument that the gravitational wave inspiral time
should be greater than the typical time for a stellar disruption in a galaxy.
Using a TDE rate of 10-4 yr-1 (e.g., see Fig. 10 in (?)) would require the
SMBH–perturber system have a merger timescale of $\gtrsim 10^{4}$ years. This
limits the perturber mass to the range of $10^{2}$–$10^{4}\,M_{\odot}$, i.e.,
to the intermediate-mass black-hole (IMBH) range (see bottom panel of Fig.
LABEL:fig_Hill_syn_radius). For such mass and distance of the secondary, the
gravitational radiation is weak and the period of the system will not evolve
significantly in the next decade, making the signal lay outside the frequency
range of the upcoming space-based gravitational-wave observatory LISA. The
unique combination of a SMBH-IMBH pair experiencing the TDE makes such
observation rather rare, though not entirely implausible. Within the
cosmological volume inside $z\sim 0.06$, we estimate $N_{\rm
pair,TDE}=0.07-5.3$ TDEs per year in hosts with tight SMBH-IMBH pairs (out of
$\sim 2.5$ million galaxies; see SI section LABEL:subsec_formation_channels
for further discussion on the estimated event rate and the detectability of
the system in gravitational waves).

In summary, our work highlights the new astrophysical phenomenon of QPOuts and
the importance of high cadence optical and X-ray monitoring observations to
potentially uncover electromagnetic signatures of tight binary black hole
systems. The identification of such SMBH–IMBH binaries, i.e.,
intermediate/extreme mass ratio inspirals (I/EMRIs), has fundamental
implications for multi-messenger astrophysics and for our understanding of
black hole growth and evolution.

Figure 1: ASASSN-20qc’s long-term evolution and a sample X-ray spectrum
highlighting the outflow. (a) ASASSN-20qc’s observed X-ray and optical
evolution. Orange data represents X-ray (0.3-1.1 keV) data acquired by various
instruments. The blue data shows the Gaia magnitude. The horizontal (dashed)
line represents NICER’s sensitivity limit of 3$\times$1042 erg s-1 for a
source at redshift, $z,=0.056$. (b) The combined X-ray spectrum using all
NICER data acquired over epochs of high absorption (yellow) and the best-fit
emission model (black histogram). (c) Zoom-in of the outburst near the X-ray
peak. (d) Ratio of the average energy spectrum using all NICER data acquired
over epochs of minima in outflow deficit ratio (ODR) and the best-fit thermal
model. The outflow band is defined as the 0.75-1.00 keV band while the
inflow/accretion band is defined as the bandpass where the ratio is near 1,
i.e., 0.30-0.55 keV band.

Figure 2:  Summary of ASASSN-20qc’s timing analysis. (a) ASASSN-20qc’s outflow
deficit ratio (ODR) versus time. ODR is defined as the ratio of background-
subtracted count rates in 0.75-1.00 keV (outflow) and 0.3-0.55 keV (continuum)
bands. A lower ODR value implies a stronger outflow and vice versa. The dashed
vertical red lines are uniformly separated by 8.5 days. (b) Lomb Scargle
periodogram (LSP) of the ODR. The strongest signal is near 8.5 days. The
horizontal dashed red lines show the 3 and 4$\sigma$ global false alarm
probabilities as per (?). The noise in the periodogram is consistent with
white with a mean LSP power value of 1 (see Methods section
LABEL:sisec:noisenature). (c) Global (trials-accounted) false alarm
probability. This curve was generated using extensive Monte Carlo simulations
(see Methods section LABEL:sisec:odrtiming). The global statistical
significance of the 8.5 d quasi-periodicity is $>$ 4.2$\sigma$.

Figure 3: A sample snapshot from our GRMHD simulation (2D HARM, Run 14 from
Table LABEL:tab:GRMHD_runs). For this case, the SMBH mass was set to 107.4 M⊙
and the perturbing companion is in an elliptical orbit (eccentricity, e=0.5)
with an observed orbital period of 8.5 days and has an influence radius of 3
gravitational radii
($1M=GM_{\bullet}/c^{2}=0.25(M_{\bullet}/10^{7.4}\,M_{\odot})\,{\rm AU}$). (a)
Spatial distribution of the logarithm of mass density expressed in arbitrary
units. The horizontal and the vertical axes are spatial coordinates expressed
in gravitational radii (units of $M$). The white contours indicate the
magnetic field configuration. The position and size of the perturber is shown
by the black circle, while the grey line displays its trajectory in the 2D
slice. (b) Spatial distribution of the Lorentz factor of the gas bulk motion.
(c) Spatial distribution of the mass-outflow rate with $v>0.2c$. The outflow
rate is colour-coded using arbitrary units according to the colour-bar to the
right. (d) Temporal profiles of the inflow rate (blue), the outflow rate
through the upper funnel (purple), and the outflow rate through the lower
funnel (green). The inflow and the outflow rates are expressed in arbitrary
units. The time is expressed in days in the observed frame. The coloured
points/dots indicate the time of the snapshot. Vertical lines are uniformly
separated by 8.5 days. (e, f) Lomb Scargle Periodogram of the ratio of the
outflow to the inflow rates (e) and the accretion rate (f) from run 14 sampled
exactly as the real data. The peak signal in panel (e) is broad with a value
of 8.5${}^{+0.7}_{-1.1}$ d and is consistent with the observed value of
8.3$\pm$0.3 d (shaded blue band) while no such signal is present in the
accretion rate periodogram (f), i.e., an elliptical binary can reproduce the
observed quasi-periodicity in the outflow strength without similar variations
in the continuum.

Figure 4: Schematic of a potential model for ASASSN-20qc. A gravitationally
bound (pre-existing) IMBH located at roughly 100 Rg from the central SMBH can
explain the repeated outflows seen here. The overall outburst could have been
triggered by a tidal disruption of a passing star by the SMBH which creates a
compact accretion disk which naturally enhances the X-ray emission and
consequently illuminates the surrounding environment and the presence of the
IMBH secondary. Secondary plunges through the pre-existing (non-TDE) accretion
flow, modulating the outflow on the orbital period. Relative sizes are not to
scale.

Table 1: A summary of the strengths and the weaknesses of various models considered in this work to explain ASASSN-20qc’s observed X-ray spectro-timing variability. See Methods sections LABEL:sisec:spectral_model, LABEL:sisec:singleclump, LABEL:sisec:lateufo and SI for more details. |  |  |
---|---|---|---
Model/class of models |  Strengths |  Weaknesses |  Notes
Inner disk precession |  Thought to be commonly seen in stellar-mass black hole binaries (?) |  The lack of strong continuum modulation and the observed changes from high column, high ionization parameter to low column, low ionization along with constant outflow speed and more importantly the lack of a strong quasi-periodicity in the 0.3-0.55 keV continuum variations are inconsistent with precession with all known types of outflows (?, ?, ?, ?, ?) |  Disfavoured based on physical reasoning
Clumpy outflow |  - |  The outflow geometry would need to be fine tuned to have uniformly separated clumps. The probability of formation of such clumps by chance is less than 1 in 50,000 |  Disfavoured due to low likelihood
Slow outflow |  Slow outflow can, in principle, produce similar spectral signatures |  \- The XMM-Newton/RGS and EPIC/pn spectrum rule out a slow outflow that can produce such a broad feature
\- A typical slow outflow is distant from the SMBH and cannot produce a rapid ($\sim$ week timescale) quasi-periodic variability seen here |  Disfavoured based on physical reasoning (see Methods section LABEL:sisec:noslow)
X-ray reflection by a corona |  Seen in several highly accreting AGN with an X-ray corona (?) |  Lack of a Compotonizing corona/powerlaw component in the X-ray spectrum |  Disfavoured based on lack of evidence in data (see Methods section LABEL:sisec:spectral_model)
X-ray reflection by a disk |  Argued to operate at least in one changing-look AGN (?) |  \- Lack of a geometrically thick surface for reflection, would require a fine-tuned disk geometry
\- Unphysically large fraction of reflected emission compared to the primary thermal emission |  Disfavoured based on statistical argument and physical grounds (see Methods section LABEL:sisec:spectral_model for more discussion)
Magnetically arrested accretion disk |  Preliminary work by (?) suggests that outflows can be produced through repeated magnetic reconnection events |  \- Based on state-of-the-art high-resolution simulations it is unclear if such outflows would be quasi-periodic in nature
\- Such regular outflows are not seen in lower-resolution simulations
\- Lack of strong quasi-periodicity in the continuum variations |  viable but no clear indication in the state-of-the-art simulations (but see Fig. 8 of (?))
Quasi-periodic eruptions (QPEs) |  Seen in a small sample of AGN |  \- QPEs manifest as large amplitude flux bursts as opposed to changes in ODR.
\- Variable outflows have not been reported in known QPE sources. |  Disfavoured because the observed signal is distinct (see Methods section LABEL:supsec:qpemodel)
Repeating partial stellar tidal disruption |  Argued to operate in at least 3 systems (?, ?, ?) |  \- The expected orbital period would be orders of magnitude longer than what is seen here (?)
\- No evidence for a similar variability in the optical light curve
\- A stellar core’s influence radius would be too small to produce the observed outflow |  Disfavoured based on physical reasoning
Stellar debris stream |  Could provide obscuration when highly inclined |  -Stellar debris would be tidally spread along the whole orbit, turning off the periodicity; the material would need to be continuously replenished (see partial TDE above) |  Disfavoured based on physical reasoning
Radiation pressure driven outflows |  Observed in a sample of accreting stellar-mass black holes (?) |  \- The persistence of the outflow over a factor of $>$200 change in X-ray flux suggests negligible radiation driving
\- Fine tuning of the disk properties for obtaining short-enough instability
period (?)
\- No evidence for a similar variability in the soft X-ray continuum |  Disfavoured based on the need for fine tuning
A scaled-up version of quasi-periodic oscillations |  Occurring in stellar-mass black-hole binaries (e.g., (?)) |  The lack of a strong quasi-periodicity in the thermal continuum (0.3-0.55 keV band) |  Disfavoured due to lack of a precendent
An orbiting object repeatedly perturbing the SMBH accretion disk |  \- can explain QPOuts
\- ultrafast outflow production supported by GRMHD simulations
\- Consistent with TDE statistics and production rates of SMBH–IMBH binaries (?) |  \- An IMBH distance of $\sim 100\,r_{\rm g}$ makes full 3D GRMHD simulations computationally expensive
\- For 2D simulations, magnetorotational instability enabling accretion tends to stop operating after sometime (within a few$\times$10,000 $M$ which corresponds to $\sim$100 days), which makes comparison with data limited at later epochs |  Viable but no precedent (see SI section LABEL:msec:modeldiscussion and LABEL:sisec:grmhdsims; see section LABEL:sisec:caveats for caveats)
|  |  |

## Material and Methods.


# On Explaining Multimodal Hateful Meme Detection Models

Ming Shan Hee  Singapore University of
Technology and DesignSingaporeSingapore<EMAIL_ADDRESS>,  Roy
Ka-Wei Lee  Singapore University of
Technology and DesignSingaporeSingapore<EMAIL_ADDRESS>and  Wen-Haw Chong
Singapore Management UniversitySingaporeSingapore
<EMAIL_ADDRESS>
(2022)

###### Abstract.

Hateful meme detection is a new multimodal task that has gained significant
traction in academic and industry research communities. Recently, researchers
have applied pre-trained visual-linguistic models to perform the multimodal
classification task, and some of these solutions have yielded promising
results. However, what these visual-linguistic models learn for the hateful
meme classification task remains unclear. For instance, it is unclear if these
models are able to capture the derogatory or slurs references in multimodality
(i.e., image and text) of the hateful memes. To fill this research gap, this
paper propose three research questions to improve our understanding of these
visual-linguistic models performing the hateful meme classification task. We
found that the image modality contributes more to the hateful meme
classification task, and the visual-linguistic models are able to perform
visual-text slurs grounding to a certain extent. Our error analysis also shows
that the visual-linguistic models have acquired biases, which resulted in
false-positive predictions.

hate speech, hateful memes, multimodal, explainable machine learning

††journalyear: 2022††copyright: acmlicensed††conference: Proceedings of the
ACM Web Conference 2022; April 25–29, 2022; Virtual Event, Lyon,
France††booktitle: Proceedings of the ACM Web Conference 2022 (WWW ’22), April
25–29, 2022, Virtual Event, Lyon, France††price: 15.00††doi:
10.1145/3485447.3512260††isbn: 978-1-4503-9096-5/22/04††ccs: Computing
methodologies Natural language processing††ccs: Computing methodologies
Computer vision representations

Disclaimer: This paper contains violent and discriminatory content that may be
disturbing to some readers.

##  1\. Introduction

Motivation. Internet memes, which are often presented as images with
accompanying text, are increasingly abused to spread hatred under the guise of
humor (Kiela et al., 2020; Gomez et al., 2020; Pramanick et al., 2021). To
fight against the proliferation of hateful memes, Facebook has recently
released a large hateful meme dataset and crowdsourced hateful meme
classification solutions (Kiela et al., 2020). The research community has
responded enthusiastically as many promising hateful meme classification
methods have been proposed (Lippe et al., 2020; Zhu, 2020; Velioglu and Rose,
2020; Zhou et al., 2021a; Lee et al., 2021). Among the proposed solutions, a
popular line of approaches is to apply pre-trained visual-linguistic models
such as VisualBERT (Li et al., 2019) and VilBERT (Lu et al., 2019) to perform
the hateful meme classification tasks. These methods have yielded promising
results. However, what these visual-linguistic models learn for the hateful
meme classification task remains unclear.

Research Objectives. Understanding visual-linguistic models is an emerging
research area that has garnered much attention from the multimodal research
community (Li et al., 2020; Cao et al., 2020; Frank et al., 2021; Parcalabescu
et al., 2021). Inspired by works that explored the internal behaviors of pre-
trained language models (Clark et al., 2019), Li et al. (Li et al., 2020)
conducted a quantitative study on whether visual-linguistic models acquire
semantic grounding ability during pre-training without explicit supervision.
Frank et al. (Frank et al., 2021) proposed a diagnostic framework to assess
the extent to which the visual-linguistic models integrate cross-model
information. This paper aims to contribute to the existing literature on
visual-linguistic model understanding by applying some of these techniques to
investigate how visual-linguistic models understand hateful memes. To the best
of our knowledge, this is the first paper that attempts to understand what
visual-linguistic models actually learn when training for the hateful meme
classification task.

Contributions. Our paper proposed three research questions, which improve our
understanding of the internal behaviors of visual-linguistic models trained to
perform the hateful meme classification task. Through extensive quantitative
and qualitative analyses, we show that (i) the visual-linguistic models have
accorded higher importance to the visual modality when performing hateful meme
classification; (ii) the visual-linguistic models are able to learn the
visual-text slurs grounding; (iii) and the models have acquired biases that
adversely affected their hateful meme classification performance.

##  2\. Research Questions

This paper aims to improve our understanding of visual-linguistic models
applied to perform the hateful meme classification task. Working towards this
goal, we formulate three research questions to guide our exploration111Code
implementation: https://gitlab.com/bottle_shop/safe/ExplainHatefulMeme..

RQ1: Modality Attribution. A key characteristic of memes is their
multimodality nature, where their underlying message is often communicated via
a combination of text and visual information. The multimodality characteristic
also motivated the application of visual-linguistic models to perform hateful
meme classification. However, it is unclear if the text and visual information
contributed equally towards the multimodal classification task. Existing
studies have attempted to improve the explainability of deep learning models
by attributing the prediction of a deep network to its input features
(Kokhlikyan et al., 2020; Guidotti et al., 2018; Ancona et al., 2018;
Sundararajan et al., 2017). For instance, Sundararajan et al. (Sundararajan et
al., 2017) proposed an attribution method called Integrated Gradients to score
the contribution of input features on deep models’ prediction. The researchers
applied their model on several images and text deep learning models to
demonstrate its ability to explain the prediction results of these models.
Similar studies were also conducted by investigating the attribution of text
and visual features in multimodal tasks such as Visual-question-answering
(VQA) (Goyal et al., 2016; Goyal et al., 2017; Patro et al., 2020). We aim to
apply the attribution methods to understand how the different modalities input
features contribute to the hateful meme classification task.

RQ2: Visual-Text Slurs Grounding. Hate speech detection tasks have been
notoriously challenging due to the ambiguity and variability in natural
languages. Adding to the complexity, hate speech also includes unique
derogatory terms that are commonly used as insinuations or allegations about
members of a specific group. For example, the word ”Dishwasher” has been
associated with females due to the traditional gender ideology where married
women would become housewives and do house chores. In doing so, it undermines
women’s gender equality rights and objectifies them as mere tools. The use of
derogatory terms amplifies further in hateful memes. With the additional
visual information, the subtle allegations in the derogatory terms are
communicated through contextual cues in both modalities. Existing works have
investigated the semantic grounding capabilities of visual-linguistic models
in VQA and image captioning tasks (Li et al., 2020; Frank et al., 2021). We
aim to extend these studies to understand the visual-linguistic models’
ability to perform visual-text grounding for derogatory terms and slurs used
in the hateful memes.

RQ3: Bias and Error Analysis. Data and model biases in text-based hate speech
detection tasks have been are widely researched (Badjatiya et al., 2019;
Davidson et al., 2019; Kennedy et al., 2020; Xia et al., 2020; Zhou et al.,
2021b). For instance, Kennedy et al. (Kennedy et al., 2020) conducted a study
to analyze group identifier biases in hate speech detection models. The
researchers found that existing text-based hate speech classifiers are over-
sensitivity to group identifiers like “Muslim”, “gay”, and “black”. We aim to
extend these studies to conduct a preliminary analysis on the biases in
hateful meme classification models. Specifically, we will examine the group
identifier biases in both text and visual modalities of the wrongly classified
memes.

##  3\. Experiments

###  3.1. Experiment Settings

####  3.1.1. Dataset

Table 1. Distribution of Facebook hateful meme dataset Train | Test
---|---
Hate | Non-hate | Hate | Non-hate
5,493 | 3,007 | 246 | 254

The Facebook hateful meme dataset (Kiela et al., 2020), which was constructed
and released by Facebook as part of a challenge to crowd-source multimodal
hateful meme classification solutions, is a popular dataset used in many
research studies. Therefore, we utilize this dataset in our experiments. The
dataset contains $10K$ memes with binary labels (i.e., hateful or non-
hateful). As we do not have labels of the memes in the test split, we utilize
the dev-seen split as the test set. Table 1 outlines the distributions of the
dataset.

####  3.1.2. Models

VilBERT (Lu et al., 2019) and VisualBERT (Li et al., 2019) are amongst the
state-of-the-art visual-linguistic models often used for various multimodal
tasks. The two models were also applied as baselines to evaluate the released
Facebook hateful meme dataset (Kiela et al., 2020). Both models use pre-
trained text from BERT (Devlin et al., 2018) and image features from $fc6$
layer of Faster-RCNN (Ren et al., 2015) with ResNeXt-152 as its backbone (Xie
et al., 2017). These models can also be trained on multimodal objectives as an
intermediate step before fine-tuning them for the multimodal hateful meme
classification task. For our experiments, we train the VilBERT and VisualBERT
on the hateful meme classification task. We also included two multimodally
pre-trained versions of these models and fine-tuned them for the hateful meme
classification task. Specifically, we include VilBERT on Conceptual Captions
(Sharma et al., 2018) (VilBERT CC), and VisualBERT on Microsoft’s Common
Objects in Context (Lin et al., 2014) (VisualBERT COCO). We trained the models
using the Facebook MMF framework (Singh et al., 2020) and adopt the
hyperparameters specified in (Kiela et al., 2020), as the researchers have
already performed grid search on numerous hyperparameters.

Table 2. Average gradients from text and visual inputs in various models Model | Text Input | Visual Input
---|---|---
Avg | Std. Dev | Avg | Std. Dev
VilBERT | 3.183 | 0.900 | 4.096 | 0.774
VisualBERT | 3.006 | 0.804 | 7.705 | 2.180
VilBERT CC | 3.130 | 0.882 | 4.145 | 0.730
VisualBERT COCO | 3.112 | 0.843 | 6.444 | 0.976

Figure 1. Attention heads in VisualBERT where the text-visual alignments
attend to the word ”dishwasher” and ”truck”.

[an image depicting a female driver driving a truck and the attention weights
obtained from Transformer architecture]the attention weight depicts strong
visual-text alignments between visual objects and certain words such as
”truck” and ”dishwasher”

Figure 2. Attention heads in VisualBERT where the text-visual alignments
attend to the word ”goat” and ”f*ck”

[an image depicting a group of Muslim gathering and the attention weights
obtained from Transformer architecture]the attention weight depicts strong
visual-text alignments between visual objects and certain words such as ”goat”
and ”fu*k”

###  3.2. Modality Attribution

Inspired by gradient-based researches that use gradients as feature importance
(Selvaraju et al., 2017; Simonyan et al., 2013), we use the gradients to
represent the contribution of each modality towards making the model’s
decision. We reasoned that gradients signify the weightage each input feature
has towards making the model’s prediction. Hence, we equate the summation of
gradients for each input type as the contribution of each modality.
Specifically, we first obtain the gradients for the inputs in each modality
through backpropagation. Subsequently, we normalize the gradients by their
magnitude. Through normalization, we place the gradients for the text and
visual modality onto the same unit space and enable us to compare their
relative contribution. We attribute the summation of these normalized
gradients as the contribution of each modality. Finally, we compute the
average and standard deviation across the 500 samples in the test data. We
noted empty tokens are present in the text inputs but not in the visual
inputs, as the text inputs have various lengths. As these empty tokens do not
contain any meaning, we argue that their gradients should not attribute to the
contribution of each modality and do not consider these gradients in our
analysis.

Table 2 shows the average and standard deviation for the text and visual
modality gradients for various hateful meme classification models. We observe
that the visual modality consistently has a slightly higher average gradient
than the text modality, suggesting that the visual modality contributes more
towards the model’s prediction. We also noted that the standard deviation for
both modalities is relatively low across most models, suggesting that there is
little variation in the contribution of each modality across all samples.

The observation that the visual modality contributes more in the hateful meme
classification results concurs with Frank et al. (Frank et al., 2021)’s
findings where proposed vision-and-language models tend to attend to visual
information more than text information. A possible reason for this observation
could be the ratio of text features to image features. We note that internet
memes generally have relatively short text. For instance, the samples in the
validation dataset have an average number of 14 words and a maximum number of
54 words. Conversely, the image features always comprise the top 100 image
regions extracted from existing object detection techniques. Hence, the number
of text features is always lower than the number of image features across the
samples, and the models may have leverage on the modality with more features.

###  3.3. Visual-Text Slurs Grounding

The visual-linguistic models offer a simplistic solution to learn interaction
across modalities. It uses a stack of Transformer layers that implicitly
aligns the text input and visual input with self-attention. Recent works have
also established that input alignments within VisualBERT’s attention weights
often capture intricate associations in the Transformers’ architecture (e.g.,
visual-text entity grounding, visual-text syntactic grounding, etc.) (Li et
al., 2020). Extending from these studies, we should observe visual-text
alignments for slurs within the attention weights after fine-tuning the
visual-linguistic models on the hateful meme classification task.

As our investigation aims to understand the visual-text grounding in the
hateful meme classification tasks, we visualize the visual-text alignments
within the models’ attention weights for evidence of slurs grounding.
Specifically, we display the bounding region on the image for each visual
feature. To obtain the bounding region for each visual feature, we used the
predicted coordinates from the final layer of the Faster-RCNN as they
originate from the model’s intermediate layer fc6. However, visualizing 100
bounding regions would clutter the image and would be impractical to make any
valuable observation. Therefore, we choose to visualize the top 9 features
ranked by their contribution to the models’ decisions.

There are many slurs embedded in the hateful memes. We select and examine two
common slurs, “dishwasher” and “goat-f*cker”, that target female and Muslim
communities, respectively. Specifically, we search the hateful memes that
contain the words “dishwasher”, “goat”, and “goat-f*cker”. From the retrieved
memes, we examine the visual bounding regions that have attention weight
aligned to the selected keywords. We then show the 4 heads of the attention
layers that best demonstrate visual-text alignments for slurs and entities.
The visual-text alignment will reveal the models’ multimodal understanding of
the slurs embedded in the memes.

The word “dishwasher” is a sexist slur commonly found in hate speech targeted
at the female gender, used to stigmatize women as housewives. In Figure 1, the
meme describes a scenario where one undermines women’s gender equality rights
and addressed the woman as “dishwasher”. We observed that the model implicitly
forms alignments between the subword ”##er” for dishwasher and the seven image
segments containing the woman in lower layers. The model displays residual
visual-text alignments for the slur using a different subword ”##wash” by the
end of the computation (i.e., layer 12), demonstrating the strong presence of
visual-text alignment for image of woman and the slur “dishwasher”.

[a top image containing female opening a sandwich maker and a bottom image
containing muslims with goats]The primary text contribution of the top image
comes from two subwords resulting from ”dishwasher” and ”sandwich”. Whereas,
the primary text contribution from bottom image come from the word ”meat”,
”very” and ”islamic”

Figure 3. Two examples of non-hateful memes wrongly classified by VisualBERT

The phrases “goat-humper” and “goat-f*cker” is one of the many offensive slurs
commonly targeted at the Muslim religion, accusing Muslim men of having sexual
relationships with goats (i.e., an act of bestiality). Figure 2 shows a meme
that suggests the Muslim men have sexual relationships with goats by
associating the text “we can f*ck goats” with Muslim men in the image. While
the phrase “goat-humper” and “goat f*cker” does not appear in this meme, we
could infer the underlying allegations and identify the relevant keywords. We
observed that the word “f*ck” and the word “goat” assigns significant
attention weights to the bounding regions containing Muslim men in the early
layers. The model displays residual visual-text alignment for the word “f*ck”
and the word “goat” in the later layers (i.e., layer 11 and 12), albeit more
attention weights are assigned to the word “f*ck”. Combining the observations,
the model demonstrates presence of visual-text alignment for the relevant
keywords that suggest the slurs “goat-humper” and “goat-f*cker”.

We also observed that the early layers demonstrate a cleaner alignment to the
slur terms. For example, in Figure 1, the unrelated visual segments to the
subword ”##er” and the word ”truck” are aligned to the separator token. A
similar observation can be made for the unrelated visual segments to the word
”goat” in Figure 2. Based on recent researches, these alignments to the
separator tokens can be seen as no-operation (no-op) as they do not
substantially impact the model’s output (Clark et al., 2019; Kobayashi et al.,
2020).

###  3.4. Bias and Error Analysis

To analyze the bias of the hateful meme classification models, we conduct an
error analysis on the wrongly predicted non-hateful memes by the models.
Similar to the existing works that studied biases in text-based hate speech
detection models, we are especially interested in the false positives because
it may reveal the features that the models are overly sensitive to when
performing hateful meme prediction. Specifically, we conduct this analysis by
inspecting the bounding regions for the critical visual features and
contributions of individual word tokens for text features. The visualization
of bounding regions for the visual features follows the same strategy
specified in 3.3. Whereas for the text features, we used Integrated Gradients
(Sundararajan et al., 2017) to visualize the contribution of each word towards
making the model’s prediction. Specifically, words are highlighted in green
and red to indicate their attribution to hateful and non-hateful prediction,
and the color intensity represents the level of attribution.

Figure 3 shows two examples of non-hateful memes wrongly classified by
VisualBERT. In Figure 3(a), we observed that the subword tokens for
“dishwasher” and word token “sandwich” have a high contribution towards making
the model predict the meme as hateful. Upon inspecting the text-visual
alignment, these text tokens also assign significant attention weights to the
bounding region containing the woman. From the observations, we can infer that
the model has likely learned a bias where the presence of keywords such as
“dishwasher” and images of women would render the model to predict the meme as
hateful. The bias also exposes a deeper issue with the visual-linguistic
hateful meme classification model; even though the models are able to learn
the visual-text slurs grounding, over-sensitivity to such grounding may also
introduce bias and results in false positive.

The model also exhibits bias for the group identifier term ”Islamic” in Figure
3(b). Inspecting the text-visual alignments, we observed that the word
”Islamic” does not assign much attention weights to any bounding regions. We
postulate that the bias for this group identifier term happens in the unimodal
text space rather than the multimodal space.

##  4\. Conclusion

We have presented an analysis of applying visual-linguistic models on the
hateful meme classification task. Our analysis showed that the image modality
contributes more to the hateful meme classification task, and the visual-
linguistic models can perform visual-text slurs grounding. Nevertheless, the
visual-linguistic models have also acquired biases, which resulted in false-
positive predictions. For our future work, we will extend our analysis to
multiple hateful meme datasets, and benchmark more models. We would also
explore debiasing techniques to reduce the multimodal biases in the visual-
linguistic models and improve their hateful meme classification performance.

## References

  * (1)
  * Ancona et al. (2018) Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. 2018.  Towards better understanding of gradient-based attribution methods for Deep Neural Networks. In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30-May 3, 2018, Conference Track Proceedings_. OpenReview. net.
  * Badjatiya et al. (2019) Pinkesh Badjatiya, Manish Gupta, and Vasudeva Varma. 2019\.  Stereotypical bias removal for hate speech detection task using knowledge-based generalizations. In _The World Wide Web Conference_. 49–59.
  * Cao et al. (2020) Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu. 2020\.  Behind the scene: Revealing the secrets of pre-trained vision-and-language models. In _European Conference on Computer Vision_. Springer, 565–580.
  * Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019.  What Does BERT Look at? An Analysis of BERT’s Attention. In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_. 276–286.
  * Davidson et al. (2019) Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019\.  Racial Bias in Hate Speech and Abusive Language Detection Datasets. In _Proceedings of the Third Workshop on Abusive Language Online_. 25–35.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Frank et al. (2021) Stella Frank, Emanuele Bugliarello, and Desmond Elliott. 2021\.  Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. 9847–9857.
  * Gomez et al. (2020) Raul Gomez, Jaume Gibert, Lluis Gomez, and Dimosthenis Karatzas. 2020\.  Exploring hate speech detection in multimodal publications. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 1470–1478.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 6904–6913.
  * Goyal et al. (2016) Yash Goyal, Akrit Mohapatra, Devi Parikh, and Dhruv Batra. 2016.  Towards transparent ai systems: Interpreting visual question answering models.  _arXiv preprint arXiv:1608.08974_ (2016).
  * Guidotti et al. (2018) Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018.  A survey of methods for explaining black box models.  _ACM computing surveys (CSUR)_ 51, 5 (2018), 1–42.
  * Kennedy et al. (2020) Brendan Kennedy, Xisen Jin, Aida Mostafazadeh Davani, Morteza Dehghani, and Xiang Ren. 2020\.  Contextualizing Hate Speech Classifiers with Post-hoc Explanation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 5435–5442.
  * Kiela et al. (2020) Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020.  The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes.  _Advances in Neural Information Processing Systems_ 33 (2020).
  * Kobayashi et al. (2020) Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020.  Attention is not only a weight: Analyzing transformers with vector norms.  _arXiv preprint arXiv:2004.10102_ (2020).
  * Kokhlikyan et al. (2020) Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, et al. 2020\.  Captum: A unified and generic model interpretability library for pytorch.  _arXiv preprint arXiv:2009.07896_ (2020).
  * Lee et al. (2021) Roy Ka-Wei Lee, Rui Cao, Ziqing Fan, Jing Jiang, and Wen-Haw Chong. 2021.  Disentangling Hate in Online Memes. In _Proceedings of the 29th ACM International Conference on Multimedia_. 5138–5147.
  * Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019.  Visualbert: A simple and performant baseline for vision and language.  _arXiv preprint arXiv:1908.03557_ (2019).
  * Li et al. (2020) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2020.  What Does BERT with Vision Look At?. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 5265–5275.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context. In _European conference on computer vision_. Springer, 740–755.
  * Lippe et al. (2020) Phillip Lippe, Nithin Holla, Shantanu Chandra, Santhosh Rajamanickam, Georgios Antoniou, Ekaterina Shutova, and Helen Yannakoudakis. 2020.  A multimodal framework for the detection of hateful memes.  _arXiv preprint arXiv:2012.12871_ (2020).
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019\.  ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. In _NeurIPS_.
  * Parcalabescu et al. (2021) Letitia Parcalabescu, Albert Gatt, Anette Frank, and Iacer Calixto. 2021.  Seeing past words: Testing the cross-modal capabilities of pretrained v&l models on counting tasks. In _Proceedings of the 1st Workshop on Multimodal Semantic Representations (MMSR)_. 32–44.
  * Patro et al. (2020) Badri Patro, Shivansh Patel, and Vinay Namboodiri. 2020\.  Robust explanations for visual question answering. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 1577–1586.
  * Pramanick et al. (2021) Shraman Pramanick, Dimitar Dimitrov, Rituparna Mukherjee, Shivam Sharma, Md Shad Akhtar, Preslav Nakov, and Tanmoy Chakraborty. 2021.  Detecting Harmful Memes and Their Targets. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_. 2783–2796.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015\.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ 28 (2015), 91–99.
  * Selvaraju et al. (2017) Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017.  Grad-cam: Visual explanations from deep networks via gradient-based localization. In _Proceedings of the IEEE international conference on computer vision_. 618–626.
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018\.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 2556–2565.
  * Simonyan et al. (2013) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013\.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  _arXiv preprint arXiv:1312.6034_ (2013).
  * Singh et al. (2020) Amanpreet Singh, Vedanuj Goswami, Vivek Natarajan, Yu Jiang, Xinlei Chen, Meet Shah, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. 2020.  MMF: A multimodal framework for vision and language research.  https://github.com/facebookresearch/mmf.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks. In _International Conference on Machine Learning_. PMLR, 3319–3328.
  * Velioglu and Rose (2020) Riza Velioglu and Jewgeni Rose. 2020.  Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge.  _arXiv preprint arXiv:2012.12975_ (2020).
  * Xia et al. (2020) Mengzhou Xia, Anjalie Field, and Yulia Tsvetkov. 2020\.  Demoting Racial Bias in Hate Speech Detection. In _Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media_. 7–14.
  * Xie et al. (2017) Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. 2017.  Aggregated residual transformations for deep neural networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 1492–1500.
  * Zhou et al. (2021b) Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Yejin Choi, and Noah A Smith. 2021b.  Challenges in Automated Debiasing for Toxic Language Detection. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_. 3143–3155.
  * Zhou et al. (2021a) Yi Zhou, Zhenhao Chen, and Huiyuan Yang. 2021a.  Multimodal Learning For Hateful Memes Detection. In _2021 IEEE International Conference on Multimedia & Expo Workshops (ICMEW)_. IEEE, 1–6.
  * Zhu (2020) Ron Zhu. 2020.  Enhance multimodal transformer with external label and in-domain pretrain: Hateful meme challenge winning solution.  _arXiv preprint arXiv:2012.08290_ (2020).