# Magnetohydrodynamics evolution of three-dimensional magnetic null in NOAA
active region 11515 initiated using non-force-free field extrapolation

Sanjay Kumar  Department of Physics, Patna University, Patna 800005, India
Avijeet Prasad  Rosseland Centre for Solar Physics, University of Oslo,
Postboks 1029 Blindern, 0315 Oslo, Norway  Institute of Theoretical
Astrophysics, University of Oslo, Postboks 1029 Blindern, 0315 Oslo, Norway
Ranadeep Sarkar  Department of Physics, University of Helsinki, Helsinki
00560, Finland  R. Bhattacharyya  Udaipur Solar Observatory, Physical Research
Laboratory, Dewali, Bari Road, Udaipur 313001, India

###### Abstract

Magnetohydrodynamics simulation of active region NOAA 11515 is performed to
examine the initiation of the M5.6 flaring event that starts around 10:43 UT
on 2012 July 2. The simulation is conducted using an extrapolated non-force-
free magnetic field generated from the photospheric vector magnetogram of the
active region as the initial magnetic field. The magnetic field shows the
presence of a three-dimensional (3D) magnetic null with the corresponding dome
overlying a filament and a low-lying magnetic flux rope, observed in 304 Å and
131 Å respectively. The simulated dynamics, triggered by the initial Lorentz
force, lead to the bifurcations of the flux rope, which is similar to the
observed bifurcation in the 131 Å brightenings. Additionally, the rope
exhibits a rise and reconnects at the 3D null. These reconnections convert
field lines of the rope into the anchored outer spine of the 3D null —
explaining the occurrence of a nearby confined C-class flare. Further, the
results show that the field lines of the flux rope reach the vicinity of the
filament and become non-parallel to the field lines of the filament. This
initiates the reconnections between the rope and the field lines of the
filament — activating the filament for the eruption. This interesting
interaction of the flux rope and filament seems to contribute to the onset of
the M-class flare.

Magnetohydrodynamics (MHD), Magnetic reconnection, Magnetic fields, Solar
flares, Extrapolation

##  1 Introduction

Solar flares are the manifestation of the abrupt and explosive release of
energy in the solar atmosphere. The flaring events generally relax the
stressed coronal magnetic fields by releasing the stored magnetic energy
(Aschwanden, 2004). The coronal magnetic field lines (MFLs) being rooted in
the solar surface with high plasma-$\beta$ are stressed by the flows at the
surface and, consequently store the energy (Shibata & Magara, 2011; Priest,
2014). However, the physics of the underlying processes which trigger the
onset of the flaring events and the associated energy release is not fully
understood (Shibata & Magara, 2011; Priest, 2014). In this direction, the
magnetic reconnection (MR) — a diffusive process in which magnetic energy
stored in the plasma gets converted into kinetic energy and heat accompanied
by a change in magnetic topology — is widely accepted to be the physical
process central to the sudden energy release (Shibata & Magara, 2011; Priest,
2014).

The three-dimensional (3D) magnetic nulls (the locations where the magnetic
field ${\bf{B}}=0$) are considered to be the potential sites for hosting the
reconnections (Pontin et al., 2005; Al-Hachami & Pontin, 2010; Kumar &
Bhattacharyya, 2016). In the solar atmosphere, the 3D null is characterized by
its spine and dome-shaped fan structures resulting from the avoidance of the
${\bf{B}}=0$ location by the MFLs. As a consequence of the spine-fan
structure, reconnections naturally commence at the 3D nulls (Pontin et al.,
2013, 2016; Prasad et al., 2018; Nayak et al., 2020). Notably, recent
magnetohydrodynamic (MHD) simulations document the occurrence of circular
flare ribbons and blowout coronal jets (Masson et al., 2009; Wang & Liu, 2012;
Nayak et al., 2019; Liu et al., 2020; Prasad et al., 2020) along with confined
flares due to the reconnections at the null-point (Ugarte-Urra et al., 2007;
Prasad et al., 2018). In addition to the 3D nulls, also important are the
magnetic flux ropes (MFRs), the magnetic structures made of MFLs winding
around a common axis (Priest, 2014). Under favorable conditions, the flaring
events are believed to be initiated by the evolution of the MFRs, which is
governed by the reconnections (Inoue et al., 2015; Kumar et al., 2016; Prasad
et al., 2020).

To construct the suitable coronal magnetic field, the nonlinear-force-free-
fields (NLFFFs) extrapolations are widely used which correspond to zero
Lorentz force equilibrium state (Wiegelmann, 2008; Wiegelmann & Sakurai, 2012;
Duan et al., 2017). Recently, magnetohydrodynamic (MHD) simulations initiated
with the NLFFF extrapolations successfully explained the various eruptive
events in solar corona (Jiang et al., 2013; Kliem et al., 2013; Amari et al.,
2014; Inoue et al., 2014, 2015; Inoue, 2016). However, the NLFFF
extrapolations are less accurate near the photosphere where plasma is non-
force-free. In the photosphere, where the vector magnetograms are taken, the
plasma-$\beta$ is non-negligible and approaches to unity (Gary, 2001), so that
the Lorentz force can not be neglected. To tackle the issue, a technique
called “preprocessing” is used on the photospheric vector magnetograms that
minimize the force and provide a suitable boundary condition for the NLFFF
extrapolations (Wiegelmann, 2008; Wiegelmann & Sakurai, 2012). Recently, non-
force-free-fields (NFFFs) extrapolation (Hu & Dasgupta, 2008; Hu et al., 2010)
has emerged as a viable alternative to the NLFFF extrapolations. In the NFFFs
extrapolations, the extrapolated field supports non-zero Lorentz force, which
is crucial for the spontaneous onset of evolution. Utilizing the NFFF
extrapolations, recent MHD simulations successfully simulated coronal dynamics
leading to solar flares, coronal jets, and coronal dimmings (Prasad et al.,
2018; Nayak et al., 2019; Prasad et al., 2020; Bora et al., 2022).

In this paper, we aim to explore the magnetohydrodynamics evolution of NOAA AR
11515 to understand the onset of an eruptive M5.6 flare on 2012 July 2 at
10:43 UT, which is suggested to be triggered by a nearby confined C2.9 flare
that occurred at 10:37 UT (Louis, Rohan E. et al., 2014). Importantly, the
presented MHD simulation initiated by the NFFF extrapolation provides a
plausible explanation of the occurrence of the C-class flare and the
subsequent initiation of the M-class flare through the interactions of a pre-
existing 3D null and the flux ropes.

The rest of the paper is arranged as follows: Section 2 briefly mentions the
important aspects of the flaring event and the details of the initial non-
force-free extrapolated field. Section 3 presents the results of the
simulation along with their relation to the multi-wavelength observations.
Important findings of the paper are summarized in Section 4.

##  2 Event observations and extrapolation

###  2.1 Flares in the NOAA AR 11515 on 2012 July 2

The eruptive M5.6 flare occurs in NOAA AR 11515 on 2012 July 2. The various
observational aspects of the flare are already studied by Louis, Rohan E. et
al. (2014). The temporal profile of GOES soft X-ray flux indicates that the
flare starts at around 10:43 UT and peaks at $\approx$ 10:52 UT (not shown).
Also notable is a preceding confined C2.9 flare in the vicinity of the M5.6
flare that initiated at 10:33 UT and peaked at 10:37 UT (Louis, Rohan E. et
al., 2014). Here, in Figures 1 and 2, we highlight a few important multi-
wavelength observations of the flares obtained from the Atmospheric Imaging
Assembly (AIA) (Lemen et al., 2012) on board the Solar Dynamic Observatory
(SDO). Figure 1 illustrates the dynamics of the AR during the precursor and
impulsive phase of the M5.6 flare as observed in SDO/AIA 131 Å images. Panel
(a) marks the presence of brightenings that extends in the north-east to
south-west direction (marked by the red arrow) at around 10:30 UT. Typically,
such brightenings are associated with the existence of a magnetic flux rope
(Joshi et al., 2017; Prasad et al., 2020). Further, the rope maintains its
structure until around 10:34 UT (panel (b)) and shows bifurcations in its
eastern part at around 10:37 UT (marked by the red arrow in panels (c) and
(d)), which is possibly related to the occurrence of the C2.9 flare. The
panels (c) and (d) also show the flare to be associated with a faint,
southwards propagating spray (marked by the green arrows), without any
eruptive behaviour (Louis, Rohan E. et al., 2014).

Figure 1: The appearance of the brightenings and its subsequent bifurcations
in 131 Å are marked by red arrows in panels (a)-(d). Also notable is the faint
southward spray of plasma (marked by green arrow in panels (c)-(d)).  Figure
2: Panels (a)-(d) show the pre-existing filament and its subsequent evolution
(marked by white arrows) in 304 Å. The green-dashed line marks the location of
filament at $10:30$ UT. Important is the activation of the filament around
10:43 UT (panel (c)) and subsequent onset of filament eruption around 10:45 UT
(panel (d)).

Figure 2 depicts the presence of a pre-existing filament (marked by a white
arrow) at around 10:30 UT, which almost remains in a static situation until
around the onset time of the M5.6 flare. The filament starts to rise and
expand around 10:43 UT onwards (panel (c)) — pointing towards the activation
of the filament. The activation is then followed by the onset of the filament
eruption (panel (d)). The erupting filament possesses close spatial and
temporal associations with the M-class flare, indicating that the filament
eruption possibly triggers the impulsive phase of the flare (Louis, Rohan E.
et al., 2014). Moreover, Louis, Rohan E. et al. (2014) suggested the onset of
the M-class flare to be triggered by the interactions of the filament with the
nearby brightening (flux rope) associated with the preceding C-class flare,
which points towards a causal connection between the flux rope dynamics and
the filament eruption.

###  2.2 Extrapolated coronal magnetic field of AR 11515

We utilize the non-force free extrapolation model developed by Hu & Dasgupta
(2008); Hu et al. (2008, 2010) to extrapolate the coronal magnetic field of AR
11515 at 10:34 UT on 2012 July 2 corresponding to the photospheric vector
magnetogram shown in Figure 3. Relevantly, the magnetogram is obtained from
the Helioseismic Magnetic Imager (HMI; Schou et al., 2012) on board the Solar
Dynamic Observatory (SDO). The magnetogram is taken from the
‘hmi.sharp_cea_720s’ data series. This data series provides the magnetic field
on a Cartesian grid which is initially remapped onto a Lambert cylindrical
equal-area (CEA) projection and then transformed into heliographic coordinates
(Bobra et al., 2014). We further crop the field of view to the region of
interest to obtain a cutout of 640$\times$416 pixels in the $x$ and $y$
directions respectively. Noticeably, the cropping maintains the flux balance.
In Figure 3, the dark green line shows the polarity inversion line (PIL). It
is noteworthy that, in the absence of significant dynamical changes from 10:30
UT to 10:34 UT in the multiwavelength observations of the events (Figures 1
and 2), we select the magnetogram at 10:34 UT for the extrapolation to reduce
the computational cost of the MHD simulation.

Figure 3: Figure illustrates HMI vector magnetogram of AR 11515 with start
time (DATE_OBS) of 10:34 UT on 2012 July 2. The red and blue arrows show the
strength and direction of the transverse magnetic field, while green line mark
PIL.  Figure 4: Panel (a) shows the distribution of the magnitude of the
initial Lorentz force density in the computational domain. Panels (b) depicts
the variation of magnitude of the horizontally averaged Lorentz force density
with height (Mm). The force density is normalized with respect to its maximum
value, as our focus is on its decay rate with height. Both the panels clearly
illustrate the exponential decrease of the force density with height. Figure
5: Panels (a) and (b) depict the zoomed-in view of the extrapolated MFLs with
the magnetogram as the bottom boundary. The location of the 3D null is marked
by a red arrow in panel (a). The yellow MFLs represent the dome-shaped fan
surface and spine axis of the 3D null. Panel (b) is further overlaid with the
twist parameter — showing the existence of two twisted magnetic structures
below the dome. Panel (c) plots the structures with 304 Å image of around
10:34 UT at the bottom boundary. The filament is almost co-located with one of
the twisted structures plotted in green. Panel (d) is overplotted with 131 Å
image of around 10:34 UT at the bottom boundary — indicating the similar
location of the magenta-colored twisted structure and the observed
brightenings in 131 Å.

The non-force-free extrapolation model is based on the principle of minimum
energy dissipation rate, according to which, a plasma system prefers to relax
toward a state which has a minimum dissipation rate (Bhattacharyya et al.,
2007). The magnetic field ${\bf{B}}$ is then shown to satisfy the double-curl
Beltrami equation (Bhattacharyya et al., 2007). The field ${\bf{B}}$ then can
be written as the superposition of two linear-force-fields and a potential
field (Hu & Dasgupta, 2008). Consequently, the Lorentz force associated with
the field is non-zero. An iterative approach based on the minimization of the
average deviation between the observed and the calculated transverse field on
the photospheric boundary is then employed to get the extrapolated NFFF (Hu et
al., 2010).

Notably, in order to minimize the computational cost, the original domain with
the spatial extension of $640\times 416$ pixels is re-scaled to $320\times
208$ pixels grid in $x$ and $y$ directions, respectively. The vertical
extension of the domain is taken as $208$ pixels. We have checked that the
magnetic structures remain preserved after the re-scaling by comparing the
extrapolated field of the original magnetogram. Panel (a) of Figure 4
illustrates the direct volume renderings of the Lorentz force density, which
shows the presence of the Lorentz force at lower heights. To further confirm,
in panel (b) of the figure, we plot the variation of horizontally averaged
Lorentz force density with height. The averaged Lorentz force density falls
off sharply with height. Hence, Figure 4 suggests that the Lorentz force
mostly resides near the photosphere and almost vanishes at coronal heights,
similar to the typical description of the solar corona. The averaged value of
the Lorentz force associated with the extrapolated field at the bottom
boundary is of the order of $10^{22}$ dyne. Interestingly, a recent
observational study by Sarkar et al. (2019) has estimated the change in the
Lorentz force during flaring events, which is approximately $(2-5)\times
10^{22}$ dyne. In our simulation, the non-zero Lorentz force at lower heights
plays a crucial role in initiating the simulated dynamics of the active region
that can lead to the flaring events.

The magnetic field lines of the extrapolated field are plotted in Figure 5. In
this and subsequent figures, the $x$, $y$, and $z$ axes of the computational
domain are represented by red, green, and blue axes, respectively. The figure
documents the existence of a pre-existing 3D null (marked by a red arrow in
Fig. 5(a)) and, the corresponding spine axis and the dome-shaped fan surface
are represented by the yellow field lines. Notable are the field lines of the
outer spine (extends in the southward direction) to be anchored at the bottom
boundary (panel (a)). Further important is the presence of two twisted
magnetic structures (having co-located high values of twist parameter),
residing under the dome of the 3D null (see panel (b) of Figure 5).
Relevantly, the twist parameter measures the twist number of a field line and
is calculated by integrating field-aligned current $\bf{J}\cdot\bf{B}/|B|^{2}$
along a field line (Berger & Prior, 2006; Liu et al., 2016). The maximum
magnitude of the twist parameter for the magenta-colored magnetic structure is
close to 1.5, while the maximum magnitude of the parameter is around 1.1 for
the green-colored structure. This suggests that these structures represent
magnetic flux ropes (Prasad et al., 2020). Notably, the flux rope presented by
the green field lines is almost co-spatial with the filament observed in 304 Å
(Fig. 5(c)) — indicating the rope to be the magnetic structure of the
filament. Furthermore, the other rope plotted by magenta field lines has a
higher magnitude of the twist parameter in comparison to the green-colored
rope and co-locates with the brightenings observed in 131 Å around 10:34 UT
(Fig. 5(d)). Due to the higher twist, the magenta-colored rope supports a
larger electrical current and, therefore, seems to be responsible for the
brightenings in 131 Å.

##  3 Magnetohydrodynamics evolution of AR 11515

###  3.1 Governing MHD equations

The evolution of AR 11515 is simulated by numerically solving equations of
magnetohydrodynamics in implicit large eddy simulation mode (Grinstein et al.,
2007). With a focus on exploring the changes in field line topology, here we
consider the incompressible, thermally inactive plasma with zero physical
resistivity (Bhattacharyya et al., 2010; Kumar et al., 2014, 2015). As the
involved magnetic field is sufficiently strong, the magnetic pressure
dominates over other pressures such as thermal, gravitational, and therefore,
their negligence is justifiable. The set of dimensional-less MHD equations is
then given as:

$\displaystyle\frac{\partial{\bf{v}}}{\partial
t}+\left({\bf{v}}\cdot\nabla\right){\bf{v}}=-\nabla
p+\left(\nabla\times{\bf{B}}\right)\times{\bf{B}}+\frac{\tau_{a}}{\tau_{\nu}}\nabla^{2}{\bf{v}},$
(1) $\displaystyle\nabla\cdot{\bf{v}}=0,$ (2)
$\displaystyle\frac{\partial{\bf{B}}}{\partial
t}=\nabla\times({\bf{v}}\times{\bf{B}}),$ (3)
$\displaystyle\nabla\cdot{\bf{B}}=0.$ (4)

The magnetic field strength ${\bf{B}}$ and the plasma velocity ${\bf{v}}$ are
normalized by the average magnetic field strength ($B_{0}$) and the Alfvén
speed ($v_{a}\equiv B_{0}/\sqrt{4\pi\rho_{0}}$ with $\rho_{0}$ representing
the constant mass density), respectively. The plasma pressure $p$, the
spatial-scale $L$, and the temporal scale $t$ are normalized by
${\rho{v_{a}}^{2}}$, length-scale of the vector magnetogram ($L_{0}$), and the
Alfvénic transit time ($\tau_{a}=L_{0}/v_{a}$), respectively. Here
$\tau_{\nu}$ represents viscous diffusion time scale
($\tau_{\nu}=L_{0}^{2}/\nu$), with $\nu$ being the kinematic viscosity. The
pressure $p$ satisfies the elliptic boundary value problem, generated by
imposing the incompressibility constraint (2) on the momentum transport
equation (1) (see (Bhattacharyya et al., 2010)).

The MHD equations are solved using the well-established numerical model EULAG-
MHD (Smolarkiewicz & Charbonneau, 2013). The details of the model are
described in Smolarkiewicz & Charbonneau (2013) and references therein.
Importantly, in the absence of the physical magnetic diffusivity (3), the
dissipative property of the model, intermittently and adaptively, regularizes
the under-resolved scales by simulating magnetic reconnections. Notably, in
our previous works (Prasad et al., 2018; Nayak et al., 2019; Prasad et al.,
2020; Bora et al., 2022), we successfully simulated the dynamics of various
active regions with the model and explained the solar transients such as
flares, coronal jets, circular brightenings in the active regions.

It is noteworthy that, in the simulation, the initial velocity is taken to be
$\bf{v}=0$. The dynamics is then developed by the Lorentz force associated
with the initial extrapolated field (see Figure 4). The direct volume
renderings of the Lorentz force density (Figure 4(a)) shows the presence of
Lorentz force at lower heights and the higher values of the magnitude of the
force are cospatial with the strong-field regions (having high values of
$B_{z}$). This Lorentz force existing at lower height initiates the simulated
evolution by pushing the plasma from its initial stationary state and
generates plasma flow. The field lines being frozen into the plasma get
deformed by the plasma flow which, modify the initial distribution of the
Lorentz force. Moreover, once the flow is generated the other forces such the
pressure gradient and the viscous drag also come into play. Therefore, once
initiated by the Lorentz force, the other forces also contribute to the
simulated dynamics.

###  3.2 Simulated evolution

Figure 6: Evolution of the magenta-colored flux rope co-relating the
brightenings in 131 Å. The evolution is overplotted with the co-current 131 Å
images. Notable is the similar bifurcations of the rope and brightenings in
131 Å (panel (b)). Panels (c) and (d) indicates the rise of the rope and its
reconnection at the 3D null.

The presented simulation is conducted in a computational domain having
$320\times 208\times 208$ grid points for a physical domain spanning
$[0,1.54]\times[0,1]\times[0,1]$ units in $x$, $y$, and $z$, respectively,
where an unit length approximately corresponds to $145$ Mm. The simulation is
initiated with the NFFF extrapolated magnetic field as shown in Figure 5. As
mentioned above, with an initial motion-less state, the simulated dynamical
evolution is generated by the initial non-zero Lorentz force associated with
the NFFF. The resulted flow is incompressible, an assumption also unitized by
Dahlburg et al. (1991); Aulanier et al. (2005). As our focus is to understand
the topological changes responsible for the flare onset, the assumption seems
to be justifiable in the tenuous coronal medium. $B_{z}$ is kept constant at
the bottom boundary of the computational domain, because the change of
magnetic flux at the boundary is minimal during the flares (not shown). Other
boundaries are kept open by continuing all the components of ${\bf{B}}$ to the
boundaries in order to vanish the net magnetic flux passing through them
(Prasad et al., 2018). The value of the dimensional-less constant
$\tau_{a}/\tau_{\nu}$ is chosen to be $5\times 10^{-4}$, which is around 20
times larger than its coronal value. The higher $\tau_{a}/\tau_{\nu}$ for the
simulation only speeds up the dynamical evolution and, hence, reduces the
computational cost, without any effect on the magnetic topology. The spatial
unit step $\Delta x=0.0048$ and time step (normalized by the Alfvén transit
time $\tau_{a}\sim 20s$) $\Delta t=2\times 10^{-3}$ are selected to satisfy
the Courant-Friedrichs-Lewy (CFL) stability condition (Courant et al., 1967).
The simulation is carried out for 1500 $\Delta t$. Because of the higher
$\tau_{a}/\tau_{\nu}$, the simulated dynamics is expected to be around 20
times faster than the coronal dynamics and, therefore, we need to multiply
1500 $\Delta t$ by 20 to directly compare the simulation results with
observations. As a result, the total simulation time roughly corresponds to an
observation time of 20 Minutes. For a direct comparison of the simulated
evolution with the observations, we present the time in units of
$3.2\tau_{a}\approx 1$ minute in describing the simulation results.

Figure 7: The figure depicts the evolution of the 3D null (in yellow) and the
flux rope (in magenta). The time is shown in units of $3.2\tau_{a}\approx 1$
minute. The rope located below the dome rises to the 3D null (panel (b)) and
undergo magnetic reconnections (panels (c) and (d)).

To understand the dynamics of the pre-flare stage of the M-class flare, we
first describe the evolution of the flux rope, correlating the brightenings in
131 Å (Fig. 5(c)). The evolution is shown in Figure 6. In the figure, the
bottom boundary is overlaid with concurrent 131 Å images. Relevantly, for
plotting the evolution of the magnetic field lines, in this and the subsequent
figures, we utilize “field line advection” technique in-built in the VAPOR
visualization package (Clyne & Rast, 2005), In the technique, one
representative point for a selected field line is advected by the velocity
field and, then the advected point is used as a seed to plot the field line at
later time (Mininni et al., 2008). For a detailed description of the technique
and its successful illustration in ideal as well as non-ideal magnetofluids,
readers are referred to (Clyne et al., 2007; Mininni et al., 2008). In Figure
6, the flux rope exhibits the bifurcations, similar to the ones in the
brightenings in 131 Å (see Fig. 6(b)) around at $t=2.5$, which approximately
corresponds to 10:37 UT in the observations. In addition, the rope appears to
rise and ultimately, reconnect at the 3D null (see Fig. 6(c)-(d)). To clearly
illustrate the null-point reconnections of the flux rope, in Figure 7, we
present the evolution of the rope (in magenta) overlaid with the field lines
of the 3D null (in yellow). Important are the rise of the field lines of the
rope toward the 3D null (Fig. 7(b)) and, their subsequent reconnections at the
null (Fig. 7(c)). These reconnections repeat in time and allow the field lines
of the rope to come out of the dome and transform into the field lines of the
outer spine of the 3D null. The conversion of the rope-MFLs into the outer
spine can transport the plasma ( confined in the rope) along the outer spine.
However, the outer spine being anchored in the southward direction does not
allow the plasma to escape from the corona — explaining the observed faint
southward brightenings during the C-class flare (see Fig. 1(d) and Fig. 6(d)).
Hence, the bifurcations of the rope and its repeated reconnections at the 3D
null seem to contribute to the occurrence of the C-class flare.

To explore the dynamical evolution of the pre-existing filament, Figure 8
illustrates the evolution of the flux rope (in green) representing the
filament along with the magenta-colored flux rope. The figure is further
overlaid with the concurrent 304 Å images at the bottom boundary. Noticeably,
the green-colored rope does not show much dynamical changes until around $t=9$
(Fig. 8(a)-(c)) — matching the observed behaviour of the filament until around
10:43 UT (Figure 2). Subsequently the rope appears to undergo reconnections at
around $t=11$, as indicated by the opening of the rope-MFLs (Fig. 8(d)).

Figure 8: The figure shows the dynamics of the flux rope representing the
filament (in green) along with the magenta-colored rope. The figure is
overlaid with the 304 Å images. Notably, the green-colored rope shows a few
dynamical changes until $t=9$ and starts reconnecting after that.  Figure 9:
Panels (a) and (b) illustrate the evolution of the green-colored rope
(filament) and the magenta-colored rope overplotted with
$|\bf{J}|$/$|\bf{B}|$. Here, the time is in units of around $1$ minute. Non-
parallel field lines of these ropes reach in the close vicinity and develop
high $|\bf{J}|$/$|\bf{B}|$ regions, which explain the onset of reconnections
in green-colored rope. Panels (c) and (d) shows the green-colored rope along
yellow field lines of the 3D null (marked by black arrow in panel (c)) at
$t=10$ and $t=12$. Green field lines are appear to move toward the 3D null and
get reconnected.  Figure 10: Here the dynamics of the filament associated rope
(in green) and the magenta-colored rope (as in Figure 8) are overplotted with
the co-temporal 193 Å observations. The post-reconnected field lines of the
ropes match well with the bright structures extending along the right bottom
in the observational images.

To examine the reconnections of the rope associated with the filament, in
panels (a) and (b) of Figure 9, we plot the evolution of the green-colored
rope and the magenta-colored rope. The figure is overplotted with
$|\bf{J}|$/$|\bf{B}|$. The evolution illustrates that the green-colored rope
and the magenta-colored rope (located below the dome of the 3D null) deform in
such a way that the corresponding non-parallel MFLs move in close proximity.
Consequently, the gradient of $\bf{B}$ steepens and strong electric currents
originate in the vicinity of the non-parallel field lines at $t=9$, evidenced
by the presence of high values of $|\bf{J}|$/$|\bf{B}|$. As a result, the
scales become under-resolved, which initiates magnetic reconnections that
repeatedly occur in time, and are responsible for the opening-up of the green-
colored rope (clearly identifiable in Fig. 9(b)). Similar to the magenta-
colored rope evolution (Figure 7), the reconnected green field lines appear to
become part of the longer outer spine field lines. Hence, the opening of the
filament seems to be initiated by the reconnections between the parts of the
ropes, which are located below the dome. Additionally, we also notice that the
green-colored rope seems to move toward the 3D null (marked by black arrow in
Fig. 9(c)) and starts to reconnect at the null (Fig. 9(d)). Such reconnections
can further contribute to the generation of the longer green outer spine field
lines and speed up the filament eruption process.

To further support the occurrence of such repeated reconnections, in Figure
10, we plot the evolution of filament-associated rope (in green) and the
magenta-colored rope (same as Figure 8) overlaid with co-current SDO/AIA 193 Å
images. Noticeably, the bifurcation of the magenta-colored rope at $t=5$
matches the observed bifurcation in the brightenings (Fig. 10(b)). Moreover,
the panel (d) of the figure also suggests that there are bright features
(situated at the right bottom in the observational 193 Å images) corresponding
to the post-reconnected semi-parallel field lines of the ropes (in green and
magenta) — suggesting a direct connection between the reconnections and the
features.

Remarkably, the interactions of two ropes through reconnections indicate that
the occurrence of the C-class flare serves as the pre-cursor for the M-class
flare. Because of the reconnections, the field lines of the green-colored rope
associated with filament start to open up, which enable the otherwise confined
cool plasma of the filament to escape along the post-reconnection opened field
lines. Hence, the opening of the field lines through the reconnections can
provide a potential explanation of the onset of the filament eruption that
leads to the initiation of the M-class flare.

##  4 Summary and Discussion

The paper reports an MHD simulation of the initiation of an M5.6 flare in the
NOAA AR 11515 on 2012 July 2. The focus of the paper is to understand the
dynamics and evolution of the pre-existing 3D null and the magnetic flux ropes
during the initiation phase of the flare. The initial magnetic field is
obtained by extrapolating the photospheric vector magnetogram of the active
region obtained from HMI/SDO at 10:34 UT on 2012 July 2 using the non-force-
free extrapolation technique. The extrapolated field supports a Lorentz force
at lower heights which becomes negligible in the corona — matching the
standard picture of the coronal magnetic field. The force, however, plays a
crucial role in generating the self-triggered dynamical evolution from the
initial static state. As the non-parallel MFLs approach in close proximity
during the evolution, the under-resolved scales develop. The employed numerics
regularize the scales with simulated magnetic reconnections by producing
locally adaptive residual dissipation.

SDO/AIA multiwavelength observations (particularly in 131 Å channel) show the
signature of a flux rope and its bifurcations during the pre-flaring
activities. Subsequent dynamics documents the rise of the rope and appearances
of the faint brightenings extending along the southward direction — indicating
the reconnections of the rope at the 3D null, which appear to contribute to
the occurrence of a C-class flare in the vicinity of the M5.6 flare. A pre-
existing filament is also observed in 304 Å channel which initially remains
almost stable and exhibits a sudden rising motion with the loss of its
material — pointing towards the initiation of the eruption during the onset of
the M-class flare.

The non-force-free extrapolation is able to successfully capture the presence
of a flux rope (represented by the highly twisted field lines), located under
the magnetic topology of a 3D null point located close to the flaring region.
Importantly, the flux rope field lines explain the signature of the flux rope
observed in the SDO/AIA 131 Å channel. Another set of highly twisted field
lines, implying the presence of a second flux rope, also exists which is
almost co-spatial with the filament observed in the SDO/AIA 304 Å channel.

The simulated MFL evolution documents the bifurcations of the flux rope co-
related to the 131 Å brightenings and its subsequent rise, which lead to the
reconnections of the rope at the 3D null. The evolution explains the dynamical
changes in the brightenings observed in the SDO/AIA 131 Å channel during the
C-class flare. The other flux rope associated with the filament shows a little
changes in its early phase of evolution and then manifests reconnections.
These reconnections are attributed to the movement of the non-parallel field
lines of these two ropes in the close vicinity and the consequent development
of high gradients in the magnetic field. These repeated reconnections allow to
open-up the field lines of the filament and, ultimately trigger the eruption
of the filament — initiating the M-class flare.

Overall, the presented simulation successfully captures the dynamical
evolution of the flux ropes and the 3D null, which leads to the precursor
C-class flare and, ultimately initiates the M-class flare by activating the
filament eruption. On the flip side, the simulated dynamics lack the
significant eruption corresponding to the sudden and rapid rise of the flux
rope, which is important to understand the consequent CME dynamics. This can
be attributed to the continuously operating viscous relaxation, which keeps on
decaying the available free magnetic energy needed for generating the sudden
rise. Additionally, the employed simplification of the incompressibility and
constant initial density may also play a role in the sudden rise of the flux
rope. Therefore, the simulation can be further advanced to simulate the
eruption by relaxing the incompressibility and including an apt physical
resistivity, which is kept for future assignments.

## Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any
commercial or financial relationships that could be construed as a potential
conflict of interest.

## Author Contributions

SK, AP and RS contributed to the initial conception of the paper. The MHD
simulation was conducted by AP and the analysis was carried out primarily by
SK. The main draft was prepared by SK and, RB helped in the interpretation of
the simulation results and the overall presentation of the paper. All authors
participated in discussions and revisions on the draft.

## Funding

AP is supported from the Research Council of Norway through its Centres of
Excellence scheme, project number 262622, as well as through the Synergy Grant
number 810218 459 (ERC-2018-SyG) of the European Research Council. AP also
obtains partial support from NSF award AGS-2020703. RS is supported from the
project EFESIS (Exploring the Formation, Evolution and Space-weather Impact of
Sheath-regions), a personal grant of RS (Academy of Finland Grant 350015).

## Acknowledgments

We acknowledge the use of the visualization software VAPOR
(www.vapor.ucar.edu) for generating relevant graphics. Data and images are
courtesy of NASA/SDO and the HMI and AIA science teams. SDO/HMI is a joint
effort of many teams and individuals to whom we are greatly indebted for
providing the data. The authors thank the reviewers for providing insightful
comments and suggestions to enhance the scientific content as well as the
presentation of the article.

## Data Availability Statement

The datasets generated for this study are available on request to the
corresponding author.

## References

  * Al-Hachami & Pontin (2010) Al-Hachami, A. K., & Pontin, D. I. 2010, Astron Astrophys., 512, A84, doi: 10.1051/0004-6361/200913002
  * Amari et al. (2014) Amari, T., Canou, A., & Aly, J.-J. 2014, Nature, 514, 465, doi: 10.1038/nature13815
  * Aschwanden (2004) Aschwanden, M. J. 2004, Physics of the Solar Corona. An Introduction (Praxis Publishing Ltd)
  * Aulanier et al. (2005) Aulanier, G., Pariat, E., & Démoulin, P. 2005, Astron Astrophys., 444, 961, doi: 10.1051/0004-6361:20053600
  * Berger & Prior (2006) Berger, M. A., & Prior, C. 2006, Journal Phys. A: Mathe. and Gen., 39, 8321, doi: 10.1088/0305-4470/39/26/005
  * Bhattacharyya et al. (2007) Bhattacharyya, R., Janaki, M. S., Dasgupta, B., & Zank, G. P. 2007, Sol. Phys., 240, 63, doi: 10.1007/s11207-006-0280-5
  * Bhattacharyya et al. (2010) Bhattacharyya, R., Low, B. C., & Smolarkiewicz, P. K. 2010, Phys. Plasmas, 17, 112901, doi: 10.1063/1.3496379
  * Bobra et al. (2014) Bobra, M. G., Sun, X., Hoeksema, J. T., et al. 2014, Sol. Phys., 289, 3549, doi: 10.1007/s11207-014-0529-3
  * Bora et al. (2022) Bora, K., Bhattacharyya, R., Prasad, A., Joshi, B., & Hu, Q. 2022, The Astrophysical Journal, 925, 197, doi: 10.3847/1538-4357/ac3bce
  * Clyne et al. (2007) Clyne, J., Mininni, P., Norton, A., & Rast, M. 2007, New Jour. Phys., 9, 301, doi: 10.1088/1367-2630/9/8/301
  * Clyne & Rast (2005) Clyne, J., & Rast, M. 2005, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 5669, Visualization and Data Analysis 2005, ed. R. F. Erbacher, J. C. Roberts, M. T. Gröhn, & K. Börner, 284–294
  * Courant et al. (1967) Courant, R., Friedrichs, K., & Lewy, H. 1967, IBM journal of Research and Development, 11, 215
  * Dahlburg et al. (1991) Dahlburg, R. B., Antiochos, S. K., & Zang, T. A. 1991, Astrophys J., 383, 420, doi: 10.1086/170799
  * Duan et al. (2017) Duan, A., Jiang, C., Hu, Q., et al. 2017, Astrophys J., 842, 119, doi: 10.3847/1538-4357/aa76e1
  * Gary (2001) Gary, G. A. 2001, Sol. Phys., 203, 71, doi: 10.1023/A:1012722021820
  * Grinstein et al. (2007) Grinstein, F. F., Margolin, L. G., & Rider, W. J. 2007, Implicit large eddy simulation: computing turbulent fluid dynamics (Cambridge university press)
  * Hu & Dasgupta (2008) Hu, Q., & Dasgupta, B. 2008, Sol. Phys., 247, 87, doi: 10.1007/s11207-007-9090-7
  * Hu et al. (2008) Hu, Q., Dasgupta, B., Choudhary, D. P., & Büchner, J. 2008, Astrophys J., 679, 848, doi: 10.1086/587639
  * Hu et al. (2010) Hu, Q., Dasgupta, B., Derosa, M. L., Büchner, J., & Gary, G. A. 2010, Journal of Atmospheric and Solar-Terrestrial Physics, 72, 219, doi: 10.1016/j.jastp.2009.11.014
  * Inoue (2016) Inoue, S. 2016, Progress in Earth and Planetary Science, 3, 19, doi: 10.1186/s40645-016-0084-7
  * Inoue et al. (2014) Inoue, S., Hayashi, K., Magara, T., Choe, G. S., & Park, Y. D. 2014, Astrophys J., 788, 182, doi: 10.1088/0004-637X/788/2/182
  * Inoue et al. (2015) —. 2015, Astrophys J., 803, 73, doi: 10.1088/0004-637X/803/2/73
  * Jiang et al. (2013) Jiang, C., Feng, X., Wu, S. T., & Hu, Q. 2013, Astrophys J. Lett., 771, L30, doi: 10.1088/2041-8205/771/2/L30
  * Joshi et al. (2017) Joshi, N. C., Sterling, A. C., Moore, R. L., Magara, T., & Moon, Y.-J. 2017, The Astrophysical Journal, 845, 26, doi: 10.3847/1538-4357/aa7c1b
  * Kliem et al. (2013) Kliem, B., Su, Y. N., van Ballegooijen, A. A., & DeLuca, E. E. 2013, Astrophys J., 779, 129, doi: 10.1088/0004-637X/779/2/129
  * Kumar & Bhattacharyya (2016) Kumar, S., & Bhattacharyya, R. 2016, Phys. Plasmas, 23, 044501, doi: 10.1063/1.4945634
  * Kumar et al. (2016) Kumar, S., Bhattacharyya, R., Joshi, B., & Smolarkiewicz, P. K. 2016, Astrophys J., 830, 80, doi: 10.3847/0004-637X/830/2/80
  * Kumar et al. (2014) Kumar, S., Bhattacharyya, R., & Smolarkiewicz, P. K. 2014, Phys. Plasmas, 21, 052904, doi: 10.1063/1.4878955
  * Kumar et al. (2015) —. 2015, Phys. Plasmas, 22, 082903, doi: 10.1063/1.4928883
  * Lemen et al. (2012) Lemen, J. R., Title, A. M., Akin, D. J., et al. 2012, Sol. Phys., 275, 17, doi: 10.1007/s11207-011-9776-8
  * Liu et al. (2020) Liu, C., Prasad, A., Lee, J., & Wang, H. 2020, Astrophys J., 899, 34, doi: 10.3847/1538-4357/ab9cbe
  * Liu et al. (2016) Liu, R., Kliem, B., Titov, V. S., et al. 2016, Astrophys J., 818, 148, doi: 10.3847/0004-637X/818/2/148
  * Louis, Rohan E. et al. (2014) Louis, Rohan E., Puschmann, Klaus G., Kliem, Bernhard, Balthasar, Horst, & Denker, Carsten. 2014, A&A, 562, A110, doi: 10.1051/0004-6361/201321106
  * Masson et al. (2009) Masson, S., Pariat, E., Aulanier, G., & Schrijver, C. J. 2009, Astrophys J., 700, 559, doi: 10.1088/0004-637X/700/1/559
  * Mininni et al. (2008) Mininni, P., Lee, E., Norton, A., & Clyne, J. 2008, New Journ. Phys., 10, 125007, doi: 10.1088/1367-2630/10/12/125007
  * Nayak et al. (2019) Nayak, S. S., Bhattacharyya, R., Prasad, A., et al. 2019, Astrophys J., 875, 10, doi: 10.3847/1538-4357/ab0a0b
  * Nayak et al. (2020) Nayak, S. S., Bhattacharyya, R., Smolarkiewicz, P. K., Kumar, S., & Prasad, A. 2020, Astrophys J., 892, 44, doi: 10.3847/1538-4357/ab75bb
  * Pontin et al. (2016) Pontin, D., Galsgaard, K., & Démoulin, P. 2016, Sol. Phys., 291, 1739, doi: 10.1007/s11207-016-0919-9
  * Pontin et al. (2005) Pontin, D. I., Galsgaard, K., Hornig, G., & Priest, E. R. 2005, Phys. Plasmas, 12, 052307, doi: 10.1063/1.1891005
  * Pontin et al. (2013) Pontin, D. I., Priest, E. R., & Galsgaard, K. 2013, Astrophys J., 774, 154, doi: 10.1088/0004-637X/774/2/154
  * Prasad et al. (2018) Prasad, A., Bhattacharyya, R., Hu, Q., Kumar, S., & Nayak, S. S. 2018, Astrophys J., 860, 96, doi: 10.3847/1538-4357/aac265
  * Prasad et al. (2020) Prasad, A., Dissauer, K., Hu, Q., et al. 2020, The Astrophysical Journal, 903, 129, doi: 10.3847/1538-4357/abb8d2
  * Priest (2014) Priest, E. 2014, Magnetohydrodynamics of the Sun (Cambridge University Press)
  * Sarkar et al. (2019) Sarkar, R., Srivastava, N., & Veronig, A. M. 2019, Astrophys. J. Lett., 885, L17, doi: 10.3847/2041-8213/ab4da2
  * Schou et al. (2012) Schou, J., Scherrer, P. H., Bush, R. I., et al. 2012, Sol. Phys., 275, 229, doi: 10.1007/s11207-011-9842-2
  * Shibata & Magara (2011) Shibata, K., & Magara, T. 2011, Living Reviews in Solar Physics, 8, 6, doi: 10.12942/lrsp-2011-6
  * Smolarkiewicz & Charbonneau (2013) Smolarkiewicz, P. K., & Charbonneau, P. 2013, Journal of Computational Physics, 236, 608, doi: 10.1016/j.jcp.2012.11.008
  * Ugarte-Urra et al. (2007) Ugarte-Urra, I., Warren, H. P., & Winebarger, A. R. 2007, Astrophys J., 662, 1293, doi: 10.1086/514814
  * Wang & Liu (2012) Wang, H., & Liu, C. 2012, Astrophys J., 760, 101, doi: 10.1088/0004-637X/760/2/101
  * Wiegelmann (2008) Wiegelmann, T. 2008, Journal of Geophysical Research (Space Physics), 113, A03S02, doi: 10.1029/2007JA012432
  * Wiegelmann & Sakurai (2012) Wiegelmann, T., & Sakurai, T. 2012, Living Reviews in Solar Physics, 9, doi: 10.12942/lrsp-2012-5


# Multi-Modal Graph Neural Network for Joint Reasoning
on Vision and Scene Text

Difei Gao1,2*, Ke Li1,2*, Ruiping Wang1,2, Shiguang Shan1,2, Xilin Chen1,2
1Key Laboratory of Intelligent Information Processing of Chinese Academy of
Sciences (CAS),
Institute of Computing Technology, CAS, Beijing, 100190, China
2University of Chinese Academy of Sciences, Beijing, 100049, China
{difei.gao<EMAIL_ADDRESS>{wangruiping, sgshan<EMAIL_ADDRESS>
###### Abstract

Answering questions that require reading texts in an image is challenging for
current models. One key difficulty of this task is that rare, polysemous, and
ambiguous words frequently appear in images, _e.g_. names of places, products,
and sports teams. To overcome this difficulty, only resorting to pre-trained
word embedding models is far from enough. A desired model should utilize the
rich information in multiple modalities of the image to help understand the
meaning of scene texts, _e.g_. the prominent text on a bottle is most likely
to be the brand. Following this idea, we propose a novel VQA approach, Multi-
Modal Graph Neural Network (MM-GNN). It first represents an image as a graph
consisting of three sub-graphs, depicting visual, semantic, and numeric
modalities respectively. Then, we introduce three aggregators which guide the
message passing from one graph to another to utilize the contexts in various
modalities, so as to refine the features of nodes. The updated nodes have
better features for the downstream question answering module. Experimental
evaluations show that our MM-GNN represents the scene texts better and
obviously facilitates the performances on two VQA tasks that require reading
scene texts.

††footnotetext: * indicates equal contribution.

##  1 Introduction

The texts in a scene convey rich information that is crucial for performing
daily tasks like finding a place, acquiring information about a product, etc.
An advanced Visual Question Answering (VQA) model which is able to reason over
scene texts and other visual contents could have extensive applications in
practice, such as assisting visually impaired users, and education of
children. Our focus in this paper is to endow VQA models the ability of better
representing the image containing the scene texts, to facilitate the
performances of answering on VQA tasks [44, 8] that requires reading in
images.

Figure 1:  An image could contain information in multiple modalities, thus it
looks different to models with different abilities. For example, the image in
the eye of a human (top left) combines multi-modal contents. The visual
modality contains the visual appearances of objects and texts. The semantic
modality involves the semantics of the texts, yet it cannot determine the
semantics of rare words like “STP” in the image. The numeric modality is about
the numeric relation between numbers, like 65 is larger than 50. Q2 to Q4 are
three common questions involving reasoning on one of these modalities; while
Q1 requires using visual context to infer the semantic of “STP”. Random
characters within green dashed boxes represent modalities out of the
observer’s capability.

What are the unique challenges of modeling scene texts compared to the pure
visual entities (such as objects and scenes) and the natural language texts
(sentences or phrases)? A scene text inherently contains information in
multiple modalities, visual information, including color, shape, and semantic
information, _e.g_. “New York” is the name of a city, and numeric information
for numbers, _e.g_. “65” is larger than “50”. These types of information are
frequently used in answering daily questions. For example in Fig. 1, Q2
requires the model to find the target scene text with its visual information;
Q3 needs the model to understand the semantic of “65” which indicates the
amount of the money; Q4 requires the understanding of numeric relation between
numbers. Therefore, to correctly answer the questions involving scene texts,
it is indispensable to clearly depict each modality of the scene texts. In
addition, among these three modalities, it is more difficult to determine the
semantics of scene texts, because the scene texts encountered in daily
environments have a large possibility to be unknown, rare or polysemous words,
_e.g_., the name of a product “STP” as shown in Fig. 1. To tackle this
problem, the model should be able to determine the semantics of these texts
beyond only using word embedding [38, 26] pre-trained on a text corpus. In
this paper, we propose to teach the model how to utilize the context in
different modalities in surrounding of the words to determine their meanings
like a human, that is, 1) visual context: the prominent word on the bottle is
most likely to be its brand, as Q1 in Fig. 1 and Q1 in Fig. 2, 2) semantic
context: the surrounding texts of a rare or ambiguous word may help to infer
its meaning, _e.g_. Q2 shown in Fig. 2. In addition, utilizing semantics of
numbers can also depict more informative numeric relations between numbers, as
Q3 shown in Fig. 2.

Figure 2: Three example questions requiring leveraging different types of
multi-modal contexts to answer the questions. Q1: the model should use visual
context (the “EPIC” is a prominent word on the bottle) to infer the semantics
of the word “EPIC”. Q2: the model should infer that “Ged” indicates the owner
of the mug by using the semantic context of the word. Q3: The model should be
able to utilize the semantic of the numbers to depict more informative numeric
relations between numbers, such as “40” is the largest measurement among “40”,
“30”.

Following the aforementioned ideas, we propose a novel approach, Multi-Modal
Graph Neural Networks (MM-GNN), to obtain a better representation of the
multi-modal contents in an image and facilitate question answering. Our
proposed MM-GNN contains three sub-graphs for representing three modalities in
an image, i.e., visual modality for visual entities (including texts and
objects), semantic modality for scene texts, and numeric modality for number-
related texts, as shown in Fig. 3. The initial representations of nodes in
three graphs are obtained from priors, such as word embedding learned from the
corpora and Faster R-CNN features. Then, MM-GNN dynamically updates the
representations of nodes by three attention-based aggregators, corresponding
to utilizing three typical types of contexts in Fig. 2. These aggregators
calculate the relevance scores of two nodes considering their visual
appearances and layout information in the image, together with questions.
Besides relevance between nodes, by attending on the basis of layout
information, we are actually linking texts to their _physical carriers_ (the
object a text is printed or carved on); and given language hints, attention
models can pass messages more accurately, by considering the directives
implied by questions. Three different aggregators guide the message passing
from one modality to another modality (or to itself) to leverage different
types of contexts to refine the node features in a certain order. The updated
representation contains richer and more precise information, facilitating the
answering model to attend to the correct answer.

Finally, we conduct experiments with our proposed MM-GNN and its variants on
two recently proposed datasets TextVQA [44] and ST-VQA [8]. The results show
that our MM-GNN with newly designed aggregators effectively learns the
representations of the scene texts and facilitates the performance of VQA
tasks that require reading texts.

##  2 Related Work

Visual Question Answering Tasks. In recent years, numerous works have proposed
diverse VQA tasks [39, 34, 4, 16, 48, 42, 53, 24, 23] for evaluating different
types of core skills for answering visual questions. One line of datasets [39,
34, 4, 16], such as COCO-QA and VQA, studies questions about querying the
visual information of an image. Relevant works [33, 14, 41, 1, 6, 35, 50]
propose various attention mechanisms and multi-modal fusion techniques to
better locate the image region for a given question to facilitate the
answering procedure. Another line of works, such as CLEVR and GQA, introduces
questions demanding complex and compositional spatial reasoning skills.
Relevant works on these tasks introduce modular networks [2, 3, 20, 25, 22]
and neural-symbolic model [43, 51] which can robustly generate answer by
performing explicit multi-step reasoning on an image.

In this paper, we focus on a new type of questions recently proposed by the
TextVQA [44] and ST-VQA [8]. Compared to other VQA tasks, these two tasks are
unique in introducing questions about images that contain multi-modal
contents, including visual objects and diverse scene texts. To solve these
tasks, this paper focuses on how to formulate the multi-modal contents and
obtain better representations of scene texts and objects.

Representation Learning in VQA. Some inspiring works have studied the
representation of images to improve the performance of VQA tasks. The VQA
models [33, 14, 41, 35, 50] in the early stage mainly use the VGG or ResNet
feature pre-trained on the ImageNet to represent images. However, this type of
grid-level feature is limited to perform object-level attention. Therefore,
[1] proposes to represent one image as a list of detected object features.
Besides, to solve complex compositional questions, [43, 51] propose some
symbolic structural representations of the synthetic images (_e.g_. a scene
graph extracted from an image) in CLEVR which allow a VQA system to perform
explicit symbolic reasoning on them. More recently, [36, 32, 21] represent the
natural image as a fully connected graph (can be viewed as an implicit scene
graph where the relations between objects are not explicitly represented).
This type of graph allows the model to predict dynamic edge weights to focus
on a sub-graph related to the question and is widely used in natural images
QA.

The above-mentioned methods all focus on the representation of visual objects,
while this paper extends it to represent images with multi-modal contents. We
represent one image as a graph composed of three sub-graphs to separately
depict the entities in each modality and build the connections between
entities in different modalities.

Graph Neural Network. Graph Neural Network (GNN) [40, 10, 29, 46, 49] is a
powerful framework for representing graph-structured data. The GNN follows an
aggregation scheme that controls how the representation vector of a node
calculated by its neighboring nodes to capture specific patterns of a graph.
Recently, numerous variants of GNN are proposed to capture different types of
patterns of the graph in many tasks. For graph classification tasks, many
works on text classification [40, 46, 11], and protein interface prediction
[13] utilize the GNN to iteratively combine the information of the neighboring
nodes to capture the structure information of the graph.

In addition, many interesting works [45, 36, 32, 21, 47] introduce GNN for
grounding related task, such as referring expression [27] and visual question
answering [54, 16, 23]. These works [45, 36, 32, 21, 47] propose GNN with
language conditioned aggregator to dynamically locate a sub-graph of the scene
for a given query (e.g. a referring expression or a question), then GNN
updates the features of the nodes in the sub-graph to encode the relations
among objects. The updated nodes have better features for latter grounding
related tasks.

Similar to the previous GNNs [45, 36, 32, 21, 47] for grounding related tasks,
we utilize the GNN to obtain better features. But this paper extends GNN from
performing reasoning on a single-modal graph to a multi-modal graph. Besides,
our proposed new aggregation schemes can explicitly capture different types of
multi-modal contexts to update the representation of the nodes.

##  3 Method

In this section, we elaborate on the proposed multi-modal graph neural
networks (MM-GNN) for answering visual questions that require reading. Given
an image, which contains visual objects and scene texts, and a question, the
goal is to generate the answer. Our model answers the question in three steps:
(1) extract the multi-modal contents of an image and construct a three-layer
graph, (2) perform multi-step message passing among different modalities to
refine the representation of the nodes, and (3) predict the answer based on
the graph representation of the image.

Figure 3: The architecture of Multi-Modal Graph Neural Network (MM-GNN). It
first runs three offline models to initialize the representation of each
graph. After that, three aggregators successively update the representations
of nodes by passing the information in an inter-graph or intra-graph way to
obtain better representations of the nodes. Finally, the answer prediction
module uses these features to output the answer. The blue or red arrows on
lines between two nodes indicate the directions of information aggregation,
with deeper lines representing higher attention. [;] indicates the
concatenating operation.

###  3.1 Multi-Modal Graph Construction

As shown in Fig. 3, given an image, we first construct a multi-modal graph
composed of three sub-graphs, i.e., visual graph, semantic graph and numeric
graph for representing the information in three modalities. The _visual graph_
$G_{v}$ is a fully connected graph, where each node
$v_{i}\in{V_{v}}=\\{v_{i}\\}_{i=1}^{N}$ encodes the pure visual information of
entities (i.e., objects and scene texts) and $N$ is the number of candidate
objects generated by the extractor. The initial representation $v_{i}^{(0)}$
of $v_{i}$ is obtained by using an image feature extractor, _e.g_. Faster
R-CNN [15] detector.

The _semantic graph_ $G_{s}$ is also a fully connected graph, and each node
$s_{i}\in V_{s}=\\{s_{i}\\}_{i=1}^{M}$ represents the semantic meaning of a
scene text, _e.g_. “New York” is the name of a city, “Sunday” is one day in a
week, and $M$ is the number of extracted tokens. Concretely, to obtain the
semantic graph, we first use an Optical Character Recognition (OCR) model to
extract word tokens in images. Then, the $i$-th token is embedded by a pre-
trained word embedding model as the initial representation $s_{i}^{(0)}$ of
node $s_{i}$.

Besides, for number-type strings, _e.g_. “2000”, they not only contain
semantic meanings indicating string type, _e.g_. year (or dollars), but also
numeric meanings which indicate the numeric relations among other number-type
strings, _e.g_. “2000” is larger than “1900”. Thus, we construct a fully
connected _numeric graph_ $G_{n}$ to represent such information of number-type
texts $x_{i}\in V_{n}=\\{x_{i}\\}_{i=1}^{K}$. We categorize common numeric
texts into several types, _e.g_. number, time, etc. Then number-type texts are
embedded into -1 to 1, denoted as $x_{i}^{(0)}$, with sigmoid function (for
monotone number, like “12”) or cosine function (for period number, like
“10:00”) according to their categories, where $K$ is the number of number-type
texts. More details of the number encoder are in the Supp. Besides, the entire
graph composed of three sub-graphs is overall fully connected, but only a
specific part of nodes and edges is used in one aggregator.

###  3.2 Aggregation Scheme

After constructing the graph and initializing the representation of each node,
we propose three aggregators which guide the information flow between one sub-
graph to another or itself to utilize the different types of context to refine
the representation of the nodes, as shown in Fig. 3.

Visual-Semantic Aggregator. The first aggregator is the Visual-Semantic
aggregator, goal of which is two-fold: 1) leverage the visual context to
refine a semantic node (for solving questions like Q1 in Fig. 2) and 2)
utilize the semantic context to refine a visual node, making the visual
representation of texts’ physical carriers aware of the text (for solving
questions like Q3 in Fig. 5). Here, we first illustrate the implementation of
the first goal. For each node $s_{i}$ in semantic graph $G_{s}$, the
aggregator updates the representation of $s_{i}$ by first attending on
relevant neighbour nodes in visual graph
$v_{j}\in\mathcal{N}_{s_{i}}^{v}=\\{v_{j}\\}_{j=1}^{N}$, then aggregating the
information of attended nodes to update the representation of $s_{i}$.
Concretely, we first calculate the relevance score $a_{v_{j},s_{i}}^{s}$
between the node $s_{i}$ and its neighboring node $v_{j}$ based on their
visual representation and their location features $b_{s_{i}}$ and $b_{v_{j}}$
(i.e. the coordinates of bounding boxes) and the question feature $q$ obtained
by embedding the question words and going through an LSTM [19], formulated as,

$\displaystyle a^{\prime}_{v_{j},s_{i}}$
$\displaystyle=f_{s}([s_{i}^{(0)};f_{b}(b_{s_{i}})])^{T}(f_{v}([v_{j}^{(0)};f_{b}(b_{v_{j}})])\odot
f_{q}(q)),$ $\displaystyle a_{v_{j},s_{i}}^{s}$
$\displaystyle=\frac{\exp(a^{\prime}_{v_{j},s_{i}})}{\sum_{v_{j}\in\mathcal{N}_{s_{i}}^{v}}\exp(a^{\prime}_{v_{j},s_{i}})},$
(1)

where $f_{s}$, $f_{v}$, $f_{b}$ and $f_{q}$ are the MLPs for encoding the
semantic nodes, visual nodes, bounding boxes features, and question feature
respectively, $[;]$ indicates concatenating two vectors, and $\odot$ is
element-wise multiplication. Here, we also consider the question information
in calculating the attention score, because we hope the model can aggregate
the related nodes considering the information in the question. Then, we
aggregate the information of attended nodes, and append the aggregated
features to $s_{i}^{(0)}$ depicting the additional information of this node to
obtain the updated semantic representation, formulated as,

$\displaystyle
s_{i}^{(1)}=[s_{i}^{(0)};\sum_{v_{j}\in\mathcal{N}_{s_{i}}^{v}}a_{v_{j},s_{i}}^{s}f_{v^{\prime}}(v_{j}^{(0)})],$
(2)

where $s_{i}^{(1)}$ is the updated node representation at $t$=$1$ (shown in
Fig. 3), and $f_{v^{\prime}}$ is an MLP to encode the features of neighboring
nodes.

Similar to the scheme of refining semantic nodes, we obtain the updated
representation of nodes $v^{(1)}_{j}$ in $G_{v}$ by

$\displaystyle a_{v_{j},s_{i}}^{v}$
$\displaystyle=\frac{\exp(a^{\prime}_{v_{j},s_{i}})}{\sum_{s_{i}\in\mathcal{N}_{v_{j}}^{s}}\exp(a^{\prime}_{v_{j},s_{i}})}$
(3) $\displaystyle v_{j}^{(1)}$
$\displaystyle=[v_{i}^{(0)};\sum_{s_{i}\in\mathcal{N}_{v_{j}}^{s}}a_{v_{j},s_{i}}^{v}f_{s^{\prime}}(s_{j}^{(0)})],$
(4)

where $f_{s^{\prime}}$ is an MLP to encode the $s_{j}$, and
$\mathcal{N}_{v_{j}}^{s}$ indicates the neighboring nodes of $v_{j}$ in
semantic graph. Note that in all aggregators, the additional information is
appended after original features; specifically, after Visual-Semantic
aggregation, the dimensions of both semantic and visual features are
multiplied by two.

Semantic-Semantic Aggregator. This aggregator then refines the representation
of each semantic node by considering its semantic context (for solving
questions like Q2 in Fig. 2). For each node $s_{i}$, the aggregator finds the
proper neighboring nodes in semantic graph $\mathcal{N}_{s_{i}}^{s}=\\{s_{j}|$
$j\in\\{1,...,M\\}$ and $j\notin{i}\\}$ by attention mechanism, then
aggregating the information of attended nodes.

More specifically, the relevance score $a_{s_{j},s_{i}}$ of the node $s_{i}$
and its neighboring node $s_{j}$ is computed by their semantic representation
and their location features $b_{s_{i}}$ and $b_{s_{j}}$ in images, formulated
as,

$\displaystyle a^{\prime}_{s_{j},s_{i}}$
$\displaystyle=g_{s_{1}}([s_{i}^{(1)};g_{b}(b_{s_{i}})])^{T}(g_{s_{2}}([s_{j}^{(1)};g_{b}(b_{s_{j}})])\odot
g_{q}(q)),$ $\displaystyle a_{s_{j},s_{i}}$
$\displaystyle=\frac{\exp(a^{\prime}_{s_{j},s_{i}})}{\sum_{j\in\mathcal{N}_{s_{i}}^{s}}\exp(a^{\prime}_{s_{j},s_{i}})},$
(5)

where $g_{s_{1}}$, $g_{s_{2}}$, $g_{b}$, and $g_{q}$ are the MLPs for encoding
the node features (the first two), bounding boxes features and question
features. Then, we aggregate the information of attended nodes, and append the
aggregated features to $s_{i}$ as,

$\displaystyle
s_{i}^{(2)}=[s_{i}^{(1)};\sum_{s_{j}\in\mathcal{N}_{s_{i}}^{s}}a_{s_{j},s_{i}}g_{s_{3}}(s_{j}^{(1)})],$
(6)

where $s_{i}^{(2)}$ is the updated node representation at $t$=$2$, and
$g_{s_{3}}$ is an MLP to encode the features of neighboring nodes.

Semantic-Numeric Aggregator. The goal of this aggregator is to leverage the
semantic context to refine the value nodes to depict more informative numeric
relations between numbers (for solving questions like Q3 in Fig. 2). The
mechanism of semantic-numeric aggregator is similar to the mechanism of
achieving the first goal in Visual-Semantic aggregator. We first calculate the
relevance score $a_{s_{j},x_{i}}$ between nodes $s_{j}$ and $x_{i}$, then
aggregate the information of semantic nodes to numeric nodes, formulated as,

$\displaystyle
x_{i}^{(3)}=[x_{i}^{(0)};\sum_{s_{j}\in\mathcal{N}_{x_{i}}^{s}}a_{s_{j},x_{i}}h(s_{j}^{(2)})],$
(7)

where $h$ is for encoding the semantic nodes and
$\mathcal{N}_{x_{i}}^{s}=\\{s_{j}\\}_{j=1}^{M}$. Finally, we append the
numeric nodes to their corresponding semantic nodes as the representation of
OCR tokens, denoted as $c=[c_{1},...,c_{M}]$. For OCR tokens which are not
number-type, we concatenate a vector where the elements are all 0.

###  3.3 Answer Prediction

The answer prediction module takes the updated visual features
$v=[v_{1},...,v_{N}]$ and OCR features $c=[c_{1},...,c_{M}]$ as inputs, and
outputs the answer with copy mechanism [17]. Concretely, first, the size of
output space is extended to the vocabulary size + OCR number, where some
indexes in the output space indicate copying the corresponding OCR as the
answer, as shown in Fig. 3. Then, we calculate the attention score on features
of two modalities, and use attended features to generate the score of each
answer, formulated as,

$\displaystyle y$
$\displaystyle=f_{a}([f_{\text{att}}^{v}(v,q)^{T}v;f_{\text{att}}^{c}(c,q)^{T}c]),$
(8)

where $f_{\text{att}}^{v}$ and $f_{\text{att}}^{c}$ are Top-down attention
networks in [1] and $f_{a}$ is an MLP to output the scores on all candidate
answers. Finally, we optimize the binary cross entropy loss to train the whole
network. This allows us to handle cases that the answer is in both the pre-
defined answer space and the OCR tokens without penalizing for predicting
either one.

##  4 Experiments

###  4.1 Experiments Setup

Datasets. We evaluate our model using the TextVQA dataset and Scene-Text VQA
(ST-VQA) dataset.

For TextVQA dataset, it contains a total of 45,336 human-asked questions on
28,408 images from Open Image dataset [30]. Each question-answer pair comes
along with a list of tokens extracted by Object Character Recognition (OCR)
models, Rosetta [9]. These questions are evaluated by VQA accuracy metric
[16].

For ST-VQA dataset, it is composed of 23,038 images, paired with 31,791 human-
annotated questions. In the Weakly Contextualized task of ST-VQA, a dictionary
of 30,000 words are provided for all questions in this task; and the Open
Dictionary task is open-lexicon. These questions are evaluated by two metrics,
Average Normalized Levenshtein Similarity (ANLS) [31] and accuracy.

Implementation Details. For experiments on TextVQA dataset, we use answers
that appear at least twice in the training set as our vocabulary. Thus, the
size of our output space is the sum of the vocabulary size and the OCR number,
that is, $3997+50$. For question features, we use GloVe [38], which is widely
used in VQA models, to embed the words, then feed word embeddings to an LSTM
[19] with self-attention [52] to generate the question embedding. For encoding
OCR tokens, GloVe only can represent out-of-vocabulary (OOV) words as
0-vectors which are not suitable for initialization them, so we use fastText
[26], which can represent OOV words as different vectors, to initialize OCR
tokens. For image features, we use two kinds of pre-extracted visual features
for each image provided by the TextVQA dataset, 1) 196 grid-based features
obtained from pre-trained ResNet-152, and 2) 100 region-based features
extracted from pre-trained Faster R-CNN model like [1]. Both two visual
features are 2048-dimensional. Note that, the Faster R-CNN provides the visual
features of both objects and the scene texts because the detector produces an
excessive amount of bounding boxes, where some bounding boxes will bound the
scene texts.

Bounding box coordinates of objects and OCR tokens are first normalized into
the interval of $[0,1]$. Then we concatenate its center point, lower-left
corner and upper-right corner’s coordinates, width, height, area, and aspect
ratio, into a 10-dimensional feature. We used AdaMax optimizer [28] for
optimization. A learning rate of 1e-2 is applied on all parameters except the
fc7 layer for finetuning, which are trained with 5e-3.

For experiments on ST-VQA dataset, due to no available OCR results are
provided, we use TextSpotter [18] to extract scene text in images. For
question and OCR token embedding, we use the same models as in TextVQA; and
for image features, we only use Faster R-CNN features. Besides, we swap the
prediction vocabulary to suit the change of the dataset. For Open Dictionary
task, we collect answers which appear at least twice together with single-word
answers which appear once in the training set as our vocabulary. For Weakly
Contextualized task, given vocabulary of size 30,000 is directly utilized.
Besides, the source codes are implemented with PyTorch [37] 111Our source
codes are available at http://vipl.ict.ac.cn/resources/codes..

Method | Val | Test
---|---|---
Pythia | 13.04 | 14.01
LoRRA (BAN) | 18.41 | -
LoRRA (Pythia) | 26.56 | 27.63
BERT + MFH | 28.96 | -
MM-GNN (ours) | 31.44 | 31.10
BERT + MFH (ensemble) | 31.50 | 31.44
MM-GNN (ensemble) (ours) | 32.92 | 32.46
LA+OCR UB | 67.56 | 68.24

Table 1: VQA accuracy (%) on the TextVQA dataset, comparison with baselines
and state-of-the-art models. LA+OCR UB refers to maximum accuracy achievable
by models using Large Vocabulary of LoRRA and OCR results provided by TextVQA
dataset [44].

###  4.2 Results

Comparison with state-of-the-arts. Table 1 shows the comparison between our
method and state-of-the-art approaches on validation and test set of TextVQA
dataset. In the table, LoRRA (Pythia) is the baseline provided by TextVQA
dataset [44]. BERT + MFH is the winner of CVPR 2019 TextVQA challenge, which
is considered as the state-of-the-art, and its results are quoted from its
challenge winner talk. LA+OCR UB refers to maximum accuracy achievable by
models using current OCR results and Large Vocabulary of LoRRA provided by
TextVQA dataset [44]. LoRRA and BERT+MFH utilize advanced fusion techniques to
attend to OCR tokens which are encoded by pre-trained FastText [26]. BERT+MFH
additionally introduces the powerful question encoder BERT [12] into the
answering model. Our approach outperforms the above methods which mainly rely
on pre-trained word embedding, and achieves state-of-the-art results. Table 2
compares our method and state-of-the-art approaches on Weakly Contextualized
and Open Dictionary tasks of Scene-Text VQA datasets, where VTA is the winner
model of ICDAR 2019 Competition on STVQA, which extends the Bottom-Up VQA
model [1] with BERT to encode the question and text. From the results, we can
see that MM-GNN obtains an obvious improvement over baseline methods, e.g.
SAN(CNN)+STR, and achieves comparable accuracies with VTA.

Method | Weakly Contextualized | Open Dictionary
---|---|---
ANLS | Acc. | ANLS | Acc.
SAAA | 0.085 | 6.36 | 0.085 | 6.36
SAAA+STR | 0.096 | 7.41 | 0.096 | 7.41
SAN(LSTM)+STR | 0.136 | 10.34 | 0.136 | 10.34
SAN(CNN)+STR | 0.135 | 10.46 | 0.135 | 10.46
VTA [7] | 0.279 | 17.77 | 0.282 | 18.13
MM-GNN (ours) | 0.203 | 15.69 | 0.207 | 16.00

Table 2: Average Normalized Levenshtein Similarity (ANLS) and accuracies (%)
of different methods on Weakly Contextualized and Open Dictionary tasks on the
test set of ST-VQA dataset.

Effectiveness of Multi-Modal GNN. Our model’s advantage lies in the
introduction of a multi-modal graph and a well-designed message passing
strategy between different sub-graphs to capture different types of contexts.
Thus, we propose several variants of our model, where each variant ablates
some aggregators to show their indispensability.

  * •

No-GNN: This variant directly uses the object and OCR token features extracted
from pre-trained models to answer the questions without going through the
multi-modal GNNs. Other modules (output, embeddings) are kept the same to MM-
GNN.

  * •

Vanilla GNN: This variant puts object and OCR token features in a single
graph. It then performs an aggregation similar to Semantic-Semantic aggregator
to update the representation of the nodes. Other modules are kept the same to
MM-GNN.

  * •

Combinations of VS, SS, and SN: These variants construct the multi-modal graph
like MM-GNN, but only use one or two of the aggregators to update
representations. VS, SS, and SN represent Visual-Semantic, Semantic-Semantic,
and Semantic-Numeric aggregators respectively.

In addition, to better compare the results in detail, we categorize the
questions in TextVQA into three types. The first type of question is
Unanswerable, including questions that are unanswerable for given currently
provided OCR tokens in TextVQA dataset. We obtain this type of question by
checking whether the ground-truth answer absent from predefined answer
vocabulary and provided OCR tokens. The second type of question has answers
which can only be found in predefined answer vocabulary, such as “red”, “bus”,
and are not in OCR tokens, denoted as Vocab. The third type of question is OCR
related questions where the answers derive from the OCR tokens. Due to
Unanswerable type of questions cannot effectively evaluate the power of
different variants, we report scores of Vocab and OCR, which are under the
category of Answerable, and the Overall accuracy (including Unanswerable).

We evaluate the variants on the validation set of TextVQA dataset and report
their accuracies on each type of question, as shown in Table 3. Comparing the
performances of our full model MM-GNN with baseline No-GNN, we can see that
MM-GNN outperforms NO-GNN with about 4% on overall accuracy, and over 8% on
OCR related questions which are the main focus of TextVQA. This demonstrates
that introducing the graph representation into TextVQA model can effectively
help the answering procedure. Comparing the results of Vanilla GNN with MM-GNN
series, we find that if message passing in GNN is not well-designed, directly
applying GNN to TextVQA task is of little help. By comparing the results of
SS, SN, and VS, we find that Visual-Semantic aggregator contributes most
performance gain to OCR-related questions and overall accuracies. This
demonstrates our idea that multi-modal contexts are effective in improving the
quality of scene text representation.

However, we find that Numeric-Semantic aggregator contributes smaller than the
other two aggregators, probably because the portion of questions querying the
relations between the numbers, such as “what is the largest number in the
image?”, is relatively small. Thus, it limits the space to show the
effectiveness of this aggregator.

Method | Answerable | Overall
---|---|---
Vocab | OCR
No-GNN | 28.88 | 35.38 | 27.55
Vanilla GNN | 28.29 | 37.70 | 28.58
VS | 27.54 | 41.38 | 30.14
SS | 29.75 | 38.89 | 29.71
SN | 25.67 | 40.30 | 28.82
VS + SS | 28.81 | 42.16 | 30.99
VS + SN | 28.61 | 41.30 | 30.44
SS + SN | 25.69 | 41.99 | 29.78
VS + SS + SN (ours) | 27.85 | 43.36 | 31.21

Table 3: VQA accuracy (%) of VQA models with different kinds of Graph Neural
Networks on validation set of the TextVQA dataset.

Impact of different combining methods. Choosing combination schemes
controlling the fusion of a source node and the aggregated features of its
neighboring nodes is one crucial part of Graph Neural Network design. Original
MM-GNN is designed to gradually append additional information to each node to
serve as hints to distinguish OCR tokens from each other and facilitate the
answering model to locate the proper OCR token. Here, we replace our
concatenation updater by several variants which are broadly used in other
GNNs:

  * •

Sum: this variant combines the features of source nodes and its neighboring
nodes by sum operation, which is widely used in existing GNN works, such as
[5].

  * •

Product: this variant updates each node by computing the element-wise
multiplication of the node feature and aggregated features of its neighboring
nodes.

  * •

Concat + MLP: this variant updates each node by concatenating the node feature
and aggregated features of its neighboring nodes, then uses an MLP to encode
the concatenated features, which is used in previous visual language-related
methods [21].

Method | Answerable | Overall
---|---|---
Vocab | OCR
Sum | 27.40 | 40.40 | 29.59
Product | 27.89 | 32.18 | 25.79
Concat+MLP | 28.11 | 38.44 | 28.73
Concat (ours) | 27.85 | 43.36 | 31.21

Table 4: VQA accuracy (%) of variants of MM-GNN with different combination
schemes on validation set of TextVQA dataset. Figure 4:  Visualization of
attention results generated by MM-GNN. White boxes in the images bound the
finally predicted OCR; red boxes are the objects most related to predicted OCR
generated by Visual-Semantic aggregator; green boxes are the OCR tokens most
related to predicted OCR generated by Semantic-Semantic aggregator. We only
show boxes with attention value above a fixed threshold, with boxes more
attended having thicker lines. It shows that, our attentions are sharp and
truly attend on a few objects or texts that are relevant to answering
questions. Figure 5: Visualization of the reasoning procedure of MM-GNN model.
We only display the attention from the OCR token selected as the answer in the
answering module. The predicted OCR tokens are in white boxes. In the Visual-
Semantic Attention column, we show the attention from OCR tokens to the most
attended two visual objects, which are in red bounding boxes. The Semantic-
Semantic Attention column displays attention between the predicted OCR token
to the most attended OCR tokens, which are in yellow bounding boxes. In the
Semantic-Numeric Attention column, the attentions from the predicted OCR token
to other OCR tokens are shown (if any) in cyan. Images most important for
answering the question are framed in orange, and the thickness of bounding
boxes is proportional to their attention weights. These satisfying
visualizations demonstrate that our model learns to do step-by-step reasoning
in an explainable way.

We evaluate their performances on the validation set of TextVQA dataset, and
the performances are shown in Table 4. We can see that all three schemes harm
the performances more or less. Empirically, this is because the information
between nodes and their neighborhoods are compressed, gradually averaging out
the differences between node features, thereby bewildering the answering
module when it tries to locate the question-related OCR token. Note that all
three above combination schemes have the superiority of not changing node
feature size through iterations; while our concatenation scheme looses this
restriction to preserve more information in combination stage.

###  4.3 Qualitative Analysis

To gain an intuition of the attention distribution in aggregators, we
visualized them in Fig. 4. It shows that our model can produce very sharp
attentions to do reasoning on graphs, and the attentions have good
interpretability. In Q1, with the question querying about the player with a
ball, OCR tokens are guided by attention module to incorporate more
information related to the basketball; besides question hints, “WDOVER”
naturally attends to the area of the player. In Q2, the OCR token “Panera”
incorporates the position and semantic information from “BREAD” according to
questions, and can be chosen in answering module because the model learns that
the word above “BREAD” is very likely to be the name.

To better illustrate the answering procedure of the MM-GNN, we visualize the
attention results of each aggregator when answering questions and compare the
final answers of MM-GNN and baseline No-GNN. In Fig. 5, we show the results of
several typical questions: Q1 requires the model to utilize the visual context
“prominent texts on the signboard of a building” to infer the semantic of
unknown OCR token “transilvania”. Besides, the OCR context “banca” also helps
to find out that “transilvania” is the name of a bank. Q2 requires to link the
text “Boston” to its _physical carriers_ “the white book” and copy the OCR
token as the answer, Q3 requires to link the text “PIE” to its _physical
carriers_ “the region containing black letters”, Q4 requires to infer the
semantic of the OCR token from its surrounding OCR tokens, and Q5 evaluates
the ability of finding the largest number.

##  5 Conclusion

In this paper, we introduce a novel framework Multi-Modal Graph Neural Network
(MM-GNN) for VQA with scene texts. The MM-GNN represents the image with multi-
modal contents as a composition of three graphs, where each graph represents
one modality. In addition, the designed multi-modal aggregators in MM-GNN
utilize multi-modal contexts to obtain a finer representation of elements in
the image, especially for unknown, rare or polysemous words. Experimentally,
we show that our new image representation and message passing schemes greatly
improve the performance of the VQA with scene texts and provide interpretable
intermediate results.

Acknowledgements. This work is partially supported by Natural Science
Foundation of China under contracts Nos. 61922080, U19B2036, 61772500, and CAS
Frontier Science Key Research Project No. QYZDJ-SSWJSC009.

## References

  * [1] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang.  Bottom-up and top-down attention for image captioning and vqa.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6077–6086, 2018.
  * [2] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein.  Learning to compose neural networks for question answering.  In Proceedings of NAACL-HLT, pages 1545–1554, 2016.
  * [3] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein.  Neural module networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 39–48, 2016.
  * [4] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh.  Vqa: Visual question answering.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2425–2433, 2015.
  * [5] James Atwood and Don Towsley.  Diffusion-convolutional neural networks.  In Advances in Neural Information Processing Systems (NIPS), pages 1993–2001, 2016.
  * [6] Hedi Ben-Younes, Rémi Cadene, Matthieu Cord, and Nicolas Thome.  Mutan: Multimodal tucker fusion for visual question answering.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2612–2620, 2017.
  * [7] Ali Furkan Biten, Rubèn Tito, Andrés Mafla, Lluís Gómez, Marçal Rusiñol, Minesh Mathew, C. V. Jawahar, Ernest Valveny, and Dimosthenis Karatzas.  Icdar 2019 competition on scene text visual question answering.  CoRR, abs/1907.00490, 2019.
  * [8] Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusiñol, Ernest Valveny, C. V. Jawahar, and Dimosthenis Karatzas.  Scene text visual question answering.  Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 4291–4301, 2019.
  * [9] Fedor Borisyuk, Albert Gordo, and Viswanath Sivakumar.  Rosetta: Large scale system for text detection and recognition in images.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 71–79, 2018.
  * [10] Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann Lecun.  Spectral networks and locally connected networks on graphs.  In International Conference on Learning Representations (ICLR), 2014\.
  * [11] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst.  Convolutional neural networks on graphs with fast localized spectral filtering.  In Advances in Neural Information Processing Systems (NIPS), pages 3844–3852, 2016.
  * [12] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2019.
  * [13] Alex Fout, Jonathon Byrd, Basir Shariat, and Asa Ben-Hur.  Protein interface prediction using graph convolutional networks.  In Advances in Neural Information Processing Systems (NIPS), pages 6530–6539, 2017.
  * [14] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach.  Multimodal compact bilinear pooling for visual question answering and visual grounding.  In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 457–468, 2016.
  * [15] Ross Girshick.  Fast r-cnn.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1440–1448, 2015.
  * [16] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6904–6913, 2017.
  * [17] Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li.  Incorporating copying mechanism in sequence-to-sequence learning.  In In Annual Meeting of the Association for Computational Linguistics (ACL), pages 1631–1640, 2016.
  * [18] Tong He, Zhi Tian, Weilin Huang, Chunhua Shen, Yu Qiao, and Changming Sun.  An end-to-end textspotter with explicit alignment and attention.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5020–5029, 2018.
  * [19] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [20] Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko.  Learning to reason: End-to-end module networks for visual question answering.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 804–813, 2017.
  * [21] Ronghang Hu, Anna Rohrbach, Trevor Darrell, and Kate Saenko.  Language-conditioned graph networks for relational reasoning.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 10294–10303, 2019.
  * [22] Drew A. Hudson and Christopher D. Manning.  Compositional attention networks for machine reasoning.  International Conference on Learning Representations (ICLR), 2018\.
  * [23] Drew A. Hudson and Christopher D. Manning.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6700–6709, 2019.
  * [24] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1988–1997, 2017.
  * [25] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick.  Inferring and executing programs for visual reasoning.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2989–2998, 2017.
  * [26] Armand Joulin, Édouard Grave, Piotr Bojanowski, and Tomáš Mikolov.  Bag of tricks for efficient text classification.  In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 427–431, 2017.
  * [27] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg.  Referitgame: Referring to objects in photographs of natural scenes.  In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014.
  * [28] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In International Conference on Learning Representations (ICLR), 2015\.
  * [29] Thomas N. Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  In International Conference on Learning Representations (ICLR), 2017\.
  * [30] Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Murphy.  Openimages: A public dataset for large-scale multi-label and multi-class image classification.  Dataset available from https://github.com/openimages, 2017.
  * [31] Vladimir I. Levenshtein.  Binary codes capable of correcting deletions, insertions, and reversals.  In Soviet Physics Doklady, volume 10, pages 707–710, 1966.
  * [32] Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu.  Relation-aware graph attention network for visual question answering.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 10313–10322, 2019.
  * [33] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh.  Hierarchical question-image co-attention for visual question answering.  In Advances in Neural Information Processing Systems (NIPS), pages 289–297, 2016.
  * [34] Mateusz Malinowski and Mario Fritz.  A multi-world approach to question answering about real-world scenes based on uncertain input.  In Advances in Neural Information Processing Systems (NIPS), pages 1682–1690, 2014.
  * [35] Hyeonwoo Noh, Paul Hongsuck Seo, and Bohyung Han.  Image question answering using convolutional neural network with dynamic parameter prediction.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 30–38, 2016.
  * [36] Will Norcliffe-Brown, Stathis Vafeias, and Sarah Parisot.  Learning conditioned graph structures for interpretable visual question answering.  In Advances in Neural Information Processing Systems (NIPS), pages 8334–8343, 2018.
  * [37] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  pages 8024–8035, 2019.
  * [38] Jeffrey Pennington, Richard Socher, and Christopher Manning.  Glove: Global vectors for word representation.  In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.
  * [39] Mengye Ren, Ryan Kiros, and Richard Zemel.  Exploring models and data for image question answering.  In Advances in Neural Information Processing Systems (NIPS), pages 2953–2961, 2015.
  * [40] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini.  The graph neural network model.  IEEE Transactions on Neural Networks, 20(1):61–80, 2008.
  * [41] Idan Schwartz, Alexander Schwing, and Tamir Hazan.  High-order attention models for visual question answering.  In Advances in Neural Information Processing Systems (NIPS), pages 3667–3677, 2017.
  * [42] Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar.  Kvqa: Knowledge-aware visual question answering.  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 8876–8884, 2019.
  * [43] Jiaxin Shi, Hanwang Zhang, and Juanzi Li.  Explainable and explicit visual reasoning over scene graphs.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8376–8384, 2019.
  * [44] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.  Towards vqa models that can read.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8317–8326, 2019.
  * [45] Damien Teney, Lingqiao Liu, and Anton van den Hengel.  Graph-structured representations for visual question answering.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 804–813, 2017.
  * [46] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio.  Graph attention networks.  In International Conference on Learning Representations (ICLR), 2017\.
  * [47] Peng Wang, Qi Wu, Jiewei Cao, Chunhua Shen, Lianli Gao, and Anton van den Hengel.  Neighbourhood watch: Referring expression comprehension via language-guided graph attention networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1960–1968, 2019.
  * [48] Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton van den Hengel.  Fvqa: Fact-based visual question answering.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(10):2413–2427, 2018.
  * [49] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka.  How powerful are graph neural networks?  In International Conference on Learning Representations (ICLR), 2019\.
  * [50] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola.  Stacked attention networks for image question answering.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 21–29, 2016.
  * [51] Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum.  Neural-symbolic vqa: Disentangling reasoning from vision and language understanding.  In Advances in Neural Information Processing Systems (NIPS), pages 1039–1050, 2018.
  * [52] Zhou Yu, Jun Yu, Chenchao Xiang, Jianping Fan, and Dacheng Tao.  Beyond bilinear: Generalized multimodal factorized high-order pooling for visual question answering.  IEEE Transactions on Neural Networks and Learning Systems, 29(12):5947–5959, 2018.
  * [53] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  From recognition to cognition: Visual commonsense reasoning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6720–6731, 2019.
  * [54] Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Yin and yang: Balancing and answering binary visual questions.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5014–5022, 2016.

Supplementary Material

## Overview

In the supplementary material, we display more implementation details of
number encoder in graph construction (Sec.3.1 of the main body) and more
experimental results about our method Multi-Modal Graph Neural Network (MM-
GNN), including quantitative (on Sec. 7 ) and qualitative analysis (on Sec.
8).

##  6 Number encoder

In MM-GNN, digital numbers and days of the week which are similar to time-
related strings containing periodic information, e.g. “Sunday”, are considered
as numeric type strings. Besides, for periodic numbers, e.g. 10:00, we first
normalize it into $\frac{10}{24}$, then we apply the cosine embedding function
$x\rightarrow cos\,2\pi x$. Note that, how to encode the numeric strings is
still an open problem, different encoders can capture different relations
between numbers, e.g. an alternative representation that uses the polar
coordinate system which uses two functions cosine and sine to encode a number
could be better in representing the periodic numbers.

##  7 Quantitative Analysis

The impact of the order of aggregators. In Multi-Modal Graph Neural Network
(MM-GNN), the three aggregators which update the representations of nodes in
different sub-graphs are performed in a particular order, that is, first
perform Visual-Semantic aggregator (VS), then perform Semantic-Semantic
aggregator (SS), finally Semantic-Numeric aggregator (SN). In this part, we
evaluate the influences of all different orders of aggregators. The results
are shown in Table 5.

From the results, we can see that the performances of different variants are
similar to each other, which indicates that our proposed MM-GNN is robust to
changes in the order. This probably thanks to the functions of three
aggregators have relatively low dependencies on each other.

Method | Answerable | Overall
---|---|---
Vocab | OCR
SS-VS-SN | 26.71 | 42.99 | 30.54
SS-SN-VS | 26.88 | 43.11 | 30.65
VS-SN-SS | 25.80 | 43.08 | 30.27
SN-SS-VS | 26.58 | 42.97 | 30.46
SN-VS-SS | 27.66 | 41.63 | 30.33
VS-SS-SN (ours) | 27.85 | 43.36 | 31.21

Table 5: VQA accuracy (%) of variants of MM-GNN with different aggregators
order on validation set of TextVQA dataset.

Results on different question types. Similar to VQA dataset [4], we categorize
the questions in TextVQA into three groups based on their question-type, i.e.,
yes/no, number and others. The performances of MM-GNN and baseline No-GNN on
different question types are shown in Table 6. We can see that our method
mainly outperforms the baseline on others-type questions as expected, because
these questions are mostly related to understanding diverse scene texts.

Model | yes/no | number | others | Final
---|---|---|---|---
No-GNN | 88.79 | 35.14 | 22.65 | 27.55
MM-GNN | 88.93 | 36.13 | 27.36 | 31.21

Table 6: VQA accuracy (%) of MM-GNN with on different types questions of
TextVQA dataset compared to No-GNN.

##  8 Qualitative Analysis

In Fig. 6, we show more successful cases for our MM-GNN model on TextVQA
dataset. We show that MM-GNN obtains the correct answer, along with reasonable
attention results when utilizing the multi-modal contexts. In Fig. 7, we show
some failure cases of MM-GNN and analyze the possible reason for each example
in the image.

Figure 6:  Successful case analysis. The predicted OCR is bounded in a white
box. We show the attention from the predicted OCR tokens to the most attended
five visual objects in Visual-Semantic aggregator (in red bounding boxes) and
the attention between OCR tokens to the most attended two OCR tokens in
Semantic-Semantic aggregator (in green bounding boxes), where bolder bounding
box indicates higher attention value.

Figure 7:  Failure case analysis. The predicted OCR is bounded in a white box.
We show the attention from OCR tokens to the most attended five visual objects
in Visual-Semantic aggregator (in red bounding boxes) and the attention
between OCR tokens to the most attended two OCR tokens in Semantic-Semantic
aggregator (in green bounding boxes), where bolder bounding box indicates
higher attention value.