# Applications of physics informed neural operators

Shawn G. Rosofsky  Data Science and Learning Division, Argonne National
Laboratory, Lemont, Illinois 60439, USA  Department of Physics, University of
Illinois at Urbana-Champaign, Urbana, Illinois 61801, USA  NCSA, University of
Illinois at Urbana-Champaign, Urbana, Illinois 61801, USA  Hani Al Majed  Data
Science and Learning Division, Argonne National Laboratory, Lemont, Illinois
60439, USA  NCSA, University of Illinois at Urbana-Champaign, Urbana, Illinois
61801, USA  Department of Electrical and Computer Engineering, University of
Illinois at Urbana-Champaign, Urbana, Illinois 61801, USA  E. A. Huerta  Data
Science and Learning Division, Argonne National Laboratory, Lemont, Illinois
60439, USA  Department of Physics, University of Illinois at Urbana-Champaign,
Urbana, Illinois 61801, USA  Department of Computer Science, The University of
Chicago, Chicago, Illinois 60637, USA

###### Abstract

We present an end-to-end framework to learn partial differential equations
that brings together initial data production, selection of boundary
conditions, and the use of physics-informed neural operators to solve partial
differential equations that are ubiquitous in the study and modeling of
physics phenomena. We first demonstrate that our methods reproduce the
accuracy and performance of other neural operators published elsewhere in the
literature to learn the 1D wave equation and the 1D Burgers equation.
Thereafter, we apply our physics-informed neural operators to learn new types
of equations, including the 2D Burgers equation in the scalar, inviscid and
vector types. Finally, we show that our approach is also applicable to learn
the physics of the 2D linear and nonlinear shallow water equations, which
involve three coupled partial differential equations. We release our
artificial intelligence surrogates and scientific software to produce initial
data and boundary conditions to study a broad range of physically motivated
scenarios. We provide the source code, an interactive website to visualize the
predictions of our physics informed neural operators, and a tutorial for their
use at the Data and Learning Hub for Science.

##  I Introduction

The description of physical systems has a common set of elements, namely: the
use of fields (electromagnetic, gravitational, etc.) on a given spacetime
manifold, a geometrical interpretation of these fields in terms of the
spacetime manifold, partial differential equations (PDEs) on these fields that
describe the change of a system over spacetime, and an initial value
formulation of these equations with suitable boundary conditions [1]. Given
that the evolution of physical fields over spacetime may be naturally
expressed in terms of PDEs, a plethora of numerical methods have been
developed to accurately and rapidly solve these class of equations [2].

In time, and even with the advent of extreme scale computing, some physical
systems have become increasingly difficult to model, e.g., multi-scale and
multi-physics systems that combine disparate time and spatial scales, and
which demand the use of subgrid-scale precision to accurately resolve the
evolution of physical fields. This is a well known problem in multiple
disciplines, including general relativistic simulations [3, 4, 5], weather
forecasting [6], ab initio density functional theory simulations [7], among
many other computational grand challenges.

The realization that large scale computing resources are finite and will
continue to be oversubscribed [8, 9], has impelled scientists to explore novel
approaches to address computational bottlenecks in scientific software [10].
Some approaches include rewriting modules of software stacks to leverage GPUs,
leading to significant speedups [11, 12, 13]. Other contemporary approaches
have harnessed advances in machine learning to accelerate specific
computations in software modules [14], while others have developed entirely
new solutions by combining GPU-accelerated computing and novel signal
processing tools that have at their core machine learning or artificial
intelligence applications [15, 16, 17, 18, 19, 20, 21].

The creation of AI surrogates aims not only to enhance the science reach of
advanced computing facilities. Most importantly, this emergent area of
research aims to combine AI and extreme scale computing to enable research
that would otherwise remain unfeasible with traditional approaches.
Furthermore, AI surrogates aim to capture known knowledge, and first
principles to stir AI learning in the right direction, and then refine AI
surrogates performance and predictions by using experimental scientific data.
In time, it is expected that by exposing AI to detailed simulations, first
principles and experimental data, AI surrogates will capture the nonlinear
behaviour of experimental phenomena and guide the planning, automation and
execution of new experiments, leading to breakthroughs in science and
engineering.

In this article we contribute to the construction of AI surrogates by
demonstrating their application to solve a number of PDEs that are ubiquitous
in physics, and which have not been presented before in the literature.
Specifically, we have three levels of applicability of these new problems.
First, we performed sanity checks with simple PDEs to verify that our model
behaves as expected. Second, we tested new numerically challenging cases to
assess the applicability of PINOs under such conditions, which include non-
constant coefficients, coupled PDEs, and shocks. Third, we applied PINOs to
coastal and tsunami modeling with the 2D linear and nonlinear shallow water
equations. Beyond these original contributions, we also provide an end-to-end
framework that unifies initial data production, construction of boundary
conditions and their use to train, validate and test the performance and
reliability of AI surrogates. These activities aim to create FAIR (findable,
accessible, interoperable and reusable) and AI-ready datasets and models [22,
23].

In the following sections we introduce key concepts and ideas that will
facilitate the understanding and use of physics-inspired neural operators
(PINOs) to solve PDEs [24]. This article is organized as follows. In Section
II we introduce the AI tools we use to learn the physics of PDEs. We describe
our methods and approaches to create PINOs in Section III. We summarize the
PDEs we consider in this study in Section IV, and present a direct comparison
between numerical solutions of these PDEs and predictions from our PINOs in
Section V. We describe future directions of work in Section VI.

Figure 1: Neural network architecture The top panel shows the architecture of
our neural operators, whereas the inset in the bottom panel shows the
structure of one of the five Fourier layers, $\\{F_{1},...,F_{5}\\}$, we use
in our PINOs. The top left panel shows the data we feed into our models,
labelled as “Initial Condition $u(x,y)$”. This input data are initially lifted
into a higher dimension representation by the neural network, $P_{1}$.
Thereafter, we apply a series of updates that consist of non-local integral
operators and nonlinear activation functions, $\sigma$, shown in the inset
labelled as “Fourier Layer F”. Eventually, the neural network $P_{2}$ projects
back the updates, producing the output shown on the right panels, which
describe the time evolution of the system. In the inset, $T_{1}$ represents a
linear transform, $T_{2}$ a local linear transform, $\sigma$ a nonlinear
activation function. $\mathcal{F}$ and $\mathcal{F}^{-1}$ stand for Fourier
transform and inverse Fourier transform, respectively.

##  II Modeling physics with artificial neural networks

###  II.1 Physics informed neural networks

Physics informed neural networks (PINNs) provide a method of using known
physical laws to predict the results of various physical systems at high
accuracy [25, 26, 27, 28, 29]. These methods estimate the results for a given
physical system consisting of a PDE, initial conditions (ICs), and boundary
conditions (BCs) by minimizing constraints in the loss function. PINNs utilize
the automatic differentiation of deep learning frameworks to compute the
derivatives of the PDE to compute the residual error. Despite the success of
PINNs, their inability to produce results for different initial conditions
prevents them from being useful surrogate models.

###  II.2 Neural operators

Neural operators use neural networks to learn operators rather than single
physical systems. In our case, PDEs are the operator these networks try to
learn. Specifically, we provide neural operators with input fields
$\mathcal{A}$ that are composed of coordinates and relevant data such as
coefficient fields, ICs, and BCs. Neural operators then output the solutions
of that operator at those coordinates which we will denote as $\mathcal{U}$.
We can mathematically represent the neural operator $\mathcal{G}_{\theta}$ as
a mapping between the input fields $\mathcal{A}$ and the output fields
$\mathcal{U}$ as

$\displaystyle\mathcal{G}_{\theta}:\mathcal{A}\rightarrow\mathcal{U}$ (1)

where $\theta$ are the weights of the neural operator [30].

There are various types of neural operators that have been studied in recent
works such as DeepONets, physics informed DeepONets, low-rank neural operators
(LNO), graph neural operators (GNO), multipole graph neural operators (MGNO),
Fourier neural operators (FNO), and physics informed neural operators [31, 32,
33, 34, 35, 30, 24]. These networks have all illustrated their ability to
reproduce the results of operators much faster than computing with the
operator directly. For a more detailed look at neural operators, we refer the
reader to [30], which provides rigorous definitions of neural operators and
examines a large number of different neural operators.

###  II.3 Physics informed neural operators (PINOs)

PINOs are a variation of neural operators that incorporate knowledge of
physical laws into their loss functions [24]. PINOs have been shown to
reproduce the results of operators with remarkable accuracy. They employ the
FNO architecture which applies a fast Fourier transform (FFT) to the data and
applies its fully connected layers in Fourier space before performing an
inverse FFT back to real space [35]. Moreover, this architecture has
demonstrated the ability to perform zero-shot super-resolution, predicting on
higher resolution data having only seen low resolution data [35, 30]. Figure 1
illustrates the FNO architecture that we use in this study.

PINOs improve upon the FNO architecture by adding physics information such as
PDEs, ICs, BCs, and other conservation laws. By including the violation of
such laws into the loss function, the network can learn these laws in addition
to the data. Rather than using automatic differentiation, these networks use
Fourier derivatives to compute the derivatives for the PDE constraints as
automatic differentiation is very memory intensive for this type of
architecture. This physics knowledge enables the network to learn operators
faster and with less training data.

##  III Methods

Here we describe the approach we followed to design, train, validate and test
our PINOs, and then how we quantified their accuracy by comparing our
predictions with actual numerical solutions of PDEs we consider in this study.

###  III.1 Initial data

The first step in this process was to generate initial data that matches the
boundary conditions of the problem. This was done using the Gaussian random
fields (GRF) method in a similar fashion to [24] where the kernel was
transformed into Fourier space to match our periodic boundary conditions. We
employed a general Matern kernel, but discovered that the PINOs performed
better when the smoothness parameter was set to infinity. In this limit, this
Matern kernel becomes the radial basis function kernel (RBF) as used in [31]
defined as
$k_{l}\left(x_{1},x_{2}\right)=\exp\left(-\left\|x_{1}-x_{2}\right\|^{2}/2l^{2}\right)$,
where $l$ is the length scale of typical spatial deviations in the data. For
this work, we set $l=0.1$ for all cases to provide features at our desired
spatial scale except when explicitly stated otherwise. A number of these
random fields were produced for each of the test problems described Section
IV.

During this work, we found that the magnitude of the training data affected
the results of the training even in linear problems. In particular, the models
seem to have difficulty with higher magnitude initial data, especially when
the data had a magnitude greater than 1. We found that by reducing the
magnitude of the initial data, we could improve our results. We believe this
is attributed to the highly nonlinear nature of the neural network model.
Other neural network models also experience similar effects and therefore
normalize their data to improve performance. Thus, we were careful in
selecting the magnitude of our input fields during this study.

###  III.2 Boundary conditions

For our boundary conditions, we used periodic boundary conditions in all
cases. This boundary condition is good for problems with significant symmetry
over long length scales or for cases where the boundary is sufficiently far
from the region of interest. We are particularly focused on the latter case
where the boundary condition does not significantly impact the problem.
Moreover, the neural network architecture implements periodic boundary
conditions by default via FFTs. This periodicity of a certain dimension can be
removed by zero padding for said dimension before feeding it to the neural
network. We employed this padding in the time dimension with a length of
$l_{pad}=5$ for all cases except the 1D wave equation, which is periodic in
time for a time interval of $t=1$.

Although we do not do so in this work, one could also use this technique for
PDEs with non-periodic BCs by zero padding the spatial dimensions of the data.
We could then add an additional loss term for the BC to the loss in Equation 2
to describe our desired BC. We describe how to model similar terms in more
detail in Section III.4.

###  III.3 Training data generation

To generate training data, we took the random IC fields that we had previously
generated and evolved them in space and time. Specifically, we evolve each of
these equations in time with RK4 time stepping starting at $t=0$ until $t=1$
with the timestep $\delta t$ varying depending on the problem. To compute
spatial derivatives, we employed a finite difference method (FDM) with 4th
order central difference for most cases. The lone exception was the inviscid
Burgers equation in 2D which required a shock capturing method. Therefore, we
employed a finite volume method (FVM) with local Lax-Friedrichs (LLF) fluxes
and MP5 reconstruction.

###  III.4 Training approach

We set up the problem as follows. We are given some space and time coordinates
as well as the initial conditions at those coordinates. Our objective is to
compute the solution at each given space and time coordinate from the initial
data. To train the network to reproduce the simulated results, we employ
multiple losses to ensure the network properly reproduces the correct loss.
These losses are the data loss $\mathcal{L}_{data}$, the physics loss
$\mathcal{L}_{phys}$, and the IC loss $\mathcal{L}_{IC}$. We do not include a
loss term for the BC here because the FNO architecture of the PINO assures the
desired periodic BCs as long as those dimensions are not padded.

$\mathcal{L}_{data}$ attempts to fit the model predictions directly to the
training data. This loss is computed via the relative mean squared error (MSE)
between the training data and the network outputs. This relative MSE is
computed by dividing the MSE of the predictions by the norm of the true
values. For cases with multiple equations where the outputs vary in magnitude
such as the linear and nonlinear shallow water equations, care must be taken
to ensure that each of those outputs contributes equally to
$\mathcal{L}_{data}$. Therefore, we compute the relative MSE for each of the
output fields separately before combining them together.

$\mathcal{L}_{phys}$ ensures that the PINO predictions obey known physical
laws such as PDEs or more general conservation laws. We define the loss as the
MSE between the violation of the physical law and the value of said law if it
was perfectly satisfied which is $0$ in most cases. Again, one must be careful
in cases like the linear and nonlinear shallow water equations with multiple
physical laws whose violations differ significantly in their magnitude. In
such cases, compute the physical violations separately and multiply each term
by some weight before combining them and computing the MSE of the combined
term to obtain $\mathcal{L}_{phys}$.

$\mathcal{L}_{IC}$ allows the PINO to learn how the input initial condition
given to the network is the value of the output at that point at $t=0$.
Moreover, by minimizing $\mathcal{L}_{IC}$, $\mathcal{L}_{phys}$ more easily
converges to the correct solution for cases where the physical law is a PDE.
We defined $\mathcal{L}_{IC}$ as the relative MSE between the PINO prediction
at $t=0$ and the initial condition fed into the network. There were no cases
where the ICs differed significantly in magnitude for the PDEs since for the
linear and nonlinear shallow water equations, the velocity fields were taken
to be zero at $t=0$ and were not fed into the initial data.

To combine these terms into the total loss $\mathcal{L}_{tot}$, we perform a
weighted sum defined as

$\displaystyle\mathcal{L}_{total}$
$\displaystyle=w_{data}\mathcal{L}_{data}+w_{phys}\mathcal{L}_{phys}+w_{IC}\mathcal{L}_{IC}\,,$
(2)

where $w_{data}$ is the data weight, $w_{phys}$ is the physics weight, and
$w_{IC}$ is the IC weight. We varied these values between different cases and
even during the training. Typically, we would set $w_{data}$ to be $5$ or
$10$, $w_{phys}$ to be $1$ or $2$, and $w_{IC}$ to be $5$ or $10$.

We emphasize that we select the weights in the loss function through a process
of trial and error rather than a sophisticated hyperparameter optimization
process. This is typically the case with PINNs as well. If we take some
limits, we can understand why this is the case. If $w_{phys}$ is very high,
the PINO will try to minimize $\mathcal{L}_{phys}$ at the cost of the other
parameters. One solution that satisfies the most PDEs is if the output field
is zero for all space and time coordinates. Therefore, we add the initial
condition loss $\mathcal{L}_{IC}$ to ensure that $\mathcal{L}_{phys}$ evolves
the correct initial data. However, if $\mathcal{L}_{IC}$ is too large, the
output field is correct at $t=0$, but it does not evolve with time. Thus, we
used a process of trial and error to determine the correct weights.

For the training itself, we used the PyTorch framework [36]. We employed an
Adam optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.999$ with an initial
learning rate of 0.001. To fine tune the latter training steps, we employed a
multistep scheduler to reduce the learning rate at several intervals
throughout the training with a gamma value of 0.5. The specific number of
epochs and the epoch decay milestones vary between the different test cases. A
common setup that was used from several of the 2D cases was to train for 150
epochs and put the scheduler milestones at [25, 50, 75, 100, 125, 150] epochs.
To further improve the performance, we would use the checkpoints after 150
epochs of training and retrain again after for this same duration. In between
restarting from these checkpoints, we would sometimes modify the weights of
the various training losses if we observed some losses were lagging behind the
others.

###  III.5 Model architecture

As noted in II.3, the model uses the FNO architecture that is shown in Figure
1. We used a Gaussian error linear units (GELUs) for our nonlinearity for all
cases [37]. We note that for physics informed deep learning, we require the
nonlinear activation function to have a non-zero second derivative, ruling
some commonly used activation functions such as the rectified linear unit
(ReLU) [27].

The FNO architecture can be described by the widths, modes, and number of
layers. The widths and modes are given as arrays representing the width and
modes for each entry. The size of the array gives the number of layers and
each entry corresponds to a different layer. We used 4 layers in all cases.
For the 1D cases, we used the $width=[16,24,24,32]$ and $modes=[15,12,9,9]$.
For the 2D cases, we used $width=[64,64,64,64]$ and $modes=[8,8,8,8]$.

In addition, the network ends with a fully connected layer. We selected a
width of 128 for the fully connected layer for all cases.

###  III.6 Performance quantification

To quantify the performance, we ran PINO on the test dataset, then calculated
the MSE of their predictions with the test data and the MSE of the physics
loss that accompanied violations of physical laws. The test dataset is
composed of approximately $10\%$ of the simulations produced from the random
initial data that was separated from the rest of the data to ensure that the
PINO did not train on it.

##  IV Test Problems

We use PINOs to learn nine different PDEs. We consider the wave equation in 1D
and 2D to demonstrate that our methods produce accurate and reliable results.
We also present results for the 1D Burgers equation, which was used in [24] to
quantify the performance of PINOs to learn PDEs. We then put our methods at
work to solve a variety of PDEs with different levels of complexity.

Specifically, many past works including [33, 34, 35, 30, 24] investigating
using nonlinear neural operators study only 3 cases, the 1D Burgers equation,
2D Darcy Flow, and the 2D Navier Stokes equations. While these cases provide a
good way to compare neural operators to each other, they limited the
applicability of neural operators to other problems. Moreover, the
aforementioned cases fail to include various physical and numerical phenomena
such as shocks, coupled PDEs, and non-constant coefficients. Therefore, we
chose a wide array of linear and nonlinear PDEs to evaluate the PINO models
and isolate the places where they may have difficulty.

###  IV.1 Wave Equation 1D

Our first test was the wave equation in 1D with periodic boundary conditions.
This is a computationally simple PDE that is second order in time and models a
variety of different physics phenomena. The equation for the evolved field
$u(x,t)$ takes the form

$\displaystyle u_{tt}+c^{2}u_{xx}$ $\displaystyle=0,$ (3) $\displaystyle
u\left(x,0\right)$ $\displaystyle=u_{0}\left(x\right)\,,$ $\displaystyle
x\in\left[0,1\right),$ $\displaystyle\ t\in\left[0,1\right]\,,$

where $c=1$ is the speed of the wave.

###  IV.2 Wave Equation 2D

We extend the wave equation in 2D with periodic boundary conditions to explore
the requirements for adding the additional spatial dimension. The equation for
the evolved field $u(x,y,t)$ is given by

$\displaystyle u_{tt}+c^{2}\left(u_{xx}+u_{yy}\ \right)=0$ (4) $\displaystyle
u\left(x,y,0\right)=u_{0}\left(x,y\right)\,,$ $\displaystyle
x,y\in\left[0,1\right),\ t\in\left[0,1\right]\,,$

where as before the speed of the wave is set to $c=1$.

###  IV.3 Wave Equation 2D Non-Constant Coefficients

By adding a spatially variable wave speed, we study the performance of PINOs
in problems with non-constant coefficients. This variable wave speed $c(x,y)$
was incorporated into the PINO by treating it as another randomly generated
input field. To produce smoother training data, we set the spatial scale
$l=0.5$ parameter for the wave speed input field $c(x,y)$, though we still
used $l=0.1$ for the initial data. The equation for the evolved field
$u(x,y,t)$ is now given by

$\displaystyle u_{tt}+c(x,y)^{2}\left(u_{xx}+u_{yy}\ \right)=0$ (5)
$\displaystyle u\left(x,y,0\right)=u_{0}\left(x,y\right)\,,$ $\displaystyle
x,y\in\left[0,1\right),\ t\in\left[0,1\right]\,.$

###  IV.4 Burgers Equation 1D

The 1D Burgers equation with periodic boundary conditions serves as a
nonlinear test case with for a variety of numerical methods. This allowed us
to verify that our PINOs can learn and reconstruct nonlinear phenomena. The
equation for the field $u(x,y,t)$ is given in conservative form by

$\displaystyle u_{t}+\partial_{x}\left(u^{2}/2\right)$ $\displaystyle=\nu
u_{xx},$ (6) $\displaystyle u(x,0)$ $\displaystyle=u_{0}(x)\,,$ $\displaystyle
x\in[0,1),$ $\displaystyle\ t\in[0,1]\,,$

where the viscosity $\nu=0.01$.

###  IV.5 Burgers Equation 2D Scalar

To verify our model can handle nonlinear phenomena in 2D, we extend the
Burgers equation into 2D by assuming the field $u(x,y,t)$ is a scalar. The
equations take the form

$\displaystyle
u_{t}+\partial_{x}\left(u^{2}/2\right)+\partial_{y}\left(u^{2}/2\right)$
$\displaystyle=\nu\left(u_{xx}+u_{yy}\right)\,,$ (7) $\displaystyle u(x,y,0)$
$\displaystyle=u_{0}(x,y)\,,$ $\displaystyle x,y\in[0,1),$ $\displaystyle\
t\in[0,1]\,,\ $

where the viscosity $\nu=0.01$.

###  IV.6 Burgers Equation 2D Inviscid

We also looked at cases involving the inviscid Burgers equation in 2D in which
we set the viscosity $\nu=0$. This setup is known to produce shocks that can
result in numerical instabilities if not handled correctly. We used a finite
volume method (FVM) to generate this data to ensure stability in the presence
of shocks. In turn, this allowed us to investigate the network’s performance
when processing shocks. The equations are given by

$\displaystyle
u_{t}+\partial_{x}\left(u^{2}/2\right)+\partial_{y}\left(u^{2}/2\right)$
$\displaystyle=0\,,$ (8) $\displaystyle x,y\in[0,1),$ $\displaystyle\
t\in[0,1]\,,$ $\displaystyle u(x,y,0)$ $\displaystyle=u_{0}(x,y).$

We observed that the presence of the shock prevented our Fourier derivative
method from producing accurate residuals when we included the full equation in
our physics loss term. Instead, for our physics loss, we used the conserved
quantity,

$\displaystyle\int_{\Omega}u(x,y,t)\ dx\ dy=\int_{\Omega}u(x,y,0)\ dx\
dy=C\,,$ (9)

where $C$ is a constant and $\Omega$ is the domain. In other words, we ensured
that the total $u$ at every time instance is equal to the total $u$ at $t=0$.

###  IV.7 Burgers Equation 2D Vector

We then looked at a vectorized form of the 2D Burgers equation with periodic
boundary conditions. This allowed us to test how well the model handles the
coupled fields $u(x,y,t)$ and $v(x,y,t)$ that parameterize the system. The
equations take the form

$\displaystyle u_{t}+uu_{x}+vu_{y}=$
$\displaystyle\nu\left(u_{xx}+u_{yy}\right)\,,$ (10) $\displaystyle
v_{t}+uv_{x}+vv_{y}=$ $\displaystyle\nu\left(v_{xx}+v_{yy}\right)\,,$ (11)
$\displaystyle u(x,y,0)=u_{0}(x,y),$ $\displaystyle\ v(x,y,0)=v_{0}(x,y),$
$\displaystyle x,y\in[0,1),$ $\displaystyle\ t\in[0,1]$

where the viscosity $\nu=0.01$. We note that this system of equations does not
have a conservative form as there is not a continuity equation in this system.

Although coupled equations might seem like a trivial case, there are actually
a number of complexities that arise when changing the number of fields. First,
having multiple inputs and outputs results in an expanded parameter space for
the PINO. Thus, one would expect the network to require a larger volume of
training data to produce accurate results. Moreover, PINOs must solve multiple
equations simultaneously in the coupled case. In turn, the models must not
only solve for single fields, but also compute the contribution of those
fields on the other fields. Therefore, it is important to understand how PINOs
can resolve coupled fields in this relatively simple 2D vectorized Burgers
equation.

###  IV.8 Linear Shallow Water Equations 2D

To examine the properties of PINOs with 3 coupled equations, we examined the
ability of the networks to reproduce the linear shallow water equations with
periodic boundary conditions. We assumed that the height of the perturbed
surface $h(x,y,t)$ is initially perturbed, but the initial velocity fields
$u(x,y,t)$ and $v(x,y,t)$ are initially zero. These equations can be expressed
as

$\displaystyle\frac{\partial h}{\partial t}+H\left(\frac{\partial u}{\partial
x}+\frac{\partial v}{\partial y}\right)=0\,,$ (12)
$\displaystyle\frac{\partial u}{\partial t}-fv=-g\frac{\partial h}{\partial
x}\,,$ (13) $\displaystyle\frac{\partial v}{\partial t}+fu=-g\frac{\partial
h}{\partial y}\,,$ (14) $\displaystyle\textrm{with}\quad h(x,y,0)=h_{0}(x,y),\
u(x,y,0)=0,\ v(x,y,0)=0,\ $ $\displaystyle x,y\in[0,1),\ t\in[0,1],$

where the gravitational constant $g=1$, the mean fluid height $H=100$, and we
considered two cases for the Coriolis coefficient $f=\\{0,1\\}$.

The challenge in this case is that we have 3 coupled fields with one of them,
the perturbed surface height $h$ having a very different physical meaning than
the others. Moreover, $h$ is typically of much larger magnitude than either of
the velocity fields. By simplifying to a linear problem, we assess the ability
of PINOs to reproduce the results of magnitude varying fields without
complicated nonlinear terms. We note that coupled equations with terms of
varying magnitude are known to be difficult for traditional PINNs as one needs
to carefully weight and normalize the equations.

###  IV.9 Nonlinear Shallow Water Equations 2D

Finally, we examined the network performance on the nonlinear shallow water
equations. We assumed a similar setup as in the linear case where we assumed
the total fluid column height $\eta(x,y,t)$, was given by a mean value of $1$
plus some initial perturbation. We again assumed the initial velocity fields
$u(x,y,t)$ and $v(x,y,t)$ were zero. These equations are given by

$\displaystyle\frac{\partial(\eta)}{\partial t}+\frac{\partial(\eta
u)}{\partial x}+\frac{\partial(\eta v)}{\partial y}$ $\displaystyle=0\,,$ (15)
$\displaystyle\frac{\partial(\eta u)}{\partial t}+\frac{\partial}{\partial
x}\left(\eta u^{2}+\frac{1}{2}g\eta^{2}\right)+\frac{\partial(\eta
uv)}{\partial y}$ $\displaystyle=\nu\left(u_{xx}+u_{yy}\right)\,,$ (16)
$\displaystyle\frac{\partial(\eta v)}{\partial t}+\frac{\partial(\eta
uv)}{\partial x}+\frac{\partial}{\partial y}\left(\eta
v^{2}+\frac{1}{2}g\eta^{2}\right)$
$\displaystyle=\nu\left(v_{xx}+v_{yy}\right)\,,$ (17)
$\displaystyle\textrm{with}\quad\eta(x,y,0)=\eta_{0}(x,y),\ u(x,y,0)=0,\
v(x,y,0)=0,\ $ $\displaystyle x,y\in[0,1),\ t\in[0,1],$

where the gravitational coefficient $g=1$ and the viscosity coefficient
$\nu=0.002$ to prevent the formation of shocks.

This case combines the difficulty of having 3 coupled fields with complicated
nonlinear governing equations. This case provides a very useful and physically
interesting benchmark for PINOs as these equations model tsunamis. Moreover,
these equations take a similar form to the equations of compressible flow, but
without an additional equation for energy that is dependent on the equation of
state.

All these different PDEs serve the purpose of establishing the accuracy and
reliability of our PINOs, and then explore their application for more
interesting scenarios for the 2D Burgers equation, and 2D linear and nonlinear
shallow waters equations, which involve several coupled equations.

Figure 2: The training loss curves for the first 150 epochs of training for the nonlinear shallow water equations. Here we compare the different losses from the data, PDE, and ICs as well as their weighted total training loss.  Table 1: Summary of PINO results The first column describes the modeled equation. The second and third columns describe the spatial and temporal resolution, respectively. The fifth and sixth columns display the number of training and testing samples used. The final column provides the relative mean squared error (MSE) of our physics informed neural operators on the test set. Model |  Spatial Resolution |  Time Steps |  Training Samples |  Testing Samples |  Relative MSE
---|---|---|---|---|---
Wave Equation 1D |  128 |  101 |  900 |  100 |  1.22E-03
Wave Equation 2D |  128 $\times$ 128 |  101 |  45 |  25 |  6.60E-03
Wave Equation 2D Non-Constant Coefficients |  128 $\times$ 128 |  101 |  175 |  25 |  4.86E-02
Burgers Equation 1D |  128 |  101 |  90 |  10 |  6.94E-03
Burgers Equation 2D Scalar |  128 $\times$ 128 |  101 |  45 |  25 |  3.56E-03
Burgers Equation 2D Scalar FNO |  128 $\times$ 128 |  101 |  45 |  25 |  6.29E-03
Burgers Equation 2D Inviscid |  128 $\times$ 128 |  101 |  90 |  10 |  3.56E-02
Burgers Equation 2D Vector |  128 $\times$ 128 |  101 |  475 |  25 |  8.49E-03
Linear Shallow Water Equations 2D f=0 |  128 $\times$ 128 |  101 |  5 |  25 |  3.61E-02
Linear Shallow Water Equations 2D f=1 |  128 $\times$ 128 |  101 |  45 |  25 |  9.19E-03
Nonlinear Shallow Water Equations 2D |  128 $\times$ 128 |  101 |  45 |  25 |  1.50E-02

##  V Results

We now quantify the ability of our PINOs to learn the physics described by the
PDEs described above. In Figures 3-12 we present qualitative and quantitative
results that illustrate the performance of our PINOs. We use two types of
quantitative metrics, namely, absolute error (shown in each figure) and mean
squared error (summarized in Table 1) between PINO predictions and ground
truth simulations. While the figures below provide snapshots of the
performance of our PINOs at a given time, $t$, we also provide interactive
visualizations of these results at this website. We also provide a tutorial to
use our PINOs and reproduce our results in the Data and Learning Hub for
Science [38, 39].

Before discussing specific results, we also present the training curves of the
nonlinear shallow water equations over its first 150 epochs in Figure 2. Once
our PINOs are fully trained, they are computationally efficient. Averaging
over 25 test sets, our PINOs produce full simulations within 0.165 seconds for
the most computationally intensive case, the nonlinear shallow water
equations.

Figure 3: PINO for 1D wave equation Test set initial conditions (left column)
fed into our neural networks. The center left column displays the ground truth
value of $u_{true}(x,t)$ as produced by our simulations. The center right
column shows the value of $u_{pred}(x,t)$ predicted by our PINO. The right
column illustrates the error between the PINO predictions and the ground
truth, $u_{pred}-u_{true}$.

###  V.1 Wave Equation 1D Results

Figure 3 shows that our PINO for the 1D wave equation learns and describes the
physics of this PDE with remarkable accuracy. These results serve the purpose
of validating our methods with a simple PDE. We found that the MSE in this
case was $1.22\times 10^{-3}$ on the test dataset. We note that for this case,
we experimented with using a very high resolution of $4096$ grid points to
generate the training data, but downsampled by a factor of $32$ to a final
resolution of $128$ grid points that were fed into the network. This simulates
having available less data to reproduce a result than was required to compute
it.

Figure 4: PINO for 2D wave equation As Figure 3, but now in 2D. The left
column represents the test set initial conditions that we feed into our PINOs.
We evolved the systems until $t=1$ and present ground truth solutions (center
left) and PINO predictions (center right). Even after evolving these
simulations until the end of the time domain under consideration, our PINOs
predict with excellent accuracy the evolution of the system, as shown in the
right column which shown at $t=1$, $u_{pred}-u_{true}$.

###  V.2 Wave Equation 2D Results

Next we consider the 2D wave equation. Figure 4 summarizes our results. As in
the 1D scenario, our PINOs have learned the physics of the 2D wave equations
with excellent accuracy. Specifically, we found the MSE on the test dataset
for this case to be $6.70\times 10^{-3}$.

Figure 5: PINO for 2D non-constant coefficients wave equation Similar to
Figure 4, but here we have a non-constant wave speed. The top row depicts the
input fields into the ANN, which are the initial condition $u$, the wave speed
coefficient $c$, and the wave speed coefficient squared $c^{2}$ from left to
right respectively. As in the constant coefficient case, we evolved the
systems until $t=1$. On the bottom row, we present the values at $t=1$ for the
ground truth solutions (left), the PINO predictions (center), the error
$u_{pred}-u_{true}$.

###  V.3 Wave Equation 2D Non-Constant Coefficients Results

Next we consider the variation of the 2D wave equation with non-constant
coefficients. Figure 5 illustrates the results for this problem. As this
scenario is considerably more difficult than the uniform coefficient wave
equation, the PINO performance is slightly worse, with an MSE of on the test
dataset of $0.0486$.

Figure 6: PINO for 1D Burgers equation The left column shows the test set
initial conditions fed into our PINOs. The center left column shows the ground
truth value, $u_{true}(x,t)$, produced by our simulations. The center right
column presents the predicted values $u_{pred}(x,t)$ by our PINOs. The right
column shows the discrepancy between PINO and ground truth predictions,
$u_{pred}-u_{true}$.

###  V.4 Burgers Equation 1D Results

We now turn our attention to the Burgers equation, and begin this analysis
with a simple and illustrative case, namely the 1D Burgers equation, shown in
Figure 6. These results show that our PINOs have accurately learned the
physics described by this PDE with an excellent level of accuracy.
Quantitatively, the MSE on the test dataset was $6.94\times 10^{-3}$. These
results furnish evidence that our methods can reproduce results published
elsewhere in the literature [24].

We now turn our attention to several 2D Burgers equations that, to the best of
our knowledge, have not been explored previously in the literature.

Figure 7: PINO and FNO for 2D scalar Burgers equation Displays the results of
the PINO and FNO models for this equation. The left column shows a sample of
test set initial conditions fed into our models. The center left column
displays the ground truth value, $u_{true}(x,y)$, of our simulations once they
have evolved up to $t=1$. The center right column shows model predictions for
this PDE at $t=1$. We have selected this time to quantify the accuracy of our
model once the system has evolved sufficient time to accumulate errors. The
right column illustrates the discrepancy between model predictions and the
ground truth, $u_{pred}-u_{true}$, at $t=1$. The top row shows the results for
the PINO and the bottom row shows the results for the FNO.

###  V.5 Burgers Equation 2D Scalar Results

This PDE is given by Equation (7). As shown in Figure 7 our PINOs can learn
and describe the physics of this PDE accurately. Note that we present results
for this PDE once the system has been evolved throughout the time domain under
consideration, i.e., $t\in[0,1]$. By presenting results at $t=1$ we gain a
good understanding of the actual performance of our PINOs once they have
evolved in time and accumulated numerical errors that may depart from ground
truth values. For this case, we compared the results to a FNO which was
trained with the same architecture, but without a physics informed loss
component. In Figure 7 we present the results of the PINO and the FNO for the
same initial condition. We found that the MSE for the full test dataset was
$3.56\times 10^{-3}$. for the PINO and was $6.29\times 10^{-3}$ for the FNO.

Figure 8: PINO for 2D inviscid Burgers equation As Figure 7 but now in the
inviscid case given by Equation 8.

###  V.6 Burgers Equation 2D Inviscid Results

This PDE is given by Equation 8. As we mentioned before, shocks are a common
occurrence for this PDEs, and may lead to numerical instabilities if we do not
use shock capturing schemes. We have quantified the ability of our PINOs to
handle shocks. We found the PINO to have an MSE of $0.0356$ on the test
dataset. In Figure 8 we notice that our PINOs are able to describe the physics
of this PDE with excellent accuracy. However, we observe a significant
discrepancy between ground truth and AI predictions right at the regions where
shocks occur. These findings indicate that further work is needed to better
handle PDEs that involve shocks. This is a specific area of work that we will
pursue in the future.

Specifically, such work would look at improvements to the way the model and
loss function handle discontinuities in the data. For example, Fourier
transforms, which are employed in the neural network and to represent
derivatives in the loss function, are known to be highly sensitive to
discontinuities. Thus, potential improvements may encompass the use of
Galerkin neural networks [40] to better handle discontinuities, and loss
functions that incorporate particle number conservation. These improvements
should be considered in future work.

###  V.7 Burgers Equation 2D Vector Results

This is the most complex PDE of the Burgers family we consider in this study,
see Equations (10) and (11). The novel feature of this PDE is that we now need
to treat two different fields $(u,v)$. In Figure 9 we present a sample result
from the test dataset, which demonstrate that our PINOs have learned the
physics of this PDE and describe it with remarkable precision even after we
have evolved this system until the end of the time domain under consideration.
Quantitatively, we achieved an MSE of $8.49\times 10^{-3}$ on the test
dataset.

Figure 9: PINO for 2D vector Burgers equation The top and bottom rows show the
$u$ and $v$ fields, respectively. The left column shows test set initial
conditions for $(u,v)$. The center left column shows ground truth values for
the $(u,v)$ fields once this PDE has been evolved up to $t=1$. The center
right column shows PINO results for the $(u,v)$ fields at $t=1$. The right
column show the discrepancy between PINO and ground truth values at $t=1$.

These results complete our analysis for a variety of PDEs that involve the
Burgers equation, and indicate that for various levels of complexity and
initial conditions our PINOs are capable of learning and describing the
physics of these PDEs with excellent accuracy. We have also realized that we
need to further develop these methods for PDEs that involve shocks. We are
keenly interested in this particular case, and will explore in future work.

Figure 10: PINO for 2D linear shallow water equation Assuming a system with a
Coriolis coefficient $f=0$, we show the $h$, $u$, and $v$ fields in the top,
middle, and bottom rows, respectively. The left column shows the initial
condition provided to the network. The center left column displays the ground
truth value at $t=1$ as produced by the simulation. The center right column
shows the value of prediction at $t=1$ predicted by our PINO. The right column
indicates the discrepancy between PINO predictions and the ground truth at
$t=1$. Figure 11: PINO for 2D linear shallow water equation As Figure 10 but
now with a Coriolis coefficient $f=1$.

###  V.8 Linear Shallow Water Equations 2D Results

Another original result in this study is the use of PINOs to learn the physics
of 3 coupled PDE equations. The first case under consideration is the 2D
linear shallow water equation given by Equations (12)-(14). In this case we
now consider the height of the perturbed surface, $h$, and the fields $(u,v)$.
Figures 10 and 11 present results assuming two Coriolis coefficient
$f=\\{0,1\\}$, respectively. The discrepancy between PINO predictions and
ground truth values in the figures is very small, which furnishes strong
evidence for the adequacy of PINOs to learn the physics of this PDE Highlights
of these results include:

  * $\bullet$

$\mathbf{f=0}$ case: We only required 5 training samples to produce PINOs that
exhibit optimal performance. In this case, the MSE on the test dataset was
$0.0361$. This illustrates the ability of our PINOs to generalize from very
sparse training data.

  * $\bullet$

$\mathbf{f=1}$ case: We increased the number of training samples to 45, which
is comparable to some of the other 2D cases. The MSE on the test dataset was
calculated to be $9.19\times 10^{-3}$.

These results provide a glimpse of the capabilities of PINOs to learn the
physics of these linear, coupled PDEs. We have extensively tested these
equations and found that they are robust to a broad range of initial data.
These results provided enough motivation to explore the use of PINOs for a
more challenging set of coupled PDEs, namely, the 2D nonlinear shallow water
equations that we discuss next.

Figure 12: PINO for 2D nonlinear shallow water equation The $(\eta,u,v)$
fields are shown in the top, middle, and bottom rows, respectively. The left
column presents the test set initial condition that is fed into our network.
The center left column displays the ground truth value at $t=1$ as produced by
our simulations. The center right column shows the value of prediction at
$t=1$ predicted by the PINO. The right column illustrates the discrepancy
between the PINO predictions and the ground truth at $t=1$.

###  V.9 Nonlinear Shallow Water Equations 2D Results

The final original contribution of this study is the use of PINOs to learn the
physics of the 2D nonlinear shallow water equation, given by Equations (15)-
(17). Even though this PDE is significantly more complex than its linear
counterpart, we notice in Figure 12 that our PINOs learn the physics described
by the fields $(\eta,u,v)$ with remarkable accuracy. We found the MSE for the
test dataset of this case to be 0.0150

Finally, we provide an additional metric to quantify the overall performance
of all the PDEs we have used in this study in Table 1. These results indicate
that our PINOs provide state-of-the-art results to model simple PDEs (1D wave
equation & 1D Burgers equations), and excellent performance for a variety of
PDEs that are solved for the first time in the literature with physics
informed neural operators.

##  VI Conclusions

We have introduced an end-to-end framework to learn PDEs that range from
simple equations that serve the purpose of testing our methods (1D wave
equation & 1D Burgers equation) to increasingly more complex equations (2D
scalar, 2D inviscid and 2D vector Burgers equation), and coupled PDEs that are
solved with PINOs for the first time in the literature (2D linear & nonlinear
shallow waters equations). The methods we introduce in this study provide the
flexibility to produce initial data to test the robustness and applicability
of AI surrogates for a broad range of physically motivated scenarios. We
provide scientific visualizations of our results through an interactive
website. We also release our PINOs and scientific software through the Data
and Learning Hub for Science so that AI practitioners may download, use and
further develop our neural networks. In addition to this Data and Learning Hub
for Science implementation, we release the source code used in this paper.

Future work may focus on the extension of these PINOs to high-dimensional PDEs
that are relevant for the modeling of complex phenomena that demand subgrid
scale precision, and which typically lead to computationally expensive
simulations. We will also focus on developing methods that handle shocks
effectively, since these phenomena are common in fluid dynamics and
relativistic astrophysics, to mention a few. We will also continue our
research program combining scientific visualization and accelerated computing
to gain insights about what PINOs learn from data, and how this information
may be used to enhance their performance when applied to experimental datasets
[41, 42, 43].

It is our expectation that our AI surrogates may be used to replace
computationally demanding numerical methods to learn PDEs in scientific
software used to model multi-scale and multi-physics phenomena—e.g., numerical
relativity simulations of gravitational wave sources, weather forecasting,
etc.,—and eventually provide data-driven and physics informed solutions that
more accurately describe and identify novel features and patterns in
experimental data.

## Acknowledgments

This material is based upon work supported by Laboratory Directed Research and
Development (LDRD) funding from Argonne National Laboratory, provided by the
Director, Office of Science, of the U.S. Department of Energy under Contract
No. DE-AC02-06CH11357. This research used resources of the Argonne Leadership
Computing Facility, which is a DOE Office of Science User Facility supported
under Contract DE-AC02-06CH11357. S.R. and E.A.H. gratefully acknowledge
National Science Foundation award OAC-1931561. This work used the Extreme
Science and Engineering Discovery Environment (XSEDE), which is supported by
National Science Foundation grant number ACI-1548562. This work used the
Extreme Science and Engineering Discovery Environment (XSEDE) Bridges-2 at the
Pittsburgh Supercomputing Center through allocation TG-PHY160053. This
research used the Delta advanced computing and data resource which is
supported by the National Science Foundation (award OAC-2005572) and the State
of Illinois. Delta is a joint effort of the University of Illinois Urbana-
Champaign and its National Center for Supercomputing Applications.

## References

  * Geroch [1996] R. P. Geroch, Partial differential equations of physics, in  _46th Scottish Universities Summer School in Physics: General Relativity_ (1996) arXiv:gr-qc/9602055
  * Press  _et al._ [2007] W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery,  _Numerical Recipes 3rd Edition: The Art of Scientific Computing_ , 3rd ed. (Cambridge University Press, USA, 2007)
  * Radice [2020] D. Radice, Binary Neutron Star Merger Simulations with a Calibrated Turbulence Model, Symmetry 12, 1249 (2020), arXiv:2005.09002 [astro-ph.HE]
  * Radice  _et al._ [2021] D. Radice, S. Bernuzzi, A. Perego, and R. Haas, A New Moment-Based General-Relativistic Neutrino-Radiation Transport Code: Methods and First Applications to Neutron Star Mergers, arXiv e-prints , arXiv:2111.14858 (2021), arXiv:2111.14858 [astro-ph.HE]
  * Foucart [2020] F. Foucart, A brief overview of black hole-neutron star mergers, Frontiers in Astronomy and Space Sciences 7, 10.3389/fspas.2020.00046 (2020)
  * Schalkwijk  _et al._ [2015] J. Schalkwijk, H. J. J. Jonker, A. P. Siebesma, and E. V. Meijgaard, Weather forecasting using gpu-based large-eddy simulations, Bulletin of the American Meteorological Society 96, 715 (2015)
  * Erba  _et al._ [2017] A. Erba, J. Baima, I. Bush, R. Orlando, and R. Dovesi, Large-scale condensed matter dft simulations: Performance and capabilities of the crystal code, Journal of Chemical Theory and Computation 13, 5019 (2017), pMID: 28873313, https://doi.org/10.1021/acs.jctc.7b00687
  * Asch  _et al._ [2018] M. Asch, T. Moore, R. Badia, M. Beck, P. Beckman, T. Bidot, F. Bodin, F. Cappello, A. Choudhary, B. de Supinski, E. Deelman, J. Dongarra, A. Dubey, G. Fox, H. Fu, S. Girona, W. Gropp, M. Heroux, Y. Ishikawa, K. Keahey, D. Keyes, W. Kramer, J.-F. Lavignon, Y. Lu, S. Matsuoka, B. Mohr, D. Reed, S. Requena, J. Saltz, T. Schulthess, R. Stevens, M. Swany, A. Szalay, W. Tang, G. Varoquaux, J.-P. Vilotte, R. Wisniewski, Z. Xu, and I. Zacharov, Big data and extreme-scale computing: Pathways to convergence-toward a shaping strategy for a future software and data ecosystem for scientific inquiry, The International Journal of High Performance Computing Applications 32, 435 (2018)
  * Gropp  _et al._ [2020] W. Gropp, S. Banerjee, and I. Foster, Infrastructure for Artificial Intelligence, Quantum and High Performance Computing, arXiv e-prints , arXiv:2012.09303 (2020), arXiv:2012.09303 [cs.CY]
  * Huerta  _et al._ [2020] E. A. Huerta, A. Khan, E. Davis, C. Bushell, W. D. Gropp, D. S. Katz, V. Kindratenko, S. Koric, W. T. C. Kramer, B. McGinty, K. McHenry, and A. Saxton, Convergence of Artificial Intelligence and High Performance Computing on NSF-supported Cyberinfrastructure, Journal of Big Data 7, 88 (2020), arXiv:2003.08394 [physics.comp-ph]
  * Taher [2009] M. Taher, Accelerating scientific applications using gpu’s, in  _2009 4th International Design and Test Workshop (IDT)_ (2009) pp. 1–6
  * Rodrigues  _et al._ [2008] C. I. Rodrigues, D. J. Hardy, J. E. Stone, K. Schulten, and W.-M. W. Hwu, Gpu acceleration of cutoff pair potentials for molecular modeling applications, in  _Proceedings of the 5th Conference on Computing Frontiers_, CF ’08 (Association for Computing Machinery, New York, NY, USA, 2008) p. 273–282
  * Wysocki  _et al._ [2019] D. Wysocki, R. O’Shaughnessy, J. Lange, and Y.-L. L. Fang, Accelerating parameter inference with graphics processing units, Phys. Rev. D 99, 084026 (2019), arXiv:1902.04934 [astro-ph.IM]
  * Graff  _et al._ [2012] P. Graff, F. Feroz, M. P. Hobson, and A. Lasenby, BAMBI: blind accelerated multimodal Bayesian inference, MNRAS  421, 169 (2012), arXiv:1110.2997 [astro-ph.IM]
  * Rosofsky and Huerta [2020] S. G. Rosofsky and E. A. Huerta, Artificial neural network subgrid models of 2D compressible magnetohydrodynamic turbulence, Phys. Rev. D 101, 084024 (2020), arXiv:1912.11073 [physics.comp-ph]
  * Huerta and Zhao [2020] E. A. Huerta and Z. Zhao, Advances in machine and deep learning for modeling and real-time detection of multi-messenger sources, in  _Handbook of Gravitational Wave Astronomy_, edited by C. Bambi, S. Katsanevas, and K. D. Kokkotas (Springer Singapore, Singapore, 2020) pp. 1–27
  * Cuoco  _et al._ [2021] E. Cuoco, J. Powell, M. Cavaglià, K. Ackley, M. Bejger, C. Chatterjee, M. Coughlin, S. Coughlin, P. Easter, R. Essick, H. Gabbard, T. Gebhard, S. Ghosh, L. Haegel, A. Iess, D. Keitel, Z. Marka, S. Marka, F. Morawski, T. Nguyen, R. Ormiston, M. Puerrer, M. Razzano, K. Staats, G. Vajente, and D. Williams, Enhancing Gravitational-Wave Science with Machine Learning, Mach. Learn. Sci. Tech. 2, 011002 (2021)
  * Huerta  _et al._ [2021] E. A. Huerta, A. Khan, X. Huang, M. Tian, M. Levental, R. Chard, W. Wei, M. Heflin, D. S. Katz, V. Kindratenko, D. Mu, B. Blaiszik, and I. Foster, Accelerated, scalable and reproducible AI-driven gravitational wave detection, Nature Astronomy 5, 1062 (2021), arXiv:2012.08545 [gr-qc]
  * Huerta  _et al._ [2019] E. A. Huerta, G. Allen, I. Andreoni, J. M. Antelis, E. Bachelet, G. B. Berriman, F. B. Bianco, R. Biswas, M. Carrasco Kind, K. Chard, M. Cho, P. S. Cowperthwaite, Z. B. Etienne, M. Fishbach, F. Forster, D. George, T. Gibbs, M. Graham, W. Gropp, R. Gruendl, A. Gupta, R. Haas, S. Habib, E. Jennings, M. W. G. Johnson, E. Katsavounidis, D. S. Katz, A. Khan, V. Kindratenko, W. T. C. Kramer, X. Liu, A. Mahabal, Z. Marka, K. McHenry, J. M. Miller, C. Moreno, M. S. Neubauer, S. Oberlin, A. R. Olivas, D. Petravick, A. Rebei, S. Rosofsky, M. Ruiz, A. Saxton, B. F. Schutz, A. Schwing, E. Seidel, S. L. Shapiro, H. Shen, Y. Shen, L. P. Singer, B. M. Sipocz, L. Sun, J. Towns, A. Tsokaros, W. Wei, J. Wells, T. J. Williams, J. Xiong, and Z. Zhao, Enabling real-time multi-messenger astrophysics discoveries with deep learning, Nature Reviews Physics 1, 600 (2019), arXiv:1911.11779 [gr-qc]
  * Khan  _et al._ [2022] A. Khan, E. A. Huerta, and H. Zheng, Interpretable AI forecasting for numerical relativity waveforms of quasicircular, spinning, nonprecessing binary black hole mergers, Phys. Rev. D 105, 024024 (2022), arXiv:2110.06968 [gr-qc]
  * Chaturvedi  _et al._ [2022] P. Chaturvedi, A. Khan, M. Tian, E. A. Huerta, and H. Zheng, Inference-Optimized AI and High Performance Computing for Gravitational Wave Detection at Scale, Front. Artif. Intell. 5, 828672 (2022), arXiv:2201.11133 [gr-qc]
  * Wilkinson  _et al._ [2016] M. Wilkinson, M. Dumontier, I. J. Aalbersberg, G. Appleton, M. Axton, A. Baak, N. Blomberg, J.-W. Boiten, L. O. Bonino da Silva Santos, P. Bourne, J. Bouwman, A. Brookes, T. Clark, M. Crosas, I. Dillo, O. Dumon, S. Edmunds, C. Evelo, R. Finkers, and B. Mons, The fair guiding principles for scientific data management and stewardship, Scientific Data 3 (2016)
  * Chen  _et al._ [2022] Y. Chen, E. A. Huerta, J. Duarte, P. Harris, D. S. Katz, M. S. Neubauer, D. Diaz, F. Mokhtar, R. Kansal, S. Eon Park, V. V. Kindratenko, Z. Zhao, and R. Rusack, A FAIR and AI-ready Higgs boson decay dataset, Scientific Data 9, 31 (2022), arXiv:2108.02214 [hep-ex]
  * Li  _et al._ [2021] Z. Li, H. Zheng, N. Kovachki, D. Jin, H. Chen, B. Liu, K. Azizzadenesheli, and A. Anandkumar, Physics-Informed Neural Operator for Learning Partial Differential Equations, arXiv e-prints , arXiv:2111.03794 (2021), arXiv:2111.03794 [cs.LG]
  * Raissi  _et al._ [2017a] M. Raissi, P. Perdikaris, and G. E. Karniadakis, Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations (2017a), arXiv:1711.10561 [cs.AI]
  * Raissi  _et al._ [2017b] M. Raissi, P. Perdikaris, and G. E. Karniadakis, Physics informed deep learning (part ii): Data-driven discovery of nonlinear partial differential equations (2017b), arXiv:1711.10566 [cs.AI]
  * Raissi  _et al._ [2019] M. Raissi, P. Perdikaris, and G. Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, Journal of Computational Physics 378, 686 (2019)
  * Pang  _et al._ [2019] G. Pang, L. Lu, and G. E. Karniadakis, fpinns: Fractional physics-informed neural networks, SIAM Journal on Scientific Computing 41, A2603 (2019), https://doi.org/10.1137/18M1229845
  * Lu  _et al._ [2021a] L. Lu, X. Meng, Z. Mao, and G. E. Karniadakis, DeepXDE: A deep learning library for solving differential equations, SIAM Review 63, 208 (2021a)
  * Kovachki  _et al._ [2021] N. Kovachki, Z. Li, B. Liu, K. Azizzadenesheli, K. Bhattacharya, A. Stuart, and A. Anandkumar, Neural Operator: Learning Maps Between Function Spaces, arXiv e-prints , arXiv:2108.08481 (2021), arXiv:2108.08481 [cs.LG]
  * Lu  _et al._ [2021b] L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis, Learning nonlinear operators via deeponet based on the universal approximation theorem of operators, Nature Machine Intelligence 3, 218–229 (2021b)
  * Wang  _et al._ [2021] S. Wang, H. Wang, and P. Perdikaris, Learning the solution operator of parametric partial differential equations with physics-informed DeepONets, Science Advances 7, eabi8605 (2021)
  * Li  _et al._ [2020a] Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar, Neural operator: Graph kernel network for partial differential equations (2020a), arXiv:2003.03485 [cs.LG]
  * Li  _et al._ [2020b] Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. M. Stuart, and A. Anandkumar, Multipole graph neural operator for parametric partial differential equations, CoRR abs/2006.09535 (2020b), 2006.09535
  * Li  _et al._ [2020c] Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar, Fourier neural operator for parametric partial differential equations (2020c), arXiv:2010.08895 [cs.LG]
  * Paszke  _et al._ [2019] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, _et al._ , Pytorch: An imperative style, high-performance deep learning library, Advances in neural information processing systems 32 (2019)
  * Hendrycks and Gimpel [2016] D. Hendrycks and K. Gimpel, Gaussian error linear units (gelus) (2016), arXiv:1606.08415 [cs.LG]
  * Chard  _et al._ [2019] R. Chard, Z. Li, K. Chard, L. Ward, Y. Babuji, A. Woodard, S. Tuecke, B. Blaiszik, M. J. Franklin, and I. Foster, Dlhub: Model and data serving for science, 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS) 10.1109/ipdps.2019.00038 (2019)
  * Blaiszik  _et al._ [2019] B. Blaiszik, L. Ward, M. Schwarting, J. Gaff, R. Chard, D. Pike, K. Chard, and I. Foster, A data ecosystem to support machine learning in materials science, MRS Communications 9, 1125 (2019)
  * Ainsworth and Dong [2021] M. Ainsworth and J. Dong, Galerkin neural networks: A framework for approximating variational equations with error control, SIAM Journal on Scientific Computing 43, A2474 (2021)
  * Doshi-Velez and Kim [2017] F. Doshi-Velez and B. Kim, Towards A Rigorous Science of Interpretable Machine Learning, arXiv e-prints , arXiv:1702.08608 (2017), arXiv:1702.08608 [stat.ML]
  * Safarzadeh  _et al._ [2022] M. Safarzadeh, A. Khan, E. A. Huerta, and M. Wattenberg, Interpreting a Machine Learning Model for Detecting Gravitational Waves, arXiv e-prints , arXiv:2202.07399 (2022), arXiv:2202.07399 [gr-qc]
  * Carvalho  _et al._ [2019] D. V. Carvalho, E. M. Pereira, and J. S. Cardoso, Machine learning interpretability: A survey on methods and metrics, Electronics 8, 10.3390/electronics8080832 (2019)


# Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages

Yasmine Karoui$\mu$ Rémi Lebret$\lambda$ Negar Foroutan$\lambda$ Karl
Aberer$\lambda$
$\mu$Technical University of Munich, Germany
$\lambda$EPFL, Switzerland  Yasmine performed this work while interning at
EPFL.

###### Abstract

Vision-Language Pre-training (VLP) has advanced the performance of many
vision-language tasks, such as image-text retrieval, visual entailment, and
visual reasoning. The pre-training mostly utilizes lexical databases and image
queries in English. Previous work has demonstrated that the pre-training in
English does not transfer well to other languages in a zero-shot setting.
However, multilingual pre-trained language models (MPLM) have excelled at a
variety of single-modal language tasks. In this paper, we propose a simple yet
efficient approach to adapt VLP to unseen languages using MPLM. We utilize a
cross-lingual contextualized token embeddings alignment approach to train text
encoders for non-English languages. Our approach does not require image input
and primarily uses machine translation, eliminating the need for target
language data. Our evaluation across three distinct tasks (image-text
retrieval, visual entailment, and natural language visual reasoning)
demonstrates that this approach outperforms the state-of-the-art multilingual
vision-language models without requiring large parallel corpora. Our code is
available at https://github.com/Yasminekaroui/CliCoTea.

##  1 Introduction

Inspired by the recent advancements in language model pre-training, Vision-
Language Pre-trained Models (VLPMs) have demonstrated state-of-the-art
performance across a wide range of vision-language (VL) tasks such as text-to-
image retrieval, visual reasoning, visual entailment, and visual QA Chen et
al. (2020); Li et al. (2021, 2022).

(a) Cross-Lingual Text Alignment

(b) Zero-shot Transfer on VL downstream task

Figure 1: Overview of our approach. We adapt the text encoder of a monolingual
VL model to an unseen language (a). Then we use the adapted model for a VL
downstream task in a zero-shot setting (b).

However, extending VLPMs to multilingual scenarios is still challenging. On
one hand, the majority of these models are trained on monolingual (English)
corpora and thus cannot perform well for other languages. On the other hand,
the multilingual pre-trained language models Devlin et al. (2018); Conneau et
al. (2019) cannot handle vision data (e.g., images or videos) directly.

Lately, there have been attempts (M3P, nUNITER, UC2) to pivot on images or
English texts to align multilingual representations with vision features Chen
et al. (2020); Ni et al. (2021); Zhou et al. (2021). However, a recent
benchmark on multilingual multimodal pre-training (IGLUE) Bugliarello et al.
(2022) shows that although these models achieve promising zero-shot cross-
lingual transfer performance on some VL tasks, they still fall short in
comparison to the “translate-test” baseline (using an English-only VLPM on the
translations of the text examples).

A more recent work (CCLM) achieves promising performance on the IGLUE
benchmark by exploiting massive parallel text and image-text corpora to pre-
train a VL model Zeng et al. (2022). This approach is motivated by a key
observation that multilingual and multimodal pre-training essentially achieves
the same goal of aligning two different views of the same object into a common
semantic space. Although this framework performs well on the IGLUE benchmark,
it requires a large amount of parallel data. Its pre-training phase relies on
19M multilingual parallel sentence pairs extracted from WikiMatrix Schwenk et
al. (2021), jointly trained with 4 million image-text pairs in multiple
languages.

In this work, we are proposing a simple yet efficient way to adapt VLP models
to unseen languages without requiring large parallel corpora. We propose to
align a VLPM monolingual text encoder (achieving start-of-the-art performance
on English downstream VL tasks) with a multilingual pre-trained language model
(e.g., mBERT), using only small in-domain parallel text corpus. The recent
progress in Neural Machine Translation (NMT) has enabled us to create such a
parallel corpus from automatically translating the data from English to any
other language, even for low-resource languages (i.e., Swahili). However,
since our approach relies on token alignment, it is robust to errors made by
NMT. Our zero-shot evaluation across three of the four IGLUE tasks shows that
the proposed method achieves state-of-the-art results while using small set of
in-domain parallel sentences. The key steps of our approach are illustrated in
Figure 1.

##  2 CLiCoTEA : Cross-Lingual Contextualised Token Embedding Alignment

We propose CLiCoTEA , an approach to transfer a monolingual vision-language
(VL) pre-trained model in one language $L_{1}$ where there is an abundant
number of training pairs of image and text (i.e., English) to a second
language $L_{2}$. As we focus in this paper on the zero-shot setting, we do
the transfer after fine-tuning the pre-trained monolingual VL model on a
downstream task $t$, where training samples are available in language $L_{1}$.

CLiCoTEA consists of six steps:

  1. 1.

Pre-train a monolingual VL model on a massive collection of image-text pairs,
where text is written in language $L_{1}$.

  2. 2.

Fine-tune the VL pre-trained model on the downstream task $t$ in language
$L1$.

  3. 3.

Create a parallel text corpus by translating the training set from step 2 in
the target language $L_{2}$. Note that this step can be done automatically
using neural machine translation.

  4. 4.

Create a list of aligned tokens for each (potentially noisy) parallel sentence
using a token alignment model.

  5. 5.

Cross-lingual transfer by aligning contextualised token embeddings. As
illustrated in Figure 1a, it transfers the VL fine-tuned model to the new
language $L_{2}$ by aligning a pre-trained multilingual LM (e.g., mBERT or
XLM-R) with the text encoder of the VL pre-trained model using the list of
aligned tokens created in step 4.

  6. 6.

Zero-shot transfer to $L_{2}$ by swapping the monolingual text encoder from
the VL pre-trained model with the aligned multilingual text encoder learned in
step 5. An example of visual reasoning in Indonesian is illustrated in Figure
1b.

In practice, steps 1 and 2 are the most computationally expensive. Therefore,
we propose to adapt VL fine-tuned models to new languages by only doing the
steps from 3 to 5 which can be computed in a few hours on a single GPU.

We note that CLiCoTEA could be used with any multimodal pre-trained model
where one of the modalities is a monolingual text encoder. We focus in this
paper on VL models, but CLiCoTEA could be applied for instance to a language-
knowledge model such as GreaseLM Zhang et al. (2021) or DRAGON Yasunaga et al.
(2022).

##  3 Experiment

###  3.1 Pre-trained Models

#### Vision-Language Model

In step 1 of CLiCoTEA , we use the Align BEfore Fuse (ALBEF) framework111Code
and models are available at https://github.com/salesforce/ALBEF. Li et al.
(2021) as our Vision-Language Pre-trained Model (VLPM). ALBEF has been fine-
tuned on multiple downstream VL tasks and achieves state-of-the-art
performance. We use the ALBEF fine-tuned models in step 2 for the downstream
tasks described in Section 3.3. Unlike other competitive VL pre-trained models
(such as BLIP Li et al. (2022)) that inject visual information by inserting
cross-attention for each transformer block of the text encoder, ALBEF first
encodes the image and text independently with a detector-free image encoder
and a text encoder. Then it uses a multimodal encoder to fuse the image
features with the text features through cross-modal attention. All encoders
are based on transformer networks with the text encoder being a 6-layer
transformer initialised using the first 6 layers of the BERTbase. We thus
extract this 6-layer text encoder for cross-lingual transfer training in step
5.

#### Multilingual Language Model

As a multilingual pre-trained language model, we use the multilingual BERT
(mBERT)222Available on HuggingFace hub at https://huggingface.co/bert-base-
multilingual-cased. Devlin et al. (2018). It has been trained on the top 104
languages with the largest Wikipedia using a masked language modeling (MLM)
objective and has demonstrated remarkable zero-shot cross-lingual transfer
capabilities Wu and Dredze (2019); Pires et al. (2019); Hu et al. (2020);
Conneau et al. (2018). We extract the first 6-layer transformer to be aligned
with the text encoder of ALBEF in step 5.

###  3.2 Implementation Details

#### Word Alignment

Since the parallel sentences do not contain word-level alignment information,
in step 4 of CLiCoTEA we utilize awesome-
align333https://github.com/neulab/awesome-align Dou and Neubig (2021) which is
a tool that automatically extracts word alignments from mBERT. The generated
word pairs are then filtered for keeping only one-to-one, one-to-many or many-
to-one alignments and removing many-to-many alignments. This is done for all
languages except Chinese because otherwise less than 3% of the training data
would remain in the set. The advantage of this filtering is twofold: a) it
removes the noise from the matching word pairs; b) it reduces the training
time and computation. For words that are split into sub-word tokens, we
consider either the left-most token embedding alignment (i.e., the first sub-
word token of a word) or, the average embedding across all sub-word tokens.

#### Contextualised Token Alignment Training

Given a set of aligned contextual word pairs extracted from parallel
sentences, we define $\\{\mathrm{x}_{i},\mathrm{y}_{i}\\}^{n}_{i=1}$, where
$\mathrm{x}_{i}\in\mathbb{R}^{d}$ is the contextualised embedding of token $i$
in the target language (obtained from mBERT), and
$\mathrm{y}_{i}\in\mathbb{R}^{d}$ is the contextualised embedding of its
alignment in the source language (obtained from the fine-tuned ALBEF)444Note
that the special $\mathtt{[CLS]}$ token is always included.. In step 5 of
CLiCoTEA , we minimise the following training objective:
$\sum_{i=1}^{n}||\mathrm{x}_{i}-\mathrm{y}_{i}||^{2}$.

The parameters of the source language encoder are frozen, while the ones of
the target language encoder are fine-tuned at training time. The learning rate
is set to $5.10^{-5}$. The batch size is set to 128. These hyperparameters are
set through the NLVR2, Flickr30k, SNLI validation sets, for each task
respectively. For each target language, the training is done on a single
GeForce GTX TITAN X in a few hours.

#### Data Augmentation

As multilingual language models are generally pre-trained on the source
language $L_{1}$, the contextualised token alignment can be trained not only
with sentences from the target language $L_{2}$, but also with sentences from
the source language $L_{1}$. This strategy doubles the training size, and
consequently, the training time but it could be used with tasks where the
number of available training sentences is limited.

###  3.3 Downstream Tasks

In step 6, we evaluate CLiCoTEA on three tasks from the IGLUE benchmark555We
do not include the Cross-lingual Grounded Question Answering task (xGQA
Pfeiffer et al. (2021)) as it requires aligning the answer decoder too. We
leave it as future work. in the zero-shot setting:

  * •

xFlickr&CO: The dataset is composed of 1000 images from Flickr30K Plummer et
al. (2015) and 1000 images from MSCOCO dataset Lin et al. (2014). These images
come along with croudsourced image captions in 6 different languages.
xFlickr&CO is a _retrieval_ task dataset. It is composed of two subtasks:
image-to-text retrieval (TR) and text-to-image retrieval (IR).

  * •

XVNLI: The dataset consists in merging SNLI hypothesis with Flickr30K Plummer
et al. (2015) images and translate the test set in four languages. The task is
called _visual entailment_ (VE) which is a fine-grained reasoning task to
determine whether a text hypothesis “contradicts”, “entails”, or is “neutral”
with respect to an image.

  * •

MaRVL: The dataset is a multilingual expansion of NLVR2 dataset Suhr et al.
(2017), with images related to concepts of five languages and cultures. The
task is called _visual reasoning_ (VR) which consists in determining whether a
statement is correct given a pair of images.

Step | Retrieval | VE | VR
---|---|---|---
Fine-tuning | Flickr30K | SNLI | NLVR2
Alignment | Flickr30K∗ | SNLI∗ | NLVR2∗
Zero-shot Test | xFlickr&CO | XVNLI | MaRVL

Table 1: The datasets used in the different steps of CLiCoTEA . Translated
train and validation captions are denoted with $*$.

Table 1 shows the datasets used for a) fine-tuning the monolingual VL pre-
trained model in step 2, b) training the alignment of contextualised token
embeddings in step 5, and c) testing the zero-shot cross-lingual transfer in
step 6. For creating the parallel corpus in step 3, all datasets used for
fine-tuning the monolingual pre-trained VL model are translated to the
corresponding test dataset languages from the IGLUE benchmark using
GoogleTrans Python API666https://pypi.org/project/googletrans/. Statistics
about the translation datasets can be found in Section A.1. MaRVL being the
smallest dataset, the data augmentation strategy described in Section 3.2 is
applied only for this task. Detailed results on data augmentation can be found
in Section 3.2.

###  3.4 Experimental Results

Model | VE |  | VR |  | Retrieval
---|---|---|---|---|---
| XVNLI |  | MaRVL |  | xFlickr&CO
|  |  |  |  | IR | TR
mUNITER | 53.69 |  | 53.72 |  | 8.06 | 8.86
xUNITER | 58.48 |  | 54.59 |  | 14.04 | 13.51
UC2 | 62.05 |  | 57.28 |  | 20.31 | 17.89
M3P | 58.25 |  | 56.00 |  | 12.91 | 11.90
CCLM3M | 74.64 |  | 65.91 |  | 67.35 | 65.37
CCLM4M | 73.32 |  | 67.17 |  | 76.56 | 73.46
CLiCoTEA | 78.15 |  | 68.09 |  | 67.45 | 65.07

Table 2: Zero-shot performance on IGLUE benchmark. Recall@1 and Accuracy are
reported for retrieval tasks (xFlickr&CO) and understanding tasks (XVNLI,
MaRVL) respectively. Results of compared models are directly copied from Zeng
et al. (2022).

Results reported in Table 2 shows that CLiCoTEA outperforms the state-of-the-
art CCLM models for all downstream tasks except retrieval. The larger
improvement against CCLM models is obtained in visual entailment with an
increase of almost 5%. The superiority of CLiCoTEA is especially high for
Spanish (+7.68%), as can be seen from Table 10 in Section A.4. The average
performance on visual reasoning is similar to CCLM, but CLiCoTEA significantly
outperforms CCLM by $\pm 4\%$ on the low-resource languages such as Tamil and
Swahili (results per language can be seen in Table 8 in Section A.3). For
retrieval, CLiCoTEA outperforms all models except CCLM4M. It is worth
mentioning that, unlike the other models, CCLM4M has been pre-trained on COCO
which could explain its superiority on Flickr&CO dataset. More details about
the results on retrieval can be found in Section A.2.

##  4 Conclusion

In this paper, we present CLiCoTEA an approach for adapting Vision-Language
pre-trained models to unseen languages. Unlike other approaches that rely on
an expensive pre-training phase (both in terms of data and computation), our
approach adapts the contextualised token embeddings of a multilingual pre-
trained language model by aligning them with the contextualised token
embeddings of the VLPM text encoder. By aligning ALBEF text encoder with
mBERT, we show that CLiCoTEA outperforms CCLM, which exploits massive parallel
text and image-text corpora. CLiCoTEA achieves start-of-the-art performance on
visual entailment and visual reasoning, with an increase of almost 5% on
visual entailment. It also demonstrates its effectiveness, especially for low-
resource languages, as it does not require large corpora to do the adaptation.

##  5 Limitations

The general performance of CLiCoTEA could be improved with a better MPLM than
mBERT, such as XLM-R which has a larger token vocabulary and has been pre-
trained on a much larger dataset. Our approach is currently not applicable to
generation tasks where a multilingual text decoder is needed to generate text
in unseen languages. We leave this adaptation for future work. Unlike the
statement made in Zeng et al. (2022), current multilingual VL models still do
not surpass the _Translate-Test_ baseline of the tasks from IGLUE benchmark.
The performance of CLiCoTEA is promising but the best scores are still
obtained when translating everything to English and using the (English-only)
ALBEF model. The smallest difference in accuracy on MaRVL dataset between
CLiCoTEA and ALBEF with _Translate-Test_ is obtained in Swahili (-2%), while
the gap is much larger (around -6%) for the other languages. Outperforming the
_Translate-Test_ achieved by ALBEF still remains an open challenge, especially
for high-resource languages.

## References

  * Bugliarello et al. (2022) Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, and Ivan Vulić. 2022.  Iglue: A benchmark for transfer learning across modalities, tasks, and languages.  _arXiv preprint arXiv:2201.11732_.
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020.  Uniter: Universal image-text representation learning.  In _European conference on computer vision_ , pages 104–120. Springer.
  * Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Unsupervised cross-lingual representation learning at scale.  _arXiv preprint arXiv:1911.02116_.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  Xnli: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dou and Neubig (2021) Zi-Yi Dou and Graham Neubig. 2021.  Word alignment by fine-tuning embeddings on parallel corpora.  _arXiv preprint arXiv:2101.08231_.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In _International Conference on Machine Learning_ , pages 4411–4421. PMLR.
  * Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  _arXiv preprint arXiv:2201.12086_.
  * Li et al. (2021) Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021.  Align before fuse: Vision and language representation learning with momentum distillation.  _Advances in neural information processing systems_ , 34:9694–9705.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , pages 740–755. Springer.
  * Ni et al. (2021) Minheng Ni, Haoyang Huang, Lin Su, Edward Cui, Taroon Bharti, Lijuan Wang, Dongdong Zhang, and Nan Duan. 2021.  M3p: Learning universal representations via multitask multilingual multimodal pre-training.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3977–3986.
  * Pfeiffer et al. (2021) Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin O Steitz, Stefan Roth, Ivan Vulić, and Iryna Gurevych. 2021.  xgqa: Cross-lingual visual question answering.  _arXiv preprint arXiv:2109.06082_.
  * Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019.  How multilingual is multilingual bert?  _arXiv preprint arXiv:1906.01502_.
  * Plummer et al. (2015) Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2641–2649.
  * Schwenk et al. (2021) Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2021.  WikiMatrix: Mining 135M parallel sentences in 1620 language pairs from Wikipedia.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1351–1361, Online. Association for Computational Linguistics.
  * Suhr et al. (2017) Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. 2017.  A corpus of natural language for visual reasoning.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 217–223.
  * Wu and Dredze (2019) Shijie Wu and Mark Dredze. 2019.  Beto, bentz, becas: The surprising cross-lingual effectiveness of bert.  _arXiv preprint arXiv:1904.09077_.
  * Yasunaga et al. (2022) Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D Manning, Percy Liang, and Jure Leskovec. 2022.  Deep bidirectional language-knowledge graph pretraining.  _arXiv preprint arXiv:2210.09338_.
  * Zeng et al. (2022) Yan Zeng, Wangchunshu Zhou, Ao Luo, and Xinsong Zhang. 2022.  Cross-view language modeling: Towards unified cross-lingual cross-modal pre-training.  _arXiv preprint arXiv:2206.00621_.
  * Zhang et al. (2021) Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D Manning, and Jure Leskovec. 2021.  Greaselm: Graph reasoning enhanced language models.  In _International Conference on Learning Representations_.
  * Zhou et al. (2021) Mingyang Zhou, Luowei Zhou, Shuohang Wang, Yu Cheng, Linjie Li, Zhou Yu, and Jingjing Liu. 2021.  Uc2: Universal cross-lingual cross-modal vision-and-language pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4155–4165.

##  Appendix A Appendix

###  A.1 Details of Alignment Datasets

Tables 3, 4, and 5 show the average number of aligned tokens extracted from
the translated sentences of Flickr30k, SNLI, and NLVR2, respectively.

Language | Total number | Avg. number of
---|---|---
| of sentences | aligned tokens
German | 144935 | 8.74
Spanish | 144990 | 10.04
Indonesian | 144858 | 7.46
Russian | 144526 | 6.44
Turkish | 143664 | 4.83
Table 3: Statistics about Flickr30k translation set. Language | Total number | Avg. number of
---|---|---
| of sentences | aligned tokens
Arabic | 513683 | 2.95
Spanish | 549785 | 6.31
French | 549260 | 5.78
Russian | 524308 | 3.60
Table 4: Statistics about SNLI translation set. Language | Total number | Avg. number of
---|---|---
| of sentences | aligned tokens
Indonesian | 86325 | 8.27
Swahili | 85415 | 5.46
Tamil | 85241 | 4.53
Turkish | 85050 | 5.42
Chinese | 86373 | 10.76
Table 5: Statistics about NLVR2 translation set.

###  A.2 Results on Retrieval

Zero-shot performance on the Flickr&CO dataset, the image-text and text-image
retrieval tasks from the IGLUE benchmark, for four available languages (DE:
German, ES: Spanish, ID: Indonesian, RU: Russian, TR: Turkish) are reported in
Table 6 and Table 7, respectively. CLiCoTEA outperforms all models except
CCLM4M. Compared with CCLM3M, CCLM4M has been trained with 1M additional
image-text pairs from Visual Genome and COCO datasets. The gap in performance
between the two models on retrieval tasks suggests that pre-training with COCO
text-image pairs gives a clear advantage to CCLM4M as Flickr&CO contains 1000
images from COCO, while all other models have been fine-tuned only on
Flickr30K.

Model | Language
---|---
| DE | ES | ID | RU | TR
mUNITER | 12.05 | 13.15 | 5.95 | 5.85 | 1.75
xUNITER | 14.55 | 16.10 | 16.50 | 15.90 | 9.05
UC2 | 28.60 | 15.95 | 14.60 | 20.00 | 7.15
M3P | 13.35 | 13.40 | 13.20 | 15.95 | 7.75
CCLM3M | 67.67 | 71.23 | 62.38 | 72.83 | 55.15
CCLM4M | 73.65 | 79.62 | 69.50 | 80.65 | 65.08
CLiCoTEA | 61.48 | 74.50 | 64.98 | 73.50 | 62.80

Table 6: Zero-shot performance on multi-lingual image-text retrieval with
Flickr&CO dataset. Recall@1 is reported.

Model | Language
---|---
| DE | ES | ID | RU | TR
mUNITER | 11.85 | 13.05 | 7.55 | 6.80 | 3.25
xUNITER | 13.25 | 15.10 | 16.75 | 14.80 | 10.05
UC2 | 23.90 | 15.30 | 13.60 | 16.75 | 6.95
M3P | 11.85 | 12.15 | 12.10 | 14.45 | 8.35
CCLM3M | 66.88 | 68.58 | 60.33 | 69.90 | 54.22
CCLM4M | 73.60 | 78.38 | 67.67 | 80.35 | 63.22
CLiCoTEA | 70.34 | 71.42 | 57.77 | 69.80 | 56.00

Table 7: Zero-shot performance on multi-lingual text-image retrieval with
Flickr&CO dataset. Recall@1 is reported.

###  A.3 Results on Natural Language Visual Reasoning

Table 8 shows the zero-shot performance on the MaRVL dataset, and the natural
language visual reasoning task from the IGLUE benchmark, for all available
languages (ID: Indonesian, SW: Swahili, TA: Tamil, TR: Turkish, ZH: Chinese).

Model | Language
---|---
| ID | SW | TA | TR | ZH
mUNITER | 54.79 | 51.17 | 52.66 | 54.66 | 55.34
xUNITER | 55.14 | 55.51 | 53.06 | 56.19 | 53.06
UC2 | 56.74 | 52.62 | 60.47 | 56.70 | 59.88
M3P | 56.47 | 55.69 | 56.04 | 56.78 | 55.04
CCLM3M | 67.81 | 61.55 | 60.28 | 69.60 | 70.52
CCLM4M | 71.66 | 67.21 | 60.36 | 66.75 | 69.86
CLiCoTEA | 69.55 | 71.30 | 63.93 | 70.72 | 64.93

Table 8: Zero-shot performance on visual reasoning with MaRVL dataset.
Accuracy is reported.

As MaRVL is the smallest dataset among the three tasks from IGLUE, we apply
the data augmentation for training the alignment as described in Section 3.2.
Results reported in Table 9 show that there is drop of 3.35% for Turkish, and
9.99% for Chinese when training only using the target language $L_{2}$, while
there is no significant difference for the three other languages (Indonesian,
Swahili, and Tamil). As explained in Section 3.2, our noise filtering
technique does not work well with Chinese. Aligning the English sentences with
half of the original training set helped the model infer knowledge from
English and reduced the number of wrong matching words. For Turkish, the
increase in performance could be explained by the similarity between the two
alphabets.

Training Set | Language
---|---
| ID | SW | TA | TR | ZH
$L_{1}$ | 69.55 | 71.30 | 63.45 | 67.37 | 54.94
$L_{1}+L_{2}$ | 68.53 | 70.31 | 63.93 | 70.72 | 64.93

Table 9: Zero-shot performance of CLiCoTEA on visual reasoning with MaRVL
dataset using monolingual ($L_{1}$) or bilingual ($L_{1}+L_{2}$) alignment
training. Accuracy is reported.

###  A.4 Results on Visual Entailment

Zero-shot performance on the XVNLI dataset, the visual entailment task from
the IGLUE benchmark, for all available languages (AR: Arabic, ES: Spanish, FR:
French, RU: Russian) are reported in Table 10. CLiCoTEA outperforms other
models by a significant margin for all languages, except Russian where CCLM3M
achieves comparable performance.

Model | Language
---|---
| AR | ES | FR | RU
mUNITER | 46.73 | 56.96 | 59.36 | 51.72
xUNITER | 51.98 | 58.94 | 63.32 | 59.71
UC2 | 56.19 | 57.47 | 69.67 | 64.86
M3P | 55.24 | 58.85 | 56.36 | 62.54
CCLM3M | 71.04 | 75.80 | 78.14 | 73.56
CCLM4M | 69.68 | 73.65 | 77.54 | 72.40
CLiCoTEA | 75.83 | 83.48 | 80.17 | 73.13
Table 10: Zero-shot performance on visual entailment with XVNLI dataset.
Accuracy is reported.

###  A.5 In-domain vs Open-domain Data

Language | Total number | Accuracy
---|---|---
| of sentences | in %
Swahili | 50400 | 63.27
Turkish | 50418 | 66.61
Chinese | 51159 | 59.09
Table 11: Zero-shot performance on visual reasoning with MaRVL dataset.
Alignment is done with a subset from XNLI dataset.

In order to eliminate the need for machine translations from CLiCoTEA in step
3, we created a parallel text corpus with sentences obtained from XNLI Conneau
et al. (2018) which is publicly available and covers 15 languages. A subset of
XNLI has been used for training the alignment by considering only the
sentences that were semantically close to the captions in NLVR2. To do so, we
used the Sentence-Transformers framework777Available at https://www.sbert.net.
to compute sentence embeddings similarities between NLVR2 captions and XNLI
English sentences and kept only the sentences with a cosine similarity higher
than 0.5. About 50k English sentences from XNLI are semantically close to
NLVR2 captions, we thus selected their parallel sentences in Swahili, Turkish
and Chinese to perform an evaluation on MaRVL dataset. After the
contextualised token alignment training on XNLI-based datasets, our results in
Table 11 suggest that a multilingual open-domain dataset gives better results
than mUNITER and xUNITER but underperforms the results obtained by translating
in-domain training sets. This could be explained by the fact that although
these datasets are multilingual, the sentences are not semantically close
enough to NLVR2 captions.