# Machine Learning for Computational Science and Engineering – a brief
introduction and some critical questions

Chennakesava Kadapa,
University of Bolton, Bolton BL3 5AB, United Kingdom.
Email<EMAIL_ADDRESS>
Version 1.0.0

###### Abstract

Artificial Intelligence (AI) is now entering every sub-field of science,
technology, engineering, arts, and management. Thanks to the hype and
availability of research funds, it is being adapted in many fields without
much thought. Computational Science and Engineering (CS&E) is one such sub-
field. By highlighting some critical questions around the issues and
challenges in adapting Machine Learning (ML) for CS&E, most of which are often
overlooked in journal papers, this contribution hopes to offer some insights
into the adaptation of ML for applications in CS&E and related fields. This is
a general-purpose article written for a general audience and researchers new
to the fields of ML and/or CS&E. This work focuses only on the forward
problems in computational science and engineering. Some basic equations and
MATLAB code are also provided to help the reader understand the basics.

##  1 Introduction

Artificial Intelligence (AI) is now everywhere; at least, that is how it
appears to be. Every sector, every field uses AI in some or other form; at
least, that is what they claim. AI is making its strides in every area, and
Engineering is no exception. While AI has proven its worth for many
applications, for example, pattern recognition, natural language processing,
etc., its validity and worth in other fields, especially computational science
and engineering (CS&E), is yet to be established. This is not to say that AI
is useless in computational science and engineering but to scrutinise its
current use and highlight the issues and challenges.

When it comes to the use of AI in CS&E, Machine Learning, particularly the
algorithms based on artificial neural networks (ANN), matter the most. While
it is nice to see fancy colourful contours and animations of results, the
reality is not as colourful as it appears to be when it comes to the
adaptation of ML for CS&E. Several questions arise – some are open while many
are ignored. Looking at the top-level of adaption of ML for CS&E, two
questions are crucial.

  * •

Whether or not it is worth using ML in CS&E?

  * •

What are the requirements, bottlenecks, challenges and harms?

To be able to answer these two questions, or at least make a correct step in
the right direction in this endeavour, one needs to understand, first of all,
how CS&E and ML work under the hood.

###  1.1 Computational Science and Engineering (CS&E)

The forward problem in CS&E involves (numerically) solving the partial
differential equation (PDE) that govern the physics of the problem, e.g. solid
mechanics, fluid mechanics, electromagnetism etc., for a given (i) geometry,
(ii) material properties, (iii) constitutive laws and (iv) boundary and
initial conditions. The solution is often obtained using a numerical
technique, for example, finite difference method (FDM), finite volume method
(FVM) or finite element method (FEM) etc. [1, 2], implemented in the software
of user’s choice.

For the purpose of discussion, let us consider Poisson’s equation in one
dimension, which is one of the simplest differential equations, and it has
numerous applications in heat transfer, electrostatics, diffusion, and solid
mechanics. The one-dimensional Poisson equation for the variable of interest
$y$ is given as

$\displaystyle-\,\frac{d^{2}y}{dx^{2}}=g,$ (1)

with the source term $g$ in the domain $x\in\left[x_{0},x_{1}\right]$, and
boundary conditions,

$\displaystyle y(x=x_{0})$ $\displaystyle=y_{0},\mathrm{and}$ (2)
$\displaystyle y(x=x_{1})$ $\displaystyle=y_{1}.$ (3)

Solutions of the one-dimensional Poisson equation (1) for four different
combinations of the source term $g$, and boundary conditions $y_{0}$ and
$y_{1}$, are presented in Figure 1. It is apparent that the solution is
different for different values of the source term and boundary conditions.
Therefore, even if we ignore the particular discretisation method used, or for
that matter, how the software provides a solution for the given problem, we
cannot and should not overlook that the solution of the PDE heavily depends on
all the four inputs mentioned above. For the same PDE, the solution changes
with a change in either of the four inputs: geometry, material properties,
constitutive laws and boundary conditions. This necessitates that we must
solve the PDE from scratch to account for the changes in either one or all
four inputs of the problem, particularly so for any significant changes in
shape and dimensions of the part and boundary conditions.

Figure 1: Solutions of the one-dimensional Poisson equation for a fixed domain
$x\in\left[0,1\right]$ for four different combinations of the source term $g$,
and boundary conditions $y_{0}$ and $y_{1}$.

###  1.2 Machine learning (ML)

ML works completely different to CS&E. There is input data and output data,
and the ML algorithm finds the functional relation between the inputs and
outputs. This functional relationship is found by _fitting_ the parameters
using the _least-squares method_. That is, ML based on ANN is nothing but
(least-squares) fitting of input data in a hyper-dimensional space; at least,
this is what is done in the usage of ML for CS&E. There is only input and
output data but no information about the underlying physics. In AI
terminology, this type of learning is called _supervised learning_ , as the
network is being _supervised_ using error measures based on the expected
outputs during the training process. This can be understood relatively easily
using _linear regression_ model [3].

Linear regression is an approach for modelling the relationship between the
output and input, also known as the dependent and independent variables. A
typical example in linear regression is shown in Figure 2. The objective is to
find the slope ($w$) and intercept ($b$) in the linear relation $y=wx+b$. For
the given set of data points, $w$ and $b$ are calculated based on some error
measure, e.g. mean-squared error. This method is called the least-squares
fitting. The corresponding equations and the associated MATLAB code for the
least-squares fitting method are given in Appendix A.

Training of artificial neural networks works very similarly to least-squares
fitting. A representation of the linear regression model as a neural network
model is shown in Figure 3. In ML terminology [4], the parameters $w$ and $b$
are called as weight and bias, respectively. The parameters $w$ and $b$ are
calculated by minimising the _loss function_ using a numerical algorithm. This
process of finding the weights and bias of a neural network model is called
_training_. It is only when a NN model is trained and tested thoroughly, it
can be deployed for _predictions_. Due to its applicability to complex, large-
scale neural network models, the _backpropagation algorithm_ [4] is the widely
used algorithm for calculating the weights and bias for typical applications
in CS&E. For the interested readers, the corresponding equations and the
associated MATLAB code for the backpropagation algorithm used to train the
neural network are given in Appendix B. For the comprehensive details, refer
to Hagan et al. [4] and Demuth et al. [5]. PyTorch [9], a Python-based
software framework for machine learning, is an alternative to MATLAB’s deep
learning toolbox.

An extension of the neural network model to a more complex problem that
consists of multiple inputs and multiple outputs is shown in Figure 4. The
greater the number of inputs and outputs, the higher the number of hidden
layers and the number of neurons in each hidden layer, which increases the
number of unknown weight and bias parameters. On top of this, there is an
additional requirement of choosing the excitation function for each (hidden)
layer.

Figure 2: A typical example in linear regression. Here, $x$ is the input, $y$
is the output, $w$ is the slope, and $b$ is the intercept. Figure 3: Linear
regression model represented as a single-input single-output (SISO) neural
network model. Here, $x$ is the input, $y$ is the output, $w$ is the weight,
$b$ is the bias and $f$ is the transfer (or excitation) function. Figure 4:
Multi-input multi-output (MIMO) neural network model.

Therefore, ML based on ANN works completely different to the philosophy of
CS&E. A conventional ML algorithm knows nothing about the problem - neither
the PDE nor any of the four inputs. It knows only about the input data and
associated output data, popularly called the _ground truth_. It must be
emphasised here that _ground truth_ is not the exact solution; it is only a
reference solution, often obtained using high-fidelity simulations. The
recently introduced version of ANNs, namely Physics informed neural networks
(PINN) [6], or other variants of NN that claim to have accounted for physics,
incorporate the PDE into the ML algorithm; this is done by replacing the
conventional _loss function_ with the discrete version of the strong form of
the associated PDE. Nevertheless, whether conventional NN or PINN, the
fundamental idea is the same; the network gets trained based on the labelled
data generated for a particular problem defined by the geometry, material
properties and boundary conditions. PINN uses an improved loss function;
however, it does not change, in any way, how ML works at the fundamental
level. Besides, due to the inclusion of differential operators, one needs to
use automatic differentiation techniques, which introduce additional numerical
errors into the model, on top of the numerical errors coming from the
simulation data and algorithms used for _training_.

Once the network is trained sufficiently accurately, which requires a
substantial amount of resources and significant effort from the user in
collecting/preparing the data and going through the iterative process of
training the NN model, the solution can be _predicted_ for _a valid point_ in
the space of input parameters. It must be emphasised here that, although the
trained network may compute the solution outside the physical domain by way of
_extrapolation_ , such a solution outside the physical domain is meaningless.
The only meaningful extrapolation would be for material and geometric
properties beyond the range of the trained input data.

An ML model does not know anything about the PDE, geometry, boundary
conditions or constitutive laws. To get an accurate prediction, a significant
amount of data is required. The amount of data required for training is
directly proportional to the number of input and output parameters and,
therefore, the complexity of the associated neural network model required for
that particular problem. You might be wondering, at this point is, _”how will
this data be generated?”_. Well, there is a magic tree on which this data
grows. But, _”how can one pick this data from the magic tree?”_ is likely to
be your next question. The answer is, by running hundreds, or perhaps of
thousands or even more, of high-fidelity simulations. Remember, ML is data-
hungry. This means that one must first carry out the high-fidelity
simulations, thousands of them, to train a NN model.

##  2 Key issues and questions

It is not a question of whether ML works for applications in CS&E; it does
since it is all about _fitting_ the parameters. But, the main challenge for ML
in CS&E is not the ML technology itself but the data, particularly the process
of generation of data that is used for training and testing of the NN model,
and the computational resources required to accomplish this. Due to the user
effort required to prepare hundreds of input files for different parameters,
3D problems compound the difficulties associated with data generation. We all
can agree that most real-world engineering applications require solutions to
3D problems.

Moreover, even for the same PDE, the nitty-gritty details of a NN model: the
number of layers, the number of neurons in each layer, the excitation
function, the number of epochs etc., required to successfully train a NN
within an acceptable level of accuracy vary from one problem to the other.
This means that every major component that differs in geometry and shape
requires a different NN model; when the geometry is changed significantly, a
different NN model needs to be trained; when the initial and boundary
conditions are changed considerably, the NN model needs to be re-trained, in
the best-case scenario.

Therefore, it is not sufficient to show that the methodology works for simple
1D and/or 2D problems because it is established knowledge that one can _fit_
the curves to the data by tweaking functions and parameters. So, the
conventional approach of demonstration for toy problems is useless when it
comes to machine learning. The benefits of Machine Learning to CS&E, if any,
can only be understood from comprehensive studies of large-scale industrial 3D
cases. Journal papers focused on 1D and 2D problems, for example, 1D
advection-diffusion, 1D elastic wave propagation, 2D Navier-Stokes in square
domains, are practically useless since a new NN model must be trained for each
problem, irrespective of the dimension.

ML for applications in CS&E may make sense only for problems that lack a
functional relationship between output and input data, and such problems are
not likely to be the ones defined by PDEs. For those problems where ML might
be appropriate, due to the data-hungry nature of ML, its benefits over
conventional methods are of primary concern.

An important aspect that is often overlooked is the cost of generation and
preparation of data and training. Most of the papers on ML for CS&E present
only equations and colour contour plots comparing the predicted solution with
some reference solution (aka ground truth). There are several crucial details
of the ML models that these papers omit, for example, number of layers, number
of neurons, excitation function, tolerances, computing resources, etc. This
omission is believed to be due to some obvious reasons. However, these details
are crucial for any user training a NN model. Some important questions on this
aspect are:

  1. Q1)

What are the computational resources used?

  2. Q2)

What is the cost of generating the input data?

  3. Q3)

What is the cost of training a neural network?

  4. Q4)

What are the number of layers, number of neurons in each layer, and the
excitation function for each layer, and how to choose them?

  5. Q5)

How to avoid over-fitting and under-fitting?

  6. Q6)

What are the tolerances and learning rate used and how they affect the
training, for example, number of epochs?

  7. Q7)

How much data is necessary to obtain a reasonably accurate solution to a
problem?

The above-listed questions are not the ones that are actually concerned with
the fundamental aspects of the core problems in CS&E. Nevertheless, their
importance concerns the development of a good NN model. This is what makes ML
a challenging technology to make any meaningful impact in CS&E. All of these
questions are concerned with how ML operates and are problem-specific even in
a given field. Moreover, questions particularly concerning data - its
collection and preparation, and training of NN model, are crucial to the
success of any project on the use of ML in CS&E. Without discussing the
details of resources required for collecting/preparing data and training the
NN model, which is substantial for 3D problems, the adaption of ML for
practical engineering problems is going to be more of a pipe dream than
reality. One or two start-ups and established firms might have done this, but
at what cost?

Moreover, concerned with the prediction of a trained NN model, the following
three additional questions arise:

  1. Q8)

How accurate is the output of the trained network beyond the range of material
parameters considered in the input?

  2. Q9)

How does the trained network generalise to other problems, e.g. changes in
geometry, topology, discretisation, initial conditions, etc.?

  3. Q10)

What are the margins of error and the respective tolerances for input
parameters? For example, how much can the input parameters vary for a given
error margin?

Most of the published papers on the use of ML for CS&E do not discuss anything
on questions Q8, Q9 and Q10. But these are essential if such NN models were to
be deployed as surrogate models in digital twin technologies.

Furthermore, it is not fair to compare the prediction time of a trained NN
model against that of high-fidelity simulations. Since the data for training
and testing of NN models must be generated by running high-fidelity
simulations of hundreds or thousands of cases, which requires a substantial
amount of computational resources, this cost cannot be ignored. Therefore, the
total time ($T_{t}$) for predicting the end result using a NN model is the sum
of time taken for data generation ($T_{dg}$), time taken for network training
($T_{nt}$) and time taken for predictions ($T_{pr}$). That is,

$\displaystyle T_{t}=T_{dg}+T_{nt}+N\,T_{pr}.$ (4)

with $N$ as the number of predictions in the deployment stage.

Taking all the costs into consideration, the use of ML for CS&E would prove
worthwhile only when the number of predictions in the deployment stage is so
substantial that the time for predictions overshadows the cost of data
generation and training beyond a break-even point.

##  3 Summary and conclusion

This contribution provides a brief introduction to the application of Machine
Learning in computational science and engineering. A set of ten questions,
which are often overlooked in journal papers on ML for CS&E yet essential to
practitioners, are highlighted in the hope that they help the novice users and
practitioners to better their understanding of the adaptation of ML for CS&E.

A lack of an honest discussion around these questions is to be treated as a
cause for concern because the points highlighted are important for the end-
users gives the non-experts and researchers from other fields a false
impression of the potential of ML for CS&E. This can be particularly dangerous
when ML is adapted for applications in fields such as biomedical and nuclear
engineering. Can doctors use ML for cardiovascular flow simulations? Since
cardiovascular simulations are patient-specific, is it possible to obtain a
sufficient number of data samples to train a NN model to compute results that
the doctors can trust? Can we use ML for applications in nuclear engineering,
where the price we must pay for a miscalculation is quite heavy? These are
still open questions.

Without answering all these questions and addressing the associated issues,
the adaptation of ML for real-world applications of computational science and
engineering might prove to be a costly and risky affair. The onus is on the
CS&E community, particularly those adopting ML for CS&E. Academia must follow
ethical engineering practices and facilitate honest discussions that can help
solve our problems rather than innovating for the namesake of fantasising and
sensationalising technologies, which is counter-productive.

##  Appendix Appendix A Linear regression and least-squares fitting

The functional relation in linear regression is model given by

$\displaystyle y=wx+b.$ (A.1)

If there are $N$ data points $(x_{1},y_{1})$, $(x_{1},y_{1})$, $\ldots$,
$(x_{N},y_{N})$, we can write equation (A.1) in the matrix-vector format as

$\mathbf{y}\;=\;\begin{bmatrix}y_{1}\\\ y_{2}\\\ \vdots\\\
y_{N}\end{bmatrix}\;=\;\begin{bmatrix}x_{1}&1\\\ x_{2}&1\\\ \vdots&\vdots\\\
x_{N}&1\end{bmatrix}\;\begin{bmatrix}w\\\
b\end{bmatrix}=\;\mathbf{X}\,\bm{\beta}\quad\mathrm{or}\quad\mathbf{y}\;=\;\begin{bmatrix}x_{1}\\\
x_{2}\\\ \vdots\\\ x_{N}\end{bmatrix}w+\begin{bmatrix}1\\\ 1\\\ \vdots\\\
1\end{bmatrix}b=\mathbf{x}\,w+\mathbf{1}\,b,$ (A.2)

from which the error vector becomes,

$\displaystyle\mathbf{e}=\mathbf{y}-\mathbf{X}\,\bm{\beta}$ (A.3)

The unknowns $w$ and $b$ are computed by minimising the error (or loss)
function. Sum of squared errors is the widely used loss function, and it is
given as

$L=\mathbf{\|y}-\mathbf{X}\,\bm{\beta}\|^{2}=\left[\mathbf{y}-\mathbf{X}\,\bm{\beta}\right]^{\;\mathrm{T}}\left[\mathbf{y}-\mathbf{X}\,\bm{\beta}\right]=\mathbf{e}^{\;\mathrm{T}}\,\mathbf{e}.$
(A.4)

For $\displaystyle L$ to be minimum with respect to $\displaystyle\bm{\beta}$,
its derivative should vanish, i.e.,

$\frac{\partial L}{\partial\bm{\beta}}=0,$ (A.5)

which leads to the equation for unknowns ($\bm{\beta}$) as

$\bm{\beta}=\mathbf{X}^{+}\,\mathbf{y},\quad\mathrm{with}\quad\mathbf{X}^{+}=\left(\mathbf{X}^{\;\mathrm{T}}\,\mathbf{X}\right)^{-1}\mathbf{X}^{\;\mathrm{T}},$
(A.6)

where $\mathbf{X}^{+}$ is the Moore-Penrose inverse, also called the
pseudoinverse.

In the literature, this procedure is called _the least-squares fitting_. The
MATLAB code for calculating the parameters using the least-squares fitting is
given in code block LABEL:listing-regression.

⬇

clear all; clc;

N = 100; % number of points

w = 2.0; b = -4.0;

x_true = linspace(-4.0,4.0,N);

y_true = w*x_true + b;

% generate data

rng(1,'philox'); % set the seed of random number generator

x = -4.0 + (4.0-(-4.0))*rand(N,1);

e = -2.0 + (2.0-(-2.0))*rand(N,1); % noise

y = w*x + b + e;

X = [x ones(N,1)]; % matrix X

params = pinv(X)*y; % solver for parameters

w_hat = params(1)

b_hat = params(2)

% Plotting

plot(x, y, 'ko');

hold on;

plot(x_true, y_true, 'r', linewidth=2);

hold on;

y_fit = w_hat*x_true + b_hat;

plot(x_true, y_fit, 'k', linewidth=2);

legend('Data', 'True model', 'Fitted model', 'Location', 'southeast');

Listing 1: MATLAB code using the least-squares fitting method.

##  Appendix Appendix B Artificial Neural Networks and backpropagation
algorithm

The functional relation between the output and input in an artificial neural
network model shown in Figure 3 is given by

$\displaystyle y=f(n),\;\mathrm{with}\;n=w\,x+b,$ (B.1)

where $n$ is the output of the neuron and $f$ is the transfer (or excitation)
function.

Accordingly, the data points can be written in the matrix-vector form as

$\displaystyle\mathbf{y}=f(\mathbf{X}\,\bm{\beta}),$ (B.2)

for which the error vector becomes,

$\displaystyle\mathbf{e}=\mathbf{y}-f(\mathbf{X}\,\bm{\beta}).$ (B.3)

The loss function is assumed to be the sum of squared errors. That is,

$L=\mathbf{\|y}-f(\mathbf{X}\,\bm{\beta})\|^{2}=\left[\mathbf{y}-f(\mathbf{X}\,\bm{\beta})\right]^{\;\mathrm{T}}\left[\mathbf{y}-f(\mathbf{X}\,\bm{\beta})\right]=\mathbf{e}^{\;\mathrm{T}}\,\mathbf{e}.$
(B.4)

The parameters $w$ and $b$ are determined by minimising the loss function,
similar to the least-squares method. However, instead of solving directly as
in the least-squares method, they are calculated using the backpropagation
algorithm [4] in which the parameters are updated iteratively until the chosen
convergence criterion is satisfied. In the backpropagation algorithm, the
error computed at the end of the feed-forward (or forward propagation) step is
_propagated_ backwards to update the weights and bias; hence, the name. Since
the weights are _updated_ iteratively, the algorithm starts with an initial
guess for each parameter.

Different methods, for example, steepest descent, Newton’s, Gauss-Newton’s and
Levenberg-Marquardt, are used to update the weights [4, 7, 8]. The Levenberg-
Marquardt method is commonly used for training small- and medium-sized
networks, but it is mathematically more complicated compared to the rest. For
demonstration, the parameters are updated using the steepest descent
algorithm.

Following the steepest descent algorithm, we get

$\displaystyle w_{(k+1)}=w_{(k)}\ -\alpha\ \frac{\partial L}{\partial w},$
(B.5)

$\displaystyle b_{(k+1)}=b_{(k)}\ -\alpha\ \frac{\partial L}{\partial b},$
(B.6)

where $\displaystyle\alpha$ is the training rate and $\displaystyle k$ is the
iteration counter. From the loss function, we get

$\displaystyle\frac{\partial L}{\partial
w}=2\,\mathbf{e}^{\;\mathrm{T}}\,\frac{\partial\mathbf{e}}{\partial
w}=-\,2\,\mathbf{e}^{\;\mathrm{T}}\,\frac{\partial f}{\partial
n}\,\frac{\partial(\mathbf{X}\,\bm{\beta})}{\partial
w}=-\,2\,\mathbf{e}^{\;\mathrm{T}}\,\frac{\partial f}{\partial
n}\,\mathbf{x},$ (B.7)

$\displaystyle\frac{\partial L}{\partial
b}=2\,\mathbf{e}^{\;\mathrm{T}}\,\frac{\partial\mathbf{e}}{\partial
b}=-\,2\,\mathbf{e}^{\;\mathrm{T}}\,\frac{\partial f}{\partial
n}\,\frac{\partial(\mathbf{X}\,\bm{\beta})}{\partial
b}=-\,2\,\mathbf{e}^{\;\mathrm{T}}\,\frac{\partial f}{\partial
n}\,\mathbf{1}.$ (B.8)

Several different transfer functions are available that are appropriate to the
application at hand [4]. If the transfer function is assumed to be a purely
linear transfer function, we have

$\displaystyle f(n)=n,\quad\mathrm{and}\quad\frac{\partial f}{\partial n}=1.$
(B.9)

The backpropagation algorithm is sensitive not only to the initial guesses of
the weight and bias parameters but also to the learning rate parameter
$\alpha$. For guaranteed convergence, the learning rate ($\alpha$ should be
small enough; however, for smaller values of $alpha$, the convergence of
iterations is slower. The interested user can experiment with different values
of $alpha$ and initial guess for $w$ and $b$ using the MATLAB code for the
backpropagation given in code block LABEL:listing-backprop.

The backpropagation algorithm, in fact, the entire training process, is
available as a few functions in MATLAB’s Deep Learning Toolbox or any other
sophisticated software libraries such as TensorFlow. A sample MATLAB code
using MATLAB’s Deep learning toolbox for the problem of linear regression is
presented in code block LABEL:listing-DLT.

⬇

ar all; clc;

N = 100; % number of points

w = 2.0; b = -4.0;

x_true = linspace(-4.0,4.0,N);

y_true = w*x_true + b;

% generate data

rng(1,'philox'); % set the seed of random number generator

x = -4.0 + (4.0-(-4.0))*rand(N,1);

e = -2.0 + (2.0-(-2.0))*rand(N,1); % noise

y = w*x + b + e;

w_hat = 1.0; b_hat = -1.0; % initial guess

alpha = 0.001; % learning rate

loss_fun = zeros(1,N);

loss = 1.0e-10;

for iter=1:N

err = y - (w_hat*x + b_hat);

lossPrev = loss;

loss = sum(err.^2);

loss_fun(iter) = loss;

if( abs(loss-lossPrev) < 1.0e-6)

break;

end

w_hat = w_hat + alpha*2.0*err'*x;

b_hat = b_hat + alpha*2.0*sum(err);

end

plot(x, y, 'ko'); hold on;

plot(x_true, y_true, 'r', linewidth=2); hold on;

y_fit = w_hat*x_true + b;

plot(x_true, y_fit, 'k', linewidth=2);

legend('Data', 'True model', 'Fitted model', 'Location', 'southeast');

figure(); plot(log10(loss_fun))

Listing 2: MATLAB code using backpropagation algorithm.

⬇

ar all; clc;

N = 100; % number of points

w = 2.0; b = -4.0;

x_true = linspace(-4.0,4.0,N);

y_true = w*x_true + b;

% generate data

rng(1,'philox'); % set the seed of random number generator

x = -4.0 + (4.0-(-4.0))*rand(N,1);

e = -2.0 + (2.0-(-2.0))*rand(N,1); % noise

y = w*x + b + e;

% Set the neural network model

hiddenLayerSize = 1; % One neuron in the hidden layer

trainFcn = 'trainlm'; % Levenberg-Marquardt algorithm

net = fitnet(hiddenLayerSize, trainFcn); % Fitting neural network

net.layers{1}.transferFcn = 'purelin'; % Pure linear transfer function

% Division of data for Training, Validation, Testing

net.divideParam.trainRatio = 80/100; % 80% for training

net.divideParam.valRatio = 10/100; % 10% for validation

net.divideParam.testRatio = 10/100; % 10% for testing

net = train(net, x', y'); % Train the network

y_fit = net(x_true); % Test the network

% Plotting

plot(x, y, 'ko');

hold on;

plot(x_true, y_true, 'r', linewidth=2);

hold on;

plot(x_true, y_fit, 'k', linewidth=2);

legend('Data', 'True model', 'Fitted model', 'Location', 'southeast');

Listing 3: MATLAB code using MATLAB’s Deep Learning Toolbox.

## References

  * [1] B. Radi, A. El Hami.  Advanced Numerical Methods with Matlab® 2: Resolution of Nonlinear, Differential and Partial Differential Equations.  Volume 7, ISTE Ltd and John Wiley & Sons, Inc, 2018.
  * [2] G. F. Pinder.  Numerical Methods for Solving Partial Differential Equations: A Comprehensive Introduction for Scientists and Engineers. John Wiley and Sons, Inc, USA, 2018.
  * [3] G. A. Seber, and A. J. Lee.  Linear Regression Analysis.  2nd Edition, John Wiley and Sons, Inc, 2012.
  * [4] M. T. Hagan, H. B. Demuth, M. H. Beale, O. De Jesús.  Neural Network Design.  2nd Edition, https://hagan.okstate.edu/nnd.html.
  * [5] H. B. Demuth, M. H. Beale, and M. T. Hagan.  Neural Network Toolbox 5. User’s Guide, Version 5, 2007.
  * [6] M. Raissi, P. Perdikaris, and G. E. Karniadakis.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  Journal of Computational Physics, 378:686-707, 2019.
  * [7] J. Nocedal, and S. J. Wright.  Numerical Optimization. Springer-Verlag, 2006.
  * [8] A. Björck.  Numerical methods for least squares problems. SIAM, Philadelphia, 1996.
  * [9] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M.. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai and S. Chintala.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  Advances in Neural Information Processing Systems 32, Edited by H. Wallach and H. Larochelle and A. Beygelzimer and F. d'Alché-Buc and E. Fox and R. Garnett, Curran Associates, Inc., 8024-8035, 2019.


# Training Transitive and Commutative Multimodal Transformers with LoReTTa

Manuel Tran1,2,3 &Amal Lahiani1 &Yashin Dicente Cid1 &Fabian J. Theis2,3
Tingying Peng3,† &Eldad Klaiman1,† 1Roche Diagnostics
2Technical University Munich
3Helmholtz Munich

###### Abstract

Collecting a multimodal dataset with two paired modalities $A$ and $B$ or $B$
and $C$ is difficult in practice. Obtaining a dataset with three aligned
modalities $A$, $B$, and $C$ is even more challenging. For example, some
public medical datasets have only genetic sequences and microscopic images for
one patient, and only genetic sequences and radiological images for another –
but no dataset includes both microscopic and radiological images for the same
patient. This makes it difficult to integrate and combine all modalities into
a large pre-trained neural network. We introduce LoReTTa (Linking mOdalities
with a tRansitive and commutativE pre-Training sTrAtegy) to address this
understudied problem. Our self-supervised framework combines causal masked
modeling with the rules of commutativity and transitivity to transition within
and between different modalities. Thus, it can model the relation
$A\rightarrow C$ with $A\rightarrow B\rightarrow C$. Given a dataset
containing only the disjoint combinations $(A,B)$ and $(B,C)$, we show that a
transformer pre-trained with LoReTTa can handle any modality combination at
inference time, including the never-seen pair $(A,C)$ and the triplet
$(A,B,C)$. We evaluate our approach on a multimodal dataset derived from MNIST
containing speech, vision, and language, as well as a real-world medical
dataset containing mRNA, miRNA, and RPPA samples from TCGA. Compared to
traditional pre-training methods, we observe up to a 100-point reduction in
perplexity for autoregressive generation tasks and up to a 15% improvement in
classification accuracy for previously unseen modality pairs during the pre-
training phase.

##  1 Introduction

Integrating multiple modalities, such as images, text, or speech, promises to
provide a more comprehensive and holistic understanding of otherwise
intractable problems. By leveraging the unique and complementary strengths of
each data type, this approach results in a more synergistic representation of
the underlying phenomena – enabling deep learning systems to excel and adapt
to real-world scenarios. Thus, there has been impressive progress in the field
of multimodal learning in recent years (Section 2). While most of the research
focuses on multimodal datasets where all data points have all modalities,
there is also work on missing modalities [49, 50]. However, all of these
models and methods rely heavily on at least a subset of samples where all
modality combinations are still present (Figure 1). In practice, it is not
easy to obtain such datasets if we consider more than two modalities, for
example, three modalities. In the worst case, we might end up with one dataset
containing only modalities $A$ and $B$, and another disjoint dataset
containing modalities $B$ and $C$. Thus, the combination $A$ and $C$ is
unconditionally missing. So we asked ourselves ”Is it possible to pre-train a
multimodal neural network that can handle any combination of modalities at
inference time?” Optimally, the model will achieve better downstream
performance the more modality combinations are given. This also includes the
never-seen input combinations $(A,C)$ and $(A,B,C)$.

To the best of our knowledge, this scenario has never been considered before
in the literature. We propose a novel transitive pre-trained transformer
(Figure 2) that can deal with such situations. LoReTTa (Linking mOdalities
with a tRansitive and commutativE pre-Training sTrAtegy) can link modalities
and transition between them using generative pre-training (predicting the next
token), the commutative property $(A,B)=(B,A)$, and the transitive relation
$(A\rightarrow B)\wedge(B\rightarrow C)\Rightarrow(A\rightarrow C)$. We show
theoretically and experimentally that by learning the relationship within and
between tokens from different data distributions, LoReTTa is well suited for
training multimodal models.

This has far-reaching implications for safety-critical domains such as
healthcare, infrastructure, and transportation. In medicine, for example, it
is already difficult to obtain a large dataset with a single modality to train
powerful foundation models – such as those used in general computer vision and
natural language processing. Finding a dataset with two or more modalities is
even more challenging. But combining them is important because it is common
practice to look at different types of data, such as radiological images,
genetic sequences, and microscopic slides, to determine the optimal treatment.
Similarly, infrastructure monitoring models or automated driving systems
require input from a variety of sensors. However, not all modality
combinations may be present in the same training dataset. With LoReTTa, we
demonstrate a unique way to address this challenging and under-researched
problem.

Figure 1: Venn diagrams showing the logical relationship between datasets
containing different modalities $A$, $B$, and $C$. Overlapping datasets
indicate that the dataset contains samples with aligned modalities (i.e., an
audio, image, and text file describing the same concept). While recent work
has mostly focused on datasets where at least some samples have all modality
combinations available (Venn diagrams (a) and (b)), we investigate the case
where some modality combinations, e.g. $(A,C)$ and $(A,B,C)$, are missing
entirely (Venn diagram (c)).

##  2 Related work

With the introduction of transformers [77], the fields of natural language
processing (NLP) and computer vision (CV) have experienced a significant shift
in performance and capabilities. These models excel at a wide range of complex
tasks, including machine translation, text summarization, question answering,
sentiment analysis, natural language generation, image classification, image
captioning, semantic segmentation, object detection, and face recognition –
achieving state-of-the-art performance in each area [45]. Transformers have
also opened up new possibilities for multimodal learning due to their lack of
inductive bias [51]. They can process and fuse different types of data, such
as audio, images, text, and video, using self- and cross-attention mechanisms
[77]. As a result, transformers are considered to be universal, modality-
agnostic learners [48, 4].

However, their lack of inductive bias comes at a price. Unlike other neural
network architectures, transformers make no assumptions about the structure of
the data [62]. This makes them highly expressive, but they also require a
large amount of data to learn effectively and generalize well [25, 31].
Consequently, self-supervised learning (SSL) on vast amounts of unlabeled data
has been proposed for transformers. In NLP, this strategy has been highly
successful, with methods such as masked modeling and causal modeling used to
train large language models like BERT variants [23, 42, 67, 38] and GPT
variants [56, 57, 14, 53, 60, 31, 22].

Recently, attempts have been made to unify both approaches through frameworks
like prefix modeling [61], permutation modeling [85], and causal masked
modeling [1, 28, 11]. When applied to image classification, token-based pre-
training methods [10, 7, 30, 17] have demonstrated their ability to outperform
established computer vision approaches such as SimCLR [19] and DINO [15]. As a
result, these sequence modeling techniques have been successfully adapted and
extended to a wide range of domains, encompassing programming [26, 18],
chemistry [46, 27], and biology [84, 70].

Adding or combining different modalities during pre-training could open up
three possibilities: positive transfer, cross-modal comprehension, and cross-
modal generation. Positive transfer can occur when learning in one modality
helps to improve performance in another modality. For example, it has been
shown that aligning images with text can lead to better classification
performance on ImageNet [58, 80]. Cross-modal understanding refers to the
explanation of one modality using the other. For example, it is possible to
describe an image using text [5, 20] or to understand proteins from sequences
[34]. And with cross-modal generation, one can generate one modality using the
other. Examples are text-to-image generation [63, 64] and text-to-music
generation [3]. All of these models are based on either contrastive learning
[40, 58, 87, 29, 82, 71], masked modeling [42, 47, 32, 55, 72, 80, 73, 21, 89,
79], or causal modeling [80, 24, 5, 66, 86, 1, 63, 53]

Overall, we introduce a novel pre-training method that combines recent
advances in self-supervised learning into one unified framework. LoReTTa uses
causal modeling for generative tasks and (causal) masked modeling for
discriminative tasks. It extends both approaches by using commutative and
transitive modeling to seamlessly integrate and impute modalities. This allows
us to learn expressive features and relationships within multimodal datasets.
The pre-trained models can be used for various downstream tasks such as
imputation or classification.

Figure 2: LoReTTa consists of two novel self-supervised training strategies:
commutative and transitive generative pre-training. (a) It applies the well-
known causal modeling technique in a commutative way to generate modality $A$
from $B$ and modality $B$ from $A$. The same idea is applied to modalities $B$
and $C$. (b) To ensure that the model learns bidirectional relations between
the input tokens (classical causal modeling is unidirectional), we use a
modified variant of generative pre-training called causal masked modeling
(also known as filling-the-middle), similar to [1, 28, 11]. (c, d) Next, we
apply transitive pre-training to learn all missing conditional joint
distributions. In transitive modeling, we randomly select a sample and use the
linking modality to predict the missing modality, which is then used to
reconstruct the existing modality, ensuring alignment across all modalities.
This process resembles cycle consistency [90] but is much more general.

##  3 Methods

Model architecture: LoReTTa uses the autoregressive transformer architecture
described in Radford et al. [57]. This model includes a modified
initialization, pre-normalization, and GELU activation. In addition, we choose
RMSNorm [88] over LayerNorm [9], as in Rae et al. [60] and Brown et al. [14].
We also replace the StandardAttention [77] with FlashAttention [75] to reduce
memory usage and runtime – similar to [75]. Following Chowdhery et al. [22],
we train without dropout and do not use biases in any of the dense kernels or
layer norms. While any general sequence model can work for next token
prediction, we choose the transformer [77] in particular for its simplicity
and scalability. Theoretically, however, our method also works with other
backbones, such as the recent RWKV [12] and Hyena [54].

Since transformers take a sequence of tokens as input, we need to tokenize our
data first. There are several ways to transform data into tokens, including
directly using the underlying raw byte stream [66]. In our experiments, we use
some of the listed techniques, but are not limited to them:

  * •

Lookup tables encode text, nominal, and other discrete data. For example, when
tokenizing sentences, we employ common techniques such as SentencePiece [37],
Byte Pair Encoding [69], and WordPiece [68].

  * •

Low-dimensional audio, image, and other continuous data are encoded by their
raw byte values [17, 33]. In the case of images, each pixel is treated as a
token, while in the case of audio files, each frequency value serves as an
individual token.

  * •

When dealing with high-dimensional continuous data, such as large images or
audio, methods like principal component analysis (PCA) or convolutional neural
networks (CNNs) are applied to reduce the spatial dimension [25, 8]. To
discretize these patches/tokens, we use traditional techniques such as the
centroids of KMeans, as well as more advanced methods like VQ-VAEs [76, 65],
as illustrated in DALLE [63] and MusicLM [3].

After tokenization, we use a parameterized embedding function to project all
tokens into a common feature space. In addition, we add modality-specific
tokens to indicate the start and end of a modality. We also learn modality-
specific absolute position encodings that are added to the feature vectors.
The embeddings for each modality are then concatenated, but only if they
belong to the same subject/object. For example, if we have an audio-image pair
or an image-text pair describing the same concept, we would concatenate their
token sequences and feed them into the transformer. However, if only one
modality is available, we use only that modality. In summary, our pre-
processing step is described below:

  * •

We tokenize and project all samples $a,b,c$ into a common vector space,
obtaining the token embeddings $[a_{1},\dots,a_{l}],[b_{1},\dots,b_{m}]$, and
$[c_{1},\dots,c_{n}]$.

  * •

If $a$ and $b$ are aligned, then we concatenate them as
$s=[a_{1},\dots,a_{l};b_{1},\dots,b_{m}]$ or
$s=[b_{1},\dots,b_{m};a_{1},\dots,a_{l}]$, which ensures commutativity.

Pre-training strategy: We train the transformer using causal language
modeling. Given a sequence of tokens $s_{1:L}$ and parameters $\theta$, the
data is modeled using the chain rule of probability

$\log p_{\theta}(s_{1},\dots,s_{L})=\sum_{l=1}^{L}\log
p_{\theta}(s_{l}|s_{1},\dots,s_{l-1})$ (1)

Thus, the task of the model is to predict the next token given the sequence of
previous tokens. We ensure commutativity by randomly mixing the order of
concatenation when appending tokens from two modalities to an input sequence
as described above. This allows the model to infer modality $A$ from $B$ and
modality $B$ from $A$. We call this technique commutative modeling.
Interestingly, this simple modification has not been used in recent multimodal
foundation models.

Since generative pre-trained transformers only consider information from the
left and ignore information from the right, they are known to lack
bidirectional context [78], which is crucial for many downstream tasks [61].
Since our goal is to train a universal model that can be fine-tuned for
different endpoints, we use causal masked modeling [1, 28, 11], mitigating
these problems. It randomly masks out a span (or several spans) of tokens and
moves them to the end. This change does not alter the training process based
on Equation 1; however, it now allows the model to use the information to the
right of the mask to predict the masked positions, which is not possible with
traditional one-directional causal modeling.

We return to the problem formulation outlined in Section 1. In particular, it
requires us to combine the modalities $A$ and $C$, given only the combinations
$(A,B)$ and $(B^{\prime},C)$. We propose to randomly sample data points
$(a,b),(b,a),(b^{\prime},c)$, or $(c,b^{\prime})$ during training and apply
transitive modeling. For simplicity, we will consider the pair $(a,b)$. Since
the model is trained to predict modality $C$ from $B$, we can infer the pseudo
data point $\hat{c}$ from $b$. Now we train the transformer to predict the
point $\hat{a}$ using the new inferred sample $\hat{c}$ in such a way that the
loss with respect to the original $a$ is minimized, ensuring that all our
modalities are aligned (Figure 2c). This way we are modeling the conditional
distribution of the missing pair $(C,A)$ – which is equal to $(A,C)$ due to
commutativity. The described approach is analogous to the concept of cycle
consistency used in CycleGANs [90]. However, our method is much more general
as it can be applied to many combinations of modalities as long as there is a
linking modality, e.g. $B$, as shown in Figure 2d.

In summary, LoReTTa uses masked modeling to transition within modalities,
causal modeling to transition between modalities, and vice versa thanks to
commutative modeling. Moreover, not all modality combinations need to be
available at training time, as we use transitive modeling to predict the
missing modalities and connect them to the original data. In our experiments,
we show that this approach allows us to learn very rich multimodal feature
representations.

##  4 Theoretical analysis

In this section, we provide definitions of the terms used and outline the
theoretical motivations for our approach described above. First, we define
modality: Modalities typically refer to different types of data sources,
domains, or sensor affinities. However, the distinction between modalities is
not always straightforward, such as when comparing different languages within
the text domain.

Using the $\sigma$-membership definition [2] for modalities, we can determine
whether two datasets belong to different modalities by examining their
perplexity:

$\text{PPL}(X)=\exp\left(-\frac{1}{N}\sum_{i=1}^{N}\log(p(x_{i}))\right)$ (2)

Perplexity is defined as the exponentiated average negative log-likelihood of
a sequence; it measures how well a probabilistic model predicts a given
dataset. Intuitively, a lower perplexity indicates a more accurate model with
better predictive capabilities. This concept is related to maximizing log-
likelihood because increasing the accuracy while reducing the uncertainty of a
model’s predictions ultimately leads to lower perplexity values.

In practice, to determine whether two datasets, $X_{i}$ and $X_{j}$, belong to
different modalities, we compare their perplexities under each other’s
probability distributions. If the perplexity of $X_{i}$ evaluated using the
distribution of $X_{j}$ is significantly higher (e.g., more than $\sigma^{2}$)
than the perplexity of $X_{j}$ evaluated using its own distribution, it
indicates that the model trained on $X_{j}$ struggles to predict $X_{i}$ well.
This suggests that the two datasets probably belong to different modalities,
as they have different statistical properties. On the other hand, if the
perplexities are similar, it indicates that the model can predict both
datasets reasonably well, suggesting that they have similar properties and
belong to the same modality. This gives us the following definition [2]:

$X_{i}\in X_{j}\Leftrightarrow\mathbb{E}_{x\sim
X_{i}}[L_{X_{j}}(x)]\leq\sigma^{2}\mathbb{E}_{x\sim X_{j}}[L_{X_{j}}(x)]$ (3)

By applying the above equation for a $\sigma$ value of 3, one can distinguish
various modalities such as text, image, speech, code, and molecules [2].

With a clearer understanding of modality, we now address the question of
whether two modalities are aligned. We consider a dataset with modalities
$X_{1},...,X_{N}$ to be aligned if there exists a subset $S$ sampled from a
joint probability distribution $P(X_{1},...,X_{N};Y)$ [43]. Here, $Y$
represents the label space. This form of alignment, where only labels are
shared across modalities, is called weak alignment [16]. Conversely, when
there is a functional relationship at the level of physical objects, we
consider the alignment as strong alignment [16].

The above definition also extends to scenarios where we are unable to observe
full modality combinations. For example, during data collection, we might only
get the paired samples $(x_{1},x_{2})$ and $(x^{\prime}_{2},x^{\prime}_{3})$,
where $x_{2}$ and $x^{\prime}_{2}$ does not belong to the same subject/object.
Thus, we have datasets of the form $(X_{1},X_{2})$ and $(X_{2},X_{3})$ instead
of $(X_{1},X_{2},X_{3})$. This is in contrast to the case where the dataset
still contains some samples of the form $(x_{1},x_{2},x_{3})$, which was
considered in previous work.

Table 1: Perplexity on the test set. The subscript describes the pre-training datasets containing image (I), text (T), or speech (W), while the blue colors highlight the results on the unseen modality pairs. Training | Testing
---|---
| I | T | W | (I, T) | (I, W) | (T, W)
CM2${}_{\text{(I)}}$ | 1.55 | - | - | - | - | -
CM2${}_{\text{(T)}}$ | - | 2.05 | - | - | - | -
CM2${}_{\text{(W)}}$ | - | - | 18.63 | - | - | -
C2M3${}_{\text{(I,T)}}$ | 1.62 | 2.28 | - | 1.76 | - | -
C2M3${}_{\text{(I,W)}}$ | 1.61 | - | 20.09 | - | 4.42 | -
C2M3${}_{\text{(T,W)}}$ | - | 2.25 | 19.85 | - | - | 10.68
C2M3${}_{\text{(I,T), (I,W)}}$ | 1.64 | 2.19 | 21.37 | 1.76 | 4.39 | 30.88
C2M3${}_{\text{(T,I), (T,W)}}$ | 1.65 | 2.18 | 19.69 | 1.76 | 104.27 | 11.26
C2M3${}_{\text{(W,I), (W,T)}}$ | 1.64 | 2.19 | 23.45 | 9.70 | 4.66 | 11.92
LoReTTa${}_{\text{(I,T), (I,W)}}$ | 1.56 | 2.17 | 18.77 | 1.72 | 3.14 | 11.21
LoReTTa${}_{\text{(T,I), (T,W)}}$ | 1.59 | 2.10 | 18.95 | 1.70 | 5.04 | 10.03
LoReTTa${}_{\text{(W,I), (W,T)}}$ | 1.60 | 2.14 | 20.35 | 3.13 | 4.36 | 10.65

To model the relationship of $(X_{1},X_{3})$ in a self-supervised way, we
cannot use the previous techniques of masked or causal modeling. The neural
network would have to implicitly learn

$p(x_{3}|x_{1})=\frac{p(x_{1},x_{3})}{p(x_{1})},$ (4)

by deriving $p(x_{1},x_{3})=\sum_{x_{2}}p(x_{1},x_{2},x_{3})$ and
$p(x_{1},x_{2},x_{3})=p(x_{3}|x_{2},x_{1})p(x_{2}|x_{1})p(x_{1})$ from the
chain rule of probability. Empirically [44, 83, 6], both unidirectional and
bidirectional sequence models fail this task as it is difficult to model
$p(x_{3}|x_{2},x_{1})$ due to the lack of $(X_{1},X_{2},X_{3})$. This is where
our proposed transitive modeling comes in.

Suppose we have a dataset of the form
$(X_{1},X_{2}),(X_{2},X_{3}),\dots,(X_{N-1},X_{N})$, where $X_{k}$ and $X_{l}$
are not aligned when $k+1\neq l$. By applying causal modeling to all pairs, we
can learn the transition $X_{1}\rightarrow X_{2}\rightarrow\dots\rightarrow
X_{N-1}\rightarrow X_{N}$. Adding our proposed commutative modeling, we get an
even stronger bidirectional relation: $X_{1}\leftrightarrow
X_{2}\leftrightarrow\dots\leftrightarrow X_{N-1}\leftrightarrow X_{N}$. Now,
let us interpret each modality as a node on a high-dimensional graph. This
gives us a connected graph (minimum spanning tree) where we can go from one
modality to any other modality. Thus, given a real data point $x_{i}$, we can
use transitive modeling to sample pseudo data points $x_{j}$ by transitioning
from $x_{i}$ to $x_{j}$. Having sampled many pairs $(x_{i},x_{j})$, we obtain
a new dataset $(X_{i},X_{j})$. It can be used again in the generative pre-
training process to learn the new transition $X_{i}\leftrightarrow X_{j}$. We
apply this idea to all $i$ and $j$. In the end, we learn the conditional
distribution between all samples and get a fully connected graph. This idea
holds for more complicated modality combinations as long as there is a
connecting modality, i.e., we have a spanning tree (see Figure 2d). We can
also sample any missing tuples $(x_{k},...,x_{l})$. Thus, LoReTTa allows us to
approximate any missing distribution $P(X_{k},...,X_{l})$ by sampling and
training with the missing modality combinations.

Ideally, there would be no generalization error after pre-training. In
practice, however, we would not be able to exactly recover the ground truth of
the missing modality if it were available. For the datasets $(X_{1},X_{2})$
and $(X_{2},X_{3})$, the learned generative model $f$ would make the following
errors:

$\displaystyle f(x_{1}|x_{2})=x_{1}+e_{2,1},\quad
f(x_{2}|x_{1})=x_{2}+e_{1,2},\quad f(x_{2}|x_{3})=x_{2}+e_{3,2},\quad
f(x_{3}|x_{2})=x_{3}+e_{2,3}$ (5)

where all $e_{i,j}$ are vector-valued error terms. Without loss of generality,
we do not track the sign of the error, since we can always write
$e_{i,j}^{\prime}=-1\cdot e_{i,j}$. Assuming, $f$ is differentiable and using
the multivariate Taylor expansion, we obtain

$\displaystyle x_{3}=f(x_{3}|f(x_{2}|x_{1})+e_{1,2})+e_{2,3}\approx
f(x_{3}|f(x_{2}|x_{1}))+J_{f}(x_{3}|f(x_{2}|x_{1}))J_{f}(x_{2}|x_{1})e_{1,2}+e_{2,3},$
(6)

where $J_{f}$ is the Jacobian of $f$. The iterative application of the Taylor
rule yields a similar error propagation formula for longer chains.
Intuitively, errors from the first transitions are amplified more and more.
However, because we use the predicted $x_{3}$ to reconstruct $x_{1}$, we can
actually bound the error
$e_{1,3}:=J_{f}(x_{3}|f(x_{2}|x_{1}))J_{f}(x_{2}|x_{1})e_{1,2}+e_{2,3}$.
Recall that we have a ground truth $x_{1}$ for which the following is true

$\displaystyle x_{1}$
$\displaystyle=f(x_{1}|x_{3})+e_{3,1}=f(x_{1}|f(x_{3}|x_{2})+e_{2,3})+e_{3,1}$
$\displaystyle=f(x_{1}|f(x_{3}|f(x_{2}|x_{1})+e_{1,2})+e_{2,3})+e_{3,1}$ (7)

Since we are minimizing the reconstruction error of $x_{1}$ given
$\hat{x}_{3}=x_{3}+e_{1,3}$ during backpropagation, the error terms $e_{1,2}$
and $e_{2,3}$ must be bounded. And since they make up the error term
$e_{1,3}$, it too must be bounded, as long as $f$ and $J_{f}$ are.

##  5 Experimental evaluation

Table 2: Average accuracy over three linear probing trials with 100 samples
per class. The subscript denotes the pre-training datasets with image (I),
text (T), or speech (W) modalities, while the blue colors highlight the
results on the unseen modality tuple during pre-training.

Training | Testing
---|---
| I | T | W | (I, T) | (I, W) | (T, W) | (I, T, W)
CM2${}_{\text{(I)}}$ | $79.7_{\pm 0.8}$ | - | - | - | - | - | -
CM2${}_{\text{(T)}}$ | - | $57.0_{\pm 0.5}$ | - | - | - | - | -
CM2${}_{\text{(W)}}$ | - | - | $80.3_{\pm 0.6}$ | - | - | - | -
C2M3${}_{\text{(I,T)}}$ | $82.9_{\pm 0.6}$ | $61.9_{\pm 0.7}$ | - | $89.7_{\pm 0.2}$ | - | - | -
C2M3${}_{\text{(I,W)}}$ | $83.8_{\pm 0.5}$ | - | $79.9_{\pm 0.2}$ | - | $89.9_{\pm 0.1}$ | - | -
C2M3${}_{\text{(T,W)}}$ | - | $61.1_{\pm 0.4}$ | $79.5_{\pm 0.2}$ | - | - | $85.2_{\pm 0.3}$ | -
C2M3${}_{\text{(I,T), (I,W)}}$ | $83.1_{\pm 0.8}$ | $59.8_{\pm 0.4}$ | $80.7_{\pm 0.4}$ | $87.9_{\pm 0.6}$ | $89.3_{\pm 0.2}$ | $71.2_{\pm 0.4}$ | $87.3_{\pm 0.4}$
C2M3${}_{\text{(T,I), (T,W)}}$ | $81.7_{\pm 0.1}$ | $63.2_{\pm 0.6}$ | $78.5_{\pm 0.6}$ | $89.1_{\pm 0.3}$ | $72.9_{\pm 0.2}$ | $84.8_{\pm 0.5}$ | $83.2_{\pm 0.4}$
C2M3${}_{\text{(W,I), (W,T)}}$ | $80.8_{\pm 0.4}$ | $63.9_{\pm 0.6}$ | $82.8_{\pm 0.7}$ | $77.8_{\pm 0.5}$ | $89.8_{\pm 0.1}$ | $88.1_{\pm 0.5}$ | $89.9_{\pm 0.5}$
LoReTTa${}_{\text{(I,T), (I,W)}}$ | $82.7_{\pm 0.9}$ | $62.8_{\pm 0.4}$ | $82.5_{\pm 0.6}$ | $88.5_{\pm 0.7}$ | $89.7_{\pm 0.1}$ | $84.0_{\pm 0.2}$ | $90.7_{\pm 0.6}$
LoReTTa${}_{\text{(T,I), (T,W)}}$ | $81.2_{\pm 0.9}$ | $63.3_{\pm 0.4}$ | $81.0_{\pm 0.5}$ | $89.0_{\pm 0.3}$ | $80.9_{\pm 0.6}$ | $85.5_{\pm 0.5}$ |  $87.8_{\pm 0.3}$
LoReTTa${}_{\text{(W,I), (W,T)}}$ | $80.1_{\pm 0.3}$ | $62.1_{\pm 0.6}$ | $84.2_{\pm 0.5}$ | $83.0_{\pm 0.7}$ | $90.4_{\pm 0.3}$ | $89.0_{\pm 0.2}$ | $91.6_{\pm 0.7}$

Evaluation protocol: We empirically analyze LoReTTa on a constructed dataset
and a real-world medical dataset with three modalities. For the downstream
tasks, we choose object classification and survival prediction, respectively.
As discussed in Section 2, sequence modeling is well suited for such
discriminative tasks. This is true for both masked [10, 46] and causal [20, 1,
59, 80, 52] models, even surpassing SOTA alternative methods. The usual way to
evaluate these self-supervised methods is to freeze the pre-trained network
and train a linear layer on top of it using supervised labels (linear
probing). This allows us to directly assess the quality of the learned
features. We evaluate our method in two steps: First, we perform an ablation
study on the constructed dataset to analyze the effect of each design choice.
Second, we compare LoReTTa with other methods such as masked (BERT [23]),
causal (GPT [14]), and contrastive (CLIP [58]) modeling on the real-world
dataset.

SVL-MNIST dataset: We test our method on a custom dataset derived from real-
world data that includes vision (I), language (T), and speech (W) modalities
representing 10 different categories. The speech dataset features about 40,000
spectrograms from AudioMNIST [35], the vision dataset comprises 70,000 images
from MNIST [39], and the language dataset consists of 130,000 documents from
WineReviews [74]. We link the datasets via their labels, i.e., wine number
zero is linked to the spoken digit zero, the handwritten number zero, and the
corresponding written review (code available at anonymous_url). This approach
results in weakly aligned modalities. The rationale behind this approach is
that the spoken number, the handwritten number, and the written description
each serve as different ways of referring to the same wine. Ambiguous
descriptions, handwriting, or pronunciation can lead to misclassification, but
having at least two modalities can help clarify and distinguish the referenced
objects more accurately. For the sake of clarity, we will refer to this
dataset as Speech-Vision-Language-MNIST (SVL-MNIST). We randomly split the
dataset such that they have exactly the same relationship, as shown in Figure
1c. In particular, our bimodal datasets (I, T), (T, W), and (I, W) have 12,000
non-overlapping samples each. All remaining samples are part of the unimodal
datasets I, T, and W. Only for testing, we use a dataset with three modalities
(I, T, W).

TCGA-OMICS dataset: The medical dataset contains omics (e.g., genomics,
proteomics, and transcriptomics) expression values from The Cancer Genome
Atlas Program (TCGA) [81]. We select the three largest subsets, which include
messenger ribonucleic acid (mRNA), micro-ribonucleic acid (miRNA), and
reverse-phase protein array (RPPA) from up to $\sim$11,000 patients. We align
the dataset at the patient level and obtain approximately 7,000 patients with
all three modalities. To simulate a setting with missing modality
combinations, we split the dataset into two subsets (mRNA, miRNA) and (mRNA,
RPPA) with non-overlapping 3,500 samples each. We choose mRNA as the linking
modality because it is one of the most common genetic modalities available.
This experiment can only be done on TCGA because it is the only public medical
dataset that contains multiple modalities for the same patient – making it
ideal for evaluating our method.

Implementation details: Our transformer decoder has 8 layers and 8 attention
heads, with an embedding dimension of 512 and query, key, and value dimensions
of 64 each. This compact architecture has proven effective in various tasks
and has been widely used in the literature [23, 31, 41]. We use a single
parameterized embedder to encode tokens and add modality-specific learned
positional embeddings [63, 66, 2]. For optimization, we use the AdamW
algorithm with a learning rate of 6e-4, a weight decay factor of 0.1, and
gradient clipping at 1. We have found that this set of hyperparameters works
very well for all experiments. The learning rate undergoes a 10-fold decay
using cosine annealing and a linear warm-up during the first 1,500 steps. For
more details on how we chose the hyperparameters, see the Appendix.

##  6 Results

###  6.1 SVL-MNIST

In the first set of experiments, we train three models (one for each modality
I, T, and W) with causal masked modeling (CM2). Next, we train three
additional models with commutative causal masked multimodal modeling (C2M3)
(Figure 2a,b) on the bimodal datasets (I, T), (T, W), and (I, W). These six
models serve as unimodal and multimodal baselines, respectively. We then train
on both (I, T) and (T, W) simultaneously using C2M3 with T as the linking
modality. We do the same for all other combinations. Finally, we initialize
LoReTTa (Figure 2a,b,c) with the C2M3 weights to combine the bimodal datasets
through the linking modality. After pre-training, we freeze the models and
evaluate them in a low-data regime using only 1000 labeled samples (100 per
class). We report the results of three runs; each run is performed with a
newly sampled subset. We also report the perplexity on the test set. Note that
we only tune the models on the modalities that the model saw during pre-
training. This also means that during probing, the linear classifier will see
some samples of the unseen modality pair. However, the pre-trained backbone
itself did not see them. In this way, we can analyze how the model combines
previously unseen modality combinations.

Table 1 shows the perplexity of the pre-trained models on the respective test
sets. When pre-trained only on the pairs (I, T) and (T, W), LoReTTa achieves
an impressively low perplexity of 5.04 on the unseen pair (I, W), which is
significantly lower than the perplexity of the baseline non-transitive model
trained with C2M3 (104.27). The same trend is observed for other combinations.
For example, the model trained on (I, W) and (W, T) and evaluated on (I, T)
has a perplexity of 3.13 (LoReTTa) compared to 9.70 (C2M3). Similarly, the
model trained on (I, T) and (I, W) and evaluated on (T, W) has a perplexity of
11.21 (LoReTTa) compared to 30.88 (C2M3). Remarkably, LoReTTa consistently
achieves perplexities for unseen modality combinations that are in a similar
range to those for seen modality combinations (non-blue values in Table 1),
demonstrating that LoReTTa is indeed capable of learning the missing
distribution as described in Section 4. Looking at Table 2, we see a similar
trend. In two out of three cases, LoReTTa achieves the highest classification
accuracy by combining two modalities. Moreover, it far outperforms C2M3 on the
unseen modality pairs. In other words, if either (IMG, TXT), (TXT, WAV), or
(IMG, WAV) is missing, it is advisable to train a model using LoReTTa, since
it is the only model capable of effectively integrating these modalities and
achieving higher accuracy compared to any single-modality model. In fact, if a
few samples with three modalities are given during linear probing (last column
of Table 2), then models pre-trained with LoReTTa can improve the
classification score even further. On the other hand, models trained with the
bimodal datasets in parallel cannot handle three modalities simultaneously.
The accuracy with three modalities is often worse than with two modalities
(negative transfer).

###  6.2 TCGA-OMICS

Table 3: We train a Cox proportional hazard model with an elastic net penalty on the extracted features of each frozen model and report the C-index on the test sets. In particular, the models are pre-trained on the (mRNA, miRNA) and (mRNA, RPPA) datasets and fine-tuned with all available labels. Due to space limitations, we abbreviate mRNA as M, miRNA as I, and RPPA as R. Blue highlights the results on the unseen modality tuples, and the underline the second highest value. Training | Testing
---|---
| M | I | R | (M, I) | (M, R) | (I, R) | (M, I, R)
BERT | 0.592 | 0.621 | 0.594 | 0.595 | 0.613 | 0.594 | 0.610
GPT | 0.575 | 0.590 | 0.611 | 0.585 | 0.576 | 0.606 | 0.588
CLIP | 0.561 | 0.603 | 0.587 | 0.610 | 0.600 | 0.623 | 0.612
C2M3 | 0.621 | 0.599 | 0.599 | 0.624 | 0.620 | 0.571 | 0.624
LoReTTa | 0.652 | 0.623 | 0.563 | 0.660 | 0.652 | 0.623 | 0.657

Omics data can be very high dimensional. For example, an mRNA expression array
contains more than 20,500 entries. Therefore, we first reduce the dimensions
by a factor of 10 using PCA and then bin the values to a dictionary of 1,000
each according to Bolis et al. [13] and Yang et al. [84]. The final input
sizes are 2,048, 64, and 16, respectively – retaining their original relative
lengths. We then pre-train our transformer with C2M3 and initialize LoReTTa
with the weights of the C2M3 model, as we did in our SVL-MNIST experiments.
For GPT, we simply use the algorithm of C2M3, but disable causal masking and
communicative switching. BERT uses the same transformer architecture; we just
have to change the unidirectional attention mask to a bidirectional attention
mask. The model based on CLIP consists of 3 encoders (each using our same
transformer architecture) and a contrastive loss [36] to align the features.
After pre-training, we extract the features and fit a Cox proportional hazards
model with elastic net penalty for each modality combination using all
available labels.

On 6 out of 7 test sets, the LoReTTa pre-trained transformer achieves the
highest C-index (Table 3). Most importantly, this includes the unseen modality
pair (miRNA, RPPA) and the triplet (mRNA, miRNA, RPPA). The only framework on
par with LoReTTa is CLIP on the unseen dataset (miRNA, RPPA). However, the
main disadvantage of CLIP is that it requires the expensive training of three
encoders – one for each modality. Remarkably, CLIP and LoReTTa are the only
strategies that consistently allow for positive or neutral transfer, which is
not always the case with other methods.

##  7 Discussion & Conclusion

With LoReTTa, we introduced an innovative self-supervised learning framework
for multimodal integration that learns any missing joint conditional
distribution given a linking modality. We demonstrated this theoretically and
practically. In particular, our method combines the rules of commutativity and
transitivity to model the multimodal datasets $(A,C)$ and $(A,B,C)$ given the
two non-overlapping datasets $(A,B)$ and $(B,C)$. In fact, we showed that the
extracted features of a transformer pre-trained with LoReTTa are very
expressive for all modality combinations, including the unseen ones. While we
only evaluated our algorithm on datasets with three modalities, it is much
more applicable because it can be used with any number and combination of
modalities. As long as there is a chain of aligned modalities in the dataset,
LoReTTa is able to learn the transition $(X_{i}\rightarrow...\rightarrow
X_{j}),...,(X_{i}\rightarrow...\rightarrow X_{k})$ to model the missing
distribution $P(X_{i},...,X_{k})$ (as shown in Figure 2d). To the best of our
knowledge, this scenario has never been considered before in the literature
without training an ensemble of contrastive models based on the ideas of CLIP
[29, 82].

Broader impacts: We believe that LoReTTa will serve as a basis for training
very powerful multimodal generative and discriminative models that will be
useful in many areas of research. While our method itself cannot be misused,
future models trained with LoReTTa could be used to generate higher-quality
harmful content. However, LoReTTa was designed to improve the quality of
neural networks that contribute to society, for example by training more
accurate multimodal systems that can help doctors and other practitioners in
their daily work.

Environmental impacts. We trained all of our models on an NVIDIA
A100-SXM4-40GB GPU using PyTorch 2.0. Techniques such as mixed-precision
training, FlashAttention, and TF32 helped us reduce memory usage and runtime.
Pre-training a model with causal masked modeling took us about 12 hours on
average. On the other hand, fine-tuning with LoReTTa took longer – about 3
days. This is because we had to autoregressively decode the predicted samples
one token at a time.

Limitations. LoReTTa is best used in a setting where we have aligned
modalities that always miss a modality combination. This only works if there
is at least one linking modality, otherwise, our method cannot be applied. Can
our method also be applied to datasets where some samples have all modality
combinations? Yes, this is possible. Although we have not analyzed this
scenario, LoReTTa is likely to provide benefits by augmenting those samples
that are missing the remaining modalities.

## References

  * Aghajanyan et al. [2022] Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al.  Cm3: A causal masked multimodal model of the internet.  _arXiv preprint arXiv:2201.07520_ , 2022.
  * Aghajanyan et al. [2023] Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer.  Scaling laws for generative mixed-modal language models.  _arXiv preprint arXiv:2301.03728_ , 2023.
  * Agostinelli et al. [2023] Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al.  Musiclm: Generating music from text.  _arXiv preprint arXiv:2301.11325_ , 2023.
  * Akbari et al. [2021] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong.  Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 34, pages 24206–24221, 2021.
  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 35, pages 23716–23736, 2022.
  * Asai and Hajishirzi [2020] Akari Asai and Hannaneh Hajishirzi.  Logic-guided data augmentation and regularization for consistent question answering.  In _58th Annual Meeting of the Association for Computational Linguistics_ , pages 5642–5650. Association for Computational Linguistics, 2020\.
  * Assran et al. [2022] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas.  Masked siamese networks for label-efficient learning.  In _2022 European Conference on Computer Vision (ECCV)_ , pages 456–473. Springer Nature Limited, 2022.
  * Avsec et al. [2021] Žiga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska-Barwinska, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley.  Effective gene expression prediction from sequence by integrating long-range interactions.  _Nature Methods_ , 18(10):1196–1203, 2021.
  * Ba et al. [2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Baevski et al. [2022] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli.  data2vec: A general framework for self-supervised learning in speech, vision and language.  In _39th International Conference on Machine Learning (ICML)_ , pages 1298–1312. PMLR, 2022.
  * Bavarian et al. [2022] Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen.  Efficient training of language models to fill in the middle, 2022.
  * Bo [2021] Peng Bo.  Blinkdl/rwkv-lm: 0.01, Aug 2021.  URL https://doi.org/10.5281/zenodo.5196577.
  * Bolis et al. [2021] Marco Bolis, Daniela Bossi, Arianna Vallerga, Valentina Ceserani, Manuela Cavalli, Daniela Impellizzieri, Laura Di Rito, Eugenio Zoni, Simone Mosole, Angela Rita Elia, et al.  Dynamic prostate cancer transcriptome analysis delineates the trajectory to disease progression.  _Nature Communications_ , 12(1):7033, 2021.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 33, pages 1877–1901, 2020.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 9650–9660. IEEE, 2021.  doi: 10.1109/ICCV48922.2021.00951.
  * Castrejon et al. [2016] Lluis Castrejon, Yusuf Aytar, Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba.  Learning aligned cross-modal representations from weakly aligned data.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2940–2949, 2016.
  * Chen et al. [2020a] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever.  Generative pretraining from pixels.  In _International Conference on Machine Learning (ICML)_ , pages 1691–1703. PMLR, 2020a.
  * Chen et al. [2021] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ , 2021.
  * Chen et al. [2020b] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _37th International Conference on Machine Learning (ICML)_ , pages 1597–1607. PMLR, 2020b.
  * Chen et al. [2023] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme Ruiz, Andreas Peter Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut.  PaLI: A jointly-scaled multilingual language-image model.  In _11th International Conference on Learning Representations (ICML)_ , 2023.
  * Chen et al. [2020c] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In _Computer Vision – ECCV 2020: 16th European Conference_ , pages 104–120. Springer, 2020c.
  * Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics, 2019.
  * Diao et al. [2023] Shizhe Diao, Wangchunshu Zhou, Xinsong Zhang, and Jiawei Wang.  Write and paint: Generative vision-language models are unified modal learners.  In _11th International Conference on Learning Representations (ICLR)_ , 2023.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _9th International Conference on Learning Representations (ICLR)_ , pages 1–21. ICLR, 2021.
  * Feng et al. [2020] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou.  CodeBERT: A pre-trained model for programming and natural languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1536–1547. Association for Computational Linguistics, 2020\.
  * Ferruz et al. [2022] Noelia Ferruz, Steffen Schmidt, and Birte Höcker.  Protgpt2 is a deep unsupervised language model for protein design.  _Nature Communications_ , 13(1):4348, 2022.
  * Fried et al. [2023] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis.  Incoder: A generative model for code infilling and synthesis.  In _11th International Conference on Learning Representations (ICLR)_ , 2023.
  * Girdhar et al. [2023] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra.  Imagebind: One embedding space to bind them all.  _arXiv preprint arXiv:2305.05665_ , 2023.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 16000–16009. IEEE, 2022.
  * Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.  An empirical analysis of compute-optimal large language model training.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 35, pages 30016–30030, 2022.
  * Huang et al. [2020] Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu.  Pixel-bert: Aligning image pixels with text by deep multi-modal transformers.  _arXiv preprint arXiv:2004.00849_ , 2020.
  * Jaegle et al. [2021] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira.  Perceiver: General perception with iterative attention.  In _International Conference on Machine Learning (ICML)_ , pages 4651–4664. PMLR, 2021.
  * Jumper et al. [2021] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al.  Highly accurate protein structure prediction with alphafold.  _Nature_ , 596(7873):583–589, 2021.
  * Khacef et al. [2019] Lyes Khacef, Laurent Rodriguez, and Benoit Miramond.  Written and spoken digits database for multimodal learning.  https://zenodo.org/record/3515935, 2019.
  * Khosla et al. [2020] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan.  Supervised contrastive learning.  In _33th Advances in Neural Information Processing Systems (NeurIPS)_ , pages 18661–18673. Curran Associates, Inc., 2020.
  * Kudo and Richardson [2018] Taku Kudo and John Richardson.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP)_ , pages 66–71. Association for Computational Linguistics, 2018.
  * Lan et al. [2020] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  Albert: A lite bert for self-supervised learning of language representations.  In _International Conference on Learning Representations (ICLR)_ , 2020.
  * LeCun et al. [2010] Yann LeCun, Corinna Cortes, and Christopher J. C. Burges.  Mnist handwritten digit database.  http://yann.lecun.com/exdb/mnist/, 2010.
  * Li et al. [2021] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 34, pages 9694–9705, 2021.
  * Li et al. [2023] Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg.  Emergent world representations: Exploring a sequence model trained on a synthetic task.  In _11th International Conference on Learning Representations (ICLR)_ , 2023.
  * Li et al. [2019] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  _arXiv preprint arXiv:1908.03557_ , 2019.
  * Liang et al. [2021] Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Yufan Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al.  Multibench: Multiscale benchmarks for multimodal representation learning.  In _35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2021.
  * Lin and Ng [2022] Ruixi Lin and Hwee Tou Ng.  Does bert know that the is-a relation is transitive?  In _60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 94–99. Association for Computational Linguistics, 2022.
  * Lin et al. [2022a] Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu.  A survey of transformers.  _AI Open_ , 3:111–132, 2022a.
  * Lin et al. [2022b] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, and Alexander Rives.  Language models of protein sequences at the scale of evolution enable accurate structure prediction.  _bioRxiv_ , 2022b.  doi: 10.1101/2022.07.20.500902.
  * Lu et al. [2019] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 32, 2019.
  * Lu et al. [2021] Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch.  Pretrained transformers as universal computation engines.  _arXiv preprint arXiv:2103.05247_ , 1, 2021.
  * Ma et al. [2021] Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng.  Smil: Multimodal learning with severely missing modality.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 2302–2310, 2021.
  * Ma et al. [2022] Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng.  Are multimodal transformers robust to missing modality?  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 18177–18186, 2022.
  * Matsoukas et al. [2022] Christos Matsoukas, Johan Fredin Haslum, Moein Sorkhei, Magnus Söderberg, and Kevin Smith.  What makes transfer learning work for medical images: Feature reuse & other factors.  In _2022 IEEE/CVF CVPR_ , pages 9225–9234. IEEE, 2022.
  * Oord et al. [2016] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu.  Wavenet: A generative model for raw audio.  _arXiv preprint arXiv:1609.03499_ , 2016.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Poli et al. [2023] Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré.  Hyena hierarchy: Towards larger convolutional language models.  _arXiv preprint arXiv:2302.10866_ , 2023.
  * Qi et al. [2020] Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti.  Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data.  _arXiv preprint arXiv:2001.07966_ , 2020.
  * Radford et al. [2018] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training, 2018.
  * Radford et al. [2019] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners, 2019.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning (ICML)_ , pages 8748–8763. PMLR, 2021.
  * Radford et al. [2022] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever.  Robust speech recognition via large-scale weak supervision.  _arXiv preprint arXiv:2212.04356_ , 2022.
  * Rae et al. [2021] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al.  Scaling language models: Methods, analysis & insights from training gopher, 2021.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research (JMLR)_ , 21(1):5485–5551, 2020.
  * Raghu et al. [2022] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy.  Do vision transformers see like convolutional neural networks?  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 34, pages 12116–12128, 2022.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning (ICML)_ , pages 8821–8831. PMLR, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Razavi et al. [2019] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals.  Generating diverse high-fidelity images with vq-vae-2.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 32, 2019.
  * Reed et al. [2022] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas.  A generalist agent.  _Transactions on Machine Learning Research (TMLR)_ , 2022.  ISSN 2835-8856.
  * Sanh et al. [2019] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_ , 2019.
  * Schuster and Nakajima [2012] Mike Schuster and Kaisuke Nakajima.  Japanese and korean voice search.  In _2012 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 5149–5152. IEEE, 2012.
  * Sennrich et al. [2016] Rico Sennrich, Barry Haddow, and Alexandra" Birch.  Neural machine translation of rare words with subword units.  In _54th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 1715–1725. Association for Computational Linguistics, 2016.
  * Shen et al. [2022] Hongru Shen, Xilin Shen, Jiani Hu, Jilei Liu, Chao Zhang, Dan Wu, Mengyao Feng, Meng Yang, Yang Li, Yichen Yang, Wei Wang, Qiang Zhang, Jilong Yang, Kexin Chen, and Xiangchun Li.  Generative pretraining from large-scale transcriptomes: Implications for single-cell deciphering and clinical translation.  _bioRxiv_ , 2022.
  * Shih et al. [2023] Yi-Jen Shih, Hsuan-Fu Wang, Heng-Jui Chang, Layne Berry, Hung-yi Lee, and David Harwath.  Speechclip: Integrating speech with pre-trained vision and language model.  In _2022 IEEE Spoken Language Technology Workshop (SLT)_ , pages 715–722. IEEE, 2023.
  * Su et al. [2020] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  In _International Conference on Learning Representations (ICLR)_ , 2020.
  * Tan and Bansal [2019] Hao Tan and Mohit Bansal.  LXMERT: Learning cross-modality encoder representations from transformers.  In _2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5100–5111. Association for Computational Linguistics, 2019.
  * Thoutt [2018] Zack Thoutt.  Wine reviews: 130k wine reviews with variety, location, winery, price, and description.  https://www.kaggle.com/datasets/zynicide/wine-reviews, 2018.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Van Den Oord et al. [2017] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 30, 2017.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _31st Conference on Neural Information Processing Systems (NIPS)_ , page 6000–6010. Curran Associates, Inc., 2017.
  * Wang et al. [2022a] Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel.  What language model architecture and pretraining objective works best for zero-shot generalization?  In _International Conference on Machine Learning (ICML)_ , pages 22964–22984. PMLR, 2022a.
  * Wang et al. [2022b] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al.  Image as a foreign language: Beit pretraining for all vision and vision-language tasks.  _arXiv preprint arXiv:2208.10442_ , 2022b.
  * Wang et al. [2022c] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao.  SimVLM: Simple visual language model pretraining with weak supervision.  In _International Conference on Learning Representations (ICLR)_ , 2022c.
  * Weinstein et al. [2013] John N Weinstein, Eric A Collisson, Gordon B Mills, Kenna R Shaw, Brad A Ozenberger, Kyle Ellrott, Ilya Shmulevich, Chris Sander, and Joshua M Stuart.  The cancer genome atlas pan-cancer analysis project.  _Nature Genetics_ , 45(10):1113–1120, 09 2013\.
  * Wu et al. [2022] Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello.  Wav2clip: Learning robust audio representations from clip.  In _2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 4563–4567. IEEE, 2022.
  * Yanaka et al. [2021] Hitomi Yanaka, Koji Mineshima, and Kentaro Inui.  Exploring transitivity in neural nli models through veridicality.  In _16th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 920–934. Association for Computational Linguistics, 2021.
  * Yang et al. [2022] Fan Yang, Wenchuan Wang, Fang Wang, Yuan Fang, Duyu Tang, Junzhou Huang, Hui Lu, and Jianhua Yao.  scbert as a large-scale pretrained deep language model for cell type annotation of single-cell rna-seq data.  _Nature Machine Intelligence_ , 4(10):852–866, 2022.
  * Yang et al. [2019] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.  Xlnet: Generalized autoregressive pretraining for language understanding.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 32, 2019.
  * Yu et al. [2022] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  _arXiv preprint arXiv:2206.10789_ , 2022.
  * Yuan et al. [2021] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al.  Florence: A new foundation model for computer vision.  _arXiv preprint arXiv:2111.11432_ , 2021.
  * Zhang and Sennrich [2019] Biao Zhang and Rico Sennrich.  Root mean square layer normalization.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 32, 2019.
  * Zhang et al. [2021] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao.  Vinvl: Revisiting visual representations in vision-language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5579–5588, June 2021.
  * Zhu et al. [2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In _IEEE International Conference on Computer Vision (ICCV)_ , pages 2223–2232, 2017.