# Training Neural Networks on RAW and HDR Images for Restoration Tasks

Lei Luo${}^{\textrm{1}}$ Alexandre Chapiro${}^{\textrm{1}}$ Xiaoyu
Xiang${}^{\textrm{1}}$ Yuchen Fan${}^{\textrm{1}}$ Rakesh
Ranjan${}^{\textrm{1}}$ Rafal Mantiuk${}^{\textrm{2}}$
${}^{\textrm{1}}$Meta ${}^{\textrm{2}}$University of Cambridge

###### Abstract

The vast majority of standard image and video content available online is
represented in display-encoded color spaces, in which pixel values are
conveniently scaled to a limited range (0–1) and the color distribution is
approximately perceptually uniform. In contrast, both camera RAW and high
dynamic range (HDR) images are often represented in linear color spaces, in
which color values are linearly related to colorimetric quantities of light.
While training on commonly available display-encoded images is a well-
established practice, there is no consensus on how neural networks should be
trained for tasks on RAW and HDR images in linear color spaces. In this work,
we test several approaches on three popular image restoration applications:
denoising, deblurring, and single-image super-resolution. We examine whether
HDR/RAW images need to be display-encoded using popular transfer functions
(PQ, PU21, mu-law), or whether it is better to train in linear color spaces,
but use loss functions that correct for perceptual non-uniformity. Our results
indicate that neural networks train significantly better on HDR and RAW images
represented in display-encoded color spaces, which offer better perceptual
uniformity than linear spaces. This small change to the training strategy can
bring a very substantial gain in performance, up to 10–15 dB.

##  1 Introduction

Neural networks are typically trained on images represented in _display-
encoded_ color spaces, such as BT.709 with the sRGB non-linearity [2]. These
are known as _display-encoded_ color spaces because they are commonly used to
drive displays. HDR images, as well as RAW images containing camera sensor
values, are often represented in _linear color spaces_ , in which the pixel
values are proportional to colorimetric or photometric light quantities111The
proportionality is approximate, as camera spectral sensitivity does not allow
for direct measurement of colorimetric or photometric quantities.. Notably,
linear color spaces are very far from being perceptually uniform [16]: the
same change in physical units is much less visible at high brightness than
low. Modern perceptual uniform transfer functions, such as the SMPTE PQ
(perceptual quantizer) [19] and PU21 (perceptually uniform transform) [17]
have been developed to address this. By way of example, according to the PQ
function, a change of 1 cd/m2 starting from near darkness (0.005 cd/m2) is
over 150 times more visible than the same change starting at 100 cd/m2.

There are arguments both for and against training neural networks with images
in linear spaces. If a plain L1 or L2 loss is used to train networks for
linear data, we run the risk that the network will overfit for bright colors,
and introduce unacceptable inaccuracies for darker colors. Additionally, if we
wish to avoid visible quantization errors, pixel values in linear color spaces
must be represented with more bits than display-encoded spaces. However, any
physical phenomena involving light, such as lens blur, motion blur, or noise,
can only be modeled in a physically plausible manner in linear color spaces.
Therefore, physically plausible image formation models for blur, noise and
sampling must be defined in linear color spaces, and by analogy, the inverse
problems of deblurring, denoising and super-resolution should also be
formulated in linear color spaces. It is unknown, however, whether neural
networks can benefit from the physical properties of linear color spaces.

In this work, we want to determine what training strategy leads to the best
results. Should we train neural networks in display-encoded or linear color
spaces (Section 3)? If we train networks in linear color spaces, should we use
a loss function that compensates for perceptual non-uniformity (Section 3.2)?
There is no consensus in the literature on these questions. Many works train
on images in linear color spaces and use an L1 loss [26, 6]. Some use a loss
with perceptual encoding [7, 11, 18]. Our goal is to find empirical evidence
to motivate a training strategy for image restoration techniques trained on
HDR/RAW images. To that end, we collect a dataset of HDR images and train six
networks to perform common image restoration tasks: single-image super-
resolution, denoising, and deblurring (Section 4).

Rather than proposing a new method, this paper is a benchmark and reproduction
study intended to settle these open questions. Our results serve as best
practice guidelines for future ML methods operating on HDR and RAW content.
Our results (Section 5) show a clear benefit in training networks in display-
encoded color spaces, offering a gain in performance up to 10–15 dB for these
applications.

##  2 Related work

###  2.1 Imaging RAW pixels

RAW image data, captured directly from image sensors, preserves unprocessed
and uncompressed pixel information, thereby providing rich information for
restoration techniques. RAW image formats also represent pixels in linear
color spaces, which facilitate physically plausible modeling of camera noise
[1]. Early works [9, 10] in this area mainly focused on noise reduction and
artifact suppression based on statistical models of noise. With the advent of
deep learning, restoration in the RAW domain has witnessed a paradigm shift.
[24, 4, 20] pioneered by the integration of deep neural networks for
denoising. Those early works, however, targeted sRGB images, typically with
synthetic additive Gaussian noise. Since denoising is likely to be integrated
in the very early stages of the ISP pipeline, it should operate on linear RAW
pixel values instead.

Surprisingly, many deep-learning methods proposed for processing RAW images
were never tested on RAW images [25, 15]. Instead, they opted to work with
display-encoded images in the sRGB space, and assumed additive Gaussian noise.
Both assumptions make input data very different from RAW sensor pixel values,
raising the question on whether these methods would be equally effective for
real camera data. Other methods that were trained and tested on RAW images
[26, 6], did so directly on linear color values and used an L1 loss. Our work
demonstrates this strategy often fails to achieve good results.

###  2.2 High Dynamic Range (HDR) imaging

The dynamic range is the contrast between the brightest and darkest parts of
an image or video. Notably, the dynamic range the human eye can see is
extremely high and much larger than what most standard dynamic range (SDR)
capture and display pipelines are capable of [13]. High Dynamic Range (HDR) is
a term for a range of technologies that enhance the visual quality and realism
of content by providing a wider than normal dynamic range [21].

A key problem in HDR imaging involves merging multiple exposures into an HDR
image while reducing “ghosting” artifacts due to motion [11]. Another problem
is tone-mapping of HDR scenes to the dynamic range that can be shown on a
display [21]. Yet another is the reconstruction of an HDR image from a single
exposure (i.e. inverse-tone mapping) [7, 12]. However, standard reconstruction
problems like denoising, single-image super-resolution, or
deconvolution/deblurring, are also relevant for HDR imaging.

This work focuses on how HDR data in linear color spaces was handled in these
previous works. Both Eilertsen et al. [7] and Kalantari et al. [11] trained
the networks to directly predict HDR images in a linear color space. However,
to compensate for the perceptual non-uniformity of that space, Eilertsen et
al. used the logarithmic function, and Kalantari et al. introduced the
$\mu$-law (explained in detail in Section 3). Mildenhall et al. [18] used a
relative loss, similar to SMAPE (explained in Section 3.2), to train a NeRF on
RAW images. The networks intended to reconstruct HDR UHD video content [12]
were directly trained to predict display-encoded frames using the PQ OETF
[19].

##  3 Representations

###  3.1 Pixel encodings

In this work, we explore popular perceptual pixel value representations from
the literature for use in machine learning applications. The main question
that arises is what representation – linear or display encoded – is most
suitable for deep-learning networks. The linear representation should be
better to model physical phenomena (assuming neural networks are able to take
advantage of this benefit). However, the display-encoded representations are
more data-efficient (requiring fewer bits) and have better perceptual
uniformity. Moreover, most existing methods have been trained on display-
encoded representations, as these are commonly used for SDR content widely
available online. To answer the question above, we tested four
representations: one linear and three display-encoded:

#### Linear:

linear RGB pixel values with BT.709 primaries, scaled between 0 and 1;

####  $\mu$-law:

a transfer function commonly used for perceptual audio encoding. Kalantari et
al. [11] proposed to encode HDR images using the $\mu$-law function:

$v_{\mu}(l)=\frac{\log(1+\mu\,l)}{\log(1+\mu)}$ (1)

with the constant $\mu=5000$. $l$ is a relative linear (RGB) color value
scaled to the range 0–1.

#### PQ (perceptual quantizer):

display-encoding for HDR content [19], used in video coding and display
standards. The PQ encoding was applied separately to each RGB channel. Input
linear values were given in absolute units and scaled to the range 0.005 to 10
000 (see Section 4.2).

#### PU21 (perceptually uniform transform):

a transform similar to PQ but based on more modern data. It was originally
proposed as an encoding of HDR images to be used with existing “standard”
dynamic range metrics [17, 3]. For the PU21 encoding we use the quadratic
function fitted to the original curve (which was derived numerically)

$v_{\text{PU21}}(L)=a\,(\log_{2}(L)-L_{\text{min}})^{2}+b\,(\log_{2}(L)-L_{\text{min}})\,,$
(2)

where $L$ is an absolute linear (RGB) color value (in the range 0.005–10 000)
and the fitted parameters are $a=0.001908$, $b=0.0078$ and
$L_{\text{min}}=\log_{2}(0.005)$. The encoded values are between 0 and 1. The
inverse is given by:

$v^{-1}_{\text{PU21}}(V)=2^{\frac{2\,a\,L_{\text{min}}-b+\sqrt{b^{2}+4\,a\,V}}{2\,a}}\,.$
(3)

The functions above were fitted to the PU21 encoding for a banding model
without the influence of glare (details in [17]) and are more computationally
efficient than the PQ encoding or the original source code for PU21. This
encoding was applied to each linear RGB channel in the same manner as PQ (see
Section 4.2).

The pixel encodings described above are plotted in Figure 1. It can be
observed that both PQ and PU21 share very similar shapes, but PQ does not
reach 0 for the minimum encoded value (0.005). The $mu$-law has a much
shallower shape for values below 1, which means these values are encoded with
less precision. Finally, linear encoding over-emphasizes large pixel values.

Figure 1: Perceptual encoding functions used to transform linear RGB color
values to an approximately perceptually uniform space. A logarithmic scale is
used for the x-axis to account for Weber’s law. Following BT.2100, the range
of interest for encoded values is between 0.005 and 10 000. Note, however,
that in our experiments, we used a smaller range between 0 and 4 000. Since
$\mu$-law encoding expects input values between 0 and 1, the values were
divided by 10 000 before passing to the function in Eq. (1).

###  3.2 Loss functions

In addition to the way pixels are encoded, the loss function is a key feature
of training neural networks. Most image restoration problems can be
effectively trained using L1 or L2 norms on pixel values as a loss function.
However, this is not necessarily true when pixel values are represented in
linear color spaces (e.g. RAW or HDR images), as they are perceptually
nonuniform — a small error at low brightness is more visible than the same
error at large brightness levels. This problem can be avoided by first
converting the pixel values we mean to compare to a color space that is more
perceptually uniform, so that the loss function becomes

$\ell=||p(f_{\bm{\phi}}(\bm{x}))-p(\bm{y})||_{1}$ (4)

where $f_{\bm{\phi}}(\cdot)$ is a network with the weights $\bm{\phi}$,
$\bm{x}$ is input, $\bm{y}$ is a reference (label) and the $p(\cdot)$ is a
perceptual encoding function such as $\mu$-law, PQ or PU21 (all introduced in
Section 3). One important consideration is that the values passed to the
function $p(\cdot)$ must be within the allowed range (e.g. greater than 0.005
for PQ and PU21). This means that out-of-range values must be clamped. Both PQ
and PU21 require absolute photometric values, which correspond to light
emitted from a display (e.g. between 0.005 and 10 000).

Another perceptual uniformity strategy is using the symmetric mean absolute
percentage error (SMAPE) as a loss:

$\ell=\left\lVert\frac{|f_{\bm{\phi}}(\bm{x})-\bm{y}|}{|f_{\bm{\phi}}(\bm{x})|+|\bm{y}|+\epsilon}\right\rVert_{1}\,,$
(5)

where $\epsilon$ is a small constant avoiding division by 0. Dividing by the
sum of absolute values makes the error relative and more perceptually uniform,
following Weber’s law.

##  4 Experiments

In this work, we focus on image restoration problems (denoising, deblurring,
and single-image super-resolution), which aim to generate a higher-quality
version of the input image. We exclude higher-level tasks (e.g. recognition)
as they involve very different losses to those used for restoration. We also
do not consider the tasks in which input and output images differ in dynamic
range — tone mapping and single-image HDR reconstruction, as these tasks would
only allow us to test either the representations (for tone mapping) or the
loss functions (single-image HDR reconstruction) but not both at once.

To compare the effectiveness of different loss functions and pixel encodings,
we train networks with HDR images on three tasks: image denoising, deblurring,
and super-resolution. For each task, we choose two different neural networks:
one a traditional technique and the second a more recent method. We follow the
training settings proposed in the original works. EDSR (2017) [14] and Real-
ESRGAN (2021) [23] are used for the task of image super-resolution (4x). The
downsampled images used for training were created using a bilinear filter. For
image deblurring, we train GFNNet (2018) [29] and MirNet-v2 (2022) [27]. The
networks are trained on images affected by Gaussian blur with sigma values of
8 in both directions. DnCNN (2017) [28] and SADNet (2020) [5] are used for
denoising of HDR images. Physically plausible simulated camera noise (photon
and readout noise) was added to the training images [9].

All networks are trained using the code provided by the authors on their
GitHub repositories. We adopted the same training schemes as the original
papers to train each network, including the optimizer and learning rate. We
trained each network for a different number of epochs to ensure that
convergence was reached. EDSR was trained for 3k epochs, 1.5k epochs for Real-
ESRGAN, 1k epochs for MirNet-v2, GFNNet, DnCNN, and SADNet.

###  4.1 Datasets

We used 106 HDR images from the HDR Photographic Survey [8], as well as HDR
videos from the SJTU HDR Video Sequences [22]. As frames originating from the
same video share similarities, we used one demo HDR frame provided for each
video222The frames can be downloaded from the website., adding an additional
16 reference images to the dataset. 60% of images were selected for training,
20% for validation, and the remaining 20% were used for testing. To verify the
networks’ generalization ability to exposure variations, the testing data set
was augmented five-fold by multiplying it with a random exposure coefficient
drawn from a uniform distribution between 0.1 and 0.9. We did not directly use
RAW images in our experiments because they tend to contain more noise than
their HDR counterparts (merging multiple exposures can radically reduce
noise). However, the simulated noise used for denoising and the exposure
augmentation technique made the prepared HDR images very close to demosaiced
RAW images.

###  4.2 HDR/RAW training cases

Table 1: The 8 combinations of pixel encoding and loss functions tested in our experiments. Label | Pixel encoding | Loss function
---|---|---
Linear-L1 | Linear | L1
PQ-L1 | PQ | L1
PU21-L1 | PU21 | L1
$\mu$-L1 |  $\mu$-law (mu-law) | L1
Linear-PQ | Linear | PQ
Linear-PU21 | Linear | PU21
Linear-$\mu$ | Linear |  $\mu$-law (mu-law)
Linear-smape | Linear | SMAPE

We tested 8 combinations of pixel encodings and loss functions, listed in
Table 1. All perceptual pixel encodings (PQ, PU21 and $\mu$-law, see Section
3) were used with a regular L1 loss as the encoded values are expected to be
approximately perceptually uniform and so should not require a custom loss
function. However, when no encoding was used (labeled as _linear_), we used
either an L1 loss or one of the losses described in Section 3.2. We also
experimented with losses that were a sum of two terms, such as L1 and PQ-
encoded L1, but they performed no better than individual loss terms. Figure 2
shows a diagram of our training setup.

Since PQ and PU21 both require pixel values in absolute units (of light, as
emitted by a display), the linear pixel values were scaled to the range
0.005–4 000. Although both encodings allow values up to 10 000 (nit), 4 000
was selected as being more representative of realistic peak luminance values
for a high-quality HDR display. For other pixel encodings, the linear color
values were scaled in the 0–1 range. By following these steps, we ensure all
values passed to the networks (for every encoding) were between 0 and 1, as is
typical for training.

Figure 2: Training on HDR/RAW images. Training pairs were first (optionally)
encoded from a linear to a perceptually uniform color space using PQ, PU21 or
a $\mu$-law transfer function (explained in Section 3). If no pixel encoding
was applied (linear), one of the loss functions from Section 3.2 was used to
account for the perceptual non-uniformity.  Figure 3: Single-image super-
resolution results for the two networks (columns) and three metrics (rows).
The violin shape represents the distribution across the testing set. A thin
black line shows the range of quality scores (excluding outliers), and the
thicker line shows the region between the 25th and 75th percentiles. The dot
and numerical value represent the median. Red horizontal lines above the
violin plots denote the groups of conditions for which we have no evidence of
statistically significant differences. Results are sorted from the best on the
left to the worst on the right. The colors are different for each combination
of representation and loss and are consistent across plots. Figure 4: Example
results for single-image super-resolution with EDSR [14]. The numbers in
parentheses show PSNR, SSIM, and ColorVideoVDP quality values (the higher, the
better) for the reconstructed image. Figure 5: Example results for single-
image super-resolution with Real-ESRGAN [23].

##  5 Results

To evaluate the results (see Table 2), we used three image quality metrics:
PSNR, SSIM and ColorVideoVDP333ColorVideoVDP GitHub repository. PSNR and SSIM
were selected as the two most widely used metrics. ColorVideoVDP was selected
because it natively supports HDR images and was designed to compare color
images and videos. To adapt PSNR and SSIM to HDR images, we used the PU21
encoding [17], i.e. PSNR was computed on the PU21-encoded RGB pixels. SSIM was
computed on the luma channel of the PU21-encoded pixels.

###  5.1 Single-image super-resolution

The numerical results for the two single-image super-resolution methods can be
found in the top row of Table 2, in Figure 3, and examples of visual results
in Figure 4 and Figure 5. In Figure 3 we visualize the distribution of the
quality scores across the entire test set and also report the results of pair-
wise t-tests (two-tailed, $\alpha=0.05$, $N=144$). The violin plots are sorted
left-to-right from the best-performing condition to the worst. Red horizontal
lines connecting two or more violin plots signify no evidence of statistically
significant differences within that group. Conversely, conditions that are not
joined by a common line were found to have performances that differ at a
statistically significant level.

The results in Figure 3 indicate a substantial gain in performance of both
super-resolution methods (10–15 dB, as compared to Linear-L1) when one of the
perceptual pixel representations is used. However, we have no statistical
evidence showing that one perceptual representation is better than the others.
Most perceptual loss functions improve the results of Real-ESRGAN as compared
to plain L1 (1–3 dB), but they also make the results worse for ESDR. The
visual results in Figure 4 and Figure 5 demonstrate that all trained networks
can produce sharp images (as compared to bilinear upsampling), but some result
in incorrect colors, in particular in bright areas. The networks that used
perceptual pixel encodings do not suffer from these color artifacts.

Figure 6: Deblurring results for the two networks (columns) and three metrics
(rows). A few configurations are missing for GFNet as network training failed
to converge for those. The notation is the same as in Figure 3.

###  5.2 Deblurring

Deblurring results are different for the two tested methods and will be
discussed separately. GFNNet failed to train and converge for several of the
tested loss functions; it did, however, converge for all perceptual pixel
encodings. As shown in Figure 6 and Table 2, the best performance was obtained
using either the $\mu$-law or PU21 pixel encodings, resulting in an
improvement of about 3.8 dB compared to the plain Linear-L1 configuration. The
improvement was smaller (but still significant) for the PQ pixel encoding and
linear encoding using the $\mu$-law loss function. Visually (see
supplementary, Figure 10) the Linear-L1 configuration resulted in artifacts in
the form of vertical stripes. These artifacts were also present but less
noticeable for other configurations. The PQ-L1 format resulted in a slightly
softer image.

Deblurring with MirNet-v2 produced better results than for GFNNet. Similarly
as for GFNNet, the worst results were achieved by the plain linear-L1
configuration. The best results were obtained by a linear encoding with the
$\mu$-law loss, PQ and $\mu$-law pixel encodings (gain of 2–3 dB).
Interestingly, the PU21 pixel encoding resulted in worse results (see
supplementary Figure 10, note the visual results’ softer appearance), though
the reason is unclear.

Figure 7: Denoising results for the two networks (columns) and three metrics
(rows) The notation is the same as in Figure 3.

| PU21-L1 | PQ-L1 |  $\mu$-L1 | Linear-L1 | Linear-PU21 | Linear-PQ | Linear-$\mu$ | Linear-SMAPE | Naive
---|---|---|---|---|---|---|---|---|---
Super-resolution | EDSR | SSIM | 0.96 | 0.96 | 0.96 | 0.74 | 0.51 | 0.39 | 0.27 | 0.39 | 0.95
PSNR | 35.91 | 35.86 | 35.74 | 21.09 | 23.576 | 16.82 | 14.34 | 15.16 | 34.18
CVVDP | 8.40 | 8.37 | 8.38 | 6.89 | 7.22 | 6.34 | 5.58 | 5.81 | 7.99
Real-ESRGAN | SSIM | 0.95 | 0.95 | 0.96 | 0.84 | 0.86 | 0.86 | 0.87 | 0.88 | 0.95
PSNR | 35.45 | 35.37 | 34.97 | 25.55 | 26.80 | 28.94 | 28.96 | 28.28 | 34.18
CVVDP | 8.33 | 8.33 | 8.38 | 7.29 | 7.44 | 7.82 | 7.92 | 7.57 | 7.99
Deblurring | GFNNet | SSIM | 0.94 | 0.93 | 0.92 | 0.83 | N/A | N/A | 0.88 | N/A | 0.87
PSNR | 33.52 | 32.57 | 33.65 | 29.98 | N/A | N/A | 32.80 | N/A | 27.30
CVVDP | 7.59 | 7.27 | 7.66 | 7.09 | N/A | N/A | 7.41 | N/A | 5.51
MirNet-v2 | SSIM | 0.93 | 0.95 | 0.94 | 0.93 | 0.93 | 0.95 | 0.95 | 0.95 | 0.87
PSNR | 32.99 | 34.65 | 34.10 | 31.98 | 32.06 | 33.57 | 34.80 | 34.10 | 27.30
CVVDP | 7.40 | 7.89 | 7.77 | 7.31 | 7.35 | 7.73 | 7.93 | 7.72 | 5.51
Denoising | DnCNN | SSIM | 0.85 | 0.84 | 0.86 | 0.80 | 0.83 | 0.83 | 0.82 | 0.80 | 0.24
PSNR | 23.10 | 22.44 | 23.65 | 17.93 | 20.56 | 21.03 | 20.56 | 18.57 | 10.60
CVVDP | 6.07 | 5.96 | 6.15 | 5.68 | 5.87 | 5.74 | 5.76 | 5.66 | 4.37
SADNet | SSIM | 0.90 | 0.91 | 0.90 | 0.88 | 0.89 | 0.89 | 0.88 | 0.89 | 0.24
PSNR | 23.94 | 27.76 | 26.47 | 24.62 | 26.03 | 25.32 | 26.32 | 26.08 | 10.60
CVVDP | 6.52 | 6.86 | 6.61 | 6.71 | 6.73 | 6.68 | 6.67 | 6.71 | 4.37

Table 2: The median values for evaluation results. The last column ’Naive’ is
bilinear-upsampling results for super-resolution experiments, original blurry
images for deblurring experiments, and the original noisy images for denoising
experiments. The best performing scores are annotated in red, while the second
best one is in blue. The ”N/A” entries correspond to the cases for which the
training failed.

###  5.3 Denoising

For denoising networks, PSNR had more distinctive and slightly different
results than ColorVideoVDP and SSIM, as seen in Figure 7 and Table 2. We
investigated this discrepancy further by comparing pairs of conditions for
which the metric predictions differed the most. We observed that both
denoising methods, in particular DnCNN, resulted in images that differed in
brightness, tone, color, and contrast from the original. Even though such
changes may be less perceptually objectionable (and difficult to notice
without a reference), PSNR was very sensitive to them. Both ColorVideoVDP and
SSIM put more emphasis on structural distortions and seemed to correlate
better with perceived quality.

For both denoising methods, perceptual pixel encodings produced the best
results in most cases. $\mu$-law and PU21 performed the best for DnCNN,
followed by PQ, which performed slightly worse according to PSNR and
ColorVideoVDP. In the case of SADNet, ColorVideoVDP could not distinguish
between any of the configurations (no evidence of statistical differences).
While $\mu$-law consistently produced the highest metric scores for SADNet,
PU21 was the second best in terms of SSIM, but the last in terms of PSNR. We
attribute this discrepancy to color shifts (SSIM is insensitive to color
changes). The visual results (see supplementary, Figure 10) show changes in
color for Linear-PQ and Linear-PU. As a result, these two configurations have
lower PSNR than the input noisy image (but higher ColourVideoVDP index because
of the denoising).

##  6 Conclusions

The results across six tested methods clearly indicate that networks for image
restoration should be trained on HDR/RAW images that are encoded using one of
the perceptual transforms: PU21, PQ, or $\mu$-law. We did not find sufficient
evidence to recommend one transform over another across all applications.
However, we observed slightly worse performance of PQ for GFNNet, and PU21 for
MirNet-v2 and SADNet. $\mu$-law encoding was robust across different methods
but did not always produce the best results. Both $\mu$-law and PU21 use
simpler and more computationally efficient formulas. PQ fails to scale the
resulting values to a 0–1 range (see Figure 1), which may cause practical
issues. Unlike the $\mu$-law, PQ and PU21 were derived from psychophysical
data and have stronger perceptual bases. As the differences in performance
between the three transforms are mostly small, we recommend using any of them,
as this offers a substantial gain in performance over not using them at all.
The actual gain in performance is application-specific and can be as large as
10–15 dB as compared to the “default” approach of training linear color values
with an L1 loss. Using linear color values in tandem with a loss function that
employs perceptual encoding seems to be less effective across the tested
applications.

The empirical findings in this work are relevant as they show that very small
changes in training strategy (“one-liners”) can substantially improve the
results of neural networks trained for popular tasks on HDR or RAW images.
Moreover, the winning strategy of using perceptual pixel coding is rarely used
in practice. More often, networks are trained in linear color spaces using an
L1 cost function [26, 6], or at most, use a perceptual encoding loss [7, 11,
18] — both strategies shown in this work to under-perform. We hope our
findings help practitioners in designing better training configurations for
HDR/RAW images.

## References

  * Aguerrebere et al. [2013] Cecilia Aguerrebere, Julie Delon, Yann Gousseau, and Pablo Musé.  _Study of the digital camera acquisition process and statistical modeling of the sensor raw data_.  2013\.  hal-00733538.
  * Anderson et al. [1996] Matthew Anderson, Ricardo Motta, Srinivasan Chandrasekar, and Michael Stokes.  Proposal for a standard default color space for the internet - sRGB.  In _Color and Imaging Conference_ , pages 238–245. Society for Imaging Science and Technology, 1996.
  * Aydın et al. [2008] Tunç O. Aydın, Rafal Mantiuk, and Hans-Peter Seidel.  Extending quality metrics to full luminance range images.  In _Human Vision and Electronic Imaging_ , page 68060B. Spie, 2008\.
  * Burger et al. [2012] Harold C Burger, Christian J Schuler, and Stefan Harmeling.  Image denoising: Can plain neural networks compete with BM3D?  In _CVPR_ , pages 2392–2399. IEEE, 2012.
  * Chang et al. [2020] Meng Chang, Qi Li, Huajun Feng, and Zhihai Xu.  Spatial-adaptive network for single image denoising.  In _Computer Vision – ECCV 2020_ , pages 171–187. Springer, 2020\.
  * Chen et al. [2018] Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun.  Learning to see in the dark.  In _CVPR_ , page 3291–3300, Salt Lake City, UT, 2018. IEEE.
  * Eilertsen et al. [2017] Gabriel Eilertsen, Joel Kronander, Gyorgy Denes, Rafał K. Mantiuk, and Jonas Unger.  HDR image reconstruction from a single exposure using deep CNNs.  _ACM Transaction on Graphics_ , 36(6):Article 178, 2017.
  * Fairchild [2007] Mark D. Fairchild.  The HDR Photographic Survey.  _Color and Imaging Conference_ , 15(1):233–238, 2007.
  * Foi et al. [2008] Alessandro Foi, Mejdi Trimeche, Vladimir Katkovnik, and Karen Egiazarian.  Practical Poissonian-Gaussian noise modeling and fitting for single-image raw-data.  _IEEE Transactions on Image Processing_ , 17(10):1737–54, 2008.
  * Hwang et al. [2011] Youngbae Hwang, Jun-Sik Kim, and In So Kweon.  Difference-based image noise modeling using skellam distribution.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 34(7):1329–1341, 2011.
  * Kalantari and Ramamoorthi [2017] Nima Khademi Kalantari and Ravi Ramamoorthi.  Deep high dynamic range imaging of dynamic scenes.  _ACM Transactions on Graphics_ , 36(4):1–12, 2017\.
  * Kim et al. [2019] Soo Ye Kim, Jihyong Oh, and Munchurl Kim.  Deep SR-ITM: Joint learning of super-resolution and inverse tone-mapping for 4k UHD HDR applications.  In _CVPR_ , pages 3116–3125, 2019.
  * Kunkel and Reinhard [2010] Timo Kunkel and Erik Reinhard.  A reassessment of the simultaneous dynamic range of the human visual system.  In _Proceedings of the 7th Symposium on Applied Perception in Graphics and Visualization_ , 2010.
  * Lim et al. [2017] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee.  Enhanced deep residual networks for single image super-resolution.  In _CVPR workshops_ , pages 136–144, 2017.
  * Liu et al. [2020] Lin Liu, Xu Jia, Jianzhuang Liu, and Qi Tian.  Joint Demosaicing and Denoising With Self Guidance.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2237–2246. IEEE, 2020.
  * Mantiuk et al. [2004] Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, and Hans-Peter Seidel.  Perception-motivated high dynamic range video encoding.  _ACM Transactions on Graphics (TOG)_ , 23(3):733–741, 2004.
  * Mantiuk and Azimi [2021] Rafal K. Mantiuk and Maryam Azimi.  PU21: A novel perceptually uniform encoding for adapting existing quality metrics for HDR.  In _2021 Picture Coding Symposium (PCS)_ , pages 1–5. IEEE, 2021\.
  * Mildenhall et al. [2022] Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul P. Srinivasan, and Jonathan T. Barron.  NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 16169–16178. IEEE, 2022.
  * Miller et al. [2013] S. Miller, M. Nezamabadi, and S. Daly.  Perceptual Signal Coding for More Efficient Usage of Bit Codes.  _SMPTE Motion Imaging Journal_ , 122(4):52–59, 2013.
  * Nam et al. [2016] Seonghyeon Nam, Youngbae Hwang, Yasuyuki Matsushita, and Seon Joo Kim.  A holistic approach to cross-channel image noise modeling and its application to image denoising.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1683–1691, 2016.
  * Reinhard et al. [2010] Erik Reinhard, Wolfgang Heidrich, Paul Debevec, Sumanta Pattanaik, Greg Ward, and Karol Myszkowski.  _High dynamic range imaging: acquisition, display, and image-based lighting_.  Morgan Kaufmann, 2010.
  * Song et al. [2016] Li Song, Yankai Liu, Xiaokang Yang, Guangtao Zhai, Rong Xie, and Wenjun Zhang.  The SJTU HDR video sequence dataset.  In _Proceedings of International Conference on Quality of Multimedia Experience (QoMEX 2016)_ , page 100, 2016.
  * Wang et al. [2021] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan.  Real-ESRGAN: Training real-world blind super-resolution with pure synthetic data.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 1905–1914, 2021.
  * Xie et al. [2012] Junyuan Xie, Linli Xu, and Enhong Chen.  Image denoising and inpainting with deep neural networks.  _Advances in neural information processing systems_ , 25, 2012.
  * Xing and Egiazarian [2021] Wenzhu Xing and Karen Egiazarian.  End-to-End Learning for Joint Image Demosaicing, Denoising and Super-Resolution.  In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3506–3515. IEEE, 2021.
  * Xu et al. [2020] Xiangyu Xu, Yongrui Ma, Wenxiu Sun, and Ming-Hsuan Yang.  Exploiting Raw Images for Real-Scene Super-Resolution.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pages 1–1, 2020.
  * Zamir et al. [2022] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao.  Learning enriched features for fast image restoration and enhancement.  _IEEE transactions on pattern analysis and machine intelligence_ , 45(2):1934–1948, 2022.
  * Zhang et al. [2017] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang.  Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising.  _IEEE transactions on image processing_ , 26(7):3142–3155, 2017.
  * Zhang et al. [2018] Xinyi Zhang, Hang Dong, Zhe Hu, Wei-Sheng Lai, Fei Wang, and Ming-Hsuan Yang.  Gated fusion network for joint image deblurring and super-resolution.  _arXiv preprint arXiv:1807.10806_ , 2018.

Supplementary Material

Figure 8: Example results for deblurring with GFNNet. The results are missing
for Linear-PQ, Linear-PU21, and Linear-SMAPE, as the networks failed to
converge for these configurations.

Figure 9: Example results for deblurring with MirNet-v2

Figure 10: Example results for denoising with DnCNN (top) and SADNet (bottom).

##  7 Visual results

This supplementary material contains additional visual results for the
networks tested in our work.

  * •

_Deblurring_ , GFNNet - Figure 10

  * •

_Deblurring_ , MirNet-v2 - Figure 10

  * •

_Denoising_ , GFNNet - Figure 10, top

  * •

_Denoising_ , SADNet - Figure 10, bottom


# Memory-Inspired Temporal Prompt Interaction for Text-Image Classification

Xinyao Yu
Zhejiang University
<EMAIL_ADDRESS>Hao Sun
Zhejiang University
<EMAIL_ADDRESS>Ziwei Niu
Zhejiang University
<EMAIL_ADDRESS>Rui Qin
Zhejiang University
<EMAIL_ADDRESS>Zhenjia Bai
Zhejiang University
<EMAIL_ADDRESS>Yen-wei Chen
Ritsumeikan University
<EMAIL_ADDRESS>Lanfen Lin
Zhejiang University
<EMAIL_ADDRESS>
###### Abstract

In recent years, large-scale pre-trained multimodal models (LMM) generally
emerge to integrate the vision and language modalities, achieving considerable
success in various natural language processing and computer vision tasks. The
growing size of LMMs, however, results in a significant computational cost for
fine-tuning these models for downstream tasks. Hence, prompt-based interaction
strategy is studied to align modalities more efficiently. In this contex, we
propose a novel prompt-based multimodal interaction strategy inspired by human
memory strategy, namely Memory-Inspired Temporal Prompt Interaction (MITP).
Our proposed method involves in two stages as in human memory strategy: the
acquiring stage, and the consolidation and activation stage. We utilize
temporal prompts on intermediate layers to imitate the acquiring stage,
leverage similarity-based prompt interaction to imitate memory consolidation,
and employ prompt generation strategy to imitate memory activation. The main
strength of our paper is that we interact the prompt vectors on intermediate
layers to leverage sufficient information exchange between modalities, with
compressed trainable parameters and memory usage. We achieve competitive
results on several datasets with relatively small memory usage and 2.0M of
trainable parameters (about 1% of the pre-trained foundation model).

##  1 Introduction

Modern Internet platforms, encompassing social media and e-commerce, host a
myriad of content expressed across various modalities, predominantly vision
and language. Harnessing information from different modalities has
demonstrated its potential to enhance performance on diverse multimodal tasks,
such as image-text classification [5], recommendation [20], and sentiment
analysis [3]. Multimodal learning commonly employs a strategy of interacting
representations from involved modalities. Previous multimodal interaction
methods [31, 32, 6, 26] involves extracting unimodal features separately with
distinct backbones, and then blends these unimodal representations using a
fusion module. However, this approach faces challenges in fully addressing
comprehensive inter- and intra-modality relationships, resulting in
insufficient inter-modality interaction.

As large-scale multimodal models (LMM) gain prominence in multimodal learning,
fine-tuning methods [11, 9, 34, 33, 10] for downstream tasks become prevalent,
albeit with a significant drawback of large memory usage. To mitigate this
issue when exploiting pre-trained models in downstream tasks, a prompt-based
cross-modal interaction strategy [17, 14] has emerged, albeit still in its
infancy, as depicted in Fig 1 (a) and (b). The prompt-based strategy leverages
prompt vectors to facilitate modality interaction on intermediate layers and
mid-level features of each modality. This approach enables inter-modality
information exchange through trainable prompts, reducing memory usage due to
the low spatial complexity of prompt operations. However, existing methods
lack direct interaction between trainable prompts, leading to insufficient
interaction between modalities. The question arises: Is it possible for
prompt-based interaction strategy to achieve sufficient inter-modality
information exchange parameters through direct interaction of trainable
prompts?

(a) The first proposed prompt-based interaction strategy.[17]

(b) Existing prompt-based interaction strategy.[14]

(c) Our proposed prompt interaction strategy MITP.

Figure 1: Comparisons among existing prompt-based interaction strategies
(a)/(b), and our proposed strategy (c). (a) extracts unimodal representations
separately and then utilizes prompt vectors to align the extracted
representations with PLM. (b) leverages prompt vectors for each branch of the
model, and utilizes the prompts to deliver information from the other branch
on intermediate layers. In our proposed MITP (c), prompt vectors are leveraged
for each modality on intermediate layers, and are then blended to generate
prompts of the next layer. Direct interactions of prompts from different
branches allow interactions between modalities.

Following this trend, we propose a novel multimodal interaction method, namely
Memory-Inspired Temporal Prompt Interaction (MITP). The proposed prompt-based
interaction strategy is as shown in Fig 1 (c). The proposed method is inspired
by human memory mechanism, to be specific, working memory [2] and memory
activation [4]. In working memory, information from uni-modality is acquired
and registered in sensory buffers, and then consolidated into long-term memory
through inter-modality interaction. Long-term memory can be activated when
processing temporal information. We leverage temporal prompts on intermediate
layers to acquire temporal uni-modal information, imitating the acquiring
stage. After acquiring stage, temporal prompts are fed into a memory hub for
prompt interaction, imitating the consolidation and activation stage. To
imitate memory consolidation, we calculate similarity between temporal prompts
of each modality to highlight memory logits of importance. Then, to imitate
memory activation, we calculate the activation vector of prompts from each
modality, and blend the activated prompts from each modality, so as to
generate temporal prompts for the next layer. The interaction of temporal
prompts enables a two-way information flow between both modalities without
participation of modality features. Note that only the prompts and projection
functions are trainable, while the rest of the model is frozen, reducing
trainable parameters and memory usage significantly. Despite the variety of
downstream tasks in multimodal learning, we focus on text-image classification
in this paper. We achieve competitive results on several public datasets on
image-text classification, such as UPMC-Food101 [27], MM-IMDB [1], and SNLI-VE
[29], with only 2.0M of trainable parameters and relatively small memory
usage.

In summary, our contribution can be shown as follows:

  1. 1.

We propose a new form of multimodal interaction framework, namely Memory-
Inspired temporal prompt interaction, which enables sufficient information
flow between both modalities through direct prompt interaction.

  2. 2.

We propose temporal prompts for information storage on intermediate layers,
and a memory hub for temporal prompts interaction and generating prompts for
next layer, to imitate the mechanism of human memory. We further compress the
number of trainable parameters by a calculative similarity-based strategy for
prompt interaction, rather than a strategy with learnable parameters.

  3. 3.

Our proposed strategy significantly reduces the trainable parameters and
memory usage of the model, yet we achieve competitive results on several
public multimodal classification datasets. In particular, we achieve the best
performance among prompt-based methods on UPMC-Food101 and SNLI-VE and surpass
all other methods with the same foundation models on all three datasets.

Figure 2: The pipeline of our proposed method. We utilize pre-trained
foundation model in frozen for basic feature extraction of image and text
branches, and leverage temporal prompts on intermediate layers to store
information on temporal layer and act as media for information exchange.
Temporal prompts of different modalities are blended in memory hub to generate
prompts for the next layer. Only the prompts and the memory hub are trainable,
requiring backward propagation; while the pre-trained foundation model of
model is in frozen, which do not participate in backward propagation.

##  2 Related works

In this section, we introduce previous works on multimodal interaction.

###  2.1 Multimodal interaction

Multimodal interaction aims to leverage information from different modalities
for multimodal tasks. Early multimodal interaction strategy, such as TFN [31]
and CentralNet [26], blends extracted features of each modality via tensor
operation. Sun et al. [25] proposed CubeMLP to fuse the extracted features
with an MLP-based fusion module. Although methods using tensor operation have
achieved great success, the drawback of high space complexity and insufficient
inter-modality interaction still exists. Therefore, attention-based methods,
such as PixelBERT [8], MBT [22] are proposed to explore inter-modality
correlation by leveraging cross modality attention to capture inter-modality
correlations. Following this trend, Hazarika et al. [6] proposed MISA to
disentangle modality-specific and -invariant information for comprehensive and
disentagled view of multimodal data. Hence, numerous works hammer at
strengthening inter-modality relationship. CLMLF [15], HyCon [21], and
TupleInfoNCE [19] utilize contrastive learning strategy to explore inter-
modality, inter-class and inter-sample relationships. As LMMs become a common
practice, Fu et al. [5] tailored an attention-based fusion module for the
backbone of CLIP [24].

###  2.2 Prompt-based strategy

Recent research has made impressive progress in large-scale multimodal pre-
training [24, 18]. Prompt-tuning methods are proposed to finetune pre-trained
large-scale multimodal models for downstream tasks. CoOp [34] finetunes CLIP
at its language branch for few-shot transfer by a set of learnable prompt
vectors, and and Co-CoOp [33] promotes generalization ability of CoOp by
input-conditional prompt vectors. MaPLe [10] extends prompt finetuning on CLIP
to both textual and visual branches. In the context of the rapid growth of
model size, prompt-based interaction methods are proposed to seek efficient
and flexible methods other than finetuning. Liang et al. [17] proposed
PromptFuse to use prompt vectors to align the vision and language modalities,
feeding prompt vectors and extracted features of each modality into PLMs as
input. Li et al.[14] proposed PMF to perform multimodal interaction on
intermediate layers via interacting prompts.

This paper is in line with prompt-based interaction strategy. We store layer-
wise information in temporal prompts, and perform direct interactions between
prompts with memory-inspired strategy to generate temporal prompts for the
next layer, to leverage in-depth interaction on mid-level features on
intermediate layers.

##  3 Method

In this section, we present our proposed method called Memory-Inspired
Temporal Prompt Interaction for image-text classification.

###  3.1 Problem statement

Given an image-text pair
$\mathit{X_{\mathit{i}}}=\left\\{\mathcal{V_{\mathit{i}}},\mathcal{T_{\mathit{i}}}\right\\}$
containing an image $\mathcal{V_{\mathit{i}}}\in\mathbb{R^{\mathit{c\times
h\times w}}}$ and a textual description
$\mathcal{T_{\mathit{i}}}\in\mathbb{R}^{\mathit{d_{i}}}$, where $c$ denotes
channel, $h*w$ denotes the image size, and $d_{i}$ denotes the length of text.
$\mathit{Y_{i}}\in\\{1,2,...,K\\}$ denotes the class label. The goal is to
learn a classifier to predict the class label $\mathit{Y}_{i}$ of each image-
text pair
$\left\\{\mathcal{V_{\mathit{i}}},\mathcal{T_{\mathit{i}}}\right\\}$.

###  3.2 Model overview

The pipeline of the model is as shown in Fig 2. This work adopts LMM in frozen
as foundation model to extract base features of each modality. The whole
structure imitates working memory and memory activation of human memory, which
can be divided into two stages: acquiring stage, and consolidation and
activation stage. Temporal prompts are leveraged on several intermediate
layers to acquire temporal information of this store. Then we consolidate the
acquired temporal information on the basis of similarity. Finally we produce
an activation mask to decide the activation rate of each logit, and add the
consolidated information from the other modality, so as to generate prompts
for the next layer. In practice, we select several layers to perform prompt
interaction, denoted as interaction layer, while those layers without prompts
are denoted as extraction layer. The consolidation and activation stages are
performed in a memory hub associated to corresponding interaction layer. The
selection of interaction layers and similarity type will be discussed in 2.
Note that the inter-modality interactions are conducted by prompt vectors,
without participation of uni-modal features.

###  3.3 Temporal prompt

In this section, we introduce the acquiring stage. The adopted LMM foundation
model contains a visual branch $Encoder_{v}$ and a textual branch
$Encoder_{t}$. Given a pair of image and tokenized text
$\left\\{\mathcal{V_{\mathit{i}},T_{\mathit{i}}}\right\\}$, we first feed the
image $\mathcal{V_{\mathit{i}}}$ into $Encoder_{v}$ and the tokenized text
$\mathcal{T_{\mathit{i}}}$ into $Encoder_{t}$ for basic feature extraction.
The image feature and text feature on intermediate layer $l$ are denoted as
$u_{m}^{l}$, where $\mathit{m}\in\left\\{v,t\right\\}$ denotes modality.
Temporal prompts $\mathit{p_{m}^{l}}$ are introduced on interaction layers of
each modality branch, where $\mathit{m}\in\left\\{v,t\right\\}$ denotes
modality and $\mathit{l}$ refers to the interaction layer.

Suppose this strategy starts at layer $\mathit{l}_{0}$. The first temporal
prompt for text $\mathit{p_{t}^{l_{0}}}$ is initialized to be a trainable
vector with Gaussion distribution. The first temporal prompt for image
$\mathit{p_{v}^{l_{0}}}$ is produced by $\mathit{p_{t}^{l_{0}}}$ through MLP
projection, as follows:

$p_{v}^{l_{0}}=W\cdot p_{t}^{l_{0}}+b.$ (1)

On each interaction layer of branch $Encoder_{m}$, we first concatenate the
modality-input ${u}_{m}^{l}$ and the corresponding ${p_{m}^{l}}$. $p_{m}^{l}$
provides information from previous layers of both modalities, and acts as
temporal memory prompt for ${u}_{m}^{l}$. Then the concatenated input is fed
into the pre-trained encoder layer $l$ as:

$[\hat{p}_{m}^{l};{u}_{m}^{l+1}]=Encoder_{m}^{l}([p_{m}^{l};{u}_{m}^{l}]),$
(2)

where $[\cdot;\cdot]$ refers to concatenation operation, and
$[\hat{p}_{m}^{l};{u}_{m}^{l+1}]$ is the output. Through this process,
information of layer $l$ from modality $m$ is acquired and registered in
temporal prompt. Similarly, the input for the next layer is
$[p_{m}^{l+1};{u}_{m}^{l+1}]$. Each interaction layer is associated to a
memory hub, to integrate the information from both modalities and generate
temporal prompts ${p_{v}^{l+1},p_{t}^{l+1}}$ for the next layer, as:

$\left\\{p_{v}^{l+1},p_{t}^{l+1}\right\\}=\text{MemoryHub}(p_{v}^{l},p_{t}^{l}).$
(3)

###  3.4 Memory hub

In this section, we introduce the memory hub, which imitates the consolidation
and activation stages of human memory. We first project $p_{m^{\prime}}^{l}$
into the space of $p_{m}^{l}$ through a two-layer MLP, denoted as
$\tilde{p}_{m^{\prime}}^{l}$. In human memory consolidation, analogical
mapping connection is leveraged to discover correspondences between tokens,
working as $Mapping(A,B)=f_{M}(sim(A,B))$, where $f_{M}$ means the mapping
function, and $sim$ means similarity function.[4] Therefore, we mimic memory
consolidation by imitating analogical mapping connection, where logits with
high correspondence are to be consolidated. We consider two types of
correspondences between temporal prompts: those intra-modality and those
inter-modality. The correspondence mapping connections are calculated as:

$\displaystyle map_{m\leftarrow m}$ $\displaystyle=RELU(sim(W\cdot
p_{m}^{l},p_{m}^{l})),$ (4) $\displaystyle map_{m\leftarrow m^{\prime}}$
$\displaystyle=RELU(sim(p_{m}^{l},\tilde{p}_{m^{\prime}}^{l})),$

where $W$ denotes a two-layer MLP projection. We leverage $map_{m\leftarrow
m}$ to highlight important logits of $p_{m}^{l}$, and leverage
$map_{m\leftarrow m^{\prime}}$ to highlight logits of
$\tilde{p}_{m^{\prime}}^{l}$ with high correspondence to $p_{m}^{l}$. In human
memory activation, the feature unit activation is calculated by
$activation_{i}=n_{i}/max(n_{j})$, where $n_{i}$ denotes feature input for
unit $i$ and $max(n_{j})$ denotes the maximum feature input.[4] In our work,
activation of each logit is considered on the basis of mapping connections
$map_{m\leftarrow m}$ and $map_{m\leftarrow m^{\prime}}$, because the mapping
connections represent correspondence relationships. Moreover, we consider
activation in soft max manner instead of hard max. The activation of prompts
is considered in intra- and inter-modality types, calculated as:

$\displaystyle z_{m}$ $\displaystyle=SoftMax(map_{m\leftarrow m}),$ (5)
$\displaystyle r_{m}$ $\displaystyle=SoftMax(map_{m\leftarrow m^{\prime}}),$

where $z_{m}$ denotes the intra-modality activation of $p_{m}^{l}$, and
$r_{m}$ denotes the inter-modality activation of $\tilde{p}_{m^{\prime}}^{l}$.
Finally we activate $p_{m}^{l}$ and $\tilde{p}_{m^{\prime}}^{l}$ to generate
$p_{m}^{l+1}$, as:

$p_{m}^{l+1}=z_{m}\cdot{p}_{m}^{l}+(1-z_{m})\cdot
r_{m}\cdot\tilde{p}_{m^{\prime}}^{l}.$ (6)

$p_{m^{\prime}}^{l+1}$ is generated by the same process as above. Through the
mapping connections and activation, we can consolidate information in
$p_{m}^{l}$ and integrate information from $p_{m^{\prime}}^{l}$. Note that he
trainable parameters are shared in all interaction layers, so that the number
of trainable parameters remains the same when the interaction layers pile up.

###  3.5 Predictions

For classification, we add hand-crafted text prompts with class labels
$y\in\left\\{1,2,...,K\right\\}$ having $K$ classes. The prediction is made by
the cosine similarity score with temperature $\tau$ between the final output
of the vision encoder $x$ and the text encoder $z$. The prediction logits
$\hat{y}=(\hat{y}_{1},...,\hat{y}_{K})$ is calculated as:

$p(\hat{y}\mid
x)=\frac{\mathrm{exp}(sim(x,z_{\hat{y}})/\tau)}{\sum_{1}^{K}\mathrm{exp}(sim(x,z_{i}))}.$
(7)

###  3.6 Loss function

As we conduct experiments on uni-label classification task and multi-label
classification task, we define loss for both tasks. For uni-label
classification task, we define the loss as:

$L_{uni}=CrossEntropy(\hat{y}_{i},y_{i}),$ (8)

where $y_{i}\in\left\\{0,1\right\\}$ is the ground truth label. For multi-
label classification task, we define the loss as:

$L_{multi}=MultiCrossEntropy(\hat{y_{i}},y_{i}).$ (9)

Table 1: Results on employed datasets compared with other methods. $MaPLe^{*}$ means that we re-implement MaPLe under our settings. Acc means the accuracy on UPMC-Food101 and SNLI-VE of uni-label classification task. F1-micro/F1-macro are the evaluation metrics on MM-IMDB dataset for multi-label classification task. MITP achieves competitive results and outperforms most late fusion methods as well as all prompt-based methods. Note that on SNLI-VE, we only use the hypothesis text as input for text branch, rather than both premise text and hypothesis text. Methods | UPMC-Food101 | MM-IMDB | SNLI-VE | Avg.
---|---|---|---|---
Acc(%) | F1-micro/macro | Acc($\%$) |
HUSE[23] | 92.30 | - | - | -
VisualBERT[9] | 92.30 | - | 75.06 |
Late Fusion[16] | - | 59.6 / 51.0 | - | -
DynMM[30] | - | 61.0 / 51.6 | - | -
UniT[7] | - | - | 73.16 | -
CMA-CLIP[5] | 93.10 | 65.3 / 52.7 | - | -
VilT[12] | 92.90 | - | - | -
MMBT[11] | 92.10 |  66.8 / 61.8 | 74.69 | 77.03
MaPLe*[10] | 90.80 | 60.9 / 51.2 | 71.52 | 72.79
P-CLIP | 90.30 | 60.0 / 50.6 | 70.56 | 72.05
PromptFuse[17] | 82.21 | 54.5 / 48.6 | 64.53 | 66.09
BlindPrompt[17] | 84.56 | 56.5 / 50.2 | 65.54 | 67.81
PMF[14] | 91.51 | 64.5 / 58.8 | 71.92 | 75.02
MITP-base(ours) | 93.95 |  65.9 / 56.3 | 73.45 | 76.17
MITP-small(ours) | 93.17 | 64.8 / 55.4 | 72.51 | 75.26

##  4 Experiments

In this section, we introduce our datasets employed and corresponding
evaluation metrics, and bring out our implementation details.

###  4.1 Datasets

We evaluate our method on three public datasets: UPMC-Food101 [27], MM-IMDB
[1], SNLI-VE [28].

  * •

UPMC-Food101 is a popular image-text classification dataset on food to
categorize food images and its paired textual recipe descriptions into 101
categories. The dataset consists of 90,840 image-text pairs in total,
including 67,971 for training and 22,715 for test.

  * •

MM-IMDB is a mainstream dataset for multi-label image-text classification on
movie. The task is to classify the movie into one or more of the 23 genres,
using the poster image and textual plot outline as image-text pairs. The
dataset contains a total of 25,956 image-text pairs, with 15,510 for training,
2,599 for validation and 7,779 for test.

  * •

SNLI-VE is a visual-entailment classification dataset built on the top of
Flickr30K and SNLI, in which each image-text pair includes an image premise
($P_{image}$) and a text hypothesis ($H_{text}$). The task is to reason the
semantic relationship (entailment, neutral, or contradiction) between
$P_{image}$ and $H_{text}$, where $entailment$ stands if there is enough
evidence in $P_{image}$ to deduct $H_{text}$ as True, $contradiction$ holds if
enough evidence exists in $P_{image}$ implying $H_{text}$ to be False;
Otherwise if the relationship is $neutral$, it means that the evidence in
$P_{image}$ is insufficient to draw a conclusion about $H_{text}$. The dataset
has 565,286 image-text pairs in total, with 529,527 for training, 17,858 for
validation and 17,901 for test. In this work, we only use the hypothesis
description as text input, in line with prior work [14].

###  4.2 Implementation details and evaluation metrics

  * •

Pre-trained foundation model and initialization. We use pre-trained CLIP
$ViT-L/14$[24] as foundation model for image encoder and text encoder in
frozen. The foundation model model is pre-trained on 400 billion WIT data. Our
first temporal prompt for text branch is initialized through a Gaussion
distribution ($mean=0,std=0.02)$), by which other prompts are generated.

  * •

Network training. We choose Adam optimizer in all experiments, with learning
rate set to 0.0002. The batch size ($bs$) is set to 10 for UPMC-Food101, 32
for MM-IMDB, and 64 for SNLI-VE. All our experiments are conducted on one
NVIDIA RTX A6000 GPU card. Our approach is implemented in the PyTorch
framework.

  * •

Metrics. We employ accuracy$(\%)$ as metric for uni-label classification task
on UPMC-Food101 and SNLI-VE datasets, and F1-micro/F1-macro as metric for
multi-label classification task on MM-IMDB dataset, following previous works
[5, 14, 30].

  * •

Interaction settings. We set the length of prompt vectors to 3 for UPMC-
Food101, 6 for SNLI-VE, and 2 for MM-IMDB. We leverage 3 interaction layers
empirically for the base MITP. We also propose a small version of the proposed
method, utilizing one interaction layer, namely MITP-small. The selection of
interaction layers will be discussed in 2.

##  5 Results and discussion

In this section, we report the performance of several baselines and analyze
the performance of our proposed MITP on public image-text classification
datasets, UPMC-Food101, MM-IMDB, and SNLI-VE.

###  5.1 Baselines

We implement several baseline models on our employed datasets and report the
results for comparison. Our baseline models are as follows:

Pre-trained LMM foundation model. We re-implement the employed pre-trained
foundation model with a linear probe under our settings and report the
results.

Pre-trained LMM foundation model with late fusion module. CMA-CLIP utilizes
the same foundation model as our proposed MITP, complementing unimodal
representations with an attention-based late fusion module. We report the
results of CMA-CLIP for comparison between our proposed MITP and late fusion
strategy.

Pre-trained LMM foundation model with prompt-tuning. MaPLe [10] is a
multimodal prompt-tuning method on the same LMM foundation model as ours,
utilizing a one-way projection from textual-branch prompts to vision-branch
prompts. We re-implement MaPLe under our experimental settings on our employed
image-text classification datasets and report the results. We also propose a
prompt-tuning CLIP without interactions between modalities, denoted as P-CLIP,
to compare our proposed method with prompt-tuning strategy. In P-CLIP, we
implement deep prompt-tuning on both vision and text branches, with the pre-
trained foundation model in frozen. All the prompt vectors are initialized
through a Gaussion distribution ($mean=0,std=0.02$). We analysis the
efficiency of trainable parameters and memory usage of both prompt-tuning
methods.

Existing prompt-based methods for multimodal interaction. PromptFuse [17],
BlindPrompt [17], and PMF [14] are existing prompt-based interaction methods.
We compare the results on employed datasets, and analyze the efficiency of
trainable parameters and memory usage.

Late fusion methods with a fusion module. [23, 16, 7, 13, 30] are late fusion
methods, we report the results of them for comparison.

(a) Comparisons on overall efficiency.

(b) Comparison on efficiency of trainable parameters.

(c) Comparison on efficiency of memory usage in training.

Figure 3: The comparison of overall efficiency among MITP and other prompt-
based methods. MITP reaches high performance with only 2.0M of parameters
(about 1% of the pre-trained model) to update and relatively small memory
usage in training.

(a) The results on interaction layers with interval=1.

(b) The results on interaction layers with interval=2.

(c) The results of MITP-small on a single interaction layer.

Figure 4: Selection of interaction layers and similarity type. In (a) and (b),
the intervals of interaction layers are selected from ${1,2}$, and similarity
type is selected from $\\{cos,mmd,c-p\\}$, where $cos$ means cosine
similarity, $mmd$ means MMD similarity, and $c-p$ denotes a combination of
covariance and pearson’s correlation coefficient. In (c), we perform prompt
interaction on one layer, with cosine similarity. Results of accuracy and
memory usage is reported in (c). The results are on UPMC-Food101.

###  5.2 Main results

MITP is effective on diverse image-text classification task types. To
demonstrate the effect of our method, we compare MITP with various existing
late fusion strategies, prompt-tuning strategies and prompt-based interaction
strategies, including CMA-CLIP [5], MMBT [11], PromptFuse [17], PMF [14], etc.
The results of our method are shown in Table 1. From the results, we observe
that our proposed method achieves competitive results on all three datasets
for diverse image-text task types. In detail, we can achieve an accuracy of
93.95$\%$ on uni-label classification dataset UPMC-Food101, F1-micro/F1-macro
of 65.9/56.3 on multi-label classification dataset MM-IMDB, and an accuracy of
73.45$\%$ on multimodal visual entailment dataset SNLI-VE. The small version
of our proposed method (MITP-small) also achieves competitive results on all
three datasets. In particular, we outperform all existing prompt-based methods
on datasets UPMC-Food101 and SNLI-VE.

MITP is efficient in terms of trainable parameters and memory usage. We
compare the efficiency our proposed method MITP with other prompt-based
methods. The efficiency is considered from perspectives of the number of
trainable parameters and the maximum memory usage in training. We present
comparisons of overall efficiency, efficiency of parameters, and efficiency of
memory usage in Fig 3. MITP achieves the second highest accuracy on UPMC-
Food101 with only 2.0M parameters (about 1$\%$ of its pre-trained foundation
model) to update and a relatively small training memory usage. The
illustrations show that MITP is balanced on performance and efficiency of
trainable parameters and memory usage.

###  5.3 Ablation study

In this section, we conduct ablation study on UPMC-Food101. We carry out
ablation study on components, to examine the effectiveness of temporal
prompts, prompt interaction, and memory-inspired prompt generation strategy
separately. We also illustrate illustrate which layers to perform temporal
prompt interaction, and impact of different similarity types in our prompt
generation strategy. Finally we discuss the robustness of the proposed method
when data is insufficient, and demonstrate the effectiveness.

Ablation study. To verify the impact of each component in our method, we
conduct component ablation study on UPMC-Food101. The results are presented in
Table 2. The baseline model does not introduce temporal prompts and is the
same as the employed pre-trained foundation model. After leveraging prompt
interaction (Model 2) and memory-inspired prompt generation strategy (Model 1)
to temporal prompts (Model 3), the performance increases significantly,
proving the effectiveness of our each component. The performance gap between
the baseline model and other models demonstrates the importance to introduce
prompts on intermediate layers, as well as the importance of prompt
interaction and prompt generation strategy.

Table 2: Ablation study on components on three employed datasets. $Tmp.$ $prompt$ means temporal prompts on intermediate layers of each branch. $P.$ $interaction$ means prompt interaction through naive MLP. $Mem.$ $strategy$ means prompt generation strategies introduced in 3.4. |  | $Tmp.$
---
prompt
| $P.$
---
interaction
| $Mem.$
---
strategy
| UMPC-Food101
---
Acc(%)
| MM-IMDB
---
F1-micro/-macro
| SNLI-VE
---
Acc(%)
Model 1 | ✓ | ✓ | ✓ | 93.95 | 65.9 / 56.3 | 73.45
Model 2 | ✓ | ✓ |  | 92.13 | 63.5 / 54.0 | 71.89
Model 3 | ✓ |  |  | 90.32 | 60.9 / 51.6 | 70.65
Baseline |  |  |  | 88.80 | 60.0 / 50.8 | 68.47

Selection of interaction layers and similarity type. In the memory hub,
various types of similarity can be chosen for prompt generation. Therefore we
discuss the selection of similarity type for memory hub. We also discuss the
selection of interaction layers. For the base MITP, we perform temporal prompt
interaction on three layers. Empirically the intervals of selected three
layers are the same, for example, (4,5,6) or (6,8,10). For the small version
MITP-small, temporal prompt interaction is operated on a single layer. For
similarity type, we choose cosine similarity, distribution-based similarity
maximum mean discrepancy (MMD), and correlation-based combination of
covariance and Pearson’s correlation coefficient (denoted as covariance-
pearsonr), where covariance is used to measure the similarity within temporal
prompt from the same modality, and Pearson’s correlation coefficient is used
to measure the similarity between temporal prompts from different modalities.

We conduct experiments to explore the performance of different layers and
different similarity types. The experimental results on selection of
interaction layers and similarity types are shown in Fig 4. The results
indicate that the model reaches its highest performance at interaction layers
(4,5,6) with cosine similarity. (On MM-IMDB and SNLI-VE, the best performance
is achieved at interaction layers (7,8,9) with cosine similarity.) The
performance of cosine similarity is generally higher than that of MMD. Cosine
similarity also outperforms covariance-pearsonr when interval is 1, while
covariance-pearsonr achieves higher performance when interval is 2. The reason
may be that distributions differ a lot between mid-level features of different
modalities. In common knowledge, deeper layers imply more semantic
information, leading to better experimental results when modality interaction
is conducted at deeper layers. However, this law is only supported by the
model with distribution-based similarities MMD and correlation-based
covariance-pearsonr. The performance of model with cosine similarity peaks
when modality interaction is conducted at the mid-level layers. Interactions
on shallower layers provide small improvement of performance, due to high
heterogeneity between shallow-level modality features. However, interactions
on shallower layers do not harm the performance. This indicates that
interactions via prompts may alleviate the modality gap during interaction. As
for interval of interaction layers, we can see that when cosine similarity is
applied, interaction layers with interval=1 generally achieve better
performance than those with interval=2; when MMD or covariance-pearsonr is
applied, interaction layers with interval=2 achieve better performance. The
reason may lie in the similarity-based prompt generation strategy. With cosine
similarity applied, it is hard for the generated prompts to bridge the gap
between layers with interval larger than 1.

Table 3: Results on UMPC-Food101 with insufficient data. The proposed method is robust with insufficient data, and achieves an accuracy over 90% with only 10% of the training data used. | Training data
---
(%)
| Results
---
Acc(%)
| Performance
---
degradation(%)
1 | 87.59 | 6.36
5 | 89.81 | 4.14
10 | 90.36 | 3.59
30 | 92.62 | 1.33

Robustness with insufficient data. For further discussion, we demonstrate the
robustness of the proposed method when the data is insufficient. We conduct
experiments on UPMC-Food101 with randomly selected 1$\%$, 5$\%$, 10$\%$, and
30$\%$ of training data, the results are shown in Table 3. The proposed MITP
achieves an accuracy over 90$\%$ with 10$\%$ of training data. There is merely
1.33$\%$ of performance degradation when the model is trained by 30$\%$ of
training data. The results indicate that our proposed method is robust with
insufficient data.

###  5.4 Discussions

Why is MITP able to reduce trainable parameter and memory usage? In MITP,
information of extracted features is stored in temporal prompts on
intermediate layers, and two-way information exchange is conducted through
prompt interaction, without participation of extracted features. Prompt
vectors contain much lower dimensions than extracted features, reducing
spatial complexity of inter-modality information exchange. Moreover,
calculative similarity-based prompt interaction ensures effective information
exchange without increasing trainable parameters or model complexity.
Comparing to fine-tuning methods with fine-tuning prompts leveraged on
multiple layers from shallower ones to deeper ones, MITP leverages temporal
prompts on selected deeper layers, significantly reduces the memory usage.

##  6 Conclusions

In this paper, we propose a memory-inspired temporal prompt interaction method
for image-text classification. Our approach interacts temporal prompts on
intermediate layers of the LMM foundation model to imitate human memory.
Sufficient inter-modality interactions are leveraged through direct prompt
interactions, while the number of parameters to update and the memory usage in
training are compressed. The results can demonstrate our effectiveness. In
particular, we achieve the best results among prompt-based interaction methods
and outperforms prompt-tuning methods with the same LMM foundation model.
However, a limitation of our proposed method is that temporal prompts are
added in prefix of input tokens, so that the knowledge inside the foundation
model is not fully leveraged. In the future, we plan to design prompt strategy
to conduct more sufficient interaction with the input tokens.

## References

  * Arevalo et al. [2017] John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González.  Gated multimodal units for information fusion.  _arXiv preprint arXiv:1702.01992_ , 2017.
  * Baddeley [2010] Alan Baddeley.  Working memory.  _Current biology_ , 20(4):R136–R140, 2010.
  * Deng et al. [2021] Huan Deng, Peipei Kang, Zhenguo Yang, Tianyong Hao, Qing Li, and Wenyin Liu.  Dense fusion network with multimodal residual for sentiment classification.  In _2021 IEEE International Conference on Multimedia and Expo (ICME)_ , pages 1–6. IEEE, 2021.
  * Doumas et al. [2022] Leonidas AA Doumas, Guillermo Puebla, Andrea E Martin, and John E Hummel.  A theory of relation learning and cross-domain generalization.  _Psychological review_ , 2022.
  * Fu et al. [2022] Jinmiao Fu, Shaoyuan Xu, Huidong Liu, Yang Liu, Ning Xie, Chien-Chih Wang, Jia Liu, Yi Sun, and Bryan Wang.  Cma-clip: Cross-modality attention clip for text-image classification.  In _2022 IEEE International Conference on Image Processing (ICIP)_ , pages 2846–2850. IEEE, 2022.
  * Hazarika et al. [2020] Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria.  Misa: Modality-invariant and-specific representations for multimodal sentiment analysis.  In _Proceedings of the 28th ACM international conference on multimedia_ , pages 1122–1131, 2020.
  * Hu and Singh [2021] Ronghang Hu and Amanpreet Singh.  Unit: Multimodal multitask learning with a unified transformer.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1439–1449, 2021.
  * Huang et al. [2020] Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu.  Pixel-bert: Aligning image pixels with text by deep multi-modal transformers.  _arXiv preprint arXiv:2004.00849_ , 2020.
  * Jia et al. [2022] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In _European Conference on Computer Vision_ , pages 709–727. Springer, 2022.
  * Khattak et al. [2023] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan.  Maple: Multi-modal prompt learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19113–19122, 2023.
  * Kiela et al. [2019] Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Ethan Perez, and Davide Testuggine.  Supervised multimodal bitransformers for classifying images and text.  _arXiv preprint arXiv:1909.02950_ , 2019.
  * Kim et al. [2021] Wonjae Kim, Bokyung Son, and Ildoo Kim.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , pages 5583–5594. PMLR, 2021.
  * [13] Liunian Harold Li, Mark Yatskar, D Yin, CJ Hsieh, and KW Chang.  Visualbert: A simple and performant baseline for vision and language. arxiv 2019.  _arXiv preprint arXiv:1908.03557_.
  * Li et al. [2023] Yaowei Li, Ruijie Quan, Linchao Zhu, and Yi Yang.  Efficient multimodal fusion via interactive prompting.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2604–2613, 2023.
  * Li et al. [2022] Zhen Li, Bing Xu, Conghui Zhu, and Tiejun Zhao.  Clmlf: a contrastive learning and multi-layer fusion method for multimodal sentiment detection.  _arXiv preprint arXiv:2204.05515_ , 2022.
  * Liang et al. [2021] Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al.  Multibench: Multiscale benchmarks for multimodal representation learning.  _arXiv preprint arXiv:2107.07502_ , 2021.
  * Liang et al. [2022] Sheng Liang, Mengjie Zhao, and Hinrich Schütze.  Modular and parameter-efficient multimodal fusion with prompting.  _arXiv preprint arXiv:2203.08055_ , 2022.
  * Liu et al. [2023] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang.  Gpt understands, too.  _AI Open_ , 2023.
  * Liu et al. [2021] Yunze Liu, Qingnan Fan, Shanghang Zhang, Hao Dong, Thomas Funkhouser, and Li Yi.  Contrastive multimodal fusion with tupleinfonce.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 754–763, 2021.
  * Lu et al. [2021] Yujie Lu, Shengyu Zhang, Yingxuan Huang, Luyao Wang, Xinyao Yu, Zhou Zhao, and Fei Wu.  Future-aware diverse trends framework for recommendation.  In _Proceedings of the Web Conference 2021_ , pages 2992–3001, 2021.
  * Mai et al. [2022] Sijie Mai, Ying Zeng, Shuangjia Zheng, and Haifeng Hu.  Hybrid contrastive learning of tri-modal representation for multimodal sentiment analysis.  _IEEE Transactions on Affective Computing_ , 2022.
  * Nagrani et al. [2021] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun.  Attention bottlenecks for multimodal fusion.  _Advances in Neural Information Processing Systems_ , 34:14200–14213, 2021.
  * Narayana et al. [2019] Pradyumna Narayana, Aniket Pednekar, Abishek Krishnamoorthy, Kazoo Sone, and Sugato Basu.  Huse: Hierarchical universal semantic embeddings.  _arXiv preprint arXiv:1911.05978_ , 2019.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Sun et al. [2022] Hao Sun, Hongyi Wang, Jiaqing Liu, Yen-Wei Chen, and Lanfen Lin.  Cubemlp: An mlp-based model for multimodal sentiment analysis and depression estimation.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 3722–3729, 2022.
  * Vielzeuf et al. [2018] Valentin Vielzeuf, Alexis Lechervy, Stéphane Pateux, and Frédéric Jurie.  Centralnet: a multilayer approach for multimodal fusion.  In _Proceedings of the European Conference on Computer Vision (ECCV) Workshops_ , pages 0–0, 2018.
  * Wang et al. [2015] Xin Wang, Devinder Kumar, Nicolas Thome, Matthieu Cord, and Frederic Precioso.  Recipe recognition with large multimodal food dataset.  In _2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW)_, pages 1–6. IEEE, 2015.
  * Xie et al. [2018] Ning Xie, Farley Lai, Derek Doran, and Asim Kadav.  Visual entailment task for visually-grounded language learning.  _arXiv preprint arXiv:1811.10582_ , 2018.
  * Xie et al. [2019] Ning Xie, Farley Lai, Derek Doran, and Asim Kadav.  Visual entailment: A novel task for fine-grained image understanding.  _arXiv preprint arXiv:1901.06706_ , 2019.
  * Xue and Marculescu [2023] Zihui Xue and Radu Marculescu.  Dynamic multimodal fusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2574–2583, 2023.
  * Zadeh et al. [2017] Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency.  Tensor fusion network for multimodal sentiment analysis.  _arXiv preprint arXiv:1707.07250_ , 2017.
  * Zadeh et al. [2018] Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency.  Memory fusion network for multi-view sequential learning.  In _Proceedings of the AAAI conference on artificial intelligence_ , 2018.
  * Zhou et al. [2022a] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16816–16825, 2022a.
  * Zhou et al. [2022b] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  _International Journal of Computer Vision_ , 130(9):2337–2348, 2022b.