# Frequency-Aware Physics-Inspired Degradation Model for Real-World Image
Super-Resolution

Zhenxing Dong1 1John Hopcroft Center for Computer Science, Shanghai Jiao Tong
University
2Institute of Image Communication and Network Engineering,Shanghai Jiao Tong
University
3Department of Electrical and Computer Engineering, University of Alabama,
Tuscaloosa, AL 35401 United States
4State Key Lab of Advanced Optical Communication Systems and Networks,
Shanghai Jiao Tong University
<EMAIL_ADDRESS>
Hong Cao1 1John Hopcroft Center for Computer Science, Shanghai Jiao Tong
University
2Institute of Image Communication and Network Engineering,Shanghai Jiao Tong
University
3Department of Electrical and Computer Engineering, University of Alabama,
Tuscaloosa, AL 35401 United States
4State Key Lab of Advanced Optical Communication Systems and Networks,
Shanghai Jiao Tong University
<EMAIL_ADDRESS>
Wang Shen2 1John Hopcroft Center for Computer Science, Shanghai Jiao Tong
University
2Institute of Image Communication and Network Engineering,Shanghai Jiao Tong
University
3Department of Electrical and Computer Engineering, University of Alabama,
Tuscaloosa, AL 35401 United States
4State Key Lab of Advanced Optical Communication Systems and Networks,
Shanghai Jiao Tong University
<EMAIL_ADDRESS>
Yu Gan3 1John Hopcroft Center for Computer Science, Shanghai Jiao Tong
University
2Institute of Image Communication and Network Engineering,Shanghai Jiao Tong
University
3Department of Electrical and Computer Engineering, University of Alabama,
Tuscaloosa, AL 35401 United States
4State Key Lab of Advanced Optical Communication Systems and Networks,
Shanghai Jiao Tong University
<EMAIL_ADDRESS>
Yuye Ling1* 1John Hopcroft Center for Computer Science, Shanghai Jiao Tong
University
2Institute of Image Communication and Network Engineering,Shanghai Jiao Tong
University
3Department of Electrical and Computer Engineering, University of Alabama,
Tuscaloosa, AL 35401 United States
4State Key Lab of Advanced Optical Communication Systems and Networks,
Shanghai Jiao Tong University
<EMAIL_ADDRESS>

Guangtao Zhai2 1John Hopcroft Center for Computer Science, Shanghai Jiao Tong
University
2Institute of Image Communication and Network Engineering,Shanghai Jiao Tong
University
3Department of Electrical and Computer Engineering, University of Alabama,
Tuscaloosa, AL 35401 United States
4State Key Lab of Advanced Optical Communication Systems and Networks,
Shanghai Jiao Tong University
<EMAIL_ADDRESS>
Yikai Su4 1John Hopcroft Center for Computer Science, Shanghai Jiao Tong
University
2Institute of Image Communication and Network Engineering,Shanghai Jiao Tong
University
3Department of Electrical and Computer Engineering, University of Alabama,
Tuscaloosa, AL 35401 United States
4State Key Lab of Advanced Optical Communication Systems and Networks,
Shanghai Jiao Tong University
<EMAIL_ADDRESS>

###### Abstract

Current learning-based single image super-resolution (SISR) algorithms
underperform on real data due to the deviation in the assumed degradation
process from that in the real-world scenario. Conventional degradation
processes consider applying blur, noise, and downsampling (typically bicubic
downsampling) on high-resolution (HR) images to synthesize low-resolution (LR)
counterparts. However, few works on degradation modelling have taken the
physical aspects of the optical imaging system into consideration. In this
paper, we analyze the imaging system optically and exploit the characteristics
of the real-world LR-HR pairs in the spatial frequency domain. We formulate a
real-world physics-inspired degradation model by considering both optics and
sensor degradation; The physical degradation of an imaging system is modelled
as a low-pass filter, whose cut-off frequency is dictated by the object
distance, the focal length of the lens, and the pixel size of the image
sensor. In particular, we propose to use a convolutional neural network (CNN)
to learn the cutoff frequency of real-world degradation process. The learned
network is then applied to synthesize LR images from unpaired HR images. The
synthetic HR-LR image pairs are later used to train an SISR network. We
evaluate the effectiveness and generalization capability of the proposed
degradation model on real-world images captured by different imaging systems.
Experimental results showcase that the SISR network trained by using our
synthetic data performs favorably against the network using the traditional
degradation model. Moreover, our results are comparable to that obtained by
the same network trained by using real-world LR-HR pairs, which are
challenging to obtain in real scenes.

##  1 Introduction

Figure 1: Illustration of the conventional and our degradation modes. We show
conventional degradation model, including blur, noise and downsampling
(denoted by $g[m,n]$) in (a), which only considers image-to-image (i.e., HR-
to-LR) degradation.However, for the camera acquisition of LR, the object
should not be simply regarded as the corresponding HR.Our proposed model
considers that HR and LR are both degraded images of a real-world 3D object
captured by different devices in (b).Our novel degradation model (denoted by
Eq. 11) is analyzed in the frequency domain.Our goal is to solve the inverse
problem, which is to recover an HR image obtained by the better device from
its corresponding LR recording.

With the rise of digital imaging and digital photography in the past decades,
persistent endeavors have been devoted to researches that attempt to distill
high-spatial-resolution contents from low-spatial-resolution measurements.
Image super-resolution is a prominent example, in which a high-resolution (HR)
image is recovered from a low-resolution (LR) counterpart. Image super
resolution can be categorized as single-image super resolution (SISR) [8] and
multi-frame super resolution based on the input data formats; the latter
requires a series of slightly misaligned LR images to provide extra temporal
information, while the former only needs one LR measurement.

Considering that multiple LR images might not be accessible in practical
settings, SISR thus becomes an attractive research topic in various fields
including medical imaging [10], small target detection [1], and remote sensing
[25].

Unlike its optical counterparts such as structured illumination microscopy
(SIM) [20], localization microscopy [14], and stimulated emission depletion
(STED) microscopy [13], SISR features a pure algorithmic effort which requires
little hardware modifications but fully leverages advanced signal processing
techniques. Conventionally, SISR was performed by traditional methods such as
neighborhood embedding [3] and dictionary learning [24]. With the recent surge
in deep learning (DL), SISR algorithms based on the neural networks have been
put in the spotlight and their performances have been constantly improved via
better network architecture design [6, 19, 28] and refined training strategies
[17, 18].

Despite of the great successes made by the neural network-based SISR
algorithms in various public datasets, their capability under real-world
circumstances has long been under question [5]. It is because most of the
existing SISR networks are trained and evaluated on synthetic datasets, in
which the LR images are not physically but numerically acquired by applying
degradation models including Gaussian blur, Bicubic downsampling, and Gaussian
noise on the corresponding HR images in the spatial domain, as shown in
Fig.1(a).

Nonetheless, the inherent mappings of the real-world HR-LR image pairs could
be far more sophisticated than the aforementioned analytical models. To bridge
this gap, researchers [4, 27] attempt to train the SISR neural networks by
using HR-LR image pairs that are collected in the real world, so that the
trained networks could implicitly learn the real-world degradation model and
get its generalization ability enhanced. Unfortunately, preparing such a
database is very challenging; excessive labours, specially designed hardware,
and careful postprocessings are often required to construct a small-scale
dataset. For example, RealSR [2] only contains 559 scenes captured by two
different devices.

In this paper, we propose a physics-inspired degradation model for real-world
SISR tasks. We suggest that the conventional degradation model, which is
illustrated in Fig. 1(a), might not be accurate: for the camera acquisition of
LR, the object should not be simply regarded as the corresponding HR. We
instead argue that HR and LR are both degraded images of a real-world 3D
object as shown in Fig. 1(b), in which, for example, the HR is acquired by a
DSLR camera and the LR is captured by a cellphone camera. We further elucidate
that the goal of a real-world SISR task is to recover an HR image obtained by
a better device from its corresponding LR recording. To validate our
hypothesis, we study the optical imaging systems in Gaussian optics’ regime,
analytically derive the degradation model, which could be viewed as a low-pass
transfer function, between HR and LR in the spatial frequency domain, and
experimentally show that the characteristics of the transfer function is
jointly determined by the object distance, the focal length of the lens, and
the pixel size of the image sensor (see Fig. 3).

Once the degradation model is properly defined, we move on to solve the
inverse problem. We propose to use a convolutional neural network to learn the
SISR degradation in the spatial frequency domain. It is worth noting that the
network is supposed to be aware of the object distance as well as the camera
specifications via the supervised learning of HR image’s spectrum and the
corresponding low-pass filter’s cutoff bandwidth.

The learned prediction network can thus be utilized to synthesize real-world-
like LR images from arbitrary HR images. The thus generated HR-LR pairs could
be further used to train SISR networks. Experimental results (Fig. 6) show
that the SISR network trained by our method obtain the best performance.

The contributions of this paper are summarized below:

  * •

We propose a novel physics-inspired degradation model to incorporate both
optics and sensor degradation in the spatial frequency domain for real-world
SISR problems.

  * •

We suggest the abovementioned degradation model could be implicitly learned by
a neural network and later applied on arbitrary HR images to synthesize their
real-world-_like_ LR counterparts.

  * •

We demonstrate the generalization and effectiveness of our degradation model
in various imaging systems for real-world image super-resolution.

##  2 Related Work

Degradation Model:

As mentioned in the previous section, SISR networks trained by traditionally
synthesized HR-LR pairs might not necessarily learn the actual degradation
process. Some works [7, 16, 26] attempted to approximate the real-world
degradation model by complicating the blur, downsampling, and noise models.
While impressive results were obtained in this way, the resultant degradation
models often lack intrinsic interpretability capability due to the absence of
physics during the model constructions. Moreover, their proposed methods only
work on the same datasets as their degradation models.

Blind SISR Networks:

The research community has also spent a lot of efforts on blind super-
resolution, in which key parameters of the degradation process are assumed
unknown. We can roughly divide the blind SISR related works into two genres:

One approach is to directly estimate the unknown blur kernel. Gu et al. [11]
propose an Iterative Kernel Correction (IKC) architecture, which uses the
intermediate SR results to iteratively correct the estimated blur kernel. Zhou
et al. [29] study a real-world super-resolution dataset (DSLR) [15] and
propose a Kernel Modeling Super-Resolution (KMSR) network. While the
performance of the network is improved compared with other works in real
world, and the estimated kernel lacks a physical interpretation and does not
perform well if being generalized to other imaging devices.

The other way is to adopt a data-driven methodology: some researchers also
seek to construct real-world datasets to implicitly infer the underlying
degradation model. Zhang et al. [27] collect 500 HR-LR image pairs at multiple
focal lengths by using Sony cameras (SR-RGB). Unfortunately, the image pairs
are not registered, so that the SR-RGB dataset cannot be directly used to
train SISR networks. Cai et al. [2] capture HR and LR data pairs in multiple
real-world scenes at multiple focal lengths, and propose a new pairing
algorithm to align the data pairs. However, capturing the real-world SISR
dataset requires a lot of manpower and material resources, and the size of the
dataset is also limited.

##  3 Frequency-Aware Physics-Inspired Degradation Model

In the following section, we propose a frequency-aware physics-inspired
degradation model that could characterize both optics and sensor degradation
processes for digital photography. The model is derived on the basis of linear
system theory and Gaussian optics [22].

Image formation:

We focus our discussion on the optics of consumer-level cameras, e.g., digital
single-lens reflex (DSLR) cameras and cellphone cameras. Optical model of
those cameras can be approximately considered as a thin lens [22]. We depicted
the image formation in Fig. 2, and the corresponding thin lens formulae are
given by:

$\left\\{\begin{aligned} \frac{1}{f}&=\frac{1}{s^{\prime}}+\frac{1}{s},\\\
\frac{h}{h^{\prime}}&=\frac{s}{s^{\prime}},\end{aligned}\right.$ (1)

where $f$ is the focal length of the equivalent thin lens, $s$ is the object
distance, $s^{\prime}$ is the image distance, $h$ is the object height, and
$h^{\prime}$ is the image height. Considering that the lens’ focal length $f$
is usually much smaller than the object distance $s$ in photography except for
macro photography, we could safely omit its contribution and obtain:

$f\approx s^{\prime}.$ (2)

By plugging Eq. 2 back to the second line of Eq. 1, we could obtain the
following equation with regard to the magnification $M$ between the image and
object:

$M=\frac{h^{\prime}}{h}=\frac{{{f}}}{s}.$ (3)

Figure 2: Thin lens imaging process. We denote focal length, object distance,
object height, image distance, imaging height, and pixel size by $f$, $s$,
$h$, $s^{\prime}$, $h^{\prime}$, and $\Delta y^{\prime}$.

Optics degradation model:

Despite of the demagnification, the formed image is also distorted by the
optical aberrations of the image system. This effect can be fully described by
a point spread function (PSF) $p_{s}(x,y)$ in the context of linear system
theory [9]. It is should be noted that $p_{s}(x,y)$ is shift-invariant in the
transverse directions ($XY$ plane) but has a strong axial dependency on the
object distance $s$. Therefore, the optically degraded image $I_{o}(x,y)$
could be given by:

${I_{o}(x,y)}=o(x,y)*p_{s}(x,y),$ (4)

where the originally three dimensional object $o(x,y,z)$ is now replaced by a
flat projection $o(x,y)$, and $*$ is the convolution operation. This
approximation is largely accurate if the object distance $s$ is much larger
than the axial extent of the object.

Sensor degradation model:

We further take the sensor into account. Without loss of generality, we assume
the pixel size of the sensor ($\Delta x^{\prime}$ and $\Delta y^{\prime}$) has
an aspect ratio of 1, i.e. $\Delta x^{\prime}=\Delta y^{\prime}$. Therefore,
the maximum spatial frequency $\xi^{\prime}_{\textrm{sensor}}$ could be
detected by the sensor in the image plane is given by Nyquist–Shannon sampling
theorem:

$\xi^{\prime}_{\textrm{sensor}}=\frac{1}{2\Delta x^{\prime}}=\frac{1}{2\Delta
y^{\prime}}.$ (5)

Consider the scaling relationship between the object plane and image plane as
shown in Fig. 2, we could obtain the corresponding spatial sampling interval
$\Delta x$ and spatial sampling frequency $\xi_{\textrm{object}}$ in the
object plane:

$\left\\{\begin{aligned} \Delta x&=\frac{1}{M}\cdot\Delta
x^{\prime}=\frac{s}{f}\cdot\Delta x^{\prime},\\\
\xi_{\textrm{object}}&=\frac{1}{M}\cdot\xi^{\prime}_{\textrm{sensor}}=\frac{s}{f}\cdot\frac{1}{2\Delta
x^{\prime}}.\end{aligned}\right.$ (6)

It is important to recognize that the spatial sampling interval $\Delta x$ is
determined not only by the sensor pixel size $\Delta x^{\prime}$ and the
camera’s focal length $f$ as described in previous works [21] but also the
object distance $s$.

.

Frequency-aware physics-inspired degradation model:

The complete imaging model could be constructed by combining the results
mentioned above:

$\displaystyle I(m\Delta x^{\prime},n\Delta y^{\prime})$
$\displaystyle=\mathcal{S}\left[I_{o}(x,y)\right]+\eta$ (7)
$\displaystyle=\mathcal{S}[o(x,y)*p_{s}(x,y)]+\eta$
$\displaystyle=o(x,y)*p_{s}(m\Delta x,n\Delta y)+\eta$
$\displaystyle=o(x,y)\\!*\\!p_{s}(\frac{s}{f}m\Delta
x^{\prime},\\!\frac{s}{f}n\Delta y^{\prime})\\!+\\!\eta,$

where $I$ is the sensor image, $\mathcal{S}$ represents for the sampling
operation, $m$ and $n$ are the sampling indices, and
$\eta\sim\mathcal{N}(0,{\sigma^{2}})$ denotes the Gaussian read noise.
Properties [29] regarding to the $S$ operation are utilized to simplify the
equation.

We can further derive the explicit expressions for HR and LR images of the
same object acquired by two different imaging systems as follows:

$\centering\left\\{\begin{aligned} \text{HR}(m_{1}\Delta
x_{1}^{\prime},\\!n_{1}\Delta
y_{1}^{\prime})\\!=\\!o(x,y)\\!*\\!p_{1,s}(\frac{s}{f_{1}}m_{1}\Delta
x_{1}^{\prime},\\!\frac{s}{f_{1}}n_{1}\Delta
y_{1}^{\prime})\\!+\\!\eta_{1},\\\ \text{LR}(m_{2}\Delta
x_{2}^{\prime},\\!n_{2}\Delta
y_{2}^{\prime})\\!=\\!o(x,y)\\!*\\!p_{2,s}(\frac{s}{f_{2}}m_{2}\Delta
x_{2}^{\prime},\\!\frac{s}{f_{2}}n_{2}\Delta
y_{2}^{\prime})\\!+\\!\eta_{2}.\end{aligned}\right.\@add@centering$ (8)

It is obvious that HR and LR images are in general of distinct sizes.
Moreover, the field-of-views (FOVs) are also different, which makes cropping,
upsampling, and registration mandatory before analyzing its degradation [2].

Without loss of generality, we could assume the sensors used in two systems
are identical ($m_{1}=m_{2}$, $n_{1}=n_{2}$, $\Delta x_{1}^{\prime}=\Delta
x_{2}^{\prime}=\Delta y_{1}^{\prime}=\Delta y_{2}^{\prime}$) and the ratio
between the effective focal length is $2:1$. In this case, the central part of
the LR image need to be cropped out and upsampled to match the FOV of the
whole HR image. Once this is done, we can transfer the matched HR-LR image
pair to the spatial frequency domain:

$\centering\left\\{\begin{aligned} \mathcal{F}(\textrm{HR}(m\Delta
x^{\prime},n\Delta
y^{\prime}))\\!&=\\!\mathcal{O}(f_{x},\\!f_{y})\\!\cdot\\!{\mathcal{P}_{1,s}}(m\Delta
f_{x}^{\prime},\\!n\Delta f_{y}^{\prime})\\!+\\!\mathcal{F}(\eta_{1}),\\\
\mathcal{F}(\textrm{LR}(m\Delta x^{\prime},n\Delta
y^{\prime}))\\!&=\\!\mathcal{O}(f_{x},\\!f_{y})\\!\cdot\\!{\mathcal{P}_{2,s}}(m\Delta
f_{x}^{\prime},\\!n\Delta
f_{y}^{\prime})\\!+\\!\mathcal{F}(\eta_{2}),\end{aligned}\right.\@add@centering$
(9)

where $\mathcal{F}$ denotes 2-D Fourier transform (FT), $f_{x,y}$ are the
spatial frequency in $x$ and $y$ direction, $\mathcal{O}$ and
$\mathcal{P}_{1,2}$ represents for the 2-D FT of $o(x,y)$ and $p_{1,2}(x,y)$,
respectively. The transfer function $\mathcal{G}$ of the proposed degradation
model can be retrieved by simply dividing the spectra of HR image by that of
the LR image:

$\displaystyle\mathcal{G}_{s}(m\Delta f_{x}^{\prime},n\Delta f_{y}^{\prime})$
$\displaystyle=\frac{\mathcal{F}(\textrm{LR}(m\Delta x^{\prime},n\Delta
y^{\prime}))}{\mathcal{F}(\textrm{HR}(m\Delta x^{\prime},n\Delta
y^{\prime}))}$ (10) $\displaystyle\approx\frac{\mathcal{P}_{{2,s}}(m\Delta
f_{x}^{\prime},n\Delta f_{y}^{\prime})}{\mathcal{P}_{{1,s}}(m\Delta
f_{x}^{\prime},n\Delta f_{y}^{\prime})},$

where we omit $\mathcal{F}(\eta_{1})$ and $\mathcal{F}(\eta_{2})$ based on the
assumption that the signal-to-noise-ratio (SNR) of both HR and LR images are
relatively high.

Therefore, we can formally write the forward model for the SISR problem as:

$\centering\textrm{LR}[m,n]\\!=\\!\left(\mathcal{F}^{-1}\\{\mathcal{F}(\textrm{HR}[m,n])\\!\cdot\\!\mathcal{G}_{s}[m,n]\\}\right){\downarrow_{N}}\\!+\\!\eta,\@add@centering$
(11)

where $\mathcal{F}^{-1}$ is the inverse 2-D Fourier transform,
$\downarrow_{N}$ is downsampling operation with a factor of $N$.

In summary, we first systematically and theoretically demonstrate the real-
world SISR degradation process based on Gaussian optics, and propose a
frequency-aware physics-inspired degradation model, which associated with
object distance, focal length and pixel size of the imaging system.

Figure 3: Visualization of real-world degradation model. (a) A series of
operations on three HR-LR pairs of RealSR [2] in order to extract real
degradation information in the frequency domain. (b) Visualization the 1-D
degradation function and mark the cut-off frequency points of the degenerate
function of Canon building and Nikon building; (c) Some important information
for three image pairs and cameras. $\mathcal{F}$ is 2-D Fourier transform and
$\mathcal{T}$ is the conversion of Cartesian coordinate to polar coordinate.

Interpretability:

To illustrate the dependencies of the degradation model on the object distance
$s$ and sensor pixel size $\Delta x^{\prime}$, we evaluate the model on a
popular real-world SISR dataset RealSR [2]. All image pairs are captured by
Canon EOS 5D Mark III and Nikon D810, both of which are equipped with one
24-105mm, f/4.0 zoom lens.

We select three pairs of HR-LR images from the dataset. The HR and LR images
are captured using the focal lengths of 105 mm and 55 mm, respectively, and
more detailed information is provided in Fig. 3(c). A series of operations are
performed on the HR-LR pairs to acquire the corresponding transfer function
$\mathcal{G}_{s}$ as shown in Fig. 3(a). Firstly, we perform 2-D FTs on both
HR and LR images, and take the logarithm of spectra. We could then obtain the
transfer function $\mathcal{G}_{s}$ by simply subtracting the spectrum of HR
from that of LR. After that, the coordinate system is changed from Cartesian
to polar to better visualize the isotropic low-pass characteristic of the
transfer function $\mathcal{G}_{s}$. Finally, we take the exponential of
$\mathcal{G}_{s}$ and average it over angular direction. The resultant curves
for three HR-LR pairs are plotted in Fig. 3(b). We have the following two
observations:

(1) The dependency of the degradation model on the object distance $s$: One
interesting observation could be made by comparing the transfer functions
$\mathcal{G}_{s}$ for HR-LR pair $\\#1$ and $\\#2$ as shown in Fig. 3(b) is
that the cutoff frequency for the former is lower than that of the latter.
This result matches the prediction made by Eq. 6 exactly, which states that
the spatial sampling frequency in the object plane $\xi_{\textrm{object}}$ is
inversely proportional to the object distance $s$, if we consider the building
of the image pair $\\#1$ is located much farther than the palette of pair
$\\#2$. More similar results could be found in the supplementary materials.

(2) The dependency of the degradation model on sensor pixel size $\Delta
x^{\prime}$: We analyze the influence of pixel size by comparing the results
of pair $\\#1$ and pair $\\#3$, where the same building (i.e. fixed object
distance $s$) is captured by distinct cameras:

  * •

Object sampling frequency ratio $\alpha$: The ratio of the object sampling
frequency between the two cameras used in the experiment could be
theoretically calculated by using published technical specifications:

$\centering\alpha=\frac{\xi_{\textrm{object,Nikon}}}{\xi_{\textrm{object,Canon}}}=\frac{\Delta
x^{\prime}_{\textrm{Canon}}}{\Delta
x^{\prime}_{\textrm{Nikon}}}=\frac{6.25\,\mu m}{4.88\,\mu m}\approx
1.28.\@add@centering$ (12)

  * •

Cutoff frequency ratio $\beta$: The actual ratio between the cutoff
frequencies, on the other hand, could be directly inferred from Fig. 3(b):

$\beta=\frac{{{f_{c,\textrm{Nikon}}}}}{{{f_{c,\textrm{Canon}}}}}=\frac{{167}}{{130}}\approx
1.28.$ (13)

It is clear that the theoretically predicted $\alpha$ matches the
experimentally derived $\beta$, which verifies the inverse proportionality
between the object sampling frequency $\xi_{\textrm{object}}$ and sensor pixel
size $\Delta x^{\prime}$.

In summary, we showcase the physical soundness of the proposed degradation
model by conducting a series of controlled experiments. Most importantly, to
our best knowledge, this is the first time that both object distance $s$ and
sensor pixel size $\Delta x^{\prime}$ are explicitly integrated in a
degradation model for SISR.

##  4 Proposed Method

The proposed pipeline for real-world SISR is showed in Fig. 4. It consists of
three stages.

Figure 4: Pipeline of our proposed model. Stage1: Utilize existing real-world
SR datasets to learn a real-world imaging system; Stage2: Use the learned
imaging system to synthesize proposed LR images from unpaired real HR images;
Stage3: Train an SISR network on our proposed dataset. Figure 5: Learning the
cutoff $f_{c}$ of a imaging system. Although we present that the paired HR
image is an RGB image, the neural network is actually trained in color
channel.

Stage 1: Learn a real-world degradation model:

In the previous section, we have demonstrated that the transfer function
$\mathcal{G}_{s}$ could be largely viewed as a low-pass filter (LPF) in the
spatial frequency domain, and the most important parameter for the LPF is the
cutoff frequency $f_{c}$ which is dictated by both $s$ and $\Delta
x^{\prime}$. Therefore, we proposed to simply learn the cutoff frequency
$f_{c}$ instead of the full transfer function $\mathcal{G}_{s}$.

First of all, we divide each image pair
$\left\\{{\left.{{I^{\textrm{HR}}},{I^{\textrm{LR}}}}\right\\}}\right.$ that
is captured by the same device from RealSR [2] into $J$ pairs of overlapping
patches
${\left\\{{\left.{{I_{j}}^{\textrm{HR}},{I_{j}}^{\textrm{LR}}}\right\\}}\right.}$,
where $j=1,2,...,J$. For every pair of patches, we follow the procedure
described in Fig. 3 to obtain the cutoff frequency ${f_{{c_{j}}}}$. More
detailed process can be found in supplementary material Algorithm 1.

Table 1: Comparison quantitative results of average PSNR and SSIM on Canon and
Nikon dataset from RealSR [2], including FSSR-DPED, FSSR-JPEG, RealSR-DPED ,
RealSR-JPEG, BSRGAN, Proposed, and Proposed† The numbers in red and blue
represent the best and second-best results.

Testing set |  Metric |  FSSR-DPED |  FSSR-JPEG |  RealSR-DPED |  RealSR-JPEG |  BSRGAN |  Proposed |  Proposed†
---|---|---|---|---|---|---|---|---
Canon |  PSNR |  25.18 |  23.46 |  23.08 |  25.67 |  25.17 |  25.59 |  25.69
|  SSIM |  0.835 |  0.753 |  0.777 |  0.820 |  0.834 |  0.847 |  0.847
Nikon |  PSNR |  24.48 |  22.32 |  21.84 |  25.54 |  24.60 |  25.70 |  25.58
|  SSIM |  0.807 |  0.674 |  0.731 |  0.791 |  0.805 |  0.832 |  0.830

We then train a prediction network $\mathcal{R}$ to learn the mapping from the
HR spectrum $\mathcal{F}(I_{j}^{\textrm{HR}})$ to the corresponding cutoff
frequency ${f_{{c_{j}}}}$ for a specific device by supervised learning as is
depicted in Fig. 5. In this study, we choose ResNet34 [12] as the prediction
network.

Stage 2: Synthesize LR images from unpaired HR images:

Once the mapping from an input HR spectrum to cutoff frequency in source
imaging system is learned, we can use it to generate real-world-like from
arbitrary HR images for a target imaging system. Here, we select SR-RGB [27]
captured with the focal length of 105mm as the source of HR images.

We first preprocess the selected HR images in the same way as laid out in
Stage 1: each unpaired HR image $I^{\textrm{HR,unpaired}}$ from SR-RGB is
divided into $J$ overlapping patches $I_{j}^{\textrm{HR,unpaired}}$. We then
input $\mathcal{F}(I_{j}^{\textrm{HR,unpaired}})$ to the trained $\mathcal{R}$
to predict the corresponding cutoff frequency $f_{c_{j},\textrm{source}}$. For
simplicity, We neglect the spatial dependency of the degradation model within
the same image, and use the image-wise average cutoff frequency
$f_{c,\textrm{source}}=\frac{1}{J}\sum\limits_{j}{f_{j_{c},\textrm{source}}}$
instead.

After acquiring $f_{c,\textrm{source}}$, a correction is made to handle the
misfit between the pixel size of the source system and that of the target
system: ratio $\alpha$ is first calculated based on Eq. 12 and later
multiplied by $f_{c,\textrm{source}}$ to retrieve the correct
$f_{c,\textrm{target}}$.

Finally, a second-order Butterworth low-pass filter $\mathcal{B}$ is
constructed to approximate the transfer function $\mathcal{G}_{s}$, and we
apply Eq. 11 to synthesize the real-world-like LR images from the input
unpaired HR images. Algorithm 2 in the supplementary material shows the
detailed process.

Stage3: Train an SISR Network:

Since the focus of the current study is on the degradation model instead of
the SISR network, we utilize an existing SISR network EDSR [19]. It should be
noted that the proposed degradation model does not limit the use of other SISR
networks.

##  5 Experiments

###  5.1 Implementation Details

Dataset and training details:

In Stage 1, two distinct prediction networks are used to learn Canon EOS 5D
Mark III and Nikon D810 imaging systems based on RealSR dataset [2],
respectively. The parameters are set the same for both networks: the batch
size is 16, the patch size is $320\times 320$, the number of epochs is 100,
the number of training samples is 12,800, the learning rate is
$1\times{10^{{\rm{-}}3}}$, and the optimizer is RMSprop
(`weight_decay=10^{-8}`, `momentum=0.9`).

In Stage 3, a total number of four synthetic datasets are generated, and they
can be categorized into two sets (Ours and Ours†) based on whether the source
system is identical to the target system:

  * •

Ours:

(1) Use the predicting network learned from a Canon system to synthesize
_Canon-like_ LR images.

(2) Use the predicting network learned from a Nikon system to synthesize
_Nikon-like_ LR images.

  * •

Ours†:

(3) Use corrected predicting network learned from a Canon system to synthesize
_Nikon-like_ LR images.

(4) Use corrected predicting network learned from a Nikon system to synthesize
_Canon-like_ LR images.

For the training of EDSR, the patch size is set to $96\times 96$, the batch
size is fixed to 16, the initial value of the learning rate is
$1\times{10^{{\rm{-}}4}}$, and the number of training epochs is 200. The Adam
default parameters (${\beta_{1}}$ $=$ 0.9, ${\beta_{2}}$ $=$ 0.999 and
$\varepsilon$ $=$ ${10^{{\rm{-}}8}}$) are used in the process of optimizing
the network. All experiments are trained using NVIDIA GeForce RTX 3090.

Evaluation metric:

Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) [23]
are employed to evaluate the performance of competing methods.

###  5.2 Comparison to state-of-the-arts

Baselines:

We compare our network (denoted by Proposed) on RealSR dataset [2] with
several state-of-the-art methods, including FSSR-DPED [7], FSSR-JPEG [7],
RealSR-DPED [16], RealSR-JPEG [16], and BSRGAN [26].

Objective results:

As listed in Table 1, Proposed surpasses BSRGAN by 0.42 dB in PSNR and 0.013
in SSIM, respectively, in the Canon testing set. For the Nikon testing set,
Proposed further widens the lead by 1.1 dB in PSNR and 0.027 in SSIM,
respectively.

Visual results:

Fig. 6 shows the visual results obtained by the abovementioned networks. It is
clear that network trained by our dataset possesses the best spatial
resolution. In particular, Proposed can completely resolve the periodic black
and white stripes in both horizontal (first row of Fig. 6) and vertical
directions (third row of Fig. 6), while artifacts are observed in SOTAs.

Figure 6: Comparison Qualitative $\times 4$ SR results of on Canon and Nikon
dataset from RealSR [2], including FSSR-DPED, FSSR-JPEG, RealSR-DPED, RealSR-
JPEG, BSRGAN, Proposed, and Proposed†.

Experiments on author-captured images:

To further confirm that the proposed model can be adapted to real-world SISR,
we use another DSLR camera (Canon EOS 5D Mark IV, $\Delta x^{\prime}=5.36$
$\mu$m, $f=28$ mm) to capture real-world images as testing LRs. The super-
resolved results by using different models are all visualized in Fig. 7. It is
clear that the network trained on our dataset achieves better visual quality
than SOTAs with more natural rendering and less artifacts. Although BSRGAN’s
visual quality results are comparable with Proposed, we can’t neglect its much
larger training set and superior network.

Figure 7: Visual $\times 4$ SR results of on real-captured images, including
FSSR-DPED, FSSR-JPEG, RealSR-DPED, RealSR-JPEG, BSRGAN, EDSR trained on RealSR
(denoted by RealSR), and Proposed.

###  5.3 Model Analysis

Reconstruction fidelity:

We further investigate the reconstruction fidelity by looking at the profile
along the yellow line marked in the top of Fig. 8; GT, LR, BSRGAN, and
Proposed are included for the comparison and the results are plotted in the
bottom of Fig. 8.

Despite of the similar visual conception of BSRGAN and Proposed in Fig. 6,
Fig. 8 tells a completely different story. While HR, LR, and Proposed all
manifest same periodicity, artifacts are observed from BSRGAN’s reconstruction
as denoted by the “*”s.

Figure 8: Comparison the frequency fidelity of SR results on yellow line
(location in the third row of Fig. 6), including GT, LR, BSRGAN, and Proposed.
“*” denotes artifacts in BSRGAN.

Cross-camera evaluation:

To demonstrate the generalization ability of our proposed model from a source
system to a known target system, we also train EDSR by Ours† datasets (denoted
by Proposed†). As shown Table 1, Proposed† are comparable to Proposed with a
mere 0.1 dB lead in PSNR for Canon testing dataset, and a 0.12 dB and a 0.002
gap in PSNR and SSIM for Nikon testing dataset, respectively. Visually, we can
hardly notice the difference between Proposed and Proposed† as shown in Fig.
6.

##  6 Conclusion

To address the challenges imposed by the real-world SISR tasks, we propose a
frequency-aware physics-inspired degradation model. Unlike the existing
approaches to generate synthetic dataset, the proposed model is rooted in
Gaussian optics and the physical modelling of camera sensors, which makes it
more interpretable and generalizable. Based on the SISR dataset synthesized by
our proposed degradation model, we train a deep SISR network for real-world
SISR. Experimental results on real-world images have shown that the network
trained on our dataset could successfully recover higher spatial-frequency
components with less artifacts. In the future, we plan to further refine the
degradation-learning process and design a novel SISR algorithm that could full
exploit the proposed datasets.

## References

  * [1] Yancheng Bai, Yongqiang Zhang, Mingli Ding, and Bernard Ghanem.  Sod-mtgan: Small object detection via multi-task generative adversarial network.  In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  * [2] Jianrui Cai, Hui Zeng, Hongwei Yong, Zisheng Cao, and Lei Zhang.  Toward real-world single image super-resolution: A new benchmark and a new model.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3086–3095, 2019.
  * [3] Hong Chang, Dit-Yan Yeung, and Yimin Xiong.  Super-resolution through neighbor embedding.  In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 1, pages I–I, 2004.
  * [4] Chang Chen, Zhiwei Xiong, Xinmei Tian, Zheng-Jun Zha, and Feng Wu.  Camera lens super-resolution.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1652–1660, 2019.
  * [5] Victor Cornillere, Abdelaziz Djelouah, Wang Yifan, Olga Sorkine-Hornung, and Christopher Schroers.  Blind image super-resolution with spatially variant degradations.  ACM Transactions on Graphics (TOG), 38(6):1–13, 2019.
  * [6] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Learning a deep convolutional network for image super-resolution.  In European conference on computer vision, pages 184–199. Springer, 2014.
  * [7] Manuel Fritsche, Shuhang Gu, and Radu Timofte.  Frequency separation for real-world super-resolution.  In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 3599–3608, 2019.
  * [8] Daniel Glasner, Shai Bagon, and Michal Irani.  Super-resolution from a single image.  In 2009 IEEE 12th international conference on computer vision, pages 349–356. IEEE, 2009.
  * [9] Joseph W Goodman.  Introduction to fourier optics.  Introduction to Fourier optics, 3rd ed., by JW Goodman. Englewood, CO: Roberts & Co. Publishers, 2005, 1, 2005.
  * [10] Hayit Greenspan.  Super-Resolution in Medical Imaging.  The Computer Journal, 52(1):43–63, 02 2008.
  * [11] Jinjin Gu, Hannan Lu, Wangmeng Zuo, and Chao Dong.  Blind super-resolution with iterative kernel correction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1604–1613, 2019.
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [13] Stefan W. Hell and Jan Wichmann.  Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy.  Opt. Lett., 19(11):780–782, Jun 1994.
  * [14] Samuel T. Hess, Thanu P.K. Girirajan, and Michael D. Mason.  Ultra-high resolution imaging by fluorescence photoactivation localization microscopy.  Biophysical Journal, 91(11):4258–4272, 2006.
  * [15] Andrey Ignatov, Nikolay Kobyshev, Radu Timofte, Kenneth Vanhoey, and Luc Van Gool.  Dslr-quality photos on mobile devices with deep convolutional networks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3277–3285, 2017.
  * [16] Xiaozhong Ji, Yun Cao, Ying Tai, Chengjie Wang, Jilin Li, and Feiyue Huang.  Real-world super-resolution via kernel estimation and noise injection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
  * [17] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In European conference on computer vision, pages 694–711. Springer, 2016.
  * [18] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al.  Photo-realistic single image super-resolution using a generative adversarial network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4681–4690, 2017.
  * [19] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee.  Enhanced deep residual networks for single image super-resolution.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 136–144, 2017.
  * [20] Manish Saxena, Gangadhar Eluru, and Sai Siva Gorthi.  Structured illumination microscopy.  Adv. Opt. Photon., 7(2):241–275, Jun 2015.
  * [21] Vincent Sitzmann, Steven Diamond, Yifan Peng, Xiong Dun, Stephen Boyd, Wolfgang Heidrich, Felix Heide, and Gordon Wetzstein.  End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging.  ACM Trans. Graph., 37(4), July 2018.
  * [22] Warren J Smith.  Modern optical engineering: the design of optical systems.  McGraw-Hill Education, 2008.
  * [23] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  * [24] Jianchao Yang, John Wright, Thomas Huang, and Yi Ma.  Image super-resolution as sparse representation of raw image patches.  In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8, 2008.
  * [25] Qiangqiang Yuan, Huanfeng Shen, Tongwen Li, Zhiwei Li, Shuwen Li, Yun Jiang, Hongzhang Xu, Weiwei Tan, Qianqian Yang, Jiwen Wang, Jianhao Gao, and Liangpei Zhang.  Deep learning in environmental remote sensing: Achievements and challenges.  Remote Sensing of Environment, 241:111716, 2020.
  * [26] Kai Zhang, Jingyun Liang, Luc Van Gool, and Radu Timofte.  Designing a practical degradation model for deep blind image super-resolution, 2021.
  * [27] Xuaner Zhang, Qifeng Chen, Ren Ng, and Vladlen Koltun.  Zoom to learn, learn to zoom.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3762–3770, 2019.
  * [28] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu.  Image super-resolution using very deep residual channel attention networks.  In Proceedings of the European conference on computer vision (ECCV), pages 286–301, 2018.
  * [29] Ruofan Zhou and Sabine Susstrunk.  Kernel modeling super-resolution on real low-resolution images.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2433–2443, 2019.


# Chain-of-Thought Prompt Distillation for Multimodal Named Entity Recognition
and Multimodal Relation Extraction

Feng Chen
Ant Group
Hangzhou, China
<EMAIL_ADDRESS>
&Yujian Feng
Nanjing University of Posts and Telecommunications
Nanjing, China
<EMAIL_ADDRESS>

###### Abstract

Multimodal Named Entity Recognition (MNER) and Multimodal Relation Extraction
(MRE) necessitate the fundamental reasoning capacity for intricate linguistic
and multimodal comprehension. In this study, we explore distilling the
reasoning ability of large language models (LLMs) into a more compact student
model by generating a chain of thought (CoT) – a sequence of intermediate
reasoning steps. Specifically, we commence by exemplifying the elicitation of
such reasoning ability from LLMs through CoT prompts covering multi-grain
(noun, sentence, multimodality) and data-augmentation (style, entity, image)
dimensions. Subsequently, we present a novel conditional prompt distillation
method to assimilate the commonsense reasoning ability from LLMs, thereby
enhancing the utility of the student model in addressing text-only inputs
without the requisite addition of image and CoT knowledge. Extensive
experiments reveal that our approach attains state-of-the-art accuracy and
manifests a plethora of advantages concerning interpretability, data
efficiency, and cross-domain generalization on MNER and MRE datasets.

##  1 Introduction

Figure 1: Illustration of retrieving knowledge from Wikipedia, sample database
and LLMs.

Multimodal named entity recognition (MNER) Wang et al. (2022c); Lu et al.
(2022) and multimodal relation extraction (MRE) Chen et al. (2022); Zheng et
al. (2021a) aim to use auxiliary visual clues from images to improve the
recognition of multisense or out-of-vocabulary words/relations. However, text-
image pairs, amassed from diverse real-world domains such as social media,
film critiques, and news articles, pose challenges in comprehending linguistic
context and multimodal relationships. A majority of prior methodologies Wang
et al. (2021b); Zhao et al. (2022); Wang et al. (2022a) employ retrieval
augmentation from Wikipedia or sample databases to seek pertinent knowledge
concerning images and text, thereby facilitating model reasoning. For
instance, KB-NER Wang et al. (2021b) builds a multilingual knowledge base
based on Wikipedia to provide related context to the NER model. MoRe Wang et
al. (2022a) utilizes KNN with ViT-B/32 from CLIP Radford et al. (2021) to
retrieve the top-k related images from Wikipedia. Consequently, the knowledge
procured from these methods exhibits inconsistencies with the current domain
Kai Zhang (2021). For example, the definition of ‘Harry Potter’ in Figure 1
(a) is misleading, and the retrieved images in (b) tend to be irrelevant for
understanding the query sample. Additionally, these approaches are indirect
for interpreting image-text pairs, given that the retrieved knowledge might
exhibit semantic discrepancies from the query in the task at hand. As shown in
Figure 1 (b), the retrieved text from the sample database is hard for
understanding the original example. Finally, few of the existing methods
provide auxiliary multimodal knowledge that jointly explains the text and
image, which is important for MNER and MRE.

Recently, large-scale models have demonstrated remarkable performance on
intricate tasks that simulate the reasoning process one might employ when
addressing a problem Yao et al. (2023). Thus, this study investigates the
distillation of the reasoning capabilities of large language models (LLMs)
into a small model through Chain-of-Thought Prompt Distillation (CoTPD).
Previous researches Brown et al. (2020); Kojima et al. (2022); Wei et al.
(2022) reveal that the chain of thought (CoT) approach favorably elicits
multi-step reasoning abilities from LLMs. By generating intermediate natural
language rationales that culminate in the final response via CoT prompting,
these chain of thought demonstrations enable models to delineate a reasoning
pathway that deconstructs intricate reasoning into several simpler steps. In
this paper, our objective is to harness CoT prompts to synthesize explicit and
direct CoT knowledge to understand each sample, and subsequently distill the
reasoning capabilities of LLMs into the student model by using such CoT
knowledge.

How is CoT knowledge synthesized? We propose prompting LLMs to interpret each
sample in multi-grain, and data-augmentation perspectives. We term the
combination of demonstrations with respect to different CoT prompts as CoT
knowledge. Multi-grain CoT knowledge includes noun, sentence, and
multimodality perspectives. The noun perspective inquires about the
definitions of potential entities and specialized vocabulary in the text. The
sentence perspective elucidates ambiguous semantics in the original text and
supplies necessary background information. The multimodality perspective
enables LLMs to interpret the correlation between images and text, explicitly
and jointly determining whether an image is helpful for understanding the
text.

In addition to multi-grain reasoning, our method inherits the zero-shot
reasoning abilities of LLMs through data augmentation. Existing text-based NER
methods Chen et al. (2021, 2020) typically employ rule-based data augmentation
to address low-resource and cross-domain challenges. However, the augmented
sample is usually homogeneous and unrealistic. In this study, we extend this
strategy to the multimodal case through fact-based entity, style, and image
augmentation. LLMs are encouraged to perform augmentation according to common
sense, enabling the generation of augmented samples that are both diverse and
realistic. To the best of our knowledge, our research is the first attempt to
utilize multimodal data augmentation in MNER and MRE.

How to distill reasoning ability using CoT knowledge? We believe, combining
text-image pair with CoT knowledge as knowledge-enhanced input, i.e., [TXT] +
[IMG] + [Knowledge], a small model can easier recognize the entity and
relation. As shown in Figure 1 (c), CoT knowledge can exceptionally infer the
vague relation between ‘Harry Potter’ and the boy. However, such reasoning
ability of LLM is not inherited by the small model, since it just simply
summarizes the extensive analysis to the final answer, rather than figuring
everything out by itself. To solve this issue, we propose a conditional prompt
distillation. Specifically, we leverage prompt learning to hint the student
model by combining text and a learnable conditional prompt as input, i.e.,
[TXT] + [Prompt]. By aligning the output distributions predicted from the
knowledge-enhanced input view and the prompt-enhanced input view, such
contextual CoT knowledge is expected to be distilled in the parameterized
prompt. It allows our method to be more practical in dealing with text-only
inputs.

The contributions of our method can be summarized in three aspects:

1 We propose a simple but effective method to distill the reasoning ability of
LLM to student model, which largely improves the performance with trivial
additional cost.

2 Additionally, we propose a fact-based multimodal data augmentation strategy,
which exhibits effectiveness in low-resource and cross-domain settings.

3 Our method substantially surpasses existing state-of-the-art approaches with
significant improvement in MNER and MRE.

Figure 2: Overview of extracting multi-grain CoT knowledge and data
augmentation CoT knowledge via querying LLMs with noun, sentence,
multimodality, style, entity, and image CoT prompts.

##  2 Related Work

Chain-of-Thought: Recently, chain of thought (CoT) has been widely used to
elicit the multi-step reasoning ability of LLMs, which encourages the LLMs to
generate intermediate reasoning chains for solving a problem. Wei et al.
(2022) prompts LLMs with ‘Let’s think step by step’ to facilitate arithmetic,
commonsense, and symbolic reasoning. Tree-of-Thoughts Yao et al. (2023)
explores performing deliberate decision-making by considering multiple
different reasoning paths and self-evaluating choices. In this paper, we
propose to use CoT to interpret each sample where the response from LLM is
used as reliable knowledge to help the model understand.

Knowledge-based NER and RE Existing NER and RE methods tend to exploit
additional knowledge to assist model reasoning. For example, Wang et al.
(2021b) obtains external contexts of a sentence by retrieving and selecting a
set of semantically relevant texts through a search engine. Later, Wang et.al
Wang et al. (2022a) extend it to the visual case that MoRe utilizes KNN with
ViT-B/32 from CLIP to retrieve the top-k related images from Wikipedia.
However, the retrieved knowledge is still vague or misleading for a model to
understand. Thus, we propose to use LLMs as knowledge providers which can
accurately explain every detail of each sample.

##  3 Our Method

###  3.1 Motivation

Large language models (LLMs), with their capacity to interpret and process
human language in a manner akin to human cognition, facilitate the relation
understanding of complex tasks. However, the enormous size of LLMs renders
them impractical for industrial use. In the present study, we endeavor to
explore an efficient methodology to distill the reasoning prowess of LLMs into
a compact model that retains superior performance while ensuring minimal
inference latency.

However, traditional knowledge distillation methods Gou et al. (2021) are
incompatible to achieve this goal. The reasons are mainly twofold. (1)
Prevailing LLMs are predominantly generative models that either produce
unstructured output logits Touvron et al. (2023) or are solely accessible
through official API 111https://openai.com/blog/chatgpt. (2) Existing LLMs
exhibit suboptimal performance in NER and RE tasks Li et al. (2023a); Kai
Zhang (2023), attributable to their lack of training under task-specific
supervision. To address these issues, we introduce Chain-of-Thought Prompt
Distillation (CoTPD), a novel approach designed to distill the explicit CoT
knowledge into the student model. This method seeks to facilitate knowledge
transfer from the parameterized LLM $\stackrel{{\scriptstyle
prompt}}{{{\longrightarrow}}}$ contextual demonstration
$\stackrel{{\scriptstyle distillation}}{{{\longrightarrow}}}$ parameterized
student model. Moreover, by training on NER and RE tasks, the student model
can obtain promising performance with the help of CoT knowledge.

In addition, existing methods Wang et al. (2022c); Xu et al. (2022); Li et al.
(2020) typically demonstrate their intuition and superiority through case
studies or Class Activation Mapping (CAM) visualization. However, these
strategies still fall short in providing comprehensive interpretability about
how models interpret the multimodal context to reach the final prediction Sun
et al. (2019). In this investigation, we demonstrate that CoT knowledge, which
is akin to how humans understand each sample, benefits the model performance
in various domains and tasks.

###  3.2 Overview of Our Method

Given an text and image pair $(\bm{x},\bm{I})$ where
$\bm{x}=\\{x_{1},x_{2},..,x_{n}\\}$ has $n$ words, our method aspires to
predict the outputs $\bm{y}$, which could represent the label sequence or the
relations for NER and RE tasks respectively. As shown in Figure 2, we
initially employ image caption technology, i.e., BLIP2 Li et al. (2023b), to
convert image $\bm{I}$ into an image caption. Subsequently, we utilize
langchain222https://github.com/hwchase17/langchain with preordained CoT
prompts to query the LLM, thereby acquiring CoT knowledge $\bm{c}$ pertaining
to each sample. Ultimately, the student language model amalgamates the text-
image pair with CoT knowledge as the knowledge-enhanced view, and concatenates
the text with a learnable conditional prompt to create a prompt-enhanced view.
The objective is to minimize the Kullback-Leibler divergence over the
probability distributions of the two views, thereby distilling the contextual
CoT knowledge into the parameterized prompt.

###  3.3 CoT Prompts for LLMs

Multi-grain CoT Prompts. To fully understand the text-image pair, we design
CoT prompts in noun, sentence, and multimodality perspectives to query
LLMs333Weaker LLM may need accurate descriptions in prompts. We take GPT4 in
the Twitter dataset as an example., as shown in Figure 2.

Noun: We query LLMs with ‘Help me explain the meaning of special words for
understanding. + $\bm{x}$’, which can explain potential entities, slang, and
terminology.

Sentence: For each sample, we query LLMs with ‘Explain the sentence to me with
necessary background. + $\bm{x}$’ . It can explain the sentiment, cause, and
subject of users.

Multimodality: Similarly, we obtain multimodal relation by prompt LLMs with
‘What is the relation between the text and the attached image? + $\bm{x+I}$’.
In this way, LLM can illustrate the vague relation between entity and object
and explicitly determine whether the visual information is noise.

Data Augmentation CoT Prompts. We also leverage LLMs to do fact-based
multimodal data augmentation, involving linguistic style, entity, and image
perspectives.

Style: We query LLMs with ‘Transform the sentence in Twitter style without
changing the meaning. + $\bm{x}$’ It diversifies the textual expression with
consistent entity and meaning.

Entity: Initially, we adhere to Wu et al. (2022) to substitute the entity in
$\bm{x}$ with an entity of the same type sourced from the MNER dataset. For
instance, we replace the entity ’Tobey Maguire [PER]’ with ’Barack Obama
[PER]’. Then, we assess the factual accuracy of the augmented sample by
prompting the LLM with the query ’Whether the sentence is possible in fact,
answer yes or no. + $\bm{x}$’. We retain only the samples that receive a
positive response indicating factual plausibility.

Image: We generate the visual information of text by querying LLMs ‘What is a
possible image with the text in a tweet? + $\bm{x}$’

We unify multi-grain CoT knowledge with the original image and text pair. For
data augmentation CoT knowledge, we reorganize it as a new sample. Notably,
the total knowledge may exceed the max token limits for the student model,
therefore, we first summarize the over-length knowledge.

###  3.4 Conditional Prompt Distillation

To distill CoT knowledge $\bm{c}$ into a compact model, we first try multi-
view alignment Wang et al. (2021a) to match the txt-only output distribution
and text-image-CoT output distribution. However, we find it results in
suboptimal performance (Sec 4.3). We believe the reasons are twofold: (1) CoT
knowledge is higher-level context than previous retrieve knowledge. (2) The
knowledge is personalized which needs extra parameters to distill. Therefore,
inspired by prompt learning Li and Liang (2021); Zhou et al. (2022a), we
propose a conditional prompt distillation with respect to the text to hint the
student model adjustably.

We introduce our conditional prompt generator, which is composed of a
transformer decoder with learnable queries $\bm{q}\in\mathbb{R}^{N\times d}$.
The conditional prompt $\bm{p}$ is generated by:

$\displaystyle\bm{p}=Softmax(\frac{(\bm{q}W_{q})(\bm{x}W_{k})^{T}}{\sqrt{d}})\cdot\bm{x}W_{v},$
(1)

where $W_{q},W_{k}$, and $W_{v}$ are learnable projectors for query, key, and
value. We concatenate text, image caption, and CoT knowledge as the knowledge-
enhanced view $[\bm{x};\bm{I};\bm{c}]$. Similarly, the prompt-enhanced view is
composed by concatenating text and conditional prompt as $[\bm{x};\bm{p}]$. We
feed these two views to the same text encoder $E$ separately:

$\begin{split}H_{k}=\\{h_{1},..,h_{n},..,h_{n+l}\\}=E([\bm{x};\bm{I};\bm{c}]),\\\
H_{p}=\\{h_{1},..,h_{n},..,h_{n+l}\\}=E([\bm{x};\bm{p}]),\end{split}$ (2)

where $H=\\{h\\}_{1}^{n+l}$ denotes the output of encoder with max token limit
$n+l$. The subscripts $k$ and $t$ represent knowledge-enahcned and prompt-
enhanced view respectively. The former $n$ tokens correspond to the embedding
of the text part in two views, which are used for following NER or RE
prediction.

Then, we reduce the gap between these two views over the output distributions,
so that the CoT knowledge and image are distilled into the student model under
specific task. We denote it as conditional prompt distillation loss:

$\begin{split}\mathcal{L}_{CPD}&=KL\left(p_{\theta}(\boldsymbol{y}\mid\bm{x},\bm{I},\bm{c})\|p_{\theta}(\boldsymbol{y}\mid\bm{x},\bm{p})\right)\\\
&=KL(H_{k}[1:n]||H_{t}[1:n])\end{split}$ (3)

Besides, for MNER and MRE, we use the negative log-likelihood (NLL) as the
training loss for the knowledge-enhanced view with gold labels $\bm{y}^{*}$:

$\mathcal{L}_{\mathrm{NLL}}=-\log
p_{\theta}\left(\boldsymbol{y}^{*}\mid\boldsymbol{x},\boldsymbol{I},\boldsymbol{c}\right)$
(4)

Thus, the total loss $\mathcal{L}_{total}$ is the combination of
$\mathcal{L}_{CPD}$ and $\mathcal{L}_{NLL}$ with coefficient factor $\alpha$:

$\mathcal{L}_{total}=\mathcal{L}_{NLL}+\alpha\cdot\mathcal{L}_{\mathrm{CPD}}$
(5)

##  4 Experiments

###  4.1 Settings

Datasets. For MNER evaluation, we verify the effectiveness of our method on
Twitter2015 Zhang et al. (2018), Twitter2017 Yu et al. (2020), SNAP Lu et al.
(2018) and WikiDiverse Wang et al. (2022a). These datasets containing
4,000/1,000/3,357, 3,373/723/723 and 4,290/1,432/1,459, 6,312/755/757
sentences in train/development/test split respectively. The former three
datasets are collected from social media domain, while WikiDiverse is
reorganized by Wang et al. (2022a) from Wikinews as a multi-modal NER dataset.
For MRE, we use MNRE dataset Zheng et al. (2021b) constructed by user tweets,
which contains 12,247/1,624/1,614 sentences in train/development/test split.

Model Configuration. To fairly compare with most of the recent works, we adopt
BERT-base-uncased as our text encoder. Besides, we follow Wang et al. (2021a,
2022a) to adopt XLM-RoBERTa-large (XLMR) to achieve state-of-the-art
performance. For LLM prompting, we use langchain to query each sample in a
dialogue way and record the answer of LLMs as CoT knowledge. The version of
ChatGPT used in our experiments is gpt-3.5-turbo and the version of GPT4 is
gpt-4. All LLMs used in our method set the sampling temperature to 0 for
stable output.

Training Configuration. Our approach is implemented in Pytorch framework on an
NVIDIA A100 GPU. We adopt AdamW optimizer to minimize the loss function.
Besides, we follow Wang et al. (2022a) to grid search the learning rate of our
models within [$1\times 10^{-6}$, $5\times 10^{-5}$]. The max length of
sentence input is set to 512 to cover more CoT knowledge. The model undergoes
a fixed 15 epochs of training with 32 mini-batch.

Baseilne Model and Variants. We refer to a model training and testing with
[TXT]+[IMG] input as our baseline model. For the knowledge variants, we take
retrieve knowledge from MoRe Wang et al. (2022a) as Retri, and knowledge
queried from Wikipedia Wang et al. (2021b) as Wiki, and our CoT knowledge as
CoT. For Retri, MoRe provides image-retrieved knowledge Retritxt and text-
retrieved knowledge Retriimg. For distillation variants, we refer to multi-
view alignment from ITA Wang et al. (2021a) as MV. Furthermore, in our method,
considering different kinds of learnable prompts, we compare prefix Li and
Liang (2021) and unconditional prompt Zhou et al. (2022b), which are denoted
as PrefixD and UPD respectively, to verify the effectiveness of the proposed
conditional prompt distillation (CPD).

###  4.2 Main Result

We compare our approach with other BERT-based Wang et al. (2022c); Lu et al.
(2022); Zhao et al. (2022) and XLMR-based Wang et al. (2021a, 2022b, 2022a,
2022c) state-of-the-art methods on MNER and MRE. To fully show the
effectiveness of our approach, we evaluate ChatGPT and GPT4 on Twitter2015,
Twitter2017 and MNRE. For our approach, we also show the model performance
using different LLM teacher and student models. As shown in Table 1, we
observe LLMs are inferior to existing state-of-the-art methods, but the CoT
knowledge significantly boosts our method. We think the training data of LLMs
includes the dataset but the training of LLM is not in NER or RE supervision.
Therefore, LLMs can generally understand each sample but still underperform in
specific tasks. Besides, compared with other previous SOTA methods, our
approach suppresses them by a large margin. Compared to MoRe, our method
achieves 0.82% and 5.96% improvement on Twitter2015 and MNRE respectively. We
attribute such significant gain to the direct and comprehensive CoT knowledge
from LLM, so that the student model can easily understand the multimodal
sample, instead of analogically learning from the related samples. We also
observe that the ability of LLMs and student models significantly influences
the final performance. For example, with respect to the same student XMLR
model, GPT4 can generally provide a further 0.4%-1.2% improvement over
ChatGPT. Similarly, a powerful student model also brings large performance
boosts.

Method T-15 T-17 SNAP Wiki MNRE BERT CAT-MNER 75.41 85.99 - - - R-GCN 75.00
87.11 - - - FMIT 76.25 86.79 - - - XMLR ITA 78.03 89.75 90.15 76.87 66.89
promptMNER 78.60 90.27 - - - CAT-MNER 78.72 90.47 - - - MoRe 79.21 90.67 91.10
79.33 68.60 LLM ChatGPT 50.21 57.50 - - 35.20 GPT4 57.98 66.61 - - 42.11
our(ChatGPT-BERT) 76.53 87.79 88.12 76.42 66.42 our(GPT4-BERT) 76.90 88.27
88.51 77.09 67.88 our(ChatGPT-XMLR) 80.31 91.78 91.97 80.02 73.67
our(GPT4-XMLR) 80.03 92.20 92.41 81.20 74.56

Table 1: Performance comparison (F1 score) with existing methods and LLMs. The
evaluation of LLM is based on Li et al. (2023a) with multimodal modification.
For our method, such as ‘ours(GPT4-XMLR)’, the two models in brackets denote
the teacher LLM and student model respectively.

T-15 T-17 SNAP Wiki MNRE our method 80.03 92.20 92.41 81.20 74.56 w/o Noun
78.63 89.70 90.56 77.32 70.20 w/o Sentence 79.04 89.87 91.21 79.09 69.41 w/o
Multimodality 79.10 91.20 91.56 79.32 73.21 w/o Style 79.65 91.70 91.81 81.04
74.16 w/o Entity 79.00 90.98 92.35 79.56 - w/o Image 79.65 91,43 91.40 79.66
73.98

Table 2: Effect of multi-grain and data augmentation CoT knowledge. Notably,
‘w/o Image’ means the augmented samples do not have any visual information.

###  4.3 Ablation Study

Effectiveness of Different CoT Knowledge. Multi-grain CoT knowledge focuses on
explaining each sample from different perspectives, leading to significant
improvement by enhancing the understanding of the student model. Specifically,
as shown in Table 2, noun knowledge contributes more to MNER than MRE while
sentence knowledge brings more improvement to MRE. We believe these two kinds
of knowledge meet the demand of two tasks where MNER focuses on entity mention
and MRE concentrates on the entity relation. Besides, it is worth noting that
data augmentation CoT knowledge, i.e., style, entity, and image augmentation,
contributes less than multi-grain CoT knowledge. It is because data
augmentation mainly generates more samples for training, rather than directly
explaining each sample. Therefore, it is more effective in cross-domain and
low-resource cases, which will be elaborated on next section.

T-15 T-17 SNAP Wiki MNRE baseline 77.21 88.77 89.53 76.04 65.85 baseline+Wiki
78.15 89.60 - - 67.21 baseline+Retritxt 78.05 89.87 90.44 78.24 -
baseline+Retriimg 78.35 90.17 90.02 77.94 67.60 baseline+CoT 80.24 91.92 92.40
81.00 74.74 baseline+Wiki+MV 78.26 89.22 - - 67.01 baseline+Retritxt+MV 78.29
90.47 90.31 77.82 - baseline+Retriimg+MV 78.20 90.12 90.11 78.03 67.54
baseline+CoT+MV 79.62 91.90 91.98 81.32 70.41 baseline+CoT+UPD 79.34 91.02
91.50 80.00 70.45 baseline+CoT+PrefixD 79.62 91.57 92.20 80.62 71.62
baseline+CoT+CPD 80.03 92.20 92.41 81.20 74.56

Table 3: Ablation study of knowledge variants and distillation variants.

Comparison with Previous Retrieved Knowledge. In Table 3, we also report the
results using retrieved knowledge to verify the effectiveness of our CoT
knowledge. For retrieved knowledge from Wikipedia and the sample database, our
CoT knowledge is better than them by a large margin. For example, CoT
suppresses Wiki with 2.09% gains and outperforms Retritxt with 2.19%
improvements on Twitter2015. It demonstrates that such direct CoT knowledge is
more effective to understand each sample. Furthermore, we investigate how
knowledge difference influences the final prediction in Appendix. It shows
accurate and direct CoT knowledge contains better potential in
interpretability and performance over existing methods.

Distilling Knowledge into Student Model. As shown in Table 3, we also compare
the results of different knowledge+distillation variants on MNER and MRE. It
is obvious that multi-view alignment is unsuitable for distilling CoT
knowledge to the student model, where ‘baseline+CoT+MV’ is 4.33% lower than
‘baseline+CoT’ on MNRE. We think such explicit and direct knowledge is higher-
level than the retrieved knowledge. Thus, distilling it into the student model
is harder. Alternatively, our conditional prompt distillation can maintain the
performance with minimal decrease. Moreover, we test other two kinds of
prevailing learnable prompts for distillation, i.e., prefix and unconditional
prompt. It shows the proposed conditional prompt is better than them. We think
the personalized/conditional prompt with respect to the text can hint the
student model easier.

###  4.4 Detailed Analysis

Cross-domain Generalization. The difference in type distribution and data
characteristics often brings significant performance gaps in practice. Thus,
we examine the cross-domain generalization of our method in Table 4. First,
without data augmentation, our method achieves comparable results as the
previous best model where our method suppresses CAT-MNER with 0.68% F1
improvement but is inferior to it with 0.28% decrease in two cross-domain
settings respectively.

Second, when the entity augmentation uses entities from the source dataset
(in-domain aug), our method achieves 1.59% and 0.73% F1 gains over CAT-MNER.
It demonstrates the data augmentation strategy can substantially boost
generalization ability. Besides, in this work, we aim to let the student model
inherit the zero-shot ability of LLM. Therefore, we first query LLMs to
provide a list of 400 Twitter entities, covering four types but without
overlapping the entities in the source set, and then apply the novel entities
for entity replacement. We refer to this implementation as zero-shot
augmentation. As shown in Table 4, augmenting novel entities can further
provide a 0.3%-0.6% F1 improvement. This demonstrates the effectiveness of
utilizing data-augmentation CoT knowledge for achieving cross-domain
generalization

Methods Twitter2017$\rightarrow$Twitter2015
Twitter2015$\rightarrow$Twitter2017 P R F1 P R F1 UMT 64.67 63.59 64.13 67.80
55.23 60.87 UMGF 67.00 62.18 66.21 69.88 56.92 62.74 FMIT 66.72 69.73 68.19
70.65 59.22 64.43 CAT-MNER 74.86 63.01 68.43 70.69 59.44 64.58 our(w/o aug)
75.76 64.06 69.11 70.08 59.67 64.30 our(w/ in-domain aug)) 76.51 64.21 70.02
71.64 59.49 65.31 our (w/ zero-shot aug) 76.04 66.28 70.40 71.34 60.16 66.05

Table 4: Comparison of the cross-domain generalization ability. For fairness,
we use GPT4 as teacher LLM and BERTbase as student backbone. Results of other
methods are from Wang et al. (2022c); Li et al. (2020). Figure 3: Influence of
augmentation times with $\\{50,100,200,400\\}$ training data on Twitter2015
and Twitter2017. Times=0 means using training data without any augmentation.

Augmentation Times for Data Efficiency. The size of augmented samples is
essential for final model performance. Typically, we randomly sample
$\\{50,100,200,400\\}$ image-text pairs from Twitter2015 and Twitter2017 with
different augmentation times to examine the effectiveness of our multimodal
augmentation strategy. As depicted in Figure 3, increased augmentation yields
improved performance, particularly when larger examples are utilized. Notably,
our augmentation strategy demonstrates a linear enhancement with the
progression of augmentation times. We attribute this improvement to the
diversity and authenticity from linguistic style, factual entity replacement
and image imagination, rather than simply substituting entities Wu et al.
(2022).

Influence of BLIP2. We observe that BLIP2 Li et al. (2023b) has basic visual-
entity alignment ability. It means it may recognize the entity mentions in the
image, which may be unfair for performance comparison. We discuss the
influence of BLIP2 in Table 5 and observe that, for both ITA and our method,
BLIP2 provides 0.3%-0.5% improvement over VinVL Zhang et al. (2021) on two
datasets. However, even using VinVL for image caption, our method can still
achieve significant improvement over the previous best method MoRe. We think
LLM can infer visual-entity relation from the general noun description, as
shown in the multimodal CoT knowledge in Figure 2.

Image Caption | Methods | T-15 | T-17
---|---|---|---
VinVL | ITA | 78.03 | 89.75
our | 79.65 | 92.03
BLIP2 | ITA | 78.32 | 90.22
our | 80.03 | 92.20
Table 5: Influence of using different image caption methods. Figure 4: Case
studies of how CoT knowledge can help model predictions. The subscript $know$
and $CPD$ denote inference with and without CoT knowledge respectively.

Case Study. We showcase how CoT knowledge can help improve the predictive
performance of the model. As illustrated in Figure 4, LLM first provides a
detailed explanation for ‘special’ words in the text, largely enriching the
linguistic semantics. Especially for abbreviation, slang, and hashtag, LLMs
directly return crucial definitions for easy understanding of the whole
context while KB-NER and MoRe only provide related or vague context needing
further consideration. For multimodal CoT knowledge, our method can
distinguish the relatedness of text-image pair. For example, in (b) and (d),
the images do not have a corresponding entity in the text. However, LLMs can
also interpret the long-range image-text relation.

Furthermore, compared with the baseline model and MoRe, our method can easily
recognize hard entities like ‘North and South Korea’ in (b) and
‘fikile_nhleko’ in (c). Besides, the correct prediction with and without CoT
knowledge indicates that the student model inherits the reasoning ability of
LLMs through conditional prompt distillation.

Limitations and Boarder Impacts. We notice that the quality of generated CoT
knowledge heavily depends on the well-posed LLM that understands the prompt
and example comprehensively. We believe next-generation multimodal large
models, such as GPT5, will make the CoT knowledge more reliable.

Our method is simple yet effective to distill the general task-agonist
knowledge from LLMs to a small model. It can be easily expanded to other tasks
which need sophisticated reasoning.

##  5 Conclusion

In this paper, we propose a novel CoT knowledge covering multi-grain
understanding of examples and data-augmentation generalization from LLMs.
Furthermore, we introduce conditional prompt distillation to distill the
reasoning ability of LLMs to a compact student model. Extensive experiments on
five datasets show that our method outperforms all existing state-of-the-art
methods and manifests a plethora of advantages concerning interpretability,
data efficiency, and cross-domain generalization.

## References

  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  _ArXiv_ , abs/2005.14165.
  * Chen et al. (2020) Jiaao Chen, Zhenghui Wang, Ran Tian, Zichao Yang, and Diyi Yang. 2020.  Local additivity based data augmentation for semi-supervised ner.  _arXiv preprint arXiv:2010.01677_.
  * Chen et al. (2021) Shuguang Chen, Gustavo Aguilar, Leonardo Neves, and Thamar Solorio. 2021.  Data augmentation for cross-domain named entity recognition.  _arXiv preprint arXiv:2109.01758_.
  * Chen et al. (2022) Xiang Chen, Ningyu Zhang, Lei Li, Shumin Deng, Chuanqi Tan, Changliang Xu, Fei Huang, Luo Si, and Huajun Chen. 2022.  Hybrid transformer with multi-level fusion for multimodal knowledge graph completion.  _arXiv preprint arXiv:2205.02357_.
  * Gou et al. (2021) Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021.  Knowledge distillation: A survey.  _IJCV_ , 129:1789–1819.
  * Kai Zhang (2021) Ruobing Xie Xu Han Zhiyuan Liu Fen Lin Leyu Lin Maosong Sun Kai Zhang, Yuan Yao. 2021.  Open hierarchical relation extraction.  In _NAACL_.
  * Kai Zhang (2023) Yu Su Kai Zhang, Bernal Jiménez Gutiérrez. 2023.  Aligning instruction tasks unlocks large language models as zero-shot relation extractors.  In _Findings of ACL_.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  _ArXiv_ , abs/2205.11916.
  * Li et al. (2023a) Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang. 2023a.  Evaluating chatgpt’s information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness.  _arXiv preprint arXiv:2304.11633_.
  * Li et al. (2023b) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  _arXiv preprint arXiv:2101.00190_.
  * Li et al. (2020) Xiaonan Li, Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2020.  Flat: Chinese ner using flat-lattice transformer.  _arXiv preprint arXiv:2004.11795_.
  * Lu et al. (2018) Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. 2018.  Visual attention model for name tagging in multimodal social media.  In _ACL_ , pages 1990–1999.
  * Lu et al. (2022) Junyu Lu, Dixiang Zhang, and Pingjian Zhang. 2022.  Flat multi-modal interaction transformer for named entity recognition.  _arXiv preprint arXiv:2208.11039_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  In _ICML_ , pages 8748–8763.
  * Sun et al. (2019) Lin Sun, Kai Zhang, Fule Ji, and Zhenhua Yang. 2019.  TOI-CNN: a solution of information extraction on Chinese insurance policy.  In _NAACL_.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Wang et al. (2022a) Xinyu Wang, Jiong Cai, Yong Jiang, Pengjun Xie, Kewei Tu, and Wei Lu. 2022a.  Named entity and relation extraction with multi-modal retrieval.  In _EMNLP_.
  * Wang et al. (2021a) Xinyu Wang, Min Gui, Yong Jiang, Zixia Jia, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021a.  Ita: Image-text alignments for multi-modal named entity recognition.  _arXiv preprint arXiv:2112.06482_.
  * Wang et al. (2021b) Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021b.  Improving named entity recognition by external context retrieving and cooperative learning.  _arXiv preprint arXiv:2105.03654_.
  * Wang et al. (2022b) Xuwu Wang, Junfeng Tian, Min Gui, Zhixu Li, Jiabo Ye, Ming Yan, and Yanghua Xiao. 2022b.  Prompt-based entity-related visual clue extraction and integration for multimodal named entity recognition.  In _International Conference on Database Systems for Advanced Applications_ , pages 297–305.
  * Wang et al. (2022c) Xuwu Wang, Jiabo Ye, Zhixu Li, Junfeng Tian, Yong Jiang, Ming Yan, Ji Zhang, and Yanghua Xiao. 2022c.  Cat-mner: Multimodal named entity recognition with knowledge-refined cross-modal attention.  In _ICME_ , pages 1–6.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, Quoc Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  _ArXiv_ , abs/2201.11903.
  * Wu et al. (2022) Linzhi Wu, Pengjun Xie, Jie Zhou, Meishan Zhang, Ma Chunping, Guangwei Xu, and Min Zhang. 2022.  Robust self-augmentation for named entity recognition with meta reweighting.  In _NAACL_ , pages 4049–4060.
  * Xu et al. (2022) Bo Xu, Shizhou Huang, Chaofeng Sha, and Hongya Wang. 2022.  Maf: A general matching and alignment framework for multimodal named entity recognition.  In _WSDM_ , pages 1215–1223.
  * Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023.  Tree of thoughts: Deliberate problem solving with large language models.  _arXiv preprint arXiv:2305.10601_.
  * Yu et al. (2020) Jianfei Yu, Jing Jiang, Li Yang, and Rui Xia. 2020.  Improving multimodal named entity recognition via entity span detection with unified multimodal transformer.  ACL.
  * Zhang et al. (2021) Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021.  Vinvl: Revisiting visual representations in vision-language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5579–5588.
  * Zhang et al. (2018) Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018.  Adaptive co-attention network for named entity recognition in tweets.  In _AAAI_ , volume 32.
  * Zhao et al. (2022) Fei Zhao, Chunhui Li, Zhen Wu, Shangyu Xing, and Xinyu Dai. 2022.  Learning from different text-image pairs: A relation-enhanced graph convolutional network for multimodal ner.  In _MM_ , pages 3983–3992.
  * Zheng et al. (2021a) Changmeng Zheng, Junhao Feng, Ze Fu, Yiru Cai, Qing Li, and Tao Wang. 2021a.  Multimodal relation extraction with efficient graph alignment.  _MM_.
  * Zheng et al. (2021b) Changmeng Zheng, Zhiwei Wu, Junhao Feng, Ze Fu, and Yi Cai. 2021b.  Mnre: A challenge multimodal dataset for neural relation extraction with visual evidence in social media posts.  In _ICME_ , pages 1–6.
  * Zhou et al. (2022a) Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022a.  Conditional prompt learning for vision-language models.  In _CVPR_ , pages 16816–16825.
  * Zhou et al. (2022b) Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022b.  Learning to prompt for vision-language models.  _IJCV_ , 130(9):2337–2348.