# TAP: The Attention Patch for Cross-Modal Knowledge Transfer
from Unlabeled Data

Yinsong Wang  Shahin Shahrampour

###### Abstract

This work investigates the intersection of cross-modal learning and semi-
supervised learning, where we aim to improve the supervised learning
performance of the primary modality by borrowing missing information from an
unlabeled modality. We investigate this problem from a Nadaraya–Watson (NW)
kernel regression perspective and show that this formulation implicitly leads
to a kernelized cross-attention module. To this end, we propose The Attention
Patch (TAP), a simple neural network plugin that allows data-level knowledge
transfer from the unlabeled modality. We provide numerical simulations on
three real-world datasets to examine each aspect of TAP and show that a TAP
integration in a neural network can improve generalization performance using
the unlabeled modality.

Cross-modal Learning, Semi-supervised Learning, Nadaraya-Watson Kernel
Regression, Attention

##  1 Introduction

In supervised learning, the difference between various learning frameworks can
be defined by the data availability at the learning stage using the data
partition in Fig. 1. For example, cross-modal learning aims at combining
complementary information from two input sources to improve learning tasks.
With the advancements in data collection and deep learning, cross-modal
learning has become more prevalent in different communities. The general
consensus is that data from different modalities, like data matrix
$\mathbf{Z}$ and $\mathbf{X}$ in Fig. 1, often contain “extra” information
with respect to each other that one can make use of. This consensus has
inspired different branches of cross-modal learning, including cross-modal
retrieval (Wang et al., 2016), cross-modal generative learning (Gu et al.,
2018; Peng & Qi, 2019), and cross-modal supervised learning (Castellano et
al., 2008; Gehler & Nowozin, 2009; Ramirez et al., 2011; Bucak et al., 2013).
Specifically, cross-modal supervised learning focuses on learning the function
mapping $f:\mathcal{X}\times\mathcal{Z}\rightarrow\mathcal{Y}$ using cross-
modal input $\mathbf{X}$ and $\mathbf{Z}$ and their corresponding labels
$\mathbf{y}$.

Despite the difference in focus, the general framework of cross-modal learning
is often the same. Given a pair of cross-modal inputs, one should devise a
model or learning algorithm for finding shared information, complementary
information, or a latent space that can describe both modalities based on the
matching pair information. The general cross-modal learning framework relies
on the alignment or weak alignment (sequence input) between modalities
$\mathbf{X}$ and $\mathbf{Z}$ to learn anything meaningful. Furthermore, most
cross-modal learning algorithms need both modalities as input at the time of
evaluation.

Figure 1: Visualization of data structure.
$\mathbf{X},\mathbf{X}^{\prime}\in\mathcal{X}$ and
$\mathbf{Z},\mathbf{Z}^{\prime}\in\mathcal{Z}$ are data from different
modalities. $\mathbf{y},\mathbf{y}^{\prime}\in\mathcal{Y}$ are the
corresponding labels.

The existence of alignment between modalities in the field of cross-modal
learning has never been a major concern in research works. However, it is not
hard to imagine that there are cases when one would know that a modality
contains “extra” information for a specific task, but it cannot be used in the
learning process due to reasons like expensive labeling costs, or the dataset
is sampled at a different time and space comparing to the primary dataset used
for this learning task.

On the other hand, the area of semi-supervised learning has been solely
focusing on utilizing unlabeled “extra” information. The motivation behind
semi-supervised learning is mainly the limited availability of labeled data.
Thus, different approaches have been devised for the purpose of utilizing
unlabeled data to improve the learning objective. That is, learning a function
$f:\mathcal{X}\rightarrow\mathcal{Y}$ using labeled data pairs
$\mathbf{X},\mathbf{y}$ and unlabeled data $\mathbf{X}^{\prime}$ (Fig. 1).
There also exists literature that studies the case where the full dataset,
both labeled and unlabeled, is cross-modal (Zhang et al., 2017; Yu et al.,
2018; Liang et al., 2020; Hong et al., 2020; Zhang et al., 2021), i.e.,
$\mathbf{X},\mathbf{Z},\mathbf{y},\mathbf{X}^{\prime},\mathbf{Z}^{\prime}$ are
all available during the learning stage.

However, there is no study for the case where only
$\mathbf{X},\mathbf{y},\mathbf{Z}^{\prime}$ are available during the learning
process, which is the main motivation of this work.

###  1.1 Contribution

  * •

This is the first work, to the best of our knowledge, that investigates the
intersection of cross-modal supervised learning and semi-supervised learning,
where the “extra” information $\mathbf{Z}^{\prime}$ is not only unlabeled and
unpaired but also comes from a different modality than the limited primary
data $\mathbf{X}$.

  * •

In Section 3, we propose the Nadaraya-Watson (NW) kernel regression
formulation of missing information estimation using unlabeled cross-modal
data. Borrowing two well-known techniques in cross-modal learning and semi-
supervised learning, we show that this formulation recovers a kernelized
version of the popular cross-attention mechanism (Vaswani et al., 2017).

  * •

Based on our observation, we propose The Attention Patch (TAP) neural network
plugin for cross-modal knowledge transfer from unlabeled modality. We show
that the NW kernel regression formulation yields an asymptotically vanishing
error. We further propose a batch training strategy from a practical
standpoint.

  * •

We provide detailed simulations on three real-world datasets to examine each
aspect of TAP and show a TAP integration in a neural network can improve
generalization performance using the unlabeled modality.

##  2 Related Literature

###  2.1 Cross-Modal Learning

Cross-modal learning focuses on learning with data from different modalities.
The most representative topic application-wise is cross-modal retrieval. This
topic focuses on finding relevant samples in one modality given a query in
another modality (Wang et al., 2016). The most important step in cross-modal
retrieval tasks is learning a coupled space that can correctly describe the
correlation between data points from different modalities. Traditional
techniques including canonical correlation analysis (Hardoon et al., 2004;
Andrew et al., 2013; Wang & Shahrampour, 2021), partial least squares (Geladi
& Kowalski, 1986; Cha, 1994), and bilinear model (Sharma et al., 2012;
Tenenbaum & Freeman, 2000) find a simple projection of the matching pairs that
minimizes certain pre-defined loss function. This framework, though have seen
several variants and extensions (Ngiam et al., 2011; Hong et al., 2015; Sohn
et al., 2014; Xu et al., 2015), remains the most popular framework in cross-
modal correlation learning.

In addition to cross-modal retrieval, cross-modal supervised learning also
follows the same principle. Examples include but are not limited to (Lin &
Tang, 2006; Evangelopoulos et al., 2013; Jing et al., 2014; Feichtenhofer et
al., 2016; Peng et al., 2017; Li et al., 2019). All of these models involve
coupled space learning through either implicit or explicit learning loss with
matching pairs of cross-modal inputs.

With the rise in popularity of natural language processing and time series
analysis, a new concept of weak alignment appears in cross-modal learning.
Weak alignment refers to the missing alignment of sub-components in instances
from different modalities (Baltrušaitis et al., 2018). More specifically, this
often means cross-modal input sequences with different sampling intervals or
orders. For example, in the case of vision and language models, a text
description sequence of a video will usually differ from the time and order of
information that appeared in the video (Venugopalan et al., 2015; Tsai et al.,
2019), or a text description of an image will need to map a sequence of words
to a collection of objects without any specific orders (Mitchell et al., 2012;
Kulkarni et al., 2013; Chen et al., 2015; Karpathy & Fei-Fei, 2015).

There exists a research direction that taps into instance-wise unalignment for
cross-modal learning, which is called non-parallel co-learning (Baltrušaitis
et al., 2018). Non-parallel co-learning aims at improving the model learned on
a single modality using another modality that is unaligned with the primary
data. However, this is a concept that has only been studied with very specific
applications, and it also requires the reference modality to be labeled during
the training process. For example, cross-modal transfer learning (Frome et
al., 2013; Kiela & Bottou, 2014; Mahasseni & Todorovic, 2016) mainly focuses
on transferring supervised pre-trained embedding networks for improved cross-
model prediction accuracy.

In a nutshell, almost all cross-modal learning frameworks focus on the case
where there are known alignments between different modalities of data at least
during the learning phase. Our work considers the case where both alignment
and labels do not exist during learning, and our proposed architecture can be
applied to arbitrary modalities.

###  2.2 Semi-Supervised Learning

Semi-supervised learning focuses on addressing the challenge of limited
labeled data availability in building a learning algorithm (Zhu, 2005; Van
Engelen & Hoos, 2020). Among all kinds of semi-supervised learning methods,
self-training is the closest class of methods that relates to our study. Self-
training refers to the class of methods that train a supervised learning
algorithm using labeled and unlabeled data together (Triguero et al., 2015).
This approach is usually done by assigning pseudo labels to the unlabeled
data, and jointly refining the supervised learning model and the pseudo labels
by iterative training (Yarowsky, 1995; Rosenberg et al., 2005; Dópido et al.,
2013; Wu et al., 2012; Tanha et al., 2017). For most traditional learners,
this means re-training the learning algorithm many times as the pseudo labels
are being updated. However, this pseudo-label training approach naturally
works with incremental learning algorithms like neural networks, where the
model is gradually learned through optimizing an objective function (Lee et
al., 2013; Berthelot et al., 2019; Zoph et al., 2020; Xie et al., 2020; Sohn
et al., 2020). Strictly speaking, all pseudo-labeling/self-training methods
focus on assigning labels in the prediction space to the data points in the
primary space that is the same as the labeled data.

There exists another line of research that studies semi-supervised learning in
the context of multi-view or multi-modal learning. For example, multi-view co-
training (Blum & Mitchell, 1998; Kiritchenko & Matwin, 2001; Wan, 2009; Du et
al., 2010) propose to train different classifiers on different views of the
same data. Multi-view co-training generally assumes the two views are
independent of each other but can perform equally well in a single-view
prediction task. The development of deep neural networks also introduced a lot
of cross-modal semi-supervised learning works, which mainly focus on computer
vision and natural language processing (Nie et al., 2017b, a, 2019; Jia et
al., 2020). Again, all of these mentioned works rely on the availability of
labels in all the modalities.

As we can see, all existing semi-supervised learning frameworks consider the
case where the unlabeled data resides in the same space or joint space as the
labeled data. Our work considers the case where the unlabeled data resides in
a completely different space than the labeled data.

##  3 Estimating the Missing Information

As discussed in the Introduction (Fig. 1), we are concerned with knowledge
transfer from unlabeled modality
$\mathbf{Z}^{\prime}\in\mathbb{R}^{n^{\prime}_{z}\times d_{z}}$ for improved
generalization. That is, to devise a framework to extract the missing
information of $\mathbf{X}$ that is contained in $\mathbf{Z}^{\prime}$. To
this end, we first define two latent spaces $\mathcal{M}^{1}$ and
$\mathcal{M}^{2}$, where a mapping
$\phi_{1}:\mathcal{Z}\rightarrow\mathcal{M}^{1}$ extracts the shared
information between $\mathcal{Z}$ and $\mathcal{X}$, and
$\phi_{2}:\mathcal{Z}\rightarrow\mathcal{M}^{2}$ extracts the exclusive
information in $\mathcal{Z}$ that is not present in $\mathcal{X}$.

In this section, we start by investigating the missing information estimation,
specifically $\mathbb{E}(\phi_{2}(\mathbf{z})|\mathbf{x})$, from the Nadaraya-
Watson kernel regression perspective and we formulate it using unlabeled
reference $\mathbf{Z}^{\prime}$. Then, we identify two major challenges under
this formulation (Remark 3.1) and show how we can potentially solve them using
two well-known techniques in cross-modal learning and semi-supervised
learning. Finally, we show that the entire estimation process recovers a
kernelized cross-attention formulation in (6).

###  3.1 Nadaraya-Watson Kernel Regression

We first write the conditional expectation as its standard definition

$\mathbb{E}(\phi_{2}(\mathbf{z})|\mathbf{x})=\int\phi_{2}(\mathbf{z})\frac{p_{1}(\phi_{2}(\mathbf{z}),\mathbf{x})}{p_{2}(\mathbf{x})}d\phi_{2}(\mathbf{z}),$
(1)

where $p_{1}(\phi_{2}(\mathbf{z}),\mathbf{x})$ is the joint density of
$\phi_{2}(\mathbf{z})$ and $\mathbf{x}$, and $p_{2}(\mathbf{x})$ is the
marginal density of $\mathbf{x}$.

To bring in the information contained in the unlabeled data modality
$\mathbf{Z}^{\prime}\in\mathbb{R}^{n^{\prime}_{z}\times d_{z}}$, we carry out
the kernel density estimations using solely instances
${\mathbf{z}^{\prime}_{i}}$ ($i$-th row of $\mathbf{Z}^{\prime}$) as the
reference points. By defining a generalized kernel density function
$g(\cdot,\cdot)$ that can describe the correlation of an input pair from two
modalities with shared information, we have the following

$\displaystyle p_{1}(\phi_{2}(\mathbf{z}),\mathbf{x})$
$\displaystyle\approx\frac{1}{n^{\prime}_{z}}\sum_{i=1}^{n^{\prime}_{z}}g(\mathbf{x},\phi_{1}(\mathbf{z}^{\prime}_{i}))k_{0}(\phi_{2}(\mathbf{z}),\phi_{2}(\mathbf{z}^{\prime}_{i}))$
(2) $\displaystyle\triangleq\hat{p}_{1}(\phi_{2}(\mathbf{z}),\mathbf{x})$
$\displaystyle p_{2}(\mathbf{x})$
$\displaystyle\approx\frac{1}{n^{\prime}_{z}}\sum_{i=1}^{n^{\prime}_{z}}g(\mathbf{x},\phi_{1}(\mathbf{z}^{\prime}_{i}))\triangleq\hat{p}_{2}(\mathbf{x}),$

where $k_{0}$ can be a standard kernel function with inputs from the same
modality. In an ideal situation, the first expression should be in terms of
$k(\mathbf{x},\mathbf{x}^{\prime}_{i})k_{0}(\phi_{2}(\mathbf{z}),\phi_{2}(\mathbf{z}^{\prime}_{i}))$
for some standard kernel functions $k$ and $k_{0}$, where
$\mathbf{x}^{\prime}_{i}$ is the $i$-th row of $\mathbf{X}^{\prime}$. However,
since $\mathbf{X}^{\prime}$ is not available, we can not use this approach.

Next, plugging the above density estimations back into (1), we have the
following

$\displaystyle\mathbb{E}(\phi_{2}(\mathbf{z})|\mathbf{x})$ (3)
$\displaystyle\approx$
$\displaystyle\int\phi_{2}(\mathbf{z})\frac{\hat{p}_{1}(\phi_{2}(\mathbf{z}),\mathbf{x})}{\hat{p}_{2}(\mathbf{x})}d\phi_{2}(\mathbf{z})$
$\displaystyle\approx$
$\displaystyle\sum_{i=1}^{n^{\prime}_{z}}\frac{\int\phi_{2}(\mathbf{z})g(\mathbf{x},\phi_{1}(\mathbf{z}^{\prime}_{i}))k_{0}(\phi_{2}(\mathbf{z}),\phi_{2}(\mathbf{z}^{\prime}_{i}))d\phi_{2}(\mathbf{z})}{\sum_{j=1}^{n^{\prime}_{z}}g(\mathbf{x},\phi_{1}(\mathbf{z}^{\prime}_{j}))}$
$\displaystyle=$
$\displaystyle\sum_{i=1}^{n^{\prime}_{z}}\frac{g(\mathbf{x},\phi_{1}(\mathbf{z}^{\prime}_{i}))\int\phi_{2}(\mathbf{z})k_{0}(\phi_{2}(\mathbf{z}),\phi_{2}(\mathbf{z}^{\prime}_{i}))d\phi_{2}(\mathbf{z})}{\sum_{j=1}^{n^{\prime}_{z}}g(\mathbf{x},\phi_{1}(\mathbf{z}^{\prime}_{j}))}.$

We now investigate the relation between (3) and the well-known Nadaraya-Watson
kernel regression estimator (Nadaraya, 1964; Watson, 1964). In the context of
missing information estimation, the exact formulation of NW kernel regression
is as follows

$\mathbb{E}(\phi_{2}(\mathbf{z})|\mathbf{x})\approx\sum_{i=1}^{n^{\prime}_{z}}\frac{k(\mathbf{x},\mathbf{x}^{\prime}_{i})\phi_{2}(\mathbf{z}^{\prime}_{i})}{\sum_{j=1}^{n^{\prime}_{z}}k(\mathbf{x},\mathbf{x}^{\prime}_{j})},$
(4)

where the matching pairs $\mathbf{x}^{\prime}_{i}$ and
$\mathbf{z}^{\prime}_{i}$ are available. Note that the above formulation is
identical to (3) when
$g(\mathbf{x},\phi_{1}(\mathbf{z}^{\prime}_{i}))=k(\mathbf{x},\mathbf{x}^{\prime}_{i})$,
and
$\int\phi_{2}(\mathbf{z})k_{0}(\phi_{2}(\mathbf{z}),\phi_{2}(\mathbf{z}^{\prime}_{i}))d\phi_{2}(\mathbf{z})=\phi_{2}(\mathbf{z}^{\prime}_{i})$,
where the latter is true for several shift-invariant kernels, such as the
Gaussian kernel. Basically, this is satisfied for any kernel
$k_{0}(\cdot-{\boldsymbol{\mu}})$ that is a density function with a mean
${\boldsymbol{\mu}}$.

######  Remark 3.1.

To account for the missing modality $\mathbf{X}^{\prime}$ in the NW kernel
regression, we need to introduce the generalized kernel function $g$ that
takes input from two different modalities. This implies the reference modality
$\mathbf{Z}^{\prime}$ needs to be information-rich, such that it shares
information with $\mathbf{X}$ for correlation calculation in addition to the
“extra” information that $\mathbf{X}$ does not have. However, there are still
two major challenges that prohibit using (3) to estimate the missing
information:

  1. 1.

How do we define the generalized kernel function $g$, such that it is a proper
kernel density function that works on cross-modal inputs?

  2. 2.

How do we find the mapping $\phi_{2}:\mathcal{Z}\rightarrow\mathcal{M}^{2}$?

###  3.2 Neural Network Integration

To address the challenges mentioned in Remark 3.1, we propose a direct neural
network integration of (3), so that the learning process can be done
implicitly. In fact, a neural network integration allows us to define a pair
of transformations $\phi_{3}:\mathcal{X}\rightarrow\mathcal{M}^{1}$ and
$\phi_{1}:\mathcal{Z}\rightarrow\mathcal{M}^{1}$ that transform $\mathbf{x}$
and $\mathbf{z}$ to the same space $\mathcal{M}^{1}$ for correlation
calculation. In this case, we can recover the exact NW kernel regression
estimator

$\mathbb{E}(\phi_{2}(\mathbf{z})|\phi_{3}(\mathbf{x}))\approx\sum_{i=1}^{n^{\prime}_{z}}\frac{k(\phi_{3}(\mathbf{x}),\phi_{1}(\mathbf{z}^{\prime}_{i}))\phi_{2}(\mathbf{z}^{\prime}_{i})}{\sum_{j=1}^{n^{\prime}_{z}}k(\phi_{3}(\mathbf{x}),\phi_{1}(\mathbf{z}^{\prime}_{j}))}.$
(5)

The transformations $\phi_{3}$ and $\phi_{1}$ can be simply defined with
trainable linear transformation matrices
$\phi_{3}(\mathbf{x})\approx\mathbf{W}_{q}\mathbf{x}$ and
$\phi_{1}(\mathbf{z}^{\prime}_{i})\approx\mathbf{W}_{k}\mathbf{z}^{\prime}_{i}$.
This workaround is identical to cross-modal correlation learning as mentioned
in Section 2.

On the other hand, finding the missing information in $\mathcal{M}^{2}$
requires learning the transformation $\phi_{2}$. This can also be made
possible with pseudo-labeling techniques, where we iteratively refine the
pseudo-labels throughout network training. Since we know the representation is
only dependent on $\mathbf{z}^{\prime}_{i}$, we can approximate the labeling
logic with another trainable linear transformation
$\phi_{2}(\mathbf{z}^{\prime}_{i})\approx\mathbf{W}_{v}\mathbf{z}^{\prime}_{i}$.

Putting together all of these transformations, we have the following
expression

$\displaystyle\mathbb{E}(\phi_{2}(\mathbf{z})|\phi_{3}(\mathbf{x}))\approx\sum_{i=1}^{n^{\prime}_{z}}\frac{k(\mathbf{W}_{q}\mathbf{x},\mathbf{W}_{k}\mathbf{z}^{\prime}_{i})\mathbf{W}_{v}\mathbf{z}^{\prime}_{i}}{\sum_{j=1}^{n^{\prime}_{z}}k(\mathbf{W}_{q}\mathbf{x},\mathbf{W}_{k}\mathbf{z}^{\prime}_{j})},$
(6)

where the RHS becomes the popular kernelized cross-attention module.

The above formulation shows that at any point in a neural network, we can
potentially feed in unlabeled reference data points
$\mathbf{Z}^{\prime}\in\mathbb{R}^{n^{\prime}_{z}\times d_{z}}$ from a
different modality to extract useful information from the seemingly unusable
data. This reference mechanism is approximately equivalent to a cross-
attention module with keys and values $\mathbf{Z}^{\prime}$ of sequence length
$n^{\prime}_{z}$ and dimension $d_{z}$ (before transformation).

Figure 2: The Attention Patch (TAP) neural network integration visualization.
TAP takes in the output $\mathbf{x}^{l-1}$ of layer $u_{l-1}$ to calculate the
missing representation $\phi_{2}(\mathbf{z})$ using a batch of reference data
$\mathbf{Z}^{\prime}_{i}$, $\phi_{2}(\mathbf{z})$ will be concatenated with
$\mathbf{x}^{l-1}$ to feed to the next layer, and the neural network yields a
corresponding output $y_{i}$ with respect to $\mathbf{Z}^{\prime}_{i}$.

######  Remark 3.2.

The neural network integration of NW kernel regression ultimately employs two
approaches in the fields of cross-modal and semi-supervised learning. First, a
similarity measure $g$ between two modalities is defined after a latent space
transformation, similar to the class of correlation learning methods we
mentioned in Section 2. Second, the approximation of the integral term can be
viewed as a multi-dimensional pseudo-labeling approach, which will be
iteratively updated during neural network training. Everything will be learned
implicitly during network training just like self-training.

##  4 The Attention Patch

In this section, we will formally define The Attention Patch (TAP) neural
network plugin. We will first introduce the reasons behind the kernelization
of TAP following NW kernel regression by showing the estimation error
asymptotically goes to zero (Theorem 4.3), and then we will propose a batch
training strategy from a practical standpoint.

###  4.1 Kernelization

Under the view of NW kernel regression, we see that the plug-in module is
approximately a cross-attention module. Furthermore, we note that the
formulation in the RHS of (6) extends the popular dot-product attention
proposed in (Vaswani et al., 2017) to a class of kernelized attention.
Although there is no study investigating the attention mechanism from an NW
kernel regression perspective, the kernelized attention approach has appeared
in several studies (Peng et al., 2020; Choromanski et al., 2020; Chen et al.,
2021; Luo et al., 2021; Xiong et al., 2021). In all of these studies, the
kernelization approach is raised based on the similarity between an
exponential lift of dot-product in the softmax function and a Gaussian kernel
function. It is also empirically shown in these studies that an actual kernel
function, especially the Gaussian kernel, will generally improve the model
performance and stability. However, there has been no study on why the actual
kernel attention enjoys a better performance from a theoretical perspective.
Here, we theoretically investigate why kernelization can be helpful from an NW
kernel regression perspective.

Given that (5) is motivated by (4), we want to characterize the estimation
error of the NW kernel regression estimator. Without loss of generality, we
use the following generic version of the NW kernel regression in this
subsection,

$\mathbb{E}(\mathbf{z}|\mathbf{x})\approx\hat{f}(\mathbf{x})\triangleq\sum_{i=1}^{n}\frac{k_{\mathbf{H}}(\mathbf{x},\mathbf{x}^{\prime}_{i})\mathbf{z}^{\prime}_{i}}{\sum_{j=1}^{n}k_{\mathbf{H}}(\mathbf{x},\mathbf{x}^{\prime}_{j})},$
(7)

where $\mathbf{H}=h\mathbf{I}$ is the bandwidth matrix of the kernel function,
such that
$k_{\mathbf{H}}(\mathbf{x},\mathbf{z})=\frac{1}{|\mathbf{H}|}k(\frac{\mathbf{x}-\mathbf{z}}{|\mathbf{H}|})$,
where $|\mathbf{H}|$ is the determinant of $\mathbf{H}$. The parameter $h$
depends on $n$, which we make it explicit in the next assumption but disregard
it in notation throughout.

######  Assumption 4.1.

(Wand & Jones, 1994) The bandwidth matrix $\mathbf{H}=h_{n}\mathbf{I}$ with
$|\mathbf{H}|=h_{n}^{d}$ (the subscript $n$ shows the dependence of $h$ to the
number of data points) has the following properties

$\displaystyle\underset{n\rightarrow\infty}{\lim}h_{n}$ $\displaystyle=0$ (8)
$\displaystyle\underset{n\rightarrow\infty}{\lim}nh^{d}_{n}$
$\displaystyle=\infty,$

which implies that the bandwidth parameter $h_{n}$ decays slower than
$n^{-1/d}$ and converges to $0$. The standard kernel function $k(\cdot)$ is a
bounded, symmetric probability density function with a zero first moment and a
finite second moment. That is, the following properties hold

$\displaystyle\int_{\mathbb{R}^{d}}k(\mathbf{x})d\mathbf{x}$
$\displaystyle=1,~{}~{}~{}~{}~{}~{}~{}\int_{\mathbb{R}^{d}}\mathbf{x}k(\mathbf{x})d\mathbf{x}=\mathbf{0}$
(9)
$\displaystyle\int_{\mathbb{R}^{d}}\mathbf{x}\mathbf{x}^{\top}k(\mathbf{x})d\mathbf{x}$
$\displaystyle=\mu_{2}(k)\mathbf{I},~{}~{}~{}~{}~{}~{}\mu_{2}(k)<\infty$
$\displaystyle\int_{\mathbb{R}^{d}}k^{2}(\mathbf{x})d\mathbf{x}$
$\displaystyle=R(k)<\infty,$

where $\mu_{2}(k)$ and $R(k)$ are constants decided by the choice of kernel
$k$.

It is easy to verify that several popular shift-invariant kernels
$k(\mathbf{x}-\mathbf{y})$ (e.g., Gaussian kernel) satisfy the above
assumption with proper normalization. We will later use the Gaussian kernel in
our simulations.

######  Assumption 4.2.

The underlying true function $f:\mathcal{X}\rightarrow\mathcal{Z}$ and the
true density function $p(\mathbf{x})$ satisfy the followings:

  * •

The true density function $p(\mathbf{x})$ is differentiable and the
$\ell_{2}$-norm of its gradient is bounded.

  * •

We have for the true function that

$\mathbf{z}=f(\mathbf{x})+\boldsymbol{\epsilon},$ (10)

where $\boldsymbol{\epsilon}$ is an isotropic noise vector from
$\mathcal{N}(\mathbf{0},\sigma^{2}\mathbf{I})$.

  * •

The true function $f(\mathbf{x})$ has a bounded gradient and Hessian in the
norm sense.

######  Theorem 4.3.

Under assumption 4.1 and 4.2, an NW kernel regression estimator
$\hat{f}(\mathbf{x})$ with an isotropic shift-invariant kernel of bandwidth
$\mathbf{H}=h\mathbf{I}$, as shown in RHS of equation (7) will yield an
estimation error
$\mathbf{e}(\mathbf{x})\triangleq\hat{f}(\mathbf{x})-f(\mathbf{x})$ that
asymptotically converges in distribution as

$\sqrt{nh^{d}}\bigg{(}\mathbf{e}(\mathbf{x})-\frac{h^{2d}\mu_{2}(k)\boldsymbol{\Psi}(\mathbf{x})}{p(\mathbf{x})}\bigg{)}\xrightarrow[]{d}\mathcal{N}(\mathbf{0},\frac{R(k)\sigma^{2}}{p(\mathbf{x})}\mathbf{I}),$
(11)

where $\mathbf{x}\in\mathbb{R}^{d}$, and the $i$-th entry of
$\boldsymbol{\Psi}(\mathbf{x})$ is

$\boldsymbol{\Psi}^{(i)}(\mathbf{x})=\frac{1}{2}p(\mathbf{x})\text{Tr}\left[\nabla^{2}f^{(i)}(\mathbf{x})\right]+\text{Tr}\left[\nabla
f^{(i)}(\mathbf{x})\nabla p(\mathbf{x})^{\top}\right],$ (12)

where $\nabla^{2}f^{(i)}(\mathbf{x})$ and $\nabla f^{(i)}(\mathbf{x})$ are the
Hessian and gradient of $i$-th entry of the true function with respect to
$\mathbf{x}$, $\nabla p(\mathbf{x})$ is the gradient of the true density
function, and $\text{Tr}\left[\cdot\right]$ is the trace operator.

The proof is provided in the Appendix. The above theorem shows that with
proper kernel function, the estimation error
$\mathbf{e}(\mathbf{x})\rightarrow\mathbf{0}$ as $n\rightarrow\infty$ under
Assumptions 4.1 and 4.2. This suggests that under an ideal latent space
transformation and pseudo-labeling logic, the reference mechanism can benefit
from more reference data points.

###  4.2 Batch Training

The attention formulation of cross-modal learning in (6) requires setting the
keys and values to be the set of unlabeled reference data points
$\mathbf{Z}^{\prime}$. Theorem 4.3 suggests that using more reference data for
the model will potentially result in a lower estimation error. However, the
computation complexity of the cross-attention module scales linearly with
respect to the sequence length of keys and values, which is equivalent to the
number of reference points $n^{\prime}_{z}$ in
$\mathbf{Z}^{\prime}\in\mathbb{R}^{n^{\prime}_{z}\times d_{z}}$. So, it is not
practical to feed all the reference points at once, mainly due to memory
limitations.

Therefore, we can break the reference dataset $\mathbf{Z}^{\prime}$ into $m$
batches $\\{\mathbf{Z}^{\prime}_{i}\\}_{i=1}^{m}$ and train each epoch by
iterating over the set of reference batches together with input batches of
primary data points. This is similar in vein to training with stochastic
gradient descent.

Each batch of reference points $\mathbf{Z}^{\prime}_{i}$ will make the neural
network yield an output in space $\mathcal{Y}$. Evaluating all batches
$\\{\mathbf{Z}^{\prime}_{i}\\}_{i=1}^{m}$ will in turn yield $m$ outputs,
which can be used in different ways depending on the application, like the
ensemble approach (Dietterich et al., 2002; Sagi & Rokach, 2018) in
classification tasks.

###  4.3 The Attention Patch Structure

The attention patch structure is shown in Fig. 2. We assume a neural network
can be decomposed into a sequence of layers $\\{u_{l}\\}_{l=1}^{L}$. The
Attention Patch can be inserted between two consecutive layers
$u_{l-1}:\mathcal{X}^{l-1}\rightarrow\mathcal{X}^{l}$ and
$u_{l}:\mathcal{X}^{l}\rightarrow\mathcal{X}^{l+1}$. The Attention Patch takes
the layer $u_{l-1}$’s output $\mathbf{x}^{l}\in\mathcal{X}^{l}$ and a batch of
reference points $\mathbf{Z}^{\prime}_{i}\in\mathcal{Z}$ to estimate the
missing information. Then, the output of The Attention Patch
$\phi_{2}(\mathbf{z})\in\mathcal{M}^{2}$ will be concatenated with
$\mathbf{x}^{l}$ to form the input for the modified layer
$u_{l}:\mathcal{X}^{l}\times\mathcal{M}^{2}\rightarrow\mathcal{X}^{l+1}$. The
insertion of The Attention Patch only requires one modification to the
original neural network, which is enlarging the input size of layer $u_{l}$
from the original dimension to the concatenation dimension.

Figure 3: Benchmark models, from left to right are Baseline, FFN, Control
Group, and TAP w/o Batch respectively.

Figure 4: Simulation results on three real-world datasets. TAP integration
shows a consistent performance advantage compared to other variants.

Figure 5: Reference batch size comparison on three real-world datasets.

Before TAP integration, the neural network is learned by minimizing the
following objective function

$\underset{q_{l-1},q_{l}}{\min}\mathbb{E}_{\mathbf{x},y}[\mathcal{L}(q_{l}(q_{l-1}(\mathbf{x})),y)],$
(13)

where $q_{l-1}$ denotes the neural network before layer $u_{l}$, and $q_{l}$
denotes the neural network starting from layer $u_{l}$. After TAP integration,
the new learning objective can be viewed as

$\underset{q_{l-1},q_{l},TAP}{\min}\mathbb{E}_{\mathbf{x},y}\left[\sum_{i=1}^{m}\mathcal{L}(q_{l}(q_{l-1}(\mathbf{x}),TAP(\mathbf{x}^{l};\mathbf{Z}^{\prime}_{i})),y)\right],$
(14)

where $TAP(\mathbf{x}^{l};\mathbf{Z}^{\prime}_{i})$ is defined as the RHS of
equation (6), which is parameterized by trainable attention weights
$\mathbf{W}_{q},\mathbf{W}_{k},\mathbf{W}_{v}$ and (batches of) frozen
reference data $\mathbf{Z}^{\prime}_{i}$.

######  Remark 4.4.

The above loss formulation shows why our proposed TAP integration has the
potential to tackle the challenge of knowledge transfer from unlabeled cross-
modal data. The cross-modal data points are considered parameters of the
neural network rather than another input modality. This resembles the
intuition behind kernel learning where all training data points are considered
the functional basis in the kernel space and ultimately become the parameters
of the learned function.

##  5 Numerical Experiments

In this section, we evaluate TAP by plugging it into neural network
classifiers. We show the effectiveness of TAP by comparing the performance of
TAP-integrated networks with other variants. The simulations are implemented
on three real-world datasets in different areas. Implementation details are
included in the Appendix for reproducibility.

###  5.1 Datasets

To ensure a comprehensive test on the performance of TAP integration, we
select three real-world cross-modal datasets in three different areas. All
datasets are open-access datasets that can be found online. A detailed dataset
and preprocessing description can be found in the Appendix.

Computer Vision: We use the MNIST dataset (MNIST) (Deng, 2012). We crop the
upper half of all images as the primary modal $\mathbf{X}$ for digit
prediction, and the lower half of all images as the reference modal
$\mathbf{Z}^{\prime}$ (without labels).

Healthcare: We use the Activity dataset (Activity) (Mohino-Herranz et al.,
2019). The Electrodermal Activity (EDA) signals will be the primary signal
$\mathbf{X}$ for predicting the subject activity. Thoracic Electrica
Bioimpedance (TEB) signals are used as the reference dataset
$\mathbf{Z}^{\prime}$ (without labels).

Remote Sensing: We choose the Crop dataset (Crop) (Khosravi et al., 2018;
Khosravi & Alavipanah, 2019). The optical features $\mathbf{X}$ are used to
predict the crop type, and the radar readings are used as reference dataset
$\mathbf{Z}^{\prime}$ (without labels).

There is no overlapping instance among the training data (in space
$\mathcal{X}$), reference data (in space $\mathcal{Z}$), and evaluation data
(in space $\mathcal{X}$).

###  5.2 Performance Evaluation

####  5.2.1 Models

It is difficult to directly compare the effectiveness of TAP against existing
state-of-the-art neural network architectures since this work is the first to
propose cross-modal learning from unlabeled data modalities. However, we can
show the effectiveness of TAP by carefully examining the performance
difference between different variants of TAP.

The baseline is a single-modal neural network without TAP integration at all.
However, even if TAP integration shows a performance advantage against
traditional supervised learning models without unlabeled cross-modal
information input, there are still several factors that might be contributing
to the performance difference, which we list as follows and propose
corresponding models to examine these factors.

  1. 1.

Factor: Is there an actual performance difference between TAP integration and
traditional supervised learning models?

Model: To show the performance advantage of TAP integration, we directly
compare the simple multilayer perceptron (MLP) classifier and its TAP
integration counterpart. This is the baseline model that will demonstrate the
benefits of TAP integration. This is the Baseline, shown in the first graph of
Fig. 3.

  2. 2.

Factor: The increase in depth of the neural network might contribute to the
performance difference.

Model: We propose a variant model that replaces the TAP with a feedforward
neural network (FFN) that takes the output of the last layer as input, and the
output of the FFN will be concatenated with the last layer’s output for the
next layer. This is the FFN, shown in the second graph of Fig. 3.

  3. 3.

Factor: The increase in parameter counts might be contributing to the
performance difference, and the model is not learning the missing information
from $\mathbf{Z}^{\prime}$.

Model: To show that TAP indeed makes use of unlabeled modality
$\mathbf{Z}^{\prime}$ rather than fitting a more complicated function on
irrelevant data, we replace the cross-modality input $\mathbf{Z}^{\prime}$
with random noises with identical mean and standard deviation to
$\mathbf{Z}^{\prime}$. This is the Control Group, shown in the third graph of
Fig. 3.

  4. 4.

Factor: Batch training might contribute negatively to the model performance.

Model: We compare it with a TAP integration that uses all of
$\mathbf{Z}^{\prime}$ throughout training and testing. This is TAP w/o Batch,
shown in the last graph of Fig. 3.

####  5.2.2 Results

The simulation results are shown in Fig. 4. The error bars are calculated over
$20$ Monte-Carlo simulations to reflect the statistical significance of the
results. As we can see, there is a consistent performance hierarchy among all
the benchmark models throughout all datasets in different areas. First, we see
that TAP integration always leads to a performance improvement compared to the
baseline classifier. Second, we see that the FFN variant shows no performance
advantage against the baseline model, which rules out the factor that the
increase in depth is responsible for performance improvement. Third, we
observe the worst generalization performance across all benchmark models for
the control group. This shows that feeding irrelevant information will
exacerbate the generalization performance. Finally, we see that batch training
and evaluation suffers from a slightly worse generalization performance than
training directly with the whole reference dataset.

###  5.3 Reference Batch Size Comparison

To further evaluate the effect of batch training, we look at the performance
of different reference batch sizes. The total number of reference data
$n^{\prime}_{z}$ is five times of labeled data $n_{x}$. Fig. 5 shows the
prediction accuracy with respect to “unlabel to label ratio”, which is defined
as the reference batch size divided by $n_{x}$.

The performance of the baseline model is shown as the horizontal dotted line.
We observe that the generalization performance improves as the reference batch
size increases, even though the total number of reference data remains the
same. This observation is consistent with the estimation error
characterization in Theorem 4.3. In practice, this suggests one can increase
the reference batch size as much as possible until the memory limit is
reached.

##  6 Conclusion

In this paper, we adopt a novel perspective of supervised learning, where we
aim to transfer knowledge from unlabeled cross-modal data to improve
generalization on the primary modality. We formulate the problem from a
missing information estimation perspective and provide an NW kernel regression
interpretation of the problem. The formulation approximately leads to a cross-
attention module in neural network integration. We name this module The
Attention Patch (TAP). We implement the TAP integration on three real-world
cross-modal datasets in different areas to show the performance advantage of
TAP. We hope that this work can inspire a new perspective in knowledge
transfer from the data level for neural networks, where seemingly unusable
data can be utilized for improved generalization. There are also future
directions that can potentially boost the method. For example, an approach
that can efficiently store and reference the unlabeled modality will provide
great benefits considering the wide availability of huge datasets. A
principled framework for selecting relevant reference datasets will also be
useful, just like in cross-modal learning and semi-supervised learning.

## References

  * Andrew et al. (2013) Andrew, G., Arora, R., Bilmes, J., and Livescu, K.  Deep canonical correlation analysis.  In _International conference on machine learning_ , pp. 1247–1255. PMLR, 2013.
  * Baltrušaitis et al. (2018) Baltrušaitis, T., Ahuja, C., and Morency, L.-P.  Multimodal machine learning: A survey and taxonomy.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(2):423–443, 2018.
  * Berthelot et al. (2019) Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., and Raffel, C. A.  Mixmatch: A holistic approach to semi-supervised learning.  _Advances in neural information processing systems_ , 32, 2019.
  * Blum & Mitchell (1998) Blum, A. and Mitchell, T.  Combining labeled and unlabeled data with co-training.  In _Proceedings of the eleventh annual conference on Computational learning theory_ , pp. 92–100, 1998.
  * Bucak et al. (2013) Bucak, S. S., Jin, R., and Jain, A. K.  Multiple kernel learning for visual object recognition: A review.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 36(7):1354–1369, 2013.
  * Castellano et al. (2008) Castellano, G., Kessous, L., and Caridakis, G.  Emotion recognition through multiple modalities: face, body gesture, speech.  In _Affect and emotion in human-computer interaction_ , pp. 92–103. Springer, 2008.
  * Cha (1994) Cha, J.  Partial least squares.  _Advanced methods of marketing research_ , 407:52–78, 1994\.
  * Chen et al. (2015) Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollár, P., and Zitnick, C. L.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_ , 2015.
  * Chen et al. (2021) Chen, Y., Zeng, Q., Ji, H., and Yang, Y.  Skyformer: Remodel self-attention with gaussian kernel and nyström method.  _Advances in Neural Information Processing Systems_ , 34:2122–2135, 2021.
  * Choromanski et al. (2020) Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al.  Rethinking attention with performers.  _arXiv preprint arXiv:2009.14794_ , 2020.
  * Deng (2012) Deng, L.  The mnist database of handwritten digit images for machine learning research.  _IEEE Signal Processing Magazine_ , 29(6):141–142, 2012.
  * Devroye & Wagner (1979) Devroye, L. and Wagner, T.  The $l\\_1$ convergence of kernel density estimates.  _The Annals of Statistics_ , 7(5):1136–1139, 1979\.
  * Dietterich et al. (2002) Dietterich, T. G. et al.  Ensemble learning.  _The handbook of brain theory and neural networks_ , 2(1):110–125, 2002.
  * Dópido et al. (2013) Dópido, I., Li, J., Marpu, P. R., Plaza, A., Dias, J. M. B., and Benediktsson, J. A.  Semisupervised self-learning for hyperspectral image classification.  _IEEE transactions on geoscience and remote sensing_ , 51(7):4032–4044, 2013.
  * Du et al. (2010) Du, J., Ling, C. X., and Zhou, Z.-H.  When does cotraining work in real data?  _IEEE Transactions on Knowledge and Data Engineering_ , 23(5):788–799, 2010.
  * Evangelopoulos et al. (2013) Evangelopoulos, G., Zlatintsi, A., Potamianos, A., Maragos, P., Rapantzikos, K., Skoumas, G., and Avrithis, Y.  Multimodal saliency and fusion for movie summarization based on aural, visual, and textual attention.  _IEEE Transactions on Multimedia_ , 15(7):1553–1568, 2013.
  * Feichtenhofer et al. (2016) Feichtenhofer, C., Pinz, A., and Zisserman, A.  Convolutional two-stream network fusion for video action recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 1933–1941, 2016.
  * Frome et al. (2013) Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M., and Mikolov, T.  Devise: A deep visual-semantic embedding model.  _Advances in neural information processing systems_ , 26, 2013.
  * Gehler & Nowozin (2009) Gehler, P. and Nowozin, S.  On feature combination for multiclass object classification.  In _2009 IEEE 12th International Conference on Computer Vision_ , pp. 221–228. IEEE, 2009.
  * Geladi & Kowalski (1986) Geladi, P. and Kowalski, B. R.  Partial least-squares regression: a tutorial.  _Analytica chimica acta_ , 185:1–17, 1986.
  * Gu et al. (2018) Gu, J., Cai, J., Joty, S. R., Niu, L., and Wang, G.  Look, imagine and match: Improving textual-visual cross-modal retrieval with generative models.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 7181–7189, 2018.
  * Hardoon et al. (2004) Hardoon, D. R., Szedmak, S., and Shawe-Taylor, J.  Canonical correlation analysis: An overview with application to learning methods.  _Neural computation_ , 16(12):2639–2664, 2004\.
  * Hong et al. (2015) Hong, C., Yu, J., Wan, J., Tao, D., and Wang, M.  Multimodal deep autoencoder for human pose recovery.  _IEEE transactions on image processing_ , 24(12):5659–5670, 2015.
  * Hong et al. (2020) Hong, D., Yokoya, N., Xia, G.-S., Chanussot, J., and Zhu, X. X.  X-modalnet: A semi-supervised deep cross-modal network for classification of remote sensing data.  _ISPRS Journal of Photogrammetry and Remote Sensing_ , 167:12–23, 2020.
  * Jia et al. (2020) Jia, X., Jing, X.-Y., Zhu, X., Chen, S., Du, B., Cai, Z., He, Z., and Yue, D.  Semi-supervised multi-view deep discriminant representation learning.  _IEEE transactions on pattern analysis and machine intelligence_ , 43(7):2496–2509, 2020.
  * Jing et al. (2014) Jing, X.-Y., Hu, R.-M., Zhu, Y.-P., Wu, S.-S., Liang, C., and Yang, J.-Y.  Intra-view and inter-view supervised correlation analysis for multi-view feature learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 28, 2014.
  * Karpathy & Fei-Fei (2015) Karpathy, A. and Fei-Fei, L.  Deep visual-semantic alignments for generating image descriptions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 3128–3137, 2015.
  * Khosravi & Alavipanah (2019) Khosravi, I. and Alavipanah, S. K.  A random forest-based framework for crop mapping using temporal, spectral, textural and polarimetric observations.  _International Journal of Remote Sensing_ , 40(18):7221–7251, 2019.
  * Khosravi et al. (2018) Khosravi, I., Safari, A., and Homayouni, S.  Msmd: maximum separability and minimum dependency feature selection for cropland classification from optical and radar data.  _International Journal of Remote Sensing_ , 39(8):2159–2176, 2018.
  * Kiela & Bottou (2014) Kiela, D. and Bottou, L.  Learning image embeddings using convolutional neural networks for improved multi-modal semantics.  In _Proceedings of the 2014 Conference on empirical methods in natural language processing (EMNLP)_ , pp. 36–45, 2014.
  * Kiritchenko & Matwin (2001) Kiritchenko, S. and Matwin, S.  Email classification with co-training.  In _Proceedings of the 2001 conference of the Centre for Advanced Studies on Collaborative research_ , pp. 8, 2001.
  * Kulkarni et al. (2013) Kulkarni, G., Premraj, V., Ordonez, V., Dhar, S., Li, S., Choi, Y., Berg, A. C., and Berg, T. L.  Babytalk: Understanding and generating simple image descriptions.  _IEEE transactions on pattern analysis and machine intelligence_ , 35(12):2891–2903, 2013.
  * Lee et al. (2013) Lee, D.-H. et al.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In _Workshop on challenges in representation learning, ICML_ , volume 3, pp. 896, 2013.
  * Li et al. (2019) Li, C., Gao, S., Deng, C., Xie, D., and Liu, W.  Cross-modal learning with adversarial samples.  _Advances in neural information processing systems_ , 32, 2019.
  * Liang et al. (2020) Liang, J., Li, R., and Jin, Q.  Semi-supervised multi-modal emotion recognition with cross-modal distribution matching.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , pp. 2852–2861, 2020.
  * Lin & Tang (2006) Lin, D. and Tang, X.  Inter-modality face recognition.  In _European conference on computer vision_ , pp. 13–26. Springer, 2006.
  * Luo et al. (2021) Luo, S., Li, S., Cai, T., He, D., Peng, D., Zheng, S., Ke, G., Wang, L., and Liu, T.-Y.  Stable, fast and accurate: Kernelized attention with relative positional encoding.  _Advances in Neural Information Processing Systems_ , 34:22795–22807, 2021.
  * Mahasseni & Todorovic (2016) Mahasseni, B. and Todorovic, S.  Regularizing long short term memory with 3d human-skeleton sequences for action recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 3054–3062, 2016.
  * Mitchell et al. (2012) Mitchell, M., Dodge, J., Goyal, A., Yamaguchi, K., Stratos, K., Han, X., Mensch, A., Berg, A., Berg, T., and Daumé III, H.  Midge: Generating image descriptions from computer vision detections.  In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_ , pp. 747–756, 2012.
  * Mohino-Herranz et al. (2019) Mohino-Herranz, I., Gil-Pita, R., Rosa-Zurera, M., and Seoane, F.  Activity recognition using wearable physiological measurements: Selection of features from a comprehensive literature study.  _Sensors_ , 19(24), 2019.
  * Nadaraya (1964) Nadaraya, E. A.  On estimating regression.  _Theory of Probability & Its Applications_, 9(1):141–142, 1964.
  * Ngiam et al. (2011) Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., and Ng, A. Y.  Multimodal deep learning.  In _ICML_ , 2011.
  * Nie et al. (2017a) Nie, F., Cai, G., and Li, X.  Multi-view clustering and semi-supervised classification with adaptive neighbours.  In _Thirty-first AAAI conference on artificial intelligence_ , 2017a.
  * Nie et al. (2017b) Nie, F., Li, J., and Li, X.  Convex multiview semi-supervised classification.  _IEEE Transactions on Image Processing_ , 26(12):5718–5729, 2017b.
  * Nie et al. (2019) Nie, F., Tian, L., Wang, R., and Li, X.  Multiview semi-supervised learning model for image classification.  _IEEE Transactions on Knowledge and Data Engineering_ , 32(12):2389–2400, 2019.
  * Peng et al. (2020) Peng, H., Pappas, N., Yogatama, D., Schwartz, R., Smith, N., and Kong, L.  Random feature attention.  In _International Conference on Learning Representations_ , 2020.
  * Peng & Qi (2019) Peng, Y. and Qi, J.  Cm-gans: Cross-modal generative adversarial networks for common representation learning.  _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_ , 15(1):1–24, 2019.
  * Peng et al. (2017) Peng, Y., Qi, J., Huang, X., and Yuan, Y.  Ccl: Cross-modal correlation learning with multigrained fusion by hierarchical network.  _IEEE Transactions on Multimedia_ , 20(2):405–420, 2017.
  * Ramirez et al. (2011) Ramirez, G. A., Baltrušaitis, T., and Morency, L.-P.  Modeling latent discriminative dynamic of multi-dimensional affective signals.  In _International Conference on Affective Computing and Intelligent Interaction_ , pp. 396–406. Springer, 2011.
  * Rosenberg et al. (2005) Rosenberg, C., Hebert, M., and Schneiderman, H.  Semi-supervised self-training of object detection models.  In _Applications of Computer Vision and the IEEE Workshop on Motion and Video Computing, IEEE Workshop on_ , volume 1, pp. 29–36. IEEE Computer Society, 2005.
  * Sagi & Rokach (2018) Sagi, O. and Rokach, L.  Ensemble learning: A survey.  _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ , 8(4):e1249, 2018.
  * Sharma et al. (2012) Sharma, A., Kumar, A., Daume, H., and Jacobs, D. W.  Generalized multiview analysis: A discriminative latent space.  In _2012 IEEE conference on computer vision and pattern recognition_ , pp. 2160–2167. IEEE, 2012.
  * Sohn et al. (2014) Sohn, K., Shang, W., and Lee, H.  Improved multimodal deep learning with variation of information.  _Advances in neural information processing systems_ , 27, 2014.
  * Sohn et al. (2020) Sohn, K., Berthelot, D., Carlini, N., Zhang, Z., Zhang, H., Raffel, C. A., Cubuk, E. D., Kurakin, A., and Li, C.-L.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  _Advances in neural information processing systems_ , 33:596–608, 2020.
  * Tanha et al. (2017) Tanha, J., Van Someren, M., and Afsarmanesh, H.  Semi-supervised self-training for decision tree classifiers.  _International Journal of Machine Learning and Cybernetics_ , 8(1):355–370, 2017.
  * Tenenbaum & Freeman (2000) Tenenbaum, J. B. and Freeman, W. T.  Separating style and content with bilinear models.  _Neural computation_ , 12(6):1247–1283, 2000\.
  * Triguero et al. (2015) Triguero, I., García, S., and Herrera, F.  Self-labeled techniques for semi-supervised learning: taxonomy, software and empirical study.  _Knowledge and Information systems_ , 42(2):245–284, 2015.
  * Tsai et al. (2019) Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., and Salakhutdinov, R.  Multimodal transformer for unaligned multimodal language sequences.  In _Proceedings of the conference. Association for Computational Linguistics. Meeting_ , volume 2019, pp. 6558. NIH Public Access, 2019.
  * Van Engelen & Hoos (2020) Van Engelen, J. E. and Hoos, H. H.  A survey on semi-supervised learning.  _Machine Learning_ , 109(2):373–440, 2020.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Venugopalan et al. (2015) Venugopalan, S., Xu, H., Donahue, J., Rohrbach, M., Mooney, R. J., and Saenko, K.  Translating videos to natural language using deep recurrent neural networks.  In _HLT-NAACL_ , 2015.
  * Wan (2009) Wan, X.  Co-training for cross-lingual sentiment classification.  In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_ , pp. 235–243, 2009.
  * Wand & Jones (1994) Wand, M. P. and Jones, M. C.  _Kernel Smoothing_.  CRC press, 1994.
  * Wang et al. (2016) Wang, K., Yin, Q., Wang, W., Wu, S., and Wang, L.  A comprehensive survey on cross-modal retrieval.  _arXiv preprint arXiv:1607.06215_ , 2016.
  * Wang & Shahrampour (2021) Wang, Y. and Shahrampour, S.  Orcca: Optimal randomized canonical correlation analysis.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2021\.
  * Watson (1964) Watson, G. S.  Smooth regression analysis.  _Sankhyā: The Indian Journal of Statistics, Series A_ , pp. 359–372, 1964.
  * Wu et al. (2012) Wu, Z., Wu, J., Cao, J., and Tao, D.  Hysad: A semi-supervised hybrid shilling attack detector for trustworthy product recommendation.  In _Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pp. 985–993, 2012.
  * Xie et al. (2020) Xie, Q., Luong, M.-T., Hovy, E., and Le, Q. V.  Self-training with noisy student improves imagenet classification.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 10687–10698, 2020.
  * Xiong et al. (2021) Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., and Singh, V.  Nyströmformer: A nyström-based algorithm for approximating self-attention.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pp. 14138–14148, 2021.
  * Xu et al. (2015) Xu, R., Xiong, C., Chen, W., and Corso, J.  Jointly modeling deep video and compositional text to bridge vision and language in a unified framework.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 29, 2015.
  * Yarowsky (1995) Yarowsky, D.  Unsupervised word sense disambiguation rivaling supervised methods.  In _33rd annual meeting of the association for computational linguistics_ , pp. 189–196, 1995.
  * Yu et al. (2018) Yu, E., Sun, J., Li, J., Chang, X., Han, X.-H., and Hauptmann, A. G.  Adaptive semi-supervised feature selection for cross-modal retrieval.  _IEEE Transactions on Multimedia_ , 21(5):1276–1288, 2018.
  * Zhang et al. (2017) Zhang, L., Ma, B., He, J., Li, G., Huang, Q., and Tian, Q.  Adaptively unified semi-supervised learning for cross-modal retrieval.  In _IJCAI_ , pp. 3406–3412, 2017.
  * Zhang et al. (2021) Zhang, S., Chen, M., Chen, J., Li, Y.-F., Wu, Y., Li, M., and Zhu, C.  Combining cross-modal knowledge transfer and semi-supervised learning for speech emotion recognition.  _Knowledge-Based Systems_ , 229:107340, 2021.
  * Zhu (2005) Zhu, X. J.  Semi-supervised learning literature survey.  _Technical Report TR 1530_ , 2005.
  * Zoph et al. (2020) Zoph, B., Ghiasi, G., Lin, T.-Y., Cui, Y., Liu, H., Cubuk, E. D., and Le, Q.  Rethinking pre-training and self-training.  _Advances in neural information processing systems_ , 33:3833–3845, 2020.

##  Appendix A Appendix

We first provide the experimental details for reproducibility. The code is
also provided in the supplementary material. Then, we introduce a simple lemma
that shows multivariate kernel density estimator is a consistent density
estimator that converges in probability. The last section is the proof of
Theorem 4.3.

###  A.1 Experimental Details

####  A.1.1 Dataset

Dataset Description:

MNIST dataset is a well-known dataset with $784$ pixel value features. We crop
the upper half of images as the primary modality, which corresponds to the
first $392$ features in the dataset, and the rest $392$ features form the
lower images. In this case, we know for sure that the two modalities are
located in different spaces and carry complementary information.

Activity dataset is an activity prediction dataset that predicts the status of
a person through wearable physiological measurements. There are $151$ features
for Thoracic Electrical Bioimpedance (TEB) readings and $208$ features for
Electrodermal Activity (EDA) readings including $104$ for left arm readings
and $104$ for right arm readings. The objective is to predict the activity
that the subject is currently going through, including neutral, mental,
emotional, and physical activities.

Crop dataset is a crop type prediction dataset that predicts the type of crop
a land is growing using satellite and radar readings. There are $76$ optical
features collected with RapidEye satellites and $98$ radar features collected
with the Unmanned Aerial Vehicle Synthetic Aperture Radar system. The labels
contain seven crop types, including Corn, Peas, Canola, Soybeans, Oats, Wheat,
and Broadleaf.

Data Preprocessing:

Just like in semi-supervised learning, the motivation behind utilizing
unlabeled data points is the limited availability of labeled data. So, we
randomly sample $200$ data points in the primary modality to serve as the
training data for each dataset. We further randomly sample $1000$ data points
in the secondary modality to serve as the cross-modal reference data. For
MNIST, it means $200$ upper images as primary modality training data and
$1000$ lower images as reference data. All the remaining data points will be
the evaluation data.

####  A.1.2 Performance Evaluation

Parameter Setting:

In the performance evaluation, the reference batch size for TAP is chosen as
$250$, which is $1.25$ times the training data. The training data batch size
is set to $100$. As shown in the reference batch size comparison, $250$ is not
the best-performing choice, but it always yields better generalization
accuracy on these three datasets.

The backbone neural network structure for all three datasets is a two-hidden-
layer neural network with $64$ hidden neurons at each layer. The activation
function is ReLU, with a dropout rate of $0.5$. Layer normalization is
implemented after each hidden layer. TAP is integrated between the two hidden
layers. For TAP, TAP w/o Batch, FFN, and Control Group, the second hidden
layer still has $64$ neurons, except the linear layer takes input from
$\mathbb{R}^{128}$ and outputs $\mathbb{R}^{64}$.

The hidden dimension in TAP is also set to $64$, and the output is also $64$.
FNN replaces the attention module in TAP with a linear layer from
$\mathbb{R}^{64}$ to $\mathbb{R}^{64}$. Layer normalization is also
implemented in TAP (including TAP w/o Batch and Control Group) and FNN as
well. The normalization parameter in the attention module of TAP is set to
$\sqrt{d}(n^{\prime}_{z}/m)^{-1/d}$ such that it follows the bandwidth
Assumption 4.1 while being close to the general recommended normalization
constant $\sqrt{d}$.

At each Monte-Carlo Simulation, the set of training data, reference data, and
evaluation data are shuffled while keeping the amount the same. The control
group also generates a new set of random reference data at each Monte-Carlo
simulation.

Network Training:

All benchmark variant models share a similar backbone structure, therefore the
training process is also similar. All models are trained with the cross-
entropy loss using the Adam optimizer with a fixed learning rate of $0.0001$.
All models are trained for $1000$ epochs ($8000$ in the Crop dataset) except
for TAP. Since TAP updates the model several times within one epoch, the total
training epoch for TAP is reduced to $1000/m$, where $m$ is the number of
reference batches. For example, in the case of performance evaluation where
the reference batch size is $250$, there are four batches of reference data,
and the total training epoch is $250$ for TAP. The prediction accuracy is
calculated as the lowest five-moving-average of validation accuracy throughout
the training process, which is a common practice in neural network performance
evaluation.

The batch evaluation of TAP is done by having each batch of reference data
generate a prediction class, and finding the majority vote among all batches
just like ensemble methods.

####  A.1.3 Reference Batch Size Comparison

The experimental settings in this section of the simulation are pretty much
identical to the previous section. The reference batch sizes are chosen as
$100,200,250,500,1000$, which corresponds to the unlabeled to labeled ratio
$1/2,1,5/4,5/2,5$. The corresponding training epochs are
$100,200,250,500,1000$ respectively. For the Crop dataset, all training epochs
are multiplied by $8$ since it takes longer to train.

###  A.2 Lemma

######  Lemma A.1.

Under Assumptions 4.1 and 4.2, a kernel density estimator

$\hat{p}(\mathbf{x})=\frac{1}{n}\sum_{i=1}^{n}k_{\mathbf{H}}(\mathbf{x},\mathbf{x}^{\prime}_{i}),$
(15)

is consistent, i.e., it converges in probability to the true density function
$p(\mathbf{x})$,

$\hat{p}(\mathbf{x})\xrightarrow[]{p}p(\mathbf{x}).$ (16)

This is a well-known result that can be found in existing literature (Devroye
& Wagner, 1979).

###  A.3 Proof of Theorem 4.3

Throughout the proof, all integrals are Riemann integral
$\int_{\mathbb{R}^{d}}$, and we omit the subscript for simplicity. Since we
are strictly talking about shift-invariant kernels, we will use
$k_{\mathbf{H}}(\mathbf{x}-\mathbf{x}^{\prime}_{i})$ to represent an isotropic
kernel function with bandwidth $\mathbf{H}=h\mathbf{I}$, where
$k_{\mathbf{H}}(\mathbf{x}-\mathbf{x}^{\prime}_{i})=\frac{1}{|\mathbf{H}|}k(\frac{\mathbf{x}-\mathbf{x}^{\prime}_{i}}{|\mathbf{H}|})$.
The determinant of the bandwidth matrix by definition is $|\mathbf{H}|=h^{d}$.

Consider the $j$-th element in $\mathbf{z}^{\prime}_{i}$, where we simply
write as $z^{(j)}$. Then, we have the following

$\displaystyle\hat{f}^{(j)}(\mathbf{x})$
$\displaystyle=\frac{\frac{1}{n}\sum_{i=1}^{n}k_{\mathbf{H}}(\mathbf{x}-\mathbf{x}^{\prime}_{i})z^{(j)}}{\hat{p}(\mathbf{x})}$
(17)
$\displaystyle=\frac{\frac{1}{n}\sum_{i=1}^{n}k_{\mathbf{H}}(\mathbf{x}-\mathbf{x}^{\prime}_{i})(f^{(j)}(\mathbf{x})+f^{(j)}(\mathbf{x}^{\prime}_{i})-f^{(j)}(\mathbf{x})+\epsilon^{(j)}_{i})}{\hat{p}(\mathbf{x})}$
$\displaystyle=f^{(j)}(\mathbf{x})+\frac{\frac{1}{n}\sum_{i=1}^{n}k_{\mathbf{H}}(\mathbf{x}-\mathbf{x}^{\prime}_{i})(f^{(j)}(\mathbf{x}^{\prime}_{i})-f^{(j)}(\mathbf{x}))}{\hat{p}(\mathbf{x})}+\frac{\frac{1}{n}\sum_{i=1}^{n}k_{\mathbf{H}}(\mathbf{x}-\mathbf{x}^{\prime}_{i})\epsilon^{(j)}_{i}}{\hat{p}(\mathbf{x})}$
$\displaystyle\triangleq
f^{(j)}(\mathbf{x})+\frac{B(\mathbf{x})}{\hat{p}(\mathbf{x})}+\frac{V(\mathbf{x})}{\hat{p}(\mathbf{x})},$

where we theoretically characterize the properties of $B(\mathbf{x})$ and
$V(\mathbf{x})$. We refer to a single term in the summation $B(\mathbf{x})$
and $V(\mathbf{x})$ as $b_{i}(\mathbf{x})$ and $v_{i}(\mathbf{x})$, and the
subscripts will be omitted from now on for simplicity.

We first look at the term $v(\mathbf{x})$, and it is easy to see that
$\mathbb{E}[v(\mathbf{x})]=0$ due to our assumption on the noise term
$\boldsymbol{\epsilon}$. For the variance, we have the following asymptotic
equality

$\displaystyle\mathbb{E}[|\mathbf{H}|v^{2}(\mathbf{x})]$
$\displaystyle=\mathbb{E}[\epsilon^{2}]|\mathbf{H}|\int
k_{\mathbf{H}}^{2}(\mathbf{x}-\mathbf{y})p(\mathbf{y})d\mathbf{y}$ (18)
$\displaystyle=\frac{\sigma^{2}}{|\mathbf{H}|}\int
k^{2}(\frac{\mathbf{x}-\mathbf{y}}{|\mathbf{H}|})p(\mathbf{y})d\mathbf{y}$
$\displaystyle=\sigma^{2}\int
k^{2}(\mathbf{z})p(\mathbf{x}-|\mathbf{H}|\mathbf{z})d\mathbf{z}$
$\displaystyle=\sigma^{2}\int
k^{2}(\mathbf{z})(p(\mathbf{x})+o(1))d\mathbf{z}$ $\displaystyle\asymp
R(k)p(\mathbf{x})\sigma^{2},$

where the last line follows from omitting the higher order term. Since
$V(\mathbf{x})$ is the mean of random variables $v(\mathbf{x})$, we can apply
the central limit theorem to show that

$\sqrt{nh^{d}}V(\mathbf{x})\xrightarrow[]{d}\mathcal{N}(0,R(k)p(\mathbf{x})\sigma^{2}).$
(19)

Since $\hat{p}(\mathbf{x})\xrightarrow[]{p}p(\mathbf{x})$ following Lemma A.1,
we can make the claim following Slutsky’s theorem

$\frac{\sqrt{nh^{d}}V(\mathbf{x})}{\hat{p}(\mathbf{x})}\xrightarrow[]{d}\mathcal{N}(0,\frac{R(k)\sigma^{2}}{p(\mathbf{x})}).$
(20)

Then, let us look at the term $b(\mathbf{x})$. We have the following

$\displaystyle\mathbb{E}[b(\mathbf{x})]$ (21) $\displaystyle=$
$\displaystyle\frac{1}{|\mathbf{H}|}\mathbb{E}\Big{[}k(\frac{\mathbf{x}^{\prime}_{i}-\mathbf{x}}{|\mathbf{H}|})(f^{(j)}(\mathbf{x}^{\prime}_{i})-f^{(j)}(\mathbf{x}))\Big{]}$
$\displaystyle=$ $\displaystyle\frac{1}{|\mathbf{H}|}\int
k(\frac{\mathbf{y}-\mathbf{x}}{|\mathbf{H}|})(f^{(j)}(\mathbf{y})-f^{(j)}(\mathbf{x}))p(\mathbf{y})d\mathbf{y}$
$\displaystyle=$ $\displaystyle\int
k(\mathbf{z})(f^{(j)}(\mathbf{x}+|\mathbf{H}|\mathbf{z})-f^{(j)}(\mathbf{x}))p(\mathbf{x}+|\mathbf{H}|\mathbf{z})d\mathbf{z}$
$\displaystyle\asymp$ $\displaystyle\int
k(\mathbf{z})\Big{(}(|\mathbf{H}|\mathbf{z})^{\top}\nabla
f^{(j)}(\mathbf{x})+\frac{1}{2}(|\mathbf{H}|\mathbf{z})^{\top}\nabla^{2}f^{(j)}(\mathbf{x})(|\mathbf{H}|\mathbf{z})\Big{)}\Big{(}p(\mathbf{x})+(|\mathbf{H}|\mathbf{z})^{\top}\nabla
p(\mathbf{x})\Big{)}d\mathbf{z}$ $\displaystyle\asymp$
$\displaystyle|\mathbf{H}|\int k(\mathbf{z})\mathbf{z}^{\top}\nabla
f^{(j)}(\mathbf{x})p(\mathbf{x})d\mathbf{z}+|\mathbf{H}|^{2}\text{Tr}\left[\int
k(\mathbf{z})p(\mathbf{x})\frac{1}{2}\nabla^{2}f^{(j)}(\mathbf{x})\mathbf{z}\mathbf{z}^{\top}d\mathbf{z}\right]$
$\displaystyle\quad\quad\quad\quad\quad\quad+|\mathbf{H}|^{2}\text{Tr}\left[\int
k(\mathbf{z})\nabla f^{(j)}(\mathbf{x})\nabla
p(\mathbf{x})^{\top}\mathbf{z}\mathbf{z}^{\top}d\mathbf{z}\right]$
$\displaystyle=$
$\displaystyle|\mathbf{H}|^{2}\mu_{2}(k)\Big{(}\frac{1}{2}p(\mathbf{x})\text{Tr}\left[\nabla^{2}f^{(j)}(\mathbf{x})\right]+\text{Tr}\left[\nabla
f^{(j)}(\mathbf{x})\nabla p(\mathbf{x})^{\top}\right]\Big{)}$
$\displaystyle\triangleq$
$\displaystyle|\mathbf{H}|^{2}\mu_{2}(k)\boldsymbol{\Psi}^{(j)}(\mathbf{x})=h^{2d}\mu_{2}(k)\boldsymbol{\Psi}^{(j)}(\mathbf{x}).$

Any term that is $o(|\mathbf{H}|^{2})$ is omitted as it asymptotically
vanishes.

For the term $\mathbb{E}[b^{2}(\mathbf{x})]$, we use a similar expansion by
omitting terms of $o(|\mathbf{H}|^{2})$ as follows

$\displaystyle\mathbb{E}[b^{2}(\mathbf{x})]=$
$\displaystyle\frac{1}{|\mathbf{H}|^{2}}\mathbb{E}\Big{[}k^{2}(\frac{\mathbf{x}^{\prime}_{i}-\mathbf{x}}{|\mathbf{H}|})(f^{(j)}(\mathbf{x}^{\prime}_{i})-f^{(j)}(\mathbf{x}))^{2}\Big{]}$
(22) $\displaystyle=$ $\displaystyle\frac{1}{|\mathbf{H}|^{2}}\int
k^{2}(\frac{\mathbf{y}-\mathbf{x}}{|\mathbf{H}|})(f^{(j)}(\mathbf{y})-f^{(j)}(\mathbf{x}))^{2}p(\mathbf{y})d\mathbf{y}$
$\displaystyle=$ $\displaystyle\frac{1}{|\mathbf{H}|}\int
k^{2}(\mathbf{z})(f^{(j)}(\mathbf{x}+|\mathbf{H}|\mathbf{z})-f^{(j)}(\mathbf{x}))^{2}p(\mathbf{x}+|\mathbf{H}|\mathbf{z})d\mathbf{z}$
$\displaystyle\asymp$ $\displaystyle\frac{1}{|\mathbf{H}|}\int
k^{2}(\mathbf{z})((|\mathbf{H}|\mathbf{z})^{\top}\nabla
f^{(j)}(\mathbf{x}))^{2}p(\mathbf{x})d\mathbf{z}$ $\displaystyle=$
$\displaystyle O\bigg{(}|\mathbf{H}|\nabla f^{(j)}(\mathbf{x})^{\top}\nabla
f^{(j)}(\mathbf{x})p(\mathbf{x})\mu_{2}(k)\bigg{)}.$

This suggests that the variance of $b(\mathbf{x})$ is of a higher order than
the variance of $\sqrt{|\mathbf{H}|}v(\mathbf{x})$. Thus, we know

$\displaystyle\sqrt{nh^{d}}(B(\mathbf{x})-h^{2d}\mu_{2}(k)\boldsymbol{\Psi}^{(j)}(\mathbf{x}))\xrightarrow[]{p}0.$
(23)

Combining with the convergence in probability for density estimator
$\hat{p}(\mathbf{x})$, we have

$\sqrt{nh^{d}}(\frac{B(\mathbf{x})}{\hat{p}(\mathbf{x})}-\frac{h^{2d}\mu_{2}(k)\boldsymbol{\Psi}^{(j)}(\mathbf{x})}{p(\mathbf{x})})\xrightarrow[]{p}0.$
(24)

Finally, we conclude that

$\displaystyle\sqrt{nh^{d}}(\mathbf{e}(\mathbf{x})-\frac{h^{2d}\mu_{2}(k)\boldsymbol{\Psi}(\mathbf{x})}{p(\mathbf{x})})$
(25) $\displaystyle=$
$\displaystyle\sqrt{nh^{d}}(\frac{\mathbf{B}(\mathbf{x})}{\hat{p}(\mathbf{x})}-\frac{h^{2d}\mu_{2}(k)\boldsymbol{\Psi}(\mathbf{x})}{p(\mathbf{x})}+\frac{\mathbf{V}(\mathbf{x})}{\hat{p}(\mathbf{x})})\xrightarrow[]{d}\mathcal{N}(\mathbf{0},\frac{R(k)\sigma^{2}}{p(\mathbf{x})}\mathbf{I}),$

where $\mathbf{B}(\mathbf{x})$ and $\mathbf{V}(\mathbf{x})$ are the
vectorization of $B(\mathbf{x})$ and $V(\mathbf{x})$, respectively.

The proof is complete.


# Vision Transformer Pruning Via Matrix Decomposition

Tianyi Sun  Committee on Computational and Applied Mathematics
Department of Computer Science
Department of Statistics
University of Chicago

(First version: Dec. 2021)

###### Abstract.

This is a further development of Vision Transformer Pruning (Zhu et al.,
2021b) via matrix decomposition. The purpose of the Vision Transformer Pruning
is to prune the dimension of the linear projection of the dataset by learning
their associated importance score in order to reduce the storage, run-time
memory, and computational demands. In this paper we further reduce dimension
and complexity of the linear projection by implementing and comparing several
matrix decomposition methods while preserving the generated important
features. We end up selected the Singular Value Decomposition as the method to
achieve our goal by comparing the original accuracy scores in the original
Github repository and the accuracy scores of using those matrix decomposition
methods, including Singular Value Decomposition, four versions of QR
Decomposition, and LU factorization.

Computer Vision, Transformer Pruning, Matrix Computation

††copyright: none††doi: ††isbn: ††conference: ; Transformer Pruning; Matrix
Computation

##  1\. Introduction

Transformer (Vaswani et al., 2017) is widely used in Natural Language
Processing tasks(Sun, 2021), (Sun and Nelson, 2023), (Sun and Gini, 2021) and
now it has attracted much attention on computer vision applications (Chen et
al., 2021), (Chen et al., 2020), (Han et al., 2021a), (Liu et al., 2021), such
as image classification (Dosovitskiy et al., 2021), (Han et al., 2021b),
(Touvron et al., 2021), object detection (Carion et al., 2020), (Zhu et al.,
2021a), and image segmentation (Hu et al., 2021), (Wang et al., 2021c), (Wang
et al., 2021b). However, most of the transformer used in the set of computer
vision tasks are highly storage, run-time memory, and computational resource
demanding. Convolutional neural networks (CNNs) are also sharing the same
issues. Developed ways of addressing this issues in CNNs’ settings are
including low-rank decomposition (Denton et al., 2014), (Lee et al., 2019),
(Lin et al., 2019), (Yu et al., 2017), (Tai et al., 2016), quantization (Cai
et al., 2017), (Gupta et al., 2015), (Rastegari et al., 2016), (Yang et al.,
2020), and network pruning (Hassibi and Stork, 1992), (Cun et al., 1990),
(Tang et al., 2021), (Tang et al., 2020), (Wen et al., 2016).

Inspired by those methods, several methods has been developed in the setting
of vision transformer. Star-Transformer (Guo et al., 2019) changes fully-
connected structure to the star-shaped topology. There are some methods focus
on compressing and accelerating the transformer for the natural language
processing tasks. With the emergence of vision transformers such as ViT
(Dosovitskiy et al., 2021), PVT (Wang et al., 2021a), and TNT (Han et al.,
2021b) an efficient transformer is urgently need for computer vision
applications. Vision Transformer Pruning (VTP) (Zhu et al., 2021b), inspired
by the pruning shceme in network slimming (Liu et al., 2017), has proposes an
approach of pruning the vision transformer according to the learnable
important scores. We add the learning importance scores before the components
to be prune and sparsify them by training the network with $L_{1}$ regulation.
The dimension with smaller importance scores will be pruned and the compact
network can be obtained. The (VTP) method largely compresses and accelerates
the original ViT models. Inspired by the low-rank decomposition (Denton et
al., 2014), (Lee et al., 2019), (Lin et al., 2019), (Yu et al., 2017), (Tai et
al., 2016) methods for compressing and accelerating CNNs, we proposed a matrix
decomposition method building up on the VTP to further reduce the storage,
run-time memory, and computational resource demanding.

##  2\. Method

In this section, we first review the key components of Vision Transformer
(ViT) (Dosovitskiy et al., 2021) and Vision Transformer Pruning according to
important features (Zhu et al., 2021b). Then we introduce matrix decomposition
methods that is fit particularly for our case.

Figure 1. Further prune Vision Transformer via Singular Value Decomposition
(SVD)

###  2.1. Vision Transformer (ViT)

The ViT model includes fully connected layer, Multi-Head Self-Attention(MHSA),
linear transformer, Multi-Layer Perception, normalization layer, activation
function, and shortcut connection.

####  2.1.1. Fully connected layer

: A fully connected layer is used to transform the input into characters that
can be used in the next step, i.e. MHSA. Specifically, it can be denoted as:

$I\rightarrow(Q_{I},K_{I},V_{I}).$

The Input $I\in\mathbb{R}^{n\times d}$ is transformed to Query
$Q\in\mathbb{R}^{n\times d}$, and Value $V\in\mathbb{R}^{n\times d}$ via
fully-connected layers, where $n$ is the number of patches $d$ in the embedded
dimension.

####  2.1.2. MHSA

: The self-attention mechanism is utilized to model the relationship between
patches. It can be denoted as:

$Attention(Q_{I},K_{I},V_{I})=Softmax\left(\frac{Q_{I}K_{I}^{T}}{\sqrt{d}}\right)V.$

####  2.1.3. Linear Transformer

: A linear transformer is used to transform the output of MHSA. It can be
denoted as:

$Y=I+f_{out}(Attention(Q_{I},K_{I},V_{I})),$

where the details of normalization layer and activation function are included
in it.

####  2.1.4. Two-layer MLP

: Two feed forward MLP layers can be denoted as:

$Z=Y+MLP_{2}(MLP_{1}(Y)).$

###  2.2. Vision Transformer Pruning

The approach of Vision Transformer Pruning is to preserve the generated
important features and remove the useless ones from the components of linear
projection.

The pruning process is depending on the score of feature importance. We denote
the matrix of feature importance scores is Diag$(a)\in\mathbb{R}^{n\times n}$,
where $a\in\mathbb{R}^{d}$ represents each feature importance score. In order
to enforce the sparsity of importance scores, we apply $l_{1}$ regularization
on the importance scores $a$, which is denoted as $\hat{a}$, where
$\hat{a}=\lambda||a||_{1}$ and $\lambda$ is the sparsity hyper-parameter. And
optimize it by adding the training objective. We, therefore, obtain the
transformer with some important scores near zero. We rank all the values of
regularized importance scores in the transformer and obtain a threshold $\tau$
according to the predefined pruning rate. With the threshold, we obtain the
discrete $a^{*}\in\mathbb{R}^{n\times n}$ score, which is denoted as

$a^{*}=\left\\{\begin{array}[]{rcrcrc@{\qquad}l}1&\text{ if
}\hat{a}\geq\tau\\\ 0&\text{ if }\hat{a}<\tau.\\\ \end{array}\right.$

The matrix $X$ after applying the pruning processes, denoted by

$X^{*}=X\text{ Diag}(a^{*}),$

and removing features with a zero score is transformed to
$X^{*}\in\mathbb{R}^{r\times d}$. This pruning procedure is denoted as
$\text{Prune}(X).$

###  2.3. Matrix Decomposition

We will introduce an experiential study of Matrix Decomposition methods
particularly applied in our case. And a comparison of run-time complexity and
the costs of those decomposition.

####  2.3.1. Eigenvalue Value Decomposition (EVD) of $X^{*}$

Note that a matrix has an EVD if and only if the matrix is diagonalizable.
However, $X^{*}\in\mathbb{R}^{r\times d}$. When $r\neq d$, the $X^{*}$ has no
EVD. So EVD has limitations while applying in our cases.

####  2.3.2. Singular Value Decomposition (SVD) of $X^{*}$

SVD is computed not depending on the dimension of the matrix. SVD can be
applied to any matrix. Assume that $X^{*}\in\mathbb{R}^{m\times n}$ is the
matrix after we have done the pruning procedure. Then $X^{*}$ has a
decomposition that

$X^{*}=U\Sigma V$

where the left singular vectors $U\in\mathbb{R}^{m\times m}$ and right
singular vectors $V\in\mathbb{R}^{n\times n}$ are both unitary matrices, which
means $U^{*}U=I_{m}=UU^{*}$ and $V^{*}V=I_{n}=VV^{*}$.
$\Sigma\in\mathbb{R}^{m\times n}$ is a diagonal matrix of $X^{*}$ in the sense
that

$\sigma_{ij}=\left\\{\begin{array}[]{rcrcrc@{\qquad}l}0&\text{ if }i\neq j\\\
\mathbb{R}_{+}&\text{ if }i=j\\\ \end{array}\right.,$

where the diagonal values are the set of singular values of the matrix
$X^{*}$. This is the full SVD. We can also remove those singular values that
are equal to zero from the matrix, which means $\sigma_{ii}=0$ for
$i\in\text{max}\\{m,n\\}$. Then we get the form of $X^{*}$ that

$X^{*}=U_{r}\Sigma_{r}V_{r},$

where $U_{r}$, $\Sigma_{r}$, and $V_{r}$ $\in\mathbb{R}^{n\times n}$. This is
the condensed SVD. $r$ is the rank of matrix $X^{*}$. The columns of $U_{r}$
forms an orthonormal basis for the image of $X^{*}$. The columns of $V_{r}$
forms an orthonormal basis for the coimage of $X^{*}$.

####  2.3.3. QR Decomposition of $X^{*}$

The QR decomposition method can solve many problems on the SVD list. The QR
decomposition is cheaper to compute in the sense that there are direct
algorithms for computing QR and complete orthogonal decomposition in a finite
number of arithmetic steps. There are several versions of QR decomposition.

  * •

_Full QR Decomposition of $X^{*}$_: For any matrix of
$X^{*}\in\mathbb{R}^{m\times n}$, where $n\neq m$, there exits an unitary
matrix $Q\in\mathbb{R}^{m\times m}$, $Q^{*}Q=QQ^{*}=I_{n}$ and an upper-
triangular matrix $R\in\mathbb{R}^{m\times n}$, where $r_{ij}=0$ whenever
$i>j$, such that

$X^{*}=QR=Q\begin{bmatrix}R_{1}\\\ 0\end{bmatrix}.$

In which, $R_{1}\in\mathbb{R}^{n\times n}$ is an upper-triangular square
matrix in general. If $X^{*}$ has full column rank, which is $rank(X^{*})=n$
then $R_{1}$ is non-singular.

  * •

_Reduced QR Decomposition of $X^{*}$_: For any matrix of
$X^{*}\in\mathbb{R}^{m\times n}$, where $n\neq m$, there exits an orthonormal
matrix $Q\in\mathbb{R}^{m\times n}$, $Q^{*}_{1}Q_{1}=I_{n}$ but
$Q_{1}Q^{*}_{1}\neq I_{m}$ unless $m=n$ and an upper-triangular square matrix
$R\in\mathbb{R}^{n\times n}$, such that

$X^{*}=Q_{1}R_{1}.$

In which, $Q_{1}$ is the first $n$ columns of $Q$ in previous Full QR
decomposition. $Q=[Q_{1},Q_{2}]$, where $Q_{2}\in\mathbb{R}^{m\times(m-n)}$ is
the last $m-n$ columns of $Q$. In fact, we can obtain reduced QR decomposition
of $X^{*}$ from the full QR decomposition of $X^{*}$ by simply multiplying out

$X^{*}=QR=[Q_{1},Q_{2}]\begin{bmatrix}R_{1}\\\
0\end{bmatrix}=Q_{1}R_{1}+Q_{2}0=Q_{1}R_{1}.$

If $X^{*}$ has full column rank, which means $rank(A)=n$, then $R_{1}$ is non-
singular.

  * •

_Rank-Retaining QR Decomposition of $X^{*}$_: For matrix of
$X^{*}\in\mathbb{R}^{m\times n}$, where $rank(X^{*})=r$, there exists a
permutation matrix $\Pi\in\mathbb{R}^{n\times n}$, a unitary matrix
$Q\in\mathbb{R}^{m\times m}$, and a non-singular, upper-triangular square
matrix $R_{1}\in\mathbb{R}^{r\times r}$, such that

$X^{*}\Pi=Q\begin{bmatrix}R_{1}&S\\\ 0&0\end{bmatrix}.$

In which, $S\in\mathbb{R}^{r\times(n-r)}$ is just some matrix with no special
properties. We can also write it as the form that

$X^{*}=QR\Pi^{T}=Q\begin{bmatrix}R_{1}&S\\\ 0&0\end{bmatrix}\Pi^{T}.$

  * •

_Complete Orthogonal Decomposition of $X^{*}$_: For matrix of
$X^{*}\in\mathbb{R}^{m\times n}$, where rank$(X^{*})=r$, there exists an
unitary matrix $Q\in\mathbb{R}^{m\times m}$, an unitary matrix
$U\in\mathbb{R}^{n\times n}$, and a non-singular, lower-triangular square
matrix $L\in\mathbb{R}^{r\times r}$, such that

$X^{*}=Q\begin{bmatrix}L&0\\\ 0&0\end{bmatrix}U^{*}.$

It can be obtained from a full QR decomposition of
$\begin{bmatrix}R^{*}_{1}\\\ S^{*}\end{bmatrix}\in\mathbb{R}^{m\times r}$,
which has full column rank,

$\begin{bmatrix}R^{*}_{1}\\\ S^{*}\end{bmatrix}=Z\begin{bmatrix}R_{2}\\\
0\end{bmatrix},$

where $Z\in\mathbb{R}^{m\times m}$ in unitary and $R_{2}\in\mathbb{R}^{r\times
r}$ in non-singular, upper-triangular square matrix. Observe the Rank-
Retaining QR Decomposition of $X^{*}$ that

$X^{*}=Q\begin{bmatrix}R_{1}&S\\\
0&0\end{bmatrix}\Pi^{T}=Q\begin{bmatrix}R_{2}^{*}&0\\\
0&0\end{bmatrix}Z*\Pi^{T}=Q\begin{bmatrix}L&0\\\ 0&0\end{bmatrix}U^{*}.$

####  2.3.4. LU Decomposition of $X^{*}$

For unstructured matrix $X^{*}$, the most common method is to obtain a
decomposition $X^{*}=LU$, where $L$ is lower triangular and $U$ is upper
triangular. LU decomposition is a sequence of constant LU decomposition. We
are not just interested in the final upper triangular matrix, we are also
interested in keeping track of all the elimination steps. We will omit this
method, because it is intuitively expensive to storage and keep track of the
complete L and U factors.

####  2.3.5. Cholesky Decomposition of $X^{*}$

The Cholesky Decomposition can be computed directly from the matrix equation
$A=R^{T}R$ where $R$ is upper-triangular.

The matrix $X^{*}$ in our vision is dense, the Table 1 compares the relative
merits of normal equations, QR decomposition, and SVD. By Table 1 and an
experimental implementation. We finally selected SVD as the method to further
prune Vision Transformer. As shown in Figure 1, we apply the SVD operation
after each steps of Dimension Pruning operation, which is after pruning
operation on all the MHSA and MLP blocks.

Merits | Evaluation
---|---
Accuracy | Normal Equation $<$ QR Decomposition $<$ SVD
Speed | Normal Equation $>$ QR Decomposition $>$ SVD
Table 1. A comparison of the relative merits among Normal Equations, QR
Decomposition, and SVD.

##  3\. Experimental Evaluation

In this section, we verify the effectiveness of the purposed methods to
further prune the Vision Transformer via Matrix Decomposition on the CIFAR-10
dataset (Krizhevsky et al., 2010).

###  3.1. Dataset

The CIFAR-10 dataset (Krizhevsky et al., 2010) is divided into five training
batches and one test batch, each with $10000$ images. The test batch contains
exactly $1000$ randomly-selected images from each class. The training batches
contain the remaining images in random order, but some training batches may
contain more images from one class than another. Between them, the training
batches contain exactly $5000$ images from each class.

####  3.1.1. CIFAR-10.

The CIFAR-10 is labeled subsets of the $80$ million tiny images dataset. They
were collected by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The
CIFAR-10 dataset consists of $60000$ images in $10$ classes, with $6000$
images per class. There are $50000$ training images and $10000$ test images.

###  3.2. Implementation Details

We modified the code book111https://github.com/Cydia2018/ViT-cifar10-pruning
by adding Matrix Decomposition Methods we mentioned about into the places
after doing the Vision Transformer Pruning operation. Since the code book is
using the CIFAR$-10$, we do not modify other components of the code book.
Figure 2 shows the result of those implementation. We will discuss it in the
next section.

####  3.2.1. Baseline

We evaluate developed pruning method via matrix decomposition on a popular
vision transformer implementation. In our experiments, a $12-$layer
transformer with $10$ heads and $768$ embedding dimensions is evaluated on
CIFAR$-10$. For a fair comparison, we utilize the official implementation of
CIFAR$-10$. On the CIFAR$-10$, we take the released model of Vision
Transformer Pruning as the baseline. We fine-tune the model on the CIFAR$-10$
using batch size $64$ for $30$ epochs. The initial learning rate is set to
$6.25\times 10^{-7}$. Following Deit (Touvron et al., 2021), we use AdamW
(Loshchilov and Hutter, 2018) with cosine learning rate decay strategy to
train and fine-tune the models.

####  3.2.2. Training with Pruning

Based on the baseline model, we train the vision transformer with $l_{1}$
regularization using different sparse regularization rates. We select the
optimal sparse regularization rate (i.e. $0.0001$) on CIFAR$-10$ and apply it
on CIFAR$-10$. The learning rate for training with sparsity is $6.25\times
10-6$ and the number of epochs is $100$. The other training setting follows
the baseline model. After sparsity, we prune the transformer by setting
different pruning thresholds and the threshold is computed by the predefined
pruning rate. Which is very similar with the original dataset.

####  3.2.3. Fine-tuning

We finetune the pruned transformer with the same optimization setting as in
training, except for removing the $l_{1}$ regularization.

###  3.3. Results and Analysis

In Figure 2 shows a result of comparison of accuracy after implemented matrix
decomposition methods, including SVD, QR decomposition, and LU decomposition.
It is easy to see that the SVD performs pretty well comparing with the other
two matrix decomposition methods including QR decomposition, and LU
decomposition. By experience, the computation is not time consuming. The times
of computation is nearly the same except the LU factorization. However, while
comparing VTPSVD and VTP, the accuracy score of VTPSVD is not lower than the
original VTP. which means that we need to further modify and develop our
mathod to achieve a better performance in our case.

| VTP | VTPSVD | VTPQRD | VTPLUD
---|---|---|---|---
ViT patch=2 | 80% | 85.0% | 71.1% | 63.7%
ViT patch=4 | 80% | 62.8% | 68.3% | 57.1%
ViT patch=8 | 30% | 53.2% | 53.5% | 52.3%
CIFAR-10 | 93% | 53.2% | 53.5% | 49.2%
Table 2. A comparison of accuracy score among Vision Transformer Pruning
(VTP), Vision Transformer Pruning with Singular Value Decomposition (VTPSVD),
Vision Transformer Pruning with QR Decomposition (VTPQRD), and Vision
Transformer Pruning with LU Decomposition (VTPLUD) with different patches.

##  4\. Conclusion

In this paper, we build up on and further develop the method of Vision
Transformer Pruning by adding a matrix decomposition operation to meet the
same goal of reduce the storage, run-time memory, and computational demands.
The experiments conducted on CIFAR-10 demonstrate that the pruning with
Singular Value Decomposition method can largely reduce the computation costs
and storage while maintaining a comparingly high accuracy with some patches.
However, this method needs to be further develop to obtain a much robust
optimal performance.

##  5\. Future Work

There are several future works that I am planning to do:

  * •

Test the method on other datasets, i.e. ImageNet (Russakovsky et al., 2015),
which is the one used in the original Vision Transformer Pruning (Zhu et al.,
2021b).

  * •

Further develop and fine-tune this method.

## References

  * (1)
  * Cai et al. (2017) Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. 2017\.  Deep Learning with Low Precision by Half-wave Gaussian Quantization.  arXiv:cs.CV/1702.00953
  * Carion et al. (2020) Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020.  End-to-End Object Detection with Transformers.  arXiv:cs.CV/2005.12872
  * Chen et al. (2021) Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. 2021\.  Pre-Trained Image Processing Transformer.  arXiv:cs.CV/2012.00364
  * Chen et al. (2020) Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. 2020.  Generative Pretraining From Pixels.  , 1691–1703 pages.
  * Cun et al. (1990) Yann Le Cun, John S. Denker, and Sara A. Solla. 1990\.  Optimal Brain Damage.  , 598–605 pages.
  * Denton et al. (2014) Emily Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. 2014.  Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation.  arXiv:cs.CV/1404.0736
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021\.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.
  * Guo et al. (2019) Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019\.  Star-Transformer.  arXiv:cs.CL/1902.09113
  * Gupta et al. (2015) Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. 2015.  Deep Learning with Limited Numerical Precision.  arXiv:cs.LG/1502.02551
  * Han et al. (2021a) Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, Zhaohui Yang, Yiman Zhang, and Dacheng Tao. 2021a.  A Survey on Vision Transformer.  arXiv:cs.CV/2012.12556
  * Han et al. (2021b) Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. 2021b.  Transformer in Transformer.  arXiv:cs.CV/2103.00112
  * Hassibi and Stork (1992) Babak Hassibi and David Stork. 1992.  Second order derivatives for network pruning: Optimal Brain Surgeon.
  * Hu et al. (2021) Jie Hu, Liujuan Cao, Yao Lu, ShengChuan Zhang, Yan Wang, Ke Li, Feiyue Huang, Ling Shao, and Rongrong Ji. 2021\.  ISTR: End-to-End Instance Segmentation with Transformers.  arXiv:cs.CV/2105.00637
  * Krizhevsky et al. (2010) Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. 2010\.  Cifar-10.  _URL http://www. cs. toronto. edu/kriz/cifar. html_ 5 (2010), 4.
  * Lee et al. (2019) Dongsoo Lee, Se Jung Kwon, Byeongwook Kim, and Gu-Yeon Wei. 2019.  Learning Low-Rank Approximation for CNNs.  arXiv:cs.LG/1905.10145
  * Lin et al. (2019) Shaohui Lin, Rongrong Ji, Chao Chen, Dacheng Tao, and Jiebo Luo. 2019.  Holistic CNN Compression via Low-Rank Decomposition with Knowledge Transfer.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 41, 12 (2019), 2889–2905.
  * Liu et al. (2017) Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. 2017\.  Learning Efficient Convolutional Networks through Network Slimming.  arXiv:cs.CV/1708.06519
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.  arXiv:cs.CV/2103.14030
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018.  Fixing Weight Decay Regularization in Adam.  https://openreview.net/forum?id=rk6qdGgCZ
  * Rastegari et al. (2016) Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. 2016.  XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks.  arXiv:cs.CV/1603.05279
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015\.  ImageNet Large Scale Visual Recognition Challenge.  arXiv:cs.CV/1409.0575
  * Sun (2021) Tianyi Sun. 2021\.  How Personal Perceptions Of COVID-19 Have Changed Over Time.
  * Sun and Gini (2021) Tianyi Sun and Maria Gini. 2021.  Study of Natural Language Understanding.  _Journal of Chang Chun Teachers College_ 47 (2021).
  * Sun and Nelson (2023) Tianyi Sun and Bradley Nelson. 2023.  Topological Interpretations of GPT-3.  arXiv:cs.CL/2308.03565
  * Tai et al. (2016) Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, and Weinan E. 2016.  Convolutional neural networks with low-rank regularization.  arXiv:cs.LG/1511.06067
  * Tang et al. (2021) Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang Xu. 2021.  SCOP: Scientific Control for Reliable Neural Network Pruning.  arXiv:cs.CV/2010.10732
  * Tang et al. (2020) Yehui Tang, Shan You, Chang Xu, Jin Han, Chen Qian, Boxin Shi, Chao Xu, and Changshui Zhang. 2020.  Reborn Filters: Pruning Convolutional Neural Networks with Limited Data.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 34, 04 (2020), 5972–5980.
  * Touvron et al. (2021) Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2021.  Training data-efficient image transformers & distillation through attention.  arXiv:cs.CV/2012.12877
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention Is All You Need.  arXiv:cs.CL/1706.03762
  * Wang et al. (2021c) Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. 2021c.  MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers.  arXiv:cs.CV/2012.00759
  * Wang et al. (2021a) Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. 2021a.  Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions.  arXiv:cs.CV/2102.12122
  * Wang et al. (2021b) Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. 2021b.  End-to-End Video Instance Segmentation with Transformers.  arXiv:cs.CV/2011.14503
  * Wen et al. (2016) Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. 2016.  Learning Structured Sparsity in Deep Neural Networks.  arXiv:cs.NE/1608.03665
  * Yang et al. (2020) Zhaohui Yang, Yunhe Wang, Kai Han, Chunjing Xu, Chao Xu, Dacheng Tao, and Chang Xu. 2020.  Searching for Low-Bit Weights in Quantized Neural Networks.  arXiv:cs.CV/2009.08695
  * Yu et al. (2017) Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. 2017\.  On compressing deep models by low rank and sparse decomposition.  , 7370–7379 pages.
  * Zhu et al. (2021b) Mingjian Zhu, Yehui Tang, and Kai Han. 2021b.  Vision transformer pruning.
  * Zhu et al. (2021a) Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. 2021a.  Deformable DETR: Deformable Transformers for End-to-End Object Detection.  arXiv:cs.CV/2010.04159