# CNN-Augmented Visual-Inertial SLAM with Planar Constraints

Pan Ji, Yuan Tian, Qingan Yan, Yuxin Ma, and Yi Xu The authors are with OPPO
US Research Center, Palo Alto, CA 94303, USA<EMAIL_ADDRESS>
###### Abstract

We present a robust visual-inertial SLAM system that combines the benefits of
Convolutional Neural Networks (CNNs) and planar constraints. Our system
leverages a CNN to predict the depth map and the corresponding uncertainty map
for each image. The CNN depth effectively bootstraps the back-end optimization
of SLAM and meanwhile the CNN uncertainty adaptively weighs the contribution
of each feature point to the back-end optimization. Given the gravity
direction from the inertial sensor, we further present a fast plane detection
method that detects horizontal planes via one-point RANSAC and vertical planes
via two-point RANSAC. Those stably detected planes are in turn used to
regularize the back-end optimization of SLAM. We evaluate our system on a
public dataset, _i.e._ , EuRoC, and demonstrate improved results over a state-
of-the-art SLAM system, _i.e._ , ORB-SLAM3.

##  I INTRODUCTION

Simultaneous Localization and Mapping (SLAM) aims to reconstruct the 3D map of
a scene and localize the position of an agent with respect to the map at the
same time. It has been extensively researched over the years [3] and has found
various applications, such as autonomous navigation, Virtual Reality (VR), and
Augmented Reality (AR).

The sensor types that SLAM can work with include monocular/stereo cameras,
depth sensor, and the inertial measurement unit (IMU). The monocular mode is
the minimal setup for visual SLAM but does not produce scale information.
Stereo and RGB-D modes can provide metric scale recovery for map points and
camera poses; however, extra sensor requirements (_e.g._ , stereo cameras and
depth sensors) hinder their wide applicability. Nowadays, with inertial
sensors becoming cheaper and easily accessible via devices such as mobile
phones, visual-inertial SLAM has become a popular option for self-
localization, _e.g._ , in mobile-based AR applications [24], as inertial
sensors can help SLAM achieve metric-scale reconstruction. Nonetheless,
visual-inertial SLAM still suffers from scale drift because the inertial
sensor inevitably accumulates errors over time [24].

On the other hand, depth prediction from a single image using CNNs has also
achieved great success [10, 11], demonstrating complimentary features to
traditional geometric methods. Hence, a few works [27, 29, 21, 26, 30] try to
combine the best of both worlds, _i.e._ , deep-learning-based depth prediction
and geometry-based SLAM. However, they do not incorporate the inertial sensors
that are currently easy to access and the structural constraints that are
commonly presented in real-world scenes.

In this paper, we target an accurate and robust visual-inertial SLAM system
that uses a monocular camera and an IMU sensor to capture input data. To this
end, we propose to augment the visual-inertial SLAM with CNNs that predict the
depth map and the uncertainty map for each image, forming a pseudo RGB-D
inertial SLAM system. As such our SLAM system combines the best of geometry
and deep learning, albeit still using a minimal sensor set-up. Furthermore,
since gravity direction can be estimated from inertial measurements, we can
fit the dominant planes much faster with fewer points required to sample. We
further enhance the robustness of plane detection leveraging the depth and
uncertainty estimated by the CNN, and in turn use the detected planes to
regularize the back-end optimization of SLAM.

In summary, our contributions include:

  * (a)

a robust visual-Inertial SLAM system augmented with CNN depth and uncertainty;

  * (b)

a fast and robust plane fitting method using gravity direction from inertial
measurements, map points from SLAM, and depth and uncertainty from the CNN;

  * (c)

a simple yet effective plane regularization energy term for SLAM back-end
optimization.

##  II RELATED WORK

###  II-A Geometric SLAM

SLAM has been a long standing problem in the community of robotics and
computer vision. Existing geometric SLAM systems can be roughly divided into
two categories: feature-based SLAM [25, 22] and direct SLAM [8, 7]. Feature-
based SLAM systems first detect feature points and then compute the geometry
using feature correspondences. The performance of those systems degrades
significantly when feature detection and tracking work poorly, _e.g._ , in
scenes with little texture. In contrast, direct SLAM systems rely on the
photometric error, _i.e._ , the difference of pixel intensities, to optimize
the geometry, and thus bypass the need of extracting feature points and
computing feature descriptions. However, direct methods assume brightness
constancy and suffer a lot in environments that have rapid lighting changes.

The robustness of SLAM can be improved using multi-modality sensors, such as
the RGB-D camera and the IMU sensor. Along this line, the ORB-SLAM family of
methods [22, 23, 4] gradually support more sensors. Starting from the
monocular ORB-SLAM [22], ORB-SLAM2 [23] adds both stereo and RGB-D modes, and
ORB-SLAM3 [4] further incorporates the inertial sensor, demonstrating the
state-of-the-art performance. Due to its robustness and versatility, we build
our system upon ORB-SLAM3.

There have also been other efforts [19, 1, 13, 16, 31, 12] that exploit
structural information, _e.g._ , planar constraints, to improve the
performance of geometric SLAM. A few of them [19, 1, 32] detect and fit 3D
planes within feature-based monocular SLAM, which have difficulties in finding
large textureless planes. Others [13, 16, 31, 12] enforce planar constraints
for RGB-D SLAM; thus achieving more reliable results. However, none of them
utilize the inertial sensor to simplify the plane fitting problem. In our
work, we combine the inertial measurements, sparse map points, dense CNN depth
and uncertainty, to build a fast and robust method for plane detection and
fitting.

###  II-B CNN-Augmented SLAM

Depth prediction using deep CNNs has recently achieved great success [10, 11,
27, 33, 14, 20, 15]. Hence, a few works [27, 29, 21, 26, 30] try to combine
deep learned depth/pose and geometric SLAM. In particular, [26], [21] and [29]
leverage deep depth, pose and/or uncertainty to bootstrap a semi-dense visual
odometry system. [27] uses the CNN depth as the input to an RGB-D SLAM system
and builds a self-improving loop for depth prediction and monocular SLAM.
However, none of them incorporates the inertial sensors, which are widely
available on mobile devices, and the planar constraints, which commonly exist
in indoor environments. In contrast, our system is incarnated with deep neural
networks, visual-inertial SLAM, plane detection, and planar constraints.

##  III METHOD

In this section, we present our CNN-augmented visual-inertial SLAM and show
how we can detect and fit 3D planes fast and robustly under our framework.

###  III-A Overall Pipeline

Our visual-inertial SLAM combines conventional geometric SLAM and modern CNN-
based depth/uncertainty predictions, and further enforces planar constraints
to improve performance. The overall pipeline is depicted in Figure 1. We first
train a CNN to predict the depth map $D_{t}$ and its uncertainty map $U_{t}$
from a single image $I_{t}$. The CNN can be trained in a supervised [17] or
self-supervised manner [29]. The image $I_{t}$, its predicted depth map
$D_{t}$, uncertainty map $U_{t}$, and inertial measurements $Y_{t}$ are then
used as the input to an uncertainty-guided visual-inertial SLAM system. Planes
are detected in a separate thread of the front-end. Planar constraints are
enforced in the back-end through the factor graph optimization after a set of
reliable planes are detected. The outputs of the system include map points and
camera poses in the metric scale and steadily detected 3D planes.

Figure 1: Overall pipeline of the proposed CNN-augmented visual-inertial SLAM
with planar constraints.

###  III-B CNN Depth and Uncertainty

Deep learning is complementary to geometry in many scenarios. For instance,
geometric SLAM tends to fail totally in textureless regions, whereas CNN-based
methods can still predict reasonable depth values in such regions leveraging
rich priors in the training data. So we leverage a CNN that predicts the depth
and the uncertainty to augment the geometric SLAM. Following [29], we train
the network in a self-supervised manner. However, note that our system is
agnostic to the training strategy of the deep neural network and a supervised
method shall also work. The network structure for depth and uncertainty
prediction from an image is outlined in Figure 2. The CNN takes an auto-
encoder structure with skip-connections between the encoder and the decoder.
We refer the reader to [29] for the detailed network structure.

CNN depth is not equally accurate in different regions of an image [17]. We
thus design the network to simultaneously predict the confidence or
uncertainty score for the depth prediction. Specifically, we follow [17, 29]
and define the uncertainty weighted self-supervised loss as follows,

$L_{self}=\sum\frac{r_{pho}}{u}+\log{u},$ (1)

where $r_{pho}$ is the photometric loss as defined in [11], and $u$ is the
uncertainty value predicted for each pixel. The total loss function is the
summation of the uncertainty weighted self-supervised loss and a depth
smoothness loss [11],

$L_{total}=L_{self}+\lambda L_{smooth},$ (2)

where $\lambda$ is a weighting parameter and is set to $1.0\times 10^{-3}$.
After the network is trained, we do inference for each coming image to predict
its depth map and the uncertainty map, and feed them into our SLAM system.

Figure 2: Network structure for CNN depth and uncertainty prediction. The
network follows an auto-encoder structure with skip connections, and takes one
image as input to predict its depth map and uncertainty map.

###  III-C Uncertainty-Guided Visual-Inertial SLAM

Our system is built upon ORB-SLAM3 [4], but is compatible with any existing
feature-based SLAM that supports RGB-D inertial input. Instead of relying a
depth sensor to get the depth image, we use the CNN predicted depth as the
depth input to form a pseudo RGB-D inertial SLAM. We further augment SLAM with
the uncertainty prediction to adaptively weigh the contribution of feature
points in the back-end optimization of bundle adjustment [28].

Given map points $X^{j}\in\mathbb{R}^{3}$ in the world coordinates and the 2D
observations $x_{i}^{j}\in\mathbb{R}^{2}$ in frame $i$, their re-projection
errors can be computed as,

$e_{ij}=x_{i}^{j}-\pi(T_{i}\bar{X}^{j}),$ (3)

where $\pi$ is the projection function, $T_{i}\in\mathbb{R}^{3\times 4}$ is
the transformation from the world coordinate system to frame $i$, and
$\bar{X}^{j}\in\mathbb{R}^{4}$ corresponds to the homogeneous coordinates of
$X^{j}$. Bundle adjustment aims to minimize the reprojection errors of map
points across a bundle of frames,

$C^{l}=\sum_{i,j}\rho_{h}(e_{ij}^{T}\Omega_{ij}^{-1}e_{ij}),$ (4)

where $\rho_{h}$ is the robust Huber cost function and $\Omega_{ij}$ is the
covariance matrix measuring the uncertainty of each detected feature point. In
a conventional geometric SLAM system, _e.g._ , ORB-SLAM, the covariance is
pre-computed according to the scale at which the feature point is detected;
thus all feature points detected from the same scale share the same
covariance. In this paper, we instead propose to use the CNN learned
uncertainties to adaptively weigh the re-projection errors. Specifically, we
use a simple method to convert the uncertainty value to a covariance metrix,
_i.e._ ,

$\Omega_{ij}=1/(I-U_{ij}),$ (5)

where $U_{ij}=u_{ij}I_{2\times 2}$. Therefore, the uncertainty-guided
reprojection errors become

$C_{uncer}^{l}=\sum_{i,j}\rho_{h}(e_{ij}^{T}(I-U_{ij})e_{ij})).$ (6)

Since our SLAM system takes CNN predicted depth as input and CNN depth is not
always accurate, this uncertainty guidance help to downweigh the contributions
of those map points that potentially have high depth errors.

Following ORB-SLAM3 [23], we use the depth and left features to generate
feature points on a virtual right view. We then compute the reprojection
errors for the virtual right view so that we have the overall reprojection
cost as follows,

$C_{reproj}=C_{uncer}^{l}+\alpha C_{uncer}^{r},$ (7)

where $C_{uncer}^{r}$ is the reprojection cost for the virtual right view and
$\alpha\leq 1$.

###  III-D Fast Plane Fitting

The world is mostly made of planes, especially for indoor environments. At the
application side, AR often requires generating virtual objects on top of real-
world planar surfaces. So, in our system, we also make use of 3D planar
information for back-end regularization. However, solely utilizing 3D map
points suffers a lot from under-constrained situations, as many walls are
texture-less. Instead, we draw upon both the dense CNN depth cue and sparse 3D
map points.

It would be slow if we naïvely fit planes using the dense depth map and all
map points. We thus propose a novel fast plane fitting method, leveraging the
prior information of gravity direction and uncertainty measurements. In
particular, we seek to find two types of dominant planes: i) horizontal planes
that are perpendicular to the gravity direction; ii) vertical planes that are
parallel to the gravity direction.

In the general case where the gravity direction is unknown, we would have to
sample three points to fit a plane. In contrast, given the gravity direction,
the degree-of-freedom (DoF) of horizontal planes becomes one, so we only need
to sample one point to determine a horizontal plane. Similarly, the DoF of
vertical planes is two and we only need two points to determine a vertical
plane. Note that, reduction in DoF of a parametric model can significantly
speed up the RANSAC algorithm [9]. Theoretically, the number of iterations for
RANSAC equals

$k=\frac{\log(1-p)}{\log(1-w^{n})},$ (8)

where $p$ is the desired success probability, $w$ is the inlier ratio, and $n$
is the number of points to fit the model. Assume $p=0.99$, $w=0.2$. If $n=3$,
then $k=249$; if $n=2$, then $k=49$; if $n=1$, then $k=9$. From this example,
we can see the benefits of applying the one-point RANSAC (for horizontal
planes) and two-point RANSAC (for vertical planes) in terms of computational
complexity, as compared to the vanilla three-point RANSAC.

We further assume that indoors scenes consist mainly of planes that are
orthogonal to one another in a “Manhattan world” manner [5, 6]. This
assumption allows us to further accelerate the detection of wall planes. Once
we have detected one main wall plane using two-point RANSAC method, we assume
there are additional planes parallel or orthogonal to the main wall plane.
This enables us to remove one DoF from the problem and test subsequent plane
hypotheses with only one point per sample.

When detecting planes from SLAM map points, we observe that false positive
plane inliers such as points belonging to other surfaces can introduce errors
in plane normal direction estimation. We also find that clusters of colinear
points can often lead to false positive detection. We mitigate these
challenges by augmenting map points with normal direction computed from CNN
depth, which is computed as follows:

$\displaystyle n_{x,y}=$ $\displaystyle
R\cdot(K^{-1}([x,y-\delta,1]^{T}-[x,y,1]^{T}))$ (9)
$\displaystyle\times(K^{-1}([x-\delta,y,1]^{T}-[x,y,1]^{T})),$

where $n_{x,y}$ is the normal direction at image coordinate $(x,y)$, $K$ is
the camera intrinsic matrix, $\delta$ stands for a small pixel shift where the
corresponding surface is assumed to be piece-wise planar and $R$ represents
the rotation matrix from the camera to the world coordinate system. Map point
normals are maintained and updated with each CNN depth frame. Map point
normals can help significantly reduce false positive inlier points, resulting
in more robust plane detection especially for vertical planes.

Moreover, we observe that CNN depth often has a global scale shift. To
overcome this issue, we additionally employ a simple yet effective method to
refine CNN depth by using SLAM map points. As map points are continually
optimized in the back-end and in general more accurate, it is therefore able
to reproject them onto images to compute an average scale difference with CNN
depth:

$s_{correction}=\frac{1}{N}\sum_{i=1}^{N}\frac{D_{i}^{SLAM}}{D_{i}^{CNN}},$
(10)

where $N$ is the number of visible map points, $D_{i}^{SLAM}$ is the depth
from the $i^{th}$ map point, and $D_{i}^{CNN}$ is the corresponding CNN depth.
Then, the computed factor is applied to CNN depth for scale correction.

We use the refined dense CNN depth map to assist recovering planes. Instead of
equally sampling the CNN depth map, we utilize CNN uncertainties to guide the
sampling process. In regions of higher uncertainties, the depth points are
sampled with lower probability, and vice versa. The inlier count of CNN depth
points is downweighed by $0.5$ as they are less accurate than SLAM map points.

$T_{0}$$T_{1}$$T_{3}$$T_{t}$$X^{i}_{v}$$X^{j}$$x_{0}$0$x_{1}$$x_{2}$$x_{3}$$x_{4}$
Figure 3: Factor graph for bundle adjustment with planar constraints. Note
that our planar constraints are implemented as a unary term in the factor
graph (_i.e._ , a self-connecting edge for the node $X_{v}^{i}$).

###  III-E Planar Constraints

Given steadily detected planes, we build an extra energy term on top of the
re-projection error term for back-end optimization. If a point has a high
probability lying on a plane, the distance between them should be as small as
possible. Therefore, for each detected plane $v\in\mathbb{R}^{4}$, we first
find its inlying map points $X_{v}^{j}$ by thresholding the distance from map
points to planes, which is controlled by $\theta$. Then the plane constraint
term in the back-end optimization can be expressed as:

$C_{plane}=\sum_{j,v}\rho_{h}(v^{T}\bar{X}_{v}^{j}),$ (11)

where $\bar{X}_{v}^{j}$ corresponds to the homogeneous coordinates of
$X_{v}^{j}$. Since the plane types used in our work, such as walls and the
ground, are well defined, it is not necessary to optimize plane parameters
along with other variances in the back-end. Instead, we adopt an iterative
manner, where planes are recomputed asynchronously in every $N_{d}$ frames. We
set $N_{d}=30$ in all of our experiments. Then the newly updated inlying
planar points and plane parameters are used in the following graph
optimization. In this manner, we can effectively express the planar constraint
as a serial of unary edges in the factor graph optimization [18], as shown in
Figure 3. The observations on these edges are set to $0$, as theoretically the
residual between planes and inlying points should be $0$. Our overall cost for
bundle adjustment is the summation of $C_{reproj}$ and $C_{plane}$, _i.e._ ,

$C=C_{reproj}+\beta C_{plane},$ (12)

where $\beta\leq 1$, and is set to $0.1$ in our experiments. The planar
constraint can be applied in both local and full bundle adjustment processes
as a guidance for map points optimization.

##  IV EXPERIMENTS

We evaluate our system on the EuRoC MAV dataset [2] which provides stereo
images, IMU data, and groundtruth camera 6-DoF poses. We choose the VICON Room
sequences of EuRoC to perform the evaluation because these sequences contain
large planes, such as the ground plane and walls. We exclude Sequence V203 due
to many missing frames in one of the cameras [29]. In the following
subsections, we first discuss the implementation details of our system, and
then present both quantitative and qualitative results of ORB-SLAM3 [4] and
our method on EuRoC.

###  IV-A Implementation Details

Figure 4: Depth and uncertainty prediction. From the left to the right are
respectively the original input, the CNN depth and the CNN uncertainty.

We follow [29] to train the network to predict the depth and uncertainty using
monocular and stereo images in a self-supervised manner. Due to the limited
size of this dataset, we use five VICON Room sequences for training to achieve
the best model. A supervised method can also be used to predict the depth map.
The depth and uncertainty maps of some sample images are shown in Figure 2. We
replace the covariance matrix in the back-end bundle adjustment with the CNN
uncertainty prediction as in Eq. (6) and set $\alpha$ to 1. To better gauge
the drift in camera poses, we disable loop closure for all competing methods.

In plane detection, map point surface normal estimated from CNN depth are used
to help filter out outliers. We maintain a hash map from map point ID to their
surface normals as well as their last observed distance. Surface normals of
map points in view are updated by new CNN depth frames. The new normal vector
is a weighted average based on inverse of observation distance,

$\overrightarrow{n}=\frac{\overrightarrow{n}_{new}\cdot{d}+\overrightarrow{n}\cdot{d_{new}}}{d_{new}+d},$
(13)

where $\overrightarrow{n}$ is the estimated map point surface normal, $d$ is
the observation distance of the map point and subscript $new$ indicates new
values estimated from the current frame.

We implement multi-plane detection by iteratively detecting the max inlier
count plane, recording its inlier points and detecting the next plane from the
remaining points. We found the inlier distance threshold $\theta=0.05m$ and
ratio $w=0.1$ work well for VICON sequences of the EuRoC dataset.

###  IV-B Experimental Results

TABLE I: Performance comparison on EuRoC (RMSE ATE in meters).

| V101 | V102 | V103 | V201 | V202 | Mean
---|---|---|---|---|---|---
ORB-SLAM3 Mono-Inertial | 0.0367 | 0.0605 | 0.0767 | 0.0480 | 0.0620 | 0.0568
ORB-SLAM3 Stereo-Inertial | 0.0293 | 0.0596 | 0.0729 | 0.0699 | 0.0637 | 0.0591
Our Mono-CNN Inertial | 0.0352 | 0.0151 | 0.0381 | 0.0320 | 0.0404 | 0.0322
Our Mono-CNN Inertial + P | 0.0346 | 0.0143 | 0.0352 | 0.0315 | 0.0393 | 0.0310
Our Mono-CNN Inertial + U | 0.0355 | 0.0127 | 0.0315 | 0.0244 | 0.0583 | 0.0325
Our Mono-CNN Inertial + U + P | 0.0352 | 0.0126 | 0.0283 | 0.0234 | 0.0468 | 0.0293

Following [4], we use the RMSE absolute translation error (ATE) as the
quantitative evaluation metric. We compare our method with both the mono-
inertial mode and the stereo-inertial mode of ORB-SLAM3, and show the
performance comparison in Table I, where all numbers are computed by taking
the median of 5 runs. We note that the mono-inertial mode and the stereo-
inertial mode of ORB-SLAM3 work on par with each other. To ablate the
contribution of each new component, we add one new module each time to the
baseline system (_i.e._ , ORB-SLAM3 Mono-Inertial), leading to four versions
of our system, denoted as Our Mono-CNN Inertial, Our Mono-CNN Inertial +
P(lane), Our Mono-CNN Inertial + U(ncertainty), and Our Mono-CNN Inertial +
U(ncertainty) + P(lane), respectively. By comparing Our Mono-CNN Inertial and
ORB-SLAM3 Mono-Inertial in Table I, it is evident that CNN depth consistently
improves the accuracy of camera poses in all five sequences, and in some cases
by a large margin (_e.g._ , for V102). This verifies the contribution of CNN
depth in bootstraping the performance of feature-based SLAM. Adding
uncertainty prediction into the system (_i.e._ , Our Mono-CNN Inertial + U)
further improves the performance in 3 out of 5 sequences. Comparing Our Mono-
CNN Inertial + P and Our Mono-CNN Inertial + U + P, we can also observe that
the mean RMSE of ATE improves by using CNN uncertainty. Our final system
(_i.e._ , Our Mono-CNN Inertial + U + P) consistently outperforms Our Mono-CNN
Inertial + U and achieves the best performance, combining the benefits of CNN
depth, uncertainty, and structural constraints.

To investigate the stability of our system, we show the boxplot of the RMSE
values of ATE over 5 runs for all the 5 sequences in Figure 6. We can observe
that, on V101 and V102, our system not only achieves lower the RMSE ATE than
ORB-SLAM3, but is also more stable with smaller variance. On V103 and V201,
our system is on par with ORB-SLAM3 in terms of performance stability.
However, our system demonstrates larger variance on V202. We conjecture that
this is due to much more violent motions in V202 which cause difficulties in
self-supervised depth and uncertainty prediction [29].

|
---|---
V101 | V102
|
V103 | V201
|
| V202
Figure 5: Boxplot of the RMSE values of ATE over 5 runs on VICON sequences.
Best viewed on screen with zoom-in.

In Figure 6, we further give a visual comparison of camera trajectories
computed by ORB-SLAM3 and our system. From the figure, it is clear that our
system recovers more accurate camera poses.

Figure 6: Camera trajectory visual comparison on EuRoC VICON sequences. We
plot the difference between the ORB-SLAM3 trajectory and the groundtruth in
red and the difference between our trajectory and the groundtruth in blue. We
can clearly see that our trajectories align better with the groundtruth than
ORB-SLAM3. Best viewed on screen with zoom-in.

Our system can detect and fit multiple dominant planes that are present in the
scene. We visualize the plane detection in a sample scene in Figure 7. Our
system is able to detect and find 3D horizontal planes and vertical planes
reliably. We also encourage the reader to watch the supplementary demo video
for our plane detection.

---
Figure 7: Plane visualization. The horizontal ground plane is plotted in blue,
the vertical planes are in red, and the invisible planes (in the current time
step) are in black.

##  V CONCLUSIONS

The proposed system maintains the minimal sensor setup of using a monocular
camera and an inertial sensor (_e.g._ , IMU) for SLAM, yet achieving the
benefits of using the RGB-D inertial input via a CNN to predict depth and
uncertainty. In this sense, the proposed system combines the best of both
worlds (_i.e._ , geometry and deep learning). In addition, we present a fast
and effective plane detection method along with our SLAM system. Our system is
fast (real-time) and accurate in reconstructing map points, camera poses, and
major planes.

## References

  * [1] C. Arndt, R. Sabzevari, and J. Civera.  From points to planes-adding planar constraints to monocular slam factor graphs.  In IROS, 2020.
  * [2] M. Burri, J. Nikolic, P. Gohl, T. Schneider, J. Rehder, S. Omari, M. W. Achtelik, and R. Siegwart.  The euroc micro aerial vehicle datasets.  The International Journal of Robotics Research, 2016.
  * [3] C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira, I. Reid, and J. J. Leonard.  Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age.  IEEE Transactions on robotics, 32(6):1309–1332, 2016.
  * [4] C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. Montiel, and J. D. Tardós.  ORB-SLAM3: An accurate open-source library for visual, visual-inertial and multi-map slam.  arXiv preprint arXiv:2007.11898, 2020.
  * [5] J. Coughlan and A. Yuille.  Manhattan world: Compass direction from a single image by bayesian inference.  In IEEE International Conference on Computer Vision, 1999.
  * [6] J. Coughlan and A. Yuille.  Manhattan world.  Neural Computation, 15:1063–1088, 2003.
  * [7] J. Engel, V. Koltun, and D. Cremers.  Direct sparse odometry.  TPAMI, 40(3):611–625, 2017.
  * [8] J. Engel, T. Schöps, and D. Cremers.  LSD-SLAM: Large-scale direct monocular slam.  In ECCV, 2014.
  * [9] M. A. Fischler and R. C. Bolles.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.  Communications of the ACM, 24(6):381–395, 1981.
  * [10] H. Fu, M. Gong, C. Wang, K. Batmanghelich, and D. Tao.  Deep ordinal regression network for monocular depth estimation.  In CVPR, 2018.
  * [11] C. Godard, O. Mac Aodha, M. Firman, and G. J. Brostow.  Digging into self-supervised monocular depth estimation.  In ICCV, 2019.
  * [12] M. Hosseinzadeh, Y. Latif, T. Pham, N. Suenderhauf, and I. Reid.  Structure aware slam using quadrics and planes.  In ACCV, 2018.
  * [13] M. Hsiao, E. Westman, and M. Kaess.  Dense planar-inertial slam with structural constraints.  In ICRA, 2018.
  * [14] P. Ji, R. Li, B. Bhanu, and Y. Xu.  Monoindoor: Towards good practice of self-supervised monocular depth estimation for indoor environments.  In ICCV, pages 12787–12796, 2021.
  * [15] P. Ji, Q. Yan, Y. Ma, and Y. Xu.  Georefine: Self-supervised online depth refinement for accurate dense mapping.  arXiv preprint arXiv:2205.01656, 2022.
  * [16] M. Kaess.  Simultaneous localization and mapping with infinite planes.  In ICRA, 2015.
  * [17] A. Kendall and Y. Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  In NuerIPS, 2017.
  * [18] R. Kümmerle, G. Grisetti, H. Strasdat, K. Konolige, and W. Burgard.  g 2 o: A general framework for graph optimization.  In ICRA, 2011.
  * [19] G. H. Lee, F. Fraundorfer, and M. Pollefeys.  Mav visual slam with plane constraint.  In ICRA, 2011.
  * [20] J. Liu, P. Ji, N. Bansal, C. Cai, Q. Yan, X. Huang, and Y. Xu.  Planemvs: 3d plane reconstruction from multi-view stereo.  arXiv preprint arXiv:2203.12082, 2022.
  * [21] S. Y. Loo, A. J. Amiri, S. Mashohor, S. H. Tang, and H. Zhang.  CNN-SVO: Improving the mapping in semi-direct visual odometry using single-image depth prediction.  In ICRA, 2019.
  * [22] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos.  ORB-SLAM: a versatile and accurate monocular slam system.  IEEE transactions on robotics, 31(5):1147–1163, 2015.
  * [23] R. Mur-Artal and J. D. Tardós.  ORB-SLAM2: An open-source slam system for monocular, stereo, and rgb-d cameras.  IEEE Transactions on Robotics, 33(5):1255–1262, 2017.
  * [24] T. Qin, P. Li, and S. Shen.  Vins-mono: A robust and versatile monocular visual-inertial state estimator.  IEEE Transactions on Robotics, 34(4):1004–1020, 2018.
  * [25] S. Song, M. Chandraker, and C. C. Guest.  Parallel, real-time monocular visual odometry.  In ICRA, 2013.
  * [26] K. Tateno, F. Tombari, I. Laina, and N. Navab.  CNN-SLAM: Real-time dense monocular slam with learned depth prediction.  In CVPR, 2017.
  * [27] L. Tiwari, P. Ji, Q.-H. Tran, B. Zhuang, S. Anand, and M. Chandraker.  Pseudo rgb-d for self-improving monocular slam and depth prediction.  In ECCV, 2020.
  * [28] B. Triggs, P. F. McLauchlan, R. I. Hartley, and A. W. Fitzgibbon.  Bundle adjustment—a modern synthesis.  In International workshop on vision algorithms, pages 298–372. Springer, 1999.
  * [29] N. Yang, L. v. Stumberg, R. Wang, and D. Cremers.  D3VO: Deep depth, deep pose and deep uncertainty for monocular visual odometry.  In CVPR, 2020.
  * [30] N. Yang, R. Wang, J. Stuckler, and D. Cremers.  Deep virtual stereo odometry: Leveraging deep depth prediction for monocular direct sparse odometry.  In ECCV, 2018.
  * [31] S. Yang, Y. Song, M. Kaess, and S. Scherer.  Pop-up SLAM: Semantic monocular plane slam for low-texture environments.  In IROS, 2016.
  * [32] L. Zhou, D. Koppel, H. Ju, F. Steinbruecker, and M. Kaess.  An efficient planar bundle adjustment algorithm.  In ISMAR, 2020.
  * [33] Y. Zou, P. Ji, Q.-H. Tran, J.-B. Huang, and M. Chandraker.  Learning monocular visual odometry via self-supervised long-term modeling.  arXiv preprint arXiv:2007.10983, 2020.


∎

11institutetext: Derya Soydaner 22institutetext: Department of Brain and
Cognition, University of Leuven (KU Leuven), Leuven, Belgium
Tel.: +32-16710471
22email<EMAIL_ADDRESS>
# Attention Mechanism in Neural Networks:

Where it Comes and Where it Goes

Derya Soydaner

(Received: 22 July 2021 / Accepted: 27 April 2022)

###### Abstract

A long time ago in the machine learning literature, the idea of incorporating
a mechanism inspired by the human visual system into neural networks was
introduced. This idea is named the _attention mechanism_ , and it has gone
through a long development period. Today, many works have been devoted to this
idea in a variety of tasks. Remarkable performance has recently been
demonstrated. The goal of this paper is to provide an overview from the early
work on searching for ways to implement attention idea with neural networks
until the recent trends. This review emphasizes the important milestones
during this progress regarding different tasks. By this way, this study aims
to provide a road map for researchers to explore the current development and
get inspired for novel approaches beyond the attention.

###### Keywords:

Attention mechanism Neural networks Deep learning Survey

##  1 Introduction

Human eye sees the world in an interesting way. We suppose as if we see the
entire scene at once, but this is an illusion created by the subconscious part
of our brain goodfellow2016 . According to the Scanpath theory noton1971 ;
noton1971a , when the human eye looks at an image, it can see only a small
patch in high resolution. This small patch is called the _fovea_. It can see
the rest of the image in low resolution which is called the _periphery_. To
recognize the entire scene, the eye performs feature extraction based on the
fovea. The eye is moved to different parts of the image until the information
obtained from the fovea is sufficient for recognition alpaydin1996 . These eye
movements are called _saccades_. The eye makes successive fixations until the
recognition task is complete. This sequential process happens so quickly that
we feel as if it happens all at once.

Biologically, this is called _visual attention system_. Visual attention is
defined as the ability to dynamically restrict processing to a subset of the
visual field ahmad1991 . It seeks answers for two main questions: _What_ and
_where_ to look? Visual attention has been extensively studied in psychology
and neuroscience; for reviews see posner1990 ; bundesen1990 ; desimone1995 ;
corbetta2002 ; petersen2012 . Besides, there is a large amount of literature
on modeling eye movements rimey1990 ; sheliga1994 ; sheliga1995 ; hoffman1995
. These studies have been a source of inspiration for many artificial
intelligence tasks. It has been discovered that the attention idea is useful
from image recognition to machine translation. Therefore, different types of
attention mechanisms inspired from the human visual system have been developed
for years. Since the success of deep neural networks has been at the forefront
for these artificial intelligence tasks, these mechanisms have been integrated
into neural networks for a long time.

This survey is about the journey of attention mechanisms used with neural
networks. Researchers have been investigating ways to strengthen neural
network architectures with attention mechanisms for many years. The primary
aim of these studies is to reduce computational burden and to improve the
model performance as well. Previous work reviewed the attention mechanisms
from different perspectives chaudhari2019 , or examined them in context of
natural language processing (NLP) galassi2019 ; lee2019a . However, in this
study, we examine the development of attention mechanisms over the years, and
recent trends. We begin with the first attempts to integrate the visual
attention idea to neural networks, and continue until the most modern neural
networks armed with attention mechanisms. One of them is the _Transformer_ ,
which is used for many studies including the _GPT-3_ language model brown2020
, goes beyond convolutions and recurrence by replacing them with only
attention layers vaswani2017 . Finally, we discuss how much more can we move
forward, and what’s next?

##  2 From the Late 1980s to Early 2010s: The Attention Awakens

The first attempts at adapting attention mechanisms to neural networks go back
to the late 1980s. One of the early studies is the improved version of the
_Neocognitron_ fukushima1980  with selective attention fukushima1987 . This
study is then modified to recognize and segment connected characters in
cursive handwriting fukushima1993 . Another study describes VISIT, a novel
model that concentrates on its relationship to a number of visual areas of the
brain ahmad1991 . Also, a novel architecture named Signal Channelling
Attentional Network (SCAN) is presented for attentional scanning postma1997 .

Early work on improving the attention idea for neural networks includes a
variety of tasks such as target detection schmidhuber1991 . In another study,
a visual attention system extracts regions of interest by combining the
bottom-up and top-down information from the image milanese1994 . A recognition
model based on selective attention which analyses only a small part of the
image at each step, and combines results in time is described alpaydin1996 .
Besides, a model based on the concept of selective tuning is proposed
tsotsos1995 . As the years go by, several studies that use the attention idea
in different ways have been presented for visual perception and recognition
culhane1992 ; reisfeld1995 ; rybak1998 ; keller1999 .

By the 2000s, the studies on making attention mechanisms more useful for
neural networks continued. In the early years, a model that integrates an
attentional orienting _where_ pathway and an object recognition _what_ pathway
is presented miau2001 . A computational model of human eye movements is
proposed for an object class detection task Zhang2006 . A serial model is
presented for visual pattern recognition gathering Markov models and neural
networks with selective attention on the handwritten digit recognition and
face recognition problems salah2002 . In that study, a neural network analyses
image parts and generates posterior probabilities as observations to the
Markov model. Also, attention idea is used for object recognition walther2002
, and the analysis of a scene schill2001 . An interesting study proposes to
learn sequential attention in real-world visual object recognition using a
Q-learner paletta2005 . Besides, a computational model of visual selective
attention is described to automatically detect the most relevant parts of a
color picture displayed on a television screen lemeur2006 . The attention idea
is also used for identifying and tracking objects in multi-resolution digital
video of partially cluttered environments gould2007 .

In 2010, the first implemented system inspired by the fovea of human retina
was presented for image classification larochelle2010 . This system jointly
trains a restricted Boltzmann machine (RBM) and an attentional component
called the fixation controller. Similarly, a novel attentional model is
implemented for simultaneous object tracking and recognition that is driven by
gaze data bazzani2011 . By taking advantage of reinforcement learning, a novel
recurrent neural network (RNN) is described for image classification mnih2014
. _Deep Attention Selective Network (DasNet)_ , a deep neural network with
feedback connections that are learned through reinforcement learning to direct
selective attention to certain features extracted from images, is presented
stollenga2014 . Additionally, a deep learning based framework using attention
has been proposed for generative modeling tang2014 .

##  3 2015: The Rise of Attention

It can be said that 2015 is the golden year of attention mechanisms. Because
the number of attention studies has grown like an avalanche after three main
studies presented in that year. The first one proposed a novel approach for
neural machine translation (NMT) bahdanau2015 . As it is known, most of the
NMT models belong to a family of encoder-decoders sutskever2014 ; cho2014a ,
with an encoder and a decoder for each language. However, compressing all the
necessary information of a source sentence into a fixed-length vector is an
important disadvantage of this encoder-decoder approach. This usually makes it
difficult for the neural network to capture all the semantic details of a very
long sentence goodfellow2016 .

Figure 1: The extension to the conventional NMT models that is proposed by
bahdanau2015 . It generates the _t_ -th target word $y_{t}$ given a source
sentence $(x_{1},x_{2},...,x_{T})$.

The idea that bahdanau2015  introduced is an extension to the conventional NMT
models. This extension is composed of an encoder and decoder as shown in Fig
1. The first part, encoder, is a bidirectional RNN (BiRNN) schuster1997  that
takes word vectors as input. The forward and backward states of BiRNN are
computed. Then, an _annotation_ $a_{j}$ for each word $x_{j}$ is obtained by
concatenating these forward and backward hidden states. Thus, the encoder maps
the input sentence to a sequence of annotations $(a_{1},...,a_{T_{x}})$. By
using a BiRNN rather than conventional RNN, the annotation of each word can
summarize both the preceding words and the following words. Besides, the
annotation $a_{j}$ can focus on the words around $x_{j}$ because of the
inherent nature of RNNs that representing recent inputs better.

In decoder, a weight $\alpha_{ij}$ of each annotation $a_{j}$ is obtained by
using its associated energy $e_{ij}$ that is computed by a feedforward neural
network _f_ as in Eq. (1). This neural network _f_ is defined as an _alignment
model_ that can be jointly trained with the proposed architecture. In order to
reduce computational burden, a multilayer perceptron (MLP) with a single
hidden layer is proposed as _f_. This alignment model tells us about the
relation between the inputs around position _j_ and the output at position
_i_. By this way, the decoder applies an attention mechanism. As it is seen in
Eq. (2), the $\alpha_{ij}$ is the output of softmax function:

$e_{ij}=f(h_{i-1},a_{j})$ (1)

$\alpha_{ij}=\frac{\exp(e_{ij})}{\sum_{k=1}^{T_{x}}\exp(e_{ik})}$ (2)

Here, the probability $\alpha_{ij}$ determines the importance of annotation
$a_{j}$ with respect to the previous hidden state $h_{i-1}$. Finally, the
context vector $c_{i}$ is computed as a weighted sum of these annotations as
follows bahdanau2015 :

$c_{i}=\sum_{j=1}^{T_{x}}\alpha_{ij}a_{j}$ (3)

Based on the decoder state, the context and the last generated word, the
target word $y_{t}$ is predicted. In order to generate a word in a
translation, the model searches for the most relevant information in the
source sentence to concentrate. When it finds the appropriate source
positions, it makes the prediction. By this way, the input sentence is encoded
into a sequence of vectors and a subset of these vectors is selected
adaptively by the decoder that is relevant to predicting the target
bahdanau2015 . Thus, it is no longer necessary to compress all the information
of a source sentence into a fixed-length vector.

The second study is the first visual attention model in image captioning
xu2015 . Different from the previous study bahdanau2015 , it uses a deep
convolutional neural network (CNN) as an encoder. This architecture is an
extension of the neural network vinyals2015  that encodes an image into a
compact representation, followed by an RNN that generates a corresponding
sentence. Here, the annotation vectors $a_{i}\in R^{D}$ are extracted from a
lower convolutional layer, each of which is a _D_ -dimensional representation
corresponding to a part of the image. Thus, the decoder selectively focuses on
certain parts of an image by weighting a subset of all the feature vectors
xu2015 . This extended architecture uses attention for salient features to
dynamically come to the forefront instead of compressing the entire image into
a static representation.

The context vector $c_{t}$ represents the relevant part of the input image at
time _t_. The weight $\alpha_{i}$ of each annotation vector is computed
similar to Eq. (2), whereas its associated energy is computed similar to Eq.
(1) by using an MLP conditioned on the previous hidden state $h_{t-1}$. The
remarkable point of this study is a new mechanism $\phi$ that computes $c_{t}$
from the annotation vectors $a_{i}$ corresponding to the features extracted at
different image locations:

$c_{t}=\phi(\big{\\{}a_{i}\big{\\}},\big{\\{}\alpha_{i}\big{\\}})$ (4)

The definition of the $\phi$ function causes two variants of attention
mechanisms: The _hard (stochastic)_ attention mechanism is trainable by
maximizing an approximate variational lower bound, i.e., by REINFORCE
williams1992 . On the other side, the _soft (deterministic)_ attention
mechanism is trainable by standard backpropagation methods. The hard attention
defines a location variable $s_{t}$, and uses it to decide where to focus
attention when generating the _t-th_ word. When the hard attention is applied,
the attention locations are considered as intermediate latent variables. It
assigns a multinoulli distribution parametrized by ${\alpha_{i}}$, and $c_{t}$
becomes a random variable. Here, $s_{t,i}$ is defined as a one-hot variable
which is set to 1 if the _i-th_ location is used to extract visual features
xu2015 :

$p(s_{t,i}=1|s_{j<t},a)=\alpha_{t,i}$ (5)

$c_{t}=\sum_{i}s_{t,i}a_{i}$ (6)

Whereas learning hard attention requires sampling the attention location
$s_{t}$ each time, the soft attention mechanism computes a weighted annotation
vector similar to bahdanau2015  and takes the expectation of the context
vector $c_{t}$ directly:

$E_{p(s_{t}|\alpha)}[c_{t}]=\sum_{i=1}^{L}\alpha_{t,i}a_{i}$ (7)

Furthermore, in training the deterministic version of the model, an
alternative method namely _doubly stochastic attention_ , is proposed with an
additional constraint added to the training objective to encourage the model
to pay equal attention to all parts of the image.

The third study should be emphasized presents two classes of attention
mechanisms for NMT: the _global_ attention that always attends to all source
words, and the _local_ attention that only looks at a subset of source words
at a time luong2015 . These mechanisms derive the context vector $c_{t}$ in
different ways: Whereas the global attention considers all the hidden states
of the encoder, the local one selectively focuses on a small window of
context. In global attention, a variable-length alignment vector is derived
similar to Eq. (2). Here, the current target hidden state $h_{t}$ is compared
with each source hidden state $\bar{h}_{s}$ by using a _score_ function
instead of the associated energy $e_{ij}$. Thus, the alignment vector whose
size equals the number of time steps on the source side is derived. Given the
alignment vector as weights, the context vector $c_{t}$ is computed as the
weighted average over all the source hidden states. Here, _score_ is referred
as a _content-based_ function, and three different alternatives are considered
luong2015 .

On the other side, the local attention is differentiable. Firstly, an aligned
position $p_{t}$ is generated for each target word at a time $t$. Then, a
window centered around the source position $p_{t}$ is used to compute the
context vector as a weighted average of the source hidden states within the
window. The local attention selectively focuses on a small window of context,
and obtains the alignment vector from the current target state $h_{t}$ and the
source states $\bar{h}_{s}$ in the window luong2015 .

The introduction of these novel mechanisms in 2015 triggered the rise of
attention for neural networks. Based on the proposed attention mechanisms,
significant research has been conducted in a variety of tasks. In order to
imagine the attention idea in neural networks better, two visual examples are
shown in Fig. 2. A neural image caption generation task is seen in the top row
that implements an attention mechanism xu2015 . Then, the second example shows
how the attention mechanisms can be used for visual question answering lu2017a
. Both examples demonstrate how attention mechanisms focus on parts of input
images.

Figure 2: Examples of the attention mechanism in visual. _(Top)_ Attending to
the correct object in neural image caption generation xu2015 . _(Bottom)_
Visualization of original image and question pairs, and co-attention maps
namely word-level, phrase-level and question-level, respectively lu2017a .

##  4 2015-2016: Attack of the Attention

During two years from 2015, the attention mechanisms were used for different
tasks, and novel neural network architectures were presented applying these
mechanisms. After the _memory networks_ weston2014  that require a supervision
signal instructing them how to use their memory cells, the introduction of the
_neural Turing machine_ graves2014  allows end-to-end training without this
supervision signal, via the use of a content-based soft attention mechanism
goodfellow2016 . Then, _end-to-end memory network_ sukhbaatar2015  that is a
form of _memory network_ based on a recurrent attention mechanism is proposed.

In these years, an attention mechanism called _self-attention_ , sometimes
called _intra-attention_ , was successfully implemented within a neural
network architecture namely _Long Short-Term Memory-Networks (LSTMN)_
cheng2016 . It modifies the standard LSTM structure by replacing the memory
cell with a memory network weston2014 . This is because memory networks have a
set of key vectors and a set of value vectors, whereas LSTMs maintain a hidden
vector and a memory vector cheng2016 . In contrast to attention idea in
bahdanau2015 , memory and attention are added _within_ a sequence encoder in
LSTMN. In order to compute a representation of a sequence, self-attention is
described as relating different positions of it vaswani2017 . One of the first
approaches of self-attention is applied for natural language inference
parikh2016 .

Many attention-based models have been proposed for neural image captioning
you2016 , abstractive sentence summarization rush2015 , speech recognition
yu2016 ; chorowski2015 , automatic video captioning zanfir2016 , neural
machine translation cheng2016a , and recognizing textual entailment
rocktaschel2016 . Different attention-based models perform visual question
answering zhu2016 ; chen2015 ; xu2016 . An attention-based CNN is presented
for modeling sentence pairs yin2016 . A recurrent soft attention based model
learns to focus selectively on parts of the video frames and classifies videos
sharma2016 .

On the other side, several neural network architectures have been presented in
a variety of tasks. For instance, _Stacked Attention Network (SAN)_ is
described for image question answering yang2016 . Deep Attention Recurrent
Q-Network (DARQN) integrates soft and hard attention mechanisms into the
structure of Deep Q-Network (DQN) sorokin2015 . Wake-Sleep Recurrent Attention
Model (WS-RAM) speeds up the training time for image classification and
caption generation tasks ba2015 . alignDRAW model, an extension of the _Deep
Recurrent Attention Writer (DRAW)_ gregor2015 , is a generative model of
images from captions using a soft attention mechanism mansimov2016 .
_Generative Adversarial What-Where Network (GAWWN)_ synthesizes images given
instructions describing what content to draw in which location reed2016 .

##  5 The Transformer: Return of the Attention

After the proposed attention mechanisms in 2015, researchers published studies
that mostly modifying or implementing them to different tasks. However, in
2017, a novel neural network architecture, namely the _Transformer_ , based
entirely on _self-attention_ was presented vaswani2017 . The Transformer
achieved great results on two machine translation tasks in addition to English
constituency parsing. The most impressive point about this architecture is
that it contains neither recurrence nor convolution. The Transformer performs
well by replacing the conventional recurrent layers in encoder-decoder
architecture used for NMT with self-attention.

The Transformer is composed of encoder-decoder stacks each of which has six
identical layers within itself. In Fig. 3, one encoder-decoder stack is shown
to illustrate the model vaswani2017 . Each stack includes only attention
mechanisms and feedforward neural networks. As this architecture does not
include any recurrent or convolutional layer, information about the relative
or absolute positions in the input sequence is given at the beginning of both
encoder and decoder using _positional encodings_.

Figure 3: The Transformer architecture and the attention mechanisms it uses in
detail vaswani2017 . _(Left)_ The Transformer with one encoder-decoder stack.
_(Center)_ Multi-head attention. _(Right)_ Scaled dot-product attention.

The calculations of self-attention are slightly different from the mechanisms
described so far in this paper. It uses three vectors namely _query_ , _key_
and _value_ for each word. These vectors are computed by multiplying the input
with weight matrices _$W_{q}$_ , _$W_{k}$_ and _$W_{v}$_ which are learned
during training. In general, each value is weighted by a function of the query
with the corresponding key. The output is computed as a weighted sum of the
values. Based on this idea, two attention mechanisms are proposed: In the
first one, called _scaled dot-product attention_ , the dot products of the
query with all keys are computed as given in the right side of Fig. 3. Each
result is divided to the square root of the dimension of the keys to have more
stable gradients. They pass into the softmax function, thus the weights for
the values are obtained. Finally each softmax score is multiplied with the
value as given in Eq. (8). The authors propose computing the attention on a
set of queries simultaneously by taking queries and keys of dimension $d_{k}$,
and values of dimension $d_{v}$ as inputs. The keys, queries and values are
packed together into matrices _K_ , _Q_ and _V_. Finally, the output matrix is
obtained as follows vaswani2017 :

$Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V$ (8)

This calculation is performed by every word against the other words. This
leads to having _values_ of each word relative to each other. For instance, if
the word $x_{2}$ is not relevant for the word $x_{1}$, then the softmax score
gives low probability scores. As a result, the corresponding value is
decreased. This leads to an increase in the value of relevant words, and those
of others decrease. In the end, every word obtains a new value for itself.

As seen from Fig. 3, the Transformer model does not directly use scaled dot-
product attention. But the attention mechanism it uses is based on these
calculations. The second mechanism proposed, called the _multi-head attention_
, linearly projects the queries, keys and values h times with different,
learned linear projections to $d_{q}$, $d_{k}$ and $d_{v}$ dimensions,
respectively vaswani2017 . The attention function is performed in parallel on
each of these projected versions of queries, keys and values, i.e., _heads_.
By this way, $d_{v}$-dimensional output values are obtained. In order to get
the final values, they are concatenated and projected one last time as shown
in the center of Fig. 3. By this way, the self-attention is calculated
multiple times using different sets of query, key and value vectors. Thus, the
model can jointly attend to information at different positions vaswani2017 :

$\displaystyle MultiHead(Q,K,V)=Concat(head_{1},...,head_{h})W^{O}$ (9)
$\displaystyle where\;head_{i}=Attention(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V})$

In the decoder part of the Transformer, _masked multi-head attention_ is
applied first to ensure that only previous word embeddings are used when
trying to predict the next word in the sentence. Therefore, the embeddings
that shouldn’t be seen by the decoder are masked by multiplying with zero.

An interesting study examines the contribution made by individual attention
heads in the encoder voita2019 . Also, there is an evaluation of the effects
of self-attention on gradient propagation in recurrent networks kerg2020 . For
a deeper analysis of multi-head self-attention mechanism from a theoretical
perspective see cordonnier2020 .

Self-attention has been used successfully in a variety of tasks including
sentence embedding lin2017  and abstractive summarization paulus2018 . It is
shown that self-attention can lead to improvements to discriminative
constituency parser kitaev2018 , and speech recognition as well povey2018 ;
vyas2020 . Also, the _listen-attend-spell_ model chan2016  has been improved
with the self-attention for acoustic modeling sperber2018 .

As soon as these self-attention mechanisms were proposed, they have been
incorporated with deep neural networks for a wide range of tasks. For
instance, a deep learning model learned a number of large-scale tasks from
multiple domains with the aid of self-attention mechanism kaiser2017 . Novel
self-attention neural models are proposed for cross-target stance
classification xu2018a  and NMT maruf2019 . Another study points out that a
fully self-attentional model can reach competitive predictive performance on
ImageNet classification and COCO object detection tasks ramachandran2019 .
Besides, developing novel attention mechanisms has been carried out such as
_area attention_ , a novel mechanism that can be used along multi-head
attention li2019 . It attends to areas in the memory by defining the key of an
area as the mean vector of the key of each item, and defining the value as the
sum of all value vectors in the area.

When a novel mechanism is proposed, it is inevitable to incorporate it into
the GAN framework goodfellow2014 . _Self-Attention Generative Adversarial
Networks (SAGANs)_ zhang2019  introduce a self-attention mechanism into
convolutional GANs. Different from the traditional convolutional GANs, SAGAN
generates high-resolution details using cues from all feature locations.
Similarly, _Attentional Generative Adversarial Network (AttnGAN)_ is presented
for text to image generation xu2018 . On the other side, a machine reading and
question answering architecture called QANet yu2018  is proposed without any
recurrent networks. It uses self-attention to learn the global interaction
between each pair of words whereas convolution captures the local structure of
the text. In another study, _Gated Attention Network (GaAN)_ controls the
importance of each attention head’s output by introducing gates zhang2018a .
Another interesting study introduces _attentive group convolutions_ with a
generalization of visual self-attention romero2020 . A deep transformer model
is implemented for language modeling over long sequences alrfou2019 .

Table 1: Summary of Notation Symbol | Definition
---|---
a | annotation
c | context vector
$\alpha$ | weight
e | energy
f | feedforward neural network
h | hidden state
$\phi$ | hard (stochastic) / soft (deterministic) attention
s | location variable
p | source position
$K$, $Q$, $V$ | keys, queries and values matrices, respectively
$W_{q}$, $W_{k}$, $W_{v}$ | weight matrices for queries, keys and values, respectively

###  5.1 Self-attention variants

In recent years, self-attention has become an important research direction
within the deep learning community. Self-attention idea has been examined in
different aspects. For example, self-attention is handled in a multi-instance
learning framework du2018 . The idea of _Sparse Adaptive Connection (SAC)_ is
presented for accelerating and structuring self-attention li2020a . The
research on improving self-attention continues as well yang2019 ; yang2018a ;
bello2019 . Besides, based on the self-attention mechanisms proposed in the
Transformer, important studies that modify the self-attention have been
presented. Some of the most recent and prominent studies are summarized below.

#### Relation-aware self-attention

It extends the self-attention mechanism by regarding representations of the
relative positions, or distances between sequence elements shaw2018 . Thus, it
can consider the pairwise relationships between input elements. This type of
attention mechanism defines vectors to represent the edge between two inputs.
It provides learning two distinct edge representations that can be shared
across attention heads without requiring additional linear transformations.

#### Directional self-attention (DiSA)

A novel neural network architecture for learning sentence embedding named
_Directional Self-Attention Network (DiSAN)_ shen2017  uses _directional_
self-attention followed by a _multi-dimensional_ attention mechanism. Instead
of computing a single importance score for each word based on the word
embedding, multi-dimensional attention computes a feature-wise score vector
for each token. To extend this mechanism to the self-attention, two variants
are presented: The first one, called _multi-dimensional ‘token2token’ self-
attention_ generates context-aware coding for each element. The second one,
called _multi-dimensional ‘source2token’ self-attention_ compresses the
sequence into a vector shen2017 . On the other side, directional self-
attention produces context-aware representations with temporal information
encoded by using positional masks. By this way, directional information is
encoded. First, the input sequence is transformed to a sequence of hidden
states by a fully connected layer. Then, multi-dimensional token2token self-
attention is applied to these hidden states. Hence, context-aware vector
representations are generated for all elements from the input sequence.

#### Reinforced self-attention (ReSA)

A sentence-encoding model named _Reinforced Self-Attention Network (ReSAN)_
uses _reinforced self-attention (ReSA)_ that integrates soft and hard
attention mechanisms into a single model. ReSA selects a subset of head
tokens, and relates each head token to a small subset of dependent tokens to
generate their context-aware representations shen2018 . For this purpose, a
novel hard attention mechanism called _reinforced sequence sampling (RSS)_ ,
which selects tokens from an input sequence in parallel and trained via policy
gradient, is proposed. Given an input sequence, RSS generates an equal-length
sequence of binary random variables that indicates both the selected and
discarded ones. On the other side, the soft attention provides reward signals
back for training the hard attention. The proposed RSS provides a sparse mask
to self-attention. ReSA uses two RSS modules to extract the sparse
dependencies between each pair of selected tokens.

#### Outer product attention (OPA)

_Self-Attentive Associative Memory (SAM)_ is a novel operator based upon
_outer product attention (OPA)_ le2020 . This attention mechanism is an
extension of dot-product attention vaswani2017 . OPA differs using element-
wise multiplication, outer product, and _tanh_ function instead of _softmax_.

#### Bidirectional block self-attention (Bi-BloSA)

Another mechanism, _bidirectional block self-attention (Bi-BloSA)_ which is
simply a _masked block self-attention (mBloSA)_ with forward and backward
masks to encode the temporal order information is presented shen2018a . Here,
mBloSA is composed of three parts from its bottom to top namely _intra-block
self-attention_ , _inter-block self-attention_ and _the context fusion_. It
splits a sequence into several length-equal blocks, and applies an intra-block
self-attention to each block independently. Then, inter-block self-attention
processes the outputs for all blocks. This stacked self-attention model
results a reduction in the amount of memory compared to a single one applied
to the whole sequence. Finally, a feature fusion gate combines the outputs of
intra-block and inter-block self-attention with the original input, to produce
the final context-aware representations of all tokens.

#### Fixed multi-head attention

The _fixed multi-head attention_ proposes fixing the head size of the
Transformer in the aim of improving the representation power bhojanapalli2020
. This study emphasizes its importance by setting the head size of attention
units to input sequence length.

#### Sparse sinkhorn attention

It is based on the idea of differentiable sorting of internal representations
_within_ the self-attention module tay2020 . Instead of allowing tokens to
only attend to tokens within the same block, it operates on block sorted
sequences. Each token attends to tokens in the sorted block. Thus, tokens that
may be far apart in the unsorted sequence can be considered. Additionally, a
variant of this mechanism named _SortCut sinkhorn attention_ applies a post-
sorting truncation of the input sequence.

#### Adaptive attention span

_Adaptive attention span_ is proposed as an alternative to self-attention
sukhbaatar2019 . It learns the attention span of each head independently. To
this end, a masking function inspired by jernite2017  is used to control the
attention span for each head. The purpose of this novel mechanism is to reduce
the computational burden of the Transformer. Additionally, _dynamic attention
span_ approach is presented to dynamically change the attention span based on
the current input as an extension luong2015 ; shu2017 .

###  5.2 Transformer variants

Different from developing novel self-attention mechanisms, several studies
have been published in the aim of improving the performance of the
Transformer. These studies mostly modify the model architecture. For instance,
an additional recurrence encoder is preferred to model recurrence for
Transformer directly hao2019 . In another study, a new weight initialization
scheme is applied to improve Transformer optimization huang2020 . A novel
positional encoding scheme is used to extend the Transformer to tree-
structured data shiv2019 . Investigating model size by handling Transformer
width and depth for efficient training is also an active research area li2020
. Transformer is used in reinforcement learning settings hoshen2017 ; hu2021 ;
parisotto2021  and for time series forecasting in adversarial training setting
wu2020a .

Besides, many Transformer variants have been presented in the recent past.
_COMmonsEnse Transformer (COMET)_ is introduced for automatic construction of
commonsense knowledge bases bosselut2019 . _Evolved Transformer_ applies
neural architecture search for a better Transformer model so2019 .
_Transformer Autoencoder_ is a sequential autoencoder for conditional music
generation choi2020 . _CrossTransformer_ takes a small number of labeled
images and an unlabeled query, and computes distances between spatially-
corresponding features to infer class membership doersch2020 . _DEtection
TRansformer (DETR)_ is a new design for object detection systems carion2020 ,
and _Deformable DETR_ is an improved version that achieves better performance
in less time zhu2021 . _FLOw-bAsed TransformER (FLOATER)_ emphasizes the
importance of position encoding in the Transformer, and models the position
information via a continuous dynamical model liu2020 . _Disentangled Context
(DisCo) Transformer_ simultaneously generates all tokens given different
contexts by predicting every word in a sentence conditioned on an arbitrary
subset of the rest of the words kasai2020 . _Generative Adversarial
Transformer (GANsformer)_ is presented for visual generative modeling
hudson2021 .

Recent work has demonstrated significant performance on NLP tasks. In _OpenAI
GPT_ , there is a left-to-right architecture, where every token can only
attend to previous tokens in the self-attention layers of the Transformer
radford2018 . _GPT-2_ radford2019  and _GPT-3_ brown2020  models have improved
the progress. In addition to these variants, some prominent Transformer-based
models are summarized below.

#### Universal Transformer

A generalization of the Transformer model named the _Universal Transformer_
dehghani2018  iteratively computes representations $H^{t}$ at step t for all
positions in the sequence in parallel. To this end, it uses the scaled dot-
product attention in Eq. (8) where d is the number of columns of Q, K and V.
In the Universal Transformer, the multi-head self-attention with k heads is
used. The representations $H^{t}$ is mapped to queries, keys and values with
affine projections using learned parameter matrices $W^{Q}\in\Re^{d\times
d/k}$, $W^{K}\in\Re^{d\times d/k}$, $W^{V}\in\Re^{d\times d/k}$ and
$W^{O}\in\Re^{d\times d}$ dehghani2018 :

$\displaystyle MultiHead(H^{t})=Concat(head_{1},...,head_{k})W^{O}$ (10)
$\displaystyle
where\;head_{i}=Attention(H^{t}W_{i}^{Q},H^{t}W_{i}^{K},H^{t}W_{i}^{V})$

#### Image Transformer

_Image Transformer_ parmar2018  demonstrates that self-attention based models
can also be well-suited for images instead of text. This Transformer type
restricts the self-attention mechanism to attend to local neighborhoods. Thus,
the size of images that the model can process is increased. Its larger
receptive fields allow the Image Transformer to significantly improve the
model performance on image generation as well as image super-resolution.

#### Transformer-XL

This study aims to improve the fixed-length context of the Transformer
vaswani2017  for language modeling. _Transformer-XL_ dai2019  makes modeling
very long-term dependency possible by reusing the hidden states obtained in
previous segments. Hence, information can be propagated through the recurrent
connections. In order to reuse the hidden states without causing temporal
confusion, Transformer-XL uses relative positional encodings. Based on this
architecture, a modified version named _the Gated Transformer-XL (GTrXL)_ is
presented in the reinforcement learning setting parisotto2020 .

#### Tensorized Transformer

_Tensorized Transformer_ ma2019  compresses the multi-head attention in
Transformer. To this end, it uses a novel self-attention model _multi-linear
attention_ with Block-Term Tensor Decomposition (BTD) lathauwer2008 . It
builds a _single-block attention_ based on the Tucker decomposition tucker1966
. Then, it uses a multi-linear attention constructed by a BTD to compress the
multi-head attention mechanism. In Tensorized Transformer, the factor matrices
are shared across multiple blocks.

#### BERT

The Bidirectional Encoder Representations from Transformers _(BERT)_ aims to
pre-train deep bidirectional representations from unlabeled text devlin2019 .
BERT uses a multilayer bidirectional Transformer as the encoder. Besides,
inspired by the Cloze task taylor1953 , it has a _masked language model_ pre-
training objective. BERT randomly masks some of the tokens from the input, and
predicts the original vocabulary id of the masked word based only on its
context. This model can pre-train a deep bidirectional Transformer. In all
layers, the pre-training is carried out by jointly conditioning on both left
and right context. BERT differs from the left-to-right language model pre-
training from this aspect.

Recently, BERT model has been examined in detail. For instance, the behaviour
of attention heads are analysed clark2019 . Various methods have been
investigated for compressing sun2019 ; wang2020a , pruning mccarley2020 , and
quantization zafrir2019 . Also, BERT model has been considered for different
tasks such as coreference resolution joshi2019b . A novel method is proposed
in order to accelerate BERT training gong2019 .

Furthermore, various BERT variants have been presented. _ALBERT_ aims to
increase the training speed of BERT, and presents two parameter reduction
techniques lan2020 . Similarly, _PoWER-BERT_ goyal2020  is developed to
improve the inference time of BERT. This scheme is also used to accelerate
ALBERT. Also, _TinyBERT_ is proposed to accelerate inference and reduce model
size while maintaining accuracy jiao2019 . In order to obtain better
representations, _SpanBERT_ is proposed as a pre-training method joshi2020 .
As a robustly optimized BERT approach, _RoBERTa_ shows that BERT was
significantly undertrained liu2019 . Also, _DeBERTa_ improves RoBERTa using
the disentangled attention mechanism he2021 . On the other side, _DistilBERT_
shows that it is possible to reach similar performances using much smaller
language models pre-trained with knowledge distillation sanh2019 .
_StructBERT_ proposes two novel linearization strategies wang2020 . _Q-BERT_
is introduced for quantizing BERT models shen2020 , _BioBERT_ is for
biomedical text mining lee2020 , and _RareBERT_ is for rare disease diagnosis
prakash2021 .

Since 2017 when the Transformer was presented, research directions have
generally focused on novel self-attention mechanisms, adapting the Transformer
for various tasks, or making them more understandable. In one of the most
recent studies, NLP becomes possible in the mobile setting with _Lite
Transformer_. It applies _long-short range attention_ where some heads
specialize in the local context modeling while the others specialize in the
long-distance relationship modeling wu2020 . A deep and light-weight
Transformer _DeLighT_ mehta2021  and a hypernetwork-based model namely
_HyperGrid Transformers_ tay2021  perform with fewer parameters. _Graph
Transformer Network_ is introduced for learning node representations on
heterogeneous graphs yun2019  and different applications are performed for
molecular data rong2020  or textual graph representation yang2021 . Also,
_Transformer-XH_ applies _eXtra Hop attention_ for structured text data
zhao2020 . _AttentionXML_ is a tree-based model for extreme multi-label text
classification you2019 . Besides, attention mechanism is handled in a Bayesian
framework fan2020 . For a better understanding of Transformers, an
identifiability analysis of self-attention weights is conducted in addition to
presenting _effective attention_ to improve explanatory interpretations
brunner2020 . Lastly, _Vision Transformer (ViT)_ processes an image using a
standard Transformer encoder as used in NLP by interpreting it as a sequence
of patches, and performs well on image classification tasks dosovitskiy2021 .

###  5.3 What about complexity?

All these aforementioned studies undoubtedly demonstrate significant success.
But success not make one great. The Transformer also brings a very high
computational complexity and memory cost. The necessity of storing attention
matrix to compute the gradients with respect to queries, keys and values
causes a non-negligible quadratic computation and memory requirements.
Training the Transformer is a slow process for very long sequences because of
its quadratic complexity. There is also time complexity which is quadratic
with respect to the sequence length. In order to improve the Transformer in
this respect, recent studies have been conducted to improve this issue. One of
them is _Linear Transformer_ which expresses the self-attention as a linear
dot-product of kernel feature maps katharopoulos2020 . Linear Transformer
reduces both memory and time complexity by changing the self-attention from
the softmax function in Eq. (8) to a feature map based dot-product attention.
Its performance is competitive with the vanilla Transformer architecture on
image generation and automatic speech recognition tasks while being faster
during inference. On the other side, _FMMformers_ which use the idea of the
_fast multipole method (FMM)_ leslie1987  outperform the linear Transformer by
decomposing the attention matrix into near-field and far-field attention with
linear time and memory complexity nguyen2021 .

Another suggestion made in response to the Transformer’s quadratic nature is
The _Reformer_ that replaces dot-product attention by one that uses _locality-
sensitive hashing_ kitaev2020 . It reduces the complexity but one limitation
of the Reformer is its requirement for the queries and keys to be identical.
_Set Transformer_ aims to reduce computation time of self-attention from
quadratic to linear by using an attention mechanism based on sparse Gaussian
process literature lee2019 . _Routing Transformer_ aims to reduce the overall
complexity of attention by learning dynamic sparse attention patterns by using
_routing attention with clustering_ roy2020 . It applies k-means clustering to
model sparse attention matrices. At first, queries and keys are assigned to
clusters. The attention scheme is determined by considering only queries and
keys from the same cluster. Thus, queries are routed to keys belonging to the
same cluster roy2020 .

_Sparse Transformer_ introduces sparse factorizations of the attention matrix
by using _factorized self-attention_ , and avoids the quadratic growth of
computational burden child2019 . It also shows the possibility of modeling
sequences of length one million or more by using self-attention in theory. In
the Transformer, all the attention heads with the softmax attention assign a
non-zero weight to all context words. _Adaptively Sparse Transformer_ replaces
softmax with $\alpha$-entmax which is a differentiable generalization of
softmax allowing low-scoring words to receive precisely zero weight
correia2019 . By means of context-dependent sparsity patterns, the attention
heads become flexible in the Adaptively Sparse Transformer. _Random feature
attention_ approximates softmax attention with random feature methods peng2021
. _Skyformer_ replaces softmax with a Gaussian kernel and adapts Nyström
method chen2021 . A sparse attention mechanism named _BIGBIRD_ aims to reduce
the quadratic dependency of Transformer-based models to linear zaheer2020 .
Different from the similar studies, BIGBIRD performs well for genomics data
alongside NLP tasks such as question answering.

_Music Transformer_ huang2019  shows that self-attention can also be useful
for modeling music. This study emphasizes the infeasibility of the relative
position representations introduced by shaw2018  for long sequences because of
the quadratic intermediate relative information in the sequence length.
Therefore, this study presents an extended version of relative attention named
_relative local attention_ that improves the relative attention for longer
musical compositions by reducing its intermediate memory requirement to linear
in the sequence length. A softmax-free Transformer (_SOFT_) is presented to
improve the computational efficiency of ViT. It uses Gaussian kernel function
instead of the dot-product similarity lu2021 .

Additionally, various approaches have been presented in _Hierarchical Visual
Transformer_ pan2021 , _Long-Short Transformer (Transformer-LS)_ zhu2021a ,
_Perceiver_ jaegle2021 , and _Performer_ choromanski2021 . Image Transformer
based on the cross-covariance matrix between keys and queries is applied
elnouby2021 , and a new vision Transformer is proposed yu2021 . Furthermore, a
Bernoulli sampling attention mechanism decreases the quadratic complexity to
linear zeng2021 . A novel linearized attention mechanism performs well on
object detection, instance segmentation, and stereo depth estimation shen2021
. A study shows that kernelized attention with relative positional encoding
can be calculated using Fast Fourier Transform and it leads to get rid of the
quadratic complexity for long sequences luo2021 . A linear unified nested
attention mechanism namely _Luna_ uses two nested attention functions to
approximate the softmax attention in Transformer to achieve linear time and
space complexity ma2021 .

##  6 Concluding Remarks: A New Hope

Inspired by the human visual system, the attention mechanisms in neural
networks have been developing for a long time. In this study, we examine this
duration beginning with its roots up to the present time. Some mechanisms have
been modified, or novel mechanisms have emerged in this period. Today, this
journey has reached a very important stage. The idea of incorporating
attention mechanisms into deep neural networks has led to state-of-the-art
results for a large variety of tasks. Self-attention mechanisms and _GPT-n_
family models have become a new hope for more advanced models. These promising
progress bring the questions whether the attention could help further
development, replace the popular neural network layers, or could be a better
idea than the existing attention mechanisms? It is still an active research
area and much to learn we still have, but it is obvious that more powerful
systems are awaiting when neural networks and attention mechanisms join
forces.

## Conflict of interest

The author declares that she has no conflict of interest.

## References

  * (1) I. Goodfellow, Y. Bengio, A. Courville, The MIT Press (2016)
  * (2) D. Noton, L. Stark, Scientific American 224(6), 34 (1971)
  * (3) D. Noton, L. Stark, Vision Research 11, 929 (1971)
  * (4) E. Alpaydın, Advances in Neural Information Processing Systems 8 pp. 771–777 (1995)
  * (5) S. Ahmad, Advances in Neural Information Processing Systems 4 pp. 420–427 (1991)
  * (6) M. Posner, S. Petersen, Annual Review of Neuroscience 13(1), 25 (1990)
  * (7) C. Bundesen, Psychological Review 97(4), 523 (1990)
  * (8) R. Desimone, J. Duncan, Annual Review of Neuroscience 18(1), 193 (1995)
  * (9) M. Corbetta, G. Shulman, Nature Reviews Neuroscience 3(3), 201 (2002)
  * (10) S. Petersen, M. Posner, Annual Review of Neuroscience 35, 73 (2012)
  * (11) R. Rimey, C. Brown, Technical Report, University of Rochester (1990)
  * (12) B. Sheliga, L. Riggio, G. Rizzolatti, Experimental Brain Research 98(3), 507 (1994)
  * (13) B. Sheliga, L. Riggio, G. Rizzolatti, Experimental Brain Research 105(2), 261 (1995)
  * (14) J. Hoffman, B. Subramaniam, Perception and Psychophysics 57(6), 787 (1995)
  * (15) S. Chaudhari, et al., ACM Transactions on Intelligent Systems and Technology (TIST) pp. 1–32 (2021)
  * (16) A. Galassi, et al., IEEE Transactions on Neural Networks and Learning Systems (2020)
  * (17) J. Lee, et al., ACM Transactions on Knowledge Discovery from Data (TKDD) 13(6), 1 (2019)
  * (18) T. Brown, et al., Advances in Neural Information Processing Systems 33 pp. 1877–1901 (2020)
  * (19) A. Vaswani, et al., Advances in Neural Information Processing Systems 30 pp. 5998–6008 (2017)
  * (20) K. Fukushima, Biological Cybernetics 36, 193 (1980)
  * (21) K. Fukushima, Applied Optics 26(23), 4985 (1987)
  * (22) K. Fukushima, T. Imagawa, Neural Networks 6(1), 33 (1993)
  * (23) E. Postma, H.V. den Herik, P. Hudson, Neural Networks 10(6), 993 (1997)
  * (24) J. Schmidhuber, R. Huber, International Journal of Neural Systems pp. 125–134 (1991)
  * (25) R. Milanese, et al., IEEE Computer Society Conference on Computer Vision and Pattern Recoginition, Seattle, WA, USA pp. 781–785 (1994)
  * (26) J. Tsotsos, et al., Artificial Intelligence 78(1-2), 507 (1995)
  * (27) S. Culhane, J. Tsotsos, Proceedings of the 11th IAPR International Conference on Pattern Recognition, The Hague, Netherlands pp. 36–40 (1992)
  * (28) D. Reisfeld, H. Wolfson, Y. Yeshurun, International Journal of Computer Vision 14(2), 119 (1995)
  * (29) I. Rybak, et al., Vision Research 38(15-16), 2387 (1998)
  * (30) J. Keller, et al., Pattern Analysis and Applications 2(3) (1999)
  * (31) F. Miau, L. Itti, Proceedings of the 23rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Istanbul, Turkey pp. 789–792 (2001)
  * (32) W. Zhang, et al., Advances in Neural Information Processing Systems 19 pp. 1609–1616 (2006)
  * (33) A. Salah, E. Alpaydın, L. Akarun, IEEE Transactions on Pattern Analysis and Machine Intelligence 24(3), 420 (2002)
  * (34) D. Walther, et al., International Workshop on Biologically Motivated Computer Vision, Springer, Berlin, Heidelberg pp. 472–479 (2002)
  * (35) K. Schill, et al., Journal of Electronic Imaging 10(1), 152 (2001)
  * (36) L. Paletta, G. Fritz, C. Seifert, International Conference on Machine Learning (2005)
  * (37) O.L. Meur, et al., IEEE Transactions on Pattern Analysis and Machine Intelligence 28(5), 802– (2006)
  * (38) S. Gould, et al., International Joint Conference on Artificial Intelligence (IJCAI) pp. 2115–2121 (2007)
  * (39) H. Larochelle, G. Hinton, Advances in Neural Information Processing Systems 23 pp. 1243–1251 (2010)
  * (40) L. Bazzani, et al., International Conference on Machine Learning (2011)
  * (41) V. Mnih, et al., Advances in Neural Information Processing Systems 27 pp. 2204–2212 (2014)
  * (42) M. Stollenga, et al., Advances in Neural Information Processing Systems 27 pp. 3545–3553 (2014)
  * (43) Y. Tang, N. Srivastava, R. Salakhutdinov, Advances in Neural Information Processing Systems 27 (2014)
  * (44) D. Bahdanau, K. Cho, Y. Bengio, International Conference on Learning Representations (2015)
  * (45) I. Sutskever, O. Vinyals, Q. Le, Advances in Neural Information Processing Systems 27 pp. 3104–3112 (2014)
  * (46) K. Cho, et al., Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) pp. 1724–1734 (2014)
  * (47) M. Schuster, K. Paliwal, IEEE Transactions on Signal Processing 45(11), 2673 (1997)
  * (48) K. Xu, et al., International Conference on Machine Learning pp. 2048–2057 (2015)
  * (49) O. Vinyals, et al., In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition pp. 3156–3164 (2015)
  * (50) R. Williams, Machine Learning 8(3-4), 229 (1992)
  * (51) M.T. Luong, H.P..C. Manning, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal pp. 1412–1421 (2015)
  * (52) J. Lu, et al., Advances in Neural Information Processing Systems 29 (2016)
  * (53) J. Weston, S. Chopra, A. Bordes, International Conference on Learning Representations (2014)
  * (54) A. Graves, G. Wayne, I. Danihelka, arXiv preprint arXiv:1410.5401 (2014)
  * (55) S. Sukhbaatar, et al., Advances in Neural Information Processing Systems 28 pp. 2440–2448 (2015)
  * (56) J. Cheng, L. Dong, M. Lapata, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing pp. 551–561 (2016)
  * (57) A. Parikh, et al., Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas pp. 2249–2255 (2016)
  * (58) Q. You, et al., In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV pp. 4651–4659 (2016)
  * (59) A. Rush, S. Chopra, J. Weston, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal pp. 379–389 (2015)
  * (60) D. Yu, et al., Interspeech pp. 17–21 (2016)
  * (61) J. Chorowski, et al., Advances in Neural Information Processing Systems 28 pp. 577–585 (2015)
  * (62) M. Zanfir, E. Marinoiu, C. Sminchisescu, In Asian Conference on Computer Vision, Springer, Cham pp. 104––119 (2016)
  * (63) Y. Cheng, et al., Proceedings of the 25th International Joint Conference on Artificial Intelligence (2016)
  * (64) T. Rockt International Conference on Learning Representations (2016)
  * (65) Y. Zhu, et al., Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition pp. 4995–5004 (2016)
  * (66) K. Chen, et al., arXiv preprint arXiv:1511.05960 (2015)
  * (67) H. Xu, K. Saenko, In European Conference on Computer Vision pp. 451–466 (2016)
  * (68) W. Yin, et al., Transactions of the Association for Computational Linguistics 4, 259 (2016)
  * (69) S. Sharma, R. Kiros, R. Salakhutdinov, International Conference on Learning Representations (2016)
  * (70) Z. Yang, et al., In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition pp. 21–29 (2016)
  * (71) I. Sorokin, et al., arXiv preprint arXiv:1512.01693 (2015)
  * (72) J. Ba, et al., Advances in Neural Information Processing Systems 28 pp. 2593–2601 (2015)
  * (73) K. Gregor, et al., International Conference on Machine Learning pp. 1462–1471 (2015)
  * (74) E. Mansimov, et al., International Conference on Learning Representations (2016)
  * (75) S. Reed, et al., Advances in Neural Information Processing Systems 29 pp. 217–225 (2016)
  * (76) E. Voita, et al., In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy pp. 5797–5808 (2019)
  * (77) G. Kerg, et al., Advances in Neural Information Processing Systems 33 (2020)
  * (78) J.B. Cordonnier, A. Loukas, M. Jaggi, International Conference on Learning Representations (2020)
  * (79) Z. Lin, et al., International Conference on Learning Representations (2017)
  * (80) R. Paulus, C. Xiong, R. Socher, International Conference on Learning Representations (2018)
  * (81) N. Kitaev, D. Klein, In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Long papers) pp. 2676–2686 (2018)
  * (82) D. Povey, et al., IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE pp. 5874–5878 (2018)
  * (83) A. Vyas, et al., Advances in Neural Information Processing Systems 33 (2020)
  * (84) W. Chan, et al., IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai pp. 4960––4964 (2016)
  * (85) M. Sperber, et al., In proceedings of Annual Conference of the International Speech Communication Association (InterSpeech) pp. 3723–3727 (2018)
  * (86) L. Kaiser, et al., arXiv preprint arXiv:1706.05137 (2017)
  * (87) C. Xu, et al., Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Short papers), Melbourne, Australia pp. 778–783 (2018)
  * (88) S. Maruf, A. Martins, G. Haffari, Proceedings of NAACL-HLT, Minneapolis, Minnesota pp. 3092–3102 (2019)
  * (89) P. Ramachandran, et al., Advances in Neural Information Processing Systems 32 pp. 68–80 (2019)
  * (90) Y. Li, et al., International Conference on Machine Learning (2019)
  * (91) I. Goodfellow, et al., Advances in Neural Information Processing Systems 27 pp. 2672–2680 (2014)
  * (92) H. Zhang, et al., International Conference on Machine Learning pp. 7354–7363 (2019)
  * (93) T. Xu, et al., In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 1316–1324 (2018)
  * (94) A. Yu, et al., International Conference on Learning Representations (2018)
  * (95) J. Zhang, et al., Conference on Uncertainty in Artificial Intelligence (2018)
  * (96) D. Romero, et al., International Conference on Machine Learning (2020)
  * (97) R. Al-Rfou, et al., AAAI Conference on Artificial Intelligence 33, 3159 (2019)
  * (98) J. Du, et al., Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing pp. 2216–2225 (2018)
  * (99) X. Li, et al., Advances in Neural Information Processing Systems 33 (2020)
  * (100) B. Yang, et al., AAAI Conference on Artificial Intelligence 33, 387 (2019)
  * (101) B. Yang, et al., Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium pp. 4449–4458 (2018)
  * (102) Proceedings of the IEEE International Conference on Computer Vision pp. 3286–3295
  * (103) P. Shaw, J. Uszkoreit, A. Vaswani, Proceedings of NAACL-HLT, New Orleans, Louisiana pp. 464–468 (2018)
  * (104) T. Shen, et al., AAAI Conference on Artificial Intelligence pp. 5446–5455 (2018)
  * (105) T. Shen, et al., In Proceedings of the 27th International Joint Conference on Artificial Intelligence, (IJCAI-18) pp. 4345–4352 (2018)
  * (106) H. Le, T. Tran, S. Venkatesh, International Conference on Machine Learning (2020)
  * (107) T. Shen, et al., International Conference on Learning Representations (2018)
  * (108) S. Bhojanapalli, et al., International Conference on Machine Learning (2020)
  * (109) Y. Tay, et al., International Conference on Machine Learning (2020)
  * (110) S. Sukhbaatar, et al., Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy pp. 331–335 (2019)
  * (111) Y. Jernite, et al., International Conference on Learning Representations (2017)
  * (112) R. Shu, H. Nakayama, In Proceedings of the First Workshop on Neural Machine Translation, Vancouver, Canada pp. 1–10 (2017)
  * (113) J. Hao, et al., Proceedings of NAACL-HLT, Minneapolis, Minnesota pp. 1198–1207 (2019)
  * (114) X. Huang, et al., International Conference on Machine Learning (2020)
  * (115) V. Shiv, C. Quirk, Advances in Neural Information Processing Systems 32 pp. 12,081–12,091 (2019)
  * (116) Z. Li, et al., International Conference on Machine Learning (2020)
  * (117) Y. Hoshen, Advances in Neural Information Processing Systems 30, Long Beach, CA, USA (2017)
  * (118) S. Hu, et al., International Conference on Learning Representations (2021)
  * (119) E. Parisotto, R. Salakhutdinov, International Conference on Learning Representations (2021)
  * (120) S. Wu, et al., Advances in Neural Information Processing Systems 33 (2020)
  * (121) A. Bosselut, et al., Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (2019)
  * (122) D. So, C. Liang, Q. Le, International Conference on Machine Learning (2019)
  * (123) K. Choi, et al., International Conference on Machine Learning (2020)
  * (124) C. Doersch, A. Gupta, A. Zisserman, Advances in Neural Information Processing Systems 33 pp. 21,981–21,993 (2020)
  * (125) N. Carion, et al., European Conference on Computer Vision pp. 213––229 (2020)
  * (126) X. Zhu, et al., International Conference on Learning Representations (2021)
  * (127) X. Liu, et al., International Conference on Machine Learning pp. 6327–6335 (2020)
  * (128) J. Kasai, et al., International Conference on Machine Learning (2020)
  * (129) D. Hudson, L. Zitnick, International Conference on Machine Learning pp. 4487–4499 (2021)
  * (130) A. Radford, et al., Technical Report, OpenAI (2018)
  * (131) A. Radford, et al., OpenAI blog p. 9 (2019)
  * (132) M. Dehghani, et al., International Conference on Learning Representations (2019)
  * (133) N. Parmar, International Conference on Machine Learning (2018)
  * (134) Z. Dai, et al., Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics pp. 2978–2988 (2019)
  * (135) E. Parisotto, International Conference on Machine Learning (2020)
  * (136) X. Ma, et al., Advances in Neural Information Processing Systems 32 pp. 2232–2242 (2019)
  * (137) L. Lathauwer, SIAM Journal on Matrix Analysis and Applications 30(3), 1033 (2008)
  * (138) L. Tucker, Psychometrika 31(3), 279 (1966)
  * (139) J. Devlin, et al., Proceedings of NAACL-HLT 2019 pp. 4171–4186 (2019)
  * (140) W. Taylor, Journalism Bulletin 30(4), 415 (1953)
  * (141) K. Clark, et al., arXiv preprint arXiv:1906.04341 (2019)
  * (142) S. Sun, et al., Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Hong Kong, China pp. 4323–4332 (2019)
  * (143) W. Wang, et al., Advances in Neural Information Processing Systems 33 (2020)
  * (144) J. McCarley, R. Chakravarti, A. Sil, arXiv preprint arXiv:1910.06360 (2020)
  * (145) O. Zafrir, et al., The 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS (2019)
  * (146) M. Joshi, et al., In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing pp. 5803–5808 (2019)
  * (147) L. Gong, et al., International Conference on Machine Learning pp. 2337–2346 (2019)
  * (148) Z. Lan, et al., International Conference on Learning Representations (2020)
  * (149) S. Goyal, et al., International Conference on Machine Learning (2020)
  * (150) X. Jiao, et al., arXiv preprint arXiv:1909.10351 (2019)
  * (151) M. Joshi, et al., Transactions of the Association for Computational Linguistics 8, 64 (2020)
  * (152) Y. Liu, et al., arXiv preprint arXiv:1907.11692 (2019)
  * (153) P. He, et al., International Conference on Learning Representations (2021)
  * (154) V. Sanh, et al., the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS (2019)
  * (155) W. Wang, et al., International Conference on Learning Representations (2020)
  * (156) S. Shen, et al., AAAI Conference on Artificial Intelligence 34, 8815 (2020)
  * (157) J. Lee, et al., Bioinformatics 36(4), 1234 (2020)
  * (158) P. Prakash, et al., AAAI Conference on Artificial Intelligence 35, 453 (2021)
  * (159) Z. Wu, et al., International Conference on Learning Representations (2020)
  * (160) S. Mehta, et al., International Conference on Learning Representations (2021)
  * (161) Y. Tay, et al., International Conference on Learning Representations (2021)
  * (162) S. Yun, et al., International Conference on Learning Representations (2018)
  * (163) Y. Rong, et al., Advances in Neural Information Processing Systems 33 (2020)
  * (164) J. Yang, et al., Advances in Neural Information Processing Systems 34 (2021)
  * (165) C. Zhao, et al., International Conference on Learning Representations (2020)
  * (166) R. You, et al., Advances in Neural Information Processing Systems 32 (2019)
  * (167) X. Fan, et al., Advances in Neural Information Processing Systems 33 (2020)
  * (168) G. Brunner, et al., International Conference on Learning Representations (2020)
  * (169) A. Dosovitskiy, et al., International Conference on Learning Representations (2021)
  * (170) A. Katharopoulos, et al., International Conference on Machine Learning (2020)
  * (171) L. Greengard, V. Rokhlin, Journal of Computational Physics 73(2), 325– (1987)
  * (172) T. Nguyen, et al., Advances in Neural Information Processing Systems 34 (2021)
  * (173) N. Kitaev, L. Kaiser, A. Levskaya, International Conference on Learning Representations (2020)
  * (174) J. Lee, et al., International Conference on Machine Learning pp. 3744–3753 (2019)
  * (175) A. Roy, et al., Transactions of the Association for Computational Linguistics pp. 53–68 (2020)
  * (176) R. Child, et al., arXiv preprint arXiv:1904.10509 (2019)
  * (177) G. Correia, V. Niculae, A. Martins, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing pp. 2174–2184 (2019)
  * (178) H. Peng, et al., International Conference on Learning Representations (2021)
  * (179) Y. Chen, et al., Advances in Neural Information Processing Systems 34 (2021)
  * (180) M. Zaheer, et al., Advances in Neural Information Processing Systems 33 (2020)
  * (181) C.Z. Huang, et al., International Conference on Learning Representations (2019)
  * (182) J. Lu, et al., Advances in Neural Information Processing Systems 34 (2021)
  * (183) Z. Pan, et al., Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 377–386 (2021)
  * (184) C. Zhu, et al., Advances in Neural Information Processing Systems 34 (2021)
  * (185) A. Jaegle, et al., International Conference on Machine Learning pp. 4651–4664 (2021)
  * (186) K. Choromanski, et al., International Conference on Learning Representations (2021)
  * (187) A. El-Nouby, et al., Advances in Neural Information Processing Systems 34 (2021)
  * (188) Q. Yu, et al., Advances in Neural Information Processing Systems 34 (2021)
  * (189) Z. Zeng, et al., International Conference on Machine Learning pp. 12,321–12,332 (2021)
  * (190) Z. Shen, et al., Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision pp. 3531–3539 (2021)
  * (191) S. Luo, et al., Advances in Neural Information Processing Systems 34 (2021)
  * (192) X. Ma, et al., Advances in Neural Information Processing Systems 34 (2021)