# Gait Recognition Based on Deep Learning: A Survey

Claudio Filipi Gonçalves dos Santos<EMAIL_ADDRESS>Federal Institute of
São Carlos - UFSCarRod. Washington Luiz, 235São CarlosSão PauloBrazil Eldorado
Research InstituteAv. Alan Turing, 275CampinasSão PauloBrazil ,  Diego de
Souza Oliveira<EMAIL_ADDRESS>,  Leandro A. Passos
<EMAIL_ADDRESS>0000-0003-3529-3109 ,  Rafael Gonçalves Pires
<EMAIL_ADDRESS>0000-0001-9597-055X ,  Daniel Felipe Silva Santos
<EMAIL_ADDRESS>,  Lucas Pascotti Valem<EMAIL_ADDRESS>0000-0002-3833-9072 ,  Thierry P. Moreira<EMAIL_ADDRESS>0000-0002-3410-6247 ,  Marcos Cleison S. Santana<EMAIL_ADDRESS>0000-0003-2568-8019 ,  Mateus Roder<EMAIL_ADDRESS>0000-0002-3112-5290
,  João Paulo Papa<EMAIL_ADDRESS>0000-0002-6494-7514 São Paulo State
University - UNESPAv. Eng. Luís Edmundo Carrijo Coube, 14-01BauruSão
PauloBrazil and  Danilo Colombo<EMAIL_ADDRESS>Cenpes,
Petroleo Brasileiro S.A. - PetrobrasRio de JaneiroRio de JaneiroBrazil

###### Abstract.

In general, biometry-based control systems may not rely on individual expected
behavior or cooperation to operate appropriately. Instead, such systems should
be aware of malicious procedures for unauthorized access attempts. Some works
available in the literature suggest addressing the problem through gait
recognition approaches. Such methods aim at identifying human beings through
intrinsic perceptible features, despite dressed clothes or accessories.
Although the issue denotes a relatively long-time challenge, most of the
techniques developed to handle the problem present several drawbacks related
to feature extraction and low classification rates, among other issues.
However, deep learning-based approaches recently emerged as a robust set of
tools to deal with virtually any image and computer-vision related problem,
providing paramount results for gait recognition as well. Therefore, this work
provides a surveyed compilation of recent works regarding biometric detection
through gait recognition with a focus on deep learning approaches, emphasizing
their benefits, and exposing their weaknesses. Besides, it also presents
categorized and characterized descriptions of the datasets, approaches, and
architectures employed to tackle associated constraints.

Gait Recognition, Biometrics, Deep Learning

††copyright: acmcopyright††journal: CSUR††journalyear: 2021††journalvolume:
1††journalnumber: 1††article: 1††publicationmonth: 1††price: 15.00††doi:
10.1145/3490235††journal: JACM††journalvolume: 37††journalnumber: 4††article:
111††publicationmonth: 8††ccs: Computing methodologies Machine learning††ccs:
Security and privacy Biometrics

##  1\. Introduction

Gait recognition emerged in the last decades as a branch of biometric
identification that focuses on detecting individuals through personal
measurements and relationships, e.g., trunk and limbs’ size, as well as space-
time information related to the intrinsic patterns in individuals movements
(Bolle et al., 2013). Such an approach presented itself extremely useful in
the contexts of surveillance systems or hazy environments monitoring, for
instance, where unique elements usually employed for biometric identification
such as the fingerprint and the face are hard or impossible to distinguish.

Besides, gait recognition approaches possess some advantages regarding other
biometric identification models, since hacking such architectures poses a hard
task (Nixon and Carter, 2004). The difficulty primary lies in the concept’s
intrinsic characteristics, i.e., identification based on the silhouette and
its movement, whose reproduction is particularly complicated. The same is not
valid for other techniques, where the individual can trick the system by
hiding the face, for instance. Further, gait recognition models do not require
high-resolution images and specialized equipment for proper identification,
like iris and fingerprints, for instance. Furthermore, gait recognition
methods are independent of individual cooperation, while other methods require
the analyzed person collaborates with the identification system.

Moreover, gait information is easily collected from distance, which stands for
an enormous advantage regarding other techniques, especially when the
identification is not assisted by the analyzed person, e.g., criminal
investigations. Besides, since it does not require sophisticated equipment for
data extraction, these methods are commonly cheaper than other approaches,
majorly due to the popularization of surveillance systems and the advent of
cellphones equipped with accelerometers, which transformed the burden of
extracting data signals into a straightforward task.

Despite the simplicity regarding the tools mentioned above, identifying people
by walking and moving is far from a trivial task. Standard gait recognition
methods, i.e., which comprise data pre-processing and features extracted in a
handcrafted fashion for further recognition, often suffer from several
constraints and challenges imposed by the complexity of the task, such as
viewing angle and large intra-class variations, occlusions, shadows, and
locating the body segments, among others. A new trend on machine learning,
known as deep learning, emerged in the last years as a revolutionary tool to
handle topics in image and sound processing, computer vision, and speech,
overwhelmingly outperforming virtually any baseline established until then.
The new paradigm exempts the necessity of manually extract representative
features from experts and also provides paramount results regarding gait
recognition, surpassing existing challenges and opening room for further
research.

Due to the increasing number of biometric and gait recognition works developed
in the last years, many authors provided studies summarizing each field’s main
achievements. Pisani et al. (Pisani et al., 2019), Khan et al. (Khan et al.,
2020), and Sundararajan et al. (Sundararajan et al., 2019), for instance,
exposed the main advances in the last years regarding biometrics
identification in general, while Gui et al. (Gui et al., 2019) explored the
novelties regarding brain biometrics methods. Regarding gait recognition, both
Wan et al. (Wan et al., 2018), and Rida et al. (Rida et al., 2018) provided a
comprehensive survey comprising general topics on the context in 2018.
Meanwhile, Singh et al. (Singh et al., 2018) published a study on vision-based
gait recognition, while Bouchrika (Bouchrika, 2018) proposed a similar work
considering smart visual surveillance. Further in 2019, Nambiar et al.
(Nambiar et al., 2019) summarized the works regarding gait-based person re-
identification, while Marsico and Mecca (Marsico and Mecca, 2019) focused on
gait recognition via wearable sensors. However, none of them focus on
presenting the main advances regarding deep learning-based approaches for gait
recognition. Moreover, even though there are reviews in this context (Alharthi
et al., 2019), most of the surveyed works comprise Convolutional Neural
Networks (CNN) (LeCun et al., 1998) and Long short-term memory (LSTM)
(Hochreiter and Schmidhuber, 1997; Lobo et al., 2020), leaving behind some
relevant and novel architectures, such as Autoencoders (Vincent et al., 2010),
Capsule Networks (CapsNet) (Sabour et al., 2017), Deep Belief Networks (Hinton
et al., 2006), and Generative Adversarial Networks (GANs) (Goodfellow et al.,
2014). Therefore, the main objectives of this work are three-fold:

  * •

to systematically introduce the most recent and significant works comprising
strategies for gait recognition through deep learning approaches; and

  * •

to provide the reader with a substantial and illustrated theoretical
background regarding gait recognition, exploring its roots on biometric
recognition and exposing the most popular tools employed to gait feature
extraction and the architectures used to tackle the associated constraints;
and

  * •

to present an illustrated, categorized, and characterized catalog of the
public datasets available for the task of gait recognition.

Regarding the selection of the reviewed papers, the keywords employed in the
search were “gait recognition using _deep learning technique_ ”, such that
_deep learning technique_ was replaced by the architectures themselves, e.g.,
convolutional neural networks, LSTM, and so on. Another filter considered was
the work’s innovation, i.e., papers with similar techniques and architectures
were discarded. Finally, the studies considered in this research were
published five years ago at most.

The remainder of this work is presented as follows. Section 2 presents a
theoretical background regarding the concepts and available methods for
biometric detection, bestows an overview regarding the deep learning
techniques employed in this work, and revisits the main concepts regarding
gait recognition. Further, Section 3 introduces the most recent approaches
using deep learning for gait recognition. Besides, the section also provides a
brief discussion regarding the surveyed works. Datasets employed for the task
are presented in Section 4. Finally, Section 5 presents the conclusions and
future works.

##  2\. Theoretical Background

This section presents a theoretical background introducing the problem of
biometric identification, describing the deep learning approaches employed for
gait recognition, and a detailed introduction regarding the problem of gait
recognition.

###  2.1. Biometric Identification

The problem of people identification poses a challenging task for humankind
since long before the existence of computers, when specialists were
responsible for analyzing and comparing documents, signatures, and other
features in a handcrafted fashion to present some restricted information or to
allow some banking transaction, for instance. The importance of an accurate
identification was intensified insofar as the society informatization
progressed, leading to the necessity of robust solutions for individuals’
recognition. In this context, a wide range of techniques emerged in the
literature as alternatives for effectively identifying people through images
or another biometric means.

To be considered a biometric qualification criterion, a candidate feature must
meet the following conditions (Jain et al., 2004):

  * •

Universality: each person should have the characteristic;

  * •

Distinctiveness: any two persons should be sufficiently different in terms of
the characteristic;

  * •

Permanence: the characteristic should be sufficiently invariant over a period
of time;

  * •

Collectability: the characteristic can be measured quantitatively.

However, in a practical biometric system, other issues should also be
considered, including:

  * •

Performance: which refers to the achievable recognition accuracy and speed,
the resources required to achieve the desired recognition accuracy and speed,
as well as the operational and environmental factors that can affect the
accuracy and the speed;

  * •

Acceptability: which indicates the extent to which people are willing to
accept the use of a particular biometric identifier in their daily lives;

  * •

Circumvention: which denotes the easiness the system is fooled through
fraudulent methods.

Among such techniques, stand out the fingerprint, iris, face, and gait
recognition, among others. The next section briefly revisits some of these
biometric approaches.

####  2.1.1. Fingerprint Recognition

Formally known as dactyloscopy, fingerprint recognition systems are widely
used due to the singularity of the ridges and furrows on the surface of a
finger, which provides an intrinsic characteristic for each individual (Li,
2009). Moreover, such features are steady and poorly degraded over time,
making the creation of a digital fingerprint image database extremely
reliable.

The first model for fingerprint recognition was designed in the late 1960s,
based on a system created in the XIX century by Francis Galton called Galton
points (Mitchell, 1920). Since then, many works addressed the problem through
different perspectives, such as digital image processing (Alsmirat et al.,
2019; Zneit et al., 2017), Generative Adversarial Networks (Yu et al., 2019),
and filter representation (Lee et al., 2017), to cite a few.

Fingerprint recognition systems are considered the most reliable and accurate
biometric identification system. Nevertheless, the field is still facing
several challenges, such as unsatisfactory accuracy under non-ideal conditions
and security issues such as spoofing attacks. In this context, Hemalatha et
al. (Hemalatha, 2020) present a systematic review comprising the most recent
techniques employed to deal with the drawbacks mentioned above and others.

####  2.1.2. Iris Recognition

The term iris denotes a colorful thin circular structure in the eye
responsible for controlling the pupil diameter, as well as the amount of light
that reaches the retina. Such a structure presents a great advantage for
people’s identification since it is consistent against alterations on
environmental conditions and generally suffers low degradation over time.
Moreover, iris recognition is among the most accurate, low-cost, and
convenient identification methods, since it is performed by image and do not
require contact with the person (Wang and Kumar, 2019).

Most commercial models employing iris recognition are developed based on the
identification of iris’ lower and upper limits using an integral-differential
operator (Karn et al., 2020), even when using eyelid demarcation. Such an
operator assumes the pupil is circular and acts as an orbicular border
detector. Further researches introduced distinct mathematical operations to
the process to make it more flexible and robust, such as parabolic curve
identifiers and normalizations, to cite a few.

Moreover, a different approach was proposed by Garagad and Iyer (Garagad and
Iyer, 2014), which considers a scale- and slope-invariant method that radially
draws the iris region for feature extraction. The method segments the pupil’s
central area using concentric circles and discards irrelevant regions using
discontinuity detection techniques. Besides, the method is straightforward if
compared to the approach mentioned above since it does not use integral-
differential operators, even though its effectiveness is confirmed by hit
rates above $83\%$.

Recent studies toward iris recognition aim to overcome some challenges in the
field, such as the adverse noise caused by specular reflections, the absence
of iris, gaze deviation, motion/defocus blur, iris rotation, and occlusions
due to hair/eyelid/glasses/eyelash (Wang et al., 2020). Besides, He et al. (He
et al., 2020) discuss how new trends towards long-distance iris recognition
have been tackled recently and the future trends to deal with the problem.

####  2.1.3. Face Recognition

Face recognition is a prominent biometric system extensively used in
identification and authentication systems in the most diverse areas, like
banks, military services, and public security (Zou et al., 2018a). Such
methods became popular in the earlies 90’s when Turk and Pentland proposed the
Eigenfaces method (Turk and Pentland, 1991). In the following decade emerged
several approaches so-called holistic, which are derived from low-dimensional
distribution representations, like linear sub-spaces (Deng et al., 2014), and
sparse reproduction (Deng et al., 2018b), among others.

Nevertheless, the major drawback regarding holistic methods lies is the
intolerance to face positioning change, thus leading to a search for
characteristics based methods. In this context, emerged the Local Binary
Patters (LBP) and the Gabor feature based classification, achieving relevant
results through filters embodying invariant properties. Later on, subsequent
works started to focus on local descriptors-based approaches for face
recognition (Lei et al., 2014). In short, the objective of these methods is to
train spatial filters for image feature extraction, such that the difference
between images from the same person is minimized. On the other hand, the
distance between features extracted from different people should be maximized.
Even though such methods obtained consistent results, techniques considered
“shallow”, in general, suffer from inevitable limitations due to the non-
linear complexity of the face variations.

A few years ago, a counterpoint for such approaches arose after the AlexNet
neural network had won the ImageNet (Krizhevsky et al., 2012) competition.
Since then, deep learning models gained attention in diverse areas of
biometrics, including face recognition. A particular family of networks known
as Convolutional Neural Networks was the first to achieve a level of
recognition near humans, i.e., the DeepFace (Taigman et al., 2014). The model
composed of six convolutional layers achieved accuracies less than $0.2\%$
different from one human competitor ($97.35\%$ against $97.53\%$). Nowadays,
the model is considered relatively simple when compared to popular
architectures, like Inception (Szegedy et al., 2016).

Even though face recognition approaches evolved to unprecedented standards in
the last years, the field is still facing several challenges in terms of
expression, pose, illumination, aging, and face partially or entirely
obstructed by masks or veils. Recent research presented by Tiong et al. (Tiong
et al., 2020) suggest using multimodal biometrics to tackle the issue.
Besides, Jaraman et al. (Jayaraman et al., 2020) describe the most relevant
alternatives freshly employed for face recognition in the last years, which
comprise local and global features, neural networks, and deep learning-based
approaches.

####  2.1.4. Multimodal Biometric Recognition

One of the main advantages of the human brain towards individual recognition
over computer-based approaches concerns the ability to simultaneously evaluate
multiple modalities of descriptive information, such as face, gait, and hair
and eyes color. To adapt to this trend, a new paradigm of biometrics
recognition emerged, the so-called “multimodal biometrics”. Such an approach
aims at combining distinct biometric recognition methods and auxiliary
information to improve the performance and reliability when considering a
single technique. In this context, Sultana et al. (Sultana et al., 2017)
proposed a person recognition approach based on a combination of visual cues,
such as face and ear, with the person’s social behavior. Experiments over
semi-real datasets provided a significant advantage over standard biometric
systems.

A recent work proposed by Li et al. (Li et al., 2021) implements a finger-
based multimodal biometrics system that considers a fusion strategy to extract
correlated features from different modalities of finger patterns, i.e., finger
veins and knuckle print, obtaining state-of-the-art results over finger
recognition methods. A similar work proposed by Tiong et al. (Tiong et al.,
2020) implements a multi-feature fusion convolutional neural network for
multimodal facial biometrics, obtaining competing results over several
benchmarking datasets.

The major advances of multimodal biometrics recognition are described in a
systematic review presented by Dargan et al. (Dargan and Kumar, 2020). The
authors provide a literature review concerning unimodal and multimodal
biometric systems, datasets, feature extraction techniques, classifiers,
results, efficiency, and reliability. Besides, they also discuss how
multimodal biometric approaches can be employed to overcome some common
challenges faced by unimodal methods, such as face, fingerprint, and iris
recognition.

####  2.1.5. Other Identifiers

For different reasons, the traits mentioned so far may not be interesting in
certain environments, either due to the imprecision of the model or the lack
of specialized equipment for the correct identification. Among some unusual
approaches, Ragan et al. (Ragan et al., 2016) showed that ear-based biometrics
has intrinsic characteristics capable of holding unique information from
individuals. Applications can be observed in smart-phone-based
applications111https://www.popsci.com/article/technology-tested-app-
authenticates-you-shape-your-ear. Meanwhile, Ramli et al. (Ramli et al., 2016)
employed the heartbeat rate pattern for biometric identification. The method
obtained significant accuracy and also has been employed for unlocking
electronic devices222https://www.extremetech.com/computing/165537-nymi-
wristband-turns-your-heartbeat-into-an-electronic-key-that-unlocks-your-
devices. Similarly, Chen et al. (Chen et al., 2020) offered a Raspberry Pi-
based system for gesture recognition on an Internet of Things environment. The
same group of authors (Dewi and Chen, 2019) presented a study comparing
several machine learning approaches for human activity recognition.

As described in Section 2.1.2, the characteristics of the eye present powerful
descriptive biometric features, such that iris recognition plays the leading
role in this context. However, some distinct approaches have been successfully
employed for the task. The Research developed by Ma et al. (Ma et al., 2019),
for instance, showed the eyes’ movement pattern while following objects is
unique in each person and employed such a feature for portable remote
authentication smart grids. A study proposed by Zehngut et al. (Zehngut et
al., 2015) has shown that it is possible to divide the types of noses between
six different groups and, combined with specific proportions of the width and
height of the nose, enable the identification of individuals with accuracy as
good as recognition by iris. Among the method’s advantages, one can highlight
its visibility in almost $100\%$ of cases, even when people use glasses or
hats. The major disadvantage regards plastic surgery, changing features
relevant to the correct identification of the individual.

The main drawback regarding image-based biometric systems, e.g., fingerprints
or irises methods, lies in the susceptibility to malicious attempts to fool
the system. In short, a simple image could fool such systems, granting access
to someone else’s information in a bank account, for instance. On the other
hand, some more sophisticated methods can identify people by fancier
characteristics, such as the veins configuration (Alariki et al., 2018). The
method is more robust against fraud than simple image-based techniques since
it demands specialized equipment for proper identification.

###  2.2. Deep Learning Approaches Considered for Gait Recognition

Even though standard machine learning and gait recognition strategies provided
relatively satisfactory results in the past years, such methods are usually
constrained to hand-crafted features and limited capacity for learning
intrinsic patterns in data. In this context, deep learning-based approaches
emerged as an elegant solution for tackling image/video and sequential
problems, among others, also revealing themselves as a powerful tool for gait
recognition. Thus, this section presents the most popular deep learning
architectures once employed for gait recognition, namely Convolutional Neural
Networks (LeCun et al., 1998), Recurrent Neural Networks (Jun et al., 2020),
Autoencoders (Babaee et al., 2019), Capsule Networks (Sabour et al., 2017),
Generative Adversarial Networks (Goodfellow et al., 2014), and Deep Belief
Networks (Hinton et al., 2006).

####  2.2.1. Convolutional Neural Networks

Convolutional Neural Networks (LeCun et al., 1998) obtained exceptional
popularity since the beginning of the 2010s, becoming paramount to solve image
processing-related problems, such as image classification (Tan and Le, 2019)
and segmentation (Zoph et al., 2020). As the name suggests, CNNs basic blocks
are composed of convolutional neurons, usually composed of $3\times 3$ or
$5\times 5$ kernels, which are responsible for performing convolutional
operations over the input data. Briefly speaking, this process’s output
generates a new set of matrices, which are then employed as the subsequent
layers of the model. In signal processing, convolution is described as a
multiplication of two signals to generate a third one (Oppenheim et al.,
2001). Figure 1 depicts the idea.

011$1_{\times 1}$$0_{\times 0}$$0_{\times 1}$0001$1_{\times 0}$$1_{\times
1}$$0_{\times 0}$0001$1_{\times 1}$$1_{\times 0}$$1_{\times 1}$00011 1 0 00010
0 0 00100 0 0 01100 0 0 0$\ast$101010101$=$14341l2433123411331133110 Figure 1.
An example of $3\times 3$ convolution.

Therefore, a CNN can be interpreted as a stack of convolutional kernels
generating the successive layers’ input. Notice some intermediate pooling
layers may be included in the process, and a fully connected layer is coupled
at the top of the architecture for classification purposes. The learning
process is performed by the backpropagation algorithm working with a gradient
descent calculation. Figure 2, inspired in LeNet-5, pictures a CNN
architecture.

Figure 2. A standard CNN architecture’s example.

####  2.2.2. Capsule Networks

Even though CNNs work very well for image features’ understanding, they are
prone to confuse spatial relationships between complex peculiarities. In other
words, a trained CNN is usually capable of recognizing dogs if it finds a
dog’s body, face, tail, and so on, even if the dog is assembled in a different
sequence or if its parts are located in distinct sections of the image. On the
other hand, Capsule Networks (Sabour et al., 2017) consider a hierarchical
approach to tackle this problem. In short, the model comprises a two-layer
structure. The first is a convolutional encoder, which performs the
recognition of low-level features. The second stands for a fully-connected
linear decoder that employs the routing by agreement algorithm (Sabour et al.,
2017) to address such low-level features to the correct position in a
hierarchical higher-level. Therefore, CapsNets are more robust to object
orientation. Besides, they may perform better for identifying multiple or
overlapping objects in a scene.

####  2.2.3. Recurrent Neural Networks

Many works addressed the gait recognition problem as a sequence of images
defining the individual’s movement. A standard method to acknowledge such
strategy using deep learning concerns Recurrent Neural Networks (RNNs) (Jun et
al., 2020), which compute each neuron’s activation considering the information
from the input data, as well as other neuron’s output, in a recurrent fashion.
Occasionally, the architecture is combined with a CNN to extract more
information about the input images to perform the inference.

Since describing a person’s gait usually requires a considerable amount of
sequential features, a particular set of RNNs, namely gated RNN, are more
suitable for the task due to their abilities to deal with long sequences. In
this context, one can refer to two main architectures, i.e., the Long-Short
Term Memory (Hochreiter and Schmidhuber, 1997) and the Gated Recurrent Unit
(GRU) (Chung et al., 2014).

  1. (1)

Long-Short Term Memory:

The Long-Short Term Memory was first implemented in 1997 by Hochreiter and
Schmidhuber (Hochreiter and Schmidhuber, 1997), where the main objective was
to improve results on long sequences of data. In a nutshell, LSTMs work
similarly to the traditional RNN, i.e., the output of a given neuron depends
on recurrent information from previous neurons’ outcomes. The main difference
lies in the LSTM cell’s architecture, which comprises more complex and fancy
relationships. Such an architecture is composed of three main gates that
control the flow of information, described as follows:

     * •

The forget gate: this gate defines how much of the information should be kept.
The previous and current state data is passed through a sigmoid function,
which outputs values between $0$ and $1$. The closer to $1$, the more
information is preserved.

     * •

The input gate: computes a new value to update the current hidden state. The
input gate considers two central values: (i) a sigmoid function calculates the
importance of the previously hidden state, and (ii) the original value is
forwarded to a hyperbolic tangent (tanh) function, which is responsible for
squishing this value between $-1$ and $1$. The multiplication of these two
values defines the current hidden state.

     * •

The output gate: after estimating the forget and the input gates, the output
gate defines the cell’s output value. The process is performed as follows: (i)
the values from the forget and the input gate are summed up and submitted to a
tahn function, (ii) the cell’s previous state is submitted to a sigmoid
function, (iii) the output from both the sigmoid and the tahn functions are
multiplied, yilding the cell’s output.

  2. (2)

Gated Recurrent Unity:

The Gated Recurrent Unit (Cho et al., 2014) is a recurrent neural network
originally idealized to improve results on neural machine translations. Like
LSTMs, GRUs possess internal gates that control the flow of information, and
the main difference lies in the number of gates available in each model, i.e.,
GRU comprises only two, namely the forget and the output gates, instead of
three. Studies show that, even though the GRU uses fewer gates than LSTM, it
can reach similar results (Chung et al., 2014), with the advantage of a
reduced computational burden and faster performance considering both tasks of
training and inference.

####  2.2.4. Autoencoders

Autoencoders (Vincent et al., 2010) are generative neural networks usually
employed for data reduction and image denoising. The model comprises two main
steps, described as follows:

  * •

Encoder: it is responsible for encoding the input information into an,
usually, smaller feature space.

  * •

Decoder: it performs the unsupervised reconstruction of encoded data.

Figure 3 depicts the architecture of the model.

Figure 3. A standard Autoencoder architecture.

####  2.2.5. Deep Belief Networks

Deep Belief Networks (Hinton et al., 2006) are stochastic neural networks
ideally designed for generative tasks, such that each layer denotes a greedy-
fashioned trained Restricted Boltzmann Machine (RBM) (Hinton et al., 2006). In
short, RBM is a graphical model composed of a visible and a latent set of
units, namely visible and hidden layers, respectively, which are connected by
a weight matrix with no connection among neurons from the same layer. The
model’s learning approach consists of attaching a collection of input data
into the visible layer and finding a representation of this data in the hidden
units. Besides, the training procedure is performed by minimizing the system’s
energy, usually conducted using a Markov chain procedure through Gibbs
sampling for optimization purposes.

Concerning classification tasks, the most common approach considers coupling a
softmax layer at the top of the DBN architecture and, after greedily pre-
training all the RBMs, performing a fine-tuning in the weights using
Backpropagation, for instance, to adjust the weight matrices and fitting the
labels for proper identification.

####  2.2.6. Generative Adversarial Networks

Generative Adversarial Networks (Goodfellow et al., 2014) became popular in
the last years due to their outstanding ability to generate realistic
synthetic images. The model comprises two distinct networks, i.e., a
generator, which is responsible for learning the data’s distribution and
generating synthetic samples, and a discriminator, which tries to identify
whether a given instance is original or synthetically created. The generator
and the discriminator compete in an adversarial fashion such that the
generator attempts to generate samples realistic enough to fool the
discriminator. In contrast, the discriminator improves itself more and more to
recognize such fake images. Figure 4 illustrates the model.

Figure 4. A standard GAN architecture.

####  2.2.7. Deep learning techniques’ summary

Table LABEL:t.architectures presents a summary concerning the deep learning
techniques considered in this work and how they are usually employed for gait
recognition tasks. Notice that the same work can eventually use distinct
architectures, thus appearing more than once in the table.

Table 1. Deep learning-based gait recognition approaches organized byt type of neural network. References |  Technique |  Task
---|---|---
(Shiraga et al., 2016; Wu et al., 2017a; Li et al., 2017; Sokolova and Konushin, 2017a; Takemura et al., 2017; Sokolova and Konushin, 2017b; Yu et al., 2017; Zou et al., 2018a; Wang and Yan, 2020; Xu et al., 2020) |  Convolutional Neural Networks (CNN) |  Feature extraction from images or frames of a video.
(Zhang et al., 2019; Babaee et al., 2019) |  Auto Encoder |  Works by compressing and decompressing features from the input.
(Xu et al., 2019; Sepas-Moghaddam et al., 2021; Zhao et al., 2021) |  Capsule |  Improve the semantic organization of the outputs from a CNN.
(Fernandes et al., 2018; Xiong et al., 2020) |  Deep Belief Networks |  Encode features and patterns into compressed representations.
(He et al., 2018; Jia et al., 2019; Hu et al., 2018) |  Generative Adversarial Networks |  A training method that relies on the differentiaton of an original input and a generate counterpart from a model, such as a CNN.
(Zou et al., 2018a; Zhang et al., 2019; Potluri et al., 2019; Wang and Yan, 2020; Tran et al., 2021) |  Recurrent Neural Networks |  Comprise both GRUs and LSTMs, which are composed with several gates to controle the flow of information and are employed to deal with temporal information.

###  2.3. Gait Recognition

Several methods for people recognition through biological characteristics have
been presented so far. Although the methods’ reliability and safety are
confirmed by their success in banks and public governance systems, two main
hindrances must be stressed: (i) they depend on the passive provision of
personal biometric information, i.e., the person must provide or register the
required information for the recognition; and (ii) some of these systems rely
on specialized equipment.

An alternative to deal with such drawbacks may comprise gait recognition
models, especially considering video-based approaches. Such methods do not
suffer from the problems mentioned above, once the acquisition of the
biometric information depends, most of the time, only on a camera with no
specific features or non-evasive sensors and devices, and, disregarding legal
problems, the collection of such information is performed passively.
Therefore, the observed person is no longer required to cooperating in the
identification process or providing any information. On the other hand,
although video-based recognition is the most intuitive method, it does not
stand for the only possibility. Gait identification can also be performed by
distinct characteristics, e.g., the footprint (Costilla-Reyes et al., 2018),
which considers the pressure and size of the area. The obvious disadvantage of
such models is the cost of deploying the equipment necessary to acquire such
features.

Moreover, techniques proposed for gait recognition can be divided into two
main groups, i.e., template- and non-template-based methods. Template-based
methods aim to obtain the movement of the trunk or legs, i.e., they usually
focus on the dynamics of movement through space or spatio-temporal based
methods (Yeo and Park, 2020). Among such techniques, one can refer to Walking
Path Image (WPI) information (Zhao et al., 2016), Gait Information Image (GII)
(Arora et al., 2015), and Gait Energy Image (GEI) features, which can be
extracted through Canonical Correlation Analysis (Luo and Tjahjadi, 2020),
Joint Sparsity Models (Yogarajah et al., 2015), segmentation using Group Lasso
Motion (Rida et al., 2016), among others (Shiraga et al., 2016; Li et al.,
2017; Wu et al., 2017b). On the other hand, non-template based methods
consider the shape and its attributes as the more relevant characteristics,
i.e., the recognition of the individual is performed with measurements that
reflect its shape (Deng et al., 2018a).

Regarding the process of gait information acquisition, it may comprise several
sorts of sensors and devices, as illustrated in Figure 5. The image depicts a
laboratory-like environment designed with several embedded devices for gait
data acquisition, such as cameras, which are the most commonly employed device
for the task since they are quite accessible and are capable of acquiring data
over greater distances, and multispectral photographic sensors that are very
efficient, but with less accessible due to high costs. The image also
comprises gyroscope and velocity sensors on the roof, responsible for
describing the spatio-temporal context acquisition. Finally, the walking
rhythm is measured using a step-pressure carpet-like sensor, which is highly
valuable to detecte gait abnormalities, such as the ones caused by Parkinson’s
disease and other neurodegenerative problems.

Figure 5. Example of gait data acquisition environment. The scene comprises
the following devices: sidewall cameras, followed by a multi-spectrum
photographic sensor. The roof accommodates velocity sensors and gyroscopes,
which measure the space-time relationship during the walk. Finally, the floor
is equipped with a carpet-like sensor for collecting the foot pressure on the
ground, completing the gait cycle.

Despite the success obtained by the methods mentioned above, deep learning-
based approaches induced a paradigm shift in the field of gait recognition,
obtaining paramount results in a variety of applications. Therefore, the next
section introduces an in-depth presentation of several surveyed works in the
context of deep learning-based approaches for gait recognition. Figure 6
provides a schematic diagram comprising an overview of the main differences
between deep learning and the standard methods for gait recognition.

Figure 6. Schematic diagrams from deep learning (top) and standard (bottom)
gait recognition pipelines. As the image suggests, deep learning approaches
abstract the steps concerning data pre-processing, feature extraction, and
classification in a single architecture.

##  3\. Gait Recognition Through Deep Learning-Based Approaches

This section presents a systematic review of recent works using distinct deep
learning architectures, i.e., Convolutional Neural Networks, Recurrent Neural
Networks, Generative Adversarial Networks, Deep Belief Networks, and
Autoencoder-based approaches for gait recognition. Besides, it also presents a
discussion regarding the compared methods and a summary of the surveyed
studies.

###  3.1. Convolutional Neural Networks

Convolutional Neural Networks (LeCun et al., 1998) use a concept of a neuron
based on the visual cortex of mammals, which was first validated for the task
of digit classification and today is probably the most widely employed neural
network for classification, reconstruction, and object detection, among
others. In the context of gait recognition, Shiraga et al. (Shiraga et al.,
2016) used an architecture similar to LENet (LeCun et al., 1998) to create a
gleaming-based recognizer through Gait Energy Image (GEI). The model achieved
an accuracy of $91.5\%$ in the OU-ISIR (Makihara et al., 2012) Large
Population dataset. A similar work proposed by Wang et al. (Wang et al., 2019)
employ nonstandard periodic GEI approaches for gait recognition and data
augmentation.

Regarding cross-view gait-based human identification, Wu et al. (Wu et al.,
2017a) claim to propose the first work using CNNs in the context of grimace
recognition. The authors employed several network architectures and
demonstrated the power of CNNs in the context of biometric-based
identification, obtaining gains above $10\%$ to previous results. Besides,
they also compared three types of data arrangement, described as follows:

  * •

Local Bottom: a combination is made among the input data, and then it is
determined whether the inputs belong to the same or different individuals;

  * •

Mid-Level Top: the neural network extracts some characteristics of both inputs
before combining them and then determining whether or not they come from the
same person; and

  * •

Global Top: similar to the previous network. However, it has an extra level of
convolutions and Perceptrons, such that the combination of characteristics is
made in the penultimate layer.

Further, Li et al. (Li et al., 2017) proposed the DeepGait, a model that
combines deep convolutional features and Joint Bayesian for video sensor-based
gait representation. The model outperformed hand-crafted features, such as
GEI, Frequency-Domain Feature, and Gait Flow Image, obtaining state-of-the-art
results over OU-ISR large population dataset (Makihara et al., 2012).
Meanwhile, a study proposed by Sokolova and Konushin (Sokolova and Konushin,
2017a) demonstrated the difficulty in identifying people by their behavior due
to the intersection of information, despite satisfactory results over TUM-GAID
database (Hofmann et al., 2014), obtaining $97.5\%$ accuracy and $99.89\%$
Rank-5. However, the performance was severely degraded concerning CASIA B
dataset (Yu et al., 2006), bringing an accuracy of $58.20\%$ over such a
scenario, demonstrating how this type of biometry is sensitive to the data’s
context.

The same authors proposed a multiple-stage model for gait recognition using
Optical Flow (OF) (Sokolova and Konushin, 2017b). In this model, the data is
pre-processed in two main steps: motion map computation and frame-by-frame
evaluation of the individual’s pose. Further, two distinct neural networks,
i.e., VGG-19 (Simonyan and Zisserman, 2014) and Wide Residual Network (WRN)
(Zagoruyko and Komodakis, 2016), are employed to validate the technique in the
context of video-based gait recognition. To increase classification speed,
they incorporated the Principal Component Analysis (PCA) for dimensionality
reduction. Such information goes through the normalization L2, and finally,
the features produced are fed to a Nearest Neighbor (NN) classifier. Using
this combination of methods, they achieved very accurate results over TUM-GAID
(Hofmann et al., 2014), CASIA B (Yu et al., 2006), and OU-ISIR (Makihara et
al., 2012) datasets.

Recently, an architecture called Siamese Networks, which accepts two separate
entries as input and computes a similarity value between them, was
successfully employed for similar tasks, such as identifying obstructed routes
or misbehavior in hazardous environments (Santana et al., 2019). In this
context, Takemura et al. (Takemura et al., 2017) implemented four different
frameworks to create a biometric identification system. All networks rely on
GEI to make the comparisons. Two of these networks were developed based on the
Triplet Ranking Loss, which necessarily needs three inputs for execution: (i)
the information of the compared person, (ii) data from the same person, and
(iii) an entrance from any other person. The other two networks use
contrastive loss in their execution, whose entries depend only on the person’s
information for comparison. The authors obtained $91.9\%$ of accuracy over the
OU-ISIR Multi-view Large Population (OU-MVLP) (Takemura et al., 2018)
database.

Still, Xu et al. (Xu et al., 2020) proposed the Pairwise Spatial Transformer
Network, a unified model composed of pairwise spatial transformers and a
recognition network for cross-view gait recognition. The model computes non-
rigid deformation fields to match input pairs into intermediate frames, which
are compared against a deformation suffered from source view to target.
Experiments conducted over the OU-MVLP (Takemura et al., 2018), OU-ISIR Large
Population (OU-LP) (Iwama et al., 2012), and CASIA B (Yu et al., 2006)
datasets grant the model robustness.

###  3.2. Capsule Networks

Another well-known type of deep architecture employed for gait recognition is
the Capsule Neural Network (Sabour et al., 2017). The network has been
developed for image classification by modeling the hierarchical relationships
between objects, i.e., capsules, in a scene. In this context, Xu et al. (Xu et
al., 2019) explored such features to propose two architectures for gait
recognition:

  * •

using the Local Bottom Feature (LBC) combination of local features, which
employs two input images and computes the differences between then in a unique
network flow; and

  * •

employing the so-called Matching Mid-Level Feature (MMF) to combine both
images’ characteristics after passing through a specific part of the neural
network.

In summary, the first architecture merges the images before inputting them to
the neural network, while the other combines the images after going through
two layers of transformations. Experiments considering GEIs, Chrono-gait image
(CGI), and resolution of the input image present an identification accuracy of
$74.4\%$ over OU-ISIR (Makihara et al., 2012) dataset.

In a similar work, Sepas et al. (Sepas-Moghaddam et al., 2021) used capsule
networks to develop a model capable of learning more discriminative features
by transferring multi-scale partial gait representations. The model employs
Bi-directional Gated Recurrent Units (BGRU) to learn the co-occurrences and
correlations among patterns and further uses a capsule network to extract
deeper relationships among such features. Experiments conducted over CASIA B
(Yu et al., 2006) and OU-MVLP (Takemura et al., 2018) assesses the superiority
of the model against state-of-the-art approaches, especially when considering
challenging conditions.

Finally, Zhao et al. (Zhao et al., 2021) introduced an automated learning
system called Associated Spatio-Temporal Capsule Network (ASTCapsNet). The
model was trained over multi-sensor datasets to show that multi-modality data
is more conducive to gait recognition. The model’s effectiveness is confirmed
on the experiments conducted over several datasets, in which results are
compared to state-of-the-art approaches.

###  3.3. Recurrent Neural Networks

Concerning Recurrent Neural Networks (Jun et al., 2020), Wang and Yan (Wang
and Yan, 2020) used the Long short-term memory (Hochreiter and Schmidhuber,
1997), a popular RNN architecture capable of learning long-term dependencies,
for cross-view human gait recognition based on frame-by-frame GEIs.
Experiments conducted over CASIA B (Yu et al., 2006) and OU-ISIR (Makihara et
al., 2012) Large Population datasets demonstrate the model’s robustness over
several baselines.

A similar work proposed by Potluri et al. (Potluri et al., 2019) employed
LSTMs to detect gait abnormalities through a wearable sensor system. The
experiments were conducted over a set of data extracted from ten healthy
individuals, such that seven behave normally, while the three remaining
simulate specific gait abnormalities, i.e., sensory ataxic, Parkinson’s gait,
and hemiplegic gait. The experiments focus on employing advanced technologies
for gait-based diagnosis and treatment assistant systems. Other models
combined specific information, such as data obtained from accelerometer and
gyroscope, for gait recognition through deep learning-based approaches,
reaching recognition rates above $91\%$ (Zou et al., 2018b).

Recently, Tran et al. (Tran et al., 2021) proposed an Inertial Measurement
Units (IMUs)-based gait recognition approach. The authors employed LSTMs to
exploit the temporal information on video sequences, thus extracting hidden
patterns inside such sequences. Experiments conducted over whuGAIT (Zou et
al., 2018b) and OU-ISIR (Makihara et al., 2012) provided state-of-the-art
performance for both verification and identification tasks.

###  3.4. Autoencoders

Babaee at al. (Babaee et al., 2019) also proposed a multiple-stage model for
the recognition of gestures. The work tackle a problem commonly observed in
several GEI-based identification datasets, i.e., the person’s cycle of
moviment is not entirely formed. In other words, the data available for
identification is not complete. To deal with the problem, the authors proposed
an autoencoder-based approach called Incomplete to Complete GEI Network
(ITCNet). The model was trained to reconstruct the missing images using
examples from the training set with a complete cycle. The network consists of
$9$ fully convolutional networks, each of responsible for $1/9$ of the
components of the cycle. Further, PCA is employed to compute the main features
and an RNN is used for gait recognition. The model presented an index of
$86\%$ and $95\%$ for Rank-1 and Rank-5, respectively, over OU-ISR Large
Population dataset (Makihara et al., 2012) using only $20$ images per
instance.

Autoencoders can also be employed to extract distinct GEI features. In this
context, Yu et al. (Yu et al., 2017) proposed a study using Stacked
Progressive Auto-Encoders (SPAE) (Kan et al., 2014) with appropriate changes
for the extraction of invariant characteristics of an individual’s behavior.
In short, the model extracts information from independent components of the
scene, such as clothes and other objects the person may eventually be
carrying, instead of the movement exclusively. Experiments conducted over
CASIA B (Yu et al., 2006) and SZU RGB (Yu et al., 2013) datasets show that,
despite the GEI information for personal identification, intermediate and last
layers also provide relevant characteristics regarding such components.
Finally, the work combines such information and perform a dimensionality
reduction step through PCA.

Zhang et al. (Zhang et al., 2019) also employed autoencoders for gait
recognition. The authors used the technique to tackle a common issue faced by
biometrical recognition literature, i.e., dealing with clothes blocking body
members’ views, such as arms and legs. The main problem concerning such
occlusions is the difficulty of reading the movements and recognizing other
features such as ligaments and the point of contact with the ground. To
minimize the problem, they used autoencoders to disconnect the images’ pose
and appearance characteristics for further linking those characteristics to
another architecture of recurrent networks to analyze the way the person
gaits. Besides, the authors also propose the Frontal-View Gait (FVG)
dataset.The system proved to be particularly robust when filming the person
walking towards the camera, whose angulation is more complicated due to the
difficulty of obtaining specific characteristics, such as the distance walked
by each step.

###  3.5. Deep Belief Networks

Deep Belief Networks (DBNs) (Hinton et al., 2006) are stochastic neural
networks constructed using Restricted Boltzmann Machines (Hinton, 2002) as
building blocks. Such models became very popular due to their ability of
performing several tasks, such as feature selection (de Souza et al., 2021),
classification (Roder et al., ress), and image reconstruction (Passos et al.,
2019, 2017), among others. Regarding gait recognition, Fernandes et al.
(Fernandes et al., 2018) employed DBNs to support gait assessment in the
diagnosis of Parkinson’s and movement disorder diseases. The authors employed
wearable sensors to extract features from the subjects and performed a
comparative classification analysis of parkinsonian gait, confirming that DBN-
based approaches are suitable for the task.

Further, Xiong et al. (Xiong et al., 2020) demonstrated how to encode patterns
using surface electromyography (sEMG) through Deep Belief Networks. The
authors consider the knee and ankle joint angle during walking to estimate a
combination of four-time domain features. The work showed a high potential for
gait tracking problems.

###  3.6. Generative Adversarial Networks

A very interesting work by Hu et al. (Hu et al., 2018) introduces Generative
Adversarial Networks (Goodfellow et al., 2014; Souza Jr et al., 2020) to the
context of Gait recognition. The authors propose the Discriminant Gait
Generative Adversarial Network, i.e., DiGGAN, to extract view-invariant gait
characteristics for cross-view gait recognition. Experiments conducted over
OU-MVLP (Takemura et al., 2018) and CASIA B (Yu et al., 2006) outperformed
state-of-the-art results, demonstrating the model’s capabilities.

Meanwhile, He et al. (He et al., 2018) proposed the Multi-task Generative
Adversarial Networks (MGANs), a GAN-based network designed to learn view-
specific gait features through the so-called Period Energy Image (PEI), a
multi-channel gait template proposed to tackle the problems of view angles’
variation faced by cross-view methods and loss of temporal information faced
by GEI-based approaches. The model employs a view-angle manifold to extract
more significant features from video sequences, thus providing competitive
results over OU-ISIR (Makihara et al., 2012), CASIA B (Yu et al., 2006), and
USF against various works available in the literature.

Further, Jia et al. (Jia et al., 2019) studied methods to avoid attacks in
gait recognition systems through GAN-generated syntectic image sequences. In
such a scenario, the authors proposed a GAN-based approach capable of
rendering fake videos from source walking sequence with realistic details. The
method is compared against two state-of-the-art gait recognition systems, and
results are analyzed under attacking scenarios considering both CASIA A (Wang
et al., 2003) and CASIA B (Yu et al., 2006) datasets. The effectiveness of the
model is verified in both attacking detection ability and visual fidelity.

###  3.7. Discussion

Table LABEL:t.summary provides a summary of the surveyed studies presented in
this section. In general, Convolutional Neural Networks are the most popular
choice when considering deep learning solutions, especially concerning
image/video-based issues, including gait recognition. Such behavior is
expected since CNNs obtained outstanding results in various applications and
won most of the benchmark challenges in the last years. Nevertheless, the
other architectures also provide a valuable contribution to the field,
performing better for several specific tasks.

Capsule Networks, for instance, are capable of extracting partial gait
representations in a hierarchical fashion, providing better results for cases
where individuals or objects are presented in the scene with multiple
orientations or overlapped. Similarly, Recurrent Neural Networks are paramount
to deal with sequential data, such as videos, thus presenting themselves as an
essential tool for this kind of gait recognition approach.

Although most deep learning-based approaches for gait recognition comprise
image/video domain, other data sources, such as accelerometers, gyroscope,
sensor-based, and handcrafted features in general, also provided impressive
results in a considerable number of works. Most of these works address
unsupervised deep learning approaches in the process, such as Autoencoders and
DBNs, which usually are more expressive over these data types, while CNNs are
paramount when dealing with raw image/video files. These unsupervised methods
can extract information regarding the data distribution and manipulate it,
usually in a lower-dimensional space, providing more representative features
for gait recognition.

Finally, Generative Adversarial Networks describe a particular case when gait
systems can learn a broader range of features, such as orientations, clothes,
number of individuals in the scene, and so on, since they can generate
synthetic data for training models. Besides, they are also useful for
evaluating frauds in gait-based systems for security purposes by producing
fake images for testing purposes.

Concerning the most used methods to represent gait image data, Gait Energy
Image reflects the sequence of a simple energy image cycle using the weighted
average method. Further, the sequences in a travel cycle are processed to
align the binary silhouette (Jing Luo and Xiu, 2015). Therefore, GEI maintains
the static and dynamic characteristics of human walking and significantly
reduces image processing’s computational cost. From an in-depth analysis of
the method, one can observe a few points that characterize the model (Han and
Bhanu, 2016), described as follows:

  * •

GEIs are a little sensitive to silhouette noise in individual pictures.

  * •

It focuses on specific representations of the human walk, which does not
soften the context of vector images.

  * •

It represents human motion in a single image while preserving temporal
information.

Similarly, cross-view-based gait recognition is a popular approach used to
deal with different visual angles. The input type requires multiple fully
controlled cameras and cooperative environments, thus being restrained to real
scenarios. Moreover, it visually normalizes gait characteristics before
performing any combination, which allows the model to learn the relationships
among visual movements in the scene (e MS Nixon, 2011; W. Kusakunniran and Li,
2013; Yasushi Makihara and Yagi, 2006). An example of the approach can be
observed in the UK’s Newcastle University cross-view gait recognition system,
the so-called DiGGAN, which obtained state-of-the-art results over the largest
multiview gait dataset in the world (Takemura et al., 2018) (comprising more
than $10,000$ people) using cross-view approaches (Hu et al., 2018).

Despite the success obtained by such methods, they still lack in some aspects,
in which some less popular techniques aim to overcome. The Period Energy
Image, for instance, comprises a multi-channel gait template proposed to
tackle view angles’ variation faced by cross-view and GEI loss of temporal
information (He et al., 2018). Another approach observed in some works
comprises Optical-flow, which provides relevant information regarding the
objects observed in the scene, their spatial arrangements, and the changes in
sich arrangements (Luo et al., 2016). Finally, a wide range of hand-crafted
features or sensor-based input data types is also employed for deep learning-
based gait recognition approaches. Among such methods, one can find the
Inertial Measurement Units, accelerometers, gyroscopes, sensor output, and so
on.

Concerning gait recognition’s actual state-of-the-art scenario, one can
consider two main approaches, i.e., literature- and representation-based.
Regarding the literature-based, 2D-CNNs are the most widely used type of deep
neural network (DNN) for gait recognition using deep learning, with
approximately 50% of solutions just based on 2D-CNN architectures for
classification. Solutions using 3D-CNN and GAN are the next popular
categories, each corresponding to 10% of the published content. In addition,
deep autoencoder (DAE), RNN, CapsNet, DBN, and graph convolutional networks
are less considered among DNNs, corresponding to $5\%$, $3\%$, $2\%$, $1\%$,
and $1\%$, respectively. On the other hand, the hybrid methods that constitute
$26\%$ of the solutions, in which the CNN-RNN combinations are the most widely
adopted approach with about $15\%$ of presence, while the combination of DAE
with GAN and RNN corresponds to $10\%$ of the methods, followed by the RNN-
CapsNet methods that constitute 2% of the solutions.

Regarding state-of-the-art solutions based on representation, the silhouettes
are the most widely adopted for gait recognition, corresponding to more than
$85\%$ of the solutions. Although it is a promising approach, skeletons have
been considered less frequently in relation to silhouettes, corresponding to
only $10\%$ of the available solutions. There were also some methods, that is,
approximately $5\%$ of the available literature, that explore the
representations of the skeleton and the silhouette, notably using unraveled
representation learning or score of fusion strategies.

Considering deep learning-based gait recognition main challenges, one can
refer to the complexity of gait data, which arises from the interaction
between many factors, such as occlusion/obstruction, camera points of view,
the appearance of individuals, order of sequence, movement of body parts, or
light sources present in the data, among others (Z. Zhang and Liu, 2020; X. Li
and Ren, 2020). Such factors can interfere in a complicated way and can hinder
the task of gait recognition. Currently, there is an increasing number of
methods in other areas related to pattern recognition, such as face
recognition, emotion, and pose estimation. These professionals focus on
learning confusing contexts, extracting representations that separate the
various explanatory factors in the data’s high-dimensional space. However,
most gait recognition methods available using deep learning have not yet
explored such approaches. Therefore, they are not explicitly able to separate
the underlying structure of gait data in the form of significant disjoint
variables. Despite recent progress in using confusing context approaches in
some gait recognition methods, there is still room for improvement.

Table 2. Deep learning-based gait recognition approaches. Ref. |  Year |  Model |  Input Type |  Dataset |  Result |  Measure
---|---|---|---|---|---|---
(Shiraga et al., 2016) |  2016 |  CNN |  GEI |  OU-ISIR |  $94.6\%$ |  Identification rate
(Wu et al., 2017a) |  2017 |  CNN |  Cross-view |  CASIA B |  $90.8\%$ |  Accuracy
(Li et al., 2017) |  2017 |  CNN + Joint Bayesian |  Sensors |  OU-ISR |  $97.6\%$ |  Identification rate
(Sokolova and Konushin, 2017a) |  2017 |  CNN |  Optical flow |  TUM-GAID and CASIA B |  $97.52\%$ |  Accuracy
(Takemura et al., 2017) |  2017 |  CNN + Siamese networks |  Cross-view |  OU-ISIR |  $98.8\%$ |  Accuracy
(Sokolova and Konushin, 2017b) |  2017 |  CNN + Nearest Neighbor |  Optical Flow |  TUM-GAID, CASIA B, and OU-ISIR |  $99.8\%$ |  Identification rate
(Yu et al., 2017) |  2017 |  Autoencoders + PCA |  GEI |  CASIA B and SZU RGB |  $97.58\%$ |  Identification rate
(Zou et al., 2018b) |  2018 |  CNN + LSTM |  Accelerometer and Gyroscope |  whuGAIT and OU-ISIR |  $99.75\%$ |  Accuracy
(He et al., 2018) |  2018 |  GAN |  PEI |  OU-ISIR, CASIA B and USF |  $94.7\%$ |  Accuracy
(Fernandes et al., 2018) |  2018 |  DBN |  Sensors |  Data collected by the authors |  $93\%$ |  Accuracy
(Xu et al., 2019) |  2019 |  Capsule |  LBC and MMF |  OU-ISIR |  $74.4\%$ |  Accuracy
(Wang et al., 2019) |  2019 |  CNN |  GEI + Data Augmentation |  CASIA B |  $98\%$ |  Accuracy
(Zhang et al., 2019) |  2019 |  Autoencoders + LSTN |  Croos- and Frontal-view |  CASIA B, USF, and FVG |  $99.1\%$ |  Accuracy
(Babaee et al., 2019) |  2019 |  Autoencoders + PCA |  GEI |  OU-ISIR and CASIA B |  $96.15\%$ |  Accuracy
(Potluri et al., 2019) |  2019 |  LSTM |  Sensors |  Data collected by the authors |  $0.02$ |  Prediction error
(Jia et al., 2019) |  2019 |  GAN |  GEI |  CASIA A and CASIA B |  $82\%$ |  Recognition result
(Wang and Yan, 2020) |  2020 |  LSTM |  GEI |  CASIA B and OU-ISIR |  $99.1\%$ |  Recognition rate
(Xu et al., 2020) |  2020 |  CNN |  Cross-view |  OU-MVLP, OU-LP, and CASIA B |  $98.93\%$ |  Identification rate
(Hu et al., 2018) |  2020 |  GAN |  Cross-view |  OU-MVLP and CASIA B |  $93.2\%$ |  Identification rate
(Sepas-Moghaddam et al., 2021) |  2020 |  Capsule |  Multi-scale representations |  CASIA-B and OU-MVLP |  $84.5\%$ |  Identification rate
(Xiong et al., 2020) |  2020 |  DBN |  Sensors |  Data collected by the authors |  $2.61$ |  RMSE
(Tran et al., 2021) |  2021 |  LSTM |  IMU |  whuGAIT and OU-ISIR |  $94.15\%$ |  Accuracy
(Zhao et al., 2021) |  2021 |  Capsule |  Sensors |  Several |  $99.69\%$ |  Accuracy

##  4\. Datasets

Machine learning models’ training and evaluation steps, regardless of the
adopted paradigm, i.e., supervised, unsupervised, or any other, depend on a
dataset comprising the task’s subject. Besides, the employment of such
datasets makes it possible to determine how effective a method is to solve a
specific problem and compare it with other solutions.

Regarding gait recognition, the availability of specific datasets for the task
is minimal, considering both public or private solutions. Such a lack of
training data hampers the development of new artificial intelligence models
capable of recognizing people by how they walk or move. Two problems stand out
for obtaining and creating a dataset:

  * •

Gait biometry demands a reasonable amount of movement recordings for a
subject, implying recording and generating multiple videos for each
individual. Besides, such videos usually possess an inherent high
dimensionality, entailing in the dataset’s growth and thus requiring a high
storage capacity.

  * •

The extraction and public distribution of biometric data require permission
from each participant. The eventual creation of a dataset without formal
consent from each individual may bring about lawsuits.

The next section gathers the most-used datasets available for gait recognition
related tasks.

###  4.1. CMU MoBo Dataset

The CMU MoBo (Gross and Shi, 2001) dataset possesses a relatively small amount
of data, comprising several videos for gait recognition extracted from $20$
people. The dataset also provides a set of silhouette masks and bounding
boxes, thus alleviating the segmentation process.

One of the main advantages of the dataset is that it is available for download
without reservations or need to sign forms of agreement, requiring just
connecting to the Calgary University File Transfer Protocol (FTP)
server333ftp://ftp.cc.gatech.edu/pub/gvu/cpl/.

###  4.2. TUM GAID Dataset

The TUM Gait from Audio, Image, and Depth (GAID) (Hofmann et al., 2014)
database, as the name suggests, is compounded by RGB images, audio, and depth,
recorded from $305$ people in three different variations at first. Further,
$32$ people were re-recorded to have some variation, comprising $3,370$
records in total. According to the authors, it is the only dataset that allows
recognition by combining video, depth, and audio.

Variations include different shows, carrying conditions (with and without
bag), and time. The first record was performed in 2012’s winter (January) and
comprised $176$ recordings, using a jacket and winter boots. The second
performance was made in April 2012 by $161$ people using considerably
different clothes, since it was warmer. From this amount, $32$ people were
recorded both times, which grant cloth and time variation. A strong advantage
of this dataset is a well-defined evaluation protocol. Figure 7 depicts some
examples of images and depth. The dataset is available after sign a document
request444https://www.ei.tum.de/mmk/verschiedenes/tum-gaid-database/.

Figure 7. Examples of three male (top rows) and three female (bottom rows)
participants in six variations: normal (columns $1$ and $2$), backpack
(columns $3$ and $4$), coating shoes (columns $5$ and $6$), time (columns $7$
and $8$), time + backpack (columns $9$ and $10$), and time + coating shoes
(columns $10$ and $11$). Extracted from
https://www.ei.tum.de/mmk/verschiedenes/tum-gaid-database/

.

###  4.3. HID-UMD Dataset

The Human Identification at a Distance (HID)-UMD dataset comprises several
videos from people walking captured in four different angles and their
respective binary masks for foreground segmentation. Its primary purpose is to
help researchers develop new gait and facial biometrics recognition methods.
Moreover, the dataset is an aggregate composed of two datasets, described as
follows:

  * •

Dataset 1 (Kale et al., 2002): composed of walking sequences of $25$
individuas in $4$ distinct poses:

    1. (1)

Frontal view/walking-toward;

    2. (2)

Frontal view/walking-away;

    3. (3)

Frontal-parallel view/toward left; and

    4. (4)

Frontal-parallel view/toward right.

  * •

Dataset 2 (Cuntoor et al., 2003): comprises videos from $55$ individuals
walking through a T-shape pathway. The sequences were obtained by two cameras
that line orthogonal to each other.

More details can be found at http://www.umiacs.umd.edu/labs/pirl/hid/umd-
eval.html. Further, the dataset is available to download through FTP,
requiring credentials that can be requested at
http://www.umiacs.umd.edu/labs/pirl/hid/data.html.

###  4.4. CASIA

The Institute of Automation, Chinese Academy of Sciences (CASIA) provides the
CASIA Gait Database (Zheng et al., 2011), a collection of four datasets
designed for gait recognition purposes, and described as follows:

  * •

CASIA A: Created in December 2001 and formerly known as the NLPR Gait Database
(Wang et al., 2003), the CASIA A dataset includes 20 individuals, each of them
comprising $12$ videos, i.e., $4$ videos for each of the $3$ directions,
namely parallel, $45$ and $90$ degrees to the image plane. Besides, each image
sequence possesses its duration, varying with the individual walking velocity.
Figure 8 shows some image examples from different angles. The dataset’s total
size is approximately 2.2GB;

  * •

CASIA B: Created in 2005, the CASIA B (Yu et al., 2006) comprises filming
$124$ individuals from $11$ different angles. Each sequence was repeated three
times, with variations such as clothing and walking speed. Moreover, the
dataset also comprises a set of silhouettes provided for all sequences,
provided for foreground segmentation. Figures 9 and 10 show variations in
angle and clothing.

  * •

CASIA C: Acquired in 2005, the CASIA C (Tan et al., 2006) dataset contains
$153$ subjects filmed by infrared cameras (thermal spectrum) in four different
variations: normal walking, slow walking, fast walking, and normal walking
carrying a backpack. Figure 11 show some examples. All images were captured at
night;

  * •

CASIA D: The CASIA gait–footprint dataset (Zheng et al., 2012) contains both
images and cumulative foot pressure information. The dataset comprises $3,496$
gait pose images and $2,658$ cumulative foot pressure images from $88$
individuals with a broad distribution of age, $20$ female and $68$ male, in an
indoor environment. Figure 12 illustrates the data acquisition schematics.

Silhouette for datasets A, B, and C are freely available for
download555http://www.cbsr.ia.ac.cn/english/Gait%20Databases.asp. Regarding
data acquisition, candidates should fill a form and wait for the dataset
owner’s approval.

|
---|---
(a) | (b)

---
(c)

Figure 8. Example frames from CASIA A dataset from each angle. (a) corresponds
to the parallel view, (b) corresponds to $90$ degrees, and (c) is an example
from $45$ degrees. Images collectd from
http://www.cbsr.ia.ac.cn/users/szheng/?page_id=71. Figure 9. Example frames
from CASIA B dataset. Each image corresponds to one of the $11$ comprised
angles. Extracted from http://www.cbsr.ia.ac.cn/users/szheng/?page_id=71.

|
---|---
(a) | (b)

---
(c)

Figure 10. Example frames from CASIA B dataset. One can notice the change in
clothes and personal objects, like backpacks. Images collectd from
http://www.cbsr.ia.ac.cn/users/szheng/?page_id=71.

(a) (b)

Figure 11. Example frames from CASIA C dataset. The images were obtained
through an infrared camera at night and with variations in the manner of
walking. Adapted from http://www.cbsr.ia.ac.cn/users/szheng/?page_id=71.
Figure 12. Example frames from CASIA D dataset. The images illustrate the
image’s acquisition process synchronously with the footsteps data. Image
collectd from http://www.cbsr.ia.ac.cn/users/szheng/?page_id=71.

###  4.5. OU-ISIR Biometric Database

The Institute of Scientific and Industrial Research (ISIR) of Osaka University
(OU) has been creating, since 2007, the largest dataset for gait recognition
in the world. The project is an aggregation of eight different groups:

  * •

Treadmill Dataset: The group is composed of sequences from people walking on
electronic treadmills surrounded by $25$ cameras filming at $60$ frames per
second at a resolution of $640\times 480$. It has $4$ subdivisions:

    1. (1)

Treadmill dataset A – Speed variation (Tsuji et al., 2010): this subset
comprises $34$ subjects in a lateral vision with speed varying between $2$ and
$10$km/h in $1$hm/h intervals;

    2. (2)

Treadmill dataset B – Clothes variation (Hossain et al., 2010): It is composed
of $68$ people in lateral vision with $32$ clothing variations;

    3. (3)

Treadmill dataset C – View variation (Makihara et al., 2010): A large-scale
database comprising $168$ people with ages ranging from $4$ to $75$ years old.
Moreover, the subset is composed of 25 views observed through a multi-view
synchronous gait system;

    4. (4)

Treadmill dataset D – Gait fluctuation (Mori et al., 2010): this set is
composed of gait silhouette sequences with $185$ subjects, viewed from a
lateral angle and with variations in velocity. The data were subdivided into
two groups of $100$ subjects (with an overlap of 15 people) by high and low
speed variations.

  * •

Large Population Dataset (Iwama et al., 2012): Collected since 2009 through
outreach activity events, the Large Population Dataset is composed of $4,016$
subjects, each of them filmed twice from $4$ camera angles at $30$ FPS and a
resolution of $640\times 480$ pixels.

  * •

Speed Transition Dataset (Mansur et al., 2014): The Speed Transition Dataset
comprises two subsets, described as follows:

    1. (1)

Dataset A: It contains $179$ scenes from people walking at a constant velocity
of $4$km/h on a treadmill or the ground. In this set, the background has been
removed using a wallpaper;

    2. (2)

Dataset B: It comprises sequences from $25$ people walking on a treadmill with
a velocity varying between $1$ and $5$ km/h. Each person is filmed twice.
Acceleration and deacceleration are performed in three seconds, and middle
sequences with one second were extracted from both.

  * •

Multi-view Large Population Dataset (Takemura et al., 2018): This dataset is
composed of $10,307$ samples such that $5,114$ regards men and the remaining
$5,193$ stand for women, whose age ranges from $2$ to $87$ years, developed
for motion recognition methods with cross-vision. The images were filmed in
$14$ different angles, at a frame rate of $25$ frames per second and a
resolution of $1280\times 980$. The devices employed for capture were placed
at a lateral distance and height of $8$ and $5$ meters, respectively.

  * •

Large Population Dataset with Bag (Uddin et al., 2018): The dataset focuses on
gait recognition concerning people carrying objects, aiming not only to rely
on biometrics information but also on identifying the position of the
transported part (if any) regarding the body. The Large Population Dataset
with Bag comprises $62,528$ people aged from $2$ to $95$ years obtained
through a camera at a distance of approximately $8$ meters and $5$ meters
height. The sequences were filmed at $25$ frames per second with a resolution
of $1280\times 980$ pixels. Each person was filmed three times, such that the
first, i.e., A1, is carrying or not an object, while the second and third do
not bring anything. Finally, a total of four regions are marked in case of
something being carried, i.e., lower side, upper side, front, and back. All
videos are also presented with a respective binary mask for background
removal.

  * •

Large Population Dataset with Age (Xu et al., 2017): The Large Population
Dataset with Age was created to investigate gait recognition concerning
people’s age and gender. The dataset comprises $62,846$ individuals walking on
a particular path with cameras capturing $640\times 480$ pixels’ resolution at
a rate of $30$ frames per second. The sequences’ people are between $2$ and
$90$ years old, and all videos have binary masks obtained after the background
removal.

  * •

Inertial Sensor Dataset (Ngo et al., 2014): Designated for research and
evaluation of methods of individual identification by movements through motion
sensors and accelerometers, the Inertial Sensor Dataset is the largest
inertial sensor-based gait database, composed of images collected from $744$
subjects ($389$ males and $355$ females) whose ages range from $2$ to $78$
years.

  * •

Similar Actions Inertial Dataset (Ngo et al., 2015): The Similar Actions
Inertial Dataset comprises $460$ participants aged between $8$ and $78$ with
gender virtually equal distributed, whose walking properties were obtained
together with the data presented in (Ngo et al., 2014). Additionally, this
dataset also presents six distinct characteristics of the floor: invalid,
flat, stair climbing, stair climbing, ramp climbing, and ramp descent.

###  4.6. University of South Florida Dataset

The University of South Florida (USF) dataset (Sarkar et al., 2005) comprises
$1,870$ sequences from $122$ subjects using two different shoe types. The
dataset also considers individuals carrying or not a briefcase, diverse
surface conditions such as grass and concrete, and distinct camera views,
i.e., left or right viewpoints. The videos are captured at two different time
instants filmed in outdoor environments

###  4.7. Southampton Dataset

The Southampton Human ID at a Distance (SOTON) database is a contribution of
the University of Southampton composed of three major segments666More
information available at http://www.eng.usf.edu/cvprg/Gait_Data.html.:

  * •

SOTON Small database (Nixon et al., 2001): Comprises $12$ subjects walking
around an inside track at varying speeds, wearing different shoes and clothes,
and carrying or not bags;

  * •

SOTON Large database (Shutler et al., 2004): Containing $114$ subjects walking
outside, inside on the laboratory track, and inside in a treadmill. Images
were filmed from six different angles and provided in a collection with more
than $5,000$ sequences.

  * •

SOTON Temporal (Matovski et al., 2010): The data was captured using a Multi-
Biometric Tunnel, which contains $12$ synchronized cameras to capture people’s
gait over time. The dataset is composed of dynamic environments, comprising
distinct background, lighting, walking surface, and position of cameras. The
dataset includes $25$ subjects ($17$ male and $8$ female) with ages ranging
from $20$ to $55$ years old. Notice they are all filmed barefoot.

###  4.8. AVA Multi-View Dataset for Gait Recognition (AVAMVG)

The AVA Multi-View Dataset for Gait Recognition (AVAMVG)777More information
at: http://www.uco.es/grupos/ava/node/41 (David López-Fernández, Francisco J.
Madrid-Cuevas, Ángel Carmona-Poyato, Manuel J. Marín-Jiménez and Rafael Muñoz-
Salinas, 2014) is a database specifically designed for 3D-based gait
recognition algorithms, which comprises gait images from $20$ actors depicting
different trajectories. The sequences were obtained using cameras specifically
calibrated for the task, followed by a post-processing step using 3D image
reconstruction algorithms. Besides, each sequence is also provided with a
respective binary silhouette for segmentation. Finally, the database contains
$200$ six-channel multi-view videos that can also be employed as $1,200$
single view videos, i.e., $6\times 200$. Figure 13 depicts some samples.

Figure 13. Example of the multiview dataset. The image presents people walking
in different directions, from multiple points of view. Adapted from
https://www.uco.es/investiga/grupos/ava/node/41.

###  4.9. Kyushu University 4D Gait Database

The Kyushu University 4D Gait Database (KY4D)888Available at
http://robotics.ait.kyushu-u.ac.jp/~yumi/db.html (Iwashita et al., 2014b) is
composed of sequential 3D models and image sequences of $42$ subjects walking
along four straight and two curved trajectories. The videos were recorded by
$16$ cameras, at a resolution of $1032\times 776$ pixels, and divided into
three subsets, described as follows:

  * •

Dataset A (Straight): It is composed of sequential 3D models and image
sequences of people walking along straight trajectories. Figure 14(a)
illustrates the trajectory, described as a red arrow. Figure 14(b) depicts
multiple 3D reconstructed models.

|
---|---
(a) | (b)
Figure 14. The studio from KY4D Gait Database A: (a) describes the trajectory
indicated by the arrow and (b) depicts sequential 3D models of a person
walking straight. Images adapted from
http://robotics.ait.kyushu-u.ac.jp/~yumi/db/gait_b.html.

  * •

Database B (Curve): It comprises image sequences from people walking along
curved trajectories, as depicted in Figure 15(a). The radius r varies either
$1.5$m or $3.0$m. Figure 15(b) depicts multiple 3D reconstructed models
considering the curve path.

|
---|---
(a) | (b)
Figure 15. The studio from KY4D Gait Database B: (a) describes the radius
variation and (b) illustrates sequential 3D models of a person walking along a
curved trajectory. Images adapted from
http://robotics.ait.kyushu-u.ac.jp/~yumi/db/gait_b.html.

  * •

KY Infrared (IR) Shadow Gait Database: It is composed of time-series shadow
images of $54$ subjects. As indicated by the walking direction arrow in Figure
16(a), all people walk straight. Figure 16(b) depicts two infrared lights and
a camera employed to collect the shadow database (Iwashita et al., 2014a). The
infrared lights were placed obliquely, and the camera was placed on the
ceiling perpendicular to the ground. A sample result from such captures is
observed in Figure 17.

Figure 16. (a) Experimental setting, (b) actual scene. Adapted from
http://robotics.ait.kyushu-u.ac.jp/~yumi/db/gait_b.html. Figure 17. An example
movie from IR shadow images. Image collectd from
http://robotics.ait.kyushu-u.ac.jp/~yumi/db/gait_b.html.

###  4.10. WhuGAIT Datasets

The whuGAIT Datasets (Zou et al., 2018a) were released in 2018 by the Wuhan
University and made available along with the source code and pre-trained
models to replicate the paper results999Available at
https://github.com/qinnzou/Gait-Recognition-Using-Smartphones. Unlike other
data collections, whuGAIT comprises $3$D accelerometers and $3$-axis gyroscope
information collected from $118$ people, $20$ of them collected during three
days, and $98$ collected in just one day. The dataset is divided into $6$
different sub-sets, according to the desired task:

  * •

Dataset #1: composed of $33,104$ samples for training and $3,740$ for testing
from $118$ individuals, divided into a two-step segmentation.

  * •

Dataset #2: similar to Dataset #1, comprises a two-step segmentation dataset
composed of $49,275$ samples for training and $4,936$ for testing, extracted
from the $3$ days collected data of $20$ people.

  * •

Dataset #3: this subset is divided into time size windows, comprising $2.56$
seconds for each sample. The set consists of $26,283$ instances used for
training and $2,991$ employed for testing purposes.

  * •

Dataset #4: similar to Dataset #3, this subset is divided into time frames of
$2.56$ seconds, but using the data from $20$ individuals collected during
three days. The subset comprises $35,373$ for training and $3,941$ for testing
purposes.

  * •

Dataset #5: the subset is employed for authentication purposes. It is composed
of $74,142$ instances from $118$ people, such that information extracted from
$98$ individuals is utilized for training, while the remaining $20$ are used
for validation. The authentication procedure is compounded by a pair of
samples from one or two different subjects. The instances comprise two-step
acceleration and gyroscopic data.

  * •

Dataset #6: this subset employs the same structure used in Dataset #5, but
instead of horizontal align, it uses vertical alignment.

###  4.11. Datasets in Usage Contexts

This section presented the datasets most commonly employed for gait
recognition and used in the works considered in this survey. Thus, in the
sequence, we also provide an overview regarding particular aspects, such as
usage contexts, acquisition environments, and spectrum. Table 3 shows such
information for each dataset. The covariates were divided into twelve main
features, i.e., viewpoint, pace, objects, shoe, clothing, time, surface,
silhouette, gait fluctuation, treadmill walking, overground walking, and foot
pressure.

Table 3. Datasets in Usage Contexts. Context/Dataset | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 4.6 | 4.6 | 4.8 | 4.9 | 4.10
---|---|---|---|---|---|---|---|---|---|---
Viewpoint | x |  | x | x | x | x | x | x | x | x
Pace | x |  |  | x | x | x | x |  |  | x
Objects | x | x |  | x |  |  | x |  |  |
Shoe | x | x | x | x | x | x | x | x |  |
Clothing | x | x | x | x | x |  | x | x |  |
Time |  |  |  | x | x |  | x |  | x | x
Surface | x | x | x | x |  | x | x |  | x | x
Silhouette |  | x |  | x | x |  |  |  | x |
Gait Fluctuation |  |  |  |  | x |  |  |  |  |
Treadmill walking | x |  |  |  | x |  | x |  |  |
Overground walking | x | x | x | x |  | x |  | x | x | x
Foot pressure |  |  |  | x |  |  |  |  |  |

Further, Figure 18 presents the datasets’ construction environments, which is
divided into four scenarios, i.e., static indoor, static outdoor, active
indoor, and active outdoor.

Figure 18. Representation of the dataset development environment.

Finally, Figure 19 provides an illustrated schema categorizing the datasets’
spectrum representation into two main classes, i.e., color-based and thermal
information.

Figure 19. Data spectrum representation of the datasets.

##  5\. Conclusions and Future Directions

This survey provided an in-depth research towards the most significant works
developed in the last years regarding gait recognition, highlighting deep
learning techniques for such a task. Besides, it also provided a historical
background concerning both methods for biometric identification, such as
fingerprint, iris, and face, among others, as well as gait recognition,
exposing the main concerns and challenges faced by the field.

Further, it provided a detailed description of the nine most employed datasets
for the task, i.e., CMU MoBo (Gross and Shi, 2001), TUM GAID (Hofmann et al.,
2014), HID-UMD (Kale et al., 2002; Cuntoor et al., 2003), CASIA (Wang et al.,
2003; Yu et al., 2006; Tan et al., 2006; Zheng et al., 2012), OU-ISIR (Tsuji
et al., 2010; Hossain et al., 2010; Makihara et al., 2010; Mori et al., 2010;
Iwama et al., 2012; Mansur et al., 2014; Takemura et al., 2018; Uddin et al.,
2018; Xu et al., 2017; Ngo et al., 2014, 2014), USF (Sarkar et al., 2005),
SOTON (Nixon et al., 2001; Shutler et al., 2004; Matovski et al., 2010),
AVAMVG (David López-Fernández, Francisco J. Madrid-Cuevas, Ángel Carmona-
Poyato, Manuel J. Marín-Jiménez and Rafael Muñoz-Salinas, 2014), KY4D
(Iwashita et al., 2014b), and WhuGait (Zou et al., 2018a) as well as their
extraction process and limitations. Regarding such datasets, one can clearly
notice the efforts towards an detection independent from gender, angle in the
scene, clothes and shoes, carry of bags and other itens, among other common
issues.

The works surveyed in this paper discuss the main concerns and efforts made by
the scientific community towards the development of techniques to guarantee
access to monitored resources via gait identification and authorization. It
also explores the advantages of a gait-based system against commonly employed
biometric features, e.g., gait systems do not require the individuals’
willingness to be identified. In this sense, the review attempted to point out
the advantages of gait biometrics compared to other biometric techniques and
highlight the most recent methodologies and state-of-the-art architectures.

There are a few critical points about video-based models and databases that
are worth highlighting:

  * •

The presented datasets do not introduce more than one person per video, either
for training or validation purposes.

  * •

The environments in which they were recorded are fully controlled. Despite the
variation of angles in some sets, it is possible to notice no variation of
objects or even of colors in the background. Others videos were recorded with
people walking on electric treadmills, denoting even greater control of the
movements and the environment.

  * •

Consequently, the presented models cited in the work are probably prone to
show lack and misbehavior when applied to the real-world operation, such as
detecting people of interest walking on public streets.

Regarding future trends, we expect an increase in the number of works related
to:

  1. (1)

Transformer networks: the technique (Vaswani et al., 2017) targets data
streaming such as video and fits the context of gait recognition perfectly.

  2. (2)

Gender and age recognition: such an approach (Sun et al., 2019) has gained
popularity in the last years due to its importance in several applications.

  3. (3)

Hazardous environment monitoring: gait features fit perfectly the purpose of
monitoring hazardous environments since they usually are composed of limited
illumination, and extracting more descriptive biometric identification
characteristics denotes a difficult task (Santana et al., 2019).

  4. (4)

Multiple people in the scene: most of the gait recognition works focus on a
single individual in the scene in controlled environments. However, real-life
problems usually require solutions robust to non-controlled environments with
multiple people in the scene.

Additionally, we observed an increasing demand for gait recognition on
individuals with different clothes or carrying objects (Yu et al., 2017), as
well as hybrid approaches composed of both gait and complementary biometric
characteristics, such as face and ear (Mehraj and Mir, 2020).

###### Acknowledgements.

The authors are grateful to São Paulo Research Foundation (Fapesp) grants
#2013/07375-0, #2014/12236-1, #2019/07665-4, and #2020/12101-0, to the
Brazilian National Council for Research and Development (CNPq) #307066/2017-7
and #427968/2018-6, Eldorado Research Institute, as well as Petroleo
Brasileiro S.A. (Petrobras) grant #2017/00285-6.

## References

  * (1)
  * Alariki et al. (2018) Ala Abdulhakim Alariki, Muqadas Faiz, Sanaullah Balagh, and Christine Murray. 2018.  A Review of Finger-vein Biometric Recognition. In _Proceedings of the World Congress on Engineering and Computer Science_ , Vol. 1.
  * Alharthi et al. (2019) Abdullah S Alharthi, Syed U Yunas, and Krikor B Ozanyan. 2019\.  Deep learning for monitoring of human gait: A review.  _IEEE Sensors Journal_ 19, 21 (2019), 9575–9591.
  * Alsmirat et al. (2019) Mohammad A Alsmirat, Fatimah Al-Alem, Mahmoud Al-Ayyoub, Yaser Jararweh, and Brij Gupta. 2019\.  Impact of digital fingerprint image quality on the fingerprint recognition accuracy.  _Multimedia Tools and Applications_ 78, 3 (2019), 3649–3688.
  * Arora et al. (2015) P. Arora, M. Hanmandlu, and S. Srivastava. 2015.  Gait based authentication using gait information image features.  _Pattern Recognition Letters_ 68 (2015), 336–342.
  * Babaee et al. (2019) Maryam Babaee, Linwei Li, and Gerhard Rigoll. 2019.  Person identification from partial gait cycle using fully convolutional neural networks.  _Neurocomputing_ (2019).
  * Bolle et al. (2013) Ruud M Bolle, Jonathan H Connell, Sharath Pankanti, Nalini K Ratha, and Andrew W Senior. 2013\.  _Guide to biometrics_.  Springer Science & Business Media.
  * Bouchrika (2018) Imed Bouchrika. 2018\.  A survey of using biometrics for smart visual surveillance: Gait recognition.  In _Surveillance in Action_. Springer, 3–23.
  * Chen et al. (2020) Rung-Ching Chen, Christine Dewi, Wei-Wei Zhang, Jia-Ming Liu, Su-Wen Huang, et al. 2020\.  Integrating gesture control board and image recognition for gesture recognition based on deep learning.  _International Journal of Applied Science and Engineering_ 17, 3 (2020), 237–248.
  * Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014.  On the properties of neural machine translation: Encoder-decoder approaches.  _arXiv preprint arXiv:1409.1259_ (2014).
  * Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_ (2014).
  * Costilla-Reyes et al. (2018) Omar Costilla-Reyes, Ruben Vera-Rodriguez, Patricia Scully, and Krikor B Ozanyan. 2018.  Analysis of spatio-temporal representations for robust footstep recognition with deep residual neural networks.  _IEEE transactions on pattern analysis and machine intelligence_ 41, 2 (2018), 285–296.
  * Cuntoor et al. (2003) Naresh Cuntoor, Amit Kale, and Rama Chellappa. 2003\.  Combining multiple evidences for gait recognition. In _2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03)._ , Vol. 3. IEEE, III–33.
  * Dargan and Kumar (2020) Shaveta Dargan and Munish Kumar. 2020.  A comprehensive survey on the biometric recognition systems based on physiological and behavioral modalities.  _Expert Systems with Applications_ 143 (2020), 113114.
  * David López-Fernández, Francisco J. Madrid-Cuevas, Ángel Carmona-Poyato, Manuel J. Marín-Jiménez and Rafael Muñoz-Salinas (2014) David López-Fernández, Francisco J. Madrid-Cuevas, Ángel Carmona-Poyato, Manuel J. Marín-Jiménez and Rafael Muñoz-Salinas. 2014.  The AVA Multi-View Dataset for Gait Recognition.  In _Activity Monitoring by Multiple Distributed Sensing_ , Pier Luigi Mazzeo, Paolo Spagnolo, and Thomas B. Moeslund (Eds.). Springer International Publishing, 26–39.  https://doi.org/10.1007/978-3-319-13323-2_3
  * de Souza et al. (2021) Renato WR de Souza, Daniel S Silva, Leandro A Passos, Mateus Roder, Marcos C Santana, Plácido R Pinheiro, and Victor Hugo C de Albuquerque. 2021.  Computer-Assisted Parkinson’s Disease Diagnosis Using Fuzzy Optimum-Path Forest and Restricted Boltzmann Machines.  _Computers in Biology and Medicine_ (2021), 104260.
  * Deng et al. (2018a) Muqing Deng et al. 2018a.  Gait recognition under different clothing conditions via deterministic learning.  _in IEEE/CAA Journal of Automatica Sinica_ (2018).
  * Deng et al. (2018b) Weihong Deng, Jiani Hu, and Jun Guo. 2018b.  Face recognition via collaborative representation: Its discriminant nature and superposed representation.  _IEEE transactions on pattern analysis and machine intelligence_ 40, 10 (2018), 2513–2521.
  * Deng et al. (2014) Weihong Deng, Jiani Hu, Jiwen Lu, and Jun Guo. 2014\.  Transform-invariant PCA: A unified approach to fully automatic facealignment, representation, and recognition.  _IEEE transactions on pattern analysis and machine intelligence_ 36, 6 (2014), 1275–1284.
  * Dewi and Chen (2019) Christine Dewi and Rung-Ching Chen. 2019.  Human activity recognition based on evolution of features selection and random Forest. In _2019 IEEE international conference on systems, man and cybernetics (SMC)_. IEEE, 2496–2501.
  * e MS Nixon (2011) G. Ariyanto e MS Nixon. 2011\.  Model-based 3D gait biometrics.  _in IJCB_ (2011).
  * Fernandes et al. (2018) Carlos Fernandes, Luís Fonseca, Flora Ferreira, Miguel Gago, Lus Costa, Nuno Sousa, Carlos Ferreira, Joao Gama, Wolfram Erlhagen, and Estela Bicho. 2018.  Artificial neural networks classification of patients with parkinsonism based on gait. In _2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_. IEEE, 2024–2030.
  * Garagad and Iyer (2014) Vishwanath G Garagad and Nalini C Iyer. 2014.  A novel technique of iris identification for biometric systems. In _2014 International Conference on Advances in Computing, Communications and Informatics (ICACCI)_. IEEE, 973–978.
  * Goodfellow et al. (2014) Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014\.  Generative adversarial networks.  _arXiv preprint arXiv:1406.2661_ (2014).
  * Gross and Shi (2001) Ralph Gross and Jianbo Shi. 2001.  _The CMU Motion of Body (MoBo) Database_.  Technical Report CMU-RI-TR-01-18. Carnegie Mellon University.
  * Gui et al. (2019) Qiong Gui, Maria V Ruiz-Blondet, Sarah Laszlo, and Zhanpeng Jin. 2019.  A survey on brain biometrics.  _ACM Computing Surveys (CSUR)_ 51, 6 (2019), 1–38.
  * Han and Bhanu (2016) Ju Han and Bir Bhanu. 2016\.  Individual Recognition UsingGait Energy Image.  _IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE_ (2016).
  * He et al. (2020) Jing Selena He, Mingyuan Yan, and Sahil Arora. 2020.  Long Range Iris Recognition a Reality or a Myth? A Survey. In _Proceedings of the 2020 ACM Southeast Conference_. 305–306.
  * He et al. (2018) Yiwei He, Junping Zhang, Hongming Shan, and Liang Wang. 2018\.  Multi-task GANs for view-specific feature learning in gait recognition.  _IEEE Transactions on Information Forensics and Security_ 14, 1 (2018), 102–113.
  * Hemalatha (2020) S Hemalatha. 2020\.  A systematic review on Fingerprint based Biometric Authentication System. In _2020 International Conference on Emerging Trends in Information Technology and Engineering (ic-ETITE)_. IEEE, 1–4.
  * Hinton (2002) G. E. Hinton. 2002\.  Training Products of Experts by Minimizing Contrastive Divergence.  _Neural Computation_ 14, 8 (2002), 1771–1800.
  * Hinton et al. (2006) G. E. Hinton, S. Osindero, and Y.-W. Teh. 2006\.  A Fast Learning Algorithm for Deep Belief Nets.  _Neural Computation_ 18, 7 (2006), 1527–1554.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Hofmann et al. (2014) Martin Hofmann, Jürgen Geiger, Sebastian Bachmann, Björn Schuller, and Gerhard Rigoll. 2014\.  The tum gait from audio, image and depth (gaid) database: Multimodal recognition of subjects and traits.  _Journal of Visual Communication and Image Representation_ 25, 1 (2014), 195–206.
  * Hossain et al. (2010) Md Altab Hossain, Yasushi Makihara, Junqiu Wang, and Yasushi Yagi. 2010.  Clothing-invariant gait identification using part-based clothing categorization and adaptive weight control.  _Pattern Recognition_ 43, 6 (2010), 2281–2291.
  * Hu et al. (2018) BingZhang Hu, Yu Guan, Yan Gao, Yang Long, Nicholas Lane, and Thomas Ploetz. 2018\.  Robust Cross-View Gait Recognition with Evidence: A Discriminant Gait GAN (DiGGAN) Approach.  _arXiv preprint arXiv:1811.10493_ (2018).
  * Iwama et al. (2012) H. Iwama, M. Okumura, Y. Makihara, and Y. Yagi. 2012\.  The OU-ISIR Gait Database Comprising the Large Population Dataset and Performance Evaluation of Gait Recognition.  _IEEE Trans. on Information Forensics and Security_ 7, Issue 5 (Oct. 2012), 1511–1521.
  * Iwashita et al. (2014a) Y. Iwashita, R. Kurazume, and A. Stoica. 2014a.  Gait Identification Using Invisible Shadows: Robustness to Appearance Changes. In _Int. Conf. Emerging Security Technologies (EST)_. UK.
  * Iwashita et al. (2014b) Y. Iwashita, K. Ogawara, and R. Kurazume. 2014b.  Identification of people walking along curved trajectories. In _Pattern Recognition Letters_.
  * Jain et al. (2004) Anil K Jain et al. 2004\.  An Introduction to Biometric Recognition.  _in IEEE Transactions on Circuits and Systems for Video Technology, Special Issue on Image- and Video-Based Biometrics, Vol. 14, No. 1_ (2004).
  * Jayaraman et al. (2020) Umarani Jayaraman, Phalguni Gupta, Sandesh Gupta, Geetika Arora, and Kamlesh Tiwari. 2020\.  Recent development in face recognition.  _Neurocomputing_ 408 (2020), 231–245.
  * Jia et al. (2019) Meijuan Jia, Hongyu Yang, Di Huang, and Yunhong Wang. 2019\.  Attacking gait recognition systems via silhouette guided GANs. In _Proceedings of the 27th ACM International Conference on Multimedia_. 638–646.
  * Jing Luo and Xiu (2015) Chunyuan Zi Ying Niu Huixin Tian Jing Luo, Jianliang Zhang and Chunbo Xiu. 2015\.  Gait Recognition Using GEI and AFDEI.  _International Journal of Optics_ (2015).
  * Jun et al. (2020) K. Jun, D. Lee, K. Lee, S. Lee, and M. S. Kim. 2020.  Feature Extraction Using an RNN Autoencoder for Skeleton-Based Abnormal Gait Recognition.  _IEEE Access_ 8 (2020), 19196–19207.
  * Kale et al. (2002) Amit Kale, Naresh Cuntoor, and Rama Chellappa. 2002\.  A framework for activity-specific human identification. In _2002 IEEE International Conference on Acoustics, Speech, and Signal Processing_ , Vol. 4. IEEE, IV–3660.
  * Kan et al. (2014) Meina Kan, Shiguang Shan, Hong Chang, and Xilin Chen. 2014\.  Stacked progressive auto-encoders (spae) for face recognition across poses. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 1883–1890.
  * Karn et al. (2020) Pradeep Karn, XiaoHai He, Jin Zhang, and Yanteng Zhang. 2020\.  An experimental study of relative total variation and probabilistic collaborative representation for iris recognition.  _Multimedia Tools and Applications_ 79, 43 (2020), 31783–31801.
  * Khan et al. (2020) Saad Khan, Simon Parkinson, Liam Grant, Na Liu, and Stephen Mcguire. 2020.  Biometric systems utilising health data from wearable devices: applications and future challenges in computer security.  _ACM Computing Surveys (CSUR)_ 53, 4 (2020), 1–29.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012\.  Imagenet classification with deep convolutional neural networks. In _Advances in neural information processing systems_. 1097–1105.
  * LeCun et al. (1998) Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. 1998\.  Gradient-based learning applied to document recognition.  _Proc. IEEE_ 86, 11 (1998), 2278–2324.
  * Lee et al. (2017) Wonjune Lee, Sungchul Cho, Heeseung Choi, and Jaihie Kim. 2017\.  Partial fingerprint matching using minutiae and ridge shape features for small fingerprint scanners.  _Expert Systems with Applications: An International Journal_ 87, C (2017), 183–198.
  * Lei et al. (2014) Zhen Lei, Matti Pietikäinen, and Stan Z Li. 2014\.  Learning discriminant face descriptor.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 36, 2 (2014), 289–302.
  * Li et al. (2017) C. Li, X. Min, S. Sun, W. Lin, and Z. Tang. 2017.  Deepgait: A learning deep convolutional representation for view-invariant gait recognition using joint bayesian.  _Applied Sciences_ 7, 3 (2017), 210.
  * Li et al. (2021) Shuyi Li, Bob Zhang, Lunke Fei, and Shuping Zhao. 2021\.  Joint discriminative feature learning for multimodal finger recognition.  _Pattern Recognition_ 111 (2021), 107704.
  * Li (2009) Stan Z Li. 2009\.  _Encyclopedia of Biometrics: I-Z._ Vol. 2.  Springer Science & Business Media.
  * Lobo et al. (2020) V. C. Lobo, L. A. Passos, and J. P. Papa. 2020.  Evolving Long Short-Term Memory Networks. In _International Conference on Computational Science (ICCS)_. IEEE.
  * Luo and Tjahjadi (2020) Jian Luo and Tardi Tjahjadi. 2020.  Multi-Set Canonical Correlation Analysis for 3D Abnormal Gait Behaviour Recognition Based on Virtual Sample Generation.  _IEEE Access_ 8 (2020), 32485–32501.
  * Luo et al. (2016) Zhengping Luo, Tianqi Yang, and Yanjun Liu. 2016.  Gait optical flow image decomposition for human recognition. In _2016 IEEE Information Technology, Networking, Electronic and Automation Control Conference_. IEEE, 581–586.
  * Ma et al. (2019) Zhuo Ma, Yilong Yang, Ximeng Liu, Yang Liu, Siqi Ma, Kui Ren, and Chang Yao. 2019.  Emir-auth: Eye-movement and iris based portable remote authentication for smart grid.  _IEEE Transactions on Industrial Informatics_ (2019).
  * Makihara et al. (2012) Y. Makihara, H. Mannami, A. Tsuji, M.A. Hossain, K. Sugiura, A. Mori, and Y. Yagi. 2012.  The OU-ISIR Gait Database Comprising the Treadmill Dataset.  _IPSJ Trans. on Computer Vision and Applications_ 4 (Apr. 2012), 53–62.
  * Makihara et al. (2010) Yasushi Makihara, Hidetoshi Mannami, and Yasushi Yagi. 2010\.  Gait analysis of gender and age using a large-scale multi-view gait database. In _Asian Conference on Computer Vision_. Springer, 440–451.
  * Mansur et al. (2014) Al Mansur, Yasushi Makihara, Rasyid Aqmar, and Yasushi Yagi. 2014.  Gait recognition under speed transition. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 2521–2528.
  * Marsico and Mecca (2019) Maria De Marsico and Alessio Mecca. 2019.  A survey on gait recognition via wearable sensors.  _ACM Computing Surveys (CSUR)_ 52, 4 (2019), 1–39.
  * Matovski et al. (2010) Darko S Matovski, Mark S Nixon, Sasan Mahmoodi, and John N Carter. 2010.  The effect of time on the performance of gait biometrics. In _2010 Fourth IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS)_. IEEE, 1–6.
  * Mehraj and Mir (2020) Haider Mehraj and Ajaz Hussain Mir. 2020.  Feature vector extraction and optimisation for multimodal biometrics employing face, ear and gait utilising artificial neural networks.  _International Journal of Cloud Computing_ 9, 2-3 (2020), 131–149.
  * Mitchell (1920) C. Ainsworth Mitchell. 1920\.  The detection of finger-prints on documents.  _Analyst_ 45, 529 (1920), 122–129.
  * Mori et al. (2010) Atsushi Mori, Yasushi Makihara, and Yasushi Yagi. 2010\.  Gait recognition using period-based phase synchronization for low frame-rate videos. In _2010 20th International Conference on Pattern Recognition_. IEEE, 2194–2197.
  * Nambiar et al. (2019) Athira Nambiar, Alexandre Bernardino, and Jacinto C Nascimento. 2019\.  Gait-based person re-identification: A survey.  _ACM Computing Surveys (CSUR)_ 52, 2 (2019), 1–34.
  * Ngo et al. (2014) Thanh Trung Ngo, Yasushi Makihara, Hajime Nagahara, Yasuhiro Mukaigawa, and Yasushi Yagi. 2014\.  The largest inertial sensor-based gait database and performance evaluation of gait-based personal authentication.  _Pattern Recognition_ 47, 1 (2014), 228–237.
  * Ngo et al. (2015) Trung Thanh Ngo, Yasushi Makihara, Hajime Nagahara, Yasuhiro Mukaigawa, and Yasushi Yagi. 2015\.  Similar gait action recognition using an inertial sensor.  _Pattern Recognition_ 48, 4 (2015), 1289–1301.
  * Nixon et al. (2001) M Nixon, J Carter, J Shutler, and M Grant. 2001\.  Experimental plan for automatic gait recognition.  _University of Southampton, Southampton, UK_ (2001).
  * Nixon and Carter (2004) Mark S Nixon and John N Carter. 2004.  Advances in automatic gait recognition. In _null_. IEEE, 139.
  * Oppenheim et al. (2001) Alan V Oppenheim, John R Buck, and Ronald W Schafer. 2001\.  _Discrete-time signal processing. Vol. 2_.  Upper Saddle River, NJ: Prentice Hall.
  * Passos et al. (2017) Leandro A. Passos, Kelton A.P. Costa, and João P. Papa. 2017.  Deep Boltzmann Machines Using Adaptive Temperatures. In _International Conference on Computer Analysis of Images and Patterns_. Springer, 172–183.
  * Passos et al. (2019) L. A. Passos, M. C. Santana, T. Moreira, and J. P. Papa. 2019\.  $\kappa$-Entropy Based Restricted Boltzmann Machines. In _The 2019 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 1–8.
  * Pisani et al. (2019) Paulo Henrique Pisani, Abir Mhenni, Romain Giot, Estelle Cherrier, Norman Poh, André Carlos Ponce de Leon Ferreira de Carvalho, Christophe Rosenberger, and Najoua Essoukri Ben Amara. 2019.  Adaptive biometric systems: Review and perspectives.  _ACM Computing Surveys (CSUR)_ 52, 5 (2019), 1–38.
  * Potluri et al. (2019) Sasanka Potluri, Srinivas Ravuri, Christian Diedrich, and Lutz Schega. 2019.  Deep Learning based Gait Abnormality Detection using Wearable Sensor System. In _2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_. IEEE, 3613–3619.
  * Ragan et al. (2016) Elizabeth J Ragan, Courtney Johnson, Jacqueline N Milton, and Christopher J Gill. 2016.  Ear biometrics for patient identification in global health: a cross-sectional study to test the feasibility of a simplified algorithm.  _BMC research notes_ 9, 1 (2016), 484.
  * Ramli et al. (2016) DA Ramli, MY Hooi, and KJ Chee. 2016.  Development of Heartbeat Detection Kit for Biometric Authentication System.  _Procedia Computer Science_ 96 (2016), 305–314.
  * Rida et al. (2018) Imad Rida, Noor Almaadeed, and Somaya Almaadeed. 2018\.  Robust gait recognition: a comprehensive survey.  _IET Biometrics_ 8, 1 (2018), 14–28.
  * Rida et al. (2016) I. Rida, X. Jiang, , and G. L. Marcialis. 2016\.  Human body part selection by group lasso of motion for model-free gait recognition.  _IEEE Signal Processing Letters_ 23, 1 (2016), 154–158.
  * Roder et al. (ress) M. Roder, L. A. Passos, L. C. F. Ribeiro, B. C. Benato, A. L. Falcão, and J. P. Papa. In Press.  Intestinal Parasites Classification Using Deep Belief Networks. In _The 19th International Conference on Artificial Intelligence and Soft Computing (ICAISC)_. IEEE.
  * Sabour et al. (2017) Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. 2017\.  Dynamic routing between capsules. In _Advances in neural information processing systems_. 3856–3866.
  * Santana et al. (2019) Marcos C. Santana, Leandro A. Passos, Thierry P. Moreira, Danilo Colombo, Victor Hugo C. de Albuquerque, and João P. Papa. 2019\.  A Novel Siamese-Based Approach for Scene Change Detection With Applications to Obstructed Routes in Hazardous Environments.  _IEEE Intelligent Systems_ 35, 1 (2019), 44–53.
  * Sarkar et al. (2005) Sudeep Sarkar, P Jonathon Phillips, Zongyi Liu, Isidro Robledo Vega, Patrick Grother, and Kevin W Bowyer. 2005.  The humanid gait challenge problem: Data sets, performance, and analysis.  _IEEE transactions on pattern analysis and machine intelligence_ 27, 2 (2005), 162–177.
  * Sepas-Moghaddam et al. (2021) Alireza Sepas-Moghaddam, Saeed Ghorbani, Nikolaus F Troje, and Ali Etemad. 2021.  Gait Recognition using Multi-Scale Partial Representation Transformation with Capsules. In _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 8045–8052.
  * Shiraga et al. (2016) Kohei Shiraga, Yasushi Makihara, Daigo Muramatsu, Tomio Echigo, and Yasushi Yagi. 2016.  Geinet: View-invariant gait recognition using a convolutional neural network. In _2016 international conference on biometrics (ICB)_. IEEE, 1–8.
  * Shutler et al. (2004) Jamie D Shutler, Michael G Grant, Mark S Nixon, and John N Carter. 2004.  On a large sequence-based human gait database.  In _Applications and Science in Soft Computing_. Springer, 339–346.
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ (2014).
  * Singh et al. (2018) Jasvinder Pal Singh, Sanjeev Jain, Sakshi Arora, and Uday Pratap Singh. 2018.  Vision-based gait recognition: A survey.  _IEEE Access_ 6 (2018), 70497–70527.
  * Sokolova and Konushin (2017a) A Sokolova and A Konushin. 2017a.  Gait Recognition Based on Convolutional Neural Networks.  _International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences_ 42 (2017).
  * Sokolova and Konushin (2017b) Anna Sokolova and Anton Konushin. 2017b.  Pose-based Deep Gait Recognition.  _CoRR_ abs/1710.06512 (2017).  arXiv:1710.06512 http://arxiv.org/abs/1710.06512
  * Souza Jr et al. (2020) Luis A. Souza Jr, Leandro A. Passos, Robert Mendel, Alanna Ebigbo, Andreas Probst, Helmut Messmann, Christoph Palm, and João P. Papa. 2020.  Assisting Barrett’s esophagus identification using endoscopic data augmentation based on Generative Adversarial Networks.  _Computers in Biology and Medicine_ 126 (2020), 104029.
  * Sultana et al. (2017) Madeena Sultana, Padma Polash Paul, and Marina L Gavrilova. 2017.  Social behavioral information fusion in multimodal biometrics.  _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ 48, 12 (2017), 2176–2187.
  * Sun et al. (2019) Yingnan Sun, Frank P-W Lo, and Benny Lo. 2019.  A deep learning approach on gender and age recognition using a single inertial sensor. In _2019 IEEE 16th International Conference on Wearable and Implantable Body Sensor Networks (BSN)_. IEEE, 1–4.
  * Sundararajan et al. (2019) Aditya Sundararajan, Arif I Sarwat, and Alexander Pons. 2019\.  A survey on modality characteristics, performance evaluation metrics, and security for traditional and wearable biometric systems.  _ACM Computing Surveys (CSUR)_ 52, 2 (2019), 1–36.
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016.  Rethinking the inception architecture for computer vision. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2818–2826.
  * Taigman et al. (2014) Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato, and Lior Wolf. 2014\.  Deepface: Closing the gap to human-level performance in face verification. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 1701–1708.
  * Takemura et al. (2017) Noriko Takemura, Yasushi Makihara, Daigo Muramatsu, Tomio Echigo, and Yasushi Yagi. 2017.  On input/output architectures for convolutional neural network-based cross-view gait recognition.  _IEEE Transactions on Circuits and Systems for Video Technology_ (2017).
  * Takemura et al. (2018) Noriko Takemura, Yasushi Makihara, Daigo Muramatsu, Tomio Echigo, and Yasushi Yagi. 2018.  Multi-view large population gait dataset and its performance evaluation for cross-view gait recognition.  _IPSJ Transactions on Computer Vision and Applications_ 10, 1 (2018), 4.
  * Tan et al. (2006) Daoliang Tan, Kaiqi Huang, Shiqi Yu, and Tieniu Tan. 2006\.  Efficient night gait recognition based on template matching. In _18th International Conference on Pattern Recognition (ICPR’06)_ , Vol. 3. IEEE, 1000–1003.
  * Tan and Le (2019) Mingxing Tan and Quoc Le. 2019.  Efficientnet: Rethinking model scaling for convolutional neural networks. In _International Conference on Machine Learning_. PMLR, 6105–6114.
  * Tiong et al. (2020) Leslie Ching Ow Tiong, Seong Tae Kim, and Yong Man Ro. 2020\.  Multimodal facial biometrics recognition: Dual-stream convolutional neural networks with multi-feature fusion layers.  _Image and Vision Computing_ 102 (2020), 103977.
  * Tran et al. (2021) Lam Tran, Thang Hoang, Thuc Nguyen, Hyunil Kim, and Deokjai Choi. 2021.  Multi-Model Long Short-Term Memory Network for Gait Recognition Using Window-Based Data Segment.  _IEEE Access_ 9 (2021), 23826–23839.
  * Tsuji et al. (2010) Akira Tsuji, Yasushi Makihara, and Yasushi Yagi. 2010\.  Silhouette transformation based on walking speed for gait identification. In _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_. IEEE, 717–722.
  * Turk and Pentland (1991) Matthew Turk and Alex Pentland. 1991.  Eigenfaces for recognition.  _Journal of cognitive neuroscience_ 3, 1 (1991), 71–86.
  * Uddin et al. (2018) Md Zasim Uddin, Thanh Trung Ngo, Yasushi Makihara, Noriko Takemura, Xiang Li, Daigo Muramatsu, and Yasushi Yagi. 2018\.  The OU-ISIR Large Population Gait Database with real-life carried object and its performance evaluation.  _IPSJ Transactions on Computer Vision and Applications_ 10, 1 (2018), 1–11.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is all you need.  _arXiv preprint arXiv:1706.03762_ (2017).
  * Vincent et al. (2010) Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. 2010.  Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.  _Journal of machine learning research_ 11, 12 (2010).
  * W. Kusakunniran and Li (2013) Jian Zhang Yi Ma W. Kusakunniran, Qiang Wu and Hongdong Li. 2013.  A New View-Invariant Feature for Cross-View Gait Recognition.  _IEEE TIFS_ (2013), 1642–1653.
  * Wan et al. (2018) Changsheng Wan, Li Wang, and Vir V Phoha. 2018.  A survey on gait recognition.  _ACM Computing Surveys (CSUR)_ 51, 5 (2018), 1–35.
  * Wang et al. (2020) Caiyong Wang, Jawad Muhammad, Yunlong Wang, Zhaofeng He, and Zhenan Sun. 2020.  Towards complete and accurate iris segmentation using deep multi-task attention network for non-cooperative iris recognition.  _IEEE Transactions on information forensics and security_ 15 (2020), 2944–2959.
  * Wang and Kumar (2019) Kuo Wang and Ajay Kumar. 2019.  Cross-spectral iris recognition using CNN and supervised discrete hashing.  _Pattern Recognition_ 86 (2019), 85–98.
  * Wang et al. (2019) Kejun Wang, Liangliang Liu, Yilong Lee, Xinnan Ding, and Junyu Lin. 2019.  Nonstandard Periodic Gait Energy Image for Gait Recognition and Data Augmentation. In _Chinese Conference on Pattern Recognition and Computer Vision (PRCV)_. Springer, 197–208.
  * Wang et al. (2003) Liang Wang, Tieniu Tan, Huazhong Ning, and Weiming Hu. 2003\.  Silhouette analysis-based gait recognition for human identification.  _IEEE transactions on pattern analysis and machine intelligence_ 25, 12 (2003), 1505–1518.
  * Wang and Yan (2020) Xiuhui Wang and Wei Qi Yan. 2020.  Human gait recognition based on frame-by-frame gait energy images and convolutional long short-term memory.  _International journal of neural systems_ 30, 01 (2020), 1950027\.
  * Wu et al. (2017a) Zifeng Wu, Yongzhen Huang, Liang Wang, Xiaogang Wang, and Tieniu Tan. 2017a.  A comprehensive study on cross-view gait based human identification with deep cnns.  _IEEE transactions on pattern analysis and machine intelligence_ 39, 2 (2017), 209–226.
  * Wu et al. (2017b) Z. Wu, Y. Huang, L. Wang, X. Wang, and T. Tan. 2017b.  A comprehensive study on cross-view gait based human identification with deep cnns.  _IEEE transactions on pattern analysis and machine intelligence_ 39, 2 (2017), 209–226.
  * X. Li and Ren (2020) C. Xu Y. Yagi X. Li, Y. Makihara and M. Ren. 2020.  Gait recognitionvia semi-supervised disentangled representation learning to identityand covariate features.  _in Computer Vision and Pattern Recognition_ (2020).
  * Xiong et al. (2020) Dezhen Xiong, Daohui Zhang, Xingang Zhao, and Yiwen Zhao. 2020\.  Continuous human gait tracking using sEMG signals. In _2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)_. IEEE, 3094–3097.
  * Xu et al. (2020) Chi Xu, Yasushi Makihara, Xiang Li, Yasushi Yagi, and Jianfeng Lu. 2020.  Cross-view gait recognition using pairwise spatial transformer networks.  _IEEE Transactions on Circuits and Systems for Video Technology_ (2020).
  * Xu et al. (2017) Chi Xu, Yasushi Makihara, Gakuto Ogi, Xiang Li, Yasushi Yagi, and Jianfeng Lu. 2017\.  The OU-ISIR Gait Database Comprising the Large Population Dataset with Age and Performance Evaluation of Age Estimation.  _IPSJ Trans. on Computer Vision and Applications_ 9, 24 (2017), 1–14.
  * Xu et al. (2019) Zhaopeng Xu, Wei Lu, Qin Zhang, Yuileong Yeung, and Xin Chen. 2019.  Gait Recognition Based on Capsule Network.  _Journal of Visual Communication and Image Representation_ (2019).
  * Yasushi Makihara and Yagi (2006) Yasuhiro Mukaigawa Tomio Echigo Yasushi Makihara, Ryusuke Sagawa and Yasushi Yagi. 2006\.  Gait Recognition Using a View Transformation Model in the Frequency Domain.  _In ECCV: v.3953_ (2006), 151–163.
  * Yeo and Park (2020) Sang Seok Yeo and Ga Young Park. 2020.  Accuracy verification of spatio-temporal and kinematic parameters for gait using inertial measurement unit system.  _Sensors_ 20, 5 (2020), 1343.
  * Yogarajah et al. (2015) P. Yogarajah, P. Chaurasia, J. Condell, and G. Prasad. 2015\.  Enhancing gait based person identification using joint sparsity model and -norm minimization.  _Pattern Recognition_ 38 (2015), 3–22.
  * Yu et al. (2019) Ning Yu, Larry S Davis, and Mario Fritz. 2019.  Attributing fake images to gans: Learning and analyzing gan fingerprints. In _Proceedings of the IEEE International Conference on Computer Vision_. 7556–7566.
  * Yu et al. (2017) Shiqi Yu, Haifeng Chen, Qing Wang, Linlin Shen, and Yongzhen Huang. 2017.  Invariant feature extraction for gait recognition using only one uniform model.  _Neurocomputing_ 239 (2017), 81–93.
  * Yu et al. (2006) Shiqi Yu, Daoliang Tan, and Tieniu Tan. 2006.  A Framework for Evaluating the Effect of View Angle, Clothing and Carrying Condition on Gait Recognition. In _Pattern Recognition_ , Vol. 4. 441–444.  https://doi.org/10.1109/ICPR.2006.67
  * Yu et al. (2013) Shiqi Yu, Qing Wang, and Yongzhen Huang. 2013.  A large RGB-D gait dataset and the baseline algorithm. In _Chinese Conference on Biometric Recognition_. Springer, 417–424.
  * Z. Zhang and Liu (2020) F. Liu Z. Zhang, L. Tran and X. Liu. 2020.  On learning disentangled representations for gait recognition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. in press_ (2020).
  * Zagoruyko and Komodakis (2016) Sergey Zagoruyko and Nikos Komodakis. 2016.  Wide residual networks.  _arXiv preprint arXiv:1605.07146_ (2016).
  * Zehngut et al. (2015) Niv Zehngut, Felix Juefei-Xu, Rishabh Bardia, Dipan K Pal, Chandrasekhar Bhagavatula, and Marios Savvides. 2015.  Investigating the feasibility of image-based nose biometrics. In _Image Processing (ICIP), 2015 IEEE International Conference on_. IEEE, 522–526.
  * Zhang et al. (2019) Ziyuan Zhang, Luan Tran, Xi Yin, Yousef Atoum, Xiaoming Liu, Jian Wan, and Nanxin Wang. 2019.  Gait Recognition via Disentangled Representation Learning.  _arXiv preprint arXiv:1904.04925_ (2019).
  * Zhao et al. (2021) Aite Zhao, Junyu Dong, Jianbo Li, Lin Qi, and Huiyu Zhou. 2021.  Associated Spatio-Temporal Capsule Network for Gait Recognition.  _arXiv preprint arXiv:2101.02458_ (2021).
  * Zhao et al. (2016) X. Zhao, Y. Jiang, T. Stathaki, and H. Zhang. 2016\.  Gait recognition method for arbitrary straight walking paths using appearance conversion machine.  _Neurocomputing_ 173 (2016), 530–540.
  * Zheng et al. (2012) Shuai Zheng, Kaiqi Huang, Tieniu Tan, and Dacheng Tao. 2012\.  A cascade fusion scheme for gait and cumulative foot pressure image recognition.  _Pattern Recognition_ 45, 10 (2012), 3603–3610.
  * Zheng et al. (2011) Shuai Zheng, Junge Zhang, Kaiqi Huang, Ran He, and Tieniu Tan. 2011.  Robust view transformation model for gait recognition. In _2011 18th IEEE International Conference on Image Processing_. IEEE, 2073–2076.
  * Zneit et al. (2017) R Abu Zneit, Ziad AlQadi, and M Abu Zalata. 2017.  A Methodology to Create a Fingerprint for RGB Color Image.  _International Journal of Computer Science and Mobile Computing_ 16, 1 (2017), 205–212.
  * Zoph et al. (2020) Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin D Cubuk, and Quoc V Le. 2020.  Rethinking pre-training and self-training.  _arXiv preprint arXiv:2006.06882_ (2020).
  * Zou et al. (2018a) Qin Zou, Yanling Wang, Yi Zhao, Qian Wang, Chao Shen, and Qingquan Li. 2018a.  Deep Learning Based Gait Recognition Using Smartphones in the Wild.  _CoRR_ abs/1811.00338 (2018).  arXiv:1811.00338 http://arxiv.org/abs/1811.00338
  * Zou et al. (2018b) Qin Zou, Yanling Wang, Yi Zhao, Qian Wang, Chao Shen, and Qingquan Li. 2018b.  Deep Learning Based Gait Recognition Using Smartphones in the Wild.  _arXiv preprint arXiv:1811.00338_ (2018).