# Model-Heterogeneous Federated Learning for Internet of Things: Enabling
Technologies and Future Directions

Boyu Fan, Siyang Jiang, Xiang Su, and Pan Hui, Boyu Fan is with the Department
of Computer Science, University of Helsinki, Helsinki 00100, Finland. E-mail:
<EMAIL_ADDRESS>Siyang Jiang is with the School of Mathematics and
Statistics, Huizhou University, and also with the Department of Information
Engineering, The Chinese University of Hong Kong. E-mail<EMAIL_ADDRESS>Xiang Su is with the Department of Computer Science, Norwegian University of
Science and Technology, Gjøvik, Norway. E-mail<EMAIL_ADDRESS>Pan Hui is
with the Department of Computer Science and Engineering, Hong Kong University
of Science and Technology, Hong Kong, and also with the Department of Computer
Science, University of Helsinki, Helsinki 00100, Finland. E-mail:
<EMAIL_ADDRESS>Manuscript received December 7, 2023

###### Abstract

Internet of Things (IoT) interconnects a massive amount of devices, generating
heterogeneous data with diverse characteristics. IoT data emerges as a vital
asset for data-intensive IoT applications, such as healthcare, smart city and
predictive maintenance, harnessing the vast volume of heterogeneous data to
its maximum advantage. These applications leverage different Artificial
Intelligence (AI) algorithms to discover new insights. While machine learning
effectively uncovers implicit patterns through model training, centralizing
IoT data for training poses significant privacy and security concerns.
Federated Learning (FL) offers an promising solution, allowing IoT devices to
conduct local learning without sharing raw data with third parties. Model-
heterogeneous FL empowers clients to train models with varying complexities
based on their hardware capabilities, aligning with heterogeneity of devices
in real-world IoT environments. In this article, we review the state-of-the-
art model-heterogeneous FL methods and provide insights into their merits and
limitations. Moreover, we showcase their applicability to IoT and identify the
open problems and future directions. To the best of our knowledge, this is the
first article that focuses on the topic of model-heterogeneous FL for IoT.

###### Index Terms:

Federated learning, Internet of things, deep learning

##  I Introduction

Over the past decade, significant advancements in Artificial Intelligence (AI)
have led to widespread integration into various Internet of Things (IoT)
applications. AI has empowered various IoT domains, ranging from smart city
and environmental monitoring to manufacturing and healthcare. Machine learning
(ML) plays a crucial role, enabling the development of robust models by
allowing algorithms to learn patterns from data. Traditionally, data is
collected from various IoT sources and stored in centralized servers, thereby
supporting data analysis and model development for researchers and scientists.
These powerful servers offer ample computational resources, facilitating
efficient and reliable model development. However, with the rising privacy
concerns, it is evident that centralized data collection poses inherent risks
to individual privacy. To address these concerns, a series of regulatory laws
and policies, such as the General Data Protection Regulation (GDPR), are
enacted by governments to regulate the collection and handling of data, which
motivates researchers to explore alternative ML paradigms that adhere to these
privacy guidelines.

Federated Learning (FL) emerges as a promising solution for training models in
a privacy-preserving manner for IoT applications. FL obviates the need for
centralized data collection by local storage of data. Each client trains the
model with its local data and contributes to the global model by sharing only
model parameters or gradients with the central server. With this privacy-
preserving feature, FL is widely used across diverse domains, such as health,
banking, and transportation. It is imperative to highlight the growing
adoption of FL within the context of the Internet of Things (IoT), where
billions of sensors and edge devices are continuously generating vast amounts
of data on a daily basis. Characteristics of FL in the IoT context underscore
its adaptability and effectiveness in handling the unique data challenges in
IoT ecosystems.

One limitation of FL is that the server can only perform aggregation once it
has received updates from all participating clients. Training a neural network
is computationally expensive, given a huge number of model parameters and the
necessity for multiple iteration rounds. Consequently, if some clients fail to
complete the local training within the expected times due to the limited
computational resources, both the server and other clients must wait for them,
leading to low learning efficiency and resource utilization inefficiency.
However, the real-world IoT environments naturally include diverse devices,
contributing to a high level of resource heterogeneity. Moreover, the dynamic
operating environments, e.g., network traffic fluctuations and temperature
variations, also have the potential to affect the training times. Therefore,
traditional model-homogeneous FL approaches like FedAvg [1] cannot be
effectively utilized in such IoT scenarios.

To this end, model-heterogeneous FL has been investigated to solve the
aforementioned challenge, alleviating the requirement imposed by traditional
FL where all clients are expected to share a universal global model. Instead,
it enables clients to train different types of models based on their
respective capabilities, which offers two significant advantages, i.e.,
mitigating the straggler issues [2] and decreasing of resource wastage. For
instance, consider a scenario where a Raspberry Pi can efficiently train a
two-layer convolutional neural network (CNN), while a ResNet18 model would
pose a significant burden on it. Lengthy training times result in waiting
periods for other clients, leading to idle periods for other clients and
inefficient resource utilization. In contrast, model-heterogeneous FL allows a
Raspberry Pi to train a small model, while a server machine can simultaneously
train a ResNet18 model within the same FL round.

While model-heterogeneous FL holds great potential to mitigate the affect of
resource heterogeneity, a core challenge should be addressed. Since clients
train different neural networks, the server cannot directly conduct averaging
operations on the received model parameters due to variations in their
architectures. Novel approaches should be proposed to enable the server to
conduct aggregation even if the model parameters have different shapes. The
advantages of model-heterogeneous FL and its great potential in IoT motivate
us to perform a comprehensive literature review to understand the state-of-
the-art solutions. In this article, we comprehensively review the state-of-
the-art model-heterogeneous FL approaches. Beyond enumerating the advantages
and disadvantages of each method, we also summarize the mainstream solutions
and propose potential refinements.

TABLE I: Comparison of relative surveys Ref | Year | Scope
---|---|---
[3] | 2021 | FL for general IoT
[4] | 2022 | The fusion of FL and IoT
[5] | 2022 | FL on resource-constrained IoT devices
[6] | 2023 | FL on heterogeneous devices
Our work | 2023 | Model-heterogeneous FL for IoT

Table I compares our survey with existing surveys related to FL and IoT. To
the best of our knowledge, our survey marks the pioneering efforts in
addressing the domain of model-heterogeneous FL, underscoring its novelty and
significance within the research landscape. Our contributions are twofold:

  * •

We review the state-of-the-art approaches to model-heterogeneous FL and
highlight the insights of each approach.

  * •

We provide the future directions of model-heterogeneous FL research for IoT
based on our findings.

The remainder of this article is organized as follows. Section II introduces
the background of FL and our motivation for this survey. Section III presents
the state-of-the-art methods for model-heterogeneous FL and conducts
comparative analyses to show the strengths and weaknesses of each method.
Section IV details the applications of model-heterogeneous FL on IoT. Section
V discusses the open questions and future research directions based on our
findings. Finally, Section VI concludes this survey.

##  II Preliminaries and challenges

###  II-A Overview of FL

FL was first proposed by Google [1], which becomes a paradigm to train ML
models without the central collection of data. Since then, ongoing research
has developed lots of new applications for FL. The driving force behind the
continued research is the emergence of new and strict privacy regulations and
laws, exemplified by GDPR, which was enacted in 2016 and enforced from 2018
onward.

Figure 1 presents the general FL architecture, consisting of two main
entities, i.e., a central server for conducting aggregation and multiple
clients for performing local training. Each client trains models with its
local dataset without sharing with others, thereby ensuring the protection of
their privacy. During the training process, instead of collecting raw data
from different sources, the server only receives model parameters from clients
and sends the aggregated model parameters back to the clients to update the
local models.

Figure 1: A standard FL architecture.

In general, the FL training process includes three main steps. 1) Server
initialization. As the first step of FL training, the server initializes a
global model tailored to the specific training task and subsequently
distributes the model to the selected clients. 2) Local Update. Each selected
client $k$ conducts model training with its local data $D_{k}$ based on the
received global model. Once the training is completed, the clients send the
updated models back to the server. 3) Model aggregation. The server receives
the updated models from clients and then conducts model averaging aggregation
to update the model. The aggregated model will become the new global model and
continually be updated in the subsequent rounds until convergence.

###  II-B The Challenges of System Heterogeneity

System heterogeneity refers to the diversity in hardware configurations,
software platforms, communication protocols, and even run-time environments
among different participants and devices, which exists in common IoT
scenarios. The traditional FL workflow typically involves the server waiting
to receive updated parameters from the selected clients before performing
aggregation. However, due to the existence of system heterogeneity, different
clients may present varying training efficiency, leading to two potential
problems. First, due to the fact that all clients perform local updates based
on a unified global model, the required training times vary based on their
computation capabilities. Consequently, both the server and the clients with
more abundant computation resources need to wait for slower clients, resulting
in resource wastage. Second, some resource-constrained devices are not capable
of training complex neural networks, e.g., ResNet. Extremely long processing
time can cause the server to mistakenly assume that the clients are offline,
thus never selecting them for the distributed training. As the local data
owned by these devices cannot join the training, the contribution to the
global model can be disproportionately small, leading to imbalanced models. In
conclusion, system heterogeneity poses challenges to the performance,
generalization, coordination and efficiency of the FL system.

###  II-C Model-heterogeneous FL

The optimal approach for addressing system heterogeneity lies in empowering
devices to learn diverse models with varying complexities based on their
available resources. For instance, training a CNN with two convolutional
layers requires significantly fewer computing resources compared to training a
ResNet18. Therefore, model-heterogeneous FL, allowing clients to train
heterogeneous models, naturally emerges as a promising technique for
mitigating system heterogeneity issues. However, in model-heterogeneous FL,
clients may possess different model architectures, hyperparameter settings, or
optimization techniques, thereby complicating the aggregation process on the
server side. Conversely, in model-homogeneous FL, there exists only one global
model architecture that is shared by all clients, facilitating straightforward
element-to-element averaging operations on multiple model parameters. Hence,
how to achieve effective model aggregation for models with different
architectures becomes the core challenge of model-heterogeneous FL.

##  III State-of-the-art Model-heterogeneous FL approaches

In this section, we investigate the existing approaches for model-
heterogeneous FL, which can be classified into two main solutions, i.e., KD-
based approaches and PT-based approaches. We also present several alternative
approaches that fall outside of these two categories.

###  III-A KD-based Approaches

####  III-A1 Knowledge Distillation

Knowledge Distillation (KD) was proposed by Hinton et al. [7], which aims to
distill knowledge from a larger model or an ensemble of models and transfer
this understanding to a smaller, more compact model. Despite its reduced size,
this distilled model, often referred to as the “student” model, can exhibit
performance metrics remarkably similar to its larger “teacher” counterpart.
During the KD process, three types of knowledge play pivotal roles, including
response-based, feature-based, and relationship-based knowledge [8]. In
response-based knowledge, information is extracted from the output layer of
the teacher model, capturing the class probabilities. Feature-based knowledge,
on the other hand, does not restrict itself to the output of the last layer,
it also gathers insights from the outputs of intermediate layers of a neural
network. Contrasting with the prior two types, relationship-based knowledge
leverages the relationships either between different layers of a neural
network or between individual data samples. Various techniques, such as
similarity matrices, instance relationship graphs, or mutual information
flows, have been proposed to define these relationships.

####  III-A2 Knowledge Distillation in FL

The output of the last layer of a model, known as logits in response-based
knowledge, has a shape determined by the number of classes in the dataset,
which is independent of the model’s architecture. For instance, in the CIFAR10
dataset, the logits’ shape remains at $10\times 1$, whereas in the CIFAR100
dataset, it is $100\times 1$. This characteristic allows the utilization of KD
in FL, enabling aggregation at the server even when received models have
varying architectures.

Figure 2: The basic idea of KD in FL.

In recent years, several approaches have been developed to apply KD to achieve
model-heterogeneous FL. The basic idea is presented in Figure 2. Clients use
their local data to conduct hard predictions and leverage public data to serve
as soft predictions. FedMD [9] is a pioneering work to achieve model
aggregation using KD rather than averaging the model parameters. The training
process of FedMD consists of four steps: First, each client computes the class
scores on a public dataset and transfers them to the central server. Second,
the server computes the average logits to serve as a consensus and sends it to
all clients. Third, each client trains its model to approach the consensus on
the public dataset. Finally, each client trains its model based on the local
data to achieve local updates. Compared to conventional FL approaches, FedMD
does not need to exchange model parameters between server and client, allowing
models to have different architectures. Clients contribute to the global model
by only sharing their logits that are computed on the public dataset.

Although FedMD bypasses model parameter aggregation, its global logits
construction by simple averaging at the server may not optimally harness the
collective knowledge of clients. KT-pFL [10] introduces a knowledge
coefficient matrix in the FL training process. After training the clients’
personalized models over the local data, the local model logits is generated
to the public data set and shared with the server. Then, using the linear
combination of the aggregated soft predictions and the knowledge coefficient
matrix, the server calculates the local soft prediction for each client. After
each iteration, the knowledge coefficient matrix is updated on the server.
With the help of a knowledge coefficient matrix, each client can adaptively
aggregate all local soft predictions to form a personalized one instead of
being restricted to a global soft prediction.

A shared limitation of the aforementioned approaches is their dependency on a
public dataset. In practical applications, this kind of public dataset
requires careful deliberation and even prior knowledge of clients’ private
data. To eliminate this dependency, several approaches for data-free KD have
been proposed. Compared to FedMD, FedDF [11] relieves the requirement of
labeled public data. Instead, unlabeled or artificially generated data from
arbitrary domains is used to perform ensemble distillation on the server. This
involves training local models with unlabeled or generated data and using the
resulting output logits to construct the global model. In contrast to FedMD,
distillation is carried out on the server side, leaving the local training
unaffected.

Similarly, FedGen [12] achieves data-free learning by training a generative
model based on the prediction rules of the local clients. Through this
approach, the generator effectively models the global data distribution. With
this distilled joint knowledge about data distribution from the generator,
local models are then updated. This methodology allows the local models to
mutually benefit from each other without the necessity of employing an
external public data source. A notable distinction from earlier frameworks is
that the distilled knowledge is not passed on to a global model, but directly
to the local models. However, a potential drawback of FedGen pertains to its
tendency to lose information from the local models, due to the fact that the
approach sometimes overlooks the incompatibility of localized knowledge during
the distillation process.

FedZKT [13] presents an alternative approach to data-free KD for FL. In
contrast to FedGen, FedZKT keeps the computation-heavy generator on the server
side to relieve devices with low computational capacity. The on-device
parameters are not shared with the server to preserve privacy. The generative
model provides inputs for global and on-device training, which are utilized to
detect and maximize the disagreements between the global and private on-device
models. Moreover, the pre-trained generative model is also utilized to distill
the on-device model and update its parameters with global knowledge. The
approach of alleviating clients from some computation-heavy tasks results in
an increase in the number of classes per client in comparison to FedMD.

FedFTG [14] offers a solution by fine-tuning the global model through a
process of hard sample mining. This method harnesses the robust computational
capacities of the server effectively. FedFTG employs a generator to extract
knowledge from the input spaces of individual local models. Central to this
process is the utilization of a hard sample mining scheme, prompting the
generator to produce challenging samples within the data distribution that
highlight disparities between the local and global models. This approach
ensures the global model is finely attuned to the overall data distribution.
In addition, the framework integrates customized label sampling and a class-
level ensemble strategy to offset variances in label distributions among local
clients resulting from class imbalances and varying importance levels of
knowledge for the classes.

In FedHKD [15], the hyper knowledge is introduced as the average of the data
representation of a class in combination with the respective average soft
prediction. The iterative process unfolds as follows: 1) Each client sends the
local hyper-knowledge of each class to the server. To safeguard the client’s
privacy, noise is intentionally added to the data representation. 2) The
global hyper knowledge is aggregated from the collected hyper knowledge of the
clients. The added noise is compensated during the aggregation. 3) The global
hyper knowledge is sent back to the clients, who adjusts their loss function
accordingly for the subsequent iteration. In comparison to FedMD, FedHDK is
competitive in local and global test accuracy for heterogeneous data.

Building on the success of using response-based KD on FL, researchers believe
the logits implicitly possess the label information of each class, serving as
a channel for knowledge transfer among different parties. FedProto [16]
capitalizes on this idea by employing class prototypes as a mean for knowledge
exchange. Specifically, a class prototype is a representation of a class,
derived from the embedding vectors of class features and subsequently
averaged. These prototypes are sent to the server, where they are aggregated
to create a global knowledge base. The global knowledge is then sent back to
the clients, motivating them to refine their local prototypes, which in turn
bolsters the efficacy of local training. In this way, no global model is
maintained, allowing clients to employ different model architectures.

FedHeNN [17] presents an architecture-agnostic FL approach for heterogeneous
neural networks. In FedHeNN, each client trains a personalized model with
their local dataset, which is allowed to have a different architecture from
the global model. In each iteration, a Representation Alignment Dataset (RAD)
is generated by the server and distributed to the clients. The clients then
use the RAD to align their representations. Instead of sending model
parameters, the clients share their generated representation for RAD with the
server to achieve aggregation. It has been empirically proven that the method
is not vulnerable to clients with smaller data footprints. The test accuracy
experiences only minimal degradation compared to the proportion of clients
with smaller local data sets.

Regatti et al. [18] investigate a solution to preserve the model privacy of
clients in addition to data privacy. The Fed-CMA algorithm allows clients to
maintain individual model architectures while participating in FL with other
clients. This approach does not require a public dataset or maintain a global
model. Instead, the feature weights of the feature extractor and the
classification weights are distilled client-side. The server then aggregates
the weights from all clients and sends the aggregated feature and
classification weights back to the clients to update the local model. To
achieve this, a new algorithm based on conditional moment alignment and
generalization for aggregating the weights is introduced, since conventional
algorithms such as FedAvg and FedProx cannot be used due to the different
client model architectures.

####  III-A3 Limitations

The KD-based concept facilitates various kinds of new FL approaches to solve
the model-heterogeneous FL problem. However, several challenges exist. First,
despite several claims of achieving data-free KD, they still require some kind
of data that the server and client can jointly access to achieve knowledge
alignment. How to achieve truly data-free KD remains an interesting yet
challenging question. Second, most of the existing works focus on the
response-based KD, the model performance with feature-based and relationship-
based KD needs to be further investigated. Third, compared to conventional
model parameter-shared FL approaches, KD-based approaches solely rely on
exchanging logits, resulting in a much smaller amount of information exchange.
Therefore, despite they achieve model-heterogeneity and require much less
communication cost, the model performance may decrease. How to balance the
model performance and the ability to accommodate model-heterogeneity is a
challenge. Finally, privacy concerns related to knowledge transmission need
further exploration, especially given the ease with which some types of
knowledge can be decoded and potentially accessed by unintended parties.

###  III-B PT-Based Approaches

####  III-B1 Partial Training

Partial training refers to the strategy that allows clients to train specific
segments of neural networks based on their computational capabilities. This
methodology primarily divides into two branches, i.e., dynamic partial
training and post-partial training. Dynamic partial training focuses on
training only a subset of the complete network, with the objective of
generating diverse models in various FL scenarios while significantly reducing
the computational demands. On the other hand, post-partial training approaches
aim at pruning the original full-size model on the server. The pruned model is
subsequently fine-tuned on devices using their local data, a process initiated
after acquiring the aggregated model. This approach strategically leverages
local data to refine the model, thereby enhancing its applicability and
effectiveness.

####  III-B2 Partial Training in FL

Figure 3 illustrates the basic idea of PT in FL. Each client trains a part of
the global model and only aggregates the corresponding parts during the
aggregation step. HeteroFL [19], as a typical dynamic partial training method,
allows clients to train specific parts of the neural network based on their
computational capacity and an adaptive communication compression scheme that
adjusts model update compression according to clients’ communication
resources. In particular, the clients are organized into multiple groups based
on their computation capabilities and communication resources. Each group
independently trains its model locally using the partial training technique,
and the local models are then aggregated to form a global model. This method
is suitable for HeteroFL since every communication round is independent.
HeteroFL achieves significant improvements in terms of computation and
communication efficiency on a benchmark dataset, making FL feasible for a
wider range of devices and use cases.

Figure 3: The basic idea of PT in FL.

Compared to the static sub-model extraction strategy in HeteroFL, FedRolex
[20] proposes a rolling sub-model extraction method. In this scheme, the sub-
model is derived from the full-sized model using a progressively advancing
rolling window with each communication round. This innovative strategy ensures
that different sections of the global server model receive uniform training,
thereby alleviating the client drift caused by model heterogeneity. In
addition, while HeteroFL confines the global model’s size to match that of its
largest client model, FedRolex permits global models to surpass the complexity
of any individual client model, leading to enhanced performance

In contrast to HeteroFL and FedRolex dividing the clients into multiple
groups, Hermes [21] develops a two-stage technique that includes client
selection and adaptive model compression, aiming to improve the efficiency of
FL on diverse mobile clients. In the client selection stage, the authors
employ a reinforcement learning algorithm to select the most suitable clients
for participating in the training process based on their data quality,
computational capabilities, and communication bandwidth. The selected clients
then train their models locally using partial training, and the updated model
weights are sent to the central server for aggregation. In the adaptive model
compression stage, the authors propose a novel approach that uses a
combination of model pruning and quantization to compress the model size
without sacrificing accuracy. This approach facilitates efficient transmission
of model updates from the clients to the server, resulting in reduced
communication costs.

As a typical dynamic partial training method, PriSM [22] leverages the spatial
structure of neutral networks, i.e., low-rank structure and networks kernel
orthogonality to train sub-networks in the orthogonal kernel space. In
particular, PriSM uses singular value decomposition on the original kernels to
extract a set of principal orthogonal kernels. The significance of these
kernels is determined by their singular values. Following this, it also
introduces a unique sampling strategy that independently selects varying
subsets of the principal kernels. This strategy is used to create client-
specific sub-models, effectively reducing computational and communication
demands. It is worth noting that kernels with larger singular values are given
a higher likelihood of being selected during the sampling process. As a
result, each sub-model represents a low-rank approximation of the
comprehensive server model.

Compared to dynamic partial training approaches, post-partial training
approaches finetune the model by leveraging their local data. For example,
LotteryFL [23] generates heterogeneous models, which is inspired by the
Lottery Ticket hypothesis, presenting a method for identifying Lottery Ticket
Networks (LTNs) – sparse subnetworks within larger models. In this context,
LotteryFL aims to find each client’s LTN during communication rounds in FL and
only exchanges LTN parameters between clients and the server, thereby reducing
communication overhead. In particular, LotteryFL employs client-specific
pruning strategies to the updated weights, resulting in sparse subnetworks
(winning tickets). In summary, the LotteryFL framework leverages the lottery
ticket hypothesis to improve personalization and communication efficiency in
FL for non-IID datasets.

PruneFL [24] directly employs model pruning to create various local models to
fit model-heterogeneous requirements. In particular, PruneFL proposes
distributed pruning which involves initial pruning at a chosen client,
followed by additional distributed pruning integrated within the standard FL
process, enabling more efficient model compression across the network.
Subsequently, PruneFL dynamically maintains a compact model with efficient
transmission, computation, and reduced memory footprint, while preserving
essential connections for convergence to an accuracy similar to the original
model. In addition, PruneFL continuously updates the parameters to retain and
adjust the model size, aiming to minimize the time taken to reach convergence.
Experimental results demonstrate that this approach significantly reduces
model size and communication costs while maintaining high accuracy on
benchmark datasets. The PruneFL approach also shows improved efficiency in
terms of training time and energy consumption, making it suitable for FL on
resource-constrained edge devices.

####  III-B3 Limitations

The partial training methodology enables clients to tailor the training of
specific sectors within neural networks, but it comes with several challenges.
First, it is assumed that each local model is a subset of the global model,
meaning they should share common elements. This requirement limits the types
of local models, reducing flexibility in designing models for different local
data. Second, most of the existing works employ either random or fixed
strategies to obtain these partial neural networks. However, the optimal
method to select the most crucial components for enhanced learning efficiency
remains ambiguous. Third, while much of the existing research is grounded in
empirical evidence, there is a notable lack of rigorous theoretical analysis.

###  III-C Alternative approaches

In addition to the KD and PT-based methodologies, several innovative
techniques have been developed to enable model-heterogeneous FL.

HAFL-GHN [25] handles heterogeneous client architectures by employing a Graph
HyperNetwork (GHN). The framework does not require any public data and the
client architectures remain private. GHNs are highly adaptable for
accommodating any architecture, with their nodes embodying the layers of the
client model and the edges of its computational workflow. Each local client
initiates by training a copy of the GHN with its private data and then passes
the resulting weights of the graph to the server. These weights are aggregated
on the servervvside and distributed to local models for further optimization.
HAFL-GHN outperforms the alternative frameworks based on KD or non-graph
hypernetworks, especially when clients possess smaller local datasets.
However, when the clients have homogeneous architectures, the performance
results of the presented framework do not yet match that of other state-of-
the-art methods. Another notable advantage of this method pertains to
flexibility, i.e. allowing for changes in local clients or new clients can be
added with minimal effort.

DISTREAL [26] presents a Distributed Resource-Aware Learning pattern to
address systems with various resources, allowing clients to adapt
independently of the server. Such adaptation involves omitting dynamical
filters of the convolutional layers to reduce the computational complexity of
the training. This dropout mechanism is applied during runtime and uses a
dropout rate, which is determined once before training and is defined by a
Pareto-optimal vector. The rate is based on the convergence speed and the
resource requirements and is determined by an automated design space
exploration in the form of a genetic algorithm. Experimental results with
different datasets have demonstrated that DISTREAL achieves significantly
faster convergence speed in the FL system without sacrificing/compromising
accuracy. This dynamic dropout approach maximizes resource utilization,
allowing devices of varying specifications to participate seamlessly.

Yao et al. [27] propose FL for Heterogeneous Models (FEDHM) as a method to
address the problem of heterogeneous models. Low-rank factorization is used to
decompose the model parameters for different clients and divide them into
shared and individual components. The shared components capture the common
features across different models, while the individual components capture the
unique features of each model. FEDHM applies FL to update the shared and
individual components separately. This update approach for both shared and
unique components reduces both communication costs and device training
overhead, boosting performance across diverse settings. Empirical evidence
highlights FEDHM’s superior efficacy compared to the state-of-the-art methods.
However, a notable drawback remains in the steep computational overhead linked
to matrix composition.

##  IV Applications of model-heterogeneous FL in IoT

###  IV-A Healthcare

The integration of AI and smart devices into healthcare is transforming
medical services from being confined to hospital settings to becoming a part
of our everyday lives. The IoT in healthcare involves a wide variety of
applications and devices, where wearable devices play a key role. These
wearables, such as heart and blood pressure monitors, as well as wristbands
and smartwatches, are equipped with multiple monitoring functions. They
monitor real-time physiological data and transmit this information for further
analysis. While the data collected is invaluable for the development of
sophisticated analytical models, it also raises significant concerns regarding
user privacy. Model-heterogeneous FL emerges as a promising solution to this
dilemma. This approach ensures that private health data remains stored on the
user’s own device, thereby eliminating the need for data to be transmitted
externally. Detection models are updated using local data. Thanks to the
model-heterogeneous feature, different models can be trained on various types
of smartphones. This eliminates the unrealistic requirement for all users to
possess identical devices, thereby enhancing the accessibility and
applicability of this technology in diverse real-world scenarios.

###  IV-B Automobile

In the evolving landscape of the automobile industry, vehicles are
increasingly integrating with the IoT. This connectivity allows for real-time
feedback about driving conditions, enhancing both safety and the overall
driving experience. However, the integration of a vast number of vehicles with
the IoT introduces certain challenges. For instance, cloud servers may incur
substantial communication costs when processing vast amounts of data.
Furthermore, data privacy risks arise as vehicles, in theory, have the
capacity to interconnect through the IoT. FL-based vehicle IoT mitigates these
challenges. Specifically, FL enables vehicles to process and train models
without transmitting local data to the centralized system, thereby mitigating
privacy concerns. Moreover, model-heterogeneous FL supports vehicles with
varied hardware configurations, offering increased flexibility and
significantly reducing the costs associated with deploying automotive IoT
solutions.

###  IV-C Smart City

The development and deployment of IoT technologies are revolutionizing urban
landscapes worldwide by connecting billions of devices and making informed
decisions based on massive amounts of heterogeneous IoT data generated from
these devices. Challenges of processing IoT data from smart cities pertain to
efficiently analyzing generated data, developing novel services for citizens
and stakeholders, and data privacy and consent. For example, city stakeholders
make decisions, such as to ensure optimal city planning and enhance city
management, which requires a deep understanding of the massive amounts of
heterogeneous IoT data. Achieving this level of understanding necessitates
deep analytics across various sectors of smart city data. However, the sharing
of data between these sectors can often be restricted. In this context, model-
heterogeneous FL in smart cities offers a promising solution to this challenge
by training diverse ML models tailored to the specific infrastructures of
different sectors, while keeping the data localized. Moreover, model-
heterogeneous FL facilitates collaboration between cities. This is
particularly beneficial for cities lacking advanced data centers, as they can
still participate by training smaller-scale models. Such collaborative efforts
enhance the overall efficacy and intelligence of smart city initiatives,
paving the way for more responsive and adaptive urban environments.

###  IV-D Industrial Automation

IoT-empowered industrial automation facilitates efficient production and finer
management. This automation process generates vast quantities of data,
encompassing sensor readings, image captures, and more. With the assistance of
AI technology, this data holds the potential for the development of enhanced
processing models. However, concerns regarding data privacy and security deter
many enterprises and factories from sharing their sensitive data across
industrial networks, potentially stalling advancements in automation systems.
Integrating model-heterogeneous FL with industrial automation offers a
solution to this challenge. First, data integrity is assured as it remains
stored locally throughout the process. Second, given that different factories
may have distinct infrastructural and equipment setups, IoT service providers
can optimize system efficiency across diverse factories through model-
heterogeneous FL. This is achieved by collecting only model parameters, rather
than raw production data, ensuring both privacy and system enhancements.

###  IV-E Energy Networks

With the flourishing of AI technology and IoT, energy networks are also
constantly evolving and developing. The IoT serves as a foundational element
to connect different energy infrastructures, while AI technology plays the
role of a planner, determining how to distribute and transmit energy
reasonably and efficiently. A representative example is the smart electricity
meters installed in every household. They monitor and record the energy
consumption of each family and send the data to the server of the energy
manufacturer on a regular basis. The volume of this data enables energy
manufacturers to train robust AI models, predicting future electricity usage
of individual households. They can then adjust both power supply and pricing,
optimizing benefits for both power supply companies and their consumers.
However, from the perspective of privacy protection, attackers or malicious
actors might deduce a household’s living patterns and daily routines by
analyzing energy consumption data. With model-heterogeneous FL, this data can
be kept on the user’s side without being sent to the energy manufacturer.
Meanwhile, model-heterogeneous FL allows households to install different smart
meters, reducing the overhead incurred by standardization. Model training is
performed locally using real-time energy consumption data, and only the model
parameters are sent to the energy manufacturer to improve the overall energy
prediction performance.

##  V Open problems and future directions

Model-heterogeneous FL is an emerging yet practical distributed learning
paradigm. Despite the considerable research on this topic, we identify several
open problems and future directions that demand further research.

###  V-A Aggregation strategy

In model-heterogeneous FL scenarios, element-wise weight averaging for
aggregation is impractical due to differing model architectures, affecting the
efficacy of most homogeneous FL techniques. To address this, leading methods
either use latent representation to standardize the models, like employing KD
for logit processing, or employ pruning to ensure models have overlapping
sections for aggregation. However, each approach has limitations. KD-based
methods presuppose aligned logits across clients, either from an existing
dataset or synthetic data. Without this alignment, logit aggregation might
lose relevance, as they could represent entirely different features. In
pruning-based methods, focusing solely on overlapping components may neglect
essential, non-overlapping model features, resulting in suboptimal model
performance.

We envision promising future research directions toward model aggregation
strategy. First, While KD strategies often rely on logits, this is constrained
by data alignment. A more promising direction might be to investigate
relation-based knowledge, which is calculated by the inner products between
features across layers. Such knowledge contains the implicit learning logic of
a model, which is irreverent to the input data, eliminating the reliance on
public datasets. Regarding pruning-based strategy, an interesting direction is
to identify the principal components, in other words, identify the layers and
weights that most impact model performance, akin to principal component
analysis (PCA). With a clear understanding of the principal components,
efficient pruning techniques can be employed to modify model architectures,
while preserving the most important features of a model.

###  V-B Privacy and security

Despite FL allowing data to remain local, sharing model parameters can
potentially reveal information about the private data. An adversary may
conduct reverse engineering to induce the raw data from the model information,
as both weights and gradients are calculated by the input data with specific
matrix operations[28]. In model-heterogeneous FL, the transferred data is
either output logits or a subset of the full model parameters. It remains
ambiguous whether transferring this information can mitigate reverse engineer
risks. Model poisoning is another security concern for FL systems [29]. The
malicious party can manipulate the updated model before it is sent to the
server for aggregation, thereby affecting the performance of the global model.
In model-heterogeneous FL, where clients are allowed to have different types
of models, directly modifying the model has limited impact on the global model
as the update strategy does not rely on the model parameter aggregations.
However, quantifying this impact is crucial for designing more efficient
model-heterogeneous methods.

###  V-C Resource-constrained IoT devices

Model-heterogeneous FL presents a promising solution to IoT applications,
which connects to large-scale heterogeneous IoT devices, compared to the
conventional model-homogeneous FL. However, many devices in IoT are resource-
constrained, such as robots, embedding controllers, cameras, and electric
meters, making them unsuitable for training even the simplest neural networks.
The computation cost of local training also contributes to heightened power
consumption, potentially affecting the core functions of the devices.
Therefore, exploring the effective deployment of model-heterogeneous FL
frameworks on IoT is an open issue. One potential direction is offloading a
part of training tasks from resource-constrained devices to their trusted edge
nodes through split learning [30]. Another direction lies in investigating
more advanced FL training strategies. For instance, a system might allow
clients with weak computation capacities to train linear models, while
permitting others to train neural networks.

##  VI Conclusion

This article presents a thorough review of model-heterogeneous FL methods for
IoT. We start with discussing the challenges posed by conventional FL, setting
the background for understanding the motivations behind model-heterogeneous FL
within the IoT context. Subsequently, we dive into the state-of-the-art model-
heterogeneous FL strategies, classifying them into two primary directions,
i.e. KD-based and PT-based approaches. We also highlight other notable works
that defy simple classification. Drawing from comparative insights into these
methods, we spotlight the applications of model-heterogeneous FL within IoT,
culminating in discussions about unresolved challenges and potential
directions for future research. This article serves as a structured guide to
model-heterogeneous FL and underscores its promising future within the IoT
landscape.

## References

  * [1] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _Artificial intelligence and statistics_. PMLR, 2017, pp. 1273–1282.
  * [2] A. Reisizadeh, I. Tziotis, H. Hassani, A. Mokhtari, and R. Pedarsani, “Straggler-resilient federated learning: Leveraging the interplay between statistical accuracy and system heterogeneity,” _IEEE JASC_ , vol. 3, no. 2, pp. 197–205, 2022.
  * [3] D. C. Nguyen, M. Ding, P. N. Pathirana, A. Seneviratne, J. Li, and H. Vincent Poor, “Federated learning for internet of things: A comprehensive survey,” _IEEE Commun Surv Tutor_ , vol. 23, no. 3, pp. 1622–1658, 2021.
  * [4] P. Boobalan, S. P. Ramu, Q.-V. Pham, K. Dev, S. Pandya, P. K. R. Maddikunta, T. R. Gadekallu, and T. Huynh-The, “Fusion of federated learning and industrial internet of things: A survey,” _Computer Networks_ , vol. 212, p. 109048, 2022.
  * [5] A. Imteaj, U. Thakker, S. Wang, J. Li, and M. H. Amini, “A survey on federated learning for resource-constrained iot devices,” _IEEE IoT-J_ , vol. 9, no. 1, pp. 1–24, 2022.
  * [6] K. Pfeiffer, M. Rapp, R. Khalili, and J. Henkel, “Federated learning for computationally constrained heterogeneous devices: A survey,” _ACM Comput. Surv._ , vol. 55, no. 14s, jul 2023.
  * [7] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _arXiv preprint arXiv:1503.02531_ , 2015.
  * [8] J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distillation: A survey,” _IJCV_ , vol. 129, pp. 1789–1819, 2021.
  * [9] D. Li and J. Wang, “Fedmd: Heterogenous federated learning via model distillation,” _arXiv preprint arXiv:1910.03581_ , 2019.
  * [10] J. Zhang, S. Guo, X. Ma, H. Wang, W. Xu, and F. Wu, “Parameterized knowledge transfer for personalized federated learning,” _NeurIPS_ , vol. 34, pp. 10 092–10 104, 2021.
  * [11] T. Lin, L. Kong, S. U. Stich, and M. Jaggi, “Ensemble distillation for robust model fusion in federated learning,” _NeurIPS_ , vol. 33, pp. 2351–2363, 2020.
  * [12] Z. Zhu, J. Hong, and J. Zhou, “Data-free knowledge distillation for heterogeneous federated learning,” in _ICML_ , ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 18–24 Jul 2021, pp. 12 878–12 889.
  * [13] L. Zhang, D. Wu, and X. Yuan, “Fedzkt: Zero-shot knowledge transfer towards resource-constrained federated learning with heterogeneous on-device models,” in _IEEE ICDCS 2024_ , 2022, pp. 928–938.
  * [14] L. Zhang, L. Shen, L. Ding, D. Tao, and L.-Y. Duan, “Fine-tuning global model via data-free knowledge distillation for non-iid federated learning,” in _CVPR_ , June 2022, pp. 10 174–10 183.
  * [15] H. Chen, H. Vikalo _et al._ , “The best of both worlds: Accurate global and personalized models through federated learning with data-free hyper-knowledge distillation,” _arXiv preprint arXiv:2301.08968_ , 2023.
  * [16] Y. Tan, G. Long, L. Liu, T. Zhou, Q. Lu, J. Jiang, and C. Zhang, “Fedproto: Federated prototype learning across heterogeneous clients,” in _AAAI_ , vol. 36, no. 8, 2022, pp. 8432–8440.
  * [17] D. Makhija, X. Han, N. Ho, and J. Ghosh, “Architecture agnostic federated learning for neural networks,” in _ICML_. PMLR, 2022, pp. 14 860–14 870.
  * [18] J. R. Regatti, S. Lu, A. Gupta, and N. Shroff, “Conditional moment alignment for improved generalization in federated learning,” in _FL Workshop in NeurIPS 2022)_ , 2022.
  * [19] E. Diao, J. Ding, and V. Tarokh, “Heterofl: Computation and communication efficient federated learning for heterogeneous clients,” in _ICLR_ , 2020.
  * [20] S. Alam, L. Liu, M. Yan, and M. Zhang, “Fedrolex: Model-heterogeneous federated learning with rolling sub-model extraction,” _NeurIPS_ , vol. 35, pp. 29 677–29 690, 2022.
  * [21] A. Li, J. Sun, P. Li, Y. Pu, H. Li, and Y. Chen, “Hermes: An efficient federated learning framework for heterogeneous mobile clients,” ser. MobiCom ’21. New York, NY, USA: Association for Computing Machinery, 2021, p. 420–437.
  * [22] Y. Niu, S. Prakash, S. Kundu, S. Lee, and S. Avestimehr, “Federated learning of large models at the edge via principal sub-model training,” _arXiv preprint arXiv:2208.13141_ , 2022.
  * [23] A. Li, J. Sun, B. Wang, L. Duan, S. Li, Y. Chen, and H. Li, “Lotteryfl: Empower edge intelligence with personalized and communication-efficient federated learning,” in _2021 IEEE/ACM SEC_ , 2021, pp. 68–79.
  * [24] Y. Jiang, S. Wang, V. Valls, B. J. Ko, W.-H. Lee, K. K. Leung, and L. Tassiulas, “Model pruning enables efficient federated learning on edge devices,” _IEEE Trans. Neural Netw. Learn. Syst._ , pp. 1–13, 2022.
  * [25] O. Litany, H. Maron, D. Acuna, J. Kautz, G. Chechik, and S. Fidler, “Federated learning with heterogeneous architectures using graph hypernetworks,” _arXiv preprint arXiv:2201.08459_ , 2022.
  * [26] M. Rapp, R. Khalili, K. Pfeiffer, and J. Henkel, “Distreal: Distributed resource-aware learning in heterogeneous systems,” in _AAAI_ , vol. 36, no. 7, 2022, pp. 8062–8071.
  * [27] D. Yao, W. Pan, M. J. O’Neill, Y. Dai, Y. Wan, H. Jin, and L. Sun, “Fedhm: Efficient federated learning for heterogeneous models via low-rank factorization,” _arXiv preprint arXiv:2111.14655_ , 2021.
  * [28] C. Ma, J. Li, M. Ding, H. H. Yang, F. Shu, T. Q. S. Quek, and H. V. Poor, “On safeguarding privacy and security in the framework of federated learning,” _IEEE Network_ , vol. 34, no. 4, pp. 242–248, 2020.
  * [29] M. Fang, X. Cao, J. Jia, and N. Gong, “Local model poisoning attacks to Byzantine-Robust federated learning,” in _USENIX Security 20_ , Aug. 2020, pp. 1605–1622.
  * [30] C. Thapa, P. C. M. Arachchige, S. Camtepe, and L. Sun, “Splitfed: When federated learning meets split learning,” in _AAAI_ , vol. 36, no. 8, 2022, pp. 8485–8493.


# ST-Gait++: Leveraging spatio-temporal convolutions for
gait-based emotion recognition on videos

Maria Luísa Lima
Voxar Labs, Centro de Informática
Universidade Federal de Pernambuco
Av. Jorn. Aníbal Fernandes, Recife, Brazil
<EMAIL_ADDRESS>Willams de Lima Costa
Voxar Labs, Centro de Informática
Universidade Federal de Pernambuco
Av. Jorn. Aníbal Fernandes, Recife, Brazil
<EMAIL_ADDRESS>Estefania Talavera Martínez
University of Twente
Drienerlolaan 5, 7522 NB
Enschede, Netherlands
<EMAIL_ADDRESS>Veronica Teichrieb
Voxar Labs, Centro de Informática
Universidade Federal de Pernambuco
Av. Jorn. Aníbal Fernandes, Recife, Brazil
<EMAIL_ADDRESS>
###### Abstract

Emotion recognition is relevant for human behaviour understanding, where
facial expression and speech recognition have been widely explored by the
computer vision community. Literature in the field of behavioural psychology
indicates that gait, described as the way a person walks, is an additional
indicator of emotions. In this work, we propose a deep framework for emotion
recognition through the analysis of gait. More specifically, our model is
composed of a sequence of spatial-temporal Graph Convolutional Networks that
produce a robust skeleton-based representation for the task of emotion
classification. We evaluate our proposed framework on the E-Gait dataset,
composed of a total of 2177 samples. The results obtained represent an
improvement of $\approx 5\%$ in accuracy compared to the state of the art. In
addition, during training we observed a faster convergence of our model
compared to the state-of-the-art methodologies.

##  1 Introduction

Humans have a natural perception capability that allows us to capture,
process, and understand behavioral cues from other people naturally [15].
There are several biological triggers inside our brains that plan our
interactions with other people based on this perception [20]. However, human
behavior is a very broad cognitive spectrum with multiple different nuances
that can affect this planning procedure. When looking at social interactions,
however, emotion is a specific part of behavior that plays a significant role.
The ability to perceive and respond to emotional aspects is essential to
develop and maintain links with peers in society.

In multiple contexts of applications, we could argue that understanding the
emotions of users is also a significant aspect of the development of systems
that are more inclusive and fair. These systems could adapt the way they
perform according to how they perceive their users. However, to allow these
systems to understand the emotions of their user, first, they need to be able
to extract and process emotional information from them. Researchers have been
studying how humans perceive and process these affective cues for a while, and
evidence from the behavioral psychology literature suggests that a significant
part of affective information is communicated naturally and intuitively
through a medium known as nonverbal communication [2, 16, 13].

Among the vast list of nonverbal communication cues, some studies highlight
the importance of bodily expression for emotion recognition. Early studies
such as those by Wallbott and Scherer [31] suggested that there are specific
body movements that lead to an accurate perception of emotions. Therefore,
this strong evidence from the literature on behavioral psychology validates
that the body acts as an outlet for the person’s emotional state, as well as
signals that, by extracting and processing these cues, someone can perceive
emotional aspects through observation of certain characteristics. Recent
studies have shown significant success in encoding body language to recognize
affect in humans through deep learning and computer vision, indicating that
body expressions are a significant cue when building affect-aware technology
[14, 27, 4, 33].

However, body language, as well as other affective features such as facial
expressions, gaze, gestures, or physiological indicators such as cardiac
frequency or respiratory rate, share a common limitation related to
applications: they require the user to be facing toward a camera so that the
affective sources (e.g., face, eyes, arms…) are visible at all times. Although
this constraint is not harmful in some scenarios, such as when people are
facing a computer or other affective agents (such as social robots), it is
also a limiting aspect in ubiquitous applications since the user would not be
able to freely interact with the environment. We could, however, communicate
to the user this constraint and ask the user to face the system; however, this
would alter the user’s behavior, changing how they would communicate their
emotions and removing all naturality from this interaction, which would now be
an artificial interaction.

A possible way to encode body language and still not limit the user is to look
at the person’s gait to extract affective information. Gait is the description
of the way that someone walks, and previous research in behavioral psychology
has found that humans are able to identify multiple social aspects through
gait-related parameters [18]. Roether et al. [25] applied machine learning to
a set of recorded gaits and found that some features, such as movement speed
and limb flexion, were essential in correlating emotions and gaits. These
pieces of evidence suggest that evaluating gait in a spatiotemporal manner can
lead to strong classifications of emotion. With this, new applications in
multiple scenarios are enabled, for example: we could leverage gait
information to monitor freely moving citizens for public safety, where
collective negative emotions could indicate a dangerous situation taking place
[10]; wellbeing applications such as urbanism, where collective spaces can be
changed according to what people experience in them [30]; or even healthcare
where the psychological monitoring of a patient in internment can prove to be
useful not only as an overall better treatment, especially if non-invasive,
but also to enable mental health professionals to better identify mental
health issues in their patients by having another information source to look
into (Dhuheir et al. [8]).

Based on this inspiration, we propose an approach for emotion recognition
using gait. Through the use of spatiotemporal processing blocks, we overcome
limitations present in the current state-of-the-art [1], which possess a
limited processing capability in this aspect. We also show that by overcoming
this limitation, our results in the proposed quantitative metric are also
better than the state-of-the-art. This work’s contributions are as follows:

  * •

An emotion recognition method through gait and body language, compatible with
behavioral psychology studies, with a $\approx 5\%$ increase in accuracy
relative to the state-of-the-art

  * •

A model that converges 3.63 times faster in training, saving time and
computational resources, allowing for faster testing and experimentation.

The rest of the paper is organized as follows: Next, in Section 2, we visit
the state of the art. In Section 3, we present the methods utilized and in
Section 4, we expose our experimental setup. Later, in Section 5, we present
the results and discuss them. Finally, Section 6 shows our conclusions and
present ideas for future research.

##  2 Related Works

Early works for emotion recognition through gaits were based on extracting
features from motion capture data, which is acquired by using special clothing
with landmark points and a motion capture camera, and using algorithms to
calculate similarity indexes with databases. Venture et al. [29] captured
walks from four different performers and applied PCA to verify that these
emotions could be distinguished through some affective features.

A significant improvement was proposed by Daoudi et al. [7], which evaluated
this task from a geometric approach. They have represented skeleton joints
over time using covariance matrices, which were mapped to the Riemannian
manifold of symmetric positive definite matrices. This allowed the authors to
exploit multiple geometric properties for emotion classification. However,
this approach is limited by how much temporal information can be encoded,
imposing a limited sequence modeling.

The natural next step here was, then, related to improving the temporality
aspect of these previous approaches. Randhavane et al. [23] presented a new
approach, now focused on RNNs, thus allowing improved spatiotemporal
relationship. They combined affective features, such as the angles between
joints and stride length, with deep features that were learned using a Long
Short-Term Memory architecture.

However, the advances on Graph Convolutional Networks at that time, especially
the proposal of the ST-GCNs by Yan et al. [32] allowed for an even more robust
way of learning these relationships. Bhattacharya et al. [1] proposed using
such architecture to extract features from videos and classify the emotions in
an implicit manner.

Still, while ST-GCNs provide an effective approach, there are some limitations
that this work aims to address. The representational capacity of the base ST-
GCN is predefined rather than learned, which was sufficient for its originally
intended application of activity recognition. However, for perceiving
emotional cues through nonverbal behaviors like gait, these cues are often
more subtle than the movements used for activity recognition. Therefore, not
learning the topology may limit the ability to capture these subtle movement
patterns that are indicative of different emotions.

This is solved by using the ST-GCN++ architecture, which brings the novelty of
learning the topology during training. Another limitation is in the temporal
aspect, with ST-GCNs having a fixed size temporal kernel, which was rendered
adjustable in the ST-GCN++ architecture. Therefore, switching from ST-GCN to
ST-GCN++, is very advantageous for the emotion recognition using gaits
problem.

##  3 Method

Given a video
$V\in\mathcal{R}^{\mathsf{n}\times\mathsf{h}\times\mathsf{w}\times 3}$ with
$\mathsf{n}$ frames, height $\mathsf{h}$ width $\mathsf{w}$ and a set of
emotions $\mathbb{K}$, our task is to classify the perceived emotion of a
person present in such video by extracting features related to body language
and gait. We first extract a set of 3D body keypoints
$\mathbf{K}\in\mathcal{R}^{16\times 3}$, in which
$\mathbf{k}_{1},\mathbf{k}_{2},\ldots,\mathbf{k}_{16}$, each $\mathbf{k}_{i}$
represents the location of a body joint in space related to the person in the
video.

Figure 1: The proposed architecture for ST-Gait++, composed of $3$ ST-GCN++
blocks with outputs sized $32$, $64$ and $64$, followed by a $2D$ Average
Pooling and a $1\times 1$ Convolution, which reduces dimensionality from $64$
to $4$, which is followed by a Softmax. Ideally, in an application scenario,
this can be used along some Skeletal trajectory extraction which takes as
input a video and outputs the gait sequences to be analysed by ST-Gait++
automatically. This work focuses on the Skeletal trajectory classification.

#### (a) Skeletal trajectory extraction.

One of the possible ways to represent a skeleton is through a graph, since
these representations can be considered analog. Each body joint, such as the
right shoulder or right elbow can be considered a vertex, and the bones that
connects these two joints can be considered edges. This is a clear indicative
on why GCNs can be used to process these types of data. Therefore, at a given
timestamp $t$, we extract the skeleton of the person visible on the scene and
represent it as a graph: $\mathbb{G}=(\mathbb{V},\mathbb{E})$, where
$\mathbb{V}$ is the vertices (or joints) set and $\mathbb{E}$ is the edge (or
bones) set and $\mathbb{N}=|\mathbb{V}|$ the number of vertices.

#### (b) Skeletal trajectory classification.

We use this graph $\mathbb{G}$ as input for our gait processing model. We
propose using ST-GCN++ blocks [9] to learn the joint representations and
discover movement patterns related to perceived emotions. This way, nonverbal
cues such as step size, arm swinging, head angle relative to the shoulders,
among others, can be extracted automatically and without user intervention.

The extracted gait features are propagated from the body joints in the shape
of $\mathbb{X}\in\mathcal{R}^{\mathsf{n}\times\mathsf{f}}$ with
$x_{i}\in\mathcal{R}^{\mathsf{f}}$ representing a feature of the $i^{th}$
vertex. The propagation rule is done in the following manner:
$\mathbb{Z}^{(l+1)}=\sigma(\mathbb{AZ}^{(l)}\mathbb{W}^{(l)})$, where
$\mathbb{Z}^{l}$ and $\mathbb{Z}^{l+1}$ are the inputs to the network layers
$l$ and $l+1$, with $\mathbb{Z}^{0}=\mathbb{X}$. $\mathbb{W}^{l}$ and
$\mathbb{W}^{l+1}$ are the weight matrices between layers $l$ and $l+1$, and
$\mathbb{A}$ is the adjacency matrix of $\mathbb{G}$ and $\sigma(.)$ is a
nonlinear activation function. Each weight matrix $\mathbb{W}$ represents a
convolutional kernel which can be used to obtain features. For example, the
application of $k$ kernels of dimension $\mathsf{f}\times\mathsf{d}$ in an
input $\mathbb{X}$, the output corresponds to a feature vector of dimension
$\mathsf{n}\times\mathsf{d}\times\mathsf{k}$. There’s also the set of adjacent
vertices $\mathit{A}^{t}_{i}\subseteq\mathbb{V}$ to $v^{t}_{i}$ at time $t$
[1].

Our proposed model ST-Gait++ is composed of $3$ ST-GCN++ blocks with $32$,
$64$, and $64$ kernels each, followed by an average pooling, a $2$D $1$x$1$
convolution layer, and a softmax layer for the $4$ emotion categories. This
design was chosen empirically to overcome the limitations presented previously
in Section 2, but also because other works experiment with similar
architectures, such as STEP [1]. Also, according to the methodology described
by the author [1], affective features can extracted from the data, so those
are extracted and added to the used data as well. We overview our model in
Figure 1.

##  4 Experimental setup

###  4.1 Dataset

We used the Emotion-Gait (E-Gait) dataset in our experiments. The dataset that
is currently available is a modified version of the original dataset made
available by Bhattacharya et al. [1]. The authors [1] do not share specifics
of what has changed or when it did, so we try our best to keep a clear
comparison with other techniques from the state of the art. We only used the
real data from the E-Gait, because of some quality issues perceived during
early experimentation with the synthetic data, alongside the issue of the
changes on the dataset.

The data consists on $342$ samples collected by the authors and 1,835 samples
collected by the Edinburgh Locomotion Mocap Database (ELMD) [11] and its
distribution per train/validation/test split and categories can be found in
Table 1. The data is composed of already extracted and normalized skeleton
sequences forming different labeled gaits in the $4$ emotions targeted in this
paper: Happy, Neutral, Sad and Angry. Each sample is shaped $T\times V$ with
$T$ being the number of time steps and $V$ the number of coordinates which is
equal to $48$ here since there are $16$ joins with $3$ dimensions each. We
show samples of the skeletons present in the dataset in Figure 2.

Table 1: E-Gait dataset sample distribution according to training, validation
and testing splits and the four emotion labels present on E-Gait: Angry,
Neutral, Happy and Sad.

|  | Category | Total
---|---|---|---
|  | Angry | Neutral | Happy | Sad
Split | Train | 818 | 332 | 237 | 136 | 1523
Val | 220 | 107 | 65 | 46 | 438
Test | 122 | 48 | 30 | 16 | 216

(a) Angry

(b) Happy

(c) Neutral

(d) Sad

Figure 2: Examples of the four categories of the E-Gait dataset. Each item is
a sample from one of the categories of the E-Gait dataset, with each frame of
the six-frame sequence taken from the whole sample gait sequence. This was
done to provide a sense of movement to the reader, so they can better
understand E-Gait’s characteristics.

###  4.2 Validation metric

We have used the accuracy metric, which is a common metric used in multiple
emotion recognition baselines [5]. This metric is also employed in other works
that use E-Gait as the evaluation benchmark.

###  4.3 Implementation details

We have used PyTorch $1.7$ to build and train our model. We have used the
publicly available implementation for the ST-GCN++ block published by Duan et
al. [9] 111Available at https://github.com/kennymckormick/pyskl. From this
point, we have applied a Bayesian Search algorithm for hyperparameter tuning.
Bayesian Search is a powerful yet simple technique for optimizing
hyperparameters by modelling objective functions and updating its belief based
on observed results. Therefore, instead of using a random set of
hyperparameters, which would be computationally costly, this method will
choose the set of hyperparameters that has the highest chance of leading to
better results and will discard those with low chance. The search space we
navigated using this algorithm is shown in Table 2, and the set of chosen
hyperparameters in comparison to those of STEP [1] are shown in Table 3. We
have trained ST-Gait++ for $200$ epochs.

Table 2: Search space defined for the Bayesian Search. For each parameter, a
set of values are chosen as a search space. As the Bayesian Search finds an
optimal value within the search space for each parameter, this value is
represented on the third column of the table. These optimal values were used
to train ST-Gait++.

Parameter | Search Space | Value
---|---|---
Basic Learning Rate | 0.01, 0.1, 0.3 | 0.01
GCN Initializer | Importance, Offset | Offset
Optimizer | Adam, RMSProp, SGD | RMSProp
Weight Decay |  $3\times 10^{-4}$, $1\times 10^{-4}$ | $3\times 10^{-4}$

Table 3: Training hyperparameters for STEP and ST-Gait++. The learning rate decay happens after epochs 250, 375 and 438 and wasn’t used on ST-Gait++. The remaining parameters are the same as STEP, except GCN Initializer which doesn’t exist on STEP. Parameter | Model
---|---
STEP | ST-Gait++
Train/Val/Test | 7:2:1 | 7:2:1
Epochs | 500 | 200
Batch Size | 8 | 8
Optimizer | Adam | RMSProp
Basic LR | 0.1 | 0.01
LR Decay | 1/10 | N/A
Momentum | 0.9 | 0.9
Weight Decay | $5\times 10^{-4}$ | $3\times 10^{-4}$
GCN Initializer | N/A | Offset

The loss used during training for both STEP and ST-Gait++ was Cross Entropy
Loss. We used the Pytorch implementation 222Available at:
https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html.
Since the predictions for each sample were in the form of probabilities for
each class, the Cross Entropy Loss can be defined as:

$l(x,y)=\frac{\sum_{n=1}^{N}(-\sum_{c=1}^{C}w_{c}log\frac{exp(x_{n,c})}{\sum_{i=1}^{C}exp(x_{n,i})}y_{n,c})}{N}$

Where $l$ is the loss, $x$ is the input, $y$ is the target, $w$ is the weight
$C$ is the number of classes and $N$ is the minibatch size.

##  5 Results and discussion

(a)

(b)

Figure 3: Confusion matrices generated from evaluating the models on the test
set for (a) ST-Gait++ and (b) STEP. As can be seen, there is a more pronounced
diagonal on (a), emphasizing the better accuracy of ST-Gait++. Also, there is
less confusion between the emotions Neutral and Happy on ST-Gait++ than on
STEP. However, there is some increase in confusion between Happy and Sad on
ST-Gait++.

#### Quantitative analysis.

We compare our results with other different approaches in Table 5. Our
proposed method outperforms other graph-related methods, such as STEP [1]. It
is interesting to notice that our implementation of this work has led to an
increased accuracy than their reported results. In this case, our model has
had an increase of $5.4\%$ regarding their result, and $4.2\%$ regarding our
implementation. Besides the accuracy increase, our model was also able to
converge faster, highlighting several improvements, such as fewer requirements
for computational resources or training time, increased possibilities for
scaling, and better generalization of data. ST-Gait++ converged on epoch #127,
while STEP converged on epoch #462, a reduction of $72\%$ in training time. A
runtime analysis was also performed, over the test set, and can be found on
table Table 4. As observed, STEP has a better time for inference, which can be
explained by the simpler architecture.

Table 4: Runtime analysis on the test set, which has a total of 216 samples.
The test was performed on a laptop with a NVIDIA GeForce RTX 4050 GPU.

Method | Min Inference Time | Average Inference Time
---|---|---
STEP | 0.99ms ($\sim$1001 fps) | 3.28ms ($\sim$304 fps)
ST-Gait++ (ours) | 4.64ms ($\sim$215 fps) | 15.08ms ($\sim$66 fps)

Our model also highly outperforms other approaches with a temporal focus, such
as Randhavane et al. [22]. In this case, we report an accuracy increase of
$6.8\%$. Overall, this quantitative evaluation highlights that our model not
only has increased accuracy in relation to the current state-of-the-art, but
is also more optimized towards training requirements.

Table 5: Quantitative analysis of methods for emotion recognition using the
E-Gait dataset.

Methods | Acc (%)
---|---
Venture et al. (2014) | 30.8
Daoudi et al. (2017) | 42.5
Li et al. (2016) | 53.7
Crenn et al. (2016) | 66.2
Randhavane et al. (2019a) | 80.7
Narayanan et al. (2020) | 82.4
Bhattacharya et al. (2020) (STEP) | 82.1
Bhattacharya et al. (2020) (STEP) (Our implementation) | 83.3
YuMeng et al. (2024) | 85.2
ST-Gait++ (Ours) | 87.5

We have also generated confusion matrices for ST-Gait++ and STEP in order to
compare how the accuracy of these two models can be represented in how they
perceive the emotional classes differently. We show these results in Figure 3.
As expected, this result reaffirm the results we have discussed before; STEP
has a higher confusion between Neutral and Happy than our model. In the case
of facial expressions, the literature on behavioral psychology has shown that
there is a structural resemblance between neutral and happy faces that could
lead to confusion, and it is common to have this type of ambiguity in these
tasks [26, 4]; however, there is still no evidence that the same could happen
to gait perception. Therefore, it is difficult to judge if STEP’s ambiguity
between these two classes could be justified. In our model, we also notice an
ambiguity between Sad and Happy. We do not have any insights regarding this
curious behavior at this point, but we also highlight that our accuracy for
the Sad class is still higher than the reported for STEP.

Finally, we have also applied a t-SNE representation of the features extracted
from the last layer of the models to visualize and analyze the learned
representations of the data. The last layer usually contains higher-level,
abstract features of the input data, which could usually be represented by
activation maps or feature maps. In this case, since we are not directly
working with images at that point, applying t-SNE could reveal clusters or
groups that could highlight the ability of the model to distinguish classes or
categories effectively, as well as to identify outliers or anomalies in the
data. We show in Figure 4 that the Angry class has a very distinct and
separated clustering, which explains the better performance in this class as
we have shown before in Figure 3. This better performance can also be
attributed to the far larger amount of samples for this category, making the
model better able to separate it from the rest. Neutral and Happy categories
are distinguishable enough to show why they also show a good performance,
while Sad, the least numerous category, is very mingled with the others.

#### Qualitative analysis.

In Figure 5 we show some qualitative visualizations with examples of correct
and incorrect predictions. By looking just at the skeletons it’s impossible to
distinguish the emotions. For some, we may agree with the annotation provided
and understand why the model made a correct prediction. For others, it’s
dubious to infer the perceived emotion with just the skeleton, and we can
understand why the model made a mistake. On (a) we can see the wide stride,
arm swinging and attribute that to happiness, but we can also see the somewhat
lowered head, which could indicate sadness. On (b) we can se a fast stride and
arm swinging that could indicate some more energetic emotion, but the model
still correctly classifies it as sadness. (c) gives us a lowered head, which
could have lead the model to infer on sadness, even though the swinging arms
and big stride indicate anger. Lastly, (d) shows a fast stride with slightly
swinging arms and upperbody which could have misled the model into inferring
happiness and not some neutral emotion. Overall, we can understand that there
is some dubiousness in the data that leads to explainable mistakes.

However, given the temporality aspect of this evaluation, we are limited to
what we can show in this research paper. We have prepared a video with a more
in-depth overview available at to be made available upon publication.

Figure 4: T-SNE representation of the features extracted from the last
convolutional layer of ST-Gait++ and STEP on the test set, The inner color
represents the ground truth labels and the outer circle represents the model’s
inference.

(a) Ground Truth: Happy — ST-Gait++ Prediction: Happy

(b) Ground Truth: Sad — ST-Gait++ Prediction: Sad

(c) Ground Truth: Angry — ST-Gait++ Prediction: Sad

(d) Ground Truth: Neutral — ST-Gait++ Prediction: Happy

Figure 5: Examples of correct and incorrect inferences by ST-Gait++ on the
test set.

#### Limitations of the E-Gait dataset.

To the best of our knowledge, E-Gait is one of the only public datasets of
emotion recognition through gait, alongside Emotion-Walk (E-Walk) [22].
However, since they have data overlap, we decided to test on E-Gait, given
that it is the one used by the state-of-the-art approaches of emotion
recognition from gait.

In addition, the E-Gait dataset [1] is very obscure, with only the skeletons
being available to researchers. Because of this, the annotations cannot be
confirmed and neither can we know the transformations taken to process the
videos and the skeletons which are already normalized. This can be observed in
Figure 5. Furthermore, the dataset does not disclose the demographics of the
subjects and a great part of the data comes from the capture with a single
individual and adds a heavy bias. For an application in other contexts, such
as Latin American contexts, for example, it would be very interesting to have
a dataset that could show the local cultural emotion expression. Also,
publishing an open dataset also containing the original videos, not only the
skeletons, would give researchers a higher freedom for experimenting,
validating ideas and checking for biases to correct.

The first step to making such a dataset would be to gather other available
datasets for emotion recognition through gait, or curate new ones, focusing on
bringing high quality data that is representative, diverse and as unbiased as
possible. Testing ST-Gait++ on more data will certainly point to new paths of
improvement for this research.

#### Diversity and Bias in emotion recognition related datasets.

There are many studies focused on trying to find biases in intelligent
systems. One such study is Rhue [24], which found racial disparities in facial
emotion recognition and raised the question of whether artificial inteligence
could in fact determine emotion better than people. To answer this question,
we need to understand that demographics such as gender, race and ethnicity
heavily influence the perception of human characteristics, such as in facial
emotion recognition algorithms [3]. But this is also due to the fact that we
as humans perceive the world based on our own biases. To annotate the data
that will be fed into algorithms is to accept that the data will have biases
that the model will propagate. Studies such as Pahl et al. [21] bring to
attention the age bias, besides the ethnicity bias, in prominent Action Unit
Datasets. The algorithms studied had problems with glasses but not with
beards. This shows data collection problem: Are there no diverse people
available or are these people not even considered as a possible variation in
the target user public? These limitations can be extended to gaits, as it is a
characteristic and emotion expression outlet that can vary across different
demographics.

Furthermore, although some emotion expression may have universal features
[28], it is noted that cultural particularities are very influential in
affective cues encoding Kleinsmith and Bianchi-Berthouze [14]. For a general
application, having an analysis on the demographic characteristics can help
researchers gain more insight on the limitations of the data and, as a
consequence, of the real world application of their research. As such, having
such a skewed dataset, such as ELMD (Habibie et al. [11]), being a
considerable part of the total data available, brings to question the
applicability of the entire dataset in in-the-wild scenarios.

Given the ELMD dataset, of the $342$ samples recorded, 90 participants were
involved, with no demographic of this public being disclosed. It is disclosed
that the data, all $1,835$ samples, was collected using a sole male actor.
With this in mind, it’s important to point out the importance of diversity and
fairness in this data collection.

Also, it’s important to emphasize that what is being dealt with here are the
perceived emotions, since whatever actual emotion was being felt by the person
at the time of data capture is only available if the person is questioned (in
real scenarios) or if we know what they are trying to convey (actors
portraying emotions). Even in real scenarios, the reported emotion after
questioning could be biased due to the type of question asked or recall biases
[12].

##  6 Conclusion

In this work, we proposed ST-Gait++, a novel framework for emotion recognition
from gait. Its skeleton-based spatio-temporal representation approach results
in state-of-the-art classification performance on the E-Gait dataset. We also
discussed some of the limitations of the field with the objective of
presenting research opportunities.

In addition, given the faster training convergence on a consumer grade laptop,
we expect ST-Gait++ to provide new research opportunities in the field of
human behaviour analysis for researchers with a lower budget or limited
resources.

Future work will explore the relevance of the different body parts for
recognising emotion and the inclusion of additional gait descriptors.

## References

  * Bhattacharya et al. [2020] Uttaran Bhattacharya, Trisha Mittal, Rohan Chandra, Tanmay Randhavane, Aniket Bera, and Dinesh Manocha.  Step: Spatial temporal graph convolutional networks for emotion perception from gaits.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 1342–1350, 2020.
  * Buck [1991] R Buck.  Motivation, emotion and cognition: A developmental-interactionist view.  _International review of studies on emotion_ , 1:101–142, 1991.
  * Buolamwini [2017] Joy Adowaa Buolamwini.  _Gender shades: intersectional phenotypic and demographic evaluation of face datasets and gender classifiers_.  PhD thesis, Massachusetts Institute of Technology, 2017.
  * Costa et al. [2022] Willams Costa, David Macêdo, Cleber Zanchettin, Estefanía Talavera, Lucas Silva Figueiredo, and Veronica Teichrieb.  A fast multiple cue fusing approach for human emotion recognition.  _Available at SSRN 4255748_ , 2022.
  * Costa et al. [2023] Willams Costa, Estefanía Talavera, Renato Oliveira, Lucas Figueiredo, João Marcelo Teixeira, João Paulo Lima, and Veronica Teichrieb.  A survey on datasets for emotion recognition from vision: Limitations and in-the-wild applicability.  _Applied Sciences_ , 13(9):5697, 2023.
  * Crenn et al. [2016] Arthur Crenn, Rizwan Ahmed Khan, Alexandre Meyer, and Saida Bouakaz.  Body expression recognition from animated 3d skeleton.  In _2016 International Conference on 3D Imaging (IC3D)_ , pages 1–7. IEEE, 2016.
  * Daoudi et al. [2017] Mohamed Daoudi, Stefano Berretti, Pietro Pala, Yvonne Delevoye, and Alberto Del Bimbo.  Emotion recognition by body movement representation on the manifold of symmetric positive definite matrices.  In _Image Analysis and Processing-ICIAP 2017: 19th International Conference, Catania, Italy, September 11-15, 2017, Proceedings, Part I 19_ , pages 550–560. Springer, 2017.
  * Dhuheir et al. [2021] Marwan Dhuheir, Abdullatif Albaseer, Emna Baccour, Aiman Erbad, Mohamed Abdallah, and Mounir Hamdi.  Emotion recognition for healthcare surveillance systems using neural networks: A survey.  In _2021 International Wireless Communications and Mobile Computing (IWCMC)_ , pages 681–687, 2021.
  * Duan et al. [2022] Haodong Duan, Jiaqi Wang, Kai Chen, and Dahua Lin.  Pyskl: Towards good practices for skeleton action recognition.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 7351–7354, 2022.
  * Grant and Williams [2011] Dawn Grant and David Williams.  The importance of perceiving social contexts when predicting crime and antisocial behaviour in cctv images.  _Legal and Criminological Psychology_ , 16(2):307–322, 2011.
  * Habibie et al. [2017] Ikhsanul Habibie, Daniel Holden, Jonathan Schwarz, Joe Yearsley, and Taku Komura.  A recurrent variational autoencoder for human motion synthesis.  In _Proceedings of the British Machine Vision Conference (BMVC)_ , 2017.
  * Hernandez et al. [2021] Javier Hernandez, Josh Lovejoy, Daniel McDuff, Jina Suh, Tim O’Brien, Arathi Sethumadhavan, Gretchen Greene, Rosalind Picard, and Mary Czerwinski.  Guidelines for assessing and minimizing risks of emotion recognition applications.  In _2021 9th International conference on affective computing and intelligent interaction (ACII)_ , pages 1–8. IEEE, 2021.
  * Jacob et al. [2016] Heike Jacob, Benjamin Kreifelts, Sophia Nizielski, Astrid Schütz, and Dirk Wildgruber.  Effects of emotional intelligence on the impression of irony created by the mismatch between verbal and nonverbal cues.  _PloS one_ , 11(10):e0163211, 2016.
  * Kleinsmith and Bianchi-Berthouze [2012] Andrea Kleinsmith and Nadia Bianchi-Berthouze.  Affective body expression perception and recognition: A survey.  _IEEE Transactions on Affective Computing_ , 4(1):15–33, 2012.
  * Lhommet and Marsella [2014a] Margaux Lhommet and Stacy C Marsella.  19 expressing emotion through posture and gesture.  _The Oxford handbook of affective computing_ , page 273, 2014a.
  * Lhommet and Marsella [2014b] Margaux Lhommet and Stacy C Marsella.  Expressing emotion through posture.  _The Oxford handbook of affective computing_ , 273, 2014b.
  * Li et al. [2016] Baobin Li, Changye Zhu, Shun Li, and Tingshao Zhu.  Identifying emotions from non-contact gaits information based on microsoft kinects.  _IEEE Transactions on Affective Computing_ , 9(4):585–591, 2016.
  * Montepare et al. [1987] Joann M Montepare, Sabra B Goldstein, and Annmarie Clausen.  The identification of emotions from gait information.  _Journal of Nonverbal Behavior_ , 11:33–42, 1987.
  * Narayanan et al. [2020] Venkatraman Narayanan, Bala Murali Manoghar, Vishnu Sashank Dorbala, Dinesh Manocha, and Aniket Bera.  Proxemo: Gait-based emotion learning and multi-view proxemic fusion for socially-aware robot navigation.  In _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 8200–8207. IEEE, 2020.
  * Nes et al. [2023] Anders Nes, Kristoffer Sundberg, and Sebastian Watzl.  The perception/cognition distinction.  _Inquiry_ , 66(2):165–195, 2023.
  * Pahl et al. [2022] Jaspar Pahl, Ines Rieger, Anna Möller, Thomas Wittenberg, and Ute Schmid.  Female, white, 27? bias evaluation on data and algorithms for affect recognition in faces.  In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ , pages 973–987, 2022.
  * Randhavane et al. [2019a] Tanmay Randhavane, Uttaran Bhattacharya, Kyra Kapsaskis, Kurt Gray, Aniket Bera, and Dinesh Manocha.  Identifying emotions from walking using affective and deep features.  _arXiv preprint arXiv:1906.11884_ , 2019a.
  * Randhavane et al. [2019b] Tanmay Randhavane, Uttaran Bhattacharya, Kyra Kapsaskis, Kurt Gray, Aniket Bera, and Dinesh Manocha.  Learning perceived emotion using affective and deep features for mental health applications.  In _2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)_ , pages 395–399. IEEE, 2019b.
  * Rhue [2018] Lauren Rhue.  Racial influence on automated perceptions of emotions.  _Available at SSRN 3281765_ , 2018.
  * Roether et al. [2009] Claire L Roether, Lars Omlor, Andrea Christensen, and Martin A Giese.  Critical features for the perception of emotion from gait.  _Journal of vision_ , 9(6):15–15, 2009.
  * Said et al. [2009] Christopher P Said, Nicu Sebe, and Alexander Todorov.  Structural resemblance to emotional expressions predicts evaluation of emotionally neutral faces.  _Emotion_ , 9(2):260, 2009.
  * Thuseethan et al. [2022] Selvarajah Thuseethan, Sutharshan Rajasegarar, and John Yearwood.  Emosec: Emotion recognition from scene context.  _Neurocomputing_ , 492:174–187, 2022.
  * Tracy and Matsumoto [2008] Jessica L Tracy and David Matsumoto.  The spontaneous expression of pride and shame: Evidence for biologically innate nonverbal displays.  _Proceedings of the National Academy of Sciences_ , 105(33):11655–11660, 2008.
  * Venture et al. [2014] Gentiane Venture, Hideki Kadone, Tianxiang Zhang, Julie Grèzes, Alain Berthoz, and Halim Hicheur.  Recognizing emotions conveyed by human gait.  _International Journal of Social Robotics_ , 6:621–632, 2014.
  * Viderman and Knierbein [2020] Tihomir Viderman and Sabine Knierbein.  Correction to: affective urbanism: towards inclusive design praxis.  _Urban Design International_ , 25:192–202, 2020.
  * Wallbott and Scherer [1986] Harald G Wallbott and Klaus R Scherer.  Cues and channels in emotion recognition.  _Journal of personality and social psychology_ , 51(4):690, 1986.
  * Yan et al. [2018] Sijie Yan, Yuanjun Xiong, and Dahua Lin.  Spatial temporal graph convolutional networks for skeleton-based action recognition.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1), 2018.
  * Yang et al. [2022] Dingkang Yang, Shuai Huang, Shunli Wang, Yang Liu, Peng Zhai, Liuzhen Su, Mingcheng Li, and Lihua Zhang.  Emotion recognition for multiple context awareness.  In _European Conference on Computer Vision_ , pages 144–162. Springer, 2022.
  * YuMeng et al. [2024] Zhao YuMeng, Liu Zhen, Liu TingTing, Wang YuanYi, and Chai YanJie.  Affective-pose gait: perceiving emotions from gaits with body pose and human affective prior knowledge.  _Multimedia Tools and Applications_ , 83(2):5327–5350, 2024.