11institutetext: KITE Research Institute, University Health Network, 550
University Avenue, Toronto, M5G 2A2, Ontario, Canada 22institutetext:
Institute of Biomedical Engineering, University of Toronto, 164 College
Street, Toronto, M5S 3G9, Ontario, Canada
22email<EMAIL_ADDRESS>
# Engagement Measurement Based on Facial Landmarks and Spatial-Temporal Graph
Convolutional Networks

Ali Abedi 11 Shehroz S. Khan 1122

###### Abstract

Engagement in virtual learning is crucial for a variety of factors including
learner satisfaction, performance, and compliance with learning programs, but
measuring it is a challenging task. There is therefore considerable interest
in utilizing artificial intelligence and affective computing to measure
engagement in natural settings as well as on a large scale. This paper
introduces a novel, privacy-preserving method for engagement measurement from
videos. It uses facial landmarks, which carry no personally identifiable
information, extracted from videos via the MediaPipe deep learning solution.
The extracted facial landmarks are fed to a Spatial-Temporal Graph
Convolutional Network (ST-GCN) to output the engagement level of the learner
in the video. To integrate the ordinal nature of the engagement variable into
the training process, ST-GCNs undergo training in a novel ordinal learning
framework based on transfer learning. Experimental results on two video
student engagement measurement datasets show the superiority of the proposed
method compared to previous methods with improved state-of-the-art on the
EngageNet dataset with a %3.1 improvement in four-class engagement level
classification accuracy and on the Online Student Engagement dataset with a
%1.5 improvement in binary engagement classification accuracy. The relatively
lightweight ST-GCN and its integration with the real-time MediaPipe deep
learning solution make the proposed approach capable of being deployed on
virtual learning platforms and measuring engagement in real time.

###### Keywords:

Engagement Measurement Graph Convolutional Network Ordinal Classification
Transfer Learning.

##  1 Introduction

Engagement is intricately linked to the interaction of students’ attention and
interest within a specific context [1]. It emerges not just as a consequence
of pre-existing interests but also has the capability to cultivate new
interests through consistent attention [2]. This aspect is extremely crucial
in educational environments, where the development and maintenance of
engagement play a pivotal role in facilitating content comprehension [1].
Nonetheless, the challenges of measuring and sustaining engagement demand
significant mental and emotional exertion from instructors. The progression in
remote sensing technologies, Artificial Intelligence (AI), and affective
computing holds promise for better measurement of engagement in diverse
learning settings, including remote and virtual learning programs. Utilizing
technologies such as facial expression recognition [3], eye gaze tracking [4],
and physiological stress detection [5], instructors can more effectively
monitor and enhance student engagement. This paper covers the development of
AI algorithms for automated measurement of student engagement in virtual
learning programs, taking into account the definition of engagement in
educational psychology and its measurement methodologies.

Fredricks et al. [6] introduced three components of engagement: affective,
behavioral, and cognitive engagement. Affective engagement relates to the
emotions and attitudes a student has towards a task or its context, for
instance, interest in a subject or feelings of excitement or confusion about a
topic during a learning class [7]. Behavioral engagement involves the
student’s active participation in a task, with indicators including looking at
a computer screen (an on-task behavior) or playing with a mobile phone (an
off-task behavior) during a learning class [8]. Cognitive engagement pertains
to the student’s commitment to learning and their readiness to embrace
challenges, encompassing cognitive outcomes such as recall, learning, and deep
understanding [1].

Booth et al. [1] identified that the primary signals used to measure
engagement components include facial, eye gaze, and body pose features, heart
rate, Electro-Encephalo-Graphy, audio, and computer logs. Given that many of
these signals can be captured by cameras and considering the widespread
presence of cameras in personal computers and mobile phones—the devices
commonly used for participation in virtual learning programs—video data has
emerged as the predominant data modality for automatic AI-driven engagement
measurement [9, 10]. Consequently, most AI algorithms for engagement
measurement have been designed and developed to analyze video data and infer
the level of student engagement from the video [9, 10, 11].

Engagement measurement methods are typically divided into end-to-end or
feature-based categories [9, 10, 11]. While feature-based approaches usually
surpass end-to-end methods (that process raw videos) in performance [12, 9],
they are constrained by the need to extract and identify the most effective
features, often through trial and error. Typically, features from various
modalities must be extracted from videos using multiple computer vision
algorithms or deep neural networks. The extracted features are then analyzed
by other deep neural networks or traditional machine learning methods to
determine engagement [12, 9]. This requirement renders multi-modal feature-
based approaches computationally heavy and challenging for practical
implementation in real-world virtual learning platforms. These computationally
intensive approaches render them impractical for local devices and necessitate
the transfer of privacy-sensitive raw video from local devices to cloud
servers to perform automated engagement measurement. Compared to raw videos,
low-dimensional landmark information extracted from videos, such as facial
landmarks, hand landmarks, and body joints [13], offers not only more compact
data but also encapsulates essential discriminative geometric features
necessary for human affect [14] and behavior analysis [15], including
engagement indicators. Most importantly, this geometric information is devoid
of personally identifiable information [16].

In this paper, an alternative course away from conventional end-to-end and
feature-based approaches is charted, and a novel engagement measurement
technique using facial landmarks extracted from videos is presented. The
proposed method is characterized by its privacy-preserving nature and
computational efficiency. This work makes the following contributions:

  * •

This is the first instance in the field of video-based engagement measurement
[9, 10, 11, 17, 18, 19, 20, 12, 21, 22, 23, 24, 25, 26, 27, 28] where facial
landmarks extracted from videos, as the unique data modality, are analyzed
using Spatial-Temporal Graph Convolutional Networks (ST-GCNs) [29] to infer
the engagement level in the video;

  * •

To integrate the ordinal nature of the engagement variable into the training
process, ST-GCNs undergo training in a novel ordinal learning framework
utilizing transfer learning;

  * •

Extensive experiments conducted on two video-based engagement measurement
datasets demonstrate the superiority of the proposed method over previous
methods, achieving an improved state-of-the-art in engagement level
classification accuracy.

The structure of this paper is organized as follows. Section 2 offers an
overview of the relevant literature. This is succeeded by Section 3, which
details the proposed methodology. Following this, Section 4 describes the
experimental setup and discusses the results obtained with the proposed
method. Lastly, Section 5 concludes the paper and proposes directions for
future research.

##  2 Related Work

In line with the contributions of the proposed work in this paper, the
literature review presented in this section serves a dual purpose: it
discusses previous video-based engagement measurement techniques and explores
the methods previously utilized variants of graph convolutional networks for
the purpose of facial expression and affect analysis.

###  2.1 Engagement Measurement

Karimah et al. [9] conducted a systematic review and explored the measurement
of student engagement in virtual learning environments. They found that most
engagement measurement methods primarily focused on the affective and
behavioral components of engagement. These methods commonly employed datasets
annotated by external observers [11] to train feature-based and end-to-end
machine-learning and deep-learning models. In the following, some of the
relevant feature-based and end-to-end works on video-based engagement
measurement are discussed.

In the domain of end-to-end engagement measurement techniques, deep neural
networks analyze consecutive raw video frames to output the engagement level
of the student in the video. These methods do not employ the extraction of
handcrafted features from the videos; instead, the network is adept at
autonomously learning to extract the most salient features directly from the
videos, utilizing consecutive convolutional layers. The deep neural networks
implemented in these end-to-end approaches include networks capable of video
analysis, such as 3D Convolutional Neural Networks (3D CNNs), as well as
combinations of 2D CNNs with sequential neural networks. For engagement
measurement with 3D video input, networks such as C3D [24, 26], Inflated 3D
CNN (I3D) [25], 3D DenseNet [27], and Video Transformer [30] have been
utilized. In subsequent methods, 2D CNNs and sequential models are jointly
trained to automatically extract visual features from raw video frames,
analyze the sequences of these features, and output the engagement level.
Examples of this approach include the integration of 2D CNN with Long Short-
Term Memory (LSTM) [24], a combination of 2D Residual Neural Network (ResNet)
and Temporal Convolutional Network (TCN) [26], ResNet paired with LSTM
equipped with global attention [31], and EfficientNet combined with LSTM and
TCN [32].

The process of measuring engagement through feature-based techniques is
divided into two stages. The first stage involves extracting behavioral and
affective features from video frames, which depends on either specific domain
knowledge or the use of pre-trained models for facial embedding extraction.
Recent examples of pre-trained models used for extracting facial embedding
features from videos for engagement measurement include Masked Autoencoder for
facial video Representation LearnINg (MARLIN) [33] utilized by Singh et al.
[34] and Emotion Face Alignment Network (EmoFAN) [35] utilized by Abedi et al.
[12]. OpenFace [36] is notably recognized for its effectiveness in extracting
features associated with facial Action Units (AUs), eye movements, gaze
direction, and the positioning of the head. OpenFace [36] has been widely used
in feature-based engagement measurement techniques. The extracted features are
then combined in various ways to create sequences of feature vectors for the
video. These feature vector sequences, taken from consecutive video frames,
form multivariate time series data. To analyze the multivariate time series,
machine learning and deep learning models such as Bag-of-Words (BoW) models
[37], variations of Recurrent Neural Networks (RNNs) [21], TCNs [12],
Transformers [34], or ensemble models [38] are employed. These models are
trained on the sequences of features to derive levels of engagement [20, 39,
31, 40, 22, 41, 42, 17, 43, 44, 18, 19].

Prior studies have demonstrated the advantages of feature-based methods over
end-to-end approaches in measuring engagement. Yet, feature-based methods are
constrained by the need to extract and pinpoint the most discriminative
features, often through a process of trial and error. Additionally, they
necessitate the segmentation of videos into multiple segments (comprising
multiple frames) and the extraction of diverse multi-modal features from all
video segments using multiple deep neural networks. For example, techniques
employing OpenFace 2.0 [36] for extracting features such as eye gaze, head
pose, and Action Units (AUs) involve the use of multiple complex neural
networks [4, 45, 46]. This makes the methods time-intensive during the
inference phase, as the features extracted must be further analyzed by another
deep neural network to make a final inference regarding engagement [34].

###  2.2 Graph-based Facial Affect and Expression Analysis

Liu et al. [3] conducted a comprehensive review of the literature on graph-
based methods for facial affect analysis. These methods typically take an
image or a sequence of images as input and produce an affect classification or
regression as output. Based on their review, Liu et al. [3] proposed a
pipeline for graph-based facial affect analysis, which includes (i) face
preprocessing, (ii) graph-based affective representation, and (iii) graph-
based relational reasoning. The preprocessing involves necessary steps such as
face detection and registration. Graph-based affective representation involves
defining the structure of the graph, i.e., nodes and edges. The graph
structure can be spatial, or spatiotemporal when the input data is still
images, or video frames, respectively. The graph structure can be at landmark-
level, region-level, or AU-level with nodes representing facial landmarks,
facial regions of interest, or facial AUs, respectively. In the relational
reasoning step, the edges, nodes, their interrelations, and temporal dynamics
are analyzed through relational reasoning machine-/deep-learning models to
make an inference regarding affect. The machine-/deep-learning models used to
analyze graph data include dynamic Bayesian networks, RNNs, CNNs, fully-
connected neural networks, and non-temporal and temporal graph neural networks
[3].

The facial expression recognition method proposed by Zhou et al. [47] is among
the methods where affective representation was performed by a spatiotemporal
landmark-level and region-level graph. The method utilized thirty-four facial
landmarks located around the eyes, lips, and cheeks, and the connections among
these landmarks formed the intra-frame graph. These intra-frame connections
were defined manually. To establish the inter-frame connections, each node in
a frame was linked to its corresponding node in the subsequent frames. Two
parallel ST-GCNs with analogous structures were trained: one on the x and y
coordinates of the nodes, and the other on the histogram of orientation
features derived from regions surrounding each node. The outputs from these
two ST-GCNs were concatenated and then processed through a fully connected
network for recognizing facial expressions in the input video. Among the
disadvantages of this method are the independent training of the above-
mentioned two ST-GCNs rather than joint learning and the manual definition of
nodes and edges rather than allowing the ST-GCNs to learn the most relevant
nodes and edges. Nodes were selected from Dlib’s facial landmarks [48] and
additional nodes were created on the cheeks and between the eyes according to
Dlib’s facial landmarks. The selection and creation of nodes were based on
their relevance to certain AUs associated with the facial expressions that
should be recognized.

Wei et al. [14] introduced a graph-based method for micro-expression
recognition in video. The introduced approach included a dual-stream ST-GCN,
focusing on the $x$ and $y$ coordinates of facial landmarks and the distance
and angles between adjacent facial landmarks. Recognizing the association
between AUs and micro-expressions, they integrated an AU-specific loss
function into their neural network’s training process. Their methodology
employed three different sets of facial landmarks as graph nodes, comprising
sets with 14, 31, and Dlib’s [48] 68 facial landmarks. Notably, the set with
68 landmarks was the only one to include landmarks along the jawline. The
experiments demonstrated the best results of micro-expression recognition when
14 facial landmark sets were used.

Leong et al. [28] introduced a method for detecting academic emotions through
the use of facial landmarks and the roll, pitch, and yaw of head pose,
utilizing spatial-temporal graph attention networks thereafter. Facial regions
are cropped from video frames, and facial landmarks from these images are
extracted, retaining those around the brow, eye, nose, and mouth, while
excluding those that outline the face’s outer shape, with the reasoning being
they lack correlation with affective states. These features are then inputted
into spatial-temporal graph attention networks to determine academic emotions.
A notable limitation of this approach is its reliance on multiple deep neural
networks for extracting features, i.e., facial landmarks and head pose. The
method achieved lower academic emotion detection accuracy when compared to
prior feature-based methods.

###  2.3 Discussion

As reviewed in this section, while some methods have utilized facial landmarks
and ST-GCNs for video-based facial affect and expression recognition, there
has been no exploration in using facial landmarks or ST-GCNs specifically for
video-based engagement measurement. It is important to note the fundamental
differences between engagement and facial affect and expression making their
measurement different. First, engagement is a multi-component state comprising
behavioral and cognitive components in addition to its affective component. To
illustrate, key indicators of behavioral engagement, such as being off-task or
on-task, are determined by head pose, eye gaze, and blink rate. These
indicators, and therefore engagement, cannot be effectively measured using
methods designed solely for facial affect analysis. Second, engagement is not
a constant state; it varies over time and should be measured at specific time
resolutions where it remains stable and quantifiable. The ideal duration for
measuring engagement is between 10 and 40 seconds, which is longer than the
time resolution for facial affect analysis which is sometimes at frame level.
Third, engagement measurement could involve recognizing an ordinal variable
that indicates various levels of engagement, which differs from facial
expression recognition, where the goal is to identify a categorical variable
with no inherent order.

The limitations of existing engagement measurement approaches, discussed at
the end of subsection 2.1, along with the key differences outlined above
between facial affect analysis and engagement measurement, highlight a gap in
the field of engagement measurement. To address this gap, we propose a simple
yet effective engagement measurement technique. This method is also privacy-
preserving and demonstrates improvements over existing methods on two video-
based engagement measurement datasets.

##  3 Method

The input to the proposed method is a video sample of a student seated in
front of a laptop or PC camera during a virtual learning session. The
sequences of facial landmarks extracted from consecutive video frames through
MediaPipe [13, 49] are analyzed by ST-GCN [29, 15] to output the engagement
level of the student in the video.

###  3.1 Graph-based Representation

MediaPipe [13], an open-source framework developed by Google, facilitates the
construction of applied machine-learning pipelines for processing video data.
The framework’s capabilities extend to a variety of tasks such as face
tracking, hand and pose estimation, and facial landmark estimation, offering
pre-trained models and tools suitable for real-time and streaming applications
[50]. Grishchenko et al. [49] introduced Attention Mesh, which was
incorporated into MediaPipe [13]. Attention Mesh is a pre-trained, end-to-end
3D facial landmark detection model that is both efficient and lightweight. It
is enhanced with an attention mechanism focused on areas of semantic
significance, such as the eyes, iris, and lips. Attention Mesh is capable of
identifying 468 3D facial landmarks across the entire face and an additional
10 landmarks specifically for the iris. However, for the proposed method for
engagement measurement, not all 478 landmarks are utilized. In line with
existing research [3], 68 3D facial landmarks that correspond to those
identified by Dlib’s [48] framework, along with 10 3D landmarks from the iris,
totaling 78 landmarks are utilized.

The 3D facial landmarks encapsulate crucial spatial-temporal information
pertinent to head pose [51], AUs [52], eye gaze [49], and affect [3, 35]—key
features identified in prior engagement measurement studies [9, 10, 11, 17,
18, 19, 20, 12, 21, 22, 23, 24, 25, 26, 27]. Consequently, there is no
necessity for extracting additional handcrafted features from the video
frames.

The $N\ $ 3D facial landmarks extracted from $T$ consecutive video frames are
utilized to construct a spatiotemporal graph $G=(V,E)$. In this graph, the set
of nodes $V=\\{v_{ti}|t=1,\ldots,T,i=1,\ldots,N\\}$ encompasses all the facial
landmarks in a sequence. To construct $G$, first, the facial landmarks within
one frame are connected with edges according to a connectivity structure based
on Delaunay triangulation [53] which is consistent with true facial muscle
distribution and uniform for different subjects [3]. Then each landmark will
be connected to the same landmark in the consecutive frame.

###  3.2 Graph-based Reasoning

Based on the spatiotemporal graph of facial landmarks $G=(V,E)$ constructed
above, an adjacency matrix $A$ is defined as an $N\times N$ matrix where the
element at position $(i,j)$ is set to 1 if there is an edge connecting the
$i^{th}$ and $j^{th}$ landmarks, and set to 0 otherwise. An identity matrix
$I$, of the same dimensions as $A$, is created to represent self-connections.
A spatial-temporal graph, as the basic element of an ST-GCN layer, is
implemented as follows [54, 29].

$f_{\text{out}}=\left(\Lambda^{-\frac{1}{2}}((A+I)\odot
M)\Lambda^{-\frac{1}{2}}f_{\text{in}}W_{\text{spatial}}\right)W_{\text{temporal}}$
(1)

where $\Lambda^{ii}=\sum_{j}(A^{ij}+I^{ij})$. $M$ is a learnable weight matrix
that enables scaling the contributions of a node’s feature to its neighboring
nodes [29]. The input feature map, denoted $f_{\text{in}}$, is the raw
coordinates of facial landmarks for the first layer of ST-GCN, and it
represents the outputs from previous layers in subsequent layers of ST-GCN.
The dimensionality of $f_{\text{in}}$ is $(C,N,T)$, where $C$ is the number of
channels, for example, 3 in the initial input to the network corresponding to
the $x$, $y$, and $z$ coordinates of the facial landmarks. In each layer of
ST-GCN, initially, the spatial (intra-frame) convolution is applied to
$f_{\text{in}}$ based on the weight matrix $W_{\text{spatial}}$, utilizing a
standard 2D convolution with a kernel size of $1\times 1$. Subsequently, the
resulting tensor is multiplied by the normalized adjacency matrix
$\Lambda^{-\frac{1}{2}}((A+I)\odot M)\Lambda^{-\frac{1}{2}}$ across the
spatial dimension. Afterward, the temporal (inter-frame) convolution, based on
the weight matrix $W_{\text{temporal}}$, is applied to the tensor output from
the spatial convolution. This convolution is a standard 2D convolution with a
kernel size of $1\times\Gamma$, where $\Gamma$ signifies the temporal kernel
size.

Following an adequate number of ST-GCN layers that perform spatial-temporal
graph convolutions as outlined above, the resulting tensor undergoes 2D
average pooling, and the final output of the network is generated by a final
2D convolution. This convolution employs a kernel size of $1\times 1$ and
features an output channel dimensionality equal to the number of classes $K$,
i.e., the number of engagement levels to be measured.

###  3.3 Ordinal Engagement classification through Transfer Learning

The model described above, with a final layer comprising $K$ output channels,
tackles the engagement measurement problem as a categorical $K$-class
classification problem without taking into account the ordinal nature of the
engagement variable [55, 39, 11]. The model could harness the ordinality of
the engagement variable to enhance its inferences. Drawing inspiration from
[56, 12], a novel ordinal learning framework based on transfer learning is
introduced as follows.

Training phase– The original $K$-level ordinal labels, $y=0,1,\ldots,K-1$, in
the training set are converted into $K-1$ binary labels $y_{i}$ as follows: if
$y>i$, then $y_{i}=1$; otherwise, $y_{i}=0$, for $i=0,1,\ldots,K-2$.
Subsequently, $K-1$ binary classifiers are trained with the training set and
the $K-1$ binary label sets described above. The training of binary
classifiers is based on transfer learning, which proceeds as follows:
Initially, a network is trained on the dataset with the original $K$-class
labels, employing a regular final layer with $K$ output channels. After
training, the ST-GCN layers of this network are frozen, and the final layer is
removed. To this frozen network, $K-1$ separate untrained 2D convolution
layers with a single output channel each are added, resulting in $K-1$ new
networks. Each of these networks consists of a frozen sequence of ST-GCN
layers followed by an untrained 2D convolution layer. These $K-1$ new networks
are then trained on the entire dataset using the $K-1$ binary label sets
described above. During this phase, only the final 2D convolution layers are
subjected to training.

Inference phase– For the ordinal classification of a test sample, the sample
is initially input into the pre-trained (and frozen) sequence of ST-GCN
layers, followed by a 2D average pooling layer. The tensor obtained from this
process is then input into $K-1$ pre-trained final 2D convolution layers, each
yielding a probability estimate for the test sample being in the binary class
$y_{t}=y_{i}$, where $i=0,1,\ldots,K-2$. Subsequently, these $K-1$ binary
probability estimates are transformed into a single multi-class probability of
the sample belonging to class $y=0,1,\ldots,K-1$, as follows [56].

$p(y_{t}=k)=\begin{cases}1-p(y_{t}\geq 0),&\text{if }k=0,\\\
p(y_{t}>k-1)-p(y_{t}\geq k),&\text{if }0<k<K-1,\\\ p(y_{t}>K-2),&\text{if
}k=K-1.\end{cases}$ (2)

Despite the increased training time for the ordinal model within the
aforementioned ordinal learning framework, the final count of parameters in
the ordinal model remains nearly identical to that of the original non-ordinal
model.

##  4 Experiments

This section evaluates the performance of the proposed method relative to
existing methods in video-based engagement measurement. It reports and
discusses the results of multi-class and binary classification of engagement
across two datasets.

###  4.1 Evaluation Metrics

Based on the engagement measurement problem at hand, several evaluation
metrics are employed. In the context of multi-class engagement level
classification, metrics such as accuracy and confusion matrix are reported.
For binary engagement classification, accuracy, Area Under the Curve of the
Receiver Operating Characteristic (AUC-ROC), and Area Under the Curve of the
Precision and Recall curve (AUC-PR) are utilized.

Table 1:  The distribution of samples in not engaged, barely engaged, engaged, and highly engaged classes in the train and validation sets in the EngageNet dataset [34].  Engagement Level |  Train |  Validation
---|---|---
Not engaged |  1550 |  132
Barely engaged |  1035 |  97
Engaged |  1658 |  273
Highly Engaged |  3740 |  569
Total |  7983 |  1071
Table 2:  The distribution of samples in engaged and not engaged classes in the train, validation, and test sets in the IIITB Online SE dataset [23].  Engagement Level |  Train |  Validation |  Test
---|---|---|---
Not engaged |  570 |  580 |  570
Engaged |  2620 |  1080 |  720
Total |  3190 |  1660 |  1290

###  4.2 Datasets

Experiments on two video-based engagement measurement datasets were conducted,
each presenting unique challenges that further enabled the validation of the
proposed method.

EngageNet The EngageNet dataset [34], recognized as the largest publicly
available dataset for video-based student engagement measurement, includes
video recordings of 127 subjects participating in virtual learning courses.
Each video sample has a duration of 10 seconds, with a frame rate of 30 fps
and a resolution of $1280\times 720$ pixels. Three independent annotators
annotated the subjects’ video recordings as four ordinal levels of engagement:
not-engaged, barely engaged, engaged, and highly engaged. The inter-rater
reliability among the annotators was high, with values of 0.73, 0.77, and
0.79, indicating a substantial agreement level. The dataset was divided into
7983, 1071, and 2257 samples for training, validation, and testing,
respectively, using a subject-independent data split approach [34]. However,
only the training and validation sets were made available by the dataset
creators and were utilized for the training and validation of predictive
models in the experiments presented in this paper. Table 1 details the
distribution of the four engagement classes within the training and validation
sets.

Online SE: The Online SE dataset [23], comprises videos of six students
participating in online courses via the Zoom platform. These recordings span
10 seconds each, with a frame rate of 24 fps and a resolution of $220\times
155$ pixels. Two independent annotators annotated the video recordings as
engaged and not engaged using the Human Expert Labeling Process (HELP) [57,
37]. The binary engagement labels achieved an inter-rater reliability score of
0.62, denoting a significant level of agreement. Following the exclusion of
flawed videos [37], the dataset was segmented into 3190, 1660, and 1290
samples for training, validation, and testing, respectively. The distribution
of engaged and not-engaged samples across the training, validation, and test
sets is outlined in Table 2.

###  4.3 Experimental Setting

The sole information extracted from video frames is facial landmarks, which
are analyzed by ST-GCN to determine the engagement level of the student in the
video. Drawing inspiration from the pioneering works on body-joints-based
action analysis [58, 29], the proposed ST-GCN for facial-landmarks-based
engagement measurement is structured as follows. The input facial landmarks
are first processed through a batch normalization layer, followed by three
consecutive ST-GCN layers with 64, 128, and 256 output channels, respectively.
Residual connections are incorporated in the last two ST-GCN layers. A dropout
rate of 0.1 is applied to each ST-GCN layer. The temporal kernel size in the
ST-GCN layers is selected to be $9$. Subsequent to the ST-GCN layers, an
average pooling layer is utilized, and its resulting tensor is directed into a
2D convolutional layer with 256 input channels and a number of output channels
corresponding to the number of classes. A Softmax activation function then
computes the probability estimates. In cases where engagement measurement is
framed as a binary classification task, the terminal 2D convolution layer is
configured with a single output channel, substituting Softmax with a Sigmoid
function. The Sigmoid function is also employed for the individual binary
classifiers within the ordinal learning framework detailed in subsection 3.3.
The models are trained using the Adam optimizer with mini-batches of size $16$
and a learning rate of $0.001$ for $300$ epochs. The learning rate is decayed
by a factor of $0.1$ every $100$ epochs.

Table 3: Classification accuracy of engagement levels on the validation set of the EngageNet dataset [34]: comparison of various feature sets and classification models from [34] against two configurations of the proposed method - facial landmarks analyzed by ST-GCN and ordinal ST-GCN. Bolded values denote the best results.  Features |  Model |  Accuracy
---|---|---
Gaze |  LSTM |  0.6125
Head Pose |  LSTM |  0.6760
AU |  LSTM |  0.6303
Gaze + Head Pose |  LSTM |  0.6769
Gaze + Head Pose + AU |  LSTM |  0.6704
Gaze |  CNN-LSTM |  0.6060
Head Pose |  CNN-LSTM |  0.6732
AU |  CNN-LSTM |  0.6172
Gaze + AU |  CNN-LSTM |  0.6275
Head Pose + AU |  CNN-LSTM |  0.6751
Gaze + Head Pose + AU |  CNN-LSTM |  0.6751
Gaze |  TCN |  0.6256
Head Pose |  TCN |  0.6611
AU |  TCN |  0.6293
Gaze + Head Pose + AU |  TCN |  0.6779
Gaze |  Transformer |  0.5545
Gaze + Head Pose |  Transformer |  0.6445
Gaze + Head Pose + AU |  Transformer |  0.6910
Gaze + Head Pose + AU + MARLIN |  Transformer |  0.6849
Facial Landmarks |  ST-GCN∗ |  0.6937
Facial Landmarks |  Ordinal ST-GCN∗ |  0.7124
∗ the proposed method |  |

###  4.4 Experimental Results

Table 3 presents the comparative results of two settings of the proposed
method, a regular non-ordinal classifier and an ordinal classifier, with
previous methods on the validation set of the EngageNet dataset [34]. The
engagement measurement in EngageNet [34] is a four-class classification
problem and the accuracy is reported as the evaluation metric. The previous
methods used different combinations of OpenFace’s eye gaze, head pose, and AU
features [36] along with MARLIN’s facial embedding features [33]. These
features were classified by LSTM, CNN-LSTM, TCN, and Transformer, refer to
[34] for more details of the previous methods.

In the single feature configurations of previous methods in Table 3, head pose
achieves better results compared to AUs, which is better than eye gaze. While
head pose and eye gaze are indicators of behavioral engagement [12, 8], AUs,
which are associated with facial expressions and affect [35], are indicators
of affective engagement. Combining these three features is always beneficial
since engagement is a multi-component variable that can be measured by
affective and behavioral indicators when the only available data is video
[12]. Among classification models, utilizing more advanced models improves
accuracy; the Transformer is better than TCN, which is better than CNN-LSTM
and LSTM; however, it comes at the cost of increased computational complexity
and the number of model parameters. It should also be noted that the feature
extraction step for the three feature sets requires three complex deep
learning models in OpenFace [36]. Considering only the classification models,
the number of parameters in the Transformer, according to the architecture
described in [34], is 104,963,076, while the numbers of parameters in the
proposed non-ordinal and ordinal ST-GCN are 861,688, and 861,431,
respectively, indicating the efficiency of the proposed method. While being
lightweight, the proposed method with ordinal STGCN improves the state-of-the-
art by 3.1% compared to eye gaze, head pose, and AUs with Transformer.

Tables 4(a) and 4(b) depict the confusion matrices of the two configurations
of the proposed method in Table 3, facial landmarks classified by non-ordinal
and ordinal ST-GCN, respectively. As shown in Tables 4(a) and 4(b),
incorporating ordinality significantly increases the number of correctly
classified samples in the first three classes and results in a 2.7%
improvement in accuracy compared to its non-ordinal counterpart.

Table 4: Confusion matrices of the proposed method using facial landmarks with
(a) non-ordinal and (b) ordinal ST-GCN on the validation set of the EngageNet
dataset [34].

(a) Class | 1 | 2 | 3 | 4
---|---|---|---|---
1 | 99 | 14 | 13 | 6
2 | 10 | 29 | 33 | 25
3 | 9 | 19 | 100 | 145
4 | 5 | 4 | 45 | 515

(b) Class | 1 | 2 | 3 | 4
---|---|---|---|---
1 | 104 | 12 | 10 | 6
2 | 15 | 36 | 24 | 22
3 | 10 | 26 | 112 | 125
4 | 9 | 5 | 44 | 511

Table 5 displays the results of different variants of the proposed method on
the validation set of the EngageNet [34] dataset. In the first two variants,
the $x$, $y$, and $z$ coordinates of facial landmarks are converted into
multivariate time series and analyzed by an LSTM and TCN. The LSTM includes
four unidirectional layers with 256 neurons in hidden units and is followed by
a $256\times 4$ fully connected layer. The parameters of the TCN are as
follows: the number of layers, number of filters, kernel size, and dropout
rate are 8, 64, 8, and 0.05, respectively. While their results are acceptable
and better than most of the earlier methods in Table 3, they cannot outperform
ST-GCN, the last two columns of Table 3. The fact that the accuracy of the
LSTM and TCN in Table 5 is higher than those in Table 3 signifies the
efficiency of facial landmarks for engagement measurement. In the third
variant, the $z$ coordinates of facial landmarks are disregarded, and the
decrease in accuracy indicates the importance of the $z$ coordinates for
engagement measurement. Temporal kernel sizes other than 9 in the fourth and
fifth variants have a negative impact on the results. When engagement
measurement is performed using every 2, 4, 8, and 16 frames, instead of every
frame, there is a slight decrease in accuracy. However, this is a trade-off
between accuracy and computation since reducing the frame rate corresponds to
a reduction in computation.

Table 5: Classification accuracy of engagement levels on the validation set of the EngageNet dataset [34] for different variants of the proposed method.  Variant |  Accuracy
---|---
ST-GCN is replaced with LSTM |  0.6847
ST-GCN is replaced with TCN |  0.6813
Only $x$ and $y$ coordinates of joints are used |  0.6655
A temporal kernel of 3 is used |  0.6748
A temporal kernel of 15 is used |  0.6841
Every 2 frames are used |  0.6813
Every 4 frames are used |  0.6907
Every 8 frames are used |  0.6907
Every 16 frames are used |  0.6841

Table 6 presents the results of the proposed method in comparison to previous
methods on the test set of the Online SE dataset [23]. The earlier methods
listed in Table 6 are feature-based, extracting affective and behavioral
features from video frames and performing binary classification of the
features using TCN in [22], TCN in [12], LSTM with attention in [17], and BoW
in [37]. Given that engagement measurement in the Online SE dataset [23] is
posed as a binary classification problem, implementing the ordinal version of
the proposed method is not required. The proposed method, employing facial
landmarks with ST-GCN, attains the highest accuracy and AUC PR.

Table 6: Binary classification accuracy, AUC ROC, and AUC PR of engagement on the test set of the Online SE dataset [23]: comparison of previous methods against the proposed method. Bolded values denote the best results.  Method |  Accuracy |  AUC ROC |  AUC PR
---|---|---|---
[22] |  0.7803 |  0.8764 |  0.8008
[12] |  0.7637 |  0.8710 |  0.7980
[17] |  0.7475 |  0.8890 |  0.7973
[37] |  0.8191 |  0.8926 |  0.9018
Proposed |  0.8315 |  0.8806 |  0.9131

##  5 Conclusion and Future Works

Our research led to the development of a novel deep-learning framework for
student engagement measurement. In the proposed framework, sequences of facial
landmarks are extracted from consecutive video frames and analyzed by ordinal
ST-GCNs to make inferences regarding the engagement level of the subject in
the video. The successful application of our model to the EngageNet [34] and
Online SE [23] datasets not only confirms its efficacy but also establishes a
new standard in engagement level classification accuracy, outperforming
existing methods. As the sole input information to the developed engagement
measurement models, the 3D facial landmarks contain information about head
pose [51], AUs [52], eye gaze [49], and affect [3, 35], which are the key
indicators of behavioral and affective engagement. The relatively lightweight
ST-GCN and its integration with real-time MediaPipe make the proposed
framework capable of being deployed on virtual learning platforms and
measuring engagement in real time. Despite these advancements, the proposed
method is not without limitations. The interpretability and explainability of
the proposed method could be improved through techniques such as Gradient-
based Class Activation Maps [58] for ST-GCNs [59]. This would facilitate an
understanding of which facial landmarks and specific timestamps contribute to
certain levels of engagement. To improve the performance of the proposed
method, the following direction could be investigated: analyzing facial
landmarks with more advanced ST-GCN, which are equipped with attention
mechanism [60], applying augmentation techniques to video data before facial
landmark extraction [61] or to facial landmark data to improve the
generalizability of the ST-GCNs, and incorporating hand landmarks and body
joints extracted from video into the developed facial-landmarks-based
framework.

Acknowledgment  The authors express their sincere thanks to the Multimodal
Perception Lab at the International Institute of Information Technology,
Bangalore, India, for their generosity in providing the Online SE dataset,
which was instrumental in the execution of our experiments.

## References

  * [1] Brandon M. Booth, Nigel Bosch and Sidney K. D’Mello  “Engagement Detection and Its Applications in Learning: A Tutorial and Selective Review”  In _Proceedings of the IEEE_ 111.10, 2023, pp. 1398–1422  DOI: 10.1109/JPROC.2023.3309560
  * [2] Suzanne Hidi and K Ann Renninger  “The four-phase model of interest development”  In _Educational psychologist_ 41.2 Taylor & Francis, 2006, pp. 111–127
  * [3] Yang Liu et al.  “Graph-based facial affect analysis: A review”  In _IEEE Transactions on Affective Computing_ IEEE, 2022
  * [4] Erroll Wood et al.  “Rendering of eyes for eye-shape registration and gaze estimation”  In _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 3756–3764
  * [5] Shruti Gedam and Sanchita Paul  “A review on mental stress detection using wearable sensors and machine learning techniques”  In _IEEE Access_ 9 IEEE, 2021, pp. 84045–84066
  * [6] Jennifer A Fredricks, Phyllis C Blumenfeld and Alison H Paris  “School engagement: Potential of the concept, state of the evidence”  In _Review of educational research_ 74.1 Sage Publications Sage CA: Thousand Oaks, CA, 2004, pp. 59–109
  * [7] Sidney D’Mello and Art Graesser  “Dynamics of affective states during complex learning”  In _Learning and Instruction_ 22.2 Elsevier, 2012, pp. 145–157
  * [8] Jaclyn Ocumpaugh  “Baker Rodrigo Ocumpaugh monitoring protocol (BROMP) 2.0 technical and training manual”  In _New York, NY and Manila, Philippines: Teachers College, Columbia University and Ateneo Laboratory for the Learning Sciences_ 60, 2015
  * [9] Shofiyati Nur Karimah and Shinobu Hasegawa  “Automatic engagement estimation in smart education/learning settings: a systematic review of engagement definitions, datasets, and methods”  In _Smart Learning Environments_ 9.1 SpringerOpen, 2022, pp. 1–48
  * [10] M Dewan, Mahbub Murshed and Fuhua Lin  “Engagement detection in online learning: a review”  In _Smart Learning Environments_ 6.1 Springer, 2019, pp. 1–20
  * [11] Shehroz S Khan, Ali Abedi and Tracey Colella  “Inconsistencies in Measuring Student Engagement in Virtual Learning–A Critical Review”  In _arXiv preprint arXiv:2208.04548_ , 2022
  * [12] Ali Abedi and Shehroz S Khan  “Affect-driven ordinal engagement measurement from video”  In _Multimedia Tools and Applications_ Springer, 2023, pp. 1–20
  * [13] Camillo Lugaresi et al.  “Mediapipe: A framework for building perception pipelines”  In _arXiv preprint arXiv:1906.08172_ , 2019
  * [14] Jinsheng Wei et al.  “Geometric graph representation with learnable graph structure and adaptive au constraint for micro-expression recognition”  In _IEEE Transactions on Affective Computing_ IEEE, 2023
  * [15] Kaili Zheng, Ji Wu, Jialin Zhang and Chenyi Guo  “A Skeleton-Based Rehabilitation Exercise Assessment System With Rotation Invariance”  In _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ 31, 2023, pp. 2612–2621  DOI: 10.1109/TNSRE.2023.3282675
  * [16] Pratik K Mishra, Alex Mihailidis and Shehroz S Khan  “Skeletal Video Anomaly Detection using Deep Learning: Survey, Challenges and Future Directions”  In _arXiv preprint arXiv:2301.00114_ , 2022
  * [17] Xu Chen, Li Niu, Ashok Veeraraghavan and Ashutosh Sabharwal  “FaceEngage: robust estimation of gameplay engagement from user-contributed (YouTube) videos”  In _IEEE Transactions on Affective Computing_ IEEE, 2019
  * [18] Jianming Wu, Bo Yang, Yanan Wang and Gen Hattori  “Advanced multi-instance learning method with multi-features engineering and conservative optimization for engagement intensity prediction”  In _Proceedings of the 2020 International Conference on Multimodal Interaction_ , 2020, pp. 777–783
  * [19] Xiaoyang Ma, Min Xu, Yao Dong and Zhong Sun  “Automatic student engagement in online learning environment based on neural turing machine”  In _International Journal of Information and Education Technology_ 11.3, 2021, pp. 107–111
  * [20] Onur Copur, Mert Nakıp, Simone Scardapane and Jürgen Slowack  “Engagement Detection with Multi-Task Training in E-Learning Environments”  In _International Conference on Image Analysis and Processing_ , 2022, pp. 411–422  Springer
  * [21] Ali Abedi and Shehroz S.  “Detecting disengagement in virtual learning as an anomaly using temporal convolutional network autoencoder”  In _Signal, Image and Video Processing_ Springer, 2023, pp. 1–9
  * [22] Chinchu Thomas, Nitin Nair and Dinesh Babu Jayagopi  “Predicting engagement intensity in the wild using temporal convolutional network”  In _Proceedings of the 20th ACM International Conference on Multimodal Interaction_ , 2018, pp. 604–610
  * [23] Chinchu Thomas, KAV Puneeth Sarma, Srujan Swaroop Gajula and Dinesh Babu Jayagopi  “Automatic prediction of presentation style and student engagement from videos”  In _Computers and Education: Artificial Intelligence_ Elsevier, 2022, pp. 100079
  * [24] Abhay Gupta, Arjun D’Cunha, Kamal Awasthi and Vineeth Balasubramanian  “Daisee: Towards user engagement recognition in the wild”  In _arXiv preprint arXiv:1609.01885_ , 2016
  * [25] Hao Zhang et al.  “An novel end-to-end network for automatic student engagement recognition”  In _2019 IEEE 9th International Conference on Electronics Information and Emergency Communication (ICEIEC)_ , 2019, pp. 342–345  IEEE
  * [26] Ali Abedi and Shehroz S Khan  “Improving state-of-the-art in Detecting Student Engagement with Resnet and TCN Hybrid Network”  In _2021 18th Conference on Robots and Vision (CRV)_ , 2021, pp. 151–157  IEEE
  * [27] Naval Kishore Mehta et al.  “Three-dimensional DenseNet self-attention neural network for automatic detection of student’s engagement”  In _Applied Intelligence_ Springer, 2022, pp. 1–21
  * [28] Hua Leong Fwa  “Fine-grained detection of academic emotions with spatial temporal graph attention networks using facial landmarks”, 2022
  * [29] Sijie Yan, Yuanjun Xiong and Dahua Lin  “Spatial temporal graph convolutional networks for skeleton-based action recognition”  In _Proceedings of the AAAI conference on artificial intelligence_ 32.1, 2018
  * [30] Xusheng Ai, Victor S Sheng and Chunhua Li  “Class-attention Video Transformer for Engagement Intensity Prediction”  In _arXiv preprint arXiv:2208.07216_ , 2022
  * [31] Jiacheng Liao, Yan Liang and Jiahui Pan  “Deep facial spatiotemporal network for engagement prediction in online learning”  In _Applied Intelligence_ 51.10 Springer, 2021, pp. 6609–6621
  * [32] Tasneem Selim, Islam Elkabani and Mohamed A Abdou  “Students Engagement Level Detection in Online e-Learning Using Hybrid EfficientNetB7 Together With TCN, LSTM, and Bi-LSTM”  In _IEEE Access_ 10 IEEE, 2022, pp. 99573–99583
  * [33] Zhixi Cai et al.  “Marlin: Masked autoencoder for facial video representation learning”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 1493–1504
  * [34] Monisha Singh et al.  “Do I Have Your Attention: A Large Scale Engagement Prediction Dataset and Baselines”  In _Proceedings of the 25th International Conference on Multimodal Interaction_ , ICMI ’23  <conf-loc>, <city>Paris</city>, <country>France</country>, </conf-loc>: Association for Computing Machinery, 2023, pp. 174–182  DOI: 10.1145/3577190.3614164
  * [35] Antoine Toisoul et al.  “Estimation of continuous valence and arousal levels from faces in naturalistic conditions”  In _Nature Machine Intelligence_ 3.1 Nature Publishing Group, 2021, pp. 42–50
  * [36] Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim and Louis-Philippe Morency  “Openface 2.0: Facial behavior analysis toolkit”  In _2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018)_, 2018, pp. 59–66  IEEE
  * [37] Ali Abedi, Chinchu Thomas, Dinesh Babu Jayagopi and Shehroz S Khan  “Bag of States: A Non-sequential Approach to Video-based Engagement Measurement”  In _arXiv preprint arXiv:2301.06730_ , 2023
  * [38] Xinran Tian, Bernardo Pereira Nunes, Yifeng Liu and Ruben Manrique  “Predicting student engagement using sequential ensemble model”  In _IEEE Transactions on Learning Technologies_ IEEE, 2023
  * [39] Jacob Whitehill et al.  “The faces of engagement: Automatic recognition of student engagementfrom facial expressions”  In _IEEE Transactions on Affective Computing_ 5.1 IEEE, 2014, pp. 86–98
  * [40] Xuesong Niu et al.  “Automatic engagement prediction with GAP feature”  In _Proceedings of the 20th ACM International Conference on Multimodal Interaction_ , 2018, pp. 599–603
  * [41] Tao Huang et al.  “Fine-grained engagement recognition in online learning environment”  In _2019 IEEE 9th international conference on electronics information and emergency communication (ICEIEC)_ , 2019, pp. 338–341  IEEE
  * [42] Dmitrii Fedotov et al.  “Multimodal approach to engagement and disengagement detection with highly imbalanced in-the-wild data”  In _Proceedings of the Workshop on Modeling Cognitive Processes from Multimodal Data_ , 2018, pp. 1–9
  * [43] Brandon M Booth et al.  “Toward active and unobtrusive engagement assessment of distance learners”  In _2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII)_ , 2017, pp. 470–476  IEEE
  * [44] Amanjot Kaur, Aamir Mustafa, Love Mehta and Abhinav Dhall  “Prediction and localization of student engagement in the wild”  In _2018 Digital Image Computing: Techniques and Applications (DICTA)_ , 2018, pp. 1–8  IEEE
  * [45] Amir Zadeh, Yao Chong Lim, Tadas Baltrusaitis and Louis-Philippe Morency  “Convolutional experts constrained local model for 3d facial landmark detection”  In _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2017, pp. 2519–2528
  * [46] Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li and Yu Qiao  “Joint face detection and alignment using multitask cascaded convolutional networks”  In _IEEE signal processing letters_ 23.10 IEEE, 2016, pp. 1499–1503
  * [47] Jinzhao Zhou, Xingming Zhang, Yang Liu and Xiangyuan Lan  “Facial Expression Recognition Using Spatial-Temporal Semantic Graph Network”  In _2020 IEEE International Conference on Image Processing (ICIP)_ , 2020, pp. 1961–1965  DOI: 10.1109/ICIP40778.2020.9191181
  * [48] Davis King  “Dlib C++ Library” [Online; accessed [Access Year]], http://dlib.net/, 2024
  * [49] Ivan Grishchenko et al.  “Attention mesh: High-fidelity face mesh prediction in real-time”  In _arXiv preprint arXiv:2006.10962_ , 2020
  * [50] Ameur Latreche, Ridha Kelaiaia, Ahmed Chemori and Adlen Kerboua  “Reliability and validity analysis of MediaPipe-based measurement system for some human rehabilitation motions”  In _Measurement_ 214 Elsevier, 2023, pp. 112826
  * [51] Salim Malek and Silvia Rossi  “Head pose estimation using facial-landmarks classification for children rehabilitation games”  In _Pattern Recognition Letters_ 152 Elsevier, 2021, pp. 406–412
  * [52] Geethu Miriam Jacob and Bjorn Stenger  “Facial action unit detection with transformers”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 7680–7689
  * [53] Yang Liu, Xingming Zhang, Yubei Lin and Haoxiang Wang  “Facial expression recognition via deep action units graph network based on psychological mechanism”  In _IEEE Transactions on Cognitive and Developmental Systems_ 12.2 IEEE, 2019, pp. 311–322
  * [54] Thomas N Kipf and Max Welling  “Semi-supervised classification with graph convolutional networks”  In _arXiv preprint arXiv:1609.02907_ , 2016
  * [55] Georgios N Yannakakis, Roddy Cowie and Carlos Busso  “The ordinal nature of emotions: An emerging approach”  In _IEEE Transactions on Affective Computing_ 12.1 IEEE, 2018, pp. 16–35
  * [56] Eibe Frank and Mark Hall  “A simple approach to ordinal classification”  In _Machine Learning: ECML 2001: 12th European Conference on Machine Learning Freiburg, Germany, September 5–7, 2001 Proceedings 12_ , 2001, pp. 145–156  Springer
  * [57] Sinem Aslan et al.  “Human expert labeling process (HELP): towards a reliable higher-order user state labeling process and tool to assess student engagement”  In _Educational Technology_ JSTOR, 2017, pp. 53–59
  * [58] Kaili Zheng, Ji Wu, Jialin Zhang and Chenyi Guo  “A Skeleton-based Rehabilitation Exercise Assessment System with Rotation Invariance”  In _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ IEEE, 2023
  * [59] Pratyusha Das and Antonio Ortega  “Gradient-Weighted Class Activation Mapping for Spatio Temporal Graph Convolutional Network”  In _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 4043–4047  DOI: 10.1109/ICASSP43922.2022.9746621
  * [60] Lilang Lin, Jiahang Zhang and Jiaying Liu  “Actionlet-Dependent Contrastive Learning for Unsupervised Skeleton-Based Action Recognition”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 2363–2372
  * [61] Ali Abedi, Mobin Malmirian and Shehroz S Khan  “Cross-Modal Video to Body-joints Augmentation for Rehabilitation Exercise Quality Assessment”  In _arXiv preprint arXiv:2306.09546_ , 2023