11institutetext: University of Pennsylvania, USA
11email<EMAIL_ADDRESS>22institutetext: Lund
University, Sweden 33institutetext: University of Castilla-La Mancha, Spain

# Improved Segmentation of Deep Sulci in Cortical Gray Matter Using a Deep
Learning Framework Incorporating Laplace’s Equation

Sadhana Ravikumar(✉) 11 Ranjit Ittyerah 11 Sydney Lim 11 Long Xie 11 Sandhitsu
Das 11 Pulkit Khandelwal 11 Laura E.M. Wisse 22 Madigan L. Bedard 11 John L.
Robinson 11 Terry Schuck 11 Murray Grossman 11 John Q. Trojanowski 11 Edward
B. Lee 11 M. Dylan Tisdall 11 Karthik Prabhakaran 11 John A. Detre 11 David J.
Irwin 11 Winifred Trotman 11 Gabor Mizsei 11 Emilio Artacho-Pérula 33 Maria
Mercedes Iñiguez de Onzono Martin 33 Maria del Mar Arroyo Jiménez 33 Monica
Muñoz 33 Francisco Javier Molina Romero 33 Maria del Pilar Marcos Rabal 33
Sandra Cebada-Sánchez 33 José Carlos Delgado González 33 Carlos de la Rosa-
Prieto 33 Marta Córcoles Parada 33 David A. Wolk 11 Ricardo Insausti 33 Paul
A. Yushkevich(✉) 11

###### Abstract

When developing tools for automated cortical segmentation, the ability to
produce topologically correct segmentations is important in order to compute
geometrically valid morphometry measures. In practice, accurate cortical
segmentation is challenged by image artifacts and the highly convoluted
anatomy of the cortex itself. To address this, we propose a novel deep
learning-based cortical segmentation method in which prior knowledge about the
geometry of the cortex is incorporated into the network during the training
process. We design a loss function which uses the theory of Laplace’s equation
applied to the cortex to locally penalize unresolved boundaries between
tightly folded sulci. Using an ex vivo MRI dataset of human medial temporal
lobe specimens, we demonstrate that our approach outperforms baseline
segmentation networks, both quantitatively and qualitatively.

###### Keywords:

Cortical segmentation topology correction ex vivo MRI

##  1 Introduction

Segmentation of the cerebral cortex from MRI is an important first step in
many neuroimaging pipelines such as quantitative morphometry analyses aimed at
understanding the pathophysiology of neurological disorders. Automated
segmentation methods applied to the cortex are challenged by various artifacts
such as image noise, partial volume effects and intensity inhomogeneities
which make accurate identification of the tissue boundaries difficult and
result in geometrically inaccurate cortical reconstructions. The cerebral
cortex or gray matter (GM) can be defined as the space between two cortical
surfaces; the pial surface which separates the GM from the surrounding
cerebrospinal fluid (CSF), and the white matter (WM) surface which separates
the GM from WM. The cortex has a complex geometry and is often modelled as a
highly folded 2D sheet, with spatially varying curvature and thickness [14].
Geometrically accurate segmentation of the cortex requires accurate
reconstruction of both the WM and pial cortical surfaces, complete with all
cortical folds and narrow sulci. A commonly used simplification when solving
cortical surface reconstruction problems is to view the cortical surfaces as
having the topology of a 3D sphere [4]. However, unless explicitly corrected
for, imaging artifacts often introduce topological defects in the resulting
surface reconstructions. Defects due to partial volume effects are
particularly apparent in tightly folded sulci and result in opposing banks of
sulci appearing fused together. This creates either ‘bridged’ or ‘unresolved’
sulci in the resulting cortical reconstruction, which cause errors in
downstream quantitative brain morphometry measures such as cortical thickness.
While topological defects can be corrected by manual editing, these checks can
be time-consuming.

Topology-corrected reconstruction of cortical surfaces is a well studied topic
in in vivo neuroimaging literature, and several state-of-the-art methods have
been developed to address this problem [4, 11, 5]. The widely used Freesurfer
framework employs a mesh-based approach to topology-correction which consists
of two main steps [4]. First, the inner WM surface is generated by applying
mesh tessellation to a volumetric WM segmentation that has been corrected for
topological defects [4]. Second, this WM surface is expanded using a
deformable surface model to reconstruct the outer, pial surface, while
ensuring that the topology of the initial surface is preserved [4, 5, 11]. In
recent years, elements of the FreeSurfer pipeline have been implemented as
deep learning networks, resulting in significant speedups [12, 7, 1, 8].
However, these frameworks still either require the time-consuming post-
processing step of topology correction [1, 7] or rely on a predefined initial
mesh with the correct, spherical topology to reconstruct the cortical surface
[8, 12].

In this work, we are specifically interested in developing an automated
cortical segmentation method that can be applied to ex vivo brain MRI datasets
to generate geometrically valid models of the cortex. Instead of a mesh
deformation-based approach, we propose a novel volumetric deep image
segmentation method that learns to segment the cortex while explicitly
modeling the ‘sheet-like’ geometry of the cortex. In ex vivo studies, it is
common to image only a portion of the brain hemisphere, thus violating the
assumption of spherical WM topology made by mesh-based approaches.
Furthermore, many of the in vivo cortical segmentation methods contain
algorithms that are optimized for data with a standard 1 mm voxel size [4],
and would result in unrealistic computational times if applied to high-
resolution ex vivo MRI datasets. As a result, existing methods are not easily
applicable to ex vivo MRI scans. As far as we know, no prior work has focused
on topology correction of ex vivo cortical segmentations.

Previous studies have used the concept of Laplace’s equation as a tool for
modelling the cortex [10, 14, 11]. By setting different boundary conditions at
the WM/GM and GM/CSF interfaces, Laplace’s equation can be solved within the
GM volume to generate a laminar ‘potential’ field that smoothly varies in
value depending on its distance between the two cortical surfaces. The
gradient of the Laplacian field can be used to compute cortical thickness [10]
and defines an expansion path which guarantees topology-preserving deformation
between the WM and pial cortical surfaces [11, 14]. Building on this idea and
the success of deep convolutional neural networks (CNN) in medical image
segmentation tasks, here we design a differentiable numerical solver for
Laplace’s equation and incorporate it within a deep segmentation framework to
locally impose a Laplacian mapping between the predicted WM and pial surfaces.
We train the segmentation network by comparing the predicted tissue
segmentations and corresponding Laplacian field maps with the equivalent
ground truth images, thus penalizing self-intersections in the predicted
segmentations. Our results show that when compared to a baseline network
trained without Laplacian constraints, our method is able to better
reconstruct the intrinsic, layered geometry of the cortex. To our knowledge,
this is the first time that an iterative numerical solver has been
incorporated within a cortical segmentation network to directly compute
Laplacian fields in an end-to-end setting.

##  2 Methods

As illustrated in Fig. 1, our proposed framework builds upon any given
backbone segmentation network (Sect. 2.1) by appending a numerical solver for
Laplace’s equation to the output of the network. We reformulate the solver to
be differentiable with respect to the input image to allow for gradient-based
learning, used within standard CNN training (Sect. 2.2). In addition to the
standard tissue segmentation loss, we design a loss function which compares
the predicted Laplacian field to the solution of Laplace’s equation applied to
the ground truth cortical segmentation, which is assumed to have correct
topology (Sect. 2.3).

Figure 1: Schematic illustration of the proposed framework. A differentiable
numerical solver for Laplace’s equation, based on the successive over
relaxation (SOR) algorithm, is incorporated within end-to-end training of a
segmentation network.

###  2.1 Backbone Segmentation Network

The proposed Laplacian solver is compatible with any semantic segmentation
network since it only relies on the segmentation map output by the backbone
network. We conducted experiments using two backbone networks for 3D image
segmentation: the state-of-art nnU-Net framework [9] based on the U-net
architecture, and nnFormer [18], a variant framework based on the recently
popular transformer architecture. Both frameworks use image patches, deep
supervision, and a variety of data-augmentation techniques to train the
network [9].

###  2.2 Differentiable Laplacian Solver

To compute the Laplacian field corresponding to the cortical segmentation
predicted based on a given input patch, the iterative solver for Laplace’s
equation is appended after the final layer of the backbone network. Laplace’s
equation, $\Delta\varphi=0$, is a second-order partial differential equation,
where $\Delta$ is the Laplacian operator
($\partial^{2}_{xx}+\partial^{2}_{yy}+\partial^{2}_{zz}$) and $\varphi$ is a
twice-differentiable, real-valued function. To solve Laplace’s equation within
a domain (in our case, the GM volume), specific conditions need to be set that
the Laplacian field $\varphi$ must satisfy at the boundaries of the domain. In
our case, we set $\varphi_{x,y,z}=0$ at the GM/WM boundary and
$\varphi_{x,y,z}=1$ at the GM/pial boundary. Voxels within the GM domain are
initialized with $\varphi_{x,y,z}=0.5$.

Given these boundary conditions, the Laplacian field can be approximated using
the finite-difference method, which is solved using an iterative numerical
solver. Here, we use the Successive Over Relaxation (SOR) algorithm, a variant
of the Gauss-Seidel method, to solve for the Laplacian field [6]. In the
Gauss-Seidel method, given initial values for all the voxels in an image, at
each iteration, the new value for a particular voxel within the GM volume,
$\tilde{\varphi}_{xyz}$, is computed by taking the weighted sum of the most
recently updated values of its six neighboring voxels (Eq. 1). The superscript
$n$ refers to the iteration of the algorithm and the subscripts are the voxel
indices.

$\tilde{\varphi}_{xyz}^{n}=\frac{1}{6}(\varphi^{n}_{x-1,y,z}+\varphi^{n-1}_{x+1,y,z}+\\\
\varphi^{n}_{x,y-1,z}+\varphi^{n-1}_{x,y+1,z}+\varphi^{n}_{x,y,z-1}+\varphi^{n-1}_{x,y,z+1})$
(1)

The SOR algorithm accelerates this approach by taking, at each iteration $n$,
the weighted sum of the current solution and the solution from the previous
iteration (Eq. 2). The over-relaxation parameter, $\omega$ accelerates the
rate of convergence of the Gauss-Seidel method when $1<\omega<2$ [17]. It has
been shown that the optimum value, $\omega_{opt}$ is given by
$\omega_{opt}=\frac{2}{1+sin(\frac{\pi}{N+1})}$ where $N$ is the minimum
dimension of the input grid [17].

$\begin{split}\varphi^{n}_{x,y,z}=(1-\omega_{opt})\varphi^{n-1}_{x,y,z}+\frac{\omega_{opt}}{6}(\varphi^{n}_{x-1,y,z}+\varphi^{n-1}_{x+1,y,z}+\\\
\varphi^{n}_{x,y-1,z}+\varphi^{n-1}_{x,y+1,z}+\varphi^{n}_{x,y,z-1}+\varphi^{n-1}_{x,y,z+1})\end{split}$
(2)

Instead of updating the value of each voxel in the image serially, computation
of Eq. 2 can be parallelized using the Red-Black SOR approach, wherein the
voxels in an image are divided into ‘red’ and ‘black’ following a checkerboard
pattern [3]. During the update step, the ‘red’ voxels only depend on the
values of the ‘black’ voxels and vice versa. Therefore, at each iteration, the
Laplacian solution is updated in two steps; first, the update equation is
applied to all of the black voxels in parallel and second, the update equation
is applied to all of the red voxels in parallel, using the updated values
computed at the black voxels. After convergence, the resulting Laplacian field
contains values within the GM volume increasing smoothly from 0 at the WM
surface to 1 at the pial surface.

To incorporate this numerical solver within a CNN, an important consideration
is that the computations used to generate the Laplacian field must be
differentiable with respect to the predicted tissue class probabilities to
allow for back-propagation of the final loss through the network. To this end,
we initialized the boundary conditions for the Laplacian solver by taking a
weighted sum of the GM, WM and background probability maps, with weights of
0.5, 0 and 1 respectively (Fig. 1A). Additionally, the SOR update step (Eq. 2)
was reformulated as a 1$\times$1 convolutional layer with fixed neighborhood
weights specified in a 3$\times$3$\times$3 kernel (Fig. 1B). The voxels in the
image were divided into a red and black grid by generating 3D ‘red’
($mod(x+y+z,2)=0$), and ‘black’ ($mod(x+y+z,2)=1$) binary checkerboard images
which were applied as image masks to retain values of interest after each
convolutional layer (Fig. 1C). Lastly, the maximum number of iterations for
the Laplacian solver was empirically set to 60, as a trade-off between
computational time and convergence of the Laplacian solution. Since the solver
numerically computes the solution to Laplace’s equation, it does not introduce
any additional training parameters within the network.

###  2.3 Loss Function

To train the model, the backbone networks compare the predicted tissue
segmentation, $S_{pred}$ with the ground truth cortical segmentation, $S_{gt}$
using a combination of Dice and cross-entropy loss, $DCE(S_{gt},S_{pred})$
[9]. We introduce an additional loss term which compares the Laplacian field
computed from the predicted tissue segmentation, $\varphi_{pred}$, with the
solution of Laplace’s equation applied to the ground truth segmentation,
$\varphi_{gt}$. To simplify comparison between the predicted Laplacian field
and the ground truth solution, we convert the Laplacian field computed by the
solver to a multi-label segmentation, $S^{\varphi}_{pred}$ using a series of
thresholding functions. The advantage of this approach is that it enables the
use of the same combined Dice and cross-entropy loss used by the backbone
network on the outputs of the Laplacian solver, instead of the mean square
error loss, thereby allowing us to equally weight the two loss terms:
$\mathcal{L}=DCE(S_{gt},S_{pred})+DCE(S^{\varphi}_{gt},S^{\varphi}_{pred})$.
To threshold the Laplacian field and create a multi-label segmentation in a
differentiable way, the computed Laplacian field is passed through a product
of two sigmoid functions,
$(1+e^{-\beta(x-t_{lower})})^{-1}\times(1+e^{\beta(x-t_{upper})})^{-1}$, which
together create a ‘band-pass’ thresholding filter. $\beta$ controls the
steepness of the filter, and $t_{lower}$ and $t_{upper}$ control the domain of
the filter. Each filter creates an image that has values close to 1 for voxels
in the Laplacian field lying within the domain of the filter, and values close
to 0 otherwise. Therefore, by varying the lower and upper threshold values, a
one-hot encoded image can be created, where each channel corresponds to a
different label along the laminar axis of the GM. A multi-label Laplacian
segmentation is then generated by applying the argmax operation to the
computed one-hot encoded image (Fig. 1D).

##  3 Experiments

###  3.1 Dataset

##### MRI Image Acquisition

To train and evaluate the proposed framework, we used ex vivo images of intact
temporal lobe specimens, obtained from 27 brain donors from either the brain
bank operated by the National Disease Research Interchange, or autopsies
performed at the University of Pennsylvania Center for Neurodegenerative
Disease Research (CNDR) and the University of Castilla-La Mancha (UCLM) Human
Neuroanatomy Laboratory (HNL) in Spain. Brain specimens were obtained in
accordance with the University of Pennsylvania Institutional Review Board
guidelines, and the Ethical Committee of UCLM. Where possible, pre-consent
during life and, in all cases, next-of-kin consent at death was given.
Following 4+ weeks of fixation, the tissue specimens were scanned overnight on
a 9.4 Tesla 31 cm bore MRI scanner using a T2-weighted, multi-slice spin echo
sequence (TE = 9330ms, TR = 23ms), with a resolution of 0.2 x 0.2 x 0.2 mm3.
Following image acquisition, the images were corrected for bias field non-
uniformity and normalized to a common intensity range of [0,1000]. In our
work, we are specifically interested in segmenting the medial temporal lobe
(MTL), a region affected early in Alzheimer’s Disease. Therefore, to
facilitate semi-automated MTL segmentation, each scan was re-oriented so that
the long axis of the MTL aligned with the anterior-poster direction.

##### Ground Truth Tissue Segmentation

To generate 3D segmentations of the MTL cortex, we adopted a semi-automatic
interpolation technique [15]. The boundary of the MTL was manually traced
approximately every 3 mm (i.e. 12-15 slices per dataset). Given the subset of
labeled slices, the interpolation method uses contour and intensity
information to compute the intermediate segmentations. This algorithm was
applied iteratively, allowing the interpolated result to be reviewed and
manually edited at each step to refine the segmentation. When editing, we
ensured that in narrow and bridged sulci, the full extent of each sulcus was
correctly labeled as background. Additionally, in a small region surrounding
the MTL, the white matter and background voxels were semi-automatically
labeled using a combination of intensity-based thresholding and morphological
operations. The ground truth segmentations also contain a separate label for
the stratum radiatum lacunosum moleculare (SRLM), which is the thin, hypo-
intense layer within the hippocampus. We note that the ground truth
segmentations only cover the region in the image encompassing the MTL, and not
the entirety of the ex vivo MRI scan.

##### Ground Truth Laplacian maps

The proposed framework requires the Laplacian field maps corresponding to the
ground truth segmentations to train the model. To solve Laplace’s equation
within the ground truth GM volume, we used the iterative finite-differences
approach, as implemented in [2]. This implementation employs a 26-neighbour
average to compute the updated potential field and terminates the numerical
solver when the Laplacian field change is below a specified threshold (sum of
changes < 0.001% of total volume). To initialize the solver, source and sink
boundary conditions were semi-manually labeled as the WM and pial surfaces of
the MTL respectively. We note that the hippocampus voxels were not included in
the GM domain of Laplace’s equation.

###  3.2 Implementation Details

We used Pytorch 1.9.1 and Nvidia Quadro RTX 500 GPUs to train the models. We
implemented the differentiable Laplacian solver within the standardized
training framework presented in [9] that is employed by both backbone
networks. The framework includes pre-processing, automated hyper-parameter
selection and fixed techniques for data augmentation. In our experiments, we
made a few modifications to the default training parameters. First, since the
ground truth segmentations only cover a portion of the input ex vivo MRI
scans, we set the ignore_label parameter in the loss function to 0 to exclude
the unlabeled background voxels from the training process. Additionally, we
increased the oversample_foreground parameter such that only foreground
patches are sampled during training. Lastly, we used an input patch size of 96
x 96 x 96, to encourage the network to learn more local image features instead
of larger contextual information, like the anatomical boundaries of the MTL.
The networks were trained with a batch size of 2 (nnFormer) and 4 (nnU-Net),
for 250 epochs in a five-fold cross validation setting. We used the results of
the first fold to tune the network parameters. Consistent with the evaluation
scheme used within nnU-Net, we aggregated the results across the remaining
four folds for reporting test accuracy. We tested the performance of the
network when using either 5 or 10 class labels (i.e. laminar layers) for
generating the Laplacian segmentation. We found that increasing the number of
class labels and training the network using Laplacian segmentations with a
denser number of laminar layers improved the network’s ability to detect
obscured sulci. To convert the Laplacian fields to segmentations, we used
$\beta=10$ (softmax scaling parameter) and selected evenly spaced thresholds
spanning the [0,1] range of the Laplacian field. More specifically, the
following lower ($t_{lower}$) and upper ($t_{upper}$) threshold values were
used:
[(-0.3,-0.2),(0,0.1),(0.1,0.2),(0.2,0.3),(0.3,0.4),(0.4,0.5),(0.5,0.6),(0.6,0.7),(0.7,0.8),
(0.8,0.95),(0.95,1.05)]. Additionally, we tested the effect of increasing the
weight given to the Laplacian segmentation loss relative to the tissue
segmentation loss and found that it had minimal effect on cortical
segmentation accuracy.

###  3.3 Evaluation

We compared the performance of our approach with the performance of the
corresponding backbone segmentation networks, trained only with the tissue
segmentation loss. We measured segmentation accuracy by computing the DSC
between the predicted and ground truth tissue segmentations, and Laplacian
field segmentations within the MTL region of interest. We also report the
symmetric Hausdorff Distance (HD) 95th percentile between the predicted and
ground truth segmentation of the MTL cortex. Since the numerical solvers used
to generate the ground truth Laplacian fields and embedded in the network
leverage different finite-difference approximation methods, during evaluation,
we re-computed the Laplacian field for both the ground truth and predicted
cortical segmentations using 120 iterations of the SOR solver used by the
network, and computed the corresponding Laplacian segmentation with 5 laminar
layers.

In a secondary analysis we evaluated the effect of introducing the Laplacian
constraint on downstream cortical thickness measures. We applied the nnU-Net
models to a dataset of 36 temporal lobe specimens obtained from individuals
not included in the training dataset. For each specimen, we quantified MTL
thickness at 6 manually identified landmarks corresponding to the anterior and
posterior locations of MTL subregions Brodmann Area (BA) 35, BA36 and the
parahippocampal cortex (PHC). We chose these subregions since they typically
lie along the banks of the collateral sulcus (CS), and are therefore mostly
likely to be affected by topological errors in the segmentation. For each
location, we extracted the GM segmentation surrounding the landmark and
measured cortical thickness using the pipeline described in [16]. In brief,
given the GM segmentation surrounding a landmark, a maximally inscribed sphere
is computed using Voronoi skelentonization [13], and the diameter of the
sphere gives the thickness at that landmark. We compared thickness
measurements obtained when using automatic GM segmentations generated by the
baseline nnU-Net and the proposed model (nnU-Net+Laplacian), and reference
thickness measurements computed using semi-automatic segmentations of the GM
in terms of Pearson’s correlation and the average fixed-raters Intra-Class
Correlation Coefficient (ICC).

Figure 2: Qualitative comparison of the cortical segmentations generated by
the proposed method and nnU-Net. Cross-sectional views are provided through
the anterior and posterior MTL, using four different specimens. The white
dashed boxes are used to indicate cortical folds demonstrating improved
geometric accuracy. GM: Gray Matter; WM: White Matter; SRLM: Stratum Radiatum
Lacunosum Moleculare Table 1: Quantitative metrics comparing cortical
segmentation accuracy of the proposed network and the baseline networks. We
compute the Laplacian Segmentation accuracy, which reflects the networks
ability to capture the layered nature of the cortex, across the whole MTL, and
also separately for the anterior and posterior MTL. Metrics: Dice Score
Coefficient (DSC) per label; Hausdorff Distance (HD). Standard deviations are
reported in parentheses. (Statistical significance was assessed using paired
t-tests; *** < 0.001, **: p < 0.01, *: p < 0.05)

[HTML]FFFFFF DSC Laplacian Segmentation (%) DSC Tissue Segmentation (%)
[HTML]FFFFFF Method Pial Surface $\rightarrow$ WM Surface GM WM BG SRLM HD 95
(mm) nnU-Net 78.3 (3.3) 78.6 (3.1) 77.4 (3.1) 77.7 (5.0) 58.7 (12.3) 94.5
(1.5) 96.0 (1.3) 95.3 (6.1) 85.8 (3.8) 0.341 (0.134) [HTML]C0C0C0 [HTML]C0C0C0
[HTML]C0C0C0 [HTML]C0C0C0 [HTML]C0C0C0 [HTML]C0C0C0 [HTML]C0C0C0 [HTML]C0C0C0
[HTML]C0C0C0 [HTML]C0C0C0 [HTML]C0C0C0 [HTML]C0C0C0nnU-Net + Laplacian
[HTML]C0C0C0  80.7**** (2.6) [HTML]C0C0C0  81.8**** (2.4) [HTML]C0C0C0
80.4**** (3.5) [HTML]C0C0C0  78.9* (5.8) [HTML]C0C0C0  58.2 (13.4)
[HTML]C0C0C0  94.5 (1.7) [HTML]C0C0C0  95.9 (1.3) [HTML]C0C0C0  95.4 (5.9)
[HTML]C0C0C0  85.6 (3.8) [HTML]C0C0C0  0.344 (0.205) nnFormer 75.5 (4.1) 75.5
(3.7) 74.6 (3.4) 75.9 (4.7) 57.0* (10.6) 93.8** (1.9) 95.1 * (2.1) 95.1 (6.1)
83.8** (4.2) 0.559 (0.885) Whole MTL [HTML]C0C0C0nnFormer + Laplacian
[HTML]C0C0C0  78.1*** (3.4) [HTML]C0C0C0  79.8**** (3.0) [HTML]C0C0C0
78.4**** (3.6) [HTML]C0C0C0  77.3*** (5.2) [HTML]C0C0C0  55.7 (11.6)
[HTML]C0C0C0  93.5 (2.2) [HTML]C0C0C0  94.9 (2.3) [HTML]C0C0C0  95.1 (6.0)
[HTML]C0C0C0  83.2 (4.5) [HTML]C0C0C0  0.589 (0.980) nnU-Net 80.0 (3.7) 80.6
(3.4) 79.9 (3.3) 79.6 (5.0) 59.3 (12.8) 94.5 (1.6) 95.0** (1.6) 95.7 (5.1)
85.9 (4.4) 0.208 (0.025) [HTML]C0C0C0nnU-Net + Laplacian [HTML]C0C0C0  81.6**
(3.0) [HTML]C0C0C0  82.8*** (2.7) [HTML]C0C0C0  81.8*** (3.7) [HTML]C0C0C0
80.3 (5.9) [HTML]C0C0C0  58.1 (13.7) [HTML]C0C0C0  94.4 (1.9) [HTML]C0C0C0
94.7 (1.7) [HTML]C0C0C0  95.9 (5.0) [HTML]C0C0C0  85.7 (4.5) [HTML]C0C0C0
0.217 (0.049) nnFormer 76.8 (4.5) 76.9 (4.3) 76.5 (4.4) 78.0 (5.1) 58.4**
(11.4) 93.8* (2.5) 93.9 (3.6) 95.7 (5.2) 83.7** (5.0) 0.208 (0.0205) Anterior
MTL [HTML]C0C0C0nnFormer + Laplacian [HTML]C0C0C0  79.3*** (3.4) [HTML]C0C0C0
80.9**** (3.0) [HTML]C0C0C0  80.0**** (3.8) [HTML]C0C0C0  79.1** (5.2)
[HTML]C0C0C0  56.8 (12.1) [HTML]C0C0C0  93.6 (2.7) [HTML]C0C0C0  93.7 (3.9)
[HTML]C0C0C0  95.8 (5.1) [HTML]C0C0C0  83.1 (5.4) [HTML]C0C0C0  0.212 (0.030)
nnU-Net 74.8 (6.2) 74.7 (5.8) 73.2 (5.4) 74.6 (6.1) 57.2 (12.9) 94.7 (1.6)
96.9 (1.4) 94.3* (9.0) 85.3 (3.4) 0.228 (0.096) [HTML]C0C0C0nnU-Net +
Laplacian [HTML]C0C0C0  78.9**** (4.7) [HTML]C0C0C0  79.8**** (3.9)
[HTML]C0C0C0  78.1**** (4.1) [HTML]C0C0C0  76.8*** (6.5) [HTML]C0C0C0  58.0
(13.5) [HTML]C0C0C0  94.6 (1.7) [HTML]C0C0C0  97.0 (1.1) [HTML]C0C0C0  94.1
(9.0) [HTML]C0C0C0  85.2 (3.4) [HTML]C0C0C0  0.223 (0.062) nnFormer 72.7 (8.1)
72.8 (7.0) 71.1 (6.6) 72.7 (6.9) 55.0 (11.9) 93.9*** (2.5) 96.2* (2.0) 93.8*
(8.9) 84.3* (4.0) 0.348 (0.430) Posterior MTL [HTML]C0C0C0nnFormer + Laplacian
[HTML]C0C0C0  75.4*** (8.3) [HTML]C0C0C0  77.5**** (6.9) [HTML]C0C0C0
75.4**** (6.6) [HTML]C0C0C0  74.3** (7.9) [HTML]C0C0C0  54.0 (13.1)
[HTML]C0C0C0  93.2 (3.2) [HTML]C0C0C0  96.0 (2.2) [HTML]C0C0C0  93.6 (9.0)
[HTML]C0C0C0  83.6 (4.0) [HTML]C0C0C0  0.436 (0.692)

##  4 Results and Discussion

###  4.1 Segmentation Accuracy

Table 1 presents the quantitative results, averaged across four cross-
validation folds, evaluating the proposed framework using the corresponding
backbone networks as baseline. Since sulci can be very thin structures, any
improvements to mislabeled sulci would contribute minimally to the tissue
segmentation DSC. Therefore, it is not surprising that we do not see
statistically significant differences in the GM DSC measures. However, when
looking at the Laplacian segmentation DSC, the results show that the Laplacian
solver significantly improves upon the baseline performance of both nnU-Net
and nnFormer in terms of better preserving the layered structure of the
cortex. This is visualized in Fig. 2 which provides a qualitative comparison
of the predicted segmentations generated by each method. Labels 1-3 correspond
to layers closest to the pial surface, and are therefore the class labels
mostly likely to reflect errors such as bridged or unresolved sulci.

We observe that in the anterior portion of the MTL, the baseline models are
often able to distinguish the sulcus, even without the Laplacian constraint
(Fig. 2, row 2). This is likely because in the ground truth segmentation
protocol, the labeled GM extends to include both banks of the CS in the
anterior MTL, but only the medial bank of the CS in the posterior MTL. As a
result, in the ground truth segmentation, the sulcus is clearly labeled in the
anterior MTL and therefore included in the tissue segmentation loss.
Conversely, in the posterior MTL, the ground truth tissue segmentation does
not explicitly enforce the presence of the sulcus. However, in this region,
the Laplacian segmentation term implicitly includes information about the
location of the sulcus and therefore drives the network to learn the correct
pial boundary of the cortex. To further investigate the contribution of the
proposed loss function in the anterior and posterior MTL, we computed the DSC
metrics of the Laplacian segmentations separately for the anterior and
posterior MTL (Table 1). We observe that even when considering the anterior
MTL on its own, the proposed framework improves Laplacian segmentation
accuracy compared to the baseline networks, confirming that the addition of
the Laplacian term is in fact contributing towards the network better learning
the layered organization of the cortex. This is further seen in the nnU-
Net+Laplacian result in Fig. 2, row 2, where the proposed network is able to
detect a buried sulcus in a cortical fold not included within the ground truth
region of interest. We note that Laplacian segmentation label 5, which
corresponds to the innermost cortical surface at the GM/WM boundary, forms a
very thin layer and therefore has greater variation in DSC compared to the
other labels.

###  4.2 Downstream Thickness Measurements

Fig. 3 shows the results of the morphometry analysis correlating automated and
manual measurements of cortical thickness in BA35, BA36 and the PHC, when
using cortical segmentations generated with and without the Laplacian-based
loss term. Since we found that nnU-Net achieves better segmentation
performance than nnFormer, we only conducted experiments using nnU-Net in the
seconday analysis. Compared to the baseline nnU-Net, we observe that the
thickness measurements of BA36 and the PHC computed using the proposed network
are more strongly correlated with manual measurements, in terms of both
correlation coefficient and ICC. Both models achieve similar correlations in
BA35. BA36 is located in the anterior MTL, whereas PHC corresponds to the
posterior MTL. The strengthened correlations in both these regions further
demonstrate that the proposed method is able to improve the accuracy of the
predicted segmentations across the whole length of MTL.

Figure 3: A) Example scan showing the 6 landmarks where cortical thickness is
measured. For each subregion, the thickness measurement is averaged across two
landmark locations. B) Scatter plots showing the correlation between automated
segmentation-based cortical thickness measurements and reference measurements
based on semi-automatic segmentations for three MTL subregions, with (green)
and without (purple) the Laplacian constraint. C) Segmentations produced by
nnUnet and nnU-Net+Laplacian for BA36 (red) and PHC (blue) landmarks where
thickness measures derived from the two networks differed the most. BA:
Brodmann Area; PHC: parahippocampal cortex

##  5 Conclusions

We present a novel deep learning-based solution for cortical segmentation,
applied to ex vivo MRI, that is able to learn the layered geometry of the
cortex by locally imposing Laplacian mappings between the predicted WM and
pial cortical surfaces. A limitation of this approach is the long run-time of
the iterative solver during training ($\sim$9x slower/epoch relative to the
backbone). However, at inference time, the input image is only passed through
the backbone segmentation network which typically takes 3-5 minutes per scan.
Another limitation is the need for the sulci to be well delineated in the
training data. In the future, we will explore ways to relax this requirement,
perhaps using additional geometric priors. While in this work we demonstrate
the utility of our approach in the context of MTL cortical segmentation, this
approach can be extended to other high-resolution neuroimaging datasets such
as ex vivo whole hemisphere scans or other in vivo image segmentation tasks
which involve similar sheet-like structures. Future work will focus on
applying this method to in vivo brain MRI, thus allowing for evaluation of our
approach against existing cortical surface reconstruction methods.

####  5.0.1 Acknowledgements

We gratefully acknowledge the tissue donors and their families. This work was
supported by the NIH (Grants RF1 AG069474, P30 AG072979 and R01 AG056014), a
UCLM travel and research grant (to R.I), and an Alzheimer’s Association grant
(AARF-19-615258) (to L.E.M.W).

## References

  * [1] Cruz, R.S., Lebrat, L., Bourgeat, P., Fookes, C., Fripp, J., Salvado, O.: Deepcsr: A 3d deep learning approach for cortical surface reconstruction. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 806–815 (2021)
  * [2] DeKraker, J., Ferko, K.M., Lau, J.C., Köhler, S., Khan, A.R.: Unfolding the hippocampus: An intrinsic coordinate system for subfield segmentations and quantitative mapping. Neuroimage 167, 408–418 (2018)
  * [3] Epicoco, I., Mocavero, S.: The performance model of an enhanced parallel algorithm for the sor method. In: International Conference on Computational Science and Its Applications. pp. 44–56. Springer (2012)
  * [4] Fischl, B.: Freesurfer. Neuroimage 62(2), 774–781 (2012)
  * [5] Han, X., Pham, D.L., Tosun, D., Rettmann, M.E., Xu, C., Prince, J.L.: CRUISE: Cortical reconstruction using implicit surface evolution. NeuroImage 23(3), 997–1012 (2004)
  * [6] Hansen, P.B.: Numerical solution of laplace’s equation (1992)
  * [7] Henschel, L., Conjeti, S., Estrada, S., Diers, K., Fischl, B., Reuter, M.: Fastsurfer-a fast and accurate deep learning based neuroimaging pipeline. NeuroImage (2020)
  * [8] Hoopes, A., Iglesias, J.E., Fischl, B., Greve, D., Dalca, A.V.: Topofit: Rapid reconstruction of topologically-correct cortical surfaces. In: Medical Imaging with Deep Learning (2021)
  * [9] Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203–211 (2021)
  * [10] Jones, S.E., Buchbinder, B.R., Aharon, I.: Three-dimensional mapping of cortical thickness using laplace’s equation. Human brain mapping 11(1), 12–32 (2000)
  * [11] Kim, J.S., Singh, V., Lee, J.K., Lerch, J., Ad-Dab’bagh, Y., MacDonald, D., Lee, J.M., Kim, S.I., Evans, A.C.: Automated 3-d extraction and evaluation of the inner and outer cortical surfaces using a laplacian map and partial volume effect classification. Neuroimage 27(1), 210–221 (2005)
  * [12] Ma, Q., Robinson, E.C., Kainz, B., Rueckert, D., Alansary, A.: Pialnn: A fast deep learning framework for cortical pial surface reconstruction. In: International Workshop on Machine Learning in Clinical Neuroimaging. pp. 73–81 (2021)
  * [13] Ogniewicz, R., Kübler, O.: Hierarchic Voronoi skeletons. Pattern Recognition 28(3), 343–359 (mar 1995)
  * [14] Osechinskiy, S., Kruggel, F.: Cortical Surface Reconstruction from High-Resolution MR Brain Images. International Journal of Biomedical Imaging 2012 (2012)
  * [15] Ravikumar, S., Wisse, L., Gao, Y., Gerig, G., Yushkevich, P.: Facilitating Manual Segmentation of 3D Datasets using Contour and Intensity Guided Interpolation. In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). pp. 714–718 (2019)
  * [16] Wisse, L.E., Ravikumar, S., Ittyerah, R., Lim, S., Lane, J., Bedard, M.L., Xie, L., Das, S.R., Schuck, T., Grossman, M., Lee, E.B., Tisdall, M.D., Prabhakaran, K., Detre, J.A., Mizsei, G., Trojanowski, J.Q., Artacho-Pérula, E., de Iñiguez de Onzono Martin, M.M., M. Arroyo-Jiménez, M., Muñoz Lopez, M., Molina Romero, F.J., P. Marcos Rabal, M., Cebada Sánchez, S., Delgado González, J.C., de la Rosa Prieto, C., Córcoles Parada, M., Wolk, D.A., Irwin, D.J., Insausti, R., Yushkevich, P.A.: Downstream effects of polypathology on neurodegeneration of medial temporal lobe subregions. Acta Neuropathologica Communications 9(1), 1–11 (dec 2021)
  * [17] Yang, S., Matthias, K.G.: The optimal relaxation parameter for the sor method applied to a classical model problem. Tech. rep., Technical Report TR2007–6, University of Maryland, Baltimore County (2007)
  * [18] Zhou, H.Y., Guo, J., Zhang, Y., Yu, L., Wang, L., Yu, Y.: nnformer: Interleaved transformer for volumetric segmentation. arXiv preprint arXiv:2109.03201 (2021)


# Video Action Recognition Collaborative Learning with Dynamics via PSO-
ConvNet Transformer

Nguyen Huu Phong  CISUC, Department of Informatics Engineering, University of
Coimbra, Coimbra, Portugal<EMAIL_ADDRESS>Bernardete Ribeiro  CISUC,
Department of Informatics Engineering, University of Coimbra, Coimbra,
Portugal

###### Abstract

Human Action Recognition (HAR) involves the task of categorizing actions
present in video sequences. Although it presents interesting problems, it
remains one of the most challenging domains in pattern recognition.
Convolutional Neural Networks (ConvNets) have demonstrated exceptional success
in image recognition and related areas. However, these advanced techniques are
not always directly applicable to HAR, as the consideration of temporal
features is crucial. In this paper, we present a dynamic PSO-ConvNet model for
learning actions in video, drawing on our recent research in image
recognition. Our methods are based on a framework where the weight vector of
each neural network serves as the position of a particle in phase space, and
particles exchange their current weight vectors and gradient estimates of the
Loss function. We extend the approach to video by integrating a ConvNet with
state-of-the-art temporal methods such as Transformer and Recurrent Neural
Networks. The results reveal substantial advancements, with improvements of up
to 9% on UCF-101 dataset. The code is available at
https://github.com/leonlha/Video-Action-Recognition-via-PSO-ConvNet-
Transformer-Collaborative-Learning-with-Dynamics.

## Introduction

Human action recognition plays a vital role for distinguishing a particular
behavior of interest in the video. It has critical applications including
visual surveillance for detection of suspicious human activities to prevent
the fatal accidents [1, 2], automation-based driving to sense and predict
human behavior for safe navigation [3, 4]. In addition, there are large amount
of non-trivial applications such as human-machine interaction [5, 6], video
retrieval [7], crowd scene analysis [8] and identity recognition [9].

In the early days, the majority of research in Human Activity Recognition was
conducted using hand-crafted methods [10, 11, 12]. However, as deep learning
technology evolved and gained increasing recognition in the research
community, a multitude of new techniques have been proposed, achieving
remarkable results.

Action recognition preserves a similar property of image recognition since
both of the fields handle visual contents. In addition, action recognition
classifies not only still images but also dynamics temporal information from
the sequence of images. Built on these intrinsic characteristics, action
recognition’s methods can be grouped into two main approaches namely recurrent
neural networks (RNN) based approach and 3-D ConvNet based approach. Besides
of the main ones, there are other methods that utilize the content from both
spatial and temporal and coined the name two-stream 2-D ConvNet based approach
[13].

Initially, action recognition was viewed as a natural extension of image
recognition, and spatial features from still frames could be extracted using
ConvNet, which is one of the most efficient techniques in the image
recognition field. However, traditional ConvNets are only capable of
processing a single 2-D image at a time. To expand to multiple 2-D images, the
neural network architecture needs to be re-designed, including adding an extra
dimension to operations such as convolution and pooling to accommodate 3-D
images. Examples of such techniques include C3D [14], I3D [15], R3D [16], S3D
[17], T3D [18], LTC [19], among others

Similarly, since a video primarily consists of a temporal sequence, techniques
for sequential data, such as Recurrent Neural Networks and specifically Long
Short Term Memory, can be utilized to analyze the temporal information.
Despite the larger size of images, feature extraction is often employed. Long-
term Recurrent Convolutional Networks (LRCN)[20] and Beyond-Short-Snippets[21]
were among the first attempts to extract feature maps from 2-D ConvNets and
integrate them with LSTMs to make video predictions. Other works have adopted
bi-directional LSTMs [22, 23], which are composed of two separate LSTMs, to
explore both forward and backward temporal information.

To further improve performance, other researchers argue that videos usually
contain repetitive frames or even hard-to-classify ones which makes the
computation expensive. By selecting relevant frames, it can help to improve
action recognition performance both in terms of efficiency and accuracy [24].
A similar concept based on attention mechanisms is the main focus in recent
researches to boost overall performance of the ConvNet-LSTM frameworks [25,
26].

While RNNs are superior in the field, they process data sequentially, meaning
that information flows from one state to the next, hindering the ability to
speed up training in parallel and causing the architectures to become larger
in size. These issues limit the application of RNNs to longer sequences. In
light of these challenges, a new approach, the Transformer, emerged [27, 28,
29, 30, 31].

There has been a rapid advancement in action recognition in recent years, from
3-D ConvNets to 2-D ConvNets-LSTM, two-stream ConvNets, and more recently,
Transformers. While these advancements have brought many benefits, they have
also created a critical issue as previous techniques are unable to keep up
with the rapidly changing pace. Although techniques such as evolutionary
computation offer a crucial mechanism for architecture search in image
recognition, and swarm intelligence provides a straightforward method to
improve performance, they remain largely unexplored in the realm of action
recognition.

In our recent research [32], we developed a dynamic Particle Swarm
Optimization (PSO) framework for image classification. In this framework, each
particle navigates the landscape, exchanging information with neighboring
particles about its current estimate of the geometry (such as the gradient of
the Loss function) and its position. The overall goal of this framework is to
create a distributed, collaborative algorithm that improves the optimization
performance by guiding some of the particles up to the best minimum of the
loss function. We extend this framework to action recognition by incorporating
state-of-the-art methods for temporal data (Transformer and RNN) with the
ConvNet module in an end-to-end training setup.

##  1 Related Works

In recent years, deep learning (DL) has greatly succeed in computer vision
fields, e.g., object detection, image classification and action recognition
[33, 24, 30]. One consequence of this success has been a sharp increase in the
number of investments in searching for good neural network architectures. An
emerging promising approach is changing from the manual design to automatic
Neural Architecture Search (NAS). As an essential part of automated machine
learning, NAS automatically generates neural networks which have led to state-
of-the-art results [34, 35, 36]. Among various approaches for NAS already
present in the literature, evolutionary search stands out as one of the most
remarkable methods. For example, beginning with just one layer of neural
network, the model develops into a competitive architecture that outperforms
contemporary counterparts [34]. As a result, the efficacy of the their
proposed classification system for HAR on UCF-50 dataset was demonstrated [33]
by initializing the weights of a convolutional neural network classifier based
on solutions generated from genetic algorithms (GA).

In addition to Genetic Algorithms, Particle Swarm Optimization - a population-
based stochastic search method influenced by the social behavior of flocking
birds and schooling fish - has proven to be an efficient technique for feature
selection [37, 38]. A novel approach that combines a modified Particle Swarm
Optimization with Back-Propagation was put forth for image recognition, by
adjusting the inertia weight, acceleration parameters, and velocity [39]. This
fusion allows for dynamic and adaptive tuning of the parameters between global
and local search capability, and promotes diversity within the swarm. In
catfish particle swarm optimization, the particle with the worst fitness is
introduced into the search space when the fitness of the global best particle
has not improved after a number of consecutive iterations [40]. Moreover, a
PSO based multi-objective for discriminative feature selection was introduced
to enhance classification problems [41].

There have been several efforts to apply swarm intelligence to action
recognition from video. One such approach employs a combination of binary
histogram, Harris corner points, and wavelet coefficients as features
extracted from the spatiotemporal volume of the video sequence [42]. To
minimize computational complexity, the feature space is reduced through the
use of PSO with a multi-objective fitness function.

Furthermore, another approach combining Deep Learning and swarm intelligence-
based metaheuristics for Human Action Recognition was proposed [43]. Here,
four different types of features extracted from skeletal data - Distance,
Distance Velocity, Angle, and Angle Velocity - are optimized using the nature-
inspired Ant Lion Optimizer metaheuristic to eliminate non-informative or
misleading features and decrease the size of the feature set. Despite these
advances, the field remains largely uncharted, especially with respect to
recent and emerging techniques.

##  2 Proposed Methods

###  2.1 Collaborative Dynamic Neural Networks

Define $\mathcal{N}(n,t)$ as the set of $k$ nearest neighbor particles of
particle $n$ at time $t$, where $k\in\mathbb{N}$ is some predefined number. In
particular,

$\displaystyle\mathcal{N}(n,t)={}$
$\displaystyle\\{(x^{(n)}(t),v^{(n)}(t)),(x^{(i_{1})}(t),v^{(i_{1})}(t)),(x^{(i_{2})},v^{(i_{2})})(t),\ldots,(x^{(i_{k})}(t),v^{(i_{k})}(t))\\}$
(1)

where $i_{1}$, $i_{2}$,… $i_{k}$ are the $k$ closest particles to $n$ and
$x^{(i_{k})}(t)$ and $v^{(i_{k})}(t)\in\mathbb{R}^{D}$ represent the position
and velocity of particle $i_{k}$ at time $t$. Figure 1 illustrates this
concept for $k=4$ particles.

Figure 1: A demonstration of the $\mathcal{N}(n,t)$ neighborhood, consisting
of the positions of four closest particles and particle $n$ itself, is shown.
The velocities of the particles are depicted by arrows.

Given a (continuous) function $L\,:\,\mathbb{R}^{D}\longrightarrow\mathbb{R}$
and a (compact) subset $S\subset\mathbb{R}^{D}$, define

$\mathcal{Y}={\sf argmin}\left\\{L(y)\,:\,y\in S\right\\}$ (2)

as the subset of points that minimize $L$ in $S$, i.e., $L(z)\leq L(w)$ for
any $z\in\mathcal{Y}\subset S$ and $w\in S$.

Dynamic 1: We investigate a set of neural networks that work together in a
decentralized manner to minimize a Loss function $L$. The training process is
comprised of two phases: i) individual training of each neural network using
(stochastic) gradient descent, and ii) a combined phase of SGD and PSO-based
cooperation. The weight vector of each neural network is represented as the
position of a particle in a $D$-dimensional phase space, where $D$ is the
number of weights. The evolution of the particles (or neural networks) is
governed by equation (3), with the update rule specified by the following
dynamics:

$\begin{array}[]{ccl}\psi^{(n)}(t+1)&=&-\eta\nabla L\left(x^{(n)}(t)\right)\\\
&&\\\ \phi^{(n)}(t+1)&=&x^{(n)}(t)+\psi^{(n)}(t+1)\\\ &&\\\
v^{(n)}(t+1)\\!\\!\\!&\\!\\!\\!=\\!\\!\\!&\\!\\!\\!\sum\limits_{\ell\in\mathcal{N}(n,t)}w_{n\ell}\psi^{(\ell)}(t+1)+c_{1}r(t)\left(P^{(n)}(t)-\phi^{(n)}(t+1)\right)+c_{2}r(t)\left(P_{g}^{(n)}(t)-\phi^{(n)}(t+1)\right)\\\
x^{(n)}(t+1)&=&x^{(n)}(t)+v^{(n)}(t)\end{array}$ (3)

where $v^{(n)}(t)\in\mathbb{R}^{D}$ is the velocity vector of particle $n$ at
time $t$; $\psi^{(n)}(t)$ is an intermediate velocity computed from the
gradient of the Loss function at $x^{(n)}(t)$; $\phi^{(n)}(t)$ is the
intermediate position computed from the intermediate velocity $\psi^{(n)}(t)$;
$r(t)\overset{i.i.d.}{\sim}{\sf Uniform}\left(\left[0,1\right]\right)$ is
randomly drawn from the interval $\left[0,1\right]$ and we assume that the
sequence $r(0)$, $r(1)$, $r(2)$, $\ldots$ is i.i.d.;
$P^{(n)}(t)\in\mathbb{R}^{D}$ represents the _best_ position visited up until
time $t$ by particle $n$, i.e., the position with the minimum value of the
Loss function over all previous positions
$x^{(n)}(0),\,x^{(n)}(1),\,\ldots,\,x^{(n)}(t)$; $P_{g}^{(n)}(t)$ represents
its nearest-neighbors’ counterpart, i.e., the best position across all
previous positions of the particle $n$ jointly with its corresponding nearest-
neighbors $\bigcup_{s\leq t}\mathcal{N}\left(n,s\right)$ up until time $t$:

$\begin{array}[]{ccl}P^{(n)}(t+1)&\in&{\sf
argmin}\left\\{L(y)\,:\,y=P^{(n)}(t),x^{(n)}(t)\right\\}\\\ &&\\\
P_{g}^{(n)}(t+1)&\in&{\sf
argmin}\left\\{L(y)\,:\,y=P_{g}^{(n)}(t),x^{(k)}(t);\right.\\\
&&\left.k\in\mathcal{N}(n,t)\right\\}\\\ &&\\\ \end{array}.$ (4)

The weights $w_{n\ell}$ are defined as

$w_{n\ell}=f\left(\left|\left|x^{(n)}(t)-x^{(\ell)}(t)\right|\right|\right),$
(5)

with $\left|\left|\cdot\right|\right|$ being the Euclidean norm and
$f\,:\,\mathbb{R}\rightarrow\mathbb{R}$ being a decreasing (or at least non-
increasing) function. In Dynamic 1, we assume that

$f(z)=\frac{M}{\left(1+z\right)^{\beta}},$ (6)

for some constants $M,\beta>0$. This _strengthens_ the collaboration learning
between any of two particles by pushing each particle against each other.

Dynamic 2: An alternative to equation (3) is to pull back a particle instead
of pushing it in the direction of the gradient. In the previous section, the
assumption was that all particles were located on the same side of a valley in
the loss function. However, if one particle is on the opposite side of the
valley relative to the rest of the particles, it will be pulled further away
from the minimum using the first dynamic. To address this issue, we introduce
a second dynamic (Dynamic 2) that pulls the particle back. The formula for
this dynamics is as follows:

$\begin{array}[]{ccl}x_{(i)}(t+1)&=&x_{(i)}(t)+\sum_{j=1}^{N}\frac{M_{ij}}{(1+\left|\left|x_{i}(t)-x_{j}(t)\right|\right|^{2})^{\beta}}(x_{j}(t)-\nabla
L(x_{j}(t)))+cr\left(P_{nbest(i)}(t)-x_{i}(t)\right)\\\ \end{array}$ (7)

where $x_{(i)}(t)\in\mathbb{R}^{D}$ is the position of particle $i$ at time
$t$; $M$, $\beta$ and $c$ are constants set up by experiments with
$\left|\left|\cdot\right|\right|$ being the Euclidean norm;
$r(t)\overset{i.i.d.}{\sim}{\sf Uniform}\left(\left[0,1\right]\right)$
randomly drawn from the interval $\left[0,1\right]$ and we assume that the
sequence $r(0)$, $r(1)$, $r(2)$, $\ldots$ is i.i.d.;
$P_{nbest(i)}(t)\in\mathbb{R}^{D}$ represents nearest-neighbors’ best , i.e.,
the best position across all previous positions of the particle $n$ jointly
with its corresponding nearest-neighbors $\bigcup_{s\leq
t}\mathcal{N}\left(n,s\right)$ up until time $t$.

###  2.2 ConvNet Transformer Architecture for Action Recognition

In this section, we discuss a hybrid ConvNet-Transformer architecture that
replaces the traditional ConvNet-RNN block for temporal input to classify
human action in videos. The architecture is composed of several components,
including a feature extraction module using ConvNet, a position embedding
layer, multiple transformer encoder blocks, and classification and aggregation
modules. The overall diagram of the architecture can be seen in Figure 2. The
goal of the architecture is to effectively capture the temporal information
present in the video sequences, in order to perform accurate human action
recognition. The hybrid ConvNet-Transformer design leverages the strengths of
both ConvNets and Transformers, offering a powerful solution for this
challenging task.

####  2.2.1 Features Extraction via ConvNet and Position Embedding

In the early days of using Transformer for visual classification, especially
for images, the frames were typically divided into smaller patches and used as
the primary input [31, 44, 45]. However, these features were often quite
large, leading to high computational requirements for the Transformer. To
balance efficiency and accuracy, ConvNet can be utilized to extract crucial
features from images, reducing the size of the input without sacrificing
performance.

We assume that, for each frame, the extracted features from ConNet have a size
of ($w$, $h$, $c$) where $w$ and $h$ are the width and height of a 2D feature
and $c$ is the number of filters. To further reduce the size of the features,
global average pooling is applied, reducing the size from $w\times h\times c$
to $c$.

The position encoding mechanism in Transformer is used to encode the position
of each frame in the sequence. The position encoding vector, which has the
same size as the feature, is summed with the feature and its values are
computed using the following formulas. This differs from the sequential
processing of data in the RNN block, allowing for parallel handling of all
entities in the sequence.

$\begin{array}[]{ccl}PE_{(pos,2i)}&=&sin(pos/{10000^{2i/{d_{model}}}})\\\
PE_{(pos,2i+1)}&=&cos(pos/{10000^{2i/{d_{model}}}})\end{array}$ (8)

where $pos$, $i$ and $PE$ are the time step index of the input vector, the
dimension and the positional encoding matrix; $d_{model}$ refers to the length
of the position encoding vector.

Figure 2: Rendering End-to-end ConvNet-Transformer Architecture.

####  2.2.2 Transformer Encoder

The Transformer Encoder is a key component of the hybrid ConvNet-Transformer
architecture. It consists of a stack of $N$ identical layers, each comprising
multi-head self-attention and position-wise fully connected feed-forward
network sub-layers. To ensure the retention of important input information,
residual connections are employed before each operation, followed by layer
normalization.

The core of the module is the multi-head self-attention mechanism, which is
composed of several self-attention blocks. This mechanism is similar to RNN,
as it encodes sequential data by determining the relevance between each
element in the sequence. It leverages the inherent relationships between
frames in a video to provide a more accurate representation. Furthermore, the
self-attention operates on the entire sequence at once, resulting in
significant improvements in runtime, as the computation can be parallelized
using modern GPUs.

Our architecture employs only the encoder component of a full transformer, as
the goal is to obtain a classification label for the video action rather than
a sequence. The full transformer consists of both encoder and decoder modules,
however, in our case, the use of only the encoder module suffices to achieve
the desired result.

Assuming the input sequence ($X={x_{1},x_{2},...,x_{n}}$) is first projected
onto these weight matrices $Q=XW_{Q}$, $K=XW_{K}$, $V=XW_{V}$ with
$W_{Q}$,$W_{K}$ and $W_{V}$ are three trainable weights, the query
($Q=q_{1},q_{2},...,q_{n}$), key ($K=k_{1},k_{2},...,k_{n}$) of dimension
$d_{k}$, and value ($V=v_{1},v_{2},...,v_{n}$) of dimension $d_{v}$, the
output of self-attention is computed as follows:

$\begin{array}[]{ccl}Attention(Q,K,V)&=&softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V.\end{array}$
(9)

As the name suggested, multi-head attention is composed of several heads and
all are concatenated and fed into another linear projection to produce the
final outputs as follows:

$\displaystyle MultiHead(Q,K,V)$ $\displaystyle=$ $\displaystyle
Concat(head_{1},head_{2},...,head_{h})W^{O}.$ (10) $\displaystyle\text{where
}head_{i}$ $\displaystyle=$ $\displaystyle
Attention(QW^{Q}_{i},KW^{K}_{i},VW^{V}_{i})$ (11)

where parameter matrices $W^{Q}_{i}\in\mathbb{R}^{d_{model}\times d_{k}}$,
$W^{K}_{i}\in\mathbb{R}^{d_{model}\times d_{k}}$,
$W^{V}_{i}\in\mathbb{R}^{d_{model}\times d_{v}}$ and
$W^{O}\in\mathbb{R}^{hd_{v}\times d_{model}}$, $i=1,2,...,h$ with $h$ denotes
the number of heads.

####  2.2.3 Frame Selection and Data Pre-processing

Input videos with varying number of frames can pose a challenge for the model
which requires a fixed number of inputs. Put simply, to process a video
sequence, we incorporated a time distributed layer that requires a
predetermined number of frames. To address this issue, we employ several
strategies for selecting a smaller subset of frames.

One approach is the "shadow method," where a maximum sequence length is
established for each video. While this method is straightforward, it can
result in the cutting of longer videos and the loss of information,
particularly when the desired length is not reached. In the second method, we
utilize a step size to skip some frames, allowing us to achieve the full
length of the video while reducing the number of frames used. Additionally,
the images are center-cropped to create square images. The efficacy of each
method will be evaluated in our experiments.

####  2.2.4 Layers for Classification

Assuming, we have a set of videos $S(S_{1},S_{2},...,S_{m})$ with
corresponding labels $y(y_{1},y_{2},...,y_{m})$ where $m$ is the number of
samples. We select $l$ frames from the videos and obtain $g$ features from the
global average pooling 2-D layer. Each transformer encoder generates a set of
representations by consuming the output from the previous block. After $N$
transformer encoder blocks, we can obtain the multi-level representation
$H^{N}(h^{N}_{1},h^{N}_{2},...,h^{N}_{l})$ where each representation is 1-D
vector with the length of $g$ (see Figure 2 block (A) $\rightarrow$ (D)).

The classification module incorporates traditional layers, such as fully
connected and softmax, and also employs global max pooling to reduce network
size. To prevent overfitting, we include Gaussian noise and dropout layers in
the design. The ConvNet-Transformer model is trained using stochastic gradient
descent and the categorical cross entropy loss is used as the optimization
criterion.

###  2.3 ConvNet-RNN

Recent studies have explored the combination of ConvNets and RNNs,
particularly LSTMs, to take into account temporal data of frame features for
action recognition in videos [20, 21, 22, 23, 46, 47].

To provide a clear understanding of the mathematical operations performed by
ConvNets, the following is a summary of the relevant formulations:

$\displaystyle\begin{cases}O_{i}=X&\text{if $i=1$}\\\
Y_{i}=f_{i}(O_{i-1},W_{i})&\text{if $i>1$}\\\ O_{i}=g_{i}(Y_{i})\end{cases}$
(12) $\displaystyle\begin{cases}Y_{i}=W_{i}\circledast O_{i-1}&\text{$i^{th}$
layer is a convolution}\\\ Y_{i}=\boxplus_{n,m}O_{i-1}&\text{$i^{th}$ layer is
a pool}\\\ Y_{i}=W_{i}*O_{i-1}&\text{$i^{th}$ layer is a FC}\end{cases}$ (13)

where $X$ represents the input image; $O_{i}$ is the output for layer
$i^{th}$; $W_{i}$ indicates the weights of the layer; $f_{i}(\cdot)$ denotes
weight operation for convolution, pooling or FC layers; $g_{i}(\cdot)$ is an
activation function, for example, sigmoid, tanh and rectified linear (ReLU) or
more recently Leaky ReLU [48]; The symbol ($\circledast$) acts as a
convolution operation which uses shared weights to reduce expensive matrix
computation [49]; Window ($\boxplus_{n,m}$) shows an average or a max pooling
operation which computes average or max values over neighbor region of size
$n\times m$ in each feature map. Matrix multiplication of weights between
layer $i^{th}$ and the layer $(i-1)^{th}$ in FC is represented as ($*$).

The last layer in the ConvNet (FC layer) acts as a classifier and is usually
discarded for the purpose of using transfer learning. Thereafter, the outputs
of the ConvNet from frames in the video sequences are fed as inputs to the RNN
layer.

Considering a standard RNN with a given input sequence
${x_{1},x_{2},...,x_{T}}$, the hidden cell state is updated at a time step $t$
as follows:

$h_{t}=\sigma(W_{h}h_{t-1}+W_{x}x_{t}+b),$ (14)

where $W_{h}$ and $W_{x}$ denote weight matrices, b represents the bias, and
$\sigma$ is a sigmoid function that outputs values between 0 and 1.

The output of a cell, for ease of notation, is defined as

$y_{t}=h_{t},$ (15)

but can also be shown using the $softmax$ function, in which $\hat{y}_{t}$ is
the output and $y_{t}$ is the target:

$\hat{y_{t}}=softmax(W_{y}h_{t}+b_{y}).$ (16)

A more sophisticated RNN or LSTM that includes the concept of a forget gate
can be expressed as shown in the following equations:

$\displaystyle f_{t}=\sigma(W_{fh}h_{t-1}+W_{fx}x_{t}+b_{f}),$ (17)
$\displaystyle i_{t}=\sigma(W_{ih}h_{t-1}+W_{ix}x_{t}+b_{i}),$ (18)
$\displaystyle
c^{\prime}_{t}=tanh(W_{c^{\prime}h}h_{t-1}+W_{c^{\prime}x}x_{t}+b^{\prime}_{c}),$
(19) $\displaystyle c_{t}=f_{t}\odot c_{t-1}+i_{t}\odot c^{\prime}_{t},$ (20)
$\displaystyle o_{t}=\sigma(W_{oh}h_{t-1}+W_{ox}x_{t}+b_{o}),$ (21)
$\displaystyle h_{t}=o_{t}\odot tanh(c_{t}),$ (22)

where the $\odot$ operation represents an elementwise vector product, and $f$,
$i$, $o$ and $c$ are the forget gate, input gate, output gate and cell state,
respectively. Information is retained when the forget gate $f_{t}$ becomes 1
and eliminated when $f_{t}$ is set to 0.

For optimization purposes, an alternative to LSTMs, the gated recurrent unit
(GRU), can be utilized due to its lower computational demands. The GRU merges
the input gate and forget gate into a single update gate, and the mathematical
representation is given by the following equations:

$\displaystyle r_{t}=\sigma(W_{rh}h_{t-1}+W_{rx}x_{t}+b_{r}),$ (23)
$\displaystyle z_{t}=\sigma(W_{zh}h_{t-1}+W_{zx}x_{t}+b_{z}),$ (24)
$\displaystyle h^{\prime}_{t}=tanh(W_{h^{\prime}h}(r_{t}\odot
h_{t-1})+W_{h^{\prime}x}x_{t}+b_{z}),$ (25) $\displaystyle
h_{t}=(1-z_{t})\odot h_{t-1}+z_{t}\odot h^{\prime}_{t}.$ (26)

Finally, it’s worth noting that while traditional RNNs only consider previous
information, bidirectional RNNs incorporate both past and future information
in their computations:

$\displaystyle h_{t}=\sigma(W_{hx}x_{t}+W_{hh}h_{t-1}+b_{h}),$ (27)
$\displaystyle z_{t}=\sigma(W_{ZX}x_{t}+W_{HX}h_{t+1}+b_{z}),$ (28)
$\displaystyle\hat{y_{t}}=softmax(W_{yh}h_{t}+W_{yz}z_{t}+b_{y}),$ (29)

where $h_{t-1}$ and $h_{t+1}$ indicate hidden cell states at the previous time
step ($t-1$) and the future time step ($t+1$).

##  3 Results

###  3.1 Benchmark Datasets

The UCF-101 dataset, introduced in 2012, is one of the largest annotated video
datasets available, and an expansion of the UCF-50 dataset. It comprises 13320
realistic video clips collected from YouTube and covers 101 categories of
human actions, such as punching, boxing, and walking. The dataset has three
distinct official splits (rather than a pre-divided training set and testing
set), and the final accuracy in our experiments is calculated as the
arithmetic average of the results across all three splits. Our experiments
were conducted using Tensorflow-2.8.2 [50], Keras-2.6.0, and a powerful 4-GPU
system (GeForce® GTX 1080 Ti). We used Hiplot [51] for data visualization.
Figure 3 provides snapshot of samples from each of the action categories.

Figure 3: A snapshot of samples of all actions from UCF-101 dataset [52].

###  3.2 Evaluation Metric

For evaluating our results, we employ the standard classification accuracy
metric, which is defined as follows:

$Accuracy=\frac{\mbox{Number of correct predictions}}{\mbox{Total numbers of
predictions made}}.$ (30)

###  3.3 Implementation

Training our collaborative models for action recognition involves building a
new, dedicated system, as these models require real-time information exchange.
To the best of our knowledge, this is the first such system ever built for
this purpose. To accommodate the large hardware resources required, each model
is trained in a separate environment. After one training epoch, each model
updates its current location, previous location, estimate of the gradient of
the loss function, and other relevant information, which is then broadcast to
neighboring models. To clarify the concept, we provide a diagram of the
collaborative system and provide a brief description in this subsection.

Our system for distributed PSO-ConvNets is designed based on a web client-
server architecture, as depicted in Figure 4. The system consists of two main
components: the client side, which is any computer with a web browser
interface, and the server side, which comprises three essential services:
cloud services, app services, and data services.

The cloud services host the models in virtual machines, while the app services
run the ConvNet RNN or ConvNet Transformer models. The information generated
by each model is managed by the data services and stored in a data storage. In
order to calculate the next positions of particles, each particle must wait
for all other particles to finish the training cycle in order to obtain the
current information.

The system is designed to be operated through a web-based interface, which
facilitates the advanced development process and allows for easy interactions
between users and the system.

Figure 4: Dynamic PSO-ConvNets System Design. The system is divided into two
main components, client and server. The client side is accessed through web
browser interface while the server side comprises of cloud, app, and data
services. The cloud stores virtual machine environments where the models
reside. The app service is where the ConvNet-RNN or ConvNet-Transformer runs,
and the information generated by each model is managed and saved by the data
service. The particles in the system update their positions based on shared
information, including current and previous locations, after completing a
training cycle.

###  3.4 Effectiveness of The Proposed Method

Table 1 presents the results of Dynamic 1 and Dynamic 2 on action recognition
models. The experiment settings are consistent with our previous research for
a fair comparison. As shown in Figure 2, we consider two different ConvNet
architectures, namely DenseNet-201 and ResNet-152, and select eight models
from the Inception [53], EfficientNet [54], DenseNet [55], and ResNet [56]
families. In the baseline action recognition methods (DenseNet-201 RNN,
ResNet-152 RNN, DenseNet-201 Transformer, and ResNet-152 Transformer),
features are first extracted from ConvNets using transfer learning and then
fine-tuned. However, in our proposed method, the models are retrained in an
end-to-end fashion. Pretrained weights from the ImageNet dataset [57] are
utilized to enhance the training speed. Our results show an improvement in
accuracy between $1.58\%$ and $8.72\%$. Notably, the Dynamics 2 for
DenseNet-201 Transformer achieves the best result. We also report the time
taken to run each method. Fine-tuning takes less time, but the technique can
lead to overfitting after a few epochs.

Table 1: Three-fold classification accuracy (%) on the UCF-101 benchmark dataset. The results of Dynamic 1 and Dynamic 2 using DenseNet-201 Transformer and Resnet-152 Transformer models and compared to baseline models, e.g., Dynamic 1 for DenseNet-201 Transformer versus DenseNet-201 Transformer. The N/A is the abbreviation for the phrase not applicable, for instance, the transfer learning is not applicable in the Dynamic 2 for DenseNet-201 Transformer as the method uses the end-to-end training instead. | Time per fold (h)
---|---
Dynamic Method | Model | Accuracy | Improve (%) | Transfer Learning | Fine-tune/Retrain
- | DenseNet-201 Transformer | 0.7741 | N/A | 26 | 0.5
ResNet-152 Transformer | 0.7679 | N/A | 24
- | DenseNet-201 RNN | 0.8195 | N/A | 26 | 0.1
ResNet-152 RNN | 0.8118 | N/A | 24
Dynamic 1 | DenseNet-201 Transformer | 0.8579 | 8.38 | N/A | 5.5
ResNet-152 Transformer | 0.8405 | 7.26 | N/A
DenseNet-201 RNN | 0.8462 | 2.67 | N/A
ResNet-152 RNN | 0.8276 | 1.58 | N/A
Dynamic 2 | DenseNet-201 Transformer | 0.8613 | 8.72 | N/A
ResNet-152 Transformer | 0.8399 | 7.20 | N/A

###  3.5 Comparison with state-of-the-art methods

Table 2: Comparisons of the proposed method and the state-of-the-art methods on the UCF-101 benchmark dataset Method | Accuracy (%)
---|---
Bag of words [52] | 44.50
Transfer Learning and Fusions [58] | 65.40
P-ODN [59] | 78.64
Our model (Dynamics 2 for ConvNet Transformer) | 86.13

The comparison between our method (Dynamic 2 for ConvNet Transformer) and
state-of-the-art (SOTA) approaches is shown in Table 2. The second method
(which is similar to ours) trains the models on a Sports-1M Youtube dataset
and uses the features for UCF-101 recognition. However, the transfer learning
procedure is slightly different as their ConvNet architectures were designed
specifically for action recognition. While it may have been better to use a
pretrained weight for action recognition datasets, such weights are not
readily available as the models differ. Also, training the video dataset with
millions of samples within a reasonable time is a real challenge for most
research centers. Despite these limitations, the use of Transformer and RNN
seem to provide a better understanding of temporal characteristics compared to
fusion methods. The third method, Prototype Open Deep Network (P-ODN),
introduces prototype learning into open set recognition. The method learns the
unknown classes by applying a multi-class triplet thresholding method based on
the distance metric between features and prototypes. Our Dynamic 2 method
outperformed P-ODN by roughly $9\%$. The bag of words method was originally
used as a baseline for the dataset and achieved the lowest recognition
accuracy ($44.5\%$).

###  3.6 Hyperparameter Optimization

In these experiments, we aimed to find the optimal settings for each model.
Table 3 presents the results of the DenseNet-201 Transformer and ResNet-152
Transformer using transfer learning, where we varied the maximum sequence
length, number of frames, number of attention heads, and dense size. The
number of frames represents the amount of frames extracted from the sequence,
calculated by $step=\mbox{(maximum sequence length)/(number of frames)}$. The
results indicate that longer sequences of frames lead to better accuracy, but
having a large number of frames is not necessarily the best strategy; a
balanced approach yields higher accuracy. Furthermore, we discovered that
models performed best with 6 attention heads and a dense size of either 32 or
64 neurons.

Figures 5 and 6 show the results for ConvNet RNN models using transfer
learning. In the experiments, we first evaluated the performance of eight
ConvNets (Inception-v3, ResNet-101, ResNet-152, DenseNet-121, DenseNet-201,
EfficientNet-B0, EfficientNet-B4, and EfficientNet-B7). The two best
performers, DenseNet-121 and ResNet-152 ConvNet architectures, were selected
for further experimentation. The results of varying the number of frames
showed a preference for longer maximum sequence lengths.

Table 3: Three-Fold Classification Accuracy Results (%) on the UCF-101
Benchmark Dataset for DenseNet-201 Transformer and ResNet-152 Transformer with
Transfer Learning Training.

| Accuracy (%)
---|---
Model | Maximum sequence length | Number of frames | Number of Attention Heads | Dense Size | Set 1 | Set 2 | Set 3 | Avg
DenseNet-201 Transformer | 100 | 2 | 1 | 4 | 69.94 | 69.42 | 69.21 | 69.52
128 | 69.57 | 69.68 | 68.51 | 69.25
4 | 64 | 74.36 | 75.39 | 75.43 | 75.06
128 | 76 | 75.15 | 74.81 | 75.32
8 | 128 | 74.57 | 74.75 | 74.65 | 74.66
256 | 74.94 | 74.75 | 75.05 | 74.91
4 | 1 | 4 | 70.92 | 69.47 | 71.67 | 70.69
8 | 70.02 | 69.63 | 70.32 | 69.99
16 | 69.87 | 68.53 | 68.29 | 68.90
32 | 70.37 | 71 | 69.1 | 70.16
64 | 70.08 | 69.52 | 67.99 | 69.20
128 | 69.26 | 69.42 | 70.54 | 69.74
10 | 4 | 64 | 76.71 | 76 | 75.3 | 76.00
128 | 77.43 | 75.33 | 76.46 | 76.41
6 | 8 | 77.08 | 76.57 | 76.95 | 76.87
16 | 77.64 | 76.62 | 76.06 | 76.77
32 | 77.4 | 76.73 | 77.08 | 77.07
64 | 76.87 | 77.16 | 78.19 | 77.41
1024 | 74.94 | 73.57 | 72.48 | 73.66
20 | 6 | 8 | 76 | 77.34 | 77.25 | 76.86
32 | 76.71 | 75.99 | 75.3 | 76.00
64 | 77.16 | 76 | 76.46 | 76.54
128 | 77.4 | 75.87 | 77.03 | 76.77
40 | 2 | 4 | 32 | 74.65 | 74.53 | 74.3 | 74.49
64 | 74.94 | 74.26 | 73.54 | 74.25
128 | 74.46 | 74.29 | 74.46 | 74.40
1024 | 71.03 | 70.73 | 70.45 | 70.74
6 | 16 | 73.38 | 74.48 | 73.97 | 73.94
32 | 74.39 | 73.43 | 73.97 | 73.93
64 | 75.34 | 74.37 | 73.57 | 74.43
128 | 73.96 | 74.21 | 74.03 | 74.07
10 | 6 | 32 | 76.37 | 74.64 | 74.35 | 75.12
64 | 75.73 | 75.55 | 74.81 | 75.36
128 | 76.08 | 76 | 75.08 | 75.72
ResNet-154 Transformer | 100 | 2 | 1 | 4 | 71.08 | 70.65 | 68.59 | 70.11
128 | 70.05 | 71.51 | 68.89 | 70.15
4 | 128 | 75.02 | 76.03 | 74.73 | 75.26
256 | 74.76 | 75.66 | 75.19 | 75.20
6 | 64 | 75.39 | 75.52 | 74.65 | 75.19
128 | 75.57 | 75.6 | 75.78 | 75.65
10 | 6 | 4 | 76.55 | 76.78 | 75.65 | 76.33
8 | 77.08 | 76.35 | 76.14 | 76.52
16 | 76.55 | 76.25 | 76.33 | 76.38
32 | 77.24 | 76.49 | 76.65 | 76.79
64 | 75.84 | 77.53 | 76.54 | 76.64

Figure 5: Hyperparameter Optimization Results for ConvNet RNN Models with
Transfer Learning. The models are numbered as follows: 1.Inception-v3,
2.ResNet-101, 3.ResNet-152, 4.DenseNet-121, 5.DenseNet-201, 6.EfficientNet-B0,
7.EfficientNet-B4, 8.EfficientNet-B7. The abbreviations $acc$, $gn$, and $lr$
stand for accuracy, Gaussian noise, and learning rate, respectively. Figure 6:
Impact of Varying the Number of Frames on the Three-Fold Accuracy of
DenseNet-201 RNN and ResNet-152 RNN Using Transfer Learning on the UCF-101
Benchmark Dataset.

##  4 Discussion

The performance of action recognition methods such as ConvNet Transformer and
ConvNet RNN is largely dependent on various factors, including the number of
attention heads, the number of dense neurons, the number of units in RNN, and
the learning rate, among others. Collaborative learning is an effective
approach to improve the training of neural networks, where multiple models are
trained simultaneously and both their positions and directions, as determined
by the gradients of the loss function, are shared. In our previous research,
we applied dynamics to ConvNets for image classification and in this study, we
extend the concept to hybrid ConvNet Transformer and ConvNet RNN models for
human action recognition in sequences of images. We first aim to identify the
optimal settings that lead to the highest accuracy for the baseline models. As
seen in Table 1, the ConvNet Transformer models did not perform as well as the
ConvNet RNN models with transfer learning, which could be due to the limited
data available for training, as transformers typically require more data than
RNN-based models. However, our proposed method, incorporating dynamics and
end-to-end training, not only outperforms the baseline models, but also
results in the ConvNet Transformer models outperforming their ConvNet RNN
counterparts. This can be attributed to the additional data provided to the
transformer models through data augmentation and additional noise.

##  5 Conclusion

Recognizing human actions in videos is a fascinating problem in the art of
recognition, and while Convolutional Neural Networks provide a powerful method
for image classification, their application to HAR can be complex, as temporal
features play a critical role.

In this study, we present a novel video action recognition framework that
leverages collaborative learning with dynamics. Our approach explores the
hybridization of ConvNet RNN and the recent advanced method Transformer, which
has been adapted from Natural Language Processing for video sequences. The
experiments include the exploration of two dynamics models, Dynamic 1 and
Dynamic 2. The results demonstrate a round improvement of $2\%-9\%$ in
accuracy over baseline methods, such as an $8.72\%$ increase in accuracy for
the DenseNet-201 Transformer using Dynamic 2 and a $7.26\%$ increase in
accuracy for the ResNet-152 Transformer using Dynamic 1. Our approach
outperforms the current state-of-the-art methods, offering significant
improvements in video action recognition.

In summary, our work makes three key contributions: (1) We incorporate Dynamic
1 and Dynamic 2 into a hybrid model that combines ConvNet with two popular
sequence modeling techniques - RNN and Transformer. (2) We extend the
distributed collaborative learning framework to address the task of human
action recognition. (3) We conduct extensive experiments on the challenging
UCF-101 dataset, over a period of 60 days, and compare our approach against
state-of-the-art methods to validate its effectiveness.

## References

  * [1] Sultani, W., Chen, C. & Shah, M.  Real-world anomaly detection in surveillance videos.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 6479–6488 (2018).
  * [2] Li, A. _et al._ Abnormal event detection in surveillance videos based on low-rank and compact coefficient dictionary learning.  _Pattern Recognition_ 108, 107355 (2020).
  * [3] Razali, H., Mordan, T. & Alahi, A.  Pedestrian intention prediction: A convolutional bottom-up multi-task approach.  _Transportation research part C: emerging technologies_ 130, 103259 (2021).
  * [4] Yang, H., Liu, L., Min, W., Yang, X. & Xiong, X.  Driver yawning detection based on subtle facial action recognition.  _IEEE Transactions on Multimedia_ 23, 572–583 (2020).
  * [5] Presti, L. L. & La Cascia, M.  3d skeleton-based human action classification: A survey.  _Pattern Recognition_ 53, 130–147 (2016).
  * [6] Poppe, R.  A survey on vision-based human action recognition.  _Image and vision computing_ 28, 976–990 (2010).
  * [7] Zhu, H., Vial, R. & Lu, S.  Tornado: A spatio-temporal convolutional regression network for video action proposal.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 5813–5821 (2017).
  * [8] Curtis, S., Zafar, B., Gutub, A. & Manocha, D.  Right of way.  _The Visual Computer_ 29, 1277–1292 (2013).
  * [9] Paul, S. N. & Singh, Y. J.  Survey on video analysis of human walking motion.  _International Journal of Signal Processing, Image Processing and Pattern Recognition_ 7, 99–122 (2014).
  * [10] Wang, H., Kläser, A., Schmid, C. & Liu, C.-L.  Action recognition by dense trajectories.  In _CVPR 2011_ , 3169–3176, DOI: 10.1109/CVPR.2011.5995407 (2011).
  * [11] Wang, H. & Schmid, C.  Action recognition with improved trajectories.  In _Proceedings of the IEEE international conference on computer vision_ , 3551–3558 (2013).
  * [12] Gorelick, L., Blank, M., Shechtman, E., Irani, M. & Basri, R.  Actions as space-time shapes.  _IEEE transactions on pattern analysis and machine intelligence_ 29, 2247–2253 (2007).
  * [13] Simonyan, K. & Zisserman, A.  Two-stream convolutional networks for action recognition in videos.  _Advances in neural information processing systems_ 27 (2014).
  * [14] Tran, D., Bourdev, L., Fergus, R., Torresani, L. & Paluri, M.  Learning spatiotemporal features with 3d convolutional networks.  In _Proceedings of the IEEE international conference on computer vision_ , 4489–4497 (2015).
  * [15] Carreira, J. & Zisserman, A.  Quo vadis, action recognition? a new model and the kinetics dataset.  In _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 6299–6308 (2017).
  * [16] Hara, K., Kataoka, H. & Satoh, Y.  Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 6546–6555 (2018).
  * [17] Xie, S., Sun, C., Huang, J., Tu, Z. & Murphy, K.  Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification.  In _Proceedings of the European conference on computer vision (ECCV)_ , 305–321 (2018).
  * [18] Diba, A. _et al._ Temporal 3d convnets: New architecture and transfer learning for video classification.  _arXiv preprint arXiv:1711.08200_ (2017).
  * [19] Varol, G., Laptev, I. & Schmid, C.  Long-term temporal convolutions for action recognition.  _IEEE transactions on pattern analysis and machine intelligence_ 40, 1510–1517 (2017).
  * [20] Donahue, J. _et al._ Long-term recurrent convolutional networks for visual recognition and description.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2625–2634 (2015).
  * [21] Yue-Hei Ng, J. _et al._ Beyond short snippets: Deep networks for video classification.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 4694–4702 (2015).
  * [22] Ullah, A., Ahmad, J., Muhammad, K., Sajjad, M. & Baik, S. W.  Action recognition in video sequences using deep bi-directional lstm with cnn features.  _IEEE access_ 6, 1155–1166 (2017).
  * [23] He, J.-Y., Wu, X., Cheng, Z.-Q., Yuan, Z. & Jiang, Y.-G.  Db-lstm: Densely-connected bi-directional lstm for human action recognition.  _Neurocomputing_ 444, 319–331 (2021).
  * [24] Gowda, S. N., Rohrbach, M. & Sevilla-Lara, L.  Smart frame selection for action recognition.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 1451–1459 (2021).
  * [25] Ge, H., Yan, Z., Yu, W. & Sun, L.  An attention mechanism based convolutional lstm network for video action recognition.  _Multimedia Tools and Applications_ 78, 20533–20556 (2019).
  * [26] Wu, Z., Xiong, C., Ma, C.-Y., Socher, R. & Davis, L. S.  Adaframe: Adaptive frame selection for fast video recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 1278–1287 (2019).
  * [27] Vaswani, A. _et al._ Attention is all you need.  In _Advances in neural information processing systems_ , 5998–6008 (2017).
  * [28] Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G. & Jégou, H.  Going deeper with image transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 32–42 (2021).
  * [29] Dosovitskiy, A. _et al._ An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ (2020).
  * [30] Arnab, A. _et al._ Vivit: A video vision transformer.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 6836–6846 (2021).
  * [31] Liu, Z. _et al._ Video swin transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 3202–3211 (2022).
  * [32] Phong, N. H., Santos, A. & Ribeiro, B.  Pso-convolutional neural networks with heterogeneous learning rate.  _IEEE Access_ 10, 89970–89988, DOI: 10.1109/ACCESS.2022.3201142 (2022).
  * [33] Ijjina, E. P. & Chalavadi, K. M.  Human action recognition using genetic algorithms and convolutional neural networks.  _Pattern recognition_ 59, 199–212 (2016).
  * [34] Real, E. _et al._ Large-scale evolution of image classifiers.  In _International Conference on Machine Learning_ , 2902–2911 (PMLR, 2017).
  * [35] Nayman, N. _et al._ Xnas: Neural architecture search with expert advice.  _Advances in neural information processing systems_ 32 (2019).
  * [36] Noy, A. _et al._ Asap: Architecture search, anneal and prune.  In _International Conference on Artificial Intelligence and Statistics_ , 493–503 (PMLR, 2020).
  * [37] Kennedy, J. & Eberhart, R.  Particle swarm optimization.  In _Proceedings of ICNN’95-international conference on neural networks_ , vol. 4, 1942–1948 (IEEE, 1995).
  * [38] Shi, Y. & Eberhart, R.  A modified particle swarm optimizer.  In _1998 IEEE international conference on evolutionary computation proceedings. IEEE world congress on computational intelligence (Cat. No. 98TH8360)_ , 69–73 (IEEE, 1998).
  * [39] Tu, S. _et al._ Modpso-cnn: an evolutionary convolution neural network with application to visual recognition.  _Soft Computing_ 25, 2165–2176 (2021).
  * [40] Chuang, L.-Y., Tsai, S.-W. & Yang, C.-H.  Improved binary particle swarm optimization using catfish effect for feature selection.  _Expert Systems with Applications_ 38, 12699–12707 (2011).
  * [41] Xue, B., Zhang, M. & Browne, W. N.  Particle swarm optimization for feature selection in classification: A multi-objective approach.  _IEEE transactions on cybernetics_ 43, 1656–1671 (2012).
  * [42] Zhang, R.  Sports action recognition based on particle swarm optimization neural networks.  _Wireless Communications and Mobile Computing_ 2022 (2022).
  * [43] Basak, H. _et al._ A union of deep learning and swarm-based optimization for 3d human action recognition.  _Scientific Reports_ 12, 1–17 (2022).
  * [44] Liu, Z. _et al._ Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 10012–10022 (2021).
  * [45] Zhang, Y. _et al._ Vidtr: Video transformer without convolutions.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 13577–13587 (2021).
  * [46] Ullah, A., Ahmad, J., Muhammad, K., Sajjad, M. & Baik, S. W.  Action recognition in video sequences using deep bi-directional lstm with cnn features.  _IEEE Access_ 6, 1155–1166, DOI: 10.1109/ACCESS.2017.2778011 (2018).
  * [47] Chen, J., Samuel, R. D. J. & Poovendran, P.  Lstm with bio inspired algorithm for action recognition in sports videos.  _Image and Vision Computing_ 112, 104214, DOI: https://doi.org/10.1016/j.imavis.2021.104214 (2021).
  * [48] Maas, A. L., Hannun, A. Y., Ng, A. Y. _et al._ Rectifier nonlinearities improve neural network acoustic models.  In _Proc. icml_ , 3 (Citeseer, 2013).
  * [49] LeCun, Y., Kavukcuoglu, K. & Farabet, C.  Convolutional networks and applications in vision.  In _Proceedings of 2010 IEEE international symposium on circuits and systems_ , 253–256 (IEEE, 2010).
  * [50] Abadi, M. _et al._ Tensorflow: Large-scale machine learning on heterogeneous systems, software available from tensorflow. org (2015).  _URL https://www. tensorflow. org_ (2015).
  * [51] Haziza, D., Rapin, J. & Synnaeve, G.  Hiplot, interactive high-dimensionality plots.  https://github.com/facebookresearch/hiplot (2020).
  * [52] Soomro, K., Zamir, A. R. & Shah, M.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  _arXiv preprint arXiv:1212.0402_ (2012).
  * [53] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. & Wojna, Z.  Rethinking the inception architecture for computer vision.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2818–2826 (2016).
  * [54] Tan, M. & Le, Q.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In _International conference on machine learning_ , 6105–6114 (PMLR, 2019).
  * [55] Huang, G., Liu, Z., Van Der Maaten, L. & Weinberger, K. Q.  Densely connected convolutional networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 4700–4708 (2017).
  * [56] He, K., Zhang, X., Ren, S. & Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778 (2016).
  * [57] Krizhevsky, A., Sutskever, I. & Hinton, G. E.  Imagenet classification with deep convolutional neural networks.  In _Advances in neural information processing systems_ , 1097–1105 (2012).
  * [58] Karpathy, A. _et al._ Large-scale video classification with convolutional neural networks.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 1725–1732 (2014).
  * [59] Shu, Y., Shi, Y., Wang, Y., Huang, T. & Tian, Y.  P-odn: Prototype-based open deep network for open set recognition.  _Scientific reports_ 10, 1–13 (2020).

## Acknowledgements

This research is sponsored by FEDER funds through the programs COMPETE –
“Programa Operacional Factores de Competitividade” and Centro2020 – “Centro
Portugal Regional Operational Programme”, and by national funds through FCT –
“Fundação para a Ciência e a Tecnologia”, under the project UIDB/00326/2020.
The support is gratefully acknowledged.

## Author contributions statement

N.H.P: Conceptualization, Methodology, Coding, Experiments, Validation, Formal
analysis, Investigation, Writing - original draft, Writing - review and
editing; B.R.: Writing - review and editing, Supervision. All authors have
reviewed the manuscript.

## Additional information

Competing interests The authors declare that they have no conflicts of
interest.