11institutetext: Max-Planck-Institute for Biological Cybernetics, 72076
Tübingen, Germany22institutetext: University Hospital Tübingen, 72076
Tübingen, Germany
22email<EMAIL_ADDRESS>
# Pretraining is All You Need: A Multi-Atlas Enhanced Transformer Framework
for Autism Spectrum Disorder Classification

Lucas Mahler 11 Qi Wang 1122 Julius Steiglechner 1122 Florian Birk 11 2 2
Samuel Heczko 11 Klaus Scheffler 11 2 2 Gabriele Lohmann 11 2 2

###### Abstract

Autism spectrum disorder (ASD) is a prevalent psychiatric condition
characterized by atypical cognitive, emotional, and social patterns. Timely
and accurate diagnosis is crucial for effective interventions and improved
outcomes in individuals with ASD. In this study, we propose a novel Multi-
Atlas Enhanced Transformer framework, METAFormer, ASD classification. Our
framework utilizes resting-state functional magnetic resonance imaging data
from the ABIDE I dataset, comprising 406 ASD and 476 typical control (TC)
subjects. METAFormer employs a multi-atlas approach, where flattened
connectivity matrices from the AAL, CC200, and DOS160 atlases serve as input
to the transformer encoder. Notably, we demonstrate that self-supervised
pretraining, involving the reconstruction of masked values from the input,
significantly enhances classification performance without the need for
additional or separate training data. Through stratified cross-validation, we
evaluate the proposed framework and show that it surpasses state-of-the-art
performance on the ABIDE I dataset, with an average accuracy of 83.7% and an
AUC-score of 0.832. The code for our framework is available at
github.com/Lugges991/METAFormer

###### Keywords:

Deep LearningTransformers fMRI Autism Spectrum Disorder Classification

##  1 Introduction

Autism spectrum disorder (ASD) is a widespread psychiatric condition
characterized by atypical cognitive, emotional, and social patterns. With
millions of individuals affected worldwide, the early diagnosis of ASD is a
critical research priority, as it has a significant positive impact on patient
outcomes. The etiology of ASD remains elusive, with intricate interactions
among genetic, biological, psychological, and environmental factors playing a
role. Currently, diagnosing ASD relies heavily on behavioral observations and
anamnestic information, posing challenges and consuming a considerable amount
of time. Skilled clinicians with extensive experience are required for
accurate diagnosis. However, common assessments of ASD have been criticized
for their lack of objectivity and transparency [27]. Given these limitations,
there is an urgent need for a fast, cost-effective, and objective diagnostic
method that can accurately identify ASD leading to more timely interventions
and improved outcomes for affected individuals.

In recent years, magnetic resonance imaging (MRI) has emerged as a powerful
non-invasive tool for gaining insights into brain disorders’ pathophysiology.
Functional MRI (fMRI), a notable advancement in MRI technology, allows for the
investigation of brain function by measuring changes in blood oxygen levels
over time. Functional connectivity (FC) analysis[4] plays a crucial role in
fMRI data analysis, as it examines the statistical dependencies and temporal
correlations among different brain regions. Rather than considering isolated
abnormalities in specific regions, brain disorders often arise from disrupted
communication and abnormal interactions between regions. FC analysis enables
researchers to explore network-level abnormalities associated with various
disorders. This analysis involves partitioning the brain into regions of
interest (ROIs) and quantifying the correlations between their time series
using various mathematical measures.

In recent years machine learning approaches have been widely applied to the
problem of ASD classification using rs-fMRI data. The majority of these
studies use functional connectivities obtained from a predefined atlas as
input to their classifiers. A considerable amount of work used classical
machine learning algorithms such as support vector machines and logistic
regression to classify ASD[11]. However, these methods have limitations as
they are typically applied to small datasets with specific protocols and fixed
scanner parameters, which may not adequately capture the heterogeneity present
in clinical data. 3D Convolutional neural networks[20, 33, 26] have also been
applied to preprocessed fMRI data, [1] have used 2D CNNs on preprocessed fMRI
data. Though, these approaches are as well limited by the fact that they were
only using small homogeneous datasets.
More recent works tried to overcome the homogeneity limitations and have used
deep learning approaches to classify ASD based on connectomes. Multi-layer
perceptrons are suited to the vector based representations of connectomes and
have thus seen some usage in ASD classification[30, 12]. Graph convolutional
models are also well suited and have yielded high accuracies[19, 29]. Other
approaches used 1D CNNs[23], or variants of recurrent neural networks[17, 18],
and also probabilistic neural networks have been proposed[16].
However, ASD classification is not limited to fMRI data and there has been
work using, for example, EEG[5] or also more novel imaging approaches such as
functional near-infrared spectroscopy[13].

The current study aims to improve classification performance of ASD based on
resting-state fMRI (rs-fMRI) data over the entire ABIDE I dataset[22] by
leveraging the representational capabilities of modern transformer
architectures. We thus summarize our main contributions as follows:

  1. 1.

We propose a novel multi-atlas enhanced transformer framework for ASD
classification using resting-state fMRI data: METAFormer

  2. 2.

We demonstrate that self-supervised pretraining leads to significant
improvements in performance without the requirement of additional data.

  3. 3.

We show that our model outperforms state of the art methods on the ABIDE I
dataset.

##  2 Methods

###  2.1 Dataset

Our experiments are conducted on the ABIDE I dataset [22] which is a publicly
available dataset containing structural MRI as well as resting-state
functional MRI data obtained from individuals with Autism Spectrum Disorder
(ASD) and typical controls (TC) from 17 different research sites. The raw
dataset encompasses a total of 1112 subjects, 539 of which are diagnosed with
ASD and 573 are TC. Subjects ages range from 7 to 64 years with a median age
of 14.7 years across groups. The ABIDE I dataset is regarded as one of the
most comprehensive and widely used datasets in the field, offering a
combination of MRI, rs-fMRI, and demographic data.
The ABIDE I dataset exhibits significant heterogeneity and variations that
should be taken into account. It comprises data from diverse research sites
worldwide, leading to variations in scanning protocols, age groups, and other
relevant factors. Consequently, the analysis and interpretation of the ABIDE I
dataset pose challenges due this inherent heterogeneity.

####  2.1.1 Preprocessing Pipeline.

We utilize the ABIDE I dataset provided by the Preprocessed Connectomes
Project (PCP) [6] for our analysis. The PCP provides data for ABIDE I using
different preprocessing strategies. In this work we use the preprocessed data
from the DPARSF pipeline [31]. The DPARSF pipeline is based on SPM8 and
includes the following steps: The first 4 volumes of each fMRI time series are
discarded to allow for magnetization stabilization. Slice timing correction is
performed to correct for differences in acquisition time between slices. The
fMRI time series are then realigned to the first volume to correct for head
motion. Intensity normalization is not performed. To clean confounding
variation due to physiological noise, 24-parameter head motion, mean white
matter and CSF signals are regressed out. Motion realignment parameters are
also regressed out as well as linear and quadratic trends in low-frequency
drifts. Bandpass filtering was performed after regressing nuisance signals to
remove high-frequency noise and low-frequency drifts. Finally, functional to
anatomical registration is performed using rigid body transformation and
anatomical to standard space registration is performed using DARTEL[2].

####  2.1.2 Functional Connectivity.

As the dimensionality of the preprocessed data is very high, we perform
dimensionality reduction by dividing the brain into a set of parcels or
regions with similar properties according to a brain atlas. In this work we
process our data using three different atlases. The first atlas is the
Automated Anatomical Labeling (AAL) atlas [25]. This atlas, which is widely
used in the literature, divides the brain into 116 regions of interest (ROIs)
based on anatomical landmarks and was fractionated to functional resolution of
$3mm^{3}$ using nearest-neighbor interpolation. The second atlas is the
Craddock 200 (CC200) atlas [7]. It divides the brain into 200 ROIs based on
functional connectivity and was fractionated to functional resolution of
$3mm^{3}$ using nearest-neighbor interpolation. The third atlas we considered
is the Dosenbach 160 (DOS160) atlas [10] which contains uniform spheres placed
at coordinates obtained from meta-analyses of task-related fMRI studies.
After obtaining the ROI time-series from the atlases, we compute the
functional connectivity using the Pearson Correlation Coefficient between each
pair of ROIs. The upper triangular part of the correlation matrix as well as
the diagonal are then dropped and the lower triangular part is vectorized to
obtain a feature vector of length $k(k-1)/2$, where $k$ is the number of ROIs,
which then serves as input to our models.

###  2.2 Model Architecture

####  2.2.1 METAFormer: Multi-Atlas Transformer.

Here, we propose METAFormer, at the core of which lies the transformer encoder
architecture, originally proposed by Vaswani et al.[28] for natural language
processing (NLP) tasks. However, as our main goal is to perform classification
and not generation we do not use the decoder part of the transformer
architecture. In order to accommodate input from multiple different atlases,
we employ an ensemble of three separate transformers, with each transformer
corresponding to a specific atlas. As depicted in Figure 1, the input to each
transformer is a batch of flattened functional connectivity matrices. First,
the input to each transformer undergoes embedding into a latent space using a
linear layer with a dimensionality of $d_{model}=256$. The output of the
embedding is then multiplied by $\sqrt{d_{model}}$ to scale the input
features. This scaling operation aids in balancing the impact of the input
features with the attention mechanism. Since we are not dealing with
sequential data, positional encodings are not utilized.
The embedded input is subsequently passed through a BERT-style encoder[9],
which consists of $N=2$ identical layers with $d_{ff}=128$ feed forward units,
and $h=4$ attention heads. To maintain stability during training, each encoder
layer is normalized using layer normalization [3], and GELU [15] is used as
the activation function. Following the final encoder layer, the output passes
through a dropout layer. Then, a linear layer with $d_{model}$ hidden units
and two output units corresponding to the two classes is applied to obtain the
final output. The outputs of the three separate transformers are averaged, and
this averaged representation is passed through a softmax layer to derive the
final class probabilities.

To train our Multi-Atlas Transformer model, we follow a series of steps.
Firstly, we initialize the model weights using the initialization strategy
proposed by He [14], while setting the biases to 0. To optimize the model, we
employ the AdamW optimizer [21] and minimize the binary cross entropy between
predictions and labels. Our training process consists of 750 epochs, utilizing
a batch size of 256. To prevent overfitting, we implement early stopping with
a patience of 40 epochs. In order to ensure robustness of our model, we apply
data augmentation. Specifically, we randomly introduce noise to each flattened
connectome vector with an augmentation probability of 0.3. The noise standard
deviation is set to 0.01. We conduct hyperparameter tuning using grid search.
We optimize hyperparameters related to the optimizer, such as learning rate
and weight decay. We also consider the dropout rate during this process.

Figure 1: Architecture of METAFormer. Our model consists of three separate
transformers, each corresponding to a specific atlas. The input to each
transformer is a batch of flattened functional connectivity matrices with the
diagonal and the upper triangular part of the matrix removed. The output of
the transformers is averaged and passed through a softmax layer to derive the
final class probabilities.

###  2.3 Self-Supervised Pretraining

As popularized by [24], the utilization of self-supervised generative
pretraining followed by task-specific fine-tuning has demonstrated improved
performance in transformer architectures. Building upon this approach, we
propose a self-supervised pretraining task for our model. Our approach
involves imputing missing elements in the functional connectivity matrices,
drawing inspiration from the work introduced by [32]. To simulate missing
data, we randomly set 10% of the features in each connectome to 0 and train
the model to predict the missing values. The corresponding configuration is
illustrated in Figure 2. To achieve this, we begin by randomly sampling a
binary noise mask $M\in\\{0,1\\}^{n_{i}}$ for each training sample, where
$n_{i}$ denotes the number of features in the $i$-th connectome. Subsequently,
the original input $X$ is masked using element-wise multiplication with the
noise mask: $X_{masked}=X\odot M$.
To estimate the corrupted input, we introduce a linear layer with $n_{i}$
output neurons on top of the encoder stack, which predicts $\hat{x}_{i}$. We
calculate a multi atlas masked mean squared error (MAMSE) loss
$\mathcal{L}_{multi}$ between the predicted and the original input:

$\mathcal{L}_{multi}=\frac{1}{3}\sum_{i=1}^{3}\frac{1}{n_{i}}\sum_{j\in
M}^{n_{i}}||x_{i,j}-\hat{x}_{i,j}||^{2}$ (1)

where $x_{i,j}$ is the original value of the $j$-th masked input from the
$i$-th atlas and $\hat{x}_{i,j}$ is the predicted value for the masked input
at position $j$ in the $i$-th atlas.

Figure 2: Self-supervised pretraining on the imputation task of the METAFormer
architecture. The inputs to the model are masked connectomes, where 10% of the
features are randomly set to 0 (exemplified as black squares). The model is
trained to predict the missing values implying that the output of the model
has the same shape as the input.

##  3 Experiments

###  3.1 Experimental Setup

To evaluate the classification performance of our models in a robust manner,
we employed 10-fold stratified cross-validation. For each fold, the model is
trained on the remaining nine training folds and evaluated on the held-out
test fold. Further, we set aside 30% of each training fold as validation sets
which are then used for hyperparameter tuning and early stopping.

In order to assess the impact of self-supervised pretraining, we compare the
performance of our model with and without pretraining. To achieve that, we
first pretrain the model using the imputation task on the training folds and
subsequently fine-tune the model on the training folds using the
classification task after which we evaluate on the held-out test fold.

In order to verify the efficacy of using multiple atlases as input we compared
the performance of our METAFormer model with the performance of single atlas
transformer (SAT) models. For that, we trained three separate transformer
models using only one atlas as input. The SAT models are trained using the
same architecture as well as training procedure as the METAFormer model. We
also evaluated the performance of the SAT models with and without self-
supervised pretraining in order to asses its impact on the performance of the
model. To make results comparable, we use the same training and validation
folds for all model configurations under investigation.

Evaluation Metrics. By using cross-validation, we obtained 10 different sets
of performance scores per configuration. These scores were then averaged and
the standard deviation of each score was obtained, providing reliable
estimates of the model’s performance on unseen data. The classification
results were reported in terms of accuracy, precision, recall (sensitivity),
F1-score and AUC-score, which are commonly used metrics for evaluating
classification models.

###  3.2 ASD Classification Results

Table 1 shows the superior performance of our pretrained METAFormer model
compared to previously published ASD classifiers trained on atlas-based
connectomes. Importantly, our model achieves higher accuracy even when
compared to approaches with similar test set sizes that did not employ cross-
validation.
To further validate the effectiveness of our proposed Multi-Atlas Transformer
model for Autism Spectrum Disorder classification, we compare METAFormer
against single atlas transformers. The results, as presented in Table 2,
demonstrate the superiority of METAFormer over all single atlas models in
terms of accuracy, precision, recall, F1-score, and AUC-score. Moreover, the
multi-atlas model exhibits comparable or lower standard deviations in
performance metrics compared to the single atlas models. This indicates higher
robustness and stability of our multi-atlas METAFormer architecture,
attributed to the joint training of the three transformer encoders.

Study | #ASD | #TC | Model | CV | Acc.
---|---|---|---|---|---
MAGE[29] | 419 | 513 | Graph CNN | 10-fold | 75.9%
AIMAFE[30] | 419 | 513 | MLP | 10-fold | 74.5%
1DCNN-GRU[23] | – | – | 1D CNN | – | 78.0%
MISODNN[12] | 506 | 532 | MLP | 10-fold | 78.1%
3D CNN[8] | 539 | 573 | 3D CNN | 5-fold | 74.53%
CNNGCN[17] | 403 | 468 | CNN/GRU | – | 72.48%
SSRN[18] | 505 | 530 | LSTM | – | 81.1%
Ours | 408 | 476 | Transformer | 10-fold | 83.7%
Table 1: Overview of state-of-the-art ASD classification methods that use
large, heterogenous samples from ABIDE I. Note that our model achieves the
highest accuracy while still using 10-fold cross-validation.

Impact of Pretraining. We also evaluated the effect of self-supervised
pretraining on the classification performance of our models. As Table 2 shows
pretraining significantly improves the performance of all models in terms of
accuracy, precision, recall, F1-score and AUC-score. Furthermore, for our
proposed METAFormer architecture pretraining improves the performance by a
large margin.

Variant | Acc. | Prec. | Rec. | F1 | AUC
---|---|---|---|---|---
METAFormer PT |  0.837 $\pm$0.030 |  0.819$\pm$0.045 | 0.901 $\pm$0.044 |  0.856 $\pm$0.023 |  0.832 $\pm$0.032
METAFormer | 0.628 $\pm$0.041 | 0.648$\pm$0.040 | 0.688 $\pm$0.091 | 0.663 $\pm$0.047 | 0.623 $\pm$0.041
SAT (AAL) | 0.593 $\pm$0.040 | 0.585$\pm$0.042 | 0.888 $\pm$0.091 | 0.701 $\pm$0.024 | 0.568 $\pm$0.047
SAT (CC200) | 0.586 $\pm$0.037 | 0.577$\pm$0.027 | 0.888 $\pm$0.057 | 0.698 $\pm$0.019 | 0.560 $\pm$0.044
SAT (DOS160) | 0.570 $\pm$0.055 | 0.571$\pm$0.038 | 0.816 $\pm$0.101 | 0.670 $\pm$0.051 | 0.550 $\pm$0.056
SAT (AAL) PT | 0.601 $\pm$0.069 | 0.587$\pm$0.055 | 0.939 $\pm$0.059 | 0.719 $\pm$0.033 | 0.573 $\pm$0.077
SAT (CC200) PT | 0.632 $\pm$0.071 | 0.622$\pm$0.074 | 0.891 $\pm$0.102 | 0.724 $\pm$0.035 | 0.611 $\pm$0.082
SAT (DOS160) PT | 0.683 $\pm$0.094 | 0.652$\pm$0.091 |  0.964 $\pm$0.057 | 0.771 $\pm$0.047 | 0.660 $\pm$0.106
Table 2: Classification results for the different model configurations.
Reported values are the mean $\pm$ standard deviation over 10 folds. The best
results are in bold. SAT=Single Atlas Transformer, PT=Pretrained, atlases are
in braces. Note that pretraining significantly improves performance across
metrics and atlases. Using our multi-atlas METAFormer in combination with
pretraining yields impressive performance increases.

##  4 Conclusion

In this paper, we propose METAFormer, a novel multi-atlas enhanced pretrained
transformer architecture for ASD classification. We utilize self-supervised
pretraining on the imputation task on the same dataset to prime the model for
the downstream task. We conducted extensive experiments to demonstrate the
effectiveness of our approach by comparing it to several baselines that use
single-atlas and multi-atlas approaches with and without pretraing. Our
results show that our model performs better than state-of-the-art methods and
that pretraining is highly beneficial for the downstream task.

## References

  * [1] Ahammed, M.S., Niu, S., Ahmed, M.R., Dong, J., Gao, X., Chen, Y.: DarkASDNet: Classification of ASD on functional MRI using deep neural network. Frontiers in Neuroinformatics 15 (Jun 2021)
  * [2] Ashburner, J.: A fast diffeomorphic image registration algorithm. NeuroImage 38(1), 95–113 (2007)
  * [3] Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization (2016)
  * [4] Biswal, B., Yetkin, F.Z., Haughton, V.M., Hyde, J.S.: Functional connectivity in the motor cortex of resting human brain using echo-planar mri. Magnetic Resonance in Medicine 34(4), 537–541 (Oct 1995)
  * [5] Brihadiswaran, G., Haputhanthri, D., Gunathilaka, S., Meedeniya, D., Jayarathna, S.: EEG-based processing and classification methodologies for autism spectrum disorder: A review. Journal of Computer Science 15(8), 1161–1183 (Aug 2019)
  * [6] Craddock, C., Benhajali, Y., Chu, C., Chouinard, F., Evans, A., Jakab, A., Khundrakpam, B.S., Lewis, J.D., Li, Q., Milham, M., et al.: The neuro bureau preprocessing initiative: open sharing of preprocessed neuroimaging data and derivatives. Frontiers in Neuroinformatics 7, 27 (2013)
  * [7] Craddock, R.C., James, G., Holtzheimer III, P.E., Hu, X.P., Mayberg, H.S.: A whole brain fmri atlas generated via spatially constrained spectral clustering. Human Brain Mapping 33(8), 1914–1928 (2012)
  * [8] Deng, J., Hasan, M.R., Mahmud, M., Hasan, M.M., Ahmed, K.A., Hossain, M.Z.: Diagnosing autism spectrum disorder using ensemble 3d-CNN: A preliminary study. In: 2022 IEEE International Conference on Image Processing (ICIP). IEEE (Oct 2022)
  * [9] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding (2019)
  * [10] Dosenbach, N.U.F., Nardos, B., Cohen, A.L., Fair, D.A., Power, J.D., Church, J.A., Nelson, S.M., Wig, G.S., Vogel, A.C., Lessov-Schlaggar, C.N., Barnes, K.A., Dubis, J.W., Feczko, E., Coalson, R.S., Pruett, J.R., Barch, D.M., Petersen, S.E., Schlaggar, B.L.: Prediction of individual brain maturity using fmri. Science 329(5997), 1358–1361 (2010)
  * [11] Du, Y., Fu, Z., Calhoun, V.D.: Classification and prediction of brain disorders using functional connectivity: Promising but challenging. Frontiers in Neuroscience 12 (Aug 2018)
  * [12] Epalle, T.M., Song, Y., Liu, Z., Lu, H.: Multi-atlas classification of autism spectrum disorder with hinge loss trained deep architectures: Abide i results. Applied Soft Computing 107, 107375 (2021)
  * [13] Gerloff, C., Konrad, K., Kruppa, J., Schulte-Rüther, M., Reindl, V.: Autism spectrum disorder classification based on interpersonal neural synchrony: Can classification be improved by dyadic neural biomarkers using unsupervised graph representation learning? In: Lecture Notes in Computer Science, pp. 147–157. Springer Nature Switzerland (2022)
  * [14] He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification (2015)
  * [15] Hendrycks, D., Gimpel, K.: Gaussian error linear units (gelus) (2023)
  * [16] Iidaka, T.: Resting state functional magnetic resonance imaging and neural network classified autism and control. Cortex 63, 55–67 (Feb 2015)
  * [17] Jiang, W., Liu, S., Zhang, H., Sun, X., Wang, S.H., Zhao, J., Yan, J.: CNNG: A convolutional neural networks with gated recurrent units for autism spectrum disorder classification. Frontiers in Aging Neuroscience 14 (Jul 2022)
  * [18] Kang, L., Gong, Z., Huang, J., Xu, J.: Autism spectrum disorder recognition based on machine learning with roi time-series. NeuroImage: Clinical (2023)
  * [19] Lamani, M.R., Benadit, P.J., Vaithinathan, K.: Multi-atlas graph convolutional networks and convolutional recurrent neural networks-based ensemble learning for classification of autism spectrum disorders. SN Computer Science 4(3) (Feb 2023)
  * [20] Li, X., Dvornek, N.C., Zhuang, J., Ventola, P., Duncan, J.S.: Brain biomarker interpretation in ASD using deep learning and fMRI. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2018, pp. 206–214. Springer International Publishing (2018)
  * [21] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization (2019)
  * [22] Martino, A.D., Yan, C.G., et al.: The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism. Molecular Psychiatry 19(6), 659–667 (Jun 2013)
  * [23] Qayyum, A., Ahamed Khan, M.K.A., Benzinou, A., Mazher, M., Ramasamy, M., Aramugam, K., Deisy, C., Sridevi, S., Suresh, M.: An efficient 1dcnn–lstm deep learning model for assessment and classification of fmri-based autism spectrum disorder. In: Raj, J.S., Kamel, K., Lafata, P. (eds.) Innovative Data Communication Technologies and Application. pp. 1039–1048. Springer Nature Singapore, Singapore (2022)
  * [24] Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al.: Improving language understanding by generative pre-training (2018)
  * [25] Rolls, E.T., Huang, C.C., Lin, C.P., Feng, J., Joliot, M.: Automated anatomical labelling atlas 3. NeuroImage 206, 116189 (2020)
  * [26] Thomas, R.M., Gallo, S., Cerliani, L., Zhutovsky, P., El-Gazzar, A., van Wingen, G.: Classifying autism spectrum disorder using the temporal statistics of resting-state functional MRI data with 3d convolutional neural networks. Frontiers in Psychiatry 11 (May 2020)
  * [27] Timimi, S., Milton, D., Bovell, V., Kapp, S., Russell, G.: Deconstructing diagnosis: Four commentaries on a diagnostic tool to assess individuals for autism spectrum disorders. Autonomy (Birmingham, England) 1(6) (2019)
  * [28] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [29] Wang, Y., Liu, J., Xiang, Y., Wang, J., Chen, Q., Chong, J.: MAGE: Automatic diagnosis of autism spectrum disorders using multi-atlas graph convolutional networks and ensemble learning. Neurocomputing 469, 346–353 (Jan 2022)
  * [30] Wang, Y., Wang, J., Wu, F.X., Hayrat, R., Liu, J.: AIMAFE: Autism spectrum disorder identification with multi-atlas deep feature representation and ensemble learning. Journal of Neuroscience Methods 343, 108840 (Sep 2020)
  * [31] Yan, C., Zang, Y.: Dparsf: a matlab toolbox for" pipeline" data analysis of resting-state fmri. Frontiers in systems neuroscience p. 13 (2010)
  * [32] Zerveas, G., Jayaraman, S., Patel, D., Bhamidipaty, A., Eickhoff, C.: A transformer-based framework for multivariate time series representation learning. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. ACM (Aug 2021)
  * [33] Zhao, Y., Ge, F., Zhang, S., Liu, T.: 3d deep convolutional neural network revealed the value of brain network overlap in differentiating autism spectrum disorder from healthy controls. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2018, pp. 172–180. Springer International Publishing (2018)


11institutetext:  1 Tata Consultancy Services Innovation Labs, Bangalore
-560066, India,
11email: {sridhar.nithya, ns.abhinay, c.bodduluri, s.shobhit,
<EMAIL_ADDRESS>
2Indian Institute of Science, Bangalore -560012, India
11email<EMAIL_ADDRESS>
# Model Based Control of Commercial-Off-The Shelf (COTS) Unmanned Rotorcraft
for Brick Wall Construction

Nithya Sridhar 11 N. Sai Abhinay 11 B. Chaithanya Krishna 11 Shubhankar
Shobhit 11 Kaushik Das 11 Debasish Ghose 22

###### Abstract

This work proposes a systematic framework for modelling and controller design
of a Commercial-Off-The Shelf (COTS) unmanned rotorcraft using control theory
and principles, for brick wall construction. With point to point navigation as
the primary application, command velocities in the three axes of the Unmanned
Aerial Vehicle (UAV) are considered as inputs of the system while its actual
velocities are system outputs. Using the sine and step response data acquired
from a Hardware-in-Loop (HiL) test simulator, the considered system was
modelled in individual axes with the help of the proposed framework. This
model was employed for controller design where a sliding mode controller was
chosen to satisfy certain requirements of the application like robustness,
flexibility and accuracy. The model was validated using step response data and
produced a deviation of only 9%. Finally, the controller results from field
test showed fine control up to $\pm 8$ cms accuracy. Sliding Mode Control
(SMC) was also compared with a linear controller derived from iterative
experimentations and seen to perform better than the latter in terms of
accuracy, and robustness to parametric variations and wind disturbances.

###### keywords:

Unmanned Aerial Vehicles (UAV), Navigation, Modelling, Sliding Mode Control
(SMC), Hardware-in-Loop (HiL)

##  1 Introduction

The new age of unmanned systems has been witnessing rapid expansion in the
Unmanned Aerial Vehicles’ (UAVs) application horizon besides its current use
in surveillance, agriculture, defence, material delivery, etc. With a further
futuristic vision, Mohamed Bin Zayed International Robotics Challenge (MBZIRC
2020) had framed three challenges for the participating teams. One of them
consisted of a task where the UAVs are to be employed to build brick walls
using special bricks specifically made for the competition. The sequence of
tasks involved were multi-fold 1) Detection of brick piles and construction
zone 2) Picking of different coloured bricks of different lengths and weight,
with magnetic aids 3) Placing of different coloured bricks in the construction
zone as per the given pattern.

This work has made use of unmanned rotorcraft as the UAV for the given
competetion. It is important to note that UAV navigation was an integral part
in all the mentioned tasks. This paper addresses the problem of point to point
navigation for brick wall construction and attempts to fuse the benefits of
two contrasting approaches. One of which involved a heuristic experiment based
methodology and the other was analytical. This work considers the practical
constraints and challenges involved in the task, and also uses control theory
extensively in the framework. The results of the proposed framework are
compared against that of its heuristic counterpart with the help of
experimental results.

The paper begins with explaining the tasks in the competition and the
challenges involved in it. Further, section 2 discusses the importance of UAV
modelling in controller design, along with some challenges in COTS rotorcraft
modelling. Section 3 discusses two contrasting control approaches with its
features. Further, section 4 and 5 present the modelling framework and
controller design process in detail followed by field results in section 6.

###  1.1 Challenges in Brick Wall Construction

Brick wall construction involved picking of bricks from a brick pile zone
where different coloured bricks were available in a particular order similar
to that shown in Figures 1 (a) & (b). The picking action involved brick
detection using vision feedback, accurate position control and gripper
actuation for attachment. After lifting the brick in air, the UAV would
navigate to the given construction zone similar to that shown in Figures 1 (c)
& (d). This involved fast but reliable navigation of the UAV without detaching
the brick on the way. Further, the placing action involved placing area
detection in the construction zone using vision feedback, fine position
control and gripper detachment action. Also, the width of the bricks was 20
cms making finer UAV control a necessity. Hence, specific challenges involved
with this application were:

  1. 1.

During picking and placing actions, accurate and finer control was a challenge
given the small brick width of 20 cms. With this in mind and wide gripper
design, the controller variation was expected to be not more than $\pm 8$ cms.

  2. 2.

There was a maximum mass variation of 2 kg for the UAV while completing one
set of brick picking and placing operation. This would lead to system
parametric variations from control theory’s viewpoint and had to be handled.

  3. 3.

Since the tasks were carried out in an open arena, wind disturbance was a huge
concern.

  4. 4.

During the navigation from brick pile zone to construction site, the UAV
controller should be reliable enough not to drop the brick on the way.
However, in the case where it goes to pick the brick from the construction
site, it has to be faster and aggressive in order to save time.

Hence, for this particular application of point to point navigation, a high
fidelity robust controller was required which could also provide flexibility
to adapt to different task requirements and finer control to achieve the
required accuracy.

Figure 1: Field test with SMC (a) Brick picking attempted (b) Brick is lifted
in air (c) Brick placing attempted (d) Brick placed in the channel

##  2 Commercial UAV Modelling

Modelling of UAVs with multirotor configuration has been addressed in
literature using physics-based models, empirical models, data-driven models,
etc. For the physics-based models, identification of system parameters may
require certain amount of testing and test facilities, which might make the
identification process tedious [Hoffer et al(2014)Hoffer, Coopmans, Jensen,
and Chen]. Other estimation techniques like least square estimation
[Gremillion and Humbert(2010)], Kalman filtering techniques [Kallapur et
al(2007)Kallapur, Ali, and Anavatti] [Kallapur et al(2008)Kallapur, Samal,
Puttige, Anavatti, and Garratt] require prior information of the modelling
structure. Moreover, these modelling frameworks do not consists of an unknown
in-built stabilizer that is present in commercial UAVs.

Figure 2 shows the block diagram of the considered UAV system with the in-
built stabilizer. This stabilizer acts as an inner loop controller for the
under-actuated UAV system to maintain system stability. In commercial UAVs,
information about the stabilizer architecture is not available. Hence, it is
imperative to adopt a more pragmatic approach while attempting to model
commercial UAVs with unknown stabilizer details. This can be carried out using
experimental data and empirical methods. It is also because of this lack of
information that UAV application developers generally choose to go for
heuristic approach for controller design. This may consist of a linear
controller with one or more of Proportional, Derivative and/or Integral
actions. This approach may not yield an optimal performance.

Figure 2: System block diagram

However, if the system dynamics are captured, controller design based on this
would facilitate finer and accurate control within the drone’s actuation
limits. The proposed framework consists of this analytical-based approach
considering practical constraints, which is believed to provide efficient
control.

##  3 Position Control Design: Heuristic Approach vs Analytical Approach

As mentioned in the previous section, commercially available UAVs consists of
an in-built stabilizer, whose model is unknown to the user. It is also well
known that model-based controller have good accuracy and facilitate superior
control. However, with the missing piece of information, designing a model-
based controller becomes a challenge. Hence, heuristic approaches are largely
prevalent among the UAV community, which involves controller design from
experiments. By and large, a linear controller is preferred with few or all of
the following control actions: Proportional, Integral and Derivative. The
choice of control actions and controller parameter values are obtained from
iterative experimentations. This approach may address the control problem
partially and may not satisfy performance criteria all the time. With its
conservative approach, the controller performance may not be improved beyond a
certain level. Some of the features of this approach are:

  1. 1.

Time required for controller implementation is minimal

  2. 2.

Low complexity in controller design

  3. 3.

Does not require the model of in-built stabilizer

  4. 4.

Produces sub-optimal performance.

  5. 5.

Only target and actual values of the control variable are required for
implementation

On the other hand, analytical approaches [Najm and Ibraheem(2019)] [Kamel et
al(2017a)Kamel, Yasmina, Laredj, Benaoumeur, and Zoubir] require the system
model and its parameters for controller design. Consideration of system
dynamics in the control design process improves the efficacy of the
controller, provides flexibility to satisfy the desired performance criteria.
Control theory is genarally used in the design and considerably less
experimentation is required for design. With good model accuracy, close
correlation between theoretical and experimental responses can be obtained.
Some features of this approach are:

  1. 1.

Time required for controller implementation depends on the type of controller

  2. 2.

Simplicity of controller design depends on the type of controller

  3. 3.

Requires the model of in-built stabilizer

  4. 4.

Capable of producing superior performance

  5. 5.

More signal variables may be required other than target and actual control
variables

A judicious mix of both these approaches would aid the controller development
stage and is believed to provide acceptable performance.

##  4 COTS Rotorcraft Modelling Framework

With the well known advantages of model-based controller, it is imperative to
obtain a mathematical model of the commercial rotorcraft. Literature has
discussed numerous techniques to model UAV, however the presence of an in-
built stabilizer and the absence of its model information is a huge challenge.
This was addressed by adopting certain features of a framework proposed for
vehicle brake system in literature [Sridhar et al(2017)Sridhar, Subramaniyam,
Subramanian, Vivekanandan, and Sivaram]. The stabilizer and the UAV were
considered as one single unit for modelling. Since the application was point
to point navigation that involved position control, velocity in each axis was
taken as the control command variable. Also with the Inertial Measurement Unit
(IMU), the actual velocity signals of the drone in all the three axes of
translation was available for external use. Hence, the system for modelling
considered command velocities as input and actual velocities as output for
individual axes.

The Hardware-in-Loop simulator platform, provided by the manufacturer was used
to acquire data for modelling. Firstly, sine responses were obtained till 15
rad/s, which was used to check for linearity and time in-variance properties
(LTI). Later, the magnitude plots in the frequency domain were used to obtain
the model structure and its parameters. Further, experimental step response
data was compared with the simulated response of the fitted model to check its
validity.

###  4.1 Frequency Domain Analysis

The sine response data obtained using HiL simulator was analysed to check if
the system could be approximated as an LTI system. The peak frequency
components of the actual velocities were plotted against that of the input
waveform as shown in Fig. 3. It was observed that the output frequency of the
peak component was very close to input frequency value with a slight average
deviation in the order of $10^{-2}$. This ascertained that the system can be
approximated as LTI. It was also observed that the responses of X and Y axes
were very close. Hence, it was decided to obtain a single model for both X and
Y axes.

Figure 3: Spectral analysis

Frequency domain analysis was carried out using sine sweep data obtained for
the frequency range of [0.4, 15] rad/s for different amplitude values ranging
from [0.5, 5] m/s for X and Y axes, and that of [0,3] m/s for Z axis.
Magnitude plots were obtained using the acquired data, of which one sample for
each axis is shown in Fig. 4. It was seen that X and Y axes characteristics
were in close agreement supporting the previous inference from spectral
analysis.

Figure 4: Magnitude plot

With nearly zero slope in the lower asymptote region, it was clear that the
system did not have any zeros in the transfer function. In the high frequency
asymptote region, the slopes of X, Y and Z axes were approximately -20
dB/decade, indicative of system’s relative order to be one. Since, the system
has no zeros, with relative order one, it has a pole and takes the form of

$G(s)=\frac{K}{1+\tau s},$ (1)

in the Laplace domain, where $K$ is the steady state gain and $\tau$ is the
system time constant. Steady state gain $K$ was obtained from the low
frequency asymptote region using,

$K=10^{\frac{M(\omega_{l})}{20}},$ (2)

where $M(\omega_{l})$ is the magnitude of the system at the lowest frequency
considered ($\omega_{l}=0.4$ rad/s). With the family of curves obtained for
the three axes systems, the range of time constant was obtained using,

$\tau=\frac{1}{2\pi f_{c}},$ (3)

where $f_{c}$ is the experimental cut-off frequency. The ranges of time
constant for X and Z axes were [0.5, 0.9] s and [0.2 0.4] s respectively.
Using the time constant ranges, model structure and sine response data, the
time constant $\tau$ was estimated using simple least square estimation
technique [Gremillion and Humbert(2010)].

The transfer functions for X, Y, and Z axes are,

$G_{x}(s)=G_{y}(s)=\frac{1.16}{1+0.75s},$ (4)

and,

$G_{z}(s)=\frac{0.98}{1+0.30s},$ (5)

respectively.

###  4.2 Time Domain Analysis

Step response data was acquired for amplitudes ranging from 1 m/s to 5 m/s in
steps of 1 m/s. The responses were captured for each axis individually and was
compared against the simulated step response of the models given in Equations
4, LABEL:eq:sys_y and 5. The deviation between these responses were quatified
using Mean Absolute Percentage Deviation (MAPD) defined as,

$MAPD=\left[\frac{1}{N}{\sum\limits_{k=0}^{N}\Bigg{|}{\frac{g_{exp}(k)-g_{sim}(k)}{g_{exp}(k)}}\Bigg{|}}\right]100,$
(6)

where $g_{exp}$ is the experimental data point, $g_{sim}$ is the simulated
data point at time interval $k$ with $N$ number of data points.

Figure 5: Step response- X axis Figure 6: Step response- Z axis

Figures 5 and 6 show the comparison of simulated and experimental step
response. The experimental response shows a slight overshoot which is not
captured in the model, thus producing a maximum MAPD of 16% for X axis and 12%
for Z axis. However, the average MAPD was approximately only 9% and the
estimated model was used for model-based controller design with this small
deviation.

##  5 Sliding Mode Controller for Point to Point Navigation

Model-based controller development was attempted with the use of the estimated
models. Many control techniques such as sliding mode control, model predictive
control [Kamel et al(2017b)Kamel, Stastny, Alexis, and Siegwart], fuzzy logic
control [Lv et al(2018)Lv, Shen, Xia, Ma, Yu, Shen, and Zhou], etc. are
discussed in literature. However, a recent study [Nasr et al(2018)Nasr,
Ashraf, Hussein, Salem, Elias, Shehata, and Morgan] showed that sliding mode
control technique was the best when compared with other popular techniques for
trajectory tracking. Also, from competition viewpoint, the following factors
influenced controller’s choice:

  1. 1.

With a maximum MAPD of 16% in the modelling process, it was imperative to
choose an inherently robust controller that can handle unmodelled dynamics of
the system.

  2. 2.

Since the mass of the drone was expected to vary during the picking and
placing tasks of brick, robustness was required in terms of system parametric
variations.

  3. 3.

Given the variable nature of wind, robustness was a much required property.

Considering all these factors, it was decided to implement sliding mode
control, which was known to be robust against parametric variations and
external disturbances. Figure 7 shows the control system design with sliding
mode controller for position control.

Figure 7: System block diagram

Equation 1 can be written in state space form as,

$\dot{y(t)}=Ay(t)+Bu(t)$ (7)

where $y(t)$ is output velocity, $u(t)$ is the command velocity and,
$A=\frac{-1}{\tau}$, $B=\frac{K}{\tau}$ are system parameters for a single
axis system. Position of the drone is given by,

$p(t)=\int y(t)dt$ (8)

assuming the drone is starting at zero position. Since position of the drone
was to be controlled by the outer loop controller, the aim was to reduce the
position error, given by,

$e(t)=p(t)-p_{d}(t)$ (9)

where $p(t)$ is the actual position and $p_{d}(t)$ is the desired position.
The sliding surface was defined as,

$s(t)=\lambda e(t)+\dot{e}(t)$ (10)

Differentiating Equation 10 and substituting Equations 9, 8 and 7,

$\dot{s}(t)=\lambda y(t)+Ay(t)+Bu(t),$ (11)

where the rate of change of desired position was assumed to be zero. The
reaching law for SMC was considered as,

$\dot{s}(t)=-Ksign(s)-Qs$ (12)

where $K$ is the gain of constant rate reaching law and $Q$ is the
proportional gain of the second term considered in the law. This reaching law
was particularly chosen with proportional term so that the aggressiveness of
the UAV can be varied for different task requirements. From Equations 11 and
12, the control input $u(t)$ is derived as,

$u(t)=\frac{-Ksign(s)-Qs-Ay(t)-\lambda y(t)}{B}$ (13)

The derived control input was given as command velocity for position control
for navigation.

##  6 Field Results

The drone was tested in the field with the model-based SMC controller as well
as the linear controller obtained from heuristic approach. The linear
controller consisted of proportional and derivative actions (PD). The drone
was given a target point in the three dimensional space starting at the
origin. It was programmed to land when a desired accuracy of $\pm 8$ cm was
achieved and maintained in each axis for a time period of 5 seconds. It was
also ensured that safe saturation limits were implemented during drone
actuation in trials.

Figure 8: Comparison of PD and SMC performance

Figure 8 shows the comparison of SMC and PD performance. Time period for the
drone’s take off action and orientation to East direction, is marked in the
figure. Once the drone reached its minimum operational height, it started to
move to the given target in the three dimensional space. This test was
conducted in the presence of wind with an average wind speed of 5 km/h. In
Fig. 8, SMC settled in the desired accuracy band at around 23 s, and landing
occurred at around 28 s. The PD controller was not able to settle in the band
of 8 cm due to wind disturbances even after 50 s. Another important
observation was the large deviation in position for the PD controller in the
presence of wind.

For the competition, separate set of SMC’s controller parameters were obtained
for different bricks and the rise time was adjusted for laden and unladen
conditions. SMC rise time was governed by the controller parameter, $Q$. This
value was adjusted such that the drone did not behave aggressively while
carrying the brick, owing to the possibility of dropping it. Similarly, in
unladen condition, SMC was made aggressive by increasing $Q$, to save time.

From the results, it was inferred that SMC showed good robustness in windy
conditions and was believed to have addressed the modelling inaccuracies and
unmodelled dynamics. It also provided flexibility to adjust to different task
requirements.

Further, SMC controller was integrated with vision feedback algorithm and
implemented for brick picking and placing. Figure 9 shows an image of the
drone attempting to pick up a brick using SMC, during field trials. SMC was
able to carry out the required tasks with excellent accuracy, as seen from
Figure 9. Details of integration with vision algorithm, controller parameter
tuning for different scenarios and results will be included in the subsequent
papers.

Figure 9: Field test with SMC (a) UAV searches for the brick (b) UAV detects
the brick and approaches towards it (c) UAV attempts to pick the brick

##  7 Conclusion

A systematic framework was proposed for modelling of commercial UAV using the
data acquired from its simulator. This method produced a first order model
relating actual velocity of the drone to its command velocity for individual
axis of translatory motion, with an average deviation of 9% and a maximum
deviation of 16%. In order to account for the unmodelled dynamics, wind
disturbances and parametric variations in the drone mass and inertia, a
sliding mode controller was used with constant rate reaching law accompanied
by a proportional term. This controller was compared with conventional PD
controller obtained through a heuristic approach and was seen to perform
better than the latter in terms of accuracy and robustness. For a fixed
accuracy band of $\pm 8$ cm, SMC was able to maintain the position within the
band, however PD failed to do it. Thus, this framework provided a pragmatic
methodology with analytical backing to design a model based controller for the
intended application, considering task related constraints and environmental
conditions during the controller development process.

## References

  * [Gremillion and Humbert(2010)] Gremillion G, Humbert J (2010) System identification of a quadrotor micro air vehicle. In: AIAA Atmospheric Flight Mechanics Conference, p 7644
  * [Hoffer et al(2014)Hoffer, Coopmans, Jensen, and Chen] Hoffer NV, Coopmans C, Jensen AM, Chen Y (2014) A survey and categorization of small low-cost unmanned aerial vehicle system identification. Journal of Intelligent & Robotic Systems 74(1-2):129–145
  * [Kallapur et al(2008)Kallapur, Samal, Puttige, Anavatti, and Garratt] Kallapur A, Samal M, Puttige V, Anavatti S, Garratt M (2008) A UKF-nn framework for system identification of small unmanned aerial vehicles. In: 2008 11th International IEEE Conference on Intelligent Transportation Systems, IEEE, pp 1021–1026
  * [Kallapur et al(2007)Kallapur, Ali, and Anavatti] Kallapur AG, Ali SS, Anavatti SG (2007) Application of extended Kalman filter towards UAV identification. In: Autonomous Robots and Agents, Springer, pp 199–207
  * [Kamel et al(2017a)Kamel, Yasmina, Laredj, Benaoumeur, and Zoubir] Kamel B, Yasmina B, Laredj B, Benaoumeur I, Zoubir AF (2017a) Dynamic modeling, simulation and PID controller of unmanned aerial vehicle UAV. In: 2017 Seventh International Conference on Innovative Computing Technology (INTECH), IEEE, pp 64–69
  * [Kamel et al(2017b)Kamel, Stastny, Alexis, and Siegwart] Kamel M, Stastny T, Alexis K, Siegwart R (2017b) Model predictive control for trajectory tracking of unmanned aerial vehicles using robot operating system. In: Robot operating system (ROS), Springer, pp 3–39
  * [Lv et al(2018)Lv, Shen, Xia, Ma, Yu, Shen, and Zhou] Lv J, Shen J, Xia H, Ma G, Yu C, Shen Y, Zhou C (2018) Fuzzy SMC for UAV to resist strong wind with disturbance approximation. In: 2018 Ninth International Conference on Intelligent Control and Information Processing (ICICIP), IEEE, pp 93–98
  * [Najm and Ibraheem(2019)] Najm AA, Ibraheem IK (2019) Nonlinear PID controller design for a 6-DOF UAV quadrotor system. Engineering Science and Technology, an International Journal 22(4):1087–1097
  * [Nasr et al(2018)Nasr, Ashraf, Hussein, Salem, Elias, Shehata, and Morgan] Nasr M, Ashraf M, Hussein MS, Salem AS, Elias CM, Shehata OM, Morgan EI (2018) A comparitive study on the control of UAVs for trajectory tracking by MPC, SMC, backstepping, and fuzzy logic controllers. In: 2018 IEEE International Conference on Vehicular Electronics and Safety (ICVES), IEEE, pp 1–6
  * [Sridhar et al(2017)Sridhar, Subramaniyam, Subramanian, Vivekanandan, and Sivaram] Sridhar N, Subramaniyam KV, Subramanian SC, Vivekanandan G, Sivaram S (2017) Model based control of heavy road vehicle brakes for active safety applications. In: 2017 14th IEEE India Council International Conference (INDICON), IEEE, pp 1–6