# Learning Exemplar Representations
in Single-Trial EEG Category Decoding ††thanks: This publication emanated from
research supported in part by Science Foundation Ireland research grants
16/RI/3399, 18/CRT/6049, and 20/FFP-P/8853, the United States National Science
Foundation Grant 1734938-IIS, and the Defense Advance Research Projects
Agency. This publication is approved for public release; distribution is
unlimited. Any opinions, findings, views, conclusions or recommendations
expressed in this material are those of the authors and do not necessarily
reflect the views, official policies, or endorsements, either expressed or
implied, of the sponsors. Finally, we thank Hari M. Bharadwaj for suggesting
that we look into the issue discussed in this paper as a potential source of
leakage.

Jack A. Kilgallen  Hamilton Institute
Maynooth University
Maynooth, Ireland
<EMAIL_ADDRESS>Barak A. Pearlmutter  Department of Computer Science
Maynooth University
Maynooth, Ireland
<EMAIL_ADDRESS>Jeffrey Mark Siskind  Elmore Family School of Electrical
and Computer Engineering
Purdue University
West Lafayette, IN, USA
<EMAIL_ADDRESS>
###### Abstract

Within neuroimgaing studies it is a common practice to perform repetitions of
trials in an experiment when working with a noisy class of data acquisition
system, such as electroencephalography (EEG) or magnetoencephalography (MEG).
While this approach can be useful in some experimental designs, it presents
significant limitations for certain types of analyses, such as identifying the
category of an object observed by a subject. In this study we demonstrate that
when trials relating to a single object are allowed to appear in both the
training and testing sets, almost any classification algorithm is capable of
learning the representation of an object given only category labels. This
ability to learn object representations is of particular significance as it
suggests that the results of several published studies which predict the
category of observed objects from EEG signals may be affected by a subtle form
of leakage which has inflated their reported accuracies. We demonstrate the
ability of both simple classification algorithms, and sophisticated deep
learning models, to learn object representations given only category labels.
We do this using two datasets; the Kaneshiro et al. (2015) dataset and the
Gifford et al. (2022) dataset. Our results raise doubts about the true
generalizability of several published models and suggests that the reported
performance of these models may be significantly inflated.

###### Index Terms:

EEG, machine learning, leakage, category decoding

##  I Introduction

Within neuroimgaing studies when working with a noisy modality such as
electroencephalography (EEG), or magnetoencephalography (MEG), it is common
practice to perform multiple trials using a single stimulus. With some
experimental designs this repetition of trials can be useful for reducing the
impact of noise on the analysis being performed, particularly when extracting
features such as Event Related Potentials (ERPs)[1]. However, this practice
presents significant limitations for certain types of analyses such as
_single-trial category decoding_.

In single-trial category decoding experiments, subjects are shown a series of
objects from different categories while their EEG signals are recorded. The
aim of the experiment is to identify the category of an object observed by a
subject from a single-trial of their EEG signals. This is a challenging task
as the EEG signals are noisy, and the features which are relevant to the
category of an object may be subtle.

When performing single-trial category decoding, the use of multiple trials per
object can allow a model to use information which we would not expect it to
have access to at prediction time _(leakage)_. When multiple trials per object
are used, a classification algorithm may overfit to features of an EEG signal
which are specific to an observed object from a category _(an exemplar
representation)_ rather than to features of neural responses which are shared
by many or all objects from a category _(a category representation)_.
Moreover, if trials relating to specific stimuli are allowed to appear in both
the training and testing sets, then algorithms which overfit to exemplar
representations are likely to achieve higher accuracies. This occurs as their
performance is evaluated using trials which are more similar to the trials
they were trained on than would be expected in a real-world scenario. In this
way a loss of generalizability is rewarded by the evaluation procedure.

An analogy to this form of leakage can be found in the case of a machine
learning student who wants to classify images of cats and dogs. The student
collects photos from friends and family of their cats and their dogs, but each
person they ask sends multiple photos of each of their pets. If the student
trains a computer vision algorithm to classify the images of cats and dogs,
but images relating to specific cats or dogs are allowed to appear in both the
training and testing sets, then the algorithm may be affected by a similar
form of leakage. Instead of learning what a dog or a cat generally looks like,
the algorithm may learn what a specific dog or cat looks like. It would then
achieve higher accuracy by overfitting to the features of that specific dog or
cat (i.e. learning the representation of an exemplar). The situation would be
further exacerbated if the dataset didn’t contain many breeds of dogs or cats,
as the algorithm would have fewer opportunities to learn what features are
shared by all dogs or cats. Similarly, in the case of single-trial category
decoding, algorithms may achieve increased by overfitting to features of a
signal a subject’s brain generates when they observe a specific object.

While this form of leakage may seem contrived, it is increasingly present in
recent literature. Since the publication of the Kaneshiro et al. (2015)[2]
dataset (hereafter referred to as the Kaneshiro dataset) which features
multiple trials per stimulus, 12 studies have made use of their dataset for
single-trial EEG category decoding[3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14].
However, none of these studies make any mention of the potential for leakage
in their experimental design, and so it is likely that the results of these
studies are affected.

To investigate the ability of classification algorithms to exploit exemplar
representations to inflate their performance at single-trial category
decoding, we augmented two EEG category decoding datasets to remove any
semantic meaning from their category labels. We remove semantic meaning by
creating new category labels for which no reasonable distinction can be made
between categories. Therefore, when performing category decoding on such a
dataset, any model which achieves a decoding accuracy which is above chance,
in a statistically significant manner, would suggest that the model has
exploited knowledge of an exemplar representation.

Our analysis demonstrates not only that some bespoke EEG decoding algorithms
are capable of learning exemplar representations given only category labels,
but that even simple classification algorithms such as k-Nearest Neighbors
(k-NN) are capable of exploiting exemplar representations. Moreover, we
demonstrate that this form of leakage can occur using two separate datasets;
the Kaneshiro datset[2] and the Gifford et al. (2022)[15] dataset (hereafter
referred to as the Gifford dataset).

##  II Related Work

While machine learning has a broad range of applications, training models on
data without a strong understanding of the domain the data originates from can
cause confounds and leakage to go unnoticed. This has occurred in a variety of
contexts[16] and attention is being given to rooting out studies affected by
such confounds[17]. Even within a domain which might be considered niche, such
as neuroimaging, significant confounds, such as the Block Design Confound[18],
have been discovered in recent literature. The work presented here is part of
a sustained effort to identify erroneous or exaggerated claims resulting from
a misapplication of machine learning techniques in neuroimaging studies.

##  III Materials and Methods

###  III-A Datasets

The datasets used in this study are the Kaneshiro dataset[2] and the Gifford
dataset[15]. They were selected as they were publically available, and
possessed both multiple trials per stimulus, and a category hierarchy.

####  III-A1 The Kaneshiro Dataset[2]

This dataset consists of preprocessed EEG recordings taken from 10 subjects
while they viewed 72 images evenly distributed across 6 categories: Human Body
(HB), Human Face (HF), Animal Body (AB), Animal Face (AF), Fruit/Vegetable
(FV) and Inanimate Object (IO). Fig. refkaneshiro-category-structure shows the
stimuli that were presented to the subjects grouped by category. To reduce the
impact that noise would have on their analysis the dataset contains 72 trials
for each image per subject which were presented in random order. This gives a
total of 5,184 trials per participant. The data was recorded using a 128
channel EEG system with a sampling rate of 1 kHz. The EEG signals were then
preprocessed using a high-pass fourth-order Butterworth filter to attenuate
frequencies below 1 Hz, and a low-pass Chebyshev Type I filter to attenuate
frequencies above 25 Hz. Ocular artifacts were removed using the Bell-
Sejnowski Infomax independent component analysis algorithm[19], and finally
the data was subsampled to 62.5 Hz to reduce the computational cost of the
analysis.

Figure 1: The stimuli presented to subjects in the Kaneshiro dataset[2].

####  III-A2 The Gifford Dataset[15]

This dataset is made up of preprocessed EEG recordings taken from 10 subjects
while they viewed a selection of images from the THINGS image database[20].
The images were divided across 27 high-level categories, which span 1854
object concepts. The object categories were partitioned into training and test
sets with 1654 object concepts used in the training set and 200 used in the
test set. For the training set 10 exemplars per concept which were each
presented to the subjects once for a total of 16,540 trials per subject. For
the test set its 200 object concepts each contained a single exemplar which
was presented 80 times, for a total of 16,000 trials per subject. The
experiment used a rapid serial visual presentation (RSVP) paradigm. During
image presentation the subjects were asked to perform a target detection task
in which they counted the number of times a predefined target image was
presented. The data was recorded using a 64-channel EEG system sampled at
1000Hz, which was filtered using a 0.1Hz high-pass filter and a 100Hz low-pass
filter, and baseline correction was applied using pre-stimulus recordings. The
data finally subsampled to 100Hz, and 17 channels over the occipital and
parietal cortex were selected. It should be noted that this dataset is not
easily misused in the same way as the Kaneshiro dataset[2], as it explicitly
supplies the data as separate training and testing sets with no exemplars
appearing in both sets. However, its large size and repetition of exemplars in
its test set make it a very useful dataset to investigate if category decoding
algorithms can learn exemplar representations when the dataset features a
higher number of exemplars per category.

###  III-B Generating Pseudocategories

To demonstrate that category decoding algorithms can learn exemplar
representations, we constructed _pseudocategories_ from the stimulus sets used
in the original datasets. By pseudocategory we mean a collection of exemplars
with no semantic relationship to each other, but which are presented to the
model as if they were from the same category. Importantly, we ensured that the
pseudocategories were balanced in both the number of exemplars they contained,
and the distribution of those exemplars among their true categories.

####  III-B1 The Kaneshiro Dataset[2] Pseudocategories

To construct the pseudocategories for this dataset, we selected a single
exemplar from each of the 6 categories and combined all trials relating to
these exemplars into a single pseudocategory. This was repeated 12 times until
all exemplars were used, resulting in 12 pseudocategories. As there are 12
pseudocategories, which are balanced in their number of trials, it follows
chance accuracy for a model predicting the pseudocategory of a trial is
$\frac{1}{8}=8.25\%$.

Figure 2: Two of the pseudocategories generated for the Gifford dataset[15].

####  III-B2 The Gifford Dataset[15] Pseudocategories

To construct the pseudocategories for this dataset, we used only trials from
the predefined test set for each subject as only this set contained multiple
trials per exemplar. We then grouped them by their higher-level categories,
and excluded any categories which contained fewer than five exemplars, or were
deemed to be too similar to other high-level categories. Finally, we evenly
divided exemplars from each category into 5 pseudocategories. The
pseudocategories each contained: 3 animals, 1 bird, 2 items of clothing, 1
container, 1 electronic device, 6 items of food, 1 musical instrument, 2 items
of sports equipment, 3 tools, and 3 vehicles for a total of 23 exemplars per
pseudocategory. Fig. 2 shows an example of a pseudocategory constructed this
way. Again, as the pseudocategories are balanced in their number of trials,
and there are 5 categories, it follows that chance accuracy for a model
predicting the pseudocategories of a trial is $\frac{1}{5}=20\%$.

###  III-C Category Decoding Algorithms

To test the ability of classification algorithms to learn exemplar
representations, we selected a broad variety of models. Our selection was
based on capturing the spectrum of complexity of classification algorithms
commonly used in EEG decoding, from simple classical machine learning
algorithms to state-of-the-art deep learning models. While many recent models
have been developed specifically for single-trial EEG category decoding the
reproducibility of these models is limited by the availability of their
implementations. Therefore, we selected models which are well known and have
implementations available in popular libraries. The models used in this study
are:

  * •

Support Vector Machine (SVM)[21].

  * •

Linear Discriminant Analysis (LDA)[22].

  * •

EEGNet[23].

  * •

DeepConvNet[24].

  * •

ShallowConvNet[24].

The k-Nearest Neighbors (k-NN) algorithm[25] was selected to investigate the
ability of a simple classical machine learning algorithm to learn exemplar
representations. The neural network models were implemented using the PyTorch
library[26] with some modifications to the original implementations to account
for the lower temporal resolution of our data relative to the dataset these
models were originally trained on. The scikit-learn library[27] was used to
implement the SVM, LDA and kNN models. A detailed description of the model
architectures and training procedures can be found in the supplementary
material.

##  IV Learning Exemplar Representations from Categories

###  IV-A Model Implementations and Training/Testing Procedures

Within subject pseudocategory decoding was then performed for each of the
models on both datasets. A stratified k-fold cross-validation procedure was
used to train and test the models. The data was stratified using the exemplar
labels to ensure that the distribution of exemplars in each fold was
representative of the distribution in the dataset. The number of folds was set
to 12 for the Kaneshiro dataset[2] and 10 for the Gifford dataset[15]. The
accuracy of the models was then calculated as the proportion of correctly
classified trials. This resulted in 120 and 100 accuracy values for the
Kaneshiro dataset[2] and Gifford datasets[15] respectively.

##  V Statistical Analysis

To determine if the models were learning exemplar representations we compared
the decoding accuracy of the models on the pseudocategory labels to chance
accuracy using a one-sample t-test. The significance level was set at
$\alpha=0.05$, and then Bonferroni correction was applied to account for
multiple comparisons. Since we performed tests for 6 models over 2 datasets
this resulted in the significance level being adjusted to
$\alpha=\frac{0.05}{6\cdot 2}\approx 0.0042$.

TABLE I: Pseudocategory Decoding Accuracies Model | Accuracy (%)
---|---
Kaneshiro et al. (2015) |
kNN | $10.6809^{*}$
SVM | $14.5370^{*}$
LDA | $14.2477^{*}$
EEGNet | $14.4483^{*}$
DeepConvNet | $16.6532^{*}$
ShallowConvNet | $12.8106^{*}$
Gifford et al. (2022) |
kNN | $22.1076^{*}$
SVM | $30.1250^{*}$
LDA | $30.3076^{*}$
EEGNet | $30.1033^{*}$
DeepConvNet | $30.5511^{*}$
ShallowConvNet | $27.8163^{*}$

##  VI Results

The results of the statistical analysis are shown in Table I. Starred values
indicate that the accuracy was above chance with statistical significance
(i.e. $p<0.001$, given a one-sample t-test with Bonferroni correction). Given
that all models achieved decoding accuracies above chance with statistical
significance on both datasets, our results show that classification algorithms
are indeed able to learn exemplar representations from categories. While the
effect may have been more dramatic for highly complex models, even models such
as kNN with only a single parameter can exploit exemplar representations to
some degree. Moreover, it is of particular interest that the DeepConvNet had
an accuracy of over twice the chance accuracy on the Kaneshiro dataset[2].
This interesting because, as mentioned previously, the Kaneshiro dataset[2]
has been used for category decoding in 12 additional studies following its
publication; 10 of which used deep learning models. This suggests that the
true accuracy of models affected by the leakage within the literature may
differ dramatically from the published results. Box plots of the
pseudocategory decoding accuracy are shown for both of the datasets in Fig. 3
and Fig 4. Moreover, it is of interest that all the models performed above
chance on the Gifford dataset[15], even though it featured 23 exemplars per
pseudocategory. This demonstrates that the classification algorithms are able
to exploit exemplar representations even when the number of exemplars per
category is much higher than in the Kaneshiro dataset[2], and it is not a flaw
which can be easily mitigated by increasing the number of stimuli used in
experiments.

Figure 3: A box plot of the pseudocategory decoding accuracies for each model
trained using the Kaneshiro dataset[2].

Figure 4: A box plot of the pseudocategory decoding accuracies for each model
trained using the Gifford dataset[15].

##  VII Future Work

While the results of this study demonstrate the ability of classification
algorithms to learn exemplar representations given only category labels,
further work is needed in recreating a sample of published models with
reported decoding accuracies which are suspected to be inflated due to
repeated exemplar leakage. This would allow a more comprehensive quantitative
analysis of the impact of the leakage and clarify the true state-of-the-art in
single-trial category decoding. Moreover, while we know that the models are
capable of learning exemplar representations in a category decoding task, it
has not yet been demonstrated that the same form of leakage can occur in other
forms of EEG analysis. In particular if multiple categories are allowed to
appear in a single image then instead of decoding a class we would be
detecting the presence of classes. While it seems likely that the same form of
leakage would occur in this case, it has not yet been demonstrated, though
datasets featuring repeated stimuli and class presence labels are
available[28].

##  VIII Conclusion

The results of this study demonstrate that given the multiple trials per
stimulus, and improper model evaluation methods, almost any classification
algorithm is capable of learning exemplar representations given only category
labels. This has significant implications not only for the fields of
neuroimaging and machine-learning, but also for any brain computer interface
(BCI) applications which might be developed on the basis of results affected
by such leakage. We achieved this using two datasets, the Kaneshiro dataset[2]
and the Gifford dataset[15], and a variety of prominent EEG classification
algorithms. Use of the former dataset demonstrated that this leakage likely
affects several published works. And use of the latter dataset established
both the potential of this leakage to occur in other datasets, and that
classification algorithms may still learn exemplar specific features, even
when the number of exemplars per category is much higher than in the Kaneshiro
dataset[2]. The significance of our findings is not only in highlighting the
potential for leakage in single-trial category decoding studies which use
repeated trials per stimulus, but also in revealing that the true state-of-
the-art in single-trial category decoding is uncertain. It seems intuitive
that as highest reported performances of models increase on this dataset, the
extent to which they are likely affected by repeated exemplar leakage also
increases. Therefore, which models have performed best at learning category
representations is currently unclear.

## References

  * [1] P. A. Davis, “Effects of acoustic stimuli on the waking human brain,” _Journal of neurophysiology_ , vol. 2, no. 6, pp. 494–499, 1939.
  * [2] B. Kaneshiro, M. Perreau Guimaraes, H.-S. Kim, A. M. Norcia, and P. Suppes, “A representational similarity analysis of the dynamics of object processing using single-trial EEG classification,” _PLOS ONE_ , vol. 10, no. 8, p. e0135697, Aug. 2015.
  * [3] H. Ahmadieh, F. Gassemi, and M. H. Moradi, “A hybrid deep learning framework for automated visual image classification using EEG signals,” _Neural Computing and Applications_ , vol. 35, no. 28, pp. 20 989–21 005, Oct. 2023.
  * [4] A. S. Bobe, A. S. Alekseev, M. V. Komarova, and D. Fastovets, “Single-trial ERP feature extraction and classification for visual object recognition task,” in _2018 Engineering and Telecommunication (EnT-MIPT)_. Moscow, Russia: IEEE, Nov. 2018, pp. 188–192.
  * [5] Z. Jiao, H. You, F. Yang, X. Li, H. Zhang, and D. Shen, “Decoding EEG by visual-guided deep neural networks,” in _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence_ , Macao, China, Aug. 2019, pp. 1387–1393.
  * [6] S. Bagchi and D. R. Bathula, “Adequately wide 1D CNN facilitates improved EEG based visual object recognition,” in _2021 29th European Signal Processing Conference (EUSIPCO)_. Dublin, Ireland: IEEE, Aug. 2021, pp. 1276–1280.
  * [7] ——, “EEG-ConvTransformer for single-trial EEG-based visual stimulus classification,” _Pattern Recognition_ , vol. 129, p. 108757, Sep. 2022.
  * [8] Y. Deng, S. Ding, W. Li, Q. Lai, and L. Cao, “EEG-based visual stimuli classification via reusable LSTM,” _Biomedical Signal Processing and Control_ , vol. 82, p. 104588, Apr. 2023.
  * [9] A. Fares, S.-h. Zhong, and J. Jiang, “Brain-media: A dual conditioned and lateralization supported GAN (DCLS-GAN) towards visualization of image-evoked brain activities,” in _Proceedings of the 28th ACM International Conference on Multimedia_. Seattle WA USA: ACM, Oct. 2020, pp. 1764–1772.
  * [10] J. Kalafatovich, M. Lee, and S.-W. Lee, “Decoding visual recognition of objects from EEG signals based on attention-driven convolutional neural network,” in _2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC)_ , Toronto, ON, Canada, Oct. 2020, pp. 2985–2990.
  * [11] ——, “Learning spatiotemporal graph representations for visual perception using EEG signals,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 31, pp. 97–108, 2023.
  * [12] J. Kalafatovich and M. Lee, “Subject-independent object classification based on convolutional neural network from EEG signals,” in _2021 9th International Winter Conference on Brain-Computer Interface (BCI)_. Gangwon, Korea (South): IEEE, Feb. 2021, pp. 1–4.
  * [13] J. Luo, W. Cui, S. Xu, L. Wang, X. Li, X. Liao, and Y. Li, “A dual-branch spatio-temporal-spectral transformer feature fusion network for EEG-based visual recognition,” _IEEE Transactions on Industrial Informatics_ , pp. 1–11, 2023.
  * [14] M. Yavandhasani and F. Ghaderi, “Visual object recognition from single-trial EEG signals using machine learning wrapper techniques,” _IEEE Transactions on Biomedical Engineering_ , vol. 69, no. 7, pp. 2176–2183, Jul. 2022.
  * [15] A. T. Gifford, K. Dwivedi, G. Roig, and R. M. Cichy, “A large and rich eeg dataset for modeling human visual object recognition,” _NeuroImage_ , vol. 264, p. 119754, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1053811922008758
  * [16] Princeton Center for Statistics and Machine Learning, “The Reproducibility Crisis in ML‑based Science,” 28 Jul. 2022, workshop. [Online]. Available: https://sites.google.com/princeton.edu/rep-workshop/
  * [17] S. Kapoor and A. Narayanan, “Leakage and the reproducibility crisis in machine-learning-based science,” _Patterns_ , vol. 4, no. 9, p. 100804, Sep. 2023.
  * [18] R. Li, J. S. Johansen, H. Ahmed, T. V. Ilyevsky, R. B. Wilbur, H. M. Bharadwaj, and J. M. Siskind, “The perils and pitfalls of block design for EEG classification experiments,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 43, no. 1, pp. 316–333, 2021.
  * [19] A. J. Bell and T. J. Sejnowski, “An information-maximization approach to blind separation and blind deconvolution,” _Neural Computation_ , vol. 7, no. 6, pp. 1129–59, 1995.
  * [20] M. N. Hebart, A. H. Dickter, A. Kidder, W. Y. Kwok, A. Corriveau, C. Van Wicklin, and C. I. Baker, “Things: A database of 1,854 object concepts and more than 26,000 naturalistic object images,” _PLOS ONE_ , vol. 14, no. 10, pp. 1–24, 10 2019. [Online]. Available: https://doi.org/10.1371/journal.pone.0223792
  * [21] C. Cortes and V. Vapnik, “Support-vector networks,” _Machine learning_ , vol. 20, no. 3, pp. 273–297, 1995.
  * [22] R. A. Fisher, “The use of multiple measurements in taxonomic problems,” _Annals of Eugenics_ , vol. 7, no. 2, pp. 179–188, 1936. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1469-1809.1936.tb02137.x
  * [23] V. J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P. Hung, and B. J. Lance, “EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces,” vol. 15, no. 5, p. 056013.
  * [24] R. T. Schirrmeister, J. T. Springenberg, L. D. J. Fiederer, M. Glasstetter, K. Eggensperger, M. Tangermann, F. Hutter, W. Burgard, and T. Ball, “Deep learning with convolutional neural networks for eeg decoding and visualization,” _Human Brain Mapping_ , vol. 38, no. 11, pp. 5391–5420, 2017. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/hbm.23730
  * [25] T. Cover and P. Hart, “Nearest neighbor pattern classification,” _IEEE Transactions on Information Theory_ , vol. 13, no. 1, pp. 21–27, 1967.
  * [26] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in _Advances in Neural Information Processing Systems 32_. Curran Associates, Inc., 2019, pp. 8024–8035. [Online]. Available: http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * [27] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” _Journal of Machine Learning Research_ , vol. 12, pp. 2825–2830, 2011.
  * [28] J. Xu, B. Aristimunha, M. E. Feucht, E. Qian, C. Liu, T. Shahjahan, M. Spyra, S. Z. Zhang, N. Short, J. Kim, P. Perdomo, R. R. Mao, Y. Sabharwal, M. A. M. Shoura, and A. Nestor, “Alljoined1 – a dataset for eeg-to-image decoding,” 2024.


# We are Going to the Space - Part 1:
Which device to deploy in a satellite?

Robert Bayer  IT University of Copenhagen<EMAIL_ADDRESS>,  Julian Priest  IT
University of Copenhagen<EMAIL_ADDRESS>and  Pınar Tözün  IT University of
Copenhagen<EMAIL_ADDRESS>
(2018)

###### Abstract.

The shrinkage in sizes of components that make up satellites led to wider and
low cost availability of satellites. As a result, there has been an advent of
smaller organizations having the ability to deploy satellites with a variety
of data-intensive applications to run on them. One popular application is
image analysis to detect, for example, land, ice, clouds, etc. However, the
resource-constrained nature of the devices deployed in satellites creates
additional challenges for this resource-intensive application.

In this paper, we investigate the performance of a variety of edge devices for
deep-learning-based image processing in space. Our goal is to determine the
devices that satisfy the latency and power constraints of satellites while
achieving reasonably accurate results. Our results demonstrate that hardware
accelerators (TPUs, GPUs) are necessary to reach the latency requirements. On
the other hand, state-of-the-art edge devices with GPUs could have a high
power draw, making them unsuitable for deployment on a satellite.

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: ; ; ††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Introduction

In the last century, most innovation in real-world satellite applications were
only available to the largest countries such as USA and Russia. These
innovations led to significant reductions in the size of the components that
make up a satellite and in the cost of the manufacturing and deployment
process of a satellite. This, in turn, introduced a new CubeSat class of
miniature satellites. Their format is based on a 10cm cube, with the
possibility of combining multiple modules to create a larger satellite. This
standardization makes a batch deployment of satellites easier as the format
affords tight configuration. The reduction in costs that came as a result of
this new deployment method led to the advent of satellites owned by small or
private organizations.

However, the size of the CubeSat class satellites poses new complex challenges
such as the power and thermal constraints as well as the physical dimensions
of components. These CubeSats often perform resource-intensive tasks, which
clashes with the resource-constrained nature of their format.

Image processing and analysis is one class of possible satellite workloads.
Satellites with this type of workload take large-scale images, which they have
to store and send back to a ground station. The link between the satellite and
a ground station is of limited bandwidth and short-lived. The images must be
highly compressed to send all of them or filtered by quality and areas of
interest. The images already have a resolution of tens or hundreds of meters
per pixel, and lossy compression would lead to even more loss of detail.
Filtering preserves the details of the images, but is more resource-intensive.

This paper is a step toward understanding the requirements and limitations of
deep-learning-based image filtering systems on small satellites. More
specifically, we characterize the requirements and constraints of an image
processing unit (IPU) deployed on a satellite and outline possible use case
scenarios of such a system, each introducing a different degree of resource
constraints and intensity. We then further analyze the performance of multiple
edge devices on these different scenarios and their suitability for possible
deployment on one of the Danish Student CubeSat Project’s (DISCO) (DISCO,
2022) satellites.

These devices are based on different hardware architectures, ranging from a
microcontroller to a tensor processing unit (TPU). While microcontrollers or
more complex CPUs have extensive flight history (have already been deployed in
satellites before) and low power footprint, they were not built with running
deep learning workloads in mind. On the other hand, GPUs have been utilized
more in the past decade in satellites, with one of the leading workloads being
machine learning, based on the success of GPUs in terrestrial use cases of
machine learning. The latest of these architectures, TPU, has not been
extensively researched for deployment in satellites, even though TPUs were
designed to perform fast neural network inference, promising low latency with
high efficiency.

The contributions of this study are as follows:

  * •

We illustrate the differences between the terrestrial and space edge IoT. The
edge devices deployed in space have to perform very resource-intensive
workloads with high reliability and do so in a highly resource-constrained
environment.

  * •

We characterize a set of requirements specific to IPU on a CubeSat. We do this
by outlining multiple use case scenarios and showing how they affect the
required latency and possible power draw of such a system.

  * •

We compare the performance of multiple edge devices based on different
architectures and show the need and the benefit of using highly specialized
hardware for machine learning workloads on satellites.

Image processing Unit |  Nominal Power | Peak Power | Design Margin | Nominal Cycle | Peak Cycle | Power | Mass budget | Dimensions
---|---|---|---|---|---|---|---|---
Constraints |  2.00 W | 5.00 W | 5% | 20% | 20% | 1.48 W | 0.15 Kg | 10x70x80 mm
Table 1. List of constraints of the IPU on board of the DISCO2 satellite.

##  2\. Background

We first introduce DISCO project and survey related work that specifically
targets data processing in satellites.

###  2.1. DISCO

The Danish Student Cubesat Program (DISCO) (DISCO, 2022) offers Danish
university students the opportunity to design and operate a small satellite
and to gain space flight experience. DISCO is a collaboration between four
universities in Denmark, which will initially launch three student Cubesats
into Low Earth Orbit with the first launch scheduled for 2023.

DISCO2 has been designed by students to carry an Earth imaging payload,
infrastructure to capture images in space, into a solar synchronous polar
orbit. The instrument is a collaboration with the Arctic Research Centre at
Aarhus University and will support a range of field research in Greenland.
Initially focused on a long term marine systems and cryosphere monitoring
projects, the satellite will supplement ground based observations with remote
sensing data, providing both good polar coverage and on demand availability
for the projects.

The payload will include one or more high resolution cameras as well as a
dedicated IPU, which will be capable of simple machine learning applications
for image classification. Students will be able to carry out conventional on-
satellite image processing in addition to running machine learning
applications for classification, discrimination, and feature identification.

###  2.2. Related Work

The challenge of low bandwidth on small satellites creates the need for data
post-processing to filter the data on the satellite before sending elsewhere.
Neural-network-based filtering has been studied for this purpose in recent
years. Specifically, several works have tested the suitability of small
system-on-chip (SoC) devices leveraging the power of GPUs to accelerate
machine learning workloads (Lofqvist and Cano, 2020; Brown, 2018). Moreover,
some works have also explored the use of ASICs, such as Intel’s Movidius
Myriad vision processing units (VPUs) (Giuffrida et al., 2020; Esposito et
al., 2019). This VPU has even seen real-life deployment (Esposito et al.,
2019).

Evaluation of TPUs for deployment on satellites is limited, however. To the
best of our knowledge, only one work has explored this option (Goodwill et
al., 2021), with no real-life deployments. DISCO project would therefore be
the first satellite to leverage TPU for onboard deep learning applications,
based on the findings of our study.

In this work, we aim to characterize the performance of the different flavors
of off-the-shelf edge devices to check their suitability for being deployed on
a satellite in the context of DISCO project.

Data management and processing for internet-of-things and edge computing has
been important in the data management community as well (Chatziliadis et al.,
2021; O’Keeffe et al., 2018; Zeuch et al., 2020; Warnke et al., 2022). We are
complementing these works with a very specific application focus, which is
data-intensive applications deployed on satellites.

##  3\. Requirements

The requirements and constraints used in this study are based on the DISCO2
Arctic imaging mission use case. The power and mass constraints of the edge
device to be deployed on the satellite are based on the power and mass budgets
developed in the DISCO2 engineering design. The satellite design is a modular
3U Cubesat with off-the-shelf modules for attitude control, power,
communications, and flight control. The power budget values are typical for a
3U earth imaging Cubesat of this type. The resulting payload capacity is 1U
(10x10x10cm) and 1.3kg and this must include the cameras, module enclosure,
optics. and image processing unit (IPU). The full list of constraints of the
IPU is shown in Table 1.

The planned orbit of DISCO2 is a solar synchronous polar orbit at 550 km
altitude. The camera sensor is based on an Alvium 1800 C-2040 and the lens
focal length is the maximum that could be considered for the mission. Assuming
a 50% overlap between images, i.e., capturing the same land area, this
provides a minimum time of 4.42s between consecutive images ($T_{i}$), which
was derived as follows:

(1)
$T_{i}=\frac{GSD*W_{i}*\cap_{i}}{v}=\frac{14.8495m/px*4512px*0.5}{7585.16m/s}=4.42s$

, where $GSD$ (ground sample distance) is the spatial resolution of the image,
$W_{i}$ is the width of image in pixels, $\cap_{i}$ is the overlap between
images and $v$ is the orbital velocity of the satellite.

Taking this latency value as a baseline, we now describe three image
processing scenarios with different levels of difficulty based on this Arctic
mission’s goals.

Scenario 1: Real-time imaging. Images are taken with a period of 4.42s, as
derived above. The inference has to be performed and completed before the next
image is captured. This would allow for the results of an inference to be used
in decision making for the next image capture.

Scenario 2: Arctic region imaging. In this scenario, images are taken only
while the satellite is over the polar regions relaxing the latency
requirements. Images are buffered and inference is performed in the remainder
of the orbit, with the IPU available for inference before the subsequent
orbit. The capture-inference cycle completes in 5739s.

Scenario 3: Greenland imaging. The images are taken only while over Greenland
further relaxing the latency requirements. As the orbit is solar synchronous
with a period of one day. This results in subsequent bursts but only when the
swathe passes over Greenland. Images are again buffered and the capture-
inference cycle completes in 21600s.

##  4\. Methodology and Setup

Our goal is to characterize the performance of modern low-power hardware for
deployment as the IPU of the DISCO2 satellite. This section presents our
experimental methodology and setup to achieve this goal.

###  4.1. Devices under Test

As representative hardware for this study, we picked three devices, each based
on a different hardware architecture. These choices were made based on their
physical dimensions, weight, power draw, and high performance considering the
low power draw limits.

ARM Cortex-M Mirocontroller, based on the STM32H745 chip, has the potential of
the lowest power draw, while the least performant of the choices. As an
additional advantage, this chip has extensive flight history and, therefore,
would be a safe choice for deployment in space applications.

In order to use this device as an on-board IPU, we can use TensorFlow Lite for
Microcontrollers (David et al., 2021), a framework designed to allow neural
network inference with only 16 KB of memory overhead (core runtime) and
without the need for an operating system. In addition, the framework also
provides support for the use of specialized neural network kernels, such as
CMSIS-NN (Lai et al., 2018) kernels designed specifically for Cortex-M
processor cores. The X-CUBE-AI (X-CUBE-AI, 2022) framework makes the
deployment of the neural networks with TensorFlow Lite for microcontrollers
even more accessible. It provides an intuitive GUI for the libraries or code
samples based on the provided and trained TensorFlow Lite (tfl, 2022) model.

Even though this device has support for floating point operations, a quantized
model was used, in order to achieve better latency and, more importantly,
lower memory footprint, which is a large constraint of this device.

To simulate the exact performance and power characteristics of a flight
computer already available for a potential deployment in DISCO project,
OBC-P3, we used only the Cortex-M7 core of the STM32H745 and scaled its clock
from 480MHz down to 300MHz. Parameters of the OBC-P3 system can be found in
the Table 2.

Processor | 2x ARM Cortex-M7 @ 300 MHz
---|---
SRAM | 384 KB SRAM
FRAM | 32 KB
Flash | 2 MB
Storage | 64 GB eMMC
Dimensions | 94 x 94 x 13 mm
Weight | 120 g
Table 2. Specifications of the OBC-P3 system containing the ARM Cortex-M7
MCUs. The dimensions and weight includes the enclosure with aluminium
shielding.

NVIDIA Jetson Nano is a portable SoC composed of power-efficient ARM CPU and
NVIDIA GPU designed for embedded systems that require GPU-friendly
computations, e.g., image processing, video encoding/decoding, and machine
learning tasks. It is the only device in our experiments that supports
TensorFlow Core. Therefore, the deployment process to this device is
equivalent to that of servers or desktops. It is also the only evaluated
device supporting batch sizes larger than one.

The device can operate at a wattage between 5W - 10W. We configured it to
operate at 5W by disabling two of the four CPU cores. While this setting leads
to lower performance, it is necessary in order to fit into the power budget
outlined in Table 1. Parameters of this device can be found in Table 3.

CPU | Quad-core ARM A57 @ 1.43 GHz
---|---
GPU | 128-core Maxwell
GFLOPS | 472
RAM | 4 GB 64-bit LPDDR4 25.6 GB/s
Storage | 64 GB SD card
Dimensions | 100 x 80 x 29 mm
Weight | 141 g
Table 3. Specifications of the NVIDIA Jetson Nano. The dimensions and weight
are based on the developer kit version of this device.

CoralAI TPU is an ASIC, AI accelerator developed by Google to accelerate
machine learning workloads at the edge. TPU is not a standalone device and
must be deployed together with a Linux, Windows, or macOS system. Even though
this device is highly specialized, model deployment is performed by compiling
a TensorFlow Lite model containing a subset of supported layers (tpu, 2022).
This model must be quantized to an 8-bit integer, as this is only supported
data type by this device. This compiled model can then be used with the TPU’s
Python or C++ library. In our experiments, we rely on the C++ library.

We evaluated this device in two formats:

  1. (1)

CoralAI Dev Board Mini, which couples this chip and an SoC on a single board.

  2. (2)

CoralAI USB accelerator connected to a Raspberry Pi 2B.

Even though (1) has the TPU chip on the same board as the SoC, the two
communicate through the USB2 bus, as in the case of (2). In addition, both of
the devices run Linux. The parameters of the different devices can be found in
Table 4.

|  CoralAI Dev Board Mini |  Raspberry Pi 2B
---|---|---
CPU |  Quad-core Arm Cortex-A35 @ 1.5 GHz |  Quad-core ARM Cortex-A7 @ 900 MHz
RAM |  2 GB |  1 GB
Storage |  8 GB eMMC |  32 GB SD card
Dimensions |  64 x 48 x 14.6 mm |  85.6 x 56.5 x 17 mm
Weight |  25.5 g |  45 g

CoralAI TPU |  TOPS |  Interface |  Dimensions |  Weight
---|---|---|---|---
|  4 |  USB2 |  65 x 30 mm |  4.3 g

Table 4. Specifications of the CoralAI Dev Board Mini and Raspberry Pi 2B, as well as the CoralAI TPU chip. Note that the CoralAI Dev Board Mini’s physical dimensions and weight include the on-board TPU. | Scaling factor
---|---
| 0.25 | 0.5 | 1.0
Device | Latency (s) |  Power consumption (mWh) | Latency (s) |  Power consumption (mWh) | Latency (s) |  Power consumption (mWh)
ARM Cortex-M7 | 118.80 |  12.10 | N/A |  N/A | N/A |  N/A
CoralAI Dev Board Mini | 3.778 |  1.76 | 3.991 |  1.90 | 4.533 |  2.40
Raspberry Pi $+$ CoralAI TPU | 3.106 |  1.69 | 3.331 |  1.93 | 3.825 |  2.40
NVIDIA Jetson Nano | 13.900 |  10.54 | 14.109 |  12.74 | 18.581 |  22.10
NVIDIA Jetson Nano, batch size 64 | 3.529 |  3.71 | 5.382 |  5.94 | 10.171 |  12.89
Table 5. Latency and power consumption results with different scaling factors
of the model as measured on corresponding devices. Batch size of 1 is used if
not stated otherwise.

###  4.2. Metrics

The metrics to evaluate the suitability of the devices as the on-satellite
IPUs are based on the requirements Section 3 outlined.

Latency is reported in seconds per inference of a sample, where a sample is a
whole image of size 4512 x 4512 pixels or 400 tiles of size 224 x 224 pixels.
We measure the latency of the neural network inference once the data is
already in the device’s memory.

Nominal power draw is the average power draw in mW multiplied by the duty
cycle, which is the ratio between achieved and required latency for each
scenario (outlined in Section 3). We use an external appliance for CoralAI TPU
and the ARM Cortex-M7 and tegrastats for Jetson Nano to measure this metric.

Peak power draw is the maximum power draw over the period of inference in mW.
It is measured using the same tools as in nominal power draw.

Power consumption is reported per inference of a sample. This metric shows the
power efficiency of the device and will guide the choice of a more suitable
device if multiple devices fulfill the requirements of a particular scenario.

Accuracy of the deployed model is also measured to show the effect
quantization has on the model’s predictive performance.

###  4.3. Workload

To simulate the imaging scenarios of interest (Section 3), we use an image
classification workload consisting of a 5-class classification problem. For
this workload, an off-the-shelf MobileNetV1 model (Howard et al., 2017) pre-
trained on the ImageNet dataset (Deng et al., 2009) was chosen for its strong
predictive power and availability in multiple scaling factors, affecting the
number of hidden layers in the model. The availability of multiple scaling
factors is an important factor as the memory of some of the devices is highly
limited and can therefore fit only the smallest of the variants.

We further fine-tuned the model before deployment on the devices using the
Flowers dataset (Team, 2019), containing 3670 color images of size 224 x 224
pixels belonging to 5 different classes. While the weights in the model do not
affect the inference latency or power required to perform the inference, the
model was fine-tuned using this dataset because it mimics one of the possible
use cases closely (in number of classes as well as size of the images) and
allows us to quantify the effects of the size and precision of the model.

Since the models running on the Cortex-M7 and CoralAI TPU were quantized,
rescaling of images was not needed before inference. For Jetson Nano, the
pixel values must be rescaled to the range of $[0,1)$, or, in other words,
divided by 255. An additional layer was prepended to the model for this
process in order to take advantage of the GPU parallelism on Jetson Nano.

As the size of the images would stress the memory available on the evaluated
devices, we employed a tiling method for inference. Each image was divided
into 400 patches of size 224 x 224.111The image is not evenly divisible;
therefore, the borders are disregarded. This method results in a positive side
effect, where each patch can be filtered separately acting as a coarse-grained
image segmentation. This way only the tiles of interest are sent back to a
ground station saving us bandwidth.

The results are reported as an average of 10 inferences on the full 4512 x
4512 pixel image. The inference on Cortex-M7 and CoralAI TPU were performed
with batch size of 1 and on Jetson Nano with the batch size of $2^{x}$, where
x is from range of 0-6. All of the devices were tested with the scaling
factors 0.25, 0.5, and 1.0 of the model, with the exception of the ARM
Cortex-M7 microcontroller, which could not fit the larger models in memory.

##  5\. Results

The results are split into three parts: (1) analysis of latency and power draw
results with respect to the three scenarios outlined in Section 3, (2) impact
of quantization on the accuracy of models with various scaling factors, (3)
impact of batch size on the performance of NVIDIA Jetson Nano.

###  5.1. Scenarios

Table 5 shows the latency achieved and power consumption of the devices using
the various scaling factors of the model. Furthermore, Tables 6 and 7 show the
nominal and peak power draw of different configurations, respectively. The
Sections 5.1.1-5.1.3 discuss the suitability of each device for the different
scenarios based on the results on these tables. The reported latency, power
consumption, and the peak power draw are independent of the scenarios. Only
the nominal power draw depends on the use case scenarios, due to the duty
cycle (see Section 4.2).

|  | Scaling factor
---|---|---
|  | 0.25 | 0.5 | 1.0
Device |  | Power draw (mW) | Power draw (mW) | Power draw (mW)
ARM Cortex-M7 | Scenario 1 | - | - | -
Scenario 2 | - | - | -
Scenario 3 | 161 | - | -
CoralAI Dev Board Mini | Scenario 1 | 1433 | 1546 | -
Scenario 2 | 89 | 95 | 120
Scenario 3 | 23 | 25 | 32
Raspberry Pi + CoralAI TPU | Scenario 1 | 1376 | 1571 | 1954
Scenario 2 | 85 | 97 | 120
Scenario 3 | 23 | 26 | 32
NVIDIA Jetson Nano | Scenario 1 | - | - | -
Scenario 2 | 529 | 639 | 1109
Scenario 3 | 141 | 170 | 295
NVIDIA Jetson Nano, batch size 64 | Scenario 1 | 3021 | - | -
Scenario 2 | 186 | 298 | 646
Scenario 3 | 49 | 79 | 172
Table 6. Nominal power draw of devices with models of different sizes. The
combinations of devices and model sizes, which do not fulfil the latency
requirements, are omitted as their active duty cycle is greater than 100%. The
power draw is highlighted in bold when exceeding the power budget.

| Scaling factor
---|---
| 0.25 | 0.5 | 1.0
| Device
---
Power draw (mW) |  Power draw (mW) |  Power draw (mW)
Arm Cortex-M7 |  389 |  N/A |  N/A
CoralAI Dev Board Mini |  2770 |  2930 |  4790
Raspberry Pi + CoralAI TPU |  2595 |  3405 |  5050
NVIDIA Jetson Nano |  2871 |  3546 |  4717
NVIDIA Jetson Nano, batch size 64 |  4523 |  4684 |  5064

Table 7. Peak power draw of the devices during inference on full-size image.
Batch size of 1 is used if not stated otherwise. The peak power is in bold
when exceeding the power budget.

####  5.1.1. Scenario 1: Real-time imaging

Real-time imaging is the most constrained scenario. Therefore, a high degree
of specialization is necessary to fulfill the requirements.

Latency. The pipeline has to achieve a 4.42s latency, which is the spacing
between images with 50% overlap. There are no passive periods. Therefore the
latency of the pipeline has to be lower than that of the imaging for the
pipeline not to get backed up. There are multiple configurations that achieved
the required latency (Table 5). These are all the configurations utilizing a
TPU and the smallest model running on Jetson Nano with batch size of 64.

Power draw. Out of the configurations that satisfy the latency requirements
for this scenario, only of the TPU configurations fit into the nominal power
budget of the satellite (Table 6). Even though the Jetson Nano with batch size
64 achieves a latency comparable to the TPUs with a scaling factor of 0.25,
the power consumption per inference is more than twice as high.

Furthermore, the Raspberry Pi with external TPU module does not fit into the
power budget when using the largest model, due to exceeding the 5W
requirements for peak power draw (Table 7). NVIDIA Jetson Nano using the
largest model with batch size of 64 also exceeds this peak power requirement.
Since the peak power draw does not depend on neither the use case scenario nor
the duty cycle, this conclusion about peak power draw holds for the rest of
the scenarios as well.

####  5.1.2. Scenario 2: Arctic region imaging

By relaxing the constraints and performing the workload equivalent to taking
images of only the areas above the arctic polar circle, we see more
configurations passing the requirements necessary to perform such a workload.

Latency. Since there would be passive imaging periods performing the workload
for this scenario, the pipeline does not need to be lower than the latency of
imaging. It can instead buffer the images and perform the inference at higher
latency, given that the pipeline can finish inference of one burst before the
next burst of images comes, which corresponds to a total latency of 5739
seconds or 71.74 seconds per image. All the configurations except the ones
with ARM Cortex-M7 achieve this latency (Table 5).

Power draw. Because of the large margin between the required and achieved
latencies for all configurations passing the latency requirements, the active
duty cycle is relatively low. Therefore, the corresponding nominal power draw
is well below the required one in all cases (Table 6).

####  5.1.3. Scenario 3: Greenland imaging

The least constrained scenario corresponds to taking images of only Greenland.
The low constraints mean that all of the devices pass the requirements under
certain model configurations.

Latency. The maximum total latency for a burst of images is 21600 seconds,
corresponding to 270 seconds per single image. This requirement is easily
passed by all of other configurations.

Power draw. Due to the low active duty cycle, each configuration passes the
nominal power draw requirement (Table 6). There is, however, a large gap
between the nominal power draw of TPU and the rest of the devices. TPU
performs the inference much more efficiently, and its power draw scales more
favourably with increasing scaling factors in comparison to the other devices.

| Scaling factor
---|---
Precision | 0.25 | 0.5 | 1.0
32 bit float | 86.65% | 90.46% | 91.42%
8 bit integer | 83.24% | 90.32% | 91.01%
Table 8. Accuracy of the full-precision and quantized MobilenetV1 as measured
on the Flowers dataset. Figure 1. Latency and power consumption at varying
model scaling factors and batch sizes on the NVIDIA Jetson Nano.

###  5.2. Effect of quantization and scaling factor

Table 8 shows the accuracy of the model with the various scaling factors
before and after quantization to an 8-bit integer. Models with scaling factors
0.5 and 1.0 do not show a significant change in predictive performance. We
can, however, see a 3% drop in accuracy in the model with the smallest scaling
factor. However, this difference is acceptable in our case and is outweighed
by the benefits of lower memory footprint and latency, and higher overall
efficiency.

###  5.3. Batch size impact on NVIDIA Jetson Nano

NVIDIA Jetson Nano is the only device in our evaluation that allows doing
inference with a batch size higher than 1. Figure 1 shows the impact of
increasing the batch size on latency and power consumption of the inference.
The most significant difference is between batch sizes 1 and 8. The impact of
increasing batch size then tapers off. We can see that maximizing the batch
size can lead to 45.3-74.6% lower latency and 41.7-64.8% lower power
consumption.

##  6\. Discussion

TPU shows the most promising results as it is the only device to fulfill the
requirements for real-time imaging, which is the most constrained scenario.
This device is highly specialized for this purpose, utilizing the systolic
array architecture. This architecture is purpose-built to perform fast
multiply-accumulate operations in a highly parallel fashion, which allows
performing matrix multiplication without the need to load/store intermediate
values.

On the other end of the spectrum was the ARM Cortex-M7 microcontroller. Due to
its lack of parallelism, this device could only fulfill the requirements of
the least constrained scenario. Furthermore, the microcontroller has a very
low amount of memory available even after heavy optimizations, such as
quantization of the model and a stripped-down version of the TensorFlow Core.
The low amount of memory means that the device cannot hold the full-size image
in memory. Therefore, in-camera tiling is used, and the results of the
operations had to be buffered to storage before the device could perform
inference on the saved tiles.

NVIDIA Jetson Nano can fulfill the requirements of the real-time imaging
scenario. However, its nominal power draw exceeds the power budget and
therefore buffering, similar to the case of microcontroller, is used to be
able to work at latencies higher than those of the imaging. Even though the
Jetson Nano could match the latency of the devices utilizing a TPU, when doing
inference on large batches of image tiles using the smallest model, it only
could do so at the cost of a higher power draw. Even with a batch size of 64,
the Jetson Nano performs inference with power consumption 112-437% higher
compared to the devices with TPU. The GPU architecture can perform massively
parallel operations, but can only perform matrix multiplication by
loading/storing intermediate results in shared memory, which creates
inefficiency.

##  7\. Conclusion

In this paper, we characterized the performance of three devices that are
possible candidates to be deployed on a satellite focusing on different image
analysis scenarios on the satellite. Our results demonstrated that the low
latency requirements combined with the limited budget for power draw, size,
and mass necessitate highly specialized hardware architectures in this domain.
The only device that fulfilled these requirements for all scenarios was
CoralAI TPU. While NVIDIA Jetson Nano could match its performance thanks to
its GPU, it could only do so at the cost of significantly higher power draw,
which ultimately led to exceeding the power budget. ARM Cortex-M7, in
contrast, could only fulfill the requirements of the least constrained
scenario due to the low degree of parallelism and limited memory it has. Even
though it provided the lowest peak power draw, the high latency of inference
using this device led to nominal power draw higher than any of the other
devices.

## References

  * (1)
  * tpu (2022) 2022\.  _Edge TPU Compiler_.  https://coral.ai/docs/edgetpu/compiler/
  * tfl (2022) 2022\.  _TensorFlow Lite_.  https://www.tensorflow.org/lite/guide
  * Brown (2018) Adam D. Brown. 2018\.  Investigation of Deep Neural Network Image Processing for CubeSat Size Satellites.  _MSc Thesis, Morehead State University_ (2018).
  * Chatziliadis et al. (2021) Xenofon Chatziliadis, Eleni Tzirita Zacharatou, Steffen Zeuch, and Volker Markl. 2021.  Monitoring of Stream Processing Engines Beyond the Cloud: An Overview.  _OJIOT_ 7, 1 (2021), 71–82.
  * David et al. (2021) Robert David, Jared Duke, Advait Jain, Vijay Janapa Reddi, Nat Jeffries, Jian Li, Nick Kreeger, Ian Nappier, Meghna Natraj, Tiezhen Wang, Pete Warden, and Rocky Rhodes. 2021\.  TensorFlow Lite Micro: Embedded Machine Learning for TinyML Systems. In _MLSys_. 800–811.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009\.  Imagenet: A large-scale hierarchical image database. In _CVPR_. 248–255.
  * DISCO (2022) DISCO 2022.  _Danish student cubesat program_.  https://discosat.dk/
  * Esposito et al. (2019) M. Esposito, S. S. Conticello, M. Pastena, and B. Carnicero Domínguez. 2019\.  In-orbit demonstration of artificial intelligence applied to hyperspectral and thermal sensing from space. In _CubeSats and SmallSats for Remote Sensing III_ , Vol. 11131. 111310C.
  * Giuffrida et al. (2020) Gianluca Giuffrida, Lorenzo Diana, Francesco Gioia, Gionata Benelli, Gabriele Meoni, Massimiliano Donati, and Luca Fanucci. 2020\.  CloudScout: A Deep Neural Network for On-Board Cloud Detection on Hyperspectral Images.  _Remote Sensing_ 12 (07 2020), 2205.
  * Goodwill et al. (2021) Justin Goodwill, Gary Crum, James Mackinnon, Cody Brewer, Michael Monaghan, Travis Wise, and Christopher Wilson. 2021.  NASA SpaceCube Edge TPU SmallSat Card for Autonomous Operations and Onboard Science-Data Analysis. In _SSC_.
  * Howard et al. (2017) Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017\.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications.  _CoRR_ abs/1704.04861 (2017).
  * Lai et al. (2018) Liangzhen Lai, Naveen Suda, and Vikas Chandra. 2018\.  CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs.  _CoRR_ abs/1801.06601 (2018).
  * Lofqvist and Cano (2020) Martina Lofqvist and José Cano. 2020.  Accelerating Deep Learning Applications in Space.  (2020).
  * O’Keeffe et al. (2018) Dan O’Keeffe, Theodoros Salonidis, and Peter Pietzuch. 2018\.  Frontier: Resilient Edge Processing for the Internet of Things.  _PVLDB_ 11, 10 (2018), 1178–1191.
  * Team (2019) The TensorFlow Team. 2019\.  _Flowers_.  http://download.tensorflow.org/example_images/flower_photos.tgz
  * Warnke et al. (2022) Benjamin Warnke, Johann Mantler, Sven Groppe, Yuri Cotrado Sehgelmeble, and Stefan Fischer. 2022\.  A SPARQL Benchmark for Distributed Databases in IoT Environments. In _BiDEDE_. 1–6.
  * X-CUBE-AI (2022) X-CUBE-AI 2022.  _AI expansion pack for STM32CubeMX_.  https://www.st.com/en/embedded-software/x-cube-ai.html
  * Zeuch et al. (2020) Steffen Zeuch, Ankit Chaudhary, Bonaventura Del Monte, Haralampos Gavriilidis, Dimitrios Giouroukis, Philipp M. Grulich, Sebastian Bress, Jonas Traub, and Volker Markl. 2020\.  The NebulaStream Platform for Data and Application Management in the Internet of Things. In _CIDR_. 1–11.