# A Transformer Framework for Data Fusion and Multi-Task Learning in Smart
Cities

Alexander C. DeRieux, , Walid Saad, ,
Wangda Zuo, Rachmawan Budiarto, Mochamad Donny Koerniawan, and Dwi Novitasari
This work was funded by the National Science Foundation under Grants
CNS-2025377 and CNS-2241361.A. C. DeRieux and W. Saad are with Wireless@VT,
Bradley Department of Electrical and Computer Engineering, Virginia Tech,
Blacksburg, VA 24061 USA (email: {acd1797, walids}@vt.edu).W. Zuo is with the
Department of Architectural Engineering, Penn State University, State College,
PA 16801 USA (email: wangda.zuo@psu.edu).R. Budiarto and D. Novitasari are
with the Department of Nuclear Engineering and Engineering Physics,
Universitas Gadjah Mada, Indonesia (email<EMAIL_ADDRESS>dwi.novitasari@mail.ugm.ac.id).M. D. Koerniawan is with the Department of
Architecture, Institut Teknologi Bandung, Indonesia (email:
donny@ar.itb.ac.id).We thank Yizhi Yang, John Zhai, Irawan Eko Prabowo, Lily
Tambunan, Nissa Aulia Ardiani, Ferdi Mochtar, and Edward Syarif for the
various discussions within the context of the Makassar project that helped
scope some of the research. 0000-0003-1606-0668 0000-0003-2247-2458
0000-0003-2102-5592 0000-0002-5088-0574 0000-0002-1566-9124
0000-0003-0074-0642

###### Abstract

Rapid global urbanization is a double-edged sword, heralding promises of
economical prosperity and public health while also posing unique environmental
and humanitarian challenges. Smart and connected communities (S&CCs)apply
data-centric solutions to these problems by integrating artificial
intelligence (AI)and the Internet of Things (IoT). This coupling of
intelligent technologies also poses interesting system design challenges
regarding heterogeneous data fusion and task diversity. Transformers are of
particular interest to address these problems, given their success across
diverse fields of natural language processing (NLP), computer vision, time-
series regression, and multi-modal data fusion. This begs the question whether
Transformers can be further diversified to leverage fusions of IoT data
sources for heterogeneous multi-task learning in S&CC trade spaces. In this
paper, a Transformer-based AI system for emerging smart cities is proposed.
Designed using a pure encoder backbone, and further customized through
interchangeable input embedding and output task heads, the system supports
virtually any input data and output task types present S&CCs. This
generalizability is demonstrated through learning diverse task sets
representative of S&CC environments, including multivariate time-series
regression, visual plant disease classification, and image-time-series fusion
tasks using a combination of Beijing PM2.5 and Plant Village datasets.
Simulation results show that the proposed Transformer-based system can handle
various input data types via custom sequence embedding techniques, and are
naturally suited to learning a diverse set of tasks. The results also show
that multi-task learners increase both memory and computational efficiency
while maintaining comparable performance to both single-task variants, and
non-Transformer baselines.

###### Index Terms:

Artificial Intelligence, Multi-Task Learning, Data Fusion, Transformers, Smart
Cities, Internet of Things

##  I Introduction

It is projected that by 2050 nearly 68% of the global populace will live in
urban areas [1]. Rapid urbanization in many countries around the world is
ushered by promises of economical prosperity and increased population health
and wellness. However, these boons pose equal challenges pertaining to
environmental quality and food cultivation, both of which are vital to
sustaining positive growth. Parallel to this expansion is also an increased
emphasis on integrating intelligent technologies to build smart and connected
communities (S&CCs). Sparked by recent advancements in artificial intelligence
(AI) and the Internet of Things (IoT), this convergence of technology and
society is the epicenter of S&CC development. In particular, the coupling of
AI with IoT poses interesting system design challenges pertaining to data
collection, and, most importantly, _data fusion_. Specifically, the question
of how to design intelligent systems that can leverage a fusion of
heterogeneous features arises. This is critical for data-rich locales, such as
S&CC, in which many of the environmental features are captured by IoT, but
remain untapped by AI to learn correlations. Hence, next-generation AI systems
for S&CC environments must be designed to leverage a fusion of heterogeneous
data sources to learn multiple tasks concurrently.

Focusing on the perspective of AI, there have been many interesting
architectural developments to learn from sequence-based information.
Specifically, Transformers [2] have revolutionized natural language processing
(NLP) and neural machine translation (NMT) trade spaces over competitive
recurrent neural networks (RNNs) models (i.e., long short-term memory (LSTM),
gated recurrent unit (GRU), etc.). Moreover, their affinity for sequence-
modeling tasks has also been shown to be applicable in fields of time-series
regression [3, 4, 5, 6, 7], computer vision [8, 9, 10], and multi-modal data
fusion [11, 12]. This applicability across many diverse trade spaces begs the
question whether attention-based architectures can be further extended to
leverage fusions of IoT data sources for heterogeneous multi-task learning in
S&CC trade spaces.

###  I-A Related Works and their Limitations

There has been a recent influx of literature studying the application of
Transformers to time-series forecasting [3, 5, 4, 6, 7, 13], computer vision
[8, 9, 10, 14], and multi-modal data fusion [11, 12] trade spaces. In terms of
time-series forecasting, the authors of [3] propose the so-called Temporal
Fusion Transformer (TFT), which combines LSTM technology with attention
mechanisms to make quantile forecasts across multiple time horizons. In [5],
the authors propose the first time-series Transformer design consisting of
both encoder and decoder blocks to forecast univariate temporal data. In [4],
the authors propose the Informer architecture, which introduces a variation of
the traditional attention mechanism, called ProbSparse self-attention. The
authors in [6] propose an encoder-only Transformer architecture, called the
Time Series Transformer (TST), for performing both multivariate time-series
regression and classification tasks. In [7], the authors develop a further
extension of the forecasting Transformer architecture, called Spacetimeformer,
which considers both the temporal and spatial relationships between sequence
features for multivariate time-series regression. In contrast, the authors in
[13] study the viability of attention mechanisms for the multivariate time-
series forecasting task. Regarding computer vision, the authors of [8] propose
the novel Vision Transformer (ViT)architecture with patch encoding scheme. The
authors of [9] successfully apply Vision Transformer (ViT) architecture to the
agriculture trade space, specifically distinguishing crops from weeds using
aerial images. In [10], the authors also apply the ViT architecture to
agriculture, identifying diseases of casava plants using images of their
leafs. In contrast, the authors of [14] study the validity of vanilla ViT
performance claims. Lastly, regarding data fusion, in [11], the authors apply
Transformers to fuse images and LiDAR representations of space in autonomous
driving tasks. Meanwhile, the authors of [12] propose a novel approach to
multi-modal fusion learning, called Unified Transformer (UniT), which also
doubles as a multi-task learning architecture.

However, approaches pertaining to time-series forecasting only focus on either
univariate regression [3, 5], same-feature multivariate regression [6] (i.e.,
the features used as input are the same as those predicted at the output), or
unique univariate target regression [4, 7, 13] (i.e., multiple input features
and a single unique output feature). In terms of computer vision, while the
results in [15] and [14] demonstrate the overall viability of ViT variations,
and the authors of [9] and [10] prove that ViT can be applied to agriculture,
they only focus on a limited set of images and classes, with the latter only
examining diseases in cassava plants. Regarding data fusion, the limitations
of [11] are two-fold: 1) the reliance on convolutional neural networks (CNNs)
for feature extraction, rather than pure Transformer networks, and 2) the
similarity between image and LiDAR data sources, both of which represent
visual and spatial information. The former being less computationally
efficient, and the latter lacking in feature diversity. Further, the work in
[12] only focuses on image and text inputs, which are not representative of
the environmental and temporal features present in S&CCs. Hence, there is a
need for novel approaches which not only address these issues, but also
consider their broader impact on the global community through application in
S&CC environments.

###  I-B Contributions

The main contribution of this paper is the design of a novel Transformer-based
AI framework for generalized IoT data fusion and multi-task learning in S&CCs.
Our framework is built upon a backbone of purely Transformer encoder layers,
and further customized through interchangeable input embedding and output task
heads. This enables system deployments to learn from a variety of
heterogeneous input data types from IoT sources and output task types present
S&CCs environments. Moreover, the extendable nature of the output heads
facilitates learning multiple tasks parallel. We consider five unique problem
sets for our framework, which were chosen to simulate the problem diversity in
smart urban environments; these are 1) multivariate time-series forecasting of
localized meteorological data, 2) plant health and disease classification
using images, a fusion of both 3) multivariate time-series data and 4) images
to perform individual forecasting and plant health classification tasks, and
5) a fusion of these same inputs to perform multi-task learning in parallel.
Deployments of our framework are split across several input type (i.e.,
single-input, and multi-input) and task type (i.e., single-task, and multi-
task) regimes. In the _single-input single-task_ regime, a Transformer model
is associated with each task and learns it using a single data source (i.e.,
time-series, images). In the _multi-input single-task_ regime, a Transformer
model is associated with each task, and learns it using a fusion of image and
time-series data sources. Lastly, in the _multi-input multi-task_ regime, a
single Transformer model is used for all tasks, which it learns through input
data fusion. All of our source code is publicly available on
GitHub111https://github.com/news-vt/makassar-ml.

We run extensive experiments to evaluate the performance of our framework. The
results show that the proposed Transformer-based system can indeed handle
various input data types via custom sequence embedding techniques. Our results
also demonstrate that the Transformer system is well-suited to learning
heterogeneous task sets. We also compare our Transformer-based models against
competitive non-Transformer variants for each task. Specifically, our
comparisons show that our proposed framework either maintains or exceeds the
performance capabilities of non-Transformer baselines. In addition, we observe
that multi-task learners exhibit both increased memory and computational
efficiency while also maintaining comparable performance to both single-task
variants and baselines. This affinity demonstrates the flexibility of
Transformer networks to learn from a fusion of IoT data sources, their
applicability in S&CC trade spaces, and their further potential for deployment
on edge computing devices.

The rest of this paper is organized as follows. Section II presents the system
model. In Section III, we propose Transformer architectures for multi-variate
time-series regression, visual plant disease classification, single-task data
fusion, and heterogeneous multi-task learning tasks. Section IV provides
simulation results. Finally, conclusions are drawn in Section V.

##  II System Model

We consider the design and deployment of an AI framework within S&CC
environments in which there are networks of IoT sensors. These sensors collect
a wealth of visual, spatial, and ecological data from within the city. This
information paints a picture of how the city is jointly effected by these
factors, and their relationship can then be used to bolster positive growth
and development.

A real-world example of this kind of environment is Makssar City, Indonesia.
Makassar is the 5 largest urban center in Indonesia, with a population of 1.7
million [16], and aims to become a world-class metropolis through combining
both IoT and AI to transform many of its 7,520 alleyways into smart and
sustainable gardens. Supported by the U.S. National Science Foundation (NSF)
and the U.S. Department of State, the Smart Garden Alley
Project222https://sites.psu.edu/sbslab/research/city/smart-garden-alleys/
proposes to develop _smart_ garden alleys following a biomimetic philosophy;
this represents the garden alleys, IoT sensor networks, and AI-informed
government policy as a collection of cells, nerves, and a brain.

We formulate the design of an AI framework for S&CC environments, like
Makassar City, as problems of both _data fusion_ and _multi-task learning_.
The variety of data available to IoT sensor networks necessitates the design
of AI that can both input diverse feature types, such as visual and
ecological, while also learning relationships between them. Likewise, the
variety of tasks that can be performed using the IoT data, such as
classification and forecasting, further necessitates AI that can not only
implement these heterogeneous tasks, but also learn the relationships between
them.

###  II-A Heterogeneous Data Fusion

We now formulate the problem of heterogeneous data fusion in S&CC
environments. Next-generation AI systems for S&CC must learn to leverage a
fusion of IoT data sources to perform necessary tasks. This facilitates
learning relationships between loosely-correlated environmental
characteristics, and encourages generalization through diverse fusion of these
features. However, the design of such AI systems which use these diverse
feature sets is non-trivial. In particular, special considerations must be
made for both data preprocessing and the overall AI architecture.

With regards to preprocessing, the frequency of each input feature set matters
significantly for ingestion into the model. Consider for example time-
delimited meteorological data as one feature set, which may have an hourly
frequency. If the task necessitates fusion with images, the question arises of
how to combine static image frames with this time-delimited data. The key
point here is that, while many algorithmic approaches to this fusion are
viable, special consideration must be made to formulate a preprocessing
algorithm in general. This in turn adds increased layers of complexity to the
overall system design.

With regards to the overall AI architecture, several considerations must be
taken into account when fusing diverse feature sets. Extrapolating from the
aforementioned data frequency concern, this also affects the design of model
input layers as well as any embedding schemes. In particular, traditional AI
design assumes inputs to be supplied in discrete increments, and the frequency
the data affects the overall input shape. Consider the previous example in
which time-delimited data is fused with images. If static images are supplied,
then an input layer must be designed to accept just a single image. However,
if the preprocessing algorithm instead produces multiple images in a buffer,
then the design of the input layer must change to support a sequence of
frames. Likewise, knowledge of the input shape will affect the downstream
embedding schemes, which could completely change the AI architecture. For
example, single-image embedding could pool the matrix pixels, whereas multi-
image embedding could instead project the sequence of frames. The key point is
that the fusion of diverse feature sets is non-trivial, and affects the entire
AI architecture design.

###  II-B Multi-Task Learning using Attention

Now, we formulate the problem of multi-task learning of diverse task sets in
S&CC environments using attention-based Transformer architectures. Highly-
performant AI language models such as BERT [17], XLNet [18], and GPT-3 [19]
all use attention mechanisms and Transformers as their baseline architecture.
The dominance of Transformers has extended beyond fields of NLP, finding
applications in computer vision with models such as the ViT [8], and time-
series forecasting with models such as the Temporal Fusion Transformer (TFT)
[3] and Informer [4]. However, all of these models are designed with a
_singular input data type_ in mind (i.e., text, images, time-series, etc.). In
the presence of multiple data types, none of the aforementioned architectures
alone can learn correlations between the heterogeneous feature sets. Consider
now our system model of IoT sensor networks in urban environments, as shown in
Fig. 1. There exists many sensory features from these devices, such as
meteorological information, traffic camera video feeds, and citizen mobility
from wireless networks that can be leveraged to bolster smart city growth and
development. In addition to diverse feature sets, each data collection is
often also associated with more then one task. Consider for example the fusion
of time-series meteorological data and images of plants as discussed in
Section II-A. Using both inputs, it is possible to perform multiple tasks such
as forecasting meteorological events, predicting necessary irrigation changes
to promote plant growth, and even identifying the onset of disease. Next-
generation AI systems for deployment in S&CC environments must be able to
leverage diverse feature sets such as these to make more tailored decisions.
As such, one key goal of this paper is to design and implement an AI framework
which learns to S&CC growth using a fusion of heterogeneous features.

Figure 1: An example of Data Fusion Learning in smart city environments. Input
features such as city maintenance, solar radiation, air quality, fertilizer,
precipitation, and images can fused to learn plant growth, disease
identification, and meteorological forecasting tasks.

In light of the Transformer’s affinity across diverse trade spaces, we propose
to exploit them in the context of smart cities, because they are capable of
learning heterogeneous feature correlations for a variety of tasks in parallel
fashion. The Transformer assumes inputs to be of an embedded sequential
nature, but is agnostic to task-specific nuances. This generalizability makes
the Transformer architecture a prime choice for implementing fusion models, as
many data types can be represented using sequence-based embedding schemes.
Hereinafter, we refer to the process of learning relationships between these
diverse feature sets as _data fusion_ , and the art of learning multiple tasks
as _multi-task learning_. Data fusion correlates features from multiple input
sources via timestamp or geographic location and fuses them within the
Transformer network. Multi-task learning extends this relation of input
features to learning several tasks across various trade spaces in parallel.
During training the network optimizes losses for each output task
concurrently, allowing a single model to learn from shared experience and
thereby boosting generalization performance. Several challenges arise in the
process of designing Transformer architectures tailored for data fusion and
multi-task learning. Feature diversity in both dimension and physical
properties are challenges in data fusion. For example, image and meterological
datasets have completely unique shapes (e.g., pixels with RGB color channels,
multivariate time-series, etc.) and value units (e.g., pixels in range 0-255,
hourly temperature in Celsius, etc.). The challenge is designing a fusion
technique which takes these properties into account and learns across
appropriate sequence dimensions. Likewise, task diversity is a challenge in
multi-task learning. The challenge is designing an AI architecture such that
its core framework supports the diverse dimensionality requirements of the
output tasks (e.g., RGB image generation, 24-hour meteorological forecast,
etc.) and capability for various multivariate projections (e.g., sinusoidal,
exponential, etc.). Next-generation AI systems for S&CC deployments can
greatly benefit from both data fusion and multi-task architecture design
choices to leverage the wide array of IoT sensor data available.

In the following section, we address the aforementioned challenges and propose
Transformer architectures for data fusion and multi-task learning.

##  III Transformer Networks for Smart Cities

We propose five Transformer-based architectures that are tailored specifically
to smart city task sets. In particular, we focus on five specific supervised-
learning problems which simulate the tasks encountered in S&CCs, like Makassar
City. These tasks are divided into two different categories for _single-task_
and _multi-task_ learning.

In the single-task category, we first formulate a supervised multivariate
time-series regression task. Specifically, unique input features within a
historical time window are used to predict a separate set of features along a
finite-time horizon. Second, we formulate a plant health identification task
using images labeled with various healthiness categories (i.e., healthy,
diseased, etc.). In this task, a single image is used as input to predict a
class label. Third, we formulate a variation of the regression task that fuses
both time-series and image datasets to enhance performance. In this task, a
model accepts both a single image and a window of multivariate time-series
data as input. The model then uses this information to predict a unique set of
output features along a finite-time horizon, similar to the single-input case.
Fourth, we formulate a plant health classification task using a fusion of the
aforementioned datasts. Here the same image and time-series window are used as
input, whereas now the model predicts the probability distribution over a
finite set of class labels.

In the multi-task category, we extend the single-task data fusion problems to
perform both heterogeneous tasks simultaneously using a single model.
Specifically, we fuse both static images and multivariate time-series windows
as input, and use this information to predict both the healthiness class label
and a set of output features along a finite-time horizon. This configuration
allows a single AI model to optimize multiple tasks in parallel, while also
benefiting from shared experience.

###  III-A Forecast Transformer for Multivariate Regression

We propose the Forecast Transformer (FoT) architecture for supervised time-
series regression of multi-input and multi-output feature spaces. This model
was developed in conjunction with an application to the financial trade space
in mind. Specifically, forecasting stock market asset valuations using
multiple unique feature sets. Naturally, this can be directly applied to our
smart city problem, as many tasks involve the computation of arbitrary output
values as a function of some input. In the case of urban farming, the input
could, for example, be multivariate time-series meteorological data, which is
then used to predict output features such as plant irrigation requirements and
expected crop yield. A key point of the FoT design is that it is agnostic to
any specific regression task. Specifically, a clear advantage of this
architecture over alternative competitive models is its portability to
multiple trade spaces, given generalized time embedding schemes, and its
ability to learn arbitrary multivariate sequence relations.

At its core, the FoT architecture uses a pure-encoder approach to time-series
regression. Time-series relations are learned through encoding a multivariate
input sequence through cascaded TransformerEncoder layers. These layers employ
multi-head self-attention mechanisms [2] to learn both cross-feature and
cross-sequence-index relations within the given input sequence. This is in
contrast to the traditional NLP Transformer design, which uses both encoder
and decoder layers for sequence-to-sequence translation. In addition, the
model employs a Time2Vec (T2V) [20] embedding layer, which learns a vector
representation of the input time sequence. The architecture is made dynamic
via hyperparameters which control the length of the input sequence, number of
input features, number of output features, number of fully-connected sub-
layers, embedding dimension, number of cascaded encoder layers, encoder point-
wise feed forward dimension, and number of self-attention heads per encoder.
The design of the FoT architecture is shown in Fig. 2.

Figure 2: FoT model architecture. Blocks without names are Dense (orange),
GeLU activation (gray), Linear activation (green), and Dropout (red).

####  III-A1 Multivariate Time2Vec Embedding

Historically, RNNs have been the tool of choice for sequence-based modeling
and regression. The auto-regressive nature of the general RNN design allows it
to learn inter-element relations using an element’s index, or more generally
location, within a given input sequence. In the case of Transformers, entire
sequences are processed at once in Attention layers via matrix dot products.
This means that the sequence information gathered via auto-regression in RNNs
is lost in the presence of Attention-based networks. To overcome this issue
for NLP-related tasks, the original Transformer architecture adopted a fixed
position-encoded embedding scheme, which represents an element’s position
within a sequence as a sinusoid of varying frequencies [2]. In working with
time-series data, however, this encoding scheme fails to capture the time
information of an element relative to the overall dataset (i.e., hourly,
daily, or monthly frequencies). We overcame this issue by implementing a
variation of the T2V embedding scheme proposed in [20], which we call
_multivariate Time2Vec (MT2V) embedding_. This embedding scheme performs two
tasks within the network. First, it acts as an embedding layer that represents
the input features as vectors in higher-dimensional space. Second, it captures
the real time periodicity of each input feature by learning both linear and
periodic components and using these as the vector embedding; akin to a learned
Fourier Series representation. Mathematically, our MT2V embedding scheme
extends the definition of [20] to multivariate input vectors, instead of
simple scalar inputs. Given an input feature vector $\boldsymbol{\tau}$ with
$F_{\textrm{in}}$ feature dimensions, and a total embedding dimension $k$, the
embedding scheme MT2V of $\boldsymbol{\tau}$, denoted as
$\boldsymbol{MT2V}(\boldsymbol{\tau})$, is a 2-dimensional matrix defined as

$\displaystyle\boldsymbol{MT2V}_{ij}(\boldsymbol{\tau})=\begin{cases}\boldsymbol{\omega}_{ij}\boldsymbol{\tau}_{i}+\boldsymbol{\phi}_{ij},&\text{if
}j=0,\\\
\mathcal{F}(\boldsymbol{\omega}_{ij}\boldsymbol{\tau}_{i}+\boldsymbol{\phi}_{ij}),&\text{if
}1\leq j<k,\end{cases}$ (1)

where $\boldsymbol{\omega}$ is a learned frequency matrix with shape
$(F_{\textrm{in}},k)$, $\boldsymbol{\phi}$ is a learned phase shift matrix
with shape $(F_{\textrm{in}},k)$, $\mathcal{F}$ is a periodic function (i.e.,
$\sin{}$, $\cos{}$, etc.), and $\boldsymbol{MT2V}_{ij}(\boldsymbol{\tau})$ is
the $i$’th feature and $j$’th embedding dimension of the learned embedding
matrix with shape $(F_{\textrm{in}},k)$ for the given input feature vector. In
practice, the feature and embedding dimensions are flattened to create a
single vector with $F_{\textrm{in}}\times k$ dimensions. This allows the
embedding to be repeated along the sequence dimension $S_{\textrm{in}}$ to
create a final sequence embedding matrix with shape
$(S_{\textrm{in}},F_{\textrm{in}}\times k)$, which is more easily ingested
into the Transformer attention layers. The final step of the embedding
process, prior to feeding into the Transformer encoder layers, is to
concatenate the original batched sequence input with shape
$(B,S_{\textrm{in}},F_{\textrm{in}})$ with the MT2V embedding output,
resulting in a final encoder input with shape $(B,S_{\textrm{in}},D_{e})$,
where $D_{e}=F_{\textrm{in}}(1+k)$ is the inner model dimension of the
Transformer encoder.

####  III-A2 Transformer Encoder Layers

The core of the network is the Transformer Encoder, which is comprised of
cascaded TransformerEncoder layers. The design of these layers is similar to
those proposed in [20], with some minor changes to be compatible with time-
series data. Each layer consists of a two concise sub-layers. The input to the
encoder layer is a matrix with shape $(B,S_{\textrm{in}},D_{e})$, where $B$ is
number of batches, $S_{\textrm{in}}$ is the input sequence length, and $D_{e}$
is the inner model dimension (derived from the MT2V embedding output shape).
In the first sub-layer, the encoder input passes through a multi-head self-
attention layer [20], which learns, or “attends” to, cross-element and cross-
feature importance of the input. The output of this attention layer (also a
matrix of with shape $(B,S_{\textrm{in}},D_{e})$) passes through a dropout
regularization layer with probability $P_{\textrm{drop}}$, and then gets added
to the original encoder input to form a residual and normalized using a
LayerNormalization layer. The output of this normalization passes through the
second sub-layer, which consists of a point-wise feed forward network (called
PointWiseFeedForward), followed by another sequence of dropout, residual
addition, and layer normalization. The point-wise feed forward network acts as
a cascaded series of convolutions with a kernel size of 1, which upscales and
then downscales the attention output into separate feature dimension
$D_{\textrm{ff}}$ respectively. The number of encoders $E$, number of self-
attention heads $h$, embedding factor $k$, feed-forward dimension
$D_{\textrm{ff}}$, and dropout regularization rate $P_{\textrm{drop}}$ are
controlled via hyperparameters.

####  III-A3 Fully-Connected Regressor

Up to this point in the network, the model has learned a time vector embedding
and cross-feature relations through attention mechanisms in the encoder
layers. To actually perform multivariate regression, the encoder output
(recall having shape $(B,S_{\textrm{in}},D)$) must be mapped to the desired
output sequence length and number of output features. In this paper we only
consider output sequences with length 1 (i.e., 1 day in the future) to
simplify the architecture design. However, it should be noted that the design
could be extended to support longer sequences in future works. The output
feature mapping can therefore be achieved using a single fully-connected layer
with linear activation as the final layer of the model. This final regressor
layer maps the result of the previous fully-connected layers to the desired
output feature space (akin to a classification layer in classifier networks,
sans softmax activation).

###  III-B Vision Transformer for Plant Health Identification

We propose a variation of the well-known ViT [8] architecture for computer
vision tasks in smart city environments. This facilitates learning image
features using attention mechanisms, which maintains a similar architecture
structure to that of the aforementioned time-series regression task, and also
has been known to exceed performance over competitive CNN models.

Similar to the FoT, the ViT also leverages the same underlying pure-encoder
Transformer architecture for learning image feature correlations. The primary
difference between them is in the underlying embedding scheme. Images are
single static objects, but the Transformer attention mechanism requires
sequence-based input to learn relationships. To make an image compatible as
input, the ViT applies a patch extraction algorithm which converts a static
image into a sequence of small square subset frames akin to a checkerboard.
These patches are then projected (embedded) into a higher dimensional space
matching the input dimension of the cascaded TransformerEncoder layers. The
output of the encoders is then passed through a cascaded sequence of fully-
connected layers, and subsequently a final dense classification layer for
classification tasks. The ViT architecture is made dynamic via multiple
hyperparameters which control the number of patches, patch shape, number of
fully-connected sub-layers, embedding dimension, number of cascaded encoder
layers, encoder point-wise feed forward dimension, and the number of self-
attention heads per encoder. The design of the ViT architecture is shown in
Fig. 3.

Figure 3: ViT model architecture. Blocks without names are Dense (orange),
GeLU activation (gray), Linear activation (green), and Dropout (red).

####  III-B1 Image Patch Generation

Historically, CNNs have been used as the standard for image-based machine
learning (ML) tasks. Convolution-based networks were originally proposed by
[21] for document character recognition in 10.1109/5.726791, and then
popularized in NIPS2012_c399862d by [22] for successful application to object
classification tasks. CNN architectures take as input source images with 3
dimensions of $(W,H,C)$, where the former two refer to the image width and
height, and the latter the number of color channels (i.e., $C=1$ for grey-
scale, $C=3$ for RGB, etc.). Convolutional layers within a CNN learn to
extract features from source images by performing matrix dot-product
operations on small windowed regions commonly referred to as a kernels. These
kernels are convolved with the input image using various settings for padding
and stride offsets which filter the pixels within the region down to a single
value. Multiple kernels can be applied in parallel to project the resulting
convolved image into higher or lower channel dimensions, thereby varying the
number of features that can be extracted from a single image.

In contrast, the ViT architecture learns image features by first splitting a
source input image into multiple smaller regions called patches. The algorithm
generates patches similar to a convolution. Patches are extracted using a
kernel with shape $(H_{p},W_{p})$ which copies source pixels within its
region. This kernel is then convolved with the input image using a stride
offset to pad the distance between each region from its nearest neighbors. The
patch extraction operation results in a output matrix with shape
$(N_{\textrm{row}},N_{\textrm{col}},H_{p},W_{p},C)$, where
$(N_{\textrm{row}},N_{\textrm{col}})$ are the number of rows and columns in
the resulting patch grid, $(H_{p},W_{p})$ are the width and height of each
patch, and $C$ is the number of channels in the original input image. In
practice, the patch matrix for a single image is flattened into shape
$(S_{p},D_{p})$, where $S_{p}=N_{\textrm{row}}\times N_{\textrm{col}}$ serves
as the sequence dimension for the Transformer encoder, and $D_{p}=H_{p}\times
W_{p}\times C$ serves as the patch projection dimension. Knowing the original
patch size and number of color channels allows the user to extract the
resulting patches individually for closer inspection.

To better understand the patching algorithm a simple example is shown in Fig.
4. In this example, an arbitrary input image with shape $(6,6,1)$ is passed to
the extraction algorithm using a patch shape of $(2,2)$ and stride of $(4,4)$.
The image is split into 4 unique patches which are organized into a matrix
with shape $(2,2,2,2,1)$, where the first 2 dimensions are the patch grid
$(2,2)$, and the latter 3 are the patch image $(2,2,1)$.

Figure 4: Example of patch extraction algorithm. An input image (left: gray)
with shape $(6,6,1)$ pixels is split into 4 patches (middle: red, green, blue,
yellow) using patch shape of $(2,2)$ and stride of $(4,4)$. The patches are
coalesced into a grid (right) of shape $(2,2,2,2,1)$.

Further, a demonstration of the patch extraction algorithm on a real image is
shown in Fig. 5. Here, an RGB image with shape $(256,256,3)$ is used as input.
Using a patch size of $(18,18)$, and a stride offset of $(18,18)$, the image
is split into 196 unique patches along a $14\times 14$ grid333The number
patches along each row or column is computed as
$N_{\textrm{row}}=\lfloor\frac{H}{H_{p}}\rfloor$ and
$N_{\textrm{col}}=\lfloor\frac{W}{W_{p}}\rfloor$ respectively. Notice that,
since the patch size and stride offset are the same, the resulting patches
will completely cover the original image with no omitted pixels.

Figure 5: Demonstration of patch extraction algorithm result. Input image
(left) with shape $256\times 256$ pixels is split into 196 patches (right,
$14\times 14$ grid) each with shape $18\times 18$ pixels.

To incorporate the patch extraction algorithm into the ViT architecture, we
designed a static TensorFlow layer called Patches. The advantage of this
approach is that the layer can be directly inserted into model build or even
preprocessing pipelines to dynamically extract patches from batches of input
images. The layer is made dynamic through hyperparameters such as patch size
and stride offset which tune the resulting patch dimensions. Note that, by
default, the patch size and stride offset are the same so as to have complete
coverage of the input image.

####  III-B2 Image Patch Embedding

Before image patches can be fed into the TransformerEncoder layers they must
first be projected into the same inner dimension of the encoder. Recall that
output of the patch extraction algorithm for a single image is a matrix of
shape $(S_{p},D_{p})$. However, the Transformer encoder requires the input to
have dimension $(S_{p},D_{e})$, where $D_{e}$ is the encoder inner dimension.
Therefore, an embedding scheme must be applied which projects the patch
dimension $D_{p}$ to the encoder dimension $D_{e}$. In addition, the embedding
must also incorporate the sequential information of the patches as the
Transformer attention mechanism processes all sequence input concurrently. To
accomplish this, we built a custom PatchEncoder layer which does both tasks.
First, a fully-connected layer is used to learn the projection $D_{p}\to
D_{e}$. Second, a relative positional embedding is added to the projection to
encode the sequence information for each patch. The result is a matrix of
shape $(S_{p},D_{e})$ which can be fed directly into the Transformer encoder
layers.

###  III-C Fusion Transformer for Multi-Task Learning

Up to this point, our proposed model designs used each input data set
independently to perform a single task. In particular, FoT only uses
multivariate sequential data for regression, and ViT only uses images for
classification. This compartmentalization of data and tasks results in several
AI models being required for a single system. Depending on the environment,
this separation may be required. In IoT environments, however, where the data
is loosely-correlated (i.e., weather and time-stamped images), it would be
more efficient for a single model to input a fusion of heterogeneous input
data sets and learn to perform multiple tasks in parallel. Hence, we propose
the Fusion Transformer (FuT), which combines the techniques of the
aforementioned FoT and ViT architectures for both single-task data fusion and
multi-task learning. Data fusion is critical in IoT applications where data
collection sources are plentiful, and the data itself is often time-
correlated. Examples of such data in smart city environments include weather
data from meteorological sensors, and images from video cameras placed on
street corners. Using the example of Makassar City’s smart garden alleys, the
FuT architecture facilitates building an AI nerve-center for the city that can
leverage a fusion of the various IoT data sources at its disposal to forecast
plant health and environmental conditions.

The core of the FuT model is a Y-network where each branch acts as an encoding
pipeline that is uniquely associated with each input data source. The output
of these pipelines are fed into task-dependent output blocks, referred to as
task heads, that fuse the learned encodings for all input data sources and
applies them to learn a specific task (i.e., classification, regression,
etc.). A high-level view of the FuT architecture is shown in Fig. 6. Note that
the number of input embedding heads and Transformer encoder pipelines $n$ are
tightly coupled, meaning that each input head is directly linked to a specific
pipeline. In contrast, the number of task heads $m$ is independent of the
input pipelines, and can therefore be chosen relative to the desired tasks.

Figure 6: High-level outline of the FuT model architecture for data fusion
learning. Each input source has its own embedding head (gray) and Transformer
encoder pipeline (blue). The outputs of each encoder pipeline are then passed
through task-specific output head (green). The number of embedding head and
encoder pipeline pairs $n$ is dependent on the amount of input desired, and
the number of task heads $m$ is dependent on the application.

The interchangeable nature of the task head was specifically designed with
transfer learning applications in mind. Transfer learning is a process by
which a sub-section of a pre-trained model is grafted into a new architecture
for a potentially different application [23]. This means that the new overall
model can benefit from the experience that the pre-trained stub has learned.
In FuT, the encoder pipelines for each input branch can be pre-trained and
grafted into a new output head to learn a different task. Further, FuT
supports multi-task learning frameworks with the option for multiple output
task heads. This emphasis on both transfer learning and multi-task learning
offers many benefits to S&CCs as its needs and requirements change over time.
For example, initial models could fuse time-series meteorological data and
images of plants to forecast irrigation requirements. If the city then desires
to add the capability of identifying plant diseases, a classification head can
be appended to the pre-trained base model to accomplish both tasks
simultaneously.

####  III-C1 Input Embedding Heads

Each branch of the Y-network is dedicated to encoding a specific input data
source using cascaded TransformerEncoder layers. The input to each pipeline is
a pre-embedded 2-dimensional input sequence. This generalization decouples the
input data embedding scheme from the model, thereby making it agnostic to any
specific input data sources. It is therefore the job of data-specific input
heads to perform embedding and reshaping operations which ultimately conform
to the encoding pipeline format. Gleaning from techniques used in ViT and FoT
Transformers, we extract the embedding sections of these networks and
encapsulate them as self-contained embedding layers for image and multivariate
time-series data respectively. Examples of these embedding heads are shown in
Fig. 7.

(a) Images

(b) Time-series

Figure 7: Examples of FuT input embedding heads for (a) images and (b)
multivariate time-series datasets.

The image embedding head in Fig. 7a accepts 3-dimensional images of shape
$(H,W,C)$ and extracts patches according to the algorithm discussed in Section
III-B1. These patches are then reshaped into a 2-dimensional matrix and
embedded according to the technique described in Section III-B2. Note that
here we also add a positional encoding to capture the sequential nature of the
image patches in the attention mechanism. This produces a 2-dimensional matrix
of shape $(S_{p},D_{e})$, where $S_{p}$ is the number patches and $D_{e}$ is
the patch embedding dimension. The newly-encoded patches are then passed
directly to an associated Transformer encoder pipeline to learn feature
relations.

The time-series embedding head in Fig. 7b accepts 2-dimensional multivariate
sequence data of shape $(S_{\textrm{in}},F_{\textrm{in}})$, where
$S_{\textrm{in}}$ is the sequence length and $F_{\textrm{in}}$ is the number
of input features. These features pass through a MT2V embedding layer, as
discussed in Section III-A1, which represents the time-series information as a
Fourier series. This Fourier series representation is then combined with the
original input via a concatenation layer along the feature dimension to form a
2-dimensional matrix of shape $(S_{\textrm{in}},F_{\textrm{in}}(1+k))$, where
$k$ is the MT2V embedding factor. Similar to the image embedding head, the
output of the time-series embedding head is also passed directly to an
associated Transformer encoder pipeline to learn feature relations.

A particular point about these input heads is the shape of its associated
output. Each produces a 2-dimensional output matrix with shape $(S,F)$, where
$S$ is the sequence length and $F$ is the number of embedding features. Notice
that the sequence length and feature size are tightly coupled with the input
data. This standardization to a 2-dimensional form ensures that all embedding
outputs are compatible with Transformer encoder pipelines, which assume
2-dimensional input matrices.

####  III-C2 Data Encoding Pipelines

After the input data passes through each respective embedding head it is fed
through a cascaded sequence of TransformerEncoder layers that are unique to
the branch. The operation of these layers is similar to those discussed in
Section III-A2. It is important to note that Transformer encoders maintain the
dimensionality of their input data; meaning that the shapes of both input and
output matrices are identical. In FuT, each branch maintains the original
shape of its embedding head. This is important because, depending on the
embedding scheme, the resulting length and number of features for each
sequence could be different. This approach was chosen, however, to make the
encoders for each branch unique to the sequences they learn. It is then the
job of the following task-specific output head to choose how the sequences are
fused.

####  III-C3 Task-Specific Output Heads

After the input data passes through its respective Transformer encoder
pipelines, the output of each pipeline is then fused together at the final
block of the network, referred to as a task head. The task head is
interchangeable based on the desired ML task, and multiple heads can be
combined in parallel to build a multi-task learning framework. This
interchangeable nature of input heads and ouput heads provides sufficient
generalizability to learn the variety of tasks present in smart city
environments.

As input, the task head has access to all encoder pipeline outputs at its
disposal. The core Y-network learns encoded forms of the original input, and
the self-contained task head learns how to fuse these encodings to learn a
specific task. Separating the pipelines like this gives the task head full
control over how the learned encodings are fused together, which further
generalizes the architecture design.

In Fig. 8, we give examples of designs for both classification and regression
task heads. Note the similarities between ViT and FoT architectures, as
discussed in Sections III-A and III-B respectively. The final task-specific
layers of these models have been encapsulated within interchangeable blocks.
This allows encoded forms of both input data types to be fused to enhance
performance for each task.

(a) Classification

(b) Regression

Figure 8: Examples of FuT output heads for (a) classification and (b)
regression tasks. Blocks without names are Dense (orange), ReLU activation
(green), and Dropout (red).

The classification head in Fig. 8a takes the outputs from $n$ encoder
pipelines with shapes $(S_{0},F_{0}),\dots,(S_{n},F_{n})$, flattens them down
to 1-dimensional vectors, and concatenates them resulting in a single vector
with shape $(S_{0}\times F_{0}+\dots+S_{n}\times F_{n})$. This vector is then
passed through a fully-connected network with ReLU activation and dropout
regularization. The final step is a dense classification layer with softmax
activation to compute probabilities for each class.

The regression head in Fig. 8b also takes the outputs from the $n$ encoder
pipelines with shapes $(S_{0},F_{0}),\dots,(S_{n},F_{n})$ and projects each
independently into a common feature dimension $D_{\textrm{fusion}}$. This
results in $n$ sequences with shapes
$(S_{0},D_{\textrm{fusion}}),\dots,(S_{n},D_{\textrm{fusion}})$. This
projection is critical as it allows the encoder outputs to be combined along
the sequence dimension to form a unified encoded matrix with shape
$((S_{0}+\dots+S_{n},D_{\textrm{fusion}})$. This matrix is flattened and then
passed through a fully-connected regression layer with linear activation to
compute the output features with dimension $F_{\textrm{out}}$.

There are two important points about these two example task heads. First,
notice that both use the data from all encoder pipelines in different ways.
Specifically, the classification task head in Fig. 8a flattens each pipeline
matrix prior to concatenation, whereas the regression head in Fig. 8b first
projects the features of each pipeline matrix to a common dimension and then
concatenates these projections along the sequence dimension to form a single
unified sequence. The second point is that the outputs of each head are unique
and independent of the others, which is subtle yet critical. This means that
the model can learn heterogeneous tasks simultaneously using the same input
data. In addition, this design also provides the flexibility for the AI system
to grow in conjunction with the smart city. This means that as the needs of
the city evolve over time, so can the AI system with the capability for
interchangeable output heads. If new tasks are required in the future, custom
task heads can be created to generate the necessary output.

##  IV Simulation Results

For our simulations, we examine the performance of the FoT, ViT, and FuT
architectures presented in Sections III-A, III-B and III-C respectively. In
Section IV-A, we detail a multivariate time-series regression task to
demonstrate the performance of the FoT model in comparison with a non-
Transformer baseline. In Section IV-B, we detail a plant disease
identification task to demonstrate the performance of the ViT model, and also
define a non-Transformer baseline for comparison. In Section IV-C, we detail
two fusion learning experiments, combining images and time-series data for
regression and classification tasks, and demonstrate the performance of the
FuT model with a _single_ output head (i.e. single-task mode). In Section
IV-D, we detail an experiment for multi-task learning using the fusion of
images and time-series data to demonstrate the performance of the FuT model
with _multiple_ output heads (i.e., multi-task mode). Finally, in Section
IV-E, we compare model performance across all tasks. All models were trained
for 30 epochs and a batch size of 256 on Virginia Tech’s Advanced Research
Computing (ARC) high-performance computing (HPC) cluster.

###  IV-A Multivariate Time-Series Forecasting

In this task, we consider the Beijing PM2.5 [24] dataset, with a history
window of 24 hours to predict a horizon of 1 hour. These windows were selected
due to their small size (i.e., easier to train) and relative periodicity
(i.e., consider what occurred in the past day to predict 1 hour into the
future). Specifically, we train the FoT model to take an input sequence of
$S_{\textrm{in}}=24$ consecutive hours with
$F_{\textrm{in}}=\\{\texttt{TEMP},\texttt{DEWP},\texttt{PRES},\texttt{Iws}\\}$
input features per hour, which are _temperature_ , _dew point_ , _pressure_ ,
and _wind speed_ respectively. The FoT model then predicts a horizon of
$S_{\textrm{out}}=1$ hour with
$F_{\textrm{out}}=\\{\texttt{pm2.5},\texttt{Ir}\\}$ output features, which are
_air quality_ , and _inches of rain_ respectively.

To fully examine the performance of our proposed FoT architecture, we also
evaluated a non-Transformer variant as a baseline for comparison. In the case
of unique multivariate regression, we considered an LSTM architecture, which
has historically been a top-performer in many regression trade spaces. In
terms of parameters, for the LSTM model we chose 3 cascaded LSTM layers with
1024, 512, and 256 units each, followed by 2 fully-connected layers with 128,
and 64 units each, and 10% dropout regularization. Note that here we only
train a single baseline LSTM model, instead of hyperparameter tuning. This
decision was made intentionally to reduce experiment complexity, directing
resources to focus on tuning the proposed models rather than the baselines. We
therefore selected this static set of LSTM network parameters using intuition
from previous experience in the financial trade space.

The data was distributed using a train/val/test percentage split of 70/20/10
respectively. This subset size was chosen such that our models would have
sufficient historical data to learn periodic trends between seasons. The FoT
models were optimized using the Adam optimizer coupled with various learning
rate schedules as tuneable parameters to stabilize performance. We minimize
mean squared error (MSE) loss and use mean absolute error (MAE) as an added
metric to indicate performance.

###  IV-B Plant Disease Classification

In this task, we consider images in the Plant Village [25] dataset to predict
a probability distribution across 38 healthiness class labels. All images are
resized to shape $(72,72,3)$, and scaled to range $[0,1]$. In addition, the
training and validation images are augmented with random flips in both the
vertical and horizontal axes.

To fully examine the performance of the ViT architecture, we also evaluated a
non-Transformer variant for comparison on the plant disease identification
task. Here, we considered the well-known InceptionV3 [26] architecture, which
is based on CNNs. A ready-to-use implementation of this architecture is
provided through the TensorFlow
API444https://www.tensorflow.org/api_docs/python/tf/keras/applications/inception_v3/InceptionV3,
which we employed to simplify experiment complexity. However, note that for
this experiment we trained an InceptionV3 model from scratch; that is, we did
not use any pre-trained weight options. This is an important distinction, as
TensorFlow provides the option for InceptionV3 variants which have been pre-
trained on the ImageNet [27] dataset. In addition, as the InceptionV3 model
was provided through TensorFlow, its design is more rigidly defined compared
to custom implementations. This prohibits most of the hyperparameter tuning
options that we employ with our custom models. Therefore, because of this, we
only trained a single InceptionV3 model using the built-in parameter list.

The data was distributed using a train/val/test percentage split of 70/20/10
respectively. The ViT models were optimized using the Adam optimizer coupled
with a constant learning rate of $10^{-4}$. We chose a constant learning rate
here as our preliminary tests indicated that the ViT model was less sensitive
to changes in learning rate, and therefore did not require a decay schedule in
contrast to the regression tasks. We minimize the sparse categorical
crossentropy (SCCE) loss, and we use accuracy as an added metric to indicate
performance.

###  IV-C Data Fusion Learning

In this task, we consider a fusion of the Plant Village and Beijing PM2.5
datasets to ensure fair performance comparison between FoT and ViT models.
Just like in the classification task, all images are resized to shape
$(72,72,3)$, rescaled to range $[0,1]$, and augmented to improve
generalization. Likewise for the time-series dataset, we consider an input
sequence of $S_{\textrm{in}}=24$ consecutive hours with
$F_{\textrm{in}}=\\{\texttt{TEMP},\texttt{DEWP},\texttt{PRES},\texttt{Iws}\\}$
input features per hour to predict a horizon of $S_{\textrm{out}}=1$ hour with
$F_{\textrm{out}}=\\{\texttt{pm2.5},\texttt{Ir}\\}$ output features.

###  IV-D Multi-Task Learning

In this task we again consider a fusion of the Plant Village and Beijing PM2.5
datasets to ensure fair performance comparison with FoT, ViT, and FuT models
with single output heads. All input data pre-processing for this experiment is
equivalent to the single-head fusion experiments discussed in Section IV-C.

Because the multi-task FuT architecture has multiple output heads, each head
is given its own loss and metrics to optimize. The regression head uses MSE
loss with MAE metric, and the classification head uses SCCE loss with accuracy
metric. These losses are aggregated using weighted mean into a single loss
value for each train/val/test category to optimize the overall model. Model
performance can therefore be examined either for each individual head (using
its associated loss), or for the entire model using the aggregated loss.

###  IV-E Performance Comparison

For each of the tasks discussed in Sections IV-A, IV-B, IV-C and IV-D we train
48 hyperparameter variations of our FoT, ViT, single-task FuT, and multi-task
FuT models. The best configuration for each model is identified as having the
lowest validation loss metric across all variants (i.e., MSE for regression,
and SCCE for classification). We omit the complete training results across all
models for brevity, instead concisely listing the best models and their
hyperparameters in Table I. Specifically, we found FoT model 9, ViT model 37,
regression FuT model 20, classifier FuT model 43, and multi-task FuT model 42
as the best. The train/val/test performance for these models across all epochs
are shown in Fig. 9. Likewise, their train/val/test metrics are shown visually
in Fig. 10, and tabularly in Tables IIa and IIb. Further, while raw
performance is important, computational complexity should also be taken into
account, especially in IoT environments where energy and compute resources are
at a premium. To finalize the comparison between models we therefore also
consider their size (in number of parameters). The total sizes of these best
models are shown in Table III.

TABLE I: Hyperparameters of the top-performing models for each task. Unlisted
values indicate the associated parameter is not used for the model.

Model | $P_{\textrm{drop}}$ | $k$ | $D_{e}$ | $D_{\textrm{ff}}$ | $E$ | $h$ | $(H_{p},W_{p})$ | $D_{\textrm{fusion}}$
---|---|---|---|---|---|---|---|---
FoT 9 | 0.1 | 5 |  | 512 | 6 | 8 |  |
ViT 37 | 0.3 |  | 32 | 256 | 6 | 8 | (6,6) |
Regression FuT 20 | 0.3 | 5 | 32 | 256 | 3 | 8 | (6,6) | 8
Classifier FuT 43 | 0.3 | 10 | 32 | 256 | 6 | 8 | (6,6) |
Multi-Task FuT 42 | 0.3 | 5 | 32 | 256 | 6 | 8 | (6,6) | 16

(a) Regression

(b) Classification

Figure 9: Training and validation performance of best models on (a)
classification, and (b) regression tasks.

(a) MAE

(b) MSE

(c) Accuracy

(d) SCCE

Figure 10: Comparison between best model metrics for regression (a) MAE and
(b) MSE, and classification (c) Accuracy and (d) SCCE. TABLE II: Performance
of best trained models on (a) regression, and (b) classification tasks. Best
metrics in each column are highlighted in bold.

Model | MSE | MAE
---|---|---
_train_ | _val_ | _test_ | _train_ | _val_ | _test_
LSTM | 1.0078 | 0.8728 | 0.5044 | 0.5284 | 0.4957 | 0.4745
FoT 9 | 0.9939 | 0.8630 | 0.5057 | 0.5258 | 0.4956 | 0.4668
Regression FuT 20 | 0.3414 | 0.5031 | 0.5999 | 0.3194 | 0.3270 | 0.3632
Multi-Task FuT 42 | 0.3031 | 0.5767 | 0.4734 | 0.3092 | 0.3405 | 0.3554

(a) Regression

Model | SCCE | Accuracy
---|---|---
_train_ | _val_ | _test_ | _train_ | _val_ | _test_
InceptionV3 | 0.0276 | 0.1581 | 0.1490 | 0.9912 | 0.9606 | 0.9615
ViT 37 | 0.0977 | 0.3499 | 0.3307 | 0.9750 | 0.8936 | 0.8978
Classifier FuT 43 | 0.0064 | 0.3112 | 0.3057 | 0.9994 | 0.9202 | 0.9192
Multi-Task FuT 42 | 0.0135 | 0.3108 | 0.2985 | 0.9987 | 0.9176 | 0.9166

(b) Classification

TABLE III: Size comparison of best models per input and task types via total
number of parameters. Here we also show the size of combined single-task
models for comparison with multi-task variants.

Input Type | Task Type | Model | Parameters
---|---|---|---
Individual | Combined
Single-input | Single-task | LSTM | 8,191,298 | 30,071,944
InceptionV3 | 21,880,646
FoT 9 | 167,330 | 476,712
ViT 37 | 309,382
Multi-input | Single-task |  Regression FuT 20 | 120,042 | 651,928
Classifier FuT 43 | 531,886
Multi-task |  FuT 42 | 428,488 | -

Consider the performance of the regression models in Table IIa, Fig. 9a, and
Figs. 10a and 10b. First, we see that our proposed FoT model 9 exhibits
slightly better training, validation, and testing performance compared to the
baseline LSTM. Specifically, FuT model 9 achieves train/val/test MSE
$0.9939/0.8630/0.5057$ and MAE $0.5258/0.4956/0.4668$, whereas the LSTM
achieves MSE $1.0078/0.8728/0.5044$ and MAE $0.5284/0.4957/0.4745$. Second, we
see that our proposed single-task FuT regression model 20 outperforms the
previous models in all metrics, while also exhibiting the highest overall
validation scores, with MSE $0.3414/0.5031/0.5999$ and MAE
$0.3194/0.3270/0.3632$. Finally, we see that our proposed multi-task FuT model
42 outperforms all other models in terms of training and test performance,
with train/val/test MSE $0.3031/0.5767/0.4734$ and MAE $0.3092/0.3405/0.3554$.
In addition, the lack of plateau in its training and validation performance in
Fig. 9a indicates that increased training could further improve the model.

Next, consider the performance of the classification models in Table IIb, Fig.
9b, and Figs. 10c and 10d. Here we see much more variation between training,
validation, and testing SCCE loss and accuracy metrics. First, the InceptionV3
model exhibits the best validation and test scores, with SCCE
$0.0276/0.1581/0.1490$ and accuracy $0.9912/0.9606/0.9615$. However, it’s
convergence in Fig. 9b is much more unstable starting around epoch 5. Second,
we see that ViT model 37 learns the training set rather well, but performs the
worst in terms of validation and test metrics, with SCCE
$0.0977/0.3499/0.3307$ and accuracy $0.9750/0.8936/0.8978$. Lastly, we see our
proposed single-task FuT classifier model 43 (SCCE $0.0064/0.3112/0.3057$,
accuracy $0.9994/0.9202/0.9192$) and multi-task FuT model 42 (SCCE
$0.0135/0.3108/0.2985$, accuracy $0.9987/0.9176/0.9166$) exhibit very similar
performance. In addition, the convergence of all our proposed classifier
models exhibits a plateau by epoch 30. This indicates that further training on
the image dataset would likely not improve classification performance.

Next, consider the relationship between model size and overall performance for
single-task learners with and without data fusion. Here we see that the FuT
regression model 20 has 47,288 fewer parameters compared to FoT model 9, and
8,071,256 fewer parameters than the baseline LSTM, while exhibiting
significantly better performance. In contrast, the FuT classification model 43
has 222,504 more parameters than ViT model 37 and has similar performance.
This indicates that, while not guaranteed, data fusion can enhance
performance, but can also come at the cost of increased complexity. However,
comparing with the baseline InceptionV3, we see a dramatic reduction of
21,571,264 fewer parameters. As discussed in Section IV-C, the baseline
InceptionV3 indeed has better performance than ViT, however, the boost is
marginal when also comparing the memory and computational complexities
introduced by such a large increase in model size.

Finally, consider the overall performance of multi-task learners with combined
single-task variants. To perform multiple tasks using single fusion models, we
must combine FuT regression model 20 and classification model 43. However,
doing so requires a total of $120,042+531,886=651,928$ parameters. If we then
compare the results of multi-task FuT model 42 we see that equivalent
performance can be achieved with only 428,488 parameters (a reduction of
223,440). If we were to combine the simpler single-input single-task FoT and
ViT models, totaling $309,382+167,330=476,712$ parameters, multi-task FuT
model 42 is still smaller (reducing the size by 48,224). Further, multi-task
FuT model 42 is significantly smaller than the two baseline LSTM and
InceptionV3 models combined, totaling $8,191,298+21,880,646=30,071,944$
parameters. This reduction in model size is significant for two reasons.
First, the smaller size requires less computational resources for inference,
thereby making deployment more viable on IoT devices. Second, it also reduces
the complexity of having two separate models, which further reduces memory
overhead, and also reduces deployment costs, as one multi-task IoT device can
do the job of two single-task variants. These boons in performance,
efficiency, and cost therefore establish the multi-task data fusion
architecture as the prime candidate for deployment in smart city environments.

##  V Conclusion

In this paper, we have provided several AI designs based on pure-encoder
Transformer architectures for use in S&CC environments. We showed that pure-
encoder Transformers serve as a viable baseline for single-task learning
models, and demonstrated their capability on independent supervised
multivariate time-series regression and computer vision classification tasks.
We enhanced our proposed single-task AI system design by leveraging the fusion
of the aforementioned heterogeneous datasets as input, and showed that this
fusion can improve learning over single-input variants. We further extended
the fusion architecture with custom output task heads to facilitate
heterogeneous multi-task learning. The flexibility offered by these custom
heads allows our proposed architecture to support virtually any task, and any
combination of tasks desired. We showed through extensive experimentation that
our proposed multi-input multi-output enhancement either maintains or exceeds
the performance capabilities of both single-task variants and non-Transformer
baselines, while also reducing its memory footprint and computational
overhead. Hence, we have demonstrated that our proposed next-generation multi-
task Transformer-based AI system is viable for deployment in smart city
environments. It can successfully learn to perform unique multivariate
regression of time-series meteorological features, while concurrently
identifying the onset of fungal diseases through visual feature extraction.
Moreover, it performs these heterogeneous tasks using a fusion of both static
images and multivariate time-series data. Indeed, this flexibility to support
diverse feature sets from heterogeneous IoT data sources, while also learning
a variety of tasks in parallel, shows that our system can bolster sustainable
urban development practices, and, in particular, foster the growth of S&CCs.

## References

  * [1] United Nations  “World Urbanization Prospects The 2018 Revision”, 2018  URL: https://population.un.org/wup/Publications/Files/WUP2018-Report.pdf
  * [2] Ashish Vaswani et al.  “Attention Is All You Need”, 2017  URL: http://arxiv.org/abs/1706.03762
  * [3] Bryan Lim, Sercan O. Arik, Nicolas Loeff and Tomas Pfister  “Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting”  In _International Journal of Forecasting_ 37.4, 2019  URL: http://arxiv.org/abs/1912.09363
  * [4] Haoyi Zhou et al.  “Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting”, 2020  URL: http://arxiv.org/abs/2012.07436
  * [5] Neo Wu, Bradley Green, Xue Ben and Shawn O’Banion  “Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case”, 2020  URL: http://arxiv.org/abs/2001.08317
  * [6] George Zerveas et al.  “A Transformer-based Framework for Multivariate Time Series Representation Learning”, 2020  URL: http://arxiv.org/abs/2010.02803
  * [7] Jake Grigsby, Zhe Wang and Yanjun Qi  “Long-Range Transformers for Dynamic Spatiotemporal Forecasting”, 2021  URL: http://arxiv.org/abs/2109.12218
  * [8] Alexey Dosovitskiy et al.  “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”  In _The International Conference on Learning Representations_ , 2020  URL: http://arxiv.org/abs/2010.11929
  * [9] Reenul Reedha, Eric Dericquebourg, Raphael Canals and Adel Hafiane  “Transformer Neural Network for Weed and Crop Classification of High Resolution UAV Images”  In _Remote Sensing_ 14.3, 2022, pp. 592  DOI: 10.3390/rs14030592
  * [10] Huy-Tan Thai, Nhu-Y Tran-Van and Kim-Hung Le  “Artificial Cognition for Early Leaf Disease Detection using Vision Transformers”  In _2021 International Conference on Advanced Technologies for Communications (ATC)_ 2021-October IEEE, 2021, pp. 33–38  DOI: 10.1109/ATC52653.2021.9598303
  * [11] Aditya Prakash, Kashyap Chitta and Andreas Geiger  “Multi-Modal Fusion Transformer for End-to-End Autonomous Driving”, 2021  URL: http://arxiv.org/abs/2104.09224
  * [12] Ronghang Hu and Amanpreet Singh  “UniT: Multimodal Multitask Learning with a Unified Transformer”, 2021  URL: http://arxiv.org/abs/2102.10772
  * [13] Ailing Zeng, Muxi Chen, Lei Zhang and Qiang Xu  “Are Transformers Effective for Time Series Forecasting?”, 2022  URL: http://arxiv.org/abs/2205.13504
  * [14] Yulin Wang et al.  “Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition”  In _NeurIPS_ , 2021, pp. 1–16  URL: http://arxiv.org/abs/2105.15075
  * [15] Alexander Kolesnikov et al.  “Big Transfer (BiT): General Visual Representation Learning”, 2019  URL: http://arxiv.org/abs/1912.11370
  * [16] “Modernizing Cities via Smart Garden Alleys with Application in Makassar City (Award # 2025377)”  In _NSF EAGER Award_ National Science Foundation  URL: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2025377&HistoricalAwards=false
  * [17] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova  “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, 2018  URL: http://arxiv.org/abs/1810.04805
  * [18] Zhilin Yang et al.  “XLNet: Generalized Autoregressive Pretraining for Language Understanding”, 2019  URL: http://arxiv.org/abs/1906.08237
  * [19] Tom B. Brown et al.  “Language Models are Few-Shot Learners”, 2020  URL: http://arxiv.org/abs/2005.14165
  * [20] Seyed Mehran Kazemi et al.  “Time2Vec: Learning a Vector Representation of Time”, 2019  URL: http://arxiv.org/abs/1907.05321
  * [21] Y. Lecun, L. Bottou, Y. Bengio and P. Haffner  “Gradient-based learning applied to document recognition”  In _Proceedings of the IEEE_ 86.11, 1998, pp. 2278–2324  DOI: 10.1109/5.726791
  * [22] Alex Krizhevsky, Ilya Sutskever and Geoffrey E Hinton  “ImageNet Classification with Deep Convolutional Neural Networks”  In _Advances in Neural Information Processing Systems_ 25 Curran Associates, Inc., 2012  URL: https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf
  * [23] Abolfazl Farahani, Behrouz Pourshojae, Khaled Rasheed and Hamid R. Arabnia  “A Concise Review of Transfer Learning”, 2021  URL: http://arxiv.org/abs/2104.02144
  * [24] Song Chen  “Beijing PM2.5 Data”  UCI Machine Learning Repository, UCI Machine Learning Repository, 2017  URL: https://archive-beta.ics.uci.edu/ml/datasets/beijing+pm2+5+data
  * [25] David.. Hughes and Marcel Salathe  “An open access repository of images on plant health to enable the development of mobile disease diagnostics”, 2015  URL: http://arxiv.org/abs/1511.08060
  * [26] Christian Szegedy et al.  “Rethinking the Inception Architecture for Computer Vision”, 2015  URL: http://arxiv.org/abs/1512.00567
  * [27] Jia Deng et al.  “ImageNet: A large-scale hierarchical image database”  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ IEEE, 2009, pp. 248–255  DOI: 10.1109/CVPR.2009.5206848

|  Alexander C. DeRieux (Graduate Student Member, IEEE) received the B.S.
degree in electrical engineering with honors and the B.S. degree in computer
science with honors from Virginia Tech, Blacksburg, VA, USA, in 2016, and the
M.S. degree in electrical engineering in 2022. He is currently a Ph.D. student
in electrical engineering and Bradley Fellow at the Network sciEnce, Wireless,
and Security (NEWS) Laboratory in the Bradley Department of Electrical and
Computer Engineering at Virginia Tech. His research interests include machine
learning, quantum computing, quantum machine learning, quantum communications,
and wireless networks.
---|---
|  Walid Saad (Fellow, IEEE) (S’07, M’10, SM’15, F’19) received his Ph.D
degree from the University of Oslo in 2010. He is currently a Professor at the
Department of Electrical and Computer Engineering at Virginia Tech, where he
leads the Network sciEnce, Wireless, and Security (NEWS) laboratory. He is
also the Next-G Wireless research leader at Virginia Tech’s Innovation Campus.
His research interests include wireless networks (5G/6G/beyond), machine
learning, game theory, security, unmanned aerial vehicles, semantic
communications, cyber-physical systems, and network science. Dr. Saad is a
Fellow of the IEEE. He was the author/co-author of eleven conference best
paper awards. He is the recipient of the 2015 and 2022 Fred W. Ellersick Prize
from the IEEE Communications Society, of the 2017 IEEE ComSoc Best Young
Professional in Academia award, of the 2018 IEEE ComSoc Radio Communications
Committee Early Achievement Award, and of the 2019 IEEE ComSoc Communication
Theory Technical Committee. He was also a co-author of the 2019 and 2021 IEEE
Communications Society Young Author Best Paper. He is the Editor-in-Chief for
the IEEE Transactions on Machine Learning in Communications and Networking.
---|---
|  Wangda Zuo is a Professor in Architectural Engineering and Mechanical
Engineering, as well as Associate Director for Research at Global Building
Network at Pennsylvania State University. Dr. Zuo also holds a joint
appointment at the National Renewable Energy Laboratory (NREL). He is
currently an Associate Editor of Journal of Solar Energy Engineering, and
Fellow of International Building Performance Simulation Association. He is a
major contributor to multiple open-source building and community energy
modeling tools, including Modelica Buildings library and URBANopt.
---|---
|  Rachmawan Budiarto is a faculty member in Department of Nuclear Engineering
and Engineering Physics, Director of Centre for Development of Sustainable
Region (CDSR) in Centre for Energy Studies as well as a senior researcher at
Center for Economic Democracy Studies, Universitas Gadjah Mada. He is a
Greenship Professional of Green Building Council Indonesia. He is a member of
International Committee of Indonesia Accreditation Board for Engineering
Education (IABEE). He was/is involved in at least 81 sustainable energy
related projects and 20 other projects, which some of them were/are funded by
17 international institutions.
---|---
|  Dr Eng. Mochamad Donny Koerniawan, ST, MT , lecturer and researcher in
Building Technology Research Group within The School of Architecture Planning
and Policy Development. Institut Teknologi Bandung, vice president of
International Building Simulation Association, Indonesia Chapter; Green
Associate and Member of Green Building Council Indonesia (GBCI); member of
Association Green Building Expert Indonesia; Member of Indonesia Architect
Institute. Highly experienced in green building research and development.
Development of Sustainable Region (CDSR) in Centre for Energy Studies.
---|---
|  Dwi Novitasari is a Ph.D. student at the Department of Electrical
Engineering and Information Technology, Universitas Gadjah Mada. She is also a
researcher at the Centre for Energy Studies and the Centre for Development of
Sustainable Region (CDSR), Universitas Gadjah Mada. She was/is involved in
national and international projects related to renewable energy, energy
planning, energy transition, and sustainable energy. Currently, her research
is about climate, land use, energy, and water nexus in power generation
planning in Indonesia. She also received the Canada-ASEAN Exchange Scholarship
at the School of Sustainable Energy Engineering, Simon Fraser University.
---|---