# Spectral Temporal Graph Neural Network for massive MIMO CSI Prediction

Sharan Mourya  Pavan Reddy  SaiDhiraj Amuru  Kiran Kumar Kuchi

###### Abstract

In the realm of 5G communication systems, the accuracy of Channel State
Information (CSI) prediction is vital for optimizing performance. This letter
introduces a pioneering approach: the Spectral-Temporal Graph Neural Network
(STEM GNN), which fuses spatial relationships and temporal dynamics of the
wireless channel using the Graph Fourier Transform. We compare the STEM GNN
approach with conventional Recurrent Neural Network (RNN) and Long Short-Term
Memory (LSTM) models for CSI prediction. Our findings reveal a significant
enhancement in overall communication system performance through STEM GNNs. For
instance, in one scenario, STEM GNN achieves a sum rate of 5.009 bps/Hz which
is $11.9\%$ higher than that of LSTM and $35\%$ higher than that of RNN. The
spectral-temporal analysis capabilities of STEM GNNs capture intricate
patterns often overlooked by traditional models, offering improvements in
beamforming, interference mitigation, and ultra-reliable low-latency
communication (URLLC).

###### Index Terms:

Graph neural networks, STEM GNN, CSI Feedback, CSI Prediction, STNET, Massive
MIMO.

##  I Introduction

In time division duplex (TDD) cellular networks, channel reciprocity enables
downlink channel state information (DL-CSI) inference from uplink channel
state information (UL-CSI), obviating the need for extra estimation or
feedback. However, this reciprocity principle doesn’t hold for frequency
division duplex (FDD) systems, introducing challenges in DL-CSI acquisition.
Prior methods required receivers to predict and feedback DL-CSI, incurring
computational load and overheads. Addressing this, researchers have delved
into channel prediction techniques. These techniques leverage historical CSI
correlations to forecast future channel conditions, aiding multi-frame
predictions. Existing methods mainly fall into model-based or neural network-
based categories. Model-based strategies employ diverse models, like linear
extrapolation, sum-of-sinusoids, and autoregressive (AR) [1], to capture
dynamic channel behavior. However, complexities arising from factors like
multipath and Doppler effects pose challenges to precise modeling. In
contrast, neural network-driven approaches have gained traction [1] which
train neural networks to mirror real-world channel behaviors through data-
driven learning.

In [2], authors employed feed-forward neural networks for CSI prediction,
directly linking uplink and downlink CSI. However, their assumption of a
bijective user-channel matrix relationship isn’t always applicable. To address
this, a more advanced model using Convolutional Neural Networks (CNN) and Long
Short-Term Memory (LSTM) was proposed in [3], notably outperforming methods
like Maximum Likelihood (ML) and Minimum Mean Squared Error (MMSE). A similar
data-driven model based on CNN and LSTM was also presented by Wang et al. [4]
for precise CSI prediction. Recurrent Neural Networks (RNNs) were employed in
some strategies [1] [5], often relying on CNNs for feature extraction, despite
being primarily designed for real-valued image tasks. To alleviate this
limitation, Zhang et al. introduced a complex-valued 3D CNN for CSI prediction
in [6], improving accuracy over real-valued networks. While LSTMs excel in
capturing long-range correlations compared to RNNs, they face the vanishing
gradient problem. Addressing this, a transformer-based CSI prediction model
was introduced in [7]. Transformers employ self-attention mechanisms,
overcoming the vanishing gradient issue and preserving long-range correlations
more effectively than LSTMs. Additionally, transformers resolve the sequential
prediction problem, where predicting each future CSI matrix relies on
historical CSI and the latest predicted CSI, leading to error propagation.
Transformers process data in parallel, sidestepping the sequential prediction
challenge and efficiently forecasting future CSI matrices.

Another category of deep learning architectures with high retention of long-
range correlation is the Graph Neural Network (GNN) [8]. When dealing with
long-range correlations, traditional neural networks like feed-forward or
recurrent neural networks may struggle due to their local receptive fields and
sequential nature. However, GNNs are well-suited to address this challenge as
they incorporate message passing. GNNs operate by passing messages between
nodes in a graph iteratively. Each node aggregates information from its
neighboring nodes and updates its representation based on the gathered
information. This process allows information to propagate across the graph,
enabling nodes to capture correlations with more distant nodes. On the other
hand, one of the fundamental components of GNNs is the graph convolutional
layer [8]. This layer incorporates both local and non-local information,
enabling nodes to capture correlations with their neighbors as well as more
distant nodes in the graph.

In this letter, we propose to use a spectral temporal (STEM) graph neural
network [9] for CSI prediction. Originally STEM GNN was proposed to solve the
problem of time series forecasting [10] on data with complex relational
structures represented as graphs. STEM GNNs are particularly useful when
dealing with multivariate time series data, where each time step contains
multiple variables (e.g., temperature, humidity, pressure, etc.) and the
interactions between variables can be represented as a graph. Our
contributions in this letter can be summarized as follows:

  1. 1.

We model CSI prediction as a multivariate time-series forecasting problem, a
popular class of problems in deep learning.

  2. 2.

We modify the state-of-the-art STEM GNN model designed for multivariate time-
series forecasting to better suit the CSI prediction problem. We exploit
spectral and temporal correlations of the historical CSI with the future CSI
to make accurate predictions.

  3. 3.

We implement other cutting-edge CSI prediction models using LSTMs and RNNs and
make a detailed comparison with the STEM GNN approach.

To the best of our knowledge, we are the first to model CSI prediction as a
multivariate time series forecasting problem.

Figure 1: System Model

The organization of this paper is as follows: In Section II, we introduce the
system model used throughout this paper. In Section III, we introduce STEM
GNNs and their applicability to CSI prediction. In Section IV, we summarize
the simulation results followed by the conclusion in Section V.

##  II System Model

We consider a MIMO orthogonal frequency division multiplexing (OFDM) system.
In Fig. 1, the base station (BS) is outfitted with an array of antennas
organized in an $M\times N$ rectangular pattern, with each antenna featuring
dual polarization resulting in a total of $N_{t}=M\times N\times 2$ transmit
antennas and the user equipment (UE) has a single antenna. The number of sub-
carriers is $N_{c}$. Given that the CSI prediction operates independently for
various users, it is sufficient to illustrate the principle using a single
user. In this context, we consider the communication channel between the user
and the BS to be represented by a three-dimensional (3D) channel model,
conforming to the specifications outlined in the 3rd Generation Partnership
Project (3GPP) for 5G communications. This channel is denoted as ${H}$.

The received signal $y_{n}$ on the $n^{th}$ sub-carrier can be denoted by

$y_{n}=\textbf{${{h}_{n}}^{H}$}x_{n}+\textbf{$w_{n}$},$ (1)

where, ${h}_{n}\in\mathbb{C}^{N_{t}\times 1}$, $x_{n}\in\mathbb{C}$, and
$w_{n}\in\mathbb{C}$ are the channel, symbol and additive noise respectively.
The channel matrix ${H}$ of size $N_{c}\times N_{t}$ is given by

${H}=\big{[}{h}_{1},{h}_{2},\cdots,{h}_{{N_{c}}}\big{]}^{H}.$ (2)

The matrix representing this channel consists of a total of $2N_{t}N_{C}$
elements, considering that the entries within the matrix are of a complex
nature. However, in practical scenarios, transmitting such a sizable matrix
proves to be unfeasible for a massive MIMO system. To address this challenge,
we adopt a strategy to reduce the payload size. This entails compressing the
matrix ${H}$, which is of a 3D nature, into a more compact one-dimensional
vector with dimensions $C\times 1$, as depicted in Figure 1. To quantify this
compression, we introduce the compression ratio denoted by
$\gamma=\frac{C}{2N_{C}N_{t}}$. The compressed one-dimensional vector is then
transmitted to the BS. Upon reception, a CSI prediction model is applied to
this vector to forecast future CSI in its compressed form. Subsequently, a
decoder is employed to reconstruct the estimated channel matrix, denoted as
${\hat{H}}$, from the predicted vector. This entire sequence of operations can
be described through the following equations:

$s=f_{e}({H}),\quad e=f_{p}(s),\quad{\hat{H}}=f_{d}(e),$

where $f_{e}$ is the encoder, $f_{p}$ is the CSI prediction model, and $f_{d}$
is the decoder. $s$, $e$, and ${\hat{H}}$ are the compressed, predicted, and
estimated channel matrices respectively. Using the estimated channel matrix
${\hat{H}}$, BS utilizes the CSI prediction model to estimate the channel
matrices for the future frames. We address the number of such frames as the
horizon (denoted by $P$) and it would be specified in the CSI prediction model
while training.

##  III Spectral Temporal Graph Neural Network

Having established the system model, let’s delve into the CSI prediction model
depicted in Figure 1. Conventional CSI prediction relied on auto-regression
[11] [1] or hidden Markov chains [12], which capture temporal correlations
between historical and future CSI but neglect spatial matrix correlations.
Thus, deep learning emerged, employing CNNs to capture the matrix’s spatial
relationships. Here, CSI is viewed as a spatial map, with each element
representing a channel aspect. CNN-based models transform raw CSI into images
highlighting patterns like signal fluctuations and interference.

Typically, CNN-based CSI models include an RNN or LSTM to capture temporal
trends [3, 4, 1, 5]. Historical CSI sequences are fed into the network,
learning patterns, and trends. RNNs and LSTMs retain memory, accurately
inferring future states. However, this separate spatial-temporal approach
misses the concurrent spatio-temporal correlations present in CSI evolution.
We propose a unified model using GNNs, effectively extracting these
correlations for efficient CSI prediction. STEM GNNs [9] offer a profound
approach to comprehending and predicting multivariate time-series patterns
intertwined in space and time. Predicting such a series is intricate [10],
requiring simultaneous consideration of intra-series and inter-series
correlations. In CSI prediction, these map to spatial and temporal
correlations. STEM GNNs adeptly handle both correlations simultaneously within
the spectral domain. This is achieved through merging Graph Fourier Transform
(GFT) [8] for spatial correlations and Discrete Fourier Transform (DFT) for
temporal dependencies. These elements unite into an end-to-end framework,
enabling STEM GNNs to effectively tackle both spatial and temporal
correlations in CSI prediction.

###  III-A Problem Formulation

Figure 2: STEM GNN Architecture

Let us establish the graph that captures the evolution of the channel as
$G=(S,W)$, where $\\{S={s_{0},s_{1},\ldots,s_{T-1}}\\}\in\mathbb{R}^{C\times
T}$ signifies the historical compressed CSI input. In this context, $s(i,j)$
represents the latent representation vector, $C$ denotes its dimension
(nodes), and $T$ represents the count of timestamps. The compressed CSI values
observed at timestamp $t$ are symbolized as $s_{t}\in\mathbb{R}^{C}$. The
adjacency matrix is denoted as $W\in\mathbb{R}^{C\times C}$, with $w_{ij}>0$
indicating the presence of a connection between nodes $i$ and $j$, where the
magnitude of $w_{ij}$ quantifies the strength of this connection.

Given the observed values from the $K$ preceding timestamps
$\\{{s_{t-K},\ldots,s_{t-1}}\\}$, the objective of CSI prediction revolves
around anticipating the node values within a multivariate temporal graph
$G=(S,W)$ for the subsequent $P$ timestamps, noted as
$\\{{e_{t},e_{t+1},\ldots,e_{t+P-1}}\\}$. These values are deducible through
the predictive model $f_{p}$ governed by parameters $\theta$.

$e_{t},e_{t+1},\ldots,e_{t+P-1}=f_{p}(s_{t-K},\ldots,s_{t-1};G;\theta).$ (3)

Now, using the decoder $f_{d}$ we decompress the predicted compressed CSI to
obtain the future CSI as shown in Fig. 1.

###  III-B STEM GNN Architecture

STEM GNN consists of three main layers each having its own importance: latent
correlation layer, STEM GNN layer, and output layer as shown in Fig. 2. The
latent correlation layer is tasked with finding the adjacency matrix ($W$) of
the graph $G$. As it is difficult to construct an adjacency matrix of
historical compressed CSI ($S$) by hand, a self-attention layer [13] is used
to learn the latent correlations between multiple compressed CSI vectors
automatically. In this way, the model exploits correlations to form an
adjacency matrix in a data-driven fashion. The extracted adjacency matrix $W$
along with the graph structure $G$ are fed to the next layer. STEM GNN layer
has a couple of STEM GNN blocks (as shown in Fig. 2) interconnected with skip
connections and addition blocks [9]. This forms the crust of the CSI
prediction model as this takes $G$ as an input and extracts the spatio-
temporal correlations from the historical compressed CSI which are then used
by the output layer to create accurate predictions for the following $P$
timestamps.

The STEM GNN block encompasses several operations, including GFT and its
inverse IGFT, DFT and its inverse IDFT, a 1D convolution layer (1D-Conv),
followed by a gated linear unit (GLU) activation, and the pivotal graph
convolution (GConv). In the initial steps, GFT transforms the graph $G$ into a
spectral matrix representation, rendering compressed CSI independent in the
spatial domain. Subsequently, DFT shifts univariate time-series components
into the frequency domain, and 1D convolution with GLU captures feature
patterns before reverting through inverse DFT. The cornerstone GConv
operation, crucial in any GNN, propagates information across the graph. The
sequence concludes with inverse GFT. This entire process is visually depicted
in Figure 2.

##  IV Results

Utilizing the STEM GNN architecture and the problem formulation outlined
earlier, we conducted a sequence of simulations using the 3GPP channel model,
characterized by the following specifications.

TABLE I: Channel Model Scenario | Urban Macro (UMa)
---|---
Antenna Config. |  $[8,2,2]$ $\Rightarrow$ $N_{t}=8\times 2\times 2=32$
No. of Subcarriers ($N_{c}$) | 32
No. of consecutive timestamps ($T$) | 10,000
Bandwidth | 20 MHz
User Mobility | 3Kmph

The BS antenna is modelled by a uniform rectangular panel array, comprising
$8\times 2$ panels. The antenna panel is dual polarized hence the antenna
configuration is $[8,2,2]$. We choose window size (K) in Eq. 3, i.e., the
number of previous timestamps to input to the CSI prediction model to be 12.
Dimension of the compressed CSI, $C$ is chosen from $\\{128,256,512\\}$ which
translates to compression ratio of $\gamma=\\{1/16,1/8,1/4\\}$ respectively.
With these settings, we compare our approach with RNN and LSTM-based CSI
prediction models as they outperform many of the traditional CSI prediction
techniques [1]. In addition, for CSI compression and feedback we use STNet
[14] as it has the best performance per resource utilization among all the
other CSI feedback networks [15].

###  IV-A Sum Rate Performance

Using transmit precoding, a massive MIMO system can attain remarkable
capacities. In this regard, we employ the prevalent linear Zero-Forcing (ZF)
transmit precoding technique [14] to assess the overall enhancement in
communication system performance brought about by distinct CSI feedback
approaches. The evaluation process involves plotting the spectral efficiency
of each method against the signal-to-noise ratio (SNR) for varying compression
ratios. The resulting depiction can be observed in Figure 2.

(a) Horizon, $P=5$

(b) Horizon, $P=7$

(c) Horizon, $P=9$

Figure 3: Sum rate vs SNR plots comparing STEM GNN with RNN and LSTM for CSI
prediciton with various horizon values, $P=\\{5,7,9\\}$ at a fixed window
size, $K=12$.

Firstly, we note that for the STEM GNN approach, the sum rate increases with
$\gamma$, and for RNN and LSTM, the sum rate decreases with $\gamma$. This is
because RNNs and LSTMs lack the representational power of a GNN and hence fail
to generalize the historical compressed CSI vectors for larger input sizes
($C$), i.e., smaller $\gamma$. On the other hand, STEM GNN is equipped with
enough capabilities to generalize the compressed CSI for higher input sizes.
Secondly, the difference in the sum rate performances of STEM GNN, RNN, and
LSTM becomes wider and more evident as the horizon ($P$) value increases from
$5$ to $9$ which is expected behaviour. For instance, at $P=9$, STEM GNN
achieves a sum rate of 5.009 bps/Hz which is $11.9\%$ higher than that of LSTM
and $35\%$ higher than that of RNN. Which, for a 20 MHz bandwidth translates
to a total data rate of 74.2 MHz, 89.52 MHz, and 100.18 MHz for RNN, LSTM, and
STEM GNN respectively.

###  IV-B CSI Prediction Performance

The accuracy of CSI prediction by various networks can be measured using root
mean squared error (RMSE) [9] between the original future CSI and the
predicted future CSI. We plotted the RMSE values for RNN, LSTM, and STEM GNN
networks for various compression ratios and horizons as shown in Fig 4. It
should be noted that the RMSE of STEM GNN is much lower when compared to RNN
and LSTM networks. For instance, at $P=3$ and $\gamma=1/4$, STEM GNN achieved
an RMSE of 1.5832 whereas RNN and LSTM achieved an RMSE of 24.4404 and 22.2726
which are 15 and 14-fold higher than that of STEM GNN which is a generational
gap in performance.

Figure 4: RMSE performance of RNN, LSTM, and STEM GNN-based CSI prediction
models for various compression ratios ($\gamma$) and horizons ($P$). Figure 5:
NMSE performance of RNN, LSTM, and STEM GNN-based CSI prediction models for
various compression ratios ($\gamma$) and horizons ($P$). Figure 6: Training
loss vs the number of epochs for RNN, LSTM, and STEM GNN-based CSI prediction
models at the horizon, $P=3$ and $\gamma=1/4$.

###  IV-C Channel Reconstruction Performance

Similarly, the accuracy of CSI reconstruction at the BS can be measured using
normalized mean squared error (NMSE) [14] between the original CSI and the
reconstructed CSI. This measure is important as its performance will
inherently increase the system performance increasing the data rate. We
plotted the NMSE values for RNN, LSTM, and STEM GNN networks for various
compression ratios and horizons as shown in Fig. 5. It should be noted that
the NMSE of STEM GNN is much lower when compared to RNN and LSTM networks. For
instance, at $P=3$ and $\gamma=1/4$, STEM GNN achieved an NMSE of -3.5116
whereas RNN and LSTM achieved an RMSE of -0.8967 and -0.7618 which are 3.9 and
4.6-fold higher than that of STEM GNN. This substantial difference in the NMSE
performance is also reflected in the overall data rate achieved by different
networks.

###  IV-D Training Performance

Training of all three networks was achieved on an Nvidia RTX 3060 graphic
card111Source code of this paper: https://github.com/sharanmourya/CSI-
Prediction/. Training loss is an important parameter in understanding the
stability and convergence of a network with respect to the dataset. Therefore,
we plotted a training loss plot varying with the number of epochs for a
horizon of $P=3$ and a compression ratio of $\gamma=1/4$ as shown in Fig. 6.
From the figure, it can be seen that the training loss of STEM GNN is at least
two orders of magnitude less than that of RNN and LSTM. In addition, the STEM
GNN model also converges significantly faster than that of RNN and LSTM.

##  V Conclusion

In this letter, we study the Spectral-Temporal Graph Neural Network (STEM GNN)
approach for CSI prediction in 5G communication systems and demonstrate its
superiority over traditional RNN and LSTM models. Leveraging both spatial
relationships and temporal dynamics through the Graph Fourier Transform, STEM
GNNs exhibit a remarkable improvement in overall communication system
performance.

## References

  * [1] W. Jiang et al., “Neural network-based fading channel prediction: A comprehensive overview,” IEEE Access, vol. 7, pp. 118112–118124, 2019\.
  * [2] Y. Yang et al., “Deep learning-based downlink channel prediction for fdd massive mimo system,” IEEE Communications Letters, vol. 23, no. 11, pp. 1994–1998, 2019.
  * [3] C. Luo et al., “Channel state information prediction for 5g wireless communications: A deep learning approach,” IEEE Transactions on Network Science and Engineering, vol. 7, no. 1, pp. 227–236, 2020.
  * [4] J. Wang et al., “Ul-csi data driven deep learning for predicting dl-csi in cellular fdd systems,” IEEE Access, vol. 7, pp. 96105–96112, 2019.
  * [5] W. Jiang et al., “Deep learning for fading channel prediction,” IEEE Open Journal of the Communications Society, vol. 1, pp. 320–332, 2020.
  * [6] Y. Zhang et al., “Cv-3dcnn: Complex-valued deep learning for csi prediction in fdd massive mimo systems,” IEEE Wireless Communications Letters, vol. 10, no. 2, pp. 266–270, 2021.
  * [7] H. Jiang et al., “Accurate channel prediction based on transformer: Making mobility negligible,” IEEE Journal on Selected Areas in Communications, vol. 40, no. 9, pp. 2717–2732, 2022.
  * [8] Z. Wu et al., “A comprehensive survey on graph neural networks,” IEEE Trans. Neural Netw. and Learning Syst., vol. 32, pp. 4–24, jan 2021.
  * [9] D. Cao et al., “Spectral temporal graph neural network for multivariate time-series forecasting,” Advances in neural information processing systems, vol. 33, pp. 17766–17778, 2020.
  * [10] B. Lim et al., “Time-series forecasting with deep learning: a survey,” Philosophical Transactions of the Royal Society A, vol. 379, no. 2194, p. 20200209, 2021.
  * [11] C. Wu et al., “Channel prediction in high-mobility massive mimo: From spatio-temporal autoregression to deep learning,” IEEE Journal on Selected Areas in Communications, vol. 39, no. 7, pp. 1915–1930, 2021.
  * [12] Z. Chen et al., “Prediction of channel state for cognitive radio using higher-order hidden markov model,” in Proceedings of the IEEE SoutheastCon 2010 (SoutheastCon), pp. 276–282, 2010.
  * [13] A. Vaswani et al., “Attention is all you need,” in Advances in Neural Information Processing Systems, vol. 30, Curran Associates, Inc., 2017\.
  * [14] S. Mourya et al., “Graph neural networks-based user pairing in wireless communication systems,” 2023.
  * [15] J. Guo et al., “Overview of deep learning-based csi feedback in massive mimo systems,” IEEE Transactions on Communications, vol. 70, no. 12, pp. 8017–8045, 2022.


# Towards Perspective-Based Specification of Machine Learning-Enabled Systems

Hugo Villamizar, Marcos Kalinowski, and Hélio Lopes  Informatics Department
Pontifical Catholic University of Rio de Janeiro
Rio de Janeiro, Brazil
{hvillamizar, kalinowski<EMAIL_ADDRESS>
###### Abstract

Machine learning (ML) teams often work on a project just to realize the
performance of the model is not good enough. Indeed, the success of ML-enabled
systems involves aligning data with business problems, translating them into
ML tasks, experimenting with algorithms, evaluating models, capturing data
from users, among others. Literature has shown that ML-enabled systems are
rarely built based on precise specifications for such concerns, leading ML
teams to become misaligned due to incorrect assumptions, which may affect the
quality of such systems and overall project success. In order to help
addressing this issue, this paper describes our work towards a perspective-
based approach for specifying ML-enabled systems. The approach involves
analyzing a set of 45 ML concerns grouped into five perspectives: objectives,
user experience, infrastructure, model, and data. The main contribution of
this paper is to provide two new artifacts that can be used to help specifying
ML-enabled systems: (i) the perspective-based ML task and concern diagram and
(ii) the perspective-based ML specification template.

###### Index Terms:

software engineering, requirements specification, machine learning, machine
learning enabled systems

##  I Introduction

Companies from all sectors are increasingly incorporating machine learning
(ML) components into the software that supports their operations. We call this
ML-enabled systems. The main difference between ML-enabled systems and non-ML
is that data, to some extent, replaces code when determining the behavior of
ML-enabled systems. This is challenging from the point of view of software
engineering (SE). For instance, data should be tested and models be validated
just as thoroughly as code, but there is a lack of best practices on how to do
so [2].

Data scientists often work on a project just to realize that the performance
of the model is not good enough. However, it is important to understand
several aspects. For example, where will the model run? What data will it have
access to? How fast does it need to be? What is the business impact of a false
positive? A false negative? How should the model be tuned to maximize business
results? In short, a model is just one component of a system as a whole.

Within the SE discipline, requirements engineering (RE) has drawn the
attention of researchers and practitioners to the fact that ML can benefit
from this perspective [6]. Literature has shown that current research on the
intersection between RE and ML mainly focuses on using ML techniques to
support RE activities rather than on exploring how RE can improve the
development of ML-based systems [1]. In addition, most ML models lack
requirements specifications since current RE practices are not well defined
and organized for ML [7]. This indicates that there is an incredible amount of
work to be done between the development of a model, the incorporation of it
into a system and the eventual sustainable customer impact.

Given this landscape, we propose a perspective-based approach for specifying
ML-enabled systems. The approach relies on a set of concerns grouped into five
perspectives: objectives, user experience, infrastructure, model, and data.
The approach, built based on the literature [12], industrial experiences
working on ML-enabled systems [9], and an initial industrial validation of
relevant concerns [13], covers relevant perspectives and practical concerns
that have not been considered as part of related work (e.g., [5, 10, 11]).

##  II Perspective-based Specification of ML Systems

The proposed perspective-based approach provides two artifacts to support the
specification of ML-enabled systems: (i) a perspective-based ML task and
concern diagram to be analyzed; and (ii) a specification template to be
filled. These artifacts were designed based on findings from a literature
review [12], industrial experiences [9], and an initial validation of the ML
concerns [13]. While the literature review showed us how ML could benefit from
the RE perspective, what particular properties and research opportunities
could be addressed, our industrial experience allowed us to understand how ML
works in practice from early to final stages. Hence, we established links
between theory and practice to learn more about the context in which our
approach operates.

###  II-A Perspective-Based ML Task and Concern Diagram

The perspective-based ML task and concern diagram shown in Figure 1 consists
of five perspectives: objectives, user experience, infrastructure, model, and
data. Together these perspectives mediate the involvement of actors such as
business owners (BO), designers (DG), software engineers (SE), and data
scientists (DS). Each perspective contains a set of concerns that should be
analyzed by a requirement engineer with the support of at least one actor. In
a previous study [13], we conducted a focus group session with eight software
professionals with experience developing ML-enabled systems to validate the
importance, quality and feasibility of using the concerns.

Figure 1: Perspective-based ML task and concern diagram

The diagram contains five rounded rectangles that represent the ML
perspectives. Each perspective is divided into rectangles that connect a task
(at the top right) to one or more concerns (at the bottom). Note that each
task-concern has at least one actor suggested (at the top left) related to the
execution of the task and the analysis of the concerns. For instance,
typically in ML projects, DS are tasked for training, running, and evaluating
models. These tasks involve implicit concerns that are not easily identified
such as learning time, inference time, and model size (Cf. model perspective
in Figure 1). The diagram also models dashed arrows that represent
dependencies between the tasks. It is noteworthy that dependencies typically
implicitly imply an influence in the opposite direction. For example, to
transport data, that is a task of the infrastructure perspective that can
require implementing real time analytics, it is necessary first to access the
data and define the source. In the following, we describe the role of the
actors involved in our approach.

BO typically understand how to plan ML projects in order to make good
decisions. How to connect business objectives with ML outcomes? What
represents future success from a business point of view? SE typically
understand how the system will interact with the model DS produce and identify
the components that need to exist within a successful ML-enabled system. What
are the pros and cons of running a model in a client or in a service? DS
typically understand the constraints regarding the ML models. For instance,
what quality properties the model should consider? What domain restrictions
may apply? How fast does it need to be? DG typically understand how to improve
the experience of end-users of the system and its ML inferences. For analyzing
some tasks and concerns associated to the DG, if possible, end-users should
also be involved. For instance, where and how often should the ML outcomes
appear? How forcefully should they appear?

Note that specific concerns can benefit from involving more than one actor in
the analysis. E.g., the execution engine concern involves knowledge in charge
of DS and may also involve expertise of SE that provide the ML-enabled system
infrastructure. Hereafter, we further describe the perspectives and concerns
of the diagram.

####  II-A1 ML Objectives Perspective

Bridging the gap between the high-level goals and the detailed properties of
ML models is one of the most common causes of failure to succeed [3]. Table I
presents the concerns related to ML objectives that could influence the
specification of ML-enabled systems.

TABLE I: ML objectives perspective Concern | Description (Addressing this concerns involves … )
---|---
Customer expectations |  Specifying expectations of customers and end-user in terms of how the system should behave (e.g., how often they expect predictions to be right or wrong).
Leading indicators |  Specifying measures correlating with future success, from the business’ perspective. This could include the users’ affective states when using the ML-enabled system (e.g., customer sentiment and engagement).
ML functionality |  Specifying the ML results in terms of functionality that the model will provide (e.g., classify customers, predict probabilities).
Model goals |  Specifying metrics and acceptable measures the model should achieve (e.g., for classification problems this could involve accuracy $\geq$ X%, precision $\geq$ Y%, recall $\geq$ Z%).
Organizational goals |  Specifying measurable benefits ML is expected to bring to the organization. E.g., increase the revenue in X%, increase the number of units sold in Y%, number of trees saved.
Problem & Context |  Specifying the problem that ML will address and its context before coding. ML must be targeted at the right problem.
User goals |  Specifying what the users want to achieve by using ML. E.g., for recommendation systems this could involve helping users finding content they will enjoy.

####  II-A2 User Experience (UX) Perspective

Better ML includes building better experiences of using ML. Connecting the
predictions with users is critical for achieving success. However, this
requires deep analysis. For instance, one mistake in the way of interacting
with the user, and all the work spent on data pre-processing and modeling may
be wasted. The goal of this perspective is to create effective UX so that
users can interact appropriately and understand what is going on with the
predictions of the model. Table II shows the UX concerns and a brief
description of each one.

TABLE II: UX Perspective Concern | Description (Addressing this concerns involves … )
---|---
Accountability |  Specifying who is responsible for unexpected model results or actions taken based on unexpected model results.
Cost |  Specifying costs involved in executing the inferences and also the user impact of a wrong model prediction.
Forcefulness |  Specifying how strongly the system forces the user to do what the model indicates they should (e.g., automatic or assisted actions).
Frequency |  Specifying how often the system interacts with users (e.g., interact whenever the user asks for it or whenever the system thinks the user will respond).
Interactiveness |  Specifying what interactions the users will have with the ML-enabled system, (e.g., to provide new data for learning, or human-in-the-loop systems where models require human interaction).
Value |  Specifying the added value as perceived by users from the predictions to their work.
Prediction Visualization |  Specifying how the ML outcomes will be presented so that users can understand them (e.g., specifying dashboard and visualization prototypes for validation).

####  II-A3 Infrastructure Perspective

ML models need to be integrated with other services. This includes components
such as ingesting and learning from new data. The adoption of ML is growing,
but proper implementations are needed to fulfill their promise. Hence,
software engineers play an important role to orchestrate these ML components.
Table III presents the infrastructure concerns.

TABLE III: Infrastructure Perspective. Concern | Description (Addressing this concerns involves … )
---|---
Data streaming |  Specifying what data steaming strategy will be used (e.g., real time data transportation or in batches).
Execution engine |  Specifying how the model of the ML-enabled system will be executed and consumed (e.g., client-side, back-end, cloud-based, web service end-point).
Incremental learning |  Specifying the need for ML-enabled system abilities to continuously learn from new data, extending the existing model’s knowledge.
Integration |  Specifying the integration that the model will have with the rest of the system functionality.
Safety |  Specifying how the system deals with risks to prevent dangerous failures. Critical systems that incorporate ML should analyze the probability of the occurrence of harm and its severity.
Security |  Specifying how the system deals with security issues (e.g., vulnerabilities) to protect the data. ML systems often contain sensitive data that should be protected.
Storage |  Specifying where the ML artifacts (e.g., models, data, scripts) will be stored.
Telemetry |  Specifying what ML-enabled system data needs to be collected. Telemetry involves collecting data such as clicks on particular buttons and could involve other usage and performance monitoring data.

####  II-A4 Model Perspective

Building a model implies not only training an algorithm with data to predict
or classify well some phenomenon. Many other aspects determine its success.
The aim of this perspective is to provide a set of model concerns a
requirements engineer should analyze. Table IV presents these concerns.

TABLE IV: Model Perspective. Concern | Description (Addressing this concerns involves … )
---|---
Algorithms |  Specifying the set of algorithms that could be used/investigated, based on the ML problem and other concerns to be considered (e.g., constraints regarding explainability or model performance, for instance, can limit the solution options).
Explainability |  Specifying the need to understand reasons of the model inferences. The model might need to be able to summarize the reasons of its decisions. Other related concerns, such as transparency and interpretability, may apply.
Inference time |  Specifying the acceptable time to execute the model and return the predictions.
Input & Output |  Specifying the expected inputs (features) and outcomes of the model. Of course, the set of meaningful inputs can be refined/improved during pre-processing activities, such as feature selection.
Learning time |  Specifying the acceptable time to train the model.
Maintainability |  Specifying the need for preparing the model to go through changes with reasonable effort (e.g., refactoring, documentation, automated redeployment).
ML problem type |  Specifying the problem type tackled by the ML algorithm (e.g., classification, regression, clustering, extract information from text).
Model performance |  Specifying the metrics used to evaluate the model (e.g., precision, recall, F1-score, mean square error) and measurable performance expectations.
Model size |  Specifying the size of the model in terms of storage and its complexity (e.g., for decision trees there might be needs for pruning).
Reproducibility |  Specifying the need for replicating the model creation process and its experiments.

####  II-A5 Data Perspective

Data is essential for ML-enabled systems. Poor data will result in inaccurate
predictions, which is referred to in the ML context as “garbage in, garbage
out”. Hence, ML requires high-quality input data. From the viewpoint of RE, it
is clear that data constitutes a new type of requirements [4], [14]. Based on
the Data Quality model defined in the standard ISO/IEC 25012 [8], we elaborate
on the data perspective. Table V presents the data concerns.

TABLE V: Data Perspective. Concern | Description (Addressing this concerns involves … )
---|---
Accuracy |  Specifying the need to get correct data.
Baseline |  Specifying the need for a baseline dataset approved by a domain expert that reflects the problem. It is employed to monitor other data acquired afterwards.
Bias |  Specifying the need to get data fair samples and representative distributions.
Completeness |  Specifying the need to get data containing sufficient observations of all situations where the model will operate.
Consistency |  Specifying the need to get consistent data in a specific context.
Credibility |  Specifying the need to get true data that is believable and understandable by users.
Data operations |  Specifying what operations must be applied on the data (e.g., data cleaning and labeling).
Distribution mismatches |  Specifying expected data distributions and how data will be split into training and testing data.
Ethic & privacy |  Specifying the need to get data to prevent adversely impacting society (e.g., listing potential adverse impacts to be avoided).
Quantity |  Specifying the expected amount of data according to the type of the problem and the complexity of the algorithm.
Real usage |  Specifying the need to get real data representing the real problem.
Source |  Specifying from where the data will be obtained.
Timeliness |  Specifying the time between when data is expected and when it is readily available for use.

###  II-B Perspective-based ML Specification Template

Requirements specification produces a description of part of what should be
built and delivers it for approval and requirements management. Based on the
perspective-based ML task and concern diagram, the requirements engineer can
identify, with the support of the actors, which tasks and concerns should be
analyzed and described as part of the specification.

The division into perspectives, the influence and dependency relationships
between tasks and concerns and the suggested involvement of actors can help in
this analysis. For instance, it can be seen in Figure 1 that if the solution
requires to know reasons of ML decisions (task) then the explainability
concern arises. Explainability depends on the chosen algorithm, and should be
specified considering this aspect. For classification problems, decisions
taken by neural networks are typically less explainable than decisions taken
by decision trees.

To support the specification, we provide an ML requirement specification
template that allows specifying the applicable ML-enabled system concerns.
Inherently to the nature of ML-enabled system projects, some types of
requirements (e.g., required algorithm, required data pre-processing
operations, model performance requirements) are uncertain at the beginning of
the project, mainly due to a common need of experimentation with data and
algorithms to get a better understanding on achievable requirements. Hence,
they may be refined or more precisely specified as the project progresses. The
ML specification template highlights these concerns with the letter E. Our
online material 111https://doi.org/10.5281/zenodo.6503232 presents an example
of the ML-enabled system specification template, filled out retroactively with
the support of practitioners with information of two real industrial ML-
enabled system projects.

##  III Concluding Remarks

The success of ML-enabled systems involves understanding business context and
problems, translating them into ML tasks, designing and experimenting with
algorithms, evaluating models, implementing telemetry, among other tasks. This
needs to be considered from early stages of ML software development. Based on
a literature review [12], practical experiences [9], and an initial validation
of the ML specification concerns[13], we presented our work towards an
approach for specifying ML-enabled systems considering five different
perspectives: objectives, UX, infrastructure, model, and data. The main
contribution of this paper is to provide two new artifacts that can be used to
help specifying ML-enabled systems: (i) the perspective-based ML task and
concern diagram and (ii) the ML requirements specification template. We
believe that these artifacts can help requirements engineers with an overview
of ML concerns that should be analyzed together with business owners,
designers, software engineers, and data scientists to specify ML-enabled
systems. Future work includes exploring how traditional and new RE techniques
can use the perspective-based ML task and concern diagram and the ML
specification template. In addition, we plan to conduct empirical studies to
evaluate the completeness and feasibility of using the artifacts presented in
this work.

## Acknowledgment

We would like to thank the CAPES agency for financial support.

## References

  * [1] K. Ahmad, M. Bano, M. Abdelrazek, C. Arora, and J. Grundy, “What’s up with requirements engineering for artificial intelligence systems?” in _2021 IEEE 29th International Requirements Engineering Conference (RE)_. IEEE, 2021, pp. 1–12.
  * [2] A. Arpteg, B. Brinne, L. Crnkovic-Friis, and J. Bosch, “Software engineering challenges of deep learning,” in _2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)_. IEEE, 2018, pp. 50–59.
  * [3] G. Barash, E. Farchi, I. Jayaraman, O. Raz, R. Tzoref-Brill, and M. Zalmanovici, “Bridging the gap between ml solutions and their business requirements using feature interactions,” in _Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2019, pp. 1048–1058.
  * [4] H. Challa, N. Niu, and R. Johnson, “Faulty requirements made valuable: on the role of data quality in deep learning,” in _2020 IEEE Seventh International Workshop on Artificial Intelligence for Requirements Engineering (AIRE)_. IEEE, 2020, pp. 61–69.
  * [5] T. Chuprina, D. Mendez, and K. Wnuk, “Towards artefact-based requirements engineering for data-centric systems,” _arXiv preprint arXiv:2103.05233_ , 2021.
  * [6] F. Dalpiaz and N. Niu, “Requirements engineering in the days of artificial intelligence,” _IEEE Software_ , vol. 37, no. 4, pp. 7–10, 2020.
  * [7] F. Ishikawa and N. Yoshioka, “How do engineers perceive difficulties in engineering of machine-learning systems?-questionnaire survey,” in _International Workshop on Conducting Empirical Studies in Industry (CESI) and 6th International Workshop on Software Engineering Research and Industrial Practice (SER &IP)_, 2019, pp. 2–9.
  * [8] ISO/IEC. (2012) Iso/iec 25012: Software engineering – software product quality requirements and evaluation (square) – data quality model. [Online]. Available: https://www.iso.org/standard/35736.html
  * [9] M. Kalinowski, H. Lopes, A. F. Teixeira, G. da Silva Cardoso, A. Kuramoto, B. Itagyba, S. T. Batista, J. A. Pereira, T. Silva, J. A. Warrak _et al._ , “Lean r&d: An agile research and development approach for digital transformation,” in _International Conference on Product-Focused Software Process Improvement_. Springer, 2020, pp. 106–124.
  * [10] K. Nakamichi, K. Ohashi, I. Namba, R. Yamamoto, M. Aoyama, L. Joeckel, J. Siebert, and J. Heidrich, “Requirements-driven method to determine quality characteristics and measurements for machine learning software and its evaluation,” in _2020 IEEE 28th International Requirements Engineering Conference (RE)_. IEEE, 2020, pp. 260–270.
  * [11] S. Nalchigar, E. Yu, and K. Keshavjee, “Modeling machine learning requirements from three perspectives: a case report from the healthcare domain,” _Requirements Engineering_ , vol. 26, no. 2, pp. 237–254, 2021.
  * [12] H. Villamizar, T. Escovedo, and M. Kalinowski, “Requirements engineering for machine learning: A systematic mapping study,” in _2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)_. IEEE, 2021, pp. 29–36.
  * [13] H. Villamizar, M. Kalinowski, and H. lopes, “A catalogue of concerns for specifying machine learning-enabled systems,” in _Workshop on Requirements Engineering (WER)_ , 2022, p. 14.
  * [14] A. Vogelsang and M. Borg, “Requirements engineering for machine learning: Perspectives from data scientists,” in _2019 IEEE 27th International Requirements Engineering Conference Workshops (REW)_. IEEE, 2019, pp. 245–251.