Interpreting a Semantic Segmentation Model for Coastline Detection

F. M. Lastname

F. M. Lastname

Conor O'Sullivan$^{1,2}$ $^{\dagger}$,
Seamus Coveney$^{3}$,
Xavier Monteys$^{4}$,
and Soumyabrata Dev$^{1,2}$

$^{1}$ADAPT SFI Research Centre, Dublin, Ireland

$^{2}$School of Computer Science, University College Dublin, Ireland

$^{3}$Envo-Geo Environmental Geoinformatics, Skibbereen, Ireland

$^{4}$Geological Survey Ireland, Dublin, Ireland

$^{\dagger}$ Presenting author and corresponding author

We interpret a deep-learning semantic segmentation model used to classify coastline satellite images into land and water. This is to build trust in the model and gain new insight into the process of coastal water body extraction. Specifically, we seek to understand which spectral bands are important for predicting segmentation masks. This is done using a permutation importance approach. Results show that the NIR is the most important spectral band. Permuting this band lead to a decrease in accuracy of 38.12 percentage points. This is followed by Water Vapour, SWIR 1, and Blue bands with 2.58,  0.78 and 0.19 respectively. Water Vapour is not typically used in water indices and these results suggest it may be useful for water body extraction. Permuting, the Coastal Aerosol, Green, Red, RE1, RE2, RE3, RE4, and SWIR 2 bands did not decrease accuracy. This suggests they could be excluded from future model builds reducing complexity and computational requirements.

Interpretability is an important aspect of machine learning in remote sensing. It means that researchers and industry professionals can understand the reasoning behind model predictions. In remote sensing, the data is often complex and underlying processes are not fully understood. By understanding the reasons for predictions, we can gain new insight into the process being studied. Researchers and professionals are also more likely to trust the results of a model if we provide reasons for them. Finally, interpretability can help identify reasons for errors in predictions and help improve the accuracy of future models.

A particular question we seek to answer is which spectral bands are most important to the predictions of a deep learning model. The model is used for the semantic segmentation of coastline images— classifying each pixel into land or water. By understanding this, we can relate the results to existing research on spectral bands for water body extraction. The hope is that this will provide valuable insight for future model builds.

Related Work

1|c|Indices                                                                                1c|Acronym     1c|c]@c@Spectral bands
used in formula
c]@l@Normalized Difference
Water Index <cit.>                                   NDWI                                      NIR and Green
c]@l@Automated Water
Extraction Index with
Shadows Elimination <cit.>     AWEIsh                                    c]@l@Blue, Green,
NIR, SWIR1
and SWIR2
Water Index 2015 <cit.>                                                                                WI2015                                    c]@l@Green, Red,
NIR, SWIR1
and SWIR2
Water Index 2 <cit.>                                                                                WI2                                       Blue and SWIR2
Sentinel-2 Water Index <cit.>                                                                                SWI                                      RE1 and SWIR2

Indices used for water extraction. The last column gives the spectral bands used in the calculation for the given index.

Previous research has focused on quantifying the individual predictive capability of spectral bands or indices using statistical methods. In terms of individual bands, the NIR band has shown to be effective for water extraction <cit.>.  Similarly, indices developed for water body extraction have been listed in Table <ref>. In particular, SWI was developed for automated coastline detection. These indices are calculated by combining multiple spectral bands. When interpreting a deep learning model used for water extraction, we can expect similar bands to be important.

When training a deep learning model, it is not necessary to calculate indices beforehand. This is because all available spectral bands can be used as input into the model  <cit.>. Through the process of training, the most important bands and relationships between the bands are used to make predictions. In some sense, the most appropriate indices can be encoded in the model's architecture. However, due to the black-box nature of deep learning models, these relationships can be difficult to uncover.

There is research on interpreting deep learning models for semantic segmentation. Researchers have considered parameter gradients <cit.> and SHAP values <cit.>. These approaches aim to understand which individual pixels are most important for segmentation predictions. Additionally, they do not use satellite images as input. We want to understand which spectral bands are important. This involves understanding the overall importance of the band and not the individual pixels in the bands.

Our main contributions are to:

* Replicate a deep learning model in previous work used for coastal water body extraction <cit.>.

* Interpret the model using a permutation feature importance approach to understand which spectral bands are most important to predictions.

* We open source the code and model to reproduce the results of this paper. These can be found in the GitHub repository [In the spirit of reproducible research, the codes and model to reproduce the results of this paper can be found here: <https://github.com/conorosully/interpreting-coastline-unet>.]

We base our analysis on the Sentinel-2 Water Edges Dataset (SWED) <cit.>. This contains Sentinel-2 satellite images and binary segmentation masks from a variety of coastlines. This includes 28,224 training observations. Each observation includes 12 spectral bands as input and a binary segmentation mask, where each pixel was classified as land (0) or water (1), as output. These were annotated using a semi-supervised approach which used Coastal Aersoal, green, red, NIR and SWIR1 bands as input into a clustering algorithm. The dataset also contains 98 test images from 49 testing locations. These were manually annotated.

Model Architecture and Training
As seen in Figure <ref>, an adaption of the U-Net architecture has been used <cit.>. The architecture consists of an encoder, bottleneck and decoder. The encoder has four convolutional blocks (conv). Each block contains two convolutional layers with a 3 x 3 kernel followed by an Exponential Linear Unit (ELU) activation and then a batch normalisation layer. Finally, each block is down-sampled by a 2 x 2 max-pooling layer. The bottleneck has the same structure except for the max-pooling layer. The decoder has four blocks (deconv). These are symmetrical to the encoder blocks. Skip connections concatenate the final convolutional layer in each convolution block to the first layer in the decoder blocks. This first layer is a 2x2 deconvolutional layer followed by two convolutional layers with a 3 x3 kernel, ELU activation and batch normalisation. The last encoder block ends with a softmax activation function.

This architecture was selected because we wanted to replicate the model architecture and training process in previous research  <cit.>. This research experimented with different cost functions. We used the one that provided the best results based on the metrics described in the Evaluation Metrics section. This is the cross entropy loss function.  We use a training and validation set of sizes 25,401 and 2,823 respectively. The model was trained for 50 epochs and we select the one that had the lowest validation loss.

Model architecture consisting of an encoder, bottleneck and decoder. Skip connections are used to concatenate layers in the encoder to symmetrical layers in the decoder. The number below each layer is the number of channels in that layer.

There were some aspects of the training process that we were not able to replicate exactly. This includes the training batch size used and the exact way the input has been scaled. Hence, we decided on a batch size of 32 and we scaled the input by dividing each pixel value by 10,000 and reduced any pixels greater than 1 to 1. There were also differences in the training and validation set sizes. Previous research had a smaller size of 23,807 and 2,661 respectively. It is not clear which training inputs were removed from SWED.

Evaluation Metrics

1l|               1l|1                  1l|0
1c|Actual     1l|1     1l|True Positive (TP)          1l|False Negative (FN)         1l|Actual Positive (P)
1c|Value      1l|0     1l|False Positive (FP)         1l|True Negative (TN)          1l|Actual Negative (N)
2l|                                                      1l|Predicted Positive (P')     1l|Predicted Negative (N')     1l|Total (T)

Confusion matrix for pixel classification. Water pixels are represented by a value of 1 and land pixels are represented by a value of 0.

To confirm that our model produces similar results, we evaluate it with metrics based on the confusion matrix in <ref>. Water is taken as the positive class and land as the negative class. Hence, TP and FP are the numbers of correctly and incorrectly labelled water pixels. TN and FN are the numbers of correctly and incorrectly labelled land pixels. For each of the 98 test images, we calculate the metrics listed below and take the average for each metric as the final value.

$$Accuracy = \frac{TP+TN}{TP+TN+FP+FN}$$

$$Balanced Accuracy = 0.5 \left (\frac{TP}{TP+FN} + \frac{TN}{TN+FP} \right )$$

$$      Precision = \frac{TP}{TP+FP} $$

$$       Recall = \frac{TP}{TP+FN} $$

$$        F1 = 2 \left(\frac{Precision*Recall}{Precision+Recall} \right) $$

Permutation importance
We interpret this model using a permutation importance approach. We identified 3 images, listed in the appendix, with erroneous annotations. To give us a reliable comparison to previous work, we removed these from the test set leaving us with 95 images. These three images were not removed from the above validation step. Hence, permutation importance scores are calculated for a spectral band or combinations of bands by:

* Permuting the respective band(s) in each of the 95 test images. This is done by randomly shuffling the pixels within an image's band.

* Using the permuted image as input, the trained U-Net is used to predict a binary segmentation mask.

* Calculate the average accuracy over the 95 images.

* Calculate the decrease in average accuracy when compared to the average accuracy obtained using the original 95 images.

We repeated the above process 5 times and take the average decrease in accuracy as the final importance score. We calculate these permutation scores for each of the 12 spectral bands. We also include combinations of bands that correspond to the water extraction indices mentioned in Table <ref> and the three visible light bands.

Results and Discussion
Evaluation Metrics

The evaluation results are presented in Table <ref>. We note differences between the original and the replicated model results. We consider these differences to be insignificant and are likely due to the aspect of the modelling development process that could not be replicated exactly. We, therefore, expect the interpretation of the replicated model to be similar to an interpretation of the original model.

1|c|Model     1c|Accuracy     1c|c]@c@Balanced
Accuracy     1c|Precision     1c|Recall     1c|F1
Original <cit.>                  0.937                0.910                                                                    0.916                 0.948              0.922
Replicated                               0.938                                      0.916                                                                                0.924                                       0.954                                   0.930

Evaluation metrics for the original and replicated model. The difference can likely be attributed to aspects of the model development process that could not be replicated exactly.

Permutation Importance of Individual Bands

The permutation importance of each spectral band is given in Figure <ref>.  These are calculated by permuting only the given band. The three largest importance scores are given for NIR, Water Vapour and SWIR 1. Permuting these bands lead to a decrease in accuracy of 38.12, 2.58 and 0.78 percentage points. The remaining bands provided a small ($<0.5$) positive or negative percentage point change.

Permutation importance scores for spectral bands used as input into the coastal segmentation model. The scores give a decrease in average accuracy when the respective band is permuted.

Based on these importance scores, NIR is the most important band for predicting the segmentation masks. Its score was over 14 times higher than the second-highest score for Water Vapour. We should consider that this may be a result of NIR being one of the bands used in the semi-supervised process for labelling the training set. In other words, the importance may reflect a bias towards this band in the training set and not the intrinsic predictive capability of this band. Similarly, SWIR 1 was also used in the semi-supervised process.

On the other hand, Water Vapour was not used in this process. Yet, based on this analysis it is the second most important spectral band. This result is particularly interesting in light of previous research. In Table <ref>, we saw that it was not typical to use this band in indices for water extraction. This suggests that Water Vapour may have been overlooked when developing  these indices.

Permutation Importance of Indices

One consideration when analysing the above results is that spectral bands can provide similar information. In other words, the pixel values in the bands can be correlated. A robust model may use the information from all bands even if they provide similar information. As a result, permuting one band may not lead to a significant decrease in accuracy. This is because the model can still use the information in the remaining bands to make predictions.

One way to test this hypothesis is by permuting multiple bands simultaneously. For example, the "Visible Light" bar in Figure <ref> gives the importance score when we permute the Blue, Green and Red spectral bands. This leads to a decrease in accuracy of 0.25 percentage points. When we permutated these bands individually the accuracy changed by 0.19, -0.1 and  -0.1 percentage points respectively. Based on this, neither of these bands is particularly important to the prediction. Additionally, when combined their importance is not significantly improved.

Similarly, the "Not Important" band gives the importance when we permute all the bands that did not have a positive importance in Figure <ref>. This includes Coastal Aerosol, Green, Red, RE1, RE2, RE3, RE4 and SWIR 2. When permuted together, these bands changed the average accuracy by -0.47 percentage points. Based on this, we can be more certain that the model does not rely on these bands to make predictions.

Permutation importance scores for coastline segmentation model. The scores give the percentage change in average accuracy when combinations are bands are permuted.

The remaining bars in Figure <ref> give the importance score if we permute the bands in the indices listed in Table <ref>. This provides evidence for the effectiveness of these indices for water body extraction. Although we should keep in mind that this is not an indication of the individual predictive capability of the indices. It is only an indication that the model has used the bands in the given index's calculation when making predictions. For example, SWI is calculated using RE1 and SWIR2. In Figure <ref> we saw that neither of these bands was important when permuted individually. Similarly, the NDWI, AWEIsh and WI2015 all use the NIR band in their calculations and had similar importance scores to this band.

Limitations and Future Work

We interpreted a semantic segmentation model used to classify the pixels of coastline images into land and water. We used a permutation importance approach to understand which spectral bands were most important when predicting segmentation masks. This analysis showed that the NIR was the most important. Permuting this band lead to a decrease in accuracy 14 times higher than any other band. This is followed by Water Vapour, SWIR 1 and Blue bands which decreased the accuracy by 2.58,  0.78 and 0.19 percentage points respectively.

The Water Vapour score was particularly interesting. The analysis suggests that this band is useful for extracting coastal water bodies. However, it is not typical for this band to be used in water extraction indices. Future research could investigate this potentially leading to a water extraction index using this band.

The other spectral bands, Coastal Aerosol, Green, Red, RE1, RE2, RE3, RE4 and SWIR 2, were not important to the prediction. They produced a small increase in accuracy ($<0.5$ percentage points) both when permuted individually and when they when all permuted together. This suggests these bands could be removed from future model development without having a negative impact on model accuracy. At the same time, fewer data can reduce the complexity and computational requirements of future models.

Lastly, we plan to explore other methods of interpreting semantic segmentation models. In this paper, we focused on the overall importance of spectral bands. Other methods, such as SHAP values, can be used to understand which pixels are important to a prediction. This could lead to new insights such as why the model is incorrectly predicting certain pixels.

This publication has emanated from research conducted with the financial support of Science Foundation Ireland under Grant number 18/CRT/6183. For the purpose of Open Access, the author has applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission. This research was conducted with the financial support of Science Foundation Ireland under Grant Agreement No. 13/RC/2106_P2 at the ADAPT SFI Research Centre at University College Dublin. The ADAPT Centre for Digital Content Technology is partially supported by the SFI Research Centres Programme (Grant 13/RC/2106_P2) and is co-funded under the European Regional Development Fund.

§ APPENDIX

§.§ Remove images

The images listed below were removed from the SWED test set for this analysis. The first image  was removed as the mask is flipped—land is labelled as 1 and water 0. The other two are removed as part of the land has not been labelled as land.


# Digital Twin-Native AI-Driven Service Architecture for Industrial Networks

Kübra Duran12, Matthew Broadbent1, Gökhan Yurdakul3, and Berk Canberk1
1School of Computing, Engineering and The Build Environment, Edinburgh Napier
University, United Kingdom
2Computer Engineering Department, Istanbul Technical University, Turkey
3BTS Group, Istanbul, Turkey
Email: {K.Duran, M.Broadbent<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The dramatic increase in the connectivity demand results in an excessive
amount of Internet of Things (IoT) sensors. To meet the management needs of
these large-scale networks, such as accurate monitoring and learning
capabilities, Digital Twin (DT) is the key enabler. However, current attempts
regarding DT implementations remain insufficient due to the perpetual
connectivity requirements of IoT networks. Furthermore, the sensor data
streaming in IoT networks cause higher processing time than traditional
methods. In addition to these, the current intelligent mechanisms cannot
perform well due to the spatiotemporal changes in the implemented IoT network
scenario. To handle these challenges, we propose a DT-native AI-driven service
architecture in support of the concept of IoT networks. Within the proposed
DT-native architecture, we implement a TCP-based data flow pipeline and a
Reinforcement Learning (RL)-based learner model. We apply the proposed
architecture to one of the broad concepts of IoT networks, the Internet of
Vehicles (IoV). We measure the efficiency of our proposed architecture and
note 30% processing time-saving thanks to the TCP-based data flow pipeline.
Moreover, we test the performance of the learner model by applying several
learning rate combinations for actor and critic networks and highlight the
most successive model.

###### Index Terms:

digital twin, internet of things, internet of vehicles, reinforcement learning

##  I Introduction

Over the recent years, Digital Twin (DT) has been raised as one of the most
stimulating technological developments to shape tomorrow’s industrial
environments[1]. Especially for large-scale Internet of Things (IoT) networks
with an excessive amount of IoT sensors, the need for monitoring and efficient
management has increased the importance of DT integration. Moreover, Internet
of Vehicles (IoV) networks enable data exchange and intelligent collaboration
between various objects, such as vehicles, environmental sensors throughout
roads, and traffic light sensors. As is one of the broader concepts of IoT,
the similar needs and the DT integration trend apply to IoV networks in the
monitoring and management processes.

Here, several challenges need to be addressed to implement a DT integration
considering the broad concept of IoT networks. Such a design should address
the following challenges:

  * •

Perpetual Connectivity: The conventional one-way interaction between sensors
and their implemented models is insufficient to serve seamless integration
and, thus, accurate monitoring due to the lack of the feedback interface.
Therefore, the communication links between the sensors and their virtual
models should be maintained in both directions continuously.

  * •

Sensor Data Streaming: The DT representation of physical world sensors
provides exact monitoring to keep track of the congestion occurrence.
Therefore, gathering the sensor’s information plays a key role in the design
of a DT-based service architecture. At this point, the question “How much
granularity and processing is required?” should be decided according to the
application of the IoV network to design an efficient data streaming approach.

  * •

Depth of Data Insights: Although there are concrete examples of Artificial
Intelligence/Machine Learning (AI/ML) applications in the management of
industrial networks, they become insufficient against spatiotemporal changes
in the application environment. Therefore, enhanced learning models with high
stability should be developed to maintain a deep understanding and precise
behaviour modelling of the IoT environment.

Figure 1: Proposed Digital Twin-native service architecture for IoV use case.

##  II Related Works

Most of the current literature for IoV networks focuses on vehicle routing
strategies considering vehicle information rather than environmental
information. For instance, [2] uses edge and cloud computing to reduce traffic
congestion by proposing optimized routes for each connected vehicle. Likewise,
[3] proposes a route recommendation framework to produce the shortest path for
each source-destination pair promptly. Moreover, [4] uses a deep reinforcement
learning algorithm to manage traffic light timings for road management
systems. However, the whole IoV network could not be presented realistically
in these studies due to the lack of considering a divergent set of
environmental entity information. On the other hand, despite the increasing
trend in deploying DT in road management systems, the enhanced DT modelling
challenge still needs to be addressed. None of the studies on transportation
systems maintains a specific data flow scheme to handle high-quality
representation needs by preserving the implementation needs. For instance, [5]
studies DT modelling and points out a data representation framework by holding
relation information; however, it does not implement the proposed model on
road transportation networks. Also, a DT assisted Mobile Edge Computing (MEC)
framework for Industrial IoT (IIoT) networks is proposed in [6] to decrease
the end-to-end latency. Furthermore, [7] establishes a traffic
infrastructure’s efficiency assessment model based on DTs with traditional
database models, resulting in high processing time. Similarly, [8] introduces
a learning approach to investigate the impact of taxi trips from an
environmental issues perspective. Although these studies perform DT platform
development efforts, they result in high querying and processing time with the
traditional modelling approaches. Regarding AI-driven service provisioning,
[9] and [10] apply ML algorithms in order to come up with a stable network
architecture for core networks. Moreover, there is the application of AI
methods that focuses on the estimation of emission levels. For example, [11],
and [12] propose an emission regulatory framework for transportation networks
by estimating the emission levels. In addition, [13] conducts emission
prediction for vehicles using data generated by in-vehicle sensors by
utilizing LSTM models. Moreover, [14] predicts exhaust emissions by using
multiple 1D-Convolutional Neural Networks (CNN) and Artificial Neural Networks
(ANNs). Also, [15] estimates the discovery states in a core network
architecture by proposing a DT-based approach. Although the results of these
works demonstrate that DT provides accurate estimation results, there is no
recommendation scheme implemented by taking into consideration the particular
event definitions for sensors.

Recent advancements in DT for IoV networks have been described above.
Nevertheless, none of these studies deals with the perpetual connectivity,
sensor data streaming, and qualified data insights challenges, as listed in
the introduction section. Thus these challenges remain unresolved. For this
reason, this study is situated around the research question, “How to serve (i)
a perpetual communication link between a physical entity and its virtual model
to ensure seamless integration, (ii) an efficient sensor data streaming method
with the target of preserving processing time, and (iii) an intelligent
service model with the capability of accurate behaviour modelling in IoT
networks scenario?”. To address this, we propose a DT-native AI-driven service
architecture consisting of a TCP-based data flow pipeline and a Reinforcement
Learning (RL)-based learner model. We summarize the contributions of this
study below:

  * •

We implement an Internet of Vehicles (IoV) network scenario as the physical
network and present its realistic representation by enabling the connection
via the Traffic Control Interface (TraCI).

  * •

We introduce a novel DT-native TCP-based data flow pipeline between the
physical and digital twin layer of the IoV network to serve sensor data
streaming in an event-based manner and trigger the reverse data flow, in other
words, feedback interface, if required.

  * •

We propose a learner model with Deep Deterministic Policy Gradient (DDPG)
algorithm to experience the IoV environment by learning the behaviours and
recommend optimal rules to avoid congestion occurrence.

The remainder of the article is organized as follows: Section III explains the
proposed DT-native AI-driven service framework with the IoV use case
implementation. The performance of the proposed model is evaluated in Section
IV. Finally, Section V concludes the paper.

##  III DT-Native AI-Driven System Architecture

###  III-A Physical Twin Layer

Figure 2: Proposed data flow pipeline between Physical Twin Layer and Digital
Twin Layers.

In this layer, we work on an Internet of Vehicles (IoV) scenario as a part of
the industrial networks. This is because the connectivity problems between the
sensors (traffic light sensors, buildings etc.), vehicles, and the
infrastructure are becoming significant to maintain the needs of smart city
applications. For this reason, we focus on a heterogeneous IoT environment
that includes different types of sensors and moving vehicles environment.
Therefore, this layer includes all physical entities within a transport
network, such as lanes, road intersections, moving vehicles, IoT sensors,
traffic lights etc. To construct a realistic traffic management scenario, we
implement our Physical Twin Layer by using the microscopic and continuous
traffic simulator Simulation of Urban MObility (SUMO). The block diagram of
the proposed architecture is given in Fig. 1.

###  III-B Digital Twin Layer

This layer includes the digital replica of physical sensors constructed by
using the gathered event data. We deploy our Digital Twin models through a
Docker Engine due to its efficiency in packaging and deploying different
services with multiple clusters. Therefore, we deploy a Dockerfile with the
base image and the dependencies to use as a data container in hosting the
Extensible Markup Language (XML)-based twin sensor instances. In this way, we
expect to improve the delivery of realistic insights as part of the Twin
Service Layer operations.

TABLE I: IoV Network Event Types Event | Definition
---|---
$E_{1}$ | (Vehicle ID, Location, Speed, Time Interval)
$E_{2}$ | (Sensor ID, Location, Lane, Time Interval)
$E_{3}$ | (Traffic light ID, Location, Lane, Duration Rate (red/green)

In the interaction between Digital Twin Layer and Physical Twin Layer, we
define a novel data flow pipeline according to the needs of the simulation
environment. As we use the SUMO-based physical twin layer in our DT
architecture, we implement Transmission Control Protocol (TCP) to initiate the
TraCI interface of the SUMO. Thus, the pipeline entities, including the TraCI
interface, are explained below:

  * •

TraCI Server: TraCI permits the real-time streaming of implemented transport
scenarios. Therefore, to enable interaction, a port on the physical road
network side is calibrated for SUMO to act as a data server and start to
listen to incoming connections.

  * •

TraCI Client: It is implemented in a Python script file to perform data
retrieval requests.

  * •

Event Publisher (EP): After getting the successful TCP connection between the
server and the client, the realistic road network data with junctions,
vehicles, route directions, and speed information etc., are streamed by using
the function list of TraCI. Thereafter, the gathered network information with
the specified events is conveyed to the Event Broker.

  * •

Event Broker (EB): It is used to transform the gathered events into individual
virtual twins according to the event types. More specifically, we use a
traditional event data (XML in our case) format.

  * •

Event Reactor (ER): It passes the messages to the implemented Physical Twin
Layer scenario. More specifically, it behaves as a feedback interface to reach
the simulation environment and change the specified configurations.

The IoV-specific events from the experimental scenario are given in Table-I.
As given in this table, we define three types of event classes in order to
distinguish the transport network entities. In this circumstance, $E_{1}$
types imply the vehicle sensors with their location information (latitude and
longitude), speed in $km/h$, and time interval in which the event is recorded.
On the other side, $E_{2}$ type events stand for the IoT sensor recordings
located within the transportation environment, throughout the roads and
junctions. These environmental sensors include location information, lane
information to describe their presence regarding the main roads, and time
interval information. Lastly, $E_{3}$ type events stand for the traffic
lights’ recordings. These events include location information, lane
information and additional duration rate of the red and green light signal
timings. This attribute is used to decide the optimal number of vehicles in a
road network to avoid congestion situations.

The data flow pipeline we proposed is given in Fig. 2. As seen from the
figure; there are two main units, TraCI Client and TraCI Server, making the
Physical Twin Layer and Digital Twin Layer connected by realizing TCP
protocol. We denote these connection phases by marking them as the Step 1
category. More specifically, in step 1.a, client asks for a TCP connection by
sending a $SYN$ packet to the server side. In step 1.b, the server responses
with a $SYN-ACK$ packet to accept the connection request of the client. In
step 1.c, the client sends an $ACK$ packet to the server and marks itself as
connected. After the connection establishment, the client sends a $NOTF$
message to notify the EP to start data streaming. Then, the EP sets the
$STREAM$ flag, and the server starts sending the data to the EP side. Here, EP
behaves as a relay point by conveying the streamed data to the EB side. This
event gathering to the EB side continues till the $STREAM$ flag is reset. In
other words, the EB in our DT-native data flow pipeline gathers event
information from EP as long as the TCP connection is established and the data
streaming flag is set. Moreover, in case of any fault between these entities,
$EB$ disables event streaming and waits till the recovery period ends. At the
end of this period, it sends a $AWAKE$ message to $EP$ to trigger resetting
the $STREAM$ flag. After that, $EB$ places the gathered events to separate
queues to process according to their types. The event types are given in Table
I, and explained in the above paragraph. The processing of event evaluations
is performed by calculating their occurrence densities. For this, we calculate
each of the event densities as,

$\rho(E_{i})=\frac{\sum_{k}O_{i}}{L_{Q_{i}}}$ (1)

where

$1\leq i\leq 3\ \text{,}\ 1\leq k\leq n$

In this formula, $\rho(E_{i})$ is the $i^{th}$ event density, $O_{i}$ is the
occurrence of $i^{th}$ type events and $L_{Q_{i}}$ implies the number of
events waiting in the related queue. In addition, $i$ is the number of event
classes, and $k$ is the total streamed number of events. According to the
calculated event densities, the $EB$ decides the data flow direction by
considering the predefined threshold values. After deciding the data flow
direction and the new configurations, the feedback interface is enabled to
interact with the $ER$. By doing this, required changes are made to stabilize
the realistic simulation scenario (step 4.b). After completing the data flow
in both directions, the TCP connection is terminated between the client and
server.

###  III-C Twin Service Layer

This layer includes an RL engine to implement Reinforcement Learning algorithm
to decide the optimal physical environment output recommendations to avoid
congestion in an IoV scenario. More specifically, we have implemented the DDPG
algorithm, which combines deep learning and policy gradient methods to learn
optimal policies in the simulation environment with a continuous action space.
As in our simulation environment, there are three types of events related to
the vehicles, environment and traffic lights; the output of the DDPG algorithm
will consist of three types of continuous actions, such as deciding a speed
limit for vehicles flowing through a particular junction, and deciding the
traffic lights’ signalling time in order to prevent congestion scenarios. The
block diagram for the proposed learner model within this layer is given in
Fig. 3.

Figure 3: The interaction of environment and the learner model in implementing
the DDPG algorithm.

Our learning model consists of four tuple {$S_{t}$, $a$, $R$, $S_{t+1}$}. Here
$S_{t}$ is the current state and represents the total number of event classes
in our IoV implementation scenario; $a$ is action space and decides the total
number of sensor records minimizing the congestion probability in an IoV
network; $R$ is an immediate reward, and $S_{t+1}$ is the decided next state.
We apply DDPG algorithm by defining the reward function for state and action
pairs $(S_{i},a_{i})$. It executes the chosen action among a divergent set of
actions and observes the next state and reward by storing the experience
tuples in an experienced pool. As shown in Fig.3, the interaction between the
simulation environment and the learner model is realized through the TraCI
interface. The events are observed, and the actor and critic networks trace
their change.

The details of our proposed learner model consisting of an actor-network, a
critic network and two target networks are explained below:

  * •

Actor network ($\theta^{A}$): It learns the policy to select optimal actions.
For example, in our IoV scenario, the actor-network tries to implement
different speed limits to evaluate the results and decide on the optimal
action. Also, this network can be called a policy network.

  * •

Critic network ($\theta^{C}$): It learns to evaluate the quality of the chosen
actions. For instance, it evaluates if a decided speed limit for a vehicle
cause to any congestion, resulting in our IoV scenario implementation. Also,
this network can be called a value function network.

  * •

Actor target network ($\theta^{A^{{}^{\prime}}}$): It is periodically updated
during the learning process. It is a time-delayed copy of the actor-network.

  * •

Actor critic network ($\theta^{C^{{}^{\prime}}}$): It is periodically updated
during the learning process. It is a time-delayed copy of the critic network.

The DDPG algorithm updates the Q values based on the environment and learning
rate values by using the Bellman equation:

$y_{i}=R_{i}+\alpha
Q^{{}^{\prime}}+(S_{i+1},C^{{}^{\prime}}(S_{i+1}|\theta^{C^{{}^{\prime}}})|\theta^{Q^{{}^{\prime}}})$
(2)

In this equation, $\alpha$ is the learning rate and takes values from a
predefined interval. We also try to minimize the mean squared loss between the
updated Q value and the originally calculated Q value given as:

$Loss=\frac{1}{E}\sum_{i}(y_{i}-A(S_{i},a_{i}|\theta^{A}))^{2}$ (3)

In this equation, $E$ implies the total number of performed experiences during
the algorithm run with the defined optimizer function.

##  IV Performance Evaluation

In this section, we show the performance of our proposed DT-native service
architecture by considering (i) the efficiency of the proposed data flow
pipeline by changing the number of sensors in the implemented IoV scenario and
observing the average processing time and (ii) the efficiency of the proposed
learner model by measuring its error and the resultant congestion scenarios.
The simulation parameters are given in Table II.

Experimental Setup: We carried out Physical Twin Layer implementations on
SUMO© environment. We deployed the Digital Twin layer over a Docker container.
Also, as the basis of our learning model implementation, we used the
TensorFlow library.

TABLE II: Simulation Parameters Parameters | Values
---|---
Number of event classes | {1, 2, 3}
Total number of sensors | {5, 20, 90}
Learning rate for policy networks | [0.001, 0.2]
Learning rate for Q-networks | [0.001, 0.2]
Optimizer | adam
Discount factor | 0.8
Batch size |  256
Confidence interval | 95%

Figure 4: Average processing time for traditional data flow and proposed DT-
native data flow.

To measure the performance of the proposed DT-native data flow architecture,
we consider three types of events collected for the increasing number of
sensors. For this, we assume there are five IoT sensors in the initial state
of the simulation. After performing the event collection as described in the
above sections, we increased the number of sensors to twenty and then to
ninety. We perform this increment in order to observe the behaviour of
entities, such as EP, EB, and ER, which perform between the Physical Twin
Layer and Digital Twin Layers of our DT model. We observe the average
processing time of the proposed DT-native data flow and compare it with the
manual data flow trigger approach as given in [16]. Our recorded results are
given in Fig. 4. We notice that the processing time is saved by 30% on average
with the proposed DT-native data flow. This is because the EB keeps track of
the specified events and calculates density values for each of these event
types. According to the calculated values, it triggers the flow direction and
applies the optimal number of sensors in order to avoid congestion scenarios
in an IoV scenario. Thanks to this cognitive approach, the proposed pipeline
can rapidly handle dynamic scenarios.

Figure 5: Average latency for traditional data flow and proposed DT-native
data flow.

Moreover, we investigate the efficiency of the proposed DT-native data flow
architecture by measuring the latency values for the implemented simulation
run, which is realized in the former step. We record the latency values in
$ms$ for five, twenty, and ninety sensor cases. As seen in Fig. 5, the
traditional and DT-native data flow pipelines have similar latency values for
each scenario. The main reason is that the EB performs event analysis and
density calculation in DT-native data flow. This step directly causes
operational latency in the data streaming process. Therefore, if an IoT
network consists of a high amount of sensors, the proposed DT-native data flow
pipeline performs favourably in terms of average latency when compared to the
case when we implement ninety sensors.

Figure 6: The success of the DDPG algorithm with changing learning rate
combinations for actor and critic networks.

Furthermore, to measure the performance of the proposed learner model, we
perform DDPG with the changing combinations of learning rates considering the
interval $[0.001,0.2]$ for actor and critic networks. We assume that
optimizers in actor and critic networks use the same optimization function,
$adam$. Also, throughout all the episodes run, we set the discount factor
value to 0.8 to help the algorithm converges. Moreover, we use a batch size of
256. We have used the TensorFlow library in order to run the DDPG algorithm.
Our recorded results show three learning rates combination for actor and
critic networks for 135 performed episodes. As seen from Fig. 6, the most
successive learning rates combination, ($\alpha_{Actor}$, $\alpha_{Critic}$),
is observed as (0.1, 0.2) with the less mean error rates. In addition, the
case with the learning rates (0.01, 0.02) also converges at the end of the
realized episodes as compared. On the contrary, the learning rates combination
(0.001, 0.002), shown as a light blue line in Fig. 6, does not converge
through all performed episodes. This means that learning rates set in this
case are insufficient for the learner model to cover all event occurrences in
the simulation. We argue that the learning rates should be set above these
values to develop an effective DDPG run.

##  V Conclusion

In this study, we focus on three distinct challenges of IoT networks;
perpetual connectivity, sensor data streaming, and depth of the data insights.
Considering the need for continuous monitoring and enhanced analysis of IoT
networks, we propose a DT-native AI-driven service framework with a TCP-based
data flow pipeline and a Reinforcement Learning (RL)-based learner model. We
apply our proposed framework to one of the broad concepts of IoT networks, the
Internet of Vehicles (IoV) network scenario, and measure its efficiency for
processing time and latency. Our simulation results show 30% processing time
saving with the proposed TCP-based data flow pipeline as compared to manual
data flow triggering. Furthermore, we measure the efficiency of our proposed
learner model with several learning rates combination and come up with the
most successive version for actor and critic networks.

## References

  * [1] A. Sari, A. Lekidis, and I. Butun, “Industrial networks and iiot: Now and future trends,” _Industrial IoT: Challenges, Design Principles, Applications, and Security_ , pp. 3–55, 2020.
  * [2] P. Arthurs, L. Gillam, P. Krause, N. Wang, K. Halder, and A. Mouzakitis, “A taxonomy and survey of edge cloud computing for intelligent transportation systems and connected vehicles,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 7, pp. 6206–6221, 2022.
  * [3] Y. Ge, H. Li, and A. Tuzhilin, “Route recommendations for intelligent transportation services,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 33, no. 3, pp. 1169–1182, 2021.
  * [4] N. Kumar, S. S. Rahman, and N. Dhakad, “Fuzzy inference enabled deep reinforcement learning-based traffic light control for intelligent transportation system,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 22, no. 8, pp. 4919–4928, 2021.
  * [5] J. Wen, B. Gabrys, and K. Musial, “Toward digital twin oriented modeling of complex networked systems and their dynamics: A comprehensive survey,” _IEEE Access_ , vol. 10, pp. 66 886–66 923, 2022.
  * [6] T. Do-Duy, D. Van Huynh, O. A. Dobre, B. Canberk, and T. Q. Duong, “Digital twin-aided intelligent offloading with edge selection in mobile edge computing,” _IEEE Wireless Communications Letters_ , vol. 11, no. 4, pp. 806–810, 2022.
  * [7] Z. Tu, L. Qiao, R. Nowak, H. Lv, and Z. Lv, “Digital twins-based automated pilot for energy-efficiency assessment of intelligent transportation infrastructure,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 11, pp. 22 320–22 330, 2022.
  * [8] M. Ghahramani and F. Pilla, “Analysis of carbon dioxide emissions from road transport using taxi trips,” _IEEE Access_ , vol. 9, pp. 98 573–98 580, 2021.
  * [9] K. Duran, B. Karanlik, and B. Canberk, “Ai-driven partial topology discovery algorithm for broadband networks,” in _2021 IEEE 18th Annual Consumer Communications & Networking Conference (CCNC)_, 2021, pp. 1–6.
  * [10] E. Ak and B. Canberk, “Fsc: Two-scale ai-driven fair sensitivity control for 802.11ax networks,” in _GLOBECOM 2020 - 2020 IEEE Global Communications Conference_ , 2020, pp. 1–6.
  * [11] Y. Sun, Y. Hu, H. Zhang, H. Chen, and F.-Y. Wang, “A parallel emission regulatory framework for intelligent transportation systems and smart cities,” _IEEE Transactions on Intelligent Vehicles_ , vol. 8, no. 2, pp. 1017–1020, 2023.
  * [12] N. Niroomand, C. Bach, and M. Elser, “Segment-based $co^{2}$ emission evaluations from passenger cars based on deep learning techniques,” _IEEE Access_ , vol. 9, pp. 166 314–166 327, 2021.
  * [13] M. Singh and R. K. Dubey, “Deep learning model based co2 emissions prediction using vehicle telematics sensors data,” _IEEE Transactions on Intelligent Vehicles_ , vol. 8, no. 1, pp. 768–777, 2023.
  * [14] X. Fei, F. Long, F. Li, and Q. Ling, “Multi-component fusion temporal networks to predict vehicle exhaust based on remote monitoring data,” _IEEE Access_ , vol. 9, pp. 42 358–42 369, 2021.
  * [15] K. Duran and B. Canberk, “Digital twin enriched green topology discovery for next generation core networks,” _IEEE Transactions on Green Communications and Networking_ , pp. 1–1, 2023.
  * [16] E. Ak, K. Duran, O. A. Dobre, T. Q. Duong, and B. Canberk, “T6conf: Digital twin networking framework for ipv6-enabled net-zero smart cities,” _IEEE Communications Magazine_ , pp. 1–7, 2023.