# DOES AN ENSEMBLE OF GANS LEAD TO BETTER PERFORMANCE WHEN TRAINING
SEGMENTATION NETWORKS WITH SYNTHETIC IMAGES?

###### Abstract

Large annotated datasets are required to train segmentation networks. In
medical imaging, it is often difficult, time consuming and expensive to create
such datasets, and it may also be difficult to share these datasets with other
researchers. Different AI models can today generate very realistic synthetic
images, which can potentially be openly shared as they do not belong to
specific persons. However, recent work has shown that using synthetic images
for training deep networks often leads to worse performance compared to using
real images. Here we demonstrate that using synthetic images and annotations
from an ensemble of 20 GANs, instead of from a single GAN, increases the Dice
score on real test images with 4.7% to 14.0% on specific classes.

##  1 Introduction

Artificial intelligence (AI) has become a popular tool in medicine to help
clinicians work more efficiently and has been vastly discussed in recent
decades [1]. Due to the recent success of machine learning (ML) methods in
other domains and the steady increase of image-based medical examinations,
applying ML methods for medical image analysis is becoming increasingly
popular. Since ML models learn from data, the quantity and quality of medical
image data have a direct impact on the success of ML-based algorithms. These
algorithms require large datasets with annotations created by human experts.
Unfortunately, the availability of large open-source datasets in biomedical
imaging is low. The main reasons for the unavailability of large public
datasets in medical imaging can be categorized into three main groups; privacy
concerns (due to the potentially sensitive information of the patients),
expensiveness in terms of cost and time for data collection and annotation,
and lack of data due to the rarity of some diseases. Synthetic images overcome
all these problems and can help researchers in training complex models.
Recently, augmentation of datasets with synthetically generated images is one
of the techniques used to explore the insufficiency of annotated data to train
the ML algorithms [2]. These synthetically produced images can be used for
training deep networks, and can potentially be shared among peers without
breaking privacy laws and data protection regulations.

Generative adversarial networks (GANs) recently gained a lot of attention in
the computer vision community. Generative modeling is an unsupervised machine
learning task that involves discovering and learning the patterns
automatically in the input data, and then training itself to further generate
new examples. GANs have proven themselves to be useful in multiple scenarios
such as data augmentation [3], image-to-image translation [4], domain
adaption, and filling in missing information. They provide an opportunity to
produce large annotated medical datasets by producing realistic synthetic
images which can further be used to train deep models. There are also several
examples where GANs have been succesfully used for medical image applications
[5, 6, 7].
GANs work by learning a complex high-dimensional data distribution to generate
diverse high-resolution images. For image segmentation, it is necessary to
produce annotations for each synthetic image, and different approaches have
been used to achieve this. Bowles et al. [7] used a two-channel approach to
synthesize images with their corresponding annotations. Foroozandeh1 et al.
[8] used a two-step process where the authors first produce the annotations
using a noise-to-image GAN [9], and then used an image-to-image GAN [10] to
synthesize MR images from the annotations previously generated. Guibas et al.
[11] also used a similar two-step approach to produce images along with
corresponding annotations.

While the main principle of training networks with synthetic images has been
demonstrated, several papers have shown that using the synthetic images
results in lower performance when training a CNN [2, 3, 8, 12]. Eilertsen et
al. [12] demonstrated that using an ensemble of GANs leads to higher accuracy
when training classifiers with synthetic images. Therefore we investigate if
using an ensemble of GANs leads to better performance when training
segmentation networks with synthetic images and annotations. In this paper, we
follow the multi-channel approach to produce images and their corresponding
annotations at the same time.

##  2 Data

The MR images used for this project were downloaded from the Multimodal Brain
tumour Segmentation Challenge (BraTS) 2020 [13, 14, 15, 16, 17]. The training
set contains MR volumes of shape 240 × 240 × 155 from 369 patients, and for
each patient four types of MR images are available: T1-weighted (T1), post
gadolinium contrast T1-weighted (T1Gd), T2-weighted (T2), and T2 fluid
attenuated inversion recovery (FLAIR). The annotations cover three parts of
the brain tumor: peritumoural edema (ED), necrotic and non-enhancing tumour
core (NCR/NET), and GD-enhancing tumour (ET). We used 313 subjects for
training and 56 subjects for testing.

##  3 Methods

All 3D volumes were split into 2D slices, as a 2D GAN was used, and only
slices with at least 15% pixels with an intensity of more than 50 were
included in the training. This resulted in a total of 23,478 5-channel images
from the 313 training subjects, and 4,238 5-channel images from the 56 test
subjects. Each slice was zero padded from 240 x 240 to 256 x 256 pixels, as
the used GAN only works for resolutions that are a power of 2, and the
intensity was rescaled to 0 - 255. The intensities for the tumor annotations
were changed from [1,2,4] to [51,102,204], such that the intensity range is
more similar for the 5 channels.

###  3.1 GAN training

The progressive growing GAN architecture [9] was used for all experiments,
with the default settings. The code was modified to generate 5-channel images
(T1, T1Gd, T2, FLAIR, segmentation annotations) instead of 3-channel color
images. The GAN will thereby learn to jointly generate 4-channel MR images and
the corresponding tumor annotations at the same time.

The same GAN was trained 20 times and due to several sources of randomness,
each GAN will learn to sample from slightly different parts of the high
dimensional image distribution. The total training time was about 100 days on
an Nvidia V100 GPU. A total of 100,000 synthetic 5-channel images were
generated from each trained GAN. When using images from 5 GANs, each GAN
provided 20,000 synthetic images. When using images from 10 GANs, each GAN
provided 10,000 synthetic images.

There is no guarantee that the synthetic annotations will be restricted to the
same values as the real annotations ([51,102,204]). The synthetic annotations
were therefore thresholded to the closest original annotation value.

###  3.2 Tumor segmentation

The synthetic images were used to train a model to perform tumor segmentation,
and the evaluation was performed using real images in the test set. The model
structure and training setup used was inspired by the 2D segmentation code
from nnUNet [18]. The model was a U-Net [19] with an extra depth layer and
instance normalization instead of batch normalization. In addition, the ReLU
activations was swapped for leaky ReLUs with a negative slope of $10^{-2}$.
All models were trained with a loss consisting of a cross-entropy term and a
soft Dice term weighted equally. In addition, deep supervision was used,
meaning that the loss was applied on the five highest depth level with
weighting $0.5^{d}$ where $d$ is the depth.

The loss was minimized using stochastic gradient descent with Nesterov
momentum of $0.99$ and weight decay of $3\cdot 10^{-5}$. The initial learning
rate was $5\cdot 10^{-2}$ and was decreased using polynomial learning rate
decay with an exponent of $0.9$. All models were trained for $3\cdot 10^{7}$
samples or 3 days, whichever occured first. 20% of the available images were
used for validation and the model with the best mean Dice over the validation
set was used for evaluation. If both real and synthetic data was used during
training, the real dataset and the synthetic dataset were sampled equally
often.

During training a geometric, and intensity-augmentation was applied. The image
and target is first randomly rotated and scaled. Both rotation and scaling is
applied with a probability of 0.75, the image and target is rotated with an
angle uniformly sampled from $[-30^{\circ},30^{\circ}]$ and the width and
height is scaled (independently) with a scale factor uniformly sampled from
$[0.9,1.1]$. Then the four input channels are augmented by; adding Gaussian
noise (applied with probability 0.5, zero mean and standard deviation
uniformly sampled from $[0.0,0.05]$), blurring the image (applied with
probability 0.2, gaussian blurring with standard deviation sampled from
$[0.5,1.0]$), faking lower resolution (zooming with a factor between 0.75 and
1.0 and then upsampling) and changing the gamma factor (scaling it with a
factor between 0.8 and 1.2). Lastly the input image channels are normalized
using Z-score normalization.

When training the 20 different GANS, the random training is a positive side
effect. However, when training the segmentation networks this effect is
undesired. Each segmentation network was therefore trained 10 times.

##  4 Results

Figure 1 shows an example of a real 5-channel image and a randomly selected
synthetic 5-channel image.

Fig. 1: Top: a real 5-channel image. Bottom: a synthetic 5-channel image. From left to right: T1-weighted, T1-weighted after gadolinium contrast, T2-weighted, FLAIR, segmentation mask. Original data | # GANs | ET | ED | NCR/NET | Mean
---|---|---|---|---|---
✓ | 0 | $0.791\pm 0.009$ | $0.785\pm 0.003$ | $0.610\pm 0.008$ | $0.729\pm 0.004$
✓ | 1 | $0.789\pm 0.010$ | $0.790\pm 0.008$ | $0.609\pm 0.013$ | $0.730\pm 0.008$
✓ | 5 | $0.795\pm 0.012$ | $0.791\pm 0.004$ | $0.617\pm 0.008$ | $0.734\pm 0.006$
✓ | 10 | $0.791\pm 0.008$ | $0.794\pm 0.005$ | $0.620\pm 0.009$ | $0.735\pm 0.003$
✓ | 20 | $0.799\pm 0.008$ | $0.791\pm 0.005$ | $0.613\pm 0.011$ | $0.734\pm 0.006$
| 1 | $0.665\pm 0.011$ | $0.632\pm 0.027$ | $0.478\pm 0.019$ | $0.592\pm 0.016$
| 5 | $0.689\pm 0.012$ | $0.665\pm 0.023$ | $0.529\pm 0.015$ | $0.628\pm 0.014$
| 10 | $0.696\pm 0.012$ | $0.702\pm 0.013$ | $0.545\pm 0.012$ | $0.648\pm 0.008$
| 20 | $0.696\pm 0.017$ | $0.693\pm 0.021$ | $0.555\pm 0.013$ | $0.648\pm 0.014$
Table 1: Results on the test dataset (56 subjects), Dice calculated per
subject. Mean and standard deviation of Dice score across the dataset are
calculated for the labels; GD-enhancing tumor (ET), peritumoral edema (ED),
and necrotic and non-enhancing tumor core (NCR/NET). All results are presented
as mean $\pm$ standard deviation of 10 trainings.

Table 1 shows the segmentation accuracy on real images when training the U-Net
using only real images, using real and synthetic images together, or only
using synthetic images. The results are repeated for synthetic images from a
single GAN, from 5 GANs, from 10 GANs and 20 GANs. The results represent the
mean and standard deviation over 10 trainings of the U-Net. The mean Dice
score increases 4.7% for the enhancing tumor, 11.1% for the edema part, and
14.0% for the necrotic core, when training with only synthetic images from 10
GANs instead of from a single GAN. As expected, adding synthetic data to the
original dataset during training (row 1-3 in Table 1) only gives a very small
increase in segmentation performance. Since the GANs that generate the
synthetic data are trained on the same dataset, no new information in the form
of annotated data is added and the addition of synthetic data can instead be
compared to advanced data augmentation. However, the same trend with more GANs
giving better performance still holds for these experiments. For 5 GANs, the
improvement compared to 1 GAN was significant (p $<$ 0.05) for NCR/NET and ET.
For 10 GANs, the improvement compared to 1 GAN was significant for all three
tumor types. Using 10 GANs was significantly better than 5 GANs for ED and ET.
The p-values were obtained through a non-parametric sign flipping test, and
multiple comparison correction was done using Bonferroni.

Figure 2 shows segmentation results for a randomly selected slice, for the
same training setups as in Table 1.

annotation | 1 GAN | 5 GANs | 10 GANs | 20 GANs
---|---|---|---|---
|  |  |  |
original | original + 1 GAN | original + 5 GANs | original + 10 GANs | original + 20 GANs
|  |  |  |
Fig. 2: Example predictions on an image in the test set. Classes are
visualized as colored overlay where red is GD-enhancing tumor, blue is
peritumoral edema (ED) and green is necrotic and non-enhancing tumor core
(NCR/NET).

##  5 Discussion

Our results show that an ensemble of GANs indeed improves the segmentation
accuracy when using the synthetic images for training segmentation networks,
compared to using synthetic images from a single GAN. This comes at the cost
of an increased training time, which scales linearly with the number of GANs.
Our results thereby reflect the results obtained by [12] for classification.
Better results were not obtained when using 20 instead of 10 GANs. Here we use
a rather strong baseline model, which may explain why we see only minor
improvements when adding synthetic images to the real ones.

A challenge with the BraTS dataset is that the data originates from 19
different centers, and that the image quality and appearance varies between
the centers. It is possible that better results can be obtained if the site
information is used as a variable in the GAN training, to train conditional
GANs which can learn to generate images that represent each center. Although
BraTS 2020 contains images from 369 subjects (training set), it is possible
that instead using BraTS 2021 (which contains 1251 training subjects) will
give substantially better results.

Training with synthetic images still results in substantially lower
segmentation accuracy, compared to training with real images. In future work
we therefore plan to repeat the experiments with the more recent architectures
StyleGAN [20], StyleGAN2 [21] and StyleGAN3 [22] (although they are even more
computationally demanding), as well as with more recent diffusion models. It
would especially be interesting to train conditional models where the imaging
site, tumor size and tumor location can be controlled. We also plan to look
into 3D GANs [23] and 3D diffusion models [24], but a general problem is that
they will use a lot of GPU memory for generating 5-channel volumes.

##  6 Compliance with ethical standards

This research study was conducted retrospectively using human subject data
made available as open access by BraTS. Ethical approval was not required as
confirmed by the ethical review board of Linköping.

##  7 Acknowledgements

Training the segmentation networks was performed using the supercomputing
resource Berzelius (480 Nvidia A100 GPUs) provided by National Supercomputer
Centre at Linköping University, Sweden, and the Knut and Alice Wallenberg
Foundation.

This research was supported by the ITEA/VINNOVA project ASSIST (Automation,
Surgery Support and Intuitive 3D visualization to optimize workflow in IGT
SysTems, 2021-01954), the VINNOVA funded project AIDA, LiU Cancer and the Åke
Wiberg foundation. Anders Eklund has previously received graphics hardware
from Nvidia.

## References

  * [1] Fei Jiang, Yong Jiang, Hui Zhi, Yi Dong, Hao Li, Sufeng Ma, Yilong Wang, Qiang Dong, Haipeng Shen, and Yongjun Wang,  “Artificial intelligence in healthcare: past, present and future,”  Stroke and vascular neurology, vol. 2, no. 4, 2017.
  * [2] Hoo-Chang Shin, Neil A Tenenholtz, Jameson K Rogers, Christopher G Schwarz, Matthew L Senjem, Jeffrey L Gunter, Katherine P Andriole, and Mark Michalski,  “Medical image synthesis for data augmentation and anonymization using generative adversarial networks,”  in International workshop on simulation and synthesis in medical imaging. Springer, 2018, pp. 1–11.
  * [3] Kenny H Cha, Nicholas A Petrick, Aria X Pezeshk, Christian G Graff, Diksha Sharma, Andreu Badal, and Berkman Sahiner,  “Evaluation of data augmentation via synthetic images for improved breast mass detection on mammograms using deep learning,”  Journal of Medical Imaging, vol. 7, no. 1, pp. 012703, 2019.
  * [4] Seung Kwan Kang, Hyun Joon An, Hyeongmin Jin, Jung-in Kim, Eui Kyu Chie, Jong Min Park, and Jae Sung Lee,  “Synthetic CT generation from weakly paired MR images using cycle-consistent GAN for MR-guided radiotherapy,”  Biomedical Engineering Letters, vol. 11, no. 3, pp. 263–271, 2021\.
  * [5] Maayan Frid-Adar, Idit Diamant, Eyal Klang, Michal Amitai, Jacob Goldberger, and Hayit Greenspan,  “GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification,”  Neurocomputing, vol. 321, pp. 321–331, 2018.
  * [6] Ali Madani, Mehdi Moradi, Alexandros Karargyris, and Tanveer Syeda-Mahmood,  “Chest x-ray generation and data augmentation for cardiovascular abnormality classification,”  in Medical imaging 2018: Image processing. SPIE, 2018, vol. 10574, pp. 415–420.
  * [7] Christopher Bowles, Liang Chen, Ricardo Guerrero, Paul Bentley, Roger Gunn, Alexander Hammers, David Alexander Dickie, Maria Valdés Hernández, Joanna Wardlaw, and Daniel Rueckert,  “GAN augmentation: Augmenting training data using generative adversarial networks,”  arXiv preprint arXiv:1810.10863, 2018.
  * [8] Mehdi Foroozandeh and Anders Eklund,  “Synthesizing brain tumor images and annotations by combining progressive growing GAN and SPADE,”  arXiv preprint arXiv:2009.05946, 2020.
  * [9] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen,  “Progressive growing of GANs for improved quality, stability, and variation,”  ICLR, 2018.
  * [10] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu,  “Semantic image synthesis with spatially-adaptive normalization,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 2337–2346.
  * [11] John T Guibas, Tejpal S Virdi, and Peter S Li,  “Synthetic medical images from dual generative adversarial networks,”  arXiv preprint arXiv:1709.01872, 2017.
  * [12] Gabriel Eilertsen, Apostolia Tsirikoglou, Claes Lundström, and Jonas Unger,  “Ensembles of GANs for synthetic training data generation,”  ICLR 2021 workshop on Synthetic Data Generation, 2021.
  * [13] Spyridon Bakas, Hamed Akbari, Aristeidis Sotiras, Michel Bilello, Martin Rozycki, Justin Kirby, John Freymann, Keyvan Farahani, and Christos Davatzikos,  “Segmentation labels and radiomic features for the pre-operative scans of the TCGA-GBM collection. The Cancer Imaging Archive,” 2017\.
  * [14] Spyridon Bakas, Hamed Akbari, Aristeidis Sotiras, Michel Bilello, Martin Rozycki, Justin Kirby, John Freymann, Keyvan Farahani, and Christos Davatzikos,  “Segmentation labels and radiomic features for the pre-operative scans of the TCGA-LGG collection,”  The Cancer Imaging Archive, 2017.
  * [15] Spyridon Bakas, Hamed Akbari, Aristeidis Sotiras, Michel Bilello, Martin Rozycki, Justin S Kirby, John B Freymann, Keyvan Farahani, and Christos Davatzikos,  “Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features,”  Scientific data, vol. 4, pp. 170117, 2017.
  * [16] Spyridon Bakas, Mauricio Reyes, Andras Jakab, Stefan Bauer, Markus Rempfler, Alessandro Crimi, Russell Takeshi Shinohara, Christoph Berger, Sung Min Ha, Martin Rozycki, et al.,  “Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge,”  arXiv preprint arXiv:1811.02629, 2018.
  * [17] Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al.,  “The multimodal brain tumor image segmentation benchmark (BRATS),”  IEEE transactions on medical imaging, vol. 34, no. 10, pp. 1993–2024, 2014.
  * [18] Fabian Isensee, Paul F Jäger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein,  “Automated design of deep learning methods for biomedical image segmentation,”  arXiv preprint arXiv:1904.08128, 2019.
  * [19] Olaf Ronneberger, Philipp Fischer, and Thomas Brox,  “U-net: Convolutional networks for biomedical image segmentation,”  in International Conference on Medical image computing and computer-assisted intervention. Springer, 2015, pp. 234–241.
  * [20] Tero Karras, Samuli Laine, and Timo Aila,  “A style-based generator architecture for generative adversarial networks,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 4401–4410.
  * [21] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila,  “Analyzing and improving the image quality of StyleGAN,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 8110–8119.
  * [22] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila,  “Alias-free generative adversarial networks,”  Advances in Neural Information Processing Systems, vol. 34, pp. 852–863, 2021.
  * [23] Anders Eklund,  “Feeding the zombies: Synthesizing brain volumes using a 3D progressive growing GAN,”  arXiv preprint arXiv:1912.05357, 2019.
  * [24] Walter HL Pinaya, Petru-Daniel Tudosiu, Jessica Dafflon, Pedro F da Costa, Virginia Fernandez, Parashkev Nachev, Sebastien Ourselin, and M Jorge Cardoso,  “Brain imaging generation with latent diffusion models,”  arXiv preprint arXiv:2209.07162, 2022.


# FlightBERT++: A Non-autoregressive Multi-Horizon Flight Trajectory
Prediction Framework

Dongyue Guo, Zheng Zhang, Jianwei Zhang, and Yi Lin This work was supported by
the National Natural Science Foundation of China (NSFC) under grants No.
U20A20161, and 62001315, and by the Open Fund of Key Laboratory of Flight
Techniques and Flight Safety, Civil Aviation Administration of China (CAAC)
under Grant No. FZ2021KF04, also by the Fundamental Research Funds for the
Central Universities under Grant No. 2021SCU12050. (Corresponding author: Yi
Lin.)Dongyue Guo is with the National Key Laboratory of Fundamental Science on
Synthetic Vision, Sichuan University, Chengdu 610000, China (e-mail:
dongyueguo@stu.scu.edu.cn).Zheng Zhang, Jianwei Zhang, and Yi Lin are with the
College of Computer Science, Sichuan University, Chengdu 610000, China
(e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>yilin@scu.edu.cn).

###### Abstract

Flight Trajectory Prediction (FTP) is an essential task in Air Traffic Control
(ATC), which can assist air traffic controllers to manage airspace more safely
and efficiently. Existing approaches generally perform multi-horizon FTP tasks
in an autoregressive manner, which is prone to suffer from error accumulation
and low-efficiency problems. In this paper, a novel framework, called
FlightBERT++, is proposed to i) forecast multi-horizon flight trajectories
directly in a non-autoregressive way, and ii) improved the limitation of the
binary encoding (BE) representation in the FlightBERT framework. Specifically,
the proposed framework is implemented by a generalized Encoder-Decoder
architecture, in which the encoder learns the temporal-spatial patterns from
historical observations and the decoder predicts the flight status for the
future time steps. Compared to conventional architecture, an extra horizon-
aware contexts generator (HACG) is dedicatedly designed to consider the prior
horizon information that enables us to perform multi-horizon non-
autoregressive prediction. Additionally, a differential prediction strategy is
designed by well considering both the stationarity of the differential
sequence and the high-bits errors of the BE representation. Moreover, the Bit-
wise Weighted Binary Cross Entropy loss function is proposed to optimize the
proposed framework that can further constrain the high-bits errors of the
predictions. Finally, the proposed framework is validated on a real-world
flight trajectory dataset. The experimental results show that the proposed
framework outperformed the competitive baselines.

###### Index Terms:

Binary encoding representation, Differential prediction, Flight trajectory
prediction, Multi-horizon forecasting, Transformer.

##  I Introduction

Flight Trajectory Prediction (FTP) is essential for Air Traffic Management
(ATM), enabling many critical applications to help Air Traffic Controllers
(ATCOs) manage airspace more safely and efficiently, such as delay prediction
[1], conflict detection [2, 3], and arrival time estimation [4]. In
particular, the FTP is one of the vital techniques to support the Trajectory-
Based Operation (TBO) that is being promoted by both the Single European Sky
ATM Research (SESAR) and the New Generation Air Transportation Systems
(NextGen) [5]. In this context, the FTP technique has gathered more attention
from researchers and achieved significant progress in the past decade.

FTP aims to forecast the flight status in future time steps according to the
observed historical flight trajectories. Accurate FTP demands a high
prediction capacity of the model which has the ability to capture the
temporal-spatial dependencies between the observations and predictions. Our
previous study proposed the FlightBERT framework that encodes the attributes
of the trajectory points into a set of binary vectors and formulates the TP
task as multiple binary classification tasks. Benefits from the innovative
idea and dedicatedly network design, the FlightBERT framework not only
enhances the semantic representation of the trajectory points, but also avoids
the vulnerability impacted by the normalization algorithms. However, some
limitations remain in the FlightBERT framework that can be improved to further
enhance the performance of the TP task.

  * •

A limitation of the Binary Encoding (BE) representation is that the high-bit
prediction errors will lead to outliers in the predictions. For instance,
given the BE representation ”0110 0100” (decimal 100), the absolute error is
128 (decimal) if the prediction error occurs in the $8^{th}$ bit (”1110 0100”)
while that is 1 (decimal) if the prediction error occurs in $1^{st}$ bit
(”0110 0101”).

  * •

The FlightBERT recursively performs multi-horizon prediction, i.e., predicts
the flight status of the next time step based on observation and iteratively
applies the predicted values as pseudo observation to obtain multi-step
predictions. On the one hand, it is easy to suffer from the accumulative error
since the prediction errors in the pseudo observation are introduced into the
recursive inference process. On the other hand, the efficiency is limited to
the prediction horizon due to the step-by-step prediction paradigm. The
inference speed is slow in long-horizon predictions, which can be not
acceptable for real-time applications.

Focusing on the aforementioned challenges, in this paper, a non-autoregressive
multi-horizon flight trajectory prediction framework, named FlightBERT++, is
proposed to improve the performance of the TP task. Thanks to the robust
trajectory representation ability of the BE, the proposed framework inherits
the BE representation of the FlightBERT and is implemented based on the
multiple-binary classification (MBC) paradigm.

To address the outlier problems in the prediction, the following two solutions
are proposed to reduce the high-bit prediction errors from the BE
representation and model optimizing perspective. i) From BE representation
perspective: instead of predicting the real values, the differential values of
the trajectory attributes are formulated as the output objective of the
proposed framework, which can reduce the number of bits of the outputs. ii)
From the model optimizing perspective: a Bit-wise Weighted Binary Cross
Entropy (BW-BCE) loss is designed to optimize the model by enlarging the
penalty of high-bit prediction errors in the training process.

To achieve high-efficiency and -accuracy multi-horizon FTP prediction, a novel
Seq2Seq architecture is proposed to implement the FlightBERT++ framework.
Specifically, the proposed framework is composed of three components, i.e.,
trajectory encoder, Horizon-Aware Context Generator (HACG), and differential-
prompted decoder. Firstly, the trajectory encoder encodes the observations
into a trajectory-level representation, while the HACG is designed to produce
the horizon-context representation of the predicted horizons. In succession,
the differential-prompted decoder is innovatively proposed to perform the
high-confidence predictions based on the horizon-context representations,
which employ the differential of the observation sequence as the prompting.
Moreover, the backbone network of the Encoder and Decoder in the proposed
framework are both stacked by the Transformer blocks, which enables the
inference process of the model in a non-autoregressive manner. In this way,
the proposed framework not only reduces the error accumulation, but also
significantly improves the computational efficiency by performing the multi-
horizon prediction directly.

The proposed framework is evaluated on a real-world flight trajectory dataset
that was collected from the ATC systems. Various competitive baselines are
conducted to make a comprehensive comparison to demonstrate the effectiveness
and efficiency of the proposed framework. In addition, extensive ablation
studies are designed to validate the effectiveness of all proposed technical
improvements. The experimental results demonstrated that the proposed
framework achieves desired performance across all flight phase.

In summary, the contributions and novelty of this work are listed as follows:

  * •

A flight trajectory prediction framework, called FlightBERT++, is innovatively
proposed to perform high-accuracy multi-horizon forecasting in a non-
autoregressive manner.

  * •

The differential prediction strategy and BW-BCE loss are dedicatedly designed
to overcome the limitations of the high-bit prediction errors in BE
representation.

  * •

A Horizon-Aware Context Generator (HACG) is innovatively proposed to generate
horizon-context representations, which is the key to supporting non-
autoregressive predictions.

  * •

Considering the stationarity of the differential sequence in trajectory data,
the differential-prompted decoder is proposed to further improve the model
performance and reduce the learning complexity in the training process.

  * •

we conducted extensive comparison experiments on the real-world trajectory
dataset to demonstrate the effectiveness of the proposed frameworks. In
addition, some diagnostic studies are designed to show the robustness of our
framework.

The rest of this paper is organized as follows. Section II detailed presents
the methodologies of the proposed FlightBERT++ framework. The experimental
settings, baselines, and evaluation metrics are described in Section III. The
experimental results, visualization, and discussions are presented in Section
IV. Finally, the work of this paper is concluded in Section V.

##  II Methodology

###  II-A Problem Formulation

In general, the short-term FTP task can be formulated as a multi-variable
time-series forecasting problem, i.e., predicting the aircraft status of
several future timestamps based on historical observation sequences. Let an
observed trajectory point $p_{t}$ in timestamp $t$, the multi-horizon FTP aims
to forecast the $P_{t+1:t+n}=\\{p_{t+1},p_{t+2},...,p_{t+n}\\}$ based on the
observation sequence $O_{t-k:t}=\\{p_{t-k},...,p_{t-1},p_{t}\\}$.
Mathematically, it can be abstracted as Eq (1):

$P_{t+1:t+n}=\\{p_{t+1},p_{t+2},...,p_{t+n}\\}=\mathcal{F}(O_{t-k:t})$ (1)

where the $n,k$ are the number of prediction horizons and the observed
sequence length, respectively. $\mathcal{F}(\cdot)$ denotes the FTP model
which can be modeled in various approaches. Compared to the conventional
approaches, instead of performing the prediction iteratively,
$\mathcal{F}(\cdot)$ outputs the multi-horizon predictions directly in the
proposed FlightBERT++ framework.

In this work, the trajectory point $p_{t}$ is composed of the six key
attributes that can mainly describe the aircraft status, which defines as
follows.

$p_{t}=[Lon_{t},Lat_{t},Alt_{t},{Vx}_{t},{Vy}_{t},{Vz}_{t}]$ (2)

where the $Lon_{t},Lat_{t},Alt_{t},{Vx}_{t},{Vy}_{t},{Vz}_{t}$ represents the
longitude, latitude, altitude, and velocity in $x,y,z$ dimensions of the
trajectory point, respectively. The $x,y,z$ dimensions of the velocity
corresponds to the longitudinal, latitudinal, and altitudinal attributes,
respectively.

Figure 1: Overview of the proposed FlightBERT++ framework.

###  II-B The Proposed Framework

The proposed framework is illustrated in Fig 1. The framework consists of
three components, including Trajectory Encoder, Horizon-Aware Context
Generator, and differential-prompted decoder. Specifically, given the observed
trajectory sequence $O_{t-k:t}$, the object of the trajectory encoder is to
learn the temporal-spatial correlations of the observations and abstract it to
the high-dimensional representations $\mathbf{Traj}_{enc}$ as Eq (3).

$\mathbf{Traj}_{enc}=TrajectoryEncoder(O_{t-k:t})$ (3)

In succession, the HACG is designed to generate the horizon contexts
$C=\\{c_{t+1},c_{t+2},...,c_{t+n}\\}$ by both considering the prior prediction
horizon embeddings $\mathbf{Hor}=\\{h_{t+1},h_{t+2},...,h_{t+n},\\}$ and the
high-level representation of the observation.

$\mathbf{C}=HACG([\mathbf{Traj}_{enc},\mathbf{Hor}])$ (4)

The differential embeddings $\mathbf{D}_{t-k:t}$ of the $O_{t-k:t}$ and
horizon contexts $C$ are jointly fed into the differential-prompted decoder to
generate the final outputs as Eq. (5), where the $\mathbf{P}_{t+1:t+n}^{BE}$
is the BE representations of the predicted differential sequence of future
trajectory. Finally, the $\mathbf{P}_{t+1:t+n}^{BE}$ is transformed into
decimals and reconstruct the predicted trajectory based on the $O_{t}$.

$\mathbf{P}_{t+1:t+n}^{BE}=Decoder([\mathbf{D}_{t-k+1:t},\mathbf{C}])$ (5)

Note that the inputs and the outputs of the proposed framework are both the BE
representation of the trajectory attributes. Therefore, the optimizing
objective of the proposed framework is also formulated as the multiple-binary
classification tasks. The FlightBERT++ framework is trained by the proposed
BW-BCE loss which is dedicatedly designed to reduce the high-bit errors in BE
representation. More details of the BE representations can be found in [6].

Compared to existing multi-horizon FTP approaches, the proposed framework (i)
innovatively designed a HACG to generate multi-horizon contexts directly, (ii)
employ the Transformer-based architecture to conduct the backbone network of
the trajectory encoder and differential-prompted decoder, which are the core
idea of non-autoregressive prediction.

Figure 2: The deatiled implemenation of the channel-mix trajectory point
embedding and trajectory encoder.

###  II-C Trajectory Encoder

As illustrated in Fig 1, the trajectory encoder is composed of three modules,
including the Conv1D-based Trajectory Point Embedding (TPE) module,
Transformer-based temporal-spatial modeling module, and Attention-based
Sequence Aggregation (ASA) module. Specifically, the Conv1D-based TPE module
projects the BE representation of the trajectory points into high-dimensional
embedding space while the Transformer-based module employs to capture the
temporal-spatial correlations among the observation sequence. In succession,
the outputs of the Transformer module are further fed into the ASA module to
generate the trajectory-level embedding which is expected to extract the
discriminative features of the whole observations.

One of the advantages of the BE representation is expanding the initial
dimensional of the inputs that facilitate learning discriminative features for
the trajectory points. In [6], the linear-based attributes embedding block is
designed to learn attribute embeddings in a channel-independent manner and the
attributes-correlations attention module is proposed to capture the
correlations of the specific attributes. However, the channel-independent
approaches suffer from two limitations. i) Since strong spatial correlations
are implied among the trajectory attributes, learning the attributes embedding
separately might be not fully considered the global features. ii) the parallel
linear layers in the attributes embedding block increase the complexity of the
model, which not only burdens the model training but also affects the
computational performance in real-time FTP applications.

To address the aforementioned limitations, in this work, a simple yet
efficient Conv1D-based channel-mix trajectory point embedding module is
proposed to project the BE representations of trajectory points into high-
dimensional embedding space. As illustrated in Fig 2, the BE representations
of all attributes are concatenated into a joint BE representation for the
trajectory point that is further fed into a Conv1D layer to learn trajectory
point embedding. On the one hand, the channel-mix strategy remains the global
correlations of the trajectory attributes. On the other hand, the Conv1D
neural network is beneficial to learn the local features among the bits of the
joint BE representation.

The Transformer-based temporal modeling module is implemented by the stacked
Transformer blocks whose effectiveness has been demonstrated in many sequence
modeling scenarios, such as NLP, CV, and time-series forecasting. Compared to
the RNN-based architectures, the Transformer is implemented by the self-
attention mechanism which enables the model to learn the temporal correlations
of the trajectory in a non-autoregressive manner.

The ASA module performs a weighted sum operation in the temporal dimension, in
which the attention weights of the trajectory points are generated as Fig 2.
In this way, the trajectory-level embedding $\mathbf{Traj}_{enc}$ can be
obtained from the trajectory encoder, which captured the discriminative
temporal-spatial features of the observations.

###  II-D Horizon-Aware Context Generator

Conventional multi-horizon approaches usually perform the prediction in an
autoregressive manner, i.e., predicting $p_{t+1}$ based on observation and
generating the $p_{t+2}$ according to the $p_{t+1}$ sequentially. However, the
autoregressive nature of these approaches makes them slow in the training and
inference process, especially at high temporal resolution conditions, which is
not applicable to real-time forecasting scenarios. To this end, a horizon-
aware context generator is innovatively designed to support multi-horizon
predictions by generating multi-horizon contexts directly.

The architecture of the HACG is illustrated as Fig 1. Specifically, we conduct
multi-horizon contexts by considering both the prior temporal specificities of
the predicted horizons and trajectory-level embeddings of the observations.
Firstly, the horizons are encoded into a set of one-hot vectors and projected
into embedding space to learn the horizon embeddings by embedding layer.
Secondly, the horizon embeddings and the trajectory-level embedding are
concatenated to generate the context vector of each horizon. Finally, the
context vectors are further fed into an MLP block for high-dimension
projection and generate the final horizon-context vectors.

###  II-E Differential-prompted Decoder

Figure 3: The deatiled implemenation of the Differential-prompted Decoder.

As described in Section I, to reduce the high-bits prediction errors, the
differential prediction idea is introduced into the FlightBERT++ framework,
i.e., output the differential sequence in the decoder instead of the original
attributes. Additionally, since the motion of the aviation flight status
generally conforms to the linear variation, the variation trend of
differential for the observation sequence is more stationary than the original
values. It is believed that predicting the differential sequence is an
effective way to improve the performance of the FTP task.

In practice, it is challenging to learn the transition patterns of
differential sequence from the observations sequence because the differential
loses the spatial features of the trajectory attributes. In this work, a
differential-prompted decoder is proposed to reduce the learning difficulty of
the network by integrating the differential prompted mechanism, which
illustrated in Fig 1. Specifically, the differential-prompted decoder consists
of the differential sequence embedding module, Masked-Transformer module, and
predictor module. Firstly, the differential sequence is calculated from the
observed sequence and encoded into BE representations which can be consistent
with the form of the outputs. Similar to the TPE module, the BE
representations are further fed into a Conv1D-based embedding layer to learn
the embeddings of the differential sequence. Secondly, the learned embeddings
of the differential sequence are concatenated with the horizon contexts along
the temporal dimension to prompt the variation of the differential sequence.
In succession, the concatenated vectors further fed into the Masked-
Transformer module to build the inter- and intra-temporal correlations across
observation and the multi-horizon predictions. The architecture of the Masked
Transformer module is similar to the Transformer-based temporal modeling
module except it employs the masked self-attention mechanism to ensure the
temporal specificities of the sequence. Finally, the predictor is composed of
a Fully Connected (FC) layer and the Sigmoid activation, which is applied to
output BE representations for the multi-horizons.

###  II-F Loss Function

Thanks to the nature of the BE representations, the proposed FlightBERT++
framework is formulated as the MBC paradigm. A natural idea is to adopt the
Binary Cross Entropy (BCE) loss function to optimize the neural networks in
the training process. However, as described in Section I, the errors in low-
bit and high-bit makes different contributions to the final prediction. The
BCE loss makes equal contributes in each bits for the output BE
representations, which might be not the best optimization strategy.

In this work, a Bit-wise Weighted BCE (BW-BCE) loss is designed to reduce the
high-bit prediction errors. Specifically, the weights of the BCE loss of each
bit in BE representation are pre-defined to increase the punishment of high-
bit prediction errors. Let the $p,y$ are the ground truth and prediction of a
trajectory point, $p_{a_{k}}^{b}$, $y_{a_{k}}^{b}$ are the BE representation
of the $k^{th}$ attribute in the ground truth and prediction, respectively.
The calculation of the loss can be mathematically described as Eq. (6) - Eq.
(7):

$\mathcal{L}(p,y)=\frac{1}{N}\sum_{n=1}^{N}\sum_{k=1}^{K}\mathcal{L}_{BW-
BCE}^{a_{k}}(p_{a_{k}}^{b},y_{a_{k}}^{b})$ (6)

$\displaystyle\mathcal{L}_{BW-
BCE}^{a_{k}}(p_{a_{k}}^{b},y_{a_{k}}^{b})=-\frac{1}{M}\sum_{i=1}^{M}\alpha_{i}(p_{a_{k}}^{b}[i]*log(y_{a_{k}}^{b}[i])+$
(7) $\displaystyle(1-p_{a_{k}}^{b}[i])*log(1-y_{a_{k}}^{b}[i]))$

where the $N$ is the number of samples in the minibatch, while $M$ denotes the
dimension of the BE representation of each attribute. $\alpha_{i}$ is pre-
defined weight which is empirically set to $2i$ in this work.

##  III Experimental Settings

###  III-A Dataset Preprocessing and Description

To validate the proposed framework, a real-world flight trajectory dataset is
conducted, which was collected from an ATC system in China. The dataset
contains a total of 9 days of trajectory data with 20 seconds intervals from
February 19 to 27, 2021, which range of the interested (ROI) longitude and
latitude are $[94.616,113.689]$ and $[19.305,37.275]$, respectively. The key
attributes of the flight trajectory are extracted from the raw data to support
the experiments, include timestamps, call sign, longitude, latitude, altitude,
velocity in $x,y,z$ directions. The flight position (longitude, latitude, and
altitude) are in degree with the WGS-84 coordinate system, while the velocity
in $x,y,z$ dimensions are measured by $km/h$ with the Cartesian coordinates.

After aforementioned preprocess, a total of 8643 flight trajectories in the
dataset and split into train, validate, and test subsets. Specifically, the
trajectory of the first 7 days is used to train the FTP models while $8^{t}h$
and $9^{t}h$ are applied for validation and testing. The aforementioned data
splits are applied to all the experiments in this work to keep the
experimental fairness.

###  III-B Comparison Baselines

In this work, a total of 5 competitive approaches serve as baselines to
validate the effectiveness of the proposed FlightBERT++ framework. Moreover,
the baselines are divided into two groups to conduct comparisons according to
the type of multi-horizon prediction approaches. Group A performs iterative
multi-horizon prediction, i.e., the model only predicts the result for the
next time step and serves as a pseudo observation for multi-horizon inference.
Group B is the direct multi-horizon prediction approach that forecasts the
results in an inference process for multiple time steps. The detailed
description of the baselines is listed as follows.

  * •

(A1) LSTM: An FTP framework using LSTM neural networks [7]. In this work, we
employ a 4-layer LSTM network with 128 neurons to perform the FTP task.

  * •

(A2) Transformer: A villain Transformer architecture is employed to build the
FTP model to predict the six attributes of the trajectory point [8]. The model
is implemented by stacking 4 Transformer blocks with 128-dimensional hidden
states and an FC layer.

  * •

(A3) Kalman-Filter: A typical model-driven flight state estimation algorithm
based on historical observations and generally applied in target tracking
scenarios. In this work, the Kalman-Filter (KF) is employed to perform the FTP
task.

  * •

(A4) FlightBERT: A novel FTP framework proposed in our previous work, which
convert the FTP tasks into MBC paradigm and achieved desired performance [6].

  * •

(B1) LSTM+Attention: A Seq2Seq architecture to predict the flight state of
multiple future time steps directly. In this experiment, an Attention-based
Encoder-Decoder LSTM network [9] is adapted to the FTP task, which has
achieved applause performance in many NLP applications.

###  III-C Experimental Configuration

Based on ATC work, the quantization precision $0.001\degree$ (about 110
meters) is selected to adapt the transformation between BE representation and
the decimals. In this resolution, the number of bits of the BE representation
of trajectory attribute for inputs and differential values for outputs is
listed as follows:

  * •

Longitude and latitude: we use 18 and 16 bits to encode the real values
(decimals) of longitude and latitude into BE representation for the inputs.
The number of the bits for longitude and latitude differential values are both
set to 8 bits.

  * •

Altitude: the altitude is measured in $10m$, and encoded into the BE with 11
bits, while its differential value encodes to BE representation with 8 bits.

  * •

Velocity: the velocity is measured in Km/H and all are encoded into the BE
representation with 11 bits for inputs. For the model outputs, the
differential values of $Vx,Vy$ are represented by the 9-bit BE representation
while the $Vz$ is encoded with a 6-bit BE representation. Note that the
highest bit indicates the sign of the real value.

Based on the above configurations, the input of the proposed framework is a
78-dimensional vector while the output is a 48-dimensional vector. The
embedding size of the trajectory points, horizons, and differentials is set to
128, i.e., 128 channels for the Conv1D networks in the proposed framework. The
number of the Transformer blocks in the trajectory encoder and differential-
prompted decoder are both set to 4. The number of hidden states of the
Transformer blocks is set to 768. A 4-head of the attention operator is
applied to all Transformer blocks in the proposed framework.

For experiments A1, A2, and B1, the Z-Score normalization algorithm is applied
to process the value into [0,1] for longitude and latitude attributes due to
the sparse specificities of their distributions. While the altitude and
velocities in $x,y,z$ dimension are normalized into [0,1] using the Max-min
algorithm. The MSE loss function is applied to optimize the model in the
training process.

In the training phase, we use the latest 3 minutes observations to predict the
flight status of the future 5 minutes, i.e., predicting 15 trajectory points
based on 9 observed trajectory points. The Adam optimizer with $10^{-4}$
initial learning rate is applied to train all the above deep learning-based
models.

In this work, all the experiments are implemented with the open-source deep
learning framework PyTorch 1.9.0. The models are trained on the server
configured with Ubuntu 16.04 operating system, 8*NVIDIA GeForce RTX 2080 GPU,
Intel(R) Core(TM<EMAIL_ADDRESS>CPU, and 128 GB memory.

###  III-D Evaluation Metrics

In this work, the Mean Absolute Error (MAE), Mean Absolute Percentage Error
(MAPE), and Root Mean Squared Error (RMSE)are applied to evaluate the proposed
methods and baselines, which are the common criteria for the TP tasks. The
defination of the MAE, MPAE, and RMSE are described as follows:

$MAE=\frac{1}{n}\frac{1}{h}\sum_{i=1}^{n}\sum_{j=1}^{h}{\lvert
a_{ij}-a^{\prime}_{ij}\rvert}$ (8)

$MAPE=100\%\times\frac{1}{n}\frac{1}{h}\sum_{i=1}^{n}\sum_{j=1}^{h}{\lvert\frac{a_{ij}-a^{\prime}_{ij}}{a_{ij}}\rvert}$
(9)

$RMSE=\sqrt{\frac{1}{n}\frac{1}{h}\sum_{i=1}^{n}\sum_{j=1}^{h}\left(a_{ij}-a^{\prime}_{ij}\right)^{2}}$
(10)

where $h$ is the prediction horizon, $a,a^{\prime}$ are the real-value
(decimals) of trajectory attributes in the ground truth and prediction,
respectively.

Although the longitude, latitude, and altitude (LLA) are measured using the
above common metrics, it is not intuitive to evaluate the error separately due
to the trajectory point being determined by LLA together in the three-
dimensional (3D) airspace. To this end, the Mean Distance Error (MDE) is
proposed to evaluate the Euclidean distance of the predictions and the actual
trajectory points. Specifically, the LLA of both predictions and ground truth
is projected to the earth-centered and earth-fixed (ECEF) coordinate system
from the WGS-84 coordinate systems, and the distance error is measured to
evaluate the model performance.

$MDE=\frac{1}{n}\frac{1}{h}\sum_{i=1}^{n}\sum_{j=1}^{h}\Phi(p_{ij}-p^{\prime}_{ij})$
(11)

where $h$ is the prediction horizon, $p,p^{\prime}$ are the transformed values
of the ground truth and prediction in the ECEF coordinate system,
respectively. $\Phi(\dot{)}$ is the calculation function of Euclidean distance
in the 3D airspace.

TABLE I: The experimental results of the proposed framework and baselines. Style | Methods | Horizon |  MAE $\downarrow$ |  MAPE (%) $\downarrow$ |  RMSE $\downarrow$ | MDE $\downarrow$
---|---|---|---|---|---|---
Lon | Lat | Alt | Lon | Lat | Alt | Lon | Lat | Alt
Iterative | LSTM | 1 | 0.0054 | 0.0055 | 1.71 | 0.0050 | 0.0202 | 0.29 | 0.0093 | 0.0124 | 6.04 | 0.91
3 | 0.0065 | 0.0065 | 3.01 | 0.0061 | 0.0238 | 0.50 | 0.0110 | 0.0151 | 7.63 | 1.08
9 | 0.0127 | 0.0121 | 10.33 | 0.0118 | 0.0443 | 1.48 | 0.0239 | 0.0270 | 18.48 | 2.05
15 | 0.0214 | 0.0194 | 20.74 | 0.0199 | 0.0714 | 2.74 | 0.0436 | 0.0427 | 33.06 | 3.38
Transformer | 1 | 0.0048 | 0.0046 | 1.56 | 0.0045 | 0.0168 | 0.27 | 0.0074 | 0.0107 | 6.09 | 0.78
3 | 0.0093 | 0.0088 | 3.25 | 0.0086 | 0.0323 | 0.54 | 0.0139 | 0.0171 | 7.79 | 1.49
9 | 0.0223 | 0.0210 | 8.60 | 0.0206 | 0.0766 | 1.34 | 0.0336 | 0.0364 | 16.57 | 3.56
15 | 0.0352 | 0.0329 | 13.78 | 0.0327 | 0.1199 | 2.06 | 0.0541 | 0.0560 | 25.55 | 5.61
Kalman-Filter | 1 | 0.0067 | 0.0032 | 1.50 | 0.0062 | 0.0120 | 0.28 | 0.0054 | 0.0171 | 8.50 | 0.82
3 | 0.0112 | 0.0059 | 2.97 | 0.0104 | 0.0220 | 0.54 | 0.0861 | 0.0272 | 11.48 | 1.41
9 | 0.0281 | 0.0168 | 8.73 | 0.0261 | 0.0624 | 25.83 | 0.1847 | 0.0619 | 25.83 | 3.69
15 | 0.0494 | 0.0313 | 15.63 | 0.0459 | 0.1162 | 2.53 | 0.2896 | 0.1016 | 42.39 | 6.62
FlightBERT | 1 | 0.0024 | 0.0021 | 1.20 | 0.0023 | 0.0077 | 0.23 | 0.0387 | 0.0325 | 12.04 | 0.44
3 | 0.0039 | 0.0036 | 2.19 | 0.0035 | 0.0133 | 0.41 | 0.0486 | 0.0506 | 13.65 | 0.71
9 | 0.0091 | 0.0086 | 6.20 | 0.0085 | 0.0317 | 1.09 | 0.0608 | 0.0679 | 22.45 | 1.60
15 | 0.0159 | 0.0148 | 10.80 | 0.0148 | 0.0549 | 1.84 | 0.0742 | 0.0794 | 32.76 | 2.71
Direct | LSTM+Attention | 1 | 0.0064 | 0.0068 | 1.98 | 0.0059 | 0.0249 | 0.30 | 0.0109 | 0.0138 | 5.96 | 1.09
3 | 0.0058 | 0.0060 | 2.75 | 0.0054 | 0.0221 | 0.43 | 0.0101 | 0.0144 | 7.36 | 0.98
9 | 0.0082 | 0.0083 | 6.07 | 0.0076 | 0.0305 | 0.94 | 0.0178 | 0.0221 | 14.03 | 1.37
15 | 0.0125 | 0.0122 | 9.05 | 0.0116 | 0.0450 | 1.38 | 0.0299 | 0.0327 | 20.01 | 2.04
FlightBERT++ | 1 | 0.0018 | 0.0017 | 1.16 | 0.0016 | 0.0066 | 0.21 | 0.0040 | 0.0115 | 12.08 | 0.31
3 | 0.0032 | 0.0031 | 2.26 | 0.0030 | 0.0118 | 0.41 | 0.0072 | 0.0133 | 12.51 | 0.56
9 | 0.0079 | 0.0076 | 5.35 | 0.0074 | 0.0283 | 0.96 | 0.0185 | 0.0236 | 18.18 | 1.33
15 | 0.0124 | 0.0118 | 7.53 | 0.0116 | 0.0439 | 1.37 | 0.0289 | 0.0338 | 23.31 | 2.06

##  IV Result and Discussions

###  IV-A Overall Performance

Table I shows the overall performance of the proposed framework and comparison
baselines. To investigate the robustness of these models with the horizons
increase, the experimental results are divided into four (1, 3, 9, 15)
different horizons, corresponding to 20 seconds, 1, 3, and 5 minutes
trajectories in the future. It is demonstrated that the proposed FlightBERT++
framework achieves significant performance improvements against FlightBERT and
outperforms other baselines in the MAE metric across all predicted attributes.
For the RMSE metrics, thanks to the reduction of the bits of BE
representation, a great RMSE reduction of the longitude, latitude, and
altitude is obtained in the proposed framework. In addition, benefits from the
dedicated network design of the proposed framework, the error accumulation of
the multi-horizon predictions is significantly decreased than competitive
baselines.

For the iterative multi-horizon prediction approaches, the KF-based model-
driven approach suffers from a huge performance degradation with the
prediction horizon increase. The primary reason is that the KF-based FTP
approach needs to update the parameters of the state equation dynamically to
correct the prediction errors. However, the loss of real-time observation
updates procedure leads to serious error accumulation in the multi-horizon
prediction process. Compared to model-driven approaches, the data-driven based
approaches achieved better performance across all prediction horizons. The
Transformer-based model achieved desired performance of the 1 and 3 horizon
predictions but fails to output more accurate forecasting in the 9 and 15
horizon procedures. In contrast, the performance of the LSTM is slightly
better than Transformer-based models in long-term horizons but loses the
precision of the output in 1 horizon. It can be attributed to the Transformer-
based models being more sensitive to prediction errors of pseudo observations,
especially in large error accumulation conditions, while the LSTM-based model
is difficult to fit complex flight patterns of the diverse flight phases that
lead to low precision. Benefits from the representation capacity of BE, the
FlightBERT harvests comparable MAE performance in all prediction horizons.
However, the RMSE of the FlightBERT is higher than other data-driven baselines
due to the outliers caused by high-bit errors in the prediction.

The LSTM-Attention achieves superior performance than iterative multi-horizon
prediction approaches for long-term horizon predictions (in horizons 9 and
15). However, it also suffers from the precision reduction of the short-term
horizons (horizon 1) in both the MAE and RMSE metrics. One possible reason is
that the LSTM-Attention approach only captures the global trajectory trends
and dependencies over long-term horizons, and does not perform well in
learning local fine-grained features.

In summary, on the whole, the direct multi-horizon prediction approaches are
superior to the iterative approaches in long-term prediction steps since the
global dependencies are learned from the training process. Thanks to the
robust temporal-spatial dependencies obtained by the trajectory encoder and
differential-promoted decoder, the FlightBERT++ framework achieves expected
performance in both short- and long-term prediction steps, which further
demonstrates the effectiveness of the network design.

##  V Conclusion and Future Work

In this paper, we present a novel FTP framework, called FlightBERT++, that
performs the multi-horizon trajectory prediction in a non-autoregressive
manner. The proposed framework not only inherits the superior representation
ability of the BE in the FlightBERT framework but also develops an innovative
multi-horizon neural network architecture for FTP tasks. To improve the high-
bit errors of the BE representation, the differential prediction strategy and
the BW-BCE loss are designed to reduce the dimension of the output and
increase the punishment of high-bit errors. The proposed framework is
implemented by the Encoder-Decoder architecture, in which an additional HACG
is designed to generate the multi-horizon contexts by considering the prior
horizon information. In addition, a differential-promoted decoder is designed
to improve the performance of the proposed framework. The experimental results
show that the proposed framework achieves significant performance improvements
and outperforms the comparison baselines across all evaluation metrics,
especially in multi-horizon prediction scenarios.

In the future, we will further focus on technical improvements of the BE
representation and the FTP tasks. In addition, the practical applications of
the FTP in the ATC domain are also interesting research topics, such as
conflict detection, and traffic flow prediction.

## References

  * [1] Y. Lin, J. wei Zhang, and H. Liu, “Deep learning based short-term air traffic flow prediction considering temporal–spatial correlation,” _Aerospace Science and Technology_ , vol. 93, p. 105113, 2019.
  * [2] W. Liu and I. Hwang, “Probabilistic trajectory prediction and conflict detection for air traffic control,” _Journal of Guidance, Control, and Dynamics_ , vol. 34, no. 6, pp. 1779–1789, 2011.
  * [3] Z. Chen, D. Guo, and Y. Lin, “A deep gaussian process-based flight trajectory prediction approach and its application on conflict detection,” _Algorithms_ , vol. 13, no. 11, p. 293, 2020.
  * [4] Z. Wang, M. Liang, and D. Delahaye, “A hybrid machine learning model for short-term estimated time of arrival prediction in terminal manoeuvring area,” _Transportation Research Part C: Emerging Technologies_ , vol. 95, pp. 280–294, 2018.
  * [5] E. FAA, “Common trajectory prediction structure and terminology in support of sesar & nextgen,” Technical Report. USA: FAA/EUROCONTROL AP 16, Brussels, Tech. Rep., 2010.
  * [6] D. Guo, E. Q. Wu, Y. Wu, J. Zhang, R. Law, and Y. Lin, “Flightbert: Binary encoding representation for flight trajectory prediction,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 24, no. 2, pp. 1828–1842, 2023.
  * [7] Z. Shi, M. Xu, Q. Pan, B. Yan, and H. Zhang, “Lstm-based flight trajectory prediction,” in _2018 International Joint Conference on Neural Networks, IJCNN 2018, Rio de Janeiro, Brazil, July 8-13, 2018_. IEEE, 2018, pp. 1–8.
  * [8] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , 2017, pp. 5998–6008.
  * [9] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , Y. Bengio and Y. LeCun, Eds., 2015. [Online]. Available: http://arxiv.org/abs/1409.0473