# Comparative Study of Bitcoin Price Prediction

1st Ali Mohammadjafari
<EMAIL_ADDRESS>
School of Computing and Informatics
University of Louisiana at Lafayette, USA

###### Abstract

Prediction of stock prices has been a crucial and challenging task, especially
in the case of highly volatile digital currencies such as Bitcoin. This
research examineS the potential of using neural network models, namely LSTMs
and GRUs, to forecast Bitcoin’s price movements. We employ five-fold cross-
validation to enhance generalization and utilize L2 regularization to reduce
overfitting and noise. Our study demonstrates that the GRUs models offer
better accuracy than LSTMs model for predicting Bitcoin’s price. Specifically,
the GRU model has an MSE of 4.67, while the LSTM model has an MSE of 6.25 when
compared to the actual prices in the test set data. This finding indicates
that GRU models are better equipped to process sequential data with long-term
dependencies, a characteristic of financial time series data such as Bitcoin
prices. In summary, our results provide valuable insights into the potential
of neural network models for accurate Bitcoin price prediction and emphasize
the importance of employing appropriate regularization techniques to enhance
model performance.

###### Index Terms:

Bitcoin, price prediction, LSTMs, GRUs, neural network models, volatility,
financial time series data

##  I Introduction

###  I-A Motivation and Overview

Bitcoin is a digital currency that operates through a decentralized online
network and does not depend on a government or legal entity. Its security and
authenticity are maintained using cryptography and peer-to-peer networking[1].

At present, Bitcoin stands as the top-performing cryptocurrency across various
categories such as Litecoin, Ripple, Bitcoin, and Ethereum. These
cryptocurrencies utilize comparable algorithms for transactions and are known
for their robust security. Furthermore, each cryptocurrency implements its
unique set of rules and regulations to attract a wide range of investors at
different times.[2]. Figure 1 shows the Bitcoin daily prices from 29 November
2011 to 27 March 2023 on Bitstamp (https://www.investing.com/), which is the
longest-running cryptocurrency exchange.

Figure 1: Bitcoin daily prices on Bitstamp (USD) from 29 November 2011 to 27
March 2023. The upper line shows log prices, whereas the lower line shows
plain prices. Some recurring patterns seem to exist when considering the log
value of the Bitcoin price.

Forecasting is a critical component of many financial sectors, as financial
losses can cause significant problems for an industry’s capital. There are
numerous applications of forecasting, including weather prediction, stock
market prediction, and natural disaster prediction such as floods and
earthquakes.[3].

Predicting the price of Bitcoin is essential for various reasons, particularly
in helping investors and traders make informed decisions regarding the
purchase or sale of Bitcoin. With the ability to forecast future price
movements, investors can capitalize on profitable opportunities and minimize
potential losses. Forecasting also enables businesses and individuals who
accept Bitcoin as payment to plan their finances and budget accordingly.

The volatile nature of Bitcoin emphasizes the significance of forecasting.
Since Bitcoin is not tied to any government or central bank, its value can
change rapidly based on market demand and other factors. Precisely predicting
these fluctuations can help investors and traders take advantage of these
changes and avoid potential losses.

Various researchers have examined several factors that affect the price of
Bitcoin and analyzed the patterns of its fluctuations using diverse analytical
and experimental techniques. Accurately forecasting Bitcoin prices is a
challenging undertaking that necessitates advanced techniques such as machine
learning and neural networks. Consequently, many prediction models for
Bitcoin’s price have been created based on deep learning techniques[4][5].

###  I-B Goal and Contributions

This project aims to leverage two distinct neural network models to forecast
Bitcoin prices and compare their effectiveness. To achieve this objective, the
project will employ the Bitcoin Historical Data dataset from the Yahoo Finance
website, which comprises daily Bitcoin prices ranging from 2015 to the present
day. Additionally, the dataset contains other features such as volume and
market capitalization that can be utilized as input variables to train neural
network models.

The initial neural network model to be used in this project is a Long Short-
Term Memory (LSTM) network. LSTMs are a type of recurrent neural network that
are designed for time series prediction tasks. The LSTM model will be trained
on a portion of the Bitcoin Historical Data dataset, employing past Bitcoin
prices and additional features to anticipate future prices.

The other neural network model to be studied is Gated Recurrent Unit (GRU).
GRU is a specialized design of recurrent neural network (RNN) that aims to
address some of the limitations of conventional RNNs. One of the main
drawbacks of RNNs is the ”vanishing gradient problem,” which occurs when
gradients used to update the network weights during backpropagation become too
small to effectively capture long-term dependencies. GRU will also be trained
on the Bitcoin Historical Data dataset, leveraging a sliding window technique
to anticipate future prices based on past prices and additional features.

To evaluate the performance of the two models, several metrics will be used,
such as mean squared error (MSE) and mean absolute error (MAE). The models
will also be compared using visualizations, such as line charts and scatter
plots, to compare the predicted prices to the actual prices. We expecte that
the result show that our models predict the future price of the bitcoin with
80 percent accuracy.

In summery, the contributions of this work are as follows:

  * •

We develop two different neural network models (LSTM and GRU) to predict
Bitcoin prices and compare their performance.

  * •

We use the Bitcoin Historical Data dataset from Yahoo Finance, and use LSTM
and GRU models to make predictions.

  * •

Based on the result, we provide insights into the effectiveness of these
models for Bitcoin price prediction, which can have important implications for
investors and traders in the cryptocurrency market.

The rest of this paper is organized as follows. Section II discusses
background study and related prior works. We explain the overview of chosen
neural network models architecture in Section III. Next, we discuss
experimental evaluation and performance analysis in Section IV. Finally,
Section V concludes the paper and provides a few avenues for the future
studies.

##  II Related Works

###  II-A Price prediction

Price prediction using neural network models is growing in popularity due to
the availability of more data and computing power. Neural networks, which are
designed to resemble the human brain, can learn and recognize patterns in
data. These models analyze historical price data to identify patterns and
relationships that may not be immediately apparent to human analysts, and use
this information to generate predictions about future price movements.
However, such models require large amounts of historical price data and other
relevant factors such as economic indicators, market sentiment, and news
events. Hyper Parameters such as the number of layers, number of neurons per
layer, and learning rate may also need to be carefully adjusted to achieve
optimal performance[6].

Cheng et al. have developed a method that uses a two-phase attention mechanism
to jointly optimize inner-modality and inter-modality sources, allowing end-
users to examine their importance. This approach provides investors with an
interpretable and profitable option, enabling them to make informed investment
decisions[7].

Wu, Shengting, et al. incorporate investors’ sentiment index, technical
indicators, and stock historical transaction data as the feature set for stock
price prediction using a long short-term memory network. The experiments show
that the mean absolute error can achieve 2.386835, which is better than
traditional methods[8].

So, we can see that By using neural networks to analyze large amounts of data,
traders and analysts can gain valuable insights into market trends and make
more informed trading decisions.

###  II-B Bitcoin Price prediction

Aghashahi and Bamdad describe their approach to bitcoin price prediction,
which involves using a combination of technical attributes such as price-
related and lagged features as inputs for neural networks. By doing so, they
aim to improve prediction accuracy and ultimately enhance profitability[9].

In order to improve the accuracy of Bitcoin price prediction, Ye, Zi, et al
introduce a new ensemble deep learning model that uses both long short-term
memory (LSTM) and gate recurrent unit (GRU) neural networks, along with
technical indicators, price data, and sentiment indexes. The model is designed
for near-real-time prediction of Bitcoin prices and performs better than daily
predictions, with a mean absolute error (MAE) that is 88.74 percent lower[10].

Experimental results in the Ehsan Sadeghi, et al. work in 2022 demonstrated
these findings that the 1-dimensional convolutional neural network and stacked
gated recurrent unit model (1DCNN-GRU) showed superior performance compared to
existing methods for Bitcoin, Ethereum, and Ripple datasets, with the lowest
RMSE values of 43.933, 3.511, and 0.00128, respectively[11].

In conclusion, various studies have been conducted on Bitcoin price prediction
using neural network models. These studies have explored the use of various
input features such as technical indicators, sentiment analysis, and lagged
features to improve the prediction accuracy of neural network models.
Additionally, the application of different types of neural networks such as
long short-term memory (LSTM) and Gated Recurrent Unit (GRU) have been
investigated to further enhance the predictive performance. Overall, the
experimental results suggest that these models have shown promising results in
predicting Bitcoin prices. However, further research is needed to improve the
robustness and generalist of these models.

##  III Methodology

###  III-A Problem Statement

The popularity of cryptocurrency and the instability of the Bitcoin market
have resulted in an increasing interest in predicting Bitcoin prices. This is
because accurate predictions can guide investors to make informed decisions
and minimize trading risks. However, forecasting the future value of Bitcoin
is a complex task due to the market’s highly intricate and nonlinear nature.
Thus, this research aims to evaluate the performance of two different neural
network models, namely LSTM and GRU, in predicting the future price of Bitcoin
by leveraging past prices and other features. The study aims to compare the
effectiveness of these models in terms of accuracy and provide valuable
insights into their applicability to Bitcoin price prediction. The ultimate
objective of the research is to support the development of more reliable and
precise models for Bitcoin price forecasting, which can be beneficial for
investors and researchers in the cryptocurrency domain.

###  III-B Initial Experiment

Experimental Procedure:

  * •

Data Preparation: The Bitcoin Historical Data dataset will be downloaded from
Yahoo Finance website. By preprocessing, data is ready to process by network
models. By using 5-fold cross-validation, we can obtain more accurate and
reliable estimates of the model’s performance on unseen data

  * •

Neural Network Models: we train LSTM and GRU models on a given dataset and
evaluate their performance on test data. Additionally, to generalization of
the models, we apply L2 regularization to the weights of the models. To assess
generalization of the models, we use 5-fold cross-validation.

  * •

Performance Comparison: The performance of the LSTM and GRU models will be
compared using various metrics, such as mean squared error (MSE), mean
absolute error (MAE), and accuracy. The models will be evaluated on the
validation set and the testing set.

  * •

Conclusion: Based on the results of the experiment, we will conclude which
neural network model performs better for Bitcoin price prediction. The
findings of this study can be useful for investors and traders who are
interested in predicting the future price of Bitcoin.

###  III-C Data Processing

In this section, the method of gathering Bitcoin price data and transforming
it for use as training data and testing data is explained. The data is
obtained from the Yahoo Finance website (https://finance.yahoo.com/quote/BTC-
USD). As the data is originally combined with different information, it
requires preprocessing.

####  III-C1  Data Collection and Preprocessing

We collected Bitcoin price from Yahoo Finance. The collected data has six
features: open price, close price, high price, low price, adjust close price
and volume at each time period (i.e., dailly). The price and volume data were
collected from December 31, 2015 to April 6, 2023. Figure 2 displays the few
data collected for this research.

Figure 2:  Display of the Data collected

####  III-C2  K-Fold Cross Validation

We use K-fold cross-validation in our bitcoin price prediction model. In
K-fold cross-validation, the data is split into K equally sized folds, and the
model is trained and evaluated K times, with each fold serving as the
validation set once. K-fold cross-validation enables us to gauge the
efficiency of our model on data that it has not encountered before, and this
is crucial for preventing the model from over-fitting to the training data. By
training and evaluating the model on various subsets of the data, we can
obtain a more precise assessment of the model’s actual performance. In our
specific case, we used 5-fold cross-validation because it strikes a good
balance between variance reduction and computational efficiency. Figure 3
shows the schematic of 5-fold cross validation. By using 5 folds, we were able
to obtain a reasonable estimate of the model’s performance while still keeping
the computational requirements manageable. Apart from estimating the
performance of the model, K-fold cross-validation is also advantageous in
ensuring that our model has the ability to generalize effectively on fresh
data. By employing multiple folds and training the model on varying subsets of
the data, we can acquire a better understanding of how proficient the model
will be in handling new data.

This is crucial as real-world situations may involve data that differs from
the data on which the model was trained. Utilizing K-fold cross-validation
instills confidence that our model will perform well on novel and unseen data,
instead of only on the precise data used for training and testing the model.

Figure 3: schematic of 5-fold cross validation

###  III-D Model Training and Validation

The architecture of the two models used in this project for predicting the
future price of Bitcoin: Long Short-Term Memory (LSTM) and Gated Recurrent
Unit (GRU). Both LSTM and GRU are types of recurrent neural networks (RNNs)
that are designed to handle time series data. They are particularly well-
suited for tasks that involve capturing long-term dependencies and patterns in
sequential data. In this project, we will use LSTM and GRU to learn the
patterns in Bitcoin prices over time and predict future prices based on past
prices and other relevant features. Before discussing the specific
architecture of each model, we will provide a brief overview of RNNs and their
limitations, which motivate the need for LSTM and GRU.

Recurrent neural networks (RNNs) are a type of neural network designed for
sequential data processing. Each hidden layer’s output in an RNN is fed back
as input to the same layer in the next time step, allowing the network to
maintain a ”memory” of previous inputs and use this information for future
predictions.

However, RNNs have limitations that make them difficult to use for certain
tasks, such as the ”vanishing gradient problem.” This problem occurs when
gradients become too small during back propagation to capture long-term
dependencies, which is more challenging in RNNs due to the multiplication of
gradients at each time step. To overcome this limitation, specialized variants
of RNNs such as LSTM and GRU have been proposed, which use gating mechanisms
to selectively store and retrieve information from the past. This enables them
to better handle long-term dependencies and capture complex patterns in
sequential data. In the following sections, we will explore the architecture
of LSTM and GRU and how they address the limitations of traditional RNNs.

####  III-D1 LSTM

The LSTM architecture consists of several interconnected layers, including an
input layer, an output layer, and a set of ”memory cells” that maintain
information over time. Each memory cell is controlled by three gates - an
input gate, an output gate, and a forget gate - that regulate the flow of
information into and out of the cell. These gates are implemented using
sigmoid activation functions and element-wise multiplication operations, which
allow the network to selectively add or remove information from the memory
cells based on the input data. The LSTM architecture can be mathematically
represented using a set of equations that describe how the inputs and outputs
of each layer are computed. These equations involve matrix multiplications and
nonlinear activation functions, and can be trained using backpropagation
through time to optimize the network’s parameters. he equations for an LSTM
cell are as follows:

Input gate:

$i_{t}=\sigma(W_{xi}x_{t}+W_{hi}h_{t-1}+W_{ci}c_{t-1}+b_{i})$

Forget gate:

$f_{t}=\sigma(W_{xf}x_{t}+W_{hf}h_{t-1}+W_{cf}c_{t-1}+b_{f})$

Candidate state:

$\tilde{c}t=\tanh(W{xc}x_{t}+W_{hc}h_{t-1}+b_{c})$

Cell state:

$c_{t}=f_{t}*c_{t-1}+i_{t}*\tilde{c}_{t}$

Output gate:

$o_{t}=\sigma(W_{xo}x_{t}+W_{ho}h_{t-1}+W_{co}c_{t}+b_{o})$

Hidden state:

$h_{t}=o_{t}*\tanh(c_{t})$

Where:

$x_{t}$ is the input at time step $t$, $h_{t-1}$ is the hidden state at time
step $t-1$, $c_{t-1}$ is the cell state at time step $t-1$, $i_{t}$, $f_{t}$,
$o_{t}$ are the input, forget and output gates respectively, $\tilde{c}_{t}$
is the candidate state, $W$ and $b$ are the weights and biases of the network,
respectively, $\sigma$ is the sigmoid activation function, $*$ denotes
element-wise multiplication, and $\tanh$ is the hyperbolic tangent activation
function. the summarise of the parameters can be found here

xt | The input at time step t
---|---
ht1 | The hidden state at time step t1
ct1 | The cell state at time step t1
it | The input gate
ft | The forget gate
ot | The output gate
t | The candidate cell state
W | The weight matrix
b | The bias vector
sigma | The sigmoid activation function
* | Element-wise multiplication
tanh | The hyperbolic tangent activation function
Figure 4: The structure of a long short-term memory (LSTM) cell.

####  III-D2 GRU

GRU are a type of recurrent neural network that were introduced as an
alternative to LSTMs. GRUs have a simpler architecture than LSTMs, with fewer
parameters, and can be faster to train. Like LSTMs, GRUs also have a gating
mechanism that allows them to selectively learn which information to keep or
discard.

The architecture of a GRU consists of two gates: an update gate and a reset
gate. The update gate determines how much of the past hidden state should be
preserved and how much of the new input should be added to the new hidden
state. The reset gate determines how much of the past hidden state should be
forgotten when computing the new hidden state.

The formulas for the GRU update and reset gates, the candidate activation
vector, and the new hidden state are as follows:

Update gate:

$z_{t}=\sigma(W_{z}x_{t}+U_{z}h_{t-1})$

Reset gate:

$r_{t}=\sigma(W_{r}x_{t}+U_{r}h_{t-1})$

Candidate activation vector:

$\tilde{h_{t}}=\tanh(Wx_{t}+U(r_{t}*h_{t-1}))$

New hidden state:

$h_{t}=(1-z_{t})*h_{t-1}+z_{t}*\tilde{h_{t}}$

Here, $x_{t}$ is the input at time step $t$, $h_{t-1}$ is the hidden state at
time step $t-1$, $z_{t}$ is the update gate output, $r_{t}$ is the reset gate
output, $\tilde{h_{t}}$ is the candidate activation vector, $W$ and $U$ are
the weights, $\sigma$ is the sigmoid activation function, $*$ denotes element-
wise multiplication, and $\tanh$ is the hyperbolic tangent activation
function.

Figure 5: The structure of GRU cell.

###  III-E Models Implementation

To implement the LSTM and GRU models for Bitcoin price prediction, we used
Python with the Keras library. Keras is a high-level neural networks API,
written in Python and capable of running on top of TensorFlow, CNTK, or
Theano.

To ready the data for training and testing, we utilized a 5-fold cross-
validation technique. This method entails partitioning the data into five
equally-sized folds, utilizing four of the folds as the training set, and
leaving one fold for validation in each iteration. We performed this process
five times to ensure that each fold is used as the validation set once. This
strategy guarantees that the models are assessed on all aspects of the data,
and that the outcomes are not skewed toward any particular segment of the data
set.

To prevent overfitting and enhance the generalization of models in machine
learning, regularization is a popular approach. For our project, we have
decided to implement L2 regularization in our neural network models.

The main reason for employing regularization is to lower the level of noise in
the data and simplify the model, which can prevent overfitting. Specifically,
by using L2 regularization, a penalty term is incorporated into the loss
function that promotes small weights in the network. This approach reduces the
impact of outliers and other forms of noise in the data, making the model more
robust and able to generalize to unseen data.

L2 regularization provides several benefits, including improving the stability
of the model. This means that the model becomes less sensitive to small
changes in the input data, which is particularly important in financial
forecasting where even minor fluctuations in the input data can have a
significant impact on the output. The regularization also reduces the
complexity of the model, which can lead to better generalization and increased
accuracy. Additionally, L2 regularization can help to prevent overfitting,
which can occur when the model becomes too complex and begins to fit the noise
in the training data rather than the underlying patterns. Overall, L2
regularization is a powerful tool for improving the performance and stability
of neural network models.

In order to incorporate L2 regularization into our models, we introduced a
penalty term to the loss function that penalizes large weights in the model.
This penalty term is directly proportional to the square of the weights,
meaning that larger weights will be penalized more heavily than smaller ones.
By adjusting the strength of the regularization with a hyperparameter, we were
able to fine-tune the models to find the optimal level of regularization that
minimized overfitting while still maintaining good performance on the
validation and test sets. We also utilized cross-validation to determine the
optimal value of the hyperparameter and ensure that the regularization was
applied consistently across all folds of the data.

The L2 regularization term can be expressed mathematically as follows:

$L2\ regularization\ term=\lambda\sum_{i=1}^{n}w_{i}^{2}$

where $\lambda$ is the regularization parameter and $w_{i}$ is the weight of
the $i^{th}$ feature in the model. The summation is taken over all the
features. This term is added to the original loss function of the model, and
the strength of the regularization is controlled by the value of $\lambda$.

##  IV RESULT AND DISCUSSION

Our results showed that the GRU model outperformed the LSTM model in
predicting the price of Bitcoin. When we compared the predicted prices with
the actual prices in the test set data, the GRU model had a lower MSE than the
LSTM model. Specifically, the GRU model had an MSE of 4.67 while the LSTM
model had an MSE of 6.25. This indicates that the GRU model was better at
capturing the underlying patterns in the data and making accurate predictions.
We also generated a plot to visualize the performance of our models. As seen
in Fig 6, also we observed that GRU is 30% faster than LSTM in performance.

Figure 6: Final Resultant graph of LSTM and GRU that are compered with actual
Bitcoin price.

The plot of the training and validation MSE loss for our LSTM and GRU models
in Fig 7 and 8 showed that both models achieved good performance without
overfitting to the training data or producing a lot of noise in the validation
data.

Specifically, the plot revealed that the training loss decreased steadily over
time for both models, indicating that the models were learning from the data
and becoming more accurate at predicting Bitcoin prices. The validation loss
also decreased for both models, albeit with some fluctuations, suggesting that
the models were generalizing well to new data and were not overfitting.

Figure 7: Training and Validation MSE Loss for LSTM. Figure 8: Training and
Validation MAE Loss for GRU.

###  IV-A Discussion

In our experiments, using the GRU model was more effective than using the LSTM
model when predicting Bitcoin price. This aligns with previous research that
has highlighted the superiority of GRU models for specific tasks. The
explanation for this may be that GRU models are better equipped to process
sequential data that has long-term dependencies, as is the case with financial
time series data such as Bitcoin prices. Furthermore, our models benefited
from L2 regularization which aided in reducing overfitting and noise,
ultimately improving performance. As such, L2 regularization can be a valuable
tool in optimizing neural network models.

Despite the promising results of our study, it is important to acknowledge
several limitations. Firstly, our investigation focused solely on the
performance of two neural network models, LSTM and GRU, and there may be other
models that could potentially produce superior results. Secondly, our models’
performance could have been impacted by the specific parameters we used, such
as the learning rate, the number of hidden layers, and the batch size. Further
research is required to explore the impact of various parameters on the
efficacy of neural network models in predicting Bitcoin prices.

##  V Conclusion and Future Works

In conclusion, our study highlights the potential of using neural network
models for predicting the price of Bitcoin. Our results indicate that the GRU
model is more effective than the LSTM model for this task, and that L2
regularization can be a useful technique for improving the performance of
neural network models. Our study provides insights into how to improve the
accuracy of Bitcoin price predictions, and suggests avenues for further
research in this area

## References

  * [1] Y. Yu and C. H. Tan, “Bitcoin blockchain explained: Development and challenges,” in _Financial Management in the Digital Economy_ , 2022, pp. 1–21.
  * [2] Y. Kitamura, “Can we stabilize the price of a cryptocurrency? understanding the design of bitcoin and its potential to compete with central bank money,” in _Quest for Good Money: Past, Present and Future_. Springer, 2022, pp. 103–123.
  * [3] A. S. Kumar, S. Priyanka, K. Dhanashree, V. Praveen, and R. Rekha, “Efficient binary grasshopper optimization based neural network algorithm for bitcoin value prediction,” _International Journal of Nonlinear Analysis and Applications_ , vol. 13, pp. 53–60, 2022.
  * [4] Y.-S. Ren, C.-Q. Ma, X.-L. Kong, K. Baltas, and Q. Zureigat, “Past, present, and future of the application of machine learning in cryptocurrency research,” _Research in International Business and Finance_ , vol. 63, p. 101799, 2022.
  * [5] S. Corbet, B. Lucey, A. Urquhart, and L. Yarovaya, “Cryptocurrencies as a financial asset: A systematic analysis,” _International Review of Financial Analysis_ , vol. 62, pp. 182–199, 2019.
  * [6] Y. Chen, J. Wu, and Z. Wu, “China’s commercial bank stock price prediction using a novel k-means-lstm hybrid approach,” _Expert Systems with Applications_ , vol. 202, p. 117370, 2022.
  * [7] D. Cheng, F. Yang, S. Xiang, and J. Liu, “Financial time series forecasting with multi-modality graph neural network,” _Pattern Recognition_ , vol. 121, p. 108218, 2022.
  * [8] S. Wu, Y. Liu, Z. Zou, and T.-H. Weng, “S_i_lstm: stock price prediction based on multiple data sources and sentiment analysis,” _Connection Science_ , vol. 34, no. 1, pp. 44–62, 2022.
  * [9] M. Aghashahi and S. Bamdad, “Analysis of different artificial neural networks for bitcoin price prediction,” _International Journal of Management Science and Engineering Management_ , pp. 1–8, 2022.
  * [10] Z. Ye, Y. Wu, H. Chen, Y. Pan, and Q. Jiang, “A stacking ensemble deep learning model for bitcoin price prediction using twitter comments on bitcoin,” _Mathematics_ , vol. 10, no. 8, p. 1307, 2022.
  * [11] E. S. Pour, H. Jafari, A. Lashgari, E. Rabiee, and A. Ahmadisharaf, “Cryptocurrency price prediction with neural networks of lstm and bayesian optimization,” _European Journal of Business and Management Research_ , vol. 7, no. 2, pp. 20–27, 2022.


# Ego-Motion Aware Target Prediction Module for Robust Multi-Object Tracking

Navid Mahdian1†, Mohammad Jani1†, Amir M. Soufi Enayati2, and Homayoun
Najjaran1,2 ${}^{*}\,$This work was supported by MITACS under the Accelerate
Program Internship No. IT31385 and Sentire.${}^{1}\,$Department of Electrical
and Computer Engineering, University of Victoria, Victoria, BC, Canada
{navidmahdian, mhmdjani, najjaran}@uvic.ca${}^{2}\,$Department of Mechanical
Engineering, University of Victoria, Victoria, BC, Canada
<EMAIL_ADDRESS>Equivalent contribution as the common first authors.

###### Abstract

Multi-object tracking (MOT) is a prominent task in computer vision with
application in autonomous driving, responsible for the simultaneous tracking
of multiple object trajectories. Detection-based multi-object tracking (DBT)
algorithms detect objects using an independent object detector and predict the
imminent location of each target. Conventional prediction methods in DBT
utilize Kalman Filter (KF) to extrapolate the target location in the upcoming
frames by supposing a constant velocity motion model. These methods are
especially hindered in autonomous driving applications due to dramatic camera
motion or unavailable detections. Such limitations lead to tracking failures
manifested by numerous identity switches and disrupted trajectories. In this
paper, we introduce a novel KF-based prediction module called the Ego-Motion
Aware Target Prediction (EMAP) module by focusing on the integration of camera
motion and depth information with object motion models. Our proposed method
decouples the impact of camera rotational and translational velocity from the
object trajectories by reformulating the Kalman Filter. This reformulation
enables us to reject the disturbances caused by camera motion and maximizes
the reliability of the object motion model. We integrate our module with four
state-of-the-art base MOT algorithms, namely OC-SORT, Deep OC-SORT, ByteTrack,
and BoT-SORT. In particular, our evaluation on the KITTI MOT dataset
demonstrates that EMAP remarkably drops the number of identity switches (IDSW)
of OC-SORT and Deep OC-SORT by 73% and 21%, respectively. At the same time, it
elevates other performance metrics such as HOTA by more than 5%. Our source
code is available at https://github.com/noyzzz/EMAP.

##  I Introduction

Multi-object tracking stands out as a pivotal task in computer vision, finding
diverse applications, ranging from monitoring crowds via surveillance cameras
to analyzing the intricate movements of soccer players on the field to
tracking vehicles on the road and following marine life in the ocean [1].
Additionally, MOT serves as a critical subsystem in autonomous vehicle
systems, providing a medium-level representation integral to path-planning
processes [2].

Detection-Based Tracking is commonly employed in these applications due to its
efficacy in scenarios where new objects may emerge or existing objects may
exit the frame. Within a DBT framework, MOT can be represented as a graph
optimization problem. In this context, an object detector generates bounding
boxes of detected objects in every frame, which serve as graph nodes, and two
nodes are connected if and only if they belong to the track of the same
object. The basic constraints for this problem enforce three conditions: no
backward edge is selected, each node is connected to at most one other node,
and no node has multiple incoming nodes. Thus, the MOT problem is reduced to
determining the optimal association between nodes in the graph. Approaches
like Tracktor++ [3] perform such graph optimization under the assumption of a
high frame rate and available detections in every frame. However, restricting
association to every pair of consecutive frames may be overly simplistic since
it barely reflects real-world scenarios where camera motion and occlusion
exist.

(a)

(b)

(c)

(d)

(e) (a)

(f) (b)

Figure 1: In this scenario, the object detection is missed, leading to failure
in predicting the location of the object during lane-changing for Vanilla
ByteTrack (a). ByteTrack + EMAP (b) significantly improves the prediction,
successfully tracking the object in the next two frames.

In situations where detections are not consistently available in every frame,
it becomes necessary to either abandon the assumption of association between
adjacent frames or substitute the unavailable detection with object state
predictions.

In KF-based methods, a motion model is assumed for the target objects in the
frame and is described by a state transition matrix (F). The rest of the
process is modeled using an observation matrix (H), a process noise (Q), an
observation noise (R), and an optional disturbance matrix (G) The object
location in the image frame is regularly predicted at each time step using the
predict stage of Kalman Filter:

$\left\\{\begin{aligned}
&\hat{\mathbf{x}}_{n+1|n}=\mathbf{F}_{n}\hat{\mathbf{x}}_{n|n}+[\mathbf{G}_{n}\mathbf{w}_{n}]\\\
&\mathbf{P}_{n+1|n}=\mathbf{F}_{n}\mathbf{P}_{n|n}\mathbf{F}_{n}^{\top}+\mathbf{Q}_{n}\\\
\end{aligned}\right.$ (1)

Then, newly detected objects are matched with the predicted positions of
existing objects, resulting in new observations denoted as z. In the next
stage of the Kalman Filter known as the update stage, the Kalman gain (K),
state estimate ($\hat{\mathbf{x}}$) and estimate covariance (P) are all
recalibrated:

$\left\\{\begin{aligned}
\mathbf{K}_{n}&=\mathbf{P}_{n|n-1}\mathbf{H}^{\top}(\mathbf{H}\mathbf{P}_{n|n-1}\mathbf{H}^{\top}+\mathbf{R}_{n})^{-1}\\\
\hat{\mathbf{x}}_{n|n}&=\hat{\mathbf{x}}_{n|n-1}+\mathbf{K}_{n}(\mathbf{z}_{n}-\mathbf{H}\hat{\mathbf{x}}_{n|n-1})\\\
\mathbf{P}_{n|n}&=(\mathbf{I}-\mathbf{K}_{n}\mathbf{H})\mathbf{P}_{n|n-1}\end{aligned}\right.$
(2)

Many KF-based approaches are constructed upon the foundation of SORT [4],
which defines the state for each target as follows:

$\mathbf{x}=[u,v,s,r,\dot{u},\dot{v},\dot{s}]^{\top}$ (3)

In this state definition, $u$ and $v$ represent the horizontal and vertical
pixel locations of the center of the target object, while $s$ and $r$ denote
the area and height of the bounding box, respectively. Notably, this
definition operates under the assumption of a constant velocity model for the
tracked objects. While such an assumption may not hold for all objects, it
serves as a reasonable estimation in scenarios where the camera is stationary
or has minimal movements, as exemplified by the MOT20 dataset [5]. In
scenarios where the camera is mounted on a moving vehicle, the impact of ego-
vehicle movement as well as the object motion invalidates the target’s
constant velocity motion. Specifically, as the motion of the camera becomes
more dynamic, the accuracy of predicted $\dot{u}$ and $\dot{v}$ diminishes,
even when the tracked object maintains a constant velocity. In Figure 1 (a),
the performance of the Kalman Filter is compromised due to rapid camera
motion. This failure becomes particularly pronounced in scenarios such as lane
changing and taking turns, where the constant-velocity Kalman Filter struggles
to make accurate predictions when new observations are unavailable to
compensate for the changing dynamics.

Figure 2: Diagram illustrating the three phases of a detection-based multi-
object tracking algorithm. The figure highlights the integration of the EMAP
module within the system.

In this paper, we propose the Ego-Motion Aware Target Prediction module and
its integration into KF-based methods. This module incorporates the motion of
the ego-vehicle into Kalman Filter, thereby enhancing the performance of MOT
solutions. Our contributions can be summarized as follows:

  * •

We redefine the Kalman Filter state to decouple camera motion from object
trajectories, thereby rejecting disturbances caused by camera motion and
maximizing the reliability of the object motion model.

  * •

We demonstrate the power of EMAP module through our comprehensive experimental
evaluations on the KITTI dataset [6] as well as our tailored CARLA [7]
autonomous driving dataset and report a substantial improvement in tracking
performance, particularly manifested by the reduction in the number of
identity switches.

##  II Related Work

The related work is studied through two parallel approaches: Detection-based
Tracking, and Motion Modeling.

##### Detection-based Tracking

Most of the leading MOT approaches [4, 8, 9, 10, 11, 12] follow the detection-
based tracking schema which consists of three distinct consecutive tasks,
namely, detection, prediction, and association. DBT methods have gained
significant attention, driven by notable advancements in the reliability of
object detectors in recent years [13, 14, 15], enabling DBT techniques to
primarily focus on finding the optimal association between per-frame detection
bounding boxes. However, the overall performance of such techniques is highly
dependent on the detection module. As a result, when the detector fails to
accurately detect target objects, the prediction module is responsible for
predicting the imminent location of the missed object until either the object
is detected again or its track is terminated permanently. In the prediction
part, the goal is to find the best prediction of the object location in the
next frames based on the previous trajectory of the target. Many dominant
methods [9, 10, 11] use Kalman Filter as their prediction model. In [4], the
authors exploited Kalman Filter with a constant velocity motion model for the
targets to anticipate their future location. Building on their work, OC-SORT
[9] expanded upon the same concept and proposed a method to mitigate the
accumulated noise during occlusions. This is achieved by generating a virtual
trajectory for the occlusion period while maintaining the simplicity of the
Kalman filter model.

Some methods [16, 17] view the prediction and association phase as a joint
problem and try to solve the problem using RNNs. For instance, the authors of
[16] presented a structure of RNNs that uses motion, appearance, and
interaction features of the objects to perform prediction and association
based on the cues over a temporal window. Several works [11, 18, 19]
concentrated on enriching the association phase by utilizing the appearance
features. In [11] the authors proposed a deep association metric by training a
CNN on MARS [20] person reidentification dataset. [18] designed an
identification CNN network to calculate the visual resemblance among intra-
frame target bounding boxes and a Gaussian mixture probability hypothesis
density filter [21] to track multiple targets across varying densities of
targets. [19] introduced an innovative unsupervised re-identification learning
module, eliminating the need for object labels. It incorporated an occlusion
estimation module to predict occluded areas, achieving results on par with
supervised approaches.

##### Motion Modeling

SORT-based tracking approaches leverage a constant velocity model to represent
and predict the location of targets within the frame. While this simple model
can accurately represent the target movement in occlusion-free static-camera
environments, its generalizability decreases in moving camera scenarios
specifically when occlusion occurs. Several trackers [12, 22, 23, 24, 25] used
motion cues from objects and the ego-camera to provide a sturdier motion
model. MAT [12] utilized the Enhanced Correlation Coefficient Maximization
(ECC) [26] model to estimate the rotation and translational motion of the
camera. It also employs Kalman Filter to predict the location of the targets.
[25] introduced a motion-aware matching module to match tracks with the new
observations based on the motion features.

Nevertheless, while these motion modeling approaches utilized motion clues to
increase the reliability of MOT, none of them directly applied the motion
information to the Kalman Filter representation. In this work, we adopt the
SORT method and integrate ego-motion data into the state definition and Kalman
Filter formulation to reject the disturbances and increase the reliability.

Algorithm 1 Integrating EMAP with a sample base tracker. Green highlights our
main contribution.

0: Existing tracks buffer $\mathbb{T}$, New Detections $\mathbb{D}$, Camera
translational and rotational motion ($\dot{D}$, $\dot{\psi}$), Depth image
$I_{d}$

0: Updated tracks buffer $\mathbb{T}^{\mathrm{updated}}$

1: Initialize $\mathbb{T}^{\mathrm{pred}}\leftarrow\varnothing$

2: for track $t$ in $\mathbb{T}$ do

3: if RotationOnly then

4: $\mathbb{T}^{\mathrm{pred}}\leftarrow\mathbb{T}^{\mathrm{pred}}\,\cup$
Pred($t$, $\dot{\psi}$)

5: else if TranslationOnly then

6: $\mathbb{T}^{\mathrm{pred}}\leftarrow\mathbb{T}^{\mathrm{pred}}\,\cup$
Pred($t$, $\dot{D}$, $I_{d}$)

7: else

8: $\mathbb{T}^{\mathrm{pred}}\leftarrow\mathbb{T}^{\mathrm{pred}}\,\cup$
Pred($t$, $\dot{D}$, $\dot{\psi}$, $I_{d}$)

9: end if

10: end for

11: ($\mathbb{T}^{\mathrm{matched}}$, $\mathbb{D}^{\mathrm{matched}}$)
$\leftarrow$ Associate($\mathbb{T}^{\mathrm{pred}},\mathbb{D}$)

12: $\mathbb{T}^{\mathrm{unmatched}}$ $\leftarrow$
$\mathbb{T}\setminus\mathbb{T}^{\mathrm{matched}}$

13: $\mathbb{D}^{\mathrm{unmatched}}$ $\leftarrow$
$\mathbb{D}\setminus\mathbb{D}^{\mathrm{matched}}$

14: UpdateKFstates($\mathbb{T}^{\mathrm{matched}}$,
$\mathbb{D}^{\mathrm{matched}}$)

15: Identify $\mathbb{T}^{\mathrm{lost}}$ from the
$\mathbb{T}^{\mathrm{unmatched}}$

16: Create $\mathbb{T}^{\mathrm{new}}$ from the
$\mathbb{D}^{\mathrm{unmatched}}$

17:
$\mathbb{T}^{\mathrm{updated}}\leftarrow\mathbb{T}^{\mathrm{matched}}\,\cup\,\mathbb{T}^{\mathrm{new}}\cup\,\\{\mathbb{T}^{\mathrm{unmatched}}\setminus\mathbb{T}^{\mathrm{lost}}\\}$

##  III Methodology

In the following, the derivation of EMAP, including the projection,
integration with Kalman Filter, and isolation of camera motion is discussed in
detail.

###  III-A Camera motion projection

Assuming a rectified image is at hand and using the pinhole camera model, the
translational movement of the ego-vehicle affects the target object’s location
on the image frame according to the following equations:

$\hat{u}_{trans}=f\frac{d\sin(\frac{u}{f})}{d\cos(\frac{u}{f})-\Delta
t\dot{D}}$ (4)
$\hat{v}_{trans}=f\frac{d\sin(\frac{v}{f})}{d\cos(\frac{v}{f})-\Delta
t\dot{D}}$ (5)

where $f$ is the camera focal length, and $d$ is the Euclidean distance
between the object and the vehicle. Also, $\dot{D}$ denotes the ego-vehicle
forward translational velocity toward the normal vector of the image plane;
$u$ and $v$ are the horizontal and vertical pixel location of the target from
the center of image, and $\hat{u},\hat{v}$ are their respective predictions.

The impact of ego-vehicle rotation on the pixel location of an object is
purely horizontal and governed by:

$\hat{u}_{rot}=f\frac{\tan(\Delta
t\dot{\psi})+\frac{u}{f}}{1-\frac{u}{f}\tan(\Delta t\dot{\psi})}$ (6)

where $\dot{\psi}$ is the ego-vehicle yaw rate. Assuming a small time
difference between each two consecutive frames leads to linearized
formulations of the form:

$\hat{u}_{trans}\approx\frac{u\sqrt{u^{2}+f^{2}}}{fd}\cdot\dot{D}\Delta t+u$
(7) $\hat{v}_{trans}\approx\frac{v\sqrt{v^{2}+f^{2}}}{fd}\cdot\dot{D}\Delta
t+v$ (8) $\hat{u}_{rot}\approx
f\left(1+\frac{u^{2}}{f^{2}}\right)\cdot\dot{\psi}\Delta t+u$ (9)

Figure 3: Visualization of town #10 in CARLA simulator, featuring four
distinct simulation scenarios superimposed on the map. The paths illustrate
the diverse trajectories taken in our dataset, capturing a range of scenarios
for comprehensive analysis.

###  III-B Camera motion integration with Kalman filter

Considering the availability of scene depth map and camera motion information,
we can now reformulate the Kalman Filter state definition to integrate this
information into a base model. Following the similar format used by SORT in
equation 3, we approximate the motion of the object by a constant velocity
model; however, we decouple the effect of camera motion in this motion model.

Each target object is then modeled using the following state definition:

$\mathbf{x}=[u^{l},v^{t},u^{r},v^{b},{\dot{x}}^{l},{\dot{y}}^{t},{\dot{x}}^{r},{\dot{y}}^{b}]^{\top}$
(10)

Where $(u^{l},v^{t})$ indicates the top-left corner of a bounding box, while
$(u^{r},v^{b})$ represents the bottom-right corner of that. Unlike SORT which
uses derivatives of $(u,v)$ as noted in 3, we use
${\dot{x}}^{l},{\dot{y}}^{t},{\dot{x}}^{r}$, and ${\dot{y}}^{b}$. These
variables denote the projected horizontal and vertical pixel velocities of the
corners of the target object bounding box caused solely by the object’s motion
in the world frame. Assuming that the movement of the ego-vehicle can be
approximated by two independent parameters, i.e., orientation and forward
displacement, we now define the state equation of Kalman Filter as follows:

$\mathbf{\hat{x}_{n+1}}=\begin{bmatrix}I_{4\times 4}&dtI_{4\times 4}\\\
0_{4\times 4}&I_{4\times
4}\end{bmatrix}\mathbf{\hat{x}_{n}}+\begin{bmatrix}G_{n}^{\psi}&G_{n}^{D}\\\
\end{bmatrix}\begin{bmatrix}\dot{\psi}\\\ \dot{D}\end{bmatrix}dt$ (11)

where,

$G_{n}^{\psi}=\begin{bmatrix}f\left(1+\left(\frac{u_{n}^{l}}{f}\right)^{2}\right)\\\
\\\ 0\\\ \\\ f\left(1+\left(\frac{u_{n}^{r}}{f}\right)^{2}\right)\\\ \\\ 0\\\
\\\ 0_{4\times 1}\end{bmatrix},\quad
G_{n}^{D}=\begin{bmatrix}\frac{u_{n}^{l}\sqrt{(u_{n}^{l})^{2}+f^{2}}}{fd_{n}}\\\
\\\ \frac{v_{n}^{t}\sqrt{(v_{n}^{t})^{2}+f^{2}}}{fd_{n}}\\\ \\\
\frac{u_{n}^{l}\sqrt{(u_{n}^{l})^{2}+f^{2}}}{fd_{n}}\\\ \\\
\frac{v_{n}^{b}\sqrt{(v_{n}^{b})^{2}+f^{2}}}{fd_{n}}\\\ \\\ 0_{4\times
1}\end{bmatrix}$ (12)

These equations describe the Pred function in Algorithm 1.

When an associated detection of an object exists, Kalman Filter update phase
is used to re-calibrate the state variables. Then, the prediction is performed
to extrapolate the location of the target in the next frames. Figure 2 shows
an overview of the EMAP module integrated into a DBT multi-object tracking
framework. In the depicted figure, the input image, captured by the camera, is
directed into the detection module. Simultaneously, the synchronized odometry
data and depth image are fed into the EMAP module. Following the generation of
predictions by EMAP for the subsequent location of each existing track, the
tracks are then associated with the newly detected objects. Finally, the
iteration ends when the tracks are updated with their new associated detection
or discarded according to the baseline design. The pseudocode presented in
Algorithm 1 outlines the integration of our prediction module with existing
DBT algorithms. This integration requires two additional inputs, camera motion
information, and the depth map.

###  III-C Justification for isolating the camera motion

As mentioned in section I, most Kalman Filter-based DBT algorithms approximate
the motion of the target by constant velocity models independent of other
objects and camera motion. However, this assumption can be readily violated in
scenarios where camera movements introduce significant disturbances to the
bounding box location. This issue is particularly exacerbated for a constant
velocity model, as it struggles to accurately predict the target’s trajectory
in the presence of detection gaps. Such inaccurate predictions give rise to
frequent identity switches, ultimately compromising the overall MOT
performance. To address this problem, in EMAP module, we take the camera
motion into account isolated from the object motion model.

Figure 4: HOTA vs IDSW comparisons of OC-SORT, Deep OC-SORT, ByteTrack, and
BoT-SORT on 21 KITTI train sequences with or without EMAP module. The height
and width of the ellipses are the standard deviation of the distribution.

##  IV Experiments

In the following section, the datasets, metrics, baseline trackers, and the
results are presented.

###  IV-A Datasets

####  IV-A1 KITTI

The KITTI dataset serves as a cornerstone in benchmarking computer vision
algorithms for autonomous driving tasks. It consists of real-world data
captured from a car equipped with sensors such as LiDAR, camera, and GPS. In
our experiments, we use the KITTI training dataset which consists of 21
sequences of diverse driving scenarios such as urban, highway, and rural
environments. For the evaluation, we focus on the Car and Pedestrian classes
for which we take bounding boxes detected by PermaTrack [27] and also YOLOv8
[28].

####  IV-A2 CARLA Simulation Dataset

In addition to the KITTI dataset, we conducted experiments using a custom-
generated dataset within the CARLA autonomous driving simulator. The CARLA
simulator offers a realistic virtual environment for autonomous driving,
allowing for controlled and diverse scenarios. In our dataset, we have four
sequences each representing distinct features. In Scenario #1, a car navigates
a straight road. Scenario #2 involves the car initiating movement from a
stationary position, moving along a straight path, encountering a curve, and
concluding the sequence upon completing the curve. Scenario #3 features the
car starting from a stationary position, progressing along a straight
trajectory, taking a left turn, reaching a curve, and coming to a stop
afterward. In Scenario #4, the vehicle travels a path with winding movements
ending after taking a right turn. Figure 3 illustrates the paths on the CARLA
town #10 map. We integrate CARLA with MOT algorithms through the Robot
Operating System (ROS) [29], enabling communication between Python and CARLA.
Synchronized messages, including RGB images, depth map (aligned with RGB),
ego-vehicle odometry, and automatically generated ground truth tracking
bounding boxes from CARLA are streamed to Python via ROS for seamless
evaluation.

###  IV-B Evaluation Metrics

We select HOTA [30] as the primary metric for its ability to balance between
the accuracy of both object detection and association in multi-object
tracking. Also, we report IDSW as the additional primary metric because it
represents the reliability of generated trajectories. The underlying reason is
that IDSW counts the occurrences of incorrect object identity swaps or the
loss and subsequent reinitialization of identities. Our analysis extends to
Association Accuracy (AssA) for evaluating association performance, along with
IDF1 as an additional metric in the same context. It is worth mentioning that
MOTA, which used to be the predominant measure of MOT performance, is heavily
affected by the detection quality. Hence, to ensure a fair comparison, we use
identical detections among all algorithms to report the MOTA score.

###  IV-C Baseline SORT-based Algorithms

In order to evaluate the impact of EMAP on existing MOT algorithms, we
selected four state-of-the-art SORT-based algorithms and integrated EMAP with
their prediction module to evaluate the effect of EMAP on their performance.
The algorithms we selected as baseline are OC-SORT [9], Deep OC-SORT [31],
ByteTrack [8], and BoT-SORT [32]. All four baselines are Kalman Filter-based
algorithms. However, in BoT-SORT, authors use image registration to
approximate the camera motion and compensate for the effect of this motion.

###  IV-D Results

In this section, we present the performance of our solution on both the CARLA
and the KITTI datasets. Table I displays the HOTA scores and identity switch
counts for each sequence in the CARLA simulation dataset. Notably, EMAP
significantly impacts Sequence #4, characterized by substantial vehicle
rotational and translational movements. This sequence shows a decrease in
identity switches and an increase in HOTA values for all trackers,
highlighting EMAP’s effectiveness in complex motion scenarios.

Table II presents the average metric values across the 21 sequences of the
KITTI train dataset when detections are taken from PermaTrack. EMAP notably
reduces the number of identity switches across all four baselines, with the
most significant impact observed in OC-SORT, where it decreases by 76%.
Furthermore, EMAP integration improves various metrics such as HOTA, MOTA,
IDF1, FP, FN, AssA, and AssR in all four baseline trackers. Figure 4
illustrates that integration of the EMAP module notably decreases identity
switches and enhances HOTA scores compared to baseline performance.
Additionally, it demonstrates that EMAP contributes to the improved robustness
of baseline trackers, as evidenced by the reduced variance in IDSW and HOTA
across all sequences.

In Table III, we present the evaluation on KITTI train dataset using YOLOv8x
as the object detector. Despite the fact that YOLOv8x exhibits weaker
performance compared to PermaTrack indicated by higher missed detections and
lower detection accuracy, the addition of EMAP still positively affects the
performance of all baseline trackers.

To assess the individual and combined impact of the translation and rotational
submodules, we conduct an ablation study comparing the results of EMAP with
each submodule added separately and together, as shown in Table IV. The
largest reduction in the number of identity switches occurs when both
submodules are active. However, the maximum HOTA is attained when only the
rotational module is activated.

TABLE I: Performance comparison of MOT algorithms on our CARLA dataset split by sequence with YOLOv8x as the object detector. The best results are shown in bold. Tracker | Sequence #1 | Sequence #2 | Sequence #3 | Sequence #4
---|---|---|---|---
HOTA$\uparrow$ |  IDs$\downarrow$ |  HOTA$\uparrow$ |  IDs$\downarrow$ |  HOTA$\uparrow$ |  IDs$\downarrow$ |  HOTA$\uparrow$ |  IDs$\downarrow$
OC-SORT [9] |  36.15 |  5 |  24.84 |  11 |  36.3 |  15 |  24.09 |  19
OC-SORT + EMAP [9] |  36.24 |  1 |  25.06 |  3 |  37.04 |  15 |  29.09 |  7
BoT-SORT [32] |  35.08 |  7 |  21.88 |  35 |  32.64 |  30 |  23.44 |  24
BoT-SORT + EMAP [32] |  34.94 |  7 |  22.14 |  35 |  31.65 |  30 |  24.54 |  18
Deep OC-SORT [31] |  35.71 |  4 |  23.65 |  55 |  36.33 |  36 |  25.17 |  67
Deep OC-SORT + EMAP [31] |  35.55 |  9 |  20.75 |  129 |  35.55 |  61 |  26.66 |  38
ByteTrack [8] |  34.94 |  0 |  21.14 |  1 |  34.26 |  5 |  25.71 |  8
ByteTrack + EMAP |  25.05 |  0 |  21.7 |  0 |  34.91 |  4 |  28.39 |  2
TABLE II: Average performance (on 21 sequences) on KITTI-train dataset, detections are taken from PermaTrack. The best results are shown in bold. Tracker |  HOTA$\uparrow$ |  MOTA$\uparrow$ |  IDF1$\uparrow$ |  FP$\downarrow$ |  FN$\downarrow$ |  IDs$\downarrow$ |  AssA$\uparrow$ |  AssR$\uparrow$
---|---|---|---|---|---|---|---|---
OC-SORT [9] | 67.74 | 64.53 | 78.24 | 709.33 | 234.19 | 53.24 | 68.38 | 79.34
OC-SORT + EMAP | 70.21 | 67.87 | 82.17 | 662.19 | 187.05 | 14.10 | 73.11 | 85.32
Deep OC-SORT [31] | 64.69 | 62.59 | 74.20 | 893.67 | 418.52 | 108.38 | 62.84 | 74.41
Deep OC-SORT + EMAP | 66.54 | 64.38 | 76.37 | 863.14 | 388.00 | 85.24 | 66.22 | 77.80
ByteTrack [8] | 70.05 | 74.98 | 84.60 | 339.10 | 305.00 | 21.95 | 72.62 | 80.28
ByteTrack + EMAP | 71.75 | 75.82 | 85.78 | 324.33 | 295.33 | 14.71 | 74.71 | 81.73
BoT-SORT [32] | 55.40 | 47.74 | 65.89 | 602.95 | 687.29 | 72.33 | 56.36 | 62.33
BoT-SORT + EMAP | 57.23 | 50.11 | 68.49 | 578.29 | 670.76 | 57.95 | 59.58 | 66.05
TABLE III: Average performance (on 21 sequences) on KITTI-train dataset, detections are taken from YOLOv8. The best results are shown in bold. Tracker |  HOTA$\uparrow$ |  MOTA$\uparrow$ |  IDF1$\uparrow$ |  FP$\downarrow$ |  FN$\downarrow$ |  IDs$\downarrow$ |  AssA$\uparrow$ |  AssR$\uparrow$
---|---|---|---|---|---|---|---|---
OC-SORT [9] | 39.35 | 37.0 | 53.04 | 163.14 | 1187.9 | 18.86 | 46.16 | 48.82
OC-SORT + EMAP | 40.28 | 38.0 | 54.4 | 168.05 | 1179.95 | 12.29 | 47.68 | 50.25
Deep OC-SORT [31] | 38.22 | 35.77 | 50.08 | 270.67 | 1236.38 | 69.38 | 42.21 | 45.36
Deep OC-SORT + EMAP | 39.6 | 37.2 | 52.88 | 225.48 | 1191.19 | 41.33 | 45.15 | 48.41
ByteTrack [8] | 33.48 | 31.29 | 44.66 | 102.76 | 1368.76 | 14.24 | 42.11 | 44.2
ByteTrack + EMAP | 33.78 | 31.32 | 45.81 | 83.95 | 1356.81 | 11.9 | 42.69 | 44.12
BoT-SORT [32] | 25.16 | 23.5 | 34.57 | 138.0 | 1584.1 | 46.05 | 29.17 | 29.81
BoT-SORT + EMAP | 28.06 | 25.49 | 39.21 | 123.7 | 1425.65 | 35.2 | 33.82 | 34.55
TABLE IV: Ablation on KITTI dataset with Translational and Rotational sub-modules. The best results are shown in bold. Method | Tracker |  HOTA$\uparrow$ |  MOTA$\uparrow$ |  IDF1$\uparrow$ |  IDs$\downarrow$
---|---|---|---|---|---
Baseline | OC-SORT | 67.74 | 64.53 | 78.24 | 53.24
Deep OC-SORT | 64.69 | 62.59 | 74.2 | 108.38
ByteTrack | 70.05 | 74.98 | 84.6 | 21.95
BoT-SORT | 55.4 | 47.74 | 65.89 | 72.33
Translational-Only EMAP | OC-SORT | 71.45 | 74.59 | 83.35 | 24.38
Deep OC-SORT | 65.33 | 63.08 | 74.91 | 99.43
ByteTrack | 70.25 | 74.77 | 84.4 | 23.0
BoT-SORT | 55.28 | 48.65 | 65.58 | 76.71
Rotational-Only EMAP | OC-SORT | 72.63 | 75.1 | 85.05 | 15.43
Deep OC-SORT | 65.57 | 63.72 | 75.62 | 97.14
ByteTrack | 71.18 | 75.86 | 85.69 | 15.1
BoT-SORT | 55.41 | 48.65 | 65.59 | 85.81
EMAP | OC-SORT | 70.21 | 67.87 | 82.17 | 14.1
Deep OC-SORT | 66.54 | 64.38 | 76.37 | 85.24
ByteTrack | 71.75 | 75.82 | 85.78 | 14.71
BoT-SORT | 57.23 | 50.11 | 68.49 | 57.95

###  IV-E Visualization results

To showcase the impact of EMAP module, we present a visualization of the
Bytetrack MOT algorithm operating within the CARLA simulator under a scenario
where detection is lost. As depicted in Figure 1 (a), the prediction module of
the vanilla ByteTrack struggles to accurately anticipate the position of the
target after the detection becomes unavailable (denoted by black bounding
boxes). In contrast, Figure 1 (b) illustrates the enhanced performance
achieved by incorporating our module. The predicted bounding box location
aligns more closely with the actual location, underscoring the efficacy of the
ego-motion aware prediction module.

##  V Conclusions and Future Work

In this paper, we enhance the performance of detection-based MOT algorithms by
leveraging both the motion of the camera and depth information. Our
contribution, the Ego-Motion Aware Prediction (EMAP) module, serves as a
predictive component seamlessly integrated into DBT multi-object tracking
algorithms. EMAP adeptly rejects disturbances introduced by camera motion,
thereby boosting the reliability of the object motion model. Through the
reformulation of the Kalman Filter, EMAP effectively decouples the impact of
the rotational and translational velocities of the camera from the target
location within the image frame. In our experiments on KITTI train MOT
dataset, EMAP significantly enhances the performance of the base trackers,
evident in increased HOTA and reduced number of identity switches. Our in-
depth investigation in CARLA simulator further reveals the efficacy of EMAP
where ego-camera motion dominantly impacts the target location. Our module
depicts how the reliability of existing MOT algorithms can be improved in
real-world scenarios, especially autonomous driving.

As a future direction, we aim to explore the retrieval of camera motion and
depth information using Visual Odometry and depth estimation algorithms solely
from RGB cameras, with the ultimate goal of extending the versatility of our
module to systems relying exclusively on an RGB camera as their primary
sensor.

## References

  * [1] W. Luo, J. Xing, A. Milan, X. Zhang, W. Liu, and T.-K. Kim, “Multiple object tracking: A literature review,” _Artificial Intelligence_ , vol. 293, p. 103448, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0004370220301958
  * [2] C. Badue, R. Guidolini, R. V. Carneiro, P. Azevedo, V. B. Cardoso, A. Forechi, L. Jesus, R. Berriel, T. M. Paixão, F. Mutz, L. d. P. Veronese, T. Oliveira-Santos, and A. F. D. Souza, “Self-driving cars: A survey,” _Expert Systems with Applications_ , vol. 165, p. 113816, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S095741742030628X
  * [3] P. Bergmann, T. Meinhardt, and L. Leal-Taixe, “Tracking Without Bells and Whistles,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_. Seoul, Korea (South): IEEE, Oct. 2019, pp. 941–951. [Online]. Available: https://ieeexplore.ieee.org/document/9010033/
  * [4] A. Bewley, Z. Ge, L. Ott, F. Ramos, and B. Upcroft, “Simple online and realtime tracking,” in _2016 IEEE International Conference on Image Processing (ICIP)_ , Sep. 2016, pp. 3464–3468, iSSN: 2381-8549.
  * [5] P. Dendorfer, H. Rezatofighi, A. Milan, J. Q. Shi, D. Cremers, I. Reid, S. Roth, K. Schindler, and L. Leal-Taix’e, “MOT20: A benchmark for multi object tracking in crowded scenes,” _ArXiv_ , Mar. 2020.
  * [6] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2012.
  * [7] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “CARLA: An open urban driving simulator,” in _Proceedings of the 1st Annual Conference on Robot Learning_ , 2017, pp. 1–16.
  * [8] Y. Zhang, P. Sun, Y. Jiang, D. Yu, F. Weng, Z. Yuan, P. Luo, W. Liu, and X. Wang, “ByteTrack: Multi-object Tracking by Associating Every Detection Box,” in _Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXII_. Berlin, Heidelberg: Springer-Verlag, Oct. 2022, pp. 1–21.
  * [9] J. Cao, J. Pang, X. Weng, R. Khirodkar, and K. Kitani, “Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking,” in _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. Vancouver, BC, Canada: IEEE, Jun. 2023, pp. 9686–9696. [Online]. Available: https://ieeexplore.ieee.org/document/10204818/
  * [10] Y. Du, Z. Zhao, Y. Song, Y. Zhao, F. Su, T. Gong, and H. Meng, “StrongSORT: Make DeepSORT Great Again,” Feb. 2023, arXiv:2202.13514 [cs]. [Online]. Available: http://arxiv.org/abs/2202.13514
  * [11] N. Wojke, A. Bewley, and D. Paulus, “Simple Online and Realtime Tracking with a Deep Association Metric,” Mar. 2017, arXiv:1703.07402 [cs].
  * [12] S. Han, P. Huang, H. Wang, E. Yu, D. Liu, and X. Pan, “MAT: Motion-aware multi-object tracking,” _Neurocomputing_ , vol. 476, no. C, pp. 75–86, Mar. 2022. [Online]. Available: https://doi.org/10.1016/j.neucom.2021.12.104
  * [13] G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics YOLO,” Jan. 2023. [Online]. Available: https://github.com/ultralytics/ultralytics
  * [14] M. Tan, R. Pang, and Q. V. Le, “Efficientdet: Scalable and efficient object detection,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 10 778–10 787.
  * [15] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _2017 IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 2999–3007.
  * [16] A. Sadeghian, A. Alahi, and S. Savarese, “Tracking the Untrackable: Learning to Track Multiple Cues with Long-Term Dependencies,” in _2017 IEEE International Conference on Computer Vision (ICCV)_. Venice: IEEE, Oct. 2017, pp. 300–311. [Online]. Available: http://ieeexplore.ieee.org/document/8237303/
  * [17] A. Milan, S. H. Rezatofighi, A. Dick, I. Reid, and K. Schindler, “Online multi-target tracking using recurrent neural networks,” in _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence_ , ser. AAAI’17. San Francisco, California, USA: AAAI Press, Feb. 2017, pp. 4225–4232.
  * [18] N. L. Baisa, “Occlusion-robust online multi-object visual tracking using a GM-PHD filter with CNN-based re-identification,” _Journal of Visual Communication and Image Representation_ , vol. 80, p. 103279, Oct. 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1047320321001814
  * [19] Q. Liu, D. Chen, Q. Chu, L. Yuan, B. Liu, L. Zhang, and N. Yu, “Online multi-object tracking with unsupervised re-identification learning and occlusion estimation,” _Neurocomputing_ , vol. 483, pp. 333–347, Apr. 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S092523122200008X
  * [20] L. Zheng, Z. Bie, Y. Sun, J. Wang, C. Su, S. Wang, and Q. Tian, “Mars: A video benchmark for large-scale person re-identification,” in _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14_. Springer, 2016, pp. 868–884.
  * [21] B.-N. Vo and W.-K. Ma, “The Gaussian mixture probability hypothesis density filter,” _IEEE Transactions on Signal Processing_ , vol. 54, no. 11, pp. 4091–4104, 2006.
  * [22] G. Wang, Y. Wang, H. Zhang, R. Gu, and J.-N. Hwang, “Exploit the Connectivity: Multi-Object Tracking with TrackletNet,” in _Proceedings of the 27th ACM International Conference on Multimedia_. Nice France: ACM, Oct. 2019, pp. 482–490. [Online]. Available: https://dl.acm.org/doi/10.1145/3343031.3350853
  * [23] H. Liu, T. Xu, and X. Wu, “MMOT: Motion-Aware Multi-Object Tracking with Optical Flow,” in _Proceedings of the 2022 11th International Conference on Computing and Pattern Recognition_. Beijing China: ACM, Nov. 2022, pp. 115–120. [Online]. Available: https://dl.acm.org/doi/10.1145/3581807.3581824
  * [24] B. Shuai, A. Berneshawi, X. Li, D. Modolo, and J. Tighe, “SiamMOT: Siamese Multi-Object Tracking,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. Nashville, TN, USA: IEEE, Jun. 2021, pp. 12 367–12 377.
  * [25] K.-C. Huang, M.-H. Yang, and Y.-H. Tsai, “Delving into motion-aware matching for monocular 3d object tracking,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 6909–6918.
  * [26] G. D. Evangelidis and E. Z. Psarakis, “Parametric Image Alignment Using Enhanced Correlation Coefficient Maximization,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 30, no. 10, pp. 1858–1865, Oct. 2008, conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence. [Online]. Available: https://ieeexplore.ieee.org/document/4515873
  * [27] P. Tokmakov, J. Li, W. Burgard, and A. Gaidon, “Learning to track with object permanence,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 10 860–10 869.
  * [28] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 779–788.
  * [29] M. Quigley, B. Gerkey, K. Conley, J. Faust, T. Foote, J. Leibs, E. Berger, R. Wheeler, and A. Ng, “ROS: an open-source Robot Operating System,” in _IEEE International Conference on Robotics and Automation Workshop on Open Source Software_ , 2009.
  * [30] J. Luiten, A. Ossep, P. Dendorfer, P. Torr, A. Geiger, L. Leal-Taixe, and B. Leibe, “HOTA: A Higher Order Metric for Evaluating Multi-object Tracking,” _International Journal of Computer Vision_ , vol. 129, no. 2, pp. 548–578, Feb. 2021. [Online]. Available: https://link.springer.com/10.1007/s11263-020-01375-2
  * [31] G. Maggiolino, A. Ahmad, J. Cao, and K. Kitani, “Deep OC-SORT: Multi-Pedestrian Tracking by Adaptive Re-Identification,” Feb. 2023, arXiv:2302.11813 [cs]. [Online]. Available: http://arxiv.org/abs/2302.11813
  * [32] N. Aharon, R. Orfaig, and B.-Z. Bobrovsky, “BoT-SORT: Robust Associations Multi-Pedestrian Tracking,” Jul. 2022, arXiv:2206.14651 [cs]. [Online]. Available: http://arxiv.org/abs/2206.14651


# R-Matrix calculations for opacities: IV.: Convergence, completeness, and
comparison of relativistic R-matrix and distorted wave calculations for Fe
xvii and Fe xviii

L Zhao1, S N Nahar1, A K Pradhan1,2,3 1 Department of Astronomy, Ohio State
University, Columbus, Ohio 43210, USA  2 Biophysics Graduate Program, 3
Chemical Physics Program, The Ohio State University, Columbus, Ohio 43210, USA

###### Abstract

To investigate the completeness of coupled channel (CC) Breit-Pauli R-Matrix
(BPRM) calculations for opacities, we employ the relativistic distorted wave
(RDW) method to complement (“top-up”) and compare the BPRM photoionization
cross sections for high-$n\ell$ levels of both Fe xvii and Fe xviii . Good
agreement is found in background photoionization cross sections using these
two methods, which also ensures correct matching of bound level cross sections
for completeness. In order to top-up the CC-BPRM calculations, bound-bound
transitions involving additional bound levels, and a large number of doubly-
excited quasi-bound levels corresponding to BPRM autoionizing resonances
described in paper RMOPII, are calculated using the RDW method.
Photoionization cross sections in the high energy region are also computed and
compared up to about 500 $Ry$, and contributions from higher core level
excitations than BPRM are considered. The effect of configuration interaction
is investigated, which plays a significant role in correctly reproducing some
background cross sections. Owing to the fact that the additional RDW levels
correspond to high-$n\ell$ bound levels that are negligibly populated
according to the Mihalas-Hummer-Däppen equation-of-state (paper I), the effect
on opacities is expected to be small.

##  1 Introduction

Previous papers I-III in this series (hereafter RMPP1, RMOP2, RMOP3) reported
BPRM calculations and plasma effects related to iron opacity at conditions
similar to the solar radiation/convection zone boundary or the base of the
convection zone (BCZ). As outlined in paper RMOP1, opacity calculations need
to consider all possible mechanisms for photon absorption and scattering from
all atomic constituents, including all levels that might possibly contribute.
Furthermore, in order to resolve discrepancies among various theoretical
models based on the DW methods that include different sets of transition
arrays, and experimental measurements [1, 2], it is necessary to establish
convergence of the BPRM calculations and completeness of transitions
considered.

Extensive CC-BPRM calculations $R$-Matrix (BPRM) calculations were carried out
for Fe xvii including 60 fine-structure levels within the $n\leq 3$ complexes
in the Fe xviii target ion [3], and 99 $LS$ terms within $n\leq 4$ (99LS-RM).
They show strong photon absorption due to core excitation, resulting in an
increment of 35% in the Rosseland mean opacity over the Opacity Project (OP)
data [4]. Whereas these previous calculations demonstrated that in the
$R$-Matrix opacity calculations convergence of the close-coupling expansion
was a necessary condition for accuracy, completeness of all possible excited
configurations by additional contributions in the high-energy region still
remains to be ascertained [4, 5, 6]. At BCZ conditions Fe xvii , Fe xviii and
Fe xix are the three dominant iron ions. For example, at the measured iron
opacity [2] and temperature and density T=$2.1\times 10^{6}$ and Ne=$3.1\times
10^{2}2$cc, the three ionization fractions are 0.19, 0.38 and 0.29,
respectively [4].

In this paper, we consider the 218CC-BPRM calculation for Fe xvii and 276CC-
BPRM calculation for Fe xviii , as described in paper RMOP2. The additional or
topup transitions for bound-bound and bound-free data are obtained from
relativistic distorted wave (RDW) calculations using the flexible atomic code
(FAC) [7]. In the following sections, the specifications of the 218CC- and
276CC-BPRM calculations of paper RMOP2 are summarized, followed by the top-up
configurations and transitions calculated using FAC. To ensure data
correspondence from FAC, a procedure of matching the bound levels from BPRM
and FAC results is described, and the bound-bound and bound-free top-up
calculations detailed afterwards.

A key step in the matching-topup procedure is level identification. Unlike
atomic structure and DW calculations, BPRM calculations do not assign
spectroscopic designations a priori and bound states are obtained only as
eigenvalues of the (e + ion) Hamiltonian. As described in paper RMOP2, the
code BPID (Fig. 1, RMOP-I) is used to obtain relevant parameters and for
spectroscopic identification of levels computed in BPRM calculations. The RDW
calculations of course have a pre-assigned identification based on initial set
of electronic configurations specified in the configuration-interaction (CI)
basis. In some instances exited level configuration mixing is such that one
configuration does not dominate the wavefunction expansion of a given state
and RDW and BPRM assignations do not match. It is then required to carefully
examine level parameters such as quantum defects and associated bound-bound
and bound-free transitions to ascertain matching data. Another consideration
is that the precise number of BPRM bound-state eigenvalues depends on an
energy mesh or effective quantum number $\nu(E)$ obtained by ”scanning” at a
fine mesh with sufficient refinement to ensure convergence. The procedure and
results are discussed in this paper.

A potentially important factor is that the close coupling approximation
introduces autoionizing resonances in photoionization cross section, which may
be affected by radiative damping in highly charged H- or He-like ions, and
thereby reduce effective cross sections considerably [16]. However, radiation
damping occurs after photoabsorption, and for ions such as Fe xvii and Fe
xviii this effect is negligible [13, 14]. Therefore, undamped photoionization
cross sections are used in opacity calculations, as reported in paper RMOP2.

##  2 BPRM bound-free and bound-bound data

The current BPRM calculations for Fe xvii and Fe xviii are unprecedented in
terms of scope and magnitude of data produced and processed for opacity
calculations, with the maximum number of free channels 998 and 1288
respectively, from calculations reported in RMOP2. For Fe xvii , 99LS-RM
calculation [4] is extended to 218CC-BPRM by including the fine structure of
the target states. The target configurations ($1s$ is always full, so omitted
for brevity) included are ${2s^{2}2p^{5}}$, ${2s2p^{6}}$,
${2s^{2}2p^{4}n\ell}$, ${2s2p^{5}n\ell}$, ${2p^{6}3\ell^{\prime}}$, where
$n=3,~{}4$, and $\ell,~{}\ell^{\prime}\leq 2$, which have 99 $LS$ terms, or
218 fine structure levels. The continuum orbitals included are $\ell\leq 9$,
and the number of continuum $R$-Matrix basis functions included is 20. The
bound states are found by scanning the eigenvalues of the (e + ion)
Hamiltonian on an effective quantum number $\nu$ up to $\nu\leq 10.1$ [8].
However, as mentioned above, unlike atomic structure calculations where
electronic configurations are specified a priori, R-matrix calculations do not
provide spectroscopic spin-orbital quantum number designations for the bound
levels obtained, nor guarantee that all possible bound levels are found within
the $\nu$-range of interest.

To resolve the first issue, the computer program BPID [9] has been developed
as part of the RM opacity codes described in paper P1. Using the code BPID one
can identify most of the bound levels spectroscopically, albeit with a few
remaining highly mixed levels undetermined (viz. [10]). That obstacle might be
overcome by comparing some physical quantities of these levels calculated by
an atomic structure code such as SUPERSTRUCTURE [11] and FAC [7], as for
example for photoionization cross section to be described in the next section.
The second issue depends on the scanning $\Delta-\nu$-mesh employed;
$\Delta\nu=0.001$ yields fewer bound levels in the 218CC-BPRM than 60CC-BPRM,
so a finer step 0.0001 or 0.0005 is used in the region where levels are
missing, which finally gives 464 bound levels, 10 more than 60CC-BPRM.

For the larger Fe xviii case, two sets of BPRM calculations are done with
different target configurations. One includes up to $n=3$ target
configurations, i.e. $2s^{2}2p^{4}$, $2s2p^{5}$, $2p^{6}$,
$2s^{2}2p^{3}3\ell$, $2s2p^{4}3\ell$, where $\ell\leq 2$, which yields 200
target fine structure levels. In addition to target configurations above, the
other BPRM calculation includes $n=4$ configurations, i.e.
$2s^{2}2p^{3}4\ell$, where $\ell\leq 2$, which yield 276 fine structure
levels. The parameters set for the continuum orbitals and basis functions are
the same as for Fe xvii 218CC-BPRM calculation. The 200CC-BPRM calculation
finds 1149 bound levels with $\Delta\nu=0.001$, while 276CC-BPRM calculation
finds 1163 bound levels with 0.001 as the initial attempt in $\nu$-mesh, and
0.0001 or 0.0005 as the second attempt, in the region where levels are missing
compared with 200CC-BPRM. Thus we may be confident of having converged with
respect to possible number of bound levels with BPRM calculations.

To compute iron opacities for Fe xvii and Fe xviii oscillator strengths from
the 60CC-BPRM and 200CC-BPRM calculations, and photoionization cross sections
from the 218CC-BPRM and 276CC-BPRM calculations, are used respectively (paper
RMOP2). So in doing the FAC top-up calculations, matching the bound levels for
each BPRM calculation is necessary but complicated by the fact that they have
different number of bound levels, especially energy regions where levels are
densely packed (see table 1) and the order of their spectroscopic designations
may be mismatched and needs to be shuffled (see figure 1).111It bears emphasis
that the opacities per se are independent of any spectroscopic labels;
however, they are necessary for processing the bound-bound and bound-free
radiative atomic transitions, and for comparing with other data sources.
Photoionization cross sections of 6 levels of Fe xvii are plotted in figure 1
for 60CC-BPRM and 276CC-BPRM, and we find distinct difference in level 24 and
26\. Similar issue arises in figure 1 of Fe xviii . Even though these levels
have similar energy, they may have distinctive configurations (see section
3.1), which is the reason why we should redo the identification for different
CC-BPRM calculations. After being switched, these levels show good agreement
(see figure 1).

Table 1: Selected packed levels of Fe xvii ($J=4,\pi=0$) and Fe xviii ($J=5/2,\pi=0$) (Note: the energy is $z$-scaled, and in unit of $10^{-2}Ry$) | level index | 60CC-BPRM | 218CC-BPRM
---|---|---|---
Fe xvii | 23 | -1.242945 | -1.239702
| 24 | -1.239540 | -1.238049
| 25 | -1.237855 | -1.236506
| 26 | -1.236421 | -1.235733
| 27 | -1.235081 | -1.235227
| 28 | -1.234742 | -1.234723
| level index | 200CC-BPRM | 276CC-BPRM
Fe xviii | 31 | -3.996445 | -4.004678
| 32 | -3.993010 | -4.003468
| 33 | -3.989362 | -4.000058

Figure 1: Photoionization cross section of 6 closely packed levels for Fe xvii
and 3 levels for Fe xviii before and after being switched. 60CC-BPRM (red),
218CC-BPRM (black); 200CC-BPRM (red), 276CC-BPRM (black)

##  3 Complementarity between BPRM and RDW: the top-up procedure

The Opacity Project [12] employed a small (e + ion) wavefunction expansion
including outer open Shell configurations in the close-coupling approximation
and $R$-Matrix method in LS-coupling to calculate non-relativistic
photoionization cross sections in the low energy region, and adopted the
Kramer approximation (”tail”) to fit and extend to higher energies afterwards.
In previous calculations, [13] replaced the Kramer tails with the RDW results
including the contribution from inner-shell processes. Later opacity tables
were updated by also including inner-shell transitions [15]. In this section,
we describe the procedure employed to compare and complement BPRM data with
RDW data from FAC. That requires careful matching between BPRM and FAC cross
sections for all bound levels, and detailed bound-bound and bound-free top-up
calculations. We also discuss the effect of configuration interaction on
photoionization cross sections.

###  3.1 Matching

In BPRM calculations, bound levels with continuum orbitals $\ell\leq 9$ and
effective quantum number $\nu\leq 10.1$ are formed by coupling the $n=2$ core
states with the continuum $\nu$ and $\ell$ of the outer electron. So in FAC we
set the bound configurations as a permutation of the $n=2$ core configurations
and an outer electron with principle quantum number $n\leq 10$. With the same
n-complex configuration interaction included, atomic structure is solved and
sorted by total angular momentum $J$ and parity $\pi$, and ordered in energy,
we find excellent agreement in the energy between BPRM and FAC values.

In calculating photoionization cross sections we include the whole $n$-complex
of core configurations for CI (hereafter CI) purpose, but only the transitions
to core configurations that are included in BPRM calculations. To delineate
photoionization cross sections at the edges of energy grid, the energy mesh is
created in such a way that within any two adjacent thresholds 10 points are
uniformly assigned. The partial photoionization cross section is the computed
in the default 6 energy grids, and interpolated/extrapolated in our mesh, and
summed to give total cross sections for each bound level. To investigate the
effect of CI, two sets of RDW calculation are carried out. Both sets only
allow the same-$n$-complex configuration interaction for bound configurations,
but for the core configurations one of them only allows the same-$n$-complex
CI and the other allows different $n$-complexes. We mix all the core
configurations together. In RDW calculations, the photoionization cross
section is related to the dipole operator matrix
$<\psi_{i}|\mathbb{D}|\psi_{f}>$. The $|\psi_{i}>$ involves the electron in
the bound state to the continuum and all other electrons, and must stay the
same if only same-$n$-complex configuration interaction is allowed. It can be
different if different-$n$-complex CI is considered [7].

To match bound state levels from BPRM and RDW, it is necessary to compare
cross sections to ensure the correctness of matching. We plot BPRM and RDW
photoionization cross sections in the order of energy for each $J$, $\pi$
symmetry pair, and a level is matched when the energy and the photoionization
cross section agree reasonably well (here the background of the BPRM data and
RDW are compared). Photoionization cross sections of majority of bound levels
show excellent consistency at the first attempt (see figure 1 for Fe xvii and
Fe xviii . The $LS$-term notation ( $S$ and $L$) can not be determined from
FAC output for all levels, so only configuration and total angular momentum
$J$ are given). However, when several levels are almost degenerate (see table
2), distinctive differences may yet occur in cross sections. Such levels need
to be switched till good matching is achieved (See figure 3 for Fe xvii and Fe
xviii The process is justified since level identification of near-degenerate
BPRM levels may not exactly correspond in energy from different atomic
structure codes such as FAC, since spectroscopic designations depend on CI
included and coupling schemes employed for the (e + ion) system222All BPRM
photoionization cross sections include a small region below the lowest
ionization threshold for each level [12], where no RDW data are shown..

In table 2, we can see the energy levels computed in BPRM and RDW agree quite
well. For Fe xvii , levels 13 and 14 , and levels 15 and 16 lie very close to
each other, and in figure 3, levels 13 and 16 achieve good agreement, while
levels 14 and 15 don’t. So we switch the order of levels 14 and 15 in RDW
calculation and recompare with good agreement. Thus levels 13 - 16 in BPRM
calculation are matched with those in RDW calculation. The same procedure is
applied to levels 65 and 66 of Fe xviii in table 2, and the result is show in
figure 3.

In figures 2 and 3, we show two sets of RDW calculation as described above,
and study the effect of configuration interaction on photoionization cross
sections, with the upper panel of figure 2 as an example. The dominant
configuration of the bound state after being matched with RDW is
$2s^{2}2p^{5}4d$, so with only the same-$n$-complex CI of core configurations
considered, the transitions can only happen to core configurations
$2s^{2}2p^{5}$, $2s2p^{5}4\ell$, $2s^{2}2p^{4}4\ell$ and $2p^{6}4\ell$, where
$\ell=s,p,d$, while with different-$n$-complex CI of core configurations
additional contribution can be from all other core configurations. From the
upper panel of figure 2 we can see that the same-$n$-complex configuration
interaction gives reasonably good background, though with some big gaps, while
different-$n$-complex CI fills up the big gap and improves the background
significantly. Similar phenomenon can be found in the rest of the figures 2
and 3, and there are still some gaps remaining after different-$n$-complex CI
is allowed333In figure 2 and 3, the oscillation in the background of the BPRM
data can be eliminated with a larger number of continuum basis functions [13].

Table 2: Selected levels of Fe xvii ($J=3,\pi=1$) and Fe xviii ($J=1/2,\pi=1$) to be matched (Note: the energy is $z$-scaled, and in unit of $10^{-2}Ry$) | level index | BPRM | RDW
---|---|---|---
Fe xvii | 12 | -3.670657 | -3.67960
| 13 | -2.873930 | -2.84319
| 14 | -2.868775 | -2.84238
| 15 | -2.842915 | -2.83761
| 16 | -2.835625 | -2.83555
| 17 | -2.774853 | -2.77998
| level index | BPRM | RDW
Fe xviii | 64 | -1.275887 | -1.27703
| 65 | -1.234865 | -1.24175
| 66 | -1.226084 | -1.23724
| 67 | -1.136909 | -1.14691

Figure 2: Most of the levels are matched at the first attempt with excellent
consistency in photoionization cross section. The configuration is attached
with each level. BPRM (black), RDW(blue and red). “Same-n CI” means only same-
n-complex CI is considered for core configurations, and “different-n CI”
refers to both same-n- and different-n-complex CI is considered.

Figure 3: Multiple attempts are needed to ensure the correct matching when the
levels are found with discrepancy in the photoionization cross section. In ??
and ?? the discrepancy is shown in upper panel and the final matching is in
the lower one. Find the configuration attached for each level. BPRM (black),
RDW(blue and red). “Same-n CI” refers to only same-n-complex CI is considered
for core configurations, and “different-n CI” refers to both same-n- and
different-n-complex CI are considered.

###  3.2 Bound-free data

As BPRM calculations are carried out in the lower part of the whole energy
range, they include low-$n\ell$ core configurations with $n\leq 4$. We use the
RDW method to extend it to higher regions up to 500 $Ry$ in photoelectron
energy, and to include higher-$n\ell$ core configurations $n=5,6$. The
following part of this section gives detailed description of these aspects.

####  3.2.1 High energy cross sections

As shown in figure 2, the RDW data can be matched almost perfectly to
background BPRM cross sections. However, we also find there are cases where
they don’t match well in the right region of energy (see figure 4). In the top
panels different-$n$ CI introduces many transitions, but they are not strong
enough to raise to the background of BPRM. In the middle panel of figure 4,
different-$n$ CI introduces many edges at positions where the background of
BPRM jumps, and raises the background higher than BPRM. While in the middle
panel of figure 4, around $105Ry$, compared with same-$n$ CI, different-n CI
moves the background up on the left side, and down on the right side, i.e.
converging to the background of BPRM. As close-coupling approximation treats
CI more completely and accurately, we multiply the RDW data in the higher
energy region by a factor which is the ratio of BPRM value and RDW value at
the last point of BPRM calculation for each level (see figure 5), to account
for the discrepancy between BPRM and RDW. In figure 5, the distribution of the
factors applied in the higher region is very alike for Fe xvii and Fe xviii ,
and there are around $60\%$ of the bound levels lying around ratio = 1. Among
the high-ratio cases, some are caused by the oscillation of the background of
BPRM, which is due to the small number of continuum basis functions used in
the wavefunction expansion [13] (see the bottom panels of figure 4.

The energy mesh used in the region is created in such a way that 10 points are
uniformly assigned between any adjacent ionization thresholds due to the other
core configurations (see section 3.2.2).

Figure 4: Different-n CI improves the background significantly, but there is
still very large discrepancy in the right region of energy for some levels.
BPRM (black), RDW(blue and red). “Same-$n$ CI” refers to only same-n-complex
configuration interaction is considered for core configurations, and
“different-$n$ CI” refers to both same-$n$\- and different-$n$-complex
configuration interaction are considered. Figure 5: The distribution of the
factors multiplied to RDW data in the higher energy region for Fe xvii and Fe
xviii . “width” is the width of the bins.

####  3.2.2 Highly excited core configurations

Using RDW with different-n-complex CI, we calculate photoionization cross
section due to other core configurations up to $n=6$ that are not included in
the BPRM calculation. To top-up 218CC BPRM for Fe xvii , we include core
configurations $2s^{2}2p^{4}4f$, $2s2p^{5}4f$, $2p^{6}4\ell^{\prime}$,
$2s^{S}2p^{P}5\ell^{\prime\prime}$ and
$2s^{S}2p^{P}6\ell^{\prime\prime\prime}$, where
$\ell^{\prime},\ell^{\prime\prime},\ell^{\prime\prime\prime}$ are all possible
subshells in the corresponding shell, and $s$, $p$ are any possible non-
negative integers satisfying $S+P=6$. To top-up 276CC BPRM for Fe xviii , we
include core configurations $2p^{5}3\ell^{\prime}$, $2s^{2}2p^{3}4f$,
$2s2p^{4}4\ell^{\prime}$, $2p^{5}4\ell^{\prime\prime}$,
$2s^{S}2p^{P}5\ell^{\prime\prime\prime}$ and
$2s^{S}2p^{P}6\ell^{\prime\prime\prime\prime}$, where
$\ell^{\prime},\ell^{\prime\prime},\ell^{\prime\prime\prime}$ and
$\ell^{\prime\prime\prime\prime}$ are all possible subshells in the
corresponding shell, and $S$, $P$ are any possible non-negative integers
satisfying $S+P=5$. The energy mesh is same as the one used in BPRM
calculation merged with the one in the high energy region as described in
section LABEL:sec:sectiontail.

As shown in figure 6, the BPRM data is merged with the scaled RDW tail, and
the contribute from other core configurations varies from negligible to
noticeable.

Figure 6: The photoionization cross section of the same four levels as in
figure 2 are extended to higher energy region and the contribution from other
core configurations with different-n-complex CI is added. Blue: BPRM
calculation; green: RDW data multiplied by a ratio; red: total photoionization
cross section.

####  3.2.3 Highly excited bound levels

In RDW, we consider all the bound state levels with $n\leq 10$, so we collect
all such levels that are not included in BPRM calculation, and calculate the
photoionization cross section due to all core configurations, i.e. the core
configurations included in the BPRM calculation and the other ones displayed
in section 3.2.2, with different-n-complex CI.

###  3.3 Bound-bound data

To top-up the bound-bound oscillator strength, we divide it into two parts.
One is from bound states to pure bound states. i.e. between negative energy
levels. We calculate all such possible transitions, but only collect the ones
that are not calculated in BPRM calculations. The other part is from bound
states to quasi-bound states, i.e. from negative energy bound levels to
positive energy doubly excited states in the continuum. BPRM calculations
treat direct photoionization and autoionization as a single unified quantum-
mechanical process, as in section 3.2 that discusses direct photoionization is
done. To simulate autoionizing resonances, we calculate the oscillator
strengths from bound to doubly excited states, among all pairs of negative-
positive energies. We consider transitions that excite an electron from
$L$-shell to a higher one, forming a doubly excited configuration that can not
be formed by combining a core configuration used in BPRM calculations with
another electron. Take Fe xviii for an example, we consider transitions from
$2s^{S}2p^{P}3\ell$ to only $2p^{5}3\ell^{\prime}n\ell^{\prime\prime}$, where
$S$, $P$ are any possible non-negative integers satisfying $S+P=6$, and
$\ell$, $\ell^{\prime}$, $\ell^{\prime\prime}$ can be any sub-shell in the
corresponding shell, and $n=3-6$. Since $2s^{2}2p^{3}$ and $2s2p^{4}$ are
included in 276CC BPRM calculation,
$2s^{2}2p^{3}3\ell^{\prime}n\ell^{\prime\prime}$ and
$2s2p^{4}3\ell^{\prime}n\ell^{\prime\prime}$ are considered naturally. Thus
they are excluded in the top-up calculations.

The number of quasibound positive energy levels in the continuum is far larger
than bound negative energy levels. For Fe xvii we have 587 bound levels as
opposed to $\sim$72,000 positive energy levels included as top-up. For Fe
xviii we obtain 1,154 bound levels vs. $\sim$175,000 quasibound levels. All
possible oscillator strengths among these large number of levels are computed
and considered in opacities calculations444We also note that oscillator
strength data for transitions among quasibound positive energy levels are
employed for free-free contribution to plasma broadening of autoionizing
resonances, discussed in paper RMOP3.

##  4 Conclusion

In order to investigate the effect of convergence and completeness of RMOP
data for opacities calculations, complete sets of relativistic distorted wave
calculations are carried out for Fe xvii and Fe xviii to compare with and
topup the 218CC and 276CC BPRM calculations, respectively. Bound state levels
are matched between BPRM and RDW calculation by comparing quantum numbers $J$,
$\pi$, energy, and cross sections. With such level correspondence, BPRM
photoionization cross sections in the higher energy region are extended by
scaled RDW data, and contribution from other core configurations up to $n=6$
is added, to examine the effect of convergence over and above the $n\leq 4$
BPRM data. Higher bound state levels are also included with photoionization
cross sections due to all core configurations up to $n=6$. Oscillator strength
data corresponding to the additional levels are also topped-up, with
contribution from bound-bound and bound-quasibound transitions.

The effects of CI on photoionization cross sections are discussed, including
same-$n$-complex and different-$n$-complexes, showing its significant role in
reproducing the background of BPRM cross sections using RDW. However, the
extensive resonance structures that dominate BPRM photoionization cross
sections throughout the energy range considered can not be compared owing to
their absence in the RDW data. Nevertheless, the RDW method may provide useful
checks on completeness and convergence of CC-BPRM results.

We have extensively studied the correspondence and complementarity between
BPRM and RDW results with a view to ascertain possible impact on opacities.
However, the local-thermodynamic-equilibrium (LTE) Mihalas-Hummer-Däppen
equation-of-state valid in stellar interiors yields extremely small occupation
probabilities and level populations for the high energy and high (e + ion)
spin-angular momenta states $nSLJ$ (discussed in paper P1), implying that the
actual effect on opacities would be small. Indeed, preliminary opacities
calculations indicate that Rosseland Mean Opacities are enhanced by only a few
percent $<$5% (results to be reported elsewhere).

##  5 Acknowledgments

One of the authors (ZL) would like to thank Dr. Ming Feng Gu for helpful
advice in using FAC and thank the following colleagues for helping run the
BPRM calculation with their resources in ASC Unity cluster at the Ohio State
University (names in alphabetic order): Jiaxin wu, Keng Yuan Meng, Max
Westphal, Xiankun Li, Yonas Getachew and Zhefu yu. This research was supported
by a teaching assistantship from the Dept. of Physics and the Dept. of
Astronomy of Ohio State University (OSU) and by the US National Science
Foundation and Dept. of Energy. Computations were carried out at the Ohio
Supercomputer Center, the OSU Dept. of Astronomy and the ASC Unity cluster of
OSU.

## References

## References

  * [1] Pradhan A K 2018 ASP Conf. Ser. 515 7
  * [2] Bailey J Nagayama T and Loisel G P et al. 2015 Nature 517 56
  * [3] Nahar S N Pradhan A K Chen G X and Eissner W 2011 Phys. Rev. A 83 053417
  * [4] Nahar S N and Pradhan A K Phys. Rev. Lett. 2016 116 235003
  * [5] Nahar S N and Pradhan A K Phys. Rev. Lett. 2016 117 249502
  * [6] Iglesias C and Hansen S 2017 Astrophys. J. 835 5
  * [7] Gu, M. F. 2008, Can. J. Phys., 86, 675;
  * [8] Seaton M J 1985 J. Phys. B 18 2111
  * [9] Nahar S N and Pradhan A K 2000 Phys. Scr. 61 675
  * [10] Nahar S N, Delayaye F Pradhan A K et al. 2000 Astron. Astrophys. Suppl. Ser. 144 141
  * [11] Eissner W, Jones M and Nussbaumer H 1974 Comput. Phys. Commun. 8 270
  * [12] The Opacity Project Team, The Opacity Project Vol.1 1995; Ibid. Vol. 2 1997, IOP Publishing Bristol
  * [13] Zhang H 1998 Phys. Rev. A 57 2640
  * [14] Zhang H Nahar S and Pradhan A 1999 J. Phys. B 32 1459
  * [15] Badnell N R and Seaton M J 2003 J. Phys. B 36, 4367 private communication
  * [16] Pradhan A K and Zhang H L 1997 J. Phys. B 30 L571