# Deep Ensemble for Rotorcraft Attitude Prediction

Hikmat Khan
Dept. of Electrical and
Computer Engineering
Rowan University
Glassboro, New Jersey, USA
<EMAIL_ADDRESS>&Nidhal C. Bouaynaya
Dept. of Electrical and
Computer Engineering
Rowan University
Glassboro, New Jersey, USA
<EMAIL_ADDRESS>&Ghulam Rasool
Dept. of Machine Learning
Moffitt Cancer Center
Tampa, Florida, USA
<EMAIL_ADDRESS>
&Tyler Travis
Federal Aviation Administration
Atlantic City, New Jersey,
United States
&Lacey Thompson
Federal Aviation Administration
Atlantic City, New Jersey,
United States
&Charles C. Johnson
Research Engineer, Flight Test Engineer, and Program Lead,
Federal Aviation Administration
Egg Harbor Township,New Jersey,

###### Abstract

Historically, the rotorcraft community has experienced a higher fatal accident
rate than other aviation segments, including commercial and general aviation.
To date, traditional methods applied to reduce incident rates have not proven
hugely successful for the rotorcraft community. Recent advancements in
artificial intelligence (AI) and the application of these technologies in
different areas of our lives are both intriguing and encouraging. When
developed appropriately for the aviation domain, AI techniques may provide an
opportunity to help design systems that can address rotorcraft safety
challenges. Our recent work demonstrated that AI algorithms could use video
data from onboard cameras and correctly identify different flight parameters
from cockpit gauges, e.g., indicated airspeed. These AI-based techniques
provide a potentially cost-effective solution, especially for small helicopter
operators, to record the flight state information and perform post-flight
analyses. We also showed that carefully designed and trained AI systems can
accurately predict rotorcraft attitude (i.e., pitch and yaw) from outside
scenes (images or video data). Ordinary off-the-shelf video cameras were
installed inside the rotorcraft cockpit to record the outside scene, including
the horizon. The AI algorithm was able to correctly identify rotorcraft
attitude at an accuracy in the range of 80%. In this work, we combined five
different onboard camera viewpoints to improve attitude prediction accuracy to
94%. Our current approach, which is referred to as ensembled prediction,
significantly increased the reliability in the predicted attitude (i.e., pitch
and yaw). For example, in some camera views, the horizon may be obstructed or
not visible. The proposed ensemble method can combine visual details recorded
from other cameras and predict the attitude with high reliability. In our
setup, the five onboard camera views included pilot windshield, co-pilot
windshield, pilot Electronic Flight Instrument System (EFIS) display, co-pilot
EFIS display, and the attitude indicator gauge. Using video data from each
camera view, we trained a variety of convolutional neural networks (CNNs),
which achieved prediction accuracy in the range of 79% to 90%. We subsequently
ensembled the learned knowledge from all CNNs and achieved an ensembled
accuracy of 93.3%. Our efforts could potentially provide a cost-effective
means to supplement traditional Flight Data Recorders (FDR), a technology that
to date has been challenging to incorporate into the fleets of most rotorcraft
operators due to cost and resource constraints. Such cost-effective solutions
can gradually increase the rotorcraft community’s participation in various
safety programs, enhancing safety and opening up helicopter flight data
monitoring (HFDM) to historically underrepresented segments of the vertical
flight community.

##  1 Introduction

As the premier agency for promoting and ensuring aviation safety, the Federal
Aviation Administration (FAA) continually strives to improve safety. The FAA
recognizes the importance of participating in Helicopter Flight Data
Monitoring (HFDM) programs and encourages their increased utilization to
improve flight safety and operational efficiency. Indeed, rotorcraft safety
was one of the agency’s top ten most wanted list of safety improvements in
2017-2018 and continues to be a high priority still in 2021. Organizations
including the FAA, National Transportation Safety Board (NTSB), and the United
States Helicopter Safety Team (USHST) are strong proponents of flight data
recorders (FDRs). These organizations and other industry partners are working
together to promote helicopter flight data monitoring (HFDM) programs as one
possible mitigation strategy to reduce the rotorcraft fatal accident rate.
However, despite all of these efforts by various safety organizations,
barriers to widespread implementation of FDRs and adaptation of HFDM still
exist. These include, but are not limited to, the technical skills required to
certify, install, and maintain an FDR, skilled resources who can perform HFDM
analyses, and the costs associated with the acquisition, certification, and
installation of the FDR. Traditional FDRs require a Supplemental Type
Certificate (STC) or Field Approval (FA) to install and operate under the
Rotorcraft Flight Manual (RFM). On average, the initial acquisition cost of an
FDR can range from $5,000 - $50,000. Given a range of factors, rotorcraft, in
general, have a lower participation rate in FDM programs than other forms of
aviation, including commercial fixed-wing or part 121 air carriers.

(a)

(b)

(c)

(d)

(e)

Figure 1: A set of representative images from five different cameras mounted
inside the helicopter cockpit are presented. In the cases of (a) and (b), we
used the whole video frame as an input for CNNs. While in the case of (c),
(d), and (e), the recorded image was cropped. The input to the CNN included
the areas inside the highlighted green rectangles. (best viewed in color)

(a)

(b)

(c)

(d)

(e)

(f)

Figure 2: Representative images from different camera views. These images show
the prediction of attitude from one gauge can be challenging, and an ensemble
approach may benefit due to the availability of different types of data
related to the attitude of the rotorcraft.(best viewed in color)

Inexpensive and off-the-shelf video cameras mounted inside the cockpit may
offer a potential alternative to traditional FDRs. Even small helicopter
operators often have access to or have the financial means to purchase one or
more off-the-shelf video cameras. These cameras can potentially record all the
data that traditional FDRs record. Moreover, onboard cameras may provide
supplementary data that may not be available in some FDRs. The recorded video
data from onboard cameras can be used for various analyses and inquiries. The
examples including estimation of flight parameters from instrument panel
gauges, flight replay during post-accident investigations, estimation of
rotorcraft attitude, and any other visual information that can be extracted
from video data.

In this work, we recorded video data from five different onboard cameras for
the accurate prediction of rotorcraft attitude. The cameras included the pilot
windshield, co-pilot windshield, pilot Electronic Flight Instrument System
(EFIS) display, co-pilot EFIS display, and the attitude indicator gauge.
Figure 1 shows representative images from all onboard cameras. We used in-
flight videos from these cameras to build training datasets for our AI models.
We used the video datasets to train a variety of convolutional neural networks
(CNNs). Later, we combined the knowledge learned by all CNNs using the
ensemble approach, which improved the attitude prediction accuracy to 93.3%.
Our proposed ensemble approach improves attitude prediction accuracy
especially when the horizon curve is obstructed or not visible due to bad
weather or other conditions. Our results from this work and previous
publications support the viability of an inexpensive onboard camera-based
solution for flight parameters’ estimation and attitude prediction. Such
cameras-based solutions do not require any special modification to the
helicopter’s avionics, communications, or display systems and are more
suitable for legacy helicopters. Importantly, the solution’s cost-effective
nature will encourage the rotorcraft community to participate in these
voluntary safety programs.

The paper is organized as follows. The Related Work section presents a brief
description of the current AI-based approaches proposed to increase rotorcraft
safety. In Methodology section, we explain the data acquisition methodology
and describe our experimental setup. Results section presents experimental
results along with a discussion on our results. Finally, we conclude the
current research in Conclusion section.

##  2 Related Work

Various AI models based on deep neural networks have obtained above human-
level performance on many different tasks. Examples in the computer vision
domain include image classification, object detection, and image segmentation
[1, 2, 3]. Computer vision tasks are primarily approached using the well-known
type of artificial neural networks, i.e., convolutional neural network or CNN.
A great deal of research has recently focused on developing various kinds of
CNNs, including EfficientNet, VGG-16, VGG-19, ResNet, Inception, and Xception
[4, 5, 6, 7]. These CNNs are also referred to as deep neural networks as they
are built using tens or hundreds of layers of artificial neurons. CNNs have
demonstrated the ability to learn complicated features directly from the image
or video data in an increasingly complex hierarchy.

Deep neural networks and CNNs are being proposed to tackle various challenging
tasks in the aviation community [8, 9, 10, 11]. Khan _et al._ showed that CNNs
could infer different flight parameters from video data recorded during
flights [8]. The authors showed that their trained models could accurately
estimate various flight parameters, including airspeed and engine torque,
directly from the instrument panel videos, with the core purpose to facilitate
the post-flight analysis.

Alligier _et al._ used machine learning techniques to improve airspeed
estimation during aircraft climbing [10]. In another work, the same authors
used machine learning to estimate the mass of ground-based aircraft during
climb prediction [11]. Kenneth _et al._ applied a natural language processing
(NLP) technique of structural topic modeling to the Aviation Safety Reporting
System (ASRS) corpus [12]. The authors identified subjects, patterns, and
areas that needed further analysis in the ASRS corpus [12]. Gianazza _et al._
trained a variety of machine learning algorithms to predict the workload of
air traffic controllers [13]. The closest research work to ours was performed
by Shin _et al._ [14]. The author proposed a conventional computer vision
strategy using hand-engineered features to predict rotorcraft attitude from
onboard cameras [14]. The authors manually analyzed each video frame and
marked the natural horizon line. Later, the classical machine learning
approach of DBSCAN Clustering was employed to estimate the roll and bank
angles [14]. However, the proposed method was computationally expensive and
lacked scalability for the inference of multiple parameters in large video
datasets.

(a)

(b)

Figure 3: The pitch and roll values for the whole duration of the flight are
presented in figure (a) and (b), respectively. The y-axis shows pitch values
in degrees and x-axis represents flight duration. (best viewed in color)

##  3 Methodology

We trained multiple different deep learning models for each dataset to predict
rotorcraft attitudes (i.e., pitch and yaw). Each dataset was build using the
video stream from a different camera. The five cameras installed in the
cockpit collected different details on the horizon curve from their own
viewpoints. For instance, the pilot and co-pilot windshield view directly
record the horizon curve, as seen through the windshield. The pilot and co-
pilot EFIS display and the attitude indicator gauge on the instrument penal
present a different visual representation of the same horizon curve. Figure 1
shows the sample images of each of the five onboard cameras .

###  3.1 Convolutional Neural Networks (CNNs)

CNNs are inspired by the human visual system and have obtained state-of-the-
art performance in various computer vision tasks [15, 16, 17]. CNNs can learn
features from input images in a hierarchical fashion taking advantage of the
spatial coherence in images, and do not require domain-specific knowledge,
i.e., feature engineering. The early layers of a CNN learn standard features,
such as edges or lines, while subsequent layers learn complex domain-dependant
features. The convolutional layers of a CNN perform convolution operations
between the input image and learnable parameters (also referred to as filters
or kernels). A CNN may have multiple layers of convolutional kernels depending
upon the complexity of the task to be learned. The features extracted by a
convolutional layer are passed through a nonlinear function, such as rectified
linear unit (ReLU). The nonlinear functions are also referred to as activation
functions or simply activations. The activation functions are generally
followed by a max-pooling layer, which reduces the dimensionality of the
input. At the end of the last convolutional layer, the extracted features are
flattened and densely connected to the next layer, referred to as the fully-
connected layer. Finally, a softmax function is used to produce class scores
or probabilities.

####  3.1.1 Weight Initialization:

In deep neural networks, appropriate weights (or parameters) initialization
can reduce the convergence time and computational cost. In our experiments, we
initialized our weights with ImageNet [6].

####  3.1.2 Activation Functions:

The nonlinear activation functions are generally introduced into every layer
of a neural network. Some commonly used activation functions include sigmoid,
hyperbolic tangent (tanh), ReLU, Leaky ReLU (LeakyReLU), and parametric ReLU
[18]. We used ReLU activation function in all of our experiments.

####  3.1.3 Max-Pooling Layers:

Pooling is used to reduce the dimensionality of the input features by sub-
sampling. Pooling operation makes the CNN invariant to small intensity and
illumination changes as well as translations. The commonly used pooling
operations include max-pooling and average-pooling [18]. The max-pooling
operation select features with the maximum value in the pooling region. The
average pooling calculates the average of the features in the pooling region.
We used the max-pooling operation in all experiments.

####  3.1.4 Regularization:

A common problem with deep neural networks is overfitting, i.e., lack of
generalization to the unseen data. Several regularization schemes have been
proposed to avoid overfitting, i.e., $L_{1}$-Regularization,
$L_{2}$-Regularization, batch normalization, and dropout [18]. Batch
normalization is performed by calculating parameterized mean and standard
deviation of the input and processed data at various layers of the CNN during
the training phase. Dropout is a commonly used and effective technique for
regularization [19] where a randomly selected set of neurons are tuned off
(forced to zero) at each forward pass. Dropout forced each neuron to learn and
contribute independently to the overall output of the CNN. We used both batch
normalization and dropout techniques in our experiments.

####  3.1.5 Fully Connected Layers and the Softmax Function:

The output of the convolutional layers is called the feature map. The feature
map is flattened (i.e., vectorized) and densely connected to the next layer, a
fully connected layer. A CNN may have multiple fully connected layers. The
output of the last fully connected layer is referred to as class scores and
becomes an input to the softmax function. Softmax function nonlinearly
normalizes input class scores to numbers between 0 and 1. The highest score is
considered the classification decision, i.e., the predicted class label by the
CNN.

####  3.1.6 Loss Function:

The loss function measures the error between the predicted class label by the
CNN and the ground truth label. The training process of a neural network is
aimed to minimize the loss function by optimally adjusting weights or
parameters. We used the categorical cross-entropy loss function in all our
experiments.

###  3.2 Data Acquisition:

We mounted five onboard cameras with various viewpoints inside the cockpit to
record the instrument panel and the horizon inside an S-76 helicopter. The
pilot and co-pilot perspectives of the horizon were captured with pilot and
co-pilot windshield cameras mounted above the pilot and co-pilot. Similarly,
the pilot and co-pilot perspectives of the EFIS displays were recorded using
the two separate cameras. The fifth camera continuously recorded the different
gauges on the instrument panel. The representative images can be seen in
Figure 1. Table 1 presents the total duration of the available videos for each
camera view. The rotorcraft was equipped with an onboard Helicopter Flight
Data Recorder (HFDR). Both HFDR and cameras (i.e., each frame of flight video)
were timestamped using a time server. The timestamps were used to annotate
individual frames of flight videos with the corresponding HFDR recordings.

Table 1: Total flight duration for five different datasets, i.e., camera views. The flight duration is in the standard time format (i.e. hh:mm:ss). Camera veiw | Flight duration
---|---
Pilot Windshield | 44:37:10
Co-pilot Windshield | 55:38:34
Pilot EFIS | 23:42:35
Co-pilot EFIS | 34:09:50
Artificial Attitude Indicator | 15:20:00
Table 2: Definition of classes for attitude. We used a threshold $\alpha=3$ and defined 9 discrete classes. Abbreviations used: NU - nose up, ND - nose down, RP - roll positive; RN - roll negative, and L - level or steady-state. Class | Description | Pitch(P) | Roll(R)
---|---|---|---
0 | NU | P $>$ $\alpha$ |  $-\alpha\leq$ R $\leq+\alpha$
1 | ND | P $<-\alpha$ |  $-\alpha\leq$ R $\leq+\alpha$
2 | RR |  $-\alpha\leq$ P $\leq+\alpha$ | R $>\alpha$
3 | RL |  $-\alpha\leq$ P $\leq+\alpha$ | R $<-\alpha$
4 | NU & RP | P $>\alpha$ | R $>\alpha$
5 | NU & RN | P $>\alpha$ | R $<-\alpha$
6 | ND & RP | P $<-\alpha$ | R $>\alpha$
7 | ND & RN | P $<-\alpha$ | R $<-\alpha$
8 | L |  $-\alpha\leq$ P $\leq+\alpha$ |  $-\alpha\leq$ R $\leq+\alpha$

###  3.3 Classes Definition:

The HFDR recorded reading for the attitude (i.e., pitch and yaw) are real
numbers. Figure 3 present the pitch and roll attitude values for one of the
flights of S-76 helicopter, respectively. We define nine classes for the
attitude. The nine classes are: class 0 - nose down (ND), class 1 - nose up
(NU), class 2 - roll positive (RP), class 3 - roll negative (RN), class 4 - ND
and RP, class 5 - NU and RP, class 6 - ND and RN, class 7 - NU and RN, and
class 8 - level and steady-state (L). Table 2 presents definition of nine
classes. The $\alpha$ takes the user-defined values and defines the boundary
among nine classes. In our experiments we set the value as $\alpha=3$.

###  3.4 Experimental Setup:

We trained four CNNs for each video dataset. CNN architectures that we
considered for attitude prediction included EfficientNet, VGG16, VGG19,
ResNet50, and Xception and InceptionV3 [4, 20, 5, 6, 7, 21]. The models were
trained on the training set and then evaluated on a separate test set. In all
experiments, we initialized the models with ImageNet weights and fine-tuned
them to our dataset [22]. All experiments were performed using the Adam
optimizer with a consistent batch size of 256 [23]. The remaining parameters
of the Adam optimizer were initialized to default values, as discussed in
[23].

##  4 Results

Table 3: Average prediction accuracies of all models trained on five camera views. The last two rows present accuracies for two ensemble models. We used majority voting strategy for creating ensemble models. Model | Pilot | Co-pilot | Pilot | Co-pilot | Attitude | Ensemble
---|---|---|---|---|---|---
| windshield | windshield | EFIS | EFIS | Gauge | Performance
EfficientNetB0 | 86% | 83.8% | 86% | 86% | 76.67% | -
ResNet50 | 77.1% | 81.7% | 85% | - | - | -
VGG16 | 85% | 89.1% | 91.6% | 90% | 74.4% | -
VGG19 | 87% | 88% | 84% | - | 79.1% | -
InceptionV3 | - | - | - | 85.6% | - | -
Xception | - | - | - | 88.1% | 76.8% | -
Deep Ensemble (excluding attitude gauge models) | - | - | - | - | - | 92.5%
Deep Ensemble (including all 20 models) | - | - | - | - | - | 93.3%

In Table 3, we present average accuracy values for all models trained on five
different datasets. Each row of the table represents a different CNN, and each
column represents a dataset (i.e., camera view). The last column shows the
performance of two ensemble approaches. The ensemble approaches combine the
attitude predictions of all trained models (i.e., 20 models) on five camera
views using a majority voting strategy. The first ensemble approach did not
include models trained with the attitude gauge datasets. The second approach
included all the models.

As evident from Table 3, the proposed two ensemble approaches obtained higher
averaged accuracy (i.e., 92.5% and 93.3%) compared to other individual models
trained on different camera views. The improvement in the predictive
performance using the ensemble approach supports our hypothesis that combining
data from multiple cameras will improve prediction accuracy.

Figure 4 presents class-wise normalized confusion matrices of four models
trained on pilot windshield datasets. Figure 5 presents class-wise normalized
confusion matrices of four models trained on co-pilot windshield datasets. The
x-axis on for confusion matrices presents the predicted attitude class while
the y-axis is reserved for the true class. The diagonal values show the
correct classification rates for nine different attitude classes. The higher
numbers on the diagonal of a confusion matrix reflect a better model. Figure 6
present class-wise normalized confusion matrices of four models trained on
pilot EFIS display dataset. Figure 7 presents class-wise normalized confusion
matrices of four models trained on co-pilot EFIS display datasets. Figure 8
presents class-wise normalized confusion matrices of four models trained on
artificial attitude indicator gauge.

Figure 9 presents the normalized confusion matrices for both ensemble
approaches. First, we considered 16 models trained on pilot and co-pilot
windshield and pilot and co-pilot EFIS datasets in the first ensemble
approach. While in the second, we considered all 20 models trained on
datasets. In Figure 9, we use green and red rectangles to show the positive
(an increase in predicting accuracy) and negative (a decrease in the
predictive accuracy) effects of the class-wise prediction in ensemble
confusion matrices. We observed that increasing the number of models in the
ensemble improved the prediction accuracy. It is also evident (red and green
arrows) from Figure 9 that an increased number of models in the ensemble
result in an increase in prediction accuracy for most classes. It is important
to highlight that the ensemble-based approach obtained consistent results on
all nine attitude classes. Whereas the models trained on specific camera views
have comparatively lower classification accuracy values on classes 6 and 7.

##  5 Conclusion

We showed that ensemble AI models could reliably predict the rotorcraft
attitude using data from ordinary cameras mounted in the cockpit. Previously,
we only used a single camera view, which resulted in higher error rates and
low reliability. Our approach overcame several common problems, i.e., obscure
or invisible horizons during night flights, low visibility during
precipitation or during fog/low clouds. We proposed two ensemble techniques
that combine various trained CNNs models using a majority voting strategy. The
ensemble approaches combine the visual information of all five camera views
before making the final attitude predictions and obtain higher average class-
level attitude predictive accuracies. Our results establish the potential of
AI-based technologies for various rotorcraft safety applications.

Authors’ Contacts:

  * •

Hikmat<EMAIL_ADDRESS>
  * •

Dr. Ghulam Rasool<EMAIL_ADDRESS>
  * •

Dr. Nidhal C. Bouaynaya<EMAIL_ADDRESS>
  * •

Charles C. Johnson<EMAIL_ADDRESS>
##  6 Acknowledgments

This work was supported by the Federal Aviation Administration (FAA)
Cooperative Agreement Number 16-G-015 and NSF Awards OAC-2008690 and
DUE-1610911. Ghulam Rasool was also partly supported by NSF Award OAC-2008690.
This publication was in part supported by a subaward from Rutgers University,
Center for Advanced Infrastructure & Transportation, under Grant no.
69A3551847102 from the U.S. Department of Transportation, Office of the
Assistant Secretary for Research and Technology (OST-R).

(a)

(b)

(c)

(d)

Figure 4: Class-wise normalized confusion matrices for 4 CNNs (i.e. VGG16,
VGG19, EfficientNet, and ResNet50) used for attitude prediction using pilot
windshield camera view are presented.

(a)

(b)

(c)

(d)

Figure 5: Class-wise normalized confusion matrices for 4 CNNs (i.e.
EfficientNet, ResNet50, VGG16, and VGG19) used for attitude prediction using
co-pilot windshield camera view are presented.

(a)

(b)

(c)

(d)

Figure 6: Class-wise normalized confusion matrices for 4 CNN architectures
(i.e. EfficientNet, RestNet50, VGG16 and VGG19) used for attitude prediction
using pilot EFIS display are presented.

(a)

(b)

(c)

(d)

Figure 7: Class-wise normalized confusion matrices for 4 CNN architectures
(i.e. EfficientNet, Xception, VGG16, and InceptionV3) used for attitude
prediction using co-pilot EFIS display are presented.

(a)

(b)

(c)

(d)

Figure 8: Class-wise normalized confusion matrices for 4 CNN architectures
(i.e. EfficientNet, VGG16, VGG19, and Xception) used for attitude prediction
using pilot attitude indicator gauge are presented.

(a)

(b)

Figure 9: Class-wise normalized confusion matrices of ensemble attitude
prediction are presented. We used the majority voting strategy to combine
different model that are trained on five different camera views. (a) Ensemble
confusion matrix based on 16 models trained on pilot windshield, co-pilot
windshield, pilot EFIS and co-pilot EFIS views. (b) Ensemble confusion matrix
for all 20 models. The ensemble approach obtained better accuracy at the
individual class level and enhanced overall attitude predictive accuracy.

##  7 Authors Biography

Hikmat Khan is currently a PhD student at Rowan University. He is a research
fellow supporting the Federal Aviation Administration (FAA) via a research
grant/cooperative agreement by evaluating the feasibility of applying deep
learning approaches to increase safety within the rotorcraft industry. His
research interests include deep learning, continual learning, few-shot
learning and optimization.

Nidhal C. Bouaynaya received her Ph.D. degree in Electrical and Computer
Engineering and M.S. degree in Pure Mathematics from The University of
Illinois at Chicago, in 2007. From 2007-2013, she was an Assistant then
Associate Professor with the Department of Systems Engineering at the
University of Arkansas at Little Rock. In Fall 2013, she joined the Department
of Electrical and Computer Engineering at Rowan University, where she is
currently a Professor and the Associate Dean for Research and Graduate
Studies. Dr. Bouaynaya co-authored more than 100 refereed journal articles,
book chapters and conference proceedings. She won numerous Best Paper Awards,
the most recent was at the 2019 _IEEE International Workshop on Machine
Learning for Signal Processing_. She is also the winner of the Top algorithm
at the 2016 Multinomial Brain Tumor Segmentation Challenge (BRATS). Her
research interests are in Big Data Analytics, Machine Learning, Artificial
Intelligence and Mathematical Optimization. In 2017, she Co-founded and is
Chief Executive Officer (CEO) of MRIMATH, LLC, a start-up company that uses
artificial intelligence to improve patient oncology outcome and treatment
response.

Ghulam Rasool is an Assistant Professor of Electrical and Computer Engineering
at Rowan University. He received a BS. in Mechanical Engineering from the
National University of Sciences and Technology (NUST), Pakistan, in 2000, an
M.S. in Computer Engineering from the Center for Advanced Studies in
Engineering (CASE), Pakistan, in 2010, and the Ph.D. in Systems Engineering
from the University of Arkansas at Little Rock in 2014. He was a postdoctoral
fellow with the Rehabilitation Institute of Chicago and Northwestern
University from 2014 to 2016. He joined Rowan University as an adjunct
professor and later as a lecturer in the year 2018. Currently, he is the co-
director of the Rowan AI Lab. His current research focuses on machine
learning, artificial intelligence, data analytics, signal, image, and video
processing. His research is funded by National Science Foundation (NSF), U.S.
Department of Education, U.S. Department of Transportation (through the
University Transportation Center (UTC), Rutgers University), Federal Aviation
Administration (FAA), New Jersey Health Foundation (NJHF), and Lockheed
Martin, Inc. His recent work on Bayesian machine learning won the Best Student
Award at the 2019 IEEE Machine Learning for Signal Processing Workshop.

Charles C. Johnson works as a research engineer, program manager, and
technical expert for the Aviation Research Division at the FAA William J.
Hughes Technical Center in Atlantic City, NJ. During his 10+ year-career with
the FAA, he has led several rotorcraft and unmanned aircraft systems (UAS)
research and development simulation/flight test activities that seek to
improve aviation safety. Cliff is qualified on the ScanEagle UAS platform. He
holds a Bachelor’s degree in Mechanical Engineering from Rowan University. He
also holds a private pilot’s license (single engine land/fixed-wing) and is
pursuing his instrument, commercial, and helicopter add-on ratings.

Tyler Travis serves as a Research Analyst at the Federal Aviation
Administration’s (FAA) William J. Hughes Technical Center in Atlantic City,
NJ. She supports several research and development activities that seek to
improve aviation safety. During her 5+ years at the FAA Technical Center,
Tyler has worked on several UAS and Rotorcraft Human-In-The Loop simulations
involving new technologies and procedural changes impacting the National
Airspace System. Prior to joining the FAA, Tyler completed her B.S. in
Business Administration with a concentration in Management Information Systems
at Drexel University in September 2012.

Lacey Thompson works as an Operations Research Analyst for the Unmanned
Aircraft Systems (UAS) Engineering Branch at the Federal Aviation
Administration’s (FAA) William J. Hughes Technical Center in Atlantic City,
NJ. During her 7.5 years with the FAA, Lacey has managed several UAS Human-In-
the-Loop (HITL) simulations. In addition, she serves as an Aviation Science,
Technology, and Mathematics (AvSTEM) Ambassador, creating the curriculum for
the first ever module on UAS for the Aviation Monthly Mentoring Program. Lacey
holds a Bachelor’s degree in Physics from Northwestern State University of
Louisiana and a Master’s degree in Aeronautics from Embry-Riddle Aeronautical
University Worldwide.

## References

  * [1] B. Stefan, W. Roland, N. Lutz-P, and R. Mauricio.  A survey of mri-based medical image analysis for brain tumor studies.  Journal of Physics in Medicine and Biology, 58(13):R97–129, 2013\.
  * [2] S. Suraj, K. S. Ravi, M. Reddy, P. Nikita, S. S. K. Srinivas, and B. Venkatesh.  A taxonomy of deep convolutional neural nets for computer vision.  Journal of Front. Robotics and AI, 2, 2016.
  * [3] Q. Z. Zhao, P. Zheng, S. Xu, and W. Xindong.  Object detection with deep learning: A review.  IEEE Transactions on Neural Networks and Learning Systems, 2019\.
  * [4] M. Tan and Q. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 6105–6114, June 15 2019.
  * [5] S. Karen and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  In International Conference on Learning Representations Conference Proceedings, San Diego, May 2015.
  * [6] H. Kaiming, Z. Xiangyu, R. Shaoqing, and S. Jian.  Deep residual learning for image recognition.  In IEEE Conference on Computer Vision and Pattern Recognition Conference Proceedings, pages 770–778, Las Vegas, Nevada, June 2016.
  * [7] F. Chollet.  Xception: Deep learning with depthwise separable convolutions.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 1800–1807, July, 2017.
  * [8] K. Hikmat, C. B. Nidhal, R. Ghulam, and C. J. Charles.  Rotorcraft flight information inference from cockpit videos using deep learning.  In Presented at American Helicopter Society 75th Annual Forum, Philadelphia, Pennsylvania, USA, May 2019.
  * [9] K. Hikmat, C. B. Nidhal, R. Ghulam, C. J. Charles, T. Tyler, and T. Lacey.  Explainable ai: Rotorcraft attitude prediction.  In Presented at the VFS International 76th Annual Forum & Technology Display, Virginia Beach, Virginia, October 6–8 2020.
  * [10] R. Alligier, D. Gianazza, and N. Durand.  Machine learning applied to airspeed prediction during climb.  In Proceedings of the 11th USA/Europe Seminar on Air Traffic Management Research and Development, volume 16, pages 3138–3149, June 2015.
  * [11] R. Alligier, D. Gianazza, and N. Durand.  Machine learning and mass estimation methods for ground-based aircraft climb prediction.  IEEE Transactions on Intelligent Transportation Systems, 16(6):3138–3149, 2015.
  * [12] D. K. Kenneth.  Topics and trends in incident reports.  In Proceedings of the 12th USA/Europe Seminar on Air Traffic Management Research and Development, June 2017.
  * [13] D. Gianazza.  Learning air traffic controller workload from past sector operations.  In Proceedings of the 12th USA/Europe Seminar on Air Traffic Management Research and Development, June 2017.
  * [14] S. Shin and I. Hwang.  Helicopter cockpit video data analysis for attitude estimation using dbscan clustering.  In AIAA Infotech at Aerospace, 2016.
  * [15] J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu.  Squeeze-and-excitation networks.  In IEEE Conference on Computer Vision and Pattern Recognition Conference Proceedings, pages 7132–7141, Salt Lake City, Utah, June 2018.
  * [16] L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4):834–848, 2018.
  * [17] J. Redmon and A. Farhadi.  Yolo9000: better, faster, stronger.  In IEEE Conference on Computer Vision and Pattern Recognition Conference Proceedings, pages 834–848, Honolulu, HI, January 2017.
  * [18] I. Goodfellow, Y. Bengio, and A. Courville.  Deep Learning.  MIT Press, Cambridge, 2016.
  * [19] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  Journal of Machine Learning Research, 15:1929–1958, 2014.
  * [20] J. Hu, L. Shen, and G. Sun.  Squeeze-and-excitation networks.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 7132–7141, 2018.
  * [21] K. Chen, P. Jiangmiao, W. Jiaqi, Y. Xiong, X. Li, S. Shuyang, et al.  Hybrid task cascade for instance segmentation.  In IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, 2019.
  * [22] Keras Documentation.  Applications - Available models for image classification with weights trained on ImageNet.  https://keras.io/applications/.
  * [23] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  In International Conference on Learning Representations, 2015.


# SODA: a TypeScript/JavaScript Library for Visualizing Biological Sequence
Annotation

Jack W. Roddy
Department of Computer Science
University of Montana
Missoula, MT
<EMAIL_ADDRESS>
George T. Lesica
Department of Computer Science
University of Montana
Missoula, MT
<EMAIL_ADDRESS>
Travis J. Wheeler
Department of Computer Science
University of Montana
Missoula, MT
<EMAIL_ADDRESS>

###### Abstract

We present SODA, a lightweight and open-source visualization library for
biological sequence annotations that enables straightforward development of
flexible, dynamic, and interactive web graphics. SODA is implemented in
TypeScript and can be used as a library within TypeScript and JavaScript.

##  1 Introduction

Annotation of biological sequences and the visualization of these annotations
is central to molecular biology. It is common to represent features (e.g.
genes) annotated in the context of a reference sequence (e.g. a genome) using
a combination of glyphs (rectangles, lines, arrows, etc.) and plots (line
plots, bar plots, heatmaps, etc.). Countless software tools have been
developed that produce such visualizations, with modern applications typically
developed using web technologies. Most prominent among these are genome
browsers such as JBrowse [1], and hosted services such as Ensembl [2] and the
UCSC genome browser [3]. Recent innovations include the grammar-based
JavaScript toolkit for visualizing genomics data, Gosling [4]. Libraries also
exist for visualizing annotation specific to protein sequences, such as
ProtVista [5] and its successor Nightingale.

These frameworks provide extensive visualization functionality, but may not
meet the needs that arise in the development of custom visualization systems.
Custom visualizations may require visual features or data types not supported
by out-of-the-box solutions, may involve interactive response to user actions
that exceed the options provided by these frameworks, and may demand
integration of multiple visual facets into a coherent whole. Furthermore, the
use of standardized browsers may impose hefty dependencies and inclusion of
unwanted GUI components. To avoid these compromises, developers of custom
visualization systems often turn to core web browser technologies (HTML, CSS,
SVG, Canvas, WebGL), or libraries that lightly abstract over those core
technologies (D3[6], PixiJS[7]).

These developers would be well served by a front-end software library that
enables extensive control over the form and function of annotation
visualization elements useful for representing genome annotations, while
further abstracting over most of the complexity of low-level libraries. To our
knowledge, there exists no lightweight web-based library that meets this need.
Here, we present SODA (Soda Obediently Draws Annotations), an open-source
TypeScript/JavaScript library that aims to facilitate the development of
flexible, dynamic, and interactive annotation visualization systems. SODA is
designed to enable augmentation with lower-level visualization libraries where
needed, and it can also be used to generate simple one-time-use figures. The
SODA source code is available at https://github.com/sodaviz/soda, the library
is packaged through NPM (@sodaviz/soda), and the website https://sodaviz.org/
provides documentation and examples of visuals implemented with SODA.

##  2 MATERIALS AND METHODS

###  2.1 Design

SODA is a lightweight, purely front-end library implemented in TypeScript.
TypeScript is compiled to JavaScript, which means SODA can be used from either
TypeScript or JavaScript. Under the hood, SODA figures are created with
Scalable Vector Graphics (SVG). SODA is an object oriented library with few
third-party dependencies. SODA objects are designed with extension in mind,
and they expose functions that allow for a great deal of programmatically-
driven change at runtime. Currently, SODA is focused on the presentation of
annotations in linear context, with light support for representation in a
circular context (e.g. for bacterial genomes).

SODA is a toolkit with which a developer may _create_ a visualization system,
rather than a visualization system in and of itself. The main consequence of
this philosophy is that SODA does not produce visualizations using templating
mechanisms commonly found in genome browser tools and the like. Instead, SODA
places fine-grained control in the hands of developers with a modest
complexity trade-off: developers must define simple data structures to house
their data, and functions that utilize SODA’s rendering API to produce a
visualization.

###  2.2 High-level description of developing with SODA

Development using SODA is largely focused on the configuration and management
of Chart objects, which are wrapper objects that control SVG-based viewports
in web pages. Creating a SODA visualization typically involves implementing
simple objects that describe the annotation data being visualized, designing a
data rendering payload that contains collections of those objects, and using
the SODA API to configure the visual representation of the payload and the
ways in which that visualization will respond to user inputs (e.g. clicking,
panning, zooming).

###  2.3 SODA’s treatment of data

SODA is designed around a minimal abstraction of annotation data, rather than
specific annotation data formats (e.g. BED, GFF3, GTF). For gene-like
annotations that describe an interval, this is a simple object composed of an
identifier string, a start coordinate, and an end coordinate. For annotations
that describe position specific information throughout an interval, SODA uses
the same object specification with an additional array/vector of position
specific values. All of the core SODA rendering features are designed to
produce a visual representation of any JavaScript objects that extend this
simple pattern. In practice, annotations are commonly augmented with auxiliary
data (e.g. alignment strand or score) which, during visualization, can be used
to modulate some aspect of the visual representation of the annotation (e.g.
color or opacity). With this in mind, SODA rendering features are also
designed such that it is easy for a developer to use additional object data
fields to control glyph styling parameters. Additionally, SODA provides
suitable object definitions for several common annotation data formats along
with accompanying parsing routines.

###  2.4 SODA’s rendering API

To render glyphs with SODA, Annotation objects are passed into rendering
functions along with a configuration that specifies the target Chart object
and styling parameters. For each style parameter, either a static value or a
callback function may be provided. The callback functions are evaluated for
each individual glyph, and the represented Annotation object and target Chart
are passed as arguments. Styling callback functions provide a simple mechanism
to control glyph style using annotation data and may be re-evaluated at
runtime to achieve visualization dynamics (e.g. zooming and transforming).

##  3 Results

Development of SODA was motivated by a need in our group to create genome-
oriented visualizations for which existing genome visualization tools either
were unable to provide desired functionality or were deemed unnecessarily
heavyweight. Here, we present several of these visualizations, with the goal
of demonstrating SODA’s flexibility in providing options for rendering and
interactivity. Working demonstrations for each example can be found at
https://sodaviz.org, and the underlying source code can be found at
https://github.com/sodaviz/. These examples present applications of SODA for
visualizing genome annotations, but it can be used to visualize protein
annotations as well.

###  3.1 Dfam visualization

Dfam [8] is an open access database of Transposable Elements (TEs). One
feature on the Dfam website allows users to view a representation of the set
of annotated TE instances in a relatively short region (up to 100,000
nucleotides) of a chromosome. In collaboration with the maintainers of Dfam,
we replaced the original implementation with a more feature-rich variant using
SODA (Fig. 1).

The bulk of the annotations underlying the Dfam visualization are the result
of comparing a genome to a database of known TE families. The resulting TE
annotations are supplemented with annotations for instances of simple tandem
repeats (repetitive sequence such as ‘atgatgatgatg’). In the linear genome
portion of the visualization (top of Fig. 1), each annotation record is
represented by a rectangle glyph that is colored according to the TE family
assigned to the position. There are three SODA components in the
visualization, stacked vertically in the following order:

  * •

TEs annotated on the forward strand of the chromosome

  * •

Simple tandem repeat annotations (black glyph on gray genome background)

  * •

TEs annotated on the reverse strand of the chromosome

Immediately following the SODA components are two additional visual
components, not built with SODA: (i) a legend describing the colors used in
the visualization, and (ii) a tabular description of each annotation rendered
above (the table is a fixture of the Dfam website, found at
https://www.dfam.org/search/annotations; it is not found in the example on
sodaviz.org). Within the SODA components, hovering over glyphs triggers a
highlight effect along with an informational tooltip popup. Additionally, this
figure demonstrates the way that SODA objects can interact with the
surrounding website based on SODA callback functions: when a glyph is clicked
by the user, the table highlights and scrolls to the row corresponding to the
clicked glyph.

###  3.2 RepeatMasker visualization

The UCSC genome browser [3] is a popular tool that houses visualization tracks
for dozens of kinds of genomic annotation. The tracks are independently
configured and many are submitted by external groups; they can differ greatly
in visual complexity. The RepeatMasker Viz. track is a highly nuanced and
information-dense track that represents the annotation of TEs and other
repetitive DNA features as labeled by the tool RepeatMasker [9]. Using SODA,
we developed a copy-cat implementation of the RepeatMasker Viz. track (Fig. 1)
to serve as a test-case with complex visual expression requirements.

The RepeatMasker Viz. track presents the same type of annotations as the Dfam
visualization described in the previous section, but includes additional
visual indicators to represent complex relationships that are not present in
the Dfam visualization. As in the Dfam visualization, annotated TEs and simple
repeats are represented with rectangle glyphs. The rectangles in the
RepeatMasker plot, however, differ in three ways:

  * •

The outline, rather than the entire glyph, is colored according to the TE
family color scheme.

  * •

The interior of the rectangles are shaded in grayscale to indicate the
inferred biological age (determined by the quality of the sequence alignment
that defines the annotation) of the annotated feature. Younger features appear
darker, while older features appear lighter.

  * •

The interior of each rectangle is textured with a repeating chevron pattern to
indicate the chromosome strand on which the feature is found.

TE annotations in a genome are often fragments of a full-length known TE
family. These fragments can arise from partial replication (at time of
insertion), partial deletion (some time after full-length insertion), or
insertion of a newer element into the middle of a previously inserted TE
(resulting in two fragments for the older element). Two indicators are added
to the RepeatMasker visualization to represent these events:

  * •

When an annotation is identified as a fragment, dashed horizontal lines are
rendered on either side to indicate portions of the consensus TE sequence that
are missing from the fragment.

  * •

When discontiguous fragments are inferred to be the result of an older element
having been split by a newer insertion, they are joined by two angled lines
meeting at a point.

We refer to groups of joined fragments, dashed lines, and angled lines as a
compound glyph. A text label is placed immediately to the left of each
compound glyph. As the user zooms in and out, the labels automatically adjust
the level of displayed text detail, based on available space.

Figure 1: (I) depicts the Dfam-SODA visualization. At (I.A), (I.B), and (I.C)
are the forward strand TE annotations, simple repeat annotations, and reverse
strand TE annotations, respectively. At (I.D) is the repeat classification
color map, and at (I.E) is the table describing the annotations visualized –
neither (I.D) nor (I.E) are created with SODA. At (I.F), the green rectangle
has been hovered and clicked, causing the descriptive tooltip to appear and
the annotation’s corresponding row at (I.G) to be highlighted. (II) depicts
the RepeatMasker-SODA visualization. At (II.A) is a simple glyph representing
a full-length TE annotation. At (II.B) is a compound glyph representing a
joined group of TE fragment annotations. At (II.C), an image is shown of the
L1MC3 text label after the visualization is zoomed out – when the label has
less screen space, it renders a less detailed string.

###  3.3 PolyA visualization

Our group has developed a tool, called PolyA [10], that adjudicates between
competing alignment-based annotations by computing position-specific measures
of confidence, identifying a trace with maximal confidence, and recursively
splicing/stitching inserted elements. For debugging and exploration purposes,
we found it helpful to develop a visualization application that provides
insight into the differences between annotation results with PolyA and an
alternative adjudication method (see Fig 2).

Because PolyA’s development was motivated by the goal of improving annotation
of TEs, the visualization places PolyA information in the context of
RepeatMasker’s adjudication results, as produced by its internal tool,
ProcessRepeats. Specifically, the PolyA-SODA visualization has four
components:

  * •

A component displaying annotations queried from a mirror of the UCSC
RepeatMasker database (these represent the results from the adjudication
method found in RepeatMasker, called ProcessRepeats).

  * •

A component displaying the PolyA adjudicated annotations for the same region.
The user can make a brush selection on this component, resulting in changes to
the following two components.

  * •

A component that displays the exact (magnified) contents of the brushed
interval in the above component.

  * •

A component that displays a heatmap of the confidence scores for competing
alignments in the same region. The sequence alignments from which the
confidence scores were calculated are overlaid on top of the heatmap for each
alignment.

This tool enables visual inspection of complicated genomic regions,
identifying differences between the two annotation adjudication methods, and
providing quick visual access to the sequence alignments and corresponding
confidence values computed within PolyA.

Figure 2: In the PolyA-SODA visualization, pre-existing TE annotations are
shown at (I) and PolyA-adjudicated annotations are shown at (II). At (A), a
brush selection has been made, and the selected interval is magnified and
displayed at (III). The alignments and confidence scores corresponding to the
selected interval are shown at (IV). (B) shows the results of hovering over
one of the triangle glyphs that represent alignment insertions relative to the
genome – the inserted sequence replaces the triangle while hovered

###  3.4 VIBES visualization

We are currently developing a tool for the identification of phage sequences
integrated into bacterial genomes, called VIBES (Viral Integrations in
Bacterial gEnomeS). To support data exploration by VIBES users, we have
developed a SODA-based application consisting of several interconnected
components.

The VIBES pipeline accepts a batch of bacterial genomes and a batch of phage
genomes and identifies regions in the bacterial genomes that appear to be the
result of phage integration. One output from VIBES is a representation, for
each bacterial genome, of the locations of inferred (possibly partial) phage
integrations – there can be several per bacterial genome. Another VIBES output
is, for each phage genome, a nucleotide-precision representation of the
frequency with which each phage nucleotide is part of an observed integration
across the batch of genomes. VIBES also identifies all Swiss-Prot and Pfam
annotations for each phage genome.

At the top of the VIBES visualization (Fig. 3) is a chart that represents a
linearized version of the target bacterial genome. The top row of the linear
chart displays annotations of phage integrations to the bacterial genome as
identified by VIBES, and the bottom row displays gene annotations. At the
lowest zoom level, the gene annotations are condensed into groups based on
proximity in the chromosome. As the zoom level increases, the gene annotations
are continuously re-rendered as more horizontal screen real estate enables
finer-grained separation and display.

Below the linear chart on the left is a circular representation of the
bacterial genome. Here, phage integrations are shown on the outer ring and a
condensed representation of gene annotations are on the inner ring. As the
linear chart is zoomed and panned, a brush is drawn on the circular chart that
highlights the region of the bacterial genome that is currently shown in the
linear view.

On both the linear and circular charts, a single phage integration is
highlighted blue, indicating that it is currently “selected”. The selected
annotation determines the contents of the occurrences plot, which is directly
to the right of the circular chart. The top of the occurrences plot displays
the position-specific integration frequencies of the phage responsible for the
current selected annotation. In the occurrences plot, the region of the phage
genome that was identified in the current bacterial genome is highlighted in
blue (a partial integration will have blue highlight over just the
corresponding fragment of the full length phage). The bottom of the
occurrences plot indicates positions of genes and protein domains along the
entirety of the phage genome. The selected annotation can be switched by
clicking on phage annotations on either the linear or circular chart, or by
pressing the arrow buttons that appear when hovering over the circular chart.

At the bottom of the visualization is a simple table that describes the gene
annotations across the selected phage genome. Clicking on an annotation in the
phage occurrences plot will cause the corresponding entry to be highlighted in
the table.

Figure 3: In the VIBES-SODA visualization, the linear bacteria chart is
displayed at (I). At (I.A) is a handful of light blue bacterial gene
annotations – each of these rectangles represents an individual gene. At (I.B)
is a handful of dark blue bacterial gene annotations – each of these
rectangles represents a group of genes aggregated to simplify display. At
(I.C) is an orange phage integration annotation – it is highlighted in cyan to
indicate that it is the current selected annotation. At (II) is the circular
bacteria chart. At (II.A) we see the same selected phage annotation and the
gray shading that indicates the region that is currently rendered in the
linear chart. At (III) is the occurrences plot, which displays position
specific integration counts across all bacterial genomes for the selected
phage integration annotation. At (III.A) is the interval on the plot that
indicates the portion of the phage genome that was identified in the selected
phage annotation. At (III.B) is the collection of viral gene annotations for
the selected phage. At (IV) is the table that describes the viral gene
annotations shown in (III.B). One of the viral gene annotations has been
hovered and clicked, causing the tooltip at (III.B) and the row at (IV.A) to
be highlighted.

##  4 DISCUSSION

Here, we have introduced SODA, an open-source TypeScript/JavaScript library
that enables the development of web-based biological sequence annotation
visualizations. SODA is primarily intended for creating visualization systems,
but it also has utility in the generation of one-off figures. In particular,
we imagine SODA as a replacement not for existing domain-specific tools, but
for lower-level, general purpose web visualization technologies. SODA’s
feature set is broadly capable, but, in our experience, it is often useful to
augment a SODA-based visualization by integration with components built with
other web technologies. We have designed SODA to be easy to use, broadly
expressive, and extensible; we would be grateful for community engagement, in
the form of identifying feature gaps and contributing to SODA’s source code to
extend expression options.

##  5 ACKNOWLEDGEMENTS

We are grateful to Jeb Rosen and Robert Hubley for the extensive and valuable
feedback they provided during development of the library, and Daniel Olson and
Thomas Colligan for helpful comments on drafts of the manuscript. We thank
Audrey Shingleton and Conner Copeland for their aid in development of example
applications of SODA (PolyA and VIBES, respectively). This work was supported
by NIH grants R01-GM132600 and U24-HG010136, and institutional support from
the University of Montana.

####  5.0.1 Conflict of interest statement.

None declared.

## References

  * [1] Robert Buels, Eric Yao, Colin M Diesh, Richard D Hayes, Monica Munoz-Torres, Gregg Helt, David M Goodstein, Christine G Elsik, Suzanna E Lewis, Lincoln Stein, et al.  JBrowse: a dynamic web platform for genome visualization and analysis.  Genome biology, 17(1):1–12, 2016.
  * [2] Kevin L Howe, Premanand Achuthan, James Allen, Jamie Allen, Jorge Alvarez-Jarreta, M Ridwan Amode, Irina M Armean, Andrey G Azov, Ruth Bennett, Jyothish Bhai, et al.  Ensembl 2021.  Nucleic acids research, 49(D1):D884–D891, 2021.
  * [3] Jairo Navarro Gonzalez, Ann S Zweig, Matthew L Speir, Daniel Schmelter, Kate R Rosenbloom, Brian J Raney, Conner C Powell, Luis R Nassar, Nathan D Maulding, Christopher M Lee, et al.  The UCSC genome browser database: 2021 update.  Nucleic acids research, 49(D1):D1046–D1057, 2021.
  * [4] Sehi L’Yi, Qianwen Wang, Fritz Lekschas, and Nils Gehlenborg.  Gosling: A grammar-based toolkit for scalable and interactive genomics data visualization.  IEEE Transactions on Visualization and Computer Graphics, 28(1):140–150, 2021.
  * [5] Xavier Watkins, Leyla J Garcia, Sangya Pundir, Maria J Martin, and UniProt Consortium.  ProtVista: visualization of protein sequence annotations.  Bioinformatics, 33(13):2040–2041, 03 2017.
  * [6] Michael Bostock, Vadim Ogievetsky, and Jeffrey Heer.  D3 data-driven documents.  IEEE Transactions on Visualization and Computer Graphics, 17(12):2301–2309, dec 2011.
  * [7] Pixijs.  https://pixijs.com/.
  * [8] Jessica Storer, Robert Hubley, Jeb Rosen, Travis J Wheeler, and Arian F Smit.  The Dfam community resource of transposable element families, sequence models, and genome annotations.  Mobile DNA, 12(1):1–14, 2021.
  * [9] Arian FA Smit.  Repeat-Masker Open-3.0.  http://www. repeatmasker. org, 2004.
  * [10] Kaitlin M Carey, Robert Hubley, George T Lesica, Daniel Olson, Jack W Roddy, Jeb Rosen, Audrey Shingleton, Arian F Smit, and Travis J Wheeler.  PolyA: a tool for adjudicating competing annotations of biological sequences.  bioRxiv, 2021.