[affiliation=]ZhongMeng [affiliation=]ZelinWu [affiliation=]RohitPrabhavalkar
[affiliation=]CalPeyser [affiliation=]WeiranWang [affiliation=]NanxinChen
[affiliation=]
Tara N.Sainath [affiliation=]BhuvanaRamabhadran

# Text Injection for Neural Contextual Biasing

###### Abstract

Neural contextual biasing effectively improves automatic speech recognition
(ASR) for crucial phrases within a speaker's context, particularly those that
are infrequent in the training data. This work proposes contextual text
injection (CTI) to enhance contextual ASR. CTI leverages not only the paired
speech-text data, but also a much larger corpus of unpaired text to optimize
the ASR model and its neural biasing component. Unpaired text is converted
into speech-like representations and used to guide the model's attention
towards relevant bias phrases. Moreover, we introduce a contextual text-
injected (CTI) minimum word error rate (MWER) training, which minimizes the
expected WER caused by contextual biasing when unpaired text is injected into
the model. Experiments show that CTI with 100 billion text sentences can
achieve up to 43.3% relative WER reduction from a strong neural biasing model.
CTI-MWER provides a further relative improvement of 23.5%.

###### keywords:

ASR, text injection, contextual biasing

##  1 Introduction

End-to-end (E2E) models excel in automatic speech recognition (ASR) by
directly mapping speech signals to word sequences [1, 2, 3, 4, 5]. Despite
training on large audio-transcript datasets, E2E ASR on contextually relevant
words that appear infrequently in the training data remains challenging [6].
Such phrases include user's contact names, songs, applications, and current
locations, etc. Contextual biasing offers an effective solution to mitigate
this training-inference mismatch, improving ASR performance on contextual
long-tail words.

Traditional contextual biasing approaches train an external language model
(LM) [7] or construct a contextual finite state transducer (FST) [8, 9, 10,
11], and combine their scores with the ASR model during inference. These
methods separately optimizes the ASR model and biasing component, requiring
extensive tuning for the interpolation weights. E2E neural biasing [12, 13,
14, 15] addresses this by directly integrating the biasing component into the
ASR model, enabling joint optimization towards a single ASR objective. These
methods employ attention mechanism to associate bias phrases with the decoder
or the acoustic encoder of the ASR model, generating a context vector that
adapts the ASR model towards the biasing context. Neural biasing shows
improved ASR performance compared to the FST-based approach. However, these
methods face a scalability challenge: as the number of biasing phrases
increases, both the computational latency and word error rate (WER) tend to
rise. To address this, neural associate memory (NAM) [16, 17] utilizes a two-
pass hierarchical biasing scheme: the 1st pass identifies the top-K bias
phrases that are most likely to occur in the speech signal via correlation,
and the 2nd pass restricts the acoustic encoder's attention into only the
selected K phrases. Deferred NAM [18] further improves the 1st pass top-K
search by introducing a retrieval loss, and has achieved the state-of-the-art
(SOTA) performance on multi-context biasing sets.

However, training neural biasing models requires paired audio-transcript data,
which is limited by the cost of human transcription. Moreover, bias phrases
sampled from such transcripts are neither rare phrases for the model nor
contextually relevant to the test utterances. To address this, we propose a
novel _contextual text injection (CTI)_. CTI transforms large-scale _unpaired
text_ into speech-like representations through an ASR acoustic encoder, and
associates them with bias phrases extracted from the _unpaired text_ via an
attention mechanism. This enhances the model's ability to identify the most
relevant bias phrases and bias its predictions towards them. CTI offers
significant advantages: 1) unpaired text is readily accessible, and is far
more plentiful than audio-transcript pairs; 2) unpaired text contains a wealth
of rare phrases that are infrequent or even absent in audio-transcript pairs.
Further, we propose contextual text-injected (CTI) minimum word error rate
(MWER) training, where unpaired text, after undergoing contextual neural
biasing, is used to generate N-best hypotheses. Minimizing an extra text-based
MWER loss computed from such hypotheses effectively reduces ASR errors
introduced by contextual biasing. Experimented with 100 billion (B) text
sentences, CTI reduces WER by up to 43.3% relatively compared to a SOTA
deferred NAM on voice search multi-context biasing sets. CTI MWER leads to an
additional 23.5% relative WER reduction from CTI.

##  2 Related Work

###  2.1 Text Injection

Various studies have explored incorporating unpaired text into E2E model
training. One approach is to fine-tune the decoder of an E2E model with
unpaired text to minimize an additional cross-entropy LM loss [19, 20, 21, 22,
23, 24, 25]. By zeroing out the encoder output, the decoder acts as an
internal LM [26, 27, 28]. However, the decoder's limited capability restricts
the effectiveness of these methods. Text injection [29, 30, 31, 32, 33, 34,
35] addresses this by directly feeding unpaired text into the ASR acoustic
encoder, where the text is mapped to a shared latent space with speech, either
explicitly or implicitly. The latent embeddings are then used to train the E2E
models.

Text injection is initially applied to contextual biasing in [36]. Building
upon JOIST [33], [36] injects contextually relevant, unpaired text directly
into an ASR model that does _not_ contain any biasing component. This improves
performance when combined with an FST-based context model during inference.
Our CTI method differs by applying JOIST-like text injection to a neural
biasing model. This allows joint optimization of ASR model and its biasing
component using unpaired text towards text-based ASR and retrieval objectives.
Consequently, CTI integrates the advantages of both powerful neural biasing
and text injection, while eliminating the need for FST or any other external
biasing component during inference.

###  2.2 Neural Contextual Biasing with Deferred NAM

Upon removing red lines and red boxes, Fig. 1 depicts a speech transducer [37,
26] with a neural biaser, deferred NAM [18]. The transducer consists of an
acoustic encoder, a decoder and a joint network. The acoustic encoder
transforms input speech features $\mathbf{X}$ into acoustic embeddings. To
recognize named entities in speaker's context, each speech utterance
$\mathbf{X}$ is assigned $N$ bias phrases $Z=\\{z_{1},...,z_{N}\\}$. $z_{n}$
may or may not exist in $\mathbf{Y}$. Deferred NAM performs 2-pass
hierarchical biasing at layer $l$ of the acoustic encoder. In Pass 1, we
retrieve the K most relevant biasing phrases based on the correlations between
acoustic embeddings $\mathbf{H}_{l}$ and bias phrase embeddings $\mathbf{P}$,
followed by a max-pooling over $T$:

$\displaystyle\mathbf{a}=\text{MaxPool}_{t=1}^{T}\left(\mathbf{h}_{l,t}{\mathbf{P}}^{\top}/\sqrt{d}\right),$
(1)

where $\mathbf{a}\in\mathbb{R}^{N+1}$,
$\mathbf{H}_{l}=\\{\mathbf{h}_{l,1},\ldots,\mathbf{h}_{l,T}\\},\mathbf{h}_{l,t}\in\mathbb{R}^{d}$
is the sequence of acoustic embeddings at the biasing layer $l$.
$\mathbf{P}\in\mathbb{R}^{(N+1)\times d}$ is generated by feeding $Z$ through
a phrase encoder and appending the output with a $d$-dim NO_BIAS embedding.
111For simplicity, we assume all encoder embeddings in this work have the same
dimension $d$ and omit projection matrices. We select top-K phrases from $Z$
corresponding to the K largest values of the correlation vector $\mathbf{a}$,
and tokenize them into word-pieces
$Z^{\text{w}}=\\{z^{\text{w}}_{1},\ldots,z^{\text{w}}_{M}\\}$, $M\geq K$.

Then in Pass 2, we compute the cross-attention between acoustic embeddings
$\mathbf{H}_{l}$ (as queries) and the word-piece embeddings
$\mathbf{W}\in\mathbb{R}^{M\times d}$ of $Z^{\text{w}}$ (as keys and values
222For better biasing performance, values are typically obtained by left-
shifting keys by one token and appending a zero-embedding [15].):

$\displaystyle\mathbf{C}=\text{CrossAttention}(Q=\mathbf{H}_{l};K,V=\mathbf{W}),$
(2)

where $\mathbf{C}\in\mathbb{R}^{T\times d}$ contains the $T$ context vectors
generated by the cross-attention $\text{Softmax}(QK^{\top}/\sqrt{d})V$.

The _biased_ acoustic embeddings are obtained by
$\mathbf{H}^{\text{bias}}_{l}=\mathbf{H}_{l}+\lambda_{c}\mathbf{C},\lambda_{c}\in\mathbb{R}$,
which are then propagated through the upper acoustic encoder layers. The
decoder takes in previous labels to generate the current label embedding. The
joint network combines the top-layer acoustic and label embeddings via a feed-
forward network, and estimates posteriors of the ground-truth transcript
$\mathbf{Y}$. A transducer loss
$\mathcal{L}^{\text{ASR}}(\mathcal{D})=-\sum_{(\mathbf{X},\mathbf{Y},Z)\in\mathcal{D}}\log
P(\mathbf{Y}|\mathbf{X},Z)$ is calculated on supervised audio-transcript data
$\mathcal{D}$. In addition, to ensure the K phrases selected in Pass 1 are
most relevant to $\mathbf{X}$, we minimize a retrieval loss
$\mathcal{L}^{\text{retr}}$ computed by a cross-entropy between $\mathbf{a}$
and the ground-truth label $r$. $r$ indicates the longest bias phrase present
in $\mathbf{Y}$, or NO_BIAS if none exists. Likewise, we calculate a word-
piece retrieval loss based on the correlations between $\mathbf{H}_{l}$ and
$\mathbf{W}$, and add it to $\mathcal{L}^{\text{retr}}(\mathcal{D})$. We
jointly minimize the transducer loss and the retrieval loss
$\lambda_{a}\mathcal{L}^{\text{ASR}}(\mathcal{D})+\lambda_{d}\mathcal{L}^{\text{retr}}(\mathcal{D})$,
$\lambda_{a},\lambda_{d}\in\mathbb{R}$ are loss weights.

##  3 Contextual Text Injection

In this work, we propose _contextual text injection (CTI)_ , in which a large
corpus of unpaired text $\mathbf{Y}^{\text{text}}\in\mathcal{D}^{\text{text}}$
together with its contextual bias phrases $Z^{\text{text}}$ are injected into
an ASR model and its neural biasing component to enhance its biasing
capability.

Figure 1: Contextual text injection (CTI) for a speech transducer with neural
biasing. Blue and red arrows show the forward propagation paths for paired
audio-transcript data and unpaired text, respectively.

As shown in Fig. 1, we first tokenize unpaired text into phoneme sequences and
then replicate each token a fixed or random number of times to get an
upsampled sequence
$\mathbf{Y}^{\text{phn}}=\\{y^{\text{phn}}_{1},\ldots,y^{\text{phn}}_{I}\\}$.
$\mathbf{Y}^{\text{phn}}$ is then fed into a phoneme encoder to generate
phoneme embeddings $\mathbf{E}^{\text{phn}}\in\mathbb{R}^{I\times d}$. To
utilize unpaired text for neural biasing, the phoneme embeddings need to be
injected to an acoustic encoder layer $l^{\text{text}}$ that precedes the
biasing layer $l$, i.e., $l^{\text{text}}<l$. Therefore, different from JOIST
[33], the masking in CTI has to be performed after the biasing layer instead
of before the phoneme encoder. We propagate $\mathbf{E}^{\text{phn}}$ through
layer $l^{\text{text}}$ to layer $l$ of the acoustic encoder to generate
_text-based_ acoustic embeddings
$\mathbf{H}^{\text{text}}_{l}\in\mathbb{R}^{I\times d}$.

Each text sentence $\mathbf{Y}^{\text{text}}$ is associated with $N$ bias
phrases
$Z^{\text{text}}=\\{z^{\text{text}}_{1},\ldots,z^{\text{text}}_{N}\\}$,
sampled from sentences within $\mathbf{Y}^{\text{text}}$'s training mini-
batch. $z^{\text{text}}_{n}$ may or may not exist in
$\mathbf{Y}^{\text{text}}$. $\mathbf{H}^{\text{text}}_{l}$ and
$Z^{\text{text}}$ undergo the two-pass contextual biasing procedure similar to
Section 2.2. In Pass 1, correlations between $\mathbf{H}^{\text{text}}_{l}$
and $Z^{\text{text}}$ are used to select the top-K bias phrases for each text
sentence and to compute a retrieval loss. In Pass 2, cross-attention between
$\mathbf{H}^{\text{text}}_{l}$ and only the selected bias phrases is computed
to generate _text-based_ biased acoustic embeddings $\mathbf{H}^{\text{text,
bias}}_{l}$.

Specifically, in Pass 1, we generate phrase embeddings
$\mathbf{P}^{\text{text}}\in\mathbb{R}^{(N+1)\times d}$ by feeding
$Z^{\text{text}}$ into the phrase encoder and appending a $d$-dim NO_BIAS
embedding to the output. We then calculate
$\mathbf{a}^{\text{text}}\in\mathbb{R}^{N+1}$ by max-pooling the correlations
between $\mathbf{H}^{\text{text}}_{l}$ and $\mathbf{P}^{\text{text}}$ over
$I$:

$\displaystyle\mathbf{a}^{\text{text}}=\text{MaxPool}_{i=1}^{I}\left(\mathbf{h}^{\text{text}}_{l,i}{\mathbf{P}^{\text{text}}}^{\top}/\sqrt{d}\right),$
(3)

where $\mathbf{a}^{\text{text}}$ signifies the correlations between
$\mathbf{H}^{\text{text}}_{l}$ and each bias phrase (and NO_BIAS) in $Z$. From
set $Z^{\text{text}}$, we select the top-K bias phrases that have the K
largest correlation values in $\mathbf{a}^{\text{text}}$, and tokenize them
into word-pieces $Z^{\text{text, w}}=\\{z^{\text{text,
w}}_{1},\ldots,z^{\text{text, w}}_{M}\\}$, $M\geq K$. In Pass 2,
$\mathbf{H}^{\text{text}}_{l}$ first cross-attends into the word-piece
embeddings $\mathbf{W}^{\text{text}}\in\mathbb{R}^{M\times d}$ of
$Z^{\text{text, w}}$ to compute text-based context vectors
$\mathbf{C}^{\text{text}}\in\mathbb{R}^{I\times d}$. $\mathbf{H}^{\text{text,
bias}}_{l}\in\mathbb{R}^{I\times d}$ is then obtained by adding
$\mathbf{C}^{\text{text}}$ to $\mathbf{H}^{\text{text}}_{l}$:

$\displaystyle\mathbf{C}^{\text{text}}=\text{CrossAttention}(Q=\mathbf{H}^{\text{text}}_{l};K,V=\mathbf{W}^{\text{text}}),$
(4) $\displaystyle\mathbf{H}^{\text{text,
bias}}_{l}=\mathbf{H}^{\text{text}}_{l}+\lambda_{c}\mathbf{C}^{\text{text}},$
(5)

where $\lambda_{c}\in\mathbb{R}$ is the biasing strength.

To make the task sufficiently difficult for the model with text input, we mask
a portion of text-based biased acoustic embeddings $\mathbf{H}^{\text{text,
bias}}_{l^{\text{mask}}}$ at layer $l^{\text{mask}}$ after the biasing layer
$l$, i.e., $l^{\text{mask}}>l$, by following [38]. The masked embeddings are
then propagated through the rest of the transducer, contributing to the
calculation of a text-based transducer loss:

$\displaystyle\mathcal{L}^{\text{ASR}}(\mathcal{D}^{\text{text}})=-\log
P(\mathbf{Y}^{\text{text}}|\text{Mask}(\mathbf{H}^{\text{text,
bias}}_{l^{\text{mask}}}),Z^{\text{text}}).$ (6)

To accurately identify the K most relevant bias phrases from the unpaired
text, we minimize a retrieval loss
$\mathcal{L}^{\text{retr}}(\mathcal{D}^{\text{text}})$ computed as a cross-
entropy between the correlation $\mathbf{a}^{\text{text}}$ and ground-truth
label $r^{\text{text}}$. $r^{\text{text}}$ is assigned the longest bias phrase
from $Z^{\text{text}}$ that appears in $\mathbf{Y}^{\text{text}}$. Otherwise,
$r^{\text{text}}$ is assigned NO_BIAS if none of the bias phrases are present
in $\mathbf{Y}^{\text{text}}$. Similarly, a word-piece retrieval loss based on
the same unpaired text is computed and then added to
$\mathcal{L}^{\text{retr}}(\mathcal{D}^{\text{text}})$.

The speech transducer, phoneme encoder, and neural biaser are jointly trained
from scratch to minimize two ASR losses and two retrieval losses using audio-
transcript paired data $\mathcal{D}$ and unpaired text
$\mathcal{D}^{\text{text}}$:

$\displaystyle\mathcal{L}^{\text{CTI}}=$
$\displaystyle\lambda_{a}\mathcal{L}^{\text{ASR}}(\mathcal{D})+\lambda_{d}\mathcal{L}^{\text{retr}}(\mathcal{D})$
$\displaystyle\quad+\lambda^{\text{text}}_{a}\mathcal{L}^{\text{ASR}}(\mathcal{D}^{\text{text}})+\lambda^{\text{text}}_{d}\mathcal{L}^{\text{retr}}(\mathcal{D}^{\text{text}}).$
(7)

where $\lambda^{\text{text}}_{a},\lambda^{\text{text}}_{d}\in\mathbb{R}$ are
weights for the text-based losses. With CTI, the model can learn from
significantly larger pool of unpaired text the ability to extract relevant
bias phrases and improve the accuracy of named entity recognition based on
them.

##  4 Contextual Text-Injected MWER

Conventional MWER training minimizes the expected number of word errors on
supervised training data [39, 40, 41], optionally including unpaired text
[33]. However, these methods reply on N-best hypotheses produced by a non-
contextualized ASR system, limiting its ability to correct errors caused by
contextual biasing. To overcome this, we propose _contextual text-injected
(CTI) MWER_ training. This approach generates N-best hypotheses by feeding
both audio-transcript pairs $\mathcal{D}$ and unpaired text
$\mathcal{D}^{\text{text}}$ through an ASR model with neural biasing, then
computes MWER losses based on these hypotheses.

With CTI MWER, we first generate the N-best hypotheses
$\\{\mathbf{Y}_{1},\ldots,\mathbf{Y}_{N}\\}$ by inputting speech $\mathbf{X}$
and the bias phrases $Z$ into the model and applying the two-pass neural
contextual biasing outlined in Section 2.2. This aligns the N-best generation
with the actual inference, empowering the MWER training to rectify errors
introduced by contextual biasing. The contextual MWER loss on supervised data
is computed as the expected number of word errors on these N-best hypotheses:

$\displaystyle\mathcal{L}^{\text{MWER}}(\mathcal{D})=$
$\displaystyle\sum_{n=1}^{N}\frac{P(\mathbf{Y}_{n}|\mathbf{X},Z)}{\sum_{n=1}^{N}P(\mathbf{Y}_{n}|\mathbf{X},Z)}R(\mathbf{Y}_{n},\mathbf{Y}),$
(8)

where $R(\mathbf{Y}_{n},\mathbf{Y})$ is the number of word errors in a
hypothesis $\mathbf{Y}_{n}$ compared to the ground-truth transcript
$\mathbf{Y}$.

Further, we inject unpaired text $\mathbf{Y}^{\text{text}}$ and bias phrases
$Z^{\text{text}}$ into the ASR and its neural biasing model to calculate an
text-based MWER loss:

$\displaystyle\mathcal{L}^{\text{MWER}}(\mathcal{D}^{\text{text}})=\sum_{n=1}^{N}\frac{P(\mathbf{Y}^{\text{text}}_{n}|\mathbf{H}^{\text{text,
bias}}_{l^{\text{mask}}})}{\sum_{n=1}^{N}P(\mathbf{Y}^{\text{text}}_{n}|\mathbf{H}^{\text{text,
bias}}_{l^{\text{mask}}})}R(\mathbf{Y}^{\text{text}}_{n},\mathbf{Y}^{\text{text}}),$

where $\\{\mathbf{Y}^{\text{text}}_{1},\ldots,\mathbf{Y}^{\text{text}}_{N}\\}$
are N-best hypotheses generated via the text-based two-pass neural biasing in
Section 3. Finally, we minimize the CTI-MWER loss across both $\mathcal{D}$
and $\mathcal{D}^{\text{text}}$:

$\displaystyle\mathcal{L}^{\text{CTI-
MWER}}=\lambda_{m}\mathcal{L}^{\text{MWER}}(\mathcal{D})+\lambda^{\text{text}}_{m}\mathcal{L}^{\text{MWER}}(\mathcal{D}^{\text{text}}),$
(9)

where $\lambda_{m},\lambda^{\text{text}}_{m}\in\mathbb{R}$ are loss weights.

To enhance the ASR performance in the _absence_ of bias phrases, we
selectively activate the biasing component during CTI MWER training. We do
this by feeding in empty bias phrases for a specific percentage of the
training data.

##  5 Experiments

###  5.1 Data

Our supervised training data comprises 520 million multi-domain English audio-
transcript pairs (490 thousand (K) hours) from voice search (VS) traffic as in
[42]. In addition, we leverage a much larger corpus of unpaired text for text
injection, exceeding the size of audio-transcript pairs by two orders of
magnitude. This corpus consists of 100B anonymized sentences drawn from Maps,
Google Play, Web, and YouTube domains as in [33].

We assess our models across various test sets. The head common-word test set
comprises roughly 12K anonymized and hand-transcribed VS utterances with an
average duration of 5.5 seconds. For rare-word evaluation, we utilize a multi-
context biasing corpus from [16, 18], consisting of three sets:

  * •

NO_PRE: 1.3K utterances matching prefix-less patterns from $APPS, $CONTACTS,
$SONGS categories (i.e., ACS).

  * •

PRE: 2.6K utterances matching prefixed patterns like ``open $APPS'', ``call
$CONTACTS'', ``play $SONGS''.

  * •

ANTI: 1K utterances from general voice assistant queries.

In all three sets, each utterance is assigned 150, 300, 600, 1500, 3000 ACS
bias phrases. The NO_PRE and PRE sets evaluate _in-context_ performance where
one bias phrase actually appears in the transcript truth, while the ANTI set
assesses _anti-context_ performance where only distractor bias phrases are
assigned.

###  5.2 Modeling Architecture

This work utilizes a non-streaming hybrid autoregressive transducer (HAT) [26]
for ASR. We extract 128-dim log Mel filterbanks from speech and subsample it
with two 2D-convolution layers, resulting a 512-dim feature every 40 ms. The
HAT acoustic encoder is based on the architecture of Google USM [43]. It
starts with 2 convolution layers, followed by 12 conformer layers [44]. Each
conformer layer uses a 8-head local self-attention with 512 dimensions and a
convolution kernel of size 10. Funnel pooling [45] reduces the computation
latency, resulting in a frame rate of 320ms at the encoder output [46, 42].
The HAT decoder is a 640-dim $|V|^{2}$ embedding [47]. The output vocabulary
consists of 4096 lowercase word-pieces. HAT has 857M parameters in total.

The neural biasing is applied at 4th encoder layer of HAT ($l=4$). The phrase
encoder is a 4-layer deep averaging network with 256 hidden units (788K
parameters). The word-piece encoder is a 1-layer conformer with 256 hidden
units (1M parameters). The projection layers in correlation and cross-
attention have 7.3M parameters. All biasing components are significantly
smaller than the main HAT. The bias strength $\lambda_{c}$ is set to 1.0
during training and 0.6 during inference.

###  5.3 Text injection for Neural Contextual Biasing

We first train a baseline HAT with deferred NAM [18] using only the supervised
data, and show its results in Table 3. $\lambda_{a}=0.9,\lambda_{r}=0.1,K=32$
for all the experiments. Note that, for each biasing test set, the reported
WER is the average across 5 different scenarios where the model receives
varying number of bias entities.

Method |  VS | In-Context |  ANTI
---|---|---|---
NO_PRE |  PRE
Sup | 4.1 | 3.0 | 2.3 | 1.8
JOIST | 4.0 | 2.2 | 2.1 | 1.9
CTI | 3.9 | 1.7 | 1.7 | 1.6
Table 1: WERs (%) for HAT with neural contextual biasing.333Without contextual
biasing, HAT with JOIST and text-injected MWER achieves 4.0%, 22.1%, 10.5% and
1.6% WERs on VS, NO_PRE, PRE and ANTI, respectively. The models are trained
from scratch using supervised data (Sup), JOIST or contextual text injection
(CTI).

Then we perform JOIST training of HAT with deferred NAM by injecting unpaired
text _only_ to the acoustic encoder of HAT. No neural biasing (Eqs. (3) – (5))
is performed with unpaired text. The settings are consistent with those used
for JOIST on HAT: the unpaired text is tokenized into phoneme sequences. Each
token is repeated a random number of times. The random number is sampled from
a uniform distribution between 1 and 2. We mask 30% of the upsampled tokens
with spans of $S=10$ and feed it into a phoneme encoder. The phoneme
embeddings are finally inputted to the 3rd conformer layer of the acoustic
encoder. $\lambda^{\text{text}}_{a}$ is set to 0.1 for all the experiments.
JOIST improves the in-context biasing by 26.7% and 8.7% relatively on PRE and
NO_PRE, but degrades the anti-context set by 5.5%.

Further, we conduct contextual text injection as described in Section 3. We
follow the same tokenization and upsampling procedure for unpaired text as
used in JOIST. Without any masking, we directly embed these upsampled phoneme
tokens via a phoneme encoder and fed them into the 3rd conformer layer of the
acoustic encoder ($l^{\text{text}}=3$). The neural biasing is then performed
with both real and text-based acoustic embeddings at the output of the 5th
conformer layer ($l=5$). The text-based biased acoustic embeddings are masked
at the 7th conformer layer ($l^{\text{mask}}$=7). $\lambda^{\text{text}}_{r}$
is set to 0.1. CTI demonstrates substantial and consistent improvement over
JOIST on all biasing test sets and VS, resulting in total relative WER
reductions of 43.3%, 26.1% on in-context sets, 11.1% on the anti-context set,
and 4.9% on head VS from the supervised baseline.

###  5.4 MWER Training for Neural Contextual Biasing

We initialize all the MWER training with the CTI model which has achieved the
best WERs so far. We first perform MWER training with only supervised data to
minimize the loss in Eq. (8).

Method |  VS | In-Context |  ANTI
---|---|---|---
NO_PRE |  PRE
CTI | 3.9 | 1.7 | 1.7 | 1.6
\+ MWER | 3.9 | 1.7 | 1.5 | 1.9
\+ JOIST MWER | 3.8 | 1.5 | 1.4 | 1.9
\+ CTI MWER | 3.8 | 1.3 | 1.3 | 1.8
Table 2: WERs (%) for HAT with neural contextual biasing. The models are
trained in two stages: first with contextual text injection (CTI), and then
fine-tuned using MWER, JOIST-based MWER, or contextual text-injected (CTI)
MWER training.

As in Table 2, MWER significantly improves over CTI on the in-context PRE and
the head VS, but degrades on the anti-context test set. We then perform JOIST-
based MWER [33] by injecting text into only the HAT encoder. During this
process, the biasing modules are updated with only the supervised data.
Compared to the MWER training, JOIST-based MWER dramatically reduces WERs on
NO_PRE. Finally, we perform contextual text-injected MWER (Section 4),
updating both HAT and biasing modules with supervised data and unpaired text.
CTI MWER significantly outperforms JOIST-based MWER across VS and all biasing
sets. Moreover, CTI MWER offers remarkable improvement over CTI on all test
sets, achieving total relative WER reductions of a 56.7%, 43.5% on in-context
biasing sets, and 7.3% on head VS from the supervised baseline. Note that, to
reduce the WER also on head VS, we selectively drop out the neural biasing
component for 30% of the training mini-batches.

###  5.5 Scalability to Number of Bias Phrases

We gradually increase the number of bias phrases assigned to each test
utterance (0-3,000) when evaluating HAT with deferred NAM on the NO_PRE in-
context biasing set in Table 3. Despite the consistent increase in WER with
more bias phrases, CTI and CTI MWER remarkably maintain WER below 1.8% even at
the maximum 3000 bias phrases. Importantly, the performance trends observed in
Sections 5.3, 5.4 persist across all bias phrase counts: CTI and CTI MWER
outperform the baseline supervised model, with their advantage increasing as
the number of phrases decreases. These trends hold consistently for PRE and
ANTI biasing sets as well.

Method | VS | In-Context: NO_PRE
---|---|---
0 | 0 |  150 |  300 |  600 | 1500 | 3000
Sup | 4.1 | 22.2 | 2.7 | 2.7 | 2.8 | 3.4 | 3.5
CTI | 3.9 | 21.8 | 1.3 | 1.4 | 1.7 | 1.8 | 2.1
\+ CTI MWER | 3.8 | 21.8 | 1.1 | 1.2 | 1.2 | 1.4 | 1.8
Table 3: WERs (%) for supervised training, contextual text injection (CTI) and
contextual text-injected (CTI) MWER training with respect to the number of
bias phrases (0-3000) assigned to each test utterance.

##  6 Conclusion

We propose a novel contextual text injection that injects unpaired text to not
only the ASR encoder, but also its biasing component. We further introduce
contextual text-injected MWER training, a method that fine-tunes the ASR model
with neural biasing to minimize an additional text-based MWER objective. CTI
reduces the WER of a HAT with deferred NAM by 26.1%-43.3% relatively on in-
context biasing sets, 11.1% relatively on the anti-context set, and 4.9% on
voice search. Further, CTI MWER achieves additional 23.5% relative WER
reductions over CTI on in-context biasing sets.

## References

  * [1] C.-C. Chiu, T. N. Sainath, and Y. Wu, ``State-of-the-art speech recognition with sequence-to-sequence models,'' in _Proc. ICASSP_ , 2018.
  * [2] M. Jain, K. Schubert, J. Mahadeokar _et al._ , ``RNN-T for latency controlled ASR with improved beam search,'' _arXiv preprint arXiv:1911.01629_ , 2019.
  * [3] T. Sainath, Y. He, B. Li _et al._ , ``A streaming on-device end-to-end model surpassing server-side conventional model quality and latency,'' in _Proc. ICASSP_ , 2020.
  * [4] J. Li, R. Zhao, Z. Meng _et al._ , ``Developing RNN-T models surpassing high-performance hybrid models with customization capability,'' in _Interspeech_ , 2020.
  * [5] A. Zeyer, A. Merboldt, R. Schlüter _et al._ , ``A new training pipeline for an improved neural transducer,'' _Proc. Interspeech_ , 2020.
  * [6] T. Sainath, Y. He, A. Narayanan _et al._ , ``An efficient streaming non-recurrent on-device end-to-end model with improvements to rare-word modeling,'' _Proc. Interspeech_ , 2021.
  * [7] L. Vasserman, B. Haynor, and P. Aleksic, ``Contextual language model adaptation using dynamic classes,'' in _Proc. SLT_. IEEE, 2016.
  * [8] P. Aleksic, M. Ghodsi, A. Michaely _et al._ , ``Bringing contextual information to google speech recognition,'' in _Proc. Interspeech_ , 2015.
  * [9] I. McGraw, R. Prabhavalkar, R. Alvarez _et al._ , ``Personalized speech recognition on mobile devices,'' in _Proc. ICASSP_. IEEE, 2016.
  * [10] D. Zhao, T. Sainath, D. Rybach _et al._ , ``Shallow-fusion end-to-end contextual biasing.'' in _Interspeech_ , 2019, pp. 1418–1422.
  * [11] W. Wang, Z. Wu, D. Caseiro _et al._ , ``Contextual biasing with the Knuth-Morris-Pratt matching algorithm,'' _arXiv preprint arXiv:2310.00178_ , 2023.
  * [12] G. Pundak, T. N. Sainath, R. Prabhavalkar _et al._ , ``Deep context: end-to-end contextual speech recognition,'' in _Proc. SLT_. IEEE, 2018.
  * [13] M. Jain, G. Keren, J. Mahadeokar _et al._ , ``Contextual RNN-T for open domain asr,'' _Proc. Interspeech_ , 2020.
  * [14] F.-J. Chang, J. Liu, M. Radfar _et al._ , ``Context-aware transformer transducer for speech recognition,'' in _Proc. ASRU_. IEEE, 2021.
  * [15] T. Munkhdalai, K. C. Sim, A. Chandorkar _et al._ , ``Fast contextual adaptation with neural associative memory for on-device personalized speech recognition,'' in _Proc. ICASSP_. IEEE, 2022.
  * [16] T. Munkhdalai, Z. Wu, G. Pundak _et al._ , ``NAM+: Towards scalable end-to-end contextual biasing for adaptive asr,'' in _Proc. SLT_. IEEE, 2023.
  * [17] Z. Wu, T. Munkhdalai, P. Rondon _et al._ , ``Dual-mode NAM: Effective top-k context injection for end-to-end asr,'' _Proc. Interspeech_ , 2023.
  * [18] Z. Wu, G. Song, C. Li _et al._ , ``Deferred NAM: Low-latency top-K context injection via deferred context encoding for non-streaming ASR,'' _Proc. NAACL 2024 - Industry Track_ , 2024.
  * [19] J. Pylkkönen, A. Ukkonen, J. Kilpikoski _et al._ , ``Fast text-only domain adaptation of RNN-transducer prediction network,'' _Proc. Interspeech_ , 2021.
  * [20] Z. Meng, Y. Gaur, N. Kanda _et al._ , ``Internal language model adaptation with text-only data for end-to-end speech recognition,'' in _Interspeech_ , 2022.
  * [21] X. Chen, Z. Meng _et al._ , ``Factorized neural transducer for efficient language model adaptation,'' in _Proc. ICASSP_ , 2022.
  * [22] Z. Meng, T. Chen, R. Prabhavalkar _et al._ , ``Modular hybrid autoregressive transducer,'' in _Proc. SLT_. IEEE, 2022.
  * [23] Z. Meng, W. Wang, R. Prabhavalkar _et al._ , ``JEIT: Joint end-to-end model and internal language model training for speech recognition,'' in _Proc. ICASSP_. IEEE, 2023.
  * [24] S. Bijwadia, S.-y. Chang, W. Wang _et al._ , ``Text injection for capitalization and turn-taking prediction in speech models,'' _Proc. Interspeech_ , 2023.
  * [25] X. Gong, Y. Wu, J. Li _et al._ , ``Advanced long-content speech recognition with factorized neural transducer,'' _TASLP_ , 2024.
  * [26] E. Variani, D. Rybach, C. Allauzen _et al._ , ``Hybrid autoregressive transducer (HAT),'' in _Proc. ICASSP_. IEEE, 2020.
  * [27] Z. Meng, S. Parthasarathy, E. Sun _et al._ , ``Internal language model estimation for domain-adaptive end-to-end speech recognition,'' in _Proc. SLT_. IEEE, 2021.
  * [28] Z. Meng, N. Kanda, Y. Gaur _et al._ , ``Internal language model training for domain-adaptive end-to-end speech recognition,'' in _Proc. ICASSP_. IEEE, 2021.
  * [29] A. Bapna, Y.-A. Chung, N. Wu _et al._ , ``SLAM: A unified encoder for speech and language modeling via speech-text joint pre-training,'' _arXiv preprint arXiv:2110.10329_ , 2021.
  * [30] Y. Tang, H. Gong, N. Dong _et al._ , ``Unified speech-text pre-training for speech translation and recognition,'' in _Proc. ACL_ , 2022.
  * [31] S. Thomas, H. J. Kuo, B. Kingsbury _et al._ , ``Towards reducing the need for speech training data to build spoken language understanding systems,'' in _Proc. ICASSP_ , 2022.
  * [32] Z. Chen, Y. Zhang, A. Rosenberg _et al._ , ``MAESTRO: Matched speech text representations through modality matching,'' in _Proc. Interspeech_ , 2022.
  * [33] T. Sainath, R. Prabhavalkar _et al._ , ``JOIST: A joint speech and text streaming model for ASR,'' _Proc. SLT_ , 2022.
  * [34] C. Peyser, Z. Meng, K. Hu _et al._ , ``Improving joint speech-text representations without alignment,'' _Proc. Interspeech_ , 2023.
  * [35] K. Wei, B. Li, H. Lv _et al._ , ``Conversational speech recognition by learning audio-textual cross-modal contextual representation,'' _arXiv preprint arXiv:2310.14278_ , 2023.
  * [36] T. N. Sainath, R. Prabhavalkar, D. Caseiro _et al._ , ``Improving contextual biasing with text injection,'' in _Proc. ICASSP_. IEEE, 2023.
  * [37] A. Graves, ``Sequence transduction with recurrent neural networks,'' _Proc. ICML_ , 2012.
  * [38] A. Baevski, Y. Zhou, A. Mohamed _et al._ , ``wav2vec 2.0: A framework for self-supervised learning of speech representations,'' _Proc. NeurlIPS_ , 2020.
  * [39] R. Prabhavalkar, T. Sainath, Y. Wu _et al._ , ``Minimum word error rate training for attention-based sequence-to-sequence models,'' in _Proc. ICASSP_. IEEE, 2018.
  * [40] L. Lu, Z. Meng, N. Kanda _et al._ , ``On minimum word error rate training of the hybrid autoregressive transducer,'' _arXiv preprint arXiv:2010.12673_ , 2020.
  * [41] Z. Meng, Y. Wu, N. Kanda _et al._ , ``Minimum word error rate training with language model fusion for end-to-end speech recognition,'' _Proc. Interspeech_ , 2021.
  * [42] R. Prabhavalkar, Z. Meng, W. Wang _et al._ , ``Extreme encoder output frame rate reduction: Improving computational latencies of large end-to-end models,'' _Proc. ICASSP_ , 2024.
  * [43] Y. Zhang, W. Han, J. Qin _et al._ , ``Google USM: Scaling automatic speech recognition beyond 100 languages,'' _arXiv preprint arXiv:2303.01037_ , 2023.
  * [44] A. Gulati, J. Qin, C. Chiu _et al._ , ``Conformer: Convolution-augmented transformer for speech recognition,'' _Proc. Interspeech_ , 2020.
  * [45] Z. Dai, G. Lai, Y. Yang _et al._ , ``Funnel-transformer: Filtering out sequential redundancy for efficient language processing,'' _Proc. NeurlIPS_ , 2020.
  * [46] W. Wang, R. Prabhavalkar, D. Hwang _et al._ , ``Massive end-to-end models for short search queries,'' _arXiv preprint arXiv:2309.12963_ , 2023.
  * [47] R. Botros, T. N. Sainath, R. David _et al._ , ``Tied & reduced RNN-T decoder,'' _Proc. Interspeech_ , 2021.


# A Review of Modern Approaches for Coronary Angiography Imaging Analysis

Maxim Popov, Temirgali Aimyshev, Eldar Ismailov, Ablay Bulegenov, Siamac Fazli
M. Popov and S. Fazli are with the Department of Computer Science, Nazarbayev
University, Astana, Kazakhstan. T. Aimyshev, E. Ismailov and A. Bulegenov are
with CMC Technologies, Astana, Kazakhstan. Corresponding author: S. Fazli
(siamac.fazli@nu.edu.kz). The co-authors would like to acknowledge the support
of Nazarbayev University Research Grants funding (240919FD3926) and .

###### Abstract

Coronary Heart Disease (CHD) is a leading cause of death in the modern world.
The development of modern analytical tools for diagnostics and treatment of
CHD is receiving substantial attention from the scientific community. Deep
learning-based algorithms, such as segmentation networks and detectors, play
an important role in assisting medical professionals by providing timely
analysis of a patient’s angiograms. This paper focuses on X-Ray Coronary
Angiography (XCA), which is considered to be a “gold standard” in the
diagnosis and treatment of CHD. First, we describe publicly available datasets
of XCA images. Then, classical and modern techniques of image preprocessing
are reviewed. In addition, common frame selection techniques are discussed,
which are an important factor of input quality and thus model performance. In
the following two chapters we discuss modern vessel segmentation and stenosis
detection networks and, finally, open problems and current limitations of the
current state-of-the-art.

###### Index Terms:

Computer Vision, Artificial Intelligence, Image Segmentation, Image
Classification, Object Detection, Coronary Artery Disease, Coronary Heart
Disease, X-ray Coronary Angiography, Image Preprocessing, Deep Learning,
Object Tracking, Review.

##  I Introduction

Coronary Heart Disease (CHD), also known and referred to as Coronary Artery
Disease (CAD), is among the most common causes of death in the world [1]. It
is characterized by the presence of atherosclerotic plaques in coronary
vessels [2]. According to different sources, coronary artery disease takes up
to 19 million lives annually, constituting for about $32\%$ of global
mortality [3]. In particular, according to the survey on heart disease and
stroke in the US, about 15.4 million people older than 20 suffer from ischemic
heart disease [4].

X-Ray Coronary Angiography (XCA) is a gold standard in the diagnosis and
treatment of CAD [5, 6]. In this method, contrast agents (e.g. iodixanol-320)
are injected to coronary vessels through a catheter to highlight the blood
flow on x-ray angiograms. It enables clinicians to obtain detailed images of
coronary arteries. These angiograms are used by medical professionals to make
qualitative judgements about the vascular wall calcification and stenosis
problems [7, 8].

Manual and semi-automatic analysis of angiograms requires significant amount
of time, thus reducing the number of patients receiving treatment. In addition
it relies heavily on the doctor’s personal experience and capability, thus
resulting in the issue of inter- and intra-observer variability [6]. Inter-
observer variability means that annotations done by several different doctors
may have diverging labels and region borders. Intra-observer variability means
that the same doctor may, on different occasions, produce inconsistent
annotations. Reasonable attempts to automate this process in order to overcome
above mentioned limitations using different algorithms have been made over the
last 20 years [9, 10, 11, 12, 13, 14, 15, 16], and especially blossomed with
the emergence of the first fully convolutional neural networks [17] and their
further development in the form of segmentation networks [18, 19, 20, 21, 22,
23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]. Convolutional neural networks
(CNNs) have achieved remarkable results in many medical applications,
including the detection of Covid-19 [34], cancer [35, 36], retinopathy [37],
and many other serious diseases, given the public availability of data and
pretrained models.

However, to date, a number of unsolved problems prevail, which hinder accurate
and efficient diagnosis of medical X-ray coronary angiograms by means of
automated algorithms. These problems include borders from X-ray filters,
rotation of the image frames, varying levels of contrast agent, magnification
during image acquisition [38], noisy image capture, sample/patient variability
[39] and deformation of the shape of blood vessels on the image [7]. While a
number of solutions have recently been proposed, which alleviate some of these
issues, others are still unsolved at this time. Unfortunately, however, a
comprehensive review of open issues and their potential resolution is
currently lacking within the scientific literature and this review attempts to
fill this gap.

##  II Open Datasets

One of the most important issues in this area is the unavailability of large
and reliable datasets for model estimation and testing, similiar to those
which exist for retinal images – DRIVE [40] and STARE [41]. Currently, only
very few datasets are publicly available for benchmarking. For example, [42]
provides 134 angiograms and their respective ground truth images labelled by
professional clinicians for open access. Another paper also included 30
4-frame sequences from angiographic videos and corresponding binary masks
[43]. One challenge facing the production of ready-to-use datasets for this
task lies in the fact that not all images and XCA sequences are suitable for
segmentation. Vessel visibility due to the propagation of contrast agent, as
well as the cardiac phase are playing an important role for medical
specialists in determining a suitable frame for analysis. Another challenge is
given by the low availability of certified medical professionals, including
cardiologists and radiologists, for reliable dataset production.

##  III Image Preprocessing Methods

Image preprocessing is crucial for the automated analysis of angiographic
images. As it was stated before, the domain of angiography is intertwined with
highly noisy images, patient’s motions, varying levels of contrast, thus
vessel segmentation and lesion detection tasks require some level of image
enhancement, so that visual characteristics of the image can be upgraded.
Image enhancement techniques can be split into 2 parts, with the first one
being morphological operations such as dilation and erosion [44, 45, 39], and
second one being advanced mathematical algorithms like Multiscale Retinex with
Color Restoration (MSRCR) [46] and Contrast Limited Histogram Equalization
(CLAHE) [47] and even specialized deep neural networks [38].

Figure 1: Gives an overview of various preprocessing methods in cornary
angiography. A1-A3. Shows raw images; B1-B3. Shows the result of application
of Hessian-based Frangi Vesselness filter; C1-C3. Shows images after the top-
bottom hat transform was applied; D1-D3. Shows the result of applying MSRCR;
E1-E3. Shows the results of CLAHE on test images.

The first example is the top-bottom hat transform, as can be seen in the third
column of Fig. 1C. The top-bottom hat transform is one of the most common and
computationally simple tools for edge visibility enhancement. It is based on
multiscale high- and low-pass filters. High-pass filters leave only the bright
spots, which usually represent vessels on the image, while low-pass filters
aim to leave only the background of the image. The resulting filtered image is
obtained by the following formula [44]:

$I_{en}=I+I_{th}-I_{bh},$ (1)

where $I_{en}$ is the enhanced image, $I$ the original image, $I_{th}$ the
image after the top-hat transform, and $I_{bh}$ the image after the bottom-hat
transform. A slightly altered version of this transform includes adjustable
weight parameters for $I_{th}$ and $I_{bh}$ [45].

Contrast Limited Adaptive Histogram Equalization (CLAHE) is an image
preprocessing algorithm [47], for improved visibility of lesions on mammogram
images. This method is based on the application of Adaptive Histogram
Equalization (AHE) on small regions of the image, called tiles, instead of the
whole image. Individual tiles are then combined using interpolation
techniques, in order to remove artificial boundaries, occurring after
equalization. Currently, this method is widely applied in retinal image
segmentation [48], chest x-ray images [49], and also found in the application
of XCA segmentation [50] An example of CLAHE on coronary angiograms can be
found in the fifth column of Fig. 1E.

Multi-scale Retinex with Color Restoration (MSRCR) is an algorithm designed to
remove the artifacts from the image while preserving sharpness on the edges by
providing color constancy and dynamic color range suppression [46]. It is a
computationally efficient tool and results in the improvement of the image
structure, effectively removing noise from the angiogram. It is often
considered as an important step in the preparation of images for further
preprocessing methodologies such as [51], since this methodology is known to
be highly sensitive to small artifacts left by noise. Further development of
classic MSRCR architecture was proposed in [52]. Zhang and colleagues had
utilized MSRCR in a pipeline, consisting of Singular Value Decomposition in
the RGB color space, automated MSRCR and Gamma Stretching for luminance and
saturation enhancement in the HSV color space, and Guided Filtering for final
denoising in RGB space. Examples of application of classical MSRCR on XCA
images can be found in [53] and on Fig. 1D.

Hessian filters are an example of computationally expensive mathematical image
enhancement techniques. This method was initially designed to detect blob-like
structures from local maxima of a given image [51]. A Hessian filter for a 2D
image is a 2x2 matrix, composed of second order partial derivatives of the
intensity function of the image.

$H_{2}(x)=\begin{pmatrix}\frac{\delta^{2}f}{\delta
x^{2}}&\frac{\delta^{2}f}{\delta x\delta y}\\\ \frac{\delta^{2}f}{\delta
y\delta x}&\frac{\delta^{2}f}{\delta
y^{2}}\end{pmatrix}\triangleq\begin{pmatrix}f_{xx}&f_{xy}\\\
f_{yx}&f_{yy}\end{pmatrix}$ (2)

Reconstructed images with highlighted tubular structures
$I_{\sigma}(\overline{x})$ are calculated separately for different values of
$\sigma$ by convolution of the original image $I$ with a Gaussian kernel
$G_{\sigma}(\overline{x})$:

$I_{\sigma}(\overline{x})=I*G_{\sigma}(\overline{x})$ (3)

The value for $\sigma$ is selected from range $0.5$ to $6$ times the system
resolution of $165\mu$m, usually being equal to $2$ to $10$ pixels. The
normalized Hessian matrix of the resulting image $I_{\sigma}(\overline{x})$
for the scale $\sigma$ is given by:

$H(\overline{x},\sigma)=B\frac{\delta^{2}I_{\sigma}(\overline{x})}{\delta\overline{x}^{2}}$
(4)

With B being a normalization factor.

The eigenvectors of a Hessian matrix define the principal directions of local
image features. Blood vessel regions can be detected based on the properties
of eigenvalues ($\lambda_{1}$,$\lambda_{2}$), with a special condition
$|\lambda_{1}|\leq|\lambda_{2}|$. The Hessian-based Frangi Vesselness filter
is used to extract tubular structures from the image using the vesselness
function [51]. Frangi  _et al._ specify the vascular similarity formula for a
given set of eigenvalues as follows:

$V_{f}(\overset{\rightarrow}{\lambda})=\Biggl{\\{}\begin{aligned}
&~{}0,\qquad\text{if}\quad\lambda_{2}<0\\\
&\exp{\left(-\frac{R^{2}_{\beta}}{2\beta^{2}}\right)}\left(1-\exp{(1-\frac{-S^{2}}{2\gamma^{2}}})\right)\end{aligned}$
(5)

$R_{\beta}$ here is the blobness measure, used to distinguish vessels from
blob-like structures, and calculated by $\frac{\lambda_{1}}{\lambda_{2}}$. $S$
is the second order structureness measure, used to distinguish vessels from
the background and noise, calculated by
$S=\sqrt{\lambda_{1}^{2}+\lambda_{2}^{2}}$. Parameters $\beta$ and $\gamma$
are weights that measure the influence of $R_{B}$ and $S$ on the vascular
similarity response. This formula is used to reconstruct a final vessel-
enhanced image at different scales $\sigma$ by summing filter responses. This
way, vesselness function returns the value in range $(0,1)$, with maximum
value corresponding to a tubular structure of scale $\sigma$, and smallest
value corresponding to background pixels [51].

While having been discovered several decades ago, Hessian filters are still
commonly employed with new modifications being released up to this date [54].
In fact, they remain one of the most useful and popular baseline methods for
unsupervised image enhancement and segmentation to this day [55, 56, 57, 58].
Practical applications of Hessian filters for tubular structures detection can
be observed on the Fig. 1B.

Sazak and colleagues in their work provide criticisms for popular enhancement
methods, such as Hessian-based, mathematical morphology-based, and adaptive
histogram-based filters. As an alternative, they propose an improvement over
the method proposed earlier by Zana and Klein [59], called multiscale bowler-
hat transform [39]. This approach is based on mathematical morphology and uses
2 morphological operations to perform vessel enhancement: erosion and
dilation. Dilation ($\oplus$) for a given pixel $I(p)$ with a structuring
element $b(p)$ in a greyscale image is defined as:

$(I\oplus b)(p)=\underset{x\in E}{sup}[I(x)+b(p-x)]$

where “sup” is the supremum and $x\in E$ denotes all points in the Euclidean
space of the image. Erosion ($\ominus$) is defined in similar terms:

$(I\ominus b)(p)=\underset{x\in E}{inf}[I(x)+b(p-x)]$

where ‘inf’ is the infimum. These two operations are combined to form the
opening operation, which is the erosion of the original image, followed by the
dilation using the same structuring element. Sazak _et al._ proposed to use a
combination of lines with 12 different orientations and different radii, and
disks with different radii as structuring elements for the opening operation.
For each diameter of a vessel $d$ from $1$ to $d_{m}ax$, a disk shaped filter
is used to produce a stack of images, on which any blob-like element of size
smaller than $d$ are removed. The same goes for linearly structuring elements
– for each combination of direction $\theta$ and length of structuring element
$d$, a distinct image is produced using opening operation. The enhanced image
is then formed from these two stacks, using the maximum difference at each
pixel between the stacks for each $d$. This way, pixels in the background are
removed using opening with disk-shaped structuring elements, and pixels in the
foreground are preserved, i.e., vessel-like structures will have a high value
due to line-based structuring elements.

The Angiographic Processing Network (APN) is a modern approach to image
enhancement [38]. It is based on deep neural networks which are trained using
images, that are enhanced with hand-crafted filters, as targets, with the aim
of improving vessel visibility. The idea behind designing this network is that
while the perfect filter is generally unknown, it can be learned by the
network from labelled examples. APN is initialized to mimic a combination of
standard enhancement filters, such as unsharp mask filters. These filters are
constructed with an aim to improve vessel boundaries and local contrast on
vessel edges. Then, the network is fine-tuned to produce a new filter that
suits the task of image segmentation most.

In summary, image enhancements are an essential tool for the vessel
segmentation task, and sometimes they can accomplish this task, without the
need of applying more complex algorithms, such as deep neural networks.

##  IV Best Frame Selection

Frame selection is an important step for obtaining clean data, which is a
prerequisite for developing a well-performing deep neural network. Input data
may be compromised by cardiac movements, spread of color agents, noisy image
capture and movements of patient, which can negatively affect the network’s
performance. Another important reason for frame selection is that it reduces
that number of input dimensions, which makes the DNN less likely to overfit.
Frame selection is usually performed manually [60, 61, 45] either without any
additional modalities or in combination with ECG signals, which are collected
synchronously with XCA [62]. In order to automate the frame selection process
and thus assist radiologists, several algorithmic approaches have been
proposed. They range from complex deep neural networks with million of
parameters and thousands of training samples, to simpler models relying on the
developer’s domain knowledge.

A CNN + LSTM framework for the selection of ideal candidate frames has
recently been introduced [63]. Best frame was described as the one having the
best image quality, full contrast-agent penetration, clearly contrasted vessel
borders, and anatomical significance of stenosis, if present. Initially,
inception-v3 [64] was pretrained on 1073 images from 19 patients for
recognizing ideal candidates, and then its fully connected layer was connected
to a pair of bi-directional LSTM networks based on [65]. The resulting network
was trained with 18688 frames from 19 patients, with videos normalized to
contain 128 frames. Performance of this step was assessed on 582 XCA videos
from 175 patients. Initially, medical professionals indicated the Beginning
and Ending Constrasting Frames on the ICA sequences. These denote the time
interval during which vessel colouring is reached. Then, neural network was
trained to detect these key frames automatically. The result of the inference
was considered as acceptable, as long as the average absolute difference
between predicted and real key frames was lower than $3$. This way, an
acceptance and error rate of $0.83$ and $0.0498$ were obtained, respectively.
This method is suitable for data filtering for medical professionals, however,
it does not indicate which frames are suitable for the detection task, since
it does not provide cardiac phase information.

U-net [32] inspired neural network that extracts temporal features from
angiographic video was proposed in [62]. This paper focuses on cardiac phase
decoding based on angiographic images in order to determine the best frame for
stenosis detection. This is accomplished by selecting images where the
negative effects of heart movements are minimized. For this purpose, authors
designed two separate neural networks – the first one detects the time window
of optimal vessel coloring, and the second one selects the optimal images,
based on specific time points within the cardiac cycle. For training the
network, cardiac cycles are estimated from ECG. Training and validation sets
were composed of total of 17800 angiographic images from 3900 patients, with
80-20% split. This method has a precision and recall of 97.4% and 96.9%,
respectively, for the task of detecting the end-diastolic phase.

The same task of detecting frames containing the end-diastolic phase was
accomplished using the key point motion detection [66]. The accuracy of this
method reached 92.99%, utilizing data of 31 patients (as compared to 3900 in
[62]). These results show that simple models with fewer number of parameters
can achieve competitive results with relatively few samples (i.e. training
data).

To select a sequence, consisting of several consecutive images for stenosis
detection, a U-net [32] based network was designed and trained with 90 XCA
sequences. Individual frames from these sequences were divided into 3 classes
- zero contrast filling, partial, and full contrast filling [67]. To
compensate for the lack of training samples, custom data augmentation methods
were proposed. In their work Wu _et al._ introduced a hyperparameter N. This
hyperparameter indicates how many preceding and following frames should be
selected with respect to the most contrast-filled frame (resulting in $2N+1$
frames in total). Best candidate frame in this approach was chosen using a
predicted probability of a frame to represent a class of fully-filled frames.
The accuracy of key frame detection using the U-net-related architecture was
reported to be $0.867$, outperforming other considered architectures, such as
Multiscale Region Growing [68], Frangi thresholding algorithm [51], Coye’s
method [69], and Otsu’s thresholding method [70]. The reasoning to extract
several frames instead of just one was to reduce the number of false-positive
responses of the stenosis detection module by comparing the stenosis detection
results across several frames.

Overall, key frame detection is a task that can be successfully completed
using machine learning approaches with virtually any amount of data available.
Even simple models can help automating this task, and by collecting more data
or by effectively utilizing domain knowledge scientist can acquire even more
sophisticated results. Automated recognition of the most suitable time window
and detection of an end-diastolic cardiac phase should greatly assist
cardiologists and deep learning algorithm designers in diagnostics of CAD.

##  V Segmentation

Current state of vessel annotation requires doctors to manually segment vessel
regions and classify lesions and stenosis in them, relying on methods such as
the Syntax Score [71]. Syntax Score is an objective method of vessel analysis
and stenosis detection, which is commonly used in medical practice. In recent
years, the use of deep learning based neural networks for the automation of
biomedical imaging diagnostics has become a viable alternative. However, these
methodologies have not yet witnessed large-scale adoption and thus remain in a
proof-of-concept stage.

Figure 2: Segmentation of vessel into specific regions according to the Syntax
Score. Top row represents sample angiographic images. Bottom row represents
their division into regions according to Syntax Score guidelines. To the best
of our knowledge, there is no publicly available model for multi class
segmentation of vessel regions.

Semantic segmentation tasks are aimed to perform pixel classification for a
given image. In order to accomplish this task, several modern segmentation
architectures have been proposed. The two most successful representatives are
U-Net [32] and DeepLab [33]. A large number of adaptations and modifications
have also been proposed [18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30,
31]. These approaches have succeeded for a large number of related tasks in
medical image analysis, such as retinal vessel segmentation [37], renal
ultrasound image segmentation [72], chest airway segmentation [73], brain
tumor segmentation [35],breast tumor segmentation [36], and many others.
Inspired by their success, a series of meaningful improvements based on
autoencoders have recently emerged, which improve segmentation quality. In the
following, we will review a cross-section of these methodologies.

To date, only few papers compared the performance of various architectures for
coronary vessel segmentation in a comprehensive manner [74, 75]. Yang et al.
compares the efficiency of different baseline models such as U-Net [32],
ResNet [76], Dense-Net [77], and InceptionResNet-V2 [78]. Their results
indicate that DenseNet121 shows favourable performance coupled with a
comparatively simple model, e.g. DenseNet121 has 5 times less parameters as
InceptionResNet-V2, while exhibiting on-par performace in terms of F1 score.
DenseNet121 reached 0.917 for this metric, while InceptionResNet-v2 scored
insignificantly less – 0.915. The number of images used for training and
testing all architecture is 3302 images from 2042 patients.

The network architecture comparison conducted in [75] was done using a dataset
of 3200 images, which were selected and annotated by cardiologists with an
average experience of 10 years. Their study shown strong results for the
Residual Attention Network [79] for most metrics. Its average precision is
reported as $0.919$ across different modalities, and DenseNet [77] takes
second place with an average precision of $0.912$. A disadvantage of this
study lies in the fact that model complexity is not addressed.

A recent of application of conditional generative adversarial networks (cGANs)
[80] utilizes XCA images to automatically segment artery regions into 20
regions for pix2pix segmentation [80]. The proposed cGAN consists of 2
networks – a generator and a discriminator. The generator is trained to
generate an image from an input coronary angiogram, with the aim to reproduce
the features of the original image, so that the discriminator network will
classify whether each pixel is correctly labelled. The discriminator is, on
the other side, is trained to distinguish between original images and ones
produced by the generator. These two networks are trained until they reach a
Nash equilibrium, where the discriminator can distinguish fake images from
real ones with a probability of $50$% (i.e. random guessing). Their work
utilizes the biggest dataset used in this field of research - $20,612$
angiograms in total, with a segment recognition accuracy of $0.984$ and a
sensitivity of $0.852$. An outstanding feature of this paper lies in its
unique approach, which in some ways resembles the SYNTAX Score [71],
inheriting the division of the coronary tree into specific regions, unlike
most other segmentation architectures which employ a binary classification
approach that merely distinguishes vessels and background noise.

Another creative approach to image segmentation using GANs was proposed in
[44]. The authors used a popular style transfer network with an Adaptive
Instance Normalization layer [81] and added Convolution Block Attention
Modules [82] and DexiNet [83] for improved style transfer performance.
Uniqueness of this method lies in the extremely fast inference time and high
accuracy of $0.9658$, which was tested using both open datasets described
earlier [42, 43]. This method does not require ground truth images, which were
labelled by medical professionals for training purposes. Furthermore, it
produces segmented images using style transfer from black and white drawings,
which are taken from the BSDS500 dataset [84]. The result of style transfer
using AdaIN network for XCA segmentation task can be observed on Fig. 3.

Figure 3: Sample segmentation using style transfer from open source black and
white drawing using baseline AdaIN network. Original image on the left;
Inference result on the right.

A number of approaches have been proposed to incorporate the temporal dynamics
of consecutive angiographic frames for segmentation [7, 43]). Their idea was
to explicitly model the flow of the color agent and thereby alleviate the
negative impacts of artifacts and the patient’s movement on the segmentation
performance.

Semi-3D TENet, proposed by [7], uses N=1 or N=2 additional images, preceding
and following the manually selected best frame, resulting in a total of 3 or 5
images in the third input dimension. Angiographic videos, that were chosen for
the dataset, can be divided into 2 classes – short-period videos, where the
contrast-filled temporal area covers less than 10 frames, and long-period
videos, where the contrast-filled temporal area covers more than 10 frames. To
train and test this network $4904$ images were collected and manually
annotated by medical students. $3099$ images were selected from the short-
period videos, and $1805$ images were selected from the long-period videos.
Comparative analysis of model performances after training on short-period and
long-period videos separately has shown that the model achieves better score
on short videos with N=1, while with longer videos better accuracy is reached
when N=2. However, as there is much higher number of short-period videos in
the training dataset, it was decided to use N=1 when training on the whole
dataset (i.e. both short-period and long-period videos together). Results on
the test set have shown the accuracy of $0.986$ and Intersection over Union
score of $0.7221$.

The Deep Neural Network for sequential vessel segmentation (SVSNet) [43] also
uses the third dimension to effectively extract temporal correlation from the
angiographic video. To verify the effectiveness of using the consecutive frame
series as input, different numbers of input frames were tested (e.g. 2, 3, 4,
5, etc.). Experimental results proved that using 4 successive frames as an
input resulted in the smaller loss and better model convergence. It was
trained and tested using a total of 120 sequences of clinical XCA images. From
these 120 sequences, 323 sample images were selected and manually annotated.
The sample images include very low-contrast vessels as well as vessel trees
that contain thin vessel branches, thus requiring the work of 3 experts to
establish the ground truth. A more complex network, consisting of 3D
convolution layers and Channel Attention Blocks (CAB) is reported to
significantly improve performance as compared to naive 2D and 3D networks,
with recall of $0.8424$ and F1 score of $0.8428$.

A novel ensemble framework, based on gradient-boosting decision trees [85] and
Deep Forest [86] classifiers, developed in [45], is reported to outperform
U-net-based algorithms and modern DeepLabV3+ [18]. 21 filter-based features
and 16 Dense-Net [77] extracted features are combined into a 37-dimensional
feature vector for each pixel of the XCA image. This ensemble method reached a
mean precision of $0.857$, a specificity of $0.992$ and an F1 score of $0.874$
on a custom dataset of 130 manually annotated XCA images, of which 60% were
used for training, 20% for validation, and 20% for testing.

The idea of combining two different tasks to achieve better segmentation
results is introduced and developed in [87, 88]. By combining centerline and
direction extraction tasks they achieve better vessel continuity for vessel
segmentation. This resulted in increased sensitivity ($0.8561$) and F1 score
($0.8561$), compared to U-net and other modern segmentation methods with
significantly lower number of trainable parameters (2.71 million) and only 200
images used for training.

##  VI Lesion Detection, Localization, and Classification

Object detection and classification are well-studied and established fields
within computer vision [89, 90, 91, 92, 93, 94, 95, 96]. For coronary
angiography these tasks are currently performed in a semi-automated manner
using a Quantitative Coronary Angiography (QCA) tool that comes along with
medical imaging hardware. The drawbacks of this semi-automated process are
that it is time-consuming and requires the cardiologist to manually highlight
vessel structures. To date, a number of attempts have already been made to
automate this process. Delegating the responsibility of stenosis labelling on
XCA images to automatic algorithms would: i) serve as valuable assistive
technology for the medical personel; ii) lessen the impact of subjective human
perception and iii) increase the number of patients receiving timely
treatment.

Figure 4: Detection results performed on the test dataset by sample network.
To the best of our knowledge, there is no model trained and tested for lesion
detection on coronary vessels available for public testing and benchmarking.
Top row represents the original images, and bottom row represents model
response.

The Deconvolutional Single-Shot multibox Detector (DSSD) [91] was proposed for
stenosis detection [67]. This framework uses 3 adjustable hyperparameters,
which are found during cross-validation. $N_{c}f$ is the number of contrast-
filled frames, $N_{sf}$ the number of frames where stenosis occurs, and
$N_{seq-iou}$ is the threshold for the Intersection over Union (IoU) operation
for detecting whether detected stenosis occurs across several bounding boxes.
The total number of sequences used in this paper is $148$, with 20% of the
samples used for testing. In this framework, the DSSD network makes a series
of independent detections over several consecutive frames. The results of
detection on independent frames are then compared, and if the result of
detection is not persistent on all of the frames in the sequence, it is
discarded, in order to reduce the number of false positive activations. These
comparisons are carried out by a specific module called seq-fps. This approach
proved to be a very effective solution, as without the seq-fps module, simple
single-shot detection had a sensitivity of $0.871$ and Positive Prediction
Value (PPV) of $0.461$, and after the processing with seq-fps the sensitivity
remained unchanged, while PPV increased to $0.795$.

Pang  _et al._ proposed Stenosis-DetNet for stenosis detection using bounding
boxes [97]. Their dataset consists of 166 images, of which 26 were chosen for
testing, and the rest for training. In a similar way to Wu  _et al._ [67],
they propose a Sequence Feature Fusion (SFF) module, that allows to reduce the
number of false positive responses of the network utilizing several
consecutive frames for the detection, and one additional module, called
Sequence Consistency Alignment module, that allows to confirm the location and
class of a detected object using several consecutive frames. The combination
of these two modules resulted in an increase of the precision to $0.9487$ and
an F1 score of $0.881$, while method proposed by Wu and their colleagues got
only the precision of $0.795$ on DetNet dataset.

A different approach to lesion detection is provided by [63]. Instead of a
detector network, classification network is employed, which is trained to
classify images with $>25\%$ of stenosis, and extracts an activation map from
it. The dataset contains 690 XCA images and was annotated independently by two
expert cardiologists, with stenosis given a nominal ground truth bounding box
of $35\times 35$ pixels. Experiments were conducted in 2 different modes, with
3-class labelling, namely “$<25\%$, $25-99\%$, $>100\%$ Stenosis”, and 2-class
labelling, namely “$<25\%$” and “$>25\%$ Stenosis” classes. The latter showed
better results of the two in all metrics, reaching an F1 score of $0.76-0.83$.

The DeepDiscern network was proposed by [80] and provides an automated tool
for lesion detection and their localization. Du  _et al._ trained their model
with $6239$ samples and reached an average F1 score of $0.8236$ on the test
set. One of the interesting and unique features of this work is that it
discriminates lesions into 5 distinct classes: i) stenotic lesions; ii) total
occlusion; iii) calcification; iv) thrombosis and v) dissection.

However, to the best of our knowledge, none of these fully-automated and
powerful methodologies mentioned above are currently being implemented in
clinical practice. Prior to their widespread adoption, large-scale open
datasets need to be established and all methodologies need to be evaluated
consistently in order to enable comparable and robust estimates of their
quality. These types of benchmark datasets (e.g. MNIST, CIFAR, ImageNet, among
many others) have shown to greatly accelerate scientific progress in many
closely-related areas of computer vision as well as for machine learning in
general, since they increase transparency and as a result the trust of
scientific community. As a desirable future outcome, we hope that the creation
of large-scale benchmark datasets for coronary angiography will not only be
beneficial from a scientific perspective, but also increase the acceptance of
clinicians towards these types of modern, fully-automated approaches.

##  VII Open Problems and Limitations in the field

As can be deduced from the analysis of the current state of the art in
automated analysis of coronary angiograms, there are still several open
problems that need to be addressed:

  * •

There is a shortage of large-scale, annotated coronary angiographic data,
which have been labelled by experts of the field. While the above mentioned
papers claim highly accurate results, their models are trained on either very
small datasets or they are trained on private datasets. This raises multiple
concerns: i) the question remains whether these models are able to generalize
well to other data, which may contain diverse clinical cases of angiograms not
included in the datasets availavle to the authors; ii) since most of the data
is not publicly available, it is impossible to objectively verify the reported
results and iii) it hinders many research groups to utilize this data and
create their own models, thus limiting the number of approaches that can be
employed. Producing large-scale data of high quality is difficult, since
expert annotation of a given dataset for deep learning purposes is a time-
consuming task and needs to be performed by professional clinicians, who have
other clinical responsibilities and thus may be unable to annotate thousands
of images in a short time.

  * •

As for now, there is no real benchmark for angiogram abnormalities detection.
Since datasets are either very small or private and most of the models
(including their code) are not publicly available, standardized performance
benchmarks are impossible. Similarly, there are no test sets for which
proposed models can be evaluated independently. As a result efficient progress
is hindered and there is a lack of consensus for building accurate and
efficient models for this task.

  * •

Some of the metrics, commonly used for ML-based model evaluation, are not
suitable for clinical contexts. These metrics include sensitivity,
specificity, accuracy and others related metrics, which are traditionally
employed for the evaluation of ML models in general and DNNs in particular.
However, these metrics deviate from ones that are used by medical
professionals, who need to estimate the confidence of their diagnosis. This
may hinder their acceptance by medical professionals and it is thus desirable
to adapt the model evaluation strategies and their outputs to the
professional’s needs. Additionally, ML-based metrics may be complex to
comprehend and interpret for some clinical specialists.

## References

  * [1] P. A. McCullough, “Coronary artery disease,” _Clinical Journal of the American Society of Nephrology_ , vol. 2, no. 3, pp. 611–616, 2007.
  * [2] U. Ralapanawa and R. Sivakanesan, “Epidemiology and the magnitude of coronary artery disease and acute coronary syndrome: A narrative review,” _Journal of Epidemiology and Global Health_ , vol. 11, no. 2, p. 169, 2021\.
  * [3] I. Ferreira-González, “The epidemiology of coronary heart disease,” _Revista Española de Cardiología (English Edition)_ , vol. 67, no. 2, pp. 139–144, 2014.
  * [4] A. S. Go, D. Mozaffarian, V. L. Roger, E. J. Benjamin, J. D. Berry, W. B. Borden, D. M. Bravata, S. Dai, E. S. Ford, C. S. Fox, S. Franco, H. J. Fullerton, C. Gillespie, S. M. Hailpern, J. A. Heit, V. J. Howard, M. D. Huffman, B. M. Kissela, S. J. Kittner, D. T. Lackland, J. H. Lichtman, L. D. Lisabeth, D. Magid, G. M. Marcus, A. Marelli, D. B. Matchar, D. K. McGuire, E. R. Mohler, C. S. Moy, M. E. Mussolino, G. Nichol, N. P. Paynter, P. J. Schreiner, P. D. Sorlie, J. Stein, T. N. Turan, S. S. Virani, N. D. Wong, D. Woo, and M. B. Turner, “Executive summary: Heart disease and stroke statistics-2013 update: A report from the american heart association,” _Circulation_ , vol. 127, no. 1, pp. 143–152, 2013, cited By :1049. [Online]. Available: www.scopus.com
  * [5] M. Barbanti, D. Todaro, G. Costa, G. Pilato, A. Picci, S. Gulino, P. Capranzano, K. La Spina, E. Di Simone, P. D’Arrigo, _et al._ , “Optimized screening of coronary artery disease with invasive coronary angiography and ad hoc percutaneous coronary intervention during transcatheter aortic valve replacement,” _Circulation: Cardiovascular Interventions_ , vol. 10, no. 8, p. e005234, 2017.
  * [6] A. Carballal, F. J. Nóvoa, C. Fernandez-Lozano, M. García-Guimarães, G. H. López-Campos, R. Calviño-Santos, J. M. Vázquez-Naya, and A. Pazos, “Automatic multiscale vascular image segmentation algorithm for coronary angiography,” _Biomed. Signal Process. Control._ , vol. 46, pp. 1–9, 2018.
  * [7] D. Liang, L. Wang, D. Han, J. Qiu, X. Yin, Z. Yang, J. Xing, J. Dong, and Z. Ma, “Semi 3d-tenet: Semi 3d network based on temporal information extraction for coronary artery segmentation from angiography video,” _Biomedical Signal Processing and Control_ , vol. 69, p. 102894, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1746809421004912
  * [8] X. Zhu, Z. Cheng, S. Wang, X. Chen, and G. Lu, “Coronary angiography image segmentation based on pspnet,” _Computer Methods and Programs in Biomedicine_ , vol. 200, p. 105897, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0169260720317302
  * [9] M. Schaap, T. van Walsum, L. Neefjes, C. Metz, E. Capuano, M. de Bruijne, and W. Niessen, “Robust shape regression for supervised vessel segmentation and its application to coronary segmentation in cta,” _IEEE transactions on medical imaging_ , vol. 30, no. 11, pp. 1974–1986, 2011.
  * [10] J. Brieva, E. Gonzalez, F. Gonzalez, A. Bousse, and J. Bellanger, “A level set method for vessel segmentation in coronary angiography,” in _2005 IEEE Engineering in Medicine and Biology 27th Annual Conference_. IEEE, 2006, pp. 6348–6351.
  * [11] S. Bock, C. Kühnel, T. Boskamp, and H.-O. Peitgen, “Robust vessel segmentation,” in _Medical Imaging 2008: Computer-Aided Diagnosis_ , vol. 6915. SPIE, 2008, pp. 1010–1018.
  * [12] M. F. Khan, S. Wesarg, J. Gurung, S. Dogan, A. Maataoui, B. Brehmer, C. Herzog, H. Ackermann, B. Aßmus, and T. J. Vogl, “Facilitating coronary artery evaluation in mdct using a 3d automatic vessel segmentation tool,” _European Radiology_ , vol. 16, no. 8, pp. 1789–1795, 2006.
  * [13] Y. Li, S. Zhou, J. Wu, X. Ma, and K. Peng, “A novel method of vessel segmentation for x-ray coronary angiography images,” in _2012 Fourth International Conference on Computational and Information Sciences_. IEEE, 2012, pp. 468–471.
  * [14] A. Wahle, J. J. Lopez, M. E. Olszewski, S. C. Vigmostad, K. B. Chandran, J. D. Rossen, and M. Sonka, “Plaque development, vessel curvature, and wall shear stress in coronary arteries assessed by x-ray angiography and intravascular ultrasound,” _Medical image analysis_ , vol. 10, no. 4, pp. 615–631, 2006\.
  * [15] S. Zhou, J. Yang, W. Chen, and Y. Wang, “New approach to the automatic segmentation of coronary artery in x-ray angiograms,” _Science in China Series F: Information Sciences_ , vol. 51, no. 1, pp. 25–39, 2008.
  * [16] C. Köse, “Fully automatic segmentation of coronary vessel structures in poor quality x-ray angiogram images,” in _Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR)_. Springer, 2006, pp. 74–82.
  * [17] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in Neural Information Processing Systems_ , F. Pereira, C. Burges, L. Bottou, and K. Weinberger, Eds., vol. 25. Curran Associates, Inc., 2012. [Online]. Available: https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf
  * [18] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , September 2018.
  * [19] C. Liu, L.-C. Chen, F. Schroff, H. Adam, W. Hua, A. Yuille, and L. Fei-Fei, “Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation,” 2019. [Online]. Available: https://arxiv.org/abs/1901.02985
  * [20] H. Wang, Y. Zhu, H. Adam, A. Yuille, and L.-C. Chen, “Max-deeplab: End-to-end panoptic segmentation with mask transformers,” 2020. [Online]. Available: https://arxiv.org/abs/2012.00759
  * [21] S. Qiao, Y. Zhu, H. Adam, A. Yuille, and L.-C. Chen, “Vip-deeplab: Learning visual perception with depth-aware video panoptic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 3997–4008.
  * [22] B. Cheng, M. D. Collins, Y. Zhu, T. Liu, T. S. Huang, H. Adam, and L.-C. Chen, “Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 12 475–12 485.
  * [23] H. Wang, Y. Zhu, B. Green, H. Adam, A. Yuille, and L.-C. Chen, “Axial-deeplab: Stand-alone axial-attention for panoptic segmentation,” in _European Conference on Computer Vision_. Springer, 2020, pp. 108–126.
  * [24] Q. Yu, H. Wang, D. Kim, S. Qiao, M. Collins, Y. Zhu, H. Adam, A. Yuille, and L.-C. Chen, “Cmt-deeplab: Clustering mask transformers for panoptic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 2560–2570.
  * [25] Y. Meng, Z. Du, C. Zhao, M. Dong, D. Pienta, Z. Xu, and W. Zhou, “Automatic extraction of coronary arteries using deep learning in invasive coronary angiograms,” 2022. [Online]. Available: https://arxiv.org/abs/2206.12300
  * [26] H. Huang, L. Lin, R. Tong, H. Hu, Q. Zhang, Y. Iwamoto, X. Han, Y.-W. Chen, and J. Wu, “Unet 3+: A full-scale connected unet for medical image segmentation,” in _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 1055–1059.
  * [27] Z. Zhou, M. M. Rahman Siddiquee, N. Tajbakhsh, and J. Liang, “Unet++: A nested u-net architecture for medical image segmentation,” in _Deep learning in medical image analysis and multimodal learning for clinical decision support_. Springer, 2018, pp. 3–11.
  * [28] Z. Zhang, Q. Liu, and Y. Wang, “Road extraction by deep residual u-net,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 15, no. 5, pp. 749–753, 2018.
  * [29] Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3d u-net: learning dense volumetric segmentation from sparse annotation,” in _International conference on medical image computing and computer-assisted intervention_. Springer, 2016, pp. 424–432.
  * [30] F. Isensee, J. Petersen, A. Klein, D. Zimmerer, P. F. Jaeger, S. Kohl, J. Wasserthal, G. Koehler, T. Norajitra, S. Wirkert, _et al._ , “nnu-net: Self-adapting framework for u-net-based medical image segmentation,” _arXiv preprint arXiv:1809.10486_ , 2018.
  * [31] O. Oktay, J. Schlemper, L. L. Folgoc, M. Lee, M. Heinrich, K. Misawa, K. Mori, S. McDonagh, N. Y. Hammerla, B. Kainz, _et al._ , “Attention u-net: Learning where to look for the pancreas,” _arXiv preprint arXiv:1804.03999_ , 2018.
  * [32] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” 2015. [Online]. Available: https://arxiv.org/abs/1505.04597
  * [33] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 4, pp. 834–848, 2017.
  * [34] T. Ozturk, M. Talo, E. A. Yildirim, U. B. Baloglu, O. Yildirim, and U. R. Acharya, “Automated detection of covid-19 cases using deep neural networks with x-ray images,” _Computers in biology and medicine_ , vol. 121, p. 103792, 2020.
  * [35] E. Grøvik, D. Yi, M. Iv, E. Tong, L. B. Nilsen, A. Latysheva, C. Saxhaug, K. D. Jacobsen, Å. Helland, K. E. Emblem, _et al._ , “Handling missing mri sequences in deep learning segmentation of brain metastases: a multicenter study,” _NPJ digital medicine_ , vol. 4, no. 1, pp. 1–7, 2021\.
  * [36] A. Negi, A. N. J. Raj, R. Nersisson, Z. Zhuang, and M. Murugappan, “Rda-unet-wgan: an accurate breast ultrasound lesion segmentation using wasserstein generative adversarial networks,” _Arabian Journal for Science and Engineering_ , vol. 45, no. 8, pp. 6399–6410, 2020.
  * [37] D. Li, D. A. Dharmawan, B. P. Ng, and S. Rahardja, “Residual u-net for retinal vessel segmentation,” in _2019 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2019, pp. 1425–1429.
  * [38] K. Iyer, C. P. Najarian, A. A. Fattah, C. J. Arthurs, S. R. Soroushmehr, V. Subban, M. A. Sankardas, R. R. Nadakuditi, B. K. Nallamothu, and C. A. Figueroa, “Angionet: A convolutional neural network for vessel segmentation in x-ray angiography,” _medRxiv_ , 2021. [Online]. Available: https://www.medrxiv.org/content/early/2021/01/26/2021.01.25.21250488
  * [39] Çiğdem Sazak, C. J. Nelson, and B. Obara, “The multiscale bowler-hat transform for blood vessel enhancement in retinal images,” _Pattern Recognition_ , vol. 88, pp. 739–750, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0031320318303558
  * [40] J. Staal, M. Abramoff, M. Niemeijer, M. Viergever, and B. van Ginneken, “Ridge-based vessel segmentation in color images of the retina,” _IEEE Transactions on Medical Imaging_ , vol. 23, no. 4, pp. 501–509, 2004.
  * [41] A. Hoover, V. Kouznetsova, and M. Goldbaum, “Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response,” _IEEE Transactions on Medical imaging_ , vol. 19, no. 3, pp. 203–210, 2000\.
  * [42] F. Cervantes-Sanchez, I. Cruz-Aceves, A. Hernandez-Aguirre, M. A. Hernandez-Gonzalez, and S. E. Solorio-Meza, “Automatic segmentation of coronary arteries in x-ray angiograms using multiscale analysis and artificial neural networks,” _Applied Sciences_ , vol. 9, no. 24, 2019. [Online]. Available: https://www.mdpi.com/2076-3417/9/24/5507
  * [43] D. Hao, S. Ding, L. Qiu, Y. Lv, B. Fei, Y. Zhu, and B. Qin, “Sequential vessel segmentation via deep channel attention network,” _Neural Networks_ , vol. 128, pp. 172–187, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0893608020301672
  * [44] S. Mulay, K. Ram, B. Murugesan, and M. Sivaprakasam, “Style transfer based coronary artery segmentation in x-ray angiogram,” in _2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)_ , 2021, pp. 3386–3394.
  * [45] Z. Gao, L. Wang, R. Soroushmehr, A. Wood, J. Gryak, B. Nallamothu, and K. Najarian, “Vessel segmentation for x-ray coronary angiography using ensemble methods with deep learning and filter-based features,” _BMC Medical Imaging_ , vol. 22, no. 1, pp. 1–17, 2022.
  * [46] Z. Rahman, D. Jobson, and G. Woodell, “Multi-scale retinex for color image enhancement,” in _Proceedings of 3rd IEEE International Conference on Image Processing_ , vol. 3, 1996, pp. 1003–1006 vol.3.
  * [47] E. D. Pisano, S. Zong, B. M. Hemminger, M. DeLuca, R. E. Johnston, K. Muller, M. P. Braeuning, and S. M. Pizer, “Contrast limited adaptive histogram equalization image processing to improve the detection of simulated spiculations in dense mammograms,” _Journal of Digital imaging_ , vol. 11, no. 4, pp. 193–200, 1998.
  * [48] A. W. Setiawan, T. R. Mengko, O. S. Santoso, and A. B. Suksmono, “Color retinal image enhancement using clahe,” in _International Conference on ICT for Smart Society_ , 2013, pp. 1–3.
  * [49] N. Reamaroon, M. W. Sjoding, H. Derksen, E. Sabeti, J. Gryak, R. P. Barbaro, B. D. Athey, and K. Najarian, “Robust segmentation of lung in chest x-ray: applications in analysis of acute respiratory distress syndrome,” _BMC Medical Imaging_ , vol. 20, no. 1, pp. 1–13, 2020.
  * [50] Z. Jiang, C. Ou, Y. Qian, R. Rehan, and A. Yong, “Coronary vessel segmentation using multiresolution and multiscale deep learning,” _Informatics in Medicine Unlocked_ , vol. 24, p. 100602, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2352914821000927
  * [51] A. F. Frangi, W. J. Niessen, K. L. Vincken, and M. A. Viergever, “Multiscale vessel enhancement filtering,” in _Medical Image Computing and Computer-Assisted Intervention — MICCAI’98_ , W. M. Wells, A. Colchester, and S. Delp, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 1998, pp. 130–137.
  * [52] S. Zhang, M. Zhu, and K. Meng, “An automated multi-scale retinex for dim image enhancement,” in _2022 IEEE 2nd International Conference on Power, Electronics and Computer Applications (ICPECA)_ , 2022, pp. 647–651.
  * [53] X. Liu, F. Hou, H. Qin, and A. Hao, “Robust optimization-based coronary artery labeling from x-ray angiograms,” _IEEE Journal of Biomedical and Health Informatics_ , vol. 20, no. 6, pp. 1608–1620, 2016.
  * [54] H. Zhao, K. Li, N. Chen, K. Zhang, L. Wang, R. Lin, X. Gong, L. Song, Z. Liu, and C. Liu, “Multiscale vascular enhancement filter applied to in vivo morphologic and functional photoacoustic imaging of rat ocular vasculature,” _IEEE Photonics Journal_ , vol. 11, no. 6, pp. 1–12, 2019.
  * [55] Y.-C. Tsai, H.-J. Lee, and M. Y.-C. Chen, “Automatic segmentation of vessels from angiogram sequences using adaptive feature transformation,” _Computers in biology and medicine_ , vol. 62, pp. 239–253, 2015.
  * [56] S. Zai and A. Abbas, “An effective enhancement and segmentation of coronary arteries in 2d angiograms,” in _2018 International Conference on Smart Computing and Electronic Enterprise (ICSCEE)_. IEEE, 2018, pp. 1–4.
  * [57] M. A. Ansari, S. Zai, and Y. S. Moon, “Performance comparison of vesselness measures for segmentation of coronary arteries in 2d angiograms,” _Indian Journal of Science and Technology_ , vol. 9, no. 48, 2016.
  * [58] I. Cruz Aceves, A. Hernández Aguirre, and J. G. Aviña Cervantes, “Automatic segmentation of coronary arteries using a multiscale top-hat operator and multiobjective optimization,” _Nova scientia_ , vol. 7, no. 15, pp. 297–320, 2015.
  * [59] F. Zana and J.-C. Klein, “Segmentation of vessel-like patterns using mathematical morphology and curvature evaluation,” _IEEE Transactions on Image Processing_ , vol. 10, no. 7, pp. 1010–1019, 2001.
  * [60] T. Park, S. Khang, H. Jeong, K. Koo, J. Lee, J. Shin, and H. C. Kang, “Deep learning segmentation in 2d x-ray images and non-rigid registration in multi-modality images of coronary arteries,” _Diagnostics_ , vol. 12, no. 4, 2022. [Online]. Available: https://www.mdpi.com/2075-4418/12/4/778
  * [61] T. Wan, X. Shang, W. Yang, J. Chen, D. Li, and Z. Qin, “Automated coronary artery tree segmentation in x-ray angiography using improved hessian based enhancement and statistical region merging,” _Computer Methods and Programs in Biomedicine_ , vol. 157, pp. 179–190, 2018. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0169260717300081
  * [62] C. Ciusdel, A. Turcea, A. Puiu, L. Itu, L. Calmac, E. Weiss, C. Margineanu, E. Badila, M. Berger, T. Redel, T. Passerini, M. Gulsun, and P. Sharma, “Deep neural networks for ecg-free cardiac phase and end-diastolic frame detection on coronary angiographies,” 2018. [Online]. Available: https://arxiv.org/abs/1811.02797
  * [63] C. Cong, Y. Kato, H. D. Vasconcellos, J. Lima, and B. Venkatesh, “Automated stenosis detection and classification in x-ray angiography using deep neural network,” in _2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_ , 2019, pp. 1301–1308.
  * [64] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” 2015. [Online]. Available: https://arxiv.org/abs/1512.00567
  * [65] H. Ma, P. Ambrosini, and T. van Walsum, “Fast prospective detection of contrast inflow in x-ray angiograms with convolutional neural network and recurrent neural network,” in _Medical Image Computing and Computer Assisted Intervention - MICCAI 2017_. Springer International Publishing, 2017, pp. 453–461. [Online]. Available: https://doi.org/10.1007%2F978-3-319-66179-7˙52
  * [66] Y. Meng, M. Dong, X. Dai, H. Tang, C. Zhao, J. Jiang, S. Xu, Y. Zhou, F. Zhu1, Z. Xu, and W. Zhou, “Automatic identification of the end-diastolic and end-systolic cardiac frames from invasive coronary angiography videos,” 2021\. [Online]. Available: https://arxiv.org/abs/2110.02844
  * [67] W. Wu, J. Zhang, H. Xie, Y. Zhao, S. Zhang, and L. Gu, “Automatic detection of coronary artery stenosis by convolutional neural network with temporal constraint,” _Computers in Biology and Medicine_ , vol. 118, p. 103657, 2020\. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0010482520300512
  * [68] A. Kerkeni, A. Benabdallah, A. Manzanera, and M. H. Bedoui, “A coronary artery segmentation method based on multiscale analysis and region growing,” _Computerized Medical Imaging and Graphics_ , vol. 48, pp. 49–61, 2016. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0895611115001913
  * [69] T. Coye, “Novel retinal vessel segmentation algorithm: Fundus images,” 2015. [Online]. Available: https://www.mathworks.com/matlabcentral/fileexchange/50839-novel-retinal-vessel-segmentation-algorithm-fundus-images
  * [70] N. Otsu, “A threshold selection method from gray-level histograms,” _IEEE Transactions on Systems, Man, and Cybernetics_ , vol. 9, no. 1, pp. 62–66, 1979.
  * [71] F.-J. Neumann, M. Sousa-Uva, A. Ahlsson, F. Alfonso, A. P. Banning, U. Benedetto, R. A. Byrne, J.-P. Collet, V. Falk, S. J. Head, P. Jüni, A. Kastrati, A. Koller, S. D. Kristensen, J. Niebauer, D. J. Richter, P. M. Seferović, D. Sibbing, G. G. Stefanini, S. Windecker, R. Yadav, M. O. Zembala, and E. S. D. Group, “2018 ESC/EACTS Guidelines on myocardial revascularization,” _European Heart Journal_ , vol. 40, no. 2, pp. 87–165, 08 2018. [Online]. Available: https://doi.org/10.1093/eurheartj/ehy394
  * [72] Q. Zhang and Q. Wang, “Comparison of semantic segmentation methods on renal ultrasounds images,” in _2022 IEEE International Instrumentation and Measurement Technology Conference (I2MTC)_. IEEE, 2022, pp. 1–5.
  * [73] A. Garcia-Uceda Juarez, R. Selvan, Z. Saghir, and M. d. Bruijne, “A joint 3d unet-graph neural network-based method for airway segmentation from chest cts,” in _International workshop on machine learning in medical imaging_. Springer, 2019, pp. 583–591.
  * [74] S. Yang, J. Kweon, J.-H. Roh, J.-H. Lee, H. Kang, L.-J. Park, D. J. Kim, H. Yang, J. Hur, D.-Y. Kang, _et al._ , “Deep learning segmentation of major vessels in x-ray coronary angiography,” _Scientific reports_ , vol. 9, no. 1, pp. 1–11, 2019.
  * [75] X. Zhan-chao, X. Wang, S. Yan, D. Yang, J. Chen, and C. Peng, “Main coronary vessel segmentation using deep learning in smart medical,” _Mathematical Problems in Engineering_ , vol. 2020, pp. 1–9, 2020.
  * [76] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 770–778.
  * [77] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 2261–2269.
  * [78] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” 2016. [Online]. Available: https://arxiv.org/abs/1602.07261
  * [79] F. Wang, M. Jiang, C. Qian, S. Yang, C. Li, H. Zhang, X. Wang, and X. Tang, “Residual attention network for image classification,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 6450–6458.
  * [80] T. Du, L. Xie, H. Zhang, X. Liu, X. Wang, D. Chen, Y. Xu, Z. Sun, W. Zhou, L. Song, _et al._ , “Training and validation of a deep learning architecture for the automatic analysis of coronary angiography,” _EuroIntervention_ , vol. 17, no. 1, pp. 32–40, 2021.
  * [81] X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” 2017. [Online]. Available: https://arxiv.org/abs/1703.06868
  * [82] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “Cbam: Convolutional block attention module,” 2018. [Online]. Available: https://arxiv.org/abs/1807.06521
  * [83] X. Soria, E. Riba, and A. Sappa, “Dense extreme inception network: Towards a robust cnn model for edge detection,” in _2020 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2020, pp. 1912–1921.
  * [84] P. Arbeláez, M. Maire, C. Fowlkes, and J. Malik, “Contour detection and hierarchical image segmentation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 33, no. 5, pp. 898–916, 2011.
  * [85] J. H. Friedman, “Greedy function approximation: A gradient boosting machine.” _The Annals of Statistics_ , vol. 29, no. 5, pp. 1189 – 1232, 2001. [Online]. Available: https://doi.org/10.1214/aos/1013203451
  * [86] Z.-H. Zhou and J. Feng, “Deep forest,” 2017. [Online]. Available: https://arxiv.org/abs/1702.08835
  * [87] T. Han, Y. Bian, R. An, Y. Han, D. Ai, and J. Yang, “Direction- and centerline-aware joint learning network (jlnet) for vessel segmentation in x-ray angiography images,” in _2021 5th International Conference on Digital Signal Processing_ , ser. ICDSP 2021. New York, NY, USA: Association for Computing Machinery, 2021, p. 13–19. [Online]. Available: https://doi.org/10.1145/3458380.3458383
  * [88] T. Han, D. Ai, Y. Wang, Y. Bian, R. An, J. Fan, H. Song, H. Xie, and J. Yang, “Recursive centerline- and direction-aware joint learning network with ensemble strategy for vessel segmentation in x-ray angiography images,” _Computer Methods and Programs in Biomedicine_ , vol. 220, p. 106787, 2022\. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0169260722001730
  * [89] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 779–788.
  * [90] C.-Y. Wang, A. Bochkovskiy, and H.-Y. M. Liao, “Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors,” 2022\. [Online]. Available: https://arxiv.org/abs/2207.02696
  * [91] C.-Y. Fu, W. Liu, A. Ranga, A. Tyagi, and A. C. Berg, “Dssd : Deconvolutional single shot detector,” 2017. [Online]. Available: https://arxiv.org/abs/1701.06659
  * [92] L. Cui, R. Ma, P. Lv, X. Jiang, Z. Gao, B. Zhou, and M. Xu, “Mdssd: multi-scale deconvolutional single shot detector for small objects,” _arXiv preprint arXiv:1805.07009_ , 2018.
  * [93] W. Shi, S. Bao, and D. Tan, “Ffessd: An accurate and efficient single-shot detector for target detection,” _Applied Sciences_ , vol. 9, no. 20, p. 4276, 2019.
  * [94] K. Sathish, S. Ramasubbareddy, and K. Govinda, “Detection and localization of multiple objects using vggnet and single shot detection,” in _emerging research in data engineering systems and computer communications_. Springer, 2020, pp. 427–439.
  * [95] R. Huang, J. Pedoeem, and C. Chen, “Yolo-lite: a real-time object detection algorithm optimized for non-gpu computers,” in _2018 IEEE International Conference on Big Data (Big Data)_. IEEE, 2018, pp. 2503–2510.
  * [96] M. J. Shafiee, B. Chywl, F. Li, and A. Wong, “Fast yolo: A fast you only look once system for real-time embedded object detection in video,” _arXiv preprint arXiv:1709.05943_ , 2017.
  * [97] K. Pang, D. Ai, H. Fang, J. Fan, H. Song, and J. Yang, “Stenosis-detnet: Sequence consistency-based stenosis detection for x-ray coronary angiography,” _Computerized Medical Imaging and Graphics_ , vol. 89, p. 101900, 2021.