:
149 2021 Machine Learning for Healthcare

# An Interpretable Framework for Drug-Target Interaction with Gated Cross
Attention

Yeachan Kim<EMAIL_ADDRESS>
Bonggun Shin<EMAIL_ADDRESS>
Deargen Inc  Seoul  South Korea

###### Abstract

In silico prediction of drug-target interactions (DTI) is significant for drug
discovery because it can largely reduce timelines and costs in the drug
development process. Specifically, deep learning-based DTI approaches have
been shown promising results in terms of accuracy and low cost for the
prediction. However, they pay little attention to the interpretability of
their prediction results and feature-level interactions between a drug and a
target. In this study, we propose a novel interpretable framework that can
provide reasonable cues for the interaction sites. To this end, we elaborately
design a gated cross-attention mechanism that crossly attends drug and target
features by constructing explicit interactions between these features. The
gating function in the method enables neural models to focus on salient
regions over entire sequences of drugs and proteins, and the byproduct from
the function, which is the attention map, could serve as interpretable
factors. The experimental results show the efficacy of the proposed method in
two DTI datasets. Additionally, we show that gated cross-attention can
sensitively react to the mutation, and this result could provide insights into
the identification of novel drugs targeting mutant proteins.

##  1 Introduction

Predicting the interactions between drugs and targets is a critical step in
drug discovery. The importance is more evident in repurposing (or
repositioning) already-approved drugs for fast-spreading and epidemic diseases
(e.g., COVID-19) because the approval process of a new drug requires
significant development timelines and overall costs. Recently, deep neural
networks have shown promising results on this problem (Öztürk et al., 2018;
Shin et al., 2019), demonstrating that they can work well in real-world
scenarios (Beck et al., 2020; Choi et al., 2020).

Most recent studies for predicting the interaction focus on representation
learning for drugs and targets. For example, Shin et al. (2019) used a
transformer (Vaswani et al., 2017) to effectively capture salient features
from molecules and further improved the generalization capability through a
pre-training phase on a large-scale database (e.g., PubChem (Kim et al.,
2019)). Nguyen et al. (2019) replaced the sequential representations of drugs
with a molecular graph using graph convolutional networks (GCNs). Although
advancements in representation learning have improved the performance, most
studies pay little attention to $i)$ the interpretability of their prediction
results and $ii)$ the interaction of drug and target features.

To further improve the current approaches, we propose a gated cross-attention
(GCA), which is a novel interpretable and interaction framework that can
provide the interpretability of the prediction results through an explicit
interaction between the drug and target features. Interpretability is served
through the lens of attention to drugs and targets. In other words, the higher
attention score at the specific parts of the drug and target, the more
significant they are for prediction. To that end, we redesign the previous
attention mechanism as a gating function so that the significant parts for the
prediction easily affect to the prediction. Furthermore, unlike the previous
attention mechanism that typically assigns all credits to all inputs, GCA
sparsely attends a small portion of inputs to encourage the neural models to
focus on salient parts over sequences.

To validate the efficacy of GCA, we apply the proposed method to existing
neural models (e.g., convolutional neural networks (CNNs), GCNs) because the
role of our method is to put current methods on a more solid footing by
obtaining interpretability on their prediction. The comprehensive results
clearly demonstrate that the proposed interaction networks not only provide
the interpretable cues for the prediction by showing significant parts in
drugs and targets, but also achieve performance improvement owing to the
explicit interaction between features. We also conduct two case studies on
mutations and possible interactions between drugs and targets and it can be
the pharmaceutical use cases of the proposed method.

Clinical Relevance: The proposed method can help design new drugs by proposing
a binding affinity with important regions of drugs and targets. It can also be
used in drug design targeting specific mutant proteins by checking whether the
attention in the mutation sites is activated differently.

### Generalizable Insights about Machine Learning in the Context of Healthcare

Due to the large length of protein sequences, effectively handling protein
representation is difficult. In this case, focusing salient parts over the
large sequence is significant, and the sparse attention mechanism can
alleviate the problem. There are several tasks taking two inputs as an input,
such as drug-drug interaction and protein-protein interaction. Cross-attention
mechanism over feature extraction layers could improve the performance in
tasks while providing interaction interpretability.

##  2 Related Work

###  2.1 Drug Target Interaction

In silico prediction of DTI has been getting attention since it could provide
useful guidance to drug discovery with reduced overall costs. The major
approaches to DTI are three-folds which are molecular docking (Trott and
Olson, 2010; Luo et al., 2016), similarity-based (Pahikkala et al., 2015; He
et al., 2017) and deep learning-based approaches. In this paper, we mainly
focus on deep learning-based methods since they could provide the results
promptly and achieve reasonably accurate results compared to other approaches
(Öztürk et al., 2018). In the beginning, several works utilized deep neural
networks over character representations of drugs and targets. For drugs,
SMILES (Simplified Molecular Input Line Entry System) format is utilized to
represent chemical compound, and protein sequence (i.e., amino acids) is
represented by FASTA (Lipman and Pearson, 1985). Based on such character
representations, Öztürk et al. (2018) proposed DeepDTA that utilizes
convolutional neural networks (CNNs) to extract locally salient features from
drug and target, and it achieved better performance than similarity-based
approaches. Shin et al. (2019) used the same character representation with
DeepDTA, but utilized the self-attention-based transformer (Vaswani et al.,
2017) to consider a long-range relationship between atoms. Furthermore, they
performed transfer learning to further generalize the molecule representations
by pre-training the transformer on a large-scale database. Instead of
character representations, (Öztürk et al., 2019) represented drug and target
as words that correspond to substructure in compounds. Beyond such string
representations, utilizing graph structures of drugs shows promising
performance than others (Lim et al., 2019; Nguyen et al., 2019; Zhao et al.,
2020).

The most related work to ours is Gao et al. (2018). It shares a similar
strategy with ours in the aspect of attending to two inputs. However, our
method adopts a multi-head attention mechanism in the transformer. It allows
neural models to attend to different aspects of information that lie in input
features. Thus, it serves more options to interpret the prediction results.
For example, we found out that one of the heads from GCA is more committed to
predicting the binding site than other heads (detailed in Section 5.2). Beyond
the differences in architecture, the proposed model constraints the attention
map to be sparse, which guides the model to more focus salient parts in
inputs.

###  2.2 Attention Method in Neural Networks

Attention mechanism was invented to improve the quality of machine translation
by aligning two different representations (e.g., source and target languages)
(Bahdanau et al., 2015; Luong et al., 2015). There are two advantages in
utilizing attention: First, it allows neural models to effectively consider a
long-range relationship between features and it leads to the improved task
performance (Yang et al., 2016; Anderson et al., 2018). Another advantage of
the attention mechanism is that it could provide the interpretability of the
model prediction (Seo et al., 2017; Annasamy and Sycara, 2019). In the
interaction task of different domains, several works utilize the attention
mechanism to provide interpretability. For example, Santos et al. (2016)
utilized pooling operation over the interactive attention maps between
question and answer for an answer selection task. Similarly, Gong et al.
(2018) used densely-connected networks over attention maps to extract locally
salient interactions between premise and hypothesis for a natural language
inference task. For a molecule optimization task, Shin et al. (2021) utilized
a self-attention-based transformer to optimize multiple properties of a given
molecule.

For drug-target interaction, most works have shown the strengths of the
attention mechanism at generating better representations. However, they pay
little attention to the interpretability and interaction between two inputs.
In this work, we focus on the above properties to improve the task performance
and provide reasonable cues for the factors to the interaction binding, which
may have broad pharmaceutical applications. Note that our method is parallel
research lines with other works. Thus, we believe that existing methods could
benefit from ours.

##  3 Gated Cross-Attention Networks

Figure 1: Overview of gated cross attention (GCA) networks and the detailed
procedures when deriving the protein attention (right). GCA explicitly
constructs the interaction between drug and target features, which are
obtained from each feature extractor, using multi-head gated attention. The
attention map, which is the by-product of the explicit interaction, serves as
an interpretable factor for the prediction results.

In this section, we elaborate the proposed method which is denoted as gated
cross attention (GCA). Before delving into the method, we first describe the
background of drug-target interaction and formally define the task (Section
3.1). Then, we introduce the attention mechanism in the transformer with some
negative observation at the interaction task (Section 3.2.1). Lastly, we
detail gated cross attention (Section 3.2.2). The overall procedures are
straightforward and illustrated in Figure 1. The main strategy of GCA is to
generate drug and target features by considering counterpart information. The
name cross in our method comes from the directionality of the attention in our
method

###  3.1 Drug-Target Interaction

The task of this paper is to predict the affinity between a drug and a protein
(also noted as target). The affinity can be measured as either scores or
categories, the task of which is then regression or classification,
respectively. Since predicting affinity scores is much beneficial in the
process of drug discovery (Öztürk et al., 2018), we focus on regression-based
modeling. Formally, the inputs are two kinds: a molecule represented by the
SMILES (Weininger, 1988) sequence or the molecular graph, and a protein
represented by the FASTA (Lipman and Pearson, 1985) sequence. A FASTA sequence
is a series of various amino acids (also noted as residues).

The typical neural frameworks for drug-target interaction consist of two major
components: 1) two feature extraction layers for drugs and targets, and 2) a
single fully-connected layer to predict binding affinity scores based on the
resultant features from 1). Let the drug and target features from each feature
extractor be $d_{1:n_{d}}$ and $p_{1:n_{p}}$ where $n_{d},n_{p}$ are the
sequential length of each feature and the subscript indicates the sequential
range of features, we formally define the task as follows:

$r_{d}=pooling(d_{1:n_{d}}),\text{ }r_{p}=pooling(p_{1:n_{p}}),$ (1)

$\hat{y}=\phi(r_{d},r_{p};\theta_{i}).$ (2)

where $pooling(\cdot)$ is the pooling methods such as global max pooling and
average pooling, which depends on the architecture. $\hat{y}$ is the predicted
binding affinity score by an interaction layer $\phi(\cdot)$ with its
parameter $\theta_{i}$. Most works used simple feed-forward networks for an
interaction function. In this work, we enhance the interaction parts
$\phi(\cdot)$ by explicitly making the interaction between drugs $d_{1:n_{d}}$
and targets $p_{1:n_{p}}$.

###  3.2 Cross Attention

To explicitly consider the interaction, we propose a cross-attention that
considers protein representations to attend the drug representations and vice
versa. We realize this by cross-attending each feature by its counterpart
representations. We formalize this as follows:

$d^{{}^{\prime}}_{1:n_{d}}=g_{p\rightarrow
d}(d_{1:n_{d}},p_{1:n_{p}};\theta_{p\rightarrow d}),$ (3)
$p^{{}^{\prime}}_{1:n_{p}}=g_{d\rightarrow
p}(p_{1:n_{p}},d_{1:n_{d}};\theta_{d\rightarrow p}).$ (4)

where $g(\cdot)$ indicates an attention function that considers counterpart
information. For example, $g_{p\rightarrow d}$ takes protein features to
attend drug features with its learnable parameters $\theta_{p\rightarrow d}$
and returns the attended features of $d_{1:n_{d}}$. The attention weights,
which are the byproducts of $g(\cdot)$, can serve as interpretable factors
because the weights exhibit the most contributing sequences (i.e., amino acids
and atoms) to predict affinity. In the following section, we will describe the
functions are suitable for $g(\cdot)$.

####  3.2.1 Multi-head Attention

As an attention function $g$, we follow multi-head attention in the
transformer. There are two different types of attention: 1) encoder attention
and 2) decoder attention since the transformer was constructed for a sequence-
to-sequence task (e.g., machine translation). We only describe the process of
calculating attention for drug features $d_{1:n_{d}}$ because the attention
for a protein is symmetric with that of drugs.

### Encoder Attention

The key idea of the encoder attention is to attend inputs by using the same
inputs but differently transformed, which is also called self-attention. This
attention is repeatedly performed with different heads and layers. For
simplicity, we show the process of a single layer with a single head.

We first construct query, key, and value by transforming the same inputs
$d_{1:n_{d}}$ by weight matrices $W_{Q}$, $W_{K}$, and $W_{V}$. These features
are denoted as $Q_{d}$, $K_{d}$, and $V_{d}$, respectively and share the same
dimensionality with the inputs (i.e., $Q,K,V\in\mathbbm{R}^{n_{d}\times b}$).
The attention weights $A\in\mathbb{R}^{n_{d}\times n_{d}}$ are computed based
on the query and key:

$A=\text{softmax}(\frac{Q_{d}K_{d}^{T}}{\sqrt{b}}),$ (5)

where $b$ is the dimensionality of input features and it serves as temperature
to relax the distribution of the attention. Using the attention weights $A$
and the values $V_{d}$, we compute attended inputs as follows:

$V_{d}^{{}^{\prime}}=AV_{d}.$ (6)

Here, $A$ can be utilized as interpretable factor to the prediction. The above
encoder attention is widely utilized at feature extraction layers in various
domains, such as image (Cornia et al., 2020), natural language processing
(Devlin et al., 2019), and drug-discovery (Shin et al., 2019, 2021).

### Decoder Attention

Decoder attention is slightly different from the self-attention. This
attention is used at the decoder in the transformer, and the goal is to align
decoder features with encoder features. To align two different features, it
uses the counterpart features $p_{1:n_{p}}$ instead of its inputs
$d_{1:n_{d}}$ to construct the key and value, which are denoted as $K_{p}$ and
$V_{p}$, and the attended features in Eq. 6 are obtained as follows:

$V_{d}^{{}^{\prime}}=\text{softmax}(\frac{Q_{d}K_{p}^{T}}{\sqrt{b}})V_{p},$
(7)

The decoder attention can be used as a $g_{p}\rightarrow{d}(\cdot)$ in the
cross attention. However, we empirically found that the decoder attention
results in highly mixed representations between the two features. Figure 2(a)
shows the pairwise feature similarity between drugs ($d,d^{{}^{\prime}}$) and
proteins ($p,p^{{}^{\prime}}$) which are inputs and outputs from the decoder
attention-based $g(\cdot)$ function. We observe that the protein features
$p^{{}^{\prime}}$ from the decoder attention are more similar to drug features
$d$ than its input features $p$ over several training ranges (e.g., 3k-18k and
near 30k) while the attended features for drugs are more similar to its input
than protein features. In other words, a specific modality could dominate the
other, and we found that such cases lead to poor performance in the task.
These results are derived from the method of attention weighting in Eq. 7
because the attention maps $A$ are applied to counterpart features rather than
its input features. This is the intended design for many sequence-to-sequence
tasks, where the goal is to effectively align the target and source. However,
we argue that this should be redesigned for interaction tasks.

(a) Decoder Attention (b) Gated Cross Attention

Figure 2: Pair-wise similarity between drugs ($d,d^{{}^{\prime}}$) and protein
features ($p,p^{{}^{\prime}}$) over different training epochs when using
different attention mechanism. It shows that utilizing decoder attention does
not generate discriminate features between drug and target.

####  3.2.2 Multi-head Gated Attention

For the interaction task, we redesign the decoder attention. It shares the
same strategy of multi-head attention with the decoder. However, it is vastly
different in that 1) It uses context-level attention instead of token-level
attention 2) The attention is applied in the manner of gating mechanism in the
variants of recurrent neural networks (Chung et al., 2014; Bradbury et al.,
2017). The attention weights $a\in\mathbbm{R}^{n_{d}}$ for drug features are
calculated as follows:

$a=\frac{1}{n_{p}}\sum_{i=1}^{n_{p}}\text{softmax}(\frac{Q_{p}K_{d}^{T}}{\sqrt{b}})_{:,i},$
(8)

In comparison to the decoder attention, the query and key inputs are reversed,
and the attention map is aggregated by counterpart information, yielding a
context-level attention vector rather than a token-wise attention matrix (Eq.
5). The attention vector $a$ serves as a gating function in calculating the
attended features with the values $V_{d}$:

$V_{d}^{{}^{\prime}}=\text{softmax}(a)\odot V_{d}.$ (9)

where $\odot$ is an element-wise multiplication operation111If the dimensions
do not match, the operation is performed in a broadcasting manner. We follow
NumPy (Harris et al., 2020) convention
(https://numpy.org/doc/stable/user/theory.broadcasting.html). After applying
attention, we add a residual connection between attention layers, which
creates good local minima (Yun et al., 2019), and follow pre-normalization
(Xiong et al., 2020), as illustrated in Figure 1 (right).

The attended values $V^{{}^{\prime}}$ are $d^{{}^{\prime}}_{1:n_{d}}$, and
these features are pooled over the length $n_{d}$ in Eq. 1. For the protein
side, the attended features $p^{{}^{\prime}}_{1:n_{p}}$ are symmetrically
computed using the above procedure. Lastly, the concatenated features of a
drug and a target are fed to multi-layered feed-forward networks to predict
the binding affinity. To train the overall networks for the drug-target
interaction task, we minimize the mean squared error (MSE) between predicted
affinity scores and ground-truth scores. Figure 2(b) shows that gated
attention generates highly discriminative features while considering
counterpart information.

####  3.2.3 Sparse Attention

Our attention mechanism assigns probabilities (i.e., $a$ in Eq 8 and 9) over
all elements. However, most of them have a small but non-zero probability,
which could weaken the attention given to the few truly significant elements
(Shen et al., 2018; Zhao et al., 2019). In the context of drug-target
interactions, it shares the same problem because a drug interacts with small
portions of a protein. We constrain the attention map to be sparse to
encourage the model to focus on a few salient parts. We achieve this by
replacing the softmax function with the sparsemax (Martins and Astudillo,
2016) function, which considers the small weights to be unimportant and masks
those values from the features when calculating the softmax-based attention.

##  4 Experiments

In this section, we evaluate the performance of our method in a drug-target
interaction task on two different datasets.

###  4.1 Experimental Settings

####  4.1.1 Comparison Methods

As previously stated, the GCA is designed to endow existing methods with
interaction interpretability. Thus we compare the performance by applying our
method to the following baselines:

  * •

EmbDTA: Instead of a specific feature extractor, it only uses an embedding
layer. We add this to see how much the proposed method improves the
performance of the task using only the interaction. It uses SMILES and FASTA
to represent the drugs and targets, respectively.

  * •

DeepDTA (Öztürk et al., 2018): It uses CNNs for feature extraction layers to
encode locally salient features from drugs and targets. It uses the same
representations for drugs and targets as the EmbDTA.

  * •

GraphDTA (Nguyen et al., 2019): It uses GCNs to exploit the properties of the
molecular graph and utilizes CNNs for proteins as in DeepDTA. For an input
representation, it uses atoms in drugs and connects them through graph
information (i.e., adjacency matrix). To preserve sequential information, we
use a sequence-wise convolution instead of feature-wise convolution, which is
similar to Kim (2014) and we empirically found that this version produces
nearly identical results with that of feature-wise convolution.

Note that the previous methods use multi-layered feed-forward networks for an
interaction layer (i.e., $\phi(\cdot)$). We replace the networks with GCA
without modifying the feature extractor on previous methods. For a fair
comparison, we set the nearly same number of parameters used in the
interaction layers.

In our model, we use a single cross-attention layer with two heads. The
training setup is the same with the previous work Nguyen et al. (2019). We
have implemented the proposed method and all experiments with PyTorch (Paszke
et al., 2017) and trained the models on a single NVIDIA Tesla V100 with 32GB
of RAM.

####  4.1.2 Datasets

Table 1: Statistics for KIBA and Davis dataset. Len. indicates the average
string length of each format (SMILES, FASTA).

Dataset # of Drugs # of Targets Len. Drugs (SMILES) Len. Targets (FASTA) # of
Interactions Affinity range KIBA 2,111 229 56.6 656.4 118,254 [0.0, 17.2]
Davis 68 442 62.7 712.2 30,056 [5.0, 10.8]

We use two benchmark datasets to evaluate the performance in the drug-target
interaction task , which are KIBA large-scale kinase inhibitors bioactivity
dataset (Tang et al., 2014; He et al., 2017) and Davis kinase binding affinity
dataset (Davis et al., 2011). The statistics of these datasets are listed in
Table 1.

####  4.1.3 Evaluation Metrics

We evaluate the methods by using mean squared error (MSE) and concordance-
index (C-index) score following previous works (Öztürk et al., 2018; Nguyen et
al., 2019). MSE is the average of regression loss in the test dataset, and
C-index is the probability that randomly selected pairs are correctly ordered.
It is computed as follows:

$\text{C-index}=\frac{1}{N_{test}}\sum_{y_{i}>y_{j}}{\mathbbm{1}(\hat{y_{i}}>\hat{y_{j}})}$
(10)

where $N_{test}$ is the number of test pairs and, $y_{i}$, $y_{j}$ are
affinity scores of randomly selected pairs, $\hat{y_{i}}$, $\hat{y_{i}}$ are
predicted affinity scores from the trained model. $\mathbbm{1}(x)$ is a
identity function that returns 1 if the statement $x$ is true, otherwise 0.

###  4.2 Results

Table 2: Performance evaluation on KIBA and Davis. Best results are
highlighted in boldface.

Method KIBA Davis MSE C-index MSE C-index EmbDTA 0.342 0.761 0.438 0.811
EmbDTA + GCA 0.311 0.784 0.393 0.832 DeepDTA 0.186 0.865 0.246 0.881 DeepDTA +
GCA 0.170 0.881 0.238 0.888 GraphDTA 0.151 0.884 0.251 0.883 GraphDTA + GCA
0.152 0.883 0.242 0.891

Table 3: Comparison with other interaction methods.

Method | MSE | C-index
---|---|---
EmbDTA | 0.448 | 0.811
EmbDTA + DIN | 0.523 | 0.776
EmbDTA + AP | 0.439 | 0.824
EmbDTA + GCA | 0.393 | 0.832

Table 2 shows the full comparison results in two datasets. Replacing the
interaction networks with GCA consistently yields better results for most
methods and datasets. Among them, EmbDTA, which has the least feature
extraction parts, achieves greater performance improvement than others
(roughly 9% improvement in terms of MSE). These results suggest that
constructing explicit interaction is useful to improve the performance in the
task. Furthermore, we see that GCA works well with the use of other input
types (i.e., characters in EmbDTA and DeepDTA, graphs in GraphDTA), and it
shows that GCA can generally be applied to existing methods.

We also compare GCA with different interaction methods used in other domains.
These methods are attentive pooling (AP, Santos et al. (2016)) and deep
interaction networks (DIN, Gong et al. (2018)). Here, we experiment on the
settings of EmbDTA in Davis to solely compare the interaction layers while
minimizing the influence of the feature extractor. Table 3 presents the
comparison results. Again, GCA works better than the others. In addition to
achieving superior performance, we have several advantages. Compared to DIN,
GCA explicitly enables the neural models to interact with a long-range
relationships between drugs and target features owing to the attention
mechanism. AP is the most similar method to GCA, as it also calculates
attention maps and assigns weights to each feature. However, GCA adopts a
multi-head and a multi-layer mechanism using the transformer, making the
method more scalable by stacking multiple interaction layers and jointly
attending to information from different representation subspaces at different
positions (Vaswani et al., 2017). Indeed, we observe that the second head in
GCA better captures the binding sites in proteins than the first head
(detailed in Section 5.2). Such differences contribute to a better
performance.

##  5 Analysis

###  5.1 Ablation Study

Table 4: Ablation study for attention directionality

Method | MSE | C-index
---|---|---
GCA | 0.311 | 0.787
GCA w/o Drug Attention | 0.341 | 0.768
GCA w/o Target Attention | 0.323 | 0.779

In this subsection, we analyze whether cross attention is effective compared
to single direction attention and which attention is more significant for
performance improvement. To this end, we measure the task performance when we
remove the drug ($g_{p\rightarrow d}$) or target attention ($g_{d\rightarrow
p}$), respectively. Here, we use EmbDTA trained on the KIBA dataset. Table 4
lists the ablation results. As can be seen from the table, the performance
decreases as we remove each directionality from GCA. Specifically, removing
drug attention from targets results in the largest drop in performance. This
indicates that drug attention is more significant than target attention. We
also see that using both two types of attention leads to the best performance,
demonstrating that cross-attention is indeed effective and that each attention
complements usefully.

###  5.2 Binding site prediction

(a) Exact prediction (b) Surrounding prediction

Figure 3: Binding site prediction results. Left is the results of exact
prediction. Right is the results of prediction with including nearest 1
position.

The proposed method assigns attention weights to a given protein such that the
highlighted residues directly affect to the task performance. Therefore,
residues receiving relatively high attention can be important local residues
for prediction, such as binding sites. In this subsection, we analyze how the
residues that are highly ranked by the attention correspond to actual binding
sites. Here, we use GraphDTA with GCA, which is trained on KIBA, and use scPDB
(Desaphy et al., 2015) for evaluation, which has atomic-level descriptions of
proteins, ligands, and binding sites from complex structures. It contains
interactions between proteins and pharmacological ligands for 7,179 entries of
Vertebrata (the number of unique proteins is 809). We measure the probability
of whether the top $k$% ranked residues include binding sites or not, and
evaluate two predictions from two different heads in GCA. Figure 3 shows the
prediction results. We first observe that the attention mostly finds the
binding site when searching for the top 10% residues. Furthermore, the second
head from our attention better captures binding sites than first head with a
large margin. This shows that our method encode different aspects of the input
features through multiple heads. These results suggest that our method can
provide the insight in to the interpretability of binding sites by
highlighting nearby residues.

###  5.3 Case Study

####  5.3.1 Interaction Interpretability

In our method, we provide two views on the prediction results: drug attention
and the target attention. In this analysis, we confirm which atoms in drugs
are receiving high attention from our method. To this end, we conduct a case
study of the interaction between chemicals (PubChem ID: 117793281) and protein
(MAP Kinase, UniProt ID: Q9HBH9).

Figure 4: Highlighted atoms in the chemical (PubChem ID: 117793281)

Here, we use GraphDTA with GCA trained in KIBA and draw the top contributing
atoms in Figure 4. As can be seen from the figure, the highlighted atoms in
the drug attention include several nitrogen atoms that can form hydrogen bonds
with amino acids at the binding site. As we use GCNs that interact with
neighbors to represent drugs, several carbons near the nitrogen have received
high attention, and they can also formulate potential solvent interaction
since carbon has a relatively large solvent accessible surface. Overall, the
results show that the highly attended parts of the drug attention can indicate
potential interaction sites in drugs. Such interpretability can help
researchers find leads by displaying the regions near the interaction sites
with a binding affinity score.

####  5.3.2 Mutation on epidermal growth factor receptor

Table 5: Rank of highlighted residues when the T790M mutation occurs

EGFR EGFR with T790M Rank Residue Index Near Residues Rank Residue Index Near
Residues 1 655 GALLL 1 655 GALLL … … … … … … 12 792 TQLMP 5 (+7) 792 MQLMP … …
… … … … 120 791 ITQLM 11 (+109) 791 IMQLM

Lastly, we analyze how the attention from ours is activated differently when a
mutation occurs. We choose the mutation occurring at the epidermal growth
factor receptor (EGFR) which is a well-known genes related to many cancer
types. Specifically, we apply T790M mutation to EGFR, a well-known mutation
found in most lung cancer patients. We use GraphDTA with GCA trained on KIBA.
For a drug, we use Osimertinib which was developed as a third generation
tyrosine kinase inhibitor targeting the T790M mutation in the exon of the EGFR
gene (Soria et al., 2018). In this analysis, we compare the rank of residues
in the target attention when the mutation occurs. Table 5 lists the ranked
results before and after T790M mutation, in descending order of attention
weights. We denote the ranks of the near residues where mutation occurs (i.e.,
791, 792 residue)222As we use CNNs to extract features from proteins, the
surrounding regions of attended parts are meaningful.. When the mutation
occurs, the near residues of the mutation site are highly activated. While the
increased ranking is not the most highlighted, such a drastic increase shows
that GCA can react sensitively to the mutation and can be used as a guideline
for identifying new drugs targeting specific mutant proteins.

##  6 Conclusion

In this paper, we have proposed a novel interpretable framework for drug-
target interactions based on gated cross attention, denoted as GCA. GCA is
elaborately designed to make the neural model explicitly interact with drugs
and targets using the attention mechanism, and it assigns weights to each
feature by considering counterpart information. Because the weights directly
affect the prediction, the values in the attention can serve as an
interpretable factor. The experimental results support the strengths of GCA in
drug-target interaction and provide pharmaceutical insights gained from GCA
for binding sites and potential interactions between drugs and targets.
Furthermore, with the example of T790M EGFR, we have shown that GCA reacts
sensitively to the mutant protein by mainly attending the surrounding mutation
site. The comprehensive results suggest that our method can provide useful
cues for the predictions of deep learning-based DTI methods, and we believe
that existing methods can benefit from our work.

## References

  * Anderson et al. (2018) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang.  Bottom-up and top-down attention for image captioning and visual question answering.  In _Proceedings of of the International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Annasamy and Sycara (2019) Raghuram Mandyam Annasamy and Katia Sycara.  Towards better interpretability in deep q-networks.  In _Proceedings of the Association for the Advancement of Artificial Intelligenc (AAAI)_ , 2019.
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  _Proceedings of the International Conference on Learning Representation (ICLR)_ , 2015.
  * Beck et al. (2020) Bo Ram Beck, Bonggun Shin, Yoonjung Choi, Sungsoo Park, and Keunsoo Kang.  Predicting commercially available antiviral drugs that may act on the novel coronavirus (sars-cov-2) through a drug-target interaction deep learning model.  _Computational and Structural Biotechnology Journal_ , 2020.
  * Bradbury et al. (2017) James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher.  Quasi-recurrent neural networks.  _Proceedings of the International Conference on Learning Representation (ICLR)_ , 2017.
  * Choi et al. (2020) Yoonjung Choi, Bonggun Shin, Keunsoo Kang, Sungsoo Park, and Bo Ram Beck.  Target-centered drug repurposing predictions of human angiotensin-converting enzyme 2 (ace2) and transmembrane protease serine subtype 2 (tmprss2) interacting approved drugs for coronavirus disease 2019 (covid-19) treatment through a drug-target interaction deep learning model.  _Viruses_ , 2020.
  * Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _Proceedings of Neural Information Processing Systems (NeurIPS): Workshop_ , 2014.
  * Cornia et al. (2020) Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara.  Meshed-memory transformer for image captioning.  In _Proceedings of of the International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Davis et al. (2011) Mindy I Davis, Jeremy P Hunt, Sanna Herrgard, Pietro Ciceri, Lisa M Wodicka, Gabriel Pallares, Michael Hocker, Daniel K Treiber, and Patrick P Zarrinkar.  Comprehensive analysis of kinase inhibitor selectivity.  _Nature biotechnology_ , 2011.
  * Desaphy et al. (2015) Jérémy Desaphy, Guillaume Bret, Didier Rognan, and Esther Kellenberger.  sc-pdb: a 3d-database of ligandable binding sites—10 years on.  _Nucleic acids research_ , 2015.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _Proceedings of the International Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_ , 2019\.
  * Gao et al. (2018) Kyle Yingkai Gao, Achille Fokoue, Heng Luo, Arun Iyengar, Sanjoy Dey, and Ping Zhang.  Interpretable drug target prediction using deep neural representation.  In _Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)_ , 2018.
  * Gong et al. (2018) Yichen Gong, Heng Luo, and Jian Zhang.  Natural language inference over interaction space.  _Proceedings of the International Conference on Learning Representation (ICLR)_ , 2018.
  * Harris et al. (2020) Charles R Harris, K Jarrod Millman, Stéfan J van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al.  Array programming with numpy.  _Nature_ , 2020.
  * He et al. (2017) Tong He, Marten Heidemeyer, Fuqiang Ban, Artem Cherkasov, and Martin Ester.  Simboost: a read-across approach for predicting drug–target binding affinities using gradient boosting machines.  _Cheminformatics_ , 2017.
  * Kim et al. (2019) Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin A Shoemaker, Paul A Thiessen, Bo Yu, et al.  Pubchem 2019 update: improved access to chemical data.  _Nucleic acids research_ , 2019.
  * Kim (2014) Yoon Kim.  Convolutional neural networks for sentence classification.  _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2014.
  * Lim et al. (2019) Jaechang Lim, Seongok Ryu, Kyubyong Park, Yo Joong Choe, Jiyeon Ham, and Woo Youn Kim.  Predicting drug–target interaction using a novel graph neural network with 3d structure-embedded graph representation.  _Journal of chemical information and modeling_ , 2019.
  * Lipman and Pearson (1985) David J Lipman and William R Pearson.  Rapid and sensitive protein similarity searches.  _Science_ , 1985.
  * Luo et al. (2016) Heng Luo, William Mattes, Donna L Mendrick, and Huixiao Hong.  Molecular docking for identification of potential targets for drug repurposing.  _Current topics in medicinal chemistry_ , 2016.
  * Luong et al. (2015) Minh-Thang Luong, Hieu Pham, and Christopher D Manning.  Effective approaches to attention-based neural machine translation.  _Proceedings of the International Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2015.
  * Martins and Astudillo (2016) Andre Martins and Ramon Astudillo.  From softmax to sparsemax: A sparse model of attention and multi-label classification.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2016.
  * Nguyen et al. (2019) Thin Nguyen, Hang Le, and Svetha Venkatesh.  Graphdta: prediction of drug–target binding affinity using graph convolutional networks.  _BioRxiv_ , 2019.
  * Öztürk et al. (2018) Hakime Öztürk, Arzucan Özgür, and Elif Ozkirimli.  Deepdta: deep drug–target binding affinity prediction.  _Bioinformatics_ , 2018.
  * Öztürk et al. (2019) Hakime Öztürk, Elif Ozkirimli, and Arzucan Özgür.  Widedta: prediction of drug-target binding affinity.  _arXiv preprint arXiv:1902.04166_ , 2019.
  * Pahikkala et al. (2015) Tapio Pahikkala, Antti Airola, Sami Pietilä, Sushil Shakyawar, Agnieszka Szwajda, Jing Tang, and Tero Aittokallio.  Toward more realistic drug–target interaction predictions.  _Briefings in bioinformatics_ , 2015.
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  2017\.
  * Santos et al. (2016) Cicero dos Santos, Ming Tan, Bing Xiang, and Bowen Zhou.  Attentive pooling networks.  _arXiv preprint arXiv:1602.03609_ , 2016.
  * Seo et al. (2017) Sungyong Seo, Jing Huang, Hao Yang, and Yan Liu.  Interpretable convolutional neural networks with dual local and global attention for review rating prediction.  In _Proceedings of International Conference on recommender systems (RecSys)_ , 2017.
  * Shen et al. (2018) Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Sen Wang, and Chengqi Zhang.  Reinforced self-attention network: a hybrid of hard and soft attention for sequence modeling.  _Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)_ , 2018.
  * Shin et al. (2019) Bonggun Shin, Sungsoo Park, Keunsoo Kang, and Joyce C Ho.  Self-attention based molecule representation for predicting drug-target interaction.  In _Proceedings of Machine Learning for Healthcare (MLHC)_ , pages 230–248. PMLR, 2019.
  * Shin et al. (2021) Bonggun Shin, Sungsoo Park, JinYeong Bak, and Joyce C Ho.  Controlled molecule generator for optimizing multiple chemical properties.  In _Proceedings of the Conference on Health, Inference, and Learning (CHIL)_ , pages 146–153, 2021.
  * Soria et al. (2018) Jean-Charles Soria, Yuichiro Ohe, Johan Vansteenkiste, Thanyanan Reungwetwattana, Busyamas Chewaskulyong, Ki Hyeong Lee, Arunee Dechaphunkul, Fumio Imamura, Naoyuki Nogami, Takayasu Kurata, et al.  Osimertinib in untreated egfr-mutated advanced non–small-cell lung cancer.  _New England journal of medicine_ , 2018.
  * Tang et al. (2014) Jing Tang, Agnieszka Szwajda, Sushil Shakyawar, Tao Xu, Petteri Hintsanen, Krister Wennerberg, and Tero Aittokallio.  Making sense of large-scale kinase inhibitor bioactivity data sets: a comparative and integrative analysis.  _Journal of Chemical Information and Modeling_ , 2014.
  * Trott and Olson (2010) Oleg Trott and Arthur J Olson.  Autodock vina: improving the speed and accuracy of docking with a new scoring function, efficient optimization, and multithreading.  _Journal of computational chemistry_ , 2010.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Proceedings of Neural Information Processing Systems (NeurIPS)_ , 2017.
  * Weininger (1988) David Weininger.  Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules.  _Journal of chemical information and computer sciences_ , 1988.
  * Xiong et al. (2020) Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu.  On layer normalization in the transformer architecture.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2020.
  * Yang et al. (2016) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy.  Hierarchical attention networks for document classification.  In _Proceedings of the International Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_ , 2016\.
  * Yun et al. (2019) Chulhee Yun, Suvrit Sra, and Ali Jadbabaie.  Are deep resnets provably better than linear predictors?  _Proceedings of Neural Information Processing Systems (NeurIPS)_ , 32:15686–15695, 2019.
  * Zhao et al. (2019) Guangxiang Zhao, Junyang Lin, Zhiyuan Zhang, Xuancheng Ren, and Xu Sun.  Sparse transformer: Concentrated attention through explicit selection.  _arXiv preprint arXiv:1912.11637_ , 2019.
  * Zhao et al. (2020) Tianyi Zhao, Yang Hu, Linda R Valsdottir, Tianyi Zang, and Jiajie Peng.  Identifying drug–target interactions based on graph convolutional network and deep neural network.  _Briefings in bioinformatics_ , 2020.


11institutetext:  Medical Imaging, Robotics, Analytic Computing
Laboratory/Engineering (MIRACLE), Key Lab of Intelligent Information
Processing of Chinese Academy of Sciences (CAS), Institute of Computing
Technology, CAS, Beijing 100190, China
11email<EMAIL_ADDRESS>11email: {quanquan,
<EMAIL_ADDRESS>22institutetext: Peng Cheng Laboratory,
Shenzhen, China

# One-Shot Medical Landmark Detection

Qingsong Yao 11 2 2 Quan Quan 11 2 2 Li Xiao 11 S. Kevin Zhou 11 2 2

###### Abstract

The success of deep learning methods relies on the availability of a large
number of datasets with annotations; however, curating such datasets is
burdensome, especially for medical images. To relieve such a burden for a
landmark detection task, we explore the feasibility of using only a single
annotated image and propose a novel framework named Cascade Comparing to
Detect (CC2D) for one-shot landmark detection. CC2D consists of two stages: 1)
Self-supervised learning (CC2D-SSL) and 2) Training with pseudo-labels
(CC2D-TPL). CC2D-SSL captures the consistent anatomical information in a
coarse-to-fine fashion by comparing the cascade feature representations and
generates predictions on the training set. CC2D-TPL further improves the
performance by training a new landmark detector with those predictions. The
effectiveness of CC2D is evaluated on a widely-used public dataset of
cephalometric landmark detection, which achieves a competitive detection
accuracy of 81.01% within 4.0mm, comparable to the state-of-the-art fully-
supervised methods using a lot more than one training image.

###### Keywords:

Medical Landmark Detection One-shot Learning

##  1 Introduction

Accurate and reliable anatomical landmark detection is a fundamental first
step in therapy planning and intervention, thus it has attracted great
interest from academia and industry [24, 26, 31, 30]. It has been proved
crucial in many medical clinical scenarios such as knee joint surgery [25],
bone age estimation [9], carotid artery bifurcation [27], orthognathic and
maxillofacial surgeries [6], and pelvic trauma surgery [3]. Furthermore, it
plays an important role in medical image analysis [15, 18], e.g.,
initialization of registration or segmentation algorithms.

Recently, deep learning based methods have been developed to efficiently
localize anatomical landmarks in radiological images such as computed
tomography (CT) and X-rays [2, 12, 18, 16]. Zhong et al. [28] use cascade
U-Net to launch a two-stage heatmap regression. Chen et al. [6] regress the
heatmap and coordinate offset maps at the same time. Liu et al. [16] improve
the performance by utilizing relative position constraints. Li et al. [13]
adapt cascade deep adaptive graphs to capture relationships among landmarks.

Figure 1: Overview of the proposed Cascade Comparing to Detect (CC2D)
framework. CC2D-SSL and CC2D-TPL represents self-supervised learning and
training with pseudo-labels, respectively.

As is well known, expert-annotated training data are needed for all of those
supervised methods. A common wisdom is that a model with a better
generalization is learned from more data. However, it often consumes
considerable cost and time for the radiologists to annotate sufficient
training data, which might restrict further application in clinical scenarios.
Several self-supervised learning attempts have been explored to break this
limitation in classification and segmentation tasks, including patch ordering
[34], image restoration [33, 32], superpixel-wise [17] and patch-wise [4]
contrastive learning. Nevertheless, training a landmark detection model with
few annotated images is still challenging.

The landmarks are usually associated with anatomical features (e.g. the lips
are halfway between the chin and nose in a cephalometric x-ray), and hence
those features exhibit certain invariance among different images. Therefore,
we pose an interesting question: Can we determine landmarks by learning from a
few or even just one labeled image? In this paper, we challenge the hardest
scenario: _Only one annotated medical image is available_ , which defines the
number of the landmarks and their corresponding locations of interests.

To tackle the challenge, we propose a novel framework named “Cascade Comparing
to Detect (CC2D)” for one-shot landmark detection. CC2D consists of two
stages: 1) Self-supervised learning (CC2D-SSL) and 2) Training with pseudo-
labels (CC2D-TPL). The CC2D-SSL idea is motivated by contrastive learning [7,
29], which learns effective features for image classification. We instead
propose to learn feature representations that embed consistent anatomical
information for image patch matching. Our design is further inspired by our
observation learned through interactions with clinicians that they firstly
roughly locate the target regions through a coarse screening and then
progressively refine the fine-grained location. Therefore, we propose to match
the coarse-grained corresponding areas first, then gradually compare the
finer-grained areas in the selected coarse area. Finally, the targeted
landmark is localized precisely by comparing the cascade embeddings in a
coarse-to-fine fashion.

In the second CC2D-TPL stage, we first use the CC2D-SSL model to generate
predictions for the whole training set as pseudo-labels, followed by training
a CNN-based landmark detector using these pseudo-labels. This step brings two
benefits. On one hand, the inference procedure becomes more concise. On the
other hand, recent findings show that training an over-parameterized network
from scratch tends to learn noiseless information firstly [1]. In our case, as
we cannot predict every training point as accurate as ground truth in the SSL
stage, a newly trained landmark detector can improve the performance by
capturing the regular information hidden from the noisy labels produced by the
CC2D-SSL stage.

We evaluate the performance of CC2D on the public dataset from the ISBI 2015
Grand-Challenge. CC2D achieves competitive detection accuracy of 81.01% within
4.0mm, comparable to the state-of-the-art fully-supervised methods that use a
lot more than one training image.

Figure 2: Overview of the self-supervised learning part (CC2D-SSL). The two
feature extractors are penalized to embed the corresponding landmarks to
similar embeddings in the cascade feature space. The embeddings in the deepest
layer select the coarse-grained area, while the embeddings in the inner layer
improve the localization accuracy by comparing the finer areas in the selected
coarse area. As consequence, the missing landmark is localized precisely in a
coarse-to-fine fashion.

##  2 Method

In this section, we first introduce the mathematical details of the training
and inference stage of the self-supervised learning part of CC2D in Section
2.1, respectively. Then, we illustrate how to train a new landmark detector
from scratch with pseudo-labels in Section 2.2, which are the predictions of
CC2D-SSL on the training set. The resulting detector is used to predict
results for the test set.

###  2.1 Stage I: Self-supervised Learning (CC2D-SSL)

Training stage: As shown in Fig. 2, for an input image $X_{r}$ resized to
$384\times 384$, we arbitrarily select a target point $P_{r}=(x_{r},y_{r})$
and randomly crop a patch $X_{p}$ with size $192\times 192$ which contains
$P$. Then we apply data augmentation to the content of $X_{p}$ by rotation and
color jittering. The chosen landmark is moved to $P_{p}=(x_{p},y_{p})$. Two
feature extractors $E_{r}$ and $E_{p}$ project the input $X_{r}$ and the patch
$X_{p}$ into a multi-scale feature space, resulting in a cascade of
embeddings, denoted by $F_{r}$ and $F_{p}$ with a length of $L$, respectively.
Here we mark the embedding of $i^{th}$ layer as $F^{i}$. Next, we extract the
feature of the anchor point, denoted by $F_{a}^{i}$, from $F_{p}^{i}$ for each
layer, guided by its corresponding coordinates
$P_{a}^{i}=(x_{p}/2^{i},y_{p}/2^{i})$ which are down-sampled for $i$ times. To
compare the features of input $F_{r}$ with anchor features $F_{a}$, we compute
cosine similarity maps for each layer:

$\displaystyle s^{i}=\frac{\langle F_{a}^{i}\cdot
F_{r}^{i}\rangle}{||F_{a}^{i}||_{2}\cdot||F_{r}^{i}||_{2}},$ (1)

In CC2D-SSL, similarity maps at different scales have different roles. The
deepest one ($s^{5}$ in this paper) differentiates the target coarse-grained
area from every different area in the whole map, while the most shallow ones
are only in charge of distinguishing the target pixel with adjacent pixels.
The cascade similarity maps locate the target point in a coarse-to-fine
fashion. Therefore, we set the matrix of interests $m^{i}=s^{i}$ if the
$i^{th}$ layer is deepest. Otherwise, we crop the similarity map $s^{i}$ to a
square patch $m^{i}$ centered on $P_{a}^{i}$ of size $\alpha$. As we aim at
increasing the similarity of the correct pixels while decreasing others, we
use softmax function to normalize $m^{i}$ to probability matrix $y^{i}$ with a
temperature $\tau$: $y^{i}=softmax(m^{i}*\tau)$. Note that the softmax
operation is performed on all pixels. Correspondingly, we set the ground truth
$GT^{i}$ of each layer:

$\displaystyle
GT^{i}=\\{\begin{tabular}[]{l}$[\pi(x,y|x_{p}/2^{i},y_{p}/2^{i})]_{12\times
12}$ ~{}~{}\text{If layer $i$ is the deepest};\\\
$[\pi(x,y|\alpha/2,\alpha/2)]_{\alpha\times\alpha}$ ~{}~{}\text{\qquad
Otherwise},\end{tabular}$ (2)

where $[\pi(x,y)]_{M\times N}$ is a matrix of size $M\times N$ and
$\pi(x,y|x_{0},y_{0})$ is an indicator function that outputs 1 only if
$x=x_{0}~{}\&~{}y=y_{0}$ and 0 otherwise. We use cross-entropy loss $L_{CE}$
to penalize the divergence between probability matrix $y^{i}$ and ground-truth
$GT^{i}$ for each layer and summarize the losses as final loss $L_{SSL}$:

$\displaystyle L_{SSL}=\sum_{i}L_{CE}(y^{i},GT^{i}),$ (3)

Inference stage to generate pseudo-labels: As shown in Fig. 2, we first
extract template patches for landmarks defined in the annotated template
image. Then we use $E_{p},E_{r}$ to embed the patches and query image,
resulting in pixel-wise template features and query features. As well as the
training stage, we extract the anchor features $F_{a}$ according to the
corresponding coordinate (the red point in Fig. 2). Next, we compute cascade
cosine similarity maps $s^{i}$ for all layers and clip to range $[0,1]$. At
last, we multiply the similarity maps $s^{i}$ and generate the final
prediction by _argmax_ operator. In CC2D, we perform inference on the training
set to generate pseudo-labels.

Feature extractor: We set the encoder of U-Net [20] as a VGG19 [21].
Additionally, we use ASPP [5] and a convolution layer with a 1x1 kernel after
every up-sampling layer to generate cascade feature embeddings. The detailed
illustration can be found in the supplemental materials.

###  2.2 Stage II: Training with pseudo-labels (CC2D-TPL)

Figure 3: The architecture of the multi-task U-Net used in CC2D-TPL. We fine-
tune the encoder of U-Net initialized by the VGG19 network [21] pretrained on
ImageNet [8].

In this stage, we train a new CNN-based landmark detector from scratch with
the pseudo-labels generated in Sec. 2.1, using the architecture in Fig. 3. We
utilize the multi-task U-Net [26] $g$ as our backbone, which predicts both
heat map and offset maps simultaneously and has satisfactory performance [6].
Specifically, for the $k^{th}$ landmark located at $(x_{k},y_{k})$ in an image
$X$. We set the ground-truth heat map $Y_{k}^{h}$ to be 1 if
$\sqrt{(x-x_{k})^{2}+(y-y_{k})^{2}}\leq\sigma$ and 0 otherwise. Then, the
ground-truth $x$-offset map $Y_{k}^{o_{x}}$ predicts the relative offset
vector $Y_{k}^{o_{x}}=(x-x_{k})/\sigma$ from $x$ to the corresponding landmark
$x_{k}$. Similarly, its $y$-offset map $Y_{k}^{o_{y}}$ is defined [26]. The
loss function $L_{k}$ for the $k^{th}$ landmark consists of a binary cross-
entropy loss, denoted by $L^{h}$, for punishing the divergence of predicted
and ground-truth heatmaps, and the L1 loss, denoted by $L^{o}$, for punishing
the difference in coordinate offset maps.

$L_{k}=L_{k}^{h}(Y_{k}^{h},g_{k}^{h}(X))+sign(Y_{k}^{h})\sum_{o\in\\{o_{x},o_{y}\\}}L_{k}^{o}(Y_{k}^{o},g_{k}^{o}(X)),$
(4)

where $g_{k}^{h}(X)$ and $g_{k}^{o}(X)$ are the networks that predict heatmaps
and coordinate offset maps, respectively; $sign(\cdot)$ is a sign function
which is used to make sure that only the area highlighted by heatmap is
included for calculation. At last, we sum the losses $L_{k}$ for all of the
landmarks defined in the template image: $L_{TPL}=\sum_{k}L_{k}$. In the test
phase, we conduct a majority-vote for candidate landmarks among all pixels
with heatmap value $g_{i}^{h}(X,\theta)\geq 0.5$, according to their
coordinate offset maps in $g_{i}^{o}(X)$. The winning position in the $k^{th}$
channel is the final predicted $k^{th}$ landmark [6].

Table 1: Comparison of the state-of-the-art supervised approaches and our CC2D on the ISBI 2015 Challenge [23] testset. * represents the performances copied from their original papers. # represents the performances we re-implement with limited labeled images. We additionally evaluate the performance of CC2D-SSL on the test set. Model |  Labeled images |  MRE ($\downarrow$) (mm) | SDR ($\uparrow$) (%)
---|---|---|---
2mm | 2.5mm | 3mm | 4mm
Ibragimov et al. [10]* | 150 | - | 68.13 | 74.63 | 79.77 | 86.87
Lindner et al. [14]* | 150 | 1.77 | 70.65 | 76.93 | 82.17 | 89.85
Urschler et al. [22]* | 150 | - | 70.21 | 76.95 | 82.08 | 89.01
Payer et al. [19]* | 150 | - | 73.33 | 78.76 | 83.24 | 89.75
Payer et al. [19]# | 25 | 2.54 | 66.12 | 73.27 | 79.82 | 86.82
Payer et al. [19]# | 10 | 6.52 | 49.49 | 57.91 | 65.87 | 75.07
Payer et al. [19]# | 5 | 12.34 | 27.35 | 32.94 | 38.48 | 45.28
CC2D-SSL | 1 | 4.67 | 40.42 | 47.68 | 55.54 | 68.38
CC2D | 1 | 2.72 | 49.81 | 58.73 | 68.18 | 81.01
Figure 4: (a) The inference procedure of CC2D-SSL for the $6^{th}$ landmark in
the query image. The similarity maps at different scales localize the target
landmark in a coarse-to-fine fashion. We mark the correct positions by the red
circles. (b) Visualizations of the testing images predicted by CC2D-SSL. The
landmarks in green and red represent the predictions and ground truths, while
the yellow lines mark their distances. (c) Visualizations of the testing
images predicted by CC2D. The rightmost images in (b) and (c) are the failure
cases.

##  3 Experiments

###  3.1 Settings

Dataset: This study uses a widely-used public dataset for cephalometric
landmark detection containing 400 radiographs, provided in IEEE ISBI 2015
Challenge [23, 11]. Two expert doctors labeled 19 landmarks of clinical
anatomical significance for each radiograph. We compute the average
annotations by two doctors as the ground truth. The image size is $1935\times
2400$ and the pixel spacing is 0.1mm. According to the official website, the
dataset is split to a training and a test subsets with 150 and 250
radiographs, respectively.

Metrics: Following the official challenge, we use mean radial error (MRE) to
measure the Euclidean distance between prediction and ground truth, and
successful detection rate (SDR) in four radii (2mm, 2.5mm, 3mm, and 4mm).

Implementation details: All of our models are implemented in PyTorch,
accelerated by an NVIDIA RTX3090 GPU. For self-supervised training, the two
feature extractors are optimized by Adam optimizer for 3500 epochs with a
learning rate of 0.001 decayed by half every 500 epochs, which takes about 6
hours to converge with a batch size of 8. The embedding length $L$ is set to
16 and the length $\alpha$ of $m$ is 19. For training with pseudo-labels, the
multi-task U-Net is optimized by Adam optimizer for 300 epochs with a learning
rate of 0.0003, which takes about 50 minutes to converge with a batch size of
16.

###  3.2 The performance of CC2D

During the inference stage of CC2D-SSL, the similarity map in the deepest
layer detects the correct coarse area first, then the similarity maps in the
inner layers gradually improve the localization accuracy. The procedure is
illustrated in Fig. 4(a). Consequently, most of the landmarks in Fig. 2(b) are
successfully detected. Moreover, after training by the predictions of CC2D-SSL
on the training set, the new landmark detector in CC2D-TPL localizes the
landmarks with better accuracy (as shown in Fig. 4(c) and Table 1).

We quantitatively compare our CC2D with the first [14] and second [10] place
in the ISBI 2015 Challenge [23] in Table 1, as well as two recent state-of-
the-art supervised method [19, 22]. With one labeled image available, CC2D
achieves the MRE of 2.72 mm and 4mm SDR of 81.01%, which are competitive
compared to the supervised methods. Furthermore, when the available annotated
data is reduced, our CC2D shows more superiority. As reported in Table 1, our
CC2D performs better than Payer et al. [18] retrained with 10 labeled images.
However, CC2D localizes landmarks with more deviation if there is a drastic
difference between the query (failure cases in Fig. 4) and template image (in
Fig. 1).

###  3.3 Ablation study and hyper-parameter analysis

Hyper-parameter analysis:  We study the influence of the embedding length $L$
and the length $\alpha$ of the matrix of interests. According to Table 2, all
of the experimental results fluctuate slightly when changing the two
parameters, while setting $L=16$ and $\alpha=19$ leads to the best
performance. For $L$, features with larger length may tend to overfit while
smaller length may not represent the anatomical information effectively. For
$\alpha$, too small matrix has limited receptive field. On the contrary, too
large matrix involves too many negative pixels, making the convergence of
$L_{SSL}$ (in Eq. 3) difficult.

Ablation study: We compare the CC2D using similarity maps in different layers.
As shown in (Table 3), the similarity map in the fifth (deepest) layer is most
important which selects the coarsest and accurate area first, while other
similarity maps also contribute to the final detection accuracy.

Table 2: The performances of CC2D with different embedding lengths $L$ and different $\alpha$, which is the length of the matrix of interests $m$. Para. | Value |  MRE ($\downarrow$) (mm) | SDR ($\uparrow$) (%)
---|---|---|---
2mm | 2.5mm | 3mm | 4mm
$L$ | 128 | 3.54 | 37.03 | 46.06 | 56.04 | 71.13
64 | 3.22 | 40.35 | 49.32 | 60.04 | 73.34
32 | 2.96 | 45.01 | 54.33 | 54.14 | 78.06
16 | 2.72 | 49.81 | 58.73 | 68.18 | 81.01
8 | 3.22 | 42.23 | 51.20 | 60.37 | 73.66
$\alpha$ | 9 | 3.31 | 39.49 | 48.48 | 58.10 | 72.31
11 | 3.28 | 39.55 | 49.01 | 59.38 | 72.27
13 | 3.01 | 44.90 | 54.02 | 62.77 | 76.44
15 | 2.92 | 47.55 | 56.35 | 65.97 | 78.54
17 | 2.85 | 47.83 | 56.75 | 66.21 | 78.37
19 | 2.72 | 49.81 | 58.73 | 68.18 | 81.01
21 | 2.92 | 47.47 | 56.50 | 65.70 | 77.11
Table 3: The performances of CC2D using cosine similarity maps in different layers. Layer Index |  MRE ($\downarrow$) (mm) | SDR ($\uparrow$) (%)
---|---|---
5 | 4 | 3 | 2 | 1 | 2mm | 2.5mm | 3mm | 4mm
$\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 57.96 | 10.07 | 12.45 | 15.01 | 17.89
$\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 3.67 | 37.37 | 46.90 | 56.04 | 68.73
$\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | 10.63 | 23.07 | 27.81 | 31.74 | 37.85
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | 4.67 | 25.72 | 32.75 | 41.87 | 56.27
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | 4.89 | 22.73 | 29.43 | 38.48 | 54.21
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 2.72 | 49.81 | 58.73 | 68.18 | 81.01

##  4 Conclusion and Future work

In this paper, we propose Cascade Comparing to Detect (CC2D), a novel
framework for building a robust landmark detection network with only one
labeled image available. CC2D learns to map the consistent anatomical
information into cascading feature spaces by solving a self-supervised patch
matching problem. Using the self-supervised model, CC2D localizes the target
landmark according to the one-shot template image in a coarse-to-fine fashion
to generate pseudo-labels for training a final landmark detector. Extensive
experiments evaluate the competitive performance of CC2D, comparable to the
state-of-the-art fully-supervised methods. In future work, we plan to further
improve the detection accuracy by considering the usage of the spatial
relationships of different landmarks.

## References

  * [1] Arpit, D., Jastrzebski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M.S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al.: A closer look at memorization in deep networks. In: ICML. pp. 233–242. PMLR (2017)
  * [2] Bhalodia, R., Kavan, L., Whitaker, R.T.: Self-supervised discovery of anatomical shape landmarks. In: MICCAI. pp. 627–638. Springer (2020)
  * [3] Bier, B., Unberath, M., Zaech, J.N., Fotouhi, J., Armand, M., Osgood, G., Navab, N., Maier, A.: X-ray-transform invariant anatomical landmark detection for pelvic trauma surgery. In: MICCAI. pp. 55–63. Springer (2018)
  * [4] Chaitanya, K., Erdil, E., Karani, N., Konukoglu, E.: Contrastive learning of global and local features for medical image segmentation with limited annotations. arXiv preprint arXiv:2006.10511 (2020)
  * [5] Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 (2017)
  * [6] Chen, R., Ma, Y., Chen, N., Lee, D., Wang, W.: Cephalometric landmark detection by attentive feature pyramid fusion and regression-voting. In: MICCAI. pp. 873–881. Springer (2019)
  * [7] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: ICML. pp. 1597–1607. PMLR (2020)
  * [8] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR. pp. 248–255 (2009)
  * [9] Gertych, A., Zhang, A., Sayre, J., Pospiech-Kurkowska, S., Huang, H.: Bone age assessment of children using a digital hand atlas. Computerized Medical Imaging and Graphics 31(4-5), 322–331 (2007)
  * [10] Ibragimov, B., Likar, B., Pernus, F., Vrtovec, T.: Computerized cephalometry by game theory with shape-and appearance-based landmark refinement (2015)
  * [11] Kaggle: Cephalometric X-Ray Landmarks Detection Challenge (2015), https://www.kaggle.com/jiahongqian/cephalometric-landmarks/discussion/133268
  * [12] Lang, Y., Lian, C., Xiao, D., Deng, H., Yuan, P., Gateno, J., Shen, S.G., Alfi, D.M., Yap, P.T., Xia, J.J.: Automatic localization of landmarks in craniomaxillofacial cbct images using a local attention-based graph convolution network. In: MICCAI. pp. 817–826. Springer (2020)
  * [13] Li, W., Lu, Y., Zheng, K., Liao, H., Lin, C., Luo, J., Cheng, C.T., Xiao, J., Lu, L., Kuo, C.F.: Structured landmark detection viatopology-adapting deep graph learning. In: ECCV (2020)
  * [14] Lindner, C., Cootes, T.F.: Fully automatic cephalometric evaluation using random forest regression-voting. Scientific Reports 6, 33581 (2016)
  * [15] Liu, D., Zhou, S.K., Bernhardt, D., Comaniciu, D.: Search strategies for multiple landmark detection by submodular maximization. In: Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. pp. 2831–2838. IEEE (2010)
  * [16] Liu, W., Wang, Y., Jiang, T., Chi, Y., Zhang, L., Hua, X.S.: Landmarks detection with anatomical constraints for total hip arthroplasty preoperative measurements. In: MICCAI. pp. 670–679. Springer (2020)
  * [17] Ouyang, C., Biffi, C., Chen, C., Kart, T., Qiu, H., Rueckert, D.: Self-supervision with superpixels: Training few-shot medical image segmentation without annotation. In: ECCV. pp. 762–780. Springer (2020)
  * [18] Payer, C., Štern, D., Bischof, H., Urschler, M.: Regressing heatmaps for multiple landmark localization using cnns. In: MICCAI. pp. 230–238. Springer (2016)
  * [19] Payer, C., Štern, D., Bischof, H., Urschler, M.: Integrating spatial configuration into heatmap regression based cnns for landmark localization. Medical image analysis 54, 207–219 (2019)
  * [20] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI. pp. 234–241. Springer (2015)
  * [21] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)
  * [22] Urschler, M., Ebner, T., Štern, D.: Integrating geometric configuration and appearance information into a unified framework for anatomical landmark localization. Medical image analysis 43, 23–36 (2018)
  * [23] Wang, C.W., Huang, C.T., Lee, J.H., Li, C.H., Chang, S.W., Siao, M.J., Lai, T.M., Ibragimov, B., Vrtovec, T., Ronneberger, O., et al.: A benchmark for comparison of dental radiography analysis algorithms. Medical Image Analysis 31, 63–76 (2016)
  * [24] Yang, D., Xiong, T., Xu, D., Huang, Q., Liu, D., Zhou, S.K., Xu, Z., Park, J., Chen, M., Tran, T.D., et al.: Automatic vertebra labeling in large-scale 3d ct using deep image-to-image network with message passing and sparsity regularization. In: IPMI. pp. 633–644 (2017)
  * [25] Yang, D., Zhang, S., Yan, Z., Tan, C., Li, K., Metaxas, D.: Automated anatomical landmark detection ondistal femur surface using convolutional neural network. In: ISBI. pp. 17–21 (2015)
  * [26] Yao, Q., He, Z., Han, H., Zhou, S.K.: Miss the point: Targeted adversarial attack on multiple landmark detection. In: MICCAI. pp. 692–702. Springer (2020)
  * [27] Zheng, Y., Liu, D., Georgescu, B., Nguyen, H., Comaniciu, D.: 3d deep learning for efficient and robust landmark detection in volumetric data. In: MICCAI. pp. 565–572. Springer (2015)
  * [28] Zhong, Z., Li, J., Zhang, Z., Jiao, Z., Gao, X.: An attention-guided deep regression model for landmark detection in cephalograms. In: MICCAI. pp. 540–548. Springer (2019)
  * [29] Zhou, H.Y., Yu, S., Bian, C., Hu, Y., Ma, K., Zheng, Y.: Comparing to learn: Surpassing imagenet pretraining on radiographs by comparing image representations. In: MICCAI. pp. 398–407. Springer (2020)
  * [30] Zhou, S.K., Greenspan, H., Davatzikos, C., Duncan, J.S., van Ginneken, B., Madabhushi, A., Prince, J.L., Rueckert, D., Summers, R.M.: A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises. Proceedings of the IEEE (2021)
  * [31] Zhou, S.K., Rueckert, D., Fichtinger, G.: Handbook of Medical Image Computing and Computer Assisted Intervention. Academic Press (2019)
  * [32] Zhou, Z., Sodha, V., Pang, J., Gotway, M.B., Liang, J.: Models genesis. Medical image analysis 67, 101840 (2021)
  * [33] Zhou, Z., Sodha, V., Siddiquee, M.M.R., Feng, R., Tajbakhsh, N., Gotway, M.B., Liang, J.: Models genesis: Generic autodidactic models for 3d medical image analysis. In: MICCAI. pp. 384–393. Springer (2019)
  * [34] Zhu, J., Li, Y., Hu, Y., Ma, K., Zhou, S.K., Zheng, Y.: Rubik’s cube+: A self-supervised feature learning framework for 3d medical image analysis. Medical Image Analysis 64, 101746 (2020)

##  5 Supplemental Material

Figure 5: The architecture of Feature Extractor used in CC2D-SSL. The light
grey part is VGG-style encoder, and dark grey part is decoder. Each dark grey
block takes both the output of a block in encoder (dotted lines) and the prior
dark grey block (black arrows) as input. The embeddings from dark grey block
and last light grey block are carried to ASPP module [5] and a Conv block to
output the final features.