# Automatic Text Summarization of COVID-19 Medical Research Articles using
BERT and GPT-2

Bowen Tan
Laboratory of Molecular Genetics
Rockefeller University
New York, NY 10065
<EMAIL_ADDRESS>
BT designed and conducted the pre-processing and quantitative assessment.
Virapat Kieuvongngam
Laboratory of Membrane Biology and Biophysics
Rockefeller University
New York, NY 10065
<EMAIL_ADDRESS>
VK proposed the idea, designed the experiment, implemented a prototype model.
Yiming Niu
Laboratory of Molecular Neurobiology and Biophysics
Rockefeller University
New York, NY 10065
<EMAIL_ADDRESS>
YN trained the model, conducted the quantitative assessment.

###### Abstract

With the COVID-19 pandemic, there is a growing urgency for medical community
to keep up with the accelerating growth in the new coronavirus-related
literature. As a result, the COVID-19 Open Research Dataset Challenge has
released a corpus of scholarly articles and is calling for machine learning
approaches to help bridging the gap between the researchers and the rapidly
growing publications. Here, we take advantage of the recent advances in pre-
trained NLP models, BERT and OpenAI GPT-2, to solve this challenge by
performing text summarization on this dataset. We evaluate the results using
ROUGE scores and visual inspection. Our model provides abstractive and
comprehensive information based on keywords extracted from the original
articles. Our work can help the the medical community, by providing succinct
summaries of articles for which the abstract are not already available.

##  1 Introduction

###  1.1 COVID-19 Open Research Dataset Chanllenge

The global health and research community is need of a way to survey the
scientific literature to come up with a treatment and measures against the
COVID-19. In response to this challenge, the White House and leading research
groups have established the COVID-19 Open Research Dataset (CORD-19) to bring
in the NLP expertise to help finding the answer within the literature or
bringing insights to the public at large [Wang et al., 2020]. This dataset
consists of over 59,000 scholarly articles, including over 47,000 with full
text about the COVID-19 or related diseases.

###  1.2 Text Summarization

Automatic text summarization is an active area of research focusing on
condensing large piece of text to smaller text retaining the relevant
information. There are two general approaches. First is the extractive
summarization aiming at extracting and concatenating important span of the
source text. This is akin to skimming the text. The second approach focus on
generating new summaries that paraphrase the source text. The extractive
approach has been shown to maintain reasonable degree of grammaticality and
accuracy. On the contrary, abstractive approach is much more challenging, due
to the fact that the model must be able to represent semantic information of
the source text and then use this semantic representation to generate a
paraphrase. However, the model may gain the ability to make a creative use of
words or ability to make inference from the source text.

###  1.3 Existing Body of Work

Both approaches of summarization have progressed considerably thanks to the
recent advances and the availability of the large pre-trained NLP models,
often made use of the attention mechanism [Vaswani et al., 2017]. These models
include the Bidirectional Encoder Representations from Transformers (BERT)
[Devlin et al., 2018] and more recently OpenAI GPT-2 [Radford et al., 2018].
They are trained on a very large dataset of text such as the entire corpus of
wikipedia and are able to perform well across diverse NLP tasks, including
machine translation, question-answering, multiple choice question, text
classification, etc. Early text summarization models that uses pre-trained
BERT is BERTSUM [Liu and Lapata, 2019]. BERTSUM is an extractive modified
variant of BERT model trained on a general news (CNN/Daily News) summarization
dataset. The model performs binary classification task to predict whether a
sentence is to be included in the summary. However, as BERT is not built to
perform language generative task, its use for abstractive summarization is
limited.

In the past few years, sequence-to-sequence (seq2seq) models based on the
transformer encoder-decoder architecture has been widely used for abstractive
summarization [Shi et al., 2018]. From the architecture point of view, the
encoder reads the source text and transform it to hidden states and the
decoder takes the hidden states and output a summary text. The mapping from
hidden representation to output text gives the architecture language
generative capability.

Even more recently a unified text-to-text framework has been used to train the
large language model on multiple NLP tasks all at once [Raffel et al., 2019].
The basic idea is to train one single model to map the input text of various
tasks to the output text. In this work, we take a similar spirit to fine-tuned
a pre-trained OpenAIGPT2 to perform mapping from a selected keywords to a
summary text, hence generating a summary abstractly.

###  1.4 Low-resource Challenge

An additional challenge to our task is due to the low availability of the
domain-specific corpus. Unlike a more general summarization like the CNN/Daily
Mail dataset with 286k document-summary pairs, the COVID-19 related
literature, as of April 8 2020, contain approximately 35k full text-abstract
pairs. Moreover, the scientific terminology found in the peer-reviewed
literature can often be esoteric; thus are not used in the mainstream text
where the pre-training was performed. This low-resource may present
considerable impediment to the fine-tuning. However, this framework, if found
useful, can be further expand.

##  2 Approach

###  2.1 Overall Outline

The project is subdivided into two parts, the unsupervised extractive part is
used as a baseline performance and the novel abstractive part. The
unsupervised extractive summarization takes the already pre-trained BERT model
to perform a sentence embedding; whereby every individual sentence is
transformed to 768 high dimensional representation. Subsequently, K-medoid
clustering analysis is performed on the high dimensional representation
[Miller, 2019]. Representing semantic centers of text, the cluster centers are
selected extracted summary.

Comparing against the extractive summarization, the abstractive summarization
is trained to generate a summary from a set of keywords. The keywords are
extracted from the source text using existing token classification tools, such
as NLTK part of speech tagging packages, or fine-tuned BERT token classifier
for part of speech tagging 111Code for BERT token classification is adapted
from https://www.depends-on-the-definition.com/named-entity-recognition-with-
bert/. the training dataset is from
https://www.kaggle.com/abhinavwalia95/entity-annotated-corpus. The keywords
are tokens classified as three different groups: verbs, nouns, verbs and
nouns. Following the extraction, the keywords are paired with the human-
generated abstract (gold summary abstract). This keyword-summary pairs are
processed and fed to the GPT-2 model as illustrated in figure 1.

After training, summary results are generated using stochastic sampling method
described in section 3.4. The results are compared and qualitative assess by
reading inspection. Quantitatively, the generated results are compared against
the gold summary using ROUGE score[Lin, 2004].

###  2.2 Model Architecture

Many of the state-of-the-art NLP models are built using transformer
architecture [Vaswani et al., 2017], relying on attention mechanism to convert
the input sequences to output sequences. Two kinds of transformer
architectures are widely used: the transformer encoder and the transformer
decoder.

The BERT model used here for unsupervised extractive summarization is a pre-
trained transformer encoder model [Sanh et al., 2019]. The model has 12
attention heads and 6 transformer encoder layers. The output is 768
dimensional last hidden state of the model. We use pytorch-based DistilBERT
implemented in the Huggingface transformer [Wolf et al., 2019].

Due to the GPU resource constraint, the abstractive summarization model is a
pre-trained distil version of GPT-2. The DistilGPT2 can take up to 1024 token
length. It has 12 attention heads and 6 transformer decoder layers. We use the
pytorch version of GPT-2 implemented in the Huggingface transformer package
[Wolf et al., 2019].

###  2.3 Training Strategy of the Abstractive Summarization GPT-2

The GPT-2 model is trained on 2 tasks: the language modeling (lm) task, and
the multiple choice (mc) prediction task. For the lm task, the model predicts
a next word token given previous tokens and context. For the mc task, given a
set of keywords, the model choose the correct gold summary from summary
choices. Each of the tasks has an associated loss.

The lm task projects the hidden state to the word embedding ouput layer.
Cross-entropy loss is applied on the target corresponding to the gold summary
to get an lm loss. For the training, we label the start and the end of text
with special tokens. To enable the model to recognize the summarization task,
a special token, <|summarize|>, is used to separate the keywords and the gold
summary. The input are all padded with padding token to 1024 tokens, and any
input longer than 1024 tokens are truncated.

For the mc task, the hidden state of the last token, <|endoftext|>, is passed
through a linear layer to get a class likelihood score ,i.e. a classification
task. The cross-entropy loss is applied to obtain a mc loss. To create the
training dataset, we randomly select 3 summaries unrelated to the keywords,
so-called distractors, and paired the distractors with the keywords in the
similar manners as the gold summary, forming a batch of 4 input items.

The language modeling training labels are the token of summary that are right-
shifted by 1 token. This is because GPT-2 is auto-regressive in nature, and
the $n^{th}$ token output is generated from all previous n-1 token inputs to
the left.

The multiple choice training label is a tensor of a numeric $i$, indicating
the $i^{th}$ item that is the correct keyword-gold summary pair.

The total loss is a weighted sum of the two losses at ratio of 2:1 lm loss to
mc loss.

###  2.4 Intuition of the Training Strategy

The intuition behind this training strategy is the following. Because the
GPT-2 model aims at text generation, it is designed to be auto-regressive.
That is to say, the model takes the backward context of n-1 previous tokens to
predict the $n^{th}$ token. This is achieved using the masked self-attention
mechanism to block the information from tokens to the right of the current
position from being calculated. The special token <|summarize|> is used to
signify the context whereby the subsequent tokens are to be the summary of the
information before this special token. The GPT-2 model is to learn this
context clue from the fine-tuning.

The multi-loss training is used in the hope that we will be able to induce the
model to map the local semantic context in the keywords to the gold summary.
At the same time, the model retains the global contextual information of the
keywords so that in the end of the text, the model is able to distinguish the
gold summary from the distractors. This multi-loss training seem to be
prominent in recent language model training that aims at general language
understanding [Li et al., 2019, Raffel et al., 2019].

Figure 1: Overview of GPT2 multi-loss training. The example input shown has n
= 4 items in the input: 1 true abstract and 3 distractors. The true abstract
is shown as the $0^{th}$ item. The beginning of sequence, end of sequence,
padding, and summarization token are denoted <BOS>, <EOS>, <pad>, <S>,
respectively. The language modeling label contains a masked token <masked>
everywhere except the gold summary tokens. The multiple choice label is a
tensor [0] indicates the $0^{th}$ item as the correct keyword-summary pair.

##  3 Experiments and Results

###  3.1 Model Training

The training of DistilGPT2 is carried out on a Google Colab equiped with 1x
NVIDIA Tesla P100 GPU. A total of 5 epochs are performed. The training dataset
consists of 31246 training samples, each sample has 4 multiple choice options.
The validation dataset consists of 3572 samples, each also has 4 multiple
choices. The training parameters include the learning rate 3$e^{-5}$, with
batch size = 1 and gradient accumulation of 5 steps. The linearly decreasing
learning rate scheduler is used for every epoch. The training loss of the
first epoch is shown in figure 2. For the validation, the lm loss is 7.0, mc
loss is 0, indicating no sign of overfitting.

Figure 2: Training results. The two losses are shown during 1 epoch of
training (32k iterations). The language model loss (blue) is shown in the
exponentiated form of the cross-entropy loss, so-called the perplexity score
($e^{lmloss}$). The multiple choice loss (orange) is calculated from the cross
entropy loss over all the multiple choices.

###  3.2 Visualizing the Results

The attention mechanism allows us to assess the model performance. The
attention can be broadly interpreted as a vector of importance weights, i.e.
how strongly the tokens in sequences are correlated with other tokens [Vig,
2019]. To visualize the attention, we input the sequence illustrated in table
1 and plot the attention as matrix of alignment heatmap[Bahdanau et al.,
2014]. In the figure 3 below, we visualize the learned attention from the
model by comparing the attention of the pre-trained GPT2 before the fine-
tuning and after the fine-tuning. Because of the multi-layer, multi-attention
head architecture, the total unique structures are 6*12 = 64. We only select 4
attentions that seems to exhibit interesting phenomena. First obvious pattern
exihibited in layer 0 head 1 both before and after fine-tuning is the diagonal
pattern. This could be interpreted as representing self-correlation. Observed
more strongly only after fine-tuning, the second pattern is the left-shifted
diagonal line, shown in layer 3 head 7 and layer 5 head 9. This could be
interpreted as the correlation between the keyword input and the summary. This
attention, learned during the fine-tuning, became more strongly after more
epochs of fine-tuning. This indicates that our training strategy works as
expected. Thirdly, the vertical line pattern observed in the attention of
layer 2 head 2 both before and after fine-tuning. This could be interpreted as
long-range correlation within phrases or sentences in the sequence important
for maintaining coherence or grammarticality.

###  3.3 Sequence Generation

The language modeling output of the GPT2 is a tensor of size [sequence
length,vocab size]. This is a tensor of likelihood distribution over all words
before the softmax. To generate a text sequence from this output, we sample
words from this distribution in the word-by-word manner. To obtain the
$i^{th}$ word, we consider the conditional probability of previous $i-1$
words.

$P(x)=\Pi_{i=1}P(X_{i}|x_{i},...,x_{i-1})$

Firstly, before the sampling, we can apply a scaling factor, so-called
temperature ($t$), to the likelihood ($u$) to reshape the skews likelihood
distribution before the softmax [Holtzman et al., 2019].

$p(x)=\frac{exp(u/t)}{\Sigma_{l^{*}}exp(u^{*}_{l}/t)}$

The high temperature tends to skews the distribution in favor of low
probability words; whereas the low temperature skews to the distribution
towards a few high probability words. The result is a tug-of-war between
favoring generation accuracy at the cost of decreasing word diversity.

Secondly, we employs a stochastic sampling method, called top-p sampling, over
the probability distribution of words to pick conditional on previous words in
the sequence. The rule for top-p sampling is that the smallest set of
candidate words ($x$) to consider is such that the cumulative conditional
probability is greater than $p$.

$\Sigma_{x}P(x|x_{1:i-1})>=p$

Additionally, to prevent the model from sampling too many low probability
words, we limit the number of candidate words to consider to $k$ words.

We empirically tested a few of the sampling parameters and found that
temperature = 1, k = 50, and p = 0.8 yields a reasonable generations.

Table 1: Example of summary result gold summary | keyword input | generated abstractive summary
---|---|---
| 1 epoch training |
Influenza virus is the most frequently reported viral cause of rhabdomyolysis. A 7-year-old child is presented with rhabdomyolysis associated with parainfluenza type 2 virus. Nine cases of rhabdomyolysis associated with parainfluenza virus have been reported. Complications may include electrolyte disturbances, acute renal failure, and compartment syndrome. |  <|startoftext|> Rhabdomyolysis Associated with Parainfluenza VirusInfluenza virus is cause rhabdomyolysis child is rhabdomyolysis parainfluenza type 2 virus cases rhabdomyolysis parainfluenza virus have Complications include electrolyte disturbances renal failure compartment syndrome <|summarize|> |  Influenza virus is the most common cause of respiratory rhabdomyolysis in the child. It is believed that the rhabdomyolysis and parainfluenza type 2 virus cases with rhabdomyolysis in parainfluenza type 2 virus. Recent cases with rhabdomyolysis in parainfluenza virus have been described. Complications include electrolyte disturbances, kidney failure,and gastrointestinal compartment syndrome.

| 5 epoch training |
|  <|startoftext|> Rhabdomyolysis Associated with Parainfluenza’, ’VirusInfluenza virus is cause rhabdomyolysis child is rhabdomyolysis’, ’parainfluenza type 2 virus cases rhabdomyolysis parainfluenza virus’, ’have Complications include electrolyte disturbances renal failure’, ’compartment syndrome <|summarize|> |  Influenza virus is a leading cause of rhabdomyolysis in child. However, several cases of rhabdomyolysis in the parainfluenza virus have been reported. Complications include electrolyte disturbances in renal failure of the normal renal compartment syndrome

###  3.4 Analysis

The result in table 1 in noteworthy in that the model "learns" that influenza
virus is the most common cause of rhabdomyolysis (a breakdown of muscle
tissue), even though this knowledge is not presented in the keyword. In this
respect the model gains an ability to infer some knowledge from the training
corpus. However, this could be problematic because it could wrongly infers
falsehood as well. For example, the model outputs: complication include …
gastrointestinal conpartment syndrome. In fact, the compartment syndrome is
not a gastrointestinal condition. However, we should note that this could also
be attributed to our sampling method.

Figure 3: Visualizing attention mechanism. The weights of attentions layers
mapping the input to the output is shown. The input sequence shown in table 1
is passed through either the pre-trained GPT2 model or our summarization fine-
tuned model. The x-axis represents the input sequence. The y axis represents
the aligned output. The notation <B>, <S>, <E> denotes the start,
summarization, and end token respectively. The part of keyword or summary
sequences are labeled. The figure compares of the attention before and after
the fine-tuning. Selected attention layers and heads are plotted as matrix
heatmaps.

##  4 Quantitative Assessment

###  4.1 ROUGE metric

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a metric used in
NLP for evaluating text summarization. The metric compare a model-generated
summary against a human-generated reference summary. ROUGE-n measures overlap
of n-grams between the two texts. ROUGE-l measures the longest matching
sequences of words, without predefined n-gram length, and does not require
consecutive matches. ROUGE-w measures the longest matching sequences that take
consecutive matching into account [Lin, 2004]. The recall version of ROUGE
reports the ratio of n-grams in the reference that are also present in the
generated summary. The precision version of ROUGE reports the ratio of n-grams
in the generated summary that are also present in the reference summary. The F
score version of ROUGE is the harmonic mean of the precision ROUGE and recall
ROUGE. In our report, we will use the F score version of ROUGE.

###  4.2 Extractive Summarization

We applied kmeans clustering followed by k nearest neighbour to extract
sentences representing comprehensive semantic meanings of abstracts. During
the extraction, we compared the effects between 40% versus 60% compression
ratios on ROUGE scores. As shown in figure 4a and 4b, all of the ROUGE scores
of 60% extraction are higher than 40% extraction, irrespective of training
epochs. Additionally, extractive summary produces reuslts with higher rouge
scores compared to the abstractive one (Figure 4c). This is consistent with
the assumption that less compression during extraction preserves more
information compared to the original abstracts.

###  4.3 Abstractive Summarization

In the abstractive summarization stage, the effect of training time was
firstly investigated. As shown in Figure 4, the plotted weights of attentions
layers mapping the input to the output i between 1 and 5 epochs suggested a
more concentrated distribution. In other words, the GTP-2 model would benefit
significantly by longer training process and this is reflected in the better
fit of relevance between the input and output. Furthermore, the output summary
shown in Table 1 illustrates that longer training could help to generate more
’abstract’ summary: Compared to the gold summary, model with 5-epoch training
summarize special cases as ’several cases of mixed infections have been
reported’ rather than explaining the specific case, whereas model with 1-epoch
training still tries to describe it. Interestingly, the seemingly more
abstract results are not reflected in the calculated rouge score. As shown in
Figure 4a, the difference of rouge scores between model with 1 and 5 epochs is
insignificant.

To investigate the effect of keywords used for sentence generation, we then
compared all of keywords (nouns and verbs) yielded from 40% versus 60%
extraction. Figure 4a and 4b together show that the overall ROUGE scores from
60% abstractive group are higher than the 40% group. This indicates that using
more keywords as input tokens would result in more coverage of information
compared to original abstracts after GPT2 generator.

We next compared whether different word classes influence generated summaries
compared to the original ones. As shown in Figure 4c, the first observation is
that 40% abstraction tend to show lower ROUGE scores than the 60% group,
irrespective of word classes. Second, only using verbs as keywords shows very
low ROUGE scores, while only using nouns tend to show almost similar ROUGE
scores compared to the group using both verbs and nouns. This may suggest that
nouns are generally weighted more than verbs during summary generation, or
nouns themselves are representing more accurate information that the original
abstracts convey. However, this does not exclude the possibility that this
advantage of using nouns is due to using larger percentage of keywords since
nouns tend to be used more than verbs in sentences.

In figure 4d, we further evaluate whether using different word sampling
methods, the greedy search versus the top-k sampling, would influence the
results. Although the ROUGE scores between the two groups are similar, in some
cases the greedy search group (top-1) even shows slightly higher scores, the
readability and abstractive meanings are significantly worse in the greedy
search group compared to the top-k group.

Figure 4: Summary of experiments on hyper-parameters. (A) and (B) Comparison
of rouge scores between model with 1 (A) and 5 (B) epochs training. (C) Effect
of nouns and verbs in keywords. Percentages represent the fraction of specific
category of words included. (D) Effect of the greedy search approach. Top 1
and 50 stands for with and without the greedy search, respectively.

##  5 Conclusion and Future Work

Abstractive summarization still represents a standing challenge for deep-
learning NLP. Even more so when this task is applied to a domain-specific
corpus that are different from the pre-training, are highly technical, or
contains low amount of training materials. COVID-19 Open Research Dataset
Challenge exemplify all the abovementioned difficulty. Nevertheless We have
here illustrated that the text-to-text, multi-loss training strategy could be
used to fine-tune a pre-trained language model such as GPT-2 to perform
abstractive summarization. The result is interpretable and reasonable, even
though it is not near human-level performance.

First of all, We think that our model could benefit from further training as
the new coronavirus-related research publication are becoming available. This
should make the model more accurate in its ability to infer conclusion from
the keywords.

In retrospect, we think that the keyword generation phrase is flexible and
could also be significantly improved without too much investment. Our keywords
are taken from nouns and verbs present in the text. We could investigated how
much more fidelity could be gained from adding more information such as
adjective part to the keyword. More data augmentation could be performed by
randomly dropping or adding words to the keyword sets. This would in effect
create much more keyword-summary pairs from the existing ones. Finally, we
think that other strategy for extracting keywords could be experimented. For
example, one could fine-tune a token classification model to selectively
extract scientific keywords.

The evaluation process of abstractive summaries still requires more
exploration. From our data, the ROUGE scores represent more direct information
or phrases overlapping rather than the actual meanings and readability of
summaries. Clearly, all of the extractive models yield higher ROUGE scores
than abstractive ones. Intuitively, extractions of raw sentences could result
in higher similarity, but this approach would not be favored for better
summarization. The fact that the GPT2 generated abstractive summaries showing
good readability and succinct information coverage are not reflected by ROUGE
scores strongly suggest that other scores or evaluation systems on these
aspects are needed in the future. One possible idea would be combining
regularization to penalize local similarity but reward global similarity.

Finally, we think that our approach can be further leveraged if more intensive
computation resources is available. Overall, our implementation is mostly
limited by computation power. With 1 Tesla P-100 GPU, the training can only be
done on the DistilGPT2 version with the batch size = 1 because of the memory
limit of the GPU. It is likely that the result could greatly benefit from
using more bigger GPT-2 version and with more training if permitted by the
available computation power.

In the end, we hope that a text summarization approach. such as ours, can help
the medical research community keep up with the rapidly growing literature and
that it helps bring new insight to fight the pandemic.

##  6 Code availability

All source codes and models of GPT2 implemented in this study are publicly
available on GitHub https://github.com/VincentK1991/BERT_summarization_1.

## References

  * Bahdanau et al. [2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate, 2014.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2018.
  * Holtzman et al. [2019] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi.  The curious case of neural text degeneration, 2019.
  * Li et al. [2019] Jianquan Li, Xiaokang Liu, Wenpeng Yin, Min Yang, and Liqun Ma.  An empirical evaluation of multi-task learning in deep neural networks for natural language processing, 2019.
  * Lin [2004] Chin-Yew Lin.  Rouge: A package for automatic evaluation of summaries.  In _ACL 2004_ , 2004.
  * Liu and Lapata [2019] Yang Liu and Mirella Lapata.  Text summarization with pretrained encoders, 2019.
  * Miller [2019] Derek Miller.  Leveraging bert for extractive text summarization on lectures, 2019.
  * Radford et al. [2018] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2018\.  URL https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf.
  * Raffel et al. [2019] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer, 2019.
  * Sanh et al. [2019] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.
  * Shi et al. [2018] Tian Shi, Yaser Keneshloo, Naren Ramakrishnan, and Chandan K. Reddy.  Neural abstractive text summarization with sequence-to-sequence models: A survey, 2018.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need, 2017.
  * Vig [2019] Jesse Vig.  A multiscale visualization of attention in the transformer model.  _arXiv preprint arXiv:1906.05714_ , 2019.  URL https://arxiv.org/abs/1906.05714.
  * Wang et al. [2020] Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Darrin Eide, Kathryn Funk, Rodney Kinney, Ziyang Liu, William Merrill, Paul Mooney, Dewey Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D. Wade, Kuansan Wang, Chris Wilhelm, Boya Xie, Douglas Raymond, Daniel S. Weld, Oren Etzioni, and Sebastian Kohlmeier.  Cord-19: The covid-19 open research dataset, 2020.
  * Wolf et al. [2019] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew.  Huggingface’s transformers: State-of-the-art natural language processing, 2019.

##  7 Appendix

###  7.1 Generated Sample 1

Gold Summary

Publisher Summary Demyelination is a component of several viral diseases of
humans. The best known of these are subacute sclerosing panencephalitis (SSPE)
and progressive multifocal leukoencephalopathy (PML). There are a number of
naturally occurring virus infections of animals that involve demyelination and
many of these serve as instructive models for human demyelinating diseases. In
addition to the naturally occurring diseases, many viruses have been shown to
be capable of producing demyelination in experimental situations. In
discussing virus-associated demyelinating disease, the chapter reviews the
architecture and functional organization of the CNS and considers what is
known of the interaction of viruses with CNS cells. It also discusses the
immunology of the CNS that differs in several important aspects from that of
the rest of the body. Experimental models of viral-induced demyelination have
also been considered. Viruses capable of producing demyelinating disease have
no common taxonomic features; they include both DNA and RNA viruses, enveloped
and nonenveloped viruses. The chapter attempts to summarize the important
factors influencing viral demyelination, their common features, and possible
mechanisms.

Abstractive Summary 1

Abstract Demyelination is a component of many diseases humans are
experiencing. Sclerosing panencephalitis (SSPE) and leukoencephalopathy (PML)
are common causes of demyelination in many animal and human diseases. However,
the virus infections of animals involve demyelination. These situations of
demyelinating disease chapter reviews the architecture of the CNS and
considers its interactions with the viruses in the CNS. Different aspects of
the rest of the body models of demyelination have been explored. Viruses, for
example, have been identified as the most important virus infections in
humans.

Abstractive Summary 2

Abstract Demyelination is an important component of the diseases human are
suffering from. The sclerosing panencephalitis (SSPE) and leukoencephalopathy
(PML) are common causes of demyelination in various clinical conditions, and
are considered as a potential cause of demyelination in some clinical
conditions. However, there are several clinical and epidemiological factors
that may influence demyelination in some clinical situations. Demyelination in
clinical situations may be a result of a combination of viral and cellular
factors, such as viral RNA, viruses, and viruses. In this chapter, we reviews
the architecture and organization of the CNS, and considers its potential to
be a potential cause of demyelination in some clinical situations. We discuss
the immunology of the CNS

###  7.2 Generated Sample 2

Gold Summary

RNAi pathway is an antiviral defence mechanism employed by insects that result
in degradation of viral RNA thereby curbing infection. Several viruses
including flaviviruses encode viral suppressors of RNAi (VSRs) to counteract
the antiviral RNAi pathway. Till date, no VSR has been reported in
alphaviruses. The present study was undertaken to evaluate chikungunya virus
(CHIKV) proteins for RNAi suppressor activity. We systematically analyzed all
nine CHIKV proteins for RNAi suppressor activity using Sf21 RNAi sensor cell
line based assay. Two non-structural proteins, namely, nsP2 and nsP3 were
found to exhibit RNAi suppressor activity. We further validated the findings
in natural hosts, namely in Aedes and in mammalian cell lines and further
through EMSA and Agrobacterium infiltration in GFP silenced transgenic tobacco
plants. Domains responsible for maximum RNAi suppressor activity were also
identified within these proteins. RNA binding motifs in these domains were
identified and their participation in RNAi suppression evaluated using site
directed mutagenesis. Sequence alignment of these motifs across all species of
known alphaviruses revealed conservation of these motifs emphasizing on a
similar role of action in other species of alphaviruses as well. Further
validation of RNAi suppressor activity of these proteins awaits establishment
of specific virus infection models.

Abstractive Summary 1

Abstract Chikungunya virus (CHIKV) proteins are highly conserved and are
highly conserved in the host cell line. However, their activity against RNAi
is not well understood. Here, we evaluated the chikungunya virus (CHIKV)
proteins as a non-structural RNAi suppressor activity. We analyzed the CHIKV
proteins as a non-structural RNAi suppressor activity. We analyzed the CHIKV
proteins as a non- ’structural RNAi suppressor activity. Sf21 RNAi sensor cell
line assay proteins nsP2 nsP3 were found to exhibit RNAi activity further.
Ourfindings on host Aedes cell lines (EMSA, Agrobacterium infiltration, GFP,
tobacco plants, Domains RNAi suppressor activity) were consistent with the
observed RNAi suppression mutagenesis. Sequence alignment motifs of species
alphaviruses revealed the conserved conservation motifs in the role of action
species.

Abstractive Summary 2

Abstract Chikungunya virus (CHIKV) proteins are a novel and promising RNAi-
mediated antiviral protein. However, their antiviral activity against
chikungunya virus (CHIKV) proteins is not well understood. In this study, we
was able to evaluate chikungunya virus (CHIKV) proteins as a novel RNAi-
mediated antiviral protein. We detected that CHIKV proteins as a novel RNAi-
mediated antiviral protein could suppress chikungunya virus infection by
targeting the Sf21 RNAi sensor cell line assay proteins nsP2 and nsP3. We were
able to demonstrate that CHIKV proteins as a novel RNAi-mediated antiviral
protein could suppress chikungunya virus infection by targeting the Sf21 RNAi
sensor cell line assay proteins nsP2 and nsP3


# Learning Canonical Embedding for Non-rigid Shape Matching

Abhishek Sharma & Maks Ovsjanikov
LIX, Ecole Polytechnique
IP Paris, France
{sharma,maks}@@lix.polytechnique.fr

###### Abstract

This paper provides a novel framework that learns canonical embeddings for
non-rigid shape matching. In contrast to prior work in this direction, our
framework is trained end-to-end and thus avoids instabilities and constraints
associated with the commonly-used Laplace-Beltrami basis or sequential
optimization schemes. On multiple datasets, we demonstrate that learning self
symmetry maps with a deep functional map projects 3D shapes into a low
dimensional canonical embedding that facilitates non-rigid shape
correspondence via a simple nearest neighbor search. Our framework outperforms
multiple recent learning based methods on FAUST and SHREC benchmarks while
being computationally cheaper, data efficient, and robust.

##  1 Introduction

Shape correspondence is a fundamental problem in computer vision, computer
graphics and related fields (Thomas et al., 2021), since it facilitates many
applications such as texture or deformation transfer and statistical shape
analysis (Bogo et al., 2014) to name a few. Although shape correspondence has
been studied from many viewpoints, we focus here on a functional map-based
approaches (Ovsjanikov et al., 2012) as this framework is quite general,
scalable and thus, has been extended to various other applications such as
pose estimation (Neverova et al., 2020), matrix completion (Sharma &
Ovsjanikov, 2021) and graph matching (Wang et al., 2020).

While recent learning based deep functional map approaches have made
impressive gains in non rigid isometric full shape matching (Litany et al.,
2017b; Roufosse et al., 2019; Halimi et al., 2019; Sharma & Ovsjanikov, 2020),
partial shape matching (Rodolà et al., 2017; Litany et al., 2017b) has
received little attention despite it being of great interest in robotics
(Chavdar et al., 2012) and Virtual reality applications (Sharma et al., 2016).
The progress is mainly hindered by the difficulty of learning a suitable
embedding or basis functions for partial 3D data. The majority of works in
this domain use the Laplace-Beltrami basis (Ovsjanikov et al., 2017), which
are biased towards near-isometries and can be unstable under significant
partiality (Kirgo et al., 2020). As a consequence, there is no unified
framework that excels at learning both partial as well as full non-rigid shape
matching despite some efforts in this direction (Sharma & Ovsjanikov, 2020;
Marin et al., 2020). While the former approach Sharma & Ovsjanikov (2020) is
class-specific and requires retraining a network for each class, the latter
(Marin et al., 2020) employs a two stage optimization strategy without
adequate regularization which, as we show later, is suboptimal.

Instead of using predefined basis functions in the functional map framework,
learning an embedding, to be used as a basis in the functional map, is a
promising direction towards obtaining a unified shape matching framework for
both partial as well as full shape matching. However, there exist few works
that exploit prior information on such embeddings. Marin et al. (2020) make
the first attempt by assuming such embeddings to be linearly invariant between
a pair of shapes. However, as we show later, learning such an embedding
without exploiting natural priors on 3D shapes, such as their symmetry
structure, leads to overfitting as no regularization or constraint is enforced
on the linear transformation between a pair of shape embeddings (Marin et al.,
2020).

In this paper, we present a novel non-rigid shape matching method based on a
nearest neighbour approach in canonical embedding. We work with point cloud
representation of 3D shapes and assume to be given a self symmetry map for
each shape during training. We make two assumptions on the canonical
embedding: our first assumption is to learn an embedding of each shape that
would make the given self-symmetry map linear in some higher-dimensional
space. This is advantageous for two reasons: first, it significantly
simplifies the embedding learning pipeline and makes it learnable end-to-end.
This is because the linearly invariant assumption is made on a self-symmetry
map and not on the pairwise map between shapes as done in (Marin et al.,
2020). This alleviates the need to retrieve this linear transformation for
shape matching at test time and thereby, reduces the two stage sequential
optimization scheme of Marin et al. (2020) to joint optimization. Secondly,
modelling a self symmetry map enables us to explicitly enforce the pointwise
map between two shapes to take into account the intrinsic self-symmetry during
training. Thus, our second contribution is a novel regularizer that, combined
with the learned self symmetry map during training, significantly improves
generalization and robustness to sampling resolution as well as the size of
embedding. Our method obtains superior results on multiple shape matching
benchmarks such as FAUST and SHREC when compared to recent learning-based
methods while being computationally cheaper, more robust and data efficient.

##  2 Related Work

#### Functional Maps

Computing point-to-point maps between two 3D discrete surfaces is a very well-
studied problem. We refer to a recent survey (Sahillioğlu, 2019) for an in-
depth discussion. Our method is closely related to the functional map
pipeline, introduced in (Ovsjanikov et al., 2012) and then significantly
extended in follow-up works (see, e.g.,Ovsjanikov et al. (2017)). The key idea
of this framework is to encode correspondences as small matrices, by using a
reduced functional basis, thus greatly simplifying many resulting optimization
problems. The functional map pipeline has been further improved in accuracy,
efficiency and robustness by many recent works including (Kovnatsky et al.,
2013; Huang et al., 2014; Burghard et al., 2017; Rodolà et al., 2017; Nogneng
& Ovsjanikov, 2017; Ren et al., 2018; Eisenberger et al., 2020; Ginzburg &
Raviv, 2020). There also exist other works (Wei et al., 2016; Boscaini et al.,
2016; Monti et al., 2017) that treat shape correspondence as a dense labeling
problem but they typically require a lot of data as the label space is very
large.

#### Learning from raw 3D shape

Although early approaches in functional maps literature used hand-crafted
features (Ovsjanikov et al., 2017), more recent methods directly aim to
_learn_ either the optimal transformations of hand crafted descriptors (Litany
et al., 2017b; Roufosse et al., 2019) or even features directly from 3D
geometry itself (Donati et al., 2020; Sharma & Ovsjanikov, 2020). Initial
efforts in this direction used classical optimisation techniques (Corman et
al., 2014). In contrast, Deep Functional Maps (Litany et al., 2017a) proposed
a deep learning architecture called FMNet to optimize a non-linear
transformation of SHOT descriptors (Tombari et al., 2010), that was further
extended to unsupervised setting (Roufosse et al., 2019; Halimi et al., 2019).
To alleviate the sensitivity to SHOT descriptor, recent works including
(Groueix et al., 2018; Donati et al., 2020; Sharma & Ovsjanikov, 2020) learn
shape matching directly from the _raw 3D data_ without relying on pre-defined
descriptors, thus leading to improvements in both robustness and accuracy.
However, all these works are aimed at full (complete) shape correspondence and
do not handle partial shape matching effectively.

#### Self Supervised Learning

Self supervised learning has been exploited for learning representations and
embedding in various domains where a proxy task is used to learn the
representation. e.g. Sharma et al. (2016) uses an autoencoder to complete the
partial shapes and uses the resulting representation of shape completion for
shape classification task. Gidaris et al. (2018) learns to predict image
rotations and uses the resulting representation for image classification. Our
formulation is in the same spirit as we learn to inject the symmetry
information in a 3D shape and use the resulting representation for 3D shape
matching. However, we choose symmetry learning as a proxy task for embedding
learning for a principled reason which we describe in detail in the
methodology section.

#### Learning Basis from Data

Most of the functional map frameworks can not handle partiality in data as
they rely on Laplacian eigenfunctions that are shown to be unstable under
partial data. (Rodolà et al., 2017; Litany et al., 2017b; Wu et al., 2020)
deal with partiality but they are based on hand-crafted features and require
an expensive optimization scheme and are instance specific. While Sharma &
Ovsjanikov (2020) proposes to learn a suitable alignment of pre-computed
Laplacian Eigen basis functions, the approach still relies on the Laplacian
basis and can therefore be unstable. Marin et al. (2020) proposed a two stage
architecture to learn a linear transformation invariant shape embedding to
bypass the difficulties associated with LBO. However, as we demonstrate later
in experiments, the two stage architecture is suboptimal due to the lack of
adequate regularization.

#### Symmetry for Non Rigid Shape Matching

Matching shapes with intrinsic symmetries involves dealing with symmetric
ambiguity problem which has been very well studied and explored in axiomatic
methods (Raviv et al., 2010; Lipman et al., 2010; Mitra et al., 2012;
Ovsjanikov et al., 2013; Ren et al., 2020). More recently, (Shi et al., 2020;
Fernandez-Labrador et al., 2020) proposes an end to end method to learn
extrinsic 3D symmetries from a RGB-D image. However, none of the existing
learning based non-rigid shape matching method models or learn symmetry
explicitly as a regularizer for shape matching.

Rest of the paper is structured as follows: In the next section, we briefly
cover the necessary background on the functional map. Afterwards, we propose a
novel learning strategy to learn canonical embedding and introduce our novel
regularization based on intrinsic symmetry prior. Lastly, we validate our
framework on three benchmark datasets by comparing it to various state-of-the-
art methods and providing ablation studies.

##  3 Background

Before describing our method, we provide a brief overview of the basic
pipeline to compute a functional map (Ovsjanikov et al., 2012).

#### Functional Map Computation

The typical functional map pipeline (Ovsjanikov et al., 2012) assumes that we
are given a source and a target shape, $\mathbf{X},\mathbf{Y}$, containing,
respectively, $n_{x}$ and $n_{y}$ vertices, a small set of $k$ basis
functions, e.g. of the respective Laplace-Beltrami operators (LBO). We are
also given a set of descriptors on each shape, to be preserved by the unknown
map, whose coefficients in the basis functions are stored as columns of
matrices $\Phi_{\mathbf{X}},\Phi_{\mathbf{Y}}$. The optimal _functional map_
$\mathbf{C}_{\text{opt}}$ is computed by solving the following optimization
problem:

$\displaystyle\mathbf{C}_{\text{opt}}=\operatorname*{arg\,min}_{\mathbf{C}}E_{\text{desc}}\big{(}\mathbf{C}\big{)}+\alpha
E_{\text{reg}}\big{(}\mathbf{C}\big{)},$ (1)

where
$E_{\text{desc}}\big{(}\mathbf{C}\big{)}=\big{\|}\mathbf{C}\Phi_{\mathbf{X}}-\Phi_{\mathbf{Y}}\big{\|}^{2}$
aims at the descriptor preservation whereas the second term acts as a
regularizer on the map by enforcing its overall structural properties, such as
bijectivity of the map. The optimization problem in equation 1 can be solved
with any convex solver. Once the optimal functional map
$\mathbf{C}_{\text{opt}}$ is computed, one can use nearest neighbor search in
the spectral embedding to convert it to a point to point correspondence.

Note that when the basis functions are neural network-based, instead of
optimizing over $\mathbf{C}$, we are optimizing the functional in equation 1
over $\mathbf{C}$, $\Phi_{\mathbf{X}}$ and $\Phi_{\mathbf{Y}}$. In this case,
joint optimization over $\mathbf{C}$, $\Phi_{\mathbf{X}}$ and
$\Phi_{\mathbf{Y}}$ is challenging as $\mathbf{C}$ is computed via an
iterative solver itself.

##  4 Learning Canonical Embedding

In the previous section, we outlined a basic mechanism to compute a functional
map given a set of basis functions. Due to the instability of Laplace-Beltrami
operator on partial 3D shapes, our main goal is to avoid using its
eigenfunctions and instead aim to _learn_ a embedding that can replace the
spectral embedding given by the LBO. This section details how to learn such an
embedding whilst working in the symmetric space.

Input Shape Representation In contrast to several recent works (Halimi et al.,
2019; Sharma & Ovsjanikov, 2020) that assume to be given a mesh representation
of 3D shapes in non-rigid shape matching, we do not impose any such constraint
and directly work the with point cloud representation.

Type of Supervision In addition to the pointwise map between shapes, we assume
to be given a collection of shapes with a self-symmetry ground truth map for
each shape and our goal is to find an embedding that respects the given
symmetry of each shape and that ultimately can reduce shape correspondence
between a pair of shapes to a nearest neighbor search between their
embeddings. When compared to our main baseline (Marin et al., 2020), this is
an additional supervision and thus, we also evaluate Marin et al. (2020) with
the same supervision for fair comparison.

Our work is most closely related to a recent work (Marin et al., 2020) that
proposes to replace the Laplace-Beltrami basis by learning embeddings that are
related by a linear transformation across pairs of shapes. Intuitively, this
formulation aims to embed a shape from the 3D space, in which complex non-
rigid deformations could occur, to another higher-dimensional space, in which
transformations across shapes are linear. However, using a supervised loss to
learn this embedding without enforcing any structural properties on the
underlying linear transform provides little guarantee that the learned
transform will generalize from the train to test setting.

#### Linearly Invariant self-symmetry embedding

We denote a map between a pair of shapes $\mathbf{X}$ and $\mathbf{Y}$ by
$T_{\mathbf{X}\mathbf{Y}}:\mathbf{X}\rightarrow\mathbf{Y}$ such that
$T_{\mathbf{X}\mathbf{Y}}(x_{i})=y_{j}$, $\forall
i\in\\{1,\ldots,n_{\mathbf{X}}\\}$ and some
$j\in\\{1,\ldots,n_{\mathbf{Y}}\\}$. This map can be represented by a matrix
$\Pi_{\mathbf{X}\mathbf{Y}}\in\mathbb{R}^{n_{\mathbf{X}}\times
n_{\mathbf{Y}}}$ such that $\Pi_{\mathbf{X}\mathbf{Y}}(i,j)=1$ if
$T_{\mathbf{X}\mathbf{Y}}(x_{i})=y_{j}$ and $0$ otherwise. We use the same
notation $T$ for self symmetry map $\mathbf{T}_{\mathbf{X}\mathbf{X_{f}}}$ as
well. We use $P_{\mathbf{X}}$ to denote the 3D coordinates of $\mathbf{X}$.

Our network takes a shape $\mathbf{X}$ as input as well as its point to point
symmetry map denoted as $\mathbf{T}_{\mathbf{X}\mathbf{X_{f}}}$. We then
perform a reflection (flip) of each shape along one axis resulting in a shape
denoted as $\mathbf{X_{f}}$. The original and flipped shapes are then
forwarded to a Siamese architecture, based on a PointNet Qi et al. (2017)
feature extractor, that embeds these two shapes into some fixed $k$
dimensional space. We illustrate in Figure 1 one such flip. The intuition
behind this operation is to help the network learn representation that can
disambiguate left from right in shape matching. Our first key idea is to learn
an embedding of each shape that would make the given self-symmetry map linear
in some higher-dimensional space.

Figure 1: Original Shape and its Flipped version from Surreal dataset

Let $\Phi_{\mathbf{X}}$ and $\Phi_{\mathbf{X_{f}}}$ denote the matrices, whose
rows can be interpreted as embeddings of the points of $\mathbf{X}$ and
$\mathbf{X_{f}}$. In the functional map framework, there exists a functional
map $C_{\mathbf{X}\mathbf{X_{f}}}$ that aligns the corresponding embeddings.
Given a self symmetry ground truth pointwise map
$\mathbf{T}_{\mathbf{X}\mathbf{X_{f}}}$, we can estimate
$C_{\mathbf{X}\mathbf{X_{f}}}$ by solving the following optimization problem:

$C_{\mathbf{X}\mathbf{X_{f}}}=\operatorname*{arg\,min}_{C}\|\Phi_{\mathbf{X}}C^{T}-\mathbf{T}_{\mathbf{X}\mathbf{X_{f}}}\Phi_{\mathbf{X_{f}}}\|_{2}$
(2)

The optimal symmetry map $C_{\mathbf{X}\mathbf{X_{f}}}$ is given by:
$C_{\mathbf{X}\mathbf{X_{f}}}=(\Phi_{\mathbf{X}}^{+}\mathbf{T}_{\mathbf{X}\mathbf{X_{f}}}\Phi_{\mathbf{X_{f}}})^{T}$,
that is differentiable using the closed-form expression of derivatives of
matrix inverses, as also mentioned in Section $3$. Similarly, we can compute
$C_{\mathbf{Y}\mathbf{Y_{f}}}$ for shape $\mathbf{Y}$.

###  4.1 Loss functions

Given a set of pairs of shapes $\mathbf{X},\mathbf{Y}$ for which ground truth
correspondences $\mathbf{T}^{gt}_{\mathbf{X}\mathbf{Y}}$ are known along with
a pointwise symmetry map, our network computes an embedding
$\Phi_{\mathbf{X}},\Phi_{\mathbf{Y}}$ for each shape as well as a self
symmetry functional map $C_{\mathbf{X}\mathbf{X_{f}}}$ and
$C_{\mathbf{Y}\mathbf{Y_{f}}}$ respectively as described above. We then
optimize the sum of three loss functions, one each defined for linearly
invariant self symmetry embedding, nearest neighbour based loss for pairwise
(shape pair) embedding and a commutativity loss for explicitly enforcing
intrinsic symmetry during training.

#### Linearly Invariant Loss

The first two loss functions are based on a soft-correspondence matrix, also
used in (Litany et al., 2017a) and (Marin et al., 2020). To define it for self
symmetry map, we _transform_ each shape embedding
$\widehat{\Phi}_{\mathbf{X}}=\Phi_{\mathbf{X}}C_{\mathbf{X}\mathbf{X_{f}}}^{T}$
by applying the optimal symmetry map. We then compare the rows of
$\widehat{\Phi}_{\mathbf{X}}$ to those of $\Phi_{\mathbf{X_{f}}}$ to obtain
the soft correspondence matrix $S_{\mathbf{X}\mathbf{X_{f}}}$ that
approximates the self-symmetry map in a differentiable way as follows:

$\displaystyle(S_{\mathbf{X}\mathbf{X_{f}}})_{ij}=\frac{e^{-\|\widehat{\Phi}_{\mathbf{X}}^{i}-\Phi_{\mathbf{X_{f}}}^{j}\|_{2}}}{\sum_{k=1}^{n}e^{-\|\widehat{\Phi}_{\mathbf{X}}^{i}-\Phi_{\mathbf{X_{f}}}^{k}\|_{2}}}$
(3)

We then define our loss that uses this soft-map to transfer the Euclidean
coordinates and compares the result to transferring the coordinates using the
ground truth map.

$\begin{split}L(\Phi_{\mathbf{X}},\Phi_{\mathbf{X_{f}}},\Phi_{\mathbf{Y}},\Phi_{\mathbf{Y_{f}}})_{lin.}=\sum\|S_{\mathbf{X}\mathbf{X_{f}}}P_{\mathbf{X_{f}}}-\mathbf{T}^{gt}_{\mathbf{X}\mathbf{X_{f}}}P_{\mathbf{X_{f}}}\|_{2}^{2}+\sum\|S_{\mathbf{Y}\mathbf{Y_{f}}}P_{\mathbf{Y_{f}}}-\mathbf{T}^{gt}_{\mathbf{Y}\mathbf{Y_{f}}}P_{\mathbf{Y_{f}}}\|_{2}^{2}\\\
\end{split}$ (4)

Note that this does not assume that the Euclidean coordinates to correspond.
Instead, this loss measures how well the predicted map transfers a particular
set of functions, compared to the ground truth map. This loss was introduced
in Marin et al. (2020) but we enforce it on the self-symmetry map.

#### Euclidean Loss

The loss described in the previous paragraph only considers the embedding of
each shape independently and aims to promote the structural property of this
embedding: i.e., that the symmetry map should be linear in the embedding
space.

Our next loss links the embeddings of the two shapes and is designed to
preserve the given ground truth mapping. Specifically, we first compute the
soft correspondence matrix $S_{\mathbf{X}\mathbf{Y}}$ between a pair of
shapes, by comparing the rows of $\Phi_{\mathbf{X}}$ to those of
$\Phi_{\mathbf{Y}}$ in a differentiable way as done in equation 3. We then
evaluate the computed soft map, again, by evaluating how well it transfers the
coordinate functions, compared to the given ground truth mapping.

$L(\Phi_{\mathbf{X}},\Phi_{\mathbf{Y}})_{euc.}=\sum\|S_{\mathbf{X}\mathbf{Y}}P_{\mathbf{Y}}-\mathbf{T}^{gt}_{\mathbf{X}\mathbf{Y}}P_{\mathbf{Y}}\|_{2}^{2}.$
(5)

Note that unlike the linearly invariant loss that we impose on the symmetry
maps, this loss is based on comparing $\Phi_{\mathbf{X}}$ and
$\Phi_{\mathbf{Y}}$ directly, without computing any linear transformations.
This significantly simplifies the learning process and in particular, reduces
the computation of the correspondence at test time to a simple nearest-
neighbor search. Despite this, as we show below, due to our strong
regularization, our approach achieves superior results compared to the method
of Marin et al. (2020), based on computing an optimal linear transformation at
test time.

#### Symmetry Commutativity Loss

Our final loss aims to link the symmetry map computed for each shape and the
correspondence across the two shapes. We achieve this by using the algebraic
properties of the functional representation, and especially using the fact
that map composition can simply be expressed as matrix multiplication.

Specifically, given a self-symmetry pointwise map on shapes $\mathbf{X}$ and
shape $\mathbf{Y}$, we aim to promote the consistency between the computed
correspondence and the symmetries on each shape. We do this by imposing the
following commutativity loss during training:

$L(\Phi_{\mathbf{X}},\Phi_{\mathbf{Y}})_{comm.}=\|S_{\mathbf{Y_{f}}\mathbf{Y}}S_{\mathbf{X}\mathbf{Y}}-S_{\mathbf{X}\mathbf{Y}}S_{\mathbf{X_{f}}\mathbf{X}}\|_{2}$
(6)

Intuitively, this loss considers the difference between mapping from
$\mathbf{X}$ to $\mathbf{Y}$ and applying the symmetry map on $\mathbf{Y}$, as
opposed to applying the symmetry on $\mathbf{X}$ and then mapping from
$\mathbf{X}$ to $\mathbf{Y}$. Note that this is similar to the commonly used
Laplacian commutativity in the functional maps literature. However, rather
than promoting isometries, our loss enforces that the computed map respects
the self-symmetry structure of each shape, which holds regardless of the
deformation class, and is not limited to isometries.

#### Overall training Loss

We combine the two embedding losses defined in equation 4 and equation 5 with
that of commutativity loss defined in equation 6 and define the training loss
as follows:

$L_{tot.}=L_{euc.}+\lambda*L_{lin.}+\gamma*L_{comm.}$ (7)

The scalars $\lambda$ and $\gamma$ allow us to weigh the symmetry information
differently in partial and full shape matching. Naturally, we set them higher
for full shape matching where enforcing symmetry structure makes more sense
than partial setting where symmetry is partial at times. We set $\lambda$ and
$\gamma$ to $5$ for full shape matching and $1$ and $.1$ for partial shape
matching based on a small validation set.

#### Test Phase

At test time, once the network is trained, we simply compute the embedding
$\Phi_{\mathbf{X}}$ and $\Phi_{\mathbf{Y}}$ and do a nearest neighbour search
between them to find correspondence between the two shapes.

#### Implementation Details

We implemented our method in Pytorch Paszke et al. (2019). All experiments are
run on a Nvidia RTX $2080$ graphics processing card and require $16$ GB of GPU
memory. For our experiments, similar to prior work (Sharma & Ovsjanikov, 2020;
Marin et al., 2020), we train over randomly selected $5000$ shapes from the
SURREAL dataset (Varol et al., 2017), where each point cloud is resampled
randomly at $3$K vertices. We learn a $k=50$ dimensional embedding (basis) for
each point cloud.

During training, we require self-symmetry ground truth as well as pairwise
ground truth map. Following (Sharma & Ovsjanikov, 2020; Marin et al., 2020),
our feature extractor is also based on the semantic segmentation architecture
of PointNet. We use a batch size of $20$ and learning rate of $1e-4$ and
optimize our objective with Adam optimizer in Pytorch Paszke et al. (2019).
Training iterations as well as other hyperparameters are validated on a small
validation set of $500$ shapes from the SURREAL dataset. Unlike Marin et al.
(2020) that uses fixed $1000$ points during train and test time, we use full
resolution at test time. In the SURREAL dataset, all point cloud contain
$6890$ points. During training, we randomly sample $3000$ points from the
point cloud and also augment the training set with random rotations applied to
input data. We obtain an embedding of $50$ dimensions during training by
validating from the set $40,50,60$. Our results are not sensitive to small
changes in these two parameters. We probe the effect of changing size of
embedding as well as the amount of training data on the resulting performance
and report it in the last section.

##  5 Results

This section is divided into two subsections. First subsection 5.1 shows the
experimental comparison of our approach with two state-of-the art methods for
near-isometric full shape matching. Section 5.2 demonstrates the effectiveness
on partial shape matching. We evaluate all results by reporting the per-point-
average geodesic distance between the ground truth map and the computed map.
All results are multiplied by 100 for the sake of readability.

###  5.1 Full Shape Matching

We present our results on a full shape matching benchmark dataset FAUST Bogo
et al. (2014). This dataset contains $100$ shapes of $10$ different subjects
in different poses where each point cloud contains $6890$ points. Following
prior work, we use the last $20$ shapes as a test set and report the
performance on this test set. We compare our results with Marin et al. (2020);
Sharma & Ovsjanikov (2020) in Table $2$ as they are applicable, in principle,
to both partial and complete shape matching. Note that Marin et al. (2020)
presents results on the FAUST data that is subsampled to $1000$ points both
during train and test. Our method obtains significantly better results than
Marin et al. (2020). Methods based on LBO eigen functions already form a good
basis for shapes and thus, prior work based on LBO eigen basis obtains
impressive performance. However, performance of this line of work degrades
significantly under partiality, as shown in the next section and also in Marin
et al. (2020).

#### Baselines

We compare with the following two state-of-the-art methods that are shown to
outperform existing competitors and our ablated baselines:

Sharma & Ovsjanikov (2020) This baseline, although weakly supervised, assumes
to be given as input a mesh representation of a shape and LBO basis. We
include it to demonstrate we can achieve competitive performance with the
methods that excel in full shape matching where LBO basis are stable.

Marin et al. (2020) This is considered state of the art for learning embedding
directly from data. Since we are testing with point clouds of much higher
resolution compared to the experiments in Marin et al. (2020), we retrain
their models with $3k$ subsampled points for each point cloud and show their
results with the best performing resolution of $3k$. The rest of the
parameters such as embedding size are used as specified in their paper as they
were found to be optimal. We use their open source code and retrain it on our
subset of Surreal dataset. Note that this baseline is somewhat different from
others since it requires and thus, learns both basis functions and probe
functions (feature descriptors).

Euc. Emb. This baseline ablates the overall performance of our method and
quantifies the gain brought in by the euclidean loss alone during training. It
shows the performance if we learn an embedding by just projecting the shapes
into a $50$ dimensional space with a nearest neighbour euclidean loss.

Euc. Emb. + comm. This baseline combines the above baseline with the
commutativity loss and quantifies what can be achieved without the linearly
invariant assumption on self-symmetry map. We denote our results with Euc.
Emb. + comm. + Lin. Inv. in the following Table. We also show the
corresponding curves below that are consistent with average geodesic error
shown in the Table below.

Method $\backslash$ Dataset | Faust
---|---
Marin et al. (2020)-3k | 08
Marin et al. (2020) \+ sym. | 09
Euc. Emb. | 12
Euc. Emb. + comm. | 10
Euc. Emb. + comm. + Lin. Inv. (Ours) | 05
Sharma & Ovsjanikov (2020) | 05

#### Discussion

Our ablation study shows that all three loss functions improve the overall
performance. We note that the performance gains brought in by linearly
invariant loss on self-symmetry embeddings are significant. We attribute our
superior quantitative results over other learning based methods to a range of
factors. First, in contrast to Marin et al. (2020) that is based on two stage
sequential architecture, our embedding is learned end to end in one phase.
Second, none of the state-of-the-art methods takes symmetry structure into
account even though symmetry ambiguities for shape matching is a well known
problem and studied extensively in axiomatic methods. Third, performance of
Marin et al. (2020) is sensitive to the size of embedding $20$. In contrast,
our method is quite robust and can train with twice their embedding size. We
compete well with Sharma & Ovsjanikov (2020) even though this approach relies
on LBO and exploiting mesh connectivity.

###  5.2 Partial Shape Matching

Datasets For a fair comparison with (Sharma & Ovsjanikov, 2020; Litany et al.,
2017b), we follow the same experimental setup and test our method on the
challenging SHREC’16 Partial Correspondence dataset (Cosmo et al., 2016). The
dataset is composed of 200 partial shapes, each containing about few hundreds
to 9000 vertices, belonging to 8 different classes (humans and animals),
undergoing nearly-isometric deformations in addition to having missing parts
of various forms and sizes. Each class comes with a “null” shape in a standard
pose which is used as the full template to which partial shapes are to be
matched. The dataset is split into two sets, namely cuts (removal of a few
large parts) and holes (removal of many small parts). We use the same test set
following (Sharma & Ovsjanikov, 2020). Overall, this test set contains $20$
shapes each for cuts and holes datasets chosen randomly from the two sets
respectively. In addition to Marin et al. (2020), we compare with the
following two baselines:

Sharma & Ovsjanikov (2020). This baseline relies on learning LBO alignment and
thus, is dependent on class that needs to be retrained for each of the 8
classes. We include their results even though our results are class agnostic
and thus, significantly more robust and efficient. We obtain these results by
running the code provided by the authors.

Litany et al. (2017b). This baseline is not learning based and relies on hand
crafted features and an expensive optimization scheme on the Stiefel manifold
for every pair of shapes at test time. Thus, in terms of computation and
ground truth map requirement, it is most expensive.

#### Results and Discussion

We present our findings on partial shape matching in Table 1 where we obtain
superior performance on both benchmark datasets for partial shape matching. In
addition, our result outperforms the euclidean embedding competitive baseline
by a significant margin and thus, validating the importance of working in the
symmetric space while learning canonical embedding. We would like to stress
that baseline such as Sharma & Ovsjanikov (2020) are class specific and need
to be trained each time whereas our method is class agnostic and can obtain
good results with a fraction of computational time. Similarly, Marin et al.
(2020) trains a similar network as our two times. First, it learns an
embedding with a network similar to ours, followed by a similar network
training to compute the optimal linear transformation between the two
embeddings. Moreover, the test phase also requires running the network twice.
Therefore, our method is at least twice faster than this baseline in
computational complexity.

Table 1: Avg. Geodesic Error on two partial SHREC benchmarks Method $\backslash$ Dataset | Holes | Cuts
---|---|---
Litany et al. (2017b) | 16 | 13
Sharma & Ovsjanikov (2020) | 14 | 16
Marin et al. (2020)-3k | 12 | 15
Euc. Emb. | 18 | 20
Ours | 10 | 13

#### Robustness to Embedding size

Our method is not sensitive to small variations in the embedding size. We
demonstrate this by changing the embedding size and plotting the average error
on the FAUST test set. Note that in contrast, Marin et al. (2020) is extremely
sensitive to changes in the size of feature descriptors (probe functions). We
show the sensitivity in Figure $2$.

#### Training Data Efficient

We experiment with $3$ training set of different sizes $1000,3000,5000$
sampled randomly and plot the results in Figure $3$. We observe a drop in the
performance of our method but the drop is slightly better than our baseline
(Marin et al., 2020).

Figure 2:

Figure 3:

##  6 Conclusion and Future Work

In shape correspondence literature, partial shape matching and complete shape
matching are generally tackled by two different sets of methods which obtain
impressive results in one of the two respective domains. We presented a
simple, general but effective method that reduces shape matching to a nearest
neighbour search problem in a canonical embedding and apply it to both partial
and complete shape matching. Our key idea is to learn an embedding of each
shape that would make the given self-symmetry map linear in some higher-
dimensional space. Our idea of injecting symmetry into the learning pipeline
also serves as a regularizer and provides competitive performance on multiple
shape matching benchmarks in comparison to all recent learning based methods.

There are several promising future directions to our work. First, our
architecture is based on a very simple PointNet feature extractor and thus,
there is a scope to integrate more advanced feature extractor that can exploit
the neighbourhood structure better while learning the embedding. Second,
following the advances in unsupervised deep functional maps, it would be
interesting to explore learning of canonical embedding with minimal
supervision.

##  7 Acknowledgement

Parts of this work were supported by the ERC Starting Grant StG-2017-758800
(EXPROTEA), and ANR AI chair AIGRETTE.

## References

  * Bogo et al. (2014) Federica Bogo, Javier Romero, Matthew Loper, and Michael J. Black.  FAUST: Dataset and evaluation for 3D mesh registration.  In _Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , Piscataway, NJ, USA, June 2014. IEEE.
  * Boscaini et al. (2016) Davide Boscaini, Jonathan Masci, Emanuele Rodola, and Michael M. Bronstein.  Learning shape correspondence with anisotropic convolutional neural networks.  In _Proc. NIPS_ , pp. 3189–3197, 2016.
  * Burghard et al. (2017) Oliver Burghard, Alexander Dieckmann, and Reinhard Klein.  Embedding shapes with Green’s functions for global shape matching.  _Computers & Graphics_, 68:1–10, 2017.
  * Chavdar et al. (2012) P Chavdar, H Sami, P Sven, Krieger Kai, and B. Darius.  Rigid 3d geometry matching for grasping of known objects in cluttered scenes.  _IJRR_ , 31(4), 2012.
  * Corman et al. (2014) Etienne Corman, Maks Ovsjanikov, and Antonin Chambolle.  Supervised descriptor learning for non-rigid shape matching.  In _Proc. ECCV Workshops (NORDIA)_ , 2014.
  * Cosmo et al. (2016) Luca Cosmo, Emanuele Rodola, Jonathan Masci, Andrea Torsello, and Michael M Bronstein.  Matching deformable objects in clutter.  In _3D Vision (3DV), 2016 Fourth International Conference on_ , pp. 1–10. IEEE, 2016.
  * Donati et al. (2020) Nicolas Donati, Abhishek Sharma, and Maks Ovsjanikov.  Deep geometric functional maps: Robust feature learning for shape correspondence.  In _CVPR_ , 2020.
  * Eisenberger et al. (2020) Marvin Eisenberger, Zorah Lahner, and Daniel Cremers.  Smooth shells: Multi-scale shape registration with functional maps.  In _CVPR_ , 2020.
  * Fernandez-Labrador et al. (2020) Clara Fernandez-Labrador, Ajad Chhatkuli, Danda Pani Paudel, Jose J Guerrero, Cédric Demonceaux, and Luc Van Gool.  Unsupervised learning of category-specific symmetric 3d keypoints from point sets.  In _European Conference on Computer Vision (ECCV)_ , 2020.
  * Gidaris et al. (2018) Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  In _ICLR_ , 2018.
  * Ginzburg & Raviv (2020) Dvir Ginzburg and D. Raviv.  Cyclic functional mapping: Self-supervised correspondence between non-isometric deformable shapes.  In _ECCV_ , 2020.
  * Groueix et al. (2018) Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry.  3d-coded: 3d correspondences by deep deformation.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pp. 230–246, 2018.
  * Halimi et al. (2019) Oshri Halimi, Or Litany, Emanuele Rodol‘a, Alex Bronstein, and Ron Kimmel.  Unsupervised learning of dense shape correspondence.  In _CVPR_ , 2019.
  * Huang et al. (2014) Qixing Huang, Fan Wang, and Leonidas Guibas.  Functional map networks for analyzing and exploring large shape collections.  _ACM Transactions on Graphics (TOG)_ , 33(4):36, 2014.
  * Kirgo et al. (2020) Maxime Kirgo, Simone Melzi, Giuseppe Patanè, Emanuele Rodolà, and Maks Ovsjanikov.  Wavelet-based heat kernel derivatives: Towards informative localized shape analysis.  In _Computer Graphics Forum_. Wiley Online Library, 2020.
  * Kovnatsky et al. (2013) Artiom Kovnatsky, Michael M Bronstein, Alexander M Bronstein, Klaus Glashoff, and Ron Kimmel.  Coupled quasi-harmonic bases.  In _Computer Graphics Forum_ , volume 32, pp. 439–448, 2013.
  * Lipman et al. (2010) Y. Lipman, Xiaobai Chen, I. Daubechies, and T. Funkhouser.  Symmetry factored embedding and distance.  In _SIGGRAPH_ , 2010.
  * Litany et al. (2017a) Or Litany, Tal Remez, Emanuele Rodolà, Alexander M. Bronstein, and Michael M. Bronstein.  Deep functional maps: Structured prediction for dense shape correspondence.  In _ICCV_ , pp. 5660–5668, 2017a.
  * Litany et al. (2017b) Or Litany, Emanuele Rodolà, Alex M Bronstein, and Michael M Bronstein.  Fully spectral partial shape matching.  In _Computer Graphics Forum_ , volume 36, pp. 247–258. Wiley Online Library, 2017b.
  * Marin et al. (2020) Riccardo Marin, Marie-Julie Rakotosaona, Simone Melzi, and Maks Ovsjanikov.  Correspondence learning via linearly-invariant embedding.  In _NeurIPS_ , volume 33, pp. 1608–1620, 2020.
  * Mitra et al. (2012) Niloy J. Mitra, Mark Pauly, Michael Wand, and Duygu Ceylan.  Symmetry in 3d geometry: Extraction and applications.  In _EUROGRAPHICS State-of-the-art Report_ , 2012.
  * Monti et al. (2017) Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodolà, Jan Svoboda, and Michael M. Bronstein.  Geometric deep learning on graphs and manifolds using mixture model cnns.  In _CVPR_ , pp. 5425–5434. IEEE Computer Society, 2017.
  * Neverova et al. (2020) Natalia Neverova, David Novotny, Vasil Khalidov, Marc Szafraniec, Patrick Labatut, and Andrea Vedaldi.  Continuous surface embeddings.  In _NeurIPS_ , 2020.
  * Nogneng & Ovsjanikov (2017) Dorian Nogneng and Maks Ovsjanikov.  Informative descriptor preservation via commutativity for shape matching.  _Computer Graphics Forum_ , 36(2):259–267, 2017\.
  * Ovsjanikov et al. (2012) Maks Ovsjanikov, Mirela Ben-Chen, Justin Solomon, Adrian Butscher, and Leonidas Guibas.  Functional Maps: A Flexible Representation of Maps Between Shapes.  _ACM Transactions on Graphics (TOG)_ , 31(4):30, 2012.
  * Ovsjanikov et al. (2013) Maks Ovsjanikov, Quentin Mérigot, Viorica Pătrăucean, and Leonidas Guibas.  Shape matching via quotient spaces.  In _SGP_ , pp. 1–11, 2013.
  * Ovsjanikov et al. (2017) Maks Ovsjanikov, Etienne Corman, Michael Bronstein, Emanuele Rodolà, Mirela Ben-Chen, Leonidas Guibas, Frederic Chazal, and Alex Bronstein.  Computing and processing correspondences with functional maps.  In _ACM SIGGRAPH 2017 Courses_ , pp. 5:1–5:62, 2017.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In _NeurIPS_ , pp. 8024–8035, 2019.
  * Qi et al. (2017) Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In _Proc. CVPR_ , pp. 652–660, 2017.
  * Raviv et al. (2010) D. Raviv, A. Bronstein, M. Bronstein, and R. Kimmel.  Full and partial symmetries of non-rigid shapes.  _International Journal of Computer Vision_ , 89:18–39, 2010\.
  * Ren et al. (2018) Jing Ren, Adrien Poulenard, Peter Wonka, and Maks Ovsjanikov.  Continuous and orientation-preserving correspondences via functional maps.  _ACM Transactions on Graphics (TOG)_ , 37(6), 2018.
  * Ren et al. (2020) Jing Ren, Simone Melzi, Maks Ovsjanikov, and Peter Wonka.  Maptree: Recovering multiple solutions in the space of maps.  _ACM Trans. Graph._ , 39(6), November 2020.
  * Rodolà et al. (2017) Emanuele Rodolà, Luca Cosmo, Michael M Bronstein, Andrea Torsello, and Daniel Cremers.  Partial functional correspondence.  In _Computer Graphics Forum_ , volume 36, pp. 222–236. Wiley Online Library, 2017.
  * Roufosse et al. (2019) Jean-Michel Roufosse, Abhishek Sharma, and Maks Ovsjanikov.  Unsupervised deep learning for structured shape matching.  In _ICCV_ , pp. 1617–1627, 2019.
  * Sahillioğlu (2019) Yusuf Sahillioğlu.  Recent advances in shape correspondence.  _The Visual Computer_ , pp. 1–17, 2019.
  * Sharma & Ovsjanikov (2020) Abhishek Sharma and Maks Ovsjanikov.  Weakly supervised deep functional maps for shape matching.  In _NeurIPS_ , volume 33, 2020.
  * Sharma & Ovsjanikov (2021) Abhishek Sharma and Maks Ovsjanikov.  Matrix decomposition on graphs: A functional view.  _arXiv_ , 2021.
  * Sharma et al. (2016) Abhishek Sharma, Oliver Grau, and Mario Fritz.  Vconv-dae: Deep volumetric shape learning without object labels.  In _ECCV_ , 2016.
  * Shi et al. (2020) Yifei Shi, Junwen Huang, Hongjia Zhang, Xin Xu, Szymon Rusinkiewicz, and Kai Xu.  Symmetrynet: Learning to predict reflectional and rotational symmetries of 3D shapes from single-view RGB-D images.  _ACM Transactions on Graphics (Proc. SIGGRAPH Asia)_ , 39, 2020.
  * Thomas et al. (2021) Oshane O. Thomas, Hongyu Shen, Ryan L. Raaum, William E.H. Harcourt-Smith, John D. Polk, and Mark Hasegawa-Johnson.  Automated morphological phenotyping using learned shape descriptors and functional maps: A novel approach to geometric morphometrics.  _bioRxiv_ , 2021.  URL https://www.biorxiv.org/content/early/2021/05/18/2021.05.18.444628.
  * Tombari et al. (2010) Federico Tombari, Samuele Salti, and Luigi Di Stefano.  Unique signatures of histograms for local surface description.  In _International Conference on Computer Vision (ICCV)_ , pp. 356–369, 2010.
  * Varol et al. (2017) Gül Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, and Cordelia Schmid.  Learning from synthetic humans.  In _CVPR_ , 2017.
  * Wang et al. (2020) Fudong Wang, Gui-Song Xia, Nan Xue, Yipeng Zhang, and M. Pelillo.  A functional representation for graph matching.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 42:2737–2754, 2020.
  * Wei et al. (2016) Lingyu Wei, Qixing Huang, Duygu Ceylan, Etienne Vouga, and Hao Li.  Dense human body correspondences using convolutional networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 1544–1553, 2016.
  * Wu et al. (2020) Yan Wu, Jun Yang, and Jinlong Zhao.  Partial 3d shape functional correspondence via fully spectral eigenvalue alignment and upsampling refinement.  _Comput. Graph._ , 92:99–113, 2020.