11institutetext: CAS Key Lab of Network Data Science and Technology, ICT, CAS
22institutetext: University of Chinese Academy of Sciences, Beijing, China
22email<EMAIL_ADDRESS>
22email: {fanyixing, chenjiangui18z, guojiafeng, zhangruqing<EMAIL_ADDRESS>
# RIGHT: Retrieval-augmented Generation for Mainstream Hashtag Recommendation

Run-Ze Fan 1122 0000-0002-8505-7756 Yixing Fan 1122 0000-0003-4317-2702
Jiangui Chen 1122 0000-0002-6235-6526 Jiafeng Guo Corresponding author1122
0000-0002-9509-8674
Ruqing Zhang 1122 0000-0003-4294-2541 Xueqi Cheng 1122 0000-0002-5201-8195

###### Abstract

Automatic mainstream hashtag recommendation aims to accurately provide users
with concise and popular topical hashtags before publication. Generally,
mainstream hashtag recommendation faces challenges in the comprehensive
difficulty of newly posted tweets in response to new topics, and the accurate
identification of mainstream hashtags beyond semantic correctness. However,
previous retrieval-based methods based on a fixed predefined mainstream
hashtag list excel in producing mainstream hashtags, but fail to understand
the constant flow of up-to-date information. Conversely, generation-based
methods demonstrate a superior ability to comprehend newly posted tweets, but
their capacity is constrained to identifying mainstream hashtags without
additional features. Inspired by the recent success of the retrieval-augmented
technique, in this work, we attempt to adopt this framework to combine the
advantages of both approaches. Meantime, with the help of the generator
component, we could rethink how to further improve the quality of the
retriever component at a low cost. Therefore, we propose RetrIeval-augmented
Generative Mainstream HashTag Recommender (RIGHT), which consists of three
components:  (i) a retriever seeks relevant hashtags from the entire tweet-
hashtags set;  (ii) a selector enhances mainstream identification by
introducing global signals; and  (iii) a generator incorporates input tweets
and selected hashtags to directly generate the desired hashtags.  The
experimental results show that our method achieves significant improvements
over state-of-the-art baselines. Moreover, RIGHT can be easily integrated into
large language models, improving the performance of ChatGPT by more than 10%.
Code will be released at: https://github.com/ict-bigdatalab/RIGHT.

###### Keywords:

Hashtag recommendation Retrieval-augmented generation Social media.

##  1 Introduction

Millions of user-generated microblogs flood Twitter daily, surpassing users’
comprehension. To facilitate rapid and easy understanding, hashtags (e.g.,
#ChatGPT) are extensively used to convey central ideas and topics, which also
enhance content visibility to reach a broader audience [24]. Such hashtags are
commonly referred to as mainstream hashtags, denoting their status as not only
the most prevalent hashtags but also semantically accurate. For instance, in
the context of Kobe Bryant’s untimely demise, both #KobeDead and #KobeDeath
can be utilized, with both possessing accurate semantic meanings. However, the
former holds more widespread usage and has achieved the distinction of being a
mainstream hashtag.

Figure 1: Illustration of evaluating hashtag recommendation with different
methods.

To provide mainstream hashtags, two main challenges need to be addressed.
First, comprehending a new tweet presents challenges primarily attributable to
the absence of real-time information [42, 50]. This is a direct consequence of
the continuous emergence of numerous new tweets in response to new topics and
events. Second, accurately identifying mainstream hashtags beyond semantic
correctness remains a challenging task. The reason is that numerous hashtags
could be used to describe a topic, but only a few are mainstream.

To address the above challenges, a considerable amount of work has been
proposed, which could be divided into two research lines [10, 13, 43, 15, 42].
Retrieval-based methods retrieve hashtags from a fixed predefined mainstream
hashtag list [43, 15], which could alleviate the second problem. However,
their ability to fully grasp the meaning of a newly posted tweet in response
to emerging topics and events is constrained. Moreover, it is a considerable
cost to maintain the predefined list [42]. In contrast, generation-based
methods [42, 50, 30] demonstrate remarkable proficiency in comprehending new
tweets and generating semantically accurate hashtags, owing to their
substantial pretraining knowledge. Nevertheless, they may encounter
difficulties when it comes to identifying mainstream hashtags without enough
mainstream information. As a result, the tweet might fail to be indexed by a
mainstream hashtag on microblog services due to the tags’ unpopularity,
weakening the recall rate of microblog searches. Inspired by the recent
success of retrieval-augmented generation technique [1, 37, 21, 12, 48],
therefore, we try to adapt this method to mainstream hashtags recommendation,
utilizing the advantages of both retrieval and generation approaches.

Typically, retrieval-augmented techniques incorporate the results of the
retriever, whether explicitly or implicitly, into the generator to enhance the
quality of generation. Utilizing this framework, the introduction of a
generator endowed with strong comprehensive capabilities might mitigate the
dependency on the quality of the retriever [31]. Thus, we could rethink the
trade-off between the quality and the cost of the retriever. Traditional
retrieval-based methods rely on a predefined list of mainstream hashtags,
which can ensure the quality of the retrieved information, but maintains such
a list at a significant cost. To reduce the maintenance burden, we transform
the small predefined list into a larger aggregation of existing tweet-hashtags
pairs, which can be automatically collected and updated without manual cost.
However, this approach carries the risk of introducing numerous low-quality
hashtags due to the informal characteristics of social media content. Such
hashtags have the potential to mislead the generator. As illustrated in Figure
1, both #KobeDead and #ATL are results of the retriever. Nonetheless, it is
noteworthy that #ATL is law-quality, even though tweet $1$ exhibits the
highest degree of similarity with the input tweet. Consequently, it becomes
imperative to further improve the quality of retrieved information without
increasing the cost.

Therefore, in this study, we propose a RetrIeval-augmented Generative
Mainstream HashTag Recommender (RIGHT), which combines the retriever and the
generator by the retrieval-augmented technique with inserting a selector.
Specifically, our method involves three components: 1) Retriever is utilized
to acquire relevant hashtags. We retrieve the tweets most similar to the input
from the tweet-hashtags corpus and obtain the corresponding hashtags set. 2)
Selector is used to improve the capability of identifying mainstream hashtags.
we incorporate three features, the similarity between the input tweet and the
retrieved tweet and its hashtags, as well as the frequency of the hashtags, to
enhance the mainstream information. 3) Generator is leveraged to provide
strong semantic comprehension and the ability of hashtag generation. We
concatenate the selected hashtags with the input tweet and feed it into the
generator to obtain the desired hashtags. In this way, we can utilize not only
the retriever and the selector to seek the mainstream hashtags but also the
generator to produce the desired hashtags flexibly.

We conduct experiments on two large-scale datasets (i.e., English Twitter
(THG) and Chinese Weibo (WHG)). Experimental results show that our method
achieves significant improvements over state-of-the-art baselines. Moreover,
as it can be easily incorporated in black-box language models, we also apply
our framework to ChatGPT by zero-shot instruction learning, bringing a 12.7%
boost for THG and 18.3% for WHG in F1@1. Finally, to deeply understand this
method, we present a detailed analysis.

##  2 Methodology

We propose RetrIeval-augmented Generative Mainstream HashTag Recommender
(RIGHT), a simple yet effective framework for mainstream hashtag
recommendation, which includes three components: retriever, selector, and
generator.

Overall, we first utilize the retriever to retrieve the tweets most similar to
the input from the existing tweet-hashtags corpus and obtain the corresponding
hashtags set. Then, we adopt a selector to select the hashtags that are most
probable mainstream from the retrieved labels using three signals. Finally, We
concatenate the selected hashtags with the input tweet and feed it into the
generator to obtain the desired hashtags. An overview of our method is shown
in Figure 2.

Figure 2: Our RIGHT framework consists of a retriever, a selector, and a
generator.

###  2.1 Retriever

The goal of the retriever is to retrieve the top-$N$ tweet-hashtags pairs on
the same topic with the input tweet from the existing corpus, aiming to find
relevant hashtags on the same topic.

Inspired by Wang et al. [41], we view the labeled training data as our corpus
and index these as input-label pairs, i.e.,
$\mathcal{C}=\\{(\tilde{t}_{i},\tilde{H}_{i})\\}$. Then, given the input tweet
$t$, the retrieval model $\mathcal{R}$ matches it with all tweets in the
corpus and returns the top-$N$ most similar tweet-hashtags pairs together with
their scores:

$\\{(\tilde{t}_{1},\tilde{H}_{1},\tilde{s}_{1}),\dots,(\tilde{t}_{N},\tilde{H}_{N},\tilde{s}_{N})\\}=\mathcal{R}(t|\mathcal{C}),$

where we denote $\tilde{s}_{i}$ as the similarity between $t$ and the $i$-th
retrieved tweet $\tilde{t}_{i}$. Each $\tilde{H}_{i}$ consists of hashtags
$\\{{\tilde{h}_{1}^{i}},\dots,{\tilde{h}_{|\tilde{H}_{i}|}^{i}}\\}$. We report
the results with sparse retrieval (e.g., BM25 [38]) and dense retrieval (e.g.,
SimCSE [11]) in experiments.

###  2.2 Selector

The goal of the selector is to filter the low-quality and non-mainstream
hashtags existing in the results of the retriever (see Figure 1). We consider
three mainstream features: the similarity between the input tweet and the
retrieved tweet and its hashtags, as well as the frequency of the hashtags.
Thus, we train a selector to compute the similarity between the tweet and the
hashtags and propose a simple algorithm for hashtag ranking.

####  2.2.1 Training.

The training data consists of positive samples and hard negative samples. Each
hashtag labeled in a tweet can be viewed as a positive sample $t^{+}$.
However, a significant challenge lies in constructing hard negative samples
($t^{-}$) to facilitate the efficient selection of mainstream hashtags on the
same topic by the selector. Inspired by BERT [9], we propose to create a hard
negative sample by disturbing the labeled hashtag without changing the
semantic meaning. Specifically, we randomly select a word to:  (i) replace
with its synonym 70% of the time;  (ii) delete 10% of the time;  (iii) swap
with the adjacent word 10% of the time;  (iv) insert a synonym after it 10% of
the time.  Thus, we obtain a training dataset
$\\{(t_{i},t_{i}^{+},t_{i}^{-})|i=1,\dots,N\\}$. Finally, we utilize
contrastive learning to train our selector by minimizing the following loss:

$\mathcal{L}_{\mathcal{S}}=-\log\frac{e^{\mathrm{sim}(\mathbf{h}_{t_{i}},\mathbf{h}_{t_{i}}^{+})/\tau}}{\sum_{j=1}^{L}{\left(e^{\mathrm{sim}(\mathbf{h}_{t_{i}},\mathbf{h}_{t_{j}}^{+})/\tau}+e^{\mathrm{sim}(\mathbf{h}_{t_{i}},\mathbf{h}_{t_{j}}^{-})/\tau}\right)}}$

where $\mathrm{sim}$ presents similarity, $\mathbf{h}_{t}$ indicates the
representation of $t$, $L$ is mini-batch size, and $\tau$ is a temperature
hyperparameter.

####  2.2.2 Inference.

In the inference stage, we propose a simple algorithm for hashtag ranking.
Given an input tweet $t$ and the result of the retriever
$\\{(\tilde{t}_{1},\tilde{H}_{1},\tilde{s}_{1}),\dots,\\\
(\tilde{t}_{N},\tilde{H}_{N},\tilde{s}_{N})\\}$, we put retrieved hashtags
into a set $\\{\tilde{h}_{1},\dots,\tilde{h}_{M}\\}$ where we denote $M$ as
the number of different hashtags, and record the number of occurrences
$\\{f_{1},\dots,f_{M}\\}$ and the corresponding score of each retrieved
hashtag $\\{\tilde{s}_{i,1}\dots,\tilde{s}_{i,f_{i}}\\}$. Then, we match the
input tweet $t$ with all hashtags using the selector $\mathcal{S}$ to obtain
the similarity score between the tweet and all hashtags
$\\{\ddot{s}_{1},\dots,\ddot{s}_{M}\\}$:

$\ddot{s}_{m}=\mathcal{S}(t,\tilde{h}_{m}).$ (1)

Finally, we average the tweet-to-tweet similarity for each hashtag and add the
similarity between the tweet and the hashtag. Since hashtags that occur more
frequently are more likely to be mainstream, we magnify the sum of the
similarity score and ranking score with a downscaled frequency:

$s_{i}=(({\frac{1}{f_{i}}}{\sum_{j=1}^{f_{i}}{\tilde{s}_{i,j}}})+\ddot{s}_{i})\times\left(1+\left((f_{i}-1)/10\right)\right),$

and sort the hashtags by the final score from largest to smallest to select
the top-$k$ hashtags $\\{\tilde{h}_{1},\dots,\tilde{h}_{k}\\}$.

###  2.3 Generator

The goal of the generator is to generate the desired hashtags, given an input
tweet $t$ and the selected hashtags $\\{\tilde{h}_{1},\dots,\tilde{h}_{k}\\}$.
We concatenate the input tweet with the retrieved hashtags and separate each
hashtag by a special token:

$I=<t,\mathtt{SEP1},\tilde{h}_{1},\mathtt{SEP1},\tilde{h}_{2},\dots,\mathtt{SEP1},\tilde{h}_{k}>,$

and feed it into the generator $\mathcal{G}$, which will output a concatenated
sequence $O$ that includes the hashtags, with each hashtag separated by
another token:

$O=<h_{1},\mathtt{SEP2},h_{2},\dots,\mathtt{SEP2},h_{|{H}|}>.$

By easily splitting by the special token, we would obtain the hashtag list
$H=\\{h_{1},h_{2},\dots,h_{|H|}\\}$.

The generative model could be Transformer-based encoder-decoder architecture
(e.g., T5 [35], BART [25]) or decoder-only architecture (e.g., a series of GPT
[33, 34, 2]). Thus, the training stage focuses on the finetuning of generative
models by minimizing the cross-entropy loss:

$\mathcal{L}_{\mathcal{G}}=\sum_{(I,O)\in\mathcal{D}}{-\log
p(O|I;\theta_{\mathcal{G}})},$

where $I$ is the input sequence consisting of the input tweet and the selected
hashtags, $O$ is the output sequence consisting of the desired hashtags, and
$\theta_{\mathcal{G}}$ is the parameters of the generator.

Table 1: Data statistics for the English Twitter hashtag generation (THG)
dataset and the Chinese Weibo hashtag generation (WHG) dataset. # T-H pairs
denotes the number of tweet-hashtags pairs. #AvgHashtags denotes the average
number of hashtags in each tweet-hashtags pair. AvgTweetLen denotes the avenge
length (token level) of all input tweets. AvgHashtagLen denotes the average
length (token level) of all hashtags.

Dataset | THG Dataset | WHG Dataset
---|---|---
Train | Validation | Test | Train | Validation | Test
# T-H pairs | 201444 | 11325 | 11328 | 307401 | 2000 | 2000
# AvgHashtags | 4.1 | 4.1 | 4.1 | 1.0 | 1.0 | 1.0
AvgTweetLen | 39.7 | 39.6 | 39.6 | 87.1 | 86.8 | 87.8
AvgHashtagLen | 3.1 | 3.0 | 3.0 | 6.6 | 6.5 | 6.5

##  3 Experiments

###  3.1 Experimental Setup

####  3.1.1 Datasets.

Our experiments are conducted on two large-scale datasets, which were crawled
from official media and influencers of social media [30]. The details are
shown in table 1.

  * •

THG: The English Twitter hashtag generation (THG) dataset has been crawled
from official Twitter sources, encompassing organizations, media outlets, and
other authenticated users, with the primary objective of acquiring tweets of
superior quality.

  * •

WHG: The Chinese Weibo hashtag generation (WHG) dataset has been acquired
through the systematic extraction of microblogs from Weibo, encompassing
notable sources including People’s Daily, People.cn, Economic Observe press,
Xinlang Sports, and various other accounts boasting over 5 million followers.
These accounts span diverse domains, encompassing politics, economics,
military affairs, sports, and more.

We use the training datasets as our retrieval corpus.

####  3.1.2 Evaluation Metric.

Following previous work [42, 30], we utilize ROUGE metrics and F1 scores at
$K$ as our evaluation metric. The average ROUGE score measures the overlap
between the generated sequence of hashtags (excluding special tokens) and the
reference sequence, including ROUGE-1, ROUGE-2, and ROUGE-L. For F1 scores at
$K$, different $K$ values result in a similar trend, so only F1@1 and F1@5 are
reported. We report results on the test dataset. Noticeably, for the WHG
dataset, where input posts have only one hashtag, F1@1 and F1@5 are identical,
so we only report F1@1 for this dataset.

####  3.1.3 Implementation Details.

Our implementation details of the retriever, selector, and generator are the
following:

  * •

For Retriever, we utilize BM25 [38] and SimCSE [11] (i.e., RoBERTa-Large [29]
for THG and Bert-Base-Chinese [9] for WHG) as our retrievers. Following Gao et
al. [11], we train our model for 3 epochs with a learning rate of 1e-5. The
hyperparameter of $N$ is set to 10, and the batch size is 6 per device.

  * •

For Selector, we use the training datasets from THG and WHG to construct our
hard negative samples, which are subsequently employed for training our
selectors in both English and Chinese independently. We utilize RoBERTa-Large
for THG and Bert-Base-Chinese for WHG. The temperature $\tau$ is $0.05$ and
other hyperparameters are the same as the retriever.

  * •

For Generator, we fine-tune a T5-base [35] for THG and a mT5-small [44] for
WHG and use Adam [22] as an optimizer. We set the weight decay and batch size
as 1e-5 and 16 and grid-search the learning rate, training epochs, and the
number of concatenated hashtags $k$ from {3e-4, 1e-4, 5e-5}, {5, 10}, and {1,
3, 5, 7, 9} respectively. The maximum length is 180 for T5-base and 256 for
mT5-small. The special token $\mathtt{SEP1}$ and $\mathtt{SEP2}$ are
$\mathtt{<extra\\_id\\_0>}$ and $\mathtt{<extra\\_id\\_1>}$ respectively.

All models are trained on four NVIDIA Tesla K80.

Table 2: The prompt used for ChatGPT. The Chinese version is its translation. Baseline |  Instruction
---|---
ChatGPT |  I want you to act as a hashtag annotator. I will provide you a tweet and your role is to annotate the relevant hashtag. You should use the related knowledge and find the topic. I want you only reply the hashtags segmented by “#” and nothing else, do not write explanations. I want you segment the word in a hashtag by space. My first tweet is {Input Tweet}.
$\mathrm{\mathbf{RIGHT}}^{\mathrm{\mathbf{ChatGPT}}}$ |  I want you to act as a hashtag annotator. I will provide you with a tweet, and your role is to annotate the relevant hashtag. Using your related knowledge, you should identify the topic and reply with only the hashtags segmented by “#”, without any explanations. Make sure to capitalize the first letter of the word. Make sure to split every word in a hashtag by a space. There are some potential hashtags:[{Retrieved Top-$k$ Hashtags}]. You can decide whether use the part of them or not. My first tweet is {Input Tweet}.

####  3.1.4 Baselines.

Our baselines consist of retrieval-based methods, generation-based methods,
and retrieval-augmented generative methods:

  * •

Retrieval-based methods: Following Mao et al. [30], we construct the
predefined hashtags list from all hashtags in the training datasets and select
top-4 hashtags for THG and top-1 for WHG according to the average number of
hashtags in each data item. We apply BM25 and SimCSE to the hashtag
recommendation:  (i) BM25[38]is a traditional strong sparse retrieval based on
term matching.  (ii) SimCSE[11]is a representative dense retriever, which
applies a simple contrastive learning framework to present sentence embeddings
on semantic textual similarity tasks. We fine-tuned SimCSE by constructing
positive samples and hard negative samples from BM25.

  * •

Generation-based methods: We consider three predominant generative methods:
(i) ChatGPTis a powerful large language model to execute various NLP tasks
[23]. Specifically, we adopt gpt-3.5-turbo and instruction zero-shot learning
to evaluate our task (Prompts are shown in Table 2).  (ii) SEGTRM Soft[30]is
the previous SOTA on our datasets, an end-to-end generative method segments
selection-based deep transformer.  (iii) Seq2Seq[42]is the first generation-
based method for hashtag recommendation. Due to the unreality to assume the
existence of conversations before publishing the tweet [50] and the lack of
conversation contexts, we reimplement the Seq2Seq model on the pretrained
language model (T5-base for THG and mT5-small for WHG) to formulate this task
to a seq-to-seq paradigm.

  * •

Retrieval-augmented Generative Methods (Ours): We apply our
retrieval-augmented framework to ChatGPT by incorporating the retrieval
results into the instruction to prompt the model to generate mainstream
hashtags, denoted as $\mathrm{\mathbf{RIGHT}}^{\mathrm{\mathbf{ChatGPT}}}$
(Prompts are shown in Table 2). We only use the best retriever on the datasets
(i.e., SimCSE for THG and BM25 for WHG), due to the high cost of ChatGPT.
Moreover, we use BM25 and SimCSE as our retriever of RIGHT, denoted them as
$\mathrm{\mathbf{RIGHT}}_{\mathrm{\mathbf{BM25}}}$ and
$\mathrm{\mathbf{RIGHT}}_{\mathrm{\mathbf{SimCSE}}}$.

Table 3: Main results (%) on the THG and WHG datasets. Bold and underline
indicate the best and second method respectively. We donate ROUGE as RG. $*$
indicates statistically significant improvements over all baselines (p-value
$<0.05$).

Model | THG | WHG
---|---|---
RG-1 | RG-2 | RG-L | F1@1 | F1@5 | RG-1 | RG-2 | RG-L | F1@1
Retrieval-based Methods
BM25 | 16.23 | 4.17 | 15.11 | 5.92 | 9.84 | 61.98 | 58.76 | 61.81 | 48.20
SimCSE | 28.43 | 10.34 | 26.38 | 12.40 | 15.15 | 59.71 | 55.81 | 59.54 | 47.65
Generation-based Methods
ChatGPT | 44.60 | 27.67 | 39.29 | 9.72 | 26.08 | 32.27 | 24.54 | 31.80 | 7.9
SEGTRM Soft | 51.18 | 37.15 | 47.05 | 27.17 | 29.02 | 55.51 | 51.28 | 54.30 | 30.72
Seq2Seq | 59.90 | 41.39 | 59.15 | 29.75 | 41.71 | 66.64 | 61.71 | 66.39 | 48.60
Retrieval-augmented Generative Methods (Ours)
$\mathrm{RIGHT}^{\mathrm{ChatGPT}}$ | 47.54 | 25.63 | 44.47 | 22.39 | 31.09 | 48.17 | 41.51 | 47.75 | 26.15
$\mathrm{RIGHT}_{\mathrm{BM25}}$ | 61.60 | 43.77 | 60.85 | 30.27 | 42.98 | 70.62∗ | 66.12∗ | 70.35∗ | 53.85∗
$\mathrm{RIGHT}_{\mathrm{SimCSE}}$ | 62.11∗ | 43.86∗ | 61.39∗ | 30.58∗ | 43.23∗ | 68.84 | 64.19 | 68.56 | 51.50

###  3.2 Main Results

As shown in Table 3, we can observe that:

  1. 1.

Among the retrieval-based methods, the performance of SimCSE outperforms BM25
in THG, while BM25 demonstrates superior performance in WHG. This difference
may be attributed to that English hashtags tend to be concise summaries, while
Chinese hashtags often comprise small sentences extracted directly from the
input text. Consequently, dense retrieval approaches utilizing semantic
matching may be more suitable for English datasets, while Chinese datasets may
benefit more from sparse retrieval techniques based on term matching.

  2. 2.

Among the generation-based methods, Seq2Seq performs well on both datasets,
potentially attributable to the utilization of mainstream hashtag knowledge
from the training dataset during fine-tuning. However, ChatGPT lags behind
other generation methods, suggesting a deficiency in mainstream hashtag
knowledge despite its vast repository of general knowledge.

  3. 3.

Among the retrieval-augmented generative methods, retrieval augmentation
brings the performance of baselines to a new level, demonstrating the
effectiveness of our method. For ChatGPT, retrieval augmentation boosts F1@1
performance by 12.67% for THG and 18.25% for WHG, indicating the substantial
value of mainstream hashtag knowledge. For RIGHT, both sparse and dense
retrievers show the potential to enhance performance compared with Seq2Seq.
Specifically, SimCSE is particularly effective for THG, while BM25 performs
better for WHG. The reason could be attributed to the superiority in the
performance of retrieval-based methods is directly proportional to the
enhancement of the retrieval augmentation. Moreover, different retrievers
excel in different scenarios, emphasizing the importance of the careful
selection of retrievers based on specific use cases. The performance of the
retrieval-based approach serves as a preliminary guide for informed decision-
making.

Overall, our method shows robustness across various scenarios, whether applied
with a fine-tuned generation model or a large black box language model.
Regardless of the retrieval approach used, our method consistently improves
performance.

###  3.3 Analysis

####  3.3.1 Ablation Study.

We conduct an ablation study to explore the impact of each component in RIGHT
on THG: 1) w/o Retriever: We remove the retriever and randomly concatenate $k$
hashtags from the training dataset with the input tweet. 2) w/o Selector: We
remove the selector and directly use the top-$k$ hashtags from the retriever’s
results by the similarity between the input tweet and the retrieved tweet. 3)
w/o Generator: We remove the generator and output the top-4 hashtags produced
by the selector. Table 4 presents the results, indicating that:

  1. 1.

The performance improvement is considerable through the integration of the
retriever, confirming that the incorporation of mainstream hashtag knowledge
indeed facilitates accurate hashtag selection.

  2. 2.

Without the selector, the performance gains are limited, indicating that
simply being on the same topic is insufficient. It is crucial to identify and
incorporate mainstream hashtags.

  3. 3.

The generator is crucial in RIGHT, emphasizing the significant impact of
semantic comprehension on performance. In contrast to the retrieval-based
approaches, it is more powerful to directly output the hashtags in the tweet-
hashtags pair that are most similar to the input.

Table 4: Ablation study results on the THG datasets. Bold indicates the best
method.

Model | ROUGE-1 | ROUGE-2 | ROUGE-L | F1@1 | F1@5
---|---|---|---|---|---
RIGHT | 62.11 | 43.86 | 61.39 | 30.58 | 43.23
w/o Retriever | 59.91 | 41.63 | 59.23 | 29.66 | 41.70
w/o Selector | 60.49 | 42.06 | 59.76 | 30.22 | 41.95
w/o Generator | 36.24 | 16.02 | 32.86 | 24.61 | 26.73

####  3.3.2 Impact of the Number of Augmented Hashtags.

To explore the impact of the number of concatenated hashtags with the input
tweet, we conduct a series of experiments. Specifically, we concatenate
various top-$k$ ($k=1,3,5,7,9$) with the input tweet for the THG and the WHG
datasets. Figure 3 demonstrates the Rouge-1 results (other metrics show the
same trends), showing that:

  1. 1.

Retrieval augmentation aids in improving the performance when a sufficient
number is considered, suggesting that augmenting more hashtags increases the
probability of covering mainstream hashtags and makes the generator more
robust in the presence of mismatches from certain hashtags.

  2. 2.

Upon reaching a certain threshold of the number of augmented hashtags (i.e.,
$k=7$), the performance converges, suggesting that the majority of mainstream
hashtags might have already been augmented.

13579$60.7$$60.9$$61.1$$61.3$$61.5$$61.7$$61.9$$62.1$$k$ROUGE-1
(%)THG13579$68.9$$69.1$$69.3$$69.5$$69.7$$69.9$$70.1$$70.3$$70.5$$70.7$$k$ROUGE-1
(%)WHG Figure 3: Our Rouge-1 results in the different number of augmented
hashtags (i.e., $k=1,3,5,7,9$).

####  3.3.3 Case Study.

To validate the successful recall of mainstream hashtags and the potential for
further improvement, we analyze the successful and unsuccessful cases and
present a representative case study in Table 5. We conclude that:

  1. 1.

Some retrieved tweets share the same topic as the input tweets but have subpar
labeled hashtags (e.g., “fx logix” in Table 5). Fortunately, our generator
demonstrates the capability to disregard these irrelevant hashtags.

  2. 2.

Some retrieved tweets are partially relevant to the input tweet. Although the
retrieved hashtags align well with the topic of the retrieved tweet, it is not
highly pertinent to the primary topic of the input tweet (e.g., “azure” in the
retrieved hashtags). Nonetheless, our generator can filter out these
irrelevant hashtags.

  3. 3.

The generation model produces a semantically accurate but non-mainstream
hashtag “windows virtual desktop” by directly copying the original word from
the input tweet due to its limited knowledge of mainstream hashtags. However,
our retriever and selector effectively identify the corresponding mainstream
hashtag in its abbreviated form “wvd”. Our RIGHT successfully replaced the
original hashtag with the mainstream hashtag, indicating the effectiveness of
retrieval augmentation.

  4. 4.

Seq2Seq generates certain hashtags that are also retrieved by the retriever,
while RIGHT fails to generate them (e.g., “microsoft”). These cases constitute
less than 1% of the total. We speculate that this discrepancy may be due to
the selector placing the correct hashtags toward the end of the list, leading
to reduced confidence and subsequent non-adoption by the generator.

Table 5: An example from the THG test set. Correct results are marked bold.

Input: |  Geeks guide to microsoft teams optimization with windows virtual desktop citrix.
---|---
Label: |  microsoft; windows; citrix; wvd
Retriever & Selector: |  citrix; wvd; fs logix; v mware; azure; aws; microsoft
Seq2Seq: |  microsoft; windows virtual desktop; citrix; vdi
RIGHT: |  citrix; wvd

##  4 Related Works

Our work mainly builds on two streams of previous work: hashtag recommendation
and retrieval-augmented generation.

###  4.1 Mainstream hashtag recommendation

Mainstream hashtag recommendation aims to provide users with short topical and
popular tags representing the main ideas of their tweets before publication.
Three primary methods have been proposed for this task [24]:

1) Keyphrase extraction method formulates this task as keyphrases extraction
from source posts [14, 47, 49], which fails to produce hashtags that do not
appear in the microblog posts while large freedom is allowed for users to
write whatever hashtags they like. The performance of this method is much
lower than other methods. 2) Retrieval-based method aims to retrieve from a
predefined hashtag list [43, 15, 19, 46], which is limited to generating only
the hashtags that are included in the list. In reality, a wide range of
hashtags can be created every day, resulting impossibility to be covered by a
fixed list and the difficulty to maintain the list. 3) Generation-based method
was proposed [42, 50, 30, 32] to overcome the aforementioned challenges, which
formulates the task as a sequence-to-sequence generation paradigm, allowing
for the creation of a wider range of hashtags that better capture the main
ideas of the microblog post. However, previous studies pay limited attention
to mainstream hashtags. Consequently, even though it produces semantically
correct tags, the tweet might fail to be indexed by a mainstream hashtag on
microblog services due to tags’ unpopularity, thus weakening the recall rate
of microblog searches.

To the best of our knowledge, we are the first to alleviate this issue by
combining retrieval and generation methods. Meanwhile, we improve the quality
of the retriever at a low cost.

###  4.2 Retrieval-augmented Generation

The retrieval-augmented generation represents a novel paradigm that merges
pre-trained generative models with information retrieval techniques [1, 26].
Previous research in this field primarily has focused on introducing external
knowledge to address knowledge-intensive tasks [4, 45, 18, 27, 39, 40, 17, 8,
6, 7] and utilizing similar data to enhance the model performance across
various natural language processing (NLP) tasks, including image captioning
[37], keyphrase generation [21, 12], named entity recognition [48, 5], and
others. Recently, this technique has also been used in large language models
to alleviate issues like factual hallucination [3, 36, 20], knowledge out-
dating [16], and the lack of domain-specific expertise [28].

Notably, we adopt this framework to the mainstream hashtag recommendation
task, by introducing a selector combining global signals to improve mainstream
identification.

##  5 Conclusion

In this study, we have proposed a simple yet effective retrieval-augmented
generative recommender, designed to utilize the advantage of retrieval and
generation methods for mainstream hashtag recommendation. To improve the
quality of the retriever’s results at a low cost, we have integrated a
selector module into the conventional retrieval-augmented framework.
Specifically, the retriever’s role is to find relevant hashtags on the same
topic, the selector is employed to enhance the identification of mainstream
hashtags, and the generator is responsible for combining input tweets and
selected hashtags to generate desired hashtags. We have conducted extension
experiments using two extensive datasets to validate the effectiveness of our
approach.

In future work, it is valuable to explore optimal strategies for combining the
retrieval-based method with the generation-based method, as well as developing
a co-training approach that jointly refines the three components.

## Acknowledgements

This work was funded by the National Natural Science Foundation of China
(NSFC) under Grants No. 62372431, and 62006218, the Youth Innovation Promotion
Association CAS under Grants No. 2021100, the project under Grants No.
2023YFA1011602, JCKY2022130C039 and 2021QY1701, and the Lenovo-CAS Joint Lab
Youth Scientist Project. All content represents the opinion of the authors,
which is not necessarily shared or endorsed by their respective employers
and/or sponsors.

## References

  * [1] Asai, A., Min, S., Zhong, Z., Chen, D.: Retrieval-based language models and applications. In: ACL. pp. 41–46. Toronto, Canada (Jul 2023). https://doi.org/10.18653/v1/2023.acl-tutorials.6, https://aclanthology.org/2023.acl-tutorials.6
  * [2] Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.: Language models are few-shot learners. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) NeurIPS (2020), https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html
  * [3] Cao, M., Dong, Y., Wu, J., Cheung, J.C.K.: Factual error correction for abstractive summarization models. In: EMNLP (Nov 2020). https://doi.org/10.18653/v1/2020.emnlp-main.506, https://aclanthology.org/2020.emnlp-main.506
  * [4] Chen, D., Fisch, A., Weston, J., Bordes, A.: Reading Wikipedia to answer open-domain questions. In: ACL (Jul 2017). https://doi.org/10.18653/v1/P17-1171, https://aclanthology.org/P17-1171
  * [5] Chen, J., Zhang, R., Guo, J., Fan, Y., Cheng, X.: GERE: generative evidence retrieval for fact verification. In: SIGIR 2022. pp. 2184–2189. ACM (2022). https://doi.org/10.1145/3477495.3531827, https://doi.org/10.1145/3477495.3531827
  * [6] Chen, J., Zhang, R., Guo, J., Liu, Y., Fan, Y., Cheng, X.: Corpusbrain: Pre-train a generative retrieval model for knowledge-intensive language tasks. In: CIKM 2022. pp. 191–200. ACM (2022). https://doi.org/10.1145/3511808.3557271, https://doi.org/10.1145/3511808.3557271
  * [7] Chen, J., Zhang, R., Guo, J., de Rijke, M., Chen, W., Fan, Y., Cheng, X.: Continual learning for generative retrieval over dynamic corpora. In: CIKM 2023. pp. 306–315. ACM (2023). https://doi.org/10.1145/3583780.3614821, https://doi.org/10.1145/3583780.3614821
  * [8] Chen, J., Zhang, R., Guo, J., de Rijke, M., Liu, Y., Fan, Y., Cheng, X.: A unified generative retriever for knowledge-intensive language tasks via prompt learning. In: SIGIR 2023. pp. 1448–1457. ACM (2023). https://doi.org/10.1145/3539618.3591631, https://doi.org/10.1145/3539618.3591631
  * [9] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: NAACL (Jun 2019). https://doi.org/10.18653/v1/N19-1423, https://aclanthology.org/N19-1423
  * [10] Ding, Z., Zhang, Q., Huang, X.: Automatic hashtag recommendation for microblogs using topic-specific translation model. In: COLING (Dec 2012), https://aclanthology.org/C12-2027
  * [11] Gao, T., Yao, X., Chen, D.: SimCSE: Simple contrastive learning of sentence embeddings. In: EMNLP (Nov 2021). https://doi.org/10.18653/v1/2021.emnlp-main.552, https://aclanthology.org/2021.emnlp-main.552
  * [12] Gao, Y., Yin, Q., Li, Z., Meng, R., Zhao, T., Yin, B., King, I., Lyu, M.: Retrieval-augmented multilingual keyphrase generation with retriever-generator iterative training. In: NAACL (2022). https://doi.org/10.18653/v1/2022.findings-naacl.92, https://aclanthology.org/2022.findings-naacl.92
  * [13] Gong, Y., Zhang, Q., Huang, X.: Hashtag recommendation using Dirichlet process mixture models incorporating types of hashtags. In: EMNLP (2015). https://doi.org/10.18653/v1/D15-1046, https://aclanthology.org/D15-1046
  * [14] Gong, Y., Zhang, Q., Huang, X.: Hashtag recommendation using dirichlet process mixture models incorporating types of hashtags. In: Màrquez, L., Callison-Burch, C., Su, J., Pighin, D., Marton, Y. (eds.) EMNLP (2015). https://doi.org/10.18653/v1/d15-1046, https://doi.org/10.18653/v1/d15-1046
  * [15] Gong, Y., Zhang, Q.: Hashtag recommendation using attention-based convolutional neural network. In: Kambhampati, S. (ed.) IJCAI (2016), http://www.ijcai.org/Abstract/16/395
  * [16] He, H., Zhang, H., Roth, D.: Rethinking with retrieval: Faithful large language model inference. arXiv preprint (2022), https://arxiv.org/pdf/2301.00303.pdf
  * [17] He, S., Fan, R.Z., Ding, L., Shen, L., Zhou, T., Tao, D.: Mera: Merging pretrained adapters for few-shot learning. arXiv preprint arXiv:2308.15982 (2023), https://arxiv.org/abs/2308.15982
  * [18] He, S., Fan, R.Z., Ding, L., Shen, L., Zhou, T., Tao, D.: Merging experts into one: Improving computational efficiency of mixture of experts. In: Bouamor, H., Pino, J., Bali, K. (eds.) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 14685–14691. Association for Computational Linguistics, Singapore (Dec 2023), https://aclanthology.org/2023.emnlp-main.907
  * [19] Huang, H., Zhang, Q., Gong, Y., Huang, X.: Hashtag recommendation using end-to-end memory networks with hierarchical attention. In: COLING (Dec 2016), https://aclanthology.org/C16-1090
  * [20] Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y.J., Madotto, A., Fung, P.: Survey of hallucination in natural language generation. ACM Comput. Surv. (2023). https://doi.org/10.1145/3571730, https://doi.org/10.1145/3571730
  * [21] Kim, J., Jeong, M., Choi, S., Hwang, S.w.: Structure-augmented keyphrase generation. In: EMNLP (2021). https://doi.org/10.18653/v1/2021.emnlp-main.209, https://aclanthology.org/2021.emnlp-main.209
  * [22] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) ICLR (2015), http://arxiv.org/abs/1412.6980
  * [23] Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., Kocoń, A., Koptyra, B., Mieleszczenko-Kowszewicz, W., Miłkowski, P., Oleksy, M., Piasecki, M., Łukasz Radliński, Wojtasik, K., Woźniak, S., Kazienko, P.: Chatgpt: Jack of all trades, master of none. arXiv preprint (2023), https://arxiv.org/pdf/2302.10724.pdf
  * [24] Kwak, H., Lee, C., Park, H., Moon, S.: What is twitter, a social network or a news media? In: WWW (2010), https://ink.library.smu.edu.sg/cgi/viewcontent.cgi?article=7104&context=sis_research
  * [25] Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., Zettlemoyer, L.: BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J.R. (eds.) ACL (2020). https://doi.org/10.18653/v1/2020.acl-main.703, https://doi.org/10.18653/v1/2020.acl-main.703
  * [26] Lewis, P.S.H., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., Kiela, D.: Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) NeurIPS (2020), https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html
  * [27] Li, J., Sun, S., Yuan, W., Fan, R.Z., Zhao, H., Liu, P.: Generative judge for evaluating alignment. arXiv preprint arXiv:2310.05470 (2023), https://arxiv.org/abs/2310.05470
  * [28] Li, X., Zhu, X., Ma, Z., Liu, X., Shah, S.: Are chatgpt and gpt-4 general-purpose solvers for financial text analytics? an examination on several typical tasks. arXiv preprint (2023), https://arxiv.org/pdf/2305.05862.pdf
  * [29] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint (2019), https://arxiv.org/pdf/1907.11692.pdf
  * [30] Mao, Q., Li, X., Liu, B., Guo, S., Hao, P., Li, J., Wang, L.: Attend and select: A segment selective transformer for microblog hashtag generation. Knowledge-Based Systems (2022). https://doi.org/https://doi.org/10.1016/j.knosys.2022.109581, https://www.sciencedirect.com/science/article/pii/S0950705122007973
  * [31] Mialon, G., Dessì, R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu, R., Rozière, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., et al.: Augmented language models: a survey. arXiv preprint (2023), https://arxiv.org/pdf/2302.07842.pdf
  * [32] Ni, S., Bi, K., Guo, J., Cheng, X.: A comparative study of training objectives for clarification facet generation. In: Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region. pp. 1–10 (2023)
  * [33] Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al.: Improving language understanding by generative pre-training. preprint (2018), https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf
  * [34] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. preprint (2019), https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
  * [35] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. (2020), http://jmlr.org/papers/v21/20-074.html
  * [36] Raunak, V., Menezes, A., Junczys-Dowmunt, M.: The curious case of hallucinations in neural machine translation. In: ACL (Jun 2021). https://doi.org/10.18653/v1/2021.naacl-main.92, https://aclanthology.org/2021.naacl-main.92
  * [37] Rita Ramos, Desmond Elliott, B.M.: Retrieval-augmented image captioning. In: EACL (2023), https://arxiv.org/abs/2302.08268
  * [38] Robertson, S.E., Walker, S.: Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In: Croft, W.B., van Rijsbergen, C.J. (eds.) SIGIR (1994). https://doi.org/10.1007/978-1-4471-2099-5_24, https://doi.org/10.1007/978-1-4471-2099-5_24
  * [39] Wang, S., Gan, T., Liu, Y., Wu, J., Cheng, Y., Nie, L.: Micro-influencer recommendation by multi-perspective account representation learning. IEEE Transactions on Multimedia (2022)
  * [40] Wang, S., Gan, T., Liu, Y., Zhang, L., Wu, J., Nie, L.: Discover micro-influencers for brands via better understanding. IEEE Transactions on Multimedia 24, 2595–2605 (2021)
  * [41] Wang, S., Xu, Y., Fang, Y., Liu, Y., Sun, S., Xu, R., Zhu, C., Zeng, M.: Training data is more valuable than you think: A simple and effective method by retrieving from training data. In: ACL (May 2022). https://doi.org/10.18653/v1/2022.acl-long.226, https://aclanthology.org/2022.acl-long.226
  * [42] Wang, Y., Li, J., King, I., Lyu, M.R., Shi, S.: Microblog hashtag generation via encoding conversation contexts. In: NAACL (Jun 2019). https://doi.org/10.18653/v1/N19-1164, https://aclanthology.org/N19-1164
  * [43] Weston, J., Chopra, S., Adams, K.: #TagSpace: Semantic embeddings from hashtags. In: EMNLP (Oct 2014). https://doi.org/10.3115/v1/D14-1194, https://aclanthology.org/D14-1194
  * [44] Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., Raffel, C.: mt5: A massively multilingual pre-trained text-to-text transformer. In: Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tür, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., Zhou, Y. (eds.) NAACL (2021). https://doi.org/10.18653/v1/2021.naacl-main.41, https://doi.org/10.18653/v1/2021.naacl-main.41
  * [45] Zhang, H., Zhang, R., Guo, J., de Rijke, M., Fan, Y., Cheng, X.: From relevance to utility: Evidence retrieval with feedback for fact verification. In: Bouamor, H., Pino, J., Bali, K. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2023. pp. 6373–6384. Association for Computational Linguistics, Singapore (Dec 2023), https://aclanthology.org/2023.findings-emnlp.422
  * [46] Zhang, Q., Wang, J., Huang, H., Huang, X., Gong, Y.: Hashtag recommendation for multimodal microblog using co-attention network. In: Sierra, C. (ed.) IJCAI (2017). https://doi.org/10.24963/ijcai.2017/478, https://doi.org/10.24963/ijcai.2017/478
  * [47] Zhang, Q., Wang, Y., Gong, Y., Huang, X.: Keyphrase extraction using deep recurrent neural networks on Twitter. In: EMNLP (Nov 2016). https://doi.org/10.18653/v1/D16-1080, https://aclanthology.org/D16-1080
  * [48] Zhang, X., Jiang, Y., Wang, X., Hu, X., Sun, Y., Xie, P., Zhang, M.: Domain-specific NER via retrieving correlated samples. In: COLING (2022), https://aclanthology.org/2022.coling-1.211
  * [49] Zhang, Y., Li, J., Song, Y., Zhang, C.: Encoding conversation context for neural keyphrase extraction from microblog posts. In: NAACL (Jun 2018). https://doi.org/10.18653/v1/N18-1151, https://aclanthology.org/N18-1151
  * [50] Zheng, X., Mekala, D., Gupta, A., Shang, J.: News meets microblog: Hashtag annotation via retriever-generator. arXiv preprint (2021), https://arxiv.org/abs/2104.08723


$m_{b2}$ and the local nuisance variable $l_{12}$. Also given are the number
of unique configurations of these local state spaces compatible with fixed $C$
and $D$ symbols in Alice’s and Bob’s events.


# LAKe-Net: Topology-Aware Point Cloud Completion
by Localizing Aligned Keypoints

Junshu Tang1, Zhijun Gong1, Ran Yi1111Corresponding authors., Yuan Xie2,
Lizhuang Ma1,2111Corresponding authors.
1Shanghai Jiao Tong University, 2East China Normal University
{tangjs, gongzhijun<EMAIL_ADDRESS><EMAIL_ADDRESS>ma-
<EMAIL_ADDRESS>
###### Abstract

Point cloud completion aims at completing geometric and topological shapes
from a partial observation. However, some topology of the original shape is
missing, existing methods directly predict the location of complete points,
without predicting structured and topological information of the complete
shape, which leads to inferior performance. To better tackle the missing
topology part, we propose LAKe-Net, a novel topology-aware point cloud
completion model by localizing aligned keypoints, with a novel Keypoints-
Skeleton-Shape prediction manner. Specifically, our method completes missing
topology using three steps: 1) Aligned Keypoint Localization. An asymmetric
keypoint locator, including an unsupervised multi-scale keypoint detector and
a complete keypoint generator, is proposed for localizing aligned keypoints
from complete and partial point clouds. We theoretically prove that the
detector can capture aligned keypoints for objects within a sub-category. 2)
Surface-skeleton Generation. A new type of skeleton, named Surface-skeleton,
is generated from keypoints based on geometric priors to fully represent the
topological information captured from keypoints and better recover the local
details. 3) Shape Refinement. We design a refinement subnet where multi-scale
surface-skeletons are fed into each recursive skeleton-assisted refinement
module to assist the completion process. Experimental results show that our
method achieves the state-of-the-art performance on point cloud completion.

##  1 Introduction

The geometry and vision community has put huge effort into point cloud
processing, which is challenging due to the unordered, unstructured
characteristics, and complex semantics of the point clouds. However, in real
applications, occlusions and insufficient lighting lead to partial scans of
real shapes and degrade the performance of subsequent processing. Point cloud
completion focuses on predicting missing regions from partial observations,
and shows its unique significance in many fundamental applications.

Recent works [41, 26, 7, 35, 18, 34, 29, 32, 40] for point cloud completion
successfully utilized deep-learning methods and achieved more plausible and
flexible results compared with traditional geometric-based methods [19, 8, 23]
and alignment-based methods [11, 12, 16].

Figure 1: Illustration of (a) visual comparison results of current completion
methods, (b) completion process of our LAKe-Net, and (c) aligned keypoints.
Compared with GRNet [34], PMP-Net [29], PoinTr [40] and Snowflake [32], LAKe-
Net can effectively recover missing topology part (see Red ellipses).

However, most existing methods directly predict the location of complete
points, without predicting structured and topological information of the
complete shape, which leads to coarse results in missing regions (see Figure
1(a)).

Inspired by typical geometric modeling theory that a complete 3D object
includes geometry and topology, e.g., coordinates and connectivity, we tend to
predict both geometric and structured topological information for point cloud
completion, including keypoints and generated skeleton. To this end, we
propose a novel Keypoints-Skeleton-Shape prediction manner, including three
steps: keypoint localization, skeleton generation, and shape refinement.

We firstly introduce the keypoint localization. Different from down-sampled
points, keypoints are evenly distributed across semantic parts of the shape,
and are considered as a crucial representation of geometric structure which
are widely-used in many vision applications [17, 27, 37]. Therefore, we hold
the belief that once the keypoints and their connectivity are correctly
localized, the entire geometry is determined. To this end, we wish to localize
complete keypoints according to partial inputs under the supervision of the
ground truth keypoints. However, obtaining keypoints annotation for a large
number of 3D data is difficult and expensive, so we propose an asymmetric
keypoint locator including an unsupervised multi-scale keypoint detector
(UMKD) and a complete keypoint generator (CKG) for complete and partial point
cloud, respectively. For UMKD, we extract Aligned Keypoints, which means the
order of keypoints are the same on different objects within a certain category
(Figure 1(c)), so as to represent more stable and richer information, and
provide stronger supervision for predicting complete keypoints from partial
inputs by CKG.

Since discrete and sparse keypoints are not enough for representing the whole
objects, we leverage skeleton to better represent topological details.
Inspired by existing skeleton extraction methods [5, 1, 14], we propose a
novel Surface-skeleton, which is generated from keypoints based on geometric
priors. Compared with other types of skeletons, our surface-skeleton is a
mixture of curves and triangle surfaces, and can represent more complex shape
information.  We integrate surface-skeletons with different fineness generated
by multi-scale keypoints into the shape refinement step to recover finer
results. Specifically, we propose a folding-based refinement subnet including
three recursive skeleton-assisted refinement modules (RSR) following some of
other completion methods [28, 32].

In general, we propose LAKe-Net, a novel topology-aware point cloud completion
model by localizing aligned keypoints. The whole pipeline includes four parts:
auto-encoder, asymmetric keypoints locator, surface-skeleton generator and the
refinement subnet. We leverage pairs of complete and partial point cloud
during training. In detail, the input point clouds (either complete or
partial) are firstly fed into an auto-encoder to learn a feature embedding
space and generate coarse and complete results. Then, we localize multi-scale
keypoints using asymmetric keypoints locator and generate corresponding
surface-skeletons. The multi-scale structures are fed into the refinement
subnet to generate fine output. The training process includes two stages for
point cloud reconstruction and completion, respectively.

Overall, we summarize our main contributions as follows: (1) We propose LAKe-
Net, a novel topology-aware point cloud completion model that utilizes a
structured representation of the surface as assistance, including Aligned
Keypoints and Surface-skeleton, with a new Keypoints-Skeleton-Shape prediction
manner. (2) We introduce an asymmetric keypoint locator including an
unsupervised multi-scale keypoint detector and a complete keypoint generator,
which can capture accurate keypoints for complete and partial objects in
multiple categories, respectively. We theoretically prove that our detector
detects aligned keypoints within each sub-category. (3) We conduct point cloud
completion experiments on two datasets, PCN and ShapeNet55. Experimental
results show that our LAKe-Net achieves the state-of-the-art performance on
both datasets.

Figure 2: The overall architecture of LAKe-Net, which consists of two parts
including Point Cloud Reconstruction (Blue) and Point cloud Completion (Red).
We show the detailed structure of (a) Auto-encoder $E$, (b) Complete Keypoint
Generator $G$ and (c) Recursive Skeleton-assisted Refinement module $R$ on the
right side. PCN encoder is firstly proposed in [41]. UMKD denotes the
unsupervised multi-scale keypoint detector. Surface-skeletons
$\mathcal{S}_{i}$ and $\hat{\mathcal{S}}_{i}$ are generated by
$\textbf{P}_{i}$ and $\hat{\textbf{P}}_{i}$, respectively.

##  2 Related Works

Point Cloud Completion. Point cloud completion focuses on predicting missing
shapes from partial point cloud input. Recently, inspired by point cloud
analysis approaches [20, 21], PCN [41] first adopts an encoder-decoder
architecture and a coarse-to-fine manner to generate the complete shape.
Several works [26, 15, 43, 32] follow this practice and make modifications in
network structure to obtain better performance. SA-Net [28] further extends
the decoding process into multiple stages by introducing hierarchical folding.
More recently, PoinTr [40] reformulates point cloud completion as a set-to-set
translation problem and designs a new transformer-based encoder-decoder for
point cloud completion. However, these methods mostly predict the location of
complete points without predicting structured and topological information,
which leads to coarse results in missing regions. SK-PCN [18] is the most
relevant work to ours which pre-processes the dataset and uses meso-skeletons
as supervision. However, SK-PCN doesn’t predict the structured and topological
information of original shape. Our proposed LAKe-Net utilizes aligned
keypoints and corresponding surface-skeleton which can capture the shared
topological information as an assistant for completion, and obtains better
performance.

Skeleton Representation. Skeleton representation is widely-used in motion
recognition [38, 39], human pose estimation [2, 22] and human reconstruction
[4, 42]. Jiang _et al_. [10] propose to incorporate skeleton awareness into
the deep learning-based regression for 3D human shape reconstruction from
point clouds. Tang _et al_. [25] utilize topology preservation property of
skeleton to perform 3D surface reconstruction from a single RGB image. P2P-Net
[36] learns bidirectional geometric transformations between point-based shape
representations from two domains, surface-skeletons and surfaces. Our method
designs surface-skeleton representations generated by multi-scale keypoints in
a more fine-grained manner to progressively aid point cloud completion.

Unsupervised keypoint detection. While most hand-crafted 3D keypoint detectors
fail to detect accurate and well-aligned keypoints in complex objects, Li _et
al_. [13] propose the first learning-based 3D keypoint detector USIP. However,
the detected keypoints are neither ordered nor semantically salient. Fernandez
_et al_. [6] utilize symmetry prior in point clouds to capture keypoints in an
unsupervised manner. Recently, Jakab _et al_. [9] further explore the
application of unsupervised keypoints in shape deformation task.
SkeletonMerger [24] proposes a novel keypoint detector based on an autoencoder
architecture. However, different models need to be trained for different
categories. We propose UMKD, an unsupervised multi-scale keypoint detector
which can capture keypoints for objects in multiple categories. We find that
it reaches the best performance on categories with shared topology on
KeypointNet [37] and produces more salient and semantic richer keypoints.

##  3 Proposed Method

We propose a novel topology-aware point cloud completion network by localizing
aligned keypoints (LAKe-Net), whose overall architecture is shown in Fig. 2.
The pipeline includes four parts: auto-encoder, asymmetric keypoint locator,
surface-skeleton generation and shape refinement. The training includes two
stages: point cloud reconstruction and completion. The training data consists
of pairs of complete and partial point clouds ${(\textbf{X},\mathcal{X})}$,
where $\textbf{X}\in\mathbb{R}^{N_{c}\times 3}$ and
$\mathcal{X}\in\mathbb{R}^{N_{p}\times 3}$ denote the coordinates of complete
and partial point clouds, $N_{c}$ and $N_{p}$ denote the number of points of
complete and partial data, respectively.

Figure 3: The detailed structure of our proposed UMKD. Take $\textbf{P}_{3}$
as an example, we show the calculation process and dimensions of relative
tensors in Blue.

Firstly, see upper part of Fig. 2, we utilize complete data X as input to
train an unsupervised multi-scale keypoint detector (UMKD) $D$, which extracts
the multi-scale keypoints $\\{\textbf{P}_{i}\\}_{i=1}^{3}$, and an auto-
encoder $E_{1}$, which maps the inputs into a global feature space $c$ and
obtain coarse results $\textbf{X}_{c}$. Then, a surface-skeleton generation
process is employed to leverage the topology information in keypoints to
construct a finer representation. Finally, a refinement subnet adopts the
topology information $\mathcal{S}$ and coarse results $\textbf{X}_{c}$ to
generate high-resolution results $\textbf{X}_{f}$. The reconstruction part
unsupervisedly trains the UMKD to learn aligned keypoints for later completion
stage and also provides a good initialization of the network to predict
complete shape.

In the second stage, we fix the weights of keypoint detector $D$ and auto-
encoder $E_{1}$, then input partial data $\mathcal{X}$ into a new auto-encoder
$E_{2}$ and predict complete coarse results $\mathcal{X}_{c}$. It is
noteworthy that the auto-encoder $E_{1}$ and $E_{2}$ have the same
architecture which includes a PCN encoder [41] and coarse point generator. The
encoder of $E_{2}$ embeds input points into point-wise local features
$\hat{f}\in\mathbb{R}^{N_{p}\times d}$, where $d$ denotes the dimension of
feature embedding. We consider the maximum value
$\hat{c}=max_{N_{p}}(\hat{f})\in\mathbb{R}^{1\times d}$ as global features.
Then we fuse the local and global features and feed the fused feature into a
complete keypoint generator (CKG) $G$ to predict multi-scale keypoints
$\\{\hat{\textbf{P}}_{i}\\}_{i=1}^{3}$, supervised by keypoints
$\\{\textbf{P}_{i}\\}_{i=1}^{3}$ detected in the first stage. At last, we send
$\mathcal{X}_{c}$ and interpolated surface-skeleton $\hat{\mathcal{S}}$ to a
skeleton-assisted refinement subnet $R$ to generate fine outputs
$\mathcal{X}_{f}$. We describe the technical details of our proposed modules
and training losses in the following sections.

###  3.1 Unsupervised Multi-scale Keypoint Detector

Given a set of $N_{c}$ input complete point clouds
$\textbf{X}=\\{x_{j}|j=1,\cdots,N_{c}\\}\in\mathbb{R}^{N_{c}\times 3}$, our
aim is to predict $\\{K_{i}\\}_{i=1}^{3}$ numbers of keypoints
$\\{\textbf{P}_{i}\\}_{i=1}^{3}=\\{p_{k}|k=1,\cdots,K_{i}\\}\in\mathbb{R}^{K_{i}\times
3}$. Specifically, we tend to predict convex combination weights
$\textbf{W}_{i}=\\{w_{ij}\\}\in\mathbb{R}^{N_{c}\times K_{i}}$ of point clouds
instead of predicting the coordinates of keypoints directly to avoid deviating
from the original shape. So the predicted keypoints $\textbf{P}_{i}$ are
derived by:

$\displaystyle\textbf{P}_{i}=\textbf{W}_{i}^{T}\textbf{X}=\sum^{N_{c}}_{j=1}w_{ij}^{T}x_{j},s.t.,w_{ij}>0,\sum^{N_{c}}_{j=1}w_{ij}=1.$
(1)

To predict convex weight $\textbf{W}_{i}$ using a single model for all
categories which adapts to our pipeline, we assume
$\textbf{W}_{i}=\textbf{W}^{i}_{t}+\textbf{W}^{i}_{o}$, where
$\textbf{W}^{i}_{t}$ denotes a category-invariant template weight and
$\textbf{W}^{i}_{o}$ denotes a category-specific weight offset. Besides, we
expect to predict multi-scale keypoints for subsequent tasks. To this end, we
propose an Unsupervised Multi-scale Keypoint Detector (UMKD) which consists of
a category-invariant convex encoder and category-specific offset predictors.
The detailed structure of UMKD is shown in Figure 3. In detail, we apply a
PointNet++ [21] as a backbone encoder to extract local and global features. It
includes four set abstraction layers to group and downsample input points.
Then the global and local features are propagated back to each partial point.
Then we input point-wise features into three fully connected blocks and
extract multi-scale convex features
$\textbf{W}^{i}_{t}\in\mathbb{R}^{K_{i}\times N_{c}}$ progressively.

As for predicting category-specific offset $\textbf{W}^{i}_{o}$, we firstly
predict the category label $\omega$ for input shape and send the
$\textbf{W}^{i}_{t}$ to the relative offset predictor $\textbf{O}_{i\omega}$.
We add two fully-connected layers after the last pointset abstraction layer in
PointNet++ as classification head to predict the category label of every input
geometries. $\textbf{O}_{ij}\in\mathbb{R}^{K_{i}\times K_{i}}$ is a learnable
matrix. We set $j\in[1,\mathcal{Q}]$. $\mathcal{Q}$ denotes the number of
categories. Take $\textbf{P}_{3}$ as an example (shown in Figure 3), the input
points $\textbf{X}\in\mathbb{R}^{N_{c}\times 3}$ are send to the convex
encoder and output template convex weights
$\textbf{W}^{3}_{t}\in\mathbb{R}^{K_{3}\times N_{c}}$ and predicted label
$\omega=2$. Then $\textbf{W}_{t}^{3}$ are input to the selected offset
predictor $\textbf{O}_{32}$ and output
$\textbf{W}^{3}_{o}\in\mathbb{R}^{K_{3}\times N_{c}}$. At last, we get the
final convex weight $\textbf{W}_{3}=\textbf{W}^{3}_{t}+\textbf{W}^{3}_{o}$ and
normalize $\textbf{W}_{3}$ by a softmax function and predict keypoint
$\textbf{P}_{3}$.

The keypoints extracted by UMKD $D$ follow a theory that: The coordinates of
detected keypoints P are irrelevant to the order of original points X. That
is: P = D(X) = D($\mathcal{R}(\textbf{X})$), where $\mathcal{R}(\cdot)$
denotes random permutation operation. The theoretical proof is introduced in
Supplementary Materials. Therefore, the detected keypoints are aligned among
objects with shared topology within a sub-category (shown in Figure 1(c)).

Figure 4: The surface-skeleton generation operation.

###  3.2 Surface-skeleton Generation

After extracting the keypoints of original point clouds, our aim is to
reconstruct the point cloud according to the extracted keypoints. We consider
skeleton as an intermediate representation between keypoints and original
point clouds. Previous methods like SkeletonMerger [24] used skeletons
connected by every pair of keypoints, which leads to high computational
complexity and a lot of invalid points. Moreover, given a surface, previous
skeletons either are located near the medial axis of the surface [5], or are
mostly located outside the surface [24]. We aim at extracting a skeleton that
are located near the surface, so that it can better assist the later
refinement process. In order to represent both topological and geometric
information of a complex shape, we design a surface-skeleton structure which
is generated by keypoints and consists of a mixture of curves and triangle
surfaces adapted to the local 3D geometry. Specifically, given predicted
keypoints $P\in\mathbb{R}^{K\times 3}$ and reference points $X$, we follow the
two shape priors proposed in [14] and generate the skeletal graph: (1) the
topology prior that each node has links to its top-$2$ nearest nodes; (2) the
recovery prior that two keypoints are linked if they are two nearest keypoints
of a reference point.

After skeletal graph generation, we can get an adjacency matrix
$A\in\mathbb{R}^{K\times K}$. We propose a surface interpolation strategy
based on Delaunay-based triangulation region. The whole interpolation process
consists of two steps: edge interpolation and triangle surface interpolation.
Given a certain graph, we firstly interpolate points into each connected edge.
Next, we detect every triangle surface according to the skeletal graph. Then
we insert points into the triangle region. The number of interpolated points
is proportional to the triangle area. The overall skeletal graph generation
and surface interpolation process is shown in Fig 4. It is noteworthy that we
utilize the complete point clouds X as reference shape for reconstruction
while using the coarse results $\mathcal{X}_{c}$ for completion.

Overall, we get the surface-skeleton $\mathcal{S}$. Increasing the number of
keypoints leads to more complex surface-skeletons which can represent finer
shape details. We utilize the structure as topology representation of
geometric shape which is crucial for reconstruction and completion.

###  3.3 Complete Keypoint Generator

In the second stage, given the partial input
$\mathcal{X}\in\mathbb{R}^{N_{p}\times 3}$, the aim of our proposed Complete
Keypoint Generator (CKG) module is to predict multi-scale complete keypoints
from partial feature embedding. To this end, we utilize the local and global
feature $\hat{f}$ and $\hat{c}$ extracted by the encoder in $E_{2}$ as input.
Similar to Farthest Point Sampling (FPS) strategy for point cloud, in order to
downsample the point features, we use Farthest Feature Sampling (FFS) strategy
to down sample point-wise local feature $\hat{f}$ to sparse feature
$\hat{f}^{*}$ where we replace point coordinates in FPS by feature embeddings.
The number of sampling is the same as predicted keypoints. Then we utilize a
de-convolution layer to upsample the global feature $\hat{c}$ and fuse them
into a residual block and predict final keypoints. We train three similar
blocks for multi-scale keypoints prediction. Then we generate corresponding
surface-skeleton $\hat{\mathcal{S}}$ by surface interpolation introduced on
Sec. 3.2.

###  3.4 Recursive Skeleton-assisted Refinement

Our proposed shape refinement subnet $R$ includes three recursive skeleton-
assisted refinement (RSR) modules that aim to integrate multi-scale surface-
skeletons and coarse output from previous auto-encoder to predict finer
geometric details in a recursive way. The detailed design of the module is
shown in Fig 2. It follows existing methods [35, 28, 32] using a coarse-to-
fine strategy to learn the offset of integrated points. Specifically, we
progressively concatenate coarse point clouds obtained from previous steps and
surface-skeleton generated by corresponding keypoints described in Sec. 3.2.
We denote the input coarse points as
$\textbf{X}_{i-1}=\\{x_{j}\\}_{j=1}^{N_{i-1}}$ and surface-skeleton as
$\mathcal{S}_{i-1}=\\{p_{j}\\}_{j=1}^{S_{i-1}}$. The integrated points
$\hat{\textbf{X}}_{i-1}=concat(\textbf{X}_{i-1},\mathcal{S}_{i-1})$ on the
$i$-th step where $concat(\cdot)$ refers to concatenate operation. $N_{i-1}$
and $S_{i-1}$ denote numbers of coarse points and surface-skeleton,
respectively. In this paper, we set $N_{i}=S_{i}$ in each step. Therefore, the
updated points output by the $i$-th RSR module
$\textbf{X}_{i}=\hat{\textbf{X}}_{i-1}+R(\hat{\textbf{X}}_{i-1})$ will be sent
to the next step.

###  3.5 Training and Losses

Point Cloud Reconstruction. In the first stage, the UMKD $D$, auto-encoder
$E_{1}$ and refinement subnet $R$ are trained together. The training losses
are divided into two parts, one is to constrain the keypoint detection, the
other is data reconstruction. Firstly, in order to encourage the detected
keypoints P to be well-distributed and not deviate from the global shape, we
calculate the Chamfer Distance (CD) loss between the predicted keypoints and
sparse point clouds $\textbf{X}^{*}$ downsampled from input data using FPS
strategy. As for training one detector within several categories, we also
train a classification head. We denote the predicted output is $\omega$ and
certain category label is $\sigma$. We train a criterion loss
$\mathcal{L}_{cls}$. Besides, as mentioned in Sec. 3.2, we expect the surface-
skeleton can reconstruct the geometric shape of the ground truth. We calculate
CD between multi-scale surface-skeletons $\\{\mathcal{S}_{i}\\}^{3}_{i=1}$ and
the ground truth X. So the overall loss for training keypoint detector is:

$\displaystyle\mathcal{L}_{CD}=\frac{1}{|X|}\sum_{x\in X}\min_{y\in
Y}||x-y||^{2}+\frac{1}{|Y|}\sum_{y\in Y}\min_{x\in X}||y-x||^{2},$ (2)
$\displaystyle\mathcal{L}_{cls}=-\sum^{\mathcal{Q}}_{i=1}(\sigma_{i}log\omega_{i}+(1-\sigma_{i})log(1-\omega_{i})),$
(3)
$\displaystyle\mathcal{L}_{kp}=\mathcal{L}_{CD}(\textbf{P},\textbf{X}^{*})+\sum^{3}_{i=1}\mathcal{L}_{CD}(\mathcal{S}_{i},\textbf{X})+\mathcal{L}_{cls}.$
(4)

At last, we calculates CD between the ground truth X and sparse output
$\textbf{X}_{c}$, dense output $\textbf{X}_{f}$, respectively.

$\displaystyle\mathcal{L}_{rec}=\mathcal{L}_{CD}(\textbf{X}_{c},\textbf{X})+\mathcal{L}_{CD}(\textbf{X}_{f},\textbf{X}).$
(5)

In general, the overall training loss in the first stage is:

$\displaystyle\mathcal{L}_{1}=\mathcal{L}_{rec}+\lambda_{kp}^{1}\mathcal{L}_{kp},$
(6)

where $\lambda_{kp}^{1}$ denotes hyper-parameters to balance inference.

Point Cloud Completion. In the second stage, we fix the weights of UMKD $D$
and auto-encoder $E_{1}$, and train a new auto-encoder $E_{2}$ and CKG $G$.
The refinement subnet $R$ pre-trained before continues to be optimized. We
constrain keypoints prediction using absolute distance between predicted
keypoints $\hat{\textbf{P}}$ and ground truth keypoints P:

$\displaystyle\mathcal{L}^{c}_{kp}=\sum^{3}_{i=1}\sum^{K_{i}}_{j=1}||p_{ij}-\hat{p}_{ij}||^{2}.$
(7)

Same as other concurrent network [31], we align global features $c$ and
$\hat{c}\in\mathbb{R}^{1\times d}$ encoded by auto-encoders in two stages for
hidden feature space learning:

$\displaystyle\mathcal{L}_{feat}=\frac{1}{d}\sum^{d}_{i=1}||c_{i}-\hat{c}_{i}||^{2}.$
(8)

As for typical training on completion task, we follow the coarse-to-fine
process in the first stage. The coarse output $\mathcal{X}_{c}$ and fine
output $\mathcal{X}_{f}$ are optimized using CD loss:

$\displaystyle\mathcal{L}_{com}=\mathcal{L}_{CD}(\mathcal{X}_{c},\textbf{X})+\mathcal{L}_{CD}(\mathcal{X}_{f},\textbf{X}).$
(9)

In the summary, the full objective of point cloud completion in the second
stage is:

$\displaystyle\mathcal{L}_{2}=\mathcal{L}_{com}+\lambda_{kp}^{2}\mathcal{L}_{kp}^{c}+\lambda_{feat}\mathcal{L}_{feat},$
(10)

where ($\lambda^{2}_{kp}$, $\lambda_{feat}$) denote hyper-parameters.

##  4 Experiments

###  4.1 Dataset Setting and Evaluation metric

PCN: The PCN dataset is a widely-used benchmark for point cloud completion,
which is created by [41], including different objects from 8 categories:
plane, cabinet, car, chair, lamp, sofa, table, and vessel. The training set
contains 28,974 objects, while validation and test set contains 800 and 1,200
objects, respectively. The complete point cloud consists of 16,384 points
which are uniformly sampled on the original CAD model. Partial point cloud,
consisting of 2,048 points, is created by back-projecting 2.5D depth images
into 3D from 8 random viewpoints.

ShapeNet55: To explore the performance of our method on a large number of
categories, we evaluate our method on all 55 categories of ShapeNet [3], named
ShapeNet55. The ShapeNet55 dataset was first created by PoinTr [40]. The
training set contains 41,952 objects, while test set contains 10,518 objects.
We randomly sample 80$\%$ objects in each category to form training set and
use the rest 20$\%$ to form validation set.

Evaluation Metrics: We utilize two evaluation metrics between output point
cloud and the ground truth, Chamfer Distance (CD) using L2 norm and Earth
Mover’s Distance (EMD), following most of the methods on PCN and ShapeNet55
test set. CD is introduced in Equation 2 and EMD is defined as:

$\displaystyle EMD(X,Y)=\min_{\phi:X\to Y}\frac{1}{|X|}\sum_{x\in
X}||x-\phi(x)||_{2},$ (11)

where $\phi$ is a bijection. It is noteworthy that we compute these metrics
using 16,384 and 8,192 points for PCN and ShapeNet55, respectively.

###  4.2 Implementation Details

The whole training of LAKe-Net is a two-stage process: point cloud
reconstruction and point cloud completion. The input of the first stage
(reconstruction) is a set of complete point clouds with coordinates and object
category labels from training set of all datasets. We train the keypoint
detection for 60 epochs and progressively extract 256, 128, 64 keypoints. The
refinement subnet includes three RSR modules, the up factors of de-convolution
are [1,1,2]. For the second stage (the bottom completion branch of Fig. 2), we
only input partial point clouds from training set with its coordinate
information. We utilize Adam optimization to train the whole architecture of
point cloud completion for 100 epochs with batchsize 64 and learning rate
0.001. The hyper-parmeters $\lambda_{kp}^{1}=\lambda_{kp}^{2}=10$,
$\lambda_{feat}=1000$. The inference time of our method is 34.5ms per sample.

Figure 5: Visualization of point cloud completion comparison results on PCN dataset with other recent methods. Category | Bed | Bench | Bookshelf | FileCabinet | Faucet | Telephone | Can | Flowerpot | Tower | Pillow | Average
---|---|---|---|---|---|---|---|---|---|---|---
Metrics | CD | EMD | CD | EMD | CD | EMD | CD | EMD | CD | EMD | CD | EMD | CD | EMD | CD | EMD | CD | EMD | CD | EMD | CD | EMD
Folding [35] | 3.17 | 73.6 | 1.45 | 50.1 | 2.48 | 64.4 | 1.94 | 65.3 | 3.19 | 66.2 | 0.69 | 39.1 | 1.76 | 60.2 | 4.11 | 82.9 | 1.83 | 59.5 | 1.64 | 63.2 | 2.06 | 60.2
PCN [41] | 2.50 | 49.4 | 0.96 | 28.9 | 2.39 | 44.7 | 1.49 | 37.2 | 1.96 | 40.3 | 0.54 | 24.0 | 1.30 | 30.9 | 2.58 | 48.7 | 1.34 | 33.7 | 1.09 | 31.5 | 1.36 | 34.0
GRNet [34] | 0.93 | 29.7 | 0.86 | 25.8 | 0.93 | 29.6 | 1.57 | 24.9 | 0.83 | 27.6 | 0.87 | 26.2 | 1.15 | 32.3 | 1.24 | 33.5 | 0.87 | 25.3 | 1.06 | 28.8 | 1.15 | 28.2
PoinTr [40] | 2.18 | 37.6 | 0.93 | 21.4 | 1.86 | 37.1 | 3.23 | 42.7 | 1.75 | 42.4 | 0.55 | 20.8 | 2.13 | 31.2 | 2.68 | 42.7 | 1.73 | 35.9 | 1.40 | 31.8 | 1.70 | 31.7
Ours | 0.72 | 28.4 | 0.71 | 18.2 | 0.89 | 29.7 | 0.97 | 16.4 | 0.34 | 20.5 | 0.48 | 20.8 | 0.63 | 29.0 | 1.19 | 35.9 | 0.60 | 21.9 | 0.97 | 29.8 | 0.89 | 31.0
Table 1: Quantitative comparison results with other completion methods on ShapeNet55 dataset using CD-$l_{2}$($\times 10^{3}$) and EMD($\times 10^{3}$) metrics. We report the detailed results for each method on 10 sampled categories and overall average results on all 55 categories. CD-$l_{2}(\times 10^{4})$ | Airplane | Cabinet | Car | Chair | Lamp | Sofa | Table | Vessel | Average
---|---|---|---|---|---|---|---|---|---
Folding [35] | 3.151 | 7.943 | 4.676 | 9.225 | 9.234 | 8.895 | 6.691 | 7.325 | 7.142
PCN [41] | 1.400 | 4.450 | 2.445 | 4.838 | 6.238 | 5.129 | 3.569 | 4.062 | 4.016
AtlasNet [7] | 1.753 | 5.101 | 3.237 | 5.226 | 6.342 | 5.990 | 4.359 | 4.177 | 4.523
MSN [15] | 1.543 | 7.249 | 4.711 | 4.539 | 6.479 | 5.894 | 3.797 | 3.853 | 4.758
GRNet [34] | 1.531 | 3.620 | 2.752 | 2.945 | 2.649 | 3.613 | 2.552 | 2.122 | 2.723
PMP-Net [29] | 1.205 | 4.189 | 2.878 | 3.495 | 2.178 | 4.267 | 2.921 | 1.894 | 2.878
SpareNet [33] | 1.756 | 6.635 | 3.614 | 6.163 | 6.313 | 7.893 | 4.987 | 3.835 | 5.149
PointTr [40] | 0.993 | 4.809 | 2.529 | 3.683 | 3.077 | 6.535 | 3.103 | 2.029 | 3.345
Snowflake [32] | 0.913 | 3.322 | 2.246 | 2.642 | 1.898 | 3.966 | 2.011 | 1.692 | 2.336
Ours | 0.646 | 2.594 | 1.743 | 2.149 | 2.759 | 2.186 | 1.876 | 1.602 | 1.944
Table 2: Quantitative comparison results with other methods of point cloud completion on PCN using CD-$l_{2}$ (lower is better).  EMD($\times 10^{2}$) | Airplane | Cabinet | Car | Chair | Lamp | Sofa | Table | Vessel | Average
---|---|---|---|---|---|---|---|---|---
Folding [35] | 1.682 | 2.576 | 2.183 | 2.847 | 3.062 | 3.003 | 2.500 | 2.357 | 2.526
PCN [41] | 2.426 | 1.888 | 2.744 | 2.200 | 2.383 | 2.062 | 1.242 | 2.208 | 2.144
AtlasNet [7] | 1.324 | 2.582 | 2.085 | 2.442 | 2.718 | 2.829 | 2.160 | 2.114 | 2.282
MSN [15] | 1.334 | 2.251 | 2.062 | 2.346 | 2.449 | 2.712 | 1.977 | 2.001 | 2.142
GRNet [34] | 1.376 | 2.128 | 1.918 | 2.127 | 2.150 | 2.468 | 1.852 | 1.876 | 1.987
PMP-Net [29] | 1.259 | 2.058 | 2.520 | 1.798 | 1.280 | 2.579 | 1.651 | 1.760 | 1.863
SpareNet [33] | 1.131 | 2.014 | 1.783 | 2.050 | 2.063 | 2.333 | 1.729 | 1.790 | 1.862
PointTr [40] | 0.938 | 1.986 | 1.851 | 1.892 | 1.740 | 2.242 | 1.931 | 1.532 | 1.764
Snowflake [32] | 1.375 | 2.633 | 2.591 | 2.086 | 1.599 | 3.070 | 1.616 | 1.957 | 2.116
Ours | 0.958 | 1.830 | 1.564 | 1.667 | 1.782 | 1.755 | 1.499 | 1.402 | 1.557
Table 3: Quantitative comparison results with other methods of point cloud
completion on PCN using EMD (lower is better).

###  4.3 Results on PCN dataset.

We compare the performance of our proposed LAKe-Net and other state-of-the-art
completion methods. We implement other methods using their open source code
and hyper-parameters for fair comparison. Table 2 and 3 show the quantitative
comparison results of our method and other point cloud completion methods on
PCN datasets, from which we can see that our method achieves the best
performance over all counterparts on both CD and EMD metrics. Specifically,
compared with the second-ranked Snowflake which also proposed progressive
decoding modules, our method has better performance with the help of aligned
keypoints and surface-skeletons. Besides, according to experimental results,
our proposed LAKe-Net is more powerful to predict symmetrical geometries and
their topology information compared with SnowflakeNet.

Moreover, we also show the visualization of qualitative comparison results and
some recent methods in Figure 5, which show that our method has better
performance on completing missing topology. Specifically, methods which also
utilize progressive coarse-to-fine decoding like PMP-Net and SnowflakeNet,
tended to predict coarse missing shape and generate scattered points,
especially for geometry with a plane or surface. Other methods like GRNet,
SpareNet and PoinTr are weak on recovering the local details and some missing
topology like table legs. Our method can predict geometries with more clear
topology structure and fewer noises.

###  4.4 Results on ShapeNet55 dataset

Moreover, to evaluate the generalization and powerful of our method on a large
account of categories of data to adapt to real-world scenarios, we conduct
experiments on ShapeNet55 dataset and compare with other completion methods.
We drop 75$\%$ of the complete point cloud and resample the remaining partial
point clouds to 2,048 points as input for all methods. Table 1 shows the
quantitative comparison results on 10 sampled categories. The last column
shows the overall average results of 55 categories. We can see that our method
achieves the best result on CD metric and have competitive results on EMD
metric. Specifically, the results on Bed, Bench, Bookshelf, FileCabinet,
Faucet, which are similar as samples in PCN datasets or have shared topology
within a sub-category, show that our method can recover geometries more
efficiently using topology assistant. Moreover, the results on other
categories show that our method is more powerful in completing geometries with
regular and symmetrical contours, similar as Vessel in PCN dataset. We
visualize the completion process of our method on samples from ShapeNet55 in
Figure 6. It can be seen that our method can localize effect keypoints and
recover missing topological and geometric information with the help of
surface-skeletons.

Figure 6: Visualization of completion on ShapeNet55 dataset by our proposed
method. We also show the predicted keypoints and generated surface-skeletons
on the second and third columns.

##  5 Method Analysis

In this section, we examine the effectiveness of our motivations in LAKe-Net.
We conduct several ablation studies from different points of view. For fair
comparison, all methods are trained and tested on PCN dataset for completion
and KeypointNet for keypoint detection.

Unsupervised Muti-scale Keypoint Detector. To prove the effectiveness and
accuracy of our proposed UMKD, we evaluate our extracted keypoints compared
with two recent unsupervised keypoint detectors Fernandez et al. [6] and
SkeletonMerger [24]. All methods are trained and tested on KeypointNet [37],
which has keypoints annotations with semantic correspondence labels. We
evaluate these methods on five categories: airplane, car, chair, table and
vessel. Specifically, we detect 16, 32, 64 keypoints for all categories. As
for other methods, we train five models and detect 16 keypoints for each
category. We firstly down-sample the same number of predict keypoints as the
annotated keypoints using the nearest neighbor strategy.

Figure 7: Visualization of (a) multi-scale keypoint detected by ours and
SkeletonMerger on KeypointNet dataset; (b) failed case by our method without
csop; (c) drawbacks of FPS 16 points. csop denotes category-specific offset
predictors.

For evaluation metrics, we follow SkeletonMerger and utilize mean Intersection
over Unions (mIoU) metrics to evaluate the keypoint silence and accuracy. It
is calculated with a threshold of 0.1 using euclidean distance. The
quantitative results are shown in Table 4, which illustrates that our proposed
keypoint detector trained on multiple categories has competitive performance,
or even better, than other unsupervised methods trained on a single category.
The results on airplane, chair and table also show that our method has better
performance on geometries with obvious topological structures. The
visualization is shown in Figure 7(a). Our detector can produce more salient
and semantic richer keypoints. We also show our detected keypoints in multi-
scale which represent finer geometric details.

Besides, to evaluate the effectiveness of our proposed category-specific
offset predictors (csop) in UMKD, we replace all offset predictors with a
single category invariant offset predictor. The qualitative results is
visualized in Figure 7(b) and quantitative results are shown in Table 4. It is
obvious that a single offset predictor cannot handle geometries in multiple
categories. The predicted keypoints tend to aggregate together to reduce the
loss.

mIoU | Airplane | Car | Chair | Table | Vessel
---|---|---|---|---|---
Fernandez et al. | 69.7 | 50.5 | 51.2 | 49.3 | 53.5
SkeletonMerger | 72.7 | 64.6 | 63.2 | 59.6 | 62.0
Ours-16 | 73.2 | 58.1 | 69.2 | 62.5 | 61.3
Ours-32 | 73.7 | 60.2 | 70.5 | 63.2 | 62.5
Ours-64 | 74.0 | 62.9 | 71.3 | 65.4 | 64.0
Ours-32 w/o csop | 35.4 | 13.2 | 27.8 | 23.9 | 11.0
Table 4: Quantitative comparison results with other unsupervised keypoint
detector on KeypointNet using mIoU (higher is better).

Keypoints and Surface-skeletons. To evaluate the necessity of using aligned
keypoints and surface-skeletons for point cloud completion, we conduct several
ablated experiments. We consider the auto-encoder $E_{2}$ and refinement
subnet $R$ in the second stage as baseline. In particular, we replace multi-
scale complete keypoints detected in the first stage with multi-scale down-
sampled points from ground truth using FPS strategy. And we use CD loss
between down-sampled points and predicted keypoints in the second stage. We
also remove the assist of generated surface-skeletons and change the up
factors into [2,2,4] for fair comparison. The quantitative results are
illustrated in Table 5. We can see that the down-sampled points cannot
represent efficient topology information (as shown in Figure 7(c)) especially
on some joint parts, and are not helpful for completing missing geometries in
our pipeline. Besides, CD loss between two unordered and sparse point clouds
is harder to be optimized to convergence.

We also visualize different types of skeletons compared with our multi-scale
surface-skeleton in Figure 8. It can be seen that our surface-skeleton focuses
on representing surfaces of original shape, and can get competitive
performance with meso-skeletons detected by typical method [30], and better
than curve skeleton from [24].

EMD($\times 10^{2}$) | Airplane | Cabinet | Car | Chair | Lamp | Sofa | Table | Vessel | Average
---|---|---|---|---|---|---|---|---|---
Ours | 0.958 | 1.830 | 1.564 | 1.667 | 1.782 | 1.755 | 1.499 | 1.402 | 1.557
-use FPS | 1.117 | 2.295 | 1.978 | 2.157 | 1.916 | 2.607 | 1.810 | 1.823 | 1.963
-w/o S-sk | 1.469 | 2.638 | 2.386 | 2.380 | 2.221 | 2.989 | 1.906 | 2.020 | 2.251
PointDisturb | 1.031 | 1.902 | 1.554 | 2.012 | 1.945 | 2.037 | 1.684 | 1.437 | 1.700
ClassDisturb | 0.963 | 1.846 | 1.576 | 1.786 | 1.831 | 1.780 | 1.545 | 1.397 | 1.590
Table 5: Ablation studies and robustness test on PCN dataset using EMD
metrics. w/o S-sk denotes without surface-skeleton. Figure 8: Visualization of
different type of skeletons, including curve skeleton generated by [24], meso-
skeleton from [30] and our surface-skeleton in multi-scale.

Robustness Test. We also conduct ablation studies to investigate the
robustness of our method in some extreme cases. We firstly randomly disturb
$5\%$ of the detected GT keypoints with a threshold of 0.1 after the first
stage. Secondly, we deliberately misclassify geometries with simiar shapes:
table, chair and sofa. The results are reported on Table 5. Experimental
results show that our method is robust to errors in keypoints detection in the
first stage.

##  6 Conclusion

In this paper, we propose a novel topology-aware point cloud completion
method, named LAKe-Net, which focuses on completing missing topology by
localizing aligned keypoints, with a novel Keypoints-Skeleton-Shape prediction
manner, including aligned keypoints localization, surface-skeleton generation
and shape refinement. Experimental results show that our LAKe-Net achieves the
state-of-the-art performance on point cloud completion.

##  7 Acknowledgements

This work was supported by the National Key Research and Development Program
of China (2019YFC1521104), National Natural Science Foundation of China
(72192821, 61972157, 62176092), Shanghai Municipal Science and Technology
Major Project (2021SHZDZX0102), Shanghai Science and Technology Commission
(21511101200, 22YF1420300, 21511100700), CAAI-Huawei MindSpore Open Fund, and
Art major project of National Social Science Fund (I8ZD22).

## References

  * [1] J. Cao, A. Tagliasacchi, M. Olson, H. Zhang, and Z. Su.  Point cloud skeletons via laplacian based contraction.  In 2010 Shape Modeling International Conference, pages 187–197. IEEE, 2010.
  * [2] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh.  Realtime multi-person 2d pose estimation using part affinity fields.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291–7299, 2017.
  * [3] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al.  Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [4] K.-L. Cheng, R.-F. Tong, M. Tang, J.-Y. Qian, and M. Sarkis.  Parametric human body reconstruction based on sparse key points.  IEEE transactions on visualization and computer graphics, 22(11):2467–2479, 2015.
  * [5] N. D. Cornea, D. Silver, and P. Min.  Curve-skeleton properties, applications, and algorithms.  IEEE Transactions on visualization and computer graphics, 13(3):530, 2007.
  * [6] C. Fernandez-Labrador, A. Chhatkuli, D. P. Paudel, J. J. Guerrero, C. Demonceaux, and L. V. Gool.  Unsupervised learning of category-specific symmetric 3d keypoints from point sets.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16, pages 546–563. Springer, 2020.
  * [7] T. Groueix, M. Fisher, V. G. Kim, B. C. Russell, and M. Aubry.  A papier-mâché approach to learning 3d surface generation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 216–224, 2018.
  * [8] F. Han and S.-C. Zhu.  Bottom-up/top-down image parsing with attribute grammar.  IEEE transactions on pattern analysis and machine intelligence, 31(1):59–73, 2008.
  * [9] T. Jakab, R. Tucker, A. Makadia, J. Wu, N. Snavely, and A. Kanazawa.  Keypointdeformer: Unsupervised 3d keypoint discovery for shape control.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12783–12792, 2021.
  * [10] H. Jiang, J. Cai, and J. Zheng.  Skeleton-aware 3d human shape reconstruction from point clouds.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5431–5441, 2019.
  * [11] E. Kalogerakis, S. Chaudhuri, D. Koller, and V. Koltun.  A probabilistic model for component-based shape synthesis.  Acm Transactions on Graphics (TOG), 31(4):1–11, 2012.
  * [12] V. G. Kim, W. Li, N. J. Mitra, S. Chaudhuri, S. DiVerdi, and T. Funkhouser.  Learning part-based templates from large collections of 3d shapes.  ACM Transactions on Graphics (TOG), 32(4):1–12, 2013.
  * [13] J. Li and G. H. Lee.  Usip: Unsupervised stable interest point detection from 3d point clouds.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 361–370, 2019.
  * [14] C. Lin, C. Li, Y. Liu, N. Chen, Y.-K. Choi, and W. Wang.  Point2skeleton: Learning skeletal representations from point clouds.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4277–4286, 2021.
  * [15] M. Liu, L. Sheng, S. Yang, J. Shao, and S.-M. Hu.  Morphing and sampling network for dense point cloud completion.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 11596–11603, 2020.
  * [16] A. Martinovic and L. Van Gool.  Bayesian grammar learning for inverse procedural modeling.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 201–208, 2013.
  * [17] A. S. Mian, M. Bennamoun, and R. Owens.  Three-dimensional model-based object recognition and segmentation in cluttered scenes.  IEEE transactions on pattern analysis and machine intelligence, 28(10):1584–1601, 2006.
  * [18] Y. Nie, Y. Lin, X. Han, S. Guo, J. Chang, S. Cui, and J. Zhang.  Skeleton-bridged point completion: From global inference to local adjustment.  In Advances in Neural Information Processing Systems, pages 16119–16130, 2020.
  * [19] M. Pauly, N. J. Mitra, J. Giesen, M. H. Gross, and L. J. Guibas.  Example-based 3d scan completion.  In Symposium on Geometry Processing, number CONF, pages 23–32, 2005\.
  * [20] C. R. Qi, H. Su, K. Mo, and L. J. Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
  * [21] C. R. Qi, L. Yi, H. Su, and L. J. Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  arXiv preprint arXiv:1706.02413, 2017.
  * [22] H. Rhodin, M. Salzmann, and P. Fua.  Unsupervised geometry-aware representation for 3d human pose estimation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 750–767, 2018.
  * [23] T. Shao, W. Xu, K. Zhou, J. Wang, D. Li, and B. Guo.  An interactive approach to semantic modeling of indoor scenes with an rgbd camera.  ACM Transactions on Graphics (TOG), 31(6):1–11, 2012.
  * [24] R. Shi, Z. Xue, Y. You, and C. Lu.  Skeleton merger: an unsupervised aligned keypoint detector.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 43–52, 2021.
  * [25] J. Tang, X. Han, J. Pan, K. Jia, and X. Tong.  A skeleton-bridged deep learning approach for generating meshes of complex topologies from single rgb images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4541–4550, 2019.
  * [26] L. P. Tchapmi, V. Kosaraju, H. Rezatofighi, I. Reid, and S. Savarese.  Topnet: Structural point cloud decoder.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 383–392, 2019.
  * [27] H. Wang, J. Guo, D.-M. Yan, W. Quan, and X. Zhang.  Learning 3d keypoint descriptors for non-rigid shape matching.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 3–19, 2018.
  * [28] X. Wen, T. Li, Z. Han, and Y.-S. Liu.  Point cloud completion by skip-attention network with hierarchical folding.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1939–1948, 2020.
  * [29] X. Wen, P. Xiang, Z. Han, Y.-P. Cao, P. Wan, W. Zheng, and Y.-S. Liu.  Pmp-net: Point cloud completion by learning multi-step point moving paths.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7443–7452, 2021.
  * [30] S. Wu, H. Huang, M. Gong, M. Zwicker, and D. Cohen-Or.  Deep points consolidation.  ACM Transactions on Graphics (ToG), 34(6):1–13, 2015.
  * [31] Y. Xia, Y. Xia, W. Li, R. Song, K. Cao, and U. Stilla.  Asfm-net: Asymmetrical siamese feature matching network for point completion.  arXiv preprint arXiv:2104.09587, 2021.
  * [32] P. Xiang, X. Wen, Y.-S. Liu, Y.-P. Cao, P. Wan, W. Zheng, and Z. Han.  Snowflakenet: Point cloud completion by snowflake point deconvolution with skip-transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5499–5509, 2021.
  * [33] C. Xie, C. Wang, B. Zhang, H. Yang, D. Chen, and F. Wen.  Style-based point generator with adversarial rendering for point cloud completion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4619–4628, 2021.
  * [34] H. Xie, H. Yao, S. Zhou, J. Mao, S. Zhang, and W. Sun.  Grnet: Gridding residual network for dense point cloud completion.  In European Conference on Computer Vision, pages 365–381. Springer, 2020.
  * [35] Y. Yang, C. Feng, Y. Shen, and D. Tian.  Foldingnet: Point cloud auto-encoder via deep grid deformation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 206–215, 2018.
  * [36] K. Yin, H. Huang, D. Cohen-Or, and H. Zhang.  P2p-net: Bidirectional point displacement net for shape transform.  ACM Transactions on Graphics (TOG), 37(4):1–13, 2018.
  * [37] Y. You, Y. Lou, C. Li, Z. Cheng, L. Li, L. Ma, C. Lu, and W. Wang.  Keypointnet: A large-scale 3d keypoint dataset aggregated from numerous human annotations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13647–13656, 2020.
  * [38] T. Yu, K. Guo, F. Xu, Y. Dong, Z. Su, J. Zhao, J. Li, Q. Dai, and Y. Liu.  Bodyfusion: Real-time capture of human motion and surface geometry using a single depth camera.  In Proceedings of the IEEE International Conference on Computer Vision, pages 910–919, 2017.
  * [39] T. Yu, Z. Zheng, K. Guo, J. Zhao, Q. Dai, H. Li, G. Pons-Moll, and Y. Liu.  Doublefusion: Real-time capture of human performances with inner body shapes from a single depth sensor.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7287–7296, 2018.
  * [40] X. Yu, Y. Rao, Z. Wang, Z. Liu, J. Lu, and J. Zhou.  Pointr: Diverse point cloud completion with geometry-aware transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12498–12507, 2021.
  * [41] W. Yuan, T. Khot, D. Held, C. Mertz, and M. Hebert.  Pcn: Point completion network.  In 2018 International Conference on 3D Vision (3DV), pages 728–737. IEEE, 2018.
  * [42] A. Zanfir, E. Marinoiu, M. Zanfir, A.-I. Popa, and C. Sminchisescu.  Deep network for the integrated 3d sensing of multiple people in natural images.  Advances in Neural Information Processing Systems, 31:8410–8419, 2018.
  * [43] W. Zhang, Q. Yan, and C. Xiao.  Detail preserved point cloud completion via separated feature aggregation.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16, pages 512–528. Springer, 2020.