#  NoticIA: A Clickbait Article Summarization Dataset in Spanish.

Iker García-Ferrero, Begoña Altuna
HiTZ Basque Center for Language Technology - Ixa
University of the Basque Country UPV/EHU
<EMAIL_ADDRESS>
##  1 Introduction

In the digital age, the expansion of clickbait headlines has become a pressing
concern for both media consumers and publishers. Clickbait refers to
sensationalized or misleading headlines designed to lure readers into clicking
on a link, often at the expense of accurate reporting and journalistic
integrity. Clickbait headlines may mislead readers by exaggerating or
distorting the actual content, leading to misinformation and confusion.
Clickbait headlines create a need in users to discover specific information,
compelling them to click on the article, while the low quality of the piece of
news may make them quickly exit the website without engaging with other ads or
articles.

The clickbait headlines often lead readers to enter a webpage and scroll down
it while they try to find the promised information they are searching for.
Such articles often contain a plethora of irrelevant details, with the main
idea typically buried at the end. This practice tends to be annoying for the
readers and undermines their trust in online news outlets. The ultimate goal
of clickbait-led news is to show as many advertisements as possible to the
readers in a way of trying to increase the advertisement revenue of the site.
For publishers, while clickbait may offer short-term gains in terms of
increased web traffic and potential advertising revenue, it carries the risk
of damaging their reputation and alienating their reader base. It also
negatively impacts advertising revenue for legitimate content creators, who
might see their web traffic reduced.

Figure 1: Examples of clickbait headlines. The headline is followed by a long
article in which the answer to the headline is located at the end of the
article.

The task of summarizing these low-quality articles with clickbait headlines
constitutes a great benchmark for Large Language Models (LLMs). Unlike general
domain summarization, this task first requires the model to understand the
headline and the information it promises. The summary depends on accurately
interpreting the headline, as the failure to do so will lead to an incorrect
summary. Then, it must successfully navigate through the filler content to
identify and extract the key information or idea that is often hidden, not
immediately obvious, or, in some cases, completely missing because the
headline was misleading. This task demands advanced text understanding and
summarization abilities and challenges the models’ capacity to infer and
connect diverse pieces of information to fulfil the user’s informational needs
as suggested by the clickbait headline. This challenge can be described as an
“ultrasummary” generation task, where articles, designed to be as lengthy as
possible, are summarized into a single sentence or even a single word. Figure
1 illustrates examples of clickbait headlines from our dataset, together with
the human-written summaries.

To the best of our knowledge, NoticIA is the only clickbait dataset that
provides headline-news body-ultra-summary triplets that can be used to assess
the performance of information retrieval systems that will help extract
crucial information. NoticIA comprises 850 Spanish news articles featuring
single-sentence summaries written by humans. Our contributions are the
following:

  * •

We introduce a dataset containing 850 Spanish news articles, each with a
clickbait headline, news body, and human-written summary.
111https://hf.co/datasets/Iker/NoticIA222https://hf.co/datasets/somosnlp/NoticIA-
it

  * •

We evaluate several state-of-the-art text-to-text Large Language Models (LLMs)
in zero-shot settings 333https://github.com/ikergarcia1996/NoticIA. This
evaluation demonstrates the text-understanding capabilities of these models in
Spanish, addressing the current scarcity of high-quality text-understanding
benchmarks and evaluations for LLMs in this language.

  * •

We fine-tune and publicly release LLMs trained on our dataset
444https://hf.co/Iker/ClickbaitFighter-2B555https://hf.co/Iker/ClickbaitFighter-7B666https://hf.co/Iker/ClickbaitFighter-10B.
This model exhibits high proficiency in summarizing clickbait articles and has
the potential to empower individuals by providing them with tools for critical
thinking and discernment amidst the vast amounts of online content. By making
this model public, we aim to exert pressure against the use of deceptive
tactics by online news providers to increase advertising revenue.

The coming sections of the paper are structured as follows. In section 2 we go
through the most relevant works on the topic. In section 3 the corpus building
effort is described. The experimental set-up is presented in section 4 and its
development is described in section 5. Finally, the final remarks are done in
section 6.

##  2 Related works

Most of the efforts aimed at combating clickbaits have focused on clickbait
headline detection. Although it is over a decade that the task has attracted
the interest of scholars, the Clickbait Challenge[Potthast et al.,2018] in
2017 was a milestone in the advancement of the field. There were 12 teams
taking part and the Webis Clickbait Corpus 2017 was released which contained
38,517 annotated tweets and that has been used since then in a list of
clickbait detection works such as the work by [Zheng, Yu, and Wu,2021]. In
addition, clickbait detection efforts have multiplied in the last lustrum as
in the case of [Pujahari and Sisodia,2021], [Liu et al.,2021] or [Wang,
Maslim, and Liu,2023].

Our work, however, is more closely related to the clickbait spoiling task. The
PAN Clickbait Challenge at SemEval 2023 [Fröbe et al.,2023] consisted in
classifying the types of spoilers according to their structure and in
generating spoilers according to their type. 23 teams submitted their systems
which were tested on the Webis Clickbait Spoiling Corpus 2022, a 5,000
headline-news-spoiler corpus collected from social media.

The task can also be designed in a question-answering manner. In
[Heiervang,2022] a title-answering method is proposed. Fine-tuned LLMs are
requested to extract an abstractive summary from the news body to answer the
question in the clickbait headline. [Johnson et al.,2022] consider both
abstractive and extractive summaries to provide an answer for each headline.
They fine-tune a RoBERTa model to perform the task. [Intan Maharani,
Purwarianti, and Aji,2023], instead, configure the task as a single or multi-
span information extraction task for question answering.

Misleading headlines in Spanish news have also been a matter of study as in
[Sepúlveda-Torres, Bonet-Jover, and Saquete,2023]. The goal of the task is to
identify whether a headline contains any contradiction in relation to the news
body and classify the types of contradictions. For this, the
ES_Headline_Contradiction dataset is built, which contains 18,542 news
headline-body pairs. A series of BERT-based models are fine-tuned for both
detection and classification and they achieve highly satisfactory results in
both tasks.

##  3  NoticIA dataset

The NoticIA dataset comprises 850 clickbait headline, news body and summary
triplets. In this section, we will describe the construction of the dataset.

###  3.1 Corpus collection

Clickbait headlines have proliferated on the web in recent years. However,
gathering clickbait headlines is an arduous task. To avoid the need for
massive crawling of articles and the subsequent post-processing step to detect
articles with clickbait headlines, we utilized two readily available sources
of clickbait headlines to build the dataset. The first source is the posting
on the X social media platform by the @ahorrandoclick1 user, who manually
summarizes articles with clickbait headlines. We gathered 725 articles from
this source. Secondly, during the early development of this dataset, we fine-
tuned a model to summarize clickbait articles. We used this model to develop a
publicly available web app 777https://iker-clickbaitfighter.hf.space/ that
users could use to summarize articles given the article’s URL. From the user
inputs, we selected an additional 125 articles, for a total of 850 articles in
the dataset.

Figure 2: Category of the articles in the dataset.

The dataset comprises articles from a wide range of categories, as depicted in
Figure 2. All articles are in Spanish, and approximately 83% of the dataset
originates from Spanish newspapers, while the remaining 17% comes from Latin
American newspapers. The detailed sources of the dataset can be found in
Appendix C.

###  3.2 Corpus annotation

We employed human annotators to create gold-standard summaries for the
dataset. These annotators were PhD students with native-level competence in
Spanish. They were instructed to read the headline and identify the question
that the clickbait posed for the reader. Subsequently, they were tasked with
finding the excerpt of the text that provided the answer to the clickbait
headline. The annotators were asked to produce the shortest possible summary
that responded solely to the clickbait headline, excluding any other
information. Additionally, whenever possible, they were encouraged to directly
quote the original text, especially when the summary included direct
statements made by individuals. The full guidelines are available in Appendix
F. The annotation process took approximately 40 hours. Table 1 lists the
average word counts for the headlines, article bodies, and summaries. On
average, the article body was summarized with a 98% reduction.

Train Dev Test Total Article no. 700 50 100 850 Avg. Headline Len. 16 17 17 17
Avg. Article Len. 544 663 549 552 Avg. Summary Len. 12 11 11 12

Table 1: Average word counts for the headlines, article bodies, and summaries.

###  3.3 Dataset quality assessment

Test Validation Rouge1 56.73 Rouge2 43.05 RougeL 55.59 Avg. Summary len. 11 8

Table 2: Validation stats between the summaries in the test split and the
summaries produced by another independent annotator.

The corpus annotations have been used to measure the quality of the
distributed data and
guidelines888https://hf.co/datasets/Iker/NoticIA_Human_Validation. A summary
of the inter-annotator agreement assessment is presented in Table 2. The
overall agreement between the annotators has been high as they have provided
the exact same answer in 26% of the cases and have provided answers that
partially share information in 48% of the cases. This may imply that, although
the news in our dataset tend to present the information intricately, it was
easy for humans to find the information the headline refers to.

Annotator A Annotator B Same information Tenemos entre 30 y 45 minutos. Entre
30 y 45 minutos. More precise information El regreso a clases para muchos
estudiantes. El regreso a clases. Additional information Tener que saltar el
potro. “Cuando pasé a bachillerato, al BUP, estaba el potro y pasé unos días
pensando que tenía saltar el potro y hacer deporte, muy malos”. Shared
information No le ha gustado por que “es de vieja”. “El abrigo es de vieja”,
ha opinado la mujer.

Table 3: Examples of partially overlapping answers by human annotators.

In the case of the partially overlapping answers, some may be considered to
carry the same information, while others show complementary information as can
be seen in the examples in Table 3.

We have also identified a list of cases in which the annotators have offered
different but equally valid answers, which constitutes 18% of the cases. For
example (3.3) the annotators have chosen different text strings to refer to
the same monetary prize. In (3.3), instead, answers are not directly related.

Headline: El 31938, tercer premio de la Lotería de Navidad 2023: esto es lo
que cobrarás. Annotator A: 50.000€ por décimo. Annotator B: 500 mil euros por
serie. Headline: Mario Picazo alerta de las consecuencias que tendrá el
debilitamiento del vórtice polar en España. Annotator A: El aire muy frío
confinado en el Ártico puede ser liberado hacia latitudes más bajas. Annotator
B: Unas temperaturas que estarían muy por debajo de lo que sería habitual.

For what regards the guideline assessment, we venture to say that the
guidelines were overall unambiguous and that the request of selecting the
minimum amount of words to generate a valid summary will always be
interpretable. As we could see from the annotators’ answers, the minimum
extent could be understood as the focus of the question in the headline or the
minimal well-formed sentence (3.3).

Headline: Josep Pedrerol hace frente a la pérdida de un rostro conocido de ‘El
Chiringuito de Jugones’: “Nunca olvidaré”. Annotator A: Se ha ido el
periodista Borja Mazarro. Annotator B: Borja Mazarro.

Finally, we identified 8 cases of disagreement. In 3 cases, one of the
annotators produced an incorrect summary, likely due to fatigue after
annotating multiple examples. In the remaining 5 cases, the disagreement
stemmed from contradictory information in the article and different
interpretations of this information (3.3). In these instances, determining the
correct summary is subject to the reader’s interpretation. In any case, we
consider the annotator agreement to be very high, and further experiments in
Section 5 demonstrate that there is a very high correlation on model
performance when evaluated using the summaries produced by different
annotators.

Headline: “El motivo por el que ‘Espejo Público’ ha prescindido de un mítico
colaborador: “La gota que ha colmado vaso” Annotator A: Se refieren a Fran
Rivera, el motivo fue aparecer en Telecinco. Annotator B: Rivera ya no
aportaba mucho contenido al programa.

##  4 Experimental Setup

In this section, we define the evaluation protocol we use to measure the
performance of Large Language Models (LLMs) on our dataset.

###  4.1 Models

Model Family Model name Parameter Count Citation Deepseek deepseek-
llm-67b-chat 67B [Bi et al.,2024] Gemma gemma-2b-it 2.51B [Gemma-Team et
al.,2024] gemma-7b-it 8.54B [Gemma-Team et al.,2024] Llama2 Llama-2-70b-chat-
hf 70B [Touvron et al.,2023] Llama-2-13b-chat-hf 13B [Touvron et al.,2023]
tulu-2-dpo-70b 70B [Ivison et al.,2023] Mistral Mistral-7B-Instruct-v0.2 7B
[Jiang et al.,2023] Nous-Hermes-2-Mistral-7B-DPO 7B [Teknium,2023] Nous-
Hermes-2-Pro-Mistral-7B 7B [Teknium,2023] openchat-3.5-0106 7B [Wang et
al.,2023] OpenHermes-2.5-Mistral-7B 7B [Teknium,2023] Solar
SOLAR-10.7B-Instruct-v1.0 10.7B [Kim et al.,2023] Nous-Hermes-2-SOLAR-10.7B
10.7B [Teknium,2023] Mixtral Mixtral-8x7B-Instruct-v0.1 46.7B [Jiang et
al.,2024] Nous-Hermes-2-Mixtral-8x7B-DPO 46.7B [Teknium,2023] Nous-
Hermes-2-Mixtral-8x7B-SFT 46.7B [Teknium,2023] Yi Nous-Hermes-2-Yi-34B 34B
[Teknium,2023] Yi-6B-Chat 6B [01.AI et al.,2024] Yi-34B-Chat 34B [01.AI et
al.,2024] QWEN Qwen1.5-7B-Chat 7B [Bai et al.,2023] Qwen1.5-14B-Chat 14B [Bai
et al.,2023] Qwen1.5-72B-Chat 72B [Bai et al.,2023] ChatGPT gpt-4-0125-preview
unknown [OpenAI,2023] gpt-3.5-turbo-0125 unknown [OpenAI,2024]

Table 4: Model that we use in our experiments. Model families group the models
we use in our experiments, based on the base model on which they were trained.

We evaluate a diverse set of large language models (LLMs), ranging from 2
billion to 70 billion parameters. All the models assessed have been fine-tuned
for instruction following. The models include Deepseek [Bi et al.,2024],
LLama2 [Touvron et al.,2023], Mistral [Jiang et al.,2023], Solar [Kim et
al.,2023], Mixtral [Jiang et al.,2024], StableLM [stability.ai,2023], Yi
[01.AI et al.,2024] and Qwen [Bai et al.,2023], as well as further fine-tuned
versions of these models which have improved capabilities. The models are
listed in Table 4. This selection of models spans the most popular
architectures at the time of writing and includes the current best-performing
open-source models. It is important to note that most of these models were
pretrained with a primary focus on the English or Chinese languages. Our
comprehensive evaluation aims to shed light on the performance of the current
state-of-the-art LLMs in Spanish. In addition to these open-source models, we
also evaluate the GPT3.5 and GPT4 [OpenAI,2023] commercial productions.
Although these models are not designed for research, and many details about
them remain undisclosed, given their popularity and widespread usage, we
consider it valuable to assess their performance on our Spanish ultra-
summarization task. We provide further details on the models we use in
Appendix D.

###  4.2 Task Formulation

Figure 3: Input prompt used to generate summaries. The prompt defines the task
and guidelines.

We adopt a zero-shot prompt evaluation setup, wherein we construct an
instruction that describes the task and the annotation guidelines. The model
receives the headline and the article body, and we expect a summary of the
article as the output. The prompt used as input for the models is illustrated
in Figure 3. The lengthy nature of the inputs, with article bodies averaging
550 words, precludes experimentation with more advanced prompting techniques,
such as few-shot learning—this is, providing the model with a few summary
examples—or Chain-of-Thought techniques. In the case of models that have been
fine-tuned as dialogue systems, these utilize different tokens to represent a
conversation, such as using markers like “¡bot¿” and “¡human¿”, or custom
system initial prompts. To accommodate these models, we format the input
according to the recommendations provided by the authors. The input prompt is
set to the user’s input, and we expect the summary to be produced as the
assistant’s output. We do not specify any system-defined prompts and, for all
the experiments, we use greedy search to generate text.

###  4.3 Metrics

As standard in summarization tasks, we use the ROUGE score metric [Lin,2004]
to automatically evaluate the summaries produced by the models. ROUGE is a
recall-oriented summarization metric that assesses the quality of
summarization systems by determining how much of the basic units in the human
reference summaries appear in the machine-generated summaries. Our primary
metric is ROUGE-1, which considers whole words as the basic units. To compute
the ROUGE score, we lowercase both summaries and remove punctuation. In
addition to the ROUGE score metric, we also consider the average length of the
summaries. For our task, we aim for the summaries to be concise, which is an
aspect the ROUGE score does not evaluate. Longer sentences may increase the
chance of overlapping words, potentially affecting the ROUGE score. Therefore,
we consider both ROUGE-1 and average summary length when evaluating the
models. Our goal is to identify a model that achieves the highest possible
ROUGE score with the shortest summary length, balancing quality and brevity.

Recent works [Min et al.,2023] have proposed advanced metrics for the
automatic evaluation of general-domain summaries. However, our dataset
significantly deviates from general domain summarization. Our summaries are
highly concise, and we consider any information in the summary that does not
contribute towards answering the clickbait headline as incorrect, even if this
information appears in the article. Therefore, the set of possible correct
summaries is very small, with any accurate summary necessarily including a
specific set of words. Thus, we consider the ROUGE score as the best metric
for evaluation, along with average sentence length to ensure that the model
does not produce overly verbose summaries or include unnecessary information.

##  5 Experiments

In this section, we present the evaluation of a broad range of LLMs in both
zero-shot and fine-tuning scenarios using the NoticIA dataset. Figure 4
presents a summary of all the results. In this chart, the Y-axis represents
the ROUGE score, while the X-axis indicates the average number of words in the
summaries produced by each model. As outlined in Section 4.3, our goal is to
identify the models that achieve the highest possible ROUGE score with the
shortest summary length. Additionally, we establish a human baseline,
calculated based on the second set of human annotations detailed in Section
3.3, and compare it against the gold summaries in the dataset. Extended
results in numerical format are presented in Appendix E.

Figure 4: ROUGE score and average summary lengths for all models evaluated in
our dataset. The Y-axis represents the ROUGE score, while the X-axis indicates
the average number of words in the summaries. A higher ROUGE score and a
shorter summary length are considered optimal.

###  5.1 Zero-Shot summarization

First, we evaluate a wide range of Large Language Models (LLMs) spanning from
2 billion to 70 billion parameters on the NoticIA dataset in a zero-shot
setting. This means that we use the models in their original form without any
specific task fine-tuning. The models are prompted using the instructions
described in Section 4.2.

The results indicate that GPT-4 generates the highest quality summaries,
whereas Nous-Hermes-2-Yi-34B produces the shortest summaries without
compromising quality. Both, Nous-Hermes-2-SOLAR-10.7B and OpenChat-3.5-0106,
with 10.7 billion and 7 billion parameters respectively, deliver remarkably
good performance despite being significantly smaller than the 34 billion
parameter model, Nous-Hermes-2-Yi-34B. This makes these two models the
preferred options when considering computational resource constraints.

The zero-shot results indicate that for this task, the number of model
parameters holds little significance compared to the pre-training data. For
instance, Llama-2-70b-chat-hf, a 70 billion parameter model, underperforms
compared to the OpenChat-3.5-0106 model, which has only 7 billion parameters.
The models that consistently achieve better performance share a common
feature: they are trained on very large high-quality synthetic datasets. For
example, the Nous-Hermes series of models have been trained using the
OpenHermes dataset [Teknium,2023], which comprises 1 million instructions
generated mainly with GPT-4. Similarly, OpenChat-3.5-0106 was also trained
using data generated by GPT-4 and GPT-3.5 [Wang et al.,2023]. Deepseek [Bi et
al.,2024] was trained with 1.5 million instructions. Further analysis of the
publicly available OpenHermes 2.5 dataset through the Lilac platform
999https://lilacai-lilac.hf.space/datasets#lilac/OpenHermes-2.5 reveals that
this dataset contains at least 25,749 examples of summarization, a significant
portion of which are article summarizations. Therefore, this suggests that the
best-performing models in our dataset are those that have been exposed to a
large amount of summarization examples during fine-tuning. It’s important to
note that the OpenHermes dataset predominantly contains instructions in
English, with only 62 of the 1 million instructions being in Spanish, and
these are specific examples of translations. This highlights the models’
remarkable proficiency in transfer learning, demonstrating their ability to
effectively apply knowledge learned from English data to tasks in Spanish.

###  5.2 Error analysis

To better understand the models’ performance on our dataset, we conducted an
error analysis to identify the circumstances and reasons why the models
failed. We identified three different error sources:

Producing a summary of the whole article instead of following the guidelines
Headline Los turistas británicos alucinan con el nuevo destino turístico
español por su ron Gold La isla de La Palma. Nous-Hermes-2-Yi-34B Los turistas
británicos se enamoran de La Palma por su microclima y gastronomía local.
LLama-2-70b-chat-hf Al rededor de 150.000 turistas llegan cada año, sólo una
fracción de los millones de británicos que acuden en masa a sus hermanas
mayores de Gran Canaria, Tenerife y Lanzarote.̈ Esta frase revela que La Palma
es un destino turístico menos masivo que otras islas Canarias, lo que la hace
atractiva para aquellos que buscan un lugar más tranquilo y auténtico. Además,
se menciona la excelente gastronomía local, incluyendo el ron elaborado con
caña de azúcar en la isla durante cientos de años, lo que sugiere que los
visitantes pueden disfrutar de una experiencia culinaria única y auténtica.
Summaries not concise enough Headline La comedia familiar y navideña que ya
está en Netflix y puedes ver en estas fiestas Gold Mejor Navidad ¡imposible!
Nous-Hermes-2-Yi-34B La comedia familiar y navideña “Mejor Navidad
¡Imposible!’ ’ ya está disponible en Netflix. GPT-4-0125-preview “Mejor
Navidad ¡imposible! es la comedia familiar y navideña lanzada por Netflix el
16 de noviembre de 2023.’ ’ Failure to understand the Headline or Text.
Headline ¡Isabel Preysler estará en El Hormiguero! Estos son los próximos
invitados Gold Malú, Pedro Alonso, Begoña Vargas e Isabel Preysler. Nous-
Hermes-2-Yi-34B Isabel Preysler, Malú, Pedro Alonso y Begoña Vargas serán
invitados en El Hormiguero la próxima semana. openchat-3.5-0106 Isabel
Preysler hablará sobre “Isabel Preysler: mi Navidad’ ’ en Disney+. Headline
¿Qué pilotos de F1 acaban contrato en 2024? Gold Sergio Pérez, Carlos Sainz,
Fernando Alonso, Esteban Ocon, Alexander Albon, Logan Sargeant, Yuki Tsunoda,
Daniel Ricciardo, Valtteri Bottas, Guanyu Zhou, Kevin Magnussen y Nico
Hulkenberg. Nous-Hermes-2-Yi-34B Más de la mitad de la parrilla de F1 acaba
contrato en 2024. GPT-4-0125-preview Sergio Pérez, Carlos Sainz, Esteban Ocon,
Alexander Albon, Logan Sargeant, Daniel Ricciardo, Yuki Tsunoda, Valtteri
Bottas y Guanyu Zhou finalizan sus contratos en 2024.

Table 5: Error analysis, examples of the different errors produced by the
models.

##### Producing a summary of the whole article instead of following the
guidelines:

We identified that some models, instead of adhering to the provided guidelines
and producing a concise summary based on the headline, simply summarize the
entire article. Table 5 shows the summaries generated by Nous-Hermes-2-Yi-34B
and Llama-2-70b-chat-hf along with the Gold summary. The former generates a
summary that answers the question, while the latter produces an overly
extensive summary of the whole article. We attribute this to the lower
proficiency of some models in following instructions. As previously discussed,
Nous-Hermes-2-Yi-34B, having been trained with 1 million instruction examples,
has acquired much better proficiency at following instructions and adhering to
the guidelines described in such instructions. The models in Figure 4 which
generate the longest summaries, all suffer from this issue.

##### Not concise enough summaries:

The best-performing models in our dataset, which follow the guidelines and
produce summaries that address the clickbait headline, still generate
summaries that are, on average, twice as long as those produced by humans.
Even when explicitly prompted to produce an answer using the minimum number of
words possible, and clarifying that the answer does not need to be a complete
sentence, they still output a complete sentence. This likely reflects a bias
from the instruction examples used during the models’ fine-tuning phase. All
models struggle to produce very short answers. Nevertheless, it is possible
that further refining the prompt used in our experiments, or employing a
multi-step inference process where the models are asked to reduce the number
of words in their summaries, could mitigate this issue.

##### Failure to understand the headline or text:

This issue arises when the model does not correctly comprehend the clickbait
headline or the information in the text. As illustrated in Table 5, for the
headline “¡Isabel Preysler estará en El Hormiguero! Estos son los próximos
invitados”, which asks about the upcoming guests on a Spanish TV program,
OpenChat-3.5-0106, rather than listing the invited individuals, generates a
summary that details the topic of discussion for a guest. This indicates a
misunderstanding of the headline. Similarly, in response to the question “¿Qué
pilotos de F1 acaban contrato en 2024?”, which is accompanied by an article
detailing the contract status of every Formula 1 driver, we found that none of
the models could generate the correct list of drivers whose contracts end in
2024. In this case, this indicates a misunderstanding of the article body.
Although the best-performing models show significantly fewer instances in
which they fail to understand the headline or text, we found evidence of this
issue across all models. In this case, the issue is related to the text
comprehension capabilities of the models.

####  5.2.1 Evaluation Consistency with Different Human Annotations

Figure 5: ROUGE scores of the models when evaluated against the gold summaries
and the validation summaries produced by a second set of human annotators.

To further assess the quality of our dataset and the reliability of the human
annotations, we evaluated the models in zero-shot settings against both sets
of human annotations described in Section 3.3. The results are presented in
Figure 5. The summaries from the second set of annotations were, on average,
slightly shorter than those from the first set. This led to lower ROUGE scores
for the models when compared with the second set of human annotations.
Nevertheless, we observed a nearly perfect correlation in the scores (Spearman
correlation of 0.99) when evaluated against both sets of human annotations.
This indicates that the gold summaries in our dataset are robust and correlate
well with those generated by different annotators following the same
guidelines. It also demonstrates that humans can achieve a very high level of
agreement on this task.

###  5.3  Clickbait Fighter a Task specific model

In this effort, we fine-tune three task-specific models using the 700 training
examples in the NoticIA dataset. We fine-tune three models of different sizes:
ClickbaitFighter-10B, which is based on Nous-Hermes-2-SOLAR-10.7B,
ClickbaitFighter-7B, which is based on openchat-3.5-0106, and
ClickbaitFighter-2B, which is based on Gemma-2B-IT. We fine-tune all the
parameters of the models using bfloat16 precision. More details about the
fine-tuning process are available in Appendix A. The results of the models are
displayed in Figure 4. Despite being a small model, ClickbaitFighter-2B
performs better than any evaluated LLM in the zero-shot setting, making this
model ideal for situations where computational resources are constrained. On
the other hand, ClickbaitFighter-10B and 7B achieve a summary quality close to
the human baseline. Task-specific models overcome the struggle of producing
concise summaries and manage to produce summaries almost twice as short as
those from models in zero-shot settings.

##### How many training samples do we need?

Figure 6: Comparison of Average Length and ROUGE Score when fine-tuning Nous-
Hermes-2-SOLAR-10.7B with different amounts of training samples.

As an example, in Table 6, we analyze the performance of Nous-
Hermes-2-SOLAR-10.7B when trained with varying amounts of training samples.
The results indicate that when trained with a small number of samples, the
model can learn to produce concise summaries, reducing the average number of
words in the output summary by 2.5. On the other hand, the ROUGE score
continues to increase as more training samples are added. The model’s
performance does not show any signs of saturation. This suggests that
extending the dataset beyond 750 training samples could further enhance model
performance.

##  6 Conclusions

In this work, we present NoticIA, a dataset comprising 850 Spanish news
articles featuring prominent clickbait headlines, each paired with high-
quality, single-sentence summaries written by humans. To the best of our
knowledge, NoticIA is the only clickbait dataset that provides headline-news
body-ultrasummary triplets, which can be used to assess the performance of
information retrieval systems aimed at extracting crucial information.

Our experiments demonstrate that NoticIA can effectively assess the text
comprehension capabilities of large language models (LLMs) in the Spanish
language. It also proves effective for training LLMs for the task of clickbait
article summarization. Hence, our dataset holds the potential to contribute
towards the development and evaluation of LLMs for the Spanish language.

In future work, we plan to further extend the dataset, as results indicate
that expanding the dataset beyond 750 training samples could further enhance
model performance.

## Acknowledgements

This work has been partially funded by:

  * •

DeepR3 (TED2021-130295B-C31) funded by MCIN/AEI/10.13039/501100011033 and
European Union NextGeneration EU/PRTR.

  * •

Disargue (TED2021-130810B-C21) MCIN/AEI/10.13039/501100011033 and European
Union NextGenerationEU/PRTR.

  * •

DeepKnowledge (PID2021-127777OB-C21) MCIN/AEI/10.13039/501100011033 and by
FEDER, EU.

A demo of the ClickbaitFighter-7B model can be found in the SomosNLP Hugging
Face organization. This demo was developed during the SomosNLP 2024 Hackathon:
https://hf.co/spaces/somosnlp/NoticIA-demo

## References

  * [01.AI et al.,2024] 01.AI, A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang, K. Yu, P. Liu, Q. Liu, S. Yue, S. Yang, S. Yang, T. Yu, W. Xie, W. Huang, X. Hu, X. Ren, X. Niu, P. Nie, Y. Xu, Y. Liu, Y. Wang, Y. Cai, Z. Gu, Z. Liu, and Z. Dai.  2024\.  Yi: Open Foundation Models by 01.AI.
  * [Bai et al.,2023] Bai, J., S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, S. Wang, W. Wang, S. Wu, B. Xu, J. Xu, A. Yang, H. Yang, J. Yang, S. Yang, Y. Yao, B. Yu, H. Yuan, Z. Yuan, J. Zhang, X. Zhang, Y. Zhang, Z. Zhang, C. Zhou, J. Zhou, X. Zhou, and T. Zhu.  2023\.  Qwen Technical Report.  CoRR, abs/2309.16609.
  * [Bi et al.,2024] Bi, X., D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fu, H. Gao, K. Gao, W. Gao, R. Ge, K. Guan, D. Guo, J. Guo, G. Hao, Z. Hao, Y. He, W. Hu, P. Huang, E. Li, G. Li, J. Li, Y. Li, Y. K. Li, W. Liang, F. Lin, A. X. Liu, B. Liu, W. Liu, X. Liu, X. Liu, Y. Liu, H. Lu, S. Lu, F. Luo, S. Ma, X. Nie, T. Pei, Y. Piao, J. Qiu, H. Qu, T. Ren, Z. Ren, C. Ruan, Z. Sha, Z. Shao, J. Song, X. Su, J. Sun, Y. Sun, M. Tang, B. Wang, P. Wang, S. Wang, Y. Wang, Y. Wang, T. Wu, Y. Wu, X. Xie, Z. Xie, Z. Xie, Y. Xiong, H. Xu, R. X. Xu, Y. Xu, D. Yang, Y. You, S. Yu, X. Yu, B. Zhang, H. Zhang, L. Zhang, L. Zhang, M. Zhang, M. Zhang, W. Zhang, Y. Zhang, C. Zhao, Y. Zhao, S. Zhou, S. Zhou, Q. Zhu, and Y. Zou.  2024\.  DeepSeek LLM: Scaling Open-Source Language Models with Longtermism.  CoRR, abs/2401.02954.
  * [Fröbe et al.,2023] Fröbe, M., B. Stein, T. Gollub, M. Hagen, and M. Potthast.  2023\.  SemEval-2023 Task 5: Clickbait Spoiling.  In A. K. Ojha, A. S. Doğruöz, G. Da San Martino, H. Tayyar Madabushi, R. Kumar, and E. Sartori, editors, Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2275–2286, Toronto, Canada, July. Association for Computational Linguistics.
  * [Gemma-Team et al.,2024] Gemma-Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, P. Tafti, L. Hussenot, A. Chowdhery, A. Roberts, A. Barua, A. Botev, A. Castro-Ros, A. Slone, A. Héliou, A. Tacchetti, A. Bulanova, A. Paterson, B. Tsai, B. Shahriari, C. L. Lan, C. A. Choquette-Choo, C. Crepy, D. Cer, D. Ippolito, D. Reid, E. Buchatskaya, E. Ni, E. Noland, G. Yan, G. Tucker, G.-C. Muraru, G. Rozhdestvenskiy, H. Michalewski, I. Tenney, I. Grishchenko, J. Austin, J. Keeling, J. Labanowski, J.-B. Lespiau, J. Stanway, J. Brennan, J. Chen, J. Ferret, J. Chiu, J. Mao-Jones, K. Lee, K. Yu, K. Millican, L. L. Sjoesund, L. Lee, L. Dixon, M. Reid, M. Mikuła, M. Wirth, M. Sharman, N. Chinaev, N. Thain, O. Bachem, O. Chang, O. Wahltinez, P. Bailey, P. Michel, P. Yotov, P. G. Sessa, R. Chaabouni, R. Comanescu, R. Jana, R. Anil, R. McIlroy, R. Liu, R. Mullins, S. L. Smith, S. Borgeaud, S. Girgin, S. Douglas, S. Pandya, S. Shakeri, S. De, T. Klimenko, T. Hennigan, V. Feinberg, W. Stokowiec, Y. hui Chen, Z. Ahmed, Z. Gong, T. Warkentin, L. Peran, M. Giang, C. Farabet, O. Vinyals, J. Dean, K. Kavukcuoglu, D. Hassabis, Z. Ghahramani, D. Eck, J. Barral, F. Pereira, E. Collins, A. Joulin, N. Fiedel, E. Senter, A. Andreev, and K. Kenealy.  2024\.  Gemma: Open Models Based on Gemini Research and Technology.
  * [Heiervang,2022] Heiervang, M.  2022\.  Abstractive title answering for clickbait content.  Master’s thesis, University of Oslo.
  * [Intan Maharani, Purwarianti, and Aji,2023] Intan Maharani, N. P., A. Purwarianti, and A. F. Aji.  2023\.  Low-Resource Clickbait Spoiling for Indonesian via Question Answering.  In 2023 10th International Conference on Advanced Informatics: Concept, Theory and Application (ICAICTA), pages 1–6.
  * [Ivison et al.,2023] Ivison, H., Y. Wang, V. Pyatkin, N. Lambert, M. Peters, P. Dasigi, J. Jang, D. Wadden, N. A. Smith, I. Beltagy, and H. Hajishirzi.  2023\.  Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2.  CoRR, abs/2311.10702.
  * [Jiang et al.,2023] Jiang, A. Q., A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de Las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed.  2023\.  Mistral 7B.  CoRR, abs/2310.06825.
  * [Jiang et al.,2024] Jiang, A. Q., A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de Las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao, T. Gervet, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed.  2024\.  Mixtral of Experts.  CoRR, abs/2401.04088.
  * [Johnson et al.,2022] Johnson, O., B. Lou, J. Zhong, and A. Kurenkov.  2022\.  Saved You A Click: Automatically Answering Clickbait Titles.
  * [Kim et al.,2023] Kim, D., C. Park, S. Kim, W. Lee, W. Song, Y. Kim, H. Kim, Y. Kim, H. Lee, J. Kim, C. Ahn, S. Yang, S. Lee, H. Park, G. Gim, M. Cha, H. Lee, and S. Kim.  2023\.  SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling.  CoRR, abs/2312.15166.
  * [Lin,2004] Lin, C.-Y.  2004\.  ROUGE: A Package for Automatic Evaluation of Summaries.  In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, July. Association for Computational Linguistics.
  * [Liu et al.,2021] Liu, T., K. Yu, L. Wang, X. Zhang, and X. Wu.  2021\.  WCD: A New Chinese Online Social Media Dataset for Clickbait Analysis and Detection.  In 2021 7th IEEE International Conference on Network Intelligence and Digital Content (IC-NIDC), pages 368–372.
  * [Min et al.,2023] Min, S., K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, P. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi.  2023\.  FActScore: Fine-grained atomic evaluation of factual precision in long form text generation.  In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100, Singapore, December. Association for Computational Linguistics.
  * [OpenAI,2023] OpenAI.  2023\.  GPT-4 Technical Report.  CoRR, abs/2303.08774.
  * [OpenAI,2024] OpenAI.  2024\.  gpt-3.5-turbo-0125.
  * [Pal et al.,2024] Pal, A., D. Karkhanis, S. Dooley, M. Roberts, S. Naidu, and C. White.  2024\.  Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive.
  * [pansophic,2023] pansophic.  2023\.  A 3B parameter GPT-like model fine-tuned on a mix of publicly available datasets using DPO.
  * [Potthast et al.,2018] Potthast, M., T. Gollub, K. Komlossy, S. Schuster, M. Wiegmann, E. P. Garces Fernandez, M. Hagen, and B. Stein.  2018\.  Crowdsourcing a Large Corpus of Clickbait on Twitter.  In E. M. Bender, L. Derczynski, and P. Isabelle, editors, Proceedings of the 27th International Conference on Computational Linguistics, pages 1498–1507, Santa Fe, New Mexico, USA, August. Association for Computational Linguistics.
  * [Pujahari and Sisodia,2021] Pujahari, A. and D. S. Sisodia.  2021\.  Clickbait detection using multiple categorisation techniques.  Journal of Information Science, 47(1):118–128.
  * [Ren et al.,2021] Ren, J., S. Rajbhandari, R. Y. Aminabadi, O. Ruwase, S. Yang, M. Zhang, D. Li, and Y. He.  2021\.  ZeRO-Offload: Democratizing Billion-Scale Model Training.  In I. Calciu and G. Kuenning, editors, 2021 USENIX Annual Technical Conference, USENIX ATC 2021, July 14-16, 2021, pages 551–564. USENIX Association.
  * [Sepúlveda-Torres, Bonet-Jover, and Saquete,2023] Sepúlveda-Torres, R., A. Bonet-Jover, and E. Saquete.  2023\.  Detecting Misleading Headlines Through the Automatic Recognition of Contradiction in Spanish.  IEEE Access, 11:72007–72026.
  * [stability.ai,2023] stability.ai.  2023\.  Introducing Stable LM Zephyr 3B: A New Addition to Stable LM, Bringing Powerful LLM Assistants to Edge Devices.
  * [Teknium,2023] Teknium.  2023\.  OpenHermes 2.5: An Open Dataset of Synthetic Data for Generalist LLM Assistants.
  * [Touvron et al.,2023] Touvron, H., L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom.  2023\.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  CoRR, abs/2307.09288.
  * [Tunstall et al.,2023] Tunstall, L., E. Beeching, N. Lambert, N. Rajani, K. Rasul, Y. Belkada, S. Huang, L. von Werra, C. Fourrier, N. Habib, N. Sarrazin, O. Sanseviero, A. M. Rush, and T. Wolf.  2023\.  Zephyr: Direct Distillation of LM Alignment.  CoRR, abs/2310.16944.
  * [Wang et al.,2023] Wang, G., S. Cheng, X. Zhan, X. Li, S. Song, and Y. Liu.  2023\.  OpenChat: Advancing Open-source Language Models with Mixed-Quality Data.  CoRR, abs/2309.11235.
  * [Wang, Maslim, and Liu,2023] Wang, H.-C., M. Maslim, and H.-Y. Liu.  2023\.  CA-CD: context-aware clickbait detection using new Chinese clickbait dataset with transfer learning method.  Data Technologies and Applications, ahead-of-print(ahead-of-print), Jan.
  * [Zhang et al.,2024] Zhang, P., G. Zeng, T. Wang, and W. Lu.  2024\.  Tinyllama: An open-source small language model.  CoRR, abs/2401.02385.
  * [Zheng, Yu, and Wu,2021] Zheng, J., K. Yu, and X. Wu.  2021\.  A deep model based on Lure and Similarity for Adaptive Clickbait Detection.  Knowledge-Based Systems, 214:106714.

##  Appendix A Fine tuning details

In this section, we detail our fine-tuning process. We fine-tune all model
parameters using bfloat16 precision. Utilizing DeepSpeed Zero 3 [Ren et
al.,2021], we distribute the model, gradients, and optimizer across four A100
80GB GPUs. We employ the standard Next Token Prediction (NTP) loss for
training our models. To prevent the loss associated with the article body
tokens from overshadowing the loss of the summary output tokens, we compute
the loss exclusively over the summary tokens. The hyperparameter settings are
outlined in Table 6. For all the experiments, we use greedy search to generate
text.

ClickbaitFighter-2B ClickbaitFighter-10B Batch Size 64 64 Optimizer AdamW
AdamW Scheduler Cosine Cosine Learning Rate 0.000005 0.000005 Weight Decay 0.0
0.0 Warmup Ratio 0.1 0.1 adam_beta1 0.9 0.9 adam_beta2 0.95 0.95 adam_epsilon
1e-12 1e-12 Epochs 5 3 Sequence Length 8192 8192 Compute Type Bfloat16
Bfloat16

Table 6: Detailed Hyperparameter Setting.

##  Appendix B Hardware used

We conducted all our experiments on a machine equipped with four NVIDIA A100
GPUs, each with 80GB of memory, interconnected via NVLink. The machine
features two AMD EPYC 7513 32-Core Processors and 1TB (1024GB) of RAM.

##  Appendix C NoticIA Dataset: Article sources

Figure 7 lists all the sources from which we gathered the news articles. The
articles were not evenly sampled from the news outlets; therefore, this table
does not indicate the number of clickbait articles published by different
newspapers. Instead, more popular news sources tend to have more appearances
in our dataset.

Figure 7: Sources from which the articles in the dataset have been gathered.

##  Appendix D Model details

Table 7 lists all the models we have evaluated, grouped by model families
according to the base model on which they were trained. Each model has
undergone instruction-following fine-tuning. We also provide the parameter
count for each model, a citation for the model description, and the URL from
which we downloaded them. All the models are open-source, except the ChatGPT
model family.

Model Family Model name Parameter Count Citation URL Deepseek deepseek-
llm-67b-chat 67B [Bi et al.,2024] https://hf.co/deepseek-ai/deepseek-
llm-67b-chat Gemma gemma-2b-it 2.51B [Gemma-Team et al.,2024]
https://hf.co/google/gemma-2b-it gemma-7b-it 8.54B [Gemma-Team et al.,2024]
https://hf.co/google/gemma-7b-it Llama2 Llama-2-70b-chat-hf 70B [Touvron et
al.,2023] https://hf.co/meta-llama/Llama-2-70b-chat-hf Llama-2-13b-chat-hf 13B
[Touvron et al.,2023] https://hf.co/meta-llama/Llama-2-13b-chat-hf
Llama-2-7b-chat-hf 7B [Touvron et al.,2023] https://hf.co/meta-
llama/Llama-2-7b-chat-hf Nous-Hermes-2-Llama-2-70B.yaml 70B [Teknium,2023]
https://hf.co/NousResearch/Nous-Hermes-2-Llama-2-70B TinyLlama-1.1B-Chat-v1.0
1.1B [Zhang et al.,2024] https://hf.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0
tulu-2-dpo-70b 70B [Ivison et al.,2023]
https://huggingface.co/allenai/tulu-2-dpo-70b Mistral Mistral-7B-Instruct-v0.2
7B [Jiang et al.,2023] https://hf.co/mistralai/Mistral-7B-Instruct-v0.2 Nous-
Hermes-2-Mistral-7B-DPO 7B [Teknium,2023] https://hf.co/NousResearch/Nous-
Hermes-2-Mistral-7B-DPO Nous-Hermes-2-Pro-Mistral-7B 7B [Teknium,2023]
https://hf.co/NousResearch/Hermes-2-Pro-Mistral-7B openchat-3.5-0106 7B [Wang
et al.,2023] https://hf.co/openchat/openchat-3.5-0106
OpenHermes-2.5-Mistral-7B 7B [Teknium,2023]
https://hf.co/teknium/OpenHermes-2.5-Mistral-7B zephyr-7b-beta 7B [Tunstall et
al.,2023] https://hf.co/HuggingFaceH4/zephyr-7b-beta Solar
SOLAR-10.7B-Instruct-v1.0 10.7B [Kim et al.,2023]
https://hf.co/upstage/SOLAR-10.7B-Instruct-v1.0 Nous-Hermes-2-SOLAR-10.7B
10.7B [Teknium,2023] https://hf.co/NousResearch/Nous-Hermes-2-SOLAR-10.7B
Mixtral Mixtral-8x7B-Instruct-v0.1 46.7B [Jiang et al.,2024]
https://hf.co/mistralai/Mixtral-8x7B-Instruct-v0.1 Nous-
Hermes-2-Mixtral-8x7B-DPO 46.7B [Teknium,2023]
https://hf.co/NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO Nous-
Hermes-2-Mixtral-8x7B-SFT 46.7B [Teknium,2023]
https://hf.co/NousResearch/Nous-Hermes-2-Mixtral-8x7B-SFT StableLM Nous-
Capybara-3B-V1.9 2.8B [Teknium,2023] https://hf.co/NousResearch/Nous-
Capybara-3B-V1.9 rocket-3B 2.8B [pansophic,2023]
https://hf.co/pansophic/rocket-3B stablelm-zephyr-3b 3B [stability.ai,2023]
https://hf.co/stabilityai/stablelm-zephyr-3b Yi Nous-Hermes-2-Yi-34B 34B
[Teknium,2023] https://hf.co/NousResearch/Nous-Hermes-2-Yi-34B Yi-6B-Chat 6B
[01.AI et al.,2024] https://hf.co/01-ai/Yi-6B-Chat Yi-34B-Chat 34B [01.AI et
al.,2024] https://hf.co/01-ai/Yi-34B-Chat QWEN Qwen1.5-7B-Chat 7B [Bai et
al.,2023] https://hf.co/Qwen/Qwen1.5-7B-Chat Qwen1.5-14B-Chat 14B [Bai et
al.,2023] https://hf.co/Qwen/Qwen1.5-14B-Chat Qwen1.5-72B-Chat 72B [Bai et
al.,2023] https://hf.co/Qwen/Qwen1.5-72B-Chat Smaug-72B-v0.1 72B [Pal et
al.,2024] https://hf.co/abacusai/Smaug-72B-v0.1 ChatGPT gpt-4-0125-preview
unknown [OpenAI,2023] https://openai.com/ gpt-3.5-turbo-0125 unknown
[OpenAI,2024] https://openai.com/

Table 7: Models that we use in our experiments. Model families group the
models used in our experiments, based on the base model on which they were
trained.

##  Appendix E Extended Results

In this section, we present all the results in numerical format. The results
can be found in Table 8. Additionally, we include models that, due to their
significantly low performance, were not featured in the main results section.

##  Appendix F NoticIA Dataset: Annotator Guidelines

Figure 8 displays in detail the guidelines that were provided to the human
annotators for generating summaries. These guidelines encompass a definition
of “clickbait”, the specific instructions for annotation, and several
examples. Initially, we asked the annotators to summarize a few examples.
After inspecting these initial summaries, we revised and updated the
guidelines to arrive at the final refined version.

Model Name ROUGE Score Average Summary Length Human 56.43 11.03
ClickbaitFighter-10B 52.01 7.66 ClickbaitFighter-7B 49.81 8.48
ClickbaitFighter-2B 37.58 9.23 gpt-4-0125-preview 32.00 23.82 Nous-
Hermes-2-Yi-34B 29.99 18.98 deepseek-llm-67b-chat 29.09 25.15
gpt-3.5-turbo-0125 28.64 20.44 openchat-3.5-0106 28.42 23.61 Qwen1.5-72B-Chat
27.80 26.89 Nous-Hermes-2-SOLAR-10.7B 27.48 20.72 Nous-
Hermes-2-Mixtral-8x7B-SFT 26.99 31.89 Nous-Hermes-2-Mistral-7B-DPO 26.33 25.22
OpenHermes-2.5-Mistral-7B 25.91 25.21 Mistral-7B-Instruct-v0.2 25.31 26.01
Mixtral-8x7B-Instruct-v0.1 25.24 32.39 Qwen1.5-14B-Chat 23.88 25.57
tulu-2-dpo-70b 23.27 38.56 gemma-2b-it 22.24 24.61 gemma-7b-it 22.04 28.19
SOLAR-10.7B-Instruct-v1.0 21.34 38.13 Qwen1.5-7B-Chat 20.66 26.26 Nous-
Hermes-2-Mixtral-8x7B-DPO 19.74 58.68 Llama-2-70b-chat-hf 19.20 59.15
Llama-2-13b-chat-hf 18.37 54.46 zephyr-7b-beta 15.64 105.75 Llama-2-7b-chat-hf
14.65 73.50 stablelm-zephyr-3b 12.02 86.90 Nous-Hermes-2-Llama-2-70B 7.22
154.55 TinyLlama-1.1B-Chat-v1.0 3.84 191.15 Smaug-72B-v0.1 1.79 82.74
rocket-3B 0.92 228.43

Table 8: ROUGE score and average summary lengths for all the models evaluated
in our dataset.

Figure 8: Guidelines provided to the annotators.


# Contrastive Predictive Autoencoders for Dynamic Point Cloud
Self-Supervised Learning

Xiaoxiao Sheng, Zhiqiang Shen, Gang Xiao Gang Xiao is the corresponding
author.

###### Abstract

We present a new self-supervised paradigm on point cloud sequence
understanding. Inspired by the discriminative and generative self-supervised
methods, we design two tasks, namely point cloud sequence based Contrastive
Prediction and Reconstruction (CPR), to collaboratively learn more
comprehensive spatiotemporal representations. Specifically, dense point cloud
segments are first input into an encoder to extract embeddings. All but the
last ones are then aggregated by a context-aware autoregressor to make
predictions for the last target segment. Towards the goal of modeling multi-
granularity structures, local and global contrastive learning are performed
between predictions and targets. To further improve the generalization of
representations, the predictions are also utilized to reconstruct raw point
cloud sequences by a decoder, where point cloud colorization is employed to
discriminate against different frames. By combining classic contrast and
reconstruction paradigms, it makes the learned representations with both
global discrimination and local perception. We conduct experiments on four
point cloud sequence benchmarks, and report the results on action recognition
and gesture recognition under multiple experimental settings. The performances
are comparable with supervised methods and show powerful transferability.

## Introduction

The development of depth sensors facilitates the acquisition of dynamic point
clouds and makes it widely applied in many scenarios, such as autonomous
vehicles and robots. Currently, processing these real-time point clouds for
surroundings perception still relies on supervised methods. However, labeling
plenty of dynamic point clouds is quite labor-intensive and error-prone.
Meanwhile, self-supervised methods in images make great success, significantly
alleviating annotation requirement and even outperforming supervised
pretraining methods in many tasks (Zbontar et al. 2021; He et al. 2020).
Inspired by this, this paper focuses on learning dynamic point cloud
representations in a self-supervised manner.

Figure 1: Illustration of our main idea. We combine contrastive predictive
coding and reconstruction to establish a unified self-supervised framework for
dynamic point clouds.

Classical self-supervised methods in images always rely on contrasting the
samples with strong data augmentations or reconstructing the randomly masked
patches based on visible parts to obtain high-level semantics (Zbontar et al.
2021; Bao et al. 2021). For discriminative methods, rich and diverse sample
pairs are usually created. However, it is likely that the semantic consistency
of the augmented dynamic point cloud pairs is difficult to be guaranteed.
Furthermore, experimental results in (Xie et al. 2020) also demonstrate that
real multi-camera views are more effective than hand-crafted views for point
clouds. For generative self-supervised methods, the original point coordinates
need to be recovered. However, the corresponding coordinates will be added to
masked tokens as positional embeddings. This results in information leakage
and makes the task less challenging. Based on the above analysis, it is
natural to ask a question: _how can we leverage the advantages of
discriminative and generative methods for self-supervised learning on dynamic
point clouds?_

Considering that point cloud sequences are constantly evolving with geometric
dynamics, we can explore temporal-related self-supervised signals lying in the
sequence itself, rather than relying on massive augmented data. Contrastive
predictive coding is to predict about future based on recent past of the
sequences, which has been successfully validated in speeches and videos (Oord,
Li, and Vinyals 2018; Han, Xie, and Zisserman 2020a). Generally, this paradigm
aims to use powerful encoder and autoregressor to learn high-level
representations with a contrastive pretext task. In addition, to introduce
more task-independent properties, we explore to maximize the mutual
information between representations and inputs. Therefore, we integrate point
cloud sequence reconstruction into contrastive predictive coding for learning
more generalized representations. Our main idea is illustrated in Figure 1.

Specifically, we first sample multiple segments and then send them into an
encoder to extract spatiotemporal features. The features of all but the last
segments are regarded as tokens to fed into a transformer autoregressor to
predict future states. We perform contrastive tasks between the predictions
and target segment features. However, due to the unordered and irregular
properties of point cloud, the points attached with predicted spatiotemporal
features are not aligned to those target ones. We design an interpolation
based method to achieve points alignment and further update predicted features
correspondingly. Then, local contrastive learning is performed, and we also
explore hard negatives to enhance the discriminability of the model.
Meanwhile, global contrastive learning is conducted to compare the embeddings
of class tokens with sequence-level features of entire current and predicted
segments. In addition, raw point cloud sequences of the target segment are
reconstructed based on the predictions. Point cloud colorization is applied to
the target segment for further discriminating frames. Overall, contrastive
prediction task explores more on the discrimination between diverse point
cloud sequences, while generative self-supervised task perceives internal
structures of point cloud itself. By combining these two self-supervised tasks
adapted to dynamic point cloud, it enables the learned representations more
comprehensive with both powerful instance discriminability and regional
context-awareness.

Multi-granularity representations can be learned by establishing our unified
contrastive prediction and reconstruction (CPR) self-supervised framework. We
evaluate our method on four dynamic point cloud benchmarks, including
MSRAction3D (Li, Zhang, and Liu 2010), NTU-RGBD 60 (Shahroudy et al. 2016),
NvGesture (Molchanov et al. 2016), and SHREC’17 (De Smedt et al. 2017). Our
method achieves excellent performance comparable with state-of-the-art
supervised methods. Ablation studies are performed to investigate the design
of self-supervised tasks. The main contributions of this paper are as follows:

  * •

We propose a new contrastive prediction and reconstruction self-supervised
framework for dynamic point cloud understanding.

  * •

We conduct local and global contrastive predictions to learn comprehensive
representations with both context-awareness and holistic discrimination.

  * •

We reconstruct raw point cloud sequences to promote the generalization of
learned representations, and apply point cloud colorization to discriminate
different frames.

  * •

We perform extensive experiments on multiple dynamic point cloud benchmarks.
We also show detailed ablation studies to analyze how to design self-
supervised tasks.

## Related Work

### Self-supervised Methods for Images

Benefiting from massive data, self-supervised methods in images achieve
remarkable performance. This not only alleviates the demands for labeling, but
further reveals that more generalized representations are obtained in a self-
supervised manner without introducing label ambiguity. Discriminative self-
supervised methods compare two views of one sample processed by diverse data
augmentations to learn high-level semantics, and utilize plenty of negative
samples for contrast to promote instance discriminability (Chen et al. 2020).
More effective techniques are also explored to enhance the generalization of
pretrained encoder for downstream tasks, such as dynamic queues (He et al.
2020), momentum updating (He et al. 2020), stop-gradients (Chen and He 2021),
and clustering (Caron et al. 2020). Generative self-supervised methods aim to
learn the generalized representation by mask image modeling (Bao et al. 2021;
Xie et al. 2022; He et al. 2022). The masked and unmasked tokens can be
simultaneously processed by an encoder, and an extra decoder is necessary to
reconstruct raw input or low-level features based on the latent
representations (Bao et al. 2021). An asymmetric network structure is also
explored to only process the visible patches by the encoder, which
significantly improves computation efficiency (He et al. 2022).

### Self-supervised Methods for Videos

Self-supervised methods of videos further consider the properties of sequence
data itself, such as temporal continuity and redundancy. Similarly,
discriminative methods focus on how to build sample pairs from diverse video
clips (Pan et al. 2021; Qian et al. 2021). Generative methods randomly mask
the spatiotemporal tubes with a higher ratio to prevent information leakage
(Tong et al. 2022; Wang et al. 2022b). Besides, many works have been proposed
to model motions, such as co-training of two models based RGB and optical flow
separately (Han, Xie, and Zisserman 2020b), or building decoupled static and
dynamic concepts (Qian et al. 2022). Clearly, contrastive learning based
methods emphasize the discrimination between diverse samples, while
reconstruction based methods focus on the perception and understanding of
samples themselves. They are both limited in learning multi-granularity
representations.

### Self-supervised Methods for Static Point Clouds

Inspired by the success of self-supervised methods in images, many works focus
on learning point cloud representations in a self-supervised manner.
PointContrast (Xie et al. 2020) is proposed to contrast real-world multi-view
point clouds with various data augmentations. Unfortunately, multi-camera
viewpoints are not available in some scenarios. As a BERT-style self-
supervised framework, Point-BERT (Yu et al. 2022) predicts latent
representations learned by an offline tokenizer, which causes two-stage
pretraining. In addition, directly recovering raw point cloud easily leads to
information leakage due to the positional encoding of masked tokens. To
alleviate the above problem, PointMAE (Pang et al. 2022) is proposed to only
input unmasked tokens into encoder and add the masked tokens to the decoder
for reconstructing. Alternatively, MaskPoint (Liu, Cai, and Lee 2022) is
proposed to randomly select visible points into encoder and train a decoder to
discriminate between masked points and noises points. It combines the
discriminative method and the masking task to achieve excellent performance.

### Modeling Dynamic Point Clouds

Currently, modeling dynamic point cloud is still dominated by traditional
supervised methods (Zhong et al. 2022; Fan et al. 2021b; Wei et al. 2022; You
and Jiang 2019; Wen et al. 2022; Fan, Yang, and Kankanhalli 2022). MeteorNet
(Liu, Yan, and Bohg 2019) gradually learns aggregated features for each point
by constructing point-wise spatiotemporal neighborhoods. 3DV (Wang et al.
2020) encodes motion information utilizing regular voxels and then abstracts
these dynamic voxels into point sets to model spatiotemporal features. PSTNet
(Fan et al. 2021a) hierarchically extracts features of raw point cloud
sequences with spatial and temporal decoupled convolutions. P4Transformer
(Fan, Yang, and Kankanhalli 2021) first utilizes the point tube convolution to
build tokens and further aggregates these tokens by a transformer encoder.
Wang et al. (Wang et al. 2021) proposes an order prediction self-supervised
task on shuffled point cloud clips to learn dynamic point cloud features.
However, this method only mines temporal information, while ignoring
spatiotemporal modeling. Therefore, the spatiotemporal discriminability and
local context awareness of representations learned by this manner are limited.

Unlike the previous methods, we propose a new self-supervised paradigm for
modeling point cloud videos. We combine the advantages of contrastive learning
and generative methods to design contrastive prediction and reconstruction
tasks for dynamic point clouds, jointly facilitating the generalization
performance for downstream tasks.

Figure 2: The framework of our approach. It contains three main components,
i.e., point spatiotemproal encoder, transformer autoregressor, and decoder. By
organically combining these components, we construct contrastive prediction
and reconstruction self-supervised tasks.

## Contrastive Prediction and Reconstruction

We present the overall framework of our method in Figure 2, which includes
three main modules. _Point Spatiotemporal Encoder_ is first utilized to
aggregate dense point cloud segments. Then, we input the aggregated embeddings
into a _Transformer Autoregressor_ to make predictions for the last target
segment. An MLP head is introduced to further project the representations into
latent space for contrast. We perform local and global contrastive learning
between predictions and targets to progressively capture multi-granularity
features. In addition, raw point cloud sequences are reconstructed by a
_Decoder_ based on the predictions. In this section, we first briefly
introduce the encoder and autoregressor, then present contrastive prediction
and reconstruction self-supervised tasks in detail.

### Point Spatiotemporal Encoder

A point cloud sequence is denoted as $\boldsymbol{P}\in\mathbb{R}^{T\times
N\times 3}$, where $T$ is the sequence length and $N$ is point number in each
frame. PSTNet (Fan et al. 2021a) is a typical spatial and temporal decoupled
feature extractor for dynamic point clouds. Specifically, the farthest point
sampling is first utilized to sample $N^{\prime}$ anchor points within each
frame. Then, anchors are mapped into adjacent frames to construct
spatiotemporal point tubes. Spatial convolution is built to extract local
structures within the neighborhoods grouped by _k_ -nearest neighbor
algorithm. 1D convolution is utilized to capture temporal features on these
tubes. These aggregated spatiotemporal features are denoted as
$\\{\boldsymbol{x}_{i}\\}_{i=1}^{m}$, where $m$ is the number of aggregated
super-points. In this paper, we focus more on how to design self-supervised
tasks for learning point spatiotemporal representations. Therefore, we
directly adopt the backbone of PSTNet as our encoder.

### Transformer Autoregressor

The autoregressor is to predict future representations based on spatiotemporal
features extracted by the encoder. In classic contrastive prediction based
methods (Oord, Li, and Vinyals 2018; Han, Xie, and Zisserman 2020a), LSTMs are
always used as autoregressors to predict future states of sequence data.
However, when dealing with long sequences, LSTMs are prone to catastrophic
forgetting, causing inferior predictions. Considering that 3D action
recognition strongly relies on long-term temporal information, we introduce
the powerful transformer as the autoregressor. We adopt the standard
transformer which consists of multi-head self-attentions and feed-forward
networks. We treat the above $\\{\boldsymbol{x}_{i}\\}_{i=1}^{m}$ as tokens.
Positional embeddings $\\{\boldsymbol{e}_{i}\\}_{i=1}^{m}$ are obtained by
projecting the spatiotemporal coordinates ($x,y,z,t$) of super-points with a
linear function. $\mathbf{E}[\mathrm{s}]$ is the class token and
$\boldsymbol{e}_{0}$ is its positional embedding obtained by projecting
$(x_{0},y_{0},z_{0},t_{0})$, where ($x_{0},y_{0},z_{0})$ is mean value of
target segment points and $t_{0}$ is the timestamp of target segment. Finally,
the token sequence $[\mathbf{E}[\mathrm{s}]+\boldsymbol{e}_{0},\
\boldsymbol{x}_{1}+\boldsymbol{e}_{1},\ \ldots,\
\boldsymbol{x}_{m}+\boldsymbol{e}_{m}]$ is sent into the transformer to model
global spatiotemporal relations and make predictions.

### Contrastive Prediction Task

We divide the whole input sequence $\boldsymbol{P}\in\mathbb{R}^{T\times
N\times 3}$ into $S$ segments equally. Each segment is
$\boldsymbol{P}^{\prime}\in\mathbb{R}^{M\times N\times 3}$, and $T=S\times M$.
We utilize the point spatiotemporal encoder to obtain target embeddings
$\boldsymbol{Z}\in\mathbb{R}^{l\times r\times c}$ of the $S$-th target
segment, where $l$ is the frame number after aggregation, $r$ is the super-
point number, and $c$ is embedding channel. The former $S\\!\\!-\\!\\!1$
segments are input into the encoder and autoregressor to predict target
embeddings. Because of temporal adjacency with target segment, the updated
embeddings of the $(S\\!\\!-\\!\\!1)$th segment are taken as predictions,
denoted as $\boldsymbol{Q}\in\mathbb{R}^{l\times r\times c}$. However, due to
the unordered and irregular properties of point clouds, it can not directly
align predictions and target embeddings. To alleviate this dilemma, we
interpolate the predicted features using _k_ -nearest neighbor algorithm by
centering at target super-points, where $k=3$. We denote the interpolated
predictions as $\boldsymbol{\hat{Q}}\in\mathbb{R}^{l\times r\times c}$. The
aligned embeddings are regarded as positive pairs
($\boldsymbol{z}_{i},\boldsymbol{\hat{q}}_{+}$), and the remainings are
negative samples. Moreover, the embeddings belonging to former $S\\!-\\!2$
segments are explored as hard negatives to improve discriminability of the
model. Then, local contrastive learning is conducted to obtain fine-grained
features. We utilize local Info Noise Contrastive Estimation (InfoNCE) loss as
follows:

$\mathcal{L}_{l}\\!=\\!-\\!\sum_{\boldsymbol{z}_{i}\in\boldsymbol{Z}}\\!\log\\!\frac{\exp(\boldsymbol{z}_{i}^{T}\boldsymbol{\hat{q}}_{+}/\tau)}{\exp(\boldsymbol{z}_{i}^{T}\boldsymbol{\hat{q}}_{+}/\tau)\\!+\\!\sum_{\boldsymbol{q}_{j}\in{\Psi}}{\exp{(\boldsymbol{z}_{i}^{T}\boldsymbol{q}_{j}/\tau)}}},$
(1)

where $\Psi$ is a set that contains the negatives, and $\tau$ is temperature
hyper-parameter.

To explore holistic semantics, we obtain the global representations of whole
input sequence, denoted as $\boldsymbol{H}\in\mathbb{R}^{\mathcal{B}\times
c}$, by performing max-pooling on the embeddings of $S$ segments passed
through the encoder. Then, we perform global contrastive learning between the
representation of class token and $\boldsymbol{H}$. The corresponding sample
pairs are positives, otherwise negatives. The global InfoNCE loss is as
follows:

$\mathcal{L}_{g}\\!=\\!-\\!\\!\sum_{\boldsymbol{h}_{i}\in\boldsymbol{H}}\\!\log\\!\frac{\exp(\boldsymbol{h}_{i}^{T}\boldsymbol{\hat{g}}_{+}/\tau)}{\exp(\boldsymbol{h}_{i}^{T}\boldsymbol{\hat{g}}_{+}/\tau)\\!+\\!\sum_{\boldsymbol{g}_{j}\in{\Theta}}{\exp{(\boldsymbol{h}_{i}^{T}\boldsymbol{g}_{j}/\tau)}}},$
(2)

where $\boldsymbol{h}_{i}$ represents the embedding of $i$-th input sequence,
$\boldsymbol{\hat{g}}_{+}$ is the embedding of its class token, and $\Theta$
contains all negatives.

By combining local and global contrastive learning, our method effectively
captures multi-granularity features. Moreover, we explore more meaningful hard
negatives and present their effectiveness in ablation studies.

### Point Cloud Sequences Reconstruction Task

Theoretical analysis in (Wang et al. 2022a) points out that reconstructing raw
inputs maximizes mutual information between representations and inputs to
promote generalizability. Inspired by this, we design sequences reconstruction
using the predictions. Rather than recovering all points of target segment, we
perform spatial downsampling by the farthest point sampling. This avoids
paying more attention to low-level details and reducing the computational
burden. Specifically, we colorize each frame of downsampled target segment
with specific RGB values to distinguish different frames. With the increase of
the temporal index, the corresponding color changes from red to green to blue
(Yang et al. 2021). We denote the colorized target points as
$\boldsymbol{\hat{P}}_{t}^{\prime}\in\mathbb{R}^{M\times N^{\prime}\times 6}$,
where $N^{\prime}$ is the number of downsampled points in each frame and 6
means $(x,y,z,r,g,b)$.

Methods | Input | Accuracy
---|---|---
Vieira et al. (Vieira et al. 2012) | depth map | 78.20 (20 frames)
Kl$\ddot{a}$seret al. (Klaser, Marszałek, and Schmid 2008) | depth map | 81.43 (18 frames)
Actionlet (Wang et al. 2012) | skeleton | 88.21 (all frames)
Frames: | 4 | 8 | 12 | 16 | 24
MeteorNet (Liu, Yan, and Bohg 2019) | point | 78.11 | 81.14 | 86.53 | 88.21 | 88.50
PSTNet (Fan et al. 2021a) | point | 81.14 | 83.50 | 87.88 | 89.90 | 91.20
P4Transformer (Fan, Yang, and Kankanhalli 2021) | point | 80.13 | 83.17 | 87.54 | 89.56 | 90.94
PSTNet++ (Fan et al. 2021b) | point | 81.53 | 83.50 | 88.15 | 90.24 | 92.68
PST-Transformer (Fan, Yang, and Kankanhalli 2022) | point | 81.14 | 83.97 | 88.15 | 91.98 | 93.73
PST2 (Wei et al. 2022) | point | 81.14 | 86.53 | 88.55 | 89.22 | -
Kinet (Zhong et al. 2022) | point | 79.80 | 83.84 | 88.53 | 91.92 | 93.27
PPTr (Wen et al. 2022) | point | 80.97 | 84.02 | 89.89 | 90.31 | 92.33
4D MinkNet + ROP Pretraining (Wang et al. 2021) | point | - | 86.31 | - | - | -
MeteorNet + ROP Pretraining (Wang et al. 2021) | point | - | 85.40 | - | - | -
CPR (Ours) | point | 82.50 | 86.53 | 91.00 | 92.15 | 93.03
Table 1: Action recognition accuracy (%) on the MSRAction3D dataset.

We utilize average pooling on predictions $\boldsymbol{Q}$ to obtain global
semantics $\boldsymbol{q}_{g}\in\mathbb{R}^{c}$, and then duplicates it to
$\boldsymbol{Q}_{g}\in\mathbb{R}^{M\times N^{\prime}\times c}$. Moreover, we
add 1D cosine positional encodings to $\boldsymbol{Q}_{g}$ to provide temporal
clues, and the points in each frame share the same cosine encoding. Finally,
we put these updated embeddings into the decoder for spatiotemporal
reconstruction. We adopt FoldingNet (Yang et al. 2018) as the decoder and
exploit chamfer distance loss to optimize this self-supervised task as
follows:

$d(\boldsymbol{R},\boldsymbol{\hat{P}}_{t}^{\prime})\\!=\\!\frac{1}{|\boldsymbol{R}|}\sum_{\hat{x}\in\boldsymbol{R}}\mathop{\min}_{p\in\boldsymbol{\hat{P}}_{t}^{\prime}}\|\hat{x}-p\|_{2}^{2}\\!+\\!\frac{1}{|\boldsymbol{\hat{P}}_{t}^{\prime}|}\sum_{p\in\boldsymbol{\hat{P}}_{t}^{\prime}}\mathop{\min}_{\hat{x}\in\boldsymbol{R}}\|p-\hat{x}\|_{2}^{2},$
(3)

where $\boldsymbol{R}\in\mathbb{R}^{M\times N^{\prime}\times 6}$ is
reconstructed sequence.

Overall, the total loss with a regularized parameter $\lambda$ of our self-
supervised framework consists of three parts:

$\mathcal{L}_{total}=\mathcal{L}_{l}+\mathcal{L}_{g}+\lambda
d(\boldsymbol{R},\boldsymbol{\hat{P}}_{t}^{\prime}).$ (4)

## Experiments

### Datasets

We perform 3D action recognition on MSRAction3D and NTU-RGBD 60, and gesture
recognition on NvGesture and SHREC’17 datasets.

MSRAction3D (Li, Zhang, and Liu 2010) includes 567 videos collected by Kinect,
with a total of 23K frames and 20 categories performed by 10 subjects. We use
the same training and test splits as (Liu, Yan, and Bohg 2019).

NTU-RGBD 60 (Shahroudy et al. 2016) is collected by three cameras with
different angles, containing 60 categories and 56880 videos performed by 40
subjects. Cross-subject and cross-view evaluations are adopted.

NvGesture (Molchanov et al. 2016) contains 1532 gesture videos focusing on
touchless driver controlling, with a total of 25 classes. We follow the
previous work to split this dataset, where 1050 videos are used for training
and 482 videos are for test (Min et al. 2020).

SHREC’17 (De Smedt et al. 2017) collects 2800 videos performed by 28 subjects
in two ways, i.e., using one finger or the whole hand. These short videos
generally contain dozens of frames and involve both coarse and fine gestures.
We adopt the same splits of training and test data as previous work (Min et
al. 2020).

### Experimental Setting

We perform pretraining on NTU-RGBD 60. Specifically, we consecutively sample
24 frames with stride 2, and each frame contains 1024 points. This sequence is
then divided into 6 segments. Following (Fan et al. 2021a), the spatial search
radius is 0.1, neighbors for the ball query is 9, and random scaling is
adopted as data augmentation for our encoder. Three transformer layers are
utilized as our autoregressor. We pretrain 200 epochs and set the batchsize to
88. We use Adam optimizer and cosine annealing scheduler with the initial
learning rate 0.0008.

Without special instructions, we adopt the pretrained point spatiotemporal
encoder for downstream tasks, and add two linear layers with BN and ReLu for
finetuning or one linear layer for linear evaluation. We utilize SGD optimizer
with momentum 0.9 and cosine scheduler with warmup 10 epochs. The 16 batchsize
corresponds to the 0.01 learning rate, and we follow the scale up rule.

### Comparison with State-of-the-art

We perform extensive experiments and compare the performances with state-of-
the-art supervised methods and other self-supervised methods.

Transfer to MSRAction3D. The finetune results are presented in Table 1,
compared with skeleton-based, depth-based, and point-based methods. We follow
the settings of previous work (Liu, Yan, and Bohg 2019) and test the
performance under variable lengths. Although the skeleton-based method
Actionlet simultaneously feeds all frames, it is still exceeded by our CPR
with 12 frames. This shows that point cloud sequences contain richer
spatiotemporal information. Moreover, since 3D action recognition relies on
temporal information, CPR obtains higher accuracy when inputting longer
sequences. Compared with other point-based supervised methods which train from
scratch, CPR achieves improvements under diverse inputs, except that it is
slightly lower than PST-Transformer on 24 frames. This indicates that high-
level semantics can be obtained by pretrained encoder and as a good
initialization for transfer learning. Compared with the self-supervised method
that performs Recurrent Order Prediction task (ROP) (Choy, Gwak, and Savarese
2019), CPR achieves higher accuracy. ROP utilizes RGB values for pretraining
and only focuses on clip-level temporal information, while our method utilizes
local/global contrastive learning and reconstruction to explore richer
spatiotemporal information just with points.

Methods | Input | NvG | S17
---|---|---|---
Human(Molchanov et al. 2016) | RGB | 88.4 | -
DG-STA (Chen et al. 2019) | skeleton | - | 90.7
FlickerNet (Min et al. 2019) | point | 86.3 | -
PLSTM-base (Min et al. 2020) | point | 85.9 | 87.6
PLSTM-early (Min et al. 2020) | point | 87.9 | 93.5
PLSTM-PSS (Min et al. 2020) | point | 87.3 | 93.1
PLSTM-middle(Min et al. 2020) | point | 86.9 | 94.7
PLSTM-late (Min et al. 2020) | point | 87.5 | 93.5
Kinet(Zhong et al. 2022) | point | 89.1 | 95.2
PSTNet(Fan et al. 2021a) | point | 88.2 | 92.0
CPR (Ours) | point | 88.7 | 93.1
Table 2: Gesture recognition accuracy (%) on NvGesture (NvG) and SHREC’17
(S17).

Transfer to NvGesture and SHREC’17. We perform self-supervised pretraining on
human action datasets. To further verify the generalization of our method, we
also transfer the pretrained encoder to two gesture datasets. During
finetuning, the spatial search radius is 0.1, the neighbors of ball query is
9, and the frame number is 32. CPR is compared with skeleton-based, RGB-based
and other point-based supervised methods. The results are presented in Table
2. By finetuning with our self-supervised encoder, CPR facilitates the
baseline PSTNet to produce comparable results. This shows that the pretrained
encoder has strong generalization ability, and exhibits powerful
spatiotemporal modeling ability under diverse domains.

Finetune on NTU-RGBD 60. We keep the encoder and autoregressor for finetuning.
As shown in Table 3, we compare CPR with skeleton-based (Liu et al. 2017; Si
et al. 2019; Li et al. 2019; Zhang et al. 2019; Shi et al. 2019b, a), depth-
based (Xiao et al. 2019), and point-based supervised methods. Particularly,
Kinet builds a hierarchical motion branch, and 3DV explicitly encodes voxel-
based motion cues. Instead, without introducing hand-crafted motion features
and complicated two-stream design, CPR achieves competitive accuracy under two
evaluations. This demonstrates the superiority of our self-supervised
approach. By exploiting pretraining to fully mine motion information in the
raw data, it can help the network acquire rich semantics without using
additional dynamical model. Our performance under cross-view evaluation is
consistent with PSTNet++, and we will explore more advanced encoders in the
future.

Methods | Input | NTU-RGBD 60
---|---|---
Subject | View
GCA-LSTM | skeleton | 74.4 | 82.8
AGC-LSTM | skeleton | 89.2 | 95.0
AS-GCN | skeleton | 86.8 | 94.2
VA-fusion | skeleton | 89.4 | 95.0
2s-AGCN | skeleton | 88.5 | 95.1
DGNN | skeleton | 89.9 | 96.1
MVDI | depth | 84.6 | 87.3
PointNet++ | point | 80.1 | 85.1
3DV (motion) | voxel | 84.5 | 95.4
3DV-PointNet++ | voxel + point | 88.8 | 96.3
PSTNet | point | 90.5 | 96.5
P4Transformer | point | 90.2 | 96.4
PSTNet++ | point | 91.4 | 96.7
PST-Transformer | point | 91.0 | 96.4
Kinet | point | 92.3 | 96.4
CPR (Ours) | point | 91.0 | 96.7
Table 3: Action recognition accuracy (%) on the NTU-RGBD 60 dataset. SSL-Data | Linear | Semi-supervision
---|---|---
Data | Accuracy | Data | Accuracy
MSR | MSR | 85.7 | 30% MSR | 86.4
NTU | NTU | 70.0 | 30% NTU | 83.3
NTU | MSR | 79.1 | 30% MSR | 87.2
Table 4: Recognition accuracy (%) of linear evaluation and semi-supervised
learning on MSRAction3D (MSR) and NTU-RGBD 60 (NTU). SSL-Data is self-
supervised data.

Linear and Semi-Supervised Learning. To test whether the pretrained encoder
has learned high-level semantics, we evaluate it on MSRAction3D and NTU-RGBD
60 under linear evaluation and limited training data. Semi-supervised training
data consists of randomly selected 30% of the training samples from each
class. We conduct self-supervised pretraining on MSRAction3D and NTU-RGBD 60,
respectively. The results are shown in Table 4. It is observed that the linear
accuracy of _MSR to MSR_ , _NTU to NTU_ , and _NTU to MSR_ is 85.7%, 70.0%,
and 79.1%. This shows that our self-supervised pretraining captures rich
semantics beneficial for 3D action recognition. After semi-supervision with
30% data, all the performances get promoted compared with linear evaluations.
Since the 30% MSRAction3D data is too small, the improvement of finetuning on
MSR is limited. In addition, the semi-supervised performance of _NTU to MSR_
is higher than that of _MSR to MSR_. This indicates that self-supervised
pretraining with larger dataset can learn more general semantics while still
needs limited data for transfer adaptation.

### Ablation Studies

In this section, we show extensive ablation studies to explore that the length
of whole input sequence utilized for our self-supervised framework, the design
of point cloud sequences reconstruction, the effectiveness of hard negatives,
and the influences of diverse self-supervised tasks. These experiments are all
pretrained on MSRAction3D and then finetuned on it.

SS-f Fin-f | 4 | 8 | 12 | 16 | 24
---|---|---|---|---|---
12 | 78.45 | 82.83 | 88.82 | 90.91 | 91.99
16 | 79.12 | 83.84 | 88.55 | 91.03 | 92.33
20 | 80.13 | 85.70 | 89.56 | 91.92 | 93.03
24 | 82.06 | 86.20 | 90.42 | 92.07 | 93.38
Table 5: Finetune accuracy (%) at different frames. Fin-f and SS-f are the frame number for finetune and self-supervised pretraining. Reconstruction Target | Points of each frame | Acc.
---|---|---
One frame | 1024 | 91.32
One segment | 256 | 92.68
One segment + Color. | 256 | 93.38
One segment + Color. | 512 | 91.99
One segment + Color. | 1024 | 92.36
Table 6: Finetune accuracy (%) with different reconstruction targets. Color. means colorization. Method | Acc.
---|---
Local contrastive prediction | 91.34
Global contrastive prediction | 90.89
Sequence reconstruction | 87.10
Local contrastive prediction with hard negatives | 92.23
Local and global contrastive prediction | 92.55
Contrastive prediction and reconstruction | 93.38
Table 7: Ablation studies on architecture design.

How long of the point cloud sequence? We perform self-supervised pretraining
with variable input lengths and then finetune with diverse frames. The results
are shown in Table 5. Notably, when sequence length of self-supervised
pretraining is longer, it is more beneficial for finetuning under multiple
frames. We finally choose 24 frames for self-supervised pretraining to cover
as much dynamic information as possible.

How to design reconstruction tasks? For reconstructing branch, we try various
experimental settings and the results are presented in Table 6. The accuracy
of reconstructing one segment is higher than that of reconstructing one frame
under the same number of points. This may be due to the fact that
spatiotemporal prediction is more challenging for self-supervised tasks. When
applying point cloud sequence colorization, we achieve higher accuracy.
Colorization is beneficial to distinguish diverse frames by assigning
different timestamps for the target segment. We also try to reconstruct more
points in each frame. However, this does not lead to improvements. It is
possible that excessive raw points provide more low-level details and even
noise, which does not help to promote generalization and discrimination.
Finally, we choose to reconstruct one colorized segment with 256 points in
each frame.

Why utilize hard negative samples? The tokens of the former $S\\!\\!-\\!\\!2$
segments are temporally adjacent to the predictions and targets, and therefore
they contain similar spatiotemporal semantics. These tokens are mined as hard
negatives to enhance local perception and discriminability. The results in
Table 7 show that the accuracy of local contrastive prediction without hard
negatives is 91.34%, and the accuracy increases to 92.23% after adding them.
This indicates that mining hard negative samples is crucial for the design of
self-supervised tasks.

How the effects of each task? We respectively evaluate the effectiveness of
local contrastive prediction, global contrastive prediction, and point cloud
sequence reconstruction tasks. The results are shown in Table 7. The hard
negatives can increase the performance of local contrastive prediction by
about 1%. Clearly, by introducing the reconstruction task, the finetune
performance has increased by 0.83%, which indicates reconstructing raw inputs
helps to learn semantics information suitable for 3D action recognition.
Compared with only utilizing local contrastive prediction, the introduction of
global contrastive prediction increases the accuracy by 0.32%. This shows that
global discrimination and local perception are all essential for capturing
multi-granularity representations. Overall, the most beneficial paradigm is to
combine contrastive prediction and point cloud sequence reconstruction.

Figure 3: Visualization of the t-SNE features. We visualize the feature
distributions of our self-supervised encoder (a) after pretraining, (b) after
finetuning on NTU-RGBD 60, and (c) after finetuning on MSRAction3D.

### Visualization

In Figure 3, we visualize the feature distributions of t-SNE after pretraining
(a) and after finetuning (b)(c). It can be seen that after pretraining, there
are approximate boundaries between 60 categories. This illustrates that self-
supervised pretraining can learn certain high-level semantics. From Figure
3(c), it is observed that each cluster has clear outlines, indicating that the
representations learned by our self-supervised framework have strong
generalization and transferability. Moreover, the proposed framework can guide
the encoder to obtain domain-agnostic and general knowledge.

## Conclusion

In this work, we propose a unified contrastive prediction and reconstruction
self-supervised framework for dynamic point cloud understanding. By
integrating discriminative and generative self-supervised tasks, it makes the
learned representations with both powerful instance discrimination and local
perception. Extensive experiments under linear evaluation, semi-supervised
learning, and transfer learning are performed on multiple benchmarks to
demonstrate the superiority of our self-supervised framework. In the future,
we will explore more dynamic point cloud related downstream tasks.

## Acknowledgments

This paper is sponsored by National Natural Science Foundation of China
(No.61673270; No.61973212).

## References

  * Bao et al. (2021) Bao, H.; Dong, L.; Piao, S.; and Wei, F. 2021.  Beit: bert pre-training of image transformers.  In _International Conference on Learning Representations (ICLR)_.
  * Caron et al. (2020) Caron, M.; Misra, I.; Mairal, J.; Goyal, P.; Bojanowski, P.; and Joulin, A. 2020\.  Unsupervised learning of visual features by contrasting cluster assignments.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 9912–9924.
  * Chen et al. (2020) Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. 2020.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning (ICML)_ , 1597–1607.
  * Chen and He (2021) Chen, X.; and He, K. 2021.  Exploring simple siamese representation learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 15750–15758.
  * Chen et al. (2019) Chen, Y.; Zhao, L.; Peng, X.; Yuan, J.; and Metaxas, D. N. 2019.  Construct dynamic graphs for hand gesture recognition via spatial-temporal attention.  _arXiv preprint arXiv:1907.08871_.
  * Choy, Gwak, and Savarese (2019) Choy, C.; Gwak, J.; and Savarese, S. 2019.  4D spatio-temporal convnets: minkowski convolutional neural networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 3075–3084.
  * De Smedt et al. (2017) De Smedt, Q.; Wannous, H.; Vandeborre, J.-P.; Guerry, J.; Le Saux, B.; and Filliat, D. 2017.  Shrec’17 track: 3d hand gesture recognition using a depth and skeletal dataset.  In _3DOR-10th Eurographics Workshop on 3D Object Retrieval_ , 1–6.
  * Fan, Yang, and Kankanhalli (2021) Fan, H.; Yang, Y.; and Kankanhalli, M. 2021.  Point 4d transformer networks for spatio-temporal modeling in point cloud videos.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 14204–14213.
  * Fan, Yang, and Kankanhalli (2022) Fan, H.; Yang, Y.; and Kankanhalli, M. 2022.  Point spatio-temporal transformer networks for point cloud video modeling.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
  * Fan et al. (2021a) Fan, H.; Yu, X.; Ding, Y.; Yang, Y.; and Kankanhalli, M. 2021a.  Pstnet: point spatio-temporal convolution on point cloud sequences.  In _Proceedings of the International Conference on Learning Representations (ICLR)_.
  * Fan et al. (2021b) Fan, H.; Yu, X.; Yang, Y.; and Kankanhalli, M. 2021b.  Deep hierarchical representation of point cloud videos via spatio-temporal decomposition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
  * Han, Xie, and Zisserman (2020a) Han, T.; Xie, W.; and Zisserman, A. 2020a.  Memory-augmented dense predictive coding for video representation learning.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 312–329.
  * Han, Xie, and Zisserman (2020b) Han, T.; Xie, W.; and Zisserman, A. 2020b.  Self-supervised co-training for video representation learning.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 5679–5690.
  * He et al. (2022) He, K.; Chen, X.; Xie, S.; Li, Y.; Dollár, P.; and Girshick, R. 2022.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 16000–16009.
  * He et al. (2020) He, K.; Wu, Y.; Xie, S.; and Girshick, R. 2020.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 9729–9738.
  * Klaser, Marszałek, and Schmid (2008) Klaser, A.; Marszałek, M.; and Schmid, C. 2008.  A spatio-temporal descriptor based on 3d-gradients.  In _2008-19th British Machine Vision Conference (BMVC)_ , 995–1004.
  * Li et al. (2019) Li, M.; Chen, S.; Chen, X.; Zhang, Y.; Wang, Y.; and Tian, Q. 2019.  Actional-structural graph convolutional networks for skeleton-based action recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , 3595–3603.
  * Li, Zhang, and Liu (2010) Li, W.; Zhang, Z.; and Liu, Z. 2010.  Action recognition based on a bag of 3d points.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 9–14.
  * Liu, Cai, and Lee (2022) Liu, H.; Cai, M.; and Lee, Y. J. 2022.  Masked discrimination for self-supervised learning on point clouds.  _arXiv preprint arXiv:2203.11183_.
  * Liu et al. (2017) Liu, J.; Wang, G.; Hu, P.; Duan, L.-Y.; and Kot, A. C. 2017.  Global context-aware attention lstm networks for 3d action recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , 1647–1656.
  * Liu, Yan, and Bohg (2019) Liu, X.; Yan, M.; and Bohg, J. 2019.  Meteornet: deep learning on dynamic 3D point cloud sequences.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 9246–9255.
  * Min et al. (2019) Min, Y.; Chai, X.; Zhao, L.; and Chen, X. 2019.  FlickerNet: adaptive 3d gesture recognition from sparse point clouds.  In _2019-30th British Machine Vision Conference (BMVC)_ , 5.
  * Min et al. (2020) Min, Y.; Zhang, Y.; Chai, X.; and Chen, X. 2020.  An efficient pointlstm for point clouds based gesture recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (ICCV)_ , 5761–5770.
  * Molchanov et al. (2016) Molchanov, P.; Yang, X.; Gupta, S.; Kim, K.; Tyree, S.; and Kautz, J. 2016.  Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 4207–4215.
  * Oord, Li, and Vinyals (2018) Oord, A. v. d.; Li, Y.; and Vinyals, O. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_.
  * Pan et al. (2021) Pan, T.; Song, Y.; Yang, T.; Jiang, W.; and Liu, W. 2021.  Videomoco: contrastive video representation learning with temporally adversarial examples.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 11205–11214.
  * Pang et al. (2022) Pang, Y.; Wang, W.; Tay, F. E.; Liu, W.; Tian, Y.; and Yuan, L. 2022.  Masked autoencoders for point cloud self-supervised learning.  _arXiv preprint arXiv:2203.06604_.
  * Qian et al. (2022) Qian, R.; Ding, S.; Liu, X.; and Lin, D. 2022.  Static and dynamic concepts for self-supervised video representation learning.  _arXiv preprint arXiv:2207.12795_.
  * Qian et al. (2021) Qian, R.; Meng, T.; Gong, B.; Yang, M.-H.; Wang, H.; Belongie, S.; and Cui, Y. 2021\.  Spatiotemporal contrastive video representation learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 6964–6974.
  * Shahroudy et al. (2016) Shahroudy, A.; Liu, J.; Ng, T.-T.; and Wang, G. 2016.  Ntu rgb+d: a large scale dataset for 3D human activity analysis.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 1010–1019.
  * Shi et al. (2019a) Shi, L.; Zhang, Y.; Cheng, J.; and Lu, H. 2019a.  Skeleton-based action recognition with directed graph neural networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 7912–7921.
  * Shi et al. (2019b) Shi, L.; Zhang, Y.; Cheng, J.; and Lu, H. 2019b.  Two-stream adaptive graph convolutional networks for skeleton-based action recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , 12026–12035.
  * Si et al. (2019) Si, C.; Chen, W.; Wang, W.; Wang, L.; and Tan, T. 2019.  An attention enhanced graph convolutional lstm network for skeleton-based action recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , 1227–1236.
  * Tong et al. (2022) Tong, Z.; Song, Y.; Wang, J.; and Wang, L. 2022.  Videomae: masked autoencoders are data-efficient learners for self-supervised video pre-training.  _arXiv preprint arXiv:2203.12602_.
  * Vieira et al. (2012) Vieira, A. W.; Nascimento, E. R.; Oliveira, G. L.; Liu, Z.; and Campos, M. F. 2012\.  Stop: space-time occupancy patterns for 3d action recognition from depth map sequences.  In _Iberoamerican congress on pattern recognition_ , 252–259.
  * Wang et al. (2022a) Wang, H.; Guo, X.; Deng, Z.-H.; and Lu, Y. 2022a.  Rethinking minimal sufficient representation in contrastive learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 16041–16050.
  * Wang et al. (2021) Wang, H.; Yang, L.; Rong, X.; Feng, J.; and Tian, Y. 2021.  Self-supervised 4d spatio-temporal feature learning via order prediction of sequential point cloud clips.  In _Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 3762–3771.
  * Wang et al. (2012) Wang, J.; Liu, Z.; Wu, Y.; and Yuan, J. 2012.  Mining actionlet ensemble for action recognition with depth cameras.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 1290–1297.
  * Wang et al. (2022b) Wang, R.; Chen, D.; Wu, Z.; Chen, Y.; Dai, X.; Liu, M.; Jiang, Y.-G.; Zhou, L.; and Yuan, L. 2022b.  Bevt: bert pretraining of video transformers.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 14733–14743.
  * Wang et al. (2020) Wang, Y.; Xiao, Y.; Xiong, F.; Jiang, W.; Cao, Z.; Zhou, J. T.; and Yuan, J. 2020\.  3DV: 3d dynamic voxel for action recognition in depth video.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 511–520.
  * Wei et al. (2022) Wei, Y.; Liu, H.; Xie, T.; Ke, Q.; and Guo, Y. 2022.  Spatial-temporal transformer for 3d point cloud sequences.  In _Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 1171–1180.
  * Wen et al. (2022) Wen, H.; Liu, Y.; Huang, J.; Duan, B.; and Yi, L. 2022.  Point primitive transformer for long-term 4D point cloud video understanding.  _arXiv preprint arXiv:2208.00281_.
  * Xiao et al. (2019) Xiao, Y.; Chen, J.; Wang, Y.; Cao, Z.; Zhou, J. T.; and Bai, X. 2019.  Action recognition for depth video using multi-view dynamic images.  _Information Sciences_ , 480: 287–304.
  * Xie et al. (2020) Xie, S.; Gu, J.; Guo, D.; Qi, C. R.; Guibas, L.; and Litany, O. 2020.  Pointcontrast: unsupervised pre-training for 3d point cloud understanding.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 574–591.
  * Xie et al. (2022) Xie, Z.; Zhang, Z.; Cao, Y.; Lin, Y.; Bao, J.; Yao, Z.; Dai, Q.; and Hu, H. 2022\.  Simmim: a simple framework for masked image modeling.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 9653–9663.
  * Yang et al. (2021) Yang, S.; Liu, J.; Lu, S.; Er, M. H.; and Kot, A. C. 2021.  Skeleton cloud colorization for unsupervised 3d action representation learning.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 13423–13433.
  * Yang et al. (2018) Yang, Y.; Feng, C.; Shen, Y.; and Tian, D. 2018.  Foldingnet: point cloud auto-encoder via deep grid deformation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , 206–215.
  * You and Jiang (2019) You, Q.; and Jiang, H. 2019.  Action4d: online action recognition in the crowd and clutter.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 11857–11866.
  * Yu et al. (2022) Yu, X.; Tang, L.; Rao, Y.; Huang, T.; Zhou, J.; and Lu, J. 2022.  Point-bert: pre-training 3d point cloud transformers with masked point modeling.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 19313–19322.
  * Zbontar et al. (2021) Zbontar, J.; Jing, L.; Misra, I.; LeCun, Y.; and Deny, S. 2021.  Barlow twins: self-supervised learning via redundancy reduction.  In _International Conference on Machine Learning (ICML)_ , 12310–12320.
  * Zhang et al. (2019) Zhang, P.; Lan, C.; Xing, J.; Zeng, W.; Xue, J.; and Zheng, N. 2019.  View adaptive neural networks for high performance skeleton-based human action recognition.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(8): 1963–1978.
  * Zhong et al. (2022) Zhong, J.-X.; Zhou, K.; Hu, Q.; Wang, B.; Trigoni, N.; and Markham, A. 2022.  No pain, big gain: classify dynamic point cloud sequences with static models by fitting feature-level space-time surfaces.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 8510–8520.