# A Neuro-Symbolic Approach to Monitoring Salt Content in Food

###### Abstract

We propose a dialogue system that enables heart failure patients to inquire
about salt content in foods and help them monitor and reduce salt intake.
Addressing the lack of specific datasets for food-based salt content
inquiries, we develop a template-based conversational dataset. The dataset is
structured to ask clarification questions to identify food items and their
salt content. Our findings indicate that while fine-tuning transformer-based
models on the dataset yields limited performance, the integration of Neuro-
Symbolic Rules significantly enhances the system’s performance. Our
experiments show that by integrating neuro-symbolic rules, our system achieves
an improvement in joint goal accuracy of over 20% across different data sizes
compared to naively fine-tuning transformer-based models.

Keywords: Dialogue Systems, Neuro-Symbolic AI, Heart Failure

A Neuro-Symbolic Approach to Monitoring Salt Content in Food

Anuja Tayal1, Barbara Di Eugenio1, Devika Salunke2, Andrew D. Boyd2,
---
Carolyn A Dickens3, Eulalia P Abril4, Olga Garcia-Bedoya3, Paula G Allen-
Meares3
1 Department of Computer Science 2 Department of Biomedical and Health
Information Sciences
3 Department of Medicine 4 Department of Communications
University of Illinois Chicago, IL, USA
{atayal4, bdieugen, dsalun2, boyda, cdickens,eulalia, ogarciab, pameares
<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

The excessive consumption of salt poses significant public health risks,
contributing to diseases such as high blood pressure and heart failure He et
al. (2020). Reducing salt intake has been shown to mitigate these health
issues. In 2017, excessive sodium intake was associated with around three
million deaths and a significant loss of healthy life years Roth et al.
(2018). Research, including clinical trials and population studies, supports
the reduction of salt intake as a means to manage and prevent these
conditions. Despite the clear benefits of sodium reduction, public
understanding and action are lacking; only 58% of individuals can accurately
read sodium content on nutrition labels, and merely 44% can classify food
products as high or low in sodium based on standard labeling Dickson and
Riegel (2009). This gap in knowledge and practice underscores the challenge of
addressing dietary sodium intake, with only a handful of countries
implementing effective public health interventions.

Therefore, we aim to develop a dialogue system that enables patients to
inquire about the salt content in various foods. This system especially aims
to support heart failure patients, who must meticulously monitor and reduce
their salt intake. More specifically, African American individuals who are
more prone to heart failure Nayak et al. (2020), have a higher sensitivity to
salt and face challenges like food deserts and higher consumption of junk
foods. This necessitates a specialized dietary management approach to help
them monitor and reduce their salt intake effectively. Furthermore, in Gupta
et al. (2020), the authors show that African American patients with heart
failure often focus on discussions related to salt and food during heart
failure educational sessions, indicating a significant interest and need for
information in this area. By providing a tool that facilitates easy access to
information about salt content in foods, we aim to empower patients to make
healthier dietary choices, thereby addressing a critical aspect of managing
heart failure. Having said that, the dialog system can be used by anyone who
wants to inquire about the salt content in food.

Figure 1: Sample Template Conversation which begins with the user asking about
the salt content in food. The system asks clarification questions to determine
the precise food item and its salt value.

Creating a dialog agent specialized in food-related conversations and nutrient
information is challenging. This is primarily due to the lack of a
conversational dataset specifically designed for this domain. Moreover,
annotating the dataset is very costly and resource-intensive. To address this
issue, we create a template-based conversational dataset (an example is shown
in Figure 1) to identify various food items and their salt content. Our
approach involves utilizing the USFDC (U.S. Food Data Central) USFDC (2022)
dataset, which provides detailed food descriptions along with their nutrient
values. To enhance the system’s ability to recognize the different components
detailed in the food descriptions, we developed a food ontology. This ontology
is constructed using the FoodOn Dooley et al. (2018) framework and GloVe
Pennington et al. (2014) embeddings, facilitating the identification of
various attributes related to the food’s cooking and preparation methods.
Leveraging this ontology, we create a structured food conversational dataset
(Figure 1). As it is a template-based dataset, annotating it is easy and
effective. We model the dataset after the state-of-the-art task-oriented
dialog dataset MultiWOZ Budzianowski et al. (2018).

To train the dialog system, we use the end-to-end dialog system PPTOD (Plug-
and-Play Task-Oriented Dialogue System) Model Su et al. (2022). PPTOD extends
the T5 framework, especially designed for task-oriented dialogue (TOD)
applications. To accurately provide salt content information, our dialog
system will address vague user queries, (such as "What is the salt content in
pork chops?"), by asking clarification questions. This approach ensures the
model understands the specific preparation and consumption context of the food
item, allowing us to determine the precise salt content based on how the food
is prepared (For ex - beef can be consumed raw, cooked, or as part of a dish
like a burger). The dataset along with the trained model is publicly
available.111https://github.com/anujatayal/NS-Monitoring-Salt-Content-in-Food

Despite the remarkable proficiency of large pre-trained language models (PLMs)
like GPT-3 and T5 Brown et al. (2020); Raffel et al. (2020) in complex
arithmetic reasoning tasks, they occasionally make calculation errors,
especially as the mathematical operations in equations become more complex Wei
et al. (2022). Our observations align with these findings, notably that even
after fine-tuning, the PPTOD model struggled to compute the salt values for
various food items. Moreover, the USFDC provides salt content for standard
food measurements, and users may not frame their queries in these standard
terms. To rectify this, we propose NS-PPTOD, where we integrate PPTOD model
with neuro-symbolic rules. These rules are designed to harness the strengths
of PLMs while compensating for their weaknesses, enabling the system to
retrieve and accurately calculate the salt content from the database. This
ensures the system’s adaptability in responding to queries about salt content
in both standard and non-standard food quantities, thereby offering accurate
salt content information and enhancing the system’s overall accessibility and
effectiveness.

Our experiments show that just fine-tuning a transformer model to predict salt
content isn’t enough. The integration of neuro-symbolic rules significantly
enhances the system’s performance, evidenced by a 20% improvement in joint
goal accuracy across different dataset sizes. This proves that combining pre-
trained language models with neuro-symbolic rules is essential for better
accuracy.

In summary, our contributions are

  * •

We propose to develop a food conversation dataset that includes clarifying
questions to infer the correct food item and its salt content.

  * •

We finetuned PPTOD on our food conversation dataset using a few-shot approach.

  * •

We propose NS-PPTOD which integrates PPTOD with Neuro-Symbolic rules to infer
correct salt values across different food weights.

  * •

We show a 20% increase in joint accuracy compared to the finetuned PPTOD.

## 2\. Related Work

  * •

HealthCare Dialog Systems Task-oriented dialogue systems have seen a
significant rise in the healthcare sector, where they play a vital role in
enhancing various aspects of healthcare. These systems are developed for a
wide array of diseases including heart failure Moulik (2019); Gupta et al.
(2020), cancer Belfin et al. (2019), mental disorders Ali et al. (2020),
public anxiety Wang et al. (2020) etc. Their applications extend to several
areas, including disease diagnosis Wei et al. (2018), patient education Cai et
al. (2023); Gupta et al. (2020), and health coaching Zhou et al. (2022) among
others. A comprehensive survey of NLP literature conducted in Valizadeh and
Parde (2022) provides an in-depth analysis of these diverse healthcare-
oriented dialogue systems, examining them from a computational perspective and
highlighting their varied end-users.

Gupta et al. (2020); Salunke et al. (2023) discuss the development of a dialog
agent for self-care needs of heart failure patients, drawing upon insights
from educational sessions. The work in Kearns et al. (2020) explores the
Wizard of Oz (WOZ) technique to craft a persona-based health counseling dialog
dataset. Additionally, recent advancements have seen the application of Large
Language Models (LLMs) in responding to patient inquiries Chowdhury et al.
(2023), though the importance of safety is emphasized. Addressing the
limitations in the medical knowledge of LLMs, the study in Li et al. (2023)
undertakes the task of enhancing and fine-tuning the LLaMa model with a
dataset of approximately 100,000 patient-doctor dialogues.

  * •

Pretrained Language Models (PLMs) With the advancement in pre-trained language
models (PLMs), different systems based on PLMs have been proposed including
dialog systems Lei et al. (2018); Peng et al. (2021). PLMs excel in various
tasks, approaching human-like performance. Yet, they struggle in mathematical
reasoning, as noted in Wei et al. (2022). Qian et al. (2023) shows the
limitations of LLMs with complex or lengthy numerical operations. For
instance, GPT-3 Brown et al. (2020) performs well in simple two-digit
additions but falters with longer numbers. Similarly, even a fine-tuned T5
model struggles with the accurate addition or subtraction of lengthy numbers
Nogueira et al. (2021), and the challenge escalates with numbers not covered
in their training data.

  * •

NeuroSymbolic AI Integrating neuro-symbolic approaches offers a solution by
combining the inference capabilities of symbolic systems with the robustness
of neural networks, creating a composite AI framework adept at reasoning,
learning, and cognitive modeling Garcez and Lamb (2023). This blend addresses
the inherent weaknesses of each system, promising enhanced performance and
robustness.

To address the generalization issues in neural networks, particularly in task-
oriented dialogue systems, various neuro-symbolic methodologies have been
investigated. Mehri and Eskenazi (2021) proposes schema graphs to generalize
across various unseen domains and tasks. In Romero et al. (2021), the authors
fine-tuned GPT-2 to generate the text and symbolic representations. DILOG Zhou
et al. (2020) employed inductive reasoning to formulate logical rules,
enabling dialog policy training with minimal data to facilitate zero-shot
domain transfer. Arabshahi et al. (2021) used a neuro-symbolic approach to
extract multi-hop reasoning and integrate commonsense in a dialog system.
These strategies underscore the potential of neuro-symbolic integration to
significantly improve the adaptability and efficacy of language models in
complex and dynamic tasks.

  * •

Representing Food in Dialog Systems Addressing the intricacies of food
representation, FoodKG Haussmann et al. (2019); Chen et al. (2021) explored
knowledge graphs to represent food. FoodKG Haussmann et al. (2019) integrates
information from diverse recipe collections and the US Food Data Central USFDC
(2022), primarily focusing on template-based queries related to ingredients
and recipes. However, this framework has limitations, notably in identifying
only the primary item in food descriptions and missing key details like the
type of food, cooking methods, or quantity, which are included in our
methodology. Fu et al. (2022) explored the role of recommending food to
improve mental health. RecipeQA Yagcioglu et al. (2018) explored multimodal
question answering within the context of recipes, while CookDial Jiang et al.
(2022) provides a platform for users to navigate and query cooking recipes
more effectively.

## 3\. Dataset Creation

Given the absence of a specialized dataset for conversational inquiries about
salt content and the challenges in dataset collection and annotation, in this
section, we show in detail how we created the dataset. Dataset creation
involves developing a template-based conversational framework to accurately
identify food items and their salt content. First, we used the USFDC dataset
USFDC (2022), and created an ontology using FoodOn Dooley et al. (2018) and
Glove Pennington et al. (2014) to describe the different components in the
food description. By doing so, we were able to distinguish between different
slot values. Using the ontology, we created a template-based conversational
dataset that mimics human conversation, alternating between the user and the
system turns. We define an average of 7 slots while creating the dataset
namely- food, cook, type, animal, part, foodweight, metric.

#### Data Source

To construct the dataset, we leverage the extensive food descriptions and
nutritional data from the USFDC database USFDC (2022). It is renowned for its
broad representation of diverse food items and is publicly available. The
dataset was created with careful consideration of cultural differences,
sourcing its data from the U.S. Department of Agriculture (USDA).

Each food description in the USFDC database consist of unstructured, comma-
separated text detailing ingredients, cooking methods, and cutting styles.
This format lacks clarity on the significance of each component as
demonstrated in Table 6 of Appendix A. To address this, we concatenated these
segments using underscores, transforming each into a distinct entity to
enhance data clarity and interpretation.

Furthermore, we faced the difficulty of distinguishing whether a food item is
a primary ingredient or as part of a larger dish. Items like lettuce and
cheese, for example, can be both independent food items and components in
recipes like pizzas or burgers. To overcome this challenge, we developed a
food ontology. This ontology aids in categorizing each food item more
accurately, thus improving the overall understanding of the dataset.

#### Ontology Construction

To develop the food ontology, an initial framework is established using FoodON
Dooley et al. (2018), focusing on key relations of food, cook, animal and
part. These relations were chosen based on their significant impact on
altering the salt content in various foods. Moving forward, these specific
relations will be utilized to aid in creating and annotating the
conversational dataset and pose clarifying questions to users to infer the
salt amount in food. To address the limitations in the comprehensiveness of
this initial ontology, pre-trained GloVe vectors Pennington et al. (2014) are
utilized to identify words similar to those in the ontology, thus expanding
its scope. However, this method inadvertently introduces some items unrelated
to food, necessitating manual preprocessing to eliminate irrelevant elements
and maintain a focus on food context.

Further refinement of the ontology was needed to incorporate the items that do
not associate with existing relations. A new relation, type, was created to
integrate these components (For example type of cuisine, meat, other food
ingredients etc). Drawing inspiration from FoodKG Haussmann et al. (2019), the
first item in each food description is categorized under the food relation.
This enhanced ontology becomes a valuable tool for mapping each component of
the comma-separated food descriptions to the relevant keys. In instances where
multiple segments pertain to the type relation, their values are concatenated
to ensure consistency and clarity.

### 3.1. Conversational Dataset Creation

key |  Questions
---|---
food |  What is the {nutrient} content in {food} ?
How much {nutrient} in {food}?
What is the {nutrient} content in {cook} {food}?
How much {nutrient} in {foodWeight} {metric} of {food}?
Can my partner with heart issues eat {food}?
Is {food} okay for heart patients?
Table 1: Sample Template Questions that user asks to begin the conversation

Using the ontology and the food descriptions, we aim to develop a template-
based conversational dataset that mimics human conversation. The conversation
initiates with a user query about the salt content in a specific food item and
alternates between the user and the system. The system poses clarification
questions, drawing from ontology relations such as the type of food, cooking
method, and portion size, which are crucial determinants of salt content.

Leveraging the task-oriented dialogue framework, each turn t is annotated with
a belief state $B_{t}$, encompassing a list of slot-value pairs and action
states inform and request. Figure 1 illustrates a sample conversation
highlighting the belief state and action state for each turn t. The figure
demonstrates that within a single turn, it is possible to fill multiple slot
values (the 3rd turn of the user). The dataset’s format and annotations draw
inspiration from the advanced task-oriented dialog dataset, MultiWOZ
Budzianowski et al. (2018). Employing MultiWOZ as a benchmark not only
validates the dataset but also enhances its replicability for crafting
conversational datasets for other nutrients. Using template-based approach
streamlines the annotation process, ensuring uniformity and efficiency, and
reducing the cost and time needed for external annotators.

| Dialogue Statistics
---|---
# Dialogues | 87,425
# Total turns | 525,392
Avg turns per dialogue | 6
Avg slots | 7
Table 2: Dialogue Statistics of the template based conversation data

We consider 3 types of turns in a conversation. We have tried to encapsulate
the range of dynamics that can occur in dialogues, ensuring the system is
robust enough to handle the fluidity of human conversation.

  * •

Matching Answers\- This type involves turns where the user’s reply is directly
pertinent to the system’s question regarding a particular slot value. An
example is when the system queries about the cooking method, and the user
responds specifically about the food’s cooking method (the 4th turn of the
user in Figure 1).

  * •

Random Answers\- Occasionally, a user’s reply may not correspond to the query
posed by the system. For instance, if the system asks about the food type and
the user responds with information about the weight of the item, as
demonstrated in the user’s second turn in Figure 1, the system needs to adapt.
In such situations, the system should recognize and not repeat a question
about the weight, since that information has already been provided. Instead,
it should proceed to ask another question, possibly continuing to seek details
about the food type. In .45% of conversations, a turn consists of a random
answer.

  * •

Changing Answers\- People are very indecisive and often tend to change their
responses. This category captures the scenarios where users might revise their
previous responses. For example, if a user initially mentions that the cooking
method is "pan-fried" but later changes it to "boiled," the system needs to
update its understanding to reflect this new information, shifting its belief
state from “pan-fried” to “boiled.” In .45% of conversations, one user turn
involves changing the answer.

The dataset is created by generating random conversations, in which, at each
turn, the system’s questions and the user’s responses are randomly selected
from the templates. The conversation starts with the user asking about the
salt content in food. This initial question is informed by a user study of
HFChat Salunke et al. (2023), where participants frequently asked 3 categories
of questions: 1) how much salt in {food} 2) Can I eat {food}? and 3) what kind
of {food} can I eat?. These question types, along with similar ones,
constitute the initial question, as exemplified in Table 1. The system’s
objective is to engage in the dialogue by asking clarification questions to
ascertain the values of different slots (cook, type, weight etc). To keep the
dialogue dynamic and realistic, questions related to these slots are presented
randomly. The number of questions is limited; for instance, if the food in
question is eaten raw, queries about cooking methods are omitted. Users might
not be aware of all system-initiated questions, in which case default values
for each slot are assumed. This led to the creation of approximately 87k
template-based conversations, each comprising 3-4 exchanges between the user
and the system. The statistics of this extensive dataset are detailed in Table
2. As the dataset size is very large, around 87k conversations, a few-shot
method is used to train the model.

## 4\. Methodology

Once the conversational dataset was created, we built the NS-PPTOD model by
fine-tuning PPTOD on the dataset using few-shot and integrating neuro-symbolic
rules.

### 4.1. Plug and Play Task Oriented Dialog System (PPTOD)

Leveraging the T5’s model success, we adopted the PPTOD model for developing
the task-oriented dialogue system. PPTOD extends the T5 framework, especially
designed for task-oriented dialogue (TOD) applications, and pre-trained on a
diverse range of dialog datasets spanning eleven different domains. PPTOD has
integrated different TOD modules — Dialogue State Tracking (DST), Natural
Language Understanding (NLU), Dialogue Policy (POL) and Natural Language
Generation (NLG) —into a single end-to-end architecture using a pipeline
approach.

Figure 2: NS-PPTOD Model with Example: This example illustrates that PPTOD
alone is not able to identify correct salt value for the food item

The PPTOD model is adept at in-context learning, employing customized prompts
for each dialogue component, thus enhancing the relevance of model inputs to
specific dialogue scenarios. Each training sample in PPTOD is represented as
($x_{t}$, $y$, $z$), where $t$ $\in$ NLU, DST, POL, NLG specifies the type of
Task-Oriented Dialogue (TOD) task to which the sample belongs. The term
$x_{t}$ refers to the task-specific prompt, formatted as translate dialogue to
A:, where A represents different aspects such as user intent, belief state,
dialogue act, and system response corresponding to the NLU, DST, POL, and NLG
tasks respectively. The input dialogue context, a concatenation of all
preceding dialogue utterances, encompassing both the system’s and the user’s
contributions is represented by $y$ whereas $z$ indicates the target output
text. This is exemplified in Figure 2. PPTOD is trained with a maximum
likelihood objective and the loss function as shown in Equation 1 where
$\theta$ is the model parameters.

$L_{\theta}=-\sum_{i=1}^{\mathinner{\\!\left\lvert z\right\rvert}}\log
P_{\theta}(z_{i}|z_{<i};x_{t},y)$ (1)

PPTOD also has an associated tokenizer, which supports a few-shot learning
framework, enabling the system to identify new food-related terms not seen in
training. We have fine-tuned it on only 1000 food-related dialogs using the
same learning objective as PPTOD.

We chose PPTOD model for its adaptability, and its ability to support modular
decomposition which in turn facilitates the incorporation of specific rules.
To further enhance the model, we incorporated neuro-symbolic techniques into
the DST framework, aiming to boost the system’s interpretative capabilities
and its adaptability to intricate dialogue contexts. This integration seeks to
fortify the dialogue system, ensuring it remains robust and flexible in
managing diverse conversations.

In summary, we chose PPTOD because:

  * •

PPTOD is a state-of-the-art (SOTA) model designed specifically for task-
oriented dialogue and is based on the T5 model.

  * •

PPTOD consists of a tokenizer making it possible to use few-shot approach to
fine-tune it.

  * •

PPTOD can be modularised to implement neuro-symbolic AI.

### 4.2. NS-PPTOD

NS-PPTOD is an integration of PPTOD with neuro-symbolic rules. The
conversational dataset we created as described in section 3.1 is used to
finetune PPTOD. Given the extensive size of the dataset of 87k template-based
conversations, a few-shot learning approach is used. Instead of utilizing the
entire dataset, limited subsets of samples are selected randomly to finetune
PPTOD. As we use a few-shot approach, T5-small is used as the base model to
train PPTOD. PPTOD model is finetuned for 10 epochs, employing a batch size of
16 and varying the total dataset size across 100, 300, 500, and 1000 samples.
Within the dataset, 10% of the data was allocated as a development set, and
another 10% served as the test set.

|  Train Size |  Epochs |  Inform |  Success |  BLEU
---|---|---|---|---|---
PPTOD |  100 |  8 |  71.43 |  0 |  24.99
NS-PPTOD |  100 |  - |  88.90 |  77.80 |  22.50
PPTOD |  300 |  7 |  75.00 |  5.00 |  34.30
NS-PPTOD |  300 |  - |  81.50 |  63.00 |  26.90
PPTOD |  500 |  9 |  82.86 |  2.86 |  29.81
NS-PPTOD |  500 |  - |  74.50 |  58.10 |  28.90
PPTOD |  1000 |  7 |  93.50 |  2.70 |  29.00
NS-PPTOD |  1000 |  - |  85.90 |  71.70 |  30.00
Table 3: Increase in performance when using NS-PPTOD compared to PPTOD.

During the experiments - to be discussed in Section 5, we observed that the
model correctly identified slot values but struggled to determine the correct
salt values. Additionally, the dataset primarily comprised salt values for
standard food weights, like 100 grams, 3 ounces, 1 packet, etc., and lacked
data for non-standard food weights that users might inquire about. PPTOD model
is fine-tuned so that it accurately learns other slot values, even if it also
learns some incorrect salt values. Subsequently, we employ a neuro-symbolic
approach that involves two key methodologies to correct the salt values:

  * •

Retrieval of the accurate salt value from the database: This step is crucial
for standard food weights where exact values are available and can be directly
obtained. (as shown in Figure 2)

  * •

Mathematical calculation of the correct salt value for varying food weights:
This method is particularly beneficial for non-standard food weights, enabling
the model to compute salt values based on weight. (In Figure 2, if the user
requests the salt value for a different food weight, instead of the standard
100 grams of pork chops.)

Specifically, upon determining the slot values, the system queries the
database for the salt content. If the database contains the salt value, it is
then retrieved. In cases where the salt value is not available in the
database, it is calculated mathematically, based on the weight of the food. We
demonstrate this in Figure 2. When the dialog context and prompt, labeled
Translate dialog to belief state, are processed through PPTOD’s DST model to
determine belief states, the model successfully infers all slot values except
for the salt value (12). The correct salt value (81) is then retrieved from
the database by applying Neuro-Symbolic rules.

Train Size |  Epochs | Joint Accuracy
---|---|---
|  |  PPTOD |  NS-PPTOD
100 |  6 |  55.56 |  73.08
300 |  4 |  51.92 |  72.8
500 |  6 |  58.75 |  83.2
1000 |  6 |  58.53 |  85.2
Table 4: Increase in Joint Accuracy when using NS-PPTOD compared to PPTOD
across different training sizes

## 5\. Evaluation

The evaluation of NS-PPTOD encompasses two task-oriented tasks. The first task
involves end-to-end dialog modeling, assessed using metrics such as inform
rate, success rate, and BLEU score Papineni et al. (2002). The second task,
the Dialog State Tracking (DST) module of PPTOD is evaluated through joint-
accuracy. This evaluation was conducted over sample sizes of 100, 300, 500,
and 1000.

#### End-to-End Dialog Modeling

After seven epochs of training, the PPTOD model demonstrated a high inform
accuracy rate of 93.5% across 1000 samples, as detailed in Table 3. Inform
rate reflects the model’s adeptness in identifying slot values and the target
goal slot. However, its success rate in accurately determining correct salt
values was notably low, standing at just 2.7%, a point further highlighted in
Table 3. This pattern of low success rate was consistent across other training
sizes of 100, 300, and 500 samples. The limited 2% success rate is attributed
to the PPTOD model’s tendency to predict values at random.

Implementing the NS-PPTOD model resulted in a substantial enhancement,
achieving success rate of 71.7%. This improvement was not just limited to the
training size of 1000 samples but was also observed consistently across the
smaller training sizes. The integration of neuro-symbolic rules with PPTOD
evidently plays a crucial role in enhancing the model’s capability to
accurately predict and determine the correct salt values.

#### DST Module

The evaluation also included the Dialog State Tracking (DST) module of PPTOD,
trained for recognizing different belief states such as food, cook, type,
weight, and value. Joint accuracy of 58.53% was achieved when PPTOD was used.
This performance notably increased to 85.2% for 1000 samples with the addition
of neuro-symbolic rules. Similar improvements in joint accuracy were observed
for other training sizes. Table 4 displays the enhanced joint-accuracy
achieved by using NS-PPTOD compared to PPTOD.

#### Analysis

The improvement in both success rate and joint accuracy can be attributed to a
key difference in approach. PPTOD, on its own, tends to memorize values
instead of effectively retrieving them from the database, a critical process
for accurately determining salt content. However, the application of neuro-
symbolic rules in conjunction with PPTOD enhances its capability, enabling it
to effectively retrieve values from the database.

### 5.1. Comparison with ChatGPT

With the advent of ChatGPT, questions have arisen about the necessity of
systems such as our NS-PPTOD. To address this,we conducted a comparative
analysis between NS-PPTOD and ChatGPT, to highlight the distinct capabilities
and applications of NS-PPTOD that are not achievable by ChatGPT.

In this comparison, we specifically focused on their responses to queries
about the salt content in foods. Appendix B illustrates ChatGPT’s response to
the prompt What is the salt amount in a pork chop?. ChatGPT’s responses are
generally comprehensive, explaining variations in salt quantity due to
different cooking methods and weights, and often include an average value. NS-
PPTOD, conversely, poses targeted clarification questions to precisely
identify both the food item and its salt content.

There is also a marked difference in the readability of responses from these
two systems. Readability assessments, SMOG Mc Laughlin (1969), Flesch-Kincaid
Grade Level (FKGL) Kincaid et al. (1975) , and Flesch-Kincaid Reading Ease
(FKRE) Kincaid et al. (1975), which evaluates text comprehensibility in
educational and informational contexts, were used for this comparison. The
SMOG and FKGL estimates the education level needed to understand a text, while
the FKRE measures the ease of comprehension.

As indicated in Table 5, understanding ChatGPT’s content requires an 11th-
grade reading level, whereas NS-PPTOD’s content is accessible to those at a
2nd-grade reading level. NS-PPTOD’s responses align with recommendations for
content to be understandable at a 5th-grade reading level or lower Badarudeen
and Sabharwal (2010). This approach is particularly suitable for the intended
audience of our system, the patients served by the hospital at our institution
who are mostly of lower socio-economic status (and mostly minorities); this
ensures the information is accessible and understandable to a wider audience.

| SMOG | FKGL | FKRE
---|---|---|---
ChatGPT | 11.8 | 11.1 | 51.28
NS-PPTOD | 0 | 2.5 | 95.17
Table 5: Readability Comparison of NS-PPTOD with ChatGPT: A higher FKRE score
denotes better readability. Lower scores in SMOG and FKGL indicate better text
readability.

## 6\. Conclusion and Future Work

This work introduces NS-PPTOD, a conversational dialogue system specifically
designed to help patients, particularly African-American individuals suffering
from heart failure, in accurately identifying the salt content in various
foods. Managing salt intake is crucial for these patients, and NS-PPTOD offers
a specialized solution to aid in their dietary management. The study reveals
that simply applying fine-tuning techniques to a transformer model for
predicting salt content is inadequate for achieving high accuracy. However,
the integration of neuro-symbolic rules into the system resulted in a
significant performance boost, enhancing accuracy by 20% across different
dataset sizes. This enhancement underscores the necessity of combining
pretrained language models with neuro-symbolic rules to achieve superior
accuracy.

Looking ahead, there are plans to conduct an in-depth study comparing the
performance of NS-PPTOD with a system based on ChatGPT. This study will
involve African American patients with heart failure interacting with both
systems in a within-subject setting. Utilizing pre and post-interaction
surveys, the aim is to conduct both qualitative and quantitative analyses.
These analyses will evaluate and contrast the effectiveness of each system
offering insights into their respective impact in real-world scenarios.

## 7\. Acknowledgement

For financial support, we thank the DPI Cycle 1 Seed Funding Program Award and
NSF award IIS 2232307

##

  * Ali et al. (2020) Mohammad Rafayet Ali, Seyedeh Zahra Razavi, Raina Langevin, Abdullah Al Mamun, Benjamin Kane, Reza Rawassizadeh, Lenhart K Schubert, and Ehsan Hoque. 2020.  A virtual conversational agent for teens with autism spectrum disorder: Experimental results and design lessons.  In _Proceedings of the 20th ACM international conference on intelligent virtual agents_ , pages 1–8.
  * Arabshahi et al. (2021) Forough Arabshahi, Jennifer Lee, Mikayla Gawarecki, Kathryn Mazaitis, Amos Azaria, and Tom Mitchell. 2021.  Conversational neuro-symbolic commonsense reasoning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 4902–4911.
  * Badarudeen and Sabharwal (2010) Sameer Badarudeen and Sanjeev Sabharwal. 2010.  Assessing readability of patient education materials: current role in orthopaedics.  _Clinical orthopaedics and related research_ , 468(10):2572–2580.
  * Belfin et al. (2019) RV Belfin, AJ Shobana, Megha Manilal, Ashly Ann Mathew, and Blessy Babu. 2019.  A graph based chatbot for cancer patients.  In _2019 5th international conference on advanced computing & communication systems (ICACCS)_, pages 717–721. IEEE.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018.  MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5016–5026, Brussels, Belgium. Association for Computational Linguistics.
  * Cai et al. (2023) Pengshan Cai, Zonghai Yao, Fei Liu, Dakuo Wang, Meghan Reilly, Huixue Zhou, Lingxi Li, Yi Cao, Alok Kapoor, Adarsha Bajracharya, Dan Berlowitz, and Hong Yu. 2023.  PaniniQA: Enhancing Patient Education Through Interactive Question Answering.  _Transactions of the Association for Computational Linguistics_ , 11:1518–1536.
  * Chen et al. (2021) Yu Chen, Ananya Subburathinam, Ching-Hua Chen, and Mohammed J. Zaki. 2021.  Personalized food recommendation as constrained question answering over a large-scale food knowledge graph.  In _Proceedings of the 14th ACM International Conference on Web Search and Data Mining_ , WSDM ’21, page 544–552, New York, NY, USA. Association for Computing Machinery.
  * Chowdhury et al. (2023) Mohita Chowdhury, Ernest Lim, Aisling Higham, Rory McKinnon, Nikoletta Ventoura, Yajie He, and Nick De Pennington. 2023.  Can large language models safely address patient questions following cataract surgery? In _Proceedings of the 5th Clinical Natural Language Processing Workshop_ , pages 131–137, Toronto, Canada. Association for Computational Linguistics.
  * Dickson and Riegel (2009) Victoria Vaughan Dickson and Barbara Riegel. 2009.  Are we teaching what patients need to know? building skills in heart failure self-care.  _Heart & Lung_, 38(3):253–261.
  * Dooley et al. (2018) D.M. Dooley, E.J. Griffiths, and G.S Gosal. 2018.  Foodon: a harmonized food ontology to increase global food traceability, quality control and data integration.  _npj Science of Food_ , 2.
  * Fu et al. (2022) Chengcheng Fu, Zhisheng Huang, Frank van Harmelen, Tingting He, and Xingpeng Jiang. 2022.  Food recommendation for mental health by using knowledge graph approach.  In _Health Information Science: 11th International Conference, HIS 2022, Virtual Event, October 28–30, 2022, Proceedings_ , page 231–242, Berlin, Heidelberg. Springer-Verlag.
  * Garcez and Lamb (2023) Artur d’Avila Garcez and Luis C Lamb. 2023.  Neurosymbolic ai: The 3 rd wave.  _Artificial Intelligence Review_ , pages 1–20.
  * Gupta et al. (2020) Itika Gupta, Barbara Di Eugenio, Devika Salunke, Andrew Boyd, Paula Allen-Meares, Carolyn Dickens, and Olga Garcia. 2020.  Heart failure education of African American and Hispanic/Latino patients: Data collection and analysis.  In _Proceedings of the First Workshop on Natural Language Processing for Medical Conversations_ , pages 41–46, Online. Association for Computational Linguistics.
  * Haussmann et al. (2019) Steven Haussmann, Oshani Seneviratne, Yu Chen, Yarden Ne’eman, James Codella, Ching-Hua Chen, Deborah L. McGuinness, and Mohammed J. Zaki. 2019.  Foodkg: A semantics-driven knowledge graph for food recommendation.  In _The Semantic Web – ISWC 2019_ , pages 146–162, Cham. Springer International Publishing.
  * He et al. (2020) Feng J. He, Monique Tan, Yuan Ma, and Graham A. MacGregor. 2020.  Salt reduction to prevent hypertension and cardiovascular disease.  _Journal of the American College of Cardiology_ , 75(6):632–647.
  * Jiang et al. (2022) Yiwei Jiang, Klim Zaporojets, Johannes Deleu, Thomas Demeester, and Chris Develder. 2022.  Cookdial: a dataset for task-oriented dialogs grounded in procedural documents.  _Applied Intelligence_ , pages 1–19.
  * Kearns et al. (2020) William R. Kearns, Neha Kaura, Myra Divina, Cuong Vo, Dong Si, Teresa Ward, and Weichao Yuwen. 2020.  A wizard-of-oz interface and persona-based methodology for collecting health counseling dialog.  In _Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems_ , CHI EA ’20, page 1–9, New York, NY, USA. Association for Computing Machinery.
  * Kincaid et al. (1975) J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975.  Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel.
  * Lei et al. (2018) Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. 2018.  Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Melbourne, Australia. Association for Computational Linguistics.
  * Li et al. (2023) Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. 2023.  Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge.
  * Mc Laughlin (1969) G Harry Mc Laughlin. 1969.  Smog grading-a new readability formula.  _Journal of reading_ , 12(8):639–646.
  * Mehri and Eskenazi (2021) Shikib Mehri and Maxine Eskenazi. 2021.  Schema-guided paradigm for zero-shot dialog.  In _Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 499–508, Singapore and Online. Association for Computational Linguistics.
  * Moulik (2019) Sanjoy Moulik. 2019.  _DIL-A Conversational Agent for Heart Failure Patients_.  Ph.D. thesis, The Claremont Graduate University.
  * Nayak et al. (2020) A Nayak, AJ Hicks, and AA Morris. 2020.  Understanding the complexity of heart failure risk and treatment in black patients.  _Circulation: Heart Failure_ , 13(8):e007264.
  * Nogueira et al. (2021) Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2021.  Investigating the limitations of transformers with simple arithmetic tasks.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Peng et al. (2021) Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, and Jianfeng Gao. 2021.  Soloist: Building task bots at scale with transfer learning and machine teaching.  _Transactions of the Association for Computational Linguistics_ , 9.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics.
  * Qian et al. (2023) Jing Qian, Hong Wang, Zekun Li, Shiyang Li, and Xifeng Yan. 2023.  Limitations of language models in arithmetic and symbolic induction.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 9285–9298, Toronto, Canada. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Romero et al. (2021) Oscar J. Romero, Antian Wang, John Zimmerman, Aaron Steinfeld, and Anthony Tomasic. 2021.  A task-oriented dialogue architecture via transformer neural language models and symbolic injection.  In _Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 438–444, Singapore and Online. Association for Computational Linguistics.
  * Roth et al. (2018) Gregory A Roth, Degu Abate, Kalkidan Hassen Abate, Solomon M Abay, Cristiana Abbafati, Nooshin Abbasi, Hedayat Abbastabar, Foad Abd-Allah, Jemal Abdela, Ahmed Abdelalim, et al. 2018.  Global, regional, and national age-sex-specific mortality for 282 causes of death in 195 countries and territories, 1980–2017: a systematic analysis for the global burden of disease study 2017.  _The Lancet_ , 392(10159):1736–1788.
  * Salunke et al. (2023) Devika Salunke, Carolyn Dickens, Olga Garcia-Bedoya, Paula Allen-Meares, Barbara Di Eugenio, and Andrew Dallas Boyd. 2023.  Learnings from pilot testing hfchat, an interactive dialogue agent that can assist black/african american and hispanic/latino patients with heart failure.  _Journal of Cardiac Failure_ , 29(4):563.
  * Su et al. (2022) Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, and Yi Zhang. 2022.  Multi-task pre-training for plug-and-play task-oriented dialogue system.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4661–4676, Dublin, Ireland. Association for Computational Linguistics.
  * USFDC (2022) USFDC. 2022.  Us food data central.
  * Valizadeh and Parde (2022) Mina Valizadeh and Natalie Parde. 2022.  The AI doctor is in: A survey of task-oriented dialogue systems for healthcare applications.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6638–6660, Dublin, Ireland. Association for Computational Linguistics.
  * Wang et al. (2020) Jinping Wang, Hyun Yang, Ruosi Shao, Saeed Abdullah, and S. Shyam Sundar. 2020.  Alexa as coach: Leveraging smart speakers to build social agents that reduce public speaking anxiety.  In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ , CHI ’20, page 1–13, New York, NY, USA. Association for Computing Machinery.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * Wei et al. (2018) Zhongyu Wei, Qianlong Liu, Baolin Peng, Huaixiao Tou, Ting Chen, Xuanjing Huang, Kam-fai Wong, and Xiangying Dai. 2018.  Task-oriented dialogue system for automatic diagnosis.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 201–207, Melbourne, Australia. Association for Computational Linguistics.
  * Yagcioglu et al. (2018) Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler-Cinbis. 2018.  RecipeQA: A challenge dataset for multimodal comprehension of cooking recipes.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1358–1368, Brussels, Belgium. Association for Computational Linguistics.
  * Zhou et al. (2022) Yue Zhou, Barbara Di Eugenio, Brian Ziebart, Lisa Sharp, Bing Liu, Ben Gerber, Nikolaos Agadakos, and Shweta Yadav. 2022.  Towards enhancing health coaching dialogue in low-resource settings.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 694–706, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Zhou et al. (2020) Zhenpeng Zhou, Ahmad Beirami, Paul Crook, Pararth Shah, Rajen Subba, and Alborz Geramifard. 2020.  Resource constrained dialog policy learning via differentiable inductive logic programming.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6775–6787, Barcelona, Spain (Online). International Committee on Computational Linguistics.

##

  * (1)

##  Appendix A Sample Food Descriptions

Food Descriptions |  Salt Value
---|---
Pork, fresh, loin, top loin (chops), boneless, separable lean and fat, raw |  48
Pork, fresh, loin, center loin (chops), bone-in, separable lean and fat, cooked, broiled |  55
Pork, fresh, blade, (chops), boneless, separable lean and fat, cooked, broiled |  58
Pork, fresh, loin, sirloin (chops or roasts), boneless, separable lean only, raw |  63
Pork, fresh, loin, blade (chops), bone-in, separable lean only, cooked, broiled |  76
Table 6: Unstructured Food Descriptions along with their salt value for 100gms
of food

##  Appendix B Sample ChatGPT Response

ChatGPT response to the prompt How much salt in pork chops?

The salt content in pork can vary significantly based on how it is prepared.
Fresh, unprocessed pork naturally contains a small amount of sodium, roughly
60 to 70 milligrams of sodium per 3-ounce serving. However, processed pork
products like bacon, ham, sausages, and deli meats can have a much higher
sodium content due to added salt during processing for flavor and
preservation. For instance, a slice of bacon can have between 150 to 450
milligrams of sodium, and a serving of ham can contain over 1,000 milligrams.

For specific sodium content, it’s best to check the nutrition labels on
packaged pork products. For fresh pork, nutrition databases or guidelines from
health organizations can provide general estimates. Remember, if you’re
watching your sodium intake, opting for fresh, unprocessed pork and limiting
processed pork products can help manage your sodium consumption.


# Unsupervised Segmentation for Terracotta Warrior with Seed-Region-Growing
CNN(SRG-Net)

Yao Hu1  Guohua Geng1  Kang Li1  Wei Zhou1  Xingxing Hao1 and Xin Cao1  1
School of Information Science and Technology, Northwest University, Xi’an
710127, Shaanxi, China  *<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The repairing work of terracotta warriors in Emperor Qinshihuang Mausoleum
Site Museum is handcrafted by experts, and the increasing amounts of unearthed
pieces of terracotta warriors make the archaeologists too challenging to
conduct the restoration of terracotta warriors efficiently. We hope to segment
the 3D point cloud data of the terracotta warriors automatically and store the
fragment data in the database to assist the archaeologists in matching the
actual fragments with the ones in the database, which could result in higher
repairing efficiency of terracotta warriors. Moreover, the existing 3D neural
network research is mainly focusing on supervised classification, clustering,
unsupervised representation, and reconstruction. There are few pieces of
researches concentrating on unsupervised point cloud part segmentation. In
this paper, we present SRG-Net for 3D point clouds of terracotta warriors to
address these problems. Firstly, we adopt a customized seed-region-growing
algorithm to segment the point cloud coarsely. Then we present a supervised
segmentation and unsupervised reconstruction networks to learn the
characteristics of 3D point clouds. Finally, we combine the SRG algorithm with
our improved CNN using a refinement method. This pipeline is called SRG-Net,
which aims at conducting segmentation tasks on the terracotta warriors. Our
proposed SRG-Net is evaluated on the terracotta warriors data and ShapeNet
dataset by measuring the accuracy and the latency. The experimental results
show that our SRG-Net outperforms the state-of-the-art methods. Our code is
available at https://github.com/hyoau/SRG-Net.

††journal: osajournal††articletype: Research Article

##  1 Introduction

Nowadays, the repairing work of terracotta warriors is mainly accomplished by
the handwork of archaeologists in Emperor Qinshihuang’s Mausoleum Site Museum.
While more and more fragments of terracotta warriors are excavated from the
site, the restoration does not have a high repairing efficiency. Moreover, the
lack of archaeological technicians repairing the terracotta warriors makes it
more uneasy to catch up with excavation speed. As a result, more and more
fragments excavated from the archaeological site remain to be repaired.
Terracotta Warrior is a 3D natural structure and can be represented with a
point cloud. Fortunately, with the development of 3D sensors (Apple Li-DAR,
Kinect, and RealSense) and the improvement of hardware (GPU and CPU), it is
possible to improve the repairing speed digitally. We can segment the 3D
object of the terracotta warriors and store the fragments in the database. Our
work aims at simplifying the work of researchers in the museum, making it easy
to match fragments excavated from the site with the pre-segmented fragments in
the database. To improve the efficiency of the repairing work, we propose an
unsupervised 3D point cloud segmentation method for the terracotta warrior
data based on convolution neural network. With the segmentation method
proposed in this paper, the complete terracotta warriors can be segmented into
fragments corresponding to various parts automatically, e.g., head, hand, and
foot, etc. Compared with the traditional segmentation work by hand, the method
proposed in this paper can significantly improve efficiency and accuracy.

Recently, there have been many pieces of research focusing on how to voxelize
the point cloud to make them evenly distributed in regular 3D space, and then
implement 3D-CNN on them. However, voxelization owns the high space and time
complexity, and there may be quantization errors in the process of
voxelization, which would result in low accuracy. Compared with other data
formats, point cloud is a data structure suitable for 3D scene calculation of
terracotta warriors. We choose to segment the terracotta warriors in the form
of point clouds (see samples in Fig. 1).

Figure 1: Simplified Terracotta Warrior Point Clouds

Our terracotta warriors data is represented with $\\{x_{n}\in
R^{p}\\}_{n=1}^{N}$, where $R^{p}$ is the feature space, $x_{n}$ means the
features of one point, such as $x$, $y$, $z$, $N_{x}$, $N_{y}$, $N_{z}$(xyz
coordinates and normal value), $N$ means the number of points in one
terracotta warrior 3D object. Our goal is to design a function
$f:R^{p}\rightarrow L$, where $L$ means the segmentation mapping labels, where
$\\{c_{n}\in L\\}_{n=1}^{N}$ and $c_{n}$ is each point’s label after the
segmentation. In our problem, $\\{c_{n}\\}$ is fixed while the mapping
function $f$ and $\\{x_{n}\\}$ are trainable. To solve the unsupervised
segmentation problem, we can split the problem into two sub-problems. Firstly,
we need to design an algorithm to predict optimal $L$. Secondly, we need to
design a network to learn the labels $\\{c_{n}\\}$ better and make full use of
features of terracotta warrior point cloud. To get the optimal cluster labels
$\\{c_{n}\\}$, we think that a good point cloud segmentation acts like what a
human does. Intuitively, the points with similar features or semantic meanings
are easier to segment to the same cluster. In 2D images, spatially continuous
pixels tend to have similar colors or textures. In 3D point cloud, grouping
spatially continuous points with similar normal value or colors are reasonable
to be given the same label in 3D point cloud segmentation. Besides, the
Euclidean distance between the points with the same label will not be very
long. In conclusion, we design two criteria to predict the $\\{c_{n}\\}$:

  * •

Points with similar spatial features are desired to be given the same label.

  * •

The Euclidean distance between spatially continuous points should not be quite
long.

We design an SRG algorithm for learning the normal feature of point clouds as
much as possible. The experiment [1] shows that the encoder-decoder structure
of FoldingNet [2] does help neural network learn from point cloud. And DG-CNN
is good at learning local features. So we propose our SRG-Net inspired by
FoldingNet [2] and DG-CNN [3] to learn part features of terracotta warriors
better. In Section 3, we describe the details of the process of assigning
labels $\\{c_{n}\\}$. Section 4 will show that our method achieves better
methods compared with other methods. Compared with existing methods, the key
contribution of our work can be summarized as follows:

  * •

We design a novel SRG algorithm for point cloud segmentation to make the most
of point cloud xyz coordinates by estimating normal features.

  * •

We propose our CNN inspired by DG-CNN and FoldingNet to learn the local and
global features of 3D point clouds better.

  * •

We combine the SRG algorithm and CNN neural network with our refinement method
and achieve state-of-the-art segmentation results on terracotta warrior data.

  * •

Our end-to-end model not only can be used in terracotta warrior point clouds
but also can achieve quite good results on other point clouds. We also
evaluate our SRG-Net on ShapeNet dataset.

##  2 Related Work

Segmentation is typical both in the 2D image and 3D point cloud processing. In
image processing, segmentation accomplishes a task that assigns labels to all
the pixels in one image and clusters them with their features. Similarly,
point cloud segmentation is assigning labels to all the points in a point
cloud. The expected result is the points with the similar feature are given
with the same label.

In terms of image segmentation, k-means is a classical segmentation method in
both 2D and 3D, which leads to partition n observations into k clusters with
the nearest mean. This method is quite popular in data mining. The graph-based
method is another popular method like Prim and Kruskal [4] which implements
simple greedy decisions in segmentation. The above methods focus on global
features instead of the local differentiated features, so they usually cannot
get promising results when it comes to the complex context. Among unsupervised
deep learning methods, there are many learning features using the generative
methods, such as [5, 6, 7]. They follow the model of neuroscience, where each
neuron represents a specific semantic meaning. Meanwhile, CNN is widely used
in unsupervised image segmentation. For example, in [8], Kanezaki combines the
superpixel [9] method and CNN and employs superpixel for back propagation to
tune the unsupervised segmentation results. Besides, [10] uses a spatial
continuity loss as an alternative to settle the limitation of the former work
[9], whose method is also quite valuable for 3D point cloud feature learning.

In the field of 3D point cloud segmentation, the state-of-the-art methods for
2D images are not quite suitable for being directly used on the point cloud.
3D point cloud segmentation methods need to understand both the global feature
and geometric details of each point. 3D point cloud segmentation problems can
be classified into semantic segmentation, instance segmentation, and object
segmentation. Semantic segmentation focuses on scene-level segmentation
instance, segmentation emphasizes object-level segmentation, and object
segmentation is centered on part-level segmentation.

As to semantic segmentation, semantic segmentation aims at separating a point
cloud into several parts with the semantic meaning of each point. There are
four main paradigms in semantic segmentation, including projection-based
methods, discretization-based methods, point-based methods, and hybrid
methods. Projection-based methods always project a 3D point cloud to 2D
images, such as multi-view [11, 12], spherical [13, 14]. Discretization-based
methods usually project a point cloud into a discrete representation, such as
volumetric [15] and sparse permutohedral lattices [16, 17]. Instead of
learning a single feature on 3D scans, several methods are trying to learn
different parts from 3D scans, such as [16, 18, 17].

The point-based network can directly learn features on a point cloud and
separate them into several parts. Point clouds are irregular, unordered, and
unstructured. PointNet [19] can directly learn features from the point cloud
and retain the point cloud permutation invariance with a symmetric function
like maximum function and summation function. PointNet can learn point-wise
features with the combination of several MLP layers and a max-pooling layer.
PointNet is a pioneer that directly learns on the point cloud. A series of
point-based networks has been proposed based on PointNet. However, PointNet
can only learn features on each point instead of on the local structure. So
PointNet++ is presented to get local structure from the neighborhood with a
hierarchy network [20]. PointSIFT [21] is proposed to encode orientation and
reach scale awareness. Instead of using K-means to cluster and KNN to generate
neighborhoods like the grouping method PointNet++, PointWeb [22] is proposed
to get the relations between all the points constructed in a local fully-
connected web. As to convolution-based method. RS-CNN takes a local point
cloud subset as its input and maps the low-level relation to the high-level
relation to learn the feature better. PointConv [23] uses the existing
algorithm, using a Monte Carlo estimation to define the convolution. PointCNN
[24] uses $\chi-conv$ transformation to convert the point cloud into a latent
and canonical order. As to point convolution methods, Parametric Continuous
Convolutional Neural Network(PCCN) [25] is proposed based on parametric
continuous convolution layers, whose kernel function is parameterized by MLPs
and spans the continuous vector space. Graph-based methods can better learn
the features like shapes and geometric structures in point clouds. Graph
Attention Convolution(GAC) [26] can learn several relevant features from local
neighborhoods by dynamically assigning attention weights to points in
different neighborhoods and feature channels. Dynamic Graph CNN(DG-CNN) [3]
constructs several dynamic graphs in neighborhood, and concatenates the local
and global features to extract better features and update them each graph
after each layer of the network dynamically. FoldingNet adopts the auto-
encoder structure to encode the point cloud $N\times 3$ to $1\times 512$ and
decode it to $M\times 3$ with the aid of chamfer loss to construct the auto-
encoder network.

Part segmentation is more complex than semantic and instance segmentation
because there are significant geometric differences between the points with
the same labels, and the number of parts with the same semantic meanings may
differ. Z. Wang et al. [27] propose VoxSegNet to achieve promising part
segmentation results on 3D voxelized data, which presents a Spatial Dense
Extraction(SDE) module to extract multi-scale features from volumetric data.
Synchronized Spectral CNN (SyncSpecCNN) [28] is proposed to achieve fine-
grained part segmentation on irregularity and non-isomorphic shape graphs with
convolution. [29] is proposed to segment unorganized noisy point clouds
automatically by extracting clusters of points on the Gaussian sphere. [30]
uses three shape indexes: the smoothness indicator, shape index, and flatness
index based on a fuzzy parameterization. [31] presents a segmentation method
for conventional engineering objects based on local estimation of various
geometric features. Branched AutoEncoder network (BAE-NET) [32] is proposed to
perform unsupervised and weakly-supervised 3D shape co-segmentation. Each
branch of the network can learn features from a specific part shape for a
particular part shape with representation based on the auto-encoder structure.

##  3 Proposed Method

In this paper, our input data for the terracotta warrior is in the form of 3D
point clouds (see samples in Fig. 1). Point cloud data is represented as a set
of 3D points $\\{P_{i}|i=1,2,3...n\\}$, where each point is a vector $R^{n}$
containing coordinates $x$, $y$, $z$ and other features like normal, color.
Our method contains three steps:

  1. 1.

We compute normal value with the $xyz$ coordinates.

  2. 2.

We use our seed-region-growing (SRG) method to pre-segment the point cloud.

  3. 3.

We use our pointwise CNN called SRG-Net self-trained to segment the point
cloud with the refinement of pre-segmentation results.

Given the point cloud only with 3D coordinates, there are several effective
normal estimation methods like [33] [34] [35]. In our method, we follow the
simplest method in [36] because of its low average-case complexity and quite
high accuracy. We propose an unsupervised segmentation method for the
terracotta warrior point cloud by combining our seed-region-growing clustering
method with our 3D pointwise CNN called SRG-Net. This method contains two
phases, SRG segmentation and segmentation network. The problem we want to
solve can be described as follows. Firstly, the SRG algorithm is implemented
on the point cloud to do pre-segmentation. Secondly, we use SRG-Net for
unsupervised segmentation with pre-segmentation labels.

###  3.1 Seed Region Growing

Unlike 2D images, not all point cloud data has features such as color and
normal. For example, our terracotta warriors 3D objects do not have any color
feature. However, normal vectors can be calculated and predicted by point
coordinates [35]. It is worth noting that there are many similarities and
differences between color in 2D and normal features in 3D. For color features
in a 2D image, if the pixels are semantically related, the color of the pixel
in the neighborhood generally does not change a lot. For 3D point cloud normal
features, compared with the color features in 2D images, the normal values of
points in the neighborhood of the point cloud often differ. Points of
neighborhood in one point cloud share similar features.

We design a seed-region-growing method to pre-segment the point cloud based on
the above characteristics of the normal feature of the point cloud.

First, we implement KNN to the point cloud to get the nearest neighbors of
each point. Then we initialize a random point as the start seed and add to the
available points to start the algorithm. Then we choose the first seed from
the available list to judge the points in its neighborhood. If the normal
value and Euclidean distance are within the threshold we set, we think that
the two points are semantically continuous, and we can group two points into
one cluster. The outline about the SRG is given in Algorithm 1. The
description of the algorithm is as follows:

  1. 1.

Input:

    1. (a)

xyz coordinates of each point ${p_{n}\in R^{3}}$

    2. (b)

normal value of each point ${n_{n}\in R^{3}}$

    3. (c)

Euclidean distance of every two points ${e_{n}\in R^{n}}$

    4. (d)

neighborhood of each point ${b_{n}\in R^{k}}$

  2. 2.

The randomly chosen point is added to the set $A$, we can call it seed $a$.

  3. 3.

For other points which are not transformed to seeds in $A$:

    1. (a)

Each of point in seed’s neighbor $N_{a}$, check the normal distance and
Euclidean distance between the point in $N_{a}$ and the seed $a$.

    2. (b)

We set the normal distance and the Euclidean distance threshold. If both of
the two value is less than the two thresholds, we can add it to $A$.

    3. (c)

The current seed is added to the checked list $S$.

Input: $P=\\{p_{n}\in R^{3}\\}$

// x, y, z coordinates

$N=\\{n_{n}\in R^{3}\\}$

// normal value

$E=\\{e_{n}\in R^{n}\\}$

// Euclidean distance

$\Omega(\cdot)$

// nearest neighbour function

Output: $L=\\{l_{n}\in R^{1}\\}$

// segmentation label of each point

Initialize: $S:=\diameter$

// set seed list to empty

$A:=rand\\{1,2...|P|\\}$

// select random point to available point list as seed

for  _t = 1 to T_ do

while  _$A\neq\diameter$_ do

$a\leftarrow first\;item\;in\;A$

$neighbours\leftarrow\Omega(a)$ for  _$neighbours\neq\diameter$_ do

$neighbour\leftarrow rand(neighbours)$

if  _$E(a,neighbour)\leq e_{t}h\land N(a,neighbour)\leq e_{t}h\land
neighbour\not\in S$ _ then

$append\;neighbour\;to\;S$ $remove\;neighbour\;from\;neighbours$

else

$remove\;neighbour\;from\;neighbours$

$add\;a\;to\;S$

Algorithm 1 SRG unsupervised segmentation

###  3.2 SRG-Net

We propose our SRG-Net, which is inspired by the dynamic graph in DG-CNN and
auto-encoder in FoldingNet. Unlike classical graph CNN, our graph is dynamic
and updated in each layer of the network. Compared with the methods that only
focus on the relationship between points, we also propose an encoder structure
to better express the features of the entire point cloud, aiming at learning
the structure of the point cloud and optimize the pre-segmentation results of
SRG. Our network structure can be shown in Fig. 2, which consists of two sub-
network, the first part is an encoder that generates features from the dynamic
graph and the whole point cloud and the second is the segmentation network.

Figure 2: Network Structure. The brown part represents the Spatial Transformer
Network, the blue module represents the Edge Convolution, and the lower module
represents the Graph Convolution. The tensors of the upper and lower modules
will be concatenated, and be input into the Segmenter to get the final
segmentation result. (Segmenter will be explained in the following figure.)

We denote the point cloud as $S$. We use lower-case letters to represent
vectors, such as $x$, and use upper-case letter to represent matrix, such as
A. We call a matrix m-by-n or $m\times n$ if it has m rows and n columns. In
addition, the terracotta warrior point cloud data is N points with 6 features
$x$, $y$, $z$, $N_{x}$, $N_{y}$, $N_{z}$(xyz coordinates and normal values),
denoted as $X=\\{x_{1},x_{2},x_{3},...,x_{n}\\}\subseteq R^{6}$.

####  3.2.1 Encoder Architecture

The SRG-Net encoder follows a similar design of [2], the structure of SRG-Net
is shown in Fig. 2. Compared with [2], our encoder concatenate several multi-
layer perceptrons(MLP) and several dynamic graph-based max-pooling layers. The
dynamic graphs are constructed by applying KNN on point clouds.

For the entire point cloud, we compute a spatial transformer network and get a
transformer matrix of 3-by-3 to maintain invariance under transformations.
Then for the transformed point cloud, we compute three dynamic graphs and get
graph features respectively. In graph feature extracting process, we adopt the
Edge Convolution in [6] to compute the graph feature of each layer, which uses
an asymmetric edge function in Eq. (1):

${f_{i}}_{j}=h(x_{i},x_{i}-x_{j})$ (1)

where it combines the coordinates of neighborhood center $x_{i}$ with the
subtraction of neighborhood point and the center point coordinates
$x_{i}-x_{j}$ to get local and global information of neighborhood. Then we
define our operation in Eq. (2):

$g_{ij}=\Theta(\mu\cdot(x_{i}-x_{j})+\omega\cdot x_{i})$ (2)

where $\mu$ and $\omega$ are parameters and $\Theta$ is a ReLU function. Eq.
(2) is implemented as a shared MLP with Leaky ReLU. Then we define our max-
pooling operation in Eq. (3):

$x_{i}=\max_{j\in N(i)}g_{ij}$ (3)

where $N(i)$ means neighborhood of point $i$.

The bottleneck is computed by the graph feature extraction layer. The
structure is shown in Fig. 2. First, we compute the covariance $3\times 3$
matrix for every point and vectorize it to $1\times 9$. Then the $n\times 3$
matrix of point coordinates is concatenated with the $n\times 9$ covariance
matrix into a $n\times 12$ matrix. Then we put the matrix into a 3-layer
perceptron. Then we feed the output of the perceptron to two subsequent graph
layers. In each layer, max-pooling is added to the neighbor of each node. At
last, we apply a 3-layer perceptron to the former output and get the final
output. The whole process of the graph feature extraction layer is summarized
in Eq. (4):

$Y=I_{max}(X)K$ (4)

In Eq. (4), $X$ is the input matrix to the graph layer and $K$ is a feature
mapping matrix. $I_{max}(X)$ can be represented in Eq. 5:

$(I_{max}(X))_{ij}=\Theta(\max_{k\in N(i)}x_{kj})$ (5)

where $\Theta$ is a ReLU function and $N(i)$ is the neighborhood of point $i$.
The max-pooling operation in Eq. (5) can get local feature based on the graph
structure. So the graph feature extraction layer can not only get local
neighborhood features, but also global features.

Figure 3: Edge Convolution. The figure contains a KNN graph convolution
mapping the point cloud to N × k × f’, and then put into max-pooling to get
the point cloud edge feature.

####  3.2.2 Segmenter Architecture

Figure 4: Segmenter Structure. First, the segmenter concatenates the features
calculated by edge convolution and graph convolution. Then put into three-
layer-perceptron, and finally put into two-layer-perceptron to get the final
segmentation result.

Segmenter gets dynamic graph features and bottleneck as input and assign
labels to each point to segment the whole point cloud. The structure of
segmenter is shown in Fig. 2. First, bottleneck is replicated $N$ times in Eq.
(6):

$B^{\prime}=\underbrace{B\cup B\cup...\cup B}_{N}$ (6)

where $N$ is the number of points in point cloud and $B$ is the bottleneck.
The output of replication is concatenated with dynamic features in Eq. (7):

$C=B^{\prime}\cup D_{1}\cup D_{2}\cup D_{3}$ (7)

where $D_{1}$, $D_{2}$, $D_{3}$ represent dynamic graph features. At last, we
feed the output of concatenation to a multi-layer-perceptron to segment the
point cloud in Eq. (8).

$D=\Theta(\Psi\cdot C+\Omega)$ (8)

where $\Psi$ and $\Omega$ represent parameters in the linear function, and
$\Theta$ represents a ReLU function.

###  3.3 Refinement

Inspired by [37], we designed point refinement to achieve better segmentation
results. The basic concept of point clustering is to group similar points into
clusters. In point cloud segmentation, it is intuitive for the clusters of
points to be spatially continuous. We add the constraint to favor cluster
labels that are the same as those of neighboring points. We first extract $K$
superpoints $\\{S_{k}\\}_{k=1}^{K}$ from the input point cloud
$L=\\{v_{n}\\}_{n=1}^{N}$, where $S_{k}$ is a set of the indices of points
that belongs to the k-th superpoint. Then we assign points in each superpoint
to have the same cluster label. More specifically, letting $|c_{n}|_{n\in
S_{k}}$ be the number of points in $S_{k}$ that belong to the $c_{n}$th
cluster, we select the most frequent cluster label $c_{m}ax$, where
$|c_{max}|_{n\in S_{k}}\geq|c_{n}|_{n\in S_{k}}$ for all $c_{n}in{1,...,q}$.
The cluster labels are replaced by $c_{max}$ for $n\in S_{k}$.

In this paper, there is no need to set a large number in seed-region-growing
process because our neural network can learn both the local and global
features. We choose seed-region-growing method with $K=25$ for the super point
extraction. The refinement process can achieve better results compared with
straight learning in Fig. 1. In addition, instead of using ground truth to
calculate loss, we use the coarse segmentation result of seed-region-growing
method to calculate the loss.

##  4 Experiments

We conduct experiments on terracotta warrior dataset and ShapeNet dataset. We
implement the pipeline using PyTorch and Python3.7. All the results are based
on experiments under RTX 2080 Ti and i9-9900K. The performances of each method
in the experiment are evaluated by the accuracy (mIoU) and the latency.

###  4.1 Experiments on Terracotta Warrior

We use Artec Eva [38] to collect 500 intact terracotta warrior models, and we
take 400 of the 500 models as the training set and 100 as the validation set.
Each model consists of about 2 million points which include xyz coordinates,
vertical normals, triangle meshes and RGB data. Before the experiments start,
we eliminate the triangle meshes and RGB data of the original models, and
remain xyz coordinates and vertical normals. Moreover, we uniformly sample the
above point clouds to 10,000 points thus as the experimental inputting. In
reality, the terracotta warriors are generally unearthed in the form of limb
fragments.

To better assist the restoration of terracotta warriors, we split the point
cloud into six pieces both in SRG and K-Means. We also set the number of
iterations $T$ as 2000 in each method(PointNet[19], PointNet2[20], DG-CNN[3],
PointHop++[39], ECC[40]) for every point cloud. Unlike the supervised problem,
our unsupervised method solves two sub-problems: prediction of cluster labels
with fixed network parameters and training of network with predicted labels.
The former sub-problem is solved with Section 3.3. The latter sub-problem is
solved with back-propagation.

Method |  005413 |  005420 |  005422 |  005423 |  005427 |  005455
---|---|---|---|---|---|---
SRG-DGCNN |  |  |  |  |  |
SRG-Pointnet2 |  |  |  |  |  |
SRG-Pointnet |  |  |  |  |  |
K-means-DGCNN |  |  |  |  |  |
K-means-Pointnet2 |  |  |  |  |  |
Kmeans-Pointnet |  |  |  |  |  |
SRG-Net |  |  |  |  |  |
Table 1: Visualization Results of Different Methods on Terracotta Warrior

We select SGD as optimizer, choose $lr=0.005$, and set the momentum factor to
0.1. To make fair comparison in the experiments, we combine PointNet[19],
PointNet2[20], DG-CNN[3], PointHop++[39] with the seed region growing method
to have a similar structure of SRG-Net. In addition, the seed region growing
is also compared with K-means. Comparison of different methods is shown in
Table 2, and the visualization results of different methods are presented in
Table 1. As is shown in Table 1, compared with k-means methods(K-means-DGCNN,
K-means-Pointnet2, K-means-Pointnet), our seed-region-growing method can get
roughly correct segmentation results. In contrast, k-means methods get the
wrong segmentation results. Compared with SRG-DGCNN, SRG-PointNet2, SRG-
PointNet, our SRG-Net can get more accurate segmentation results.

As shown in Table 2, SRG-Net outperforms SRG-DGCNN by increasing more than
5.5% of accuracy with consuming about 50% of SRG-DGCNN latency. Our method
also outperforms SRG-PointHop++ and SRG-ECC by a large margin(6.1%, 8.2%) with
about 72.2% and 70.3% latency. Compared with K-means methods, SRG-Net improves
SRG-ECC and SRG-DGCNN by more than 10% in mIoU, and it also outperforms
Kmeans-PointNet2 and K-means&PointNet by a large margin.

Method |  Accuracy(%) |  Latency(ms)
---|---|---
SRG-DGCNN |  78.32 |  37.26
SRG-PointHop++ |  77.94 |  30.62
SRG-ECC |  76.32 |  31.45
SRG-Pointnet |  72.03 |  13.35
SRG-Pointnet2 |  70.55 |  24.10
K-means-DGCNN |  70.64 |  36.30
K-means-Pointnet2 |  52.33 |  25.38
K-means-Pointnet |  53.94 |  13.47
SRG-Net |  82.63 |  22.11
Table 2: Comparison of Different Methods

Ranking the results of different methods in Table 2, we can draw the following
conclusions:

  1. 1.

Compared with SRG-DGCNN, SRG-PointNet, SRG-PointNet2, our network can enhance
the accuracy of learning with less latency.

  2. 2.

Compared with Kmeans-DGCNN, Kmeans-PointNet, Kmeans-PointNet2, our customized
SRG method can achieve better results compared with k-means.

  3. 3.

In general, SRG-Net has obvious advantages in accuracy and latency.

###  4.2 Experiments on ShapeNet

In this section, we perform experiments on ShapeNet to evaluate the robustness
of our SRG-Net method. ShapeNet contains 16,881 objects from 16 categories. We
set the number of the parts according to different categories separately. All
models were down-sampled to 2,048 points. The evaluation metric is mean
intersection-over-union (mIoU).

Our quantitative experiment results are shown in Table 3. As in Table 3, SRG-
Net outperforms all previous models. SRG-Net improves the overall accuracy of
SRG-DGCNN by 8.2% and even larger overhead compared with SRG-PointNet2 and
SRG-PointNet. Especially, our method outperforms SRG-DGCNN on all kinds of
categories, increasing 5% accuracy on knife. Overall, our method achieves
better accuracy on ShapeNet compared with other methods.

Figure 5: SRG-Net Segmentation Results of ShapeNet

Some visualization results are shown in Fig. 5. As is shown in Fig. 5, SRG-Net
achieves good results on bag, knife, motorbike, and achieves quite good
results on airplane, earphone and laptop.

Method |  Airplane |  Bag |  Earphone |  Knife |  Laptop |  Bike |  Mug |  OA
---|---|---|---|---|---|---|---|---
SRG-DGCNN |  67.7393 |  61.2568 |  56.0633 |  77.0035 |  75.8854 |  70.7194 |  59.3024 |  72.6273
SRG-Pointnet2 |  61.1865 |  54.4515 |  51.6673 |  71.9515 |  69.9074 |  64.3308 |  53.8897 |  66.4712
SRG-Pointnet |  57.1590 |  51.0517 |  47.2101 |  67.9229 |  66.4234 |  61.2438 |  49.1265 |  61.4242
SRG-Net |  73.0493 |  67.0935 |  62.4256 |  83.2430 |  82.6076 |  76.7677 |  65.8810 |  78.1954
Table 3: Accuracy(%) of Different Methods on ShapeNet

###  4.3 Ablation Study

In order to show the influence of different modules and epochs in our method,
we conduct ablation studies on our terracotta warrior dataset, which are
described in Section 3.2, which are evaluated by overall accuracy and mIoU.

The influences of different modules. As shown in Table 4, The results of
version without the graph convolution (A, row 1) show that the network not
working well in learning the topological features of the local neighborhood of
the point cloud. The results of version without edge convolution (B, row 2)
demonstrate that our method without edge convolution will cause the network
unable to understand the relationship between points well. The results of
version without refinement operation (C, row 3) reveal that the pipeline
cannot set tags reasonably based on point cloud content because the number of
unique cluster labels should be adaptive to context. The results show that the
refinement operation increase 15.6% on the accuracy of SRG-Net.

The influences of different epochs. To visualize the influence of different
epochs, we set the number of epochs to 2500 and set the segmentation number to
6. The segmentation results of different iterations in one terracotta warrior
model are shown in Fig. 6.

Method |  mIoU(%) |  OA(%)
---|---|---
A |  77.71 |  80.17
B |  74.18 |  76.51
C |  71.46 |  73.48
Table 4: Ablation Study

(a) iter 250

(b) iter 500

(c) iter 750

(d) iter 1000

(e) iter 1250

(f) iter 1500

(g) iter 1750

(h) iter 2000

Figure 6: Results of Different Iterations in SRG-Net

##  5 Conclusion

This paper provides an end-to-end model called SRG-Net for unsupervised
segmentation on the terracotta warrior point cloud. Our method aims at
assisting the archaeologists in speeding up the repairing process of
terracotta warriors. Firstly, we design a novel seed-region-growing method to
pre-segment point clouds with coordinates and normal value of the 3D
terracotta warrior point clouds. Based on the pre-segmentation process, we
proposed a CNN inspired by the dynamic graph in DG-CNN and auto-encoder in
FoldingNet to learn the local and global characteristics of the 3D point cloud
better. At last, we propose a refinement process and append it after the CNN
process. Finally, we evaluate our method on the terracotta warrior data, and
we outperform the state-of-the-art methods with higher accuracy and less
latency. Besides, we also conduct experiments on the ShapeNet, which
demonstrates our approach is also useful for the human body and standard
object part segmentation. Our work still has some limitations. The number of
points from the hand of the terracotta warrior is relatively small, so it is
difficult for SRG-Net to learn the characteristics of hands. We will try to
work on this problem in the future. We hope our work can be helpful to the
research of terracotta warriors in archaeology and point cloud work of other
researchers.

##  6 Acknowledgments

This work is equally and mainly supported by the National Key Research and
Development Program of China (No. 2019YFC1521102, No. 2019YFC1521103) and the
Key Research and Development Program of Shaanxi Province (No. 2020KW-068).
Besides, this work is also partly supported by the Key Research and
Development Program of Shaanxi Province (No.2019GY-215, No.2019ZDLSF07-02,
No.2019ZDLGY10-01), the Major research and development project of Qinghai(No.
2020-SF-143), the National Natural Science Foundation of China under Grant
(No.61701403,No.61731015) and the Young Talent Support Program of the Shaanxi
Association for Science and Technology under Grant (No.20190107).

## Disclosures

The authors declare no conflicts of interest.

## References

  * [1] A. Tao, “Unsupervised point cloud reconstruction for classific feature learning,” https://github.com/AnTao97/UnsupervisedPointCloudReconstruction (2020).
  * [2] Y. Yang, C. Feng, Y. Shen, and D. Tian, “Foldingnet: Point cloud auto-encoder via deep grid deformation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,_ (2018), pp. 206–215.
  * [3] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon, “Dynamic graph cnn for learning on point clouds,” Acm Transactions On Graphics (tog) 38, 1–12 (2019).
  * [4] J. B. Kruskal, “On the shortest spanning subtree of a graph and the traveling salesman problem,” Proceedings of the American Mathematical society 7, 48–50 (1956).
  * [5] H. Lee, P. Pham, Y. Largman, and A. Ng, “Unsupervised feature learning for audio classification using convolutional deep belief networks,” Advances in neural information processing systems 22, 1096–1104 (2009).
  * [6] Q. V. Le, “Building high-level features using large scale unsupervised learning,” in _2013 IEEE international conference on acoustics, speech and signal processing,_ (IEEE, 2013), pp. 8595–8598.
  * [7] H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng, “Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations,” in _Proceedings of the 26th annual international conference on machine learning,_ (2009), pp. 609–616.
  * [8] A. Kanezaki, “Unsupervised image segmentation by backpropagation,” in _2018 IEEE international conference on acoustics, speech and signal processing (ICASSP),_ (IEEE, 2018), pp. 1543–1547.
  * [9] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk, “Slic superpixels compared to state-of-the-art superpixel methods,” IEEE transactions on pattern analysis and machine intelligence 34, 2274–2282 (2012).
  * [10] W. Kim, A. Kanezaki, and M. Tanaka, “Unsupervised learning of image segmentation based on differentiable feature clustering,” IEEE Transactions on Image Processing 29, 8055–8068 (2020).
  * [11] F. J. Lawin, M. Danelljan, P. Tosteberg, G. Bhat, F. S. Khan, and M. Felsberg, “Deep projective 3d semantic segmentation,” in _International Conference on Computer Analysis of Images and Patterns,_ (Springer, 2017), pp. 95–107.
  * [12] A. Boulch, B. Le Saux, and N. Audebert, “Unstructured point cloud semantic labeling using deep segmentation networks.” 3DOR 2, 7 (2017).
  * [13] B. Wu, A. Wan, X. Yue, and K. Keutzer, “Squeezeseg: Convolutional neural nets with recurrent crf for real-time road-object segmentation from 3d lidar point cloud,” in _2018 IEEE International Conference on Robotics and Automation (ICRA),_ (IEEE, 2018), pp. 1887–1893.
  * [14] A. Milioto, I. Vizzo, J. Behley, and C. Stachniss, “Rangenet++: Fast and accurate lidar semantic segmentation,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS),_ (IEEE, 2019), pp. 4213–4220.
  * [15] B. Graham, M. Engelcke, and L. Van Der Maaten, “3d semantic segmentation with submanifold sparse convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition,_ (2018), pp. 9224–9232.
  * [16] A. Dai and M. Nießner, “3dmv: Joint 3d-multi-view prediction for 3d semantic scene segmentation,” in _Proceedings of the European Conference on Computer Vision (ECCV),_ (2018), pp. 452–468.
  * [17] M. Jaritz, J. Gu, and H. Su, “Multi-view pointnet for 3d scene understanding,” in _Proceedings of the IEEE International Conference on Computer Vision Workshops,_ (2019), pp. 0–0.
  * [18] S. Wang, S. Suo, W.-C. Ma, A. Pokrovsky, and R. Urtasun, “Deep parametric continuous convolutional neural networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,_ (2018), pp. 2589–2597.
  * [19] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition,_ (2017), pp. 652–660.
  * [20] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” in _Advances in neural information processing systems,_ (2017), pp. 5099–5108.
  * [21] M. Jiang, Y. Wu, T. Zhao, Z. Zhao, and C. Lu, “Pointsift: A sift-like network module for 3d point cloud semantic segmentation,” arXiv preprint arXiv:1807.00652 (2018).
  * [22] H. Zhao, L. Jiang, C.-W. Fu, and J. Jia, “Pointweb: Enhancing local neighborhood features for point cloud processing,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,_ (2019), pp. 5565–5573.
  * [23] W. Wu, Z. Qi, and L. Fuxin, “Pointconv: Deep convolutional networks on 3d point clouds,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,_ (2019), pp. 9621–9630.
  * [24] Y. Li, R. Bu, M. Sun, W. Wu, X. Di, and B. Chen, “Pointcnn: Convolution on $\mathcal{X}$-transformed points,” (2018).
  * [25] S. Wang, S. Suo, W.-C. Ma, A. Pokrovsky, and R. Urtasun, “Deep parametric continuous convolutional neural networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,_ (2018), pp. 2589–2597.
  * [26] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, “Graph attention networks,” arXiv preprint arXiv:1710.10903 (2017).
  * [27] Z. Wang and F. Lu, “Voxsegnet: Volumetric cnns for semantic part segmentation of 3d shapes,” IEEE transactions on visualization and computer graphics (2019).
  * [28] L. Yi, H. Su, X. Guo, and L. J. Guibas, “Syncspeccnn: Synchronized spectral cnn for 3d shape segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,_ (2017), pp. 2282–2290.
  * [29] Y. Liu and Y. Xiong, “Automatic segmentation of unorganized noisy point clouds based on the gaussian map,” Computer-Aided Design 40, 576–594 (2008).
  * [30] L. Di Angelo and P. Di Stefano, “Geometric segmentation of 3d scanned surfaces,” Computer-Aided Design 62, 44–56 (2015).
  * [31] P. Benkő and T. Várady, “Segmentation methods for smooth point regions of conventional engineering objects,” Computer-Aided Design 36, 511–523 (2004).
  * [32] Z. Chen, K. Yin, M. Fisher, S. Chaudhuri, and H. Zhang, “Bae-net: Branched autoencoder for shape co-segmentation,” in _Proceedings of the IEEE International Conference on Computer Vision,_ (2019), pp. 8490–8499.
  * [33] D. OuYang and H.-Y. Feng, “On the normal vector estimation for point cloud data from smooth surfaces,” Computer-Aided Design 37, 1071–1079 (2005).
  * [34] J. Zhou, H. Huang, B. Liu, and X. Liu, “Normal estimation for 3d point clouds via local plane constraint and multi-scale selection,” Computer-Aided Design 129, 102916 (2020).
  * [35] Y. Wang, H.-Y. Feng, F.-É. Delorme, and S. Engin, “An adaptive normal estimation method for scanned point clouds with sharp features,” Computer-Aided Design 45, 1333–1348 (2013).
  * [36] R. B. Rusu and S. Cousins, “3d is here: Point cloud library (pcl),” in _2011 IEEE international conference on robotics and automation,_ (IEEE, 2011), pp. 1–4.
  * [37] A. Kanezaki, “Unsupervised image segmentation by backpropagation,” in _2018 IEEE international conference on acoustics, speech and signal processing (ICASSP),_ (IEEE, 2018), pp. 1543–1547.
  * [38] A. Europe, “3d object scanner artec eva | best structured-light 3d scanning device,” .
  * [39] M. Zhang, Y. Wang, P. Kadam, S. Liu, and C.-C. J. Kuo, “Pointhop++: A lightweight learning model on point sets for 3d classification,” arXiv preprint arXiv:2002.03281 (2020).
  * [40] M. Simonovsky and N. Komodakis, “Dynamic edge-conditioned filters in convolutional neural networks on graphs,” in _CVPR,_ (2017).