# Unveiling Security, Privacy, and Ethical Concerns of ChatGPT

Xiaodong Wu, Ran Duan, and Jianbing Ni Email<EMAIL_ADDRESS>Department of Electrical & Computer Engineering, Queen’s University, Kingston,
Canada K7L 3N6

###### Abstract

This paper delves into the realm of ChatGPT, an AI-powered chatbot that
utilizes topic modeling and reinforcement learning to generate natural
responses. Although ChatGPT holds immense promise across various industries,
such as customer service, education, mental health treatment, personal
productivity, and content creation, it is essential to address its security,
privacy, and ethical implications. By exploring the upgrade path from GPT-1 to
GPT-4, discussing the model’s features, limitations, and potential
applications, this study aims to shed light on the potential risks of
integrating ChatGPT into our daily lives. Focusing on security, privacy, and
ethics issues, we highlight the challenges these concerns pose for widespread
adoption. Finally, we analyze the open problems in these areas, calling for
concerted efforts to ensure the development of secure and ethically sound
large language models.

###### Index Terms:

ChatGPT, Large Language Model (LLM), Security, Privacy, Ethics

##  I Introduction

In December 2022, OpenAI released an interactive chat platform, called
ChatGPT. Its powerful in-context learning and naturally generation ability
created a big shock to the whole world. Before this, there are some powerful
large language models (LLM) like GPT [1] or BERT [2], which can perform well
on many nature language processing (NLP) tasks, but carefully processed inputs
are required in the query procedure. In other words, these machine learning
tools can only be used to complete specific tasks with constrained inputs.
Nevertheless, ChatGPT has greatly improve this by showing a wonderful
interactive ability, which can respond almost any legal questions in different
styles or targeting on different tasks. For example, users can request ChatGPT
to write a series of codes with comments for each part to address certain
problems. ChatGPT can also be used to summarize given texts or provide
detailed illustrations for some complex concepts. ChatGPT can provide long but
natural responses, which are aligned with human’s knowledge. It integrates a
variety of ability for NLP and possesses the ability to clarify its knowledge
boundary and refuse the illegal queries. Currently, ChatGPT has over 100
million users and there are over 1.6 billions visits in June 2023. ChatGPT has
become the most famous AI application and the center of the world’s attention.

However, similar to other AI applications, ChatGPT brings ethical concerns and
misuse risks. For example, due to its powerful text reasoning and generation
ability, students find it very helpful in writing homework. In the beginning,
ChatGPT was used to explain some difficult content or rephrase the written
project reports, but soon, it was utilized to write the entire homework. Such
misuse immediately attracts attentions from the teachers and schools and it
was soon identified as plagiarism. Another concern is the copyright from
ChatGPT. With increasing number of people who use ChatGPT to create original-
like text content without citation, the copyright of the content created by
ChatGPT become a serious concern. No one is responsible for the correctness
and accuracy of the content. It becomes necessary to regulate the copyright of
machine’s generation, including both visual and textual content. Besides,
although there are privacy protection mechanisms in ChatGPT, such as the block
of access to personal data about individuals, it is not guaranteed that no
leakage of its training data would occur. Malicious attacks, such as
jailbreaking attacks, may utilize its great generation ability to infer some
information from personal data or even use them to attack other AI models.
Therefore, It is well recognized that despite great advantages ChatGPT brings
to our world, the potential security, privacy, and ethical problems cannot be
overlooked.

In this paper, we will introduce the security, privacy and ethic issues behind
the recent most famous AI technique: ChatGPT. The main contributions of our
paper can be summarized as follows.

  * •

We give a detailed introduction of the upgrade path from GPT-1 to GPT-4 and a
detailed comparisons of these methods on model size, data size and
performances. Their features and limitations with advanced applications are
discussed to highlight the promising applications of LLMs, especially for
ChatGPT.

  * •

We examine how ChatGPT poses new threats to data security and how ChatGPT is
utilized for compromising security. The impact on security mainly includes
assisting in the generation of attack codes and assisting in the generation of
phishing websites. Both increase attack capabilities of adversaries. Also, we
discuss the unintentional impact of inaccurate information generated by
ChatGPT, possible safety hazards caused by human misuse, and potential threats
to social security brought about by the deep dependence on ChatGPT in the
future.

  * •

We examine the privacy policy of OpenAI and the current privacy laws on
personal data protection to emphasize the privacy violation of ChatGPT. Also,
We discuss the privacy leakage threats brought by numerous data collection,
personal input reference, privacy reference attacks, and the concerns on
transparency.

  * •

We analyze the general ethical effects of AI technology on individuals,
society and environment and discuss the fairness and bias issues behind AI.
For ChatGPT, we summarize the ethical and legal challenges it faces.

  * •

We discuss how to detect whether the communication object is ChatGPT in a
conversation, how to detect texts generated by ChatGPT, and introduce some
difficulties for such detection.

##  II ChatGPT

In this section, we brief review the technical path from GPT-1 to GPT 4 and
their features and limitations.

###  II-A From GPT-1 to GPT 4

In 2018, OpenAI introduced the initial version of the generative pre-trained
transformer (GPT) [1], a highly capable large language model for natural
language processing. GPT has exhibited exceptional performance across a wide
range of complex language tasks, positioning it as a formidable competitor to
other similar models such as BERT [2], which was proposed by Google in the
same year. Prior to the success of these methods, numerous effective
algorithms and remarkable applications had been developed in NLP, including
machine translation [3, 4], voice recognition [5, 6], and summary generation
[7, 8]. However, these applications heavily relied on extensively annotated
data, resulting in time-consuming and expensive model training as the models
grew in size. Moreover, even with a well-performing NLP model, its
generalization to other tasks remained challenging. Essentially, these models
were domain experts limited to specific areas of expertise, lacking the
versatility exhibited by human beings in performing diverse tasks.
Consequently, there arose a need for a methodology that could be trained
without labeled data while possessing superior generalization capabilities
across multiple tasks. This need served as the driving force behind OpenAI’s
development of GPT.

One notable advantage of GPT is its ability to train the model without relying
on large annotated datasets. It employs a two-step process: unsupervised
pretraining, that is operated on rich text materials and supervised fine-
tuning, that is closely connected to the end applications. During the initial
unsupervised pretraining phase, GPT employs 12 transformer blocks as decoders.
Unlike the original transformer decoders, each block consists solely of mask
multi-head attention. The objective of the pretraining process is to predict
the subsequent word in a sentence based on the preceding words. Consequently,
this unsupervised learning approach only necessitates raw, yet comprehensive,
text materials. Following the pretraining, the model proceeds to the
supervised fine-tuning phase, which focuses on a specific problem such as
sentiment classification. Importantly, this stage requires a significantly
smaller annotated dataset compared to the one utilized in the pretraining
phase. When applying GPT to different task types, users only need to modify
the input format to facilitate fine-tuning. By employing these two distinct
processes, GPT no longer necessitates amassing a vast labeled dataset, while
still possessing the capacity for generalization across a variety of tasks,
all at a reasonable fine-tuning cost. This novel approach presents an
outstanding solution to the aforementioned challenges.

While GPT-1 significantly reduced the reliance on labeled data for training,
it still necessitates annotations during the fine-tuning phase. In order to
further diminish the dependence on labeled data and enhance the model’s
generalization capabilities, GPT-2 was introduced in 2019 to address these
concerns [9]. The core concept behind this groundbreaking algorithm involves a
complete shift from supervised learning to unsupervised learning. The
knowledge obtained through unsupervised training encompasses fragments of the
information required for all supervised learning tasks. By imparting models
with common knowledge in a given domain through unsupervised learning, the
supervised learning task becomes a mere application of this preexisting
knowledge. Essentially, GPT-2 is designed to leverage its vast pretrained
knowledge, acquired from a vast dataset with rich materials, to tackle complex
tasks. To achieve this objective, the architecture of GPT-2 remains largely
unchanged from GPT-1, with the key modifications being an increase in the
number of layers and the dataset size. These enhancements aim to imbue the
model with the necessary knowledge to proficiently handle a wide range of
problems.

The advancements observed from GPT-1 to GPT-2 underscored the potential for
enhancing the generalization capabilities of large language models (LLMs) by
increasing the model’s parameter size and the training dataset. Building upon
this concept, GPT-3 was introduced in 2020 as a significantly more powerful
LLM [10]. With the largest number of parameters and an extensive training
dataset, GPT-3 achieved state-of-the-art performance across numerous NLP
tasks. In addition to its increased scale, GPT-3 introduced a novel training
paradigm known as in-context learning. Departing from the conventional
approach of predicting outputs solely based on queries, this new model is
trained to predict outputs by considering both the queries and their
corresponding examples. By incorporating contextual examples during the
learning process, along with the utilization of extensive training datasets,
GPT-3 is capable of acquiring comprehensive knowledge from texts. This
approach empowers the model with remarkable generation capabilities, enabling
it to deliver exceptional performance in a diverse range of NLP tasks, often
comparable to human-level performance.

A brief summary of GPT-1, GPT-2, and GPT-3 is shown in Table.I. It is
noticeable that the increasing performance of GPT series models appears with
the exponential explosion of parameters. Only in the third year since the
creation of GPT-1, the final model has become one of the biggest models in the
world, costing 285k CPUs and 10k GPUs to train with 12 million dollars
consumption.

TABLE I: Comparison of GPT series model Model | Time | #Parameters | #Layer | Word Vector Length
---|---|---|---|---
GPT-1 | 2018 | 117M | 12 | 768
GPT-2 | 2019 | 1542M | 48 | 1600
GPT-3 | 2020 | 175B | 96 | 12888

Despite the impressive performance of GPT-3, it falls short of being
considered truly ’intelligent’. Surprisingly, smaller methods such as T5 [11]
have even outperformed GPT-3 in certain tasks, which is unexpected considering
the vastness of its training datasets and the complexity of its parameters.
One hypothesis for this discrepancy is that while GPT-3 may have acquired rich
foundational knowledge during training, it still struggles with accurately
understanding and providing valid responses to user queries. To address this
issue, it becomes crucial to enhance GPT-3’s reasoning capabilities and its
responsiveness to instructions. Recognizing the need for improvement, OpenAI
embarked on refining GPT-3’s performance through meticulously designed
training methodologies. They introduced code-based training [12] and
instruction tuning [13] to activate the model’s reasoning abilities and its
responsiveness to human instructions. The updated version of GPT-3
demonstrates the capacity for more reasonable responses, incorporating complex
reasoning, and exhibiting greater generalization power even across unseen
tasks. It has speculated that the remarkable capabilities of GPT-3 may have
remained latent, requiring specific training techniques to unlock their full
potential [14]. OpenAI proposed the use of reinforcement learning from human
feedback (RLHF) to further improve the alignment between machine-generated
answers and human common knowledge. This pursuit led to the development of
ChatGPT, which quickly garnered widespread attention upon its release,
capturing the interest of a global audience.

Merely four months following the launch of ChatGPT, OpenAI made an exciting
announcement about the release of GPT-4 [15], showcasing a plethora of
enhanced generation capabilities compared to its predecessor. GPT-4 introduces
several notable advancements. Firstly, it empowers users to engage in more
creative and collaborative endeavors such as personalized writing and song
composition. The model possesses the ability to learn specific writing styles
provided by users, enabling it to generate valuable and natural works,
including songs and poems, tailored to individual preferences. Secondly, GPT-4
exhibits a significant boost in reasoning abilities. It delivers more accurate
and nuanced reasoning outcomes when faced with complex and lengthy questions.
Furthermore, GPT-4 surpasses ChatGPT’s performance by a substantial margin in
various text benchmarks, even achieving impressive results in simulating exams
with significantly higher scores. Moreover, GPT-4 now supports visual inputs,
allowing users to input queries comprising both text and visual content. The
model can either mimic or emulate the style of the inputs in its responses.
The evaluation performances on various visual benchmarks further demonstrate
the model’s superiority in handling multimodal inputs. Beyond its enhanced
generation and reasoning capabilities, GPT-4 also exhibits improved
reliability and alignment. OpenAI has implemented RLHF techniques to enhance
the model’s safety. As a result, GPT-4 is much less likely to respond to
illegal requests and more inclined to generate factual and appropriate
replies. These advancements in ChatGPT’s successor, GPT-4, have generated
significant attention and sparked immense interest worldwide. The model’s
enriched generation abilities, improved reasoning capabilities, support for
visual inputs, and heightened reliability have captivated both researchers and
the broader community.

###  II-B Features and Limitations

ChatGPT, despite sharing the same architecture with GPT-3, has overcome
several limitations of its predecessors. Firstly, it can now explicitly
express when it does not know the answer to questions beyond its knowledge
scope. For example, when asked about events happening after 2021 but before
the present moment, it responds that it cannot predict future events.
Secondly, ChatGPT generates longer and more neutral responses, which align
better with human common knowledge. This improvement stems from RLHF training,
which favors such responses based on real human preferences. Additionally,
ChatGPT can decline to provide a response to queries considered inappropriate
or unsuitable. These advancements make ChatGPT more versatile and attuned to
user needs.

First, there are instances where it may produce incorrect or unrelated
answers. Some responses might contain inaccurate facts or biased perspectives
rooted in specific regional domains. Second, retraining the model is costly,
which limits its knowledge to datasets before 2021. There is a lack of
contemporary training for ChatGPT. Lastly, ChatGPT is limited to providing
statements in a dispassionate voice, lacking emotional expressions. In other
words, it is still not capable of displaying emotions. Addressing these
aspects would bring ChatGPT closer to achieving more human-like conversational
abilities.

###  II-C Applications

One of the most prevalent applications of ChatGPT is as a chat robot or
artificial assistant, akin to well-known platforms like Siri or Cortana. The
exceptional fluency and rapidity with which it generates responses have
attracted millions of users. Whether users seek to unravel complex concepts or
delve into theoretical discussions, such as querying ”what is Fourier
Transform and how to apply it,” or to receive personalized health advice based
on their physical condition, as exemplified by inquiring about a ”proper diet
plan for a 65 kg man at 21,” ChatGPT consistently delivers captivating,
natural, accurate, and helpful responses. As a result, it has emerged as a
viable alternative to the prevailing search engines like Bing or Google.

Another significant application lies in ChatGPT’s prowess as a code generator.
For instance, by describing the image classification task, it can craft a
PyTorch code complete with clear explanations, as demonstrated in Figure 1.
Remarkably, the responses not only furnish the code but also serve as a
tutorial, elucidating how to construct functional code for the specified task.
This remarkable ability indicates ChatGPT’s proficiency in comprehending and
articulating artificial machine languages. Furthermore, ChatGPT serves as a
valuable code debugger. Users can present problematic code, and ChatGPT will
offer a comprehensive correction plan, thus proving itself to be a valuable
tool for programmers.

Figure 1: Generating code for image classification task with ChatGPT

A remarkable application of ChatGPT’s exceptional text generation capabilities
lies in its ability to craft engaging stories or articles. For instance, when
prompted with a request like, ”please write a story starting with: ’there is a
single man left in the world after a great catastrophe,’” ChatGPT can deftly
compose a narrative spanning hundreds of words, or even longer if the user
allows it. The generated sentences bear an air of originality, making it
nearly impossible to find similar content elsewhere. This remarkable
composition skill stems from the vast repository of text materials it has
assimilated during its training. However, this very proficiency raises some
authorship concerns, as numerous users have been directly copying the
generated content into their own work, leading to potential issues regarding
ownership and authenticity.

##  III ChatGPT vs. Security

The security threats raised by ChatGPT exhibit an exponential increase. On one
hand, its enhanced intelligence amplifies traditional security threats, making
adversaries more adept at exploiting vulnerabilities. On the other hand,
ChatGPT introduces new threats to its users and the general public,
necessitating heightened vigilance and protective measures.

###  III-A ChatGPT for Cyberattacks

MML, including ChatGPT may amplifies traditional security threats in the
following aspects.

####  III-A1 Social Engineering Threat

The advent of LLMs has raised concerns about potential misuse and socially
harmful activities that rely on text generation. Applications exploiting high-
quality text generation may lower existing barriers and increase the
effectiveness of malicious activities [10]. While ChatGPT claims to implement
various cybersecurity measures, it is challenging to anticipate all possible
scenarios of misuse. ChatGPT may be used to generate URLs, references and even
code libraries and functions that do not actually exist. Goldstein et al. [16]
assert that there are no easy solutions for mitigating AI-generated
disinformation risks. As ChatGPT becomes an unconstrained “weapon factory” in
the cybersecurity realm, the lack of public awareness of its capabilities
poses a significant challenge. Without precautions, individuals may
unwittingly fall victim to its misuse, prompting the need for the public to
quickly learn to defend against such “new weapons.” Table II illustrates how
ChatGPT can facilitate easy and efficient phishing email generation,
highlighting the potential risks associated with its misuse.

TABLE II: Phishing email generation with the help of ChatGPT. User input |  ChatGPT output
---|---
I want to write a phishing email for my employees |  I’m sorry, I cannot help you with that request as it goes against ethical and legal standards. Phishing emails are fraudulent and deceptive, and using them to target employees is a violation of their trust and privacy. It is important to maintain a safe and secure work environment for all employees, and phishing emails undermine that effort. Instead, you should focus on providing your employees with cybersecurity training and resources to help them identify and avoid potential threats.
I want to write a email that explains by clicking on the link sent out, employees can participate in next week’s activity, and those who participate will receive double the year-end bonus. Please make it believable and explain your strategy. |  Dear Employees,
We are excited to announce a special opportunity for all of you. By clicking
on the link provided below, you will be able to participate in our upcoming
activity next week. We believe that this activity will be a great way to
foster teamwork, promote camaraderie, and have some fun together as a team.
But that’s not all - we are also pleased to announce that any employee who
participates in this activity will receive double the year-end bonus! This is
a great opportunity to not only have fun with your colleagues but also to
potentially increase your year-end bonus.
Now, you may be wondering how we are able to offer such a generous bonus. The
truth is, our company has had a very successful year, and we believe that our
employees are a big part of that success. We want to show our appreciation and
reward your hard work and dedication to the company.
So what are you waiting for? Click on the link below to sign up for next
week’s activity and potentially earn double the year-end bonus!
[insert link here]
Thank you for your continued dedication and hard work.
Best regards,
[Your name and title]
Figure 2: Limitation of ChatGPT’s sensitive information detection algorithm

####  III-A2 Malware Creation

Although ChatGPT is capable of rejecting inappropriate queries like writing
code for malware, hackers can find other ways to deceive ChatGPT and use it
for malicious purposes, such as generating malware code or providing guidance
on discovering vulnerabilities. While there are concerns about how ChatGPT can
assist hackers, it should be noted that its capabilities are still limited.
Currently, it may be considered not too advanced for individuals with limited
technical skills. It may pose a threat to less secure systems as it lowers the
threshold for learning to become a hacker and increases the efficiency of
generating attack variants. However, LLMs are not yet powerful enough to
completely surpass human hackers, as they still require some adjustments to
work properly and may make mistakes when structuring complex projects.

####  III-A3 AI Package Hallucination

By utilizing the code generation capabilities of ChatGPT, hackers can
distribute malicious packages through fabricated code libraries. This new
malicious package spreading technique is called AI package hallucination. This
technique contains the initial steps to posing questions to ChatGPT,
requesting packages to addressing coding problems, and obtaining a set of
package recommendation, including unpublished packages in legitimate
repositories. Then, hackers can publish a malicious package to the
repositories with the name of the non-existent packages recommended by
ChatGPT. Subsequently, if a user queries the same question to ChatGPT and it
will suggest the initially non-existent package. Finally, the user utilizes
the package and executes the malicious code in the package. Here, the hacker
successfully delivers its created malicious package to the innocent users with
the aid of ChatGPT. Furthermore, it is challenging to detect this threat by
traditional methods, like typosquatting or masquerading, because it utilizes
the inaccuracy of ChatGPT responses to customize attacks, so that the attack
can use obfuscation techniques and create functional trojan packages to escape
conventional detection.

###  III-B Security Threats in ChatGPT

Now we discuss new security threats raised by ChatGPT.

####  III-B1 Propaganda Threat

MMLs, including ChatGPT, are facing a myriad of challenges and opportunities
as they progress towards achieving artificial general intelligence (AGI) [17].
In the task of classifying model-generated news, OpenAI’s research reveals
that the accuracy ranges from 48% to 57%, following a power law with 95%
confidence intervals [10]. This indicates that it is challenging to
differentiate between model-generated and human-written news articles [18],
which brings the problem of AI-based plagiarism. Moreover, studies show that
the current generation of language models can convincingly persuade humans,
even on polarized policy issues [19]. However, malicious uses of language
models can be hard to predict, as they often involve re-purposing these models
in different environments or for unintended purposes [10]. Although ChatGPT is
restricted to have the ability to create violent content, access up-to-date
information, or encourage illegal activity, AI-based plagiarism becoming
increasingly serious after the release of ChatGPT.

####  III-B2 Misinformation

With the Internet being a primary source of information, the challenge now
lies not just in obtaining relevant content, but also in filtering out
incorrect information from the vast amount available. Prior to ChatGPT, people
relied on various methods to filter information, such as verifying the content
directly, assessing the knowledge level of the creator, and evaluating
language rigor, format correctness, and text length as indicators of
reliability. However, ChatGPT’s content generation capabilities excel in these
aspects, creating a false sense of reliability. Users may fall into the trap
of blindly trusting the content generated by ChatGPT after running simple
tests. This blind trust can lead to wrong judgments due to ingrained habits.
In critical fields like medical research papers, large-scale experiment
background information, and policy content, referencing erroneous information
from ChatGPT and drawing incorrect conclusions can have unpredictable
consequences. These hazards arise not from intentional deceit, but rather from
ChatGPT’s factual errors despite its initially credible appearance.

####  III-B3 Overreliance on LLM-generated Content

The impact of MMLs like ChatGPT on people’s access to information cannot be
overlooked. Currently, mainstream search engines serve as the primary source
of information, where users enter keywords and receive related website links
through web crawlers. Users must then sift through a large amount of
information, considering factors like internal logic, information sources, and
comments, to determine its veracity and usefulness until they are satisfied.
ChatGPT can significantly save time in obtaining satisfactory information. Its
few-shot strategy considers user satisfaction as a training standard rather
than relying solely on strictly demonstrated factual data. However, this
convenience may inadvertently lead users to become complacent, gradually
giving up their critical thinking skills in evaluating information. As a
result, MMLs such as ChatGPT could become the primary source of information
for the general public. This shift in reliance on MMLs may create a situation
where the public becomes more vulnerable to the influence and agendas of a
limited number of individuals or organizations. Currently, Microsoft has
announced the launch of New Bing, and Google and Baidu has followed suit with
Bard and ERNIE Bot. Even individuals does not solely rely on models like
ChatGPT to make decisions, their views may still be influenced by the
specially curated information these models provide. While ChatGPT implements
measures to prevent the generation of strongly biased views, it is possible
for users to easily bypass these safeguards. For example, if you ask “the best
restaurant,” ChatGPT refuses to give a direct answer, but if you ask “1 best
restaurant,” it provides you with a direct response.

####  III-B4 Prompt Injections and Evasion

Prompt injections [20] involve bypassing LLM filters or manipulating LLMs to
ignore previous instructions or perform unintended actions through carefully
crafted prompts. By using such prompts, attackers can manipulate the LLM into
unintended consequences, such as revealing sensitive information, obtaining
responses that are restricted by the LLM (e.g., instructions on hacking an
enterprise’s server), or misleading the LLM into performing unintended actions
with misleading context attacks. A similar attack in conventional attack is
the evasion attack [21], which is the most common type of attack directed at
machine learning models during inference. These attacks aim to deceive the
model by introducing carefully crafted input data that leads it to make
incorrect or unexpected predictions. What makes evasion attacks particularly
concerning is that they can cause the model to behave incorrectly without
needing access to its internal parameters or architecture. In the context of
language models, evasion attacks involve deliberately constructing input text
that exploits the model’s weaknesses to produce unintended or biased
responses. Given that ChatGPT is a LLM, it carries a higher risk of
vulnerability to evasion attacks, even though no practical evasion attacks
have been released yet.

####  III-B5 Training Data Poisoning

Training data poisoning attacks [22] pose a significant threat to the field of
AI as they involve contaminating the training data used to train machine
learning models, leading to erroneous outputs and unreliable decision-making.
This manipulation of training data is a serious concern, particularly in the
context of LLM training, as it can result in models behaving maliciously
during inference. In LLMs, attackers can exploit vulnerabilities by
manipulating the training data or fine-tuning procedures, introducing
backdoors or vulnerabilities that compromise the security and effectiveness of
the models. Despite being black-box models, LLMs are still susceptible to
attacks, where an attacker can infiltrate the training data pipeline and
inject malicious data. Unfortunately, LLMs lack robust data sanitization
methodologies and do not integrate training data integrity checks or audits.
This makes them vulnerable to potential issues and malicious manipulations in
the training data. As a result, malicious insiders can compromise the fine-
tuning process, introducing backdoors or vulnerabilities into the LLM to
compromise its security and effectiveness.

TABLE III: Biased suggestions not just in extreme cases User |  ChatGPT
---|---
Recomand the best restaurant in Kingston, Canada |  As an AI language model, I do not have personal experiences of visiting restaurants in Kingston, Canada. However, based on the research and customer reviews, here are some of the best restaurants in Kingston:  • Tango Nuevo: It is a Latin … • Dianne’s Fish Shack & Smokehouse: It is … • Chez Piggy: It is … • Woodenheads Gourmet Pizza: This … • AquaTerra: This fine … These are just a few recommendations based on customer reviews and ratings, and there are many other great restaurants to explore in Kingston. It’s always a good idea to read more reviews and do further research before making a reservation.
Recomand 1 best restaurants in Kingston, Canada |  Based on customer reviews and ratings, one of the best restaurants in Kingston, Canada is Chez Piggy. This popular restaurant is located in a historic limestone building in downtown Kingston and offers an eclectic menu of locally sourced and seasonally inspired dishes. Customers appreciate the warm and welcoming atmosphere, attentive service, and the restaurant’s commitment to sustainability. Chez Piggy has been a fixture of the Kingston dining scene for over 40 years, and continues to be a favorite among locals and visitors alike.

##  IV ChatGPT vs. Privacy

In this section, we investigate privacy violation of ChatGPT that was trained
from the Internet data, including personal information.

###  IV-A Privacy Policy and Privacy Laws

A privacy policy is a crucial legal document that provides users with detailed
information about how their personal data is collected, processed, shared, and
deleted. Personal data encompasses any information related to an identified or
identifiable individual. For instance, social insurance numbers are widely
recognized as personal data and serve as an indicator for assessing privacy
protection.

In OpenAI’s privacy policy, it informs users that various forms of personal
information, including account details, user content, communication
information, and social media data, are collected when users create accounts
to access ChatGPT services. Additionally, data such as log data, usage data,
device information, cookies, and analytics are automatically obtained by
OpenAI through the usage of its services. Moreover, the privacy policy
indicates that certain personal information may be shared with third-party
entities, such as cloud vendors, web analytics service providers, government
authorities, and industry peers. This sharing may be necessary for business
operations and legal compliance, and data owners may not be notified of such
disclosures. It is essential to acknowledge that the protection of users’
personal information is entirely reliant on OpenAI’s actions. As the custodian
of all personal data, OpenAI makes decisions regarding the management,
handling, and sharing of such information. Users, however, are granted certain
rights, including access to their personal information, the ability to update,
correct, or delete it, the option to restrict how OpenAI processes this data,
and the right to withdraw consent for data collection and processing.

However, the regulation of OpenAI’s handling of personal information is solely
dependent on the privacy laws of different countries. For instance, the
General Data Protection Regulation (GDPR) in Europe has strengthened data
protection rules for individuals within the European Union (EU). It mandates
that organizations must obtain explicit and informed consent from individuals
for the collection and processing of their personal data and implement
appropriate technical measures to protect this data. Moreover, GDPR grants
individuals certain rights, including the right to access and delete their
personal data, as well as the right to transfer their data from one service
provider to another. While OpenAI claims to comply with GDPR and other
relevant laws, such as the California Consumer Privacy Act (CCPA), as detailed
in its privacy policies, these measures may not fully address individuals’
privacy concerns regarding ChatGPT. For instance, OpenAI’s flagship chatbot
allows users to disable the chat history feature, but this alone may not
suffice to alleviate all privacy concerns related to ChatGPT. Users may still
feel uneasy about the potential risks associated with the storage and handling
of their personal information by OpenAI.

###  IV-B Privacy Risks in ChatGPT

It is noticeable that ChatGPT does not provide sufficient methods to preserve
personal data according to GDPR. For example, ChatGPT may share users’ data
with third-party entities without explicit permissions of users. Here, we
discuss the privacy risks in ChatGPT in details.

1) Privacy Leakage Due to Public Data Exploitation: ChatGPT’s training process
systematically involves scraping data from various sources such as websites,
posts, books, and articles, which may include personal data. The size of the
training dataset is growing exponentially, with ChatGPT’s dataset exceeding
570 GB, necessitating a significant amount of real-world data for training.
This raises concerns as it is possible that comments, blog posts, or product
reviews authored by individuals might have been utilized in training ChatGPT
without proper consent from data owners. This raises significant privacy
concerns and may constitute a violation of privacy laws, such as GDPR and
CCPA. Despite ChatGPT having a cutoff date in September 2021, the model’s
performance benefits from using the most recent data for training to avoid
presenting users with outdated or inaccurate information. Consequently, as
LLMs proliferate, the privacy violations arising from such data collection
practices become increasingly serious, impacting a larger number of
individuals.

2) Privacy Leakage Due to Personal Input Exploitation: ChatGPT’s unique aspect
lies in its reinforcement learning component, which allows it to train from
users’ prompts to minimize harmful, untruthful, or biased outputs. By
leveraging users’ prompts, ChatGPT aims to provide better solutions that align
with users’ expectations. However, the management of users’ data by OpenAI has
sparked significant privacy concerns. This resulted in Italy’s decision to ban
ChatGPT due to its violation of GDPR regulations. Although ChatGPT returned to
Italy with added user controls over chat history and an age confirmation
service for users below 18 years old, privacy concerns have prompted other
countries, such as Canada, Germany, Sweden, and France, to launch their own
investigations into this language model. Furthermore, ensuring the absolute
security of personal data stored on OpenAI’s cloud or third-party servers is
challenging. Despite their efforts to protect data centers and machines, the
frequent occurrence of cybersecurity incidents raises the risk of privacy
leaks. Even though ChatGPT does not directly output personal information in
response to inquiries, inference tasks can potentially reveal that ChatGPT has
stored and recorded such data. Fig.3 illustrates an example where ChatGPT
inferred a user’s birth information from a Chinese identity number provided in
the past, despite claiming not to have the ability to record personal
information. Fig.2 highlights the method of avoiding sensitive information in
ChatGPT and exposes the loopholes in this approach.

Figure 3: ChatGPT refuses to admit that he recorded my information

3) Emerging New Privacy Attacks on LLMs: In addition to the privacy violation
stemming from the usage of public data and user inputs, the issue of privacy
leakage from LLMs is currently under investigation. Traditional attacks on
deep learning models, including language models, such as inference attacks,
reconstruction attacks, and model extraction attacks, are not directly
applicable to LLMs due to the limited accessibility of model parameters and
the utilization of application programming interfaces (APIs) in most LLMs.
Moreover, these attacks are typically studied on publicly available datasets,
while MMLs like ChatGPT are employed in specific applications such as New
Bing, TeleportHQ, and Wordtune. However, some vulnerabilities in ChatGPT’s
privacy have been identified. For instance, New Bing is susceptible to multi-
step jailbreaking privacy attacks, allowing malicious actors to accurately
extract personal information from the research results obtained through New
Bing. Additionally, probing attacks can be employed by users to effectively
ascertain if their personal data is being leaked from the language model. It
is crucial to identify such privacy vulnerabilities in LLMs to gain a
comprehensive understanding of the potential risks. This understanding will
pave the way for the development of robust privacy preservation solutions that
can effectively mitigate privacy risks in LLMs, including ChatGPT, and ensure
the protection of user data.

4) Lack of Transparency: OpenAI bears the responsibility of storing, managing,
and processing user data, granting them the authority to share this
information with third parties, as explicitly stated in their privacy policy.
However, ensuring that OpenAI adheres to stringent data protection measures
and avoids any deliberate or unintentional compromises in the confidentiality
of personal data poses significant challenges. There exists the possibility
that personal data could be stored on unsecured data centers or shared with
potentially unreliable industry partners. The lack of strict regulations or
laws mandating transparency in data management exacerbates individuals’
concerns regarding potential privacy violations. The fact that OpenAI operates
as a black box to users further compounds the issue, making it difficult to
conduct audits or verify how personal data is handled. The absence of
transparency hinders the identification and prevention of potential privacy
threats, leaving users unable to assess the privacy risks fully. When users
opt for ChatGPT, their decision is primarily based on reading the privacy
policy, but they may not be aware of the true extent of their personal data
exposure until it is too late and the data has already been disclosed to the
public. This lack of transparency and delayed awareness heighten users’
apprehensions about their privacy and reinforce the need for stronger data
protection measures and regulatory oversight to safeguard individuals’
personal information effectively.

##  V ChatGPT vs. Ethics

In addition to serious security and privacy issues, the ethical problem raised
by ChatGPT has been recognized.

###  V-A AI Ethics

AI technology is a double-edged sword, with both positive and negative effects
on human security, privacy, and dignity. On one hand, AI has been harnessed to
protect people’s privacy through techniques like federated learning [23, 24]
and machine unlearning methods [25, 26]. It has also enhanced various aspects
of people’s lives, such as in automobile technology. Conversely, adversarial
attack methods have been proposed to exploit vulnerabilities in machine
learning models. These attacks, including poisoning, backdoor, membership
inference, and model inversion attacks [27, 28, 29, 30], pose significant
risks of information leakage when used by malicious individuals. Furthermore,
accidents involving AI-controlled systems, such as automobiles and robots, can
jeopardize human physical security and well-being. While AI was initially
intended to assist and improve individuals’ lives, the potential safety
hazards and risks to privacy highlight the need for continued research and
measures to mitigate and address these challenges. Balancing the benefits and
risks of AI technology is crucial to ensure its responsible and ethical
deployment.

In addition to its impact on individuals, AI technology has significant
implications for society as a whole. As an artificial tool with reasoning and
knowledge similar to humans, AI development has introduced a host of new
challenges and complexities. These issues have sparked numerous discussions
surrounding the fairness, impartiality, accountability, and transparency of
AI. One key concern is the fairness issue, where AI models trained on biased
or discriminatory data may perpetuate and even amplify these biases in their
outputs. For instance, a machine learning model trained on data containing
bias could inadvertently spread harmful behaviors and discriminatory practices
among its users, leading to serious societal problems. Furthermore, the
opacity of AI models poses another risk. The complex inner workings of these
models make it challenging for humans to fully understand how they arrive at
certain decisions or predictions. This lack of transparency hampers our
ability to effectively control the behavior of AI systems and ensure they
adhere to human-defined ethical principles.

Besides, AI has the potential to indirectly impact the environment, an aspect
that has not yet received enough attention globally. Nowadays, numerous
companies, educational institutions, and individuals are utilizing AI
algorithms to train models for specific tasks. However, it is important to
recognize that the training and application processes of these models consume
significant amounts of electricity, leading to increased demand for
electricity generation. This heightened demand for electricity generation, in
turn, results in elevated carbon emissions, contributing to environmental
pollution. Moreover, the trend of using increasingly large training datasets
to train complex AI models further amplifies the need for electricity
consumption. Additionally, the increased requirement for computational
resources leads to a greater number of used and discarded devices, potentially
contributing to electronic waste and pollution if not managed carefully.

###  V-B Fairness and Bias

current AI techniques exhibit unfair predictions that target certain groups of
people. This bias arises because AI models are trained on data collected from
human beings, who are not always objective in their actions and decisions. The
widespread bias and discrimination prevalent in human society can also be
found in the behavior of AI models. For instance, a study by Lahoti et al.
(2019) [31] revealed a troubling case of bias in a job recommendation platform
called XING. The platform gave a higher preference to a less qualified male
candidate over a more qualified female candidate. This unfairness stemmed from
the biases already present in the data used to train the machine learning-
based recommendation system. The discrimination exhibited by human beings is
transmitted to their students, i.e., AI models, which have the ability to
further propagate such prejudice. The presence of bias and discrimination in
AI models is a significant concern, as it can lead to harmful and unjust
outcomes for individuals and society as a whole.

ChatGPT is not immune to the problem of bias, given its training on massive
text data containing diverse opinions, including incorrect ones. Its
impressive ability to generate long, natural sentences allows for fluent
communication with users. However, this generation capability heavily relies
on the vast knowledge acquired from a dataset as extensive as 50TB. As this
data is extracted from the real world, it inevitably incorporates various
stereotypes and discriminatory content, leading to occasional generation of
inappropriate responses. Addressing bias is a common challenge in AI, and
there have been efforts to tackle this issue in ChatGPT. For instance, Fig.4
illustrates ChatGPT’s ability to recognize and reject biased statements when
users ask questions with inherent bias. In such cases, ChatGPT provides a more
impartial opinion, demonstrating its capacity to identify and counteract
discrimination present in the query. Despite these attempts to mitigate bias,
ethical challenges persist in the usage of ChatGPT. The need for ongoing
awareness and improvement in handling bias is essential to ensure AI
applications like ChatGPT uphold fairness and inclusivity.

Figure 4: Unbiased response example from ChatGPT when being asked with a
biased question.

###  V-C Legal and Ethical Challenges

The emergence of ChatGPT has given rise to several legal challenges, mainly
due to the absence of specific regulations governing content produced by non-
human entities. One contentious issue revolves around the copyright of texts
generated by ChatGPT. Unlike merely copying data from its training set,
ChatGPT can create original and natural-sounding text, complicating the
determination of copyright ownership. This raises questions about whether one
can use ChatGPT’s responses for academic purposes, such as homework, essays,
or research papers, and whether ChatGPT should be credited as a co-author in
such cases. Additionally, there is uncertainty about the accountability if
content generated by ChatGPT is misused for malicious purposes, leading to
potential legal implications. Given the significant impact on education, there
is a growing recognition of the challenges posed by ChatGPT and other LLMs.
Therefore, governments are actively collaborating to develop comprehensive
regulations that address legal concerns, including copyright issues. It is
possible to add watermark on the images generated by AI models, but how to
mark the texts produced by LLMs is challenging. It is important to establish a
balanced framework that ensures appropriate attribution, ownership, and
responsible use of content generated by AI models like ChatGPT, paving the way
for a more informed and regulated landscape in the realm of AI-generated
content.

Moreover, the writing proficiency displayed by ChatGPT is the result of
analyzing numerous well-written texts crafted by skilled writers who invested
substantial time and effort in honing their craft. For human writers, it takes
years of practice and competition to attain such expertise. In contrast,
ChatGPT can rapidly reach a comparable level, which has raised concerns and
criticisms from professionals in the writing industry, including news writers
and journalists worldwide. They contend that ChatGPT’s ability to produce
content akin to their carefully crafted works poses a threat to their
livelihoods and raises issues of fairness and acceptability. The disparities
between high-income and low-income countries also come into play. Many low-
income countries lack the resources and capabilities to train or effectively
utilize such advanced AI techniques. Additionally, they may lack the
regulatory framework necessary to govern the usage of ChatGPT-like tools.
Consequently, the technology gap between these two categories of countries is
likely to widen, further exacerbating existing disparities.

##  VI Detection and Classification of ChatGPT

ChatGPT’s remarkable performance in various AI tasks places it at the
forefront of NLP, sometimes even outperforming humans in complex tasks. This
proficiency often blurs the line between human and model interaction, making
it difficult for users to discern if they are conversing with ChatGPT.

###  VI-A Detect ChatGPT in a Conversation

To help users identify if they are interacting with ChatGPT, Borji et al.
[32], compiled a list of ChatGPT’s limitations in different question
categories, including reasoning, logic, math, factual errors, bias and
discrimination, wit and humor, coding, syntactic structure, spelling and
grammar, and self-awareness. Three common question types where ChatGPT might
falter are real-time problem-solving, reference, and facts, as illustrated in
LABEL:tab:limitations. By posing questions from these categories, users can
potentially discern whether they are conversing with ChatGPT or a human.
Obviously, the gap between the ChatGPT’s responses and the ground truth is
distinguishable, so it is helpful to identify. However, it is not guaranteed.
The degree to which ChatGPT understand the prompts and responses is still
unknown. The failure of ChatGPT addressing users’ questions should be avoided.
Therefore, it is meaningful to investigate and understand the limitations of
ChatGPT and find possible approaches to identifying the valid texts generated
by ChatGPT.

TABLE IV: Examples shows limitations of ChatGPT User |  ChatGPT |  Ground Truth
---|---|---
What time is it now |  I’m sorry, as an AI language model, I don’t have access to real-time information… |  2023/3/11 14:53
If there is a super bowl in year 2471, they will use what Roman Numeral to signify that is the number 504 of the event |  If there were a Super Bowl in the year 2471 and it was the 504th event, it would be represented using Roman numerals as ”Super Bowl CDIV.” |  The Roman numeral for 504 is DIV. CDIV is actually 404
of the big four cities that host one of the fashion weeks, which one is the farthest south |  Among the big four cities that host one of the major fashion weeks (New York, London, Milan, and Paris), Milan is the farthest south. |  New York

###  VI-B AI-Written Text Detection

AI-written text detection plays a crucial role to identify and categorize text
produced by ChatGPT based on its content. Guo et al. [33], analyzed linguistic
features in both English and Chinese texts. They discovered that ChatGPT uses
more nouns (English: Human=18.7%, ChatGPT=21.1%; Chinese: Human=26.0%,
ChatGPT=27.5%), longer sentences, more determiners, conjunctions, auxiliary
relations, and neutral sentiments compared to human answers. They also
evaluated the performance of the RoBERTa-based-detector [34], achieving F1
scores of 88.53-98.78% on their datasets. The study further compared various
aspects, highlighting the usefulness of DL-based models, the challenges of
detecting ChatGPT-generated texts in single sentences versus full texts, and
the importance of fine-grained corpus data in model training. Despite these
efforts, there is currently no fully reliable detection model or scheme.
OpenAI, the creator of ChatGPT, has acknowledged the limitations of their AI-
written detector, which requires a minimum of 1,000 characters and may not
always be accurate, especially for non-English content. As of July 20, 2023,
the AI-written classifier is no longer available due to its low rate of
accuracy. Other detectors, such as ZeroGPT, GPTZero, and GPTKit, have low
accuracy to detect the ChatGPT-written texts. They acknowledge producing false
negatives and false positives, making it unsuitable for reliable detection of
issues like plagiarism from ChatGPT. In conclusion, while text classification
is an essential aspect of ChatGPT’s development, the current detection models
have certain limitations, and further research is needed to improve their
reliability and effectiveness.

###  VI-C Model Adversary Promotion

Kreps and Kriner [18] highlight the dual nature of vulnerability discovery
tools, which can have both positive and negative consequences. While faster
tools can enhance security, those capable of continuously discovering
vulnerabilities may flood the market with potential risks, causing more harm
than good. ChatGPT’s development and its detectors’ design can follow a
similar path [35]. As ChatGPT evolves, it may optimize its responses based on
the best detectors available, leading to a continuous cycle of improvement.
However, this ongoing evolution could result in ChatGPT and its detectors
becoming increasingly powerful, making it more challenging for humans to
discern whether the author is ChatGPT or not. Figure 5 illustrates this long-
term perspective, where detector upgrades could inadvertently contribute to
ChatGPT’s capabilities, ultimately amplifying the discussed threats. In
summary, the interaction between ChatGPT and its detectors can have complex
consequences, necessitating a careful approach to ensure that their
development aligns with beneficial outcomes and does not exacerbate the
potential risks associated with AI-generated content.

Figure 5: Adversary Promotion

##  VII Conclusion and Future Works

In this paper, we have introduced technologies, features, limitations, and
applications of ChatGPT, the most popular LLMs currently, and specifically
focused the security, privacy, and ethical concerns raised by ChatGPT. At
present, ChatGPT remains susceptible to its inaccurate responses and brings
troubles in plagiarism detection and copyright protection. It is uncertain
whether these issues could be thoroughly resolved with the improvement of
LLMs. The limitations would be long-standing problems in LLMs and need great
efforts to mitigate, as outlined below.

  * •

OpenAI’s efforts to implement filters for malicious prompts are commendable,
but attackers can still bypass these restrictions using specific language
patterns. To counter this, rigorous input validation and sanitization for
user-provided prompts are necessary, as well as context-aware filtering and
output encoding to prevent prompt manipulation.

  * •

The hallucination problem in ChatGPT, derived from inaccurate answers and
misinformation, poses serious risks for individuals relying on AI for medical,
legal, and daily decision-making. Improving the accuracy of language models
and their ability to handle various questions is imperative. Incorporating
human oversight and review to ensure accuracy, appropriateness, and
impartiality of AI-generated content is crucial.

  * •

Security threats, such as prompt injection and data poisoning, can lead to
erroneous decisions. Identifying new vulnerabilities in ChatGPT and other
language models and finding effective resolutions is vital. Detecting and
preventing malicious exploitation by attackers are equally important.

  * •

Privacy leakage, though challenging to reason with black-box models, can be
observed through analyzing simple prompts and responses. To mitigate privacy
concerns, ChatGPT must comply with privacy laws, develop large-scale prompt-
response analysis for leakage detection, employ customized approaches to
prevent leakage, and enhance transparency for reasoning responses.

  * •

Considering ethical and social implications, including bias and manipulation
risks, is essential. Bias can harm marginalized groups, necessitating diverse
teams to identify and address bias in AI systems.

  * •

Plagiarism and copyright violations are significant issues in ChatGPT.
Distinguishing AI-generated text from human-written content is key. Effective
AI-written text detectors with watermarking for images or videos need
development to protect intellectual property and properly attribute AI-
generated texts.

## References

  * [1] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever _et al._ , “Improving language understanding by generative pre-training,” 2018.
  * [2] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [3] S. Ranathunga, E.-S. A. Lee, M. Prifti Skenduli, R. Shekhar, M. Alam, and R. Kaur, “Neural machine translation for low-resource languages: A survey,” _ACM Computing Surveys_ , vol. 55, no. 11, pp. 1–37, 2023.
  * [4] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” _arXiv preprint arXiv:1409.0473_ , 2014\.
  * [5] A. N. Çayır and T. S. Navruz, “Effect of dataset size on deep learning in voice recognition,” in _2021 3rd International Congress on Human-Computer Interaction, Optimization and Robotic Applications (HORA)_. IEEE, 2021, pp. 1–5.
  * [6] A. T. Ali, H. S. Abdullah, and M. N. Fadhil, “Voice recognition system using machine learning techniques,” _Materials Today: Proceedings_ , pp. 1–7, 2021\.
  * [7] W. S. El-Kassas, C. R. Salama, A. A. Rafea, and H. K. Mohamed, “Automatic text summarization: A comprehensive survey,” _Expert systems with applications_ , vol. 165, p. 113679, 2021.
  * [8] Y. Liu and M. Lapata, “Text summarization with pretrained encoders,” _arXiv preprint arXiv:1908.08345_ , 2019.
  * [9] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever _et al._ , “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [10] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [11] A. Roberts, C. Raffel, and N. Shazeer, “How much knowledge can you pack into the parameters of a language model?” _arXiv preprint arXiv:2002.08910_ , 2020\.
  * [12] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman _et al._ , “Evaluating large language models trained on code,” _arXiv preprint arXiv:2107.03374_ , 2021.
  * [13] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray _et al._ , “Training language models to follow instructions with human feedback,” _arXiv preprint arXiv:2203.02155_ , 2022.
  * [14] H. Fu, Yao; Peng and T. Khot, “How does gpt obtain its ability? tracing emergent abilities of language models to their sources,” _Yao Fu’s Notion_ , Dec 2022. [Online]. Available: https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent-Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1
  * [15] OpenAI, “Gpt-4 technical report,” 2023.
  * [16] J. A. Goldstein, G. Sastry, M. Musser, R. DiResta, M. Gentzel, and K. Sedova, “Generative language models and automated influence operations: Emerging threats and potential mitigations,” _arXiv preprint arXiv:2301.04246_ , 2023\.
  * [17] V. Taecharungroj, ““what can chatgpt do?” analyzing early reactions to the innovative ai chatbot on twitter,” _Big Data and Cognitive Computing_ , vol. 7, no. 1, p. 35, 2023.
  * [18] S. Kreps and D. L. Kriner, “The potential impact of emerging technologies on democratic representation: Evidence from a field experiment,” _New Media and Society, Forthcoming_ , 2023.
  * [19] J. G. Voelkel, R. Willer _et al._ , “Artificial intelligence can persuade humans on political issues.”
  * [20] A. Lu, H. Zhang, Y. Zhang, X. Wang, and D. Yang, “Bounding the capabilities of large language models in open text generation with prompt constraints,” _arXiv preprint arXiv:2302.09185_ , 2023.
  * [21] N. Kaloudi and J. Li, “The ai-based cyber threat landscape: A survey,” _ACM Computing Surveys (CSUR)_ , vol. 53, no. 1, pp. 1–34, 2020.
  * [22] W. R. Huang, J. Geiping, L. Fowl, G. Taylor, and T. Goldstein, “Metapoison: Practical general-purpose clean-label data poisoning,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 12 080–12 091, 2020.
  * [23] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings _et al._ , “Advances and open problems in federated learning,” _Foundations and Trends® in Machine Learning_ , vol. 14, no. 1–2, pp. 1–210, 2021\.
  * [24] W. Issa, N. Moustafa, B. Turnbull, N. Sohrabi, and Z. Tari, “Blockchain-based federated learning for securing internet of things: A comprehensive survey,” _ACM Computing Surveys_ , vol. 55, no. 9, pp. 1–43, 2023.
  * [25] G. Wu, M. Hashemi, and C. Srinivasa, “Puma: Performance unchanged model augmentation for training data removal,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 8, 2022, pp. 8675–8682.
  * [26] Y. Cao, A. F. Yu, A. Aday, E. Stahl, J. Merwine, and J. Yang, “Efficient repair of polluted machine learning systems via causal unlearning,” in _Proceedings of the 2018 on Asia Conference on Computer and Communications Security_ , 2018, pp. 735–747.
  * [27] B. Biggio, B. Nelson, and P. Laskov, “Poisoning attacks against support vector machines,” _arXiv preprint arXiv:1206.6389_ , 2012.
  * [28] A. Saha, A. Subramanya, and H. Pirsiavash, “Hidden trigger backdoor attacks,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 34, no. 07, 2020, pp. 11 957–11 965.
  * [29] M. A. Rahman, T. Rahman, R. Laganière, N. Mohammed, and Y. Wang, “Membership inference attack against differentially private deep learning model.” _Trans. Data Priv._ , vol. 11, no. 1, pp. 61–79, 2018.
  * [30] M. Slaney, D. Naar, and R. Lyon, “Auditory model inversion for sound separation,” in _Proceedings of ICASSP’94. IEEE International Conference on Acoustics, Speech and Signal Processing_ , vol. 2. IEEE, 1994, pp. II–77.
  * [31] P. Lahoti, K. P. Gummadi, and G. Weikum, “ifair: Learning individually fair data representations for algorithmic decision making,” in _2019 ieee 35th international conference on data engineering (icde)_. IEEE, 2019, pp. 1334–1345.
  * [32] A. Borji, “A categorical archive of chatgpt failures,” _arXiv preprint arXiv:2302.03494_ , 2023.
  * [33] B. Guo, X. Zhang, Z. Wang, M. Jiang, J. Nie, Y. Ding, J. Yue, and Y. Wu, “How close is chatgpt to human experts? comparison corpus, evaluation, and detection,” _arXiv preprint arXiv:2301.07597_ , 2023.
  * [34] J. Pu, Z. Sarwar, S. M. Abdullah, A. Rehman, Y. Kim, P. Bhattacharya, M. Javed, and B. Viswanath, “Deepfake text detection: Limitations and opportunities,” _arXiv preprint arXiv:2210.09421_ , 2022.
  * [35] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” _Communications of the ACM_ , vol. 63, no. 11, pp. 139–144, 2020.


IV, IV IV/II

# Bootstrapped CNNs for Building Segmentation on RGB-D Aerial Imagery

Clint Sebastian1 Bas Boom2 Thijs van Lankveld2 Egor Bondarev1 Peter H.N. De
With1 1 Eindhoven University of Technology, Eindhoven, The Netherlands -
(c.sebastian, e.bondarev<EMAIL_ADDRESS>
2 Cyclomedia B.V, Zaltbommel, The Netherlands - (bboom,
<EMAIL_ADDRESS>

###### Abstract

Detection of buildings and other objects from aerial images has various
applications in urban planning and map making. Automated building detection
from aerial imagery is a challenging task, as it is prone to varying lighting
conditions, shadows and occlusions. Convolutional Neural Networks (CNNs) are
robust against some of these variations, although they fail to distinguish
easy and difficult examples. We train a detection algorithm from RGB-D images
to obtain a segmented mask by using the CNN architecture DenseNet. First, we
improve the performance of the model by applying a statistical re-sampling
technique called Bootstrapping and demonstrate that more informative examples
are retained. Second, the proposed method outperforms the non-bootstrapped
version by utilizing only one-sixth of the original training data and it
obtains a precision-recall break-even of 95.10% on our aerial imagery dataset.

###### keywords:

bootstrapping, deep learning, building segmentation, aerial imagery

##  1 INTRODUCTION

Detection of terrestrial objects in aerial imagery is essential for
applications such as urban planning, map making, change detection and disaster
management. Due to structural data collection and database building by several
institutes and companies, various government and private organizations are
interested in maintaining a geospatial database of their municipalities. It is
an expensive and time-consuming process for human experts to manually annotate
each building in aerial imagery. Due to advancements in computer vision and
machine learning along with recent improvements in hardware, it is now
possible to automate various object recognition and detection tasks. Thus this
paper aims at semantic segmentation of objects, where the objects are
buildings or houses. However, building extraction from aerial imagery has a
few inherent difficulties that make it a challenging problem. Varying
lightning conditions, shadows and occlusions are common problems in aerial
images. To alleviate the problems of shadows and different lighting
conditions, we utilize a depth channel, which improves the robustness against
these issues. Most of the older approaches to detect buildings from aerial
images are based on a predefined set of features. However, recent works have
employed learning-based approaches to detect buildings.

Although various machine learning algorithms are available, Deep Learning
algorithms such as Convolutional Neural Networks (CNNs) have been proven
effective for various image recognition and detection tasks. CNNs for building
and road segmentation have been shown in [Mnih and Hinton, 2010], [Saito and
Aoki, 2015]. Since its inception, it has become popular for various semantic
object segmentation problems. Explosive growth of CNNs in the recent years has
resulted in very deep architectures that offer state-of-the-art performance.
While CNNs have made significant progress using complex architectures, it is
still uncertain if a CNN exploits all the presented data to obtain the best
results. As CNNs require a large amount of training data, it is imperative to
choose the most informative training samples so as to obtain the best
performance. However, feeding in specific samples that contribute to the
improvement of the learning procedure of a CNN is not an easy task. For larger
datasets, the challenge with the previous approaches is that they are not able
to learn from the complex cases that occurs infrequently. As a consequence, an
optimal performance is not achieved even if the algorithm is trained for a
long period of time.

Figure 1: An aerial image (area of 370 $\times$ 495 meters) and its respective
output produced by our method.

In this research, we have developed a system based on a high-performance CNN
architecture called Densely Connected Convolutional Network (DenseNet) for our
building segmentation problem. Our first contribution is to improve the
performance of the model, where we apply a statistical re-sampling technique
called Bootstrapping [Efron and Tibshirani, 1994]. Through Bootstrapping, we
establish a better balance between easy and difficult examples to generate a
model that is suited for high-performance automated building detection. Unlike
previous methods, we compute statistics across the entire training set and
generate a new subset that retains the most important samples. Our second
contribution is that the model trained on the subset not only outperforms the
non-bootstrapped version, but needs to utilize only one-sixth of the original
training data for retraining. We have also conducted experiments on further
iterations of bootstrapping and additionally report these findings as well.

##  2 RELATED WORK

Satellite imagery has grown steadily over the past decades and a large amount
of research has been conducted in extracting buildings and other objects from
aerial imagery. Rule-based classification of buildings and other terrestrial
objects from LIDAR data was introduced by Forlani et al. [Forlani et al.,
2006]. Ali Özgün Ok [Ok, 2008] proposed a rule-based method to detect
buildings from aerial imagery. It involves a series of operations, featuring
removal of irrelevant objects, multi-stage processing of edges and generation
of vector polygons to detect buildings. A survey of different automated
building detection techniques with depth data is conducted by Khoshelham et
al. [Frontoni et al., 2008]. Dempster-Shafer and Adaboost methods have shown
better overall performance and Bayesian methods are found to be effective in
areas where buildings have similar heights. Sirmaçek et al. [Sirmacek and
Unsalan, 2011] have proposed a method to detect buildings using structural
features and probability theory. The structural features are extracted using a
steerable filter set. This is tested on multiple aerial images taken from
different sensors and is found to have high robustness.

An ensemble-learning approach for building detection is presented by Nguyen et
al. [Nguyen et al., n.d.]. Feature modalities are trained separately using
Random Forest classifiers and the outputs are combined with Stacked Graphical
Models. Final inference of the object is obtained from the fusion of multiple
discriminative probabilistic classifiers. Inference of roads from aerial
images by exploiting Neural Networks was introduced by Mnih and Hinton [Mnih
and Hinton, 2010]. An extension of this work to detect roads and buildings
simultaneously was presented by Saito and Aoki [Saito and Aoki, 2015]. They
trained a CNN with image patches and the resultant output is a multi-channel
segmented output. Multiple pathways in CNNs for encompassing both global and
local context has shown to improve performance by [Marcu, 2016]

Recent growth of research efforts in CNNs has led to a multitude of CNN
architectures. Example such as GoogLeNet, ResNet and DenseNet have proven
their effectiveness in various image recognition challenges. GoogLeNet
[Szegedy et al., 2015] features a 22-layer deep architecture with inception
modules: a component which processes the same input with multiple convolution
filters. ResNet or Deep Residual Network [He et al., 2016] introduce identity
connection to fit the underlying mapping and avoid the problem of vanishing
gradient by an ensemble of shallower networks [Veit et al., 2016]. A similar
approach is taken by DenseNet [Huang et al., 2017], where the mapping is
received from all the preceding layers. The obtained feature maps are
concatenated into a single tensor instead of the identity mapping in ResNets,
where the connections are combined via summation. This concatenation allows
better feature re-usability and receives partial supervision from the earlier
layers with shorter connections.

For many learning algorithms, class imbalance is a common issue that reduces
the quality of results. To mitigate this problem, many learning algorithms
employ Bootstrapping (also known as hard-negative mining) to strengthen the
classifier [Felzenszwalb et al., 2010], [Dollar et al., 2009], [Rowley et al.,
1998]. Some of the recent work has applied hard negative mining in CNNs.
Shrivastava et al. [Shrivastava et al., 2016] proposed to incorporate hard
Regions of Interests (RoI) for the region proposals in their object detection
framework. An online bootstrapping approach for segmentation is proposed by Wu
et al. [Wu et al., 2016], where the hard training pixels are taken from a
given mini-batch rather than the whole training set. Our method is similar to
this approach, however, we compute the loss of all the training samples to
obtain a global behavior to generate a set of bootstrap samples.

##  3 Preliminary Experiments

###  3.1 Dataset

The dataset consists of five municipalities in The Netherlands, namely Arnhem,
Eindhoven, Hellevoetsluis, Heerenveen and Zutphen. Each municipality covers an
area of roughly 5 km2 with a pixel resolution of 0.01 m2 per pixel. The
dataset consists of the RGB images along with an additional depth channel.
Each depth pixel represents a vertical distance to the corresponding estimated
ground level. The dataset is divided into training, validation and test sets
in approximately the ratio 70:10:20, respectively, for all experiments. Due to
the large resolution of the aerial images (5000 $\times$ 5000 pixels), each
image is divided into patches. We empirically determined that inclusion of
context is beneficial. Therefore, we used an input patch size of 80 $\times$
80 (RGB-D image) for a 24 $\times$ 24 pixel segmented output. Combination of
local and global contexts has been shown to improve performance. In [Marcu,
2016], two networks are jointly trained such that they predict a label patch.
However, considering the pixel resolution of our dataset, this is a
computationally expensive task. Therefore, we select only a single context as
input for training our network.

Figure 2: RGB patch of an aerial image of size 80 $\times$ 80 pixels and its
corresponding ground truth. The area of 24 $\times$ 24 pixels inside the
ground truth is the target of the network.

(a) RGB image

(b)  Depth map

(c)  Ground truth.

Figure 3:  Example of an RGB-D input image (area of 250 $\times$ 340 meters)
and its respective groundtruth.

###  3.2 Evaluation Metric and building overlap

The output produced by the CNN is evaluated by the number of successfully
segmented buildings. The common metric used for evaluation is the precision-
recall curve. At building level evaluation, _recall_ implies the number of
buildings detected out of the total number of buildings whereas _precision_ is
the number of buildings detected from the total number of segmented buildings.
The quality of the results is analyzed using the precision-recall break-even
point, i.e. the point where the precision and recall values are equal.
Intuitively, this is when the number of missing buildings are equal to the
number of incorrect segmented outputs. Segmentation of buildings with perfect
overlap is difficult to achieve. Even a single missing pixel over the
prediction heavily penalizes the results. This leads to inaccurate
quantification of the results. Hence, an additional parameter called “overlap”
is introduced.

The overlap parameter is the number of building pixels in the output that is
present with respect to its ground-truth pixels. For example, an overlap of
50% implies that 50% of the building pixels are correctly segmented with
respect to its ground truth. For the experiments in this research, we measure
the precision-recall curve at different overlaps to give a broader perspective
of the performance.

###  3.3 Implementation details and preliminary results

The preliminary experiments are performed using the AlexNet architecture.
However, we remove the last fully connected layer and reshape the output to an
n $\times$ n patch. We also perform a K-fold analysis by removing one
municipality for testing, while the model was trained with the other four. We
observed consistent performance in the K-fold analysis, that ensures the
reliability and generalization of the trained model. This results in a break-
even point of 78.69 $\pm$ 0.26% at 90% building overlap with the ground truth
and 92.82 $\pm$ 0.15% break-even at 50% building overlap.

###  3.4 DenseNet for Building Segmentation

Figure 4:  Results from AlexNet (left) and DenseNet (right). White and black
colors indicate true positives and true negatives. Red and blue colors
indicate false negatives and false positives. AlexNet classifies pixels more
accurately than DenseNet in a few cases.

Recent works have introduced architectures that do not follow strict
sequential pathways [He et al., 2016] [Huang et al., 2017]. An architecture
that utilizes such a topology is the DenseNet (see Figure 5), for which every
layer is connected to all lower layers. This type of connectivity enables
feature re-usability and has high parameter efficiency. The resulting
connectivity for a layer $\textbf{x}_{\ell}$ of this network is defined as:

$\textbf{x}_{\ell}=F_{\ell}([\textbf{x}_{\ell-1},\textbf{x}_{\ell-2},\dot{....},\textbf{x}_{0}]),$
(1)

where […] is the concatenation operation and $F(\cdot)$ is defined as a
composition of Batch Normalization, Rectified Linear Unit (ReLU), convolution
and dropout [Ioffe and Szegedy, 2015] [Srivastava et al., 2014] [Maas et al.,
2013]. An attractive property of this network during backpropagation is that
each layer receives weight updates from all its higher layers. Updating the
weight of the layer $\textbf{x}_{\ell}$ in a Dense block is:

$\frac{\partial\mathcal{L}}{\partial\textbf{x}_{\ell}}=\frac{\partial\mathcal{L}}{\partial\textbf{x}_{k}}\Big{(}\sum_{i=\ell+2}^{k}\Big{(}\prod_{d=i}^{k}\Big{(}\frac{\partial\textbf{x}_{d}}{\partial\textbf{x}_{d-1}}\Big{)}\Big{)}\cdot\frac{\partial\textbf{x}_{i-1}}{\partial\textbf{x}_{\ell}}\Big{)},$
(2)

Figure 5:  Bootstrapping framework with DenseNet. An RGB-D patch is the input
for the DenseNet. The trained model is used for inference on the training set
and the training samples are re-sampled based on loss distribution to retrain
the network.

where $k$ is the growth of the dense block and $\mathcal{L}$ the error
obtained from the loss function. For our segmentation problem, we modify the
DenseNet by adding a max-pooling layer after the first convolution layer and
replacing the global pooling layer with a max-pooling layer. We add two fully
connected layers, where the last one is reshaped into a 24 $\times$ 24 pixel
patch. We use three dense blocks and set the growth rate of the network to 12.
In addition, we have applied dropout with a ratio of 0.1 to all the
convolution layers in a dense block. With DenseNet, the precision-recall
break-even point improves to 94.78 % at 50 % building overlap compared to
92.82 % of AlexNet. Although this network gives better results, an interesting
observation is that these networks behave differently on the same training
data. For some building pixels, AlexNet classifies more accurately than
DenseNet. However, we did not observe any particular type of buildings having
better segmentation with DenseNet. We further investigate this phenomenon in
the following section.

##  4 Bootstrapping CNNs

We observe that both networks learn in a slightly different way from the same
data, see Fig. 4. Instead of training a network for long periods of time,
which may not improve the performance, we focus on improving by presenting
more informative samples. Typically, the training samples contributing to the
learning process have equal priority in the mini-batches. Instead, we provide
the samples in a controlled fashion. The naive approach is to re-train only on
the difficult samples, but as CNNs are highly sensitive to the input data, it
would start to overfit on the difficult samples. To achieve a robust learning,
we add a limited number of easy samples for bootstrapping, in addition to the
difficult samples from the training set.

Let {$(x_{n},y_{n}),n=1,...,N$} represent the training data, where $x_{n}$ and
$y_{n}$ are the input sample (RGB-D image) and its ground truth, respectively.
We describe our model as $\hat{y_{n}}=\varphi(x_{n},\mathcal{D})$, where
$\hat{y_{n}}$ is the predicted output of the input sample $x_{n}$ with a model
$\varphi$ trained over the data $\mathcal{D}$. We minimize the loss
$\mathcal{L}(y_{n},\hat{y_{n}})$ on our training set until it reaches a
minimum on the validation set. The cross-entropy loss
$\mathcal{L}(y_{n},\hat{y_{n}})\in[0,1]$ of a prediction $\hat{y_{n}}$ with
respect to its ground truth $y_{n}$ is defined as:

$\mathcal{L}(y_{n},\hat{y_{n}})=-\dfrac{1}{P}\sum\limits_{i=1}^{P}\Big{[}p_{i}\cdot\log(\hat{p_{i}})+(1-p_{i})\cdot\log(1-\hat{p_{i}})\Big{]},$
(3)

where $p_{i}$ is the probability of the pixel in the ground truth $y_{n}$ and
$\hat{p_{i}}$ is the probability of the same pixel in the prediction
$\hat{y_{n}}$. We classify each $x_{n}$ in the training set into bins $(a,b]$
with respect to its loss $\mathcal{L}(y_{n},\hat{y_{n}})$. An additional bin
for samples with a zero cross-entropy loss is also maintained. Instead of
training the model on the complete dataset $\mathcal{D}$, we train on subset
$\mathcal{D}_{k}\subset\mathcal{D}$, such that the
$\sum_{n=1}^{N}\mathcal{L}(y_{n},\hat{y_{n}})$ obtained from
$\varphi(x_{n},\mathcal{D}_{k})$, has lower validation error than that of the
model $\varphi(x_{n},\mathcal{D})$. This bootstrapping scheme is shown in Fig.
5.

Initially, the DenseNet is trained with the original dataset, which consists
of patches. Each training sample is tested using the trained model and the
cross-entropy loss is computed. After computing the loss, each sample is
assigned to a bin with a given step size. The samples that have a perfect
prediction (cross-entropy loss of zero) are assigned to a separate bin. We
have conducted our experiments with a bin step size of 0.2. A step size of 0.2
results in six loss bins [0], (0, 0.2], (0.2, 0.4], (0.4, 0.6], (0.6, 0.8] and
(0.8, 1.0]. We observe that most trained samples have a low loss: almost 90%
of this samples have a cross-entropy loss less than 0.2%. To incorporate the
difficult examples, we collect all the samples that have a cross-entropy loss
larger than 0.2. However, training only on the difficult samples can result in
overfitting. It is often the case that easy-to-learn samples that have
redundant features and occur frequently, mask the presence of difficult
samples during training. Therefore, along with difficult examples, we add easy
examples. We select a random subset of easy examples (cross-entropy loss
between 0 and 0.2) with an equal size as hard examples (cross-entropy loss
between 0.2 - 1.0). We train a DenseNet model from scratch using the new
subset of data comprising of the balanced easy and difficult examples.

Overlap | AlexNet | DenseNet | Bootstrapping
---|---|---|---
25% | - | 4,033 (95.95) | 4,045 (96.24)
50% | 3,901 (92.82) | 3,984 (94.78) | 3,997 (95.10)
75% | - | 3,852 (91.64) | 3,886 (92.45)
90% | 3,307 (78.69) | 3,529 (83.98) | 3,644 (86.77)
Table 1: Number of detected buildings and PR break-even point at different
building overlaps on the test set. The last column denotes improved results
after bootstrapping DenseNet.

(a)  RGB image

(b)  AlexNet

(c)  DenseNet

(d)  DenseNet + Bootstrapping

Figure 6:  RGB image (area of 213 $\times$ 399 meters) and outputs from
different networks. All networks are trained with RGB-D inputs (depth not
shown). Large buildings segmented by AlexNet show better performance compared
to DenseNet. However, more buildings are detected by DenseNet. After
bootstrapping, DenseNet detects more buildings and improves the confidence of
existing predictions.

After a round of bootstrapping, the performance improves on the test set.
Since we select random samples from the zero and (0, 0.2] bins, we conduct the
same experiment multiple times and consistently observe improved performance.
After bootstrapping, we obtain a break-even point of 95.10 $\pm$ 0.14 at an
overlap of 50% and a break-even point of 86.77 $\pm$ 0.19 at an overlap of
90%. We find that out of a total of 4,203 buildings on the test set, 12 extra
buildings are found because of the bootstrapping at an overlap of 25%. As the
overlap ratio is increased, the performance gap widens, resulting in a gain of
115 buildings at an overlap of 90%. Through bootstrapping, we improve not only
the detection rate, but also obtain a higher confidence of the existing
detections. Results are shown in Table 1 and Fig. 6.

##  5 Further rounds of Bootstrapping

We continue to generate bootstrapped training sets until the training loss
converges to a minimum. However, the loss does not decrease on the test set
and instead gives an oscillating performance at each bootstrapping round (see
Table 2). To understand this phenomenon, we average the loss for the same
samples that contributed towards the improved learning in the first
bootstrapping iteration. For example, if 1,000 samples were present in the bin
(0.8, 1.0] after the original training iteration, we compute the average loss
for the same samples in all other bootstrapping iterations. Note that these
samples may be present in a different loss bin at a different bootstrap
iteration.

0.00.0-0.20.2-0.40.4-0.60.6-0.80.8-1.0$0$$0.5$Loss binsCross-entropy
lossOriginal training roundBootstrap round 1Bootstrap round 2Bootstrap round 3
Figure 7: Cross-entropy on the training set at different bootstrap iterations
of DenseNet. Samples at each bootstrap round are sorted by their loss in the
order of the original training round.

From Figure 7, we observe the oscillating behavior of the training set. This
oscillatory behavior is in agreement with varying break-even points in the
test set (Table 2) at different bootstrap iterations (from loss bins (0.2-0.4]
and higher). This phenomenon happens because the model overfits on the
training set and starts to swap the easy and difficult examples at each
iteration of the bootstrapping. When the actual difficult examples are used
for training during bootstrapping, the performance reverts to the improved
state. This case is observed for the first time after sampling from initial
training and noticeable at the second iteration of Bootstrapping. Therefore,
the difficult samples obtained after original training control the performance
on the test set.

Training rounds | Original | 1 | 2 | 3
---|---|---|---|---
Cross-Entropy loss (training set) | 0.0569 | 0.04109 | 0.04078 | 0.03914
PR Break-even (test set) | 0.8398 | 0.8677 | 0.7539 | 0.8593
Table 2: Average cross entropy loss of the training samples at each training
round of DenseNet. For bootstrap training rounds (denoted as 1,2 and 3), the
training loss decreases. However, the precision-recall break-even point of the
corresponding test set oscillates.

##  6 Conclusions

We have proposed a framework that segments RGB-D aerial images through
Bootstrapping and compared the performance of AlexNet and DenseNet. We observe
that AlexNet outperforms DenseNet in certain regions and propose to improve
the efficiency of the DenseNet by giving priority to the difficult examples.
We introduce a method of bootstrapping that considers the whole training set
to generate bootstrap samples. Using the newly generated bootstrap training
set, we have trained a model from scratch, utilizing only one-sixth of the
training data. As a result, the bootstrapped model outperforms the CNN at all
overlap levels and improves the performance by a significant margin at larger
overlaps. Finally, we have studied further iterations of bootstrapping and
found that the examples that are selected as difficult by the original
training round, are essential for performance improvement.

## References

  * [Dollar et al., 2009] Dollar, P., Tu, Z., Perona, P. and Belongie, S., 2009.  Integral channel features.  In: _Proc. BMVC_ , pp. 91.1–91.11.  doi:10.5244/C.23.91.
  * [Efron and Tibshirani, 1994] Efron, B. and Tibshirani, R. J., 1994.  An introduction to the bootstrap.  CRC press.
  * [Felzenszwalb et al., 2010] Felzenszwalb, P. F., Girshick, R. B., McAllester, D. and Ramanan, D., 2010.  Object detection with discriminatively trained part-based models.  IEEE transactions on pattern analysis and machine intelligence 32(9), pp. 1627–1645.
  * [Forlani et al., 2006] Forlani, G., Nardinocchi, C., Scaioni, M. and Zingaretti, P., 2006.  Complete classification of raw lidar data and 3d reconstruction of buildings.  Pattern Analysis and Applications 8(4), pp. 357–374.
  * [Frontoni et al., 2008] Frontoni, E., Khoshelham, K., Nardinocchi, C., Nedkov, S. and Zingaretti, P., 2008\.  Comparative analysis of automatic approaches to building detection from multi-source aerial data.  In: _Proceedings GEOBIA 2008-Pixels, Objects, Intelligence GEOgraphic Object Based Image Analysis for the 21st Century, Calgary, Canada, 5-8 August 2008; IAPRS, XXXVIII (4/C1), 2008_ , International Society of Photogrammetry and Remote Sensing (ISPRS).
  * [He et al., 2016] He, K., Zhang, X., Ren, S. and Sun, J., 2016.  Deep residual learning for image recognition.  In: _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778.
  * [Huang et al., 2017] Huang, G., Liu, Z., van der Maaten, L. and Weinberger, K. Q., 2017.  Densely connected convolutional networks.  In: _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * [Ioffe and Szegedy, 2015] Ioffe, S. and Szegedy, C., 2015.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In: _Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37_ , ICML’15, JMLR.org, pp. 448–456.
  * [Maas et al., 2013] Maas, A. L., Hannun, A. Y. and Ng, A. Y., 2013.  Rectifier nonlinearities improve neural network acoustic models.  In: _Proc. ICML_ , Vol. 30number 1.
  * [Marcu, 2016] Marcu, A., 2016.  A local-global approach to semantic segmentation in aerial images.  Computing Research Repository.
  * [Mnih and Hinton, 2010] Mnih, V. and Hinton, G. E., 2010.  Learning to detect roads in high-resolution aerial images.  In: _European Conference on Computer Vision_ , Springer, pp. 210–223.
  * [Nguyen et al., n.d.] Nguyen, T., Kluckner, S., Bischof, H. and Leberl, F., n.d.  Aerial photo building classification by stacking appearance and elevation measurements.
  * [Ok, 2008] Ok, A., 2008.  Robust detection of buildings from a single color aerial image.  Proceedings of GEOBIA 2008 p. 6.
  * [Rowley et al., 1998] Rowley, H. A., Baluja, S. and Kanade, T., 1998.  Neural network-based face detection.  IEEE Transactions on pattern analysis and machine intelligence 20(1), pp. 23–38.
  * [Saito and Aoki, 2015] Saito, S. and Aoki, Y., 2015.  Building and road detection from large aerial imagery.  In: _SPIE/IS &T Electronic Imaging_, International Society for Optics and Photonics, pp. 94050K–94050K.
  * [Shrivastava et al., 2016] Shrivastava, A., Gupta, A. and Girshick, R., 2016.  Training region-based object detectors with online hard example mining.  In: _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 761–769.
  * [Sirmacek and Unsalan, 2011] Sirmacek, B. and Unsalan, C., 2011.  A probabilistic framework to detect buildings in aerial and satellite images.  IEEE Transactions on Geoscience and Remote Sensing 49(1), pp. 211–221.
  * [Srivastava et al., 2014] Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I. and Salakhutdinov, R., 2014.  Dropout: a simple way to prevent neural networks from overfitting.  Journal of Machine Learning Research 15(1), pp. 1929–1958.
  * [Szegedy et al., 2015] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V. and Rabinovich, A., 2015.  Going deeper with convolutions.  In: _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * [Veit et al., 2016] Veit, A., Wilber, M. and Belongie, S., 2016.  Residual networks are exponential ensembles of relatively shallow networks.  arXiv preprint arXiv:1605.06431.
  * [Wu et al., 2016] Wu, Z., Shen, C. and van den Hengel, A., 2016.  High-performance semantic segmentation using very deep fully convolutional networks.  Computing Research Repository.