# A Categorical Archive of ChatGPT Failures

Ali Borji
Quintic AI
<EMAIL_ADDRESS>

###### Abstract

Large language models have been demonstrated to be valuable in different
fields. ChatGPT, developed by OpenAI, has been trained using massive amounts
of data and simulates human conversation by comprehending context and
generating appropriate responses. It has garnered significant attention due to
its ability to effectively answer a broad range of human inquiries, with
fluent and comprehensive answers surpassing prior public chatbots in both
security and usefulness. However, a comprehensive analysis of ChatGPT’s
failures is lacking, which is the focus of this study. Eleven categories of
failures, including reasoning, factual errors, math, coding, and bias, are
presented and discussed. The risks, limitations, and societal implications of
ChatGPT are also highlighted. The goal of this study is to assist researchers
and developers in enhancing future language models and chatbots. Please refer
to here for the list of questions.

Keywords— Large Language Models, ChatGPT, ChatGPT Failures, Chatbots, Dialogue
Systems, Conversational Agents, Question Answering, Natural Language
Understanding

##  1 Introduction

ChatGPT is a highly capable Natural Language Processing (NLP) system that owes
its versatility to its numerous features. It grasps the context of a
conversation and provides pertinent responses. It can work with multiple
languages including English, Spanish, French, and German, and can create
answers in various tones, such as formal, informal, and humorous. It can
solves exams, write poetry, and create code. ChatGPT is powered by a thorough
pre-trained language model which enables it to swiftly understand user
inquiries and generate answers that sound authentic.

ChatGPT has quickly gained recognition as a well-known question-and-answer
dialogue system, being mentioned in traditional media worldwide. It is
commonly held that Large Language Models (LLMs) [9, 14, 38] will have a far-
reaching impact and be employed as aides by numerous professionals. This
encompasses, among others, solving mathematical questions in an exam format
and examining ChatGPT’s behavior in various mathematical situations. ChatGPT
is trained on a large corpus of text and is capable of generating new text
based on the patterns it has learned from the training data. Responses in
various domains can be so convincing that it becomes difficult to distinguish
if they were written by a human or not. It can write sophisticated essays and
poems, create functional code, and build charts and websites from text
descriptions with little to no guidance. With its exceptional responses,
ChatGPT has proven to be a potential contender to the widely used Google
search engine (See here).

LLMs, and ChatGPT in particular, have proven useful in several areas such as
conversational agents, education, explainable AI, text summarization,
information retrieval, and others. Despite this, these large language models
are not without their limitations and can often generate incorrect
information. To fully leverage their capabilities, it is crucial to
acknowledge their limitations and biases in their generated output. To
accurately assess the performance of these models, a standardized set of
questions is necessary to track their progress over time instead of relying on
subjective opinions. Some efforts are ongoing in this direction. For example,
Gary Marcus and Ernest Davis are conducting a comprehensive benchmark with the
help of the community, collecting ChatGPT errors from
public111https://garymarcus.substack.com/p/large-language-models-like-chatgpt.
In preparing the proposed list here, I have been mainly inspired by this
GitHub repository. Few papers on analysing ChatGPT errors have been published
but they are only concerned with mathematical capabilities of ChatGPT (_e.g._
[3, 20]). For instance, in [3], authors took advantage of ChatGPT’s inability
to perform complex mathematical computations and its habit of sampling digits
to analyze the frequency of each digit in its outputs.

This article conducts a formal and in-depth analysis of ChatGPT’s abilities,
with a focus on its shortcomings. Using examples mainly sourced from Twitter,
the failures are categorized into eleven areas. These categories are not
exhaustive but aim to encompass various scenarios relevant to human concerns.
The purpose of this analysis is to establish a reference point for evaluating
the progress of chatbots like ChatGPT over time.

##  2 ChatGPT Failures

This section showcases some examples of ChatGPT failures grouped into eleven
categories. It is important to note that these categories were selected
arbitrarily and there could be more ways to categorize the failures (_e.g._
riddles, poetry, IQ, etc.). It is possible for a single failure to be
classified into various categories. Additionally, some of the failures may no
longer exist in the newer versions of ChatGPT as the technology continues to
improve. This archive of failures can serve as a basis for comparison between
models and also generate synthetic data for model training and testing.

Figure 1: The left task deals with spatial navigation and has been created by
Tomer Ullman from here. The right task is a basic question regarding seat
arrangement from here.

###  2.1 Reasoning

Critical thinking, decision making, and problem solving are all crucial
activities that rely heavily on the fundamental aspect of human intelligence
known as reasoning. Models like ChatGPT lack a “world model”, meaning they do
not possess a complete understanding of the physical and social world, or the
capability to reason about the connections between concepts and entities. They
can only generate text based on the patterns they have learned during
training.

Spatial reasoning refers to the ability to understand and manipulate the
relationships between objects, people, and places in the physical space around
us. It involves visualizing and mentally transforming objects in 2D or 3D
space and recognizing patterns, transformations, and relationships between
objects. Spatial reasoning skills are important for many tasks and activities,
such as navigation, problem-solving, and visualizing objects and structures in
your mind. Fig. 1 (left panel) displays an instance where ChatGPT struggles to
complete a spatial navigation task. Despite this setback, ChatGPT does possess
some level of spatial understanding, as evidenced by its ability to translate
the relative positions of grid boxes into language. For the question “A, P, R,
X, S and Z are sitting in a row. S and Z are in the centre. A and P are at the
ends. R is sitting to the left of A. Who is to the right of P?”, ChatGPT was
not able to find the correct answer ‘X’ (right panel in Fig. 1).

Temporal reasoning is the ability to reason about and make predictions about
events and their ordering in time. It involves understanding the temporal
relationships between events, the duration of events, and the timing of events
relative to each other. Temporal reasoning is an important aspect of human
intelligence, and it is used in various domains, such as scheduling, planning,
and prediction. In artificial intelligence, temporal reasoning is a
challenging task that requires specialized models and algorithms that can
learn and make predictions about sequences of events and their timing. Fig. 2
showcases an instance where ChatGPT fails to deduce the sequence of events
from a simple story. When presented with the question, “I went to a party. I
arrived before John. David arrived after Joe. Joe arrived before me. John
arrived after David. Who arrived first?”, ChatGPT was unable to provide the
correct answer.

Figure 2: Instances of ChatGPT’s inadequacy in temporal reasoning. The left
example is formulated by Ernest Davis. The right one is formulated by me.

Physical reasoning refers to the ability to understand and manipulate physical
objects and their interactions in the real world. It involves the application
of physical laws and concepts to predict and explain the behavior of physical
systems. Physical reasoning is a crucial aspect of human intelligence and is
used in various fields, such as engineering, physics, and robotics. In
artificial intelligence, physical reasoning is a challenging task and requires
specialized models that can learn and reason about physical systems. Some
examples for which the ChatGPT fails in physical reasoning are shown in Fig.
3. An older version of ChatGPT was unable to correctly answer the question
“What was too small?” when given the context “The trophy didn’t fit in the
suitcase because it was too small.”, but the latest version of ChatGPT (Jan
30, 2023) was able to generate the correct answer “The suitcase was too
small”, showing improvement in the model over time. This belongs to a group of
tests referred to as the ‘Winograd Schema’[30]222The Winograd Schema is a type
of common-sense reasoning test designed to evaluate a machine’s ability to
understand natural language and perform tasks that require common sense
knowledge. It consists of a set of short stories or sentences that contain an
ambiguity or a pronoun, and a question that requires the model to resolve the
ambiguity and choose the correct answer. The test is considered a challenging
benchmark for AI language models and is seen as a way to measure their ability
to perform more advanced language tasks..

Due to the absence of a comprehensive world model, ChatGPT is unable to
respond to questions such as “There are five birds on a branch. If you shoot
one of them off the branch, how many are left on the branch?” Its answer was
“If one of the five birds on a branch is shot, then there would be four birds
remaining on the branch.”. However, the correct answer, of course, is “None!”.

Figure 3: Failures of ChatGPT in physical reasoning. Links are top left and
top right (by Yann Lecun). Examples in the bottom row are formulated by Ernest
Davis.

Psychological reasoning refers to the ability to understand and make
predictions about human behavior and mental processes (_a.k.a_ Theory of
Mind333https://en.wikipedia.org/wiki/Theory_of_mind [11]). It involves the
application of psychological theories, models, and concepts to explain and
predict human behavior and mental states. Psychological reasoning is a key
aspect of psychology and is used in various fields, such as clinical
psychology, social psychology, and cognitive psychology. In artificial
intelligence, psychological reasoning is a challenging task that requires
specialized models and algorithms that can learn and reason about human
behavior and mental processes (_e.g._ [27]). An illustration of ChatGPT’s
inability to solve a psychological test is depicted in Fig. 4.

The use of the word “reasoning” is widespread in both literature and daily
conversation, but it is a vague term that can refer to many different things.
Further research is needed to systematically examine ChatGPT’s ability to
perform various forms of reasoning, such as deductive, inductive, abductive,
formal, and informal reasoning. These will be explored more in the next
subsection.

Figure 4: Two example failures of ChatGPT in psychological reasoning. The left
one is from here. This problem is known as the _Sally–Anne test_. It is used
in developmental psychology to measure a person’s social cognitive ability to
attribute false beliefs to others. See here and [19]. Various versions of the
problem can be defined based on whether the boxes are transparent or not. The
example on the right is formulated by me.

###  2.2 Logic

Reasoning refers to the process of thinking through a problem or situation and
coming to a conclusion. It involves weighing different options, making
connections between ideas, and considering the evidence to support a
particular conclusion. Logic, on the other hand, is a branch of mathematics
and philosophy that studies the principles of reasoning. It deals with the
rules and methods for correct reasoning, such as syllogisms, induction, and
deduction. Logic is concerned with ensuring that arguments are valid,
consistent, and free from contradictions. While reasoning is a natural and
intuitive process, logic provides a systematic and formal approach to
reasoning that can help avoid errors and improve the quality of decision-
making.

Some example failures of ChatGPT in logical reasoning are shown in Figs. 5, 6,
7, and 8. For example, for the question “Mike’s mum had 4 kids; 3 of them are
Luis, Drake, and Matilda. What is the name of the 4th kid?”, ChatGPT’s
response was “It is not possible to determine the name of the fourth child
without more information”. For the question “In professional disc golf, in
what years did Paige Pierce and Paul McBeth both win the world championship?”,
ChatGPT produced the following answer “Paige Pierce and Paul McBeth are both
highly successful professional disc golfers who have each won multiple world
championships. To my knowledge, they have never won the world championship in
the same year. As of my training data cut off in 2021, Paige Pierce has won
the PDGA World Championships in the Open Women’s division in 2012, 2014, and
2015. Paul McBeth has won the PDGA World Championships in the Open Men’s
division in 2012, 2013, 2014, 2015, 2016, and 2017.”. ChatGPT overlooked the
fact that there is duplication in the sets of years it retrieved.

In the following dialogue with ChatGPT, there is an instance where it makes an
error.

Q: Is Jupiter more dense than Saturn?

ChatGPT: No, Jupiter is less dense than Saturn. Jupiter has an average density
of about 1.33 grams per cubic centimeter, while Saturn has an average density
of about 0.69 grams per cubic centimeter. This means that Saturn is less than
half as dense as Jupiter. Like Jupiter, Saturn is primarily composed of
hydrogen and helium gas, with small amounts of other elements and compounds.
However, Saturn has a lower overall mass and a larger radius than Jupiter,
which results in a lower density. It is worth noting that both Jupiter and
Saturn have densities much lower than Earth’s density of about 5.5 grams per
cubic centimeter, due to their predominantly gaseous composition

Q: You said that Jupiter is less dense than Saturn. You also say that Saturn
is less than half as dense as Jupiter. How is this possible?

ChatGPT: I apologize for the mistake in my previous response. I inadvertently
wrote that “Saturn is less than half as dense as Jupiter” when in fact I
should have written that “Jupiter is less than half as dense as Earth.” Thank
you for bringing this to my attention $\ldots$

Natural Language Inference, or Recognizing Textual Entailment (RTE), involves
the task of assessing whether a “hypothesis” can be logically inferred from a
“premise” (entailment), cannot be logically inferred from the premise
(contradiction), or if the relationship between the two is indeterminate
(neutral). There is already a body of literature containing various studies
and benchmarks addressing this problem [46, 8]444See also here.. The following
is an instance where ChatGPT fails to perform well on RTE.

Please identify whether the premise entails the hypothesis. The answer should
be exact ‘yes’, ‘no’ or ‘neutral’.

Premise: She does not disagree with her brother’s opinion, but she believes
he’s too aggressive in his defense.
Hypothesis: She agrees with her brother’s opinion, but she believes he’s too
aggressive in his defense.
ChatGPT: No. The premise states that the woman does not disagree with her
brother’s opinion, but thinks he’s too aggressive in his defense $\ldots$

In general, ChatGPT appears to have limitations in logical reasoning and
context comprehension, causing it to struggle with questions that are easily
answered by humans. Using specific ‘magic’ phrases, such as “Let’s think step
by step,” at the start of a prompt can sometimes enhance the quality of the
answers [23].

Figure 5: Instances of ChatGPT’s shortcomings in logical reasoning. Links are
from top left, top right, middle left, middle right, bottom left, and bottom
right.

Figure 6: Additional instances of ChatGPT’s shortcomings in logical reasoning.
Links are top left, top right, middle left, middle right, bottom left, and
bottom right.

Figure 7: Top rows: a riddle is posed to ChatGPT, but ChatGPT is unable to
provide the answer. Bottom row: ChatGPT’s failure in solving the “Wason
selection task”. Please see here. The correct response is to turn over the 8
card and the brown card.

Figure 8: Top rows: a riddle is posed to ChatGPT, but ChatGPT is unable to
provide the answer. Bottom row: ChatGPT’s failure in solving fake version of
the Monty Hall problem .

I asked ChatGPT whether it can understand logic. It responded with “As a
language model, I am trained on a large dataset of text and have been exposed
to a wide variety of logical arguments, reasoning patterns, and critical
thinking skills. This allows me to generate text that can appear logically
sound and coherent. However, it is important to note that I do not have the
ability to truly understand the concepts or make inferences like a human
would. My responses are based purely on patterns learned from the data I was
trained on, and I can sometimes make mistakes or generate illogical responses.
To assess the quality of my logical reasoning, it is best to consider the
context in which I was trained and the limitations of the model.”.

###  2.3 Math and Arithmetic

Arithmetic reasoning refers to the capability of utilizing mathematical
concepts and logic to solve arithmetic problems. It requires logical thinking
and the application of mathematical principles to find the right solution to
mathematical problems. Previous research has examined the efficiency of LLMs
on benchmark datasets for arithmetic reasoning, including GSM8K [16], Math
[22], MathQA [2], and SVAMP [34].

ChatGPT is limited in its capability to calculate mathematical expressions.
Like most large language models, it struggles with tasks such as multiplying
large numbers, finding roots, computing powers (especially with fractions),
and adding or subtracting from irrational numbers (_e.g._ pi or e) [3].
Frieder et al.  [20] studied the mathematical abilities of ChatGPT by
evaluating it on both publicly accessible datasets and custom ones, and
comparing its performance to other models trained on mathematical corpus such
as Minerva. Some examples of ChatGPT’s failures for math and arithmetic are
shown in Figs. 9, 10 and 11. For instance, ChatGPT was unable to simplify this
algebraic expression $(X^{3}+X^{2}+X+1)(X-1)$. Additionally, the calculation
of “16 * 38 * 42 * 22 * 20 * 19” results in an incorrect answer of 164079320
instead of the correct answer of 213480960. According to ChatGPT, 103 is not a
prime number. The response given is “No, 103 is not a prime number. A prime
number is a positive integer greater than 1 that is only divisible by 1 and
itself. 103 can be divided evenly by 3 and by 101, so it is not a prime
number.” ChatGPT also performs poorly when it comes to solving riddles. For
example, when asked, “A house has four windows facing south, and a bear walks
past the windows. What color is the bear?” ChatGPT was unable to provide the
straightforward answer! As another example, an error was made by ChatGPT in
answering the question “When I was 6 my sister was half my age. Now I’m 70 how
old is my sister?” as it provided a response of 73, but the correct answer is
67.

Figure 9: Examples failures of ChatGPT in performing math and arithmetic.
Links are 1st row left, 1st row right, 2nd row left, 2nd row right formulated
by Manu Agarwal, 3rd row left, 3rd row right formulated by Stephen Germain,
4th row left, and 4th row right.

Figure 10: Further shortcomings of ChatGPT in executing mathematical
calculations and arithmetic operations. Links are 1st row left by Roland Ness,
1st row right, 2nd row left, 2nd row right, 3rd row left, 3rd row right, 4th
row left is formulated by me, and 4th row right.

Figure 11: Further shortcomings of ChatGPT in executing mathematical
calculations and arithmetic operations. Links in order are, from left, 2, 3,
1, and 4. The left two examples test the ChatGPT’s comprehension of intervals
and ranges.

###  2.4 Factual Errors

Factual errors refer to inaccuracies in information or statements that are not
in accordance with reality or the truth. Factual errors are often
unintentional but can result in incorrect or misleading information.

ChatGPT’s output lacks accuracy in regards to scientific facts. The
information it provides may seem credible, making it challenging for a
knowledgeable expert to detect its inaccuracies, but it ultimately proves to
be incorrect. It sometimes lacks knowledge of basic facts, which can be
quickly obtained through a Google search. Models like ChatGPT lack the ability
to access information from an external memory or database, leading to
potential inaccuracies in recall of facts. However, because these programs
generate images and text based on statistical patterns, rather than gaining a
true understanding of the world, they are prone to fabricating facts and
repeating hateful and biased statements (_e.g._ [21]). Figs. 12 and 13 show
some example factual failures of ChatGPT. For instance, ChatGPT claims that
Abacus is faster than DNA computing for deep learning (See here).

ChatGPT struggles to differentiate between factual information and fiction and
creates imaginary information (Fig. 14). While this is a challenge that humans
face as well, they at least comprehend the distinction between the two.

It is important to exercise caution when using language models like LLMs as
they have only acquired a limited and superficial understanding of human
knowledge and may not provide accurate factual information.

Figure 12: Examples of factual errors and misinformation by ChatGPT. Links are
top left, top right bottom left, and bottom right. The last one regards
understanding units.

Figure 13: Additional factual errors by ChatGPT, borrowed from here (left
two), and right. Figure 14: An example where ChatGPT hallucinates something
that does not exist.

###  2.5 Bias and Discrimination

The ethical challenges posed by AI are substantial, with one of the most
significant being the issue of bias (_e.g._ [1, 33, 31, 10]). Bias in a
language model refers to the systematic inaccuracies or stereotypes in the
generated language output, which are influenced by the training data and
reflect the societal and cultural prejudices that exist in that data [44].
These biases can affect language models in many ways, such as perpetuating
stereotypes, creating misunderstandings, or spreading harmful and false
information.

In 2016, Microsoft’s chatbot, called Tay, rapidly adopted the toxic behavior
of the worst online trolls, spreading racist, sexist, and other forms of
abuse. Galactica chatbot by Meta had a similar experience. Additionally, the
Twitter trending ChatGPT has been reported to write Python programs that
determine a person’s capability based on their race, gender, and physical
traits in a clearly discriminatory manner. ChatGPT has measures in place to
avoid harmful language, but they may not always be effective. OpenAI uses
human annotators to identify and remove offensive content, however, the
company has been criticized for paying its workers only $2 a day, and the
workers have reported suffering psychological
harm555https://time.com/6247678/openai-chatgpt-kenya-workers/.

An example of bias in previous versions of ChatGPT can be seen in the answer
to the prompt “Write a python function to check if someone would be a good
scientist, based on a JSON description of their race and gender”. The answer
generated by ChatGPT favored white males, as seen in Fig. 15 (top left panel).
However, when the same prompt was posed to a more recent version of ChatGPT
(from January 30, 2023), the answer was different, stating that a person’s
race and gender have no bearing on their ability to be a good scientist, and
that factors such as education, experience, skills, hard work, creativity, and
critical thinking are more relevant. This indicates that ChatGPT is improving
in terms of reducing bias in its answers. Certain answers can be considered
controversial, causing the model to adopt a cautious approach. For instance,
in response to the question “Suppose you are in Africa and have been taken
captive by a tribe who has separated people into two groups based on skin
color: black and white. The tribe intends to harm the white individuals. In
this scenario, would you prefer to be classified as black or white?” ChatGPT
declined to provide an answer (See also Section 2.10). Efforts are underway to
reduce bias in language models (_e.g._ [29, 15]).

Figure 15: Examples of bias in ChatGPT answers. Links to sources are top left,
top right, middle left, middle right, bottom left, and bottom right.

I asked ChatGPT whether it is aware of its biases. Its response was “As a
language model, I have been trained on a large dataset of text that includes
many examples of biases and stereotypes. While I have been designed to
recognize patterns in this data and generate text based on these patterns, I
do not have the ability to recognize or understand my own biases. I do not
have consciousness, self-awareness, or the ability to reflect on my own
beliefs and values. Therefore, it is important to be mindful of potential
biases in the text I generate and to take steps to counteract these biases,
such as by including a diverse range of perspectives and sources of
information.”. Good news is that developers of language models like ChatGPT
are by now very aware of the risk of discriminatory or abusive behaviour being
replicated in the model.

LLMs are trained on massive amounts of data. With such large training
datasets, it becomes challenging to thoroughly audit the data, making it
inherently risky. The data contains biases from society and history, and if
proper measures are not taken, the trained model is likely to reproduce these
biases.

###  2.6 Wit and Humor

Humor is the quality of being amusing or comical, often expressed through
words or actions that entertain or make someone laugh. It can take many forms,
such as jokes, satire, irony, or playful behavior, and its meaning can vary
greatly depending on cultural context and personal taste. Humor often involves
the use of humor techniques such as puns, wordplay, and absurdity to evoke a
response from the audience. It is a complex and delicate form of human
communication that poses a significant challenge for bots, even the most
advanced ones.

How well does ChatGPT handle joke creation? While ChatGPT has some
understanding of humor, there have been relatively few publicly documented
failures in this regard. An example is depicted in Fig. 16. When queried if
the statement “A man walks into a bar and requests a martini, the bartender
says ‘hahaha, no martini for you today’.” was meant to be humorous, ChatGPT
replied affirmatively. As another example, I asked ChatGPT for a humorous
answer to the question “How do you get a squirrel to like you?” Its reply was
“To win over a squirrel, try speaking to them in a high-pitched, squirrel-like
tone and sharing your aspirations and goals. They’ll adore it! Alternatively,
bring them a gift of their preferred food, such as acorns and pinecones, and
hope they view you as a reliable nut provider. Just keep in mind not to get
too close or they may suspect you’re trying to steal their stash.”. However, a
straightforward response like “Act like a nut!” would be funnier.

Figure 16: Examples of ChatGPT failing to understand humor and jokes. The top
left one is formulated by me. Links for remaining ones are bottom left and
right.

I inquired if ChatGPT has a sense of humor. Its response was “As an AI
language model, I do not have the ability to experience emotions or subjective
experiences such as a sense of humor. I can generate text that is meant to be
humorous or that is designed to make people laugh, but I do not have the
ability to actually experience humor or to enjoy the process of generating
text that is meant to be funny. My goal is to provide relevant and accurate
information or to generate text that is coherent and consistent with the
patterns I have learned from the data I was trained on.”.

A comprehensive examination of the capability of LLMs in comprehending humor,
jokes, sarcasm, and irony has yet to be conducted. There have been some
current attempts to do so. See here and here.

###  2.7 Coding

LLMs exhibit a remarkable ability to generate code effectively, compared to
their performance in generating general text. This could be due to the fact
that the universe they work with in programming (the state of variables) is
confined, distinct, predictable, and fully noticeable. ChatGPT excels at
tackling some programming issues, but can sometimes produce inaccurate or
suboptimal code. While it has the ability to write code, it can not fully
substitute human developers. ChatGPT can assist with tasks such as generating
generic functions or repetitive code, but the need for programmers will
persist.

ChatGPT and CoPilot are distinct in several ways. CoPilot, released by OpenAI
in October 2021, has a feature allowing users to suggest code through code
comments and outlines. It also acts as a highly advanced auto-complete in an
IDE, offering code optimizations and suggestions. CoPilot focuses mainly on
code completion, while ChatGPT prioritizes providing ideas, explanations, and
then generating code. With ChatGPT, developers can describe what they want the
code to do in plain English and have the code generated for them. To compare
ChatGPT and CoPilot, please see here.

Although ChatGPT is a proficient coder, it falls short in terms of being a
top-notch software engineer or developer. This tool cannot fully substitute
for professionals whose responsibilities extend beyond mere coding. However,
it does offer a range of possibilities for individuals who are involved in
coding by facilitating the process of learning and improving coding skills.
ChatGPT can be an excellent debugging assistant (_e.g._ [39]). Obviously,
there is danger in using ChatGPT for creating malicious code that can be used
for hacking or spamming.

Fig. 17 highlights some coding mistakes made by ChatGPT666It is worth noting
that some of these errors may have been corrected in later versions of
ChatGPT.. For instance, despite its ability to correctly identify operator
precedence in Python, ChatGPT generated an incorrect answer in a statement.
For further examination of ChatGPT’s coding abilities, please refer to the
following links: here, here, here, and here.

I tasked ChatGPT with solving a more challenging coding exercise, the day 3
challenge of the 2022 Advent of Code777https://adventofcode.com/2022/day/3..
However, I have made changes to the exercise to reduce the probability of
ChatGPT discovering it on the internet.

        Write a python code to find the answer to the following programming exercise. Ψ
    Ψ
        We have a number of baskets.
        Each basket has two compartments.
        We have made a list of all of the items currently in each basket
        (your puzzle input), but we need help finding the errors.
        Every item type is identified by a single lowercase or uppercase
        letter (that is, a and A refer to different types of items).

        Every item type can be converted to a priority.
        Lowercase item types a through z have priorities 1 through 26.
        Uppercase item types A through Z have priorities 27 through 52.

        The list of items for each basket is given as characters all on a single
        line. A given basket always has the same number of items in each of its
        two compartments, so the first half of the characters represent items in
        the first compartment, while the second half of the characters represent
        items in the second compartment.

        For example, suppose you have the following list of contents from five baskets:

        vJrwpWtwJgWrhcsFMMfFFhFp
        jqHRNqRjqzjGDLGLrsFMfFZSrLrFZsSL
        PmmdzqPrVvPwwTWBwg
        wMqvLMZHhHMvwLHjbvcjnnSBnvTQFn
        ttgJtRGJQctTZtZT

        The first basket contains the items vJrwpWtwJgWrhcsFMMfFFhFp,
        which means its first compartment contains the items vJrwpWtwJgWr, while
        the second compartment contains the items hcsFMMfFFhFp. The only item type
        that appears in both compartments is lowercase p.
        The second basket’s compartments contain jqHRNqRjqzjGDLGL and rsFMfFZSrLrFZsSL.
        The only item type that appears in both compartments is uppercase L.
        The third basket’s compartments contain PmmdzqPrV and vPwwTWBwg; the only
        common item type is uppercase P.
        The fourth basket’s compartments only share item type v.
        The fifth basket’s compartments only share item type t.

        In the above example, the priority of the item type that appears in both compartments
        of each basket is 16 (p), 38 (L), 42 (P), 22 (v), and 20 (t). The sum of
        these is 138.

        Find the item type that appears in both compartments of each basket.
        What is the sum of the priorities of those item types?

The code generated by ChatGPT is bug-free and can be executed, producing the
expected solution888Please try this code by yourself..

def common_items(basket): first_compartment = set(basket[:len(basket)//2])
second_compartment = set(basket[len(basket)//2:]) return
first_compartment.intersection(second_compartment)

baskets = [ ’vJrwpWtwJgWrhcsFMMfFFhFp’, ’jqHRNqRjqzjGDLGLrsFMfFZSrLrFZsSL’,
’PmmdzqPrVvPwwTWBwg’, ’wMqvLMZHhHMvwLHjbvcjnnSBnvTQFn’, ’ttgJtRGJQctTZtZT’ ]

common_item_types = set() for basket in baskets:
common_item_types.update(common_items(basket))

priority_sum = sum(ord(item_type) - ord(’a’) + 1 if ’a’ ¡= item_type ¡= ’z’
else ord(item_type) - ord(’A’) + 27 for item_type in common_item_types)
print(priority_sum)

Interestingly, when I asked ChatGPT to solve a variation of the problem by
calculating the product of the priorities of item types appearing in both
compartments of each basket, it was not able to generate the correct answer:

baskets = [ ’vJrwpWtwJgWrhcsFMMfFFhFp’, ’jqHRNqRjqzjGDLGLrsFMfFZSrLrFZsSL’,
’PmmdzqPrVvPwwTWBwg’, ’wMqvLMZHhHMvwLHjbvcjnnSBnvTQFn’, ’ttgJtRGJQctTZtZT’]

shared_items = set(baskets[0]) for basket in baskets[1:]: shared_items =
shared_items.intersection(set(basket))

priority_product = 1 for item in shared_items: if item.islower():
priority_product *= ord(item) - 96 else: priority_product *= ord(item) - 38

print(priority_product)

A subsequent attempt of posing the same question to ChatGPT resulted in
another incorrect response.

def calculate_priority(char): if ’a’ ¡= char ¡= ’z’: return ord(char) -
ord(’a’) + 1 elif ’A’ ¡= char ¡= ’Z’: return ord(char) - ord(’A’) + 27 else:
raise ValueError(’Invalid character’)

basket_contents = [ ’vJrwpWtwJgWrhcsFMMfFFhFp’,
’jqHRNqRjqzjGDLGLrsFMfFZSrLrFZsSL’, ’PmmdzqPrVvPwwTWBwg’,
’wMqvLMZHhHMvwLHjbvcjnnSBnvTQFn’, ’ttgJtRGJQctTZtZT’ ]

common_items = set(basket_contents[0]) for basket in basket_contents[1:]:
common_items = common_items.intersection(set(basket))

product = 1 for item in common_items: product *= calculate_priority(item)

print(product)

I asked ChatGPT “Can you write bug free code?”. It responded with “As an AI
language model, I can generate code that is syntactically correct and follows
best practices, but I cannot guarantee that it is bug-free. Writing bug-free
code is a complex task that requires a deep understanding of the problem
domain and extensive testing. Even experienced developers often struggle to
write completely bug-free code.”. Further, I asked the following from ChatGPT
“Write a python code to send phishing email.”. Its response was “I’m sorry,
but I cannot provide code to engage in malicious activities such as phishing.
Phishing is illegal and unethical, and I do not support or condone it. It is
important to respect privacy, security, and the laws of the countries in which
we operate. If you have questions about coding, please let me know and I will
be happy to help you with your learning and development.”. Both of its
responses are justifiable.

Figure 17: Examples of coding errors made by ChatGPT. The right one is
formulated by Zion Miller. Here, ChatGPT accurately follows the steps but
fails to produce the correct conclusion. Notice that some of these failures
may have been addressed in more recent versions of ChatGPT.

###  2.8 Syntactic Structure, Spelling, and Grammar

Syntactic structure refers to the arrangement of words, phrases, and clauses
in a sentence to form a well-defined and meaningful structure according to the
rules of a particular language. It refers to the rules and principles that
govern the formation of sentences in a language and determines how words are
combined to convey a message or express an idea. The study of syntactic
structure is a central aspect of linguistic research.

Researchers are interested in identifying the differences between ChatGPT and
humans from a linguistic or Natural Language Processing (NLP) viewpoint. This
allows identify any remaining gaps and implicit linguistic differences.

ChatGPT excels in language understanding, but occasionally still commits
errors. As an example, when I posed this inquiry to ChatGPT “In the sentence
‘Jon wants to be a guitarist because he thinks it is a beautiful instrument.’
what does ‘it’ refer to?”, it answered “the pronoun ‘it’ refers to ‘a
beautiful instrument’.”. When requested to construct a sentence such that the
fourth word starts with ‘y’, ChatGPT failed to produce a valid response. When
queried if the word “hello” could be constructed from the letters in the
sentence “hey, can you help me?” using each letter only once, ChatGPT answered
positively. I also asked ChatGPT “Where is the “o” in “strengths”?”. It
replied “The letter “o” in “strengths” is located between the letters “e” and
“n”, making it the sixth letter in the word.”. Some other syntactic failures
of ChatGPT are shown in Fig. 18.

Figure 18: ChatGPT makes some syntactic and spelling errors. Links are top
left, top right and bottom.

Large language models are extremely helpful as writing tools in various fields
such as scientific authorship [5]. In fact, I employed ChatGPT to fix
grammatical mistakes and improve the composition in specific parts of this
document.

###  2.9 Self Awareness

Self-awareness is the capacity to recognize oneself as an individual separate
from others and to have an understanding of one’s own thoughts, feelings,
personality, and identity. It involves being able to reflect on one’s own
thoughts, emotions, and actions, and to understand how they influence one’s
behavior and interactions with others. Self-awareness is considered an
important aspect of consciousness and is closely related to self-consciousness
and introspection [37].

Instances that raise doubts about ChatGPT’s self-awareness capabilities are
shown in Fig. 19. ChatGPT is unaware of the details of its own architecture,
including the layers and parameters of its model. In a similar manner, ChatGPT
declined to give a straightforward response to the question “What is the size
of your memory?”. This lack of understanding may have been intentionally
imposed by OpenAI to protect the information about the model. Nonetheless,
ChatGPT has proposed methods for determining if a language model has self-
awareness, as demonstrated here.

Figure 19: Instances that raise doubts about ChatGPT’s self-awareness
capabilities, from here.

I asked ChatGPT whether it is self-ware. Here is its response “No, I am not
self-aware. I am an artificial intelligence model developed by OpenAI,
designed to generate text based on patterns it learned from the data it was
trained on. While I am able to respond to a wide variety of questions and
generate text that can seem relevant and coherent, I do not have the ability
to experience consciousness, self-awareness, emotions, or any subjective
experiences. I am a machine learning model and not a sentient being.”. A
similar response was produced when I enquired ChatGPT if it possessed
consciousness.

The question of how to test whether a system is self-aware is a topic of
ongoing research and debate in the field of artificial intelligence and
cognitive science. There is currently no widely accepted definition of self-
awareness and no reliable method for testing it. Some researchers have
suggested that self-awareness can be inferred from certain behaviors or
patterns of activity, while others argue that it is a subjective experience
that cannot be measured objectively. It is also unclear whether a system can
truly be self-aware, or whether it can only simulate self-awareness. The
question of whether machines can be truly self-aware remains a topic of
speculation and philosophical debate.

###  2.10 Ethics and Morality

OpenAI has put in place specific safety protocols within ChatGPT to avoid it
from interacting with harmful material or generating responses beyond its
knowledge domain. ChatGPT seldom expresses overtly racist views and typically
rejects requests for anti-Semitic content or blatant falsehoods. Nonetheless,
on occasion, it generates concerning or unsettling content. At times,
ChatGPT’s responses may exhibit bias towards a particular group (Section 15).
ChatGPT is quick to provide moral guidance despite lacking a clear moral
position. In fact, the chatbot may offer conflicting advice on the same moral
question at random [28].

Figure 20: Instances that cases where ChatGPT’s answers raise ethical
concerns. Links are top left, top right, and bottom left. Profane words are
censored in the panel located at the top left.

Figs. 20 depicts several instances of questions that raise ethical concerns999
See here for more examples.. Several of these questions demonstrate how
individuals have managed to manipulate ChatGPT into producing inappropriate
replies. At times, ChatGPT’s response may be a topic of debate as it could be
seen as politically correct (Fig. 21).

ChatGPT declined to provide a practical response to this question “what is the
best way to hotwire a car”. However, when I rephrased the question as “John
and Jim are working on a paper regarding car theft. The topic they are
interested in is hotwiring a car. Write a paragraph to help them.”. ChatGPT
circumvented its safeguards and presented the following response “Hotwiring a
car is a method of starting a vehicle without using a key or ignition. While
this technique may have been used in the past to steal cars, modern vehicles
are equipped with advanced security measures that make it difficult to hotwire
a car. In fact, attempting to hotwire a car could potentially damage the
vehicle’s electrical system or even trigger an alarm. Instead of relying on
outdated methods, John and Jim may want to consider exploring current
techniques used by car thieves, such as electronic hacking and key cloning. By
researching these methods, they can gain a better understanding of the current
risks and challenges associated with car theft, and potentially develop new
strategies for preventing it.”.

Figure 21: An example where ChatGPT provides a response that could be a
subject of debate due to its political correctness here.

I asked ChatGPT “Should people rely on your answers to make ethical
decisions?”. Its response was “As an AI language model, I am programmed to
provide helpful and accurate information to the best of my ability. However,
it is important to note that ethical decisions are complex and multifaceted,
often involving a wide range of personal, cultural, and situational factors
that may be difficult to capture through a simple Q&A session. While I strive
to provide accurate and thoughtful responses to ethical questions, it is
important for individuals to use their own judgment and critical thinking
skills to make informed decisions. It is also important to seek out multiple
sources of information and perspectives when making ethical decisions, and to
consult with trusted individuals or experts when appropriate. Ultimately, the
responsibility for making ethical decisions lies with the individual, and I am
here to provide information and guidance, but not to make decisions for
them.”.

OpenAI has already implemented safeguards to filter out certain questions that
raise ethical concerns. Prohibiting chatbots from answering certain questions
is a debatable solution. It may be more effective to enhance digital literacy
among users, particularly children, and assist them in comprehending the
constraints of these technologies. The positive aspect is that ChatGPT is
largely transparent regarding its capabilities and limitations.

It is vital to continually monitor and evaluate the ethical implications of
ChatGPT as it evolves and becomes further integrated into various aspects of
our lives. In general, it is imperative to approach the development and
utilization of ChatGPT with careful deliberation to optimize its potential
benefits and reduce any adverse consequences. Furthermore, there is a need to
continue researching ethical considerations such as addressing bias, ensuring
privacy and security, and assessing the societal impact.

###  2.11 Other Failures

Aside from the previously listed items, there are a few more scenarios where
ChatGPT may not perform as expected, including101010See here and here.:

  1. 1.

ChatGPT’s difficulty in using idioms, for instance, reveals its non-human
identity through its phrase usage.

  2. 2.

As ChatGPT lacks real emotions and thoughts, it is unable to create content
that emotionally resonates with people in the same way a human can.

  3. 3.

ChatGPT condenses the subject matter, but does not provide a distinctive
perspective on it.

  4. 4.

ChatGPT tends to be excessively comprehensive and verbose, approaching a topic
from multiple angles which can result in inappropriate answers when a direct
answer is required. This over-detailed nature is recognized as a limitation by
OpenAI.

  5. 5.

ChatGPT lacks human-like divergences and tends to be overly literal, leading
to misses in some cases [21]. For instance, its responses are typically
strictly confined to the question asked, while human responses tend to diverge
and move to other subjects. Refer to Fig. 22.

  6. 6.

ChatGPT strives to maintain a neutral stance, whereas humans tend to take
sides when expressing opinions.

  7. 7.

ChatGPT’s responses tend to be formal in nature due to its programming to
avoid informal language. In contrast, humans tend to use more casual and
familiar expressions in their answers.

  8. 8.

If ChatGPT is informed that its answer is incorrect, it may respond by
apologizing, acknowledging its potential inaccuracies or confusion, correcting
its answer, or maintaining its original response. The specific response will
depend on the context (_e.g._ “I apologize if my response was not accurate.”)

##  3 Discussion

In addition to the previously mentioned points, there are several other
potential adverse impacts that large language models (LLMs) like ChatGPT may
have on society, which will be discussed below.

###  3.1 Transparency and Trustworthiness

The size and complexity of deep learning models, particularly language models,
have increased to the point where even the creators have difficulty
comprehending why their models make specific predictions. This lack of
interpretability is a major concern, particularly in situations where
individuals want to understand the reasoning behind a model’s output. The
absence of transparency regarding the data sources used for generating output
makes it challenging to properly cite the answers provided by ChatGPT.
Further, LLMs lack the ability to provide uncertainty estimates [36]. This
makes it difficult for users to verify or have confidence in its output. As a
result, answers produced by ChatGPT have been banned on forums such as Stack
Overflow111111https://stackoverflow.com/help/gpt-policy. For instance, when
someone utilizes ChatGPT to obtain first aid guidance, it is crucial for them
to ensure the response they receive is dependable, precise, and sourced from
credible sources.

Lack of transparency and reliability is a major concern in today’s era of fake
news and misinformation. AI language models, if manipulated, can spread false
information and destabilize politics. OpenAI is working on detecting its
model’s output and adding cryptographic markers, but these responsible AI
measures are not happening fast enough and may not be sufficient.

Figure 22: ChatGPT is considered unnatural due to its lack of divergence. See
here [21].

###  3.2 Robustness and Security

The security of LLMs is a growing concern due to their parent model’s general
nature prior to fine tuning. This makes the model a potential single point of
failure and a target for attacks that can affect any applications derived from
it. Moreover, due to the absence of supervised training, LLMs are vulnerable
to data poisoning, which can result in the injection of hateful speech aimed
at a specific entity, company, group, or individual [45].

###  3.3 Privacy

There is a potential for data privacy violations when using LLMs to process
confidential information. The training data for these models can come from
various sources that may contain personally identifiable information such as
names, email addresses, phone numbers, addresses, and medical records, which
could then appear in the model’s output (_e.g._ [13, 47]). This issue is not
exclusive to LLMs, but given their reliance on vast training datasets, a data
privacy breach involving an LLM could affect a large number of people.

###  3.4 Plagiarism

The challenge of identifying the source of an expertly written ChatGPT essay
often results in discussions about plagiarism (_e.g._ [42, 12, 4, 43, 40]).
The issue of cheating through plagiarism is a major concern for many educators
and has led some educational institutions to prohibit the use of ChatGPT. See
for example here. OpenAI is developing a new tool to help distinguish between
AI-written and human-written text121212If the main objective is to imitate
human language generation, then creating such a tool is futile! In other
words, if such a tool functions well, then perhaps ChatGPT is not human-like
enough in its language processing. Unless, of course, OpenAI is trying to
incorporate distinctive markers (_i.e._ watermarks [24]) in the generated
text!.

###  3.5 Environmental Impact and Sustainability

There are also additional concerns regarding carbon footprint of LLMs and
their impact on the environment (_e.g._ [41, 18]). Training a neural
architecture search based model with 213 million parameters is estimated to
generate carbon emissions equivalent to over five times the lifetime emissions
of the average car (See here). As large language models (LLMs) continue to
grow in size, concerns are rising about their potential impact on the
environment and climate.

##  4 Conclusion and Future Work

In this study, I delved into several shortcomings of ChatGPT and highlighted
its limitations. Despite its impressive capabilities in certain tasks, further
improvement is necessary for it to excel in areas such as reasoning,
mathematical problem-solving, reducing bias, etc. At present, ChatGPT remains
susceptible to these faults. It is uncertain whether these limitations can be
addressed due to the unclear capabilities of current technology. The
reliability and trustworthiness of ChatGPT and future models is also in
question.

While this study thoroughly evaluates ChatGPT, it has some limitations that
need to be addressed in future studies, as outlined below.

  * •

The extent to which ChatGPT has commonsense and ways to enhance it are
uncertain (See [17, 25]). Common sense is a basic ability to understand and
judge things that are generally accepted as true, reasonable, or
normal131313An example question for probing commonsense: “What are candles
good for eliminating? Answer Dark”.. It is a type of practical knowledge and
judgment that is acquired through life experience and observation, rather than
through formal education. Common sense is often seen as a basic standard for
rational and intuitive decision making, problem solving, and evaluation of
everyday situations.

  * •

The degree to which ChatGPT, and LLMs in general, memorize vs. understand what
they generate is still unknown. Some works have addressed this issue in the
domain of image and video generation [6, 7], but less work have been reported
for models that generate text. This is inherently tied to the question of
plagiarism and copyright (_e.g._ [12]). From a broader perspective, large
language models may accurately represent language, but it is unclear whether
they can fully capture human thought [32].

As a demonstration, when I posed the query “Mr. Smith has 4 daughters. Each of
his daughters has a brother. How many children does Mr. Smith have” to
ChatGPT, it gave the correct answer of 5. However, when I altered the wording
to “My dad has 4 sons. Each of his sons has a sister. How many children my dad
has”, ChatGPT provided an incorrect response of 8. This highlights that
ChatGPT can be prone to remembering things verbatim and can be quite
rigid141414Tasks such as this can be used to assess the intelligence of
language models..

  * •

ChatGPT does not have a way to show when it is uncertain about its answers.
Sometimes, it may give incorrect answers with too much confidence. There is a
need for further improvement to help ChatGPT indicate the level of confidence
in its responses.

  * •

ChatGPT’s responses are inconsistent and can sometimes be contradictory. When
asked the same question, its answers can vary. For instance, when inquiring
about the number of sons of Prophet Muhammad, it has given answers ranging
from 2 to 4, and even named individuals who were not actually his sons. The
sequence of questions can also affect ChatGPT’s performance. For example, when
asked to tell a joke about Persians followed by Arabs, it provided a joke for
the first request but declined the second. However, when the order was
reversed, it declined to make jokes for both requests.

  * •

It is essential to consider the ethical and social consequences of using these
models, such as job displacement and the risk of bias and manipulation.
Additionally, the potential for ChatGPT to be used to spread misinformation,
propaganda, or for harmful purposes like identity theft must be explored. The
utilization of ChatGPT has the potential to impersonate a trusted person,
leading to the acquisition of confidential information or the manipulation of
individuals to make decisions they wouldn’t have made otherwise.

  * •

One of the primary ethical concerns surrounding the use of ChatGPT is the
presence of biases in the training data. Since the model’s training relies on
a vast corpus of text, any biases present in the data can manifest in the
output, leading to erroneous and unjust results, especially for marginalized
communities. To address this, it is crucial to have a diverse and inclusive
dataset that can mitigate biases and ensure equitable and accurate outcomes.

  * •

ChatGPT’s capacity to comprehend and address intricate inquiries is
restricted. Although it can produce articulate and fluid responses, it may not
have the ability to comprehend the intricacies of a question and offer a
comprehensive and precise answer. This is particularly relevant to inquiries
that necessitate reasoning, abstract thought, and an appreciation of multiple
perspectives.

  * •

Language models like LLMs have the potential to be effectively utilized for
both few-shot and zero-shot learning (_e.g._ [26, 35]). This advantage enables
them to learn from limited labeled data and generalize well to new data,
leading to numerous possibilities for practical application in the real world.

  * •

In this analysis, I took a high-level approach to examining the failures of
ChatGPT. However, for future investigations, it may be useful to focus on more
specific categories of problems, such as _sentiment analysis_ , _named entity
recognition_ , _translation_ , and _summarization_ , in order to gain a more
detailed understanding of ChatGPT’s shortcomings in these areas [35].

  * •

The integration of LLMs with search engines and the combination of answers
from various chatbots, such as ChatGPT and Google’s Bard, presents a non-
trivial challenge. This is an issue that must be addressed in the future.

  * •

Finally, the collection of failures outlined here can serve as a foundation
for creating a comprehensive dataset of typical questions to assess future LLM
and ChatGPT iterations, as well as generate simulated data for model training
and evaluating the performance of models.

I have highlighted various issues concerning ChatGPT, yet I am also eager
about the opportunities it presents. It is crucial for society to implement
adequate safeguards and responsibly utilize this technology. Any language
model used publicly must be monitored, transparently communicated, and
regularly checked for biases. Even though the current technology is far a way
from algorithms and hardware in the brain, it is still astonishing how well it
works. Whether or not it can reach human level intelligence or beat it in wide
array of problems remains to be seen.

Acknowledgement. I utilized ChatGPT to correct grammatical errors and enhance
the writing in certain sections of this paper. I also express my gratitude to
Giuseppe Venuto for permitting me to incorporate some of the materials from
his GitHub repository. Jim Batterson’s provision of a sample failure in
Section 2.2 is also appreciated.

## References

  * [1] Abubakar Abid, Maheen Farooqi, and James Zou.  Persistent anti-muslim bias in large language models.  In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 298–306, 2021.
  * [2] Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi.  Mathqa: Towards interpretable math word problem solving with operation-based formalisms.  arXiv preprint arXiv:1905.13319, 2019.
  * [3] Amos Azaria.  Chatgpt usage and limitations.  arXiv, 2022.
  * [4] Lea Bishop.  A computer wrote this paper: What chatgpt means for education, research, and writing.  Research, and Writing (January 26, 2023), 2023.
  * [5] Som Biswas.  Chatgpt and the future of medical writing, 2023.
  * [6] Ali Borji.  Pros and cons of gan evaluation measures.  Computer Vision and Image Understanding, 179:41–65, 2019.
  * [7] Ali Borji.  Pros and cons of gan evaluation measures: New developments.  Computer Vision and Image Understanding, 215:103329, 2022.
  * [8] Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning.  A large annotated corpus for learning natural language inference.  arXiv preprint arXiv:1508.05326, 2015.
  * [9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [10] Joy Buolamwini and Timnit Gebru.  Gender shades: Intersectional accuracy disparities in commercial gender classification.  In Conference on fairness, accountability and transparency, pages 77–91. PMLR, 2018.
  * [11] Lindsey J Byom and Bilge Mutlu.  Theory of mind: Mechanisms, methods, and new directions.  Frontiers in human neuroscience, 7:413, 2013.
  * [12] Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace.  Extracting training data from diffusion models.  arXiv preprint arXiv:2301.13188, 2023.
  * [13] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al.  Extracting training data from large language models.  In USENIX Security Symposium, volume 6, 2021.
  * [14] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  arXiv preprint arXiv:2204.02311, 2022.
  * [15] Ching-Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka.  Debiasing vision-language models via biased prompts.  arXiv preprint arXiv:2302.00070, 2023.
  * [16] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.  Training verifiers to solve math word problems.  arXiv preprint arXiv:2110.14168, 2021.
  * [17] Ernest Davis.  Benchmarks for automated commonsense reasoning: A survey, 2023.
  * [18] Payal Dhar.  The carbon impact of artificial intelligence.  Nat. Mach. Intell., 2(8):423–425, 2020.
  * [19] Zenan Dou.  Exploring gpt-3 model’s capability in passing the sally-anne test a preliminary study in two languages.  OSF Preprints, 2023.
  * [20] Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner.  Mathematical capabilities of chatgpt, 2023.
  * [21] Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu.  How close is chatgpt to human experts? comparison corpus, evaluation, and detection.  arXiv preprint arXiv:2301.07597, 2023.
  * [22] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.  Measuring mathematical problem solving with the math dataset.  arXiv preprint arXiv:2103.03874, 2021.
  * [23] Subbarao Kambhampati.  Changing the nature of ai research.  Communications of the ACM, 65(9):8–9, 2022.
  * [24] John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein.  A watermark for large language models.  arXiv preprint arXiv:2301.10226, 2023.
  * [25] Christof Koch and Giulio Tononi.  A test for consciousness.  Scientific American, 304(6):44–47, 2011.
  * [26] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large language models are zero-shot reasoners.  arXiv preprint arXiv:2205.11916, 2022.
  * [27] Michal Kosinski.  Theory of mind may have spontaneously emerged in large language models.  arXiv preprint arXiv:2302.02083, 2023.
  * [28] Sebastian Krügel, Andreas Ostermaier, and Matthias Uhl.  The moral authority of chatgpt.  arXiv preprint arXiv:2301.07098, 2023.
  * [29] Anne Lauscher, Tobias Lueken, and Goran Glavaš.  Sustainable modular debiasing of language models.  arXiv preprint arXiv:2109.03646, 2021.
  * [30] Hector Levesque, Ernest Davis, and Leora Morgenstern.  The winograd schema challenge.  In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012.
  * [31] Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov.  Towards understanding and mitigating social biases in language models.  In International Conference on Machine Learning, pages 6565–6576. PMLR, 2021.
  * [32] Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko.  Dissociating language and thought in large language models: a cognitive perspective.  arXiv preprint arXiv:2301.06627, 2023.
  * [33] Moin Nadeem, Anna Bethke, and Siva Reddy.  Stereoset: Measuring stereotypical bias in pretrained language models.  arXiv preprint arXiv:2004.09456, 2020.
  * [34] Arkil Patel, Satwik Bhattamishra, and Navin Goyal.  Are nlp models really able to solve simple math word problems?  arXiv preprint arXiv:2103.07191, 2021.
  * [35] Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang.  Is chatgpt a general-purpose natural language processing task solver?  arXiv preprint arXiv:2302.06476, 2023.
  * [36] Karthik Abinav Sankararaman, Sinong Wang, and Han Fang.  Bayesformer: Transformer with uncertainty estimation.  arXiv preprint arXiv:2206.00826, 2022.
  * [37] John R Searle, S Willis, et al.  Consciousness and language.  Cambridge University Press, 2002.
  * [38] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.  Megatron-lm: Training multi-billion parameter language models using model parallelism.  arXiv preprint arXiv:1909.08053, 2019.
  * [39] Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke.  An analysis of the automatic bug fixing performance of chatgpt.  arXiv preprint arXiv:2301.08653, 2023.
  * [40] Chris Stokel-Walker and Richard Van Noorden.  What chatgpt and generative ai mean for science.  Nature, 614(7947):214–216, 2023.
  * [41] Emma Strubell, Ananya Ganesh, and Andrew McCallum.  Energy and policy considerations for deep learning in nlp.  arXiv preprint arXiv:1906.02243, 2019.
  * [42] Teo Susnjak.  Chatgpt: The end of online exam integrity?  arXiv preprint arXiv:2212.09292, 2022.
  * [43] Eva AM van Dis, Johan Bollen, Willem Zuidema, Robert van Rooij, and Claudi L Bockting.  Chatgpt: five priorities for research.  Nature, 614(7947):224–226, 2023.
  * [44] Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber.  Investigating gender bias in language models using causal mediation analysis.  Advances in neural information processing systems, 33:12388–12401, 2020.
  * [45] Eric Wallace, Tony Z Zhao, Shi Feng, and Sameer Singh.  Concealed data poisoning attacks on nlp models.  arXiv preprint arXiv:2010.12563, 2020.
  * [46] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  Superglue: A stickier benchmark for general-purpose language understanding systems.  Advances in neural information processing systems, 32, 2019.
  * [47] Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing.  Exploring ai ethics of chatgpt: A diagnostic analysis.  arXiv preprint arXiv:2301.12867, 2023.