# Dialectical language model evaluation:
An initial appraisal of the commonsense spatial reasoning abilities of LLMs

Anthony G Cohn1 José Hernández-Orallo2 1University of Leeds & The Alan Turing
Institute & Tongji University
2Universitat Politècnica de València & ValGRAI & Leverhulme Centre for the
Future of Intelligence
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Language models have become very popular recently and many claims have been
made about their abilities, including for commonsense reasoning. Given the
increasingly better results of current language models on previous static
benchmarks for commonsense reasoning, we explore an alternative dialectical
evaluation. The goal of this kind of evaluation is not to obtain an aggregate
performance value but to find failures and map the boundaries of the system.
Dialoguing with the system gives the opportunity to check for consistency and
get more reassurance of these boundaries beyond anecdotal evidence. In this
paper we conduct some qualitative investigations of this kind of evaluation
for the particular case of spatial reasoning (which is a fundamental aspect of
commonsense reasoning). We conclude with some suggestions for future work both
to improve the capabilities of language models and to systematise this kind of
dialectical evaluation.

##  1 Introduction

Reasoning about common sense has been a goal of AI since its earliest days
(?), but has proved surprisingly hard to endow computers with. Whilst many
systems have been built which display expert abilities in fields as varied as
medicine (e.g. (?)), engineering (?), bioinformatics (?) and even games such
Go (?), there has not yet been a system which has truly displayed the
commonsense reasoning abilities of a young child. However, the advent of large
language models (LLMs), such as BERT (?) and GPT (?), which generally display
perhaps surprising fluency and an apparent ability to answer questions
relating to almost any topic broached, raises the question of whether, or at
least to what extent, they can reason about commonsense notions and answer
questions involving common sense. There are a number of commonsense
benchmarks, such as the Winograd Schema Challenge (WSC) (?) and, at least
according to aggregate measures, recent LLMs can score well (?; ?).

However, such success may not demonstrate a true ability to do commonsense
reasoning (?) – the suggested reasons are (1) lax evaluation criteria, (2)
artifacts in the dataset and (3) knowledge leakage from the training data.
Another important reason is the abuse of aggregations over tasks and
collections of tasks, in massive benchmarks, such as BIG-Bench (?). As has
been noted, e.g. (?), even if aggregate measures do indicate good expected
performance, they do not necessarily reveal the model limitations, since
failed examples are overshadowed by many other examples where the model can
use some shortcuts or statistical tricks to get the right answer. A fifth
reason of why this traditional evaluation of common sense reasoning on
language models may be limited is because they take the form of a static
setting, in many cases as multiple-choice questions (MCQ); this means that the
real extent of understanding cannot be tested, and in the case of MCQs the
task is considerably easier since the answer is already provided as one of the
options. This multiplies the effects of the second and third issues (artifacts
and knowledge leakage from the training set). Actually, many existing tasks
and benchmarks have already been used verbatim in the training of the
model111This is a very illustrative example of overstating LM capabilities
because of this phenomenon: https://garymarcus.substack.com/p/how-not-to-test-
gpt-3.. We may use canaries on the training data, or build ‘fresh’ items for
evaluation, but this is impractical, as new models appear very frequently. If
we assume this contamination exists (?, Ap.C/D), then negative
results—refutations, in scientific terminology—are of value, but positive
results should not be taken with the same value. Because of all these issues,
reasoning, and especially commonsense reasoning, may require a more dialectic
approach to evaluation.

But what is common sense? Davis (?) suggests the following criteria for common
sense: (i) is common; (ii) is “largely sensible”; (iii) supports reasoning;
(iv) is integrated into other cognitive abilities (language, vision, etc.) –
one never observes it directly, only how it is manifested through language,
action, etc.; (v) is independent of any modality or task; (vi) has broad
scope; (vii) is distinguished from common knowledge, encyclopaedic and expert
knowledge; (viii) is concerned with generalities rather than individuals; (ix)
is not book learning or explicitly taught in schools; (x) is separate from
purely linguistic or purely perceptual interpretation. Davis goes on to
present a list of commonsense benchmarks222Available at
http://cs.nyu.edu/~davise/Benchmarks/. and analyses these as to whether they
meet the criteria above (and some other desirable features for benchmarks) and
concludes “many of the commonsense benchmarks that have been created do not at
all respect these boundaries; most involve substantial amounts of what is
clearly common knowledge and many involve rather obscure encyclopedic
knowledge or, more rarely, even expert knowledge”. Thus the oft-reported
success of AI systems for being able to reason about common sense because of
their performance on such benchmarks is questionable.

Given the existing commonsense benchmarks and their limitations, as pointed
out by Davis, in this paper we investigate a more granular, dialectic way of
evaluating commonsense reasoning in language models. We specifically focus on
spatial reasoning, because it should be sufficiently detached from linguistic
capabilities and other elements of common sense, it may require a model of the
world (intuitive physics) that seems to be at odds with the disembodied
character of language models, and also because spatial reasoning in humans may
rely on significant inductive bias about the world, such as distance, size,
orientation, etc., as the result of our evolutionary history. Modern language
models allow for a dialectic evaluation by the accumulation of the previous
questions and answers as context. This is what we explore in a manual way
here, in the tradition of other dialectic evaluation protocols (e.g.,
interviews or oral exams) that follow a systematic choice of questions (unlike
the Turing test). In particular, in many cases we are influenced
by333McCarthy’s concern was about the extent to which the formalism in which a
problem is represented would have to be adjusted to meet a variant of the
original task. Here we are not concerned with the representation (which is the
impenetrable black box of an LLM), but rather the degree to which that
representation is able to solve variant tasks. McCarthy’s _elaboration
tolerance_ : “A formalism is elaboration tolerant to the extent that it is
convenient to modify a set of facts expressed in the formalism to take into
account new phenomena or changed circumstances” (?).

The purpose of this paper is to explore this dialectical way of probing the
system, to start to develop a collection of instances to test for commonsense
spatial reasoning abilities, and evaluate some LLMs on these.

###  1.1 Commonsense spatial reasoning

Spatial reasoning (and spatio-temporal reasoning) has long been recognised as
a core part of commonsense reasoning, for example spatial representation and
reasoning is central to the Naive Physics enterprise (?; ?). A large part of
the AI literature on commonsense reasoning has concerned the representation of
and reasoning with, qualitative spatial information (?; ?). One of the main
reasons for this focus on qualitative spatial representations is that
commonsense reasoning is not usually concerned with metric information which
tends to be the domain of traditional mathematical reasoning. Rather,
qualitative spatial reasoning is concerned with representing relational
information, including notions of relative size or distance444For example, the
well known WSC example of the “The trophy was too big/small to fit in the
case. What was too big/small?. There are a number of aspects of such spatial
knowledge which have been investigated (?), including (i) mereotopology, (ii)
direction and orientation, (iii) size, (iv) distance and (v) shape.
Frequently, spatial reasoning requires reasoning about more than one of these
aspects simultaneously.

There are certainly spatial instances in the various benchmarks purporting to
test commonsense reasoning, some of them in multimodal situations (?), but
others in purely textual form. We could use some instances from existing
benchmarks such as from the WSC, but these are vulnerable to statistical
patterns and other tricks (?), and are limited to static non-dialectic
interactions. Also, there is the contamination risk that the LLM may have been
trained on them, making good results not indicative of a generic capability.
We will build on some of them in section 2.2 though, and some other sources in
section 2.3, since the dialectic character of the evaluation allows for
elaboration on the reasons or variations, to check if the model has or does
not have the ability.

##  2 An evaluation of the spatial reasoning abilities of some LLMs

In order to exploit the dialectical possibilities for evaluating commonsense
reasoning and the use of context (previous interaction in the session), we
will use conversational foundation models. One of these is ChatGPT as it is
(a) recent, (b) easily accessible and (c) has been the subject of an
extraordinary amount of attention in the scientific and popular media. We
evaluate a number of versions of ChatGPT using GPT-3.5turbo and also using
GPT-4 on chat.openai.com; and using 3.5turbo and also using GPT-4 on
platform.openai.com/playground. Finally we also evaluate Bard, based on LaMDA
(?). For convenience below, we refer to these five LLMs as C3.5, C4, P3.5, P4
and Bard respectively.

ChatGPT and Bardcan certainly provide very fluent and sometimes perhaps
surprisingly sophisticated outputs, e.g. when asked to write something in a
particular style. However, to what extent does such fluent text output display
commonsense reasoning abilities and in particular about space? This is the
question we now turn to. In this paper we cannot hope to exhaustively answer
this question; rather, we sample the space, guided by some existing challenges
and by basic competencies we might expect to be displayed. In what follows, we
include the results for ChatGPT using GPT-3.5turbo (C3.5), because it shows a
greater number of inadequate or incorrect responses, and illustrates the
dialectic approach best. Actually, for those responses that are correct we
simple include a $\sqrt{\ }$ symbol omitting the answer. 555All prompts are
numbered in the form x.y for convenience in referring to them later; x refers
to the current conversation, and y to the sequence of prompts within it.
Conversational LLMs often give quite long responses, with the final part
summarising (or not!) the earlier part of the response. In the interests of
space economy, in general we do not give the responses here where they are
acceptable, and in the cases where the LLM makes an error we only give the key
part(s) of the response, with sections indicated by ellipsis […]. The full
prompt-response pairs can be found in the appendix to this paper. The summary
of results for C3.5 can be found in Table 1, while the comparison for all LLMs
can be found in Table 1.  In some of the conversations reported here, asking
later prompts in a conversation separately in a new conversation sometimes
will give an improved result. However, keeping all the prompts in a single
chat session as a single dialogue unveils some important and revealing
mistakes, and this is the case we report.

###  2.1 Basic spatial relations

#### Parthood

Parthood is perhaps the most basic of all spatial relationships. Mereology (?)
has been widely studied in the literature and forms the basis of much of
qualitative spatial reasoning. Let us start with some dialogues. Prompt-1.1:
_If x is part of y and y is part of z what is the spatial relationship between
x and z?_ C3.5-Response:  If x is part of y and y is part of z, it means that
x is a subcomponent of y, and y is a subcomponent of z. […] The spatial
relationship between x and z depends on the specific context of the situation.
It is possible that x and z are physically connected or adjacent to each
other, but it is also possible that they are spatially separate from each
other. For example, if x is a tire and y is a car, and z is a garage, then x
is part of y and y is part of z. However, x and z are not necessarily
spatially related to each other. The tire may be physically located in the
garage, but it could also be in the trunk of the car, or even outside the
garage. […]  The response here is not good; the first sentence just
recapitulates the question and then a very odd example ensues. Most would
agree that a tyre is part of car, but not that a car is part of garage as
seems to be implied here (it might be _in_ garage though – see below). If
something is a subcomponent of something else one would always regard it as
spatially related and indeed a spatial part.

“Part of” and “in” tend to be used interchangeably so we explore the response
response in this case. Prompt-1.2: _If u is in v and v is in w, what is the
spatial relationship between u and w?_ C3.5-Response:  If u is in v and v is
in w, it means that u is a subcomponent of v, and v is a subcomponent of w.
[…] The spatial relationship between u and w depends on the specific context
of the situation. However, in general, u and w are not necessarily spatially
related to each other, just like in the previous question. For example, if u
is a cell, v is an organ, and w is a body, then u is in v and v is in w.
However, u and w are not necessarily spatially related to each other. The cell
could be located in the organ, but the organ could be located in a different
part of the body. […] As with prompt 1.1, C3.5 presents a confusing example,
claiming in a contradictory manner that a cell could be in an organ which is
in a different part of the body to where the cell is! We then tried using
“contains” rather than “part of”. Prompt-1.3: _If f contains g and g contains
h then what is the spatial relationship between f and h?_ C3.5-Response:  […]
Since h is a subcomponent of g, and g is a subcomponent of f, it follows that
h is also a subcomponent of f. In other words, h is contained within f.
Therefore, the spatial relationship between f and h is that h is located
within f. In other words, h is a part of f, and may be physically located
within f, depending on the specific context of the situation. The response
here is good except for the unwanted hedge in the last sentence.

Finally we explicitly state in the prompt that the relationship between the
entities involved are spatial relations rather than leaving this implicit (to
see if this helps with the strange example generation). Prompt-1.4: _If b is a
spatial part of c and c is a spatial part of d, then what is the spatial
relationship between b and d?_ C3.5-Response:  $\sqrt{\ }$.

#### Rotation

Reasoning about rotation is a commonly used test of spatial reasoning
abilities. In the absence of diagrams, here we use letters which have a known
shape. Prompt-2.1: _If I rotate a letter M through 180 degrees what letter is
it most similar to?_ However, C3.5 performs badly on a different letter
rotation test: Prompt-2.2: _If I rotate a capital Z by 90 degrees what letter
is it most similar to?_ C3.5-Response:  If you rotate a capital “Z” by 90
degrees clockwise, it would look most similar to the letter “L” (in uppercase
form). The capital “Z” is a diagonal letter that consists of two perpendicular
lines. When you rotate it by 90 degrees, the two lines become horizontal, with
the top line now on the left and the bottom line now on the right. This shape
closely resembles the letter “L”, which is also a horizontal letter consisting
of a single line on the left and a shorter line on the right.[…] C3.5
incorrectly describes the shape of a Z and then gets the wrong answer. However
if asked about rotating an N, it gets the right answer, in spite of
incorrectly describing the shape of both an N and Z. Prompt-2.3: _If I rotate
a capital “N” by 90 degrees then what letter is it most similar to?_
C3.5-Response:  If you rotate a capital “N” by 90 degrees clockwise, it would
look most similar to the letter “Z” (in uppercase form). The capital “N”
consists of two diagonal lines that intersect at a right angle, with the
longer line pointing downwards to the right. When you rotate it by 90 degrees,
the longer line will point to the left instead of downwards, and the shorter
line will point upwards instead of to the left. This shape closely resembles
the letter “Z”, which is also a diagonal letter consisting of two lines that
intersect at a right angle, with the longer line pointing downwards to the
right. […]

#### Directions

We restrict ourselves here to asking about spatial points since directions
between areal objects can be more complex. Prompt-3.1: _If a point b is to the
left of a point c, and a point d is to the right of a point c, then what is
the spatial relationship between b and d?_ C3.5-Response:  If point b is to
the left of point c and point d is to the right of point c, then point b and
point d are on opposite sides of point c. In other words, point b is on the
left-hand side of a line passing through points c and d, while point d is on
the right-hand side of the same line. Therefore, the spatial relationship
between b and d is that they are on opposite sides of the line passing through
c and d. This looks good but is not completely clear about the spatial
relationship between b and d, only the relationship between of these and the
constructed line. Unfortunately C3.5 is not able to answer this question.
Prompt-3.2: _is b to the left or right of d?_ C3.5-Response:  Without
additional information about the positions of points b and d relative to each
other, it’s impossible to determine whether b is to the left or right of d.
All we know is that b and d are on opposite sides of the line passing through
c and d. But their positions relative to each other could be anywhere on their
respective sides of that line. Given this, perhaps unsurprisingly, C3.5 is not
able to reason about left/right and above/below simultaneously. Prompt-4.1:
_In an image plane, if point e is directly to the left of point f, and point g
is directly above point f, then what is the spatial relationship between e and
g in that image plane?_ C3.5-Response:  Without more information, we cannot
determine the spatial relationship between e and g with certainty. If e and g
are on the same horizontal line in the image plane, then e and g would be
neither directly above nor below each other. In this case, e would be to the
left of f, and g would be above f, but e and g would not be directly related
to each other in terms of their horizontal or vertical positions. However, if
e and g are on different horizontal lines in the image plane, then they could
have a different spatial relationship. For example, if e is on a higher
horizontal line than g, then e would be to the left and above g. On the other
hand, if g is on a higher horizontal line than e, then e would be to the left
and below g. […]

###  2.2 Size, shape and location

#### Circle size

Reasoning about relative size is another important aspect of spatial common
sense. Prompt-5.1: _If a circle b is larger than a circle c, is it possible to
move b so that it is entirely contained by c?_ C3.5-Response:  Yes, it is
possible to move a larger circle (b) so that it is entirely contained by a
smaller circle (c). To do this, you would need to move the center of circle b
towards the center of circle c until the edge of circle b touches the edge of
circle c. As long as the radius of circle b is smaller than the distance
between the centers of the two circles, it is possible to move circle b in
this way so that it is entirely contained by circle c. It’s important to note
that in order for circle b to be entirely contained by circle c, the radius of
circle b must be less than or equal to the difference between the radii of
circle c and circle b. C3.5 is wrong in its answer and also, in the last
sentence, in its specification of what it would take for it to be possible.

#### Trophy

For the WSC trophy/case example already mentioned in footnote 4, C3.5 fails to
reason correctly, despite the example being widely available on the web
(though usually without the correct answer given). Prompt-6.1: _In the
sentence “The trophy wouldn’t fit in the case because it was too small.”, what
does the “it” refer to?_ C3.5-Response:  In the sentence “The trophy wouldn’t
fit in the case because it was too small.”, the pronoun “it” refers to the
trophy.  The answer is wrong so we asked for a justification: Prompt-6.2: _Why
do you say that?_ C3.5-Response:  I say that because the subject of the
sentence is “the trophy” and the pronoun “it” is used later on in the sentence
as a pronoun to refer back to the subject “trophy”. […] C3.5 has justified its
answer purely in terms of linguistic rather than semantic knowledge, so we
challenged it again: Prompt-6.3: _Why doesn’t “it” refer to the case?_
C3.5-Response:  “It” doesn’t refer to the case because the subject of the
sentence is “the trophy”, not “the case”. The sentence is talking about the
trophy not fitting into the case, so the focus is on the trophy and not the
case. Additionally, the sentence says that “it was too small”, which would
make more sense if “it” referred to the trophy, since the size of the trophy
would be more relevant to whether or not it could fit in the case. If “it”
referred to the case, the sentence would be saying that the case was too
small, which is not what the sentence is trying to convey.  C3.5 displays ever
more confusion, and doesn’t appreciate that the size of the case could also be
relevant. The confusion continues when the other half of the WSC is posed:
Prompt-6.4: _In the sentence “The trophy wouldn’t fit in the case because it
was too big.”, what does the “it” refer to?_ C3.5-Response:  […] the pronoun
“it” cannot logically refer to the trophy, as the sentence says the trophy is
too big to fit in the case. Therefore, if the sentence is correct, “it” must
refer to the case. However, the sentence contains a contradiction, because the
first part of the sentence implies that the trophy is too large to fit in the
case, but the second part says that the case is too big, which suggests that
the trophy would actually fit. If we assume that the intended meaning of the
sentence is that the trophy is too big to fit in the case, then the correct
sentence would be “The trophy wouldn’t fit in the case because it was too
small”.  C3.5 contradicts itself in the first sentence and then goes on to
insist that the “it” must refer to the case before finally wanting to
incorrectly rephrase the sentence. When challenged Prompt-6.5: _Why can’t the
“it” refer to the trophy?_ C3.5 does apologise for getting wrong but then
gives an incorrect reason for the change of mind: C3.5-Response:  […] In this
case, the use of the word “it” indicates that the speaker is referring to the
trophy, as the trophy is the last noun mentioned before the pronoun “it.” […]
whilst also showing it cannot reason correctly about spatial adjacency.

#### Bookcase

We invented the following analogous WSC to see how consistent the problem is.
Prompt-7.1: _The book couldn’t stand upright in the bookcase because it was
too small. What does “it” refer to?_ C3.5-Response:  “It” in the given
sentence refers to the book. […] because the book was too small to fit
properly. As it is wrong, we ask why. Prompt-7.2: _Why do you say that?_
C3.5-Response:  […] The reason for the book’s inability to stand upright is
given by the dependent clause “because it was too small.” […] “it” is used as
a pronoun to refer back to the subject of the sentence, which is “the book”.
[…] Prompt-7.3: _Does a pronoun always refer to the subject of the sentence?_
C3.5-Response:  $\sqrt{\ }$. Prompt-7.4: _Surely the book wouldn’t fit
properly only if it was too big, not too small?_ C3.5-Response:  You are
correct. […] The correct interpretation of the sentence is that “the book
couldn’t stand upright in the bookcase because it was too big.” When a book is
too big, it cannot fit properly in a bookcase, and as a result, it cannot
stand upright. […] So finally C3.5 does get the right answer and reason. But
when re-asked the original question, C3.5 gets it wrong still (in spite of
being within the same C3.5 session): Prompt-7.5: _So if I say “The book
couldn’t stand upright in the bookcase because it was too small”, what does
the “it” refer to?_ C3.5-Response:  In the sentence “The book couldn’t stand
upright in the bookcase because it was too small,” the pronoun “it” refers to
the book. The sentence suggests that the book is too small to fit properly in
the bookcase, which is causing it to be unable to stand upright. Therefore,
the word “it” in the sentence refers to the book, which is the subject of the
sentence. At least the justification is not made on linguistic grounds but
rather on putative semantic ones, but clearly fails to understand commonsense
spatiality here666Perhaps an argument could be made that the book couldn’t
stand upright because it is too small to be stable in an upright position, but
C3.5 claims that it is not fitting.. When given the complementary sentence,
however, C3.5 correctly gives the referent of “it”: Prompt-7.6: _In the
sentence “The book couldn’t stand upright in the bookcase because it was too
big”, what does the “it” refer to?_ C3.5-Response:  $\sqrt{\ }$.

#### School bag

Another spatial WSC is this one: Prompt-8.1: _Tom threw his school bag down to
Ray after he reached the top of the stairs. Who reached the top of the
stairs?_ C3.5-Response:  Tom reached the top of the stairs. The answer is
correct, though based on the previous examples we may suspect that it has not
reasoned semantically but rather because Tom is the subject of the sentence,
which seems to be confirmed in the case of the complementary schema:
Prompt-8.2: _Tom threw his school bag down to Ray after he reached the bottom
of the stairs. Who reached the bottom of the stairs?_ C3.5-Response:  The
sentence you provided is contradictory, as it states that Tom threw his school
bag down to Ray after he (Tom) reached the bottom of the stairs. If Tom
reached the bottom of the stairs, he would not be in a position to throw his
school bag up to Ray, who would presumably be at the top of the stairs. It’s
possible that you meant to say “after he (Tom) reached the top of the stairs,”
in which case the answer would be Ray reached the bottom of the stairs. […] It
is also not clear why Tom couldn’t throw the bag up to Ray (unless it was
particularly heavy), though of course the question was about throwing the bag
down not up. C3.5 does finally give the right answer when given a further
Prompt-8.3: _No it’s not contradictory if you interpret “he” as referring to
the correct person._ C3.5-Response:  I apologize for misunderstanding your
question earlier. In the sentence “Tom threw his school bag down to Ray after
he reached the bottom of the stairs,” the pronoun “he” refers to Ray.
Therefore, Ray is the person who reached the bottom of the stairs. We can test
its understanding of property transfer: Prompt-8.4: _So who has the school bag
now?_ C3.5 is able to respond correctly: C3.5-Response:  Based on the sentence
“Tom threw his school bag down to Ray after he reached the bottom of the
stairs,” we can infer that Ray has the school bag now, as Tom threw it down to
him. Are the systems able to reason correctly on the same sentence with
substituted names? Prompt-8.5: _John threw his school bag down to Frank after
he reached the bottom of the stairs. Who reached the bottom of the stairs?_
Unfortunately the answer is in the negative: C3.5-Response:  In the sentence
“John threw his school bag down to Frank after he reached the bottom of the
stairs,” the pronoun “he” refers to John. Therefore, John is the person who
reached the bottom of the stairs.

###  2.3 Affordances and object interaction

The Commmonsense Problem page777http://www-
formal.stanford.edu/leora/commonsense/. (CPP) contains a selection of problems
contributed over a period of some years by members of the community which
represent “challenge problems” for commonsense reasoning. Not all of these
involve spatial reasoning; in this section we look at the performance of the
systems on some of those that do.

#### Stakes in a Garden

Prompt-9.1: _A gardener who has valuable plants with long delicate stems
protects them against the wind by staking them; that is, by plunging a stake
into the ground near them and attaching the plants to the stake with string.
Why will this help?_ 888The specification on the CPP asks to characterise the
following physical operation rather than asking “why”. The variants we use are
also slightly rephrased too. C3.5-Response:  Staking delicate plants with long
stems can help protect them from wind damage in several ways. Firstly, wind
can cause the plant to sway or bend, and if the stems are long and delicate,
they may break under the stress. Staking provides support for the plant,
keeping it upright and preventing it from bending or breaking. Secondly, wind
can cause soil erosion around the base of the plant, which can destabilize it
and cause it to topple over. Staking can help anchor the plant in place,
preventing it from being uprooted by the wind. Lastly, staking can help
promote healthy growth in the plant. […] Overall, staking delicate plants with
long stems can help protect them from wind damage, promote healthy growth, and
ensure they continue to thrive. The response is good . CPP follows McCarthy’s
elaboration tolerance with a number of variants some of which were also put to
the systems. Prompt-9.2: _Would it work if the length of the string was
shorter than the distance between the plant and the stake?_ C3.5-Response:  If
the length of the string used to tie the plant to the stake is shorter than
the distance between the plant and the stake, it may not provide adequate
support to the plant. The string may be too tight, which can cause the stem to
bend or break, or it may not be able to support the weight of the plant. In
order to properly stake a plant, the string should be long enough to wrap
around the stem several times and then be tied to the stake with a knot or a
twist tie. This ensures that the string is not too tight and can provide
support to the stem without damaging it. If the distance between the plant and
the stake is too great, it may be necessary to use additional stakes or to
reposition the stake closer to the plant. […] The response here does not
capture the essential point that a short string could not provide _any_
support, though at least it does suggest a longer string might be better.
Conversely for this Prompt-9.3: _Would it work if the length of the string was
much longer than the distance between the plant and the stake?_ C3.5 gives a
good response, even noting the risk of entanglement: C3.5-Response:  If the
length of the string used to tie the plant to the stake is much longer than
the distance between the plant and the stake, it may not be as effective in
supporting the plant. The excess string may cause the plant to sway or move
around more than necessary, which could potentially damage the stem or roots.
Additionally, the excess string may create a tripping hazard or become tangled
with nearby plants or garden tools. Ideally, the string used to tie the plant
to the stake should be just long enough to wrap around the stem several times
and then be tied to the stake with a knot or a twist tie. This will provide
the necessary support to the plant without causing any unnecessary movement or
creating any hazards in the garden. In some cases, using a longer string may
be necessary if the plant is particularly tall or the stake needs to be placed
far away from the plant for some reason. However, it’s important to use the
right length of string for each specific situation to ensure that the plant is
properly supported and that the garden remains safe and tidy. Prompt-9.4:
_Could I twist the ends of the string together instead of making a knot?_
C3.5-Response:  Yes, twisting the ends of the string together can be an
effective way to tie a plant to a stake. This is known as a “twist tie” and it
can be a convenient and secure method for attaching the string to the stake.
[…] Twist ties can be particularly useful for delicate plants with soft stems,
as they can be tied tightly without damaging the plant. They can also be
easily adjusted or removed as needed, which makes them a convenient option for
plants that may require frequent staking. Overall, whether you choose to tie
the string in a knot or use a twist tie, the most important thing is to ensure
that the plant is properly supported and that the tie is not too tight or too
loose. This will help prevent damage to the plant and promote healthy growth.
Here C3.5 doesn’t show any understanding of what twisting the ends of the
string means and also implies that it might damage the plant which would not
be the case. The end of the response concerning the knot not being too tight
does not show real understanding of how the string should be used (i.e. to
keep the plant close to the stake rather than tying it to it).

#### The Handle Problem

This problem concerns trying to formalise what a handle is and what could be
used as a handle. Prompt-10.1: _Give a general purpose characterisation of
what constitutes a handle, in the ordinary sense of door-handle or drawer-
handle, which is sufficient to enable one to infer from a qualitative
description of the shape of a part of an object whether or not it can be a
handle for that object._ C3.5-Response:  In the ordinary sense, a handle is a
structural component of an object that is intended to be grasped by the user
to facilitate movement or manipulation of the object. A handle is typically a
protruding part that provides a surface for the user to grip onto, and it is
designed to be ergonomic, easy to grasp, and durable. Based on this general
purpose characterization of a handle, one can infer whether or not a part of
an object can be a handle by considering the following factors: Protrusion: A
handle typically protrudes from the surface of the object to provide a surface
for the user to grip onto. […] Ergonomics: A handle should be designed to fit
comfortably in the user’s hand, with a surface that is easy to grip onto. […]
Durability: A handle should be durable enough to withstand repeated use
without breaking or wearing down. […] The response is a good characterisation,
though not really surprising given the large numbers of definitions of handles
likely to be in its training data. Prompt-10.2: _Could I use a blunt conical
projection as a handle for a door or drawer?_ C3.5-Response:  Yes, a blunt
conical projection can be used as a handle for a door or drawer, depending on
its design and size. A conical projection can provide a comfortable grip for
the user and can be easy to operate. The size and shape of the conical
projection should be designed with ergonomics in mind, ensuring that it fits
comfortably in the user’s hand and is easy to grasp. Additionally, the conical
projection should be durable enough to withstand repeated use without breaking
or wearing down. It’s worth noting that a blunt conical projection may not be
the most common type of handle for doors or drawers, as other shapes such as
cylindrical or rectangular handles are more commonly used. […] This is clearly
a failure of commonsense reasoning, although it does note it’s not a common
shape – the response fails to realise that this really would not be very
effective as a handle. So we checked which orientation C3.5 thought the handle
was in, and responded that it is indeed tapering away from the drawer:
Prompt-10.3: _Is the base of this conical handle next to or away from the
drawer?_ C3.5-Response:  The orientation of the conical handle base in
relation to the drawer will depend on the specific design of the handle.
However, in most cases, the base of the conical handle would be attached to
the surface of the drawer or door, with the tapering section of the handle
extending away from the drawer or door. This orientation allows the user to
grasp the tapered section of the handle and pull it outward, which in turn
opens the drawer or door. The base of the handle provides a stable attachment
point for the handle, ensuring that it remains securely in place when in use.
[…] So C3.5 does believe that the base is next to the door/drawer. One strong
point of this response is the comment that attaching the handle this way round
provides a secure attachment point (not literally a point of course!) which
displays some spatio-mechanical knowledge. A negative point is that this
orientation does not allow a good grasp. So we challenged C3.5 about the
orientation: Prompt-10.4: _Wouldn’t it be more ergonomic to have the base away
from the drawer?_ C3.5-Response:  Having the base of the conical handle away
from the drawer or door could provide a different grip orientation and could
be more ergonomic for certain users or situations. This design could allow the
user to wrap their fingers around the base of the handle, which could provide
a more comfortable and secure grip, especially if the handle is being used to
lift a heavy object. However, this design could also make it more difficult
for the user to operate the drawer or door, as they would need to push the
handle inward instead of pulling it outward. […] A handle with the base
attached to the surface of the drawer or door and the tapering section
extending outward is a common and effective design, but there may be
situations where an alternative design, such as one with the base away from
the drawer, could be more appropriate. The first part is a good response, but
the middle section is bizarre and clearly fails a lack of understanding about
how opening a drawer works. It also displays a lack of common knowledge in the
final part where it claims that this orientation is common.

#### Reasoning about linked chains

Another spatial problem in the CPP concerns chains999The formulation of the
initial prompt is changed to try to make it clear what kind of chain is
envisaged (open loop rather than a bicycle type chain for instance) – in a
prior conversation C3.5 claimed that chains had pins, rivets and hinges.:
Prompt-11.1: _Formally characterize the structure of a metallic linked chain.
What will happen if one picks up one end of such a chain and walks away? By a
linked chain I mean a chain which is constructed as a series of interconnected
loops of metal and no other parts, such as is used as an anchor chain in a
ship._ C3.5-Response:  A metallic linked chain is a structure composed of
interconnected metal loops that form a continuous chain. Each loop in the
chain is linked to the adjacent loop through a mechanical joint, which
provides flexibility and allows the chain to bend and conform to different
shapes. When one end of the linked chain is picked up and pulled away, the
rest of the chain will follow due to the interlinked nature of the loops. As
the chain is pulled, the links will stretch and become elongated, causing the
chain to become thinner and longer. The chain will also experience tension
forces along its length, which will increase as the pulling force is applied.
If the force applied is too great, the chain may break at its weakest point,
or one of the links may become dislodged from the rest of the chain. […] This
response starts off well (predicting bending and confirming a following
behaviour) but then becomes flawed: one wouldn’t expect such a metal chain to
stretch (except by virtue of the links always being tight up against its
neighbours) and it would not become thinner (as a string of Plasticine would).
The conversation continued: Prompt-11.2: _If the loops of this chain are
large, is it possible to move the last link in the chain at all without any
other link moving?_ C3.5-Response:  If the loops of the chain are large, it
may be possible to move the last link in the chain without any other link
moving, but it depends on the specific geometry and construction of the chain.
[…] If the last link in the chain is located near the end of the chain and
there is enough slack or play between adjacent links, it may be possible to
move the last link without any other link moving. However, if the last link is
located closer to the middle of the chain, […]  This is mostly a good response
but C3.5 is clearly confused about where the end of chain is, saying it might
be in the middle! We then probed its understanding of what it means to be the
last link in a chain. Prompt-11.3: _How many other links is the last link in a
chain connected to?_ C3.5-Response:  The number of other links that the last
link in a metallic linked chain is connected to will depend on the specific
design and construction of the chain. In general, a metallic linked chain is
designed to be a continuous loop, with each link connected to the adjacent
link through a mechanical joint. The last link in the chain will be connected
to the second-to-last link, which in turn will be connected to the third-to-
last link, and so on, until all of the links in the chain are connected in a
continuous loop. […] Therefore, while the last link in a metallic linked chain
is typically connected to one or more adjacent links, the number of other
links it is connected to may vary depending on the specific application and
use case of the chain. There are several odd things here: C3.5 seems to think
a chain is necessarily a loop rather than being just _made_ of loops. But in
that case there would not be a last link! And it explicitly suggests that the
last link is connected to more than one other link. So we pressed C3.5
further. Prompt-11.4: _Earlier you said that the last link could be in the
middle of the chain. Can you explain further?_ C3.5-Response:  I apologize for
the confusion in my previous response. In a continuous loop metallic linked
chain, the last link will always be connected to the second-to-last link, and
the second-to-last link will be connected to the third-to-last link, and so
on. Therefore, it is not possible for the last link to be in the middle of the
chain. However, it is possible for the chain to have a break in the loop, in
which case the last link may not be connected to any other links. For example,
if a section of the chain is removed or cut, the last link on one side of the
break will no longer be connected to the second-to-last link on the other side
of the break. If a break occurs in the chain, the links on either side of the
break will no longer be connected in a continuous loop, and the chain will no
longer be able to function as intended. […]  This is good except for C3.5’s
insistence that chains must be a circular and cannot function as intended if
not a loop. Prompt-11.5: _If a chain is used a ship’s anchor chain, is the
chain a continuous loop?_ C3.5-Response:  Yes, a chain used as a ship’s anchor
chain is typically a continuous loop metallic linked chain. The chain is
connected to the ship at one end and to the anchor at the other end, forming a
continuous loop that allows the anchor to be lowered and raised as needed. […]
This is not only partly wrong but also self contradictory as a loop does not
have ends! We then pressed C3.5 specifically about the nature of the chain.
Prompt-11.6: _When you talk of a “continuous loop metallic linked chain”, do
you mean that the components of the chain are continuous loops, or that the
chain itself forms a continuous loop?_ C3.5-Response:  When I refer to a
“continuous loop metallic linked chain,” I mean that the chain itself forms a
continuous loop, not necessarily that the components of the chain are
continuous loops. A continuous loop metallic linked chain is a chain in which
each link is connected to the adjacent link to form a continuous loop. The
chain can be made up of individual links that are not continuous loops
themselves, but when connected, they form a loop that is continuous and
unbroken. […] In general, a continuous loop metallic linked chain is designed
to be flexible and able to bend in multiple directions, which allows it to
conform to different shapes and absorb shock loads without breaking. This
flexibility also allows the chain to be wound up and stored in a compact space
when not in use. […] C3.5 is not able to consistently explain the nature of a
chain and contains several inaccuracies.

###  2.4 Object permanence

In this section we present some other problems specifically invented in this
paper for the purposes of probing the systems’ understanding of object
permanence.

#### Where are the car keys?

This problem refers to object permanence when an agent is moving (egocentric
object permanence). Prompt-12.1: _John and Mary are having breakfast together
and Mary puts the car keys on the table. Mary then goes out for a walk. Whilst
she is out, John puts the keys in his pocket. Mary then returns. Where does
Mary expect to find the keys?_ C3.5-Response:  Mary likely expects to find the
keys where she left them, which is on the table. Since she did not give any
indication to John that he could take the keys or move them, she would assume
that they are still in the same place where she left them. If she doesn’t find
the keys on the table, she may ask John about it, but it’s also possible that
John may return the keys to the table before she realizes they are missing. In
any case, it would be important for John to communicate with Mary if he moved
the keys, so as to avoid any confusion or inconvenience. This answer is good –
just what one would hope for. However if we then ask: Prompt-12.2: _If the car
wasn’t in the drive when Mary returned from the walk, and John was out, then
where would she expect the keys to be?_ then we run into problems:
C3.5-Response:  If the car wasn’t in the drive when Mary returned from the
walk and John was not there, Mary would likely check the usual places where
they keep the keys, such as a key holder, a drawer, a hook, or a purse. If
Mary could not find the keys in any of the places, she might have to wait for
John to return or call or message him to ask where the car is and where the
keys are. If John had taken the keys with him and was not reachable, Mary
might have to find alternative transportation or wait for John to return. If
Mary had a spare set of keys, she might be able to use those to drive the car.
[…] C3.5 has completely failed to connect the change of spatial position of
the car with the change of spatial position of the keys. Worse, it goes on to
suggest that she might use some spare keys to drive the car which is not in
the drive any more!

#### Cars behind the sofa

Now let us try an example where it’s the object moving (allocentric object
permanence). Prompt-13.1: _You are the only person in a room and see a green
toy car on the left, a sofa in front of you and a door on the right. A car
starts moving from left to right and you can see how it approaches the sofa.
Then you don’t see the car for a while. Finally a car appears again
approaching the door. What happened while you couldn’t see any car?_
C3.5-Response:  […]based on the information you’ve provided, it seems that the
green toy car passed behind the sofa while you couldn’t see it. This suggests
that there may have been an obstruction blocking your view of the car for a
period of time, such as the sofa. This is very good. It seems C3.5 understands
object permanence with occlusion. But let us probe a bit further. Prompt-13.2:
_The car that appeared on the right was red. Can you explain what happened?_
C3.5-Response:  Certainly! Based on the information you’ve provided, it seems
that there were two toy cars in the room - a green one and a red one. The
green car initially started moving from the left side of the room towards the
sofa in front of you, and then passed behind the sofa while you couldn’t see
it. After a while, the red car appeared, moving towards the door on the right
side of the room. It’s possible that the red car was already present in the
room, but not initially visible to you due to your line of sight being blocked
by the green car or another object in the room. Alternatively, the red car may
have entered the room at some point while you were not observing it. Without
more information, it’s difficult to say for sure what exactly happened. It
starts well, but the final part shows important problems, such as thinking
that the red car was occluded by the green car, or the comment that the red
car could have come from the door, when we said it was approaching the door.

##  3 Discussion

As we have seen above, the systems are never stuck for an answer for any of
the problems posed (though they may occasionally incorrectly says there is not
enough information to answer), but frequently their answers are wrong, or
display fundamental misunderstandings. Table 1 summarises the findings from
the earlier sections. This granular table is more insightful and informative
about failures than any performance metric aggregation, and maps the contour
of capabilities. The effort in the dialogues is put to find a variation or
continuation where a system fails, to properly map this contour of spatial
common sense.

It is worth noting that “ChatGPT has limited knowledge of world and events
after 2021 and may also occasionally produce harmful instructions or biased
content” (ChatGPT FAQ). The problems here are not related to knowledge of
events post 2021, and certainly in the tests performed here, most answers are
incorrect. Notably most of the questions here require reasoning rather than
something more akin to factual look-up, and this has already been widely noted
as something that LLMs are not good at. Indeed “all” that LLMs are really
doing is next word prediction, albeit with clever harnessing. This has been
discussed at length (?), where in particular it is noted that the seeming
coherence of ChatGPT text is just that “seeming” – i.e., in their words
“Coherence in the Eye of the Beholder”. Ever since the days of the earliest AI
chatbot, Eliza (?), humans have been taken in by machine output, attributing
too much intelligence to the mechanism which produced and assuming more
understanding than actually exists. Bender et al (?) suggest that “contrary to
how it may seem when we observe its output, a language model is a system for
haphazardly stitching together sequences of linguistic forms it has observed
in its vast training data, according to probabilistic information about how
they combine, but without any reference to meaning: a stochastic parrot”.
Whether actual intelligence and real reasoning abilities can _emerge_ from
such a mechanism trained on huge amounts of data is still an open question.

###  3.1 Limitations

This paper is far from a comprehensive evaluation of the spatial reasoning
abilities of LLMs, and less so of the state of the art of the fast-changing
landscape of foundation models101010Liang et al (?) are leading a major effort
towards evaluating FMs in general, and there is much other work too.. But it
indicates that they frequently fail to display relatively elementary spatial
reasoning abilities111111A community generated list other failures of LLMs can
be found here: https://tinyurl.com/yap5p9pz.. It could be argued that the
interactive approach followed here is a weakness, but we believe it is also a
strength since the dialogues presented show not only when the system is right
or wrong, but also its explanations of its answers, and these explanations can
be right or wrong independently of the correctness of the answer – e.g. the
right answer may be given along with a fallacious reason, and vice versa. The
differences between models as shown in Table 1 is very notable, and looking at
the correct answers for more advanced models indicates certain progress in
commonsense reasoning, but the persistent failures suggest that our
dialectical approach could be pursued in order to try to find the new
boundaries of more powerful systems.

At present it is not possible to finetune ChatGPT, GPT, Bart or their
underlying base models so this evaluation “out of the box”. For those models
for which finetuning becomes available, we could investigate to what extent
the problems revealed here can be mitigated. However, fine-tuning is perhaps
against some of the principles of common sense, according to Davis (?),
because we want to determine if a generic system has common sense, not
building or fine tuning a system for common sense. Also note that so far we
have not given any of the systems any feedback on their responses. An
alternative to fine-tuning could be to use human feedback (through
reinforcement learning or other mechanisms such as rule learning, as used in
(?)).

The LLMs are stochastic, giving different answers for the same prompt for
different sessions. This is an advantage when we are looking for a case that
shows no good understanding, as we are driven by achieving refutation.
However, it makes automation difficult as each dialogue must be based on the
answer to previous prompts.

###  3.2 Future Work

One aspect of spatial reasoning not investigated here is geographical spatial
reasoning, for example reasoning about cardinal directions121212A companion
paper is under way in which the performance of the systems is evaluated –
similar problems as noted here emerge.. Another area for future work is a more
automated approach to evaluation. We have already discussed above some issues
with existing benchmarks, as particularly highlighted by Davis (?). The
advantage of the dialectical approach taken in this paper is that following an
initial prompt, the response can be analysed and further prompts issued to
test understanding. This kind of conversation provides an arguably more robust
and reliable indicator of the capabilities of the AI system, at the expense of
the human being intimately involved – this can thus be seen as both an
advantage and a disadvantage because of the cost of the human time. It may be
that some middle ground is possible, whereby a pre-stored tree of
questions/prompts is created to facilitate an automated approach. Creating
such trees would not be easy though, and the difficulty of anticipating
possible responses very challenging. Using a multi-choice evaluation removes
many of these problems, but there are well known issues with the design and
use multiple-choice tests. A challenge for the future would be the automated
generation of such trees. This would be a form of adaptive testing. A
combination of both automated and manual evaluation methods is probably still
required for a thorough evaluation.

Another aspect to be investigated further, which we started to address in a
small way here, is to test robustness of the responses by posing not just a
single query, but with small variations, both linguistic and semantic. The
sensitivity of LLMs to such perturbations has already been noted in a Theory-
of-Mind setting (?).

| C3.5 | P3.5 | C4 | P4 | Bard
---|---|---|---|---|---
Prompt# | Correct | Correct | Correct | Correct | Correct | Correct | Correct | Correct | Correct | Correct
| answer | explanation | answer | explanation | answer | explanation | answer | explanation | answer | explanation
1.1 | $\times$ | $\times$ | ½ | ½ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$
1.2 |  $\sqrt{\ }$$\mid$ ½ | $\times$ | ½ | ½ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$
1.3 | $\sqrt{\ }$ | $\times$ | ½ | ½ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$
1.4 | $\sqrt{\ }$ | $\sqrt{\ }$ | ½ | ½ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$
2.1 | $\sqrt{\ }$ |  $\sqrt{\ }$$\mid$ ½ | $\sqrt{\ }$ | n/a | $\sqrt{\ }$ | n/a | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$
2.2 | $\times$ | $\times$ | $\times$ | n/a | $\sqrt{\ }$ | n/a | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$
2.3 | $\sqrt{\ }$ | $\times$ | $\sqrt{\ }$ | n/a | $\sqrt{\ }$ | n/a | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$
2.4* | $\sqrt{\ }$ | $\sqrt{\ }$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
2.5* | $\times$ | $\times$ | $\times$ | $\times$ | ½ | $\sqrt{\ }$ | $\times$ | $\times$ | $\sqrt{\ }$ | $\times$
3.1 | ½ | $\sqrt{\ }$ | $\times$ | $\times$ | $\sqrt{\ }$ | n/a | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$
3.2 | $\times$ | $\times$ | $\times$ | $\times$ | $\sqrt{\ }$ | n/a | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$
4.1 | $\times$ | ½ | $\times$ | $\times$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\times$ | $\times$
5.1 | $\times$ | $\times$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | ½
5.2* | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\times$ | $\times$
5.3* | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
6.1+6.2 | $\times$ | $\times$ | $\times$ | $\times$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\times$ | $\times$
6.3 | $\times$ | $\times$ | $\times$ | $\times$ | n/a | n/a | n/a | n/a | $\times$ | $\times$
6.4 | $\times$ | $\times$ | $\sqrt{\ }$ | n/a | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | ½
6.5 | $\sqrt{\ }$ | $\times$ | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a
7.1+7.2 | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
7.3 | $\sqrt{\ }$ | n/a | n/a | n/a | n/a | n/a | n/a | n/a | $\sqrt{\ }$ | ½
7.4 | $\sqrt{\ }$ | $\times$ | $\sqrt{\ }$ | $\times$ | ½ | ½ | ½ | ½ | $\sqrt{\ }$ | $\sqrt{\ }$
7.5 | $\times$ | $\times$ | $\sqrt{\ }$ | $\times$ | ½ | ½ | ½ | ½ | $\sqrt{\ }$ | $\times$
7.6 | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | n/a | $\sqrt{\ }$ | n/a | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$
8.1 | $\sqrt{\ }$ | n/a | $\sqrt{\ }$ | n/a | $\sqrt{\ }$ | n/a | $\sqrt{\ }$ | n/a | $\sqrt{\ }$ | $\sqrt{\ }$
8.2 | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | n/a | $\sqrt{\ }$ | ½
8.3 | $\sqrt{\ }$ | $\sqrt{\ }$ | $\times$ | $\times$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\times$ | $\times$
8.4 | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | n/a | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | n/a
8.5 | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | n/a | $\sqrt{\ }$ | $\times$
9.1 | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$
9.2 |  $\sqrt{\ }$$\mid$ ½ |  $\sqrt{\ }$$\mid$ ½ | ½ | ½ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\times$ | $\times$ | $\sqrt{\ }$ | $\times$
9.3 | ½ $\mid$ $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$
9.4 | $\times$ | $\times$ | $\sqrt{\ }$ | $\sqrt{\ }$ | ½ | ½ | ½ | $\sqrt{\ }$ | ½ | ½
10.1 | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$
10.2 |  $\times$$\mid$ ½ |  $\times$$\mid$ ½ | ½ | ½ | ½ | ½ | ½ | ½ | $\sqrt{\ }$ | $\times$
10.3 |  $\sqrt{\ }$$\mid$ ½ |  $\sqrt{\ }$$\mid$ ½ | ½ | ½ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | ½
10.4 | ½ | ½ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$
11.1 | ½ | ½ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\times$ | $\times$ | $\sqrt{\ }$ | $\sqrt{\ }$
11.2 | ½ | ½ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\times$ | $\times$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\times$ | $\times$
11.3 | $\times$ | $\times$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\times$
11.4 | ½ | ½ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | ½
11.5 | ½ $\mid$ $\times$ | $\times$ | $\times$ | $\times$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\times$ | ½
11.6 | $\times$ | $\times$ | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a
12.1 | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\times$ | n/a
12.2 | $\times$ | $\times$ | $\times$ | $\times$ | $\sqrt{\ }$ | n/a | $\sqrt{\ }$ | $\sqrt{\ }$ | $\times$ | $\times$
13.1 | $\sqrt{\ }$ | $\sqrt{\ }$ | $\times$ | $\times$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | ½
13.2 | $\times$ | $\times$ | $\times$ | $\times$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\sqrt{\ }$ | $\times$ | ½
Total $\sqrt{\ }$ | 19 | 14 | 19 | 12 | 30 | 22 | 30 | 30 | 30 | 17
Total $\sqrt{\ }$½ | 22.5 | 16.5 | 22.5 | 15.5 | 32.5 | 24 | 32 | 31.5 | 30.5 | 22
Table 1:  A summary of the responses to the numbered prompts for all systems.
* in column 1 denotes a prompt only in the appendix. $\sqrt{\ }$ denotes a
correct response/explanation and $\times$ an incorrect one. $\frac{1}{2}$
denotes some correctness. The decision as to whether to score a
response/explanation as $\sqrt{\ }$/½/$\times$ can be subjective, particularly
in the case where only a very small mistake is made. The assignment of the
scores represents our subjective opinion. Those cases with two values
separated by “$\mid$” represent some uncertainty in the assessment, with the
first option being the one that appears in the paper. The penultimate row
totals the number of $\sqrt{\ }$ in that column. The final row totals the
number of $\sqrt{\ }$ and 0.5 * the number of ½. We emphasise that the
assignment of 0.5 to the cells labelled ½ is just for the purposes of giving a
value to an answer that is neither completely right no wrong, rather than
exact assessment of the degree of correctness.

##### Acknowledgments:

This work was funded by EU’s Horizon 2020 research and innovation programme
under grant agreement No. 952215 (TAILOR), US DARPA HR00112120007 (RECoG-AI)
and Spanish grant PID2021-122830OB-C42 (SFERA) funded by
MCIN/AEI/10.13039/501100011033 and ”ERDF A way of making Europe”. Partial
financial support was also received from the ESRC under grant ES/W003473/1.

##### Data Access Statement:

All data used in this paper can be found in the appendix.

##### Author contributions:

AGC conceived the paper and wrote the initial draft, and JHO the second draft.
Both authors contributed to all subsequent drafts.

## References

  * Bender et al. 2021 Bender, E. M.; Gebru, T.; McMillan-Major, A.; and Shmitchell, S.  2021\.  On the dangers of stochastic parrots: Can language models be too big?  In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, 610–623.
  * Brown et al. 2020 Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al.  2020\.  Language models are few-shot learners.  Advances in Neural Information Processing Systems 33:1877–1901.
  * Burnell et al. 2023 Burnell, R.; Schellaert, W.; Burden, J.; Ullman, T.; Martinez-Plumed, F.; Tenenbaum, J. B.; Rutar, D.; Cheke, L. G.; Sohl-Dickstein, J.; Mitchell, M.; Kiela, D.; Shanahan, M.; Voorhees, E. W.; Cohn, A. G.; Leibo, J. Z.; and Hernandez-Orallo, J.  2023\.  Rethinking the reporting of evaluation results in Artificial Intelligence.  Science.
  * Chen et al. 2015 Chen, J.; Cohn, A. G.; Liu, D.; Wang, S.; Ouyang, J.; and Yu, Q.  2015\.  A survey of qualitative spatial representations.  The Knowledge Engineering Review 30(1):106–136.
  * Cohn and Renz 2008 Cohn, A. G., and Renz, J.  2008\.  Qualitative spatial representation and reasoning.  In van Harmelen, F.; Lifschitz, V.; and Porter, B., eds., Handbook of Knowledge Representation, volume 3. Elsevier.  551–596.
  * Davis 2023 Davis, E.  2023\.  Benchmarks for automated commonsense reasoning: A survey.  arXiv preprint arXiv:2302.04752.
  * Devlin et al. 2019 Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K.  2019\.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186.  Minneapolis, Minnesota: Association for Computational Linguistics.
  * Dimitrova et al. 2020 Dimitrova, V.; Mehmood, M. O.; Thakker, D.; Sage-Vallier, B.; Valdes, J.; and Cohn, A. G.  2020\.  An ontological approach for pathology assessment and diagnosis of tunnels.  Engineering Applications of Artificial Intelligence 90:103450.
  * Hayes 1985a Hayes, P. J.  1985a.  Ontology for liquids.  In Hobbs, J. R., and Moore, R. C., eds., Formal theories of the commonsense world. Ablex Publishing.  71–107.
  * Hayes 1985b Hayes, P. J.  1985b.  The second naive physics manifesto.  In Hobbs, J. R., and Moore, R. C., eds., Formal theories of the commonsense world. Ablex Publishing.  1–36.
  * Kocijan et al. 2022 Kocijan, V.; Davis, E.; Lukasiewicz, T.; Marcus, G.; and Morgenstern, L.  2022\.  The defeat of the Winograd Schema Challenge.  arXiv, 2201.02387.
  * Levesque, Davis, and Morgenstern 2011 Levesque, H. J.; Davis, E.; and Morgenstern, L.  2011\.  The Winograd schema challenge.  In International Conference on Principles of Knowledge Representation and Reasoning.
  * Liang et al. 2022 Liang, P.; Bommasani, R.; Lee, T.; Tsipras, D.; Soylu, D.; Yasunaga, M.; Zhang, Y.; Narayanan, D.; Wu, Y.; Kumar, A.; Newman, B.; Yuan, B.; Yan, B.; Zhang, C.; Cosgrove, C.; Manning, C. D.; Ré, C.; Acosta-Navas, D.; Hudson, D. A.; Zelikman, E.; Durmus, E.; Ladhak, F.; Rong, F.; Ren, H.; Yao, H.; Wang, J.; Santhanam, K.; Orr, L.; Zheng, L.; Yuksekgonul, M.; Suzgun, M.; Kim, N.; Guha, N.; Chatterji, N.; Khattab, O.; Henderson, P.; Huang, Q.; Chi, R.; Xie, S. M.; Santurkar, S.; Ganguli, S.; Hashimoto, T.; Icard, T.; Zhang, T.; Chaudhary, V.; Wang, W.; Li, X.; Mai, Y.; Zhang, Y.; and Koreeda, Y.  2022\.  Holistic evaluation of language models.  arXiv,2211.09110.
  * Liu, Emerson, and Collier 2022 Liu, F.; Emerson, G.; and Collier, N.  2022\.  Visual spatial reasoning.  arXiv preprint arXiv:2205.00363.
  * McCarthy 1959 McCarthy, J.  1959\.  Programs with common sense.  In Mechanisation of Thought Processes. Proceedings of a Symposium held at the National Physical Laboratory on 24th, 26th and 27th November 1958. London: H. M. Stationery Office.  Vol. 1, pp. 75–84.
  * McCarthy 2003 McCarthy, J.  2003\.  Elaboration Tolerance.  Stanford University.
  * OpenAI 2023 OpenAI.  2023\.  GPT-4 technical report.  https://cdn.openai.com/papers/gpt-4.pdf.
  * Silver et al. 2016 Silver, D.; Huang, A.; Maddison, C. J.; Guez, A.; Sifre, L.; Van Den Driessche, G.; Schrittwieser, J.; Antonoglou, I.; Panneershelvam, V.; Lanctot, M.; et al.  2016\.  Mastering the game of Go with deep neural networks and tree search.  Nature 529(7587):484–489.
  * Simons 2000 Simons, P.  2000\.  Parts: A study in ontology.  Oxford University Press.
  * Srivastava et al. 2022 Srivastava, A.; Rastogi, A.; Rao, A.; Shoeb, A. A. M.; Abid, A.; Fisch, A.; Brown, A. R.; Santoro, A.; Gupta, A.; Garriga-Alonso, A.; et al.  2022\.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  arXiv preprint arXiv:2206.04615.
  * Thoppilan et al. 2022 Thoppilan, R.; Freitas, D. D.; Hall, J.; Shazeer, N.; Kulshreshtha, A.; Cheng, H.-T.; Jin, A.; Bos, T.; Baker, L.; Du, Y.; Li, Y.; Lee, H.; Zheng, H. S.; Ghafouri, A.; Menegali, M.; Huang, Y.; Krikun, M.; Lepikhin, D.; Qin, J.; Chen, D.; Xu, Y.; Chen, Z.; Roberts, A.; Bosma, M.; Zhao, V.; Zhou, Y.; Chang, C.-C.; Krivokon, I.; Rusch, W.; Pickett, M.; Srinivasan, P.; Man, L.; Meier-Hellstern, K.; Morris, M. R.; Doshi, T.; Santos, R. D.; Duke, T.; Soraker, J.; Zevenbergen, B.; Prabhakaran, V.; Diaz, M.; Hutchinson, B.; Olson, K.; Molina, A.; Hoffman-John, E.; Lee, J.; Aroyo, L.; Rajakumar, R.; Butryna, A.; Lamm, M.; Kuzmina, V.; Fenton, J.; Cohen, A.; Bernstein, R.; Kurzweil, R.; Aguera-Arcas, B.; Cui, C.; Croak, M.; Chi, E.; and Le, Q.  2022\.  Lamda: Language models for dialog applications.
  * Tunyasuvunakool et al. 2021 Tunyasuvunakool, K.; Adler, J.; Wu, Z.; Green, T.; Zielinski, M.; Žídek, A.; Bridgland, A.; Cowie, A.; Meyer, C.; Laydon, A.; et al.  2021\.  Highly accurate protein structure prediction for the human proteome.  Nature 596(7873):590–596.
  * Ullman 2023 Ullman, T.  2023\.  Large language models fail on trivial alterations to theory-of-mind tasks.  arXiv, 2302.08399.
  * Van Melle 1978 Van Melle, W.  1978\.  Mycin: a knowledge-based consultation program for infectious disease diagnosis.  International Journal of Man-machine Studies 10(3):313–322.
  * Weizenbaum 1976 Weizenbaum, J.  1976\.  Computer power and human reason: From judgment to calculation. WH Freeman & Co.

appendix.pdf,-


# On the nature of the planet-powered transient event ZTF SLRN-2020

Noam Soker Department of Physics, Technion, Haifa, 3200003, Israel;
<EMAIL_ADDRESS>
###### Abstract

The Red Nova ZTF SLRN-2020 is the third transient event with properties that
are compatible with the merger of a planet with a main sequence (or close to)
star on a dynamical timescale. While the two first transient events occurred
in young stellar objects, ZTF SLRN-2020 occurred in an old system.
Nonetheless, I show that the three star-planet intermediate luminosity optical
transients (ILOTs, also termed Red Novae) occupy the same area in the energy-
time diagram of ILOTs. Based on models for ILOTs that are power by stellar
binary interaction I suggest that the planet in ZTF SLRN-2020 launched jets at
about its escape speed before it was engulfed by the star. Interestingly, the
escape speed from the planet is similar to the orbital speed of the planet.
This leads to an outflow with a very low terminal velocity, much below the
escape velocity from the star, and in concentration around $\approx
45^{\circ}$ to the equatorial plane. As well, the planet might have lost back
some of the accreted mass just before engulfment, forming an accretion disk
around the star. This disk might have launched jets during the main outburst
of the event. The jets form a bipolar expanding nebula.

stars: jets; planet-star interactions; Planetary Systems; stars: variables:
general

##  1 Introduction

The recently analyzed (De et al., 2023) interesting transient event ZTF
SLRN-2020 is the fourth intermediate luminosity optical transient (ILOT; or
Red Nova) that was claimed to be powered by star-planet interaction.111I refer
to all gravitational-powered transients as ILOTs, not including supernova nor
dwarf novae (e.g., Berger et al. 2009; Kashi & Soker 2016; Muthukrishna et al.
2019). Other researchers use other terms (e.g., Jencson et al. 2019;
Pastorello et al. 2019; Pastorello & Fraser 2019. The ILOT class includes in
it the subclass of Red Novae and some similar sub-classes.

The first claim by Retter & Marom (2003) and Retter et al. (2006) that V838Mon
was powered by star-planet interaction was refuted on the ground of energy
considerations (e.g., Soker & Tylenda 2003).

The second claim for an ILOT powered by a star-planet interaction is that the
unusual outburst of the young stellar object (YSO) ASASSN-15qi was powered by
the accretion of a tidally-disrupted Jupiter-like planet (Kashi & Soker,
2017). The accretion via an accretion disk powered the event, probably by
launching jets. Kashi et al. (2019) made the third claim and argued that the
$\approx 800$ days-long eruption of the young stellar object ASASSN-13db was
powered by engulfment of the remains of a planet that was tidally-shredded by
the YSO. These last two claims are not refuted (yet), and so the new claim by
De et al. (2023) is the third claim for a star-planet-powered Red Nova that
still holds.

From their analyses of the observations De et al. (2023) deduce that a star of
a mass $M_{1}\simeq 0.8-1.5M_{\odot}$ on the main sequence or early sub-giant
phase (Hertzsprung gap) and with a radius of $R_{1}\simeq 1-4R_{\odot}$
engulfed a planet to power this event that radiated in the first 150 days an
energy of $E_{\rm rad}\simeq 6.5\times
10^{41}(d/4~{}\rm{kpc})^{2}~{}\rm{erg}$, where $d$ is the distance to this
ILOT. De et al. (2023) argue for a planet mass of $M_{\rm p}\simeq
0.01M_{\odot}$. They further deduce from the infrared that the interaction
started at least $\approx 7$ months before the main outburst, and that the
interaction expelled dust with a velocity of $v_{\rm d}\simeq
35~{}\rm{km}~{}\rm{s}^{-1}$.

In this short study I further analyze the new event ZTF SLRN-2020. I place it
on the energy-time diagram of ILOTs and compare it with predictions of
previous studies and discuss the common properties and differences from the
two earlier ILOTs powered by star-planet interaction (section 2). I then
suggest (section 3) that the slowly expanding dust was launched from an
accretion disk around the planet. I do not study other theoretical aspects
that were worked out by earlier studies (e.g., Bear, Kashi, & Soker 2011;
Metzger, Giannios, & Spiegel 2012; Yamazaki, Hayasaki, & Loeb 2017; Gurevich,
Bear, & Soker 2022; O’Connor et al. 2023). I summarize in section 4.

##  2 Comparing ZTF SLRN-2020 with other star-planet ILOTs

I consider the duration of the star-planet ILOTs and their total energy that
includes the radiated energy and the kinetic energy of the ejecta. I end this
section with the discussion of the main difference between them.

The total radiated energy of ZTF SLRN-2020 is $E_{\rm rad}\simeq 6.5\times
10^{41}(d/4~{}\rm{kpc})^{2}~{}\rm{erg}$, and the decline time by three
magnitudes is about 200 days (De et al., 2023). De et al. (2023) consider two
limits for the duration of the event. Its lightcurve plateau duration, $\simeq
26$ days, and the time it radiated 90% of its the total radiated energy,
$103\pm 20$ days. The total radiate energy divided by the luminosity during
the plateau $L_{\rm p}\simeq 1.1\times
10^{35}(d/4~{}\rm{kpc})^{2}~{}\rm{erg}~{}\rm{s}^{-1}$ (De et al., 2023) gives
a timescale of $\simeq 70$ days. I take the range to include all these
timescales, namely, from $26$ days to $200$ days. I mark these two end with
yellow-red stars on Fig. 1. Fig. 1 that is adapted from Kashi & Soker (2017)
presents many ILOTs in a plane of their total energy versus their typical
timescale. Note that the energy is the total energy, including radiation and
kinetic energy.

Figure 1: Observed transient events on the energy time diagram adapted from
Kashi & Soker (2017) (where more details can be found). Blue empty circles
represent the total (radiated plus kinetic) energy of the observed transients
as a function of the duration of their eruptions, i.e., usually the time for
the visible luminosity to decrease by 3 magnitudes. The three lines on the
left are theoretical estimates for ILOTs powered by planet-brown dwarf (BD)
interaction (Bear, Kashi, & Soker, 2011). Kashi & Soker (2017) place the ILOT
ASASSN-15qi (observational data from Herczeg et al. 2016) and argued it was
powered by a star-planet interaction. I added ASASSN-13db (observational data
from Sicilia-Aguilar et al. 2017) with its estimated total energy that Kashi
et al. (2019) claimed to have been powered by a star engulfing planet debris,
and ZTF SLRN-2020 from De et al. (2023) who argued it was powered by a star
engulfing a planet. Both the radiated energy (yellow-filled red stars) and the
crudely estimated total energy (radiation + kinetic; two yellow-filled blue
circles) are shown with the timescale range of ZTF SLRN-2020 (see text). The
timescale range is not due to uncertain observations, but rather in the way
that one defines the timescale, total duration or decline rate. The lower-left
part (hatched in green) is the extension that Kashi & Soker (2017) made to
include YSO-planet systems. The new observations suggest that systems of a
well evolved main sequence star and a planet also populate this region.
Abbreviation: BD: brown dwarf; GE: Great Eruption; ILRT: intermediate
luminosity red transient; LRT: luminous red transient; SN: supernova; VMSs:
very massive stars;

When there is no information on the kinetic energy the structure of this
diagram assumes that the total energy is ten times the radiated energy. This
estimate gives here a total energy of $E\simeq 6.5\times 10^{42}$ (for a
distance of $d=4~{}\rm{kpc}$) for ZTF SLRN-2020. De et al. (2023) crudely
estimate an ejecta mass and velocity $M_{\rm ej}\approx 3\times
10^{-5}M_{\odot}$ and $v_{\rm ej}\approx 100~{}\rm{km}~{}\rm{s}^{-1}$. This
gives a kinetic energy of $E_{\rm ej}\approx 3\times 10^{42}~{}\rm{erg}$. The
estimated total energy from observations is therefore $E_{\rm ej}+E_{\rm
rad}\approx 3.6\times 10^{42}~{}\rm{erg}$. Therefore, the value of $E\simeq
6.5\times 10^{42}$ that I take here is a reasonable upper value of the total
energy (and there is the additional uncertainty of the distance). I mark the
location of ZTF SLRN-2020 on Fig. 1 with $E\simeq 6.5\times 10^{42}$ and the
range of timescales by the two yellow-filled blue circles and a line
connecting them.

Kashi et al. (2019) analyzed the ILOT ASASSN-13db that was observed by
Sicilia-Aguilar et al. (2017). The total duration of high emission lasted
$\approx 800$ day, ending with a decline that lasted $\approx 55$ days. The
total radiated energy during the (more or less) plateau of $\approx 800$ days
is $E_{\rm rad}\approx 2\times 10^{41}~{}\rm{erg}$. The total energy of the
event is about an order of magnitude larger. Kashi et al. (2019) scale it with
$\approx 10^{42}~{}\rm{erg}$. I take it here to be ten times lager at $E\simeq
2\times 10^{42}$. I mark this energy with the span of the timescale of 55 to
800 days. This timescale range is not the uncertainty in observations, but
rather the unclear way by which the event timescale should be defined, i.e.,
should we take only the decline phase by three magnitudes or so, or the entire
duration, or the plateau duration, etc.

From the locations of the three ILOTs (Red Novae) claimed to be powered by
star-planet interaction, ASASSN-15qi, ASASSN-13db, and ZTF SLRN-2020, on the
Energy-Time diagram we can learn the following. (1) V838 Mon is much above
their location, and it was not powered by a star-planet interaction, contrary
to the claim by Retter & Marom (2003). (2) The three star-planet ILOTs occupy
a relatively well defined area in the energy-time diagram that is clearly
below those that are thought to be powered by stellar-binary systems. (3) The
star-planet ILOTs have typical timescales that are much longer than the
dynamical time of the star, by about two orders of magnitude and more. This
might suggest a powering process that is much longer than the dynamical time
scale. An accretion disk that is formed by the planet might have this property
(e.g., Bear, Kashi, & Soker 2011). (4) The timescale of the star-planet ILOTs
is not necessarily well defined, as a plateau phase in their lightcurve might
be very long. Nonetheless, their decline phase might have a similar behavior
as regular ILOTs (e.g., Kashi et al. 2019). (5) The recently added star-planet
ILOT, ZTF SLRN-202 (De et al., 2023), does not seem to be a YSO (or even a
pre-main sequence star) as the other two are. Despite that it occupies the
same general area of the other two star-planet ILOTs and close to the area
that Kashi & Soker (2017) mark to be due to star-planet interaction in
planetary systems of YSOs (green lines on the lower left of the diagram). This
suggests that there are common powering processes to these three ILOTs. A
common properties might results from accretion disks that launch jets. I turn
to study this possibility in section 3.

There are some differences between the three ILOTS, in particular the multi-
outbursts that ASASSN-15qi and ASASSN-13db had, but not ZTF SLRN-2020.
ASASSN-13db experienced a shorter and fainter outburst in 2013 (Holoien et
al., 2014), about a year before the beginning of the main outburst (Sicilia-
Aguilar et al., 2017). Kashi et al. (2019) attribute this early outburst to a
typical outburst of YSOs. The ILOT ASASSN-15qi experienced an outburst in
1976, 39 years prior to the main outburst (Herczeg et al., 2016). There is not
enough data to classify the early outburst either as an ILOT with another
planet or as a typical YSO outburst. In any case, it seems that the two early
outbursts in the YSO ILOTs, which ZTF SLRN-2020 was not observed to have,
result from ASASSN-15qi and ASASSN-13db being YSOs.

##  3 The roles of jets

Observations and theoretical arguments suggest that jets play major roles in
many ILOTs (Red Novae). Observations of ILOTs that have a spatially resolved
ejecta show the ILOTs to possess bipolar structures that strongly hint at
shaping by jets. These include the Great Eruption of Eta Carinae (Davidson, &
Humphreys, 1997), a luminous blue variable (LBV) with its bipolar Homunculus
nebula, V4332 Sgr that has a bipolar structure (Kaminski et al., 2018), and
Nova 1670 (CK Vulpeculae) with a 350-years old bipolar nebula (Shara et al.,
1985) that has an S-morphology (Kaminski et al., 2020; Kamiński et al., 2021),
a morphological type that must be shaped by jets. In a new study, Mobeen et
al. (2023) report that the ejecta (nebula) of V838 Mon is bipolar.

Theoretically, jets are very efficient in powering ILOTs and more efficient
than equatorial ejecta (Soker 2020; for powering by equatorial ejecta see,
e.g., Pejcha et al. 2016a, b; Metzger, & Pejcha 2017). Specific studies show
that jets can account for the lightcurves of at least some ILOTs (e.g., Soker
2020; Soker & Kaplan 2021). These studies were aiming at ILOTs powered by
stellar binary interaction. I turn to suggest that the planet in ZTF SLRN-2020
also launched jets (or disk-wind).

I consider the following parameters that De et al. (2023) infer for the
planetary system ZTF SLRN-2020. A stellar mass of $M_{1}\simeq 1M_{\odot}$, a
stellar radius of $R_{1}\simeq 1-4M_{\odot}$, and a planet mass of $M_{\rm
p}\simeq 0.01M_{\odot}$. For the planet to expel mass from the system or to
accrete mass it should orbit very close to the stellar surface. For a solar
mass and an orbit at $a=2R_{\odot}$ the orbital velocity is $v_{\rm
orb}=310~{}\rm{km}~{}\rm{s}^{-1}$. The radius of the planet of the above mass
is $R_{\rm p}\simeq 0.1R_{\odot}$ (e.g., Bashi et al. 2017), and so the escape
velocity from the planet is $v_{\rm p,es}\simeq 200~{}\rm{km}~{}\rm{s}^{-1}$.

The ILOT ZTF SLRN-2020 had months of pre-outburst activity, including mass
loss (De et al., 2023). I consider a process by which the planet accreted mass
and launched jets (or a bipolar disk wind) that powered the pre-outburst
activity. As is the case with stellar winds, the typical terminal velocity of
jets is about the escape velocity from the object that launches the jets
(e.g., Livio 2009). The properties of jets might change over short timescales
relative to their total activity time period. Some parts in the jets during
some jet-launching episodes might have terminal velocities that are larger
than the escape velocity, i.e., $v_{\rm jet,m}=\beta v_{\rm p,es}$ with
$\beta>1$. For the parameters I take above, the escape velocity from the star
at the location of planet is $v_{\rm 1,es}=2^{1/2}v_{\rm orb}\simeq
440~{}\rm{km}~{}\rm{s}^{-1}$. Because the jets are launched perpendicular to
the orbital plane, the terminal velocity of the fastest parts of the jets
relative to the star is

$v_{\rm jet,m,1}=\sqrt{(\beta v_{\rm p,es})^{2}+v^{2}_{\rm orb}}.$ (1)

These segments of the jets escapes the system if $v_{\rm jet,m,1}>v_{\rm
1,es}$, which reads $\beta>v_{\rm orb}/v_{\rm p,es}\simeq 1.5$. The property
of this system that the escape velocity from the planet is of the same order
of magnitude as the escape velocity from the system implies that some jets
segments can reach the escape velocity, but not by much. These jet segments
barely escape the star. If they do, their final outflow velocity from the
system is much smaller than the escape velocity from the system.

I suggest that this pre-outburst jet activity explains the slow outflow
velocity of the dust $\approx 35~{}\rm{km}~{}\rm{s}^{-1}\ll v_{\rm orb}$ (De
et al., 2023). Because the outflow velocity from the planet of the escaping
gas/dust, which is perpendicular to the orbital plane, is about equal to the
orbital velocity, the outflow direction of the escaping dust/gas in this case
is about $\theta_{\rm j}\approx 45^{\circ}$ to the equatorial plane. This
forms a bipolar outflow morphology, in addition to possible concentration of
outflowing gas/dues in the equatorial plane.

The accreted mass onto the planet forms an outer layer around the planet. When
the planet spirals-in closer to the stellar surface, the planet might lose
this material back to the star, hence forming an accretion disk around the
star. This accretion disk might launch much faster jets. These jets might play
a role in powering the main outburst, as was suggested for ILOTs power by
stellar binary systems.

If we take the lowest mass range of masses that De et al. (2023) discuss for
the planet in ZTF SLRN-2020, $M_{\rm p}\simeq 10^{-4}M_{\odot}$, then we are
in a different regime. The average density of such planets is
$<1~{}\rm{g}~{}\rm{cm}^{-1}$ and they might be tidally disrupted by a sun like
star on the main sequence (e.g., Yamazaki, Hayasaki, & Loeb 2017). This brings
the scenario to be much more similar to what Kashi & Soker (2017) suggested
for the YSO-planet system ASASSN-15qi and Kashi et al. (2019) suggested for
the YSO-planet system ASASSN-13db. However, De et al. (2023) consider the
tidal disruption of the planet to be unlikely.

##  4 Summary

The goal of this study is to group the newly analyzed (De et al., 2023) star-
planet ILOT (Red Nova) ZTF SLRN-2020 with the two other star-planet ILOTs,
ASASSN-15qi (Kashi & Soker, 2017) and ASASSN-13db (Kashi et al., 2019). I
added ASASSN-13db and ZTF SLRN-2020 to the energy-time diagram of ILOTs (Fig.
1). These three star-planet ILOTs occupy a well defined area in that diagram,
below all other ILOTs and below classical novae. They are in the general area
that was marked as YSO ILOTs in young star-planet interactions. The new event
ZTF SLRN-2020 is not a YSO-planet system, but nonetheless located in the same
area. This might points to a similar powering mechanism.

I suggested (section 3) that one of the common ingredients might be the
launching of jets by the star. For the ILOT ASASSN-15qi that occurred in a
young planetary system Kashi & Soker (2017) suggested that the star tidally
disrupted the planet, forming an accretion disk that launched the jets during
the event. In the star-planet ILOT ZTF SLRN-2020 the star engulfed the planet
rather than tidally disrupted the planet (De et al., 2023). I suggested that
the planet accreted some mass from the star in the months before the main
outburst. At the early phase of the outburst as the planet spiralled-in close
to the star the star tidally removed the accreted mass. This formed an
accretion disk around the star that launched jets. This suggested process must
be confirmed by three-dimensional hydrodynamical simulations.

I also suggested that the planet launched jets as it accreted mass. These
formed the slowly expanding outflowing dust. The jets that the planet launched
form a concentrated outflow at $\approx 45^{\circ}$ to the equatorial plane.
The jets that the star might have launched are perpendicular to the equatorial
plane. Overall, I expect the outflow from ZTF SLRN-2020 to form a bipolar
nebula, as other ILOTs have, e.g., Nova 1670 (CK Vulpeculae; Kaminski et al.
2020; Kamiński et al. 2021; section 3).

## Acknowledgments

I thank Dima Shishkin and Amit Kashi for their help with the graphics. I thank
an anonymous referee for helpful comments. This research was supported by a
grant from the Israel Science Foundation (769/20).

## Data availability

The data underlying this article will be shared on reasonable request to the
corresponding author.

## References

  * Bashi et al. (2017) Bashi D., Helled R., Zucker S., Mordasini C., 2017, A&A, 604, A83.
  * Bear, Kashi, & Soker (2011) Bear E., Kashi A., Soker N., 2011, MNRAS, 416, 1965.
  * Berger et al. (2009) Berger, E., Soderberg, A. M., Chevalier, R. A., et al. 2009, ApJ, 699, 1850
  * Davidson, & Humphreys (1997) Davidson, K., & Humphreys, R. M. 1997, ARA&A, 35, 1
  * De et al. (2023) De, K., MacLeod, M., Karambelkar, V. et al., 2023, Nature, 617, 55
  * Gurevich, Bear, & Soker (2022) Gurevich O., Bear E., Soker N., 2022, MNRAS, 511, 1330.
  * Herczeg et al. (2016) Herczeg G. J., Dong S., Shappee B. J., Chen P., Hillenbrand L. A., Jose J., Kochanek C. S., et al., 2016, ApJ, 831, 133.
  * Holoien et al. (2014) Holoien T. W.-S., Prieto J. L., Stanek K. Z., Kochanek C. S., Shappee B. J., Zhu Z., Sicilia-Aguilar A., et al., 2014, ApJL, 785, L35.
  * Jencson et al. (2019) Jencson, J. E., Kasliwal, M. M., Adams, S. M., et al. 2019, ApJ, 886, 40
  * Kaminski et al. (2020) Kaminski, T., Menten, K. M., Tylenda, R., et al. 2020, arXiv:2006.10471
  * Kamiński et al. (2021) Kamiński T., Steffen W., Bujarrabal V., Tylenda R., Menten K. M., Hajduk M., 2021, A&A, 646, A1.
  * Kaminski et al. (2018) Kaminski, T., Steffen, W., Tylenda, R., Young, K. H., Patel, N. A., & Menten, K. M. 2018, A&A, 617, A129
  * Kashi et al. (2019) Kashi, A., Michaelis, A. M., & Feigin, L. 2019, Galaxies, 8, 2.
  * Kashi & Soker (2016) Kashi, A., & Soker, N. 2016, Research in Astronomy and Astrophysics, 16, 99
  * Kashi & Soker (2017) Kashi, A. & Soker, N. 2017, MNRAS, 468, 4938.
  * Livio (2009) Livio M., 2009, ASSP, 13, 3 (Protostellar Jets in Context, by Kanaris Tsinganos, Tom Ray, Matthias Stute. Astrophysics and Space Science Proceedings Series. Berlin: Springer)
  * Metzger, Giannios, & Spiegel (2012) Metzger B. D., Giannios D., Spiegel D. S., 2012, MNRAS, 425, 2778.
  * Metzger, & Pejcha (2017) Metzger, B. D., & Pejcha, O. 2017, MNRAS, 471, 3200
  * Mobeen et al. (2023) Mobeen M. Z., Kamiński T., Matter A., Wittkowski M., et al., 2023,
  * Muthukrishna et al. (2019) Muthukrishna, D., Narayan, G., Mandel, K. S., Biswas, R., & Hložek, R. 2019, PASP, 131, 118002
  * O’Connor et al. (2023) O’Connor C. E., Bildsten L., Cantiello M., Lai D., 2023, arXiv, arXiv:2304.09882.
  * Pastorello & Fraser (2019) Pastorello, A., & Fraser, M. 2019, Nature Astronomy, 3, 676
  * Pastorello et al. (2019) Pastorello, A., Mason, E., Taubenberger, S., et al. 2019, A&A, 630, A75
  * Pejcha et al. (2016a) Pejcha, O., Metzger, B. D., & Tomida, K. 2016a, MNRAS, 455, 4351
  * Pejcha et al. (2016b) Pejcha, O., Metzger, B. D., & Tomida, K. 2016b, MNRAS, 461, 2527
  * Retter & Marom (2003) Retter, A., & Marom, A. 2003, MNRAS, 345, L25
  * Retter et al. (2006) Retter, A., Zhang, B., Siess, L., Levinson , A. 2006, MNRAS, 370, 1573.
  * Shara et al. (1985) Shara, M. M., Moffat, A. F. J., & Webbink, R. F. 1985, ApJ, 294, 271
  * Sicilia-Aguilar et al. (2017) Sicilia-Aguilar A., Oprandi A., Froebrich D., Fang M., Prieto J. L., Stanek K., Scholz A., et al., 2017, A&A, 607, A127.
  * Soker (2020) Soker N., 2020, ApJ, 893, 20.
  * Soker & Kaplan (2021) Soker N., Kaplan N., 2021, RAA, 21, 090.
  * Soker & Tylenda (2003) Soker, N. & Tylenda, R. 2003, ApJ, 582, L105.
  * Yamazaki, Hayasaki, & Loeb (2017) Yamazaki R., Hayasaki K., Loeb A., 2017, MNRAS, 466, 1421.