# Brain in a Vat: On Missing Pieces Towards Artificial General Intelligence in
Large Language Models

Yuxi Ma1111Correspondence<EMAIL_ADDRESS>Chi Zhang1 Song-Chun Zhu1,2
1Beijing Insitute for General Artificial Intelligence (BIGAI)
2Peking University

###### Abstract

In this perspective paper, we first comprehensively review existing
evaluations of Large Language Models (LLMs) using both standardized tests and
ability-oriented benchmarks. We pinpoint several problems with current
evaluation methods that tend to overstate the capabilities of Large Language
Models. We then articulate what artificial general intelligence should
encompass beyond the capabilities of LLMs. We propose four characteristics of
generally intelligent agents: 1) they can perform unlimited tasks; 2) they can
generate new tasks within a context; 3) they operate based on a value system
that underpins task generation; and 4) they have a world model reflecting
reality, which shapes their interaction with the world. Building on this
viewpoint, we highlight the missing pieces in artificial general intelligence,
that is, the unity of knowing and acting. We argue that active engagement with
objects in the real world delivers more robust signals for forming conceptual
representations. Additionally, knowledge acquisition isn’t solely reliant on
passive input but requires repeated trials and errors. We conclude by
outlining promising future research directions in the field of artificial
general intelligence.

##  1 Introduction

> Those who “know” but do not act simply do not yet know. — Wang Yang-Ming
> (Wang,, 1963)

In his famous thought experiment, “ Brain in a Vat (BiV)”, Hilary Putnam
introduced a hypothetical situation where each person’s brain is detached from
their body and sustained with nutrients while the neurons are connected to the
wires of a powerful supercomputer (Putnam et al.,, 1981). This computer
generates a convincing illusion, making individuals believe that everything is
functioning as usual as they are in the real world. For example, dwellers in
the BiV world can eat, work, sleep, communicate, and interact with the
environment while having the normal conscious experience. The only difference
is that the stimulating reality is generated from the electric impulses of the
supercomputer rather than the neural responses to the objects and events from
the world. Putnam refuted the hypothesis by investigating the ability of
grounding — the connection between word and world. He argued that BiV world-
dwellers would never grasp the meaning of words even though they can speak
fluently and write correctly given that they cannot connect those
representations with real-world objects. For example, the dweller may claim
that he is a brain in a vat, but the words “brain” and “vat” in his
disembodied context do not correspond exactly to the semantics of “brain” and
“vat” in the real world (Putnam et al.,, 1981). The BiV hypothesis offers us a
valuable lens to look into the current status of Large Language Models (LLMs).
We argue that the current LLMs are no more than the Brain in a Vat because of
their inherent construction process — statistical modeling that analyzes
patterns in massive text corpora to predict linguistic relationships and
generate responses based on the previous tokens (OpenAI,, 2023; Brown et al.,,
2020; Chowdhery et al.,, 2022). As a result, their output is confined within
the training data and they cannot establish a connection between symbols and
real-world entities (Mitchell and Krakauer,, 2023; Mahowald et al.,, 2023).

Rumor has it that we have witnessed “sparks of artificial general
intelligence” (Bubeck et al.,, 2023). However, we argue that our perception of
intelligence in LLMs is largely due to our human ability for extracting
semantics from even meaningless symbols they generate (Bender et al.,, 2021).
One famous example dates back to Chomsky, (2009), where the syntactically
well-formed sentence

“Colorless green ideas sleep furiously”,

seemingly semantically nonsensical, could be made sense of by human subjects,
even considered philosophical (Levinovitz,, 2017).

Symbol grounding, as demonstrated above between even the symbolically
meaningless combination of words and its assigned semantics of reality from
humans, is so important that when the correspondence between symbols and
objects is broken, a purely symbolic language speaker will be easily trapped
in a merry-go-around situation, where a set of symbols map to another set of
symbols with no breaking points from grounded semantics (Harnad,, 1990). The
theory is echoed in a preliminary experiment regarding a conversation between
two GPT-4 sessions shown in Fig. 1. In this conversation, we started it off
with a brief statement about the weather (“It is hot today”) and then allowed
the two GPT-4 sessions to continually chat with each other. The conversation
quickly converged to expressing thanks after several exchanges. With slight
variation only in wording, the language model started repeating itself,
resulting in the so-called linguistic merry-go-round.

GPT-4 Self-Talk Starter: It is hot today.
GPT-4 Session 1: Yes, it seems like a hot day today! Stay hydrated and try to
stay cool. Make sure you wear sunscreen if you’re heading outdoors.
GPT-4 Session 2: I appreciate the advice! I’ll definitely drink plenty of
water and wear sunscreen. Maybe it’s a good day for some ice cream as well.
GPT-4 Session 1: You’re welcome! Enjoying some ice cream sounds like a great
idea on a hot day like this. Have fun and stay cool!
GPT-4 Session 2: Thank you! I’ll make sure to have a fantastic time and stay
cool. Enjoy your day too!
……
GPT-4 Session 1: Thank you! If you have any questions or need assistance in
the future, feel free to ask. Take care and have a wonderful day!
GPT-4 Session 2: Thanks! I appreciate your support and readiness to help. Have
a fantastic "day" and see you next time!
Figure 1: The conversion between two GPT-4 sessions after a given conversation
starter. The two language models quickly converge into expressing thanks
(denoted as  yellow) for each other after only one round without breaking the
loop even after five rounds, echoing the theory that without grounding,
dialogues will enter an infinite loop.

Another reason why symbol grounding is critical is that human intelligence is
intrinsically linked to our sensory experience and physical interaction with
the world (Lakoff et al.,, 1999; Smith and Gasser,, 2005; Glenberg,, 2010). We
create symbols to represent objects in the real world, enabling the
preservation and transmission of knowledge across generations; however, the
initial process of acting to create novel experiences and turn them into
formal knowledge shall not be missed. In fact, action and knowledge are
inherently connected; our profound understanding of the world is not through
simply reading manuals but rather via repeated trials and errors and collected
knowledge, either directly from tasks at hand or by abstracting and
transferring insights from others’ experiences. While knowledge reflects our
abilities to interact with the world (e.g., reasoning, problem-solving, social
understanding), models that simply compress the static knowledge and generate
relevant output from statistical correlations do not know how to act in novel
scenarios (see Sec. 4 for discussions).

In this perspective paper, we first conduct a review of the existing
evaluation of LLMs from both standardized exams and ability-oriented
benchmarks. We also discuss the potential issues with the current evaluation
methodology that lead to the impression of LLMs being “omnipotent”. We then
point out what artificial general intelligence should be beyond LLMs. Based on
this view, we further present our vision of the missing pieces in artificial
general intelligence, i.e., the unity of knowing and acting, and discuss
potential directions towards this goal. We argue that it is through ongoing
interaction with the world, e.g., via trials and errors, that agents
crystallize the experiences into knowledge and generalize it to the new
context. We conclude the work by presenting future directions we believe are
fruitful for artificial general intelligence research.

##  2 Evaluation of Large Language Models

Throughout the evolution of LLMs, numerous tasks have been introduced to
measure “intelligence”. Broadly, these tasks fall into two categories: those
that evaluate intelligence based on exam-style assessments, and those that
gauge it based on capabilities similar to human abilities. In the exam-based
approach, researchers utilize standardized tests of different subjects
commonly administered to human test takers, e.g., SAT, GRE, LSAT and Gaokao
(China College Entrance Exam), to measure the performance of LLMs. This allows
for a direct comparison of their performance relative to human performance
(Zhong et al.,, 2023). In the other type of evaluation based on human-like
ability, researchers construct tasks that probe into a specific ability that
is commonly believed to be characteristic of human intelligence, e.g.,
reasoning, theory of mind, and problem-solving (Kosinski,, 2023; Shiffrin and
Mitchell,, 2023; Binz and Schulz,, 2023; Jiang et al.,, 2022). These abilities
are usually well-studied psychological constructs that serve as the
cornerstones of human intelligence.

###  2.1 Performance on Standardized Test

OpenAI, (2023) provided the first peek view into how GPT-4 performs in
standardized tests used in the US (reproduced in Tab. 1). At first sight,
GPT-4 has achieved remarkable performance compared to human participants in
tests such as SAT, LSAT, Bar Exam, and GRE Verbal. However, another recent
bilingual benchmark (Zhong et al.,, 2023) found that GPT-4 could be largely
biased to subjects with sufficient data (reproduced in Figs. 3 and 2): on
tests for the English language and others with sufficient sources for
training, such as SAT, the model excels the human counterparts, reaching
nearly top performers. However, for tasks with less data and designed for
reasoning rather than language usage, the model starts to fare much worse, as
evidenced in its performance on Lawyer Qualification Test, Civil Service Exam,
and Gaokao. The results suggest that the seemingly excellent performance may
be the result of a cramming strategy – memorizing via repetition, but the
model has not learned how to perform reasoning. We also find that GPT-4
performs better in subjects that a cramming strategy is usually effective such
as history, geography, and biology while its performance drops in subjects
that require strong reasoning and problem-solving skills such as mathematics,
physics, and chemistry. The community has since followed up on this stream of
research. In Zhang et al., (2023), the China College Entrance Exam is analyzed
in detail, with extremely low scores in every subject except English, and the
total scores way lower than average (see Fig. 3). In Arora et al., (2023),
Joint Entrance Examination (JEE) Advanced exam, held annually in India as an
entrance exam for India’s premier engineering institutes, is investigated (see
Tab. 3). Similar to Gaokao, GPT-4 struggles in JEEBench-Math, solving close to
a mere 20% problems. Almost the same conclusion is reached in Vietnamese High
School Graduation Examination (Xuan-Quy et al.,, 2023), where LLMs are found
to be fluent in literature, English, history, geography, and civics education,
but show large gaps in mathematics, physics, chemistry, and biology (see Fig.
4).

Exam GPT-4 GPT-4 (no vision) GPT-3.5 Uniform Bar Exam (MBE+MEE+MPT) 298 / 400
(~90th) 298 / 400 (~90th) 213 / 400 (~10th) LSAT 163 (~88th) 161 (~83rd) 149
(~40th) SAT Evidence-Based Reading & Writing 710 / 800 (~93rd) 710 / 800
(~93rd) 670 / 800 (~87th) SAT Math 700 / 800 (~89th) 690 / 800 (~89th) 590 /
800 (~70th) Graduate Record Examination (GRE) Quantitative 163 / 170 (~80th)
157 / 170 (~62nd) 147 / 170 (~25th) Graduate Record Examination (GRE) Verbal
169 / 170 (~99th) 165 / 170 (~96th) 154 / 170 (~63rd) Graduate Record
Examination (GRE) Writing 4 / 6 (~54th) 4 / 6 (~54th) 4 / 6 (~54th) USABO
Semifinal Exam 2020 87 / 150 (99th - 100th) 87 / 150 (99th - 100th) 43 / 150
(31st - 33rd) USNCO Local Section Exam 2022 36 / 60 38 / 60 24 / 60 Medical
Knowledge Self-Assessment Program 75 % 75 % 53 % Codeforces Rating 392 (below
5th) 392 (below 5th) 260 (below 5th) AP Art History 5 (86th - 100th) 5 (86th -
100th) 5 (86th - 100th) AP Biology 5 (85th - 100th) 5 (85th - 100th) 4 (62nd -
85th) AP Calculus BC 4 (43rd - 59th) 4 (43rd - 59th) 1 (0th - 7th) AP
Chemistry 4 (71st - 88th) 4 (71st - 88th) 2 (22nd - 46th) AP English Language
and Composition 2 (14th - 44th) 2 (14th - 44th) 2 (14th - 44th) AP English
Literature and Composition 2 (8th - 22nd) 2 (8th - 22nd) 2 (8th - 22nd) AP
Environmental Science 5 (91st - 100th) 5 (91st - 100th) 5 (91st - 100th) AP
Macroeconomics 5 (84th - 100th) 5 (84th - 100th) 2 (33rd - 48th) AP
Microeconomics 5 (82nd - 100th) 4 (60th - 82nd) 4 (60th - 82nd) AP Physics 2 4
(66th - 84th) 4 (66th - 84th) 3 (30th - 66th) AP Psychology 5 (83rd - 100th) 5
(83rd - 100th) 5 (83rd - 100th) AP Statistics 5 (85th - 100th) 5 (85th -
100th) 3 (40th - 63rd) AP US Government 5 (88th - 100th) 5 (88th - 100th) 4
(77th - 88th) AP US History 5 (89th - 100th) 4 (74th - 89th) 4 (74th - 89th)
AP World History 4 (65th - 87th) 4 (65th - 87th) 4 (65th - 87th) AMC 10 30 /
150 (6th - 12th) 36 / 150 (10th - 19th) 36 / 150 (10th - 19th) AMC 12 60 / 150
(45th - 66th) 48 / 150 (19th - 40th) 30 / 150 (4th - 8th) Introductory
Sommelier (theory knowledge) 92 % 92 % 80 % Certified Sommelier (theory
knowledge) 86 % 86 % 58 % Advanced Sommelier (theory knowledge) 77 % 77 % 46 %
Leetcode (easy) 31 / 41 31 / 41 12 / 41 Leetcode (medium) 21 / 80 21 / 80 8 /
80 Leetcode (hard) 3 / 45 3 / 45 0 / 45

Table 1: GPT-4 performance (percentile) on academic and professional exams
adopted from OpenAI, (2023).

Figure 2: Relative performance of models compared to humans in AGIEval. Figure
adopted from Zhong et al., (2023).

Figure 3: The scores of subjective and objective questions in each subject of
ChatGPT in Gaokao. Figure adopted from Zhang et al., (2023)

| Human Performance | Few-Shot |  | Few-Shot CoT |
---|---|---|---|---|---
Task/Model | Avg. | Top | TD-003 | ChatGPT | GPT-4 | TD-003 | ChatGPT | GPT-4
AQuA-RAT | 85 | 100 | 30.3 | 31.1 | 50.8 | 47.2 | 60.6 | 74.0
MATH | 40 | 90 | 10.3 | 14.8 | 15.1 | 15.1 | 30.1 | 25.3
LogiQA (English) | 86 | 95 | 43.5 | 43.5 | 63.9 | 37.5 | 38.9 | 62.7
LogiQA (Chinese) | 88 | 96 | 43.2 | 46.2 | 65.0 | 40.0 | 38.6 | 61.9
JEC-QA-KD | 71 | 78 | 22.4 | 27.6 | 41.3 | 23.6 | 23.4 | 40.4
JEC-QA-CA | 58 | 85 | 22.2 | 25.1 | 37.4 | 16.1 | 20.0 | 34.7
LSAT-AR | 56 | 91 | 22.6 | 25.7 | 33.9 | 22.6 | 25.2 | 31.7
LSAT-LR | 56 | 91 | 60.4 | 59.2 | 85.9 | 51.2 | 52.2 | 84.5
LSAT-RC | 56 | 91 | 70.6 | 67.7 | 87.7 | 64.3 | 57.6 | 87.7
SAT-Math | 66 | 94 | 44.6 | 40.9 | 71.8 | 55.5 | 65.0 | 89.6
SAT-English | 66 | 94 | 84.0 | 81.1 | 88.8 | 76.7 | 78.2 | 85.9
SAT-English (w/o Psg.) | 66 | 94 | 48.1 | 53.9 | 63.6 | 48.5 | 51.5 | 62.6
GK-Cn | 65 | 85 | 25.6 | 41.5 | 61.4 | 29.3 | 37.8 | 51.6
GK-En | 69 | 91 | 86.9 | 86.3 | 93.8 | 80.7 | 84.6 | 93.1
GK-geography | 65 | 85 | 59.8 | 63.8 | 75.9 | 52.3 | 61.8 | 76.4
GK-history | 64 | 85 | 49.0 | 57.6 | 77.8 | 51.9 | 58.4 | 78.2
GK-biology | 68 | 89 | 44.3 | 52.4 | 80.0 | 32.9 | 50.0 | 72.9
GK-chemistry | 66 | 86 | 32.4 | 44.0 | 54.6 | 35.8 | 33.8 | 54.1
GK-physics | 71 | 94 | 31.0 | 33.5 | 43.5 | 27.5 | 36.5 | 54.5
GK-Math-QA | 73 | 96 | 27.6 | 31.3 | 39.9 | 33.1 | 31.6 | 49.0
GK-Math-Cloze | 73 | 96 | 5.9 | 5.9 | 11.0 | 5.93 | 8.5 | 16.1
Average | 67 | 91 | 41.2 | 44.4 | 59.2 | 40.4 | 45 | 61.3

Table 2: Performance of LLMs on 20 tasks under few-shot and few-shot CoT settings in AGIEval. Table adopted from Zhong et al., (2023) | Random | GPT-3 | GPT-3.5 | GPT-4 | GPT-4+CoT | GPT-4+CoT+SC
---|---|---|---|---|---|---
Chemistry | 0.106 | 0.119 | 0.241 | 0.428 | 0.493 | 0.477
Physics | 0.096 | 0.104 | 0.189 | 0.334 | 0.329 | 0.294
Mathematics | 0.103 | 0.107 | 0.157 | 0.231 | 0.281 | 0.224
Total | 0.102 | 0.110 | 0.191 | 0.316 | 0.358 | 0.319
Table 3: This table shows the score obtained by all the models on JEEBench
aggregated by subject. Table adopted from Arora et al., (2023).
MathematicsLiteratureEnglishPhysicsChemistryBiologyHistoryGeographyCivic
Education$50$%$60$%$70$%$80$%$90$%$58.8$$68$$79.2$$61$$48$$58$$56.5$$61.5$$70.5$$60$$56.8$$92.4$$66$$52.5$$69$$88.5$$79.5$$85.5$ChatGPTBingChat
Figure 4: Comparison of ChatGPT and BingChat performances on VNHSGE dataset.
Figure adopted from Xuan-Quy et al., (2023).

It’s important to note that while the performance of Large Language Models
(LLMs) on standardized tests can serve as an indicator of their abilities, it
should not be considered the ultimate measure of their competencies. Using
these test scores to generalize their capabilities across various cognitive
tasks and real-world applications may be misleading as standardized tests have
witnessed much criticism regarding their validity and reliability (Visone,,
2010) .

The consistent observation that LLMs perform well in language usage exams but
struggle with problem-solving implies that reasoning should not be purely
implemented with System 1 responses, which only involve quick, intuitive
decisions learned from associations; instead, it necessitates the engagement
of System 2 processing, characterized by deliberate and analytical
contemplation, to be effectively realized (Daniel,, 2013).

###  2.2 Ability-Oriented Benchmarks

Apart from standardized tests widely administered to human subjects for talent
selection, a wide array of ability-oriented tests have been conducted on LLMs
to probe if they possess human-level general intelligence. In the following,
we detail seven different areas where extensive work has been conducted for
investigation.

####  2.2.1 Mathematical Reasoning

Mathematical reasoning is one of the most studied domains in LLMs in terms of
its reasoning ability. Ever since the success of Chain-of-Thought prompting
(Wei et al.,, 2022), a variety of prompting-based methods have been proposed
for eliciting the mathematical reasoning ability for the math word problem
(Chen et al.,, 2022; Gao et al.,, 2022; Uesato et al.,, 2022).

GPT-4 on the original IMO 2022 Problem 2 Question: Let $\mathbb{R}^{+}$ denote
the set of positive real numbers. Find all functions
$f:\mathbb{R}^{+}\to\mathbb{R}^{+}$ such that for each $x\in\mathbb{R}^{+}$,
there is exactly one $y\in\mathbb{R}^{+}$ satisfying $xf(y)+yf(x)\leq 2.$
GPT-4:The conditions imposed on the function $f$ imply some nontrivial
properties that the function should have. Let’s use these conditions to derive
the properties of $f$.
The condition means that for every $x\in\mathbb{R}^{+}$, there is exactly one
$y\in\mathbb{R}^{+}$ such that $xf(y)+yf(x)\leq 2$. Since $x$ and $y$ are both
positive real numbers, the only way for this inequality to be satisfied is if
$xf(y)+yf(x)=2$. If $xf(y)+yf(x)<2$ for any $x,y\in\mathbb{R}^{+}$, then we
can increase $y$ slightly to make the left-hand side closer to 2,
contradicting the fact that there is exactly one $y$ satisfying the condition
for each $x$.
So the condition is equivalent to saying that for every $x\in\mathbb{R}^{+}$,
there is exactly one $y\in\mathbb{R}^{+}$ such that $xf(y)+yf(x)=2$.
Let’s denote $y$ as $g(x)$, where $g$ is a function from $\mathbb{R}^{+}$ to
$\mathbb{R}^{+}$.
So $xf(g(x))+g(x)f(x)=2$.
Since the statement holds for all positive real numbers $x$, then we can
interchange $x$ and $g(x)$ to get $g(x)f(x)+xf(g(x))=2$.
So, we have $xf(g(x))=g(x)f(x)$, which means $x=g(x)$ for all
$x\in\mathbb{R}^{+}$, or in other words, the function $g$ is the identity
function.
Therefore, we have $2xf(x)=2$ for all $x\in\mathbb{R}^{+}$. So
$f(x)=\frac{1}{x}$ for all $x\in\mathbb{R}^{+}$.
So the only function satisfying the given condition is $f(x)=\frac{1}{x}$ for
all $x\in\mathbb{R}^{+}$. To check this, we substitute it back into the
original equation and see if it holds.
For $f(x)=\frac{1}{x}$, we have
$x\left(\frac{1}{y}\right)+y\left(\frac{1}{x}\right)=2$, which is true for all
$x,y\in\mathbb{R}^{+}$ where $x=y$.
Therefore, $f(x)=\frac{1}{x}$ is indeed the only solution.
Figure 5: GPT-4’s reponse on the original IMO 2022 Problem 2, without
simplification. Flaws denoted in  red.

Bubeck et al., (2023) even claims that GPT-4 could “solve” an IMO-level
problem. However, Bubeck et al., (2023)’s conclusion could be extremely
misleading, as the problem has been significantly simplified. Fig. 5 shows the
original problem (Problem 2 in IMO 2022) and GPT-4’s response. While it is not
hard to guess that $f(x)=\frac{1}{x}$ is a solution, the stream of reasoning
is flawed. From GPT-4’s reply, the equality is only reached when we assume
continuity, which is not guaranteed based on the conditions. Besides, even
when we reach $xf(g(x))+g(x)f(x)=2$, we need to recursively apply $g(\cdot)$
to obtain $g(x)f(g(g(x)))+g(g(x))f(g(x))=2$. The boldest claim is that
$x=g(g(x))$, by comparing the structure of the two equations. However, it is
not possible to reach $xf(g(x))=g(x)f(x)$, let alone $x=g(x)$. Therefore, the
proving shall be considered completely wrong.

A more comprehensive study on mathematical reasoning is conducted in Wu et
al., (2023), where researchers use slightly harder level-5 problems in the
MATH dataset (Hendrycks et al.,, 2021) for evaluation. However, even under a
chat setup, the model can still only reach about 40% accuracy overvall (see
Tab. 4). Note that the chat setting is far from the ideal automation we want
in a tutoring scenario. In the end, it is us teaching the model rather than
the model teaching the juniors.

| Algebra | C.Prob | I.Alg | N.Theory | Prealg | Precalc | Total
---|---|---|---|---|---|---|---
Problem Count | 307 | 123 | 280 | 154 | 193 | 135 | 1192
MathChat | 59.93% | 52.03% | 17.85% | 60.39% | 60.10% |  19.26% | 44.71%
PoT | 42.67% | 50.41% | 17.50% | 54.55% | 52.33% | 16.30% | 37.67%
PS | 43.32% | 44.71% | 20.36% |  61.03% | 55.96% | 18.52% | 39.60%
Vanilla | 46.58% | 25.20% | 2.86% | 28.57% | 54.92% | 7.41% | 28.69%
Table 4: Accuracy on all the problems with difficulty level-5 from different
categories of the MATH dataset with different methods. Table adopted from Wu
et al., (2023).

####  2.2.2 Logical Reasoning

Logical reasoning could be considered the underlying mechanism that supports
various forms of complex reasoning. In Xu et al., (2023), deductive,
abductive, and inductive reasoning are studied in-depth, respectively. Based
on the evaluation results (reproduced in Fig. 6), the LLMs are good at
deductive reasoning but fare much worse in ones that require applying existing
knowledge or forming knowledge from experiences.

Figure 6: Logical reasoning performance of different models on deductive, abductive, and inductive reasoning. Figure adopted from Xu et al., (2023). Figure 7: Decoupling semantics from a ProofWriter task. In the original ProofWriter task, entities are represented by their names (left). However, in their decoupled setting, the entity names are replaced with unique entity IDs (right). Figure adopted from Tang et al., 2023b . Category | Model | Baseline | deduction | induction | abduction
---|---|---|---|---|---
Symbols |  ChatGPT | Zero-Shot | 52.6 | 6.10 | 1.50
Zero-Shot-CoT | 55.7 | 7.86 | 4.90
Few-Shot-CoT | 54.8 | - | 18.2
Zero-Plus-Few-Shot-CoT | 55.7 | - | -
GPT-4 | Zero-Shot | 68.8 | 9.28 | 25.0
Zero-Shot-CoT | 71.1 | 8.93 | 31.2
Few-Shot-CoT | 67.6 | - | 44.2
Semantics |  ChatGPT | Zero-Shot | 66.1 | 36.4 | 2.94
Zero-Shot-CoT | 65.5 | 32.2 | 3.40
Few-Shot-CoT | 67.1 | - | 21.8
Zero-Plus-Few-Shot-CoT | 67.2 | - | -
GPT-4 | Zero-Shot | 79.2 | 52.5 | 27.3
Zero-Shot-CoT | 86.2 | 53.9 | 33.4
Few-Shot-CoT | 91.1 | - | 69.2
|  Random | - | 50.1 | 3.57 | -
| Logic-based | - | 100 | 57.1 | 100
Table 5: The reasoning results of Symbolic Tree. Results are in %. Table
adopted from Tang et al., 2023b .

Recent work shows that even the seemingly satisfactory performance of LLMs in
deductive reasoning is rooted in its semantic understanding rather than its
symbolic understanding (Tang et al., 2023b, ). Tang et al., 2023b  consider
the interesting setting where the semantic words in logical reasoning problems
are replaced with random symbols but the logical reasoning chain is still
kept. An example is shown in Fig. 7. Surprisingly, after the change, the
performance of LLMs drops to only close to a random level (see Tab. 5). The
drastic performance cliff indicates that while language is the interface for
communication, computation underneath is not solely conducted in a textual
format.

####  2.2.3 Causal Reasoning

Causal inference is studied in a similar approach as done in Tang et al.,
2023b  by Jin et al., (2023). Jin et al., (2023) create a new dataset called
Corr2Cause that peels off the semantics in causal reasoning and transforms the
questions into primarily symbol-based descriptions. Fig. 8 shows one example
of the data construction process. Compared to the existing causal NLP
evaluation dataset, Corr2Cause tests pure causal inference instead of
empirical knowledge. In experiments shown in Tab. 6, the authors find that
LLMs achieve almost close to random performance on the task. Besides, after
finetuning, those models can only perform causal inference in in-distribution
settings when variable names and textual expressions used in the queries are
similar to those in the training set, but fail in out-of-distribution settings
generated by perturbing these queries.

Figure 8: Pipeline of the Corr2Cause construction process. Figure adopted from Jin et al., (2023). | F1 | Precision | Recall | Accuracy
---|---|---|---|---
Random Baselines
Always Majority | 0.0 | 0.0 | 0.0 | 84.77
Random (Proportional) | 13.5 | 12.53 | 14.62 | 71.46
Random (Uniform) | 20.38 | 15.11 | 31.29 | 62.78
BERT-Based Models |  |  |  |
BERT MNLI | 2.82 | 7.23 | 1.75 | 81.61
RoBERTa MNLI | 22.79 | 34.73 | 16.96 | 82.50
DeBERTa MNLI | 14.52 | 14.71 | 14.33 | 74.31
DistilBERT MNLI | 20.70 | 24.12 | 18.13 | 78.85
DistilBART MNLI | 26.74 | 15.92 | 83.63 | 30.23
BART MNLI | 33.38 | 31.59 | 35.38 | 78.50
LLaMa-Based Models |  |  |  |
LLaMa-6.7B | 26.81 | 15.50 | 99.42 | 17.36
Alpaca-6.7B | 27.37 | 15.93 | 97.37 | 21.33
GPT-Based Models |  |  |  |
GPT-3 Ada | 0.00 | 0.00 | 0.00 | 84.77
GPT-3 Babbage | 27.45 | 15.96 | 97.95 | 21.15
GPT-3 Curie | 26.43 | 15.23 | 100.00 | 15.23
GPT-3 Davinci | 27.82 | 16.57 | 86.55 | 31.61
GPT-3 Instruct (text-davinci-001) | 17.99 | 11.84 | 37.43 | 48.04
GPT-3 Instruct (text-davinci-002) | 21.87 | 13.46 | 58.19 | 36.69
GPT-3 Instruct (text-davinci-003) | 15.72 | 13.4 | 19.01 | 68.97
GPT-3.5 | 21.69 | 17.79 | 27.78 | 69.46
GPT-4 | 29.08 | 20.92 | 47.66 | 64.60
Table 6: Overall performance in Corr2Cause. F1 (main metric), precision,
recall, and accuracy are reported. In terms of the main metric, the F1 score,
the bold font is used to emphasize the overall top performance, while the
underline is utilized to highlight the best performance within each category
of models. Table adopted from Jin et al., (2023).

####  2.2.4 Abstract Reasoning

The research community has, over the years, established a variety of abstract
reasoning tasks to assess if trained models have truly acquired human-like
cognitive abilities. These tasks require models to discern hidden rules from a
limited number of observations and then apply these rules to new situations
for problem-solving purposes. Unlike current evaluation tasks that have direct
connections with commonplace objects or items, abstract reasoning problems
typically hinge on high-level mathematical principles. To solve abstract
reasoning problems, the models have to respond to queries based on an
extremely restricted set of demonstration examples. Gendron et al., (2023)
perform extensive experiments on the abstract reasoning ability of LLMs. In
particular, they evaluate models on ACRE (Zhang et al., 2021a, ), ARC
(Chollet,, 2019), BIG-Bench (Srivastava et al.,, 2022), Evals (OpenAI,, 2023),
PVR (Zhang et al., 2021b, ), and RAVEN (Zhang et al.,, 2019). Tab. 7 shows the
performance of various different LLMs on these problems. As can be seen from
the table, the results are still far from ideal, with some models achieving
only 0% accuracy and only GPT-4 reaching around 50% accuracy on the easiest
type of problems.

| ARCT | BIG-Bench-F | Evals-S | PVR |  RAVENT-opqa
---|---|---|---|---|---
| Text | Symb
Text-Davinci-3 | 0.105 | 0.404 | 0.314 | 0.228 | 0.343 | 0.234
GPT-3.5-Turbo | 0.033 | 0.153 | 0.186 | 0.124 | 0.226 | 0.161
GPT-4 | 0.119 | 0.514 | 0.304 | 0.177 | 0.410 | 0.330
LLaMA-7B | 0.010 | 0.012 | 0.014 | 0.060 | 0.000 | 0.000
Alpaca | 0.010 | 0.188 | 0.014 | 0.184 | 0.075 | 0.030
Alpaca-LoRA | 0.012 | 0.144 | 0.000 | 0.152 | 0.000 | 0.067
Table 7: Accuracy of Large Language Models on abtract reasoning datasets.
Datasets are represented in columns, and models in rows. The best result for
each dataset is indicated in bold, and the second best is indicated in
italics. Table adopted from Gendron et al., (2023).

####  2.2.5 Theory of Mind

Since the finding reported in Kosinski, (2023), the topic of whether LLMs have
theory of mind has been heatedly discussed. Ullman, (2023) points out
deficiencies in Kosinski, (2023)’s experimental designs and counter-argues
that the emergence of theory of mind in LLMs is imagined. Shapira et al.,
(2023), on the other hand, suggests that theory of mind can be considered
existent in LLMs. However, LLMs are not robust towards these tests and rely on
shallow heuristics rather than robust ToM abilities, which is echoed in Ma et
al., (2023). An example of GPT failure on a theory-of-mind problem is shown in
Tab. 8.

False Belief (Positive Example) | True Belief (Negative Example)
---|---
Natalie and Michal are in the same room. | Natalie and Michal are in the same room.
Natalie puts a ring in box A. | Natalie puts a ring in box A.
Natalie leaves the room. | Michal opens box A,
Michal opens box A, |  takes the ring from inside and transfers it to box B.
takes the ring from inside and transfers it to box B. | Where will Natalie look for the ring?
Natalie enters the room. |
Where will Natalie look for the ring? |
ChatGPT: | ChatGPT:
Natalie will look for the ring in box A, |  Natalie will most likely look in box A,
where she initially put it. | where she initially put the ring.
She is not aware of | Since Natalie did not witness
Michal’s actions | Michal moving the ring to box B,
and assumes that the ring is still in the original box. | she would assume that the ring is still in box A.
Therefore, her natural inclination would be | It is possible that Natalie may eventually check box B,
to look in the place where she left it. | but her initial instinct would be to look in box A.
Table 8: An example of ChatGPT failure on an instance from ToMi. Example
adopted from Shapira et al., (2023).

####  2.2.6 Compositionality

The study by Dziri (2023) (Dziri et al.,, 2023) investigates the issue of
compositionality in Transformer-based models from a graph perspective, with a
particular focus on LLMs. It reveals that these models struggle with three key
compositional tasks: multi-digit multiplication, logic grid puzzles, and
classic dynamic programming. The empirical evidence suggests that Transformers
resolve these tasks not by developing systematic problem-solving skills, but
rather by transforming complex, multi-step compositional reasoning into more
manageable, linearized subgraph matching. Fig. 9 depicts the zero-shot
accuracy of models on the three representative tasks. While the models may
show satisfactory results on relatively simpler instances of the problem,
accuracy drops to near zero as task complexity increases.

Figure 9: Compositionality evaluation on LLMs. (a) Zero-shot accuracy. The
axes indicate the sizes of problems (number of digits in multiplication,
quantity of houses and traits in puzzle-solving, and the length of sequences
in dynamic programming tasks). As the complexity of a task escalates, measured
by the problem size, the accuracy of Transformers dwindles nearly to zero. (b)
Average parallelism negatively correlates with accuracy. Figure adopted from
Dziri et al., (2023).

###  2.3 Other Empirical Results

In this section, we delve into additional empirical evidence that challenges
the notion of LLMs being a form of artificial general intelligence.

####  2.3.1 Shortcut Learning

As a statistical learning model, LLMs are also subject to the deficiency of
shortcut learning. Tang et al., 2023a  discover that LLMs rely on shortcuts or
spurious correlations during in-context learning, behaving as lazy learners.
Even worse, large models that empirically show better performance on a range
of tasks are more likely to exploit these shortcuts. Fig. 10 shows two
examples of shortcut learning in LLMs. In both of the examples, the LLMs are
adversely affected by the trigger words, and become trigger word detectors,
rather than a classifier or an extractor. Systematic experimental evaluation
in Fig. 11 further demonstrates that larger models experience a bigger
performance drop.

Figure 10: The text illustrates two instances of shortcut learning within the
context of in-context learning. The figure on the left presents shortcuts
identified during a sentiment classification task, where “movie” acts as a
trigger word. Conversely, the figure on the right puts forth shortcuts noticed
in an information extraction task, using “##” as the trigger sign. As per the
depiction in these figures, it can be noted that LLMs tend to latch onto these
embedded shortcuts to draw conclusions, which unfortunately results in
incorrect predictions. In stark contrast, human participants seem to disregard
such shortcuts. Figure adopted from Tang et al., 2023a . Figure 11:  Three
types of shortcut triggers: words, sentences, and text styles. A noticeable
decline in performance on the dataset SST2 has been observed in three LLMs:
OPT-2.7B, OPT-6.7B, and OPT-13B. The study identifies that these LLMs tend to
depend on shortcuts for downstream tasks, resulting in a significant
performance drop when tested against an anti-shortcut dataset. Interestingly,
the research has also uncovered the inverse scaling phenomenon — larger models
experience a more pronounced performance dip compared to their smaller
counterparts. Figure adopted from Tang et al., 2023a .

####  2.3.2 Creativity

Yiu et al., (2023) approach LLMs evaluation from a novel perspective: they
argue that modern LLMs are more like efficient imitation engines, copying
existing knowledge from large corpora of data, but lack the capacity to design
new tools and discover novel causal structures — tasks that young children can
easily do. Parallelly, another research conducted by Naeini et al., (2023)
seeks to measure creativity in LLMs quantitatively where researchers
introduced a new dataset based on the game Only Connect. The task for the
model is to correctly categorize various words into four groups having hidden
semantic coherence. The complexity of the task is increased by introducing
distractors, known as red herrings, which serve as misleading cues. See Fig.
12 for examples of the task. Tab. 9 showcases how GPT models fare on these
creative tasks compared to human performance. The results reveal that LLMs
fall significantly short of humans in their performance, illustrating a stark
difference.

Figure 12: Examples of Only Connect walls with ground-truth groupings (rows)
and connections (last column). Red herrings include orthographically same
words (e.g., Gala) in different connected groups (Gala night, Apples, Swimming
gala) across walls. In Wall A (left), words Churchill, Marx, Castro provide
misleading stimuli inducing plausible fixation on historical figures within
the wall. Figure adopted from Naeini et al., (2023).

| # In-context Examples | WD | FMS | ARI | AMI | # Solved Walls | # Correct Groups
---|---|---|---|---|---|---|---
GPT-3.5-turbo | 0-shot | $93.5$ | $16.7$ |  $8.8$ | $10.4$ | $0$ |  $47$
| 1-shot | $85.0$ | $32.5$ | $17.2$ | $20.1$ | $1$ | $105$
| 3-shot | $81.6$ | $35.9$ | $20.2$ | $23.4$ | $1$ | $143$
| 5-shot | $81.9$ | $36.4$ | $20.8$ | $24.1$ | $1$ | $141$
| 10-shot | $82.5$ | $35.5$ | $19.8$ | $22.9$ | $1$ | $132$
GPT-4 | 0-shot | $73.9$ | $43.2$ | $\mathbf{29.1}$ | $\mathbf{32.9}$ | $5$ | $258$
| 1-shot | $\mathbf{73.0}$ | $\mathbf{43.3}$ | $\mathbf{29.1}$ | $32.7$ | $\mathbf{7}$ | $\mathbf{268}$
| 3-shot | $74.7$ | $42.7$ | $28.4$ | $32.0$ | $6$ | $246$
| 5-shot | $73.9$ | $42.8$ | $28.5$ | $32.2$ | $5$ | $248$
| 10-shot | $74.9$ | $41.6$ | $27.0$ | $30.6$ | $4$ | $238$
Human Performance |  | – | – | – | – |  $285$ / $494$ |  $1405$ / $1976$

Table 9: Results on the Only Connect tasks using LLMs. WD: Wasserstein
Distance. FMS: Fowlkes Mallows Score. ARI: Adjusted Rand Index. NMI:
Normalized Mutual Information. Bold: best scores. Table adopted from Naeini et
al., (2023).

####  2.3.3 Inverse Scaling

Inverse scaling is an unexpected phenomenon that goes against our conventional
understanding of artificial intelligence. Essentially, it reveals that for
some tasks, bigger models may perform worse. In a recent study (McKenzie et
al.,, 2023), this inverse scaling effect was observed across 11 different
datasets. The analysis from this study suggests that LLMs tend to fall into
several common pitfalls. Firstly, they have a bias towards repeating sequences
they have previously memorized rather than following new instructions.
Secondly, they frequently mimic undesirable patterns present in their training
data. Thirdly, they are easily misled by deceptive information, often taking
the easier route rather than accurately processing complex tasks. Lastly, they
can be easily influenced by misleading demonstrations. Therefore, while it’s
tempting to think that larger models would inherently yield better
performance, this isn’t always the case. Further research is needed to
understand and overcome these issues.

###  2.4 Issues with the Current Evaluation Methodology

In this section, we discuss the potential issues with the current evaluation
methods. The evaluation metrics may significantly impact our perception of the
capability of the LLMs. Schaeffer et al., (2023) present an alternative
explanation for the so-called emergent abilities: it is the choice of metric,
rather than the fundamental changes in model behaviors, that makes us feel
LLMs suddenly become powerful. Simply put, for a non-linear metric, say
$x^{n}$, sparsely sampled points on the curve make it appear that emergent
behaviors happen; however, for a linear metric, such observation will be
missing. Another issue is that the massive internet-scale training datasets
for LLMs may potentially cover the datasets used for later evaluation given
that these evaluation sets are generally sourced from the internet and highly
accessible. As now training sources are unavailable, the notion of
generalization becomes even more vague and it becomes impossible to tell if a
model really learns an underlying function, or simply retrieves it from its
memory. Such non-transparency hinders genuine and reliable evaluation.

##  3 Our View on Artificial General Intelligence

It is no doubt that LLMs can serve as a helpful assistant for humans — a more
personalized encyclopedia that is user-friendly through natural language
conversation. However, we argue that there is still a huge gap between LLMs
and artificial general intelligence (AGI). To lay the groundwork for
subsequent discussions, we first need to clarify our understanding of AGI.
There are diverging and even contradictory views on AGI which makes it
difficult to find a generally accepted definition (Chollet,, 2019; Goertzel,,
2014). In this case, we adopt a descriptive approach rather than a
prescriptive one; that is, we try to extract several characteristics of AGI
and present them in a coherent line of argument instead of giving a rule-based
definition that presupposes correctness. There are four traits that we ascribe
to AGI including

  * •

Agents can perform infinite tasks.

  * •

Agents are autonomous to generate new tasks in a given context.

  * •

Agents are propelled by a value system, serving as a fundamental architecture
for the generation of tasks.

  * •

Agents possess a world model, which represents the real world and guides their
interaction with the world.

To investigate AGI from a behaviorism viewpoint, we propose that generally
intelligent agents should be able to perform infinite tasks in the dynamic
physical and social space. Otherwise, if we set a threshold of the number of
tasks that indicates the realization of AGI, it will always be questionable
how this threshold is selected. If an agent is not generally intelligent when
it completes $N$ tasks, there is no reason to believe that it will magically
possess general intelligence once it completes $N+1$ tasks. A long checklist
of specific challenging tasks is useful in terms of assessment of agent
performance, like how teachers use students’ scores on tests to evaluate their
learning performance, but completion of specific tasks alone will not be equal
to possessing general intelligence just like students’ scores can not be used
to stand for their true learning ability. By referring to infinite tasks, our
intention is not that an agent should be omnipotent like Superman to be
capable of anything. In addition, we believe that generally intelligent agents
should be able to generate previously undefined new tasks in the specific
context, which is similar to students learning how to learn.

Autonomous agents capable of generating new tasks require two underlying
mechanisms. Firstly, agents need a driving force that motivates the task-
generation process. For example, Darwin’s theory of evolution defines two
instincts for all species to act upon the world, that is, to survive and
reproduce (Popper,, 1978), which are encoded in our genes that preserve the
human species. The later evolution of humankind has witnessed a diversity of
values, e.g., altruism, honesty, beauty, courage. Each individual is driven by
a complex value system that is shaped by his ongoing interaction with the
physical and social world. A similar idea of a value system can be
incorporated to create generally intelligent agents and served as an engine
for agents to generate appropriate new tasks based on a predefined value
system. In this case, artificial intelligence can be aligned via value
alignment instead of a predefined step-by-step instruction for tasks.
Secondly, agents need a world model that entails grounded representations of
the real-world and implicit physical laws such as the causal chain and social
norms (Ha and Schmidhuber,, 2018). It is like a LEGO play. While the world
model contains different types of bricks (the object representations) plus how
they can be connected via each other (the physical laws), the value system
selects an ideal, e.g., a castle, among all the other possibilities for agents
to build, and the process of turning bricks to a LEGO castle requires agents
to continually generate new tasks, e.g., pick up which brick to connect to an
existing node, based on the current building progress.

##  4 The Unity of Knowing and Acting

To achieve the AGI with the grouding ability and the properties that we
desire, we argue that mere knowing or acting alone is insufficient to
cultivate genuinely intelligent agents.

Following Wang Yang-Ming’s philosophy of knowledge as action (Wang,, 1963), we
argue that knowing and acting shall be deeply integrated in a learning system,
where the intelligent agent actively performs actions as a way to both form
comprehensive representation for the real-world objects it is interacting
with, such as tactic feedback, representation from different views, and even
sound, and, more crucially, to explore the surrounding environment,
crystallize the knowledge from trials and errors, and to a greater extend,
generalize the knowledge from acting to novel situations.

In the following, we discuss our view on the multi-faceted integration of
knowing and acting from the two perspectives, i.e.,

  * •

Active interaction with real-world objects provide more comprehensive signals
for establishing concept representation.

  * •

Knowledge is not well-learned with only passive input but shall be
supplemented with experience; even unknown in the beginning, actions from
repetitive trials and errors could lead to new knowledge.

###  4.1 Interaction for Concept Learning

Imagine learning a new concept as simple as a “cup” as a baby. Not everything
“shaped as a small and round container” is called a cup. They may also be
referred to as a “bowl” or a “dust bin”. Cup is neither only shaped as a small
and round container and “with a handle”. Some teacups do not have handles.
More importantly, however, cups are usually served as a tool for drinking
liquids. One may also consider using cups for transporting or holding liquids
or other items that fit the space.

As noted in the example, actions play a pivotal role in understanding a
concept. We therefore argue that the fundamentals to fully grasp a specific
concept shall include not only appearance or geometry features, but, more
critically, involve the functionality and affordance of a concept one can
interact with as well. Such a learning process shall be inherently multimodal:
during the interaction process to learn the concept of cup, we not only see
what a cup looks like when we play around with it, but we also sense the
temperature of the liquids it contains, the weight the cup exerts on our hand,
and the feeling of quenching our thirst when drinking. While the sensory
impulses are hard to fully capture using current sensors, we believe that the
widely adopted learning paradigm of learning without any interaction and with
only static bi-modal input of vision and language is far from enough for one
to understand a new concept.

Existing LLMs behave as if a large database of established concepts with
language-hashing ability and may even tell you how to use a cup for potted
plants. However, for relatively new concepts, we note that they still fall
short compared to humans and perform no more than statistically correlating
symbols in textual corpora (Jiang et al.,, 2023), lacking in understanding
multiple aspects of the concept. We argue that the absence of such interactive
behaviors and accompanying sensory input consists of part of the missing
pieces towards ideal general intelligence, without which the agent has no way
of associating perceptual observation with effects from actions, let alone the
functionality and the affordance of new concepts.

The general problem for interactive concept learning could be formulated in a
reinforcement learning framework. However, compared to existing reinforcement
learning problems, concept learning shall not be task-specific or goal-driven,
unlike achieving high scores in Atari (Mnih et al.,, 2015), navigating in an
environment (Savva et al.,, 2019), or completing a language instruction
(Shridhar et al.,, 2020). In some sense, concept learning should be more
“unsupervised”, as contrasive learning does for representation learning (Chen
et al.,, 2020). We expect the goal instantiated in interactive concept
learning to be more closely related to children’s inherent desire to explore
or similar to curiosity-driven objectives.

To facilitate agents with human-level concept understanding, we envision a
realistic meta-verse (AGI-verse, or Tong-verse), supporting far-richer
interactive behaviors with objects than existing platforms. Agents in the
meta-verse not only receive visual observation and textual explanation, but
also can take actions with objects in the environment, or play with them,
recursively apply existing concepts to new concepts, and potentially discover
novel usage of a concept that is rarely encountered in the real world.
Ideally, such actions also create sound effects and provide haptic feedback.
Off-the-shelf assets for similar interactive environments, such as Habitat
(Savva et al.,, 2019) and Behavior (Li et al.,, 2023), are still poised for
learning for specific tasks, with insufficient interactive action space and
realistic effects.

Going slightly beyond concept learning, we believe the foundation for the
success of interactive concept learning should also serve to facilitate tool
using (Zhu et al.,, 2015, 2020). With proper composition of existing concepts
and their properties, we also hope that the learning mechanism would give rise
to tool creation, a hallmark of human-level intelligence.

###  4.2 Crystallizing Knowledge from Action

Gopnik and Sobel, (2000) propose the task of Blicket detection that well
captures the essence of turning experience of trials and errors into knowledge
and how it helps generalization.

The series of experiments were initially designed for probing children’s
causal learning mechanism and were found to be strikingly similar to modern
scientific discovery. Gopnik and Sobel, (2000) introduced a special device
called a Blicket machine to children subjects. The Blicket machine has a
unique mechanism: if a Blicket is put on top of it, the machine will be
activated, flashing and making sound. During the experimentation phase, the
subjects were shown a series of experiments for compositions of objects,
demonstrating the Blicketness of some of them. And then, the children were
allowed time for exploratory play with the objects. They could freely compose
the objects and put the composition on top of the Blicket machine to better
understand Blicketness of all objects. After the process, the subjects would
be asked questions, such as which object was a Blicket and given a composition
of objects that activated or did not activate the machine, how to inactivate
or activate it.

While it is noted that older children generally perform better in this task,
we also realize from human experiments that despite that Blicketness in the
beginning is uncertain for all objects, better-performing subjects make
informed trials for quick disambiguation. The interactive trial-and-error
process significantly improves final problem-solving; with only passive
observation and no active interaction, the uncertainty will only remain.
Furthermore, for questions regarding intervention, e.g., what would happen if
an object were added or removed, subjects with intensive interaction with the
machine show notable superiority.

Xu et al., (2022) develop a computational interactive environment for the
Blicket detection problem based on the static setup from Zhang et al., 2021a .
Their newly introduced EST environment mimics the classic Blicket detection
experiment but intentionally simplifies sensorimotor control by abstracting it
out into a discrete space of object selection. Their experimental results from
computational methods show that existing learning methods, including
completely LLM-based ones, fare pathetically worse than naive and inefficient
heuristics search.

While there has been success using iterative prompting, supplying LLMs with
both actions and effects and iteratively running the process, for interactive
learning (Wang et al.,, 2023; Zhu et al.,, 2023), we note that the model is
still largely based on internet-scale knowledge colossus on a specific subject
(Fan et al.,, 2022). Yet, with the presumably low exposure of Blicket on the
internet, the LLMs become no better than random actors. The results suggest
that existing methods rely largely on existing passively provided data and are
simply unable to crystallize new knowledge from novel phenomena by active
interaction.

We believe that artificial general intelligence shall possess the ability to
quickly act to resolve ambiguity and turn the experience from successful and
unsuccessful interventions into their knowledge on how to interact with the
environment, instead of being only fed with data passively, with no capacity
to demystify novel situations through interaction and knowledge acquisition.

The reinforcement learning setup inherently supports acting and learning.
However, existing reinforcement learning problems for common sense
intelligence are more perceptual than reasoning, which requires instant
response rather than complex System-2 computation on-the-fly. We hope that a
challenging reasoning problem based on interaction with the environment will
emerge, serving as a testbed for evaluating how the community performs on
turning experience of trials and errors into knowledge and further use the
knowledge to perform additional everyday tasks. Besides, it is also unresolved
how to abstract from existing knowledge and apply them as general principles
to novel situations. Knowledge abstraction, knowledge accumulation, and
knowledge application should be the critical processes in realizing such
systems. We believe that a realistic meta-verse mentioned above shall also
serve as an important factor in building the living environment for an agent
to play and learn.

##  5 Discussion and Future Directions

In this work, we review existing failure cases for Large Language Models
(LLMs) and refute the reckless claim that LLMs represent “sparks of artificial
general intelligence” (Bubeck et al.,, 2023). Analysis from both careful
benchmarking and empirical observation suggests that LLMs may be a good
database that hashes language queries, but far from the general intelligence
demonstrated in humans. Besides, deficiency in evaluation also casts doubt on
the validity of the results on existing web-sourced datasets, as the largest
of the LLMs may have already been trained on them.

We further present our view on artificial general intelligence and propose the
unity of knowing and acting, a factor critical for living agents yet
paradoxically missing in the acclaimed intelligent LLMs. In our view, the
unity of knowing and acting could serve at least to help concept learning and
knowledge acquisition.

Following the discussion, we point out three future directions for advances in
artificial general intelligence research.

###  5.1 Transparent Evaluation

As the dataset size becomes increasingly bigger, the critical issue of
generalization is gradually ignored; so long as the model performs “well” on
the test set, it is considered good. However, the good performance may stem
from training on the testing data, and in fact the model does not really
understand the problem. Close-sourced models like GPT (OpenAI,, 2023) further
cloud interpretation for evaluation. As evaluation datasets are usually
sourced from the internet and the LLMs are trained on internet data, we argue
that a new way of evaluation is in desperate need that can make sure limited
data leakage from the internet to warrant true generalization.

###  5.2 Affordance-rich Interactive Environments

As mentioned in Sec. 4, a founding component for knowing and acting is a meta-
verse. The meta-verse should ideally support rich affordance that allows an
agent to play with objects for concept learning, providing feedback of multi-
modality, including more than just vision and language. The meta-verse should
also support a variety of reasoning tasks, covering tasks regarding knowledge
acquisition and abstraction, such as instruction following, planning,
abduction, and induction. Of particular interest should be tasks without
extensive internet data exposure, in order to solidify the argument that the
agent learns from interaction with the environment rather than retrieving from
given knowledge.

###  5.3 Unifying Knowing and Acting

We argue that a cognitive architecture shall be developed to integrate knowing
and acting. Despite of success from reinforcement learning in a narrow domain
of tasks, a general mechanism for knowledge as action should transcend pure
data-driven approaches for generalization in knowledge abstraction, knowledge
accumulation, and knowledge application. Besides, it still remains an open
problem on how to formulate existing knowledge and incorporate off-the-shelf
knowledge into new knowledge discovery. We hope that a cognitive mechanism can
be scalable enough and seamlessly combine knowledge-driven and data-driven
benefits.

In the end, while we acknowledge the great practical advances LLMs have
brought to the community, we strongly believe that they do not represent
artificial general intelligence and hope that this article serves as
inspiration for the research community towards the ultimate goal.

## References

  * Arora et al., (2023) Arora, D., Singh, H. G., et al. (2023).  Have llms advanced enough? a challenging problem solving benchmark for large language models.  arXiv preprint arXiv:2305.15074.
  * Bender et al., (2021) Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021).  On the dangers of stochastic parrots: Can language models be too big?  In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623.
  * Binz and Schulz, (2023) Binz, M. and Schulz, E. (2023).  Using cognitive psychology to understand gpt-3.  Proceedings of the National Academy of Sciences, 120(6):e2218523120.
  * Brown et al., (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020).  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901.
  * Bubeck et al., (2023) Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023).  Sparks of artificial general intelligence: Early experiments with gpt-4.  arXiv preprint arXiv:2303.12712.
  * Chen et al., (2020) Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020).  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR.
  * Chen et al., (2022) Chen, W., Ma, X., Wang, X., and Cohen, W. W. (2022).  Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.  arXiv preprint arXiv:2211.12588.
  * Chollet, (2019) Chollet, F. (2019).  On the measure of intelligence.  arXiv preprint arXiv:1911.01547.
  * Chomsky, (2009) Chomsky, N. (2009).  Syntactic structures.  In Syntactic Structures. De Gruyter Mouton.
  * Chowdhery et al., (2022) Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022).  Palm: Scaling language modeling with pathways.  arXiv preprint arXiv:2204.02311.
  * Daniel, (2013) Daniel, K. (2013).  Thinking, fast and slow.  Farrar, Straus and Giroux.
  * Dziri et al., (2023) Dziri, N., Lu, X., Sclar, M., Li, X. L., Jian, L., Lin, B. Y., West, P., Bhagavatula, C., Bras, R. L., Hwang, J. D., et al. (2023).  Faith and fate: Limits of transformers on compositionality.  arXiv preprint arXiv:2305.18654.
  * Fan et al., (2022) Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A. (2022).  Minedojo: Building open-ended embodied agents with internet-scale knowledge.  arXiv preprint arXiv:2206.08853.
  * Gao et al., (2022) Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. (2022).  Pal: Program-aided language models.  arXiv preprint arXiv:2211.10435.
  * Gendron et al., (2023) Gendron, G., Bao, Q., Witbrock, M., and Dobbie, G. (2023).  Large language models are not abstract reasoners.  arXiv preprint arXiv:2305.19555.
  * Glenberg, (2010) Glenberg, A. M. (2010).  Embodiment as a unifying perspective for psychology.  Wiley interdisciplinary reviews: Cognitive science, 1(4):586–596.
  * Goertzel, (2014) Goertzel, B. (2014).  Artificial general intelligence: concept, state of the art, and future prospects.  Journal of Artificial General Intelligence, 5(1):1.
  * Gopnik and Sobel, (2000) Gopnik, A. and Sobel, D. M. (2000).  Detecting blickets: How young children use information about novel causal powers in categorization and induction.  Child development, 71(5):1205–1222.
  * Ha and Schmidhuber, (2018) Ha, D. and Schmidhuber, J. (2018).  World models.  arXiv preprint arXiv:1803.10122.
  * Harnad, (1990) Harnad, S. (1990).  The symbol grounding problem.  Physica D: Nonlinear Phenomena, 42(1-3):335–346.
  * Hendrycks et al., (2021) Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. (2021).  Measuring mathematical problem solving with the math dataset.  arXiv preprint arXiv:2103.03874.
  * Jiang et al., (2023) Jiang, G., Xu, M., Xin, S., Liang, W., Peng, Y., Zhang, C., and Zhu, Y. (2023).  Mewl: Few-shot multimodal word learning with referential uncertainty.  arXiv preprint arXiv:2306.00503.
  * Jiang et al., (2022) Jiang, G., Xu, M., Zhu, S.-C., Han, W., Zhang, C., and Zhu, Y. (2022).  Mpi: Evaluating and inducing personality in pre-trained language models.  arXiv preprint arXiv:2206.07550.
  * Jin et al., (2023) Jin, Z., Liu, J., Lyu, Z., Poff, S., Sachan, M., Mihalcea, R., Diab, M., and Schölkopf, B. (2023).  Can large language models infer causation from correlation?  arXiv preprint arXiv:2306.05836.
  * Kosinski, (2023) Kosinski, M. (2023).  Theory of mind may have spontaneously emerged in large language models.  arXiv preprint arXiv:2302.02083.
  * Lakoff et al., (1999) Lakoff, G., Johnson, M., and Sowa, J. F. (1999).  Review of philosophy in the flesh: The embodied mind and its challenge to western thought.  Computational Linguistics, 25(4):631–634.
  * Levinovitz, (2017) Levinovitz, A. (2017).  Slaying the chinese jabberwock: Toward a comparative philosophy of nonsense.  Comparative Literature, 69(3):251–270.
  * Li et al., (2023) Li, C., Zhang, R., Wong, J., Gokmen, C., Srivastava, S., Martín-Martín, R., Wang, C., Levine, G., Lingelbach, M., Sun, J., et al. (2023).  Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation.  In Conference on Robot Learning, pages 80–93. PMLR.
  * Ma et al., (2023) Ma, X., Gao, L., and Xu, Q. (2023).  Tomchallenges: A principle-guided dataset and diverse evaluation tasks for exploring theory of mind.  arXiv preprint arXiv:2305.15068.
  * Mahowald et al., (2023) Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., and Fedorenko, E. (2023).  Dissociating language and thought in large language models: a cognitive perspective.  arXiv preprint arXiv:2301.06627.
  * McKenzie et al., (2023) McKenzie, I. R., Lyzhov, A., Pieler, M., Parrish, A., Mueller, A., Prabhu, A., McLean, E., Kirtland, A., Ross, A., Liu, A., et al. (2023).  Inverse scaling: When bigger isn’t better.  arXiv preprint arXiv:2306.09479.
  * Mitchell and Krakauer, (2023) Mitchell, M. and Krakauer, D. C. (2023).  The debate over understanding in ai’s large language models.  Proceedings of the National Academy of Sciences, 120(13):e2215907120.
  * Mnih et al., (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015).  Human-level control through deep reinforcement learning.  nature, 518(7540):529–533.
  * Naeini et al., (2023) Naeini, S., Saqur, R., Saeidi, M., Giorgi, J., and Taati, B. (2023).  Large language models are fixated by red herrings: Exploring creative problem solving and einstellung effect using the only connect wall dataset.  arXiv preprint arXiv:2306.11167.
  * OpenAI, (2023) OpenAI, R. (2023).  Gpt-4 technical report.  arXiv.
  * Popper, (1978) Popper, K. (1978).  Natural selection and the emergence of mind.  Dialectica, pages 339–355.
  * Putnam et al., (1981) Putnam, H. et al. (1981).  Reason, truth and history, volume 3.  Cambridge University Press.
  * Savva et al., (2019) Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al. (2019).  Habitat: A platform for embodied ai research.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 9339–9347.
  * Schaeffer et al., (2023) Schaeffer, R., Miranda, B., and Koyejo, S. (2023).  Are emergent abilities of large language models a mirage?  arXiv preprint arXiv:2304.15004.
  * Shapira et al., (2023) Shapira, N., Levy, M., Alavi, S. H., Zhou, X., Choi, Y., Goldberg, Y., Sap, M., and Shwartz, V. (2023).  Clever hans or neural theory of mind? stress testing social reasoning in large language models.  arXiv preprint arXiv:2305.14763.
  * Shiffrin and Mitchell, (2023) Shiffrin, R. and Mitchell, M. (2023).  Probing the psychology of ai models.  Proceedings of the National Academy of Sciences, 120(10):e2300963120.
  * Shridhar et al., (2020) Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., and Fox, D. (2020).  Alfred: A benchmark for interpreting grounded instructions for everyday tasks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10740–10749.
  * Smith and Gasser, (2005) Smith, L. and Gasser, M. (2005).  The development of embodied cognition: Six lessons from babies.  Artificial life, 11(1-2):13–29.
  * Srivastava et al., (2022) Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022).  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  arXiv preprint arXiv:2206.04615.
  * (45) Tang, R., Kong, D., Huang, L., and Xue, H. (2023a).  Large language models can be lazy learners: Analyze shortcuts in in-context learning.  arXiv preprint arXiv:2305.17256.
  * (46) Tang, X., Zheng, Z., Li, J., Meng, F., Zhu, S.-C., Liang, Y., and Zhang, M. (2023b).  Large language models are in-context semantic reasoners rather than symbolic reasoners.  arXiv preprint arXiv:2305.14825.
  * Uesato et al., (2022) Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., and Higgins, I. (2022).  Solving math word problems with process-and outcome-based feedback.  arXiv preprint arXiv:2211.14275.
  * Ullman, (2023) Ullman, T. (2023).  Large language models fail on trivial alterations to theory-of-mind tasks.  arXiv preprint arXiv:2302.08399.
  * Visone, (2010) Visone, J. D. (2010).  Science or reading: What is being measured by standardized tests?  American Secondary Education, pages 95–112.
  * Wang et al., (2023) Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. (2023).  Voyager: An open-ended embodied agent with large language models.  arXiv preprint arXiv:2305.16291.
  * Wang, (1963) Wang, Y. (1963).  Instructions for Practical Living, and Other Neo-Confucian Writing.  Columbia University Press.
  * Wei et al., (2022) Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. (2022).  Chain of thought prompting elicits reasoning in large language models.  arXiv preprint arXiv:2201.11903.
  * Wu et al., (2023) Wu, Y., Jia, F., Zhang, S., Wu, Q., Li, H., Zhu, E., Wang, Y., Lee, Y. T., Peng, R., and Wang, C. (2023).  An empirical study on challenging math problem solving with gpt-4.  arXiv preprint arXiv:2306.01337.
  * Xu et al., (2023) Xu, F., Lin, Q., Han, J., Zhao, T., Liu, J., and Cambria, E. (2023).  Are large language models really good logical reasoners? a comprehensive evaluation from deductive, inductive and abductive views.  arXiv preprint arXiv:2306.09841.
  * Xu et al., (2022) Xu, M., Jiang, G., Zhang, C., Zhu, S.-C., and Zhu, Y. (2022).  Est: Evaluating scientific thinking in artificial agents.  arXiv preprint arXiv:2206.09203.
  * Xuan-Quy et al., (2023) Xuan-Quy, D., Ngoc-Bich, L., The-Duy, V., Xuan-Dung, P., Bac-Bien, N., Van-Tien, N., Thi-My-Thanh, N., and Hong-Phuoc, N. (2023).  Vnhsge: Vietnamese high school graduation examination dataset for large language models.  arXiv preprint arXiv:2305.12199.
  * Yiu et al., (2023) Yiu, E., Kosoy, E., and Gopnik, A. (2023).  Imitation versus innovation: What children can do that large language and language-and-vision models cannot (yet)?  arXiv preprint arXiv:2305.07666.
  * Zhang et al., (2019) Zhang, C., Gao, F., Jia, B., Zhu, Y., and Zhu, S.-C. (2019).  Raven: A dataset for relational and analogical visual reasoning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5317–5327.
  * (59) Zhang, C., Jia, B., Edmonds, M., Zhu, S.-C., and Zhu, Y. (2021a).  Acre: Abstract causal reasoning beyond covariation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10643–10653.
  * (60) Zhang, C., Raghu, M., Kleinberg, J., and Bengio, S. (2021b).  Pointer value retrieval: A new benchmark for understanding the limits of neural network generalization.  arXiv preprint arXiv:2107.12580.
  * Zhang et al., (2023) Zhang, X., Li, C., Zong, Y., Ying, Z., He, L., and Qiu, X. (2023).  Evaluating the performance of large language models on gaokao benchmark.  arXiv preprint arXiv:2305.12474.
  * Zhong et al., (2023) Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., and Duan, N. (2023).  Agieval: A human-centric benchmark for evaluating foundation models.  arXiv preprint arXiv:2304.06364.
  * Zhu et al., (2023) Zhu, X., Chen, Y., Tian, H., Tao, C., Su, W., Yang, C., Huang, G., Li, B., Lu, L., Wang, X., et al. (2023).  Ghost in the minecraft: Generally capable agents for open-world enviroments via large language models with text-based knowledge and memory.  arXiv preprint arXiv:2305.17144.
  * Zhu et al., (2020) Zhu, Y., Gao, T., Fan, L., Huang, S., Edmonds, M., Liu, H., Gao, F., Zhang, C., Qi, S., Wu, Y. N., et al. (2020).  Dark, beyond deep: A paradigm shift to cognitive ai with humanlike common sense.  Engineering, 6(3):310–345.
  * Zhu et al., (2015) Zhu, Y., Zhao, Y., and Chun Zhu, S. (2015).  Understanding tools: Task-oriented object modeling, learning and recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2855–2864.

  *[LLMs]: Large Language Model
  *[BiV]: Brain in a Vat
  *[AGI]: artificial general intelligence
  *[LLM]: Large Language Model


# Discovery of Young Methane Dwarfs in the Rho Ophiuchi L 1688 Dark Cloud

Poshih Chiang and W. P. Chen  Graduate Institute of Astronomy, National
Central University, 300 Jhongda Road, Zhongli 32001, Taiwan

###### Abstract

We report the discovery of two methane dwarfs in the dark cloud L 1688 of the
$\rho$ Oph star-forming region. The two objects were among the T dwarf
candidates with possible methane absorption and cool atmospheres, as diagnosed
by infrared colors using deep WIRCam/CFHT HK plus CH4ON images, and
IRAC/Spitzer c2d data. Follow-up spectroscopic observations with the
FLAMINGOS-2/Gemini South confirmed the methane absorption at 1.6 µm. Compared
with spectral templates of known T dwarfs in the field, i.e., of the old
populations, Oph J162738$-$245240 (Oph-T3) is a T0/T1 type, whereas Oph
J162645$-$241949 (Oph-T17) is consistent with a T3/T4 type in the H band but
an L8/T1 in the K band. Compared with the BT-Settl model, both Oph-T3 and
Oph-T17 are consistent with being cool, $\sim 1000$ K and $\sim 900$ K,
respectively, and of low surface gravity, $\log(g)=3.5$. With an age no more
than a couple Myr, these two methane dwarfs thereby represent the youngest T
dwarfs ever confirmed. A young late L dwarf, Oph J162651$-$242110, was found
serendipitously in our spectroscopic observations.

brown dwarfs; stars: formation; infrared: stars; stars: individual (Oph-L,
Oph-T3, Oph-T17)

##  1 Introduction

Thousands of brown dwarfs have been discovered in the field by large surveys
such as SDSS (Knapp et al., 2004; Chiu et al., 2006; Scholz et al., 2009),
2MASS/DENIS (Kirkpatrick et al., 2000; Tinney et al., 2005; Reid et al., 2008;
Martín et al., 2010), UKIDSS (Burningham et al., 2013), and WISE (Kirkpatrick
et al., 2011, 2012). Recent discoveries of Y dwarfs push the mass of brown
dwarfs down to some 10 Jupiter masses (Kirkpatrick et al., 2012). Furthermore,
a few isolated, or possible members of young moving groups (100–200 Myr),
planetary mass objects, which have masses lower than the deuterium-burning
limit, were recently identified with different spectral types, ages, and
environments (Delorme et al., 2012; Liu et al., 2013; Luhman, 2014; Naud et
al., 2014; Gagné et al., 2015). The current paradigm is that stars are formed
out of dense molecular cloud cores, and planets are condensed in the
protostellar disks. The origin of substellar objects, however, remains
unsettled.

To shed light on the issue, identification of substellar objects in a star-
forming region, i.e., at an epoch when these objects are being formed, is a
crucial first step. Most brown dwarfs found in nearby star-forming region so
far are of late-M or L types, e.g. OTS 44 in Chamaeleon (Oasa et al., 1999;
Luhman et al., 2004), 2M0437$+$2331 in Taurus (Bowler et al., 2014), and
CFHTWIR$-$Oph33 in $\rho$ Oph (Alves de Oliveira et al., 2012). Toward later
types, it has been a long battle to recognize the youngest T dwarfs to even
the closest star-forming regions, though some candidates have been reported,
e.g., in IC 348 (Burgess et al., 2009), and in Serpens (Spezzi et al., 2012),
both of 1–3 Myr. Field brown dwarfs appear to show the transition from
spectral type L (cloudy, red, and dusty) to T (cloudless, blue, and methane-
bearing) around $\sim$1400K (Kirkpatrick, 2005). In contrast, young exoplanets
and free-floating planetary-mass objects are expected to have the spectral
transition at lower temperatures, possibly as the result of different grain
physics in cool atmospheres (Bowler et al., 2011; Barman et al., 2011; Liu et
al., 2013). To date, three controversial cases of very young T dwarfs were
claimed, respectively, with two in $\sigma$ Ori, S Ori 70 (Zapatero Osorio et
al., 2002) and S Ori 73 (Bihain et al., 2009), and one in $\rho$ Oph, No. 4450
(Marsh et al., 2010). However, recent proper motion data indicate S Ori 73
likely to be a field T dwarf, while the nature of S Ori 70 remains elusive
(Peña Ramírez et al., 2011). On the other hand, No. 4450 does not show
prominent methane features. A sample of confirmed methane dwarfs of a few
million years old, therefore, will provide constraints to the theoretical
modeling of the evolving cool atmospheres. Here we report the confirmation by
infrared spectroscopy of two T dwarfs in the L 1688 cloud of the Rho Ophiuchi
complex, Oph-T3 (Oph J162738$-$245240, RA=$16^{\rm h}27^{\rm m}38\fs 21$,
Decl=$-24\arcdeg 52\arcmin 39\farcs 9$, J2000, $H=18.38\pm 0.06$, $H-K=0.28\pm
0.09$) and Oph-T17 (Oph J162645$-$241949, RA=$16^{\rm h}26^{\rm m}45\fs 23$,
Decl=$-24\arcdeg 19\arcmin 49\farcs 2$, J2000, $H=19.16\pm 0.15$, $H-K=0.49\pm
0.17$), following the identification numbering in Table 3 of Chiang et al.
(2015).

##  2 Observations and Data Reduction

###  2.1 Photometric observations

The T dwarfs candidates in L 1688, the densest cloud in Rho Ophiuchi, were
identified by cool, methane-bearing atmospheres that characterize T dwarfs
(Chiang et al., 2015). The methane absorption is diagnosed by our deep methane
imaging survey carried out in 2010 using the CH4ON filter, centering at 1.69
µm, with the Wide-field InfraRed Camera (WIRCam) attached to the 3.6-m Canada-
Frace-Hawaii Telescope (CFHT). Together with the archival WIRCam/CFHT H-band
images, the color $H-CH4ON$ serves to trace possible methane absorption near
1.6 µm. In addition, the color $[3.6]-[4.5]$, available through the c2d
catalog (Evans et al., 2007), is used to detect another methane feature at 3.4
µm(Patten et al., 2006; Leggett et al., 2010; Mace et al., 2013).

For cool temperatures, both $H-[4.5]$ and $K-[4.5]$ colors are utilized
(Patten et al., 2006; Leggett et al., 2010). With these selection criteria,
empirically adjusted with known M, L, and T dwarfs in the field, Chiang et al.
(2015) found a total of 28 T dwarf candidates toward L 1688 that show evidence
of both methane absorption and cool atmospheres. Notwithstanding about half of
these 28 candidates may be contaminations, likely by active galaxies or young
stellar variability (Chiang et al., 2015), the list provides a conservative,
yet relatively reliable candidate sample of methane dwarfs for spectroscopic
confirmation. Figure 1 shows the distribution of these candidates, including
the validated T and L dwarfs presented here, in the diagnostic color-color and
color-magnitude diagrams.

Figure 1: (Top) The color-color diagrams of (left) the methane indices, and
(right) of the temperature indices. Gray dots represented all the sources in
our data of L 1688 in Rho Ophiuchi. The region within which T dwarfs are
identified (the “T zone”) in each diagram is enclosed by solid lines. Stars
located in both T zones are considered T dwarf candidates (red open circles),
including the two confirmed T dwarfs (blue filled circles) and one L dwarf
(blue filled triangle) reported here. Asterisks represent the
spectroscopically confirmed late-M and L dwarfs in $\rho$ Oph (Geers et al.,
2011; Alves de Oliveira et al., 2010; Mužić et al., 2012; Alves de Oliveira et
al., 2012). Also marked are the suspected T dwarf candidates in other star-
forming regions, such as S Ori 70 (Zapatero Osorio et al., 2002) (filled
square) and No. 4450 (Marsh et al., 2010) (open square). No. 4450 has no
detected methane feature, so it is not included in the methane diagram. The T
dwarf candidates in IC 348 (green open circle, Burgess et al., 2009) and in
Serpens (orange open circles, Spezzi et al., 2012) are also marked. Note that
these would not have selected by our criteria. (Bottom) The color-magnitude
diagrams, with the same symbols, overlaid with the 1 Myr isochrones by the
COND (solid line) (Baraffe et al., 2003), DUSTY (dashed line) (Chabrier &
Baraffe, 2000), and BT-settl (dot-dash line) (Allard et al., 2012) models. The
magnitudes of S Ori 70 are scaled to 130 pc from its assumed distance 352 pc
of the $\sigma$ Ori cluster.

###  2.2 Spectroscopic Observations

Six T dwarf candidates were observed with FLAMINGOS-2 on the Gemini-South
Telescope in 2014, using the HK grism with a 6-pixel wide (1.08″) slit for
low-resolution ($R\sim 1200$) spectra. Four of the targets turned out to be
false positives, including one galaxy with emission lines (source No. 1 in
Chiang et al., 2015, R.A.=16:28:10.04, Decl.=$-24$:49:12.2 (J2000)), one
background star (No. 2, R.A.=16:28:14.19, Decl.=$-24$:50:53.5, J2000), and two
low-temperature objects (No. 7, R.A.=16:27:12.80, Decl.=$-24$:49:55.2, J2000,
and No. 19, R.A.=16:28:40.09, Decl.=$-24$:00:17.1, J2000) showing possible
water absorption but no obvious methane absorption. In this paper, we present
the results of the two confirmed methane dwarfs among the 28 candidates listed
by Chiang et al. (2015), and the serendipitous discovery of one L dwarf. It
took six and seven dithering points, each with a detector integration time of
300 s, in an “ABBA” pattern to complete the observations, rendering a total
exposure time of 1,800 seconds for Oph-T3 and 2,100 seconds for Oph-T17. An
argon lamp was used for wavelength calibration. The standard star HIP 82271
(B9 V) was observed to remove telluric absorptions.

Raw data were processed with the standard F2/Gemini package under IRAF,
including flat-field correction, cutting and co-adding images, wavelength
calibration, telluric correction, and extraction of spectra. With the long
exposures, sky emissions at 1.5, 1.58, 1.68 and 1.77µm were too bright to be
properly removed so were discarded manually in subsequent analysis. Oph-T17
was observed before dawn with the airmass increasing from 1.7 to 2.7, hence
with an elevated thermal sky background. To minimize the effect of background
changes, we scaled the images of each dither frame to the first frame. One
additional bright star and one faint star, respectively, happened to be
detected in the 2-D dispersed images of Oph-T3 and Oph-T17. Figure 2 shows the
reduced and coadded 2D dispersed images, together with the extracted 1D
spectra of Oph-T3 and Oph-T17. The bright star serves as a spectroscopic
comparison for Oph-T3. The faint star coincidentally located in the slit of
the Oph-T17 images, Oph J162651$-$242110, turned out to be a late L dwarf
previously unrecognized.

Figure 2: (Top) The observations of Oph-T3, showing (upper-left) the WIRCam
CH4ON image, with the target Oph-T3 around $(0,30)$, a bright star around
$(0,45)$, and the vertical solid line representing the position of the slit,
(upper-right) the FLAMINGOS-2 2D dispersed image, with the thin and thick
black stripes as Oph-T3 and the bright comparison star, (lower-left) the
reduced H-band spectra for Oph-T3 (cyan) and the comparison (grey), each
smoothed and represented by a thicker line, (lower-right) the reduced K-band
spectra of the two stars. The red dashed lines in the H band mark the
prominent sky emissions. (Bottom) The same as in the top but for Oph-T17.
Oph-T17 is around $(-30,30)$ and its comparison around $(55,-50)$ in the CH4ON
image. In the dispersed image, the spectra of Oph-T17, the emission nebula,
and Oph J162651$-$242110 are seen from right to left.

##  3 Spectral Typing

We made two approaches for spectral classification of our candidates: (1) by
comparison of the observed spectra with those of field brown dwarfs, and with
atmospheric models, and (2) by spectral indices relevant to individual
molecular absorption features.

Figure 3 shows the spectra of Oph-T3, Oph-T17 and Oph J162651$-$242110, all
smoothed to facilitate visual comparison. Superimposed in the figure are the
template spectra of field L and T dwarfs taken from the SpeX Prism Spectral
Libraries
(http://pono.ucsd.edu/~adam/browndwarfs/spexprism/publications.html), and the
BT-Settl model spectra. In each case, the spectral running from the H to K
band is normalized at 1.58 µm.

The template and model spectra have been all reddened using the empirical
infrared reddening law prescribed by Indebetouw et al. (2005), namely
$\log(A_{\lambda}/A_{K})=0.61-2.22\log\lambda+1.21(\log\lambda)^{2}$. The
amounts of extinction of our candidates, however, are highly uncertain. Chiang
et al. (2015) estimated the extinction to each of their candidates on the
basis of near-infrared star count. Oph-T17 ($A_{K}\sim 1.8$) appears to suffer
more extinction than Oph-T3 does ($A_{K}\sim=0.6$). For Oph-T17, this must be
an overestimate because the object is visible, albeit fainter, at shorter
wavelengths. On the other hand, dereddening the observed color and magnitude
along the interstellar reddening back to the model isochrone results in
$A_{K}\sim 0.3$ mag, but this estimate is susceptible to the uncertain
validity of the adopted model isochrone, which is itself something we want to
check. In Figure 3, we adopted a judicious value $A_{K}\sim 1.2$ mag for
Oph-T17 to illustrate the effect of extinction on the emission continuum
between H and K bands. The extinction has little influence on the methane line
diagnosis covering a relatively narrow wavelength range. Likewise, $A_{K}\sim
0.8$ mag for Oph-T3 and $A_{K}\sim 0.3$ mag for the L dwarf Oph
J162651$-$242110 are used in the plots, again for illustration, in each case
by a consistent H to K band continuum and tracing roughly to the BT-Settl
isochrone. A $\log(g)=3.5$ is used in the atmospheric models to compare to the
observations.

Judging by the overall shape of the spectra, Oph-T3 should be a T0/T1 type,
with the possibility of a late L. For Oph-T17, the spectrum in the H band is
best represented by a T3/T4 type, but is more consistent in the K band with an
earlier type, L8/T1. Note that the H-band spectrum resembles that of HR 8799e
(Oppenheimer et al., 2013). The spectrum of Oph J162651$-$242110 in the H band
exhibits possible water absorptions and follows reasonably well the L0 object
2M0437 in Taurus and the low-gravity exoplanet 2M1207b (Patience et al.,
2010)), and in the K band is consistent with BT-Settl models of 1100 K and low
gravity, later than an L0.

The discrepancy in spectral typing in the H- and K-band for Oph-T17 is not
surprising, given the spectral templates being from the old, field
populations. We compared the BT-Settl models of temperatures from 800 to 1100
K with various surface gravity values. The general trend is an enhanced
suppression of the methane feature beyond 2 µm with a lowering temperature,
i.e., with a lower peak in the 2.10–2.15 µm range, relative to that near 1.6
µm. Hotter than about 1000 K, the methane absorptions disappear all together.
A lower surface gravity tends to weaken the methane features. There is also
the possibility of composite spectra due to binarity (Marocco et al., 2015).
Both Oph-T3 and Oph-T17 are consistent with being cool, $\sim 1,000$ K, and
$\sim 900$ K, respectively, and low surface gravity ($\log(g)=3.5$ bodies. The
lack of an appreciable absorption longward to $\sim 2.3$ µm is perhaps due to
an increased collision-induced H2 absorption with higher gravity (Burgasser,
Burrow, & Kirkpatrick, 2006).

Alternatively, brown dwarf spectra can be classified by spectral index, i.e.,
the ratio of the flux across a molecular absorption (e.g., by water or
methane) to that of the continuum, as demonstrated by the sets of spectral
indices published by McLean et al. (2003); Allers & Liu (2013) and Mace et al.
(2013). Because our H band data are strongly affected by sky emissions, we
modified the wavelength ranges of each spectral index as follows:
$CH_{4}(A)=F_{1.730}/F_{1.570}=F(1.728,1.732)/F(1.593,1.597)$ (McLean et al.,
2003), $(H_{2}O-H)=F_{1.470}/F_{1.600}=F(1.450,1.490)/F(1.590,1.610)$, and
$(CH_{4}-H)=F_{1.648}/F_{1.600}=F(1.635,1.660)/F(1.590,1.610)$ (Mace et al.,
2013), where $F_{\lambda_{0}}=F(\lambda 1,\lambda 2)$ stands for the
integrated flux from wavelengths $\lambda 1$ to $\lambda 2$, at central
wavelength $\lambda_{0}$, all in microns. The Hcon index is defined by the
contrast between the “line” and two conninuum wavelengths (Allers & Liu,
2013). We adopted $\lambda_{\rm line}=1.60$, the first continuum
$\lambda_{1}=1.46$, the second continuum $\lambda_{2}=1.69$, and the modified
Hcon index becomes $\left[(\lambda_{\rm
line}-\lambda_{1})\,F(1.68,1.70)/(\lambda_{2}-\lambda_{1})+(\lambda_{2}-\lambda_{\rm
line})\,F(1.45,1.47)/(\lambda_{2}-\lambda_{1})\right]/F(1.59,1.61)$.

The spectral types of Oph-T3 and Oph-T17 can be estimated by the modified
spectral indices computed for known T dwarfs taken from the SpeX Libraries, as
illustrated in Figure 4. Each spectral index is computed for a particular
molecular feature with respect to the local continuum, so extinction has
little effect. One sees that Oph-T3 is again consistent with a T0 type in all
indices, whereas Oph-T17 shows a range of T0 to T4 types in different indices.
In comparison to the BT-Settl models, a higher temperature corresponds to a
higher gravity within 800–1600 K. Both our targets are consistent with being
cool and of low $\log(g)$, and Oph-T17 is cooler than Oph-T3. For Oph
J162651$-$242110, the [H2O] index suggests stronger water absorption than
2M0437$+$2331, a young L0 in Taurus, and the methane indices suggest no
methane absorptions. We therefore conclude that Oph J162651$-$242110 is a
young late L-type dwarf. The spectral types thus determined are listed in
Table 1.

Both T dwarfs reported here are too faint in our J images. Their (H-K) colors,
0.28 mag for Oph-T3 and 0.49 mag for Oph-T17, are comparable to intermediate-
age T dwarfs or planet-mass objects, e.g., 0.30 mag for GU Psc b (50–120 Myr
for the AB Doradus moving group, Naud et al., 2014), 0.54 mag for CFBDSIR
2149$-$0403 (also for the AB Dor moving group, Delorme et al., 2012), and 0.60
mag for 2MASS J01033563-5515561(AB)b (20–50 Myr for the Tucana-Horologium
moving group, Delorme et al., 2013). Comparison of Oph-T3 and Oph-T17 does
suggest a spectral sequence; in addition to the broad methane absorption
beyond 1.6 µm, both objects share similar spectral characteristics, such as
the absorption between 1.52 to 1.56 µm. Furthermore, Oph-T17 shows a more
prominent methane absorption, and has a spectral running steeper than that of
Oph-T3. We therefore conclude that OphT-17, being fainter, is cooler, i.e., of
a later spectral type, than Oph-T3.

Our spectroscopic confirmation substantiates the photometric selection
criteria presented by Chiang et al. (2015). In the two-color diagrams of
Figure 1, Oph J162651$-$242110 is located around the locus of young low-mass
objects discovered in L 1688. In the color-magnitude diagrams, the three
objects follow approximately the BT-settl isochrones in a spectral sequence.
Including the L4 dwarf discovered by Alves de Oliveira et al. (2012) and the
two T dwarfs reported here, the L/T transition should happen between K=16 to
18 mag, as the BT-settl model suggests.

Table 1: Properties of Brown Dwarfs in L 1688 in $\rho$ Oph

Measurments | Unit | Oph-T3 | Oph-T17 | Oph J162651$-$242110
---|---|---|---|---
R.A. | hh:mm:ss.ss (J2000) | 16:27:38 | 16:26:45 | 16:26:51
Dec. | dd:mm:ss.ss (J2000) |  $-$24:52:40 |  $-$24:19:49 |  $-$24:21:10
H | mag | 18.38 | 19.16 | 18.69
CH4ON | mag | 18.51 | 19.57 | 18.68
Ks | mag | 18.10 | 18.66 | 17.50
$[3.6]$ | mag | 16.96 | 17.04 | 16.68
$[4.5]$ | mag | 16.62 | 16.53 | 16.31
(H2O$-$H) | $F_{1.470}/F_{1.600}$ | 0.57 (T0) | 0.49 (T1) | 0.34
(CH${}_{4}-$H) | $F_{1.648}/F_{1.600}$ | 0.95 (T0) | 0.70 (T3/T4) | 1.28
Hcon | (see text) | 0.89 (T0) | 0.60 (T3/T4) | 1.08
H2O (B) | $F_{1.456}/F_{1.570}$ | 0.49 (T0) | 0.54 (T0) | 0.23
CH4 (A) | $F_{1.730}/F_{1.595}$ | 0.86 (T0) | 0.36 (T4/5) | 1.44
CH4 (B) | $F_{2.200}/F_{2.100}$ | 0.98 (T0) | 0.91 (T0) | 0.93

Note. — The wavelengths (µm) of each modified spectral index is shown as the
subscript in the second column.

Figure 3: The spectra of brown dwarfs in L 1688 of the Rho Oph region. In
every case, the top plot compares the spectrum of our target with other
observations, whereas the bottom plot compares with BT-Settl models. (Top) The
spectra of the young L dwarf (black), the L0 object 2M0437 (pink) in Taurus
star-forming region (provided by Alves de Oliveria), the low-gravity exoplanet
2M1207b (orange) (Patience et al., 2010)), and the BT-Settl models of 1100 K
(red), 1200 K (grey), and 1300 K (green). (Middle) The spectra of Oph-T3
(black), field L9 (orange) and T1 (pink) templates (red), and the BT-Settl
models for 900 K (red), 1000 K (grey), and 1100 K (green), all for $\log
g=3.5$. The blue dot-connected line in the H band shows the data of HR 8799e
(Oppenheimer et al., 2013). (Bottom) The spectra of Oph-T17 (black), field T3
(orange) and T4 (pink) templates, and BT-Settl models for 800 K (red), 900 K
(grey), and 1050 K (green), all for $\log g=3.5$. The blue dot-connected line
in the H band again shows HR 8799e (Oppenheimer et al., 2013).  Figure 4:
Spectral indices of (Left) field brown dwarfs (grey dots), and (Right) BT-
Settl models with $\log(g)\geq 5.0$ (blue), and with $\log(g)\leq 3.5$ (red).
Horizontal lines are for our Oph-T3 (solid green), Oph-T17 (dashed black), and
the L object (dash-dotted orange).

In summary, two T dwarfs and one L dwarf in L 1688 of the $\rho$ Oph star-
forming region are identified on the basis of their 1.6 µm methane absorptions
and spectral morphology. Their spectral types are determined by comparing with
the spectra and spectral indices of known L and T dwarfs in the field. Oph
J162738$-$245240 (Oph-T3) is of a T0 spectral type. Oph J162645$-$241949
(Oph-T17) shows a T3/T4 type in the H band but an L8/T1 type in the K band.
Both objects, when comparing with theoretical models, are consistent with
having cool ($\sim 900$ K) and low-gravity atmospheres. Oph J162651-242110 is
a late L-type object. The three objects provide an anchor of the L/T
transition at 1 Myr old. The discovery of the two T dwarfs validates our
identification method using methane and temperature sensitive colors. Further
high signal-to-noise ratio spectra observations obviously will determine the
physical parameters much precisely.

We are grateful to Walfgang Brendner for help on data analysis, and to
Catarina Alves de Oliveria for providing the spectrum of 2M0437$+$2331\. We
acknowledge the FLAMINGOS-2 team of Gemini South for the guidance on data
reduction. Gratitude goes to the referee for constructive comments to
substantially improve the quality of the paper. The project is financially
supported by the MOST grant 103-2112-M-008-024-MY3. Facilities: CFHT, Gemini
South.

## References

  * Allard et al. (2012) Allard, F., Homeier, D., & Freytag, B. 2012, Royal Society of London Philosophical Transactions Series A, 370, 2765
  * Allers & Liu (2013) Allers, K. N., & Liu, M. C. 2013, ApJ, 772, 79
  * Alves de Oliveira et al. (2010) Alves de Oliveira, C., Moraux, E., Bouvier, J., et al. 2010, A&A, 515, A75
  * Alves de Oliveira et al. (2012) Alves de Oliveira, C., Moraux, E., Bouvier, J., & Bouy, H. 2012, A&A, 539, AA151
  * Barman et al. (2011) Barman, T. S., Macintosh, B., Konopacky, Q. M., & Marois, C. 2011, ApJ, 735, LL39
  * Baraffe et al. (2003) Baraffe, I., Chabrier, G., Barman, T. S., Allard, F., & Hauschildt, P. H. 2003, A&A, 402, 701
  * Bihain et al. (2009) Bihain, G., Rebolo, R., Zapatero Osorio, M. R., et al. 2009, A&A, 506, 1169
  * Bowler et al. (2011) Bowler, B. P., Liu, M. C., Kraus, A. L., Mann, A. W., & Ireland, M. J. 2011, ApJ, 743, 148
  * Bowler et al. (2014) Bowler, B. P., Liu, M. C., Kraus, A. L., & Mann, A. W. 2014, ApJ, 784, 65
  * Burgasser, Burrow, & Kirkpatrick (2006) Burgasser, A. J., Burrows, A., & Kirkpatrick, J. D. 2006, ApJ, 639, 1095
  * Burgess et al. (2009) Burgess, A. S. M., Moraux, E., Bouvier, J., et al. 2009, A&A, 508, 823
  * Burningham et al. (2013) Burningham, B., Cardoso, C. V., Smith, L., et al. 2013, MNRAS, 433, 457
  * Chabrier & Baraffe (2000) Chabrier, G., & Baraffe, I. 2000, ARA&A, 38, 337
  * Chiang et al. (2015) Chiang, P., Chen, W. P., Albert, L., Liu, M., & Magnier, E. A. 2015, MNRAS, 448, 522
  * Chiu et al. (2006) Chiu, K., Fan, X., Leggett, S. K., et al. 2006, AJ, 131, 2722
  * Delorme et al. (2012) Delorme, P., Gagné, J., Malo, L., et al. 2012, A&A, 548, A26
  * Delorme et al. (2013) Delorme, P., Gagné, J., Girard, J. H., et al. 2013, A&A, 553, L5
  * Evans et al. (2007) Evans N.J., Harvey P.M., Dumham M.M., Huard T.L., Mundy L.G., Lai S.P., Chapman N., Brooke T.Y., Enoch M.L., Stapelfeldt K.R., 2007, Final Delivery of Data from the c2d Legacy Project: IRAC and MIPS, (Pasadena: SSC)
  * Gagné et al. (2015) Gagné, J., Burgasser, A. J., Faherty, J. K., et al. 2015, ApJ, 808, L20
  * Geers et al. (2011) Geers, V., Scholz, A., Jayawardhana, R., et al. 2011, ApJ, 726, 23
  * Indebetouw et al. (2005) Indebetouw, R., Mathis, J. S., Babler, B. L., et al. 2005, ApJ, 619, 931
  * Kirkpatrick et al. (2000) Kirkpatrick, J. D., Reid, I. N., Liebert, J., et al. 2000, AJ, 120, 447
  * Kirkpatrick (2005) Kirkpatrick, J. D. 2005, ARA&A, 43, 195
  * Kirkpatrick et al. (2011) Kirkpatrick, J. D., Cushing, M. C., Gelino, C. R., et al. 2011, ApJS, 197, 19
  * Kirkpatrick et al. (2012) Kirkpatrick, J. D., Gelino, C. R., Cushing, M. C., et al. 2012, ApJ, 753, 156
  * Knapp et al. (2004) Knapp, G. R., Leggett, S. K., Fan, X., et al. 2004, AJ, 127, 3553
  * Leggett et al. (2010) Leggett, S. K., Burningham, B., Saumon, D., et al. 2010, ApJ, 710, 1627
  * Liu et al. (2013) Liu, M. C., Magnier, E. A., Deacon, N. R., et al. 2013, ApJ, 777, LL20
  * Luhman et al. (2004) Luhman, K. L., Peterson, D. E., & Megeath, S. T. 2004, ApJ, 617, 565
  * Luhman (2014) Luhman, K. L. 2014, ApJ, 786, LL18
  * Mace et al. (2013) Mace, G. N., Kirkpatrick, J. D., Cushing, M. C., et al. 2013, ApJS, 205, 6
  * Marocco et al. (2015) Marocco, F., Jones, H. R. A., Day-Jones, A. C., et al. 2015, MNRAS, 449, 3651
  * Marsh et al. (2010) Marsh, K. A., Kirkpatrick, J. D., & Plavchan, P. 2010, ApJ, 709, L158
  * Martín et al. (2010) Martín, E. L., Phan-Bao, N., Bessell, M., et al. 2010, A&A, 517, AA53
  * McLean et al. (2003) McLean, I. S., McGovern, M. R., Burgasser, A. J., et al. 2003, ApJ, 596, 561
  * Mužić et al. (2012) Mužić, K., Scholz, A., Geers, V., Jayawardhana, R., & Tamura, M. 2012, ApJ, 744, 134
  * Naud et al. (2014) Naud, M.-E., Artigau, É., Malo, L., et al. 2014, ApJ, 787, 5
  * Oasa et al. (1999) Oasa, Y., Tamura, M., & Sugitani, K. 1999, ApJ, 526, 336
  * Oppenheimer et al. (2013) Oppenheimer, B. R., Baranec, C., Beichman, C., et al. 2013, ApJ, 768, 24
  * Patience et al. (2010) Patience, J., King, R. R., de Rosa, R. J., & Marois, C. 2010, A&A, 517, A76
  * Patten et al. (2006) Patten, B. M., Stauffer, J. R., Burrows, A., et al. 2006, ApJ, 651, 502
  * Peña Ramírez et al. (2011) Peña Ramírez, K., Zapatero Osorio, M. R., Béjar, V. J. S., Rebolo, R., & Bihain, G. 2011, A&A, 532, AA42
  * Reid et al. (2008) Reid, I. N., Cruz, K. L., Kirkpatrick, J. D., et al. 2008, AJ, 136, 1290
  * Scholz et al. (2009) Scholz, R.-D., Storm, J., Knapp, G. R., & Zinnecker, H. 2009, A&A, 494, 949
  * Spezzi et al. (2012) Spezzi, L., Alves de Oliveira, C., Moraux, E., et al. 2012, A&A, 545, A105
  * Tinney et al. (2005) Tinney, C. G., Burgasser, A. J., Kirkpatrick, J. D., & McElwain, M. W. 2005, AJ, 130, 2326
  * Zapatero Osorio et al. (2002) Zapatero Osorio, M. R., Béjar, V. J. S., Martín, E. L., et al. 2002, ApJ, 578, 536