# From LSAT: The Progress and Challenges of Complex Reasoning

Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou,
Zhongyu Wei, Zhumin Chen, and Nan Duan, Work is done during internship at
Lanboat. Corresponding author: Ming Zhou; Zhongyu Wei.Siyuan Wang and Zhongyu
Wei are with the School of Data Science, Fudan University, Shanghai 200433,
China. And Zhongyu Wei is also with the Research Institute of Intelligent and
Complex Systems, Fudan University, Shanghai 200433, China (email:
<EMAIL_ADDRESS>zywei@fudan.edu.cn).Zhongkun Liu and Zhumin Chen are
with the School of Computer Science and Technology, Shandong University,
Qingdao 266237, China (email<EMAIL_ADDRESS>chenzhumin@mail.sdu.edu.cn)Wanjun Zhong is with the School of Computer Science
and Engineering, Sun Yat-Sen University, Guangzhou 510275, China (email:
zhongwj25@mail2.sysu.edu.cn).Ming Zhou is with the Sinovation Ventures,
Beijing 100080, China (email: zhouming@chuangxin.com).Nan Duan is with
Microsoft Research, Beijing 100080, China (email: nanduan@microsoft.com).

###### Abstract

Complex reasoning aims to draw a correct inference based on complex rules. As
a hallmark of human intelligence, it involves a degree of explicit reading
comprehension, interpretation of logical knowledge and complex rule
application. In this paper, we take a step forward in complex reasoning by
systematically studying the three challenging and domain-general tasks of the
Law School Admission Test (LSAT), including analytical reasoning, logical
reasoning and reading comprehension. We propose a hybrid reasoning system to
integrate these three tasks and achieve impressive overall performance on the
LSAT tests. The experimental results demonstrate that our system endows itself
a certain complex reasoning ability, especially the fundamental reading
comprehension and challenging logical reasoning capacities. Further analysis
also shows the effectiveness of combining the pre-trained models with the
task-specific reasoning module, and integrating symbolic knowledge into
discrete interpretable reasoning steps in complex reasoning. We further shed a
light on the potential future directions, like unsupervised symbolic knowledge
extraction, model interpretability, few-shot learning and comprehensive
benchmark for complex reasoning.

###### Index Terms:

LSAT, complex reasoning, analytical reasoning, logical reasoning, reading
comprehension.

##  I Introduction

Complex reasoning aims to comprehend and analyze the given information, and
apply complex rules to draw correct inference [1, 2]. As an essential ability
for complex problem solving, it provides tremendous opportunities for many
real-world scenarios, such as mathematical word problems, negotiation and
argument, and medical diagnosis [3, 4, 5, 6]. In recent years, having a
computer pass admission examinations is a hot AI challenge towards complex
reasoning, which offers an objective and accurate measurement with a certain
difficulty. Fujita et al. [7] design a system to sit the Japanese National
Center Test for University Admissions. Gaokao, as the National College
Entrance Examination of China, also has been widely studied [8, 9]. Although
encouraging results have been achieved in taking these real high-school exams,
these works study domain-specific and limited complex reasoning capabilities.
They cover different subjects and each is separately processed with subject-
specific knowledge [10]. For example, the annotated formulas representing
physics, mathematics and biology knowledge are usually assumed to be provided
to solve corresponding problems [7, 11], while history and geography questions
are primarily solved by retrieving relevant information supplemented by
shallow reasoning to match the answer [12, 13].

Figure 1:  Three examples of LSAT tasks with the required reasoning processes.
For AR, it needs to understand the knowledge of participants, positions, and
constraints, and deduce the legitimate option. For LR, the elementary logical
symbols and expressions need to be identified to logically infer the implicit
expression. For RC, it requires locating the relevant pieces by the positional
indicators and abstract the answer. The options in green mean the correct
answers of three examples.

To take a step towards a more challenging and domain-general complex reasoning
ability, we focus on the Law School Admission Test
(LSAT)111https://www:lsac:org/lsat/., which is one of the most difficult exams
covering multiple domains. LSAT is a standardized test administered for
prospective law school candidates worldwide, which mainly assesses their
general complex reasoning skills, including analytical reasoning, logical
reasoning, and reading comprehension capabilities in general domains.
Correspondingly, the LSAT can be categorized into three tasks: (1) analytical
reasoning (AR), measures the ability to analyze a scenario ruled by a set of
constraints, and determine which option satisfies or conflicts with all the
constraints. (2) logical reasoning (LR), a task that focuses on the logical
analysis of texts and performing logical inference to deduce implications from
asserted ones. (3) reading comprehension (RC), revolves around the ability to
deeply understand long-form materials, and locate the relevant pieces to
distinguishing what is the case by summarizing or comparing highly abstract
concepts, such as attitudes and principles. Three examples of LSAT tasks are
listed in Figure 1, which are all quite challenging and involve complex
reasoning processes. Therefore, we aim to systematically explore the progress
and challenges of complex reasoning, and take the real-world LSAT tasks which
have been rarely explored [14] as a testbed.

Existing methods for complex reasoning can be summarized into three types,
namely, symbolic models, neural models, and neural-symbolic models [15, 16].
Symbolic models identify the discrete symbols (like entities and logical
functions) as basic reasoning units, and perform explicit inferences upon
symbolic representations. With controllable and interpretable reasoning steps,
yet they largely depend on expert-defined rules which are inflexible for
different datasets and lack resilience against data noises [17]. Neural models
mimic the neuron connections in the human brain to learn the semantics of data
with continuous vectors and implicitly infer the answer, which are robust to
the ambiguous and noisy data but short of interpretability. To achieve
synergies among the advantages and circumvent the limitations of symbolism and
connectionism, neural-symbolic models integrate both symbolic logic and
continuous representation to reason out the answer [18].

From the above perspectives, we design a hybrid reasoning system for three
LSAT tasks. For AR, we first propose a symbolic system [19] which designs
rules to identify symbolic participants and constraints, and deterministically
deduce the legitimate solutions. We then attempt a neural method utilizing a
graph network for modeling constraints between participants. We also come up
with a neural-symbolic model which neurally parses the textual constraints
into programs and discretely executes the programs to reach the answer. For
LR, we propose a neural-symbolic logic-driven system [20], which employs a
symbolic module to extract logic from the texts and infer the entailed logic
by logical laws, then utilizes a neural module to encode the inferred logic
for answer prediction. For RC, we propose a neural Transformer-based approach
with an external multi-head attention mechanism [21] and passage positional
information for better understanding the interaction between context and
question.

On the whole, our overall system integrating three tasks achieves an accuracy
of $56.8\%$ on the LSAT exams, which is comparable to the median for human
test taker scores. Our systems for RC and LR even have a chance to be admitted
to the top 30 and top 58 law schools, respectively. The results show the
effectiveness of our systems for modeling complex reasoning abilities, notably
the fundamental reading comprehension and challenging logical reasoning
capability. The overall illustration of our investigation on LSAT towards
complex reasoning is shown in Figure 2. Through a systematical study of three
LSAT tasks, we not only achieve great performance on LSAT, but also make some
progress towards complex reasoning. We further investigate the emerging
challenges and inspire potential future directions of complex reasoning. For
example, automatically extracting symbolic knowledge in an unsupervised
manner, few-shot complex reasoning, improving the interpretability of the
neural reasoning system, and building a comprehensive benchmark, are all
essential to be explored to promote complex reasoning research.

In the rest of the paper, we first list some related work of complex reasoning
and corresponding advanced methods. We also preliminarily introduce the tasks,
baseline models and datasets of LSAT in § II. Then we separately introduce the
challenges, detailed methods, experimental results and further analysis for
analytical reasoning, logical reasoning and reading comprehension tasks in §
IV, § V and § VI, respectively. We further summarize our overall performance
on LSAT, and discuss the major challenges and future directions in complex
reasoning in § VII. We finally draw our conclusion in § VIII.

Figure 2:  The overall illustration of our investigation from LSAT towards
complex reasoning.

##  II Related Work

###  II-A Taxonomy of Complex Reasoning

To encourage the progress of artificial intelligence systems towards deeper
human-like comprehension and reasoning, there has been a surge in complex
reasoning research in recent years. We first investigate existing works on
several major aspects of complex reasoning.

##### Logical Reasoning

An increasing number of tasks and datasets have been introduced targeting
logical reasoning. Natural Language Inference [22, 23, 24] aims to determine
the entailment relationship between a hypothesis and a premise, which requires
relatively simple logical reasoning ability at the sentence level. Several
question answering datasets have been proposed for promoting logical reasoning
ability, i.e., LogiQA [25], ReClor [14], which are sourced from public
standardized exams. However, previous methods usually fail to model the
discrete logical inference process explicitly. This work also dives into
logical reasoning, and considers understanding the elementary logical
structure and perform explicit logical inference to draw a logical conclusion.

##### Commonsense Reasoning

Commonsense reasoning requires utilizing commonsense knowledge to reason out
the answer, which attracts great concern of the research communities. Recently
many benchmarks have been introduced to assess reasoning capabilities over
different commonsense knowledge, such as domain-specific knowledge [26, 27,
28], general semantic knowledge [29] and inferential knowledge [30]. Current
methods, however, are still not robust enough to be deployed in the open
domain and ignore directly modeling commonsense through symbolic integration
[31]. How to incorporate symbolic commonsense and reason over relevant
knowledge will be substantially explored [32, 33].

##### Multi-Hop Reasoning

Multi-hop reasoning is another widely studied complex reasoning task recently
[34, 35, 36, 37], which requires reasoning across multiple pieces of sentences
or documents to model multi-hop relationships to reach the answer.
Conversational question answering tasks [38, 39] also involve multi-hop
reasoning over multi-turn utterances. Constructing correct multi-hop reasoning
chains (conversation flows) has been a key challenge for these tasks, and it
is an essential research direction to effectively model multi-hop reasoning
paths over multiple passages[40, 41].

##### Numerical Reasoning

Numerical reasoning involves performing discrete arithmetic reasoning over
quantities to solve mathematical word problems[42, 43, 44, 45], which is a
fundamental and challenging task. Previous work translates the textual math
word problem into an expression or an expression tree and utilizes arithmetic
knowledge to solve it [43, 46, 47]. With weak supervision, how to improve the
accuracy of generated expression trees is worth further study [48, 49].

###  II-B Advanced Methods of Complex Reasoning

Advanced methods to solve complex reasoning problems, no matter what specific
reasoning skill is required, can be summarized as following three types:
symbolic models, neural models and neural-symbolic models [15, 16, 17].

##### Symbolic Models

As complex reasoning requires discrete reasoning operations over reasoning
units, most previous studies for complex reasoning are symbolic expert systems
[50, 51, 52, 53]. They design a set of rules or templates to identify basic
units for reasoning as symbols, such as the quantities and arithmetic signs
for numerical reasoning [54, 55, 56], relevant triples and paths from
knowledge graph for commonsense reasoning [57], etc. They then perform
deterministic and explicit inferences upon discrete elementary units to
predict the answer. The symbolic models provide sound readability and
interpretability. However, it requires expert knowledge and tremendous human
efforts in designing rules, which makes it inflexible to scale across
different datasets and lack resilience against data noises. Moreover, the
finite and discrete symbolic representations are insufficient to depict all
the intrinsic reasoning structures.

##### Neural Models

Neural models mimic the neuron connections in the human brain [58, 59] and
apply neural networks to implicitly represent the abstracted semantics of
input text and knowledge with continuous vectors, which are robust to the
ambiguous and noisy data [60, 61]. For example, pre-trained language models
have achieved superior performance on many comprehensive tasks [62, 63, 64].
However, the decision process of neural models is always a black box, which
makes the prediction lacks explainability and reliability. Whether the neural
models show the reasoning ability or just capture the data bias to achieve
high accuracy is also a question. Graph neural networks [65, 66] and neural
module networks [67, 68] are also introduced to partly imitate the human
reasoning process to make up interpretability. However, these methods still
perform an implicit inference to reason out the answer without a clue as to
why and how. Besides, neural models depend a lot on training data and are
computationally expensive to train, and the performance will sharply decrease
with limited data and resource.

##### Neural-Symbolic Models

To combine the advantages and circumvent the shortcomings of both symbolic and
neural methods, neural-symbolic models which integrate symbolic logic and
neural representation are widely studied [69, 70, 71, 72, 73]. Some work
employs a neural module to parse the language into executable programs [74,
18] and deterministically executing the programs to find the answer in a
symbolic module. For example, the neural-symbolic models for numerical
reasoning translate the input texts into expressions through a neural
generation model and then discretely execute the expressions [43, 46, 47].
Other work first designs rules to extract the reasoning units and explicitly
conduct inference over them in a symbolic module. They then utilize a neural
module to learn continuous vectors for symbolic representations to deal with
the uncertainty of data [75, 76, 20]. Specifically, for commonsense and multi-
hop reasoning, the symbolic module can be designed to identify relevant
knowledge and multi-hop reasoning chain, respectively, and then encode them
into the neural module to match the answer [77, 78, 79, 80]. However, how to
generate high-quality programs under weak supervision and extract symbolic
reasoning units in an unsupervised manner remains an elusive challenge.

###  II-C Examination-based Question Answering

Recent years have witnessed an emerging trend in answering complex questions
collected from human standardized examinations at a different level of
education, which is more difficult and measurable. The Todai Robot project
aims to create a system that can pass the Japanese National Center Test for
University Admissions [7]. Aristo Challenge focused on solving the questions
of Elementary School Science and Math Tests which is for 6-11 year olds [81].
A similar project studying the National College Entrance Examination of China
(Gaokao) also has been launched [8, 9, 12]. These challenges deal with various
subjects, like mathematics, biology, physics, geography, history, and drive
the progress of AI systems towards complex problem solving and reasoning.
However, these questions rely heavily on domain-specific expressions and
knowledge like formulas in mathematics and physics, and quotes in Classical
Chinese, which ignore the domain-general complex reasoning ability. Besides,
the studies of these tasks have hit the bottleneck of commonsense-based
reading comprehension and general intelligence, which fail to be admitted to
key universities [82].

RACE dataset [83] is introduced to remove domain restrictions by collecting
the general English exams for middle and high school Chinese students.
However, around 70% of questions are in the category of word matching,
paraphrasing or single-sentence reasoning, which are relatively simple. LogiQA
[25] collected from the National Civil Servants Examination of China and
ReClor [14] from the Graduate Management Admission Tests and Law School
Admission Tests both require deeper logical reasoning. In this paper, we not
only examine the logical reasoning capability of LSAT, but also challenging
analytical reasoning and complicated reading comprehension abilities to
systematically explore complex reasoning.

##  III Preliminaries

###  III-A Task Definition

The LSAT problems are formulated as a multiple-choice question answering task,
which is described as follows. Given a context $c$, a question $q$ together
with five candidate options $o={o_{1},o_{2},o_{3},o_{4},o_{5}}$, only one
option is need to be predicted as the most plausible answer $o_{a}$.

###  III-B Baseline Model

Pre-trained Transformer-based language models, i.e., BERT [62], RoBERTa [63]
and ALBERT [64], achieve impressive performance on multiple-choice question
answering, which can be employed as the baseline model of all LSAT tasks.
Specifically, the context, the question and an option are concatenated as the
input for encoding, which is formulated as $[CLS]\ c\ [SEP]\ q\ ||\ o_{i}\
[SEP]$ and $||$ is the concatenation operator. Given five options, five
concatenated sequences are constructed to be encoded. The representations of
special token $[CLS]$ in five sequences are fed into a classification layer to
get the probabilities of options as their scores, and the option with the
highest score is selected as the answer. The models are fine-tuned with cross-
entropy loss on the training set.

###  III-C Dataset

The LSAT datasets are collected from previous exams from 1991 to 2016,
including a total of 90 examinations. Each exam roughly contains 100
questions, among which 1/2 are logical reasoning questions, 1/4 are analytical
reasoning questions and the rest 1/4 are reading comprehension questions. For
a small proportion of questions with only four options, we randomly select one
of the wrong choices as a supplemental option for each instance. We name
datasets of analytical reasoning, logical reasoning and reading comprehension
tasks as AR-LSAT [19], LR-LSAT and RC-LSAT, respectively. Each dataset is
further split into training, validation and testing sets. The detailed
statistics of each dataset are listed in Table I.

Datasets | Statistics | Train | Val | Test | Total
---|---|---|---|---|---
_AR-LSAT_ | _# context_ | 280 | 40 | 40 | 360
_# question_ | 1,630 | 231 | 230 | 2,091
_LR-LSAT_ | _# context_ | 3,325 | 503 | 507 | 4,335
_# question_ | 3,529 | 506 | 510 | 4,545
_RC-LSAT_ | _# context_ | 280 | 40 | 40 | 360
_# question_ | 1,880 | 270 | 269 | 2,419

TABLE I: The detailed statistics of LSAT datasets. _# context_ and _#
question_ are the numbers of contexts and questions in each split.

As collected from standardized examinations, these LSAT datasets are of high
quality and difficulty for complex reasoning, and are accompanied by an
accurate evaluation metric and relatively limited data size. The data sparsity
makes these reasoning tasks more difficult to be solved by traditional data-
driven approaches that purely learn data patterns from massive data.
Therefore, the models with solid complex reasoning abilities need to be
developed and decrease the dependency on the data size.

##  IV Analytical Reasoning

###  IV-A Challenges in Analytical Reasoning

Analytical Reasoning aims to analyze a scenario involving a set of predefined
constraints and perform deductive reasoning to draw correct solutions. As no
previous work or benchmark dataset completely studies this challenging task,
we introduce a new dataset, namely AR-LSAT [19], to foster research on this
area. Most of the questions in AR-LSAT can be viewed as a constraint
satisfaction problem [84], which needs to find legal assignments of positions
to participants satisfying the given constraints. All participants, positions
and constraints are described in the context. Take a look at the first example
in Figure 1, two positions (i.e., X committee and Y committee) need to be
assigned to seven directors (i.e., A, B, etc.) under a set of constraints. To
solve such problems, the system requires understanding the game settings
including the compositions of participants, the possible values of positions,
and interpreting the logical meaning of the constraints descriptions. Then it
involves conducting inference over constraints to deduce the answers.

The analytical reasoning problems are quite challenging. The situation
descriptions in the context are diverse with no domain restriction. A more
accurate and comprehensive understanding of the context is also required,
because each piece of the context is significant in building the whole
reasoning chain. Besides, the five options of a question tend to be similar to
each other and the answer never explicitly appears in the context. Therefore,
the AR task cannot be superficially solved by relevant information extraction
and shallow contextual matching.

###  IV-B Symbolic Model: ARM

To explicitly model discrete analytical reasoning steps, we start with a
symbolic model called Analytical Reasoning Machine (ARM) [19], which can
answer the question by pre-defined rules and deterministic deduction. To solve
questions like the first example in Figure1, we propose to perform a four-
stage reasoning process: 1) extracting participants, positions, constraints
from the context and question; 2) interpreting constraints into executable
programs, i.e., a combination of logical functions; 3) generating a set of
legitimate assignments by executing all programs; 4) selecting the most
plausible option by matching the legitimate assignments. Next, we will
introduce these steps in detail.

####  IV-B1 Arguments Extraction

We first extract participants, positions and constraints from the context to
have a primary understanding of the problem. Specifically, we use a Named
Entity Recognition model [85] to extract entities from the context and group
them into participants and positions. Entities that appear together in the
leading sentence of the context are recognized as participants or positions,
where participants always appear before positions. We also identify
constraints by judging whether a sentence restricts the assignments between
participants and positions. As the example in Figure 1, seven participants
(i.e., _A,B,C,D,E,F,G_) need to be assigned into two positions (i.e., _X
committee and Y committee_). All sentences except the first are recognized as
constraint descriptions.

####  IV-B2 Constraints Interpretation

We next interpret natural language constraints into executable programs based
on three types of predefined functions, i.e., relational, counting and
compositional functions. The relational function involves participants and
positions as arguments and focuses on the relationship between them, e.g., “
_Before(A,B)_ ” means “ _A must be in a lower-numbered position than B_ ”
while “ _To(A,X)_ ” indicates “ _Participant A is assigned to position X_ ”.
The counting function describes the numerical and order constraints over
participants, which take as arguments both participants and numbers. A
compositional function takes two sets of relational or counting functions as
arguments and formulates the relationship between them, like conditional (_if-
then_) relationship. For example, the constraint “ _If A serves on the X, then
B serves on the Y_ ” can be expressed as “ _IfThen({To(A,X)}, {To(B,Y)})_”.

We design a set of trigger words to match potential functions [74] and extract
arguments (i.e., participants, positions, and numbers) according to their
relative positions to the trigger words. For uncertain sentences with no
matched function, we also build a neural classification model based on RoBERTa
to predict their corresponding function types. The combination of functions in
each sentence is the interpreted program. In figure 1, the set of interpreted
programs corresponding to the AR example is also presented.

Figure 3: The tree-based reasoning process. Here “T/F/-” means
“True/False/Unknown”.

####  IV-B3 Program Execution

We recognize the programs that immediately determine the positions of some
participants to construct an initial assignment. For example, the constraint “
_D and F both serve on the X committee_ ” corresponds to the initial
assignment in Figure 3. Each assignment is formulated as a table with columns
as participants and rows as positions. Each cell in the assignment has three
possible states, i.e., _True, False, Unknown_ , which means whether a
participant is assigned to a position.

Starting from the initial assignment, we conduct reasoning to deduce
legitimate assignments satisfying all constraints by executing other programs.
The reasoning process is formulated into a tree-based heuristic shown in
Figure 3. Specifically, each node is an assignment while each edge is an
executable program, and the initial assignment is taken as the root node
$a_{0}$. At each iteration, for a program $f_{i}$ we enumerate all possible
positions for participants involved in this program if they are unknown, and
execute the program to check satisfaction for each assignment to remove the
illegitimate ones. Then taking each legitimate assignment as a new root and
starting a new iteration, we validate the next program to further extend the
reasoning tree. The tree is recursively expanded until all programs are
executed. The leaf nodes that satisfy all constraints are obtained as final
legitimate assignments.

####  IV-B4 Option Selection

After deducing all legitimate assignments, we analyze the options to select
the most plausible one as the answer. As each option can be interpreted as a
program in the same way described in § IV-B2, we can further extend the
reasoning tree and execute the option-based program. We take the number of
legitimate assignments of each option as its score and the option with the
maximum score is selected as the final answer.

###  IV-C Neural Model: CGAR

As ARM requires substantial effort to craft sets of rules which are far from
perfect, we attempt to design two neural models to ease the manual work. We
first adopt the state-of-the-art pre-trained models including RoBERTa [63] and
ALBERT [64] as described in § III-B. However, they struggle to capture complex
reasoning ability beyond shallow-level semantic understanding and perform
nearly a random guess [19].

We also propose a Constraint Graph-based Analytical Reasoning (CGAR)
framework. Considering that for each AR problem, participants are to be
assigned into several positions, and each constraint describes a restriction
relationship between some participants and positions. To better model the
relationship structure of involved participants and positions to deduce the
legitimate solutions, we propose constructing constraint graphs for each
question. We thus introduce a CGAR framework that utilizes a graph
convolutional network (GCN) [86] and a pre-trained model to reason over the
constraint structure. The framework is composed of three modules, namely, a
graph construction module, a graph reasoning module and an answer prediction
module.

####  IV-C1 Constraint Graph Construction

We construct a constraint graph $\mathcal{G}_{i}$ for each (context $c$,
question $q$, option $o_{i}$) triple. Then five graphs can be constructed
corresponding to five options of a question. Each is designed as a
heterogeneous undirected graph where the nodes consist of participants,
positions and constraints in the context. Each constraint node is connected to
its mentioned participant nodes and position nodes. Besides, the (question,
option) pair is also injected as a global node to establish linkages with all
constraint nodes, to indicate whether the option satisfies the constraints and
answers the question. We follow the extraction method in [19] which utilizes
named entity recognition to extract participants and positions from the
leading sentence of the context. And we simply take each sentence in the
context as a constraint.

####  IV-C2 Graph Reasoning

To initialize the node representations, we utilize the output of the pre-
trained model as the embedding of each token. For participant and position
nodes, we perform mean pooling on the constituent token embeddings to get
their representations. For constraint nodes, we take the average of the start
and end token embeddings of each sentence as their representations. Then two
$[SEP]$ token embeddings are also averaged to initialize the representation of
the global node. To model the heterogeneity of the graph, we also define three
types of nodes, including the entity node, the constraint node and the global
node. Participant and position nodes are both viewed as entity nodes. We
utilize a linear transformation onto the node representations for different
node types to get their type-specific representations.

We employ a GCN to perform reasoning over a constraint graph
$\mathcal{G}_{i}$. During each message-passing iteration, we hope to model the
feasible states of participants and positions by aggregating the constraint
information to entities. Correspondingly, constraint features and global
features are also updated. After multiple iterations, the global node is aware
of available states of participants and positions in (question, option) pair
for predicting whether option $o_{i}$ is the correct answer.

####  IV-C3 Answer Prediction

As in § III-B, we also need to compute scores of each option to find the most
plausible answer. We determine the plausibility of each option $o_{i}$ given
the question $q$ with the information from both text $c$ and graph
$\mathcal{G}_{i}$. Specifically, we concatenate the final representation of
the global node with the representation of $[CLS]$ token from the pre-trained
model and feed it into a classification layer.

###  IV-D Neural-Symbolic Model: NSAR

Although neural models are capable of well capturing language semantics, they
disregard the interpretability of predictions. To reconcile the robust
learning in neural models and the discrete reasoning of symbolic methods, we
design a neural-symbolic model to solve the AR task. We propose an approach
named NSAR (Neural-Symbolic model for Analytical Reasoning), which extracts
the arguments and parses the constraints into compositional programs in a
neural manner, and executes the programs to derive the answer with a symbolic
inference engine.

As the symbolic executor is non-differential, policy-gradient methods are
usually employed to train the model [74, 87]. However, analytical reasoning
questions can only be answered after examining the satisfiability of all the
constraints, which leads to extremely sparse rewards and makes the model hard
to be optimized. Thus, we manually annotate programs for supervision learning.

####  IV-D1 Data Annotation

We first extend the constraint function set defined in ARM [19] to improve the
scalability (e.g., _FirstPos_ and _LastPos_ are too specific). It is composed
of common logical functions (_AND, OR, IF, NOT, etc._) and operator functions
including arithmetics ($+,-,=,>,<,etc.$), sorting (_ARGMAX, MAX, etc._),
assignment (_VALUE_), selection (_SELECT_) and counting (_COUNT_) operations.

Before program annotation, we need to annotate the participant and position
sets of each problem. Then for each constraint sentence in the context and the
(question, option) pair, we annotate a program following the above definition,
which is a composition of functions over the involved participants and
positions. For example, we annotate a program as “ _VALUE(roadster) $>$
VALUE(van) AND VALUE(roadster) $<$ VALUE(hatchback)_” for the constraint “
_The roadster is serviced later in the week than the van and earlier in the
week than the hatchback_ ”. The statistics and examples of our data annotation
are listed in Appendix A.

####  IV-D2 Neural Parser

In the neural perception module, we need to extract participants and positions
from the context and interpret constraints into programs.

##### Participant/Position Extractor

We separately take participant extraction and position extraction as two
sequence tagging tasks. For each task, a pre-trained model with a linear token
classification head on top is employed for predicting which tokens are parts
of participants or positions [63].

##### Program Parser

In order to model the relevance and dependency relationship between different
constraints for program parsing, we take the whole context as an input and
generate a sequence of compositional programs. The concatenation of the
question and each option is also fed as an input to generate a program. Here
we adopt a pre-trained encoder-decoder model for program generation [18, 88].
Besides, to alleviate generating irrelevant or incorrect content in programs,
we dynamically limit the vocabulary for each input to tokens of the input and
all constraint functions.

####  IV-D3 Symbolic Executor

After obtaining programs from the context, question and each option, we need
to execute these programs and present a score for each option. To do this, we
inherit the executor from § IV-B3 and § IV-B4. We first execute programs from
the context to obtain legitimate assignments. Then we execute the program of
each (question, option) pair on these assignments and calculate the ratio of
assignments satisfying the option as the score. We finally choose the option
with the highest score as the answer.

###  IV-E Results and Analysis

####  IV-E1 Overall Comparison

We compare the performance of the above three methods with a baseline pre-
trained model and human performance [19] on the validation and test sets. The
pre-trained baseline model, CGAR and the extractor in NSAR all take RoBERTa-
large [63] as the backbone while the parser in NSAR employs a T5-based model
[88].

Methods | Val (%) | Test (%)
---|---|---
_Human Performance_ | - | 59.7
_Random Guess_ | 20.0 | 20.0
_ARM_ | 34.2 | 30.9
_RoBERTa_ | 24.2 | 23.1
_CGAR_ | 27.7 | 24.9
_NSAR_ | 24.2 | 24.8
TABLE II: The answer prediction accuracy (%) of different methods on AR-LSAT.

The results are shown in Table II. We observe that the symbolic system _ARM_
outperforms _RoBERTa_ and _CGAR_. It shows that the complex analytical
reasoning process is difficult to be completely parameterized by neural
models, while _ARM_ works better by customizing complicated rules targeted at
the AR-LSAT dataset and performing a deterministic deduction. Meanwhile, such
difficulty for neural learning is exacerbated by limited data. From the
improvement of _CGAR_ compared to _RoBERTa_ , multiple message-passing
iterations over the constraint graph are proved to partly work on modeling the
analytical reasoning process. However, _NSAR_ outperforms _RoBERTa_ while
performing worse than both _CGAR_ and _ARM_ , because there is no large-scale
supervised data. We annotate a small number of instances for extracting
participant/position and parsing program, which results in interpreting
imperfect programs and further influences the execution performance. We deem
that annotating more data or heuristically augmenting data will boost the
performance of _NSAR_. Although we have achieved an improvement over the
baseline model, our systems are still far from equivalent to human
performance, leaving significant opportunities for further exploration of this
highly challenging task.

####  IV-E2 Detailed Analysis on NSAR

We first analyze the performance of participant extraction and position
extraction in Table III. The _NSAR_ performs relatively worse than _ARM_ in
both validation and test sets, indicating that it is more appropriate to apply
a rule-based extraction method to cover low-resource but diverse datasets. In
addition, there is still a certain gap between the _ARM_ model and perfect
extraction, showing that AR-LSAT is extremely challenging and needs a finer
set of extraction rules.

Methods | Val | Test
---|---|---
Prec. | Recall | Prec. | Recall
_ARM_ | 96.2 | 92.9 | - | -
_NSAR_ | 87.5 | 86.0 | 85.8 | 87.9
_ARM_ | 84.4 | 85.8 | - | -
_NSAR_ | 56.3 | 52.2 | 44.9 | 61.5
TABLE III: Performance (%) of the participant extraction (top) and position extraction (bottom). Parser Settings | EM | Rouge-L
---|---|---
_Combination_ | 0.0 | 60.9
_Separation_ | 16.2 | 68.4
TABLE IV: Evaluation results (%) of the program parser on AR-LSAT. EM means
Exact Match score.

To investigate the performance of our program parser, we evaluate the programs
generated by combining all sentences in the context and the question-option
pair (as _Combination_ setting) and by separately feeding each sentence or the
question-option pair (as _Separation_ setting), respectively. As shown in
Table IV, our generated programs achieve a high Rouge-L while the exact match
score is low and even equals zero for the combination setting. Besides the
limited training data, it is because that program parsing itself is a
challenging task. Specifically, it is susceptible to generating invalid
programs by a small midterm mistake and suffers from the intrinsic diversity
of programs that a constraint description can be formulated as different
programs. Moreover, we observe that the program generation for long context is
much more difficult than shorter sentences. How to design a simple and generic
program language is an essential direction for saving annotation resources and
improving program generation quality.

####  IV-E3 Error Analysis

We further dive into the error cases within our methods and summarize the
major reasons causing wrong predictions for AR problems. The first reason is
that participants and positions sometimes fail to be correctly extracted,
which fundamentally causes the misunderstanding of the problem settings and
thus affects the answer prediction of the aforementioned three methods.

Some other errors occur because the predefined program language is not
designed perfectly to cover some constraint descriptions with complex
semantics. For example, “two consecutive” in the constraint “ _No breed is
featured on any two consecutive days._ ” is difficult to be formulated. The
obstacle performance of the program parsing method also hinders the correct
analytical deduction.

Although the textual descriptions in AR-LSAT are straightforward, basic
commonsense knowledge is also needed to fully understand problem descriptions.
For example, the system should realize that “ _9:00 A.M._ ” and “ _2:00 P.M._
” are respectively in the morning and afternoon when it needs to arrange the
schedule to satisfy “ _Some participants should be scheduled in the morning._
”.

Figure 4:  The overall architecture of the logic-driven context extension
framework for LR. $c$, $q$, $o_{i}$ and $e_{i}$ are the context, question,
$i$-th option and the extended context for $i$-th option, respectively. The
texts in green mean that the option $B$ is matched against its extended
context which has the highest score.

####  IV-E4 Discussion

We make different attempts on AR-LSAT, including neural, symbolic and neural-
symbolic models, and observe that there is a certain gap between human
performance and all attempted methods. Our models achieve a nearly 31%
accuracy while the powerful pre-trained models obtain only random performance.
We therefore would like to highlight the challenges of analytical reasoning
and shed a light on the potential directions.

In order to solve the AR problem, the main step is to exactly understand and
abstract the textual constraints to machine cognitive programs with minimum
manual effort. Moreover, a question is usually answered through multiple
constraints. The research of modeling multiple constraints and mitigating
error accumulation can be further studied. As commonsense knowledge is
required for better understanding constraint descriptions, how to inject
external knowledge into the AR task is also a further research direction.
Although analytical reasoning assesses both deep analytical understanding and
complex deduction reasoning capabilities, only few data is available for
exploration. How to develop solid complex reasoning ability and ease the
dependency on the data size should be given top priority in future research.
Data augmentation is yet another feasible direction.

##  V Logical Reasoning

###  V-A Challenges in Logical Reasoning

Logical reasoning requires understanding a given text at a logical level and
performing logical inference to deduce implications from asserted ones. It is
a challenging task widely studied in recent years and several logical
reasoning benchmarks have been introduced, such as ReClor [14] and LogiQA
[25]. Previous work usually treats the task as a traditional reading
comprehension problem, and utilizes large-scale pre-trained language models
[62, 63, 64] or graph neural networks [76] to match the context with candidate
options. Although promising results have been achieved, they mainly rely on
word-level semantics without capturing symbolic logic.

In order to solve questions in LR-LSAT, as the example in Figure 4, the
reasoning system needs to extract the critical constituents from the context
as logical symbols like “$\alpha$: _have keyboarding skills_ ”, “$\beta$: _be
able to use a compute_ ”, “$\gamma$: _be able to write your essay using a word
processing program_ ” and identify the logical relationships between them to
constituent existing logical expressions, like
$(\neg\alpha\rightarrow\neg\beta)$ and $(\neg\beta\rightarrow\neg\gamma)$.
Then according to equivalence laws, it performs logical inference to extend
the logical expressions that are not explicitly mentioned in the context.
Comparing the extended expressions with the expressions of candidate options,
it selects the most similar option as the answer.

###  V-B LReasoner Model

We propose a logic-driven reasoner (LReasoner) model for logical reasoning
problems [20] from a neural-symbolic perspective. It utilizes a logic-driven
context extension framework to integrate the above reasoning process. Besides,
a logic-driven data augmentation algorithm is also introduced, to construct
literally similar but logically different samples and utilize contrastive
learning to encourage our model better capture logical information.

####  V-B1 Logic-Driven Context Extension

The overall logic-driven context extension framework is illustrated in Figure
4. It first identifies the logical symbols and expressions explicitly
mentioned in the context as the elementary components for reasoning (Logic
Identification). Then it performs logical inference following equivalence laws
to extend the implicit logical expressions (Logic Extension). Finally, it
verbalizes the extended logical expressions related to each option as an
extended context and utilizes it into the pre-trained model to match the
answer (Logic Verbalization).

##### Logic Identification

It employs a constituency parser [89] to extract constituents including noun
phrases and gerundial phrases from the context as basic logical symbols. Then
the logical symbols in each sentence are combined by logical negative and
conditional connectives $\\{\neg,\rightarrow\\}$ to constitute the logical
expression as a follow-up. If any negative word among {“ _not_ ”, “ _n’t_ ”, “
_unable_ ”, “ _no_ ”, “ _few_ ”, “ _little_ ”, “ _neither_ ”, “ _none of_ ”}
is in or immediately before a logical symbol $\alpha$, we add the negation
connective $\neg$ before $\alpha$ as $\neg\alpha$. If there is a conditional
relationship between two logical symbols $\alpha$ and $\beta$ in a sentence,
such as _“if $\alpha$, then $\beta$”, “$\alpha$ thus $\beta$”, “$\beta$ due to
$\alpha$” and “$\neg\beta$ unless $\alpha$”_, we can construct the
corresponding logical expression as $(\alpha\rightarrow\beta)$. As illustrated
in Figure 4, it extract three logical symbols $\\{\alpha,\beta,\gamma\\}$ and
identify two existing logical expressions as
$(\neg\alpha\rightarrow\neg\beta)$ and $(\neg\beta\rightarrow\neg\gamma)$.

##### Logic Extension

There still exists some other implicit logical expressions which can be
deduced from asserted ones. Therefore, it integrates the identified logical
expressions in all sentences of the context, and performs logical inference
over them to further extend the implicit expressions according to logical
equivalence laws, including _contraposition_ [90] and _transitive law_ [91]:

$\displaystyle\text{Contraposition}:\
(\alpha\rightarrow\beta)\implies(\neg\beta\rightarrow\neg\alpha)$ (1)
$\displaystyle\text{Transitive Law}:\ (\alpha\rightarrow\beta)\ \land\
(\beta\rightarrow\gamma)\implies(\alpha\rightarrow\gamma)$ (2)

As shown in Figure 4, it implies a set of extended logical expressions as
$\mathcal{S}_{E}=\\{(\beta\rightarrow\alpha),(\gamma\rightarrow\beta),(\neg\alpha\rightarrow\neg\gamma),(\gamma\rightarrow\alpha)\\}$.

##### Logic Verbalization

Considering that symbolic logic is more difficult to encode, it uses the pre-
trained model as the backbone of our framework. It verbalizes extended logical
expressions into natural language and feeds them as an extended context into
the pre-trained model. Specifically, it selects the related extended
expressions for each option that have the same logical symbols as the option
and transforms such expressions into natural language by filling them into a
template. $(\neg\alpha\rightarrow\neg\gamma)$ can be verbalized as “ _If do
not $\alpha$, then will not $\gamma$_”. It takes such a sentence as an
extended context for each option and feeds $[CLS]\ c\ [SEP]\ q\ ||\ o\ [EXT]\
e\ [SEP]$ into the pre-trained model to get each option’s score.

####  V-B2 Logic-Driven Data Augmentation

In order to make our model better capture logical information from the
context, we also introduce a logic-driven data augmentation algorithm. It
utilizes logical expressions to augment challenging samples with literally
similar but logically different contexts. Taking the original context to
construct the positive sample, it constructs _logical negative samples_ by
modifying the existing logical expressions in the context and verbalizing the
modified expressions into a negative context. The modification operations
include randomly deleting a logical expression, reversing the conditional
order of a logical expression and negating a logical symbol in a logical
expression.

It then adopts contrastive learning [92] and trains our model to select the
correct context supporting the answer to encourage the model to put more focus
on logical information, especially logical negative and conditional
relationships. As a result, the model is trained in a combination of answer
prediction loss and context classification loss.

###  V-C Results and Analysis

####  V-C1 Overall Comparison

We compare our LReasoner model with several baseline models for logical
reasoning, and the comparison results are shown in Table V. We can see that
_$\text{LReasoner}_{\text{ALBERT}}$_ achieves a great performance,
outperforming all baseline models by a considerable margin on both validation
and test sets. This indicates the effectiveness of our logic-driven system for
logical reasoning. And _$\text{LReasoner}_{\text{RoBERTa}}$_ and
_$\text{LReasoner}_{\text{ALBERT}}$_ both perform better than the
corresponding baseline models _RoBERTa_ and _ALBERT_. It demonstrates that
_LReasoner_ is robust to be effective for logical reasoning on top of
different pre-trained models.

Methods | Val (%) | Test (%)
---|---|---
_Random Guess_ | 20.0 | 20.0
_BERT_ | 39.3 | 39.4
_RoBERTa_ | 49.2 | 49.6
_ALBERT_ | 57.9 | 57.8
$\text{\emph{LReasoner}}_{\text{\emph{RoBERTa}}}$ | 54.0 | 53.3
$\text{\emph{LReasoner}}_{\text{\emph{ALBERT}}}$ | 65.0 | 63.5
Ablation study |  |
$\text{\emph{LReasoner}}_{\text{\emph{ALBERT}}}$ (w/o _CE_) | 63.4 | 61.6
$\text{\emph{LReasoner}}_{\text{\emph{ALBERT}}}$ (w/o _DA_) | 61.7 | 60.0
TABLE V: The answer prediction accuracy (%) of different methods on LR-LSAT.
_CE_ and _DA_ are our logic-driven context extension framework and data
augmentation algorithm.

We also conduct ablation study which takes _ALBERT_ as our backbone model.
_$\text{LReasoner}_{\text{ALBERT}}$ (w/o CE)_ and
_$\text{LReasoner}_{\text{ALBERT}}$ (w/o DA)_ both outperform the baseline
model _ALBERT_ and perform worse than our final system
_$\text{LReasoner}_{\text{ALBERT}}$_. It demonstrates that both logic-driven
context extension framework and logic-driven data augmentation algorithm are
beneficial for logical reasoning.

####  V-C2 Analysis of Logic Identification

To investigate the performance of the heuristic logic identification method,
we randomly sample 50 instances and manually annotate the logical symbols and
expressions as labels. We report the recall score of logical symbol
identification and logical expression identification as 65.9% and 48.9%,
respectively. We can see that our generic logic extraction method which
operates in an unsupervised manner achieves relatively reliable performance.
How to design an unsupervised and generic logic extraction method is essential
to be studied to further enhance the performance of the overall system.

####  V-C3 Performance on ReClor & LogiQA

We also conduct experiments on two public logical reasoning datasets, ReClor
[14] and LogiQA [25], to investigate the robustness of our logic-driven
reasoner. ReClor dataset is proposed from GMAT and LSAT tests while LogiQA is
collected from the National Civil Servants Examination of China, and each
question is provided with a context and four answer options. As shown in Table
VI, our system is robust to be effective on both ReClor and LogiQA, and even
surpasses the human performance of ReClor.

Model | ReClor | LogiQA
---|---|---
Val | Test | Val | Test
_RoBERTa_ [63] | 62.6 | 55.6 | 35.9 | 35.3
_ALBERT_ [64] | 70.2 | 66.5 | 38.9 | 37.6
_DAGN_ [76] | 65.8 | 58.3 | 36.9 | 39.3
$\text{\emph{LReasoner}}_{\text{\emph{RoBERTa}}}$ | 66.2 | 62.4 | 38.1 | 40.6
$\text{\emph{LReasoner}}_{\text{\emph{ALBERT}}}$ | 73.2 | 70.7 | 41.6 | 41.2
_Human Performance_ | - | 63.0 | - | 86.0
TABLE VI:  Experimental results (accuracy $\%$) of different models on ReClor
and LogiQA.

####  V-C4 Error Analysis

Although our LReasoner system achieves outstanding performance, there still
exist some instances that cannot be solved. Similar to the logical reasoning
dataset ReClor [14], LSAT also integrates various types of logical reasoning
skills, including “ _Necessary Assumptions_ ”, “ _Sufficient Assumptions_ ”, “
_Implication_ ”, “ _Most Strongly Supported_ ”, “ _Strength_ ”, “ _Weaken_ ”,
“ _Match flaws_ ”, etc. We thus investigate the detailed performance with
respect to different logical reasoning types to analyze which type of
questions tend to be more challenging.

Among nearly 17 categories, our model performs relatively poorly on _Match
flaws_ and _Weaken_ with an almost 60% accuracy while accuracies of other
types are higher than 70%. _Weaken_ aims to find the statement that weaken the
argument. _Match flaws_ is even more challenging as it requires analysis of
the flaw that conflicts with the complete logical chain illustrated in the
context, and find an option exhibiting the same flaw. Our system that first
extracts logical expressions and then implies the implicit logical expressions
is not suitable for abstracting the complete logical chain for flaw matching
and modeling the different degrees of a logical statement to identify weaken
statements.

####  V-C5 Discussion

Our logic-driven system is able to model the discrete logical inference
process explicitly and achieves an outstanding performance on LR-LSAT. Our
system even surpasses human performance on another logical reasoning dataset,
ReClor. However, some challenges are still there. The first is that current
methods treat all types of logical reasoning skills alike and do not dive into
the difference between reasoning types. How to deal with different logical
reasoning types is a further research direction. Moreover, the main challenge
for logical reasoning is to point out the logical structures among the
context. Specifically, automatically extracting the logical elementary units
and identifying the logical relationships between units in an unsupervised
manner is essential to be explored and improved. Further research can also
focus on directly encoding the symbolic logical structure rather than
verbalizing them into natural language for utilization. Concretely, we can
separately model each logical connectives in different networks, and take the
involved logical symbols as the network inputs to learn the logical
expressions.

##  VI Reading Comprehension

###  VI-A RC-LSAT Challenges

Apart from analytical reasoning and logical reasoning, LSAT also involves
reading comprehension, which is a fundamental ability of human intelligence.
It requires understanding long-form, complex passages and distinguishing what
is the correct statement by synthesis, comparison, and application of
principles. Many widely studied datasets have been developed for reading
comprehension, such as SQuAD [93], MCTest [94] and RACE [95]. Recent pre-
trained language models [63, 64] and diverse attention modules [96, 21]
achieve state-of-the-art performance on them, and even exceed human
performance. We compare RC-LSAT with several similar multiple-choice reading
comprehension datasets including MCTest, RACE, and COSMOS QA [33] to
investigate its challenges. The overall statistics are presented in Table VII.

Dataset | MCTest | RACE | COSMOS QA | RC-LSAT
---|---|---|---|---
# of contexts | 660 | 27,933 | 21,866 | 360
# of questions | 2,640 | 97,687 | 35,588 | 2,419
Context Len | 210.1 | 321.9 | 70.3 | 511.5
Question Len | 7.8 | 10.0 | 10.6 | 20.5
Option Len | 3.4 | 5.3 | 8.1 | 16.8
TABLE VII: Statistics of RACE, COSMOS QA, and RC-LSAT.

We can see that the context, question and option of RC-LSAT are more
complicated than other datasets with longer sequences, which are also more
difficult to be comprehended. Take a look at the following example question “
_Which one of the following most accurately and completely describes the
function of the second paragraph of the passage?_ ” and the corresponding
answer “ _explains the ramifications of the strict constructionists claims and
helps clarify the relevance of evidence offered in subsequent paragraphs_ ”.
Not only text understanding and function abstraction, but also positional
information modeling, such as “ _the second paragraph of the passage_ ”,
should be both incorporated for answer prediction. Besides, RC-LSAT is of
relatively limited data size. The above-mentioned challenges make models stuck
in more complex reading comprehension with few data.

###  VI-B Position-aware DUMA Model

Considering the state-of-the-art performance achieved by DUMA model [21] on
most similar examination-based reading comprehension datasets RACE involving
reasoning, we also employ DUMA for RC-LSAT. Compared to the baseline Model
introduced in Sec III-B, it additionally employs a Dual Multi-head Co-
Attention module between the pre-trained encoder model and the classification
layer. It simulates human transposition thinking patterns to further capture
relationships of key information from the passage, question and answer
options. In Dual Multi-head Co-Attention module, it first separates the output
representation of the encoder to obtain
$E^{P}=[e^{p}_{1},e^{p}_{2},...,e^{p}_{l_{p}}]$ and
$E^{QA}=[e^{qa}_{1},e^{qa}_{2},...,e^{qa}_{l_{qa}}]$, where $e^{p}_{i}$,
$e^{qa}_{j}$ denote the $i$-th and $j$-th token representation of passage and
question-answer respectively and $l_{p}$, $l_{qa}$ are the corresponding
lengths. It then calculates question-answer-aware passage representation
$E^{P(QA)}$ and passage-aware question-answer representation $E^{QA(P)}$ in a
bi-directional way. It aggregates the key information from $E^{QA(P)}$ and
$E^{P(QA)}$ as $O_{i}$ by mean pooling and concatenation operations and
utilizes it for answer classification instead of $[CLS]$ representation.

$\begin{split}E^{QA(P)}&=MultiHeadAttn(E^{P},E^{QA},E^{QA}),\\\
E^{P(QA)}&=MultiHeadAttn(E^{QA},E^{P},E^{P}),\\\
O_{i}=&[Mean(E^{QA(P)});Mean(E^{P(QA)})],\\\ \end{split}$ (3)

Note that questions of RC-LSAT usually involve positional information
indicators, like “ _line 3-5_ ” and “ _second paragraph_ ”, we label the
context with position marks, e.g., “$\langle$line3$\rangle$…
$\langle$/line3$\rangle$” for line 3 and “$\langle$P2$\rangle$…
$\langle$/P2$\rangle$” for paragraph 2. We feed labeled context as input and
implement a position-aware DUMA model.

###  VI-C Results and Analysis

####  VI-C1 Overall Comparison

We compare our position-aware _DUMA_ (_P-DUMA_) model with several baseline
models and _DUMA_ model and the comparison result is shown in Table VIII. We
can see that our _P-DUMA_ model achieves outstanding performance on both
validation and test sets. Position marks are observed to help improve
performance on top of both _ALBERT_ and _DUMA_ , which shows the effectiveness
of our proposed position mark for RC-LSAT.

Model | Val (%) | Test (%)
---|---|---
_RoBERTa_ | 52.0 | 44.0
_ALBERT_ | 51.9 | 48.7
_DUMA_ | 60.0 | 52.0
_P-ALBERT_ | 58.5 | 55.4
_P-DUMA_ | 57.4 | 56.1
TABLE VIII: Overall performance (%) on RC-LSAT. _P-ALBERT_ and _P-DUMA_ mean
position-aware ALBERT and our position-aware DUMA model, respectively.

####  VI-C2 Transfer Learning

Previous work [97, 98] has shown the effectiveness of pre-training on similar
datasets then fine-tuning on the target dataset for transfer learning, which
can partly relieve the pressure of data sparsity. Due to limited samples in
the RC-LSAT, we adopt the most similar RACE as the source dataset and conduct
a set of transfer learning experiments. As shown in Table IX, a significant
improvement is obtained after transfer learning for all models, which further
validates the potential of transfer learning for reading comprehension.
However, positional information has no effect in transfer learning settings
compared to position-unaware _ALBERT_ and _DUMA_. The reason may be that the
gap between RC-LSAT and RACE becomes larger after using position marks, as no
position label is provided in RACE.

Model | Val (%) | Test (%)
---|---|---
$\emph{ALBERT}_{\text{RACE}}$ | 73.7 | 67.7
$\emph{DUMA}_{\text{RACE}}$ | 71.9 | 69.1
$\emph{P-ALBERT}_{\text{RACE}}$ | 71.5 | 68.4
$\emph{P-DUMA}_{\text{RACE}}$ | 68.5 | 63.9
TABLE IX: Transfer learning results of models first trained on RACE and then
fine-tuned on RC-LSAT.

####  VI-C3 Error Analysis

Although outstanding performance has been achieved, there still exist some
challenging instances. We illustrate the major error types of RC-LSAT as
follows.

The first type of error is caused by the reason that the context is too long
and some essential information for answer prediction is truncated and cannot
be effectively encoded. The second error type is that some comparative
questions require comparative learning ability, which still has not been
considered. Take the question “ _The authors of the passages would be most
likely to disagree over whether?_ ” as an example, the model not only needs to
understand the view of each author, but also aims to comparatively infer the
point that authors would disagree with. In addition, the lack of commonsense
knowledge is another key problem for the wrong prediction. For example,
unaware that 1934 and 1939 are within the 1930s, the model would fail to
answer the question “ _According to the passage, which one of the following
was true of the physics community during the 1930s?_ ”.

####  VI-C4 Discussion

Although previous work [64, 21] and our models have shown promising results on
RC-LSAT, we still have a way to go to improve performance. As the truncation
mechanism of pre-trained models threatens modeling long sequence and the
sparse self-attention mechanism [99] sometimes fails to be effective,
designing an appropriate hierarchical encoder for long-form sequence encoding
is emerging as an area for further exploration. Besides, current methods
rarely specialize in answering comparative questions, how to comparatively
reading multiple passages and diving into the differences also requires
potential research.

A more challenging direction is to retrieve relevant commonsense or passages
from open source and incorporate them as evidence for answer prediction.
Previous work [100, 101] can only achieve poor performance on this topic due
to intrinsic noise in retrieved passages. Further research can focus on
improving evidence retrieval and predicting answers from noised evidence.

##  VII Further Discussion

After making attempts on three tasks of the LSAT, we have achieved some
progress towards complex reasoning. We evaluate our whole system on the LSAT
tests and raise some positive findings. We further discuss the existing
challenges of complex reasoning and shed a light on the future research
directions.

###  VII-A Overall Performance of LSAT Tests

To give an intuitive overview of our machine intelligence in LSAT tests, we
convert our raw accuracy scores of three tasks to an LSAT scale, which ranges
from 120 to 180 [102]. We also integrate the accuracies of AR, LR and RC, to
calculate an overall score by weighted averaging with the original proportion
of $1:2:1$. We compare our scaled scores with the median score of candidates
taking LSAT exams during $2019\sim 2020$ [103]. We further demonstrate which
level of schools our systems could be admitted to according to the law school
rankings displayed by Internet Legal Research Group [104].

| Scaled (Raw) Score | School Ranking
---|---|---
Overall system | 151 (56.8%) | Top 104
AR system | 135 (30.9%) | $>200$
LR system | 155 (63.5%) | Top 58
RC system | 158 (69.1%) | Top 30
Candidates | 152 (58.0%) | Top 94
TABLE X: Comparison results of our systems with human candidates. _$ >200$_
means that the ranking of AR system is beyond the top 200 which is not
displayed by Internet Legal Research Group [104].

As shown in Table X, we have several positive findings. Our overall system
achieves impressive performance on the standard LSAT tests designed to examine
the reasoning ability of prospective law school candidates, and performs
comparably with the median candidate scores, which indicates the potential of
machine complex reasoning. Our systems for RC and LR even have a chance to be
accepted by the top 30 and 58 law schools, respectively, which demonstrates
that the combination of the powerful pre-trained models and task-specific
reasoning modules is effective in performing complex reasoning. Concretely,
our logic-driven system endows itself an excellent logical reasoning ability
by performing explicit logical inference while the DUMA model aware of
position information with transfer learning possesses fundamental complex
reading comprehension capability. Although AR is an extremely challenging task
requiring a more comprehensive understanding of all context pieces to build
the whole reasoning chain and only few data is available, our symbolic ARM
system is still able to get into a law school, which shows that symbolic
knowledge plus discrete interpretable reasoning steps is essential in solving
analytical reasoning task.

###  VII-B Challenges & Future Directions

Despite the positive achievement, some unsolved challenges remain in complex
reasoning. We investigate the major challenges and the corresponding potential
solutions as follows.

####  VII-B1 Unsupervised Symbolic Knowledge Extraction

The automatic extraction of elementary symbolic units or expressions builds
the foundation of complex reasoning tasks as it is required to fully
understand complex scenarios, which further affects the performance of the
overall reasoning system. For example, the extraction of mathematical
expressions composed of quantities and arithmetic signs is essential for
numerical reasoning, and logical reasoning is heavily reliant upon logical
expression identification. However, predefined rule patterns by domain experts
or large-scale annotated data for symbolic knowledge extraction are expensive,
which are impractical to be obtained for all the tasks. Therefore, the
unsupervised extraction of symbolic knowledge is a major challenge in complex
reasoning.

To handle this challenge, we can begin by utilizing formal programming
languages [105, 106] to design a generic and extensible extraction framework
for universal symbolic knowledge, supplemented by a set of task-specific
extension and modification operations. In this way, we can further modify and
extend the generic framework, and obtain the specific extraction method for
different complex reasoning tasks to automatically extract high-quality
symbolic knowledge.

####  VII-B2 Model Interpretability

Interpretability is a significant characteristic of trustworthy and
controllable reasoning systems, which makes the decision-making process of
complex reasoning comprehensible and predicts more reliable results. Besides,
the incorrect prediction can be traced to learn the cause and intervened to
make a revised prediction. For instance, the explicit walk along the multi-hop
relational paths makes the multi-hop reasoning process more interpretable to
find the answer. Although neural models achieve robust performance on complex
reasoning, their prediction is always a black box for humans to understand.
How to improve the interpretability of a neural system is essential to be
studied.

We can design the neural model structure from the perspective of simulating
human cognition and the reasoning process. It integrates multiple modules for
different reasoning steps, and injects the intermediate results into the whole
reasoning chain, which makes the neural model interpretable.

####  VII-B3 Few-Shot Learning

As the data targeted at complex reasoning with high quality and difficulty is
rare and hard to be collected or annotated, the traditional data-driven
learning methods heavily relying on a huge amount of training instances may
have poor performance on complex reasoning tasks. The analytical reasoning
task is a typical example with few data. Therefore, a few-shot learning
paradigm [107, 108, 109] urgently needs to be explored for complex reasoning
to improve the reasoning ability and the generalization capability of systems
with only a few training instances.

Transfer learning will be a potential direction of few-shot learning for
complex reasoning. It uses the pre-training models on related tasks as the
starting point which can ease the sufferings of data sparsity, and transfer
the reasoning ability learned from source task to target domain. We can also
heuristically synthesize the data by modifying original data to augment the
dataset of the current complex reasoning task.

####  VII-B4 All-Sided Benchmark for Complex Reasoning

Recent years have witnessed an increasing trend in natural language
understanding towards complex reasoning, yet there is no existing integrated
benchmark to the best of our knowledge, that comprehensively evaluates
different types and domains of complex reasoning ability. It is worthwhile to
build an all-sided benchmark dataset to promote research in complex reasoning.

First of all, the benchmark should cover the three reasoning capabilities
involved in the LSAT. Specifically, reading comprehension establishes the
foundation of complex reasoning to understand and summarize the semantics of
substances and qualities. Logical reasoning extends complex reasoning ability
with logical deduction over propositions while analytical reasoning simulates
the human analytical thinking and problem-solving capacity. Then several
widely studied complex reasoning tasks should be integrated, such as reasoning
for commonsense knowledge [27, 29, 30], multi-hop relationships [35, 36] and
numerical calculation [44, 45] described in § II-A.

Some other rarely explored complex reasoning abilities also need to be
considered. For example, we should dive into abductive reasoning [110] and
counterfactual reasoning [111, 112] to research cause-and-effect complex
reasoning. The former aims to trace the most likely explanation to partial
observations and the latter focus on the illation of how alternative events in
the past result in different outcomes. Analogical reasoning [113] that draws a
comparison between different things and based on their similarities to infer
their further shared properties is widely adopted in human daily thinking. As
no benchmark dataset so far has been proposed for mainly enhancing analogical
inference, it also should be involved in this comprehensive benchmark.

##  VIII Conclusion

In this paper, we take a step towards complex reasoning research by studying
three tasks involved in LSAT tests, namely, analytical reasoning, logical
reasoning and reading comprehension. Inspired by the advantages and
disadvantages of symbolic, neural, and neural-symbolic methods of complex
reasoning, we propose a hybrid system for three tasks and achieve promising
performance on the LSAT tests. In particular, our position-aware DUMA model
with transfer learning for reading comprehension and logic-driven model for
logical reasoning even have a chance to be recruited to the top 30 and top 58
law schools, respectively. It demonstrates our progress in modeling complex
reasoning abilities, notably the fundamental reading comprehension and
challenging logical reasoning capability. Through a systematical study of the
LSAT, we further overall discuss the unsolved challenges in complex reasoning
and investigate the potential directions. In the future, how to extract
symbolic knowledge in an unsupervised manner and improve the interpretability
of a neural reasoning system are promising directions. Few-shot complex
reasoning and building a comprehensive complex reasoning benchmark are also
worthy of exploration.

##  Appendix A Data Annotation

We annotate a subset of programs to investigate the operation mechanism of our
neural-symbolic method. An example of our annotation is shown in Figure 5. For
each problem, we not only annotate the participants and positions, but also
annotate the programs of all constraints in the context and question-option
pairs. We annotate a total of $100/40/40$ instances for participants and
positions extraction in the training/validation/test set. As the AR task has a
situational property that a group of questions share the same context but ask
different aspects of information, we annotate programs for total 77 contexts
and 428 questions, with $50/13/14$ contexts and $274/75/79$ questions in the
train/validation/test set.

Figure 5:  An example of our annotation for an AR problem. C-$i$ means the
$i$-th constraint while Q-$j$ denotes the pair of the question and option $j$.

## References

  * Duschl et al. [2007] R. A. Duschl, H. A. Schweingruber, and A. W. Shouse, “Taking science to school: Learning and teaching science in grades k-8,” _Eurasia Journal of Mathematics, Science & Technology Education_, vol. 3, no. 2, pp. 163–166, 2007.
  * Songer et al. [2009] N. B. Songer, B. Kelcey, and A. W. Gotwals, “How and when does complex reasoning occur? empirically driven development of a learning progression focused on complex reasoning about biodiversity,” _Journal of Research in Science Teaching: The Official Journal of the National Association for Research in Science Teaching_ , vol. 46, no. 6, pp. 610–631, 2009.
  * Chattopadhyay et al. [2013] S. Chattopadhyay, S. Banerjee, F. A. Rabhi, and U. R. Acharya, “A case-based reasoning system for complex medical diagnosis,” _Expert Systems_ , vol. 30, no. 1, pp. 12–20, 2013.
  * Shi et al. [2015] S. Shi, Y. Wang, C.-Y. Lin, X. Liu, and Y. Rui, “Automatically solving number word problems by semantic parsing and reasoning,” in _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , 2015, pp. 1132–1142.
  * Reinhold et al. [2020] F. Reinhold, S. Hofer, M. Berkowitz, A. Strohmaier, S. Scheuerer, F. Loch, B. Vogel-Heuser, and K. Reiss, “The role of spatial, verbal, numerical, and general reasoning abilities in complex word problem solving for young female and male adults,” _Mathematics Education Research Journal_ , vol. 32, no. 2, pp. 189–211, 2020.
  * Galassi et al. [2020] A. Galassi, K. Kersting, M. Lippi, X. Shao, and P. Torroni, “Neural-symbolic argumentation mining: An argument in favor of deep learning and reasoning,” _Frontiers in big Data_ , vol. 2, p. 52, 2020.
  * Fujita et al. [2014] A. Fujita, A. Kameda, A. Kawazoe, and Y. Miyao, “Overview of todai robot project and evaluation framework of its nlp-based problem solving,” _World History_ , vol. 36, no. 36, p. 148, 2014.
  * Cheng et al. [2016] G. Cheng, W. Zhu, Z. Wang, J. Chen, and Y. Qu, “Taking up the gaokao challenge: An information retrieval approach.” in _IJCAI_ , vol. 2016, 2016, pp. 2479–2485.
  * Yu et al. [2016] K. Yu, Q. Liu, Y. Zheng, T. Zhao, and D. Zheng, “History question classification and representation for chinese gaokao,” in _2016 International Conference on Asian Language Processing (IALP)_. IEEE, 2016, pp. 129–132.
  * Zhang et al. [2018a] Z. Zhang, L. Zhang, H. Zhang, W. He, Z. Sun, G. Cheng, Q. Liu, X. Dai, and Y. Qu, “Towards answering geography questions in gaokao: A hybrid approach,” in _China Conference on Knowledge Graph and Semantic Computing_. Springer, 2018, pp. 1–13.
  * Gunning et al. [2010] D. Gunning, V. K. Chaudhri, P. E. Clark, K. Barker, S.-Y. Chaw, M. Greaves, B. Grosof, A. Leung, D. D. McDonald, S. Mishra _et al._ , “Project halo update—progress toward digital aristotle,” _AI Magazine_ , vol. 31, no. 3, pp. 33–58, 2010.
  * Ding et al. [2018] J. Ding, Y. Wang, W. Hu, L. Shi, and Y. Qu, “Answering multiple-choice questions in geographical gaokao with a concept graph,” in _European Semantic Web Conference_. Springer, 2018, pp. 161–176.
  * Zhang and Zhao [2018] Z. Zhang and H. Zhao, “One-shot learning for question-answering in gaokao history challenge,” _arXiv preprint arXiv:1806.09105_ , 2018.
  * Yu et al. [2020] W. Yu, Z. Jiang, Y. Dong, and J. Feng, “Reclor: A reading comprehension dataset requiring logical reasoning,” in _International Conference on Learning Representations (ICLR)_ , April 2020.
  * Itzhaky et al. [2013] S. Itzhaky, S. Gulwani, N. Immerman, and M. Sagiv, “Solving geometry problems using a combination of symbolic and numerical reasoning,” in _International Conference on Logic for Programming Artificial Intelligence and Reasoning_. Springer, 2013, pp. 457–472.
  * Bach et al. [2015] S. H. Bach, M. Broecheler, B. Huang, and L. Getoor, “Hinge-loss markov random fields and probabilistic soft logic,” _arXiv preprint arXiv:1505.04406_ , 2015.
  * Zhang et al. [2021] J. Zhang, B. Chen, L. Zhang, X. Ke, and H. Ding, “Neural, symbolic and neural-symbolic reasoning on knowledge graphs,” _AI Open_ , vol. 2, pp. 14–35, 2021.
  * Chen et al. [2019a] X. Chen, C. Liang, A. W. Yu, D. Zhou, D. Song, and Q. V. Le, “Neural symbolic reader: Scalable integration of distributed and symbolic representations for reading comprehension,” in _International Conference on Learning Representations_ , 2019.
  * Zhong et al. [2021] W. Zhong, S. Wang, D. Tang, Z. Xu, D. Guo, J. Wang, J. Yin, M. Zhou, and N. Duan, “Ar-lsat: Investigating analytical reasoning of text,” 2021.
  * Wang et al. [2021] S. Wang, W. Zhong, D. Tang, Z. Wei, Z. Fan, D. Jiang, M. Zhou, and N. Duan, “Logic-driven context extension and data augmentation for logical reasoning of text,” 2021.
  * Zhu et al. [2020] P. Zhu, H. Zhao, and X. Li, “Duma: Reading comprehension with transposition thinking,” _arXiv: 2001.09415_ , 2020.
  * Dagan et al. [2005] I. Dagan, O. Glickman, and B. Magnini, “The pascal recognising textual entailment challenge,” in _Machine Learning Challenges Workshop_. Springer, 2005, pp. 177–190.
  * Bowman et al. [2015] S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning, “A large annotated corpus for learning natural language inference,” in _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Lisbon, Portugal: Association for Computational Linguistics, Sep. 2015, pp. 632–642.
  * Williams et al. [2017] A. Williams, N. Nangia, and S. R. Bowman, “A broad-coverage challenge corpus for sentence understanding through inference,” _arXiv preprint arXiv:1704.05426_ , 2017.
  * Liu et al. [2020a] J. Liu, L. Cui, H. Liu, D. Huang, Y. Wang, and Y. Zhang, “Logiqa: A challenge dataset for machine reading comprehension with logical reasoning,” in _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence_ , 2020.
  * Rashkin et al. [2018] H. Rashkin, M. Sap, E. Allaway, N. A. Smith, and Y. Choi, “Event2mind: Commonsense inference on events, intents, and reactions,” _arXiv preprint arXiv:1805.06939_ , 2018.
  * Talmor et al. [2018] A. Talmor, J. Herzig, N. Lourie, and J. Berant, “Commonsenseqa: A question answering challenge targeting commonsense knowledge,” _arXiv preprint arXiv:1811.00937_ , 2018.
  * Zhou et al. [2019] B. Zhou, D. Khashabi, Q. Ning, and D. Roth, “” going on a vacation” takes longer than” going for a walk”: A study of temporal commonsense understanding,” _arXiv preprint arXiv:1909.03065_ , 2019.
  * Speer et al. [2017] R. Speer, J. Chin, and C. Havasi, “Conceptnet 5.5: An open multilingual graph of general knowledge,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 31, no. 1, 2017.
  * Sap et al. [2019] M. Sap, R. Le Bras, E. Allaway, C. Bhagavatula, N. Lourie, H. Rashkin, B. Roof, N. A. Smith, and Y. Choi, “Atomic: An atlas of machine commonsense for if-then reasoning,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, no. 01, 2019, pp. 3027–3035.
  * Sap et al. [2020] M. Sap, V. Shwartz, A. Bosselut, Y. Choi, and D. Roth, “Introductory tutorial: Commonsense reasoning for natural language processing,” _Association for Computational Linguistics (ACL 2020): Tutorial Abstracts_ , p. 27, 2020.
  * Zhang et al. [2018b] S. Zhang, X. Liu, J. Liu, J. Gao, K. Duh, and B. Van Durme, “Record: Bridging the gap between human and machine commonsense reading comprehension,” _arXiv preprint arXiv:1810.12885_ , 2018.
  * Huang et al. [2019] L. Huang, R. L. Bras, C. Bhagavatula, and Y. Choi, “Cosmos qa: Machine reading comprehension with contextual commonsense reasoning,” _arXiv preprint arXiv:1909.00277_ , 2019.
  * Khashabi et al. [2018] D. Khashabi, S. Chaturvedi, M. Roth, S. Upadhyay, and D. Roth, “Looking beyond the surface: A challenge set for reading comprehension over multiple sentences,” in _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , 2018, pp. 252–262.
  * Welbl et al. [2018] J. Welbl, P. Stenetorp, and S. Riedel, “Constructing datasets for multi-hop reading comprehension across documents,” _Transactions of the Association for Computational Linguistics_ , vol. 6, pp. 287–302, 2018.
  * Yang et al. [2018] Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning, “Hotpotqa: A dataset for diverse, explainable multi-hop question answering,” _arXiv preprint arXiv:1809.09600_ , 2018.
  * Inoue et al. [2019] N. Inoue, P. Stenetorp, and K. Inui, “R4c: A benchmark for evaluating rc systems to get the right answer for the right reason,” _arXiv preprint arXiv:1910.04601_ , 2019.
  * Choi et al. [2018] E. Choi, H. He, M. Iyyer, M. Yatskar, W.-t. Yih, Y. Choi, P. Liang, and L. Zettlemoyer, “Quac: Question answering in context,” _arXiv preprint arXiv:1808.07036_ , 2018.
  * Reddy et al. [2019] S. Reddy, D. Chen, and C. D. Manning, “Coqa: A conversational question answering challenge,” _Transactions of the Association for Computational Linguistics_ , vol. 7, pp. 249–266, 2019.
  * Chen et al. [2019b] J. Chen, S.-t. Lin, and G. Durrett, “Multi-hop question answering via reasoning chains,” _arXiv preprint arXiv:1910.02610_ , 2019.
  * Wang et al. [2019] H. Wang, M. Yu, X. Guo, R. Das, W. Xiong, and T. Gao, “Do multi-hop readers dream of reasoning chains?” _arXiv preprint arXiv:1910.14520_ , 2019.
  * Clark et al. [2016] P. Clark, O. Etzioni, T. Khot, A. Sabharwal, O. Tafjord, P. Turney, and D. Khashabi, “Combining retrieval, statistics, and inference to answer elementary science questions,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 30, no. 1, 2016.
  * Ling et al. [2017] W. Ling, D. Yogatama, C. Dyer, and P. Blunsom, “Program induction by rationale generation: Learning to solve and explain algebraic word problems,” _arXiv preprint arXiv:1705.04146_ , 2017.
  * Dua et al. [2019] D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner, “Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs,” _arXiv preprint arXiv:1903.00161_ , 2019.
  * Amini et al. [2019] A. Amini, S. Gabriel, P. Lin, R. Koncel-Kedziorski, Y. Choi, and H. Hajishirzi, “Mathqa: Towards interpretable math word problem solving with operation-based formalisms,” _arXiv preprint arXiv:1905.13319_ , 2019.
  * Wang et al. [2018] L. Wang, Y. Wang, D. Cai, D. Zhang, and X. Liu, “Translating a math word problem to an expression tree,” _arXiv preprint arXiv:1811.05632_ , 2018\.
  * Xie and Sun [2019] Z. Xie and S. Sun, “A goal-driven tree-structured neural model for math word problems.” in _IJCAI_ , 2019, pp. 5299–5305.
  * Hong et al. [2021] Y. Hong, Q. Li, D. Ciao, S. Huang, and S.-C. Zhu, “Learning by fixing: Solving math word problems with weak supervision,” in _AAAI Conference on Artificial Intelligence_ , 2021.
  * Saha et al. [2021] A. Saha, S. Joty, and S. C. Hoi, “Weakly supervised neuro-symbolic module networks for numerical reasoning,” _arXiv preprint arXiv:2101.11802_ , 2021\.
  * Galárraga et al. [2013] L. A. Galárraga, C. Teflioudi, K. Hose, and F. Suchanek, “Amie: association rule mining under incomplete evidence in ontological knowledge bases,” in _Proceedings of the 22nd international conference on World Wide Web_ , 2013, pp. 413–422.
  * Zheng et al. [2015] W. Zheng, L. Zou, X. Lian, J. X. Yu, S. Song, and D. Zhao, “How to build templates for rdf question/answering: An uncertain graph similarity join approach,” in _Proceedings of the 2015 ACM SIGMOD international conference on management of data_ , 2015, pp. 1809–1824.
  * Minsky and Papert [2017] M. Minsky and S. A. Papert, _Perceptrons: An introduction to computational geometry_. MIT press, 2017.
  * Donadello et al. [2017] I. Donadello, L. Serafini, and A. D. Garcez, “Logic tensor networks for semantic image interpretation,” _arXiv preprint arXiv:1705.08968_ , 2017\.
  * Fuchs et al. [2013] L. S. Fuchs, D. C. Geary, D. L. Compton, D. Fuchs, C. Schatschneider, C. L. Hamlett, J. DeSelms, P. M. Seethaler, J. Wilson, C. F. Craddock _et al._ , “Effects of first-grade number knowledge tutoring with contrasting forms of practice.” _Journal of Educational Psychology_ , vol. 105, no. 1, p. 58, 2013.
  * Supekar et al. [2013] K. Supekar, A. G. Swigart, C. Tenison, D. D. Jolles, M. Rosenberg-Lee, L. Fuchs, and V. Menon, “Neural predictors of individual differences in response to math tutoring in primary-grade school children,” _Proceedings of the National Academy of Sciences_ , vol. 110, no. 20, pp. 8230–8235, 2013.
  * Donadello and Serafini [2016] I. Donadello and L. Serafini, “Integration of numeric and symbolic information for semantic image interpretation,” _Intelligenza Artificiale_ , vol. 10, no. 1, pp. 33–47, 2016.
  * Sun [1995] R. Sun, “Robust reasoning: integrating rule-based and similarity-based reasoning,” _Artificial Intelligence_ , vol. 75, no. 2, pp. 241–295, 1995\.
  * Rosenblatt [1958] F. Rosenblatt, “The perceptron: a probabilistic model for information storage and organization in the brain.” _Psychological review_ , vol. 65, no. 6, p. 386, 1958.
  * Rumelhart et al. [1986] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” _nature_ , vol. 323, no. 6088, pp. 533–536, 1986.
  * Hinton et al. [2006] G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,” _Neural computation_ , vol. 18, no. 7, pp. 1527–1554, 2006\.
  * Bengio et al. [2007] Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, “Greedy layer-wise training of deep networks,” in _Advances in neural information processing systems_ , 2007, pp. 153–160.
  * Devlin et al. [2019] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186.
  * Liu et al. [2019] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” _CoRR_ , vol. abs/1907.11692, 2019.
  * Lan et al. [2020] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Albert: A lite bert for self-supervised learning of language representations,” in _International Conference on Learning Representations_ , 2020.
  * Kipf and Welling [2016] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” _arXiv preprint arXiv:1609.02907_ , 2016.
  * Veličković et al. [2017] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, “Graph attention networks,” _arXiv preprint arXiv:1710.10903_ , 2017.
  * Andreas et al. [2016] J. Andreas, M. Rohrbach, T. Darrell, and D. Klein, “Neural module networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 39–48.
  * Gupta et al. [2019] N. Gupta, K. Lin, D. Roth, S. Singh, and M. Gardner, “Neural module networks for reasoning over text,” _arXiv preprint arXiv:1912.04971_ , 2019.
  * Gallant [1988] S. I. Gallant, “Connectionist expert systems,” _Communications of the ACM_ , vol. 31, no. 2, pp. 152–169, 1988.
  * Gallant and Gallant [1993] S. I. Gallant and S. I. Gallant, _Neural network learning and expert systems_. MIT press, 1993.
  * de Penning et al. [2011] H. L. H. de Penning, A. S. d. Garcez, L. C. Lamb, and J.-J. C. Meyer, “A neural-symbolic cognitive agent for online learning and reasoning,” in _Twenty-Second International Joint Conference on Artificial Intelligence_ , 2011.
  * Besold et al. [2017] T. R. Besold, A. d. Garcez, S. Bader, H. Bowman, P. Domingos, P. Hitzler, K.-U. Kühnberger, L. C. Lamb, D. Lowd, P. M. V. Lima _et al._ , “Neural-symbolic learning and reasoning: A survey and interpretation,” _arXiv preprint arXiv:1711.03902_ , 2017.
  * Garcez et al. [2019] A. d. Garcez, M. Gori, L. C. Lamb, L. Serafini, M. Spranger, and S. N. Tran, “Neural-symbolic computing: An effective methodology for principled integration of machine learning and reasoning,” _arXiv preprint arXiv:1905.06088_ , 2019.
  * Liang et al. [2016] C. Liang, J. Berant, Q. Le, K. D. Forbus, and N. Lao, “Neural symbolic machines: Learning semantic parsers on freebase with weak supervision,” _arXiv preprint arXiv:1611.00020_ , 2016.
  * Zhong et al. [2020] W. Zhong, D. Tang, Z. Feng, N. Duan, M. Zhou, M. Gong, L. Shou, D. Jiang, J. Wang, and J. Yin, “Logicalfactchecker: Leveraging logical operations for fact checking with graph module network,” _arXiv preprint arXiv:2004.13659_ , 2020.
  * Huang et al. [2021] Y. Huang, M. Fang, Y. Cao, L. Wang, and X. Liang, “Dagn: Discourse-aware graph network for logical reasoning,” _arXiv preprint arXiv:2103.14349_ , 2021\.
  * Arabshahi et al. [2020] F. Arabshahi, J. Lee, M. Gawarecki, K. Mazaitis, A. Azaria, and T. Mitchell, “Conversational neuro-symbolic commonsense reasoning,” _arXiv preprint arXiv:2006.10022_ , 2020.
  * Liu et al. [2020b] Y. Liu, M. Hildebrandt, M. Joblin, M. Ringsquandl, and V. Tresp, “Integrating logical rules into neural multi-hop reasoning for drug repurposing,” _arXiv preprint arXiv:2007.05292_ , 2020.
  * Liu et al. [2021] Y. Liu, M. Hildebrandt, M. Joblin, M. Ringsquandl, R. Raissouni, and V. Tresp, “Neural multi-hop reasoning with logical rules on biomedical knowledge graphs,” in _European Semantic Web Conference_. Springer, 2021, pp. 375–391.
  * Moghimifar et al. [2021] F. Moghimifar, L. Qu, Y. Zhuo, G. Haffari, and M. Baktashmotlagh, “Neural-symbolic commonsense reasoner with relation predictors,” _arXiv preprint arXiv:2105.06717_ , 2021.
  * Clark [2015] P. Clark, “Elementary school science and math tests as a driver for ai: Take the aristo challenge!” in _AAAI_. Citeseer, 2015, pp. 4019–4021.
  * Arai [2018] N. Arai, _Artificial intelligence vs. Children who can’t read textbooks_. Toyo Keizai Shimpo, 2018.
  * Lai et al. [2017a] G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy, “RACE: Large-scale ReAding comprehension dataset from examinations,” in _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. Copenhagen, Denmark: Association for Computational Linguistics, Sep. 2017, pp. 785–794.
  * Kumar [1992] V. Kumar, “Algorithms for constraint-satisfaction problems: A survey,” _AI magazine_ , vol. 13, no. 1, pp. 32–32, 1992.
  * Peters et al. [2017] M. E. Peters, W. Ammar, C. Bhagavatula, and R. Power, “Semi-supervised sequence tagging with bidirectional language models,” _arXiv preprint arXiv:1705.00108_ , 2017.
  * Kipf and Welling [2017] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in _International Conference on Learning Representations (ICLR)_ , 2017.
  * Bunel et al. [2018] R. Bunel, M. Hausknecht, J. Devlin, R. Singh, and P. Kohli, “Leveraging grammar and reinforcement learning for neural program synthesis,” _arXiv preprint arXiv:1805.04276_ , 2018.
  * Raffel et al. [2019] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” _arXiv preprint arXiv:1910.10683_ , 2019\.
  * Joshi et al. [2018] V. Joshi, M. Peters, and M. Hopkins, “Extending a parser to distant domains using a few dozen partially annotated examples,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 1190–1199.
  * Russel et al. [2013] S. Russel, P. Norvig _et al._ , _Artificial intelligence: a modern approach_. Pearson Education Limited, 2013\.
  * Zhao et al. [1997] J.-K. Zhao, E. M. Rudnick, and J. H. Patel, “Static logic implication with application to redundancy identification,” in _Proceedings. 15th IEEE VLSI Test Symposium (Cat. No. 97TB100125)_. IEEE, 1997, pp. 288–293.
  * Chen et al. [2020] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _International conference on machine learning_. PMLR, 2020, pp. 1597–1607.
  * Rajpurkar et al. [2016] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “Squad: 100,000+ questions for machine comprehension of text,” _arXiv preprint arXiv:1606.05250_ , 2016\.
  * Richardson et al. [2013] M. Richardson, C. J. Burges, and E. Renshaw, “Mctest: A challenge dataset for the open-domain machine comprehension of text,” in _Proceedings of the 2013 conference on empirical methods in natural language processing_ , 2013, pp. 193–203.
  * Lai et al. [2017b] G. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy, “RACE: large-scale reading comprehension dataset from examinations,” in _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017_ , 2017, pp. 785–794.
  * Zhang et al. [2020] Z. Zhang, J. Yang, and H. Zhao, “Retrospective reader for machine reading comprehension,” _arXiv preprint arXiv:2001.09694_ , 2020.
  * Min et al. [2017] S. Min, M. Seo, and H. Hajishirzi, “Question answering through transfer learning from large fine-grained supervision data,” _arXiv preprint arXiv:1702.02171_ , 2017.
  * Jin et al. [2020] D. Jin, S. Gao, J.-Y. Kao, T. Chung, and D. Hakkani-tur, “Mmm: Multi-stage multi-task learning for multi-choice reading comprehension,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2020, pp. 8010–8017.
  * Beltagy et al. [2020] I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” _arXiv preprint arXiv:2004.05150_ , 2020.
  * Lin et al. [2018] Y. Lin, H. Ji, Z. Liu, and M. Sun, “Denoising distantly supervised open-domain question answering,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_ , 2018, pp. 1736–1745.
  * Pang et al. [2019] L. Pang, Y. Lan, J. Guo, J. Xu, L. Su, and X. Cheng, “HAS-QA: hierarchical answer spans model for open-domain question answering,” in _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , 2019, pp. 6875–6882.
  * lsa [a] “Test prep nerds.” https://testprepnerds.com/lsat/lsat-scores/ Accessed July 1, 2021.
  * lsa [b] “Law school admission council.” https://www.lsac.org/lsat-interpretive-guide/2019-2020 Accessed July 1, 2021\.
  * lsa [c] “Internet legal research group.” https://www.ilrg.com/rankings/law/index/1/desc/LSATLow Accessed July 1, 2021\.
  * Rawlings et al. [1985] C. Rawlings, W. Taylor, J. Nyakairu, J. Fox, and M. J. Sternberg, “Reasoning about protein topology using the logic programming language prolog,” _Journal of Molecular Graphics_ , vol. 3, no. 4, pp. 151–157, 1985.
  * Boulton [1997] R. J. Boulton, “A tool to support formal reasoning about computer languages,” in _International Workshop on Tools and Algorithms for the Construction and Analysis of Systems_. Springer, 1997, pp. 81–95.
  * Fei-Fei et al. [2006] L. Fei-Fei, R. Fergus, and P. Perona, “One-shot learning of object categories,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 28, no. 4, pp. 594–611, 2006.
  * Xiong et al. [2018] W. Xiong, M. Yu, S. Chang, X. Guo, and W. Y. Wang, “One-shot relational learning for knowledge graphs,” _arXiv preprint arXiv:1808.09040_ , 2018\.
  * Du et al. [2019] Z. Du, C. Zhou, M. Ding, H. Yang, and J. Tang, “Cognitive knowledge graph reasoning for one-shot relational learning,” _arXiv preprint arXiv:1906.05489_ , 2019.
  * Bhagavatula et al. [2019] C. Bhagavatula, R. L. Bras, C. Malaviya, K. Sakaguchi, A. Holtzman, H. Rashkin, D. Downey, S. W.-t. Yih, and Y. Choi, “Abductive commonsense reasoning,” _arXiv preprint arXiv:1908.05739_ , 2019.
  * Starr [2019] W. Starr, “Counterfactuals,” 2019.
  * Qin et al. [2019] L. Qin, A. Bosselut, A. Holtzman, C. Bhagavatula, E. Clark, and Y. Choi, “Counterfactual story reasoning and generation,” _arXiv preprint arXiv:1909.04076_ , 2019.
  * Bartha [2013] P. Bartha, “Analogy and analogical reasoning,” 2013.

  *[RC]: reading comprehension