# EST: Evaluating Scientific Thinking
in Artificial Agents

Manjie Xu${}^{2,3\,\star}$
<EMAIL_ADDRESS>&Guangyuan Jiang${}^{1,2,4\,\star}$
<EMAIL_ADDRESS>&Chi Zhang${}^{2,5\,\textrm{{\char 0\relax}}}$
<EMAIL_ADDRESS>&Song-Chun Zhu1,2,5,6,7
<EMAIL_ADDRESS>&Yixin Zhu${}^{1,6\,\textrm{{\char 0\relax}}}$
<EMAIL_ADDRESS>⋆ Manjie Xu and Guangyuan Jiang contributed equally.
${}^{\textrm{{\char 0\relax}}}$corresponding authors
1Institute for Artificial Intelligence, Peking University
2Beijing Institute for General Artificial Intelligence (BIGAI)
3Kuang Yaming Honors School, Nanjing University 4Yuanpei College, Peking
University
5Center for Vision, Cognition, Learning, and Autonomy (VCLA), UCLA
6School of Artificial Intelligence, Peking University 7Department of
Automation, Tsinghua University

###### Abstract

Theoretical ideas and empirical research have shown us a seemingly surprising
result: children, even very young toddlers, demonstrate learning and thinking
in a strikingly similar manner to scientific reasoning in formal research.
Encountering a novel phenomenon, children make hypotheses against data,
conduct causal inference from observation, test their theory via
experimentation, and correct the proposition if inconsistency arises. Rounds
of such processes continue until the underlying mechanism is found. Towards
building machines that can learn and think like people, one natural question
for us to ask is: whether the intelligence we achieve today manages to perform
such a scientific thinking process, and if any, at what level. In this work,
we devise the EST environment for evaluating the scientific thinking ability
in artificial agents. Motivated by the stream of research on causal discovery,
we build our interactive EST environment based on Blicket detection.
Specifically, in each episode of EST, an agent is presented with novel
observations and asked to figure out all objects’ Blicketness. At each time
step, the agent proposes new experiments to validate its hypothesis and
updates its current belief. By evaluating Reinforcement Learning (RL) agents
on both a symbolic and visual version of this task, we notice clear failure of
today’s learning methods in reaching a level of intelligence comparable to
humans. Such inefficacy of learning in scientific thinking calls for future
research in building humanlike intelligence.

##  1 Introduction

> “Man explores the universe around him and calls the adventure Science.”
>
> — Edwin Hubble [22]

Ivan Pavlov might never imagine finding the principles of classical
conditioning during his preliminary research in dogs’ digestive glands.
However, his intellectual curiosity along with unusual energy to explore led
to his pioneering work in the reflex system via scientific control, a typical
method in years of human adventure in science. Surprisingly, such a scientific
approach has been consistently observed in even very young children: Facing a
new phenomenon, children constantly propose new hypotheses, test them by
controlled experiments, and correct the theories if the observation violates
the proposition. During this process, children not only realize when there is
information to be gained, but also formulate a theory about which variables
are relevant, and manage to distinguish the causal roles of each variable via
active experimentation [24, 16, 18, 19, 17, 20, 26, 29, 9, 15]. Scientists’
exploration has been strikingly similar to children’s approach in exploratory
play aforementioned. As Gopnik puts it, “it’s not that children are little
scientists – it’s that scientists are big children” [15].

Figure 1: Children’s exploratory play with the Blicket machine and objects. In
this process, children continuously design new experiments to test whether one
object is a Blicket. After observing an initial context panel with an
activated machine, the child first proposes to test the red cube. However, the
machine is still activated after leaving only the yellow cylinder, indicating
that the cylinder alone can set off the machine, attesting its Blicketness.
The red cube, however, needs an additional test for verification.

The omnipresent scientific thinking in children’s exploratory play [9, 40, 7]
and its close connection to formal scientific methods motivate us to ask a
question in our pursuit of an intelligent machine that can learn and think
like people:

> Do current learning methods produce agents with such a humanlike scientific
> thinking ability? And if so, how good is it?

To answer this question, we introduce the Evaluating Scientific Thinking (EST)
environment for evaluating the scientific thinking ability in artificial
agents. Specifically, the EST environment is grounded on the Blicket detection
setup [18, 26, 29, 37, 38, 43, 44, 50]. The Blicket detection experiment was
initially designed to evaluate children’s causal induction ability. In a
series of experiments, children were presented with a special device, a
Blicket machine. The Blicket machine has a very intuitive working mechanism:
Whenever a Blicket is put on top of it, the device becomes activated, lighting
up and playing music. Test subjects were demonstrated a series of experiments
to show the Blicketness of a set of objects. After exploratory play with the
objects and the machine, the subjects were asked which object was a Blicket
and how to make an intervention to stop/start the machine. Critically, during
the exploratory process, children produce and validate various kinds of
hypotheses until the theory is consistent with all observation [18, 37, 38,
43]. Fig. 1 shows an illustrative example of the baby’s exploratory play in
unravelling how the Blicket machine works and which object is a Blicket.

The proposed EST environment is built following a similar interactive setting
as in children’s exploratory play in the Blicket experiments. Specifically, we
are inspired by recent work in building synthetic reasoning datasets [23, 48,
50, 14, 49, 2] and adopt the CLEVR [23] universe in creating the interactive
environment. Following the recent work of ACRE [50] for causal reasoning, we
borrow the object combinations and the tabletop machine setup. In particular,
an agent is presented with a few observations of objects on Blicket machines
that are either activated or not (referred to as context) at the beginning of
each EST episode. Information for correctly telling which object is a Blicket
is incomplete from the context only. Therefore, to determine the Blicketness
of each object in the shortest amount of time, an agent is tasked to propose a
new experimentation setup that should be carried out in each of the upcoming
time step (referred to as trial), and in the meantime, updating its belief
over which object is an actual Blicket. The correctness of its belief at each
step serves as the motivating signal for the agent, which additionally
receives a constant penalty for every unsuccessful trial. The EST environment
is built with a clear mind in evaluating visual reasoning in scientific
thinking and therefore does not emphasize motor control for the agent. Hence,
we simplify the action space by quantifying it into a discrete space
representing the selection of each object for further experimentation. In
summary, the EST environment is designed with the following principles:

  * •

The task is light in visual effect and motor interaction but heavy in visual
reasoning and exploration. Every EST scene only depicts elementary 3D shapes
as in ACRE [50], and the agent’s action is abstracted out into object
selection instead of involving sensorimotor complexity of the body and hand
movement, making it easier for practitioners to concentrate on reasoning.

  * •

EST is more than an interactive game. Not all interactive processes are
interventional. The task is designed to focus on generating and testing
interventions to disentangle confounders via proposal making, which goes
beyond simple causal reasoning testbeds for RL.

In the experiments, the EST environment serves as a testbed for evaluating the
scientific thinking capability of artificial agents trained via today’s state-
of-the-art Reinforcement Learning (RL) algorithms [34, 27, 13, 31, 41]. Not
only do we benchmark RL algorithms with visual input from the rendering
engine, but also with the underlying symbolic representation of the
environment. However, we note that the environment is challenging enough for
agents with even the ground-truth symbolic representation; visual complexity
further adds to the difficulty of the problem. Further comparing the
performance from the heuristics algorithms and that from the RL agents,
prominent failure of today’s artificial agents in scientific thinking becomes
even more evident, calling for future investigation into building intelligence
that can learn and think like people, and more for fulfilling a dream to
create an agent that can assist in formal scientific research.

To sum up, our work makes the following contributions:

  * •

We introduce the EST environment for evaluating the scientific thinking
ability of artificial agents. In each episode, an agent is presented with a
few Blicket experiments. With an incentive to understand the underlying
working mechanism of the Blicket machine and distinguish the actual Blickets,
the agent proposes new experiments to verify and update its belief of
Blicketness for each object based on the experimental outcome. The agent is
eventually rewarded for correctly finding all Blickets in the shortest amount
of time.

  * •

We carefully designed the EST environment to ensure (i) its simplicity in
perception (but not lack of) and sensorimotor control, and (ii) its richness
in reasoning and discovery. EST introduces a new and minimal scientific
thinking paradigm of exploration and discovery in visual reasoning that offers
more than an interactive game or a causal reasoning dataset.

  * •

Using the established EST environment, we benchmark current RL learning
agents’ capability in performing this challenging task of scientific
reasoning. Analyzing the results from the trained agents and heuristics, we
note that (i) the visual complexity is not the core difficulty in this task,
which echos our design principle to minimize visual complexity, (ii) causal
discovery in EST requires strong reasoning abilities and depends on how to
efficiently make trial and intervene, and (iii) today’s learning agents fall
short in developing abilities in efficient scientific discovery.

##  2 Related Work

#### Visual Reasoning

A range of visual comprehension and reasoning tasks has been proposed
recently. On the basis of a series of Visual Question Answering (VQA)
benchmarks [1, 25, 42, 57], Johnson _et al_. [23] use synthetic images
depicting simple 3D shapes to scrutinize a suite of VQA models and discover
their potential weaknesses. From a causal and physical perspective, the video
dataset of CLEVRER was introduced by Yi _et al_. [48] to investigate the
performance of state-of-the-art (SOTA) models on learning complex spatial-
temporal-causal structures from interacting objects in a scene. At a more
abstract level, model performance in human Intelligence Quotient (IQ) tests
has been studied; Barrett _et al_. [3] and Zhang _et al_. [49] archived
datasets inspired by the Raven’s Progressive Matrices (RPM) [6, 11], featuring
reasoning on the hidden spatial-temporal transformation from a limited number
of context panels. Approaches for this abstract reasoning task range from the
neural end towards neuro-symbolism over the years [33, 21, 51, 52, 45, 39, 53,
47]. Inspired by Blicket detection and the problem of causal induction [18,
19], the ACRE dataset [50] was presented as a way to systematically evaluate
current vision systems’ capability in causal induction. It is worth noting a
visual reasoning method based on object-centric representation and self-
attention [10] has obtained notable performance on various visual reasoning
domains including CLEVRER [48], CATER [14], and ACRE [50], with the help of
unsupervised object segmentation algorithms [5].

While the proposed EST environment is based on the Blicket setup and visually
similar to the ACRE dataset, the new interactive environment emphasizes a
drastically different problem: Apart from figuring out the hidden causal
factors, the agent is also tasked with making maximally meaningful exploration
from an initial setup of high uncertainty and aggregating the collected
information to update its belief and guide its next decision, resembling the
procedure in scientific research.

#### Causal Discovery

Causality still remains a central problem for current artificial intelligence
to solve. Earlier methods approach the problem from video understanding [12,
56], image processing [28], or counterfactual prediction [32]. More recently,
researchers have started to investigate the problem from a reasoning
perspective. In particular, CoPhy [2] introduce a dataset for understanding
how objects would behave under imagined scenarios; CLEVRER [48] frame the
problem in a question-answering setup with explanatory, predictive, and
counterfactual queries; ACRE [49] propose the task of causal induction that
required a model to answer if a factor resulted in the eventual effect;
Alchemy [46] introduce a new benchmark with latent causal structures for meta-
RL research.

#### Comparison

EST differs from previous works by introducing scientific thinking and active
causal discovery into visual reasoning. We compare EST with the existing
dataset benchmarks in Tab. 1. As an interactive environment, EST enables
agents to gain information actively from the environment to test and revise
their hypotheses, fundamentally distinctive from classic visual reasoning
tasks such as CLEVR. The causal structure in EST is rich and easy for
intervention, going beyond the traditional paradigm of passive observation and
reasoning. In addition, EST also focuses on active exploration and efficiency
in reasoning; one cannot predict two steps ahead unless it interacts with the
environment and collects new observations, akin to the scientific discovery
process.

Table 1: Comparison between EST and other related visual reasoning benchmarks in terms of task (classification, video question answering, and game), size (number of scenarios), and input format. EST introduces a spatial-temporal-causal reasoning task, which allows intervention and belief update. It aims at few-shot problem solving for fast experimentation and reasoning. Benchmarks | Task | Size | Format | Temporal | Interventional | Belief | Few-shot
---|---|---|---|---|---|---|---
CLEVR [23] | vqa | 100k | image | ✗ | ✗ | ✗ | ✗
CLEVRER [48] | vqa | 20k | video | ✓ | ✗ | ✗ | ✗
CATER [14] | cls | 5.5k | video | ✓ | ✗ | ✗ | ✗
ACRE [50] | cls | 30k | image | ✓ | ✗ | ✗ | ✓
Alchemy [46] | game | - | image/symbol | ✓ | ✓ | ✗ | ✗
EST (Ours) | game | - | image/symbol | ✓ | ✓ | ✓ | ✓

Notably, while adopting a similar setting with the Blicket experiment in ACRE,
EST is more than interactive ACRE: By abstracting out the sensorimotor
complexity, EST places emphasis on the under-explored topic of interventional
causal discovery. An agent not only needs to perform causal reasoning to
figure out the hidden relations based on observation, but also and more
importantly, to propose interventional trials to collect new information in
the most efficient way to understand the system. The discovery process is at
the core of scientific thinking, bringing a new problem for agents to generate
and test interventions to disentangle confounders in causal discovery.

##  3 The EST Environment

We build the proposed EST environment under the OpenAI Gym API [4]. Different
from other vision-based RL platforms, the EST interactive environment aims at
fostering an agent to understand the visual information collected and
disentangle the causal factors underlying the observation by actively
proposing new experiments to demystify the phenomenon, validate its
hypothesis, and correct its belief. Following earlier works, the visual domain
of EST is consistent with the CLEVR [23] and ACRE [50] universe, where a
Blicket machine sits on a tabletop with three-point lighting. Lying upon the
Blicket machine are objects with potential Blicketness, which can be inferred
from the observation of the machine’s activation pattern across frames. The
objects’ attributes vary in shapes (cube, sphere, or cylinder), materials
(metal or rubber), and colors (gray, red, blue, green, brown, cyan, purple, or
yellow). We signal activation of the Blicket machine by lighting it up.

An agent in the EST environment is tasked to figure out which objects are
Blickets. At the start of each episode, the agent is presented with several
initial observations of various object combinations (referred to as contexts).
The context alone is insufficient to solve Blicketness for all objects. Hence,
in each following step (referred to as trials), the agent proposes a new
experiment of a specific object combination and updates its belief of
Blicketness based on the outcome of experiments.

An episode will be terminated if the agent works out the Blicketness of all
objects or consume all $T=10$ time steps. The agent is, therefore, rewarded at
each step based on the correctness of its belief, and as a way to encourage
efficient exploration, penalized every step it fails the problem. Please refer
to Fig. 2 for a simplified example of an episode in the EST environment.

Figure 2: A simple example episode in the EST environment. At the beginning of
the episode, an agent is given a set of context panels (4 in an actual
instance) as an introduction to the problem. Next, the agent is motivated to
figure out which object is a Blicket by proposing new experiments to validate
its hypothesis and update its belief. The agent will receive a high reward if
all Blickets have been found out.

In the following, we discuss the design choices in detail.

#### Context

To instantiate a problem at each episode, we first sample $9$ unique shape-
material-color combinations from the pool to create the objects, the
Blicketness of which is for the agent to solve. We then randomly assign $n$
objects ($3\leq n\leq 8$) to be Blickets. To build a context panel, we pick
$m$ objects ($2\leq m\leq 6$) out of the $9$ at random and place it onto a
Blicket machine. The status of the Blicket machine can be determined by
checking if the $m$ sampled objects contain a Blicket. We repeat the sampling
process for $4$ times to create a set of context panels as the initial
observation of the agent. This process resembles the demonstration researchers
presented to test subjects in the psychology study. Of note, during the
sampling process, we follow ACRE in crafting the sampling probability to
enable queries of direct, indirect, screening-off, and backward-blocking [50]
in order to invalidate the simplistic covariation-based solving strategy:
Counting co-occurrence of an object with an activated Blicket machine will not
lead to a correct answer in this scenario.

#### Trial

After observing the context panels, the agent forms an initial belief of
Blicketness over all the objects. At each time step $t$ that follows, the
agent observes the outcome of the previous experiment, updates its own belief
about the Blickets, and, if still uncertain about some object(s), proposes a
new experiment to test. This process resembles the exploratory play in
children’s problem solving.

#### Observation Space

We consider two forms of observation for EST: the symbolic version and the
pixel version. For the symbol-input version, we use a binary vector to
describe the state of the scene, where the first $9$ entries represent if the
object of interest is present or not and the last entry if the Blicket machine
is on or off. For the pixel-input version, we feed the scene description into
Blender [8] and use EEVEE to render images in real-time. For efficiency, we
only render images of shape $160\times 120$. For the pixel-input version, we
hypothesize that an agent could discover causalities at many possible levels
of abstraction; they could be defined on an attribute basis (_e.g_., color,
shape, material), an object basis, or on a group basis. The pixel version adds
more confounding variables into the problem, whereas a well-defined symbolic
version drastically simplifies it. Considerable studies in psychology and
cognition have also shown that visual stimuli are essential in causal
discovery; we perceive causality automatically and irresistibly with tight
control of spatiotemporal patterns of motion [35], which concludes the
significance of visual EST.

#### Action Space

The agent’s action space is composed of two sub-spaces: the trial sub-space
and the belief sub-space. The trial sub-space stands for the object selection
to perform experiment on in the next trial step. The belief sub-space denotes
the agent’s belief of Blicketness after analyzing all experimental results up
to the time step. For ease of training, we soften the binary sub-spaces into
continuous values in $[0,1]$, interpreting each value in the trial sub-space
as the probability of selecting that object and that in the belief sub-space
as the probability of being a Blicket.

#### Reward

The agent is rewarded based on the correctness of its belief and the
efficiency of its trial. If within the total $T$ time steps the agent figures
out the Blicketness for every object, it will receive a constant reward of
$20$. For every step it fails the problem, a small penalty of $-1$ will be
sent. We also introduce an auxiliary reward based on the partial correctness
of its belief. Specifically, at each time step, we calculate an oracle
Blicketness belief based on all the experimental results the agent has
observed via search: We search for all possible Blicket assignments that are
consistent with the observation and count the probability of an object
appearing in a feasible assignment. We then use the negative Jensen-Shannon
Distance (JSD) between the current belief and the oracle belief as the
motivating signal. Note that as negative JSD is bounded by $[-1,0]$, the
reward an agent can receive ranges from $-20$ (the agent completely fails at
each step and consumes all time) to $20$ (the agent instantly solves the
problem after observing the context).

##  4 Benchmarking EST

We detail two groups of models for the proposed EST environment: (i)
heuristics methods, and (ii) RL methods under the Partially Observable Markov
Decision Process (POMDP) formulation.

###  4.1 Heuristics Methods

For the heuristics methods, we consider five different agents. Please note
that all the heuristics agents are run on the symbol-input version of the EST
environment.

#### Random Agent

We consider a simple random agent as a baseline. The agent only randomly
samples belief and the next trial from a uniform distribution without
processing any observation. As a result, the agent does not propose any
meaningful trials during the interaction.

#### Bayes Agent

Numerous studies have shown that children and adults propose hypotheses to
explain the forms of causal relationships in terms of a Bayesian model [30,
15, 18, 19, 17, 20, 16]. Therefore, we propose to use a Bayes agent for
evaluation as well. Specifically, we implement a Naive Bayes classifier based
on the Bernoulli distribution. The classifier predicts the Blicketness for
each object via the Bayes’ rule using the observation collected to this time
step as training data. The agent then proposes a random trial to gather more
information.

#### Naive Agent

The most intuitive way to generate trials, though not efficient, is to test
one object at a time. If an agent performed this strategy, there would be no
confounding factors, and reasoning would be greatly simplified. Cook _et al_.
[9] reveals that, to learn a latent causal structure, children without formal
science education tend to perform actions that isolate relevant variables in
the naive strategy aforementioned. Following this empirical observation, we
implement a naive agent to benchmark our environment. As each new experiment
only contains one object, the agent updates its belief for the Blicketness of
the object with certainty and only randomly selects an object it has not
tested in the next round. Though feasible and certain, this agent results in
low efficiency in trials.

#### Search-based Random Agent

We also propose a search-based random agent. The agent is non-parametric in
the sense that the agent assumes knowledge of the causal structure and at each
time step searches for all possible Blicket assignments that are consistent
with observation up to now. The agent then computes the frequency of the
appearance of an object in the set of possible Blicket assignments and
randomly selects a set of objects to test.

#### Search-based Naive Agent

The search-based naive agent follows the same design as the search-based
random agent; the prior belief probability is computed in the exact same way.
However, this naive version uses the probability distribution to select
objects for the next round. Specifically, we apply the naive strategy and only
select the object with the highest uncertainty to test.

###  4.2 RL Methods

The EST environment can be modelled as a POMDP for RL training. Formally, the
POMDP problem is a tuple $(S,A,T,R,\Omega,O,\gamma)$, where $S$, $A$, $T$,
$R$, $\Omega$ are the state space, the action space, the transition
probability, the reward function, and the observation space, respectively.
Note that the state space covers the ground-truth belief up to each time step
$t$. The action space, the observation space, and the reward function have
been discussed in Sec. 3. $O$ is the observation generator for each state-
action pair $(s,a)$ and $\gamma$ is the discount factor (set to $0.99$). As
mentioned in Sec. 3, we consider two versions of the observation space: the
symbol-input version and the pixel-input version. For the POMDP formulation of
the RL algorithms, we use a recurrent agent for learning, as the underlying
state is not directly observable and has to be inferred from the history of
the observation.

The POMDP could also be transformed into an Markov Decision Process (MDP) if
we know the underlying state at each time step. Therefore, we also consider a
feed-forward architecture for this formulation: if one regards the belief sub-
space from the agent’s output as the true state space, we can use the belief
from the previous time step and the new observation to propose a new trial and
update the belief. Note that despite a feed-forward architecture being used,
the entire process is still recurrent as the belief state’s dependency is
traced back to the history observation. See Fig. 3 for a graphical
illustration of the general RL architecture we use for benchmarking EST. Here
we only depict the pixel-input version environment, where the CNN encoder
shares the parameters and the agent architecture is either an LSTM module that
takes in the history observation or a feed-forward module that uses the
agent’s belief as an approximate.

Figure 3: The general RL architecture for benchmarking EST. The architecture
follows the actor-critic design; both the policy and value function are
represented with neural networks. For the pixel-input version of the
environment depicted here, we use a shared CNN encoder to extract visual
features. The symbol-input version differs from the pixel version in that it
provides a binary vector description that is processed by an MLP.

As the action space has been softened, we consider several classic continuous
RL methods for evaluation. Specifically, we benchmark Deep Deterministic
Policy Gradient (DDPG) [27], Twin Delayed Deep Deterministic Policy Gradient
(TD-3) [13], and Proximal Policy Optimization (PPO) [36]. For a preliminary
study, we also test a RL agent that incorporates NOTEARS [54, 55] as its
reasoning component. Please refer to the supplementary material for details of
these methods.

##  5 Experiment

###  5.1 Experimental Setup

Table 2: Performance of heuristic models on the symbol-input version of EST
environment. We report two evaluation metrics: the problem solving accuracy
denoted as Acc and the total reward denoted as $R$. The context column records
results when the agent only observes the context panels, whereas the episode
column after the entire episode.

Model | Context | Episode
---|---|---
Acc | $R$ | Acc | $R$
Random | 0.71% | -5.67 | 2.02% | -14.11
Bayes | 0.24% | -4.30 | 3.39% | -11.15
Naive | 1.71% | -3.95 | 25.85% | -5.03
Search-Random | 0.24% | -4.00 | 2.70% | -9.36
Search-Naive | 0.40% | -3.91 | 36.83% | -1.65

We run experiments on the EST environment with the agents and their variants
aforementioned. For evaluation metrics, we report the agent’s average reward
together with problem solving accuracy over $10^{4}$ random test episodes. The
average reward measures how the model performs in terms of reasoning and trial
efficiency, while the problem solving accuracy is computed by counting the
number of episodes where an agent correctly figures out Blicketness for all
objects. Apart from the final results after finishing an entire episode, we
also evaluate how the agent performs after observing the initial context
panels only: a metric on how the model understands the problem without any
trials.

###  5.2 Performance of Agents

Table 3: Performance of RL models on the symbol-input version of EST
environment. The evaluation metrics are the same as in Tab. 2. FF denotes the
network with the feed-forward architecture, whereas Re using the recurrent
network as the agent.

Model | Context | Episode
---|---|---
Acc | $R$ | Acc | $R$
DDPG-FF | 6.99% | -3.21 | 11.86% | -8.73
DDPG-Re | 2.51% | -4.20 | 8.51% | -10.32
TD-3-FF | 5.03% | -3.32 | 6.39% | -9.55
TD-3-Re | 2.29% | -4.64 | 7.11% | -11.60
PPO | 1.10% | -4.11 | 2.70% | -10.60
NOTEARS | 3.21% | -3.83 | 4.30% | -9.99

#### Heuristic Agents

Tab. 2 shows the performance of the heuristic agents in the proposed EST
environment. In general, we note that the more information collected, the
better the performance of the agents: The low accuracy after the context
panels also verifies that additional trials are necessary for solving the
entire problem. However, both the random agent and the Bayes agent fail in
this task, reaching close to chance problem solving accuracy and very negative
rewards. The comparison among the random agent, the naive agent, and their
search-based variants indicates that both the search component and the naive
strategy are beneficial for a symbolic approach. Without the search component,
the naive agent does not build a reliable belief over which object is more
likely to be a Blicket; without the naive strategy, an agent only randomly
collects new experiments, doing little to no help in demystifying the
phenomenon.

#### Symbol-Input RL Agents

Tab. 3 summarises the performance of symbol-input RL agents. Comparing the
DDPG agent and the TD-3 agent, we note that contradictory to the conventional
wisdom; the DDPG agent attains slightly higher or on-par performance with the
TD-3 agent. One interesting observation is that the agents generally propose
very inefficient trials; the final reward decreases as time goes by, though it
becomes more likely for an agent to stumble on a solution by chance.

Table 4: Performance of RL models on the pixel-input version of EST environment. The evaluation metrics and notations are the same as in Tab. 3. Model | Context | Episode
---|---|---
Acc | $R$ | Acc | $R$
DDPG-FF | 0.30% | -4.19 | 0.37% | -10.76
DDPG-Re | 0.47% | -4.15 | 0.30% | -10.75
TD-3-FF | 0.47% | -4.16 | 0.43% | -10.78
TD-3-Re | 0.30% | -4.19 | 0.37% | -10.80

#### Pixel-Input RL Agents

With poor performance of PPO and NOTEARS, we only consider DDPG and TD-3 in
the pixel-input experiments. As shown in Tab. 4, all the models tested in the
pixel-input version of EST catastrophically fail with random level problem-
solving accuracy and total reward. Surprisingly, while their rewards are
slightly higher than the random agent, their problem-solving accuracy is even
worse, let alone the conspicuously large gap from the performance under the
symbol-input circumstance. We carefully check the output of these agents and
found that belief and trial predicted by agents oscillate around $0.5$,
indicating that they have difficulty learning to understand EST environment
from pixel-level input. Our results show that current models still struggle in
understanding the complexity of perceived causality, and traditional RL
methods should be further improved in representation learning to help
scientific discovery.

Figure 4: (a), (b): Number of steps an agent takes to successfully solve an
EST episode. (a): results for the search-based naive agent. (b): results for
the DDPG-FF agent. (c): Box plot of the number of objects the DDPG-FF agent
proposes at each time step.

#### Comparison

Comparing experimental results in heuristic agents, symbol-input RL agents,
and pixel-input RL agents, we note that the visual complexity adds additional
challenges into the problem. Besides, neither the heuristics nor the RL-
learned policies are effective in proposing meaningful experiments to validate
and update belief. Among others, the search-based naive agent is relatively
effective in solving the problem. However, its simple trial policy bears
potential for improvement. Looking into the failure in the pixel-input agents,
we hypothesize that causal representation learning could mitigate its learning
inefficacy: Not only do we need features to distinguish between different
experiments, but also those that support in-depth understanding of the
environment are helpful for learning efficient scientific exploration. Of
note, with the searched belief, both the naive agent and RL agents enjoy
remarkable performance gain. In addition, the learned policies achieve better
results compared to the manually defined naive one, showing that with improved
internal representation, the exploration in RL bears the potential to find a
better learning and thinking strategy. Furthermore, the naive trial policy
archives significantly better results compared to the random trial policy with
oracle belief. Taken together, these results imply that the EST problem could
not be solved just using the random experimentation, but needs careful trial
selection.

#### Analysis

With pixel-input RL agents reaching only random-level performance, we analyze
the best symbolic model (_i.e_., search-based naive agent) and the best
symbol-input RL model (_i.e_., DDPG-FF). Fig. 4 shows the number of steps the
agents take to successfully solve a problem and the number of objects proposed
at each time step in the $10^{4}$ random test episodes. The search-based
method makes reasonable choices at each step and as the information collected
amounts, the problem-solving rate becomes increasingly higher. The DDPG-FF
agent’s performance is not intuitive; most of the problems are solved with a
few number of steps or nearly exercise all time steps. We also note that the
model can only solve problems with a simple setup (only a few objects in the
context panels and most are non-Blickets) in a few steps and consistently
selects around 4 to 5 objects to test at earlier steps in the process but
starts to wildly pick more objects towards the end. Combining the analysis
from the number of steps it takes to solve the problem, we believe that the
agent in the early steps learns to use a simpler strategy to solve the problem
but becomes increasingly confused as ineffective evidence mounts towards the
end, showing very limited scientific thinking ability.

##  6 Conclusion and Discussion

#### Conclusion

In this work, we propose the EST environment, an interactive testbed for
evaluating the scientific thinking capability of artificial agents. Inspired
by the theoretical proposition and the empirical observation of infants being
little scientists, the newly introduced EST mimics the classic Blicket
detection experiment but intentionally simplifies the sensorimotor control by
abstracting it out into a discrete space of object selection. Specifically, in
the EST environment, an agent is presented with a few context panels showing
the working mechanism of the Blicket machine: The machine lights up when a
Blicket is placed on it. In the following trials, the agent is tasked with
finding all the Blickets in a set of pre-selected objects in the shortest
amount of time, in each step validating its hypothesis of Blicketness for each
object, proposing new experiments to check consistency with the observation,
and correcting the wrong belief if contradictory. EST’s design not only
requires the agent to effectively update its belief based on the information
collected so far, but also necessitates the capability to come up with
maximally efficient new experiments to disentangle confounding factors. A
successful agent in this environment shall not only know how to passively
reason, but also actively explore from both visual and symbolic input.

#### Do current learning methods produce agents with such a humanlike
scientific thinking ability?

We measure current artificial agents’ scientific thinking ability in our EST
environment. Two groups of algorithms have been evaluated: the heuristic
methods and RL methods. Due to the nature of the task being a POMDP, we adapt
the RL agents by either treating the belief from the previous time step as an
approximate or simply recruiting a recurrent agent.

Analyzing the results from the heuristic agents, we note that two general
strategies are effective in helping an agent correctly figure out Blicketness
for all objects: the naive trial proposal that only selects one object at a
time step and the search-based belief computation. The naive selection
effectively gathers information by focusing only on the most uncertain object,
serving as an intuitive but constructive method similar to scientific control
widely conducted in today’s scientific research. Based on such a strategy, the
search-based agent summarizes experimental results by traversing the entire
space and computes the feasibility of an object being a Blicket.

RL agents’ incapability in scientific reasoning becomes relatively evident in
this benchmark. Under the symbol-input environment, which also serves as a
diagnostic tool and for upper-bounding neuro-symbolic methods, we find neither
the feed-forward agents nor the recurrent agents can generate meaningful
belief and suggest de-confounding experiments, rendering them very confused
during the problem solving process. Looking deep into how they perform, we
note that the best RL agent with the symbol input mostly finishes only the
straightforward problem instances with few objects and the majority of them
being non-Blickets. The agent cannot demystify the correlation by singling out
objects of uncertainty and only nearly randomly combine them with others. The
visual complexity of the problem further adds to the challenge of this task:
With visual input, all of the RL agents only perform at a randomly level.

#### Future directions

Concluding this manuscript is not the end, but rather the beginning of our
pursuit of an intelligent agent that can learn and think like people in the
scientific setting. With today’s agents catastrophically failing in this
problem, how do humans, even very young children, successfully complete this
task of scientific nature without formal scientific training? What role does
training from nurture play in the process? And how to incubate an artificial
agent with this ability through interaction with the environment? With many
questions unanswered, we hope this preliminary work would motivate further
research.

## References

  * Antol et al., [2015] Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D. (2015).  Vqa: Visual question answering.  In International Conference on Computer Vision (ICCV).
  * Baradel et al., [2019] Baradel, F., Neverova, N., Mille, J., Mori, G., and Wolf, C. (2019).  Cophy: Counterfactual learning of physical dynamics.  arXiv preprint arXiv:1909.12000.
  * Barrett et al., [2018] Barrett, D., Hill, F., Santoro, A., Morcos, A., and Lillicrap, T. (2018).  Measuring abstract reasoning in neural networks.  In International Conference on Machine Learning (ICML).
  * Brockman et al., [2016] Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016).  Openai gym.  arXiv preprint arXiv:1606.01540.
  * Burgess et al., [2019] Burgess, C. P., Matthey, L., Watters, N., Kabra, R., Higgins, I., Botvinick, M., and Lerchner, A. (2019).  Monet: Unsupervised scene decomposition and representation.  arXiv preprint arXiv:1901.11390.
  * Carpenter et al., [1990] Carpenter, P. A., Just, M. A., and Shell, P. (1990).  What one intelligence test measures: a theoretical account of the processing in the raven progressive matrices test.  Psychological Review, 97(3):404.
  * Chu and Schulz, [2020] Chu, J. and Schulz, L. E. (2020).  Play, curiosity, and cognition.  Annual Review of Developmental Psychology, 2:317–343.
  * Community, [2016] Community, B. O. (2016).  Blender–a 3d modelling and rendering package.
  * Cook et al., [2011] Cook, C., Goodman, N. D., and Schulz, L. E. (2011).  Where science starts: Spontaneous experiments in preschoolers’ exploratory play.  Cognition, 120(3):341–349.
  * Ding et al., [2021] Ding, D., Hill, F., Santoro, A., Reynolds, M., and Botvinick, M. (2021).  Attention over learned object embeddings enables complex visual reasoning.  In Advances in Neural Information Processing Systems (NeurIPS).
  * ea Raven, [1938] ea Raven, J. (1938).  Raven’s progressive matrices.  Western Psychological Services, 2:5.
  * Fire and Zhu, [2015] Fire, A. and Zhu, S.-C. (2015).  Learning perceptual causality from video.  ACM Transactions on Intelligent Systems and Technology (TIST), 7(2):1–22.
  * Fujimoto et al., [2018] Fujimoto, S., Hoof, H., and Meger, D. (2018).  Addressing function approximation error in actor-critic methods.  In International Conference on Machine Learning (ICML).
  * Girdhar and Ramanan, [2019] Girdhar, R. and Ramanan, D. (2019).  Cater: A diagnostic dataset for compositional actions & temporal reasoning.  In International Conference on Learning Representations (ICLR).
  * Gopnik, [1996] Gopnik, A. (1996).  The scientist as child.  Philosophy of Science, 63(4):485–514.
  * Gopnik, [2012] Gopnik, A. (2012).  Scientific thinking in young children: Theoretical advances, empirical research, and policy implications.  Science, 337(6102):1623–1627.
  * Gopnik et al., [2004] Gopnik, A., Glymour, C., Sobel, D. M., Schulz, L. E., Kushnir, T., and Danks, D. (2004).  A theory of causal learning in children: causal maps and bayes nets.  Psychological Review, 111(1):3.
  * Gopnik and Sobel, [2000] Gopnik, A. and Sobel, D. M. (2000).  Detecting blickets: How young children use information about novel causal powers in categorization and induction.  Child Development, 71(5):1205–1222.
  * Gopnik et al., [2001] Gopnik, A., Sobel, D. M., Schulz, L. E., and Glymour, C. (2001).  Causal learning mechanisms in very young children: two-, three-, and four-year-olds infer causal relations from patterns of variation and covariation.  Developmental Psychology, 37(5):620.
  * Gopnik and Wellman, [2012] Gopnik, A. and Wellman, H. M. (2012).  Reconstructing constructivism: Causal models, bayesian learning mechanisms, and the theory theory.  Psychological Bulletin, 138(6):1085.
  * Hill et al., [2019] Hill, F., Santoro, A., Barrett, D. G., Morcos, A. S., and Lillicrap, T. (2019).  Learning to make analogies by contrasting abstract relational structure.  arXiv preprint arXiv:1902.00120.
  * Hubble, [1954] Hubble, E. (1954).  The nature of science, and other lectures.  Huntington Library.
  * Johnson et al., [2017] Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R. (2017).  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In Conference on Computer Vision and Pattern Recognition (CVPR).
  * Keil, [2011] Keil, F. C. (2011).  Science starts early.  Science, 331(6020):1022–1023.
  * Krishna et al., [2017] Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017).  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International Journal of Computer Vision (IJCV), 123(1):32–73.
  * Kushnir and Gopnik, [2007] Kushnir, T. and Gopnik, A. (2007).  Conditional probability versus spatial contiguity in causal learning: Preschoolers use new contingency evidence to overcome prior spatial assumptions.  Developmental Psychology, 43(1):186.
  * Lillicrap et al., [2015] Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015).  Continuous control with deep reinforcement learning.  arXiv preprint arXiv:1509.02971.
  * Lopez-Paz et al., [2017] Lopez-Paz, D., Nishihara, R., Chintala, S., Scholkopf, B., and Bottou, L. (2017).  Discovering causal signals in images.  In Conference on Computer Vision and Pattern Recognition (CVPR).
  * Lucas et al., [2014] Lucas, C. G., Bridgers, S., Griffiths, T. L., and Gopnik, A. (2014).  When children are better (or at least more open-minded) learners than adults: Developmental differences in learning the forms of causal relationships.  Cognition, 131(2):284–299.
  * Lucas and Griffiths, [2010] Lucas, C. G. and Griffiths, T. L. (2010).  Learning the form of causal relationships using hierarchical bayesian models.  Cognitive Science, 34(1):113–147.
  * Mnih et al., [2015] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015).  Human-level control through deep reinforcement learning.  Nature, 518(7540):529–533.
  * Mottaghi et al., [2016] Mottaghi, R., Rastegari, M., Gupta, A., and Farhadi, A. (2016).  “what happens if…” learning to predict the effect of forces in images.  In European Conference on Computer Vision (ECCV).
  * Santoro et al., [2017] Santoro, A., Raposo, D., Barrett, D. G., Malinowski, M., Pascanu, R., Battaglia, P., and Lillicrap, T. (2017).  A simple neural network module for relational reasoning.  arXiv preprint arXiv:1706.01427.
  * Schmidhuber, [2015] Schmidhuber, J. (2015).  Deep learning in neural networks: An overview.  Neural Networks, 61:85–117.
  * Scholl and Tremoulet, [2000] Scholl, B. J. and Tremoulet, P. D. (2000).  Perceptual causality and animacy.  Trends in Cognitive Sciences, 4(8):299–309.
  * Schulman et al., [2017] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017).  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347.
  * Sobel and Kirkham, [2006] Sobel, D. M. and Kirkham, N. Z. (2006).  Blickets and babies: the development of causal reasoning in toddlers and infants.  Developmental Psychology, 42(6):1103.
  * Sobel et al., [2004] Sobel, D. M., Tenenbaum, J. B., and Gopnik, A. (2004).  Children’s causal inferences from indirect evidence: Backwards blocking and bayesian reasoning in preschoolers.  Cognitive Ccience, 28(3):303–333.
  * Spratley et al., [2020] Spratley, S., Ehinger, K., and Miller, T. (2020).  A closer look at generalisation in raven.  In European Conference on Computer Vision (ECCV).
  * Stahl and Feigenson, [2015] Stahl, A. E. and Feigenson, L. (2015).  Observing the unexpected enhances infants’ learning and exploration.  Science, 348(6230):91–94.
  * Sutton and Barto, [2018] Sutton, R. S. and Barto, A. G. (2018).  Reinforcement learning: An introduction.  MIT press.
  * Tapaswi et al., [2016] Tapaswi, M., Zhu, Y., Stiefelhagen, R., Torralba, A., Urtasun, R., and Fidler, S. (2016).  Movieqa: Understanding stories in movies through question-answering.  In Conference on Computer Vision and Pattern Recognition (CVPR).
  * Walker and Gopnik, [2014] Walker, C. M. and Gopnik, A. (2014).  Toddlers infer higher-order relational principles in causal learning.  Psychological Science, 25(1):161–169.
  * Walker and Gopnik, [2017] Walker, C. M. and Gopnik, A. (2017).  Discriminating relational and perceptual judgments: Evidence from human toddlers.  Cognition, 166:23–27.
  * Wang et al., [2019] Wang, D., Jamnik, M., and Lio, P. (2019).  Abstract diagrammatic reasoning with multiplex graph networks.  In International Conference on Learning Representations (ICLR).
  * Wang et al., [2021] Wang, J. X., King, M., Porcel, N. P. M., Kurth-Nelson, Z., Zhu, T., Deck, C., Choy, P., Cassin, M., Reynolds, M., Song, H. F., et al. (2021).  Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents.  In Advances in Neural Information Processing Systems (NeurIPS).
  * Wu et al., [2020] Wu, Y., Dong, H., Grosse, R., and Ba, J. (2020).  The scattering compositional learner: Discovering objects, attributes, relationships in analogical reasoning.  arXiv preprint arXiv:2007.04212.
  * Yi et al., [2019] Yi, K., Gan, C., Li, Y., Kohli, P., Wu, J., Torralba, A., and Tenenbaum, J. B. (2019).  Clevrer: Collision events for video representation and reasoning.  In International Conference on Learning Representations (ICLR).
  * [49] Zhang, C., Gao, F., Jia, B., Zhu, Y., and Zhu, S.-C. (2019a).  Raven: A dataset for relational and analogical visual reasoning.  In Conference on Computer Vision and Pattern Recognition (CVPR).
  * [50] Zhang, C., Jia, B., Edmonds, M., Zhu, S.-C., and Zhu, Y. (2021a).  Acre: Abstract causal reasoning beyond covariation.  In Conference on Computer Vision and Pattern Recognition (CVPR).
  * [51] Zhang, C., Jia, B., Gao, F., Zhu, Y., Lu, H., and Zhu, S.-C. (2019b).  Learning perceptual inference by contrasting.  Advances in Neural Information Processing Systems (NeurIPS).
  * [52] Zhang, C., Jia, B., Zhu, S.-C., and Zhu, Y. (2021b).  Abstract spatial-temporal reasoning via probabilistic abduction and execution.  In Conference on Computer Vision and Pattern Recognition (CVPR).
  * Zheng et al., [2019] Zheng, K., Zha, Z.-J., and Wei, W. (2019).  Abstract reasoning with distracting features.  Advances in Neural Information Processing Systems (NeurIPS).
  * Zheng et al., [2018] Zheng, X., Aragam, B., Ravikumar, P. K., and Xing, E. P. (2018).  Dags with no tears: Continuous optimization for structure learning.  Advances in Neural Information Processing Systems (NeurIPS), 31\.
  * Zheng et al., [2020] Zheng, X., Dan, C., Aragam, B., Ravikumar, P., and Xing, E. (2020).  Learning sparse nonparametric dags.  In International Conference on Artificial Intelligence and Statistics (AISTATS).
  * Zhu et al., [2020] Zhu, Y., Gao, T., Fan, L., Huang, S., Edmonds, M., Liu, H., Gao, F., Zhang, C., Qi, S., Wu, Y. N., et al. (2020).  Dark, beyond deep: A paradigm shift to cognitive ai with humanlike common sense.  Engineering, 6(3):310–345.
  * Zhu et al., [2016] Zhu, Y., Groth, O., Bernstein, M., and Fei-Fei, L. (2016).  Visual7w: Grounded question answering in images.  In Conference on Computer Vision and Pattern Recognition (CVPR).

  *[EST]: Evaluating Scientific Thinking
  *[RL]: Reinforcement Learning
  *[VQA]: Visual Question Answering
  *[SOTA]: state-of-the-art
  *[IQ]: Intelligence Quotient
  *[RPM]: Raven’s Progressive Matrices
  *[JSD]: Jensen-Shannon Distance
  *[POMDP]: Partially Observable Markov Decision Process
  *[MDP]: Markov Decision Process
  *[MLP]: Multi-Layer Perceptron
  *[DDPG]: Deep Deterministic Policy Gradient
  *[TD-3]: Twin Delayed Deep Deterministic Policy Gradient
  *[PPO]: Proximal Policy Optimization


# MrRank: Improving Question Answering Retrieval System through Multi-Result
Ranking Model

Danupat Khamnuansin†*, Tawunrat Chalothorn†, and Ekapol Chuangsuwanich*
† Kasikorn Labs Co., Ltd., Kasikorn Business-Technology Group, Thailand
* Department of Computer Engineering, Faculty of Engineering, Chulalongkorn University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Large Language Models (LLMs) often struggle with hallucinations and outdated
information. To address this, Information Retrieval (IR) systems can be
employed to augment LLMs with up-to-date knowledge. However, existing IR
techniques contain deficiencies, posing a performance bottleneck. Given the
extensive array of IR systems, combining diverse approaches presents a viable
strategy. Nevertheless, prior attempts have yielded restricted efficacy. In
this work, we propose an approach that leverages learning-to-rank techniques
to combine heterogeneous IR systems. We demonstrate the method on two
Retrieval Question Answering (ReQA) tasks. Our empirical findings exhibit a
significant performance enhancement, outperforming previous approaches and
achieving state-of-the-art results on ReQA SQuAD.

MrRank: Improving Question Answering Retrieval System through Multi-Result
Ranking Model

Danupat Khamnuansin†*, Tawunrat Chalothorn†, and Ekapol Chuangsuwanich* †
Kasikorn Labs Co., Ltd., Kasikorn Business-Technology Group, Thailand *
Department of Computer Engineering, Faculty of Engineering, Chulalongkorn
University<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Large Language Models (LLMs) have demonstrated proficiency in comprehending
human language across diverse domains. Among these advancements, question
answering (QA) systems integrated with LLMs have gained significant attention.
Nonetheless, concerns persist regarding the susceptibility of LLMs to
hallucinations, potentially leading to inaccuracies in their responses.

To address the aforementioned challenge, the Retrieval Augmented Generation
(RAG) Lewis et al. (2020) is proposed. The RAG system generally consists of
two parts: an Information Retrieval (IR) engine which proposes the top $k$
candidate pieces from a document corpus for the query, and the QA engine which
is responsible for deriving the final answer from retrieved information.

Although LLMs demonstrate exceptional proficiency in QA tasks, contemporary
text embedding techniques employed in IR remain underdeveloped. This
deficiency poses a performance bottleneck for RAG, as shown by the results in
the Massive Text Embedding Benchmark Muennighoff et al. (2023).

To improve retrieval performance, advanced fine-tuning techniques applied to
individual retrieval systems Zhao et al. (2023) or the enhancement of model
components Zhao et al. (2021) emerge as a prominent area of investigation.
While fine-tuning approaches have demonstrated efficacy in this domain, they
commonly require substantial resources for training, posing a challenge to
real-world adoption.

Combining results from multiple IR systems can be quite effective, as
demonstrated in the work of Cormack et al. (2009), Reciprocal Rank Fusion
(RRF) is a technique that utilizes ranking information to rerank the overall
ranking, which has become the industry standard for combining multiple ranking
systems. Although the implementation of RRF is straightforward, a notable
limitation arises due to its failure to leverage output scores from individual
retrievers. Modern techniques have been developed to overcome this limitation;
FlexNeuART Boytsov and Nyberg (2020) employs the Learning to Rank (LTR)
framework to amalgamate scores from diverse retrievers into a unified score.
However, the application of LTR in the QA retrieval setting requires a
modification of the training process to avoid irrelevant samples. Moreover,
the exploration and optimization of these techniques within the domain of
question answering retrieval settings remain largely unaddressed.

In the work of Liang et al. (2021), a routing method for combining multiple
retrievers in the QA setting is introduced. This model employs a routing
classifier to select between a neural retriever and a traditional BM25
retriever. However, this method heavily relies on the score generated by BM25
and disregards ranking information from the neural model. Consequently, its
performance is limited in modern question answering systems where the neural
model typically outperforms BM25.

We propose re-ranking techniques specifically trained to improve question
answering retrieval performance. Our investigations involve combining two
different types of model architecture (term weighting and neural networks).
Additionally, we conducted experiments on two distinct styles of ReQA datasets
to demonstrate that combining multiple models using this setup can leverage
the advantages of both models through ranking loss objective training. Our
findings show that the proposed methodology yields enhanced retrieval
performance and ultimately improves the downstream QA task. Our approach
outperforms the current state-of-the-art Zhao et al. (2023) on ReQA SQuAD,
surpassing all individual retrieval models, RRF, and the statistical routing
strategy, yielding an average enhancement of 13.6% in the mean reciprocal rank
(MRR) across datasets.

##  2 Related Work

### Massive Text Embedding Benchmark

Massive Text Embedding Benchmark (MTEB) proposed by Muennighoff et al. (2023)
offers comprehensive insights and evaluations across a wide range of modern
text embedding models. This benchmark is particularly crucial for
understanding the out-of-the-box performance of various models. The evaluation
incorporated established text embedding models available on the Hugging Face
platform Wolf et al. (2020). The assessment encompassed retrieval tasks across
15 distinct datasets, each representing diverse domains and dataset
characteristics. The evaluation of re-ranking tasks was carried out using four
specifically curated re-ranking datasets, providing a robust assessment of
performance in real-world applications.

As shown in Table 1, GPT Sentence Embeddings for Semantic Search (SGPT)
(Muennighoff, 2022) is one of the top performing models in retrieval tasks.
Note that sentence embeddings constructed by simply averaging Global Vectors
for Word Representation (GloVe) Pennington et al. (2014) outperforms BERT
Devlin et al. (2019) and OpenAI’s Ada Similarity. This observation underscores
the challenges inherent in retrieval tasks, where mainstream text embedding
models still exhibit a deficiency in text retrieval proficiency.

In the context of re-ranking tasks, MPNET (Song et al., 2020) has demonstrated
superior performance, highlighting the challenges in achieving a universally
best solution in the current landscape. Notably, a model exhibiting
proficiency in retrieval tasks may not necessarily excel in re-ranking tasks.

Model | Retr. | Rerank.
---|---|---
BERT | 10.59 | 43.44
OpenAI Ada Similarity | 18.36 | 49.02
GloVe | 21.62 | 43.29
MPNet | 43.81 | 59.36
SGPT-5.8B-msmarco | 50.25 | 56.56
Table 1: Average of the retrieval and re-ranking metric per task per model on
MTEB English subsets.

### Retrieval Question Answering

Retrieval question answering (ReQA) tasks are proposed in works such as Ahmad
et al. (2019) and Chen et al. (2017). They encompass a type of information
retrieval system that takes a question as input and responds with the most
relevant sentence containing the answer. This task distinguishes itself from
traditional Question Answering (QA), where the paragraph containing the answer
is provided upfront, leading to notable achievements in machine learning
systems, some of which surpass human performance (Rajpurkar et al., 2016). QA
retrieval systems typically rely on either neural or term weighting methods.
Recent studies, such as SPARTA (Zhao et al., 2021), exhibit significant
improvements in retrieval performance by adhering to the concept of tokens-
level matching from traditional IR systems. However, the exploration of QA
retrieval systems that incorporate hybrid retrieval mechanisms remains
relatively limited.

### Hybrid Method for Evidence Retrieval for Question Answering

Previous attempts to combine a term weighting model with a neural-based model
simply involve routing between retrieval results from each model. This is
achieved by training a routing classifier that determines whether to use the
traditional IR method (BM25) or the neural-based method (USE-QA) based on
statistics derived from the training dataset Liang et al. (2021). The study
demonstrates that the hybrid approach outperforms individual retrieval
strategies in ReQA SQuAD and ReQA NQ. However, this methodology has
limitations in its support for combining multiple neural models, the
integration of more than two models, and heavily relies on BM25. These
limitations may deter real-world adoption, where there are numerous retrieval
models specialized in their own areas. Notably, USE-QA Yang et al. (2020),
employs fine-tuning techniques to tackle an asymmetric matching task present
in QA retrieval. Combining the specialized abilities of these models could
lead to better performance.

Thus, we emphasized an approach enhancing model retrieval performance through
model-level combination that aims to capitalize on the strengths of multiple
model types. Our objective is to present a robust methodology that can
effectively combine state-of-the-art or fine-tuned models for better
performance improvement.

Figure 1: Our framework for amalgamating multiple ranking information by using
the re-ranking approach.

##  3 Proposed Method

To construct a cohesive framework for merging multiple sets of ranking
information, we adopt a re-ranking approach. We have designated a neural-based
model as the primary retriever, complemented by BM25 as a supporting model.
This choice is based on the observed consistency of the neural-based model,
which demonstrates superior performance on R@10. This metric is particularly
crucial for assessing the upper limits of performance gain in re-ranking
performance. The rankings obtained from the main retriever serve as the
initial rankings, subsequently augmented by incorporating scores derived from
the support retriever. These concatenated scores are then fed into the re-
ranker model to predict the final rankings. The overall process is illustrated
in Figure 1.

The frameworks consist of two stages:

1) Retrieval Stage: off-the-shelf retrievers are used in order to generate a
candidate pool. We select top candidate retrievers from MTEB benchmarks within
the retrieval and re-ranking category. The performance is shown in Table 2.

2) Re-ranking Stage: a re-ranking network is used to construct the final
ranking from the candidate pool.

###  3.1 Retrieval Stage

Our approach focuses on incorporating multiple QA retrieval models. These
models have demonstrated significant performance improvements in question
answering retrieval tasks compared to base retrieval models. To maximize the
applicability of our approach, we have chosen widely recognized and publicly
accessible models. In accordance with Muennighoff et al. (2023) results, we
also included SGPT-5.8B-msmarco and MPNET. These models represent the current
state-of-the-art publicly accessible models in the domains of retrieval and
re-ranking tasks, respectively.

Model | MRR | R@1 | R@5 | R@10
---|---|---|---|---
ReQA SQuAD
BM25 | 0.670 | 0.591 | 0.759 | 0.814
USEqa | 0.665 | 0.561 | 0.793 | 0.854
MPNETqa | 0.549 | 0.399 | 0.748 | 0.847
SGPT5.8B | 0.783 | 0.699 | 0.887 | 0.926
ReQA NQ
BM25 | 0.529 | 0.378 | 0.723 | 0.797
USEqa | 0.582 | 0.447 | 0.751 | 0.826
MPNETqa | 0.628 | 0.439 | 0.902 | 0.950
SGPT5.8B | 0.652 | 0.528 | 0.807 | 0.856
Table 2: The off-the-shelf sentence-level retrieval performance on the ReQA
SQuAD and ReQA NQ task.

BM25 Retriever The token statistics for BM25 in the original ReQA paper were
computed over the first 10,000 questions of each dataset. This led to
significant performance losses, as shown in Table 3. In our implementation of
BM25, we have made modifications that include the following key distinctions:

1) The statistics are derived from all documents within the training set,
instead of solely utilizing the first 10,000 questions.

2) Our BM25 model is constructed using the gensim library (Rehurek and Sojka,
2011) with default parameters.

The original ReQA paper provided the rationale behind using only the initial
10,000 questions as to reduce the scoring time. We found that there is no
discernable disparity in inference time when comparing the utilization of all
question statistics versus using only the first 10,000 statistics for
constructing BM25. Consequently, we recommend adopting our approach to
implement the BM25 version instead, which yields a notable increase in
performance. It is worth mentioning that the training and inference duration
of our model remains unaffected by these variations in BM25 configurations.

Model | MRR | R@1 | R@5 | R@10
---|---|---|---|---
ReQA SQuAD
BM2510k | 0.600 | 0.523 | 0.683 | 0.741
BM25all | 0.670 | 0.591 | 0.759 | 0.814
ReQA NQ
BM2510k | 0.474 | 0.327 | 0.660 | 0.739
BM25all | 0.529 | 0.378 | 0.723 | 0.797
Table 3: Performances of BM25 on sentence-level ReQA from original paper and
our implementation.

###  3.2 Re-ranking Stage

To effectively integrate ranking information derived from multiple retrieval
systems, we opted for a neural-based architecture with a pairwise learning-to-
rank approach. The method is motivated by two primary factors.

1) QA retrieval models are initially pre-trained with a specific focus on
enhancing the ranking of question-answer document pairs. Consequently,
applying the pointwise method to train the re-ranker model would lead to
redundant training.

2) The unavailability of comprehensive ranking in the question-answer
retrieval dataset poses limitations on our ability to fully utilize the
advantageous attributes of listwise methods.

Architecture The re-ranker model adopts the RankNet (Burges et al., 2005)
architecture, which comprises of two layers of Siamese feed-forward neural
network. (Koch et al., 2015) The network consists of 10 hidden units with
leaky ReLU (Maas et al., 2013) and a single sigmoid output node.

Training Hyperparameters We optimized the re-ranker network using the Adam
optimizer, (Kingma and Ba, 2015) using a learning rate of $1e-3$, 1024 batch
size, and 100 epochs for every model setting.

Data Preparation The training features are derived from the top 64 ranking
scores of the best-performing retrieval model during the retrieval stage (main
retriever). Subsequently, the scores from support models are computed and
concatenated to form a set of re-ranking features. For the term weighting
model, we utilize the scoring output directly from the model. For the neural-
based model, the score is generated by calculating the cosine similarity
between the encoded query and the candidate document vectors.

We chose a number of re-ranked documents ($k$) equal to 64 to facilitate a
direct comparison with the routing method. We experimented on how the value of
$k$ affects the performance and inference time. The findings indicate that
augmenting the value of $k$ beyond 16 yields only marginal improvements in MRR
and is not worth trading off with prediction speed. Detailed experimental
outcomes are presented in Figure 3 and Figure 4, as outlined in the Appendix.

Figure 2: Data preparation steps include: 1) retrieving scores for each
document candidate, 2) performing permutations on each pair of documents and
filtering out tied pairs, and 3) creating negative and positive samples.

Training Objective The pairwise LTR framework objective involves training the
model to produce a value closer to one when it determines the ranking of a
particular document to be higher than another. Conversely, it generates a
value closer to zero for the opposite scenario. In the event of a tied
situation, the conventional RankNet framework designates a target value of
0.5. Since the ReQA dataset has a lot of non-target documents, selecting
random pairs will usually result in a tie. To customize the training objective
for ReQA tasks, our approach prioritizes sentences containing the correct
answer over those that do not. We have modified the conventional RankNet
learning-to-rank algorithm by excluding the training sample with a target
value of 0.5. The reduction in unnecessary training pairs resulted in a
significant decrease in training time while simultaneously enhancing
performance.

Inference Phase During the inference phase, the re-ranker model generates
ranking information for each candidate document pair to determine which
document should rank higher {(Di > Dj), (Dj > Dk), (Di > Dk)}. The final
ranking is derived using a topological sorting algorithm {Di > Dj > Dk}. In
cases where the outcome fails to yield a conclusive ranking, the original
results from the main retriever are used as the final ranking. All operations
are performed using the networkx library (Hagberg et al., 2008).

##  4 Results

We assess different retrieval models, including BM25, USE-QA,
SGPT-5.8B-msmarco and MPNET-QA (Song et al., 2020). Our baselines include
average scoring, thresholding, and Rank Reciprocal Fusion (RRF), a technique
commonly employed within modern multi-retriever systems.

We compare our proposed approach mainly on the ReQA SQuAD and ReQA NQ,
aligning with the original ReQA paper. Additionally, we conduct experiments on
seven question answering datasets from MTEB to assess its effectiveness in
contemporary retrieval tasks.

The Stanford Question Answering Dataset (SQuAD) Rajpurkar et al. (2016)
consists of questions on a set of Wikipedia articles, whereas Natural
Questions (NQ) Kwiatkowski et al. (2019) involves user questions issued to
Google search, with corresponding answers sourced from Wikipedia by
annotators. These two datasets represent distinct characteristics related to
lexical overlap between questions and contexts, with SQuAD representing a low-
overlap scenario while NQ exhibits the opposite. The publicly available
version of both datasets consists of two partitions: a training set and a
development set. The number of questions and documents is shown in Table 4.

The development set was used as the evaluation set for both datasets. We
focused on the sentence-level retrieval task, following the method outlined in
the original ReQA paper. However, our implementation of BM25 included the
complete vocabulary of the training documents, deviating slightly from the
original ReQA paper which only used the first 10,000 questions. This
difference resulted in better performance for our BM25 implementation. See
"BM25 Retriever" in Section 3.1 for additional details.

Dataset | Question | Document
---|---|---
ReQASQuAD (train) | 87,599 | 91,707
ReQASQuAD (dev) | 11,426 | 10,250
ReQANQ (train) | 74,097 | 239,013
ReQANQ (dev) | 1,772 | 7,020
Table 4: The number of questions and candidate sentences on the ReQA SQuAD and ReQA NQ datasets. Model | Type | MRR | R@1 | R@5 | R@10
---|---|---|---|---|---
ReQA SQuAD
BM25 | Single | 0.670 | 0.591 | 0.759 | 0.814
USEqa | Single | 0.665 | 0.561 | 0.793 | 0.854
MPNETqa | Single | 0.549 | 0.399 | 0.748 | 0.847
SGPT5.8B | Single | 0.783 | 0.699 | 0.887 | 0.926
RRFuse (USEqa+BM25) | Hybrid | 0.721 | 0.624 | 0.839 | 0.895
Routinguse (USEqa+BM25) | Hybrid | 0.705 | 0.626 | 0.796 | 0.851
Oursuse (USEqa+BM25) | Hybrid | 0.750 | 0.663 | 0.860 | 0.902
RRFmp (MPNETqa+BM25) | Hybrid | 0.673 | 0.546 | 0.827 | 0.896
Routingmp (MPNETqa+BM25) | Hybrid | 0.709 | 0.623 | 0.811 | 0.869
Oursmp (MPNETqa+BM25) | Hybrid | 0.748 | 0.658 | 0.856 | 0.909
RRFsgpt (SGPT5.8B+BM25) | Hybrid | 0.754 | 0.664 | 0.864 | 0.914
Routingsgpt (SGPT5.8B+BM25) | Hybrid | 0.762 | 0.686 | 0.852 | 0.897
Ourssgpt (SGPT5.8B+BM25) | Hybrid | 0.815 | 0.736 | 0.913 | 0.945
ReQA NQ
BM25 | Single | 0.529 | 0.378 | 0.723 | 0.797
USEqa | Single | 0.582 | 0.447 | 0.751 | 0.826
MPNETqa | Single | 0.628 | 0.439 | 0.902 | 0.950
SGPT5.8B | Single | 0.652 | 0.528 | 0.807 | 0.856
RRFuse (USEqa+BM25) | Hybrid | 0.603 | 0.450 | 0.802 | 0.874
Routinguse (USEqa+BM25) | Hybrid | 0.552 | 0.396 | 0.758 | 0.831
Oursuse (USEqa+BM25) | Hybrid | 0.590 | 0.452 | 0.765 | 0.832
RRFmp (MPNETqa+BM25) | Hybrid | 0.617 | 0.449 | 0.843 | 0.897
Routingmp (MPNETqa+BM25) | Hybrid | 0.638 | 0.467 | 0.873 | 0.931
Oursmp (MPNETqa+BM25) | Hybrid | 0.642 | 0.457 | 0.908 | 0.951
RRFsgpt (SGPT5.8B+BM25) | Hybrid | 0.601 | 0.443 | 0.791 | 0.872
Routingsgpt (SGPT5.8B+BM25) | Hybrid | 0.600 | 0.440 | 0.813 | 0.876
Ourssgpt (SGPT5.8B+BM25) | Hybrid | 0.655 | 0.529 | 0.816 | 0.862
Table 5: Model performances on the sentence-level retrieval of ReQA SQuAD and
ReQA NQ task. The routing model refers to the Hybrid (BM25) method described
in the original paper.

We use the Mean Reciprocal Rank (MRR) and Recall at k (R@k) as evaluation
metrics. The experiments were replicated five times, and the mean value is
reported. We performed significant testing between routing and our method,
which showed significant improvements in all settings (one sample t-test, with
p < 0.05). See Appendix for summary statistics of the results.

Table 5 presents the performance of both individual and combined models. Our
model demonstrates better performance across all datasets and evaluation
metrics, particularly in ReQA SQuAD where BM25’s performance is comparable to
those of the neural-based models. Our approach exhibits a significant
improvement compared to individual retrieval methods, RRF and routing
strategies. Our model (SGPT+BM25) outperforms MPNET-QA by 48.5%, USE-QA by
22.6%, BM25 by 21.6%, SGPT-5.8B-msmarco by 4.1%, and routing techniques on the
same model combination by 7.0%.

On the ReQA NQ dataset, the performance of BM25 is significantly inferior
compared to the neural-based systems. Our approach demonstrates the capability
to effectively integrate ranking information, resulting in an average
performance uplift of 2.6% from individual neural models. This highlights the
efficacy of our approach, even in situations where less potent models are
combined with their more superior counterparts.

### Common Ensemble Techniques

We assess standard ensemble techniques including

1) Average scoring, re-rank outputs based on normalized average scores from
each retriever.

2) Thresholding with a predefined criterion set at 0.5, routing between main
and support retriever.

Our results indicate that common ensemble techniques demonstrate inferior
performance compared to our approach across all settings, as shown in Table 6.

Model | MRR | R@1 | R@10
---|---|---|---
ReQA SQuAD
Avg (USE+BM25) | 0.687 | 0.595 | 0.869
Avg (MPNET+BM25) | 0.687 | 0.599 | 0.859
Avg (SGPT+BM25) | 0.696 | 0.598 | 0.901
Thr (USE+BM25) | 0.691 | 0.594 | 0.868
Thr (MPNET+BM25) | 0.571 | 0.424 | 0.861
Thr (SGPT+BM25) | 0.788 | 0.707 | 0.927
ReQA NQ
Avg (USE+BM25) | 0.345 | 0.198 | 0.691
Avg (MPNET+BM25) | 0.357 | 0.209 | 0.720
Avg (SGPT+BM25) | 0.346 | 0.190 | 0.720
Thr (USE+BM25) | 0.595 | 0.455 | 0.846
Thr (MPNET+BM25) | 0.630 | 0.441 | 0.949
Thr (SGPT+BM25) | 0.650 | 0.528 | 0.853
Table 6: Retrieval performances of common ensemble techniques. $Avg$ and $Thr$
denote average scoring and thresholding, respectively.

### Neural Model Combinations

Our experiments were assessed within the framework of term weighting-neural
combination in direct comparison with the routing method. We also included the
neural-neural combination experiments. The results demonstrate the efficacy of
our approach. The MPNET-QA and SGPT-5.8B-msmarco combination demonstrated
notable improvements, surpassing the best performer in the main experiments on
ReQA NQ. This is consistent with our prior observations that the fusion of
competitive models yields greater performance gains. The results of the model
combination are shown in Table 7.

Model | MRR | R@1 | R@10
---|---|---|---
ReQA SQuAD
USEqa \+ SGPT | 0.784 | 0.704 | 0.920
MPNETqa \+ SGPT | 0.786 | 0.696 | 0.939
SGPT + USEqa | 0.793 | 0.710 | 0.934
MPNETqa \+ USEqa | 0.741 | 0.634 | 0.924
USEqa \+ MPNETqa | 0.742 | 0.639 | 0.915
SGPT + MPNETqa | 0.799 | 0.710 | 0.948
ReQA NQ
USEqa \+ SGPT | 0.660 | 0.536 | 0.867
MPNETqa \+ SGPT | 0.723 | 0.569 | 0.963
SGPT + USEqa | 0.665 | 0.538 | 0.876
MPNETqa \+ USEqa | 0.712 | 0.551 | 0.961
USEqa \+ MPNETqa | 0.698 | 0.553 | 0.920
SGPT + MPNETqa | 0.714 | 0.576 | 0.922
Table 7: The performances of neural-neural models.

### Overall Question Answering Performance

In this experiment we investigated how enhancing retrieval performance
translates to the overall question answering system performance. We integrated
retrieval systems with the BERT reader model and found that an increase in
retrieval performance benefits the overall QA system performance in every
setting. We reported answer quality using two metrics, including the Exact
Match score (EM), where the answer is strictly required to match the label,
along with the less strict ROUGE-L score Lin (2004). The results show a strong
correlation coefficient of 0.865 between R@1 and ROUGE-L scores, confirming
that enhancements in retrieval performance positively impact overall QA
performance. However, there is a substantial gap between the best retrieval
method and the oracle retriever, showing the potential gain for further
improvements. The results are presented in Table 8.

Model | ROUGE-L | EM
---|---|---
SQuAD
BM25 | 0.519 | 0.392
MPNET | 0.370 | 0.264
SGPT | 0.611 | 0.458
Ours (MPNET+BM25) | 0.573 | 0.431
Ours (SGPT+BM25) | 0.636 | 0.481
Ours (SGPT+MPNET) | 0.618 | 0.466
Ours (Trio*) | 0.641 | 0.485
Oracle Retriever | 0.848 | 0.651
NQ
BM25 | 0.340 | 0.235
MPNET | 0.392 | 0.259
SGPT | 0.450 | 0.311
Ours (MPNET+BM25) | 0.403 | 0.271
Ours (SGPT+BM25) | 0.450 | 0.310
Ours (SGPT+MPNET) | 0.488 | 0.339
Ours (Trio*) | 0.488 | 0.340
Oracle Retriever | 0.781 | 0.588
Table 8: Overall performances on SQuAD and NQ. *Trio model comprises SGPT,
MPNET, and BM25.

### Effect of Sample Mining Technique

We trained the the combination model of BM25 and SGPT-5.8B-msmarco using the
original pairwise learning-to-rank approach, and compared it to our modified
version which includes sample mining. Table 9 shows that by reducing
unnecessary training pairs, there is a substantial 97.0% reduction in training
samples. At the same time, it yields an average 0.6% marginally improvement in
performance across datasets.

Model | MRR | Training Pair
---|---|---
ReQA SQuAD
Ourssgpt | 0.8146 ±0.0004 | 5,212,691
-miningsgpt | 0.8070 ±0.0012 | 176,599,584
ReQA NQ
Ourssgpt | 0.6554 ±0.0007 | 4,609,808
-miningsgpt | 0.6535 ±0.0005 | 149,379,552
Table 9: Differences in performance and training pair when unnecessary samples
are omitted.

### Low-resource Setting

To emulate a low-resource scenario, we ran an experiment using only 10% of the
training data. The findings demonstrate that the method can work even in
scenarios with limited data. This suggests that our method also works on a
smaller training size, as shown in Table 10.

Model | Data size | MRR | R@1
---|---|---|---
ReQA SQuAD
Oursuse | 100% | 0.750 | 0.663
Oursuse | 10% | 0.749 | 0.661
Oursmp | 100% | 0.748 | 0.658
Oursmp | 10% | 0.744 | 0.652
Ourssgpt | 100% | 0.815 | 0.736
Ourssgpt | 10% | 0.815 | 0.737
ReQA NQ
Oursuse | 100% | 0.590 | 0.452
Oursuse | 10% | 0.592 | 0.454
Oursmp | 100% | 0.642 | 0.457
Oursmp | 10% | 0.641 | 0.456
Ourssgpt | 100% | 0.655 | 0.529
Ourssgpt | 10% | 0.656 | 0.529
Table 10: The performances of hybrid models (BM25-neural) on low-resource
datasets.

### Combining More Than Two Models

Our framework also supports extensions beyond two models. We conducted
experiments to combine three retrieval models. The results indicate that
combining a model with a different specialization leads to a substantial
performance boost, as demonstrated by the combination of SGPT-5.8B-msmarco,
MPNET-QA, and BM25. This combination surpasses the current state-of-the-art in
ReQA SQuAD, yielding an improvement from 0.801 to 0.823 on the MRR metric Zhao
et al. (2023). The results are shown in Table 11.

Model | MRR | R@1 | R@10
---|---|---|---
ReQA SQuAD
USEmp+bm25 | 0.784 | 0.697 | 0.926
USEsgpt+bm25 | 0.807 | 0.732 | 0.930
MPNETuse+bm25 | 0.786 | 0.697 | 0.934
MPNETsgpt+bm25 | 0.815 | 0.734 | 0.946
SGPTuse+bm25 | 0.819 | 0.741 | 0.948
SGPTmp+bm25 | 0.823 | 0.742 | 0.957
ReQA NQ
USEmp+bm25 | 0.700 | 0.555 | 0.920
USEsgpt+bm25 | 0.663 | 0.537 | 0.867
MPNETuse+bm25 | 0.715 | 0.556 | 0.961
MPNETsgpt+bm25 | 0.729 | 0.576 | 0.964
SGPTuse+bm25 | 0.670 | 0.545 | 0.876
SGPTmp+bm25 | 0.714 | 0.576 | 0.922
Table 11: Retrieval performances of trio models.

### Performance on Modern QA Datasets

We performed experiments using QA datasets from MTEB to assess the efficacy of
our methods in modern retrieval tasks. We incorporated MPNET-QA and
SGPT-125M-msmarco, utilizing the development set for training and the test set
for evaluation. The results presented in Table 12 show the effectiveness of
our approach across datasets.

Dataset | MPNET | SGPT | Ours
---|---|---|---
DBPedia | 0.656 | 0.264 | 0.669
FEVER | 0.542 | 0.151 | 0.546
FiQA2018 | 0.560 | 0.120 | 0.560
HotpotQA | 0.622 | 0.234 | 0.634
MSMARCO | 0.352 | 0.425 | 0.663
NFCorpus | 0.539 | 0.130 | 0.539
QuoraRetrieval | 0.868 | 0.797 | 0.877
Table 12: Model performances on question answering datasets from MTEB,
reported in MRR.

### End-to-end Evaluation on RAG

We conducted Retrieval-Augmented Generation (RAG) experiments to assess the
end-to-end performance of our method using the mistral-7b-instruct-v0.2 Jiang
et al. (2023) as a reader model. The outcomes were evaluated based on METEOR
Banerjee and Lavie (2005), BERT-F1 Zhang et al. (2020), and ROUGE-L scores.
Our systems demonstrate performance improvements, as shown in Table 13.

Model | ME | BERT | RO-L
---|---|---|---
SQuAD
BM25 | 0.298 | 0.722 | 0.194
MPNET | 0.225 | 0.700 | 0.143
SGPT | 0.346 | 0.737 | 0.225
MPNET + BM25 | 0.328 | 0.731 | 0.213
SGPT + BM25 | 0.359 | 0.741 | 0.233
SGPT + MPNET | 0.348 | 0.738 | 0.227
Trio | 0.362 | 0.742 | 0.236
Oracle Retriever | 0.464 | 0.770 | 0.298
NQ
BM25 | 0.251 | 0.697 | 0.155
MPNET | 0.286 | 0.706 | 0.173
SGPT | 0.325 | 0.719 | 0.200
MPNET + BM25 | 0.289 | 0.707 | 0.175
SGPT + BM25 | 0.325 | 0.719 | 0.201
SGPT + MPNET | 0.344 | 0.724 | 0.212
Trio | 0.345 | 0.724 | 0.213
Oracle Retriever | 0.362 | 0.742 | 0.236
Table 13: The performances of models in RAG tasks.

### Multilingual Settings

We conducted a non-English language experiment to assess the performance of
our method when utilizing a multilingual model setup. We performed the
experiment in the context of Thai language, utilizing the iapp-wiki-qa-squad
datasets Viriyayudhakorn and Polpanumas (2021) which adhere to the SQuAD
styles. Our modifications to support the Thai language include:

1) Changing USE-QA, which only supports English language, to Multilingual
Universal Sentence Encoder Question Answering (mUSE-QA).

2) Substituting the NLTK Bird and Loper (2004) tokenizer with the PyThaiNLP
Phatthiyaphaibun et al. (2023) newmm tokenizer.

3) Using BM25 as the main retriever instead of the neural model due to a
significant gap in performance on the R@10 metric.

The result shows that our method outperforms routing techniques. Notably,
routing strategies only selected the ranking from the BM25 models, yielding a
0% uplift in performance. The results are shown in Table 14.

Model | MRR | R@1 | R@10
---|---|---|---
ReQA iAppTH
BM25 | 0.828 | 0.742 | 0.951
mUSEqa | 0.636 | 0.533 | 0.811
Routingmuse | 0.828 | 0.742 | 0.951
BM25muse | 0.835 | 0.748 | 0.960
Table 14: Model performances on multilingual settings.

##  5 Conclusion

In this paper, we propose an effective method to combine term weighting and
neural-based ReQA systems. Our approach involves training re-ranker neural
networks using a modified pairwise re-ranking objective. Through empirical
evaluations, the combined system exhibits a significant performance
improvement over other combining strategies, particularly when confronted with
significant performance disparities among the individual models. It is
noteworthy that this technique is not restricted to a specific type or
quantity of retrieval models. Our method focuses on improving the QA system by
enhancing the retrieval which we identified as a bottleneck. Further
investigation into improving the overall performance by directly optimizing
the reader along with the retrieval model might improve the overall pipeline.

## Limitations

In our work, we have focused on implementing our methodology within the
context of the retrieval question answering task. Nonetheless, it is
reasonable to anticipate that the application of our techniques to a broader
retrieval task would yield favorable results. Our methodology necessitated the
selection of a main retriever. This is helpful in reducing the computing load
of the supporting retriever, yet it can introduce a cap on the final
performance. In the future, we may try to eliminate the need for main
retrieval model selection. We also focus our experiment on techniques that do
not require full-weight updates. Nevertheless, our approach can complement
other full-weight update fine-tuning techniques to further enhance
performance. Lastly, our model’s computational cost scales with the number of
re-ranking indexes fed through the re-ranker ($k$), as the process requires
the prediction of $k\textsuperscript{2}$ pairs of attributes to determine the
final ranking. This aspect may present challenges when deploying the model in
situations with a tight compute budget.

## Acknowledgements

This work is under the joint research initiative between KASIKORN Business-
Technology Group (KBTG) and the Faculty of Engineering, Chulalongkorn
University. We would like to thank KASIKORNBANK PUBLIC COMPANY LIMITED (KBank)
and KBTG for supporting the research and providing computational resources to
conduct the experiments.

## References

  * Ahmad et al. (2019) Amin Ahmad, Noah Constant, Yinfei Yang, and Daniel Cer. 2019.  ReQA: An evaluation for end-to-end answer retrieval models.  In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_ , pages 137–146, Hong Kong, China. Association for Computational Linguistics.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Bird and Loper (2004) Steven Bird and Edward Loper. 2004.  NLTK: The natural language toolkit.  In _Proceedings of the ACL Interactive Poster and Demonstration Sessions_ , pages 214–217, Barcelona, Spain. Association for Computational Linguistics.
  * Boytsov and Nyberg (2020) Leonid Boytsov and Eric Nyberg. 2020.  Flexible retrieval with NMSLIB and FlexNeuART.  In _Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)_ , pages 32–43, Online. Association for Computational Linguistics.
  * Burges et al. (2005) Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005.  Learning to rank using gradient descent.  In _Proceedings of the 22nd International Conference on Machine Learning_ , ICML ’05, page 89–96, New York, NY, USA. Association for Computing Machinery.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading Wikipedia to answer open-domain questions.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1870–1879, Vancouver, Canada. Association for Computational Linguistics.
  * Cormack et al. (2009) Gordon V. Cormack, Charles L A Clarke, and Stefan Buettcher. 2009.  Reciprocal rank fusion outperforms condorcet and individual rank learning methods.  In _Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’09, page 758–759, New York, NY, USA. Association for Computing Machinery.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Hagberg et al. (2008) Aric A. Hagberg, Daniel A. Schult, and Pieter J. Swart. 2008.  Exploring network structure, dynamics, and function using networkx.  In _Proceedings of the 7th Python in Science Conference_ , pages 11 – 15, Pasadena, CA USA.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Koch et al. (2015) Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. 2015.  Siamese neural networks for one-shot image recognition.  In _ICML deep learning workshop_ , volume 2(1). Lille.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , NIPS’20, Red Hook, NY, USA. Curran Associates Inc.
  * Liang et al. (2021) Zhengzhong Liang, Yiyun Zhao, and Mihai Surdeanu. 2021.  Using the hammer only on nails: A hybrid method for representation-based evidence retrieval for question answering.  In _Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 – April 1, 2021, Proceedings, Part I_ , page 327–341, Berlin, Heidelberg. Springer-Verlag.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Maas et al. (2013) Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. 2013.  Rectifier nonlinearities improve neural network acoustic models.  In _Proc. icml_ , volume 30(1), page 3. Atlanta, Georgia, USA.
  * Muennighoff (2022) Niklas Muennighoff. 2022.  Sgpt: Gpt sentence embeddings for semantic search.
  * Muennighoff et al. (2023) Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023.  MTEB: Massive text embedding benchmark.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Phatthiyaphaibun et al. (2023) Wannaphong Phatthiyaphaibun, Korakot Chaovavanich, Charin Polpanumas, Arthit Suriyawongkul, Lalita Lowphansirikul, Pattarawat Chormai, Peerat Limkonchotiwat, Thanathip Suntorntip, and Can Udomcharoenchaikit. 2023.  PyThaiNLP: Thai natural language processing in python.  In _Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)_ , pages 25–36, Singapore, Singapore. Empirical Methods in Natural Language Processing.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Rehurek and Sojka (2011) Radim Rehurek and Petr Sojka. 2011.  Gensim–python framework for vector space modelling.  _NLP Centre, Faculty of Informatics, Masaryk University, Brno, Czech Republic_ , 3(2).
  * Song et al. (2020) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020.  Mpnet: Masked and permuted pre-training for language understanding.  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , NIPS’20, Red Hook, NY, USA. Curran Associates Inc.
  * Viriyayudhakorn and Polpanumas (2021) Kobkrit Viriyayudhakorn and Charin Polpanumas. 2021.  iapp_wiki_qa_squad.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Yang et al. (2020) Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2020.  Multilingual universal sentence encoder for semantic retrieval.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 87–94, Online. Association for Computational Linguistics.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020.  Bertscore: Evaluating text generation with BERT.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Zhao et al. (2023) Bo Zhao, Jun Bai, Chen Li, Jianfei Zhang, Wenge Rong, Yuanxin Ouyang, and Zhang Xiong. 2023.  Enhancing biomedical reqa with adversarial hard in-batch negative samples.  _IEEE/ACM Transactions on Computational Biology and Bioinformatics_ , 20(5):2933–2944.
  * Zhao et al. (2021) Tiancheng Zhao, Xiaopeng Lu, and Kyusong Lee. 2021.  SPARTA: Efficient open-domain question answering via sparse transformer matching retrieval.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 565–575, Online. Association for Computational Linguistics.

##  Appendix A Effect of the Amount of Re-ranked Documents

We conducted an experiment to study the effect of the number of re-ranked
documents on the retrieval performance (Figure 3) and runtime (Figure 4). The
processing was conducted on a standalone machine featuring Intel(R) Core(TM)
i7-7700 CPU @ 3.60GHz, with no GPU acceleration.

To measure the latency, we ran our trio model with $k=16$. The re-ranker
network adds a latency of 0.0133 seconds per query on average across datasets.

Figure 3: Relationship between the number of re-ranked documents ($k$) and
retrieval performance (MRR). Figure 4: Relationship between the number of re-
ranked documents ($k$) and inference time (query/sec).

##  Appendix B Correlation Between Retrieval and Overall QA Performance

We assess the correlation between retrieval and overall QA performance. The
findings reveal a strong correlation across metrics, suggesting that
improvements in retrieval performance have a positive influence on overall QA
performance.

Metric | MRR | R@1
---|---|---
SQuAD
ROUGE-L | 0.8225 | 0.8392
EM | 0.8182 | 0.8385
NQ
ROUGE-L | 0.8806 | 0.8909
EM | 0.8430 | 0.8751
Table 15: Correlation coefficient between retrieval and overall question answering performance. Model | MRR | Recall@1 | Recall@5 | Recall@10
---|---|---|---|---
ReQA SQuAD |  |  |  |
Oursuse | 0.7503 ±0.0002 | 0.6629 ±0.0002 | 0.8598 ±0.0006 | 0.9015 ±0.0004
Oursmpnet | 0.7476 ±0.0004 | 0.6576 ±0.0008 | 0.8561 ±0.0004 | 0.9090 ±0.0003
Ourssgpt | 0.8146 ±0.0004 | 0.7364 ±0.0005 | 0.9131 ±0.0006 | 0.9451 ±0.0005
ReQA NQ |  |  |  |
Oursuse | 0.5902 ±0.0012 | 0.4518 ±0.0022 | 0.7646 ±0.0005 | 0.8319 ±0.0008
Oursmpnet | 0.6422 ±0.0014 | 0.4568 ±0.0026 | 0.9077 ±0.0008 | 0.9510 ±0.0006
Ourssgpt | 0.6554 ±0.0007 | 0.5292 ±0.0011 | 0.8160 ±0.0006 | 0.8621 ±0.0019
Table 16: Summary statistics for experimental results on ReQA SQuAD and ReQA NQ. Query |  Routing |  Ours
---|---|---
What two artists came out with Coldplay during the half-time show? |  On December 3, the league confirmed that the show would be headlined by the British rock group Coldplay. |  The Super Bowl 50 halftime show was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Super Bowl XLVIII halftime shows, respectively.
What type of relationships do enthusiastic teachers cause? |  Students who had enthusiastic teachers tend to rate them higher than teachers who didn’t show much enthusiasm for the course materials. |  Enthusiastic teachers are particularly good at creating beneficial relations with their students.
When did Luther return to Wittenberg? |  When the town council asked Luther to return, he decided it was his duty to act. |  Luther secretly returned to Wittenberg on 6 March 1522.
What is the mortality rate of pneumonic plague? |  Septicemic plague is the least common of the three forms, with a mortality rate near 100%. |  Pneumonic plague has a mortality rate of 90 to 95 percent.
What was the population Jacksonville city as of 2010? |  Jacksonville has Florida’s largest Filipino American community, with 25,033 in the metropolitan area as of the 2010 Census. |  Jacksonville is the principal city in the Jacksonville metropolitan area, with a population of 1,345,596 in 2010.
What was the result of the 1967 referendum? |  However, as a result of the referendum in France and the referendum in the Netherlands, the 2004 Treaty establishing a Constitution for Europe never came into force. |  When a consolidation referendum was held in 1967, voters approved the plan.
Table 17: Comparative analysis between Routing and our model reveals distinct
characteristics. The outcomes obtained from the Routing technique demonstrate
the limitations inherent in the BM25 model, as it heavily relies on the
presence of overlapping evidence while inadequately addressing the critical
aspect of question-answer alignment. Conversely, our model exhibits a robust
approach by incorporating ranking information from the neural model.