# Exascale Grid Optimization (ExaGO) toolkit: An open-source high-performance
package for solving large-scale grid optimization problems ††thanks: This
research was supported by the Exascale Computing Project (ECP), Project
Number: 17-SC-20-SC, a collaborative effort of two DOE organizations—the
Office of Science and the National Nuclear Security Administration—responsible
for the planning and preparation of a capable exascale ecosystem—including
software, applications, hardware, advanced system engineering, and early
testbed platforms—to support the nation’s exascale computing imperative.

Shrirang Abhyankar
Slaven Peleš
Tamara Bečejac
Jesse Holzer  Electricity Infrasatructure and Buildings Division
Pacific Northwest National Laboratory (PNNL)
Richland, Washington, USA
{shrirang.abhyankar<EMAIL_ADDRESS>
{tamara.becejac<EMAIL_ADDRESS>Asher Mancinelli
Cameron Rutherford  Research Computing
Pacific Northwest National Laboratory (PNNL)
Richland, WA, USA
{asher.mancinelli<EMAIL_ADDRESS>
###### Abstract

This paper introduces the Exascale Grid Optimization (ExaGO) toolkit, a
library for solving large-scale alternating current optimal power flow (ACOPF)
problems including stochastic effects, security constraints and multi-period
constraints. ExaGO can run on parallel distributed memory platforms, including
massively parallel hardware accelerators such as graphical processing units
(GPUs). We present the details of the ExaGO library including its
architecture, formulations, modeling details, and its performance for several
optimization applications.

##  I Introduction

The electric power grid remains vulnerable to disruption from extreme events
including wildfires, severe storms, and cyber-attacks. Variable generation
resources, load volatility and system being operated closer to its limits,
also present operational challenges to grid stability. To mitigate disruptions
before they snowball, grid planners and operators must be able to see these
events coming and understand their potential impacts on grid reliability.
Planners and operators rely on optimization and real-time tools to efficiently
plan and operate the grid in a secure, cost-effective manner. The foundational
bedrock for this analysis has been the optimal power flow (OPF) analysis [1,
2, 3], which to date remains a challenging problem due to its non-convex
nature.

Over the years, the complexity of the optimal power flow grew as the grid’s
complexity increased and stringent reliability policies being mandated. Some
examples include the incorporation of stochasticity[4, 5], $N-1$ security
constraints[6, 7], multi-period aspects [8, 9], and a number of modeling
complexities [10, 11]. Figure 1 illustrates these three dimensions of the
optimal power flow complexity.

To continue to ensure secure system operation, new tools that account for
higher-level uncertainties, and are able to produce more accurate security
assessment are required. Besides uncertainties, short-term decision will
require additional attention, due to the more frequency ramping events caused
by non-conventional generation, thus modeling that looks ahead a few steps
will be desirable. Existing tools today either only address problems along a
certain dimension, for e.g., security-constrained OPF only. Moreover, the
scalability of such tools is limited only to a handful of ”what-if condition”
cases.

Figure 1: The three uncertainty axes for the optimal power flow problem –
stochasticity, security, and time. The black nodes represent optimal power
flow problems along a single dimnesion, for e.g., stochastic optimal power
flow, while the blue nodes require a combination of dimensions, for e.g.,
stochastic security-constrained optimal power flow. Tools addressing OPF
problems along two or more dimensions at scale are needed for the future low-
inertia power systems with deep penetration of renewable energy sources

In this paper, we introduce a novel open-source tool, the Exascale Grid
Optimization Toolkit (ExaGO)[12, 13], for solving Alternate Current OPF
(ACOPF)-based optimization problems. The ExaGO toolkit can solve stochastic,
security-constrained, and multiperiod ACOPF problems on high-performance with
CPUs and GPUs. It contains of an array of applications to solve problems along
a specific dimension (for e.g. Security-constrained ACOPF) or a combination of
dimensions (for e.g., stochastic security-constrained ACOPF).

ExaGO supports solving grid optimization problems expressed in general form
given in (1)-(6). Here, the subscripts $s$, $c$, and $t$ represent the
dimensions of stochasticity, security, and time, respectively.

min
$\displaystyle~{}\sum_{s\in\mathcal{S}}\pi_{s}\sum_{c\in\mathcal{C}}\sum_{t\in\mathcal{T}}f(x_{s,c,t})$
(1) s.t. $\displaystyle~{}g(x_{s,c,t})=0$ (2)
$\displaystyle~{}h(x_{s,c,t})\leq 0$ (3) $\displaystyle x^{-}$
$\displaystyle\leq x_{s,c,t}\leq x^{+}$ (4) $\displaystyle-\delta_{t}{x}$
$\displaystyle\leq x_{s,c,t}-x_{s,c,t-\Delta{t}}\leq\delta_{t}{x},~{}t\neq 0$
(5) $\displaystyle-\delta_{c}{x}$ $\displaystyle\leq
x_{s,c,0}-x_{s,0,0}\leq\delta_{c}{x},~{}c\neq 0,$ (6)
$\displaystyle-\delta_{s}{x}$ $\displaystyle\leq
x_{s,0,0}-x_{0,0,0}\leq\delta_{s}{x},~{}s\neq 0$ (7)

where the subscripts $s\in\mathcal{S}$, $c\in\mathcal{C}$, and
$t\in\mathcal{T}$ represent the dimensions of stochasticty, contingencies, and
time, respectively, and $\pi_{s}$ is the probabiity or the weight of scenario
$s$. The subscript $0$ denotes base-case for scenario or base-case for a
contingency or the first time-period. The last three equations express the
coupling between the scenarios, time, and contingencies, respectively.

ExaGO is written in C/C++ and uses high-performance computing (HPC) for
scaling to solve large-scale problems. It can also be used with NViDIA and AMD
graphical processing units (GPUs). Table I highlights the comparison of ExaGO
with other open-source grid optimization tools - MOST [14], MATPOWER [15],
Pandapower [16], PowerModels.jl [17].

TABLE I: Comparison between ExaGO and other grid optimization tools |  ExaGO |  [14] |  [15] |  [16] |  [17]
---|---|---|---|---|---
Formulation |  AC |  DC |  AC/DC |  AC/DC |  AC
Language |  C/C++ |  Matlab |  Matlab |  Python |  Julia
Contingencies |  Y |  Y |  N |  N |  N
Multi-period |  Y |  Y |  N |  N |  N
Stochasticity |  Y |  Y |  N |  N |  N
Parallel computing |  Y |  N |  N |  N |  N
GPU |  Y |  N |  N |  N |  N

##  II ExaGO architecture

ExaGO is written in C/C++ and makes heavy use of the functionality provided by
the PETSc[18] library. All ExaGO applications use a nonlinear formulation
based on full AC optimal power flow. The different applications available with
ExaGO are listed in Table II

TABLE II: ExaGO applications Application Name |  Description
---|---
OPFLOW |  AC optimal power flow
TCOPFLOW |  Multi-period AC optimal power flow
SCOPFLOW |  Single/multi-period security-constrained AC optimal power flow
SOPFLOW |  Single/multi-period no/multi-contingency stochastic AC optimal power flow
PFFLOW |  AC power flow

The hierarchy of the applications is shown in Figure 2 to show how the
applications are layered.

Figure 2: ExaGO application hierarchy

ExaGO is interfaced with three optimization solvers – Ipopt , HiOp , and TAO –
for solving the underlying optimization problems. Figure 3 shows the solvers
available for the different applications.

Figure 3: ExaGO applications and solvers

One unique computational feature of ExaGO is the ability to perform
calculations (model evaluation and optimization solves) on the GPU. For this,
it uses HiOp , RAJA [19], and Umpire [20] libraries. All the calculations are
done directly on the GPU, i.e., there is minimal exchange of data between CPU
and GPU during the optimization iterations. The reader is refered [21] for GPU
implementation details.

In the next sections, we describe the different ExaGO applications.

##  III OPFLOW: AC optimal power (ACOPF)

OPFLOW solves the full AC optimal power flow problem and provides various
flexible features that can be toggled via run-time options. It has interfaces
to different optimization solvers and includes different representations of
the underlying equations (power-balance-polar and power-balance-cartesian)
that can be used. By selecting the appropriate solver, OPFLOW can be executed
on CPUs (in serial or parallel) or on GPUs.

In compact form, the set of equations describing ACOPF is given by (8)-(11).

$\displaystyle\underset{x}{\text{min}}$ $\displaystyle~{}f(x)$ (8) s.t.
$\displaystyle g(x)=0$ (9) $\displaystyle h^{-}$ $\displaystyle\leq h(x)\leq
h^{+}$ (10) $\displaystyle x^{-}$ $\displaystyle\leq x\leq x^{+}$ (11)

The full formulation of an AC optimal power flow is digressed in this paper
due to space limitations. The reader is referred to [13, 15, 22] for
comprehensive details on the AC optimal power flow formulation. Figure 4
briefly describes the modeling details of OPFLOW.

Figure 4: ExaGO formulation includes standard ACOPF formulation plus
feasabiity (load loss, power imbalance), and advanced features (PV/PQ
switching, AGC)

The layout of the ACOPF application in ExaGO is shown in Fig. 5. A clear
separation of model (physics) and the solver is implemented to allow switching
between different models/solvers (through run-time options). OPFLOW can be
used with three different solvers: Ipopt [23], HiOp [24, 25], and TAO [18].

Figure 5: Architecture of the ACOPF application in ExaGO

Different colors in Fig. 5 denote models/solvers compatible with CPU and GPU,
respectively. The advantage of such a separation is that hardware-specific
code for a model or a solver is isolated to it and does not spill over other
portions of the code base and can be easily replaced.

##  IV TCOPFLOW: Multi-period OPF

TCOPFLOW solves a full AC multi-period optimal power flow problem with the
objective of minimizing the total cost over the given time horizon while
adhering to constraints for each period and between consecutive time-periods
(ramping constraints).

The multi-period optimal power flow problem is a series of optimal power flow
problems coupled via temporal constraints. The generator real power deviation
($p_{jt}^{\text{g}}-p_{jt-\Delta{t}}^{\text{g}}$) constrained within the ramp
limits form the temporal constraints.

An illustration of the temporal constraints is shown in Fig. 6 with four time
steps. Each time-step $t$ is coupled with its preceding time $t-\Delta{t}$,
where $\Delta{t}$ is the time-step where the objective is to find a least cost
dispatch for the given time horizon.

$t_{0}$$t_{1}$$t_{2}$$t_{3}$ Figure 6: Multi-period optimal power flow example
with four time-steps. The lines connecting the different time-periods denote
the coupling between them.

In general form, the equations for multi-period optimal power flow are given
by (12)-(16). TCOPFLOW solves to minimize the total generation cost
$\sum_{t\in\mathcal{T}}f(x_{t})$ over the time horizon $T$. At each time-step,
the equality constraints ($g(x_{t})$), inequality $h(x_{t})$, and the
lower/upper limit ($x^{-}$,$x^{+}$) constraints need to be satisfied. Equation
(16) represents the coupling between the consecutive time-steps. The most
common form of coupling are ramping constraints that limit the deviation of
the real power generation at time $t$ from its preceding time-step
$t-\Delta{t}$ to within its ramping capability $\delta_{t}{x}$.

min $\displaystyle~{}\sum_{t\in\mathcal{T}}f(x_{t})$ (12) s.t.
$\displaystyle~{}g(x_{t})=0$ (13) $\displaystyle~{}h(x_{t})\leq 0$ (14)
$\displaystyle x^{-}$ $\displaystyle\leq x_{t}\leq x^{+}$ (15)
$\displaystyle-\delta_{t}{x}$ $\displaystyle\leq
x_{t}-x_{t-\Delta{t}}\leq\delta_{t}{x},$ $\displaystyle t\neq 0$ (16)

TCOPFLOW currently supports solving the multi-period OPF problem using Ipopt
on single processor only. Figure 7 shows the results from TCOPFLOW for a
30-min. horizon with 5 minute intervals for the ACTIVSg200 [26] bus system.

Figure 7: Generation dispatch for ACTIVSg200 [26] bus system from TCOPLOW for
a 30-min horizon with 5-minute intervals

Load and/or wind generation profiles can be set for the multiperiod probem
with TCOPFLOW . The output of the base-case and the contingency subproblems is
saved to MATPOWER format files (one file per subproblem).

##  V SCOPFLOW: Single/Multi-period Security-constrained ACOPF

SCOPFLOW solves a single/multi-period contingency-constrained optimal power
flow problem. The problem is set up as a two-stage optimization problem where
the first-stage (base-case $c_{0}$) represents the normal operation of the
grid and the second-stage comprises $c\in\mathcal{C}$ contingency cases. Each
contingency case can be single or multi-period. SCOPFLOW operates in two
modes: (a) _preventive_ \- only reference generators pick up the
deficit/excess power for the contingency, or (b) _corrective_ \- the
generators are allowed to ramp up/dowm constrained by their ramping limits. In
the next subsections, we described the single-period and multi-period
formulations for SCOPFLOW .

###  V-A Single-period

The contingency-constrained optimal power flow (popularly termed as security-
constrained optimal power flow (SCOPF) in power system parlance) attempts to
find a least cost dispatch for the base case (or no contingency) while
ensuring that if any of contingencies do occur then the system will be secure.
This is illustrated in Fig. 8 for a SCOPF with a base-case $c_{0}$ and three
contingencies.

$c_{0}$$c_{1}$$c_{2}$$c_{3}$ Figure 8: Contingency constrained optimal power
flow example with three contingencies. $c_{0}$ represents the base case (or no
contingency case). $c_{1}$, $c_{2}$, $c_{3}$ are the three contingency cases.
Each of the contingency states is coupled with the base-case through ramping
constraints (denoted by red lines)

In general form, the equations for contingency-constrained optimal power flow
are given by (17) – (21). This is a two-stage stochastic optimization problem
where the first stage is the base case $c_{0}$ and each of the contingency
states are second-stage subproblems.

SCOPFLOW aims to minimize the objective $\sum_{c\in\mathcal{C}}f(x_{c})$,
while adhering to the equality $g(x_{c})$, inequality $h(x_{c})$, and the
lower/upper bound ($x^{-}$,$x^{+}$) constraints. For notational ease we
include the base-case in set $\mathcal{C}$, i.e.,
$\mathcal{C}\equiv\mathcal{C}\cup c_{0}$. Equation (21) represents the
coupling between the base-case and each of the contingency states $c_{i}$.
Equation (21) is the most typical form of coupling that limits the deviation
of the contingency variables $x_{c}$ from the base $x_{0}$ to within
$\delta_{c}{x}$. An example of this constraint could be the allowed real power
output deviation for the generators constrained by their ramp limit.

min $\displaystyle\sum_{c\in\mathcal{C}}f(x_{c})$ (17) s.t.
$\displaystyle~{}g(x_{c})=0,$ (18) $\displaystyle~{}h(x_{c})\leq 0,$ (19)
$\displaystyle x^{-}$ $\displaystyle\leq x_{c}\leq x^{+},$ (20)
$\displaystyle-\delta_{c}{x}$ $\displaystyle\leq
x_{c}-x_{0}\leq\delta_{c}{x},~{}c\neq 0$ (21)

where, $\mathcal{C}$ represents the set of contingencies, including the base-
case denoted by subscript 0. The list of contingencies is provided through a
PTI or native format file. Each contingency can be single outage (e.g. single
generator outage) or multiple (e.g. simultaneous generator and a line outage).

Three different solvers are available to solve single-period SCOPFLOW :

  * •

With the Ipopt solver, SCOPFLOW constructs a large monolithic problem
consisting of the base case and the contingencies. This solver is only
supported in serial.

  * •

The HiOp solver uses a primal decomposition approach to parallelize SCACOPF
analysis where each contingency problem (and base case) are solved
independently. HiOp uses a primal decomposition approach with second order
corrections [27] to solve the SCOPFLOW problem. This solver is supported in
serial and parallel, and can also use GPUs for solving the contingency
subproblems. Figure 9 shows strong scaling of SCOPFLOW with HiOP primal
decomposition algorithm on the ACTIVSg200 [26] test system with 100
contingencies.

Figure 9: Scalability of SCOPFLOW using primal decomposition approach. The
HIOP solver was used for solving SCOPFLOW on the ACTIVSg200 system with 100
contingencies.

  * •

The EMPAR solver is an _embarrasingly parallel_ version of SCOPFLOW that
solves the base case and contingency subproblems independently in parallel. In
other words, it relaxes the coupling constraint between the base-case and
contingency subproblems and solves them separately. Figure 10 shows the
scalability of the embarrasingly parallel solver for a large-scale
optimization run.

Figure 10: Scalability of SCOPFLOW on 512 nodes (3096 ranks) on Summit cluster
[28]. The embarrasingly parallel solver EMPAR was used for solving SCOPFLOW on
the ACTIVSg2000 [26] system with 3095 contingencies.

###  V-B Multiperiod

In the multi-period version of SCOPFLOW , each contingency comprised of
multiple time-periods. The multiple periods have variables and constraints as
described in section IV, i.e., each multiperiod problem uses TCOPFLOW
application internally. An example of multi-contingency multi-period optimal
power flow is illustrated in Fig. 11 with two contingencies $c_{0}$ and
$c_{1}$. Here, $c_{0}$ is the case with no contingencies, i.e., the base-case.
In this example, each contingency is multi-period with four time-periods. Each
time-step is coupled with its adjacent one through ramping constraints.
Currently, we assume the contingency is incident at the first time-step, i.e.
at $t_{0}$. This results in the coupling between the contingency cases
$c\in\mathcal{C}$ and the base-case $c_{0}$ only at time-step $t_{0}$ as shown
in Fig. 11.

$t_{0}$$t_{1}$$t_{2}$$t_{3}$$\mathbf{c_{0}}$$t_{0}$$t_{1}$$t_{2}$$t_{3}$$\mathbf{c_{1}}$

Figure 11: Multi-period contingency constrained optimal power flow example
with two contingencies $c_{0}$ and $c_{1}$, each with four time-periods
$t_{0}$, $t_{1}$, $t_{2}$, $t_{3}$. State $c_{0},t_{0}$ represent the base
case (no contingency) case. We assume that any contingency is incident at the
first time-step, i.e., at $t_{0}$. The contingency states $c_{1},t_{0}$ is
coupled with the no-contingency state $c_{0},t_{0}$ at time $t_{0}$. The red
line denotes the coupling between the contingency.

The overall objective of this contingency-constrained multi-period optimal
power flow is to find a secure dispatch for base-case $c_{0}$ while adhering
to contingency and temporal constraints. The general formulation of this
problem is given in Eqs. (22) – (LABEL:eq:ctopflow_end).

min $\displaystyle~{}\sum_{c\in\mathcal{C}}\sum_{t\in\mathcal{T}}f(x_{c,t})$
(22) s.t. $\displaystyle~{}g(x_{c,t})=0$ (23) $\displaystyle~{}h(x_{c,t})\leq
0$ (24) $\displaystyle x^{-}$ $\displaystyle\leq x_{c,t}\leq x^{+}$ (25)
$\displaystyle-\delta_{t}{x}$ $\displaystyle\leq
x_{c,t}-x_{c,t-\Delta{t}}\leq\delta_{t}{x},~{}t\neq 0$ (26)
$\displaystyle-\delta_{c}{x}$ $\displaystyle\leq
x_{c,0}-x_{0,0}\leq\delta_{c}{x},~{}c\neq 0$ (27)

here, $\mathcal{C}$ and ${T}$ are the sets for contingencies (including the
base-case) and time-steps, respectively.

The objective of the multi-period SCOPFLOW formulation is to reduce the total
cost for the given horizon over the set of contingencies. Equation
(LABEL:eq:ctopflow_end) represents the coupling between the base case $c_{0}$
and each contingency $c_{i}$ at time-step $t_{0}$. We use a simple box
constraint $\delta_{c}{x}$ to restrict the deviation of decision variables
$x_{c,0}$ from the base-case $x_{0,0}$. The bound $\delta_{c}{x}$ could
represent here, for example, the allowable reserve for each generator. All the
modeling details of TCOPFLOW IV can be used for this multi-period setup. The
multi-period SCOPFLOW currently supports solution using Ipopt only. It can
also do an embarrasingly parallel solution where the contingency coupling
constraints are relaxed and each multi-period problem is distributed and
solved independently on a rank.

##  VI SOPFLOW: Single/multiperiod no/multi-contingency Stochastic ACOPF

SOPFLOW solves a stochastic optimal power flow problem where the stochasticity
is described through scenarios of wind and/or load forecast deviations. The
problem is set up as a two-stage optimization problem where the first-stage
(base-case) represents the normal operation of the grid (or the most likely
forecast) and the second-stage comprises $N_{s}$ scenarios of forecast
deviation. Each scenario can have multiple contingencies and each contingency
can be multi-period. Like SCOPFLOW , SOPFLOW also operates in either a
preventive or corrective mode of generation redispatch.

In the next subsections, we describe the different variations of SOPFLOW .

###  VI-A Single period, no contingencies

This problem has a structure as illustrated in Fig. 12.

$s_{0}$$s_{1}$$s_{2}$$s_{3}$ Figure 12: Stochastic optimal power flow example
with three four. $s_{0}$ represents the base case (or the scenario with the
highest probability). $s_{1}$, $s_{2}$, $s_{3}$ are the other three scenario
forecasts. Each of the scenarios is coupled with the base-case scenario
through ramping constraints (denoted by red lines)

The formulation for the single period stochastic optimal power flow is given
in 29–33. This formulation is similar to single-period SCOPFLOW. The
difference is SOPFLOW objective is weighted by $\pi_{s}$ for each scenario.

min $\displaystyle\sum_{s\in\mathcal{S}}\pi_{s}f(x_{s})$ (29) s.t.
$\displaystyle~{}g(x_{s})=0,$ (30) $\displaystyle~{}h(x_{s})\leq 0,$ (31)
$\displaystyle x^{-}$ $\displaystyle\leq x_{s}\leq x^{+},$ (32)
$\displaystyle-\delta_{s}{x}$ $\displaystyle\leq
x_{s}-x_{0}\leq\delta_{s}{x},~{}s\neq 0$ (33)

here, ${S}$ is the set of scenarios with the base-scenario denoted by
subscript 0. While there is no real notion of a ”base” scenario, our current
implementation assumes it as the scenario with the most weight or probability.
The coupling constraints between the base-case and scenario subproblem are
ramping constraints for the generators.

The single period, no contingency SOPFLOW can be solved on a single processor
(serial) with Ipopt , in parallel using HiOp primal decomposition approach, or
in an embarassingingly parallel way with the EMPAR solver.

###  VI-B Single period, multiple contingencies

The aim of the single period, contingency-constrained stochastic OPF is to
optimize the grid dispatch to ensure the grid is secure for all the wind
forecast deviations and for all the contingencies considered therein. There
are two variations of this problem implemented in ExaGO differing in how the
contingency-scenario pairs are set up.

####  VI-B1 Full stochastic contingency-constrained structure

(34)-(39) describes the formulation for the full stochastic contingency-
constrained OPF formulation and its structure is illustrated in Figure 13 with
a two scenarios- two contingency example..

$s_{0}$$s_{0}c_{1}$$s_{0}c_{2}$$s_{1}$$s_{0}c_{1}$$s_{0}c_{2}$ Figure 13: Full
stochastic contingency-constrained optimal power flow structure

$s_{0}$ and $s_{1}$ represent the base case for the two scenarios with two
contingencies $c_{1}$ and $c_{2}$. Each of the contingency states is coupled
with its scenario base-case through ramping constraints (denoted by red lines)
and each scenario is coupled with each other through having a common real
power set-point for generators blue (see [13] for the generator set-point
formulation). This structure results in a three-stage optimization problem
that is difficult to solve in parallel since most of the optimization solvers
cater to two-stage optimization problems only. Hence, SOPFLOW only supports
solving this problem on a single processor using Ipopt currently.

$\displaystyle\centering\min\@add@centering$
$\displaystyle\sum_{s\in\mathcal{S}}\pi_{s}\sum_{c\in\mathcal{C}}f(x_{s,c})$
(34) s.t. $\displaystyle~{}g(x_{s,c})=0,$ (35) $\displaystyle~{}h(x_{s,c})\leq
0,$ (36) $\displaystyle x^{-}$ $\displaystyle\leq x_{s,c}\leq x^{+},$ (37)
$\displaystyle-\delta_{s}{x}$ $\displaystyle\leq
x_{s,0}-x_{0,0}\leq\delta_{s}{x},~{}s\neq 0$ (38)
$\displaystyle-\delta_{c}{x}$ $\displaystyle\leq
x_{s,c}-x_{s,0}\leq\delta_{c}{x},~{}c\neq 0$ (39)

###  VI-C Flattened stochastic contingency-constrained structure

This variation is a relaxation of the full structure obtained by flattening
the contingency-scenario pairs to reduce the problem to a two-stage
optimization problem. The formulation is given in (34)-(39) and its structure
is illustrated in Figure 14.

$s_{0}$$s_{0}c_{1}$$s_{0}c_{2}$$s_{1}$$s_{1}c_{1}$$s_{1}c_{2}$ Figure 14:
Flattened stochastic contingency-constrained optimal power flow structure

The scenarios and the contingencies are flattened to obtain a standard two-
stage optimization structure. This is done by choosing the most probable
scenario as the base-case and coupling (denoted by red) it with all the other
scenario-contingency pairs. This problem variation can be solved both in
serial (using Ipopt or HiOp ) or in parallel (using HiOp ). The difference
between the full structure and the flattened structure can be seen in the last
constraint in their respective formulations. While the full structture has
coupling constraints for contingencies with their respective ”base” scenario,
the flattened structured has all contingencies (and scenarios) coupled to a
single base scenario. Figure 15 shows the strong-scaling of SOPFLOW with the
flattened contingency-structure on the ACTIVSg200 bus system with 5 wind
scenarios each with 100 contingencies.

Figure 15: Scalability of SOPFLOW using primal decomposition approach. The
HIOP solver was used for solving SOPFLOW on the ACTIVSg200 system with 5
scenarios and 100 contingencies.

min
$\displaystyle\sum_{s\in\mathcal{S}}\pi_{s}\sum_{c\in\mathcal{C}}f(x_{s,c})$
(40) s.t. $\displaystyle~{}g(x_{s,c})=0,$ (41) $\displaystyle~{}h(x_{s,c})\leq
0,$ (42) $\displaystyle x^{-}$ $\displaystyle\leq x_{s,c}\leq x^{+},$ (43)
$\displaystyle-\delta_{s}{x}$ $\displaystyle\leq
x_{s,0}-x_{0,0}\leq\delta_{s}{x},~{}s\neq 0$ (44)
$\displaystyle-\delta_{c}{x}$ $\displaystyle\leq
x_{s,c}-x_{0,0}\leq\delta_{c}{x},~{}c\neq 0$ (45)

###  VI-D Multiperiod, multi-contingency

This is the most detailed formulation involving stochasticity, security
constraints, and time, i.e. all the three dimensions of uncertainty. The full
formulation for the stochastic security-constrained multi-period optimal power
flow is given in (46) – (52). In this formulation, the objective is to reduce
the expected cost, where $f(x_{s,c,t})$ is the cost for scenario $s$ with
contingency $c$ at time $t$. $\pi_{s}$ is the probability of scenario $s$.

min
$\displaystyle~{}\sum_{s\in\mathcal{S}}\pi_{s}\sum_{c\in\mathcal{C}}\sum_{t\in\mathcal{T}}f(x_{s,c,t})$
(46) s.t. $\displaystyle~{}g(x_{s,c,t})=0$ (47)
$\displaystyle~{}h(x_{s,c,t})\leq 0$ (48) $\displaystyle x^{-}$
$\displaystyle\leq x_{s,c,t}\leq x^{+}$ (49) $\displaystyle-\delta_{t}{x}$
$\displaystyle\leq x_{s,c,t}-x_{s,c,t-\Delta{t}}\leq\delta_{t}{x},~{}t\neq 0$
(50) $\displaystyle-\delta_{c}{x}$ $\displaystyle\leq
x_{s,c,0}-x_{s,0,0}\leq\delta_{c}{x},~{}c\neq 0$ (51)
$\displaystyle-\delta_{s}{x}$ $\displaystyle\leq
x_{s,0,0}-x_{0,0,0}\leq\delta_{s}{x},~{}s\neq 0$ (52)

here, $c\in\mathcal{C}$, $t\in\mathcal{T}$, and $s\in\mathcal{S}$ are the
indices for contingencies, time-step, and scenarios, respectively.

An illustration of SOPFLOW is shown in Fig. 16 for a case with two scenarios
$s_{0}$ and $s_{1}$. Each scenario has two contingencies $c_{0}$, $c_{1}$, and
each contingency has four time-periods.

$t_{0}$$t_{1}$$t_{2}$$t_{3}$$\mathbf{c_{0}}$$t_{0}$$t_{1}$$t_{2}$$t_{3}$$\mathbf{c_{1}}$$\mathbf{s_{0}}$$t_{0}$$t_{1}$$t_{2}$$t_{3}$$\mathbf{c_{0}}$$t_{0}$$t_{1}$$t_{2}$$t_{3}$$\mathbf{c_{1}}$$\mathbf{s_{1}}$

Figure 16: Stochastic multi-period contingency constrained example with two
scenarios $s_{0}$ and $s_{1}$. Each scenario has two contingencies $c_{0}$ and
$c_{1}$ and each contingency consists of four time-periods $t_{0}$, $t_{1}$,
$t_{2}$, $t_{3}$. State $s_{0},c_{0},t_{0}$ represents the base case (no
contingency) case for the two scenarios. We assume that any contingency is
incident at the first time-step, i.e., at $t_{0}$. Thus, the contingency
states $c_{1},t_{0}$ is coupled with the no-contingency state $c_{0},t_{0}$ at
time $t_{0}$ for both the scenarios. The red line denotes the coupling between
the contingency and the no-contingency states.The blue line denotes the
coupling between the scenarios

SOPFLOW currently only supports Ipopt solver to solve the stochastic security-
constrained multi-period optimal power flow problem.

##  VII Conclusions and Future Work

Dealing with increased uncertainties will be important to address reliability
issues in a deep decarbonized grid with low-inertia and extensive penetration
off renewable energy sources. The ExaGO toolkit aims to solve large-scale
stochastic, security-constrained, multi-period ACOPF problems on high-
performance computers. It has high-fidelity models with scalable numerical
algorithms. Moreover, it has a well-designed API through which new models and
solvers can be interfaced. ExaGO has been validated against PowerWorld [29] on
several test networks and we are continually validating it on more networks
and applications. It is in active development and several new features are to
be added to the library including new modeling capabilities (storage, flexible
load), improving robustness for large-scale optimization problems,and high-
performance implementation of multi-period, multi-contingency stochastic
ACOPF, and expanding its solver capabilities.

## Acknowledgements

This research was supported by the Exascale Computing Project (ECP), Project
Number: 17-SC-20-SC, a collaborative effort of two DOE organizations—the
Office of Science and the National Nuclear Security Administration—responsible
for the planning and preparation of a capable exascale ecosystem—including
software, applications, hardware, advanced system engineering, and early
testbed platforms—to support the nation’s exascale computing imperative.

The authors thank Cosmin Petra, Naiyuan Chiang, and Jingyi Wang for their
guidance and support on the usage of HiOp

The authors also want to thank Phil Roth, Christopher Oehmen, and Lori Ross
O’Neil for their support on this work.

## References

  * [1] J. Carpentier, “Optimal power flows,” _International Journal of Electrical Power and Energy Systems_ , vol. 1, no. 1, pp. 3–15, 1979.
  * [2] W. F. Tinney, “Optimal Power Flow Solutions,” _IEEE Transactions on Power Apparatus and Systems_ , vol. PAS-87, no. 10, pp. 1866–1876, 1968.
  * [3] M. B. Cain, R. P. O’neill, and A. Castillo, “History of Optimal Power Flow and Formulations,” Tech. Rep., 2012.
  * [4] T. Yong and R. Lasseter, “Stochastic optimal power flow: formulation and solution,” in _2000 Power Engineering Society Summer Meeting (Cat. No.00CH37134)_ , vol. 1, 2000, pp. 237–242 vol. 1.
  * [5] J. Luo, L. Shi, and Y. Ni, “A solution of optimal power flow incorporating wind generation and power grid uncertainties,” _IEEE Access_ , vol. 6, pp. 19 681–19 690, 2018.
  * [6] F. Capitanescu and L. Wehenkel, “A new iterative approach to the corrective security-constrained optimal power flow problem,” _IEEE Transactions on Power Systems_ , vol. 23, no. 4, pp. 1533–1541, 2008.
  * [7] F. Capitanescu, J. L. Martinez Ramos, P. Panciatici, D. Kirschen, A. Marano Marcolini, L. Platbrood, and L. Wehenkel, “State-of-the-art, challenges, and future trends in security constrained optimal power flow,” _Electric Power Systems Research_ , vol. 81, no. 8, pp. 1731–1741, 2011. [Online]. Available: http://dx.doi.org/10.1016/j.epsr.2011.04.003
  * [8] M. Schanen, F. Gilbert, C. G. Petra, and M. Anitescu, “Toward multiperiod ac-based contingency constrained optimal power flow at large scale,” in _2018 Power Systems Computation Conference (PSCC)_ , 2018, pp. 1–7.
  * [9] A. Fuchs, J. Garrison, and T. Demiray, “Aa security-constrained multi-period OPF for the locational allocation of automatic reserves,” in _2017 IEEE Manchester PowerTech_ , 2017.
  * [10] M. Farivar and S. H. Low, “Branch flow model: Relaxations and convexification-part i,” _IEEE Transactions on Power Systems_ , vol. 28, no. 3, pp. 2554–2564, 2013.
  * [11] M. Lubin, Y. Dvorkin, and L. Roald, “Chance constraints for improving the security of ac optimal power flow,” _IEEE Transactions on Power Systems_ , vol. 34, no. 3, pp. 1908–1917, 2019.
  * [12] S. Abhyankar, S. Peles, A. Mancinelli, and C. Rutherford, “Exago git repository,” https://gitlab.pnnl.gov/exasgd/frameworks/exago.
  * [13] ——, “ExaGO Manual version 1.0,” https://gitlab.pnnl.gov/exasgd/frameworks/exago/-/blob/master/docs/manual/manual.pdf.
  * [14] R. D. Zimmerman and C. E. Murillo-Sanchez, “MATPOWER Optimal Scheduling Tool MOST 1.1 User’s Manual,” https://matpower.org/docs/MOST-manual-1.1.pdf, 2020.
  * [15] R. D. Zimmerman, C. E. Murillo-Sánchez, and R. J. Thomas, “MATPOWER: Steady-state operations, planning, and analysis tools for power systems research and education,” _IEEE Transactions on Power Systems_ , vol. 26, no. 1, pp. 12–19, 2011.
  * [16] L. Thurner, A. Scheidler, F. Schäfer, J.-H. Menke, J. Dollichon, F. Meier, S. Meinecke, and M. Braun, “Pandapower—an open-source python tool for convenient modeling, analysis, and optimization of electric power systems,” _IEEE Transactions on Power Systems_ , vol. 33, no. 6, pp. 6510–6521, 2018\.
  * [17] “PowerModels.jl documentation,” https://lanl-ansi.github.io/PowerModels.jl/stable/, 2021.
  * [18] S. Balay, S. Abhyankar, M. F. Adams, J. Brown, P. Brune, K. Buschelman, L. Dalcin, A. Dener, V. Eijkhout, W. D. Gropp, D. Karpeyev, D. Kaushik, M. G. Knepley, D. A. May, L. C. McInnes, R. T. Mills, T. Munson, K. Rupp, P. Sanan, B. F. Smith, S. Zampini, H. Zhang, and H. Zhang, “PETSc users manual,” Argonne National Laboratory, Tech. Rep. ANL-95/11 - Revision 3.12, 2019. [Online]. Available: https://www.mcs.anl.gov/petsc
  * [19] D. A. Beckingsale, J. Burmark, R. Hornung, H. Jones, W. Killian, A. J. Kunen, O. Pearce, P. Robinson, B. S. Ryujin, and T. R. Scogland, “Raja: Portable performance for large-scale scientific applications,” in _2019 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC)_. IEEE, 2019, pp. 71–81.
  * [20] D. A. Beckingsale, M. J. Mcfadden, J. P. Dahm, R. Pankajakshan, and R. D. Hornung, “Umpire: Application-focused management and coordination of complex hierarchical memory,” _IBM Journal of Research and Development_ , vol. 64, no. 3/4, pp. 00–1, 2019.
  * [21] S. Abhyankar, S. Peles, A. Mancinelli, and C. Rutherford, “Evaluation of AC optimal power flow problems on graphical processing units,” in _Proceedings of IEEE PES General Meeting, 2021_ , 2021.
  * [22] R. P. O’Neill, A. Castillo, and M. B. Cain, “The IV formulation and linear approximations of the AC optimal power flow problem (OPF Paper 2),” _FERC Staff Technical Paper_ , no. December, pp. 1–18, 2012. [Online]. Available: http://www.ferc.gov/industries/electric/indus-act/market-planning/opf-papers/acopf-2-iv-linearization.pdf
  * [23] A. Waechter and L. Biegler, “On the implementation of an interior-point filter line-search algorithm for large scale nonlinear programming,” _Mathematical Programming_ , vol. 106, no. 1, pp. 25–57, 2006.
  * [24] Cosmin Petra, “HiOP User Guide version 0.3,” https://github.com/LLNL/hiop/blob/master/doc/hiop_usermanual.pdf, 2017.
  * [25] C. G. Petra, I.Aravena Solis, N. Gawande, V. Amatya, A. Li, J. Li, S. Abhyankar, S. Peles, M. Schanen, K. Kim, A. Maldonado, M. Anitescu, “ExaSGD - Optimization grid dynamics at Exascale,” https://ecpannualmeeting.com/assets/overview/posters/exasgd_poster_3column_v4.pdf, 2020\.
  * [26] A. B. Birchfield, T. Xu, K. M. Gegner, K. S. Shetye, and T. J. Overbye, “Grid structural characteristics as validation criteria for synthetic networks,” _IEEE Transactions on Power Systems_ , vol. 32, no. 4, pp. 3258–3265, 2017\.
  * [27] C. Petra, “HIOP – HPC solver for optimization.” [Online]. Available: https://github.com/LLNL/hiop
  * [28] “Summit User Guide.” [Online]. Available: https://docs.olcf.ornl.gov/systems/summit_user_guide.html
  * [29] PowerWorld Corporation, “Powerworld simulator optimal power flow reference,” https://www.powerworld.com/knowledge-base/powerworld-simulator-optimal-power-flow-reference, 2021\.


# SEER: A Knapsack approach to Exemplar Selection
for In-Context HybridQA

Jonathan Tonglet1,2, Manon Reusens2, Philipp Borchert2,3, Bart Baesens2,4
1Ubiquitous Knowledge Processing Lab (UKP Lab),
Department of Computer Science and Hessian Center for AI (hessian.AI),
TU Darmstadt
2 Research Centre for Information Systems Engineering (LIRIS),
Faculty of Economics and Business, KU Leuven
3IESEG School of Management, 3 Rue de la Digue, 59000 Lille, France
4Department of Decision Analytics and Risk, University of Southampton
www.ukp.tu-darmstadt.de

###### Abstract

Question answering over hybrid contexts is a complex task, which requires the
combination of information extracted from unstructured texts and structured
tables in various ways. Recently, In-Context Learning demonstrated significant
performance advances for reasoning tasks. In this paradigm, a large language
model performs predictions based on a small set of supporting exemplars. The
performance of In-Context Learning depends heavily on the selection procedure
of the supporting exemplars, particularly in the case of HybridQA, where
considering the diversity of reasoning chains and the large size of the hybrid
contexts becomes crucial. In this work, we present Selection of ExEmplars for
hybrid Reasoning (SEER), a novel method for selecting a set of exemplars that
is both representative and diverse. The key novelty of SEER is that it
formulates exemplar selection as a Knapsack Integer Linear Program. The
Knapsack framework provides the flexibility to incorporate diversity
constraints that prioritize exemplars with desirable attributes, and capacity
constraints that ensure that the prompt size respects the provided capacity
budgets. The effectiveness of SEER is demonstrated on FinQA and TAT-QA, two
real-world benchmarks for HybridQA, where it outperforms previous exemplar
selection methods111Code available at github.com/jtonglet/SEER.

##  1 Introduction

Hybrid documents, which combine tables and text paragraphs, are prevalent in
various industries such as finance, healthcare, and manufacturing. The
development of question-answering systems capable of effectively handling
these documents holds the potential to automate business processes and enhance
accessibility to the information they contain. Several benchmarks anchored in
the financial domain have been introduced for hybrid question answering
(HybridQA) (Chen et al., 2021c; Zhu et al., 2021; Zhao et al., 2022). Figure 1
presents an example from the FinQA dataset. Despite ongoing progress, current
HybridQA models have yet to achieve human expert performance Zhu et al.
(2021); Chen et al. (2021c).
Recently, In-Context Learning (ICL) with Large Language Models (LLMs) has
shown great performance on reasoning tasks. In this setting, a set of
exemplars, i.e. training instances with their answers, are provided as part of
the input prompt to assist LLMs in generating the correct answer. ICL is an
inference time technique that keeps the LLMs parameters frozen (Brown et al.,
2020). The performance of ICL depends heavily on the quality of the provided
exemplar set Liu et al. (2022a). Various strategies, ranging from random
selection to similarity-based retrieval, have been proposed to tackle this
problem Liu et al. (2022b); Rubin et al. (2022); Li and Qiu (2023); Lu et al.
(2023). When selecting exemplars for HybridQA, special consideration must be
given to the unique challenges of the task, including diverse reasoning
chains, large context sizes (Chen et al., 2021c; Zhao et al., 2022), and the
limited correlation between the question and its reasoning chain.

Figure 1: An instance of the FinQA dataset (Chen et al., 2021c). Text snippets
of the text paragraphs are shown in the dashed box.

In this work, we propose Knapsack Programs as a framework to model exemplar
selection for ICL. Knapsacks are a family of Integer Linear Programs that
search an optimal subset of items under linear constraints Wolsey (2020). For
a given test instance, a Knapsack Program is solved to obtain the optimal
exemplar set. This expressive framework allows balancing the diversity and
similarity of the selected exemplars while controlling the prompt size with
user-defined linear constraints. We introduce SEER, a novel method to select
exemplars for HybridQA using Knapsack Programs. SEER reduces the candidate set
with a nearest neighbor filtering, and leverages constraint modules to predict
the attributes of the test instance. The attributes of a HybridQA instance are
properties that influence the underlying reasoning chain, e.g., the modality
(table, text, hybrid) and the answer type (span extraction, arithmetic
reasoning, counting). By leveraging constraint modules, SEER shapes the
Knapsack structure to prioritize the selection of exemplars that share similar
attributes with the test instance.
The contributions of this work are as follows: (1) we introduce Knapsack
Programs as a framework for ICL exemplar selection. (2) We propose SEER, a
novel exemplar selection method for In-Context HybridQA. (3) We address all
three challenges of HybridQA exemplar selection at the same time with fine-
grained token budget constraints and attribute-guided selection. Extensive
evaluation of two real-world HybridQA benchmarks shows that SEER outperforms
state-of-the-art exemplar selection methods, especially under restricted token
capacity budgets.

##  2 Related work

In-Context Learning LLMs have shown the ability to perform a wide range of
tasks with only a few exemplars provided as a prompt while keeping all the
model parameters frozen (Brown et al., 2020). However, the performance is
highly dependent on the quality of the provided exemplars (Zhao et al., 2021;
Lu et al., 2022). Hence, several approaches have been explored for exemplar
selection, including nearest neighbor search (Liu et al., 2022a),
reinforcement learning (Zhang et al., 2022; Lu et al., 2023), clustering
(Zhang et al., 2023), search algorithms (Li and Qiu, 2023), and supervised
learning (Rubin et al., 2022; Ye et al., 2023). Rubin et al. (2022) consider
token capacity indirectly by constructing the largest possible prompt with the
selected exemplars. In contrast to previous methods, we consider exemplar
selection as an ILP Knapsack Program, which allows us to specify diversity-
enhancing constraints, directly optimize the token capacity without simple
heuristics, and leverage powerful solvers to find a performant exemplar
selection.
Hybrid Question Answering Chen et al. (2020, 2021b) introduced the task of
HybridQA on open-domain Wikipedia pages. Later on, datasets based on real-
world financial documents were introduced (Zhu et al., 2021; Chen et al.,
2021c; Zhao et al., 2022; Chen et al., 2022b). Previous work has focused on
improving the retriever-generator framework (Lei et al., 2022; Sun et al.,
2022; Zhang and Moshfeghi, 2022). Chen et al. (2022a) use few-shot chain-of-
thoughts to solve the task of HybridQA. However, they focus on improving the
prompt format, while our work focuses on selecting good exemplars.
Integer Linear Programming for NLP Integer Linear Programming has been used in
many NLP tasks (Martins, 2014), including coreference resolution (Denis and
Baldridge, 2007; De Belder and Moens, 2012), sentence compression (De Belder
and Moens, 2010), dependency parsing (Riedel and Clarke, 2006), semantic role
labeling (Roth and Yih, 2005), and translation (Germann et al., 2004). In this
work, we introduce a novel application of ILP in NLP: exemplar selection for
ICL. Furthermore, it is, to the best of our knowledge, the first time that the
Knapsack family of ILP programs is used in NLP.

##  3 Methodology

###  3.1 Integer Linear Programming

Linear Programming (LP) involves maximizing or minimizing an objective
function while adhering to a set of constraints. The objective function
consists of a weighted linear combination of variables, while the constraints
are (in)equalities that involve linear combinations of these variables. These
constraints serve to restrict the value ranges of the variables and capture
their interaction effects (De Belder and Moens, 2010). Integer Linear
Programming (ILP) is a subset of LP, wherein variables are constrained to take
only integer values. ILP is divided into program families, one of which is the
Knapsack. Given a set of items, the Knapsack’s objective is to select the
subset that maximizes the total value while remaining within the maximum
capacity Wolsey (2020).
More formally, the problem can be expressed as:

$\begin{array}[]{ll@{}ll}\text{maximize}&\displaystyle\sum\limits_{i\in
S}w_{i}\>\>x_{i}\\\ \\\ \text{subject to}&\displaystyle\sum\limits_{i\in
S}c_{i}\>\>x_{i}\leq C&\>\>\>\>i\in S\\\ &x_{i}\in\\{0,1\\}&\>\>\>\>i\in
S\end{array}$

where $x_{i}$ is a variable that takes value 1 when item $i$ in set $S$ is
selected, 0 otherwise. $w_{i}$ and $c_{i}$ are parameters representing the
value and cost of item $i$, respectively. $C$ is the maximum capacity of the
Knapsack. Depending on the setting, additional constraints and variables can
be added to the basic Knapsack program template shown above. The Knapsack is
NP-hard, but several algorithms have been developed to find an optimal
solution efficiently, including Branch-and-Bound and Cutting Planes (De Belder
and Moens, 2010). Several solvers provide efficient implementations of these
algorithms (Martins, 2014).

###  3.2 Challenges of Exemplar Selection for HybridQA

Figure 2: Four problem instances from TAT-QA. Similar questions do not always
share similar problem attributes.

When solving HybridQA problems with ICL, the task is to predict an answer $A$
given a question $Q$, a hybrid context consisting of text paragraphs $P$ and a
table $T$, and a set of $n$ exemplars $E=\\{e_{1},...,e_{n}\\}$ where $e_{i}$
is a tuple ($Q_{i}$,$A_{i}$,$P_{i}$,$T_{i}$).

$A=argmax_{a}P\left(a\mid Q,P,T,E\right)$

Prior studies (Liu et al., 2022a; Lu et al., 2023) have demonstrated that the
thoughtful selection of the exemplar set $E$ can improve and stabilize the
performance of ICL over a random selection baseline. However, selecting the
optimal exemplar set poses three challenges for HybridQA problems. First,
there is a high diversity in the type of questions and in the approaches to
solve them. The financial dataset FinQA for example contains more than 300
different numerical formulas. For questions asking to compute a “percentage
change”, the training set counts 12 unique formulas. Given this diversity, it
is not possible to cover all possible reasoning chains with a single set of
exemplars. This challenge is partially addressed by similarity-based exemplar
selection methods Liu et al. (2022a).
Secondly, Figure 2 shows the additional challenge of the low correlation
between the problem’s question and attributes. This might result in prediction
errors, as these problems seem semantically similar, however, they require
different modalities and answer types.
Thirdly, HybridQA problems, especially when dealing with real-world data like
financial documents, involve large contexts. LLMs have a limit to the number
of input and output text tokens they can process, whether it is due to
organizational resource constraints or the inherent limitation of the LLM.
Consequently, it becomes crucial to ensure that tokenized exemplars fit within
the LLM’s capacity while reserving enough tokens for generating the desired
output text. In the following, we propose a new exemplar selection method that
addresses those three challenges by modelling them as objectives and
constraints of a Knapsack program. Notably, this work addresses explicitly the
latter two challenges for the first time, contributing to the advancement of
exemplar selection in this domain.

Figure 3: Overview of SEER’s architecture to select the optimal exemplar set
for a HybridQA problem.

###  3.3 SEER

SEER generates Knapsack programs for exemplar selection in HybridQA. Given the
training set and a test instance, SEER constructs a unique Knapsack program
using nearest neighbor filtering and constraint modules. The selected exemplar
set is the optimal solution to the Knapsack program. These exemplars, along
with the test instance, are provided as prompts to an LLM for predicting the
final answer. Figure 3 provides an overview of SEER’s methodology.
Similarity computation involves calculating cosine similarity between pairs of
HybridQA problems’ questions in the embedding space. The resulting similarity
values serve as coefficients in the objective function. To ensure accurate
comparisons, preprocessing is applied to remove noise. (1) All questions are
lowercased. (2) All punctuation is removed. (3) Dates, numerical values,
locations, and companies are replaced by their NER tags.
Nearest Neighbor filtering involves applying an initial filter Liu et al.
(2022a) to identify the $k$ candidates from the training set that exhibit
substantial surface similarity with the test instance, thus narrowing down the
search space.
Constraint modules predict an attribute of the test instance. We define an
attribute as a characteristic of the reasoning chain of a HybridQA problem.
Attributes include modality, answer type, and number of reasoning steps.
Inferring these attributes involves a standard classification task, where the
question and hybrid context are provided as input, and the output corresponds
to one of the attribute values. The task can be addressed through fine-tuning
or ICL.
A SEER Knapsack is uniquely defined for a test instance by the combination of
the similarity weights and the predicted attributes. As an illustration, the
base template for Knapsack programs with predicted attribute “modality:table”
is formulated as follows, where variable $x_{i}$ takes value 1 if instance $i$
from the candidate set $S$ is selected as an exemplar, 0 otherwise. The
candidate set $S$ is a subset of the original training set, composed of the
$k$ nearest neighbor of the test instance’s. The Knapsack has a double
capacity constraint. Firstly, $M$ is the maximum allowed number of exemplars.
Secondly, $L$ is the maximum combined length of the exemplars $l_{i}$ in
number of tokens. The value of $L$ depends on the backbone LLM.

$\begin{array}[]{ll@{}ll}\text{maximize}&\displaystyle\sum\limits_{i\in
S}w_{i}\>\>x_{i}\\\ \\\ \text{subject to}&\displaystyle\sum\limits_{i\in
S}x_{i}\leq M\\\ &\displaystyle\sum\limits_{i\in S}l_{i}\>\>x_{i}\leq L\\\
&\displaystyle\sum\limits_{i\in S}table_{i}\>\>x_{i}\geq\alpha\>M\\\
&\displaystyle\sum\limits_{i\in S}other_{i}\>\>x_{i}\geq\beta\>M\\\
&x_{i}\in\\{0,1\\}\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>i\in S\\\
\text{where}&other_{i}=text_{i}+hybrid_{i}\\\ \end{array}$

The program contains two or more diversity constraints. The structure of those
constraints depends on the attributes predicted by the constraint modules. All
possible diversity constraint configurations are presented in Appendix B.
Parameter $\alpha$ ranges from 0 to 1 and controls the promotion of exemplars
possessing the predicted attribute, i.e. “table” in our case. Parameter
$\beta$, also in the range [0,1], controls the promotion of the other
attributes. It holds that $\alpha+\beta\leq 1$. $tab_{i}$, $text_{i}$, and
$hybrid_{i}$ are binary parameters with value 1 when the modality of candidate
$i$ is the table, the text, or both, respectively, and 0 otherwise. The
objective is to maximize the sum of the exemplar’s similarity with the test
instance $w_{i}$, while respecting all constraints. Framing exemplar selection
as a Knapsack program has several benefits. It is solvable with efficient
deterministic algorithms. It gives full control to the end user over the
prompt capacity in tokens and explicitly allows the user to balance the
similarity and diversity of the selected exemplars in terms of the test
instance’s attributes. SEER’s pseudo-code is shown in Appendix A.

###  3.4 LLM Code Generation

The ICL exemplars selected by SEER and the test instance are concatenated and
provided as a prompt to an LLM. To remove irrelevant text paragraphs from the
hybrid context in relation to the question, we employ a pre-trained text
retriever. Recent studies have demonstrated the benefits of formulating answer
derivation as Python code (Chen et al., 2022a; Gao et al., 2023; Mishra et
al., 2022). Inspired by these approaches, we adopt a code generation
formulation instead of text generation. The resulting code is executed using
an external engine to derive the answer. Figure 1 depicts the conversion from
the original text answer to a Python code answer.

##  4 Experiments

###  4.1 Datasets

We evaluate SEER on two HybridQA datasets with real-world contexts: FinQA
(Chen et al., 2021c) and TAT-QA (Zhu et al., 2021), both anchored in the
financial domain.
FinQA comprises 8,281 problems, each containing a context with a table and multiple text paragraphs. The answers in FinQA are expressed in a domain-specific language as a sequence of operators with two operands each. These expressions are then translated to Python code using an automated script, with each operation on a separate line. Tables are linearized with “ | ” as column delimiter and “ \n ” as row delimiters. A prompt example is provided in Figure 1. This dataset includes a constraint module for predicting the modality.
TAT-QA consists of 16,552 problems. Since the ground truth answers for the
test set of TAT-QA are not publicly available, we employ the original dev set
as the test set. To compensate, we create a new dev set of equal size by
splitting the training set. The context consists of text paragraphs and a
table. Some tables are flattened, while others exhibit complex structures with
multiple header levels. The original answers are written as text or equations.
We convert text to Python comments. Equations are written as a one-line
variable assignment. A prompt example is provided in Appendix C. Unlike FinQA
which only contains arithmetic problems, TAT-QA includes other answer types
such as (multi-)span extraction and counting problems. Consequently, TAT-QA
has two constraint modules, one for modality and one for the answer type.
The evaluation of the datasets is based on their respective metrics: execution
accuracy (EA) and program accuracy (PA) (Chen et al., 2021c) for FinQA, and
Exact Match (EM) and numeracy-focused F1 (Dua et al., 2019) for TAT-QA. EA and
EM assess the correctness of the final answer, while PA ensures that the
generated code is mathematically equivalent to the ground truth derivation.
The numeracy-focused F1 (Dua et al., 2019) computes the F1 score over the bag-
of-word representation of the generated answer. For TAT-QA, we exclude the
sub-task of scale prediction and reserve it for future work in multi-task ICL
HybridQA. The LLM has little exposure to syntactic conventions unrelated to
correct reasoning, such as determining the appropriate scale to represent a
percentage ([0,1] or [0,100]). To take this into account, we allow some
flexibility in the answer evaluation script. For further details on the
evaluation procedure, refer to Appendix E.

###  4.2 Baselines

We compare SEER with other exemplar selection strategies for ICL. The Random
baseline randomly selects a set of exemplars from the training set for each
test instance. We define CSP (Constraint Satisfaction Problem) as SEER without
the objective function. A candidate set is randomly selected among those that
meet all the Knapsack’s constraints. The Fixed set baseline employs the same
set of exemplars for every test instance. Specifically, we randomly sample 20
groups of 4 and 8 exemplars from the training set for FinQA and TAT-QA,
respectively. The group exhibiting the highest performance on the dev sets is
selected. KATE (Liu et al., 2022a) selects the k nearest neighbors to the test
instance. It corresponds to SEER without token capacity and diversity
constraints. Diverse KATE, an extension of KATE, divides the training set into
two subsets, one for text problems and one for table problems. An equal number
of nearest neighbors is selected from each subset. This ensures that both
modalities are present in the exemplar set. PromptPG (Lu et al., 2023) is a
reinforcement learning method that trains a policy network to select well-
performing exemplars among a fixed candidate pool. At inference time, the
policy decides which exemplars from the candidate set are selected for a given
test instance. For training, we use the same parameters as Lu et al. (2023),
except the size of the candidate exemplar set which is set to 20 and 40 for
FinQA and TAT-QA respectively. To disentangle the exemplar selection
performance from errors related to constraint modules, we also include results
obtained using ground truth problem attributes, denoted as SEERgold, which
serves as an upper bound estimate for SEER’s performance. Although our primary
focus is on ICL, numerous studies have explored fine-tuning approaches for
HybridQA. We report the results of the SOTA fine-tuning approach. For a
comparison of fine-tuning approaches with SEER, refer to Appendix F.

###  4.3 Implementation details

| FinQA |  | TAT-QA
---|---|---|---
Method | EA | PA |  | EM | F1
PromptPG (Lu et al., 2023) | 53.56 $\pm$ 3e-3 | 24.09 $\pm$ 1e-3 |  | 51.64 $\pm$ 0.27 | 58.86 $\pm$ 0.27
Random | 55.65 $\pm$ 3.35 | 29.5 $\pm$ 17.91 |  | 49.7 $\pm$ 1.38 | 57.31 $\pm$ 1.24
CSP | 60.77 $\pm$ 0.14 | 43.62 $\pm$ 0.23 |  | 57.47 $\pm$ 0.19 | 65.21 $\pm$ 0.13
Fixed set | 64.05 $\pm$ 0.22 | 38.15 $\pm$ 0.08 |  | 66.55 $\pm$ 0.18 | 73.8 $\pm$ 0.11
KATE (Liu et al., 2022a) | 67.07 $\pm$ 0.04 | 58.65 $\pm$ 0.04 |  | 68.9 $\pm$ 0.07 | 75.77 $\pm$ 0.08
Diverse KATE | 67.31 $\pm$ 0.24 | 59.54 $\pm$ 0.19 |  | 61.53 $\pm$ 0.23 | 68.89 $\pm$ 0.21
SEER | 68.85 $\pm$ 0.04 | 59.78 $\pm$ 0.15 |  | 69.68 $\pm$ 0.07 | 76.71 $\pm$ 0.07
SEERgold | 69.25 $\pm$ 0.11 | 60.16 $\pm$ 0.14 |  | 71.32 $\pm$ 0.07 | 78.12 $\pm$ 0.08
SOTA fine-tuned model | 71.07 | 68.94 |  | 73.6 | 81.3
Table 1: Table of main results (%). EA indicates the Execution Accuracy, PA
the Program Accuracy, EM the Exact Match, and F1 the numeracy-focused F1
score. Reported results are averages over three iterations on the test set.
CODEX is the backbone LLM for ICL methods.

We use CODEX, a 175 Billion parameters LLM available through the OpenAI
API222https://openai.com/blog/openai-codex (Chen et al., 2021a), as a backbone
for SEER and all baselines. CODEX has demonstrated exceptional performance in
code generation tasks (Chen et al., 2021a). For the computation of question
and text paragraph embeddings, we employ Sentence-BERT models (Reimers and
Gurevych, 2019). Regarding the constraint modules, we explore two strategies.
Firstly, fine-tuning a BERT model on the training set. Secondly, employing ICL
predictions with one exemplar per attribute value, leveraging CODEX as the
underlying model. The ICL strategy offers the advantage of requiring only a
small number of training instances with attribute labels. The ICL prompt
starts by the text paragraphs, the linearized table and the problem’s
question, in that order. Then, a task-specific instruction is provided at the
end of the prompt. Instruction for modality prediction: “Do you need data from
the table, the text paragraphs, or both (hybrid) to answer this question?
Answer by one of the following: table, text, hybrid.”. Instruction for answer
type prediction: “Does this question require to extract spans from the
document, to count, or to perform an arithmetic reasoning? Answer by one of
the following: span, multi-span, count, arithmetic.”. We set the maximum token
length $L$ to the maximum capacity of the LLM minus the token lengths of the
problem’s context and the longest answer in the training set. This
conservative estimate of $L$ ensures that the exemplars, the problem’s
context, and its answer can all fit within the imposed token limit. We use the
GUROBI solver333https://www.gurobi.com/solutions/gurobi-optimizer to find the
optimal Knapsack solution. The detailed parameter values are listed in
Appendix D. Our experiments are run on a single NVIDIA GeForce RTX 3050 GPU,
except for CODEX inferences, which are performed on the OpenAI servers.

###  4.4 Main results

Table 1 shows the main results, showcasing the superior performance of SEER
compared to all exemplar selection baselines. Notably, SEERgold achieves
better results than SEER by a margin of 0.4 to 1.5%, indicating that accurate
attribute prediction by the constraint modules significantly contributes to
the overall performance. The difference between KATE and SEER on TAT-QA is not
significant. However, SEERgold is significantly different from KATE, showing
that SEER ultimately outperforms KATE with better constraint modules. For the
detailed significance test of the main results, please refer to Appendix G.
The lower performance of random selection reaffirms the necessity of careful
exemplar selection. Despite its relatively low performance, CSP demonstrates
the notable improvements achievable by introducing constraints into the random
selection process, surpassing the outcomes of a purely random approach.
Interestingly, Diverse KATE is marginally stronger than KATE on FinQA but 7%
lower in EM on TAT-QA. While the SOTA fine-tuning models outperform ICL
methods, SEER contributes to closing the gap. The average run time
distribution is as follows: solving the Knapsack with Gurobi takes 0.2
seconds, predicting an attribute requires 0.05 seconds with BERT and 2.95
seconds with CODEX, and predicting the answer with CODEX takes 2.89 seconds,
on average.
Figure 4 presents a comparison between SEER and two strong baselines across
four subsets of FinQA test instances. The set of instances with attributes
correctly predicted by the constraint module, called Correct Attribute
Predicted (CAP), and its complement, Incorrect Attribute Predicted (IAP). The
set of instances with at least one selected exemplar having the correct
attribute, called Correct Attribute Selected (CAS), and its complement,
Incorrect Attribute Selected (IAS). It holds that
$\text{CAP}\subseteq\text{CAS}$, $\text{IAS}\subseteq\text{IAP}$, and
$\text{CAS}\cap\text{IAP}\neq\varnothing$. We conclude that SEER’s hedge over
baselines is the direct consequence of the prediction and selection of correct
attributes. Interestingly, the baselines struggle with instances where
incorrect attributes are predicted and selected, indicating a correlation
between the difficulty of a HybridQA instance and the challenge of predicting
its attributes. Where no correct attribute is selected (IAS), the Fixed set
baseline outperforms SEER and KATE by a slight margin.

Figure 4: EA (%) on four subsets of the FinQA test instances. Results are
averages over three iterations. CAP stands for Correct Attribute Predicted,
CAS for Correct Attribute Selected, IAP for Incorrect Attribute Predicted, IAS
for Incorrect Attribute Selected.

###  4.5 Constraint modules results

Table 2 shows the performance of fine-tuned and ICL constraint modules on the
dev and test sets. On two out of the three attribute prediction tasks, the
fine-tuned BERT module performs best. However, the ICL constraint module’s
performance is always close, making it a viable alternative when attribute
labels are not available in the training set. There is still room to improve
the performance of constraint modules, as shown by the low precision of the
“hybrid” modality in the confusion matrices in Figure 5. As a result, we
decided to treat the “hybrid” modality as an “uncertain” modality attribute.
Hence, the diversity constraints in the “hybrid” setting promote the usage of
all three modalities.

| FinQA |  | TAT-QA
---|---|---|---
| Modality |  | Modality | Answer type
| Dev | Test |  | Dev | Test | Dev | Test
Fine-TunedBERT | 63.98 | 58.93 |  | 56.89 | 55.28 | 95.69 | 91.79
ICLCODEX | 59.9 | 58.15 |  | 63.19 | 62.17 | 87.23 | 87.23
Table 2: Accuracy (%) of constraint modules on the dev and test sets. Figure
5: Confusion matrices of the best constraint modules on the test sets.

##  5 Analysis

How sensitive is SEER to variations in constraint parameters $\alpha$ and
$\beta$?
We analyze SEER’s and SEERgold’s sensitivity to different ($\alpha$,$\beta$)
value pairs. By increasing $\alpha$, we encourage the selection of exemplars
that share the predicted attributes of the test instance. Conversely,
increasing $\beta$ promotes the inclusion of exemplars with different
attributes, thus ensuring diversity and mitigating errors introduced by the
constraint modules. We evaluate the EA and EM on the dev set for the pairs
(50,25), (75,0), (75,25), (100,0). The results, depicted in Figure 6, indicate
that the difference in EA and EM between the lowest and highest performing
configurations is less than 1%. Consequently, we conclude that SEER does not
require extensive tuning of these two parameters to achieve satisfactory
performance. SEERgold performs best when $\alpha$ is set to higher values and
$\beta$ is set to 0. As SEERgold leverages the ground truth attribute, there
is no need to mitigate attribute errors with $\beta$. This finding aligns with
our intuition that guiding exemplar selection based on problem attributes
improves performance.

Figure 6: EM and EA (%) for different pairs of ($\alpha$/$\beta$) values.
Results averaged over 5 iterations on the dev set.

How does SEER perform under different token capacity budgets?
To evaluate the benefits of the double capacity constraint, we evaluate SEER
under different token capacity budgets. While the default token capacity of
CODEX is 4096, multiple reasons might lead to a reduction in that value,
including using another LLM or financial and time constraints. In the
following, we consider capacities of 2048 and 1024 tokens. The 1024 token
budget is too restrictive to even fit most TAT-QA test instances. Hence, we
only consider FinQA for that setting. Our experiment involves varying values
of M, representing the maximum number of exemplars to be included in the
selection. We compare SEER with KATE, the strongest baseline, which disregards
token capacity budgets during the search for the most similar exemplars.
Figures 7 and 8 report the coverage, i.e. the number of prompts that respect
the token budget, and the performance of SEER and KATE. SEER consistently
outperforms KATE across all capacity budgets, showcasing a margin of
improvement of up to 26% for the more restrictive budgets. While imposing
budget restrictions inevitably results in reduced performance, SEER
demonstrates a superior ability to mitigate the negative impact compared to
KATE. Those results could further be improved with a better estimate of the
maximum token capacity $L$. The current estimate is very conservative, as it
saves enough space to generate the longest solution of the training sets, 309
and 800 tokens for FinQA and TAT-QA, respectively. Most real-world solutions
will require fewer tokens than this upper bound.

Figure 7: Performance of SEER under a 2048 token capacity budget, as evaluated
on the dev set. The coverage is the percentage of exemplar set respecting the
budget. The score is the EA (%) for FinQA and EM (%) for TAT-QA. Figure 8:
Performance of SEER under a 1024 token capacity budget, as evaluated on the
dev set of FinQA.

Qualitative Error analysis
We sample 100 incorrect instances from the test sets at random and categorize
them manually in different error categories, summarized in Table 3. On TAT-QA,
incorrect values occur even when the exemplars use the required formula.
Hence, it seems the error results from the limitations of CODEX. This is
particularly true for Counting problems, where 4 or more relevant exemplars
are provided, but CODEX fails to make the correct count for the test instance.
25% of the FinQA value errors result from a poor computation of the similarity
with the candidate exemplars, based on semantically superficial elements.
Furthermore, we observed that 17% of the FinQA operator errors are due to one
missing reasoning step, e.g. the addition of a constant term. The analysis
highlights directions for future work. (1) SEER could benefit from fine-tuning
methods to compute the similarity weights (Rubin et al., 2022). (2) SEER can
be augmented with mechanisms that predict and control the required number of
reasoning steps.

Error category | FinQA | TAT-QA
---|---|---
Values error | 76% | 63 %
Operators error | 45% | 30 %
Ground truth error | 10 % | 0 %
Answer formatting | 0 % | 10 %
Execution error | 4 % | 0 %
Table 3: Percentage of errors per category.

##  6 Conclusion

This paper investigates the problem of exemplar selection for ICL in HybridQA
tasks. We propose ILP as a framework for exemplar selection and introduce
SEER, a novel method based on Knapsack programs. While existing methods only
focus on the high diversity of questions and how to solve them, SEER
explicitly tackles two other key challenges of HybridQA exemplar selection in
the form of integer linear constraints. Specifically, SEER focuses on the
following challenges, the low correlation between the problem’s questions and
attributes on one hand and the large context required to solve these problems
through diversity and capacity constraints. Diversity constraints enhance
performance by selecting exemplars that share the same attributes as the test
instance. Capacity constraints provide fine-grained control to the end-user
over the token budget allocated to the prompt, ensuring sufficient tokens to
generate the desired output. This level of control is beneficial in overcoming
the limitations of the backbone LLM and dealing with financial constraints.
Evaluation on two financial HybridQA datasets shows that SEER outperforms
previous ICL exemplar selection methods, especially under limited token
capacity.
For future research, we plan to explore additional attributes and seek ways to
enhance the overall performance of the constraint modules. Leveraging the
flexibility of the Knapsack framework, we intend to study the potential of
incorporating constraints defined by end users, such as domain experts, to
further refine exemplar selection. Additionally, SEER can be extended beyond
the HybridQA setting to encompass other modalities like images and knowledge
graphs, as well as other tasks, such as hybrid fact-checking (Aly et al.,
2021), data-to-text generation (Parikh et al., 2020), and multimodal fraud
detection (Wang et al., 2023).

## Limitations

We identify two main limitations to the method introduced in this paper.
Firstly, we assumed that we could select ICL exemplars from thousands of
training instances with annotated textual reasoning. However, in real-world
scenarios, obtaining these textual annotations is a time-consuming process
that involves manual labeling. In a recent study by Su et al. (2023), a method
called vote-k was proposed to address this challenge. It involves selecting
candidates to annotate from a large unlabeled pool before performing
similarity-based exemplar selection. By incorporating vote-k into SEER, we can
reduce the reliance on a fully annotated training set.
Secondly, while efficient solvers exist for finding the optimal solution to
SEER’s Knapsack program, it remains an NP-hard problem that can take
exponential time to complete in the worst case. To mitigate this issue, we
impose a computation budget by setting a timeout of 5 seconds for the solver’s
execution. We empirically observed that SEER convergence to the global optimum
is slower in cases where many candidates have a high similarity weight with
the test instance.

## Ethics Statement

Intended Use HybridQA demonstrates extensive applicability in real-world
scenarios, particularly in the fields of finance and education. In finance, it
can reduce the time needed for financial analysts and journalists to process
large volumes of financial reports. Additionally, it enhances accessibility to
finance-related information for a wider audience. In the education sector, it
assists students in solving textbook exercises that involve a hybrid context.
Misuse potential While we have not identified any direct potential for misuse
of SEER in particular, it is crucial to acknowledge some general concerns with
using automatic question answering systems and LLMs. Automatic question
answering systems are not infallible. Blindly relying on the outputs generated
by the model without double-checking the results can have adverse consequences
for end-users, particularly in financial investment analysis. We recommend
careful and critical usage of the content generated by LLMs, both with and
without SEER. To mitigate this risk, we see potential in research on verifiers
for program generation (Ni et al., 2023) and on new explainability methods for
LLMs.

## Acknowledgements

This research has been funded by the Statistics Flanders research cooperation
agreement on Data Science for Official Statistics and supported by the LOEWE
initiative (Hesse, Germany) within the emergenCITY center. We express our
gratitude to the Policy Research unit at OpenAI for providing access to the
language model CODEX via the OpenAI Research Access Program.

## References

  * Aly et al. (2021) Rami Aly, Zhijiang Guo, Michael Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021.  Feverous: Fact extraction and verification over unstructured and structured information.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks_ , volume 1. Curran.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Chen et al. (2021a) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021a.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_.
  * Chen et al. (2021b) Wenhu Chen, Ming-wei Chang, Eva Schlinger, William Wang, and William Cohen. 2021b.  Open question answering over tables and text.  _International Conference on Learning Representations (ICLR)_.
  * Chen et al. (2022a) Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2022a.  Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.  _arXiv preprint arXiv:2211.12588_.
  * Chen et al. (2020) Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020.  HybridQA: A dataset of multi-hop question answering over tabular and textual data.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1026–1036, Online. Association for Computational Linguistics.
  * Chen et al. (2021c) Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021c.  FinQA: A dataset of numerical reasoning over financial data.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3697–3711, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Chen et al. (2022b) Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022b.  ConvFinQA: Exploring the chain of numerical reasoning in conversational finance question answering.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 6279–6292, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * De Belder and Moens (2010) Jan De Belder and Marie-Francine Moens. 2010.  Integer linear programming for dutch sentence compression.  In _Proceedings of the 11th International Conference on Computational Linguistics and Intelligent Text Processing_ , CICLing’10, page 711–723, Berlin, Heidelberg. Springer-Verlag.
  * De Belder and Moens (2012) Jan De Belder and Marie-Francine Moens. 2012.  Coreference clustering using column generation.  In _Proceedings of COLING 2012: Posters_ , pages 245–254, Mumbai, India. The COLING 2012 Organizing Committee.
  * Denis and Baldridge (2007) Pascal Denis and Jason Baldridge. 2007.  Joint determination of anaphoricity and coreference resolution using integer programming.  In _Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference_ , pages 236–243, Rochester, New York. Association for Computational Linguistics.
  * Dua et al. (2019) Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019.  DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2368–2378, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gao et al. (2023) Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023.  Pal: Program-aided language models.  In _International Conference on Machine Learning_ , pages 10764–10799. PMLR.
  * Germann et al. (2004) Ulrich Germann, Michael Jahr, Kevin Knight, Daniel Marcu, and Kenji Yamada. 2004.  Fast and optimal decoding for machine translation.  _Artificial Intelligence_ , 154(1):127–143.
  * Lei et al. (2022) Fangyu Lei, Shizhu He, Xiang Li, Jun Zhao, and Kang Liu. 2022.  Answering numerical reasoning questions in table-text hybrid contents with graph-based encoder and tree-based decoder.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 1379–1390, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Li et al. (2022) Chenying Li, Wenbo Ye, and Yilun Zhao. 2022.  FinMath: Injecting a tree-structured solver for question answering over financial reports.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 6147–6152, Marseille, France. European Language Resources Association.
  * Li and Qiu (2023) Xiaonan Li and Xipeng Qiu. 2023.  Finding supporting examples for in-context learning.  _arXiv preprint arXiv:2302.13539_.
  * Liu et al. (2022a) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022a.  What makes good in-context examples for GPT-3? In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , pages 100–114, Dublin, Ireland and Online. Association for Computational Linguistics.
  * Liu et al. (2022b) Junpeng Liu, Kaiyu Huang, Jiuyi Li, Huan Liu, Jinsong Su, and Degen Huang. 2022b.  Adaptive token-level cross-lingual feature mixing for multilingual neural machine translation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 10097–10113, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Lu et al. (2023) Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. 2023.  Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning.  In _International Conference on Learning Representations (ICLR)_.
  * Lu et al. (2022) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8086–8098, Dublin, Ireland. Association for Computational Linguistics.
  * Martins (2014) André F. T. Martins. 2014.  Linear programming decoders in natural language processing: From integer programming to message passing and dual decomposition.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts_ , Doha, Qatar. Association for Computational Linguistics.
  * Mishra et al. (2022) Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. 2022.  LILA: A unified benchmark for mathematical reasoning.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5807–5832, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Nararatwong et al. (2022) Rungsiman Nararatwong, Natthawut Kertkeidkachorn, and Ryutaro Ichise. 2022.  Enhancing financial table and text question answering with tabular graph and numerical reasoning.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 991–1000, Online only. Association for Computational Linguistics.
  * Ni et al. (2023) Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. 2023.  Lever: Learning to verify language-to-code generation with execution.  In _International Conference on Machine Learning_ , pages 26106–26128. PMLR.
  * Parikh et al. (2020) Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020.  ToTTo: A controlled table-to-text generation dataset.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1173–1186, Online. Association for Computational Linguistics.
  * Pi et al. (2022) Xinyu Pi, Qian Liu, Bei Chen, Morteza Ziyadi, Zeqi Lin, Qiang Fu, Yan Gao, Jian-Guang Lou, and Weizhu Chen. 2022.  Reasoning like program executors.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 761–779, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Riedel and Clarke (2006) Sebastian Riedel and James Clarke. 2006.  Incremental integer linear programming for non-projective dependency parsing.  In _Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing_ , pages 129–137, Sydney, Australia. Association for Computational Linguistics.
  * Roth and Yih (2005) Dan Roth and Wen-tau Yih. 2005.  Integer linear programming inference for conditional random fields.  In _Proceedings of the 22nd International Conference on Machine Learning_ , ICML ’05, page 736–743, New York, NY, USA. Association for Computing Machinery.
  * Rubin et al. (2022) Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022.  Learning to retrieve prompts for in-context learning.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2655–2671, Seattle, United States. Association for Computational Linguistics.
  * Su et al. (2023) Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2023.  Selective annotation makes language models better few-shot learners.  In _International Conference on Learning Representations (ICLR)_.
  * Sun et al. (2022) Jiashuo Sun, Hang Zhang, Chen Lin, Yeyun Gong, Jian Guo, and Nan Duan. 2022.  Apollo: An optimized training approach for long-form numerical reasoning.  _arXiv preprint arXiv:2212.07249_.
  * Wang et al. (2023) Gang Wang, Jingling Ma, and Gang Chen. 2023.  Attentive statement fraud detection: Distinguishing multimodal financial data with fine-grained attention.  _Decision Support Systems_ , 167:113913.
  * Wolsey (2020) Laurence A. Wolsey. 2020.  _Formulations_, chapter 1. John Wiley & Sons, Ltd.
  * Xia et al. (2023) Yuancheng Xia, Feng Li, Qing Liu, Li Jin, Zequn Zhang, Xian Sun, and Lixu Shao. 2023.  Reasonfuse: Reason path driven and global–local fusion network for numerical table-text question answering.  _Neurocomputing_ , 516:169–181.
  * Ye et al. (2023) Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023.  Compositional exemplars for in-context learning.  In _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 39818–39833. PMLR.
  * Zhang and Moshfeghi (2022) Jiaxin Zhang and Yashar Moshfeghi. 2022.  Elastic: Numerical reasoning with adaptive symbolic compiler.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 12647–12661. Curran Associates, Inc.
  * Zhang et al. (2022) Yiming Zhang, Shi Feng, and Chenhao Tan. 2022.  Active example selection for in-context learning.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9134–9148, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Zhang et al. (2023) Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023.  Automatic chain of thought prompting in large language models.  In _International Conference on Learning Representations (ICLR)_.
  * Zhao et al. (2022) Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang. 2022.  MultiHiertt: Numerical reasoning over multi hierarchical tabular and textual data.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6588–6600, Dublin, Ireland. Association for Computational Linguistics.
  * Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021.  Calibrate before use: Improving few-shot performance of language models.  In _International Conference on Machine Learning_ , pages 12697–12706. PMLR.
  * Zhou et al. (2022a) Fan Zhou, Mengkang Hu, Haoyu Dong, Zhoujun Cheng, Fan Cheng, Shi Han, and Dongmei Zhang. 2022a.  TaCube: Pre-computing data cubes for answering numerical-reasoning questions over tabular data.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2278–2291, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Zhou et al. (2022b) Yongwei Zhou, Junwei Bao, Chaoqun Duan, Youzheng Wu, Xiaodong He, and Tiejun Zhao. 2022b.  UniRPG: Unified discrete reasoning over table and text as program generation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 7494–7507, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Zhu et al. (2021) Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021.  TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3277–3287, Online. Association for Computational Linguistics.
  * Zhu et al. (2023) Fengbin Zhu, Moxin Li, Junbin Xiao, Fuli Feng, Chao Wang, and Tat Seng Chua. 2023.  Soargraph: Numerical reasoning over financial table-text data via semantic-oriented hierarchical graphs.  In _Companion Proceedings of the ACM Web Conference 2023_ , WWW ’23 Companion, page 1236–1244, New York, NY, USA. Association for Computing Machinery.

##  Appendix A The SEER algorithm

SEER is explained in pseudo-code in Algorithm 1.

Algorithm 1 Selection of ExEmplars for hybrid Reasoning (SEER)

Input Test instance $X_{test}$, training set $S_{train}$, candidate pool size
$k$
Output Exemplar selection $E_{selection}$

$E_{candidates}\leftarrow KNN(X_{test},S_{train},k)$$\triangleright$ KNN
cosine similarity

$attributes\leftarrow\\{\\}$

for $module\in constraint\\_modules$ do

$attributes[module.name]\leftarrow module.predict(X_{test})$

end for

$knapsak\leftarrow get\\_ilp(X_{test},attributes)$ $\triangleright$ Generate
the Knapsack

$E_{selection}\leftarrow solve(knapsack,E_{candidates})$ $\triangleright$
Solve the Knapsack program

##  Appendix B Diversity constraint templates

We present the potential configurations of diversity constraints, which are
determined by the predictions of the constraint modules. There exists one
configuration per possible attribute value.
1) Predicted modality: table

$\begin{array}[]{ll@{}ll}\text{subject to}&\displaystyle\sum\limits_{i\in
S}table_{i}\>\>x_{i}\geq\alpha\>M\\\ &\displaystyle\sum\limits_{i\in
S}other\\_{i}\>\>x_{i}\geq\beta\>M\\\
\text{where}&other\\_{i}=text_{i}+hybrid_{i}\end{array}$

2) Predicted modality: text

$\begin{array}[]{ll@{}ll}\text{subject to}&\displaystyle\sum\limits_{i\in
S}text_{i}\>\>x_{i}\geq\alpha\>M\\\ &\displaystyle\sum\limits_{i\in
S}other\\_{i}\>\>x_{i}\geq\beta\>M\\\
\text{where}&other\\_{i}=table_{i}+hybrid_{i}\end{array}$

3) Predicted modality: hybrid

$\begin{array}[]{ll@{}ll}\text{subject to}&\displaystyle\sum\limits_{i\in
S}tab_{i}\>\>x_{i}\geq\beta\>M\\\ &\displaystyle\sum\limits_{i\in
S}text_{i}\>\>x_{i}\geq\beta\>M\\\ &\displaystyle\sum\limits_{i\in
S}hybrid_{i}\>\>x_{i}\geq\beta\>M\\\ \end{array}$

4) Predict answer type: span

$\begin{array}[]{ll@{}ll}\text{subject to}&\displaystyle\sum\limits_{i\in
S}span_{i}\>\>x_{i}\geq\alpha\>M\\\ &\displaystyle\sum\limits_{i\in
S}other_{i}\>\>x_{i}\geq\beta\>M\\\
\text{where}&other_{i}=mspan_{i}+arith_{i}+count_{i}\end{array}$

5) Predict answer type: multi-span

$\begin{array}[]{ll@{}ll}\text{subject to}&\displaystyle\sum\limits_{i\in
S}mspan_{i}\>\>x_{i}\geq\alpha\>M\\\ &\displaystyle\sum\limits_{i\in
S}other_{i}\>\>x_{i}\geq\beta\>M\\\
\text{where}&other_{i}=span_{i}+arith_{i}+count_{i}\end{array}$

6) Predict answer type: arithmetic

$\begin{array}[]{ll@{}ll}\text{subject to}&\displaystyle\sum\limits_{i\in
S}arith_{i}\>\>x_{i}\geq\alpha\>M\\\ &\displaystyle\sum\limits_{i\in
S}other_{i}\>\>x_{i}\geq\beta\>M\\\
\text{where}&other_{i}=span_{i}+mspan_{i}+count_{i}\end{array}$

7) Predict answer type: count

$\begin{array}[]{ll@{}ll}\text{subject to}&\displaystyle\sum\limits_{i\in
S}count_{i}\>\>x_{i}\geq\alpha\>M\\\ &\displaystyle\sum\limits_{i\in
S}other_{i}\>\>x_{i}\geq\beta\>M\\\
\text{where}&other_{i}=span_{i}+mspan_{i}+arith_{i}\end{array}$

##  Appendix C TAT-QA Prompt examples

Figure 9 illustrate an instance from the TAT-QA dataset. The answer type is
multi-span and the modality is text.

Figure 9: TAT-QA instance example

##  Appendix D Implementation details

Table 4 presents an overview of the parameters employed in the components of
the SEER framework.

Parameter | Value
---|---
Preprocessing
Spacy model | ’en_core_web_lg’
NER model | ’bert-base-NER-uncased’
Similarity computation
Backbone | ’all-mpnet-base-v2’
Fine-tuned constraint modules
Model | ’bert-base-cased’
Epochs | 3
Batch size | 32
Learning rate | $5e^{-5}$
ICL constraint modules
Model | ’code-davinci-002’
Temperature | 0
Top P | 1
Max tokens | 5
SEER FinQA
$k$ | 200
M | 4
Constraint module | [’Modality’]
$\alpha$ | 75%
$\beta$ | 0 %
solver | ’GUROBI CMD’
SEER TAT-QA
$k$ | 200
M | 8
Constraint module | [’Modality’,’Answer type’]
$\alpha$ | 50%
$\beta$ | 25 %
solver | ’GUROBI CMD’
Text retrieval
Backbone | ’all-MiniLM-L6-v2’
$k$ | 10
Code Generation
Backbone | ’code-davinci-002’
Temperature | 0
Top P | 1
Max tokens | 256

Table 4: Parameter values

##  Appendix E Complement on evaluation metrics

The ground truth answers of FinQA and TAT-QA have specific syntactic rules
that require access to many training examples to be learned. Those rules are
not related to the semantic correctness of the answer. ICL approaches are
limited to a few exemplars. Hence, they have a limited ability to learn those
syntactic rules. As a result, we provide some flexibility in the evaluation
scripts. The following rules are applied: equivalence of answers written as
percentage or decimal, removal of characters ($,“million”,“billion”, …),
equivalence up to a rounding of 2 decimals, removal of trailing 0s after the
comma. Examples are shown on Table 5

Ground Truth | ((4.1 - 3.9) / 3.9) * 100
---|---
Prediction | ((4.1 - 3.9) / 3.9)
Ground Truth | [’$ 3.4 million’]
Prediction | [’3.4 million’]
Ground Truth | [’29.0’,’27.0’]
Prediction | [’29’,’27’]
Ground Truth | 45.4
Prediction | 45.3981
Table 5: Examples of differences between ground truth and predictions that are
counted as correct by the evaluation scripts.

##  Appendix F Comparison with Fine-Tuned models

Tables 6 and 7 list the best-reported performance of models that followed a
fine-tuning strategy. SEER outperforms several models but lags in performance
compared to the current SOTA.

Model | EA
---|---
APOLLOensemble (Sun et al., 2022) | 71.07
ELASTIC (Zhang and Moshfeghi, 2022) | 68.96
SEER \+ CODEX | 68.85
APOLLO(Sun et al., 2022) | 67.99
ReasonFuse (Xia et al., 2023) | 66.17
SoarGraph (Zhu et al., 2023) | 64.5
FinQANet (Chen et al., 2021c) | 61.24

Table 6: FinQA Fine-Tuned models vs SEER’s performance on the test set (%). Model | EM
---|---
RegHNT Lei et al. (2022) | 73.6
SoarGraph (Zhu et al., 2023) | 70.3
UniRPG Zhou et al. (2022b) | 70.2
SEER \+ CODEX | 69.68
GANO (Nararatwong et al., 2022) | 68.4
FinMath Li et al. (2022) | 60.5
Poet-SQL Pi et al. (2022) | 59.1
TaCube Zhou et al. (2022a) | 53.9
TagOp (Baseline) Zhu et al. (2021) | 55.2
Table 7: TAT-QA Fine-Tuned models vs SEER’s performance on the dev set (%).

There are several trade-offs to consider between fine-tuning and ICL
approaches. Fine-tuning requires updating the model weights, which costs time
and resources. On the other hand, ICL models can be adjusted to new use cases
without retraining. It is interesting to note that both approaches are still
far from the human expert performance, which is 91.16% of EA for the test of
FinQA and 84.1% of EM for the private test set of TAT-QA. Hence, despite a
relatively superior performance of the SOTA fine-tuned models over ICL models,
both approaches are equally far from the human performance. Given the recent
progress made with LLMs, we believe that the importance of ICL will only
increase in the future. Hence, developing strategies for exemplar selection
like SEER is an important direction for future work.

##  Appendix G Significance tests

We evaluate the statistical significance of the results of Table 1 using the
Wilcoxon Signed-Rank non-parametric test. The significance is evaluated by
comparing the average EA and EM over 3 iterations for FinQA and TAT-QA
respectively. Results are reported in Tables 8 and 9.

| Random | CSP | Fixed set | KATE | Diverse KATE | PromptPG | SEER
---|---|---|---|---|---|---|---
CSP | 1e-15 | - | - | - | - | - | -
Fixed set | 2e-27 | 2e-3 | - | - | - | - | -
KATE | 2e-29 | 6e-6 | 0.02 | - | - | - | -
Diverse KATE | 4e-32 | 1e-5 | 0.03 | 0.68 | - | - | -
PromptPG | 4e-13 | 8e-16 | 4e-24 | 3e-29 | 1e-29 | - | -
SEER | 3e-35 | 3e-9 | 1e-4 | 7e-3 | 2e-3 | 2e-36 | -
SEERgold | 3e-38 | 6e-10 | 5e-5 | 2e-3 | 6e-3 | 6e-37 | 0.24

Table 8: Significance tests for FinQA

| Random | CSP | Fixed set | KATE | Diverse KATE | PromptPG | SEER
---|---|---|---|---|---|---|---
CSP | 1e-13 | - | - | - | - | - | -
Fixed set | 5e-41 | 1e-20 | - | - | - | - | -
KATE | 2e-49 | 1e-20 | 0.012 | - | - | - | -
Diverse KATE | 1e-20 | 1e-3 | 1e-4 | 8e-14 | - | - | -
PromptPG | 3e-9 | 2e-5 | 9e-30 | 6e-36 | 8e-12 | - | -
SEER | 2e-51 | 4e-23 | 2e-3 | 0.24 | 1e-15 | 4e-40 | -
SEERgold | 4e-56 | 2e-28 | 3e-6 | 4e-3 | 1e-19 | 5e-45 | 0.015

Table 9: Significance tests for TAT-QA