# Study of the Parent-of-origin effect in monogenic diseases with variable age
of onset.
Application on ATTRv

Flora Alarcon1, Violaine Planté-Bordeneuve2 , Gregory Nuel3

1 Laboratory MAP5 UMR CNRS 8145 Paris University, Paris, France
2 Department of Neurology, Henri Mondor University Hospital, APHP, Créteil,
France. And Paris Est-Créteil University, Créteil, France. And Inserm U.955,
Institut Mondor de Recherche Biomédicale (IMRB), Créteil, France.
3 Stochastics and Biology Group, Department of Probability and Statistics
(LPSM, UMR CNRS 8001), Sorbonne University, Paris, France

## Abstract:

In genetic diseases with variable age of onset, an accurate estimation of the
survival function for the mutation carriers and also modifying factors effects
estimations are important for the management of asymptomatic gene carriers
across life. Among the modifying factors, the gender of the parent
transmitting the mutation (i.e. the parent-of-origin effect) has been shown to
have a significant effect on survival curve estimation on transthyretin
familial amyloid polyneuropathy (ATTRv) families. However, as most genotypes
are unknown, the parent-of-origin must be calculated through a probability
estimated from the pedigree. We propose in this article to extend the method
providing mutation carrier survival estimates in order to estimate the parent-
of-origin effect. The method is both validated on simulated data and applied
to familly samples with ATTRv.

Key words:Semi-parametric estimation, Survival Curve, Parent of origin effect,
Proportionnal hazard model, Familial amyloid polyneuropathy.

##  1 Introduction

In variable age of onset diseases caused by a deleterious gene mutation,
knowledge of the survival function for carrier individual is important both to
understand the underlying mechanism of the disease (like identification of
potential factors that modulate this age of onset) and for prevention
strategies. In the literature on this subject, the age-specific cumulative
distribution function (CDF), named also penetrance function, is preferentially
used. In this paper, like in [1], we will use the classical survival function
(which is simply the complementary of the CDF) to assess the probability of
not being affected by the disease according to the age for mutation carrier
individuals. Note that our survival function hence corresponds to the _cause-
specific_ survival (disease diagnosis) and not to the _overall_ survival.

Recently, a semi-parametric method based on a Cox model was developed to
estimate survival function from familial data ascertained through affected
individual. This method is very efficient and handle the unknown genotypes
through the sum-product algorithm in Bayesian network. The method is described
in detail in [1]. This method has been applied on families affected by a
mendelian disease : the ATTRv, that is the most frequent familial amyloidosis,
with autosomal dominant transmission. A fatal outcome occurs after an average
duration of 10-13 years [2, 3]. This severe diseases shows important
differences in age of onset and thus on survival curve, according to different
covariates as countries, gender, mutations [4, 5, 6]. Especially,
heterogeneity in the survival curves according to gender of the transmitting
parent have been noted on Portugueses ATTRv families [7, 8]. In particular,
Hellman et al. [4] have find that the risk of disease in the carriers was
significantly higher when the mutation was inherited from the mother than from
the father.

In 2009, Bonaïti et al [9] investigate the parent-of-origin effect in a sample
of French and Portuguese families that have already been described in [10].
The covariate for parent of origin factor was not calculated algorithmically
according to pedigrees and genotypical information but was determined
previously to the estimation of the parameters and decided manually by expert-
based review. They found that the penetrance was higher when the mutation was
inherited from the mother than from the father (i.e. the survival curve was
lower when the mutation was inherited from the mother than from the father).
This difference was significant in the Portuguese families but not in the
French sample. This results led them to the hypothesis of a genetically
determined effect through an imprinting phenomenon.

Unfortunately, the gender of the transmitting parent is not known for most of
carriers du to the fact that most of individuals are non-genotyped. And we
should be skeptical as to the estimates made in this context. However, no
simulation studies have been proposed to evaluate the quality of survival
curve estimations in studying the parent-of-origin effect.

In this article, we propose to extend the semi-parametric method based on a
Cox model described in [1] in order to determine the gender of the
transmitting parent. The method for analysing the parent-of-origin effect in
survival curve estimations of Mendelian disease when a part of genotypic
information is unknown, is validated through simulations.

To do the link with the literature about the subject, we illustrate our
results on Portugese fand French datasets of ATTRv already analyzed in [9] and
compare our results with theirs.

##  2 Methods

Survival curve are estimated with a semi-parametric method based on a Cox
model and adapted for pedigree data. As mentioned before, the data available
are families ascertained through at least one affected individual. For all
individuals, we have information about his genotype, phenotype, age, gender,
etc. Families data consist of individuals independents from each other
conditionally to their genotype. The notion of family is taken into account
exclusively in the estimation of the probability to carry the mutation. The
method is described in detail in [1]. In this section, the method is briefly
presented in the particular case of studying the parent-of-origin effect.

### The model

If we denote by $\mathrm{ev}$ the _evidence_ which consists of all the
available information: all time $T_{i}$ and status $\delta_{i}$, the
individual genotype $X_{i}$ and covariates $Z_{i}$ (possibly multidimensional:
gender, comorbidities, ethnicity, etc.), and the genetic testing $G_{i}$ (when
available, $\mathrm{NA}$ is missing) we have:

$\mathbb{P}(X,\mathrm{ev})=\prod_{i}\underbrace{\mathbb{P}(T_{i},\delta_{i}|X_{i},Z_{i})\mathbb{P}(G_{i}|X_{i})^{\mathbf{1}_{G_{i}\neq\mathrm{NA}}}}_{\phi_{i}\left(X_{i}\right)}\prod_{i\in\mathcal{F}}\mathbb{P}\left(X_{i}\right)\prod_{i\notin\mathcal{F}}\mathbb{P}\left(X_{i}|X_{\mathrm{pat}_{i}},X_{\mathrm{mat}_{i}}\right)$

where $\mathcal{F}$ is the set of founders and where $X_{\mathrm{pat}_{i}}$
and $X_{\mathrm{mat}_{i}}$ indicate respectively the genotype of the father
and the mother on individual $i$. Thus, the genotype distribution among
founders follows the Hardy-Weinberg equilibrium with disease allele frequency
$q$, and the conditional distribution for non-founders follows the Mendelian
transmission of alleles. If $G_{i}=\mathrm{NA}$, then
$X_{i}\in\\{0,1_{m},1_{p},2\\}$ corresponding respectively to a non-mutated
individual, a mutated individual whose mutation comes from the father, a
mutated individual whose mutation comes from the mother and finally, to a
homozygous individual.

More precisely, we have :

$\mathbb{P}(T_{i}=t,\delta_{i}=0|X_{i}=x,Z_{i})=\left\\{\begin{array}[]{ll}\exp\left(-\Lambda_{0}(t)\exp(Z_{i}\gamma)\right)&\text{if
$x=1\mathrm{m}$ or $x=2$}\\\
\exp\left(-\Lambda_{0}(t)\exp(\beta+Z_{i}\gamma)\right)&\text{if
$x=1\mathrm{p}$}\\\ 1&\text{if $X_{i}=0$}\end{array}\right.$

and

$\mathbb{P}(T_{i}=t,\delta_{i}=1|X_{i}=x,Z_{i})=\left\\{\begin{array}[]{ll}\exp\left(-\Lambda_{0}(t)\exp(Z_{i}\gamma)\right)\times\left(\lambda_{0}(t)\exp(Z_{i}\gamma)\right)&\text{if
$x=1\mathrm{m}$ or $x=2$}\\\
\exp\left(-\Lambda_{0}(t)\exp(\beta+Z_{i}\gamma)\right)\times\left(\lambda_{0}(t)\exp(\beta+Z_{i}\gamma)\right)&\text{if
$x=1\mathrm{p}$}\\\ 0&\text{if $X_{i}=0$}\end{array}\right.$

which can be simplified as follows :

$\frac{1}{\lambda_{0}(t)}\mathbb{P}(T_{i}=t,\delta_{i}=1|X_{i}=x,Z_{i})=\left\\{\begin{array}[]{ll}\exp\left(-\Lambda_{0}(t)\exp(Z_{i}\gamma)\right)\exp(Z_{i}\gamma)&\text{if
$x=1\mathrm{m}$ or $x=2$}\\\
\exp\left(-\Lambda_{0}(t)\exp(\beta+Z_{i}\gamma)\right)\exp(\beta+Z_{i}\gamma)&\text{if
$x=1\mathrm{p}$}\\\ 0&\text{if $X_{i}=0$}\end{array}\right.$

where $\lambda_{0}$ is the baseline hazard and $\Lambda_{0}$ the baseline
cumulative hazard. Moreover, $\beta$ is the Cox’s model parameter, to be
estimated, for the parent-of-origin effect and $\gamma$ the parameter vector
corresponding to the other covariates. For the genetic testings, we have a
simple model with false positive rate $\varepsilon$ and false negative rate
$\eta$:

$\mathbb{P}(G_{i}=1|X_{i}\neq
0)=1-\varepsilon\quad\mathbb{P}(G_{i}=0|X_{i}=0)=1-\eta$

Note that, in our case, these rates are probably very small (e.g.
$\varepsilon<1/100$ and $\eta<1/1000$)

### EM framework

As $X_{i}$ is either partially observed or not observed, we consider this
variable as latent and use a classical Expectation-Maximization algorithm in
order to maximize the log-likelihood model in parameter of interest.

Model parameters are: $q$, $\beta$, $\gamma$, $\varepsilon$, $\eta$,
$\Lambda_{0}$ and $\lambda_{0}$. In order for the model to be identifiable, we
assume that error rates $\varepsilon$ and $\eta$ are known as well as the
disease allele frequency $q$. And since $\lambda_{0}$ appears only has a
proportional factor in the expression of $\phi_{i}(X_{i})$ we can perform
model inference without explicit value for this parameter. Our aim is
therefore to estimate $\theta=(\beta,\gamma,\Lambda_{0})$ using a classical EM
framework.

In this framework we alternate to steps until convergence:

E-Step

give the weights $w_{i}$ using current parameter $\theta_{\text{old}}$,
computed for all $i$:

$w_{i}^{\mathrm{pat}}=\mathbb{P}\left(X_{i}=1\mathrm{p}|\mathrm{ev};\theta_{\text{old}}\right)\quad\text{and}\quad
w_{i}^{\mathrm{mat}}=\mathbb{P}\left(X_{i}=1\mathrm{m}\text{ or
}X_{i}=2|\mathrm{ev};\theta_{\text{old}}\right)$

M-Step

: create an artificial weighted dataset with the following $2n$ patients

$\begin{array}[]{ccccc}\hline\cr\text{time}&\text{status}&\text{POO}&\text{covariates}&\text{weight}\\\
\hline\cr T_{1}&\delta_{1}&\mathrm{pat}&Z_{1}&w_{1}^{\mathrm{pat}}\\\
\vdots&\vdots&\vdots&\vdots&\vdots\\\
T_{n}&\delta_{n}&\mathrm{pat}&Z_{n}&w_{n}^{\mathrm{pat}}\\\ \hline\cr
T_{1}&\delta_{1}&\mathrm{mat}&Z_{1}&w_{1}^{\mathrm{mat}}\\\
\vdots&\vdots&\vdots&\vdots&\vdots\\\
T_{n}&\delta_{n}&\mathrm{mat}&Z_{n}&w_{n}^{\mathrm{mat}}\\\
\hline\cr\end{array}$

and simply:

  * •

fit a (weighted) Cox’s proportional hazard model with Factor POO and the
covariates to update $\beta$ and $\gamma$

  * •

use non-parametric estimate (_e.g._ Kaplan-Meier or similar) of $\Lambda_{0}$.

### Posterior distributions

For computing the posterior weights $w_{i}^{\mathrm{pat}}$ and
$w_{i}^{\mathrm{mat}}$ of the E-Step one has to integrate the model over all
unobserved (even when partially observed) genotypes $X_{i}$. Since the total
number of configurations for $X_{i}$ is $4^{n}$ it is clearly impossible to
perform this summation with brute force. Fortunately, geneticist know for year
that likelihood computations in genetic model can be performed efficiently
using the Elston-Stewart algorithm[11]. Thanks to this algorithm, it is
therefore possible to compute any posterior distribution as a simple
likelihood ratio:

$\mathbb{P}(X_{i}=x|\mathrm{ev})=\frac{\mathbb{P}(X_{i}=x,\mathrm{ev})}{\mathbb{P}(\mathrm{ev})}$

In the probabilistic graphical model community, such computation can be dealt
efficiently using the sum-product algorithm (also called: belief propagation,
forward/backward, inward/outward, Kalman filter, etc.) in order to obtain in a
single computational pass _all_ $w_{i}^{\mathrm{pat}}$ and
$w_{i}^{\mathrm{mat}}$. NB: in Totir [12] , the authors suggest a
generalization of the Elston-Stewart algorithm allowing to compute all
posterior distribution in a single passe rather than repeating likelhood
computations. Without surprise, Totir’s algorithm is in fact the exact
reformulation of the classical sum-product algorithm.

Practically, initialization is performed by affecting random weights $w_{i}$
(ex: drawn from a uniform distribution on $[0,1]$). EM iterations are stopped
when we observe convergence on test survival estimates (ex: baseline survival
at age $20,40,60,80$).

##  3 Results

### Simulations study

We have simulated $n$ families ($n=100$ or $400$) with 10 individuals as shown
in figure 1.

Figure 1: Simulated family structure

Genotypes were assigned respecting Mendelian transmission and heterozygous
genotypes were ordered according to the gender (pat or mat) of the parent who
transmitted the mutation. So, a new binary covariate (named POO for parent of
origin) are introduced such as $\text{POO}=\mathrm{mat}$ if the mother
transmitted the disease mutation and $\text{POO}=\mathrm{pat}$ if disease
mutation was transmitted by the father. The disease allele frequency was set
to q = 0.20 in our simulated dataset for the sake of speed (without simulating
any ascertainment process). The age at event was simulated according to a
piecewise constant hazard rate function. Moreover a censoring variable was
added that follows a uniform distribution $\mathcal{U}[15,80]$.

To study the parent-of-origin effect, the POO was simulated in a proportional
hasard model. The risk when the mutation is transmitted by the mother is given
by $\lambda_{0}$ as follows and the coefficient parameters in the hazard model
is noted $\beta$ ($\beta=-0.60$ or $\beta=-1.2$ in simulations). Thus, as
$\beta<0$, that’s mean that survival is upper when mutation provides on the
father.

$\lambda_{0}(t)=\left\\{\begin{array}[]{ll}0&\text{ if }t\in[0,20]\\\
0.02&\text{ if }t\in[20,40]\\\ 0.10&\text{ if }t\in[40,60]\\\ 0.05&\text{ if
}t>60\end{array}\right.$

To study the behavior of our method, we will consider different scenarios for
simulations. In the first scenario (S0), all genotypes are set to unobserved,
in the second more realistic scenario (S1), 80% of genotypes are observed in
affected individuals and only 10% are observed in non affected individuals and
in the third one (S2), all genotypes are observed. Moreover, we consider an
other scenario where, in addition to the genotypes, we know, for each
individual, the sex of the parent transmitting the mutation. We call this
scenario “Oracle”. In order to compare the $\beta$ estimate in the different
scenarios, 200 replications are performed.

Figure 2 shows the violin plots according to the scenario for the percentage
of observed genotypes, which allows to show the full distribution of the data.
Thus, each color corresponds to a genotypic scenario for each of the cases A
($n=100$ families was simulated with $\beta=-0.6$), B ($n=400$ families was
simulated with $\beta=-0.6$) and C ($n=100$ families was simulated with
$\beta=-1.2$). For each scenario, the mean and standard deviation are shown
and the horizontal black line represents the true $\beta$ value. In all cases
the $\beta$ estimate are unbiased whatever the scenario and the standard
deviation increases with the percentage of unobserved genotypes. As expected,
the oracle varies little. It is also interesting to look at the number of
iterations needed before convergence of the EM algorithm, depending on the
different scenarios. Figure 3 shows the violin plot including boxplots of the
number of iterations required before the convergence of the EM algorithm
according to scenarios (S0), (S1) and (S2) in the cases A and B. The violin
plot shows that this number increases with the amount of unknown genotypes.
Thus, when all genotypes are observed, the algorithm converges in 65
iterations on average while it converges in 89 iterations on average when no
genotypes are observed. We also observe a higher variability of the number of
iterations when the genotypes are not observed.

Figure 2: Violin Plots according to different scenarios. A : $n=100$ families
; $\beta=-0.6$. B : $n=400$ families ; $\beta=-0.6$. C : $n=100$ families ;
$\beta=-1.2$ Figure 3: Violin plot with box plot of the number of iterations
before EM convergence according to different scenarios. A : $n=100$ families ;
$\beta=-0.6$. C : $n=100$ families ; $\beta=-1.2$

### Application

Our method has been applied to two datasets of Portuguese families and French
families already analyzed in [9]. For data analysis, the disease allele
frequency was set to $q=0.04$ as in [4]. The ascertainment bias was corrected
by a classical method that consists in simply removing the phenotypic
information of the proband, as done in [1]. The parent of origin gender
parameter was not significantly different from zero in French families
($\hat{\beta}=0.114$ with p-value $=0.467$). However, the $\beta$ estimate was
significantly different from zero in Portuguese families ($\hat{\beta}=-0.999$
and a p-value $=5.10^{-10}$) showing that the risk of being affected is higher
when the mutation is transmitted by the mother than by the father. This
difference in survival curves estimated is shown in Figure 4 and is totally
consistent with the results of [9].

Figure 4: Estimation of the survival function according to the sex of the
transmitting parent with 95% confidence intervals for Portugese families

##  4 Conclusion

In this article, we propose an extension of our method in order to take into
account the gender of the transmitting parent in the estimation of the
survival function from familial data in cases of age-dependent genetic
diseases. The probability of the transmitting parent gender is calculated and
its effect is estimated through a proportional hazard model. Our extension is
assessed with simulations and applied on a French and a Portuguese dataset of
ATTRv families. Obviously, the method provides confidence intervals for each
estimation.

The simulation study shows that the method is unbiased and the increase in
variance of the estimate with the number of unobserved genotypes.

We have applied this method to a French and Portuguese dataset and shows a
significant difference of the penetrance function according to the gender of
the transmitted parent. These results are consistent with the analyses already
done in [9], even if the method used in [9] was less precise since the
genotypic probabilities were not calculated. Here we have proposed a unified
method able to take into account precisely these probabilities. Moreovern the
methode allowsi additional variables to be taken into account when estimating
survival curves.

Finally, in this work, genotypic probabilities are calculated using a sum-
product algorithm which is a very general method which can deal efficiently
with very complex pedigree structure (ex: $2000$ individuals with $50$ loops).
Unlike Elston-Stewart algorithm, the sum-product algorithm does not use loop
breaking approaches to deal with loop pedigrees. Instead, the sum-product
algorithm use an auxiliary tree called the junction tree (JT) which basically
is a clique decomposition of the moral graph corresponding to the pedigree
problem. JT and BP are well known is the graph theory (ex: JT can be used to
solve a graph coloring problem) and in the mathematical field of probabilistic
graphical models (Bayesian network, hidden Markov model, decision trees,
Markov networks, etc.).

Conflict of interest. The authors declare no conflict of interest.

## References

  * [1] Flora Alarcon, Violaine Planté-Bordeneuve, Malin Olsson, and Grégory Nuel.  Non-parametric estimation of survival in age-dependent genetic disease and application to the transthyretin-related hereditary amyloidosis.  PloS one, 13(9):e0203860, 2018.
  * [2] Violaine Planté-Bordeneuve and G Said.  Transthyretin related familial amyloid polyneuropathy.  Current opinion in neurology, 13(5):569–573, 2000.
  * [3] O Suhr, Å Danielsson, G Holmgren, and L Steen.  Malnutrition and gastrointestinal dysfunction as prognostic factors for survival in familial amyloidotic polyneuropathy.  Journal of internal medicine, 235(5):479–485, 1994.
  * [4] Urban Hellman, Flora Alarcon, Hans-Erik Lundgren, Ole B Suhr, Catherine Bonaïti-Pellié, and Violaine Planté-Bordeneuve.  Heterogeneity of penetrance in familial amyloid polyneuropathy, attr val30met, in the swedish population.  Amyloid, 15(3):181–186, 2008.
  * [5] G. Holmgren, PMP. Costa, C. Andersson, K. Asplund, L. Steen, L. Beckman, PO. Nylander, A. Teixeira, MJM. Saraiva, and PP. Costa.  Geographical distribution of TTR met 30 carriers in northern Sweden .  J Med Genetic, 31:351–354, 1994.
  * [6] Farida Gorram, Malin Olsson, Flora Alarcon, Gregory Nuel, Intissar Anan, and Violaine Planté-Bordeneuve.  New data on the genetic profile and penetrance of hereditary val30met transthyretin amyloidosis in sweden.  Amyloid, pages 1–7, 2020.
  * [7] A. Sousa, T. Coelho, J. Barros, and J. Sequeiros.  Genetic epidemiology of familial amyloidotic polyneuropathy (FAP)-Type I in Povoa do Varzim and Vila do Conde (North of Portugal) .  Am J Med Genet (Neuropsych Genet), 60:512–521, 1995.
  * [8] Monica Ines, Teresa Coelho, Isabel Conceicao, Filipa Duarte-Ramos, Mamede de Carvalho, and Joao Costa.  Epidemiology of transthyretin familial amyloid polyneuropathy in portugal: a nationwide study.  Neuroepidemiology, 51(3-4):177–182, 2018.
  * [9] Bernard Bonaïti, Flora Alarcon, Catherine Bonaïti-Pellié, and Violaine Planté-Bordeneuve.  Parent-of-origin effect in transthyretin related amyloid polyneuropathy.  Amyloid, 16(3):149–150, 2009.
  * [10] V. Plante-Bordeneuve, J. Carayol, A. Ferreira, D. Adams, F. Clerget-Darpoux, M. Misrahi, G. Said, and C. Bonaïti-Pellié.  Genetic study of transthyretin amyloid neuropathies: carrier risks among French and Portuguese families.  J Med Genet, 40(11):e120, 2003.
  * [11] R.C. Elston and J. Stewart.  A general model for the genetic analysis of pedigree data.  Human Heredity, 21(6):523–542, 1971.
  * [12] Liviu R Totir, Rohan L Fernando, and Joseph Abraham.  An efficient algorithm to compute marginal posterior genotype probabilities for every member of a pedigree with loops.  Genetics Selection Evolution, 41(1):1–11, 2009.


# Learning to Reduce: Towards Improving Performance of
Large Language Models on Structured Data

Younghun Lee  Sungchul Kim  Ryan Rossi  Tong Yu  Xiang Chen

###### Abstract

Large Language Models (LLMs) have been achieving competent performance on a
wide range of downstream tasks, yet existing work shows that inference on
structured data is challenging for LLMs. This is because LLMs need to either
understand long structured data or select the most relevant evidence before
inference, and both approaches are not trivial. This paper proposes a
framework, Learning to Reduce, that fine-tunes a language model with On-Policy
Learning to generate a reduced version of an input structured data. When
compared to state-of-the-art LLMs like GPT-4, Learning to Reduce not only
achieves outstanding performance in reducing the input, but shows
generalizability on different datasets. We further show that the model fine-
tuned with our framework helps LLMs better perform on table QA tasks
especially when the context is longer.

Structured data QA,Table QA,Large Language Models,LLM Prompting,Reinforcement
Learning,Policy Optimization

##  1 Introduction

Recent Large Language Models (LLMs) such as GPT-4 (OpenAI, 2023b), Llama 2
(Touvron et al., 2023), and Vicuna (Chiang et al., 2023), have shown the
ability to understand language and improve performance on a wide range of
downstream tasks (Wei et al., 2022; Kojima et al., 2022; Wang et al., 2022; Yu
et al., 2022; He et al., 2024).

Despite their ability, LLMs find it challenging to understand structured data
such as knowledge graphs, tables, and databases. Not only does the structured
data have structural dependencies among entities, but it also accompanies long
context issues. The maximum input sequence length of the recent LLMs keeps
increasing, yet one cannot put faith in the LLMs’ performance when the context
is long. Liu et al. (2023) show that the performance of ChatGPT (OpenAI,
2023a) on a multi-document QA task drops more than $20\%$ when it is prompted
with 20 documents and the key information about the answer is located in the
middle of the prompt. This implies that LLMs are likely to perform
unsatisfactory on structured data QA because most structured data tend to be
long and the information relevant to the answer can be located in the middle.

To avoid a long context issue of structured data, Jiang et al. (2023) used
ChatGPT to identify the most relevant evidence from the structured data before
performing QA tasks. Unfortunately, finding the most relevant evidence was not
trivial; experimental results show that around $74\%$ and $28\%$ of the errors
came from the incorrect selection of relevant evidence in a KGQA task and a
table QA task, respectively.

In this paper, we explore a method to improve LLMs’ reasoning ability on
structured data by Learning to Reduce; the model efficiently reduces the input
structured data by identifying the relevant evidence to the downstream task.
Specifically, we focus on a table QA as a downstream task, and train a model
to generate a table with reduced rows and columns. Using On-Policy Learning,
we fine-tune a T5 language model based on the rewards computed by whether the
model selects rows and columns that are relevant to the input question.
Experimental results show that our framework achieves better performance in
identifying relevant items from a table QA dataset, WikiTableQuestions
(Pasupat & Liang, 2015). Additionally, Learning to Reduce outperforms other
baseline models including GPT-4 (OpenAI, 2023b) on the generalizability test
on an unseen dataset. Lastly, the reduced table generated by our framework
helps LLMs perform more accurately on a table QA task, especially when the
context is longer.

Figure 1: Inference with an original table (dotted arrow) and with a reduced
table (solid arrow). Given an input question and a table, a language model
(blue hexagon) learns a policy to generate the relevant rows and columns by
getting rewards. By learning the optimal policy, our model generates reduced
tables which leads the fixed LLM model to perform more accurately on QA tasks.

To the best of our knowledge, this is the first attempt to train a language
model to reduce the input context of structured data. The training framework
we suggest is model-agnostic, thus more powerful language models that are
trained with our framework can be used as a pre-prompting tool for any
structured data QA tasks. This would ultimately maximize the reasoning ability
of LLMs as well as the cost efficiency of using them.

##  2 Learning to Reduce

We design a training framework for a language model that learns to generate
the relevant evidence for a given structured data QA instance.

Formally, we define an input space that consists of an input context $\bm{c}$,
a task description $\bm{x}$, and an output $\bm{y}$. In the table QA task, for
instance, $\bm{c}$ is an input table, $\bm{x}$ is an input question, and
$\bm{y}$ is an answer to the question. We use a heuristics $h(\cdot)$ that
identifies a subset of the context that is relevant to the task, namely
$\bm{c_{r}}=h(x,c)$. Our framework trains a language model,
$\theta(\bm{z|x,c})$, that generates a reduced input context $\bm{z}$, using a
target reduced input context $\bm{c_{r}}$. Our final objective is to help LLMs
better perform on the task, thus we prompt an LLM with reduced context
$\bm{z}$ to generate an output. Denoting $\psi_{LLM}$ as a fixed LLM model
used for a table QA task, the predicted output $\bm{\hat{y}}$ follows
$\bm{\hat{y}}\sim\psi_{LLM}(\cdot|\bm{x,z})$. Figure 1 illustrates how our
proposed model works.

###  2.1 Language Model as a Policy Network

We consider a language model as a policy network. Following an approach
proposed by Li et al. (2023), we first fine-tune the language model parameters
before applying policy learning objectives.

Supervised fine-tuning requires a dataset that has relevant rows and columns
annotated for input questions and tables. To automatically identify relevant
items, we use a table QA dataset which has a text-to-SQL annotation. The
heuristics for automatically identifying relevant rows and columns is to
execute SQL queries while iteratively removing rows and columns one by one
from the input tables; if executing a SQL query can generate an answer on a
table even after removing some rows and columns, then the removed items are
irrelevant to the input question.

Using the WikiTableQuestions (WTQ) dataset (Pasupat & Liang, 2015) and its
corresponding text-to-SQL annotation from SQUALL (Shi et al., 2020), we
identify relevant rows and columns using the aforementioned heuristics. WTQ
dataset was selected because the questions in the dataset are simple enough to
be translated into SQL queries, yet the state-of-the-art LLMs’ inference
ability on this dataset is not reliable compared to other widely used table QA
datasets111Jiang et al. (2023) reported ChatGPT’s accuracy on WTQ is 43.3,
while it achieves 51.6 on WikiSQL (Zhong et al., 2017) and 82.9 on TabFact
(Chen et al., 2019). Appendix A describes the details of the data.

We use a sequence-to-sequence language model with FLAN-T5-Large (Chung et al.,
2022) pre-trained checkpoint as a base model. For better optimization, we
train two models separately, one for the column reduction and the other for
the row reduction. The two models work in sequence; the column reduction
language model is applied and generates a table with relevant columns, then
the row reduction model gets a column-reduced table and determines relevant
rows. Thus the column reduction model gets all column headers in the prompt,
whereas the row reduction model gets rows only with relevant columns. More
details of the model choice and prompt designs are described in Appendix B.

###  2.2 Policy Optimization

We consider fine-tuned language models as initial policy network parameters
and further optimize the models by maximizing the rewards. The models get
positive rewards for selecting the correct relevant items, and get two types
of negative rewards.

Type I errors occur when the models generate rows and columns that are not
relevant. Having this error is not preferred but it is not critical; when the
policy network generates a table with some irrelevant rows and columns, LLMs
can still perform a QA task as long as the relevant items are in the table. In
this case, the policy network gets a small negative reward.

On the other hand, type II errors need to be considered more seriously, as
this happens when the models fail to generate relevant rows and columns. In
this case, LLMs would have to perform inference without the necessary
information and fail to correctly answer the question. We consider this as a
critical error and the policy network gets a high negative reward.

To formally define a reward function, consider $\bm{z}$ and $\bm{c_{r}}$ as a
concatenated text of rows and columns from a set $Z$ and $C_{r}$. For
instance, $Z=\\{\text{col1},\text{row1}\\}$ when $\bm{z}$ is “col1, row1”. The
model maximizes the reward $\mathcal{R}$, which is defined as:

$\mathcal{R}=\lambda_{p}|Z\cap
C_{r}|+\lambda_{n1}|Z-C_{r}|+\lambda_{n2}|C_{r}-Z|$

$\lambda_{p}>0$, $\lambda_{n1}<0$, $\lambda_{n2}\ll 0$ are coefficients for
correct predictions, type I errors, and type II errors, respectively. Details
of the policy network architecture, implementation, and reward optimization
are described in Appendix C.

##  3 Experiments

Baselines: First, we fine-tune a RoBERTa token classifier (Liu et al., 2019)
as a baseline. Similar to supervised fine-tuning of a language model, input is
given with question and column/row values, and the token classifier models
generate tags for each column/row whether they are relevant or not. Another
baseline is a zero-shot GPT-4 model222We use version gpt-4-0613. In this
setting, we provide input questions with column/row values, then ask the GPT-4
model to select relevant rows and columns. Lastly, we use fine-tuned FLAN-T5
models as a baseline. This is to measure the effectiveness of the On-Policy
Learning component in context reduction.

Table 1:  Input context reduction tested on WTQ test set (WTQ) and on an
unseen test set (HybQA).  Recall on Context Reduction
---
Column Reduction | WTQ | HybQA
RoBERTa Token Clf | 89.08 | 77.32
Zero-shot GPT-4 | 74.03 | 71.48
Fine-tuned FLAN-T5 | 90.19 | 82.72
$\boldsymbol{\star}$ Learning to Reduce (ours) | 91.82 | 87.22
Row Reduction | WTQ | HybQA
RoBERTa Token Clf | 95.06 | 60.35
Zero-shot GPT-4 | 86.13 | 92.12
Fine-tuned FLAN-T5 | 95.21 | 90.22
$\boldsymbol{\star}$ Learning to Reduce (ours) | 96.17 | 93.78

Evaluation Metrics: As described in 2.2, context reduction models are
considered better when they generate relevant items as many as possible, even
though their generation includes irrelevant items. Thus we use recall scores
as a metric for context reduction and measure how many relevant rows and
columns are selected from the models. More discussions and results of the
precision scores are reported in Appendix D.

Dataset: The models are tested on the WTQ test set as well as on Hybrid QA
(Chen et al., 2020). Hybrid QA is a table QA task that not only requires
reasoning on a given table but also needs more complex language understanding.
We test the models on Hybrid QA to evaluate their generalizability. The model
needs to be robust to different data distributions because fine-tuning is not
always possible. Thus we argue that the model’s performance on Hybrid QA is
more significant when we consider deploying the model for more general usages.
Table 1 shows the experimental results.

###  3.1 Performance and Generalizability

RoBERTa token classifier outperforms zero-shot GPT-4 on both column and row
reduction. However, the model’s performance significantly drops when it is
tested on Hybrid QA; the column reduction model shows a $12\%$ drop in column
reduction and a $35\%$ drop in row reduction. This result shows that fine-
tuned BERT-based token classifiers lack generalizability.

GPT-4 is a very large and highly contextual model, yet it has difficulty
understanding the relatedness between a table and a question. This result
aligns with the selection error statistics in the existing research (Jiang et
al., 2023). As the model is not fine-tuned on a specific dataset, its
performance on WTQ and Hybrid QA are similar as expected. A slight drop from
$74\%$ to $71.5\%$ on column reduction implies that Hybrid QA is a little more
challenging dataset than WTQ.

Regarding the language model approaches, On-Policy Learning helps improve
performance. Not only does the Learning to Reduce outperform other baselines
on the WTQ test set, but it also exhibits remarkable results on Hybrid QA.
This result implies that policy networks can capture more general knowledge
about context reduction and opens the possibility of applying Learning to
Reduce on other structured data QA tasks without fine-tuning again.

Figure 2: Accuracy (precision) of GPT-4 model on WTQ test set with different
input context tables. Reducing both rows and columns (red and purple) is more
powerful when the context is longer.

###  3.2 Downstream Task Performance

The ultimate goal of our framework is to help LLMs better perform table QA
tasks with reduced rows and columns. Figure 2 illustrates the accuracy of the
GPT-4 model in answering the WTQ questions when prompted with different
context inputs333We compute precision (i.e. how many GPT-4 predictions are
correct out of all prompted instances). Note that the x-axis is the number of
tokens of the input tables before context reduction.

Our hypothesis, that LLMs lack inference ability on structured data due to its
long context, is validated by the results of original input table; the
accuracy of the original input drops as the table gets longer (blue line in
Figure 2).

We assumed that the input table representations with only columns reduced can
moderately improve the inference ability of LLMs. This is because the majority
of tables in WTQ datasets are long-narrow tables (i.e. fewer columns than
rows), thus removing a few columns would reduce a great number of input
tokens. However, the green and orange line shows that reducing only columns
does not improve the overall performance of LLMs.

The performance of both rows and columns reduced (red and purple line), on the
other hand, is significantly more stable, especially when the original input
table is longer. The purple line shows that our proposed method provides
better QA results compared to pure LLM-based approaches (blue), even though
the policy network’s recall on context reduction is not 100% accurate. Our
proposed method proves to be more impactful for maximizing LLMs’ ability to
understand long structured data. More results with a different LLM,
GPT-3.5-turbo, are described in Appendix D.

##  4 Related Work

LLM on Structured Data: Various approaches have been proposed to integrate
structured data into LLM prompts. Ye et al. (2023) proposed DATER which uses
LLMs to decompose inputs into sub-questions/-tables, then executes queries to
improve the reasoning ability of the LLM. Hegselmann et al. (2023) converted
tables into natural language and fine-tuned LLMs on downstream tasks.
StructGPT (Jiang et al., 2023) iteratively gets the most relevant evidence
from structured data, then prompts LLMs for better inference.

LLM Prompt Engineering: Another direction to improve the downstream task
performance of a fixed LLM is to optimize the prompts. GrIPS (Prasad et al.,
2023) uses a gradient-free, edit-based instruction search method. AutoPrompt
(Shin et al., 2020) and RLPrompt (Deng et al., 2022) train the model to
generate tokens that help LLMs better perform on downstream tasks. Directional
Stimulus Prompting (Li et al., 2023) trains the model to generate keywords or
hints that can directly help LLMs to accomplish downstream tasks.

##  5 Discussion and Conclusion

There are a few areas that can further improve the paper. The paper would
benefit from experiments on more table QA datasets; the baseline models as
well as Learning to Reduce are only tested on Hybrid QA datasets for measuring
the models’ generalizability. Similarly, the downstream task performance
results would be more strongly supported by experimenting with more table QA
datasets—measuring whether LLMs’ inference ability improve or not on other
table QA datasets after context reduction.

The goal of this paper can be stretched to develop a framework that improves
performance on LLMs on any type of structured data, including knowledge base
and databases. Testing the framework with other types of structured data
remains as future work.

Lastly, our framework can benefit from adding task-specific rewards. There are
some cases where our model generates a very small number of rows and columns.
Even though they are correct reduction, LLMs end up being confused with the
input with too much reduced rows and columns. If there are more reward signals
from how LLMs perform on downstream tasks with the reduced context input, the
model would be able to learn better representations. We leave this as future
work.

To conclude, we propose Learning to Reduce, a framework that learns to
generate a reduced context. The framework adopts a novel reward function for
context reduction and fine-tunes a language model through On-Policy Learning.
Not only does our framework outperform a state-of-the-art LLM (e.g. GPT-4) in
context reduction, but it is also generalizable to an unseen dataset. We
further show that the output of our framework improves the LLM’s performance
on downstream tasks. Learning to Reduce is model-agnostic thus it can be
applied to different types of structured data such as knowledge base and
databases in future studies.

## References

  * Chen et al. (2019) Chen, W., Wang, H., Chen, J., Zhang, Y., Wang, H., Li, S., Zhou, X., and Wang, W. Y.  Tabfact: A large-scale dataset for table-based fact verification.  _arXiv preprint arXiv:1909.02164_ , 2019.
  * Chen et al. (2020) Chen, W., Zha, H., Chen, Z., Xiong, W., Wang, H., and Wang, W. Y.  HybridQA: A dataset of multi-hop question answering over tabular and textual data.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pp. 1026–1036, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.findings-emnlp.91.
  * Chiang et al. (2023) Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., et al.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  _See https://vicuna. lmsys. org (accessed 14 April 2023)_ , 2(3):6, 2023.
  * Chung et al. (2022) Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., et al.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ , 2022.
  * Deng et al. (2022) Deng, M., Wang, J., Hsieh, C.-P., Wang, Y., Guo, H., Shu, T., Song, M., Xing, E., and Hu, Z.  RLPrompt: Optimizing discrete text prompts with reinforcement learning.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pp. 3369–3391, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.emnlp-main.222.
  * He et al. (2024) He, Q., Zeng, J., Huang, W., Chen, L., Xiao, J., He, Q., Zhou, X., Liang, J., and Xiao, Y.  Can large language models understand real-world complex instructions?  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 38, pp. 18188–18196, 2024.
  * Hegselmann et al. (2023) Hegselmann, S., Buendia, A., Lang, H., Agrawal, M., Jiang, X., and Sontag, D.  Tabllm: Few-shot classification of tabular data with large language models.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 5549–5581. PMLR, 2023.
  * Jiang et al. (2023) Jiang, J., Zhou, K., Dong, Z., Ye, K., Zhao, W. X., and Wen, J.-R.  Structgpt: A general framework for large language model to reason over structured data.  _arXiv preprint arXiv:2305.09645_ , 2023.
  * Kojima et al. (2022) Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y.  Large language models are zero-shot reasoners.  _Advances in neural information processing systems_ , 35:22199–22213, 2022.
  * Li et al. (2023) Li, Z., Peng, B., He, P., Galley, M., Gao, J., and Yan, X.  Guiding large language models via directional stimulus prompting.  _arXiv preprint arXiv:2302.11520_ , 2023.
  * Liu et al. (2023) Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P.  Lost in the middle: How language models use long contexts.  _arXiv preprint arXiv:2307.03172_ , 2023.
  * Liu et al. (2019) Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * OpenAI (2023a) OpenAI.  Chatgpt.  _https://chat.openai.com/chat_ , 2023a.
  * OpenAI (2023b) OpenAI.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023b.
  * Pasupat & Liang (2015) Pasupat, P. and Liang, P.  Compositional semantic parsing on semi-structured tables.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 1470–1480, Beijing, China, July 2015. Association for Computational Linguistics.  doi: 10.3115/v1/P15-1142.
  * Prasad et al. (2023) Prasad, A., Hase, P., Zhou, X., and Bansal, M.  GrIPS: Gradient-free, edit-based instruction search for prompting large language models.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pp. 3845–3864, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.eacl-main.277.
  * Ramamurthy et al. (2022) Ramamurthy, R., Ammanabrolu, P., Brantley, K., Hessel, J., Sifa, R., Bauckhage, C., Hajishirzi, H., and Choi, Y.  Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization.  _arXiv preprint arXiv:2210.01241_ , 2022.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Shi et al. (2020) Shi, T., Zhao, C., Boyd-Graber, J., Daumé III, H., and Lee, L.  On the potential of lexico-logical alignments for semantic parsing to SQL queries.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pp. 1849–1864, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.findings-emnlp.167.
  * Shin et al. (2020) Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., and Singh, S.  Autoprompt: Eliciting knowledge from language models with automatically generated prompts.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 4222–4235, 2020.
  * Touvron et al. (2023) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Wang et al. (2022) Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., and Zhou, D.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_ , 2022.
  * Wei et al. (2022) Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837, 2022.
  * Ye et al. (2023) Ye, Y., Hui, B., Yang, M., Li, B., Huang, F., and Li, Y.  Large language models are versatile decomposers: Decompose evidence and questions for table-based reasoning.  _arXiv preprint arXiv:2301.13808_ , 2023.
  * Yu et al. (2022) Yu, W., Iter, D., Wang, S., Xu, Y., Ju, M., Sanyal, S., Zhu, C., Zeng, M., and Jiang, M.  Generate rather than retrieve: Large language models are strong context generators.  _arXiv preprint arXiv:2209.10063_ , 2022.
  * Zhong et al. (2017) Zhong, V., Xiong, C., and Socher, R.  Seq2sql: Generating structured queries from natural language using reinforcement learning.  _arXiv preprint arXiv:1709.00103_ , 2017.
  * Ziegler et al. (2019) Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_ , 2019.

##  Appendix A Data Statistics

The original WTQ dataset consists of around 14K, 3.5K, 4.5K instances for
training, validation, and test set. When we use the SQUALL annotation and
identify the instances with valid SQL query annotation, the number of
remaining instances is 10K, 3K, 1.5K for each of the data split. Detailed
statistics of the dataset is described in Table 2.

From the observation, most of the tables in the dataset are long-narrow tables
where there are more rows than columns. The average number of tokens of input
context table is 1,333, and there are around 300 isntances that exceed the
maximum sequence length of recent GPT checkpoints, 4,096.

##  Appendix B Experiment Details

All models are trained and tested on 8-core NVIDIA Tesla A10 GPU with 24GB
RAM. Policy networks are trained over 10 iterations and evaluated for every 3
iterations. The total amount of policy network training took 18 hours.

We design our framework to fine-tune two separate language models, one for the
column reduction and the other for the row reduction. One of the reasons we
did not train a single model for context reduction is to minimize repetition.
Consider an input table, and denote the number of all columns $N$, number of
relevant columns $n$, number of all rows $M$, and number of relevant rows $m$.
When we have a language model with the maximum input sequence length as $L$,
the single model needs to be prompted $\lceil\frac{N\times M}{L}\rceil$ times.
When we reduce the columns first, the number of prompting reduces to
$\lceil\frac{n\times M}{L}\rceil+1$ times. $+1$ represents language model
prompting for column reduction, and this can be done within a single prompting
because the maximum number of columns is 25 in WTQ dataset. Another reason is
to make the policy network optimize better. The single language model that
generates relevant rows and columns at the same time fails to be fine-tuned
and only generates columns or rows even after training the model with several
iterations.

For context reduction, language models are prompted as “Select relevant
columns from a table to answer a question. Output ‘@’ if done generating.
Question: {Table QA question}, List of column headers: {Col1, Col2, ..}”. For
row reduction, the rows are represented with each cell’s corresponding column
value as “Select relevant rows from a table to answer a question. Output ‘@’
if done generating. Question: {Table QA question}, List of rows in a format
‘rowX: (column name=value)’: {row1: (col1=val1), row2: (col2=val2), ..}”. When
there are a lot of rows in the table and representing them exceeds the maximum
token limit of the language model, we truncate the table and prompt the model.
Thus in some cases, the language model can only generate the end-of-sequence
token when the given sub-table does not contain the relevant evidence.

For the generalizability test, we manually annotate 200 instances of the
Hybrid QA test set in order to identify the relevant rows and columns for a
given question and table pair, because Hyrbid QA does not have corresponding
SQL queries annotated for the instances.

Table 2:  Statistics of the WTQ dataset with valid SQUALL annotation  WTQ with
valid SQUALL annotation
---
# of train / valid / test set | 10K / 3K / 1.5K
# of unique tables | 5.4K
# of questions per table | 2.73
Avg / Max # of columns | 9 / 25
Avg / Max # of rows | 46 / 753
Avg / Max # of cells | 230 / 3,832
Avg / Max # of context tokens | 1,333 / 20,324
# instances $>$ 4,096 tokens | 248
# instances $>$ 8,192 tokens | 60

##  Appendix C Model Architecture

The initial policy network parameters are pre-trained language model that is
small enough to fine-tune. We use a sequence-to-sequence language model with
FLAN-T5-Large (Chung et al., 2022) checkpoint as a base model and fine-tune
the model with the annotation of relevant rows and columns for each question
and table pair. Mathematically, we fine-tune the language models by maximizing
the log-likelihood as follows:

$\mathcal{L}_{FT}=-\mathbb{E}\text{log}\theta_{LM}(\bm{z|x,c})$

We further tune the language model by computing the rewards from the model’s
predictions on relevant rows and columns. The model gets positive rewards for
selecting the correct relevant items, and gets negative rewards for incorrect
selection. The model is penalized more when it does not select the relevant
items compared to when it selects irrelevant items. As described in 2.2, the
reward is defined as:

$\mathcal{R}=\lambda_{p}|Z\cap
C_{r}|+\lambda_{n1}|Z-C_{r}|+\lambda_{n2}|C_{r}-Z|$

Note that $C_{r}$ is derived from a heuristics with text-to-SQL annotation,
namely $\bm{c_{r}}=h(x,c)$. To formally define the parameter update condition,
we aim to maximize the following objective:

$\text{max}_{\theta_{LM}}\mathbb{E}_{\bm{z}\sim\theta_{LM}(\cdot|\bm{x,c})}[\mathcal{R}(\bm{x,c,z})]$

To make the optimization tractable for policy network, we employ proximal
policy optimization (PPO) method (Schulman et al., 2017). We consider a fine-
tuned language model as an initial policy network (i.e.,
$\pi_{0}=\theta_{LM}$) and update the policy network $\pi$ using PPO. The
policy network’s generation of relevant information can be considered as a
Markov Decision Process $\langle\mathcal{S},\mathcal{A},r,\mathcal{P}\rangle$,
where $\mathcal{S}$ is a state space, $\mathcal{A}$ is an action space, $r$ is
a reward function, and $\mathcal{P}$ is transition probabilities. At each time
step $t$ in an episode, the model selects an action (i.e., generating tokens
of relevant information) from $\mathcal{A}$, based on the state of the current
time step. The state at time $t$ is defined with the input and the policy
network’s previous generations, that is,
$\bm{z_{t}}=\pi(\cdot|\bm{x,c,z_{<t-1}})$. The episode ends when the policy
network generates an end-of-sequence token.

Following the existing RL approaches on NLP applications (Ziegler et al.,
2019), we employ KL divergence penalty rewards that dynamically adapt the
coefficient $\beta$ in different time steps to minimize excessive parameter
updates from the initial policy. The action space for token generation often
stretches to the size of vocabulary and it makes optimization costly. To
minimize such issues, (Ramamurthy et al., 2022) proposed NLPO, an approach
that masks out the least probable tokens using top-$p$ sampling. We set $p$ as
0.9 in the experiments. The policy network’s reward function $r\bm{(x,c,z)}$
is defined as:

$r(\bm{x,c,z})=\mathcal{R}(\bm{x,c,z})-\beta\log{\frac{\pi(\bm{z|x,c})}{\theta(\bm{z|x,c})}}$
$\bm{e}_{t}=\text{CLIP}\left(\frac{\text{KL}(\pi_{t},\theta)-\text{KL}_{\text{target}}}{\text{KL}_{\text{target}}},-0.2,0.2\right)$
$\beta_{t+1}=\beta_{t}(1+K_{\beta}\bm{e}_{t})$

##  Appendix D Additional Results

Table 3:  Precision scores of input context reduction tested on WTQ test set
(WTQ) and on an unseen test set (HybQA).  Precision on Context Reduction
---
Column Reduction | WTQ | HybQA
RoBERTa Token Clf | 93.32 | 77.61
Zero-shot GPT-4 | 72.13 | 86.78
Fine-tuned FLAN-T5 | 93.30 | 75.11
$\boldsymbol{\star}$ Learning to Reduce (ours) | 90.32 | 75.15
Row Reduction | WTQ | HybQA
RoBERTa Token Clf | 92.95 | 7.93
Zero-shot GPT-4 | 87.91 | 94.35
Fine-tuned FLAN-T5 | 89.01 | 98.22
$\boldsymbol{\star}$ Learning to Reduce (ours) | 91.12 | 97.88

###  D.1 Context Reduction Precision

Table 3 shows the precision scores of models on context reduction. We use
recall scores for evaluating different models, yet precision scores cannot be
ignored. For instance, if a model chooses all columns and rows as relevant for
all cases, it achieves 100% recall but the model would not be considered as
well trained.

Learning to Reduce does not outperform other models with regard to the
precision scores. When it is compared to its non-RL counterpart, fine-tuned
FLAN-T5, the precision scores are almost similar or decreasing. This is
expected because our framework is tuned with recall-based reward. As our
framework does not show exceptionally poor performance on precision scores
compared to other baselines, we argue that Learning to Reduce does not achieve
its outperforming recall scores by simply selecting more columns and rows.

###  D.2 Downstream Task Performance

Figure 3: Accuracy (precision) of GPT-3.5-turbo on WTQ test set with different
input context tables.

Figure 3 shows table QA performance on WTQ dataset when we use a different
LLM, GPT-3.5-turbo. Similar to the results using GPT-4 model, LLM’s inference
ability drops as the input context gets longer. We can also see the same trend
for inputs with reduced context—input tables with only columns reduced does
not improve the LLM’s performance while input tables with both rows and
columns reduced leads LLMs perform better. The gap between the gold and
predicted context reduction motivates the necessity of improving the accuracy
of context reduction in future studies.