11institutetext: Institute for Fundamental Theory, Physics Department,
University of Florida, Gainesville FL 32653, USA

# Reproducing Bayesian Posterior Distributions for Exoplanet Atmospheric
Parameter Retrievals with a Machine Learning Surrogate Model

Eyup B. Unlu 11 0000-0002-6683-6463 Roy T. Forestano 11 0000-0002-0355-2076
Konstantin T. Matchev 11 0000-0003-4182-9096 Katia Matcheva 11
0000-0003-3074-998X

###### Abstract

We describe a machine-learning-based surrogate model for reproducing the
Bayesian posterior distributions for exoplanet atmospheric parameters derived
from transmission spectra of transiting planets with typical retrieval
software such as TauRex. The model is trained on ground truth distributions
for seven parameters: the planet radius, the atmospheric temperature, and the
mixing ratios for five common absorbers: $H_{2}O$, $CH_{4}$, $NH_{3}$, $CO$
and $CO_{2}$. The model performance is enhanced by domain-inspired
preprocessing of the features and the use of semi-supervised learning in order
to leverage the large amount of unlabelled training data available. The model
was among the winning solutions in the 2023 Ariel Machine Learning Data
Challenge.

###### Keywords:

Exoplanet atmospheres semi-supervised learning feature engineering approximate
inference.

##  1 Introduction

The discovery of planets orbiting stars beyond our own has drastically
reshaped our perception of the cosmos and our role in it. With the
advancements in observational techniques and the increasing number of space-
based missions, an ever-growing catalog of exoplanets has emerged, revealing
an astounding range of varying planetary habitats. Invaluable insights about
the structure and composition of an exoplanet atmosphere come from transit
spectroscopy, whereby a planet passes in front of its host star and the
associated relative reduction in the original stellar flux $\Phi_{0}$,

$M_{\lambda}\equiv\frac{\Phi_{0}-\Phi_{i}}{\Phi_{0}},$ (1)

is measured in several different wavelengths $\lambda$ (here $\Phi_{i}$ is the
minimum stellar flux during transit).

However, extracting significant and dependable information from these observed
transit spectra necessitates a comprehensive and thorough understanding of the
underlying theory, observational limitations, numerical simulations, and
statistical inference involved in the retrieval process.

In order to model the observed transit spectra, one employs complex forward
radiative transfer models that account for the atmospheric structure,
composition and planet parameters. The inverse problem of exoplanet parameter
determination is typically tackled by Bayesian retrieval models, which, given
an observed spectrum, provide the posterior distributions of the planetary
atmospheric parameters. Bayesian retrievals are considered as the gold
standard in the field, but are notoriously slow, since sampling-based
retrievals typically require between $10^{5}-10^{8}$ calls to the forward
model before reaching convergence [4]. The arrival of the next generation of
telescopes with high-resolution spectroscopic capabilities for exoplanet
characterisation is projected to result in a substantial increase in the
availability of high-resolution (i.e., high-dimensional from machine learning
point of view) spectroscopic data for thousands of planets. The use of
traditional data analysis and retrieval tools will be prohibitively slow in
the face of the sheer volume of data and number of planets to be analyzed.
This necessities the development of a new generation of fast, efficient, and
accurate retrieval methods.

##  2 Objectives

Since 2019, the Ariel Machine Learning Data Challenge has been an annual
hackathon event that seeks innovative machine learning (ML) solutions to
pressing issues faced by the exoplanet community and the Ariel Space Telescope
in particular [13, 4, 16, 2]. Similar to the James Webb Space Telescope, the
Ariel mission will perform infrared spectroscopic observations of transiting
exoplanets at high resolution. The specific task for the 2023 installment of
the Ariel Machine Learning Data Challenge was to develop a surrogate ML model,
which, given a simulated observation, can predict the posterior distributions
for seven planet parameters: the planet radius $R_{p}$, the atmospheric
temperature $T$ and five molecular abundances ($H_{2}O$, $CH_{4}$, $CO_{2}$,
$CO$, $NH_{3}$). The ground truth was the corresponding posterior distribution
derived with the TauRex suite [1] using a MultiNest algorithm [5, 6] (for
further details, see [4, 16]).

The training dataset provided by the challenge organizers contained a total of
41423 planets. For each planet, the observed transit depth $M_{\lambda}$ was
provided at 52 different wavelengths $\lambda$ between $0.55\mu m$ and
$7.28\mu m$, along with the associated uncertainty $\epsilon_{\lambda}$ for
each spectral bin $\lambda$. Additionally, the training dataset contained
information about a number of auxiliary parameters describing the star-planet
system, e.g., the star distance, the star mass, the star radius, the star
temperature, the planet mass, the planet orbital period, the planet distance
and the planet surface gravity. The distributions of those parameters over the
entire dataset are shown in the top eight panels in Figure 1. The training
database also contained the values of the forward model parameters used to
produce the synthetic transit spectrum for each planet. Those parameters were
the planet radius $R_{p}$, the atmospheric temperature $T$, and the mixing
ratios $X_{i}$ for five common absorbers, where
$i\in\\{H_{2}O,CH_{4},NH_{3},CO,CO_{2}\\}$. Their distributions are shown in
the bottom seven panels in Figure 1, where $R_{p}$ is given in units of the
Jupiter radius $R_{J}$, $T$ is in degrees Kelvin ($K$), and the mixing ratios
are on a log base 10 scale. The forward model parameters ranges are
$0.075R_{J}$ to $2.43R_{J}$ for the planet radius $R_{p}$, $101K$ to $4965K$
for the atmospheric temperature $T$, from -9 to -3 for $\log X_{H_{2}O}$,
$\log X_{CH_{4}}$ and $\log X_{NH_{3}}$, from -6 to -3 for $\log X_{CO}$ and
from -9 to -4 for $\log X_{CO_{2}}$. Note that while the forward model
parameters were provided in the training dataset, and thus could be used
during training, they were not available in the test data, and therefore
should not be among the inputs of the trained surrogate ML model.

Figure 1: Distribution of the auxiliary parameters (top eight panels) and the
forward model parameters (bottom seven panels) in the training database.

For 6766 of the planets in the dataset, posterior distributions derived with
TauRex were provided in the form of weighted samples. From now on we shall
refer to those planets as labelled. The remaining 34657 planets in the dataset
did not have such posterior distributions and are therefore unlabelled.

##  3 Methodology

We develop three separate custom ML models to predict the chemical composition
$X_{i}$, the planet radius $R_{p}$, and the atmospheric temperature $T$,
respectively.

###  3.1 Feature Engineering

For all models, we expand the initial set of auxiliary parameters $\\{Aux\\}$
to a larger set $\\{Aux+\\}$ by adding i) an approximate estimate
$\hat{R}_{p}$ of the planet radius and ii) four dimensionless features
motivated by domain knowledge [10, 12]

$\\{Aux+\\}=\left\\{Aux,\hat{R}_{p},\frac{\hat{R}_{p}}{R_{s}},\frac{D}{H},\frac{R_{s}}{H},\max(M^{\prime}_{\lambda})\right\\}.$
(2)

The minimum transit depth, $M_{min}\equiv\min_{\lambda}(M_{\lambda})$,
provides the most stringent upper bound on the planet radius, therefore we
take $\hat{R}_{p}=R_{s}\sqrt{M_{min}}$ as a useful proxy for $R_{p}$. Another
quantity of interest entering (2) is the scale height of the atmosphere
$H=\frac{k_{B}T_{e}}{mg}$, which depends on the Boltzmann constant $k_{B}$,
the mean molecular mass $m$ (approximated as 2.29 amu for a typical gas
giant), the specific gravity $g$, and the planet equilibrium temperature
$T_{e}$. The latter can be estimated from the available auxiliary parameters
as $T_{e}=T_{s}\sqrt{\frac{R_{s}}{2D}}$, where $T_{s}$ is the star
temperature, $R_{s}$ is the star radius, and $D$ is the distance from the
planet to the star.

Subsequently, the auxiliary information (2) was standardized by subtracting
the mean and dividing by the standard deviation over the training set for each
individual parameter, respectively.

In order to focus on the absorption of the atmosphere alone, we modify the
transit spectrum $M_{\lambda}$ by subtracting the opaque disk of the planet as
[12, 11]

$M_{\lambda}^{\prime}=M_{\lambda}-\frac{\hat{R}_{p}^{2}}{R_{s}^{2}},$ (3)

and rescaling as

$M_{\lambda}^{\prime\prime}=\frac{M^{\prime}_{\lambda}}{\max_{\lambda}(M_{\lambda}^{\prime})}.$
(4)

The noise data was rescaled in a similar manner:

$\epsilon_{\lambda}^{\prime}=\frac{\epsilon_{\lambda}}{\max_{\lambda}(M_{\lambda}^{\prime})}.$
(5)

Due to the rescalings in eqs. (4) and (5), we include
$\max_{\lambda}(M_{\lambda}^{\prime})$ in the extended list of parameters in
(2).

In addition to (2), (4) and (5), the models for the temperature and the planet
radius also used the unmodified spectra $M_{\lambda}$ among their inputs. In
summary, the inputs for the radius and the temperature ML models were
$(M_{\lambda},M_{\lambda}^{\prime\prime},\epsilon_{\lambda}^{\prime},Aux+)$,
while the inputs for the chemical ML model were simply
$(M_{\lambda}^{\prime\prime},\epsilon_{\lambda}^{\prime},Aux+)$.

###  3.2 Posterior Distribution Parametrization

The posterior pdf (probability density function) for each of the seven target
parameters was suitably parameterized with an analytical ansatz. The ansatz
assumed no correlations for the chemical mixing ratios and represented each
one as a weighted sum of a Gaussian distribution and a uniform distribution:

$\displaystyle f_{i}(x_{i};\alpha_{i},\mu_{i},\sigma_{i},m_{i})$
$\displaystyle=$
$\displaystyle\alpha_{i}\times\frac{1}{\sigma_{i}\sqrt{2\pi}}e^{-\left(x_{i}-\mu_{i}\right)^{2}/2\sigma_{i}^{2}}$
(6) $\displaystyle+$
$\displaystyle\left(1-\alpha_{i}\right)\times\frac{\Theta\left(x_{i}+12\right)\Theta\left(m_{i}-x_{i}\right)}{m_{i}+12},$

where $x_{i}=\log X_{i}$ is the base 10 logarithm of the mixing ratio for the
$i$-th chemical absorber, and $\alpha_{i}$ and ($1-\alpha_{i}$) are the
fractional contributions of the Gaussian and the uniform distribution,
respectively, to the posterior distribution $f_{i}$ for the $i$-th absorber.
The remaining quantities in eq. (6) are as follows: $\mu_{i}$ and $\sigma_{i}$
are respectively the mean and the standard deviation of the Gaussian, and
$m_{i}$ is the upper endpoint of the uniform distribution (the lower endpoint
is taken as $-12$, hence the range is $m_{i}-(-12)=m_{i}+12$). This ansatz has
a good physical justification, which is illustrated in Figure 2, where we
compare the given ground truth distributions (blue histograms) and the fitted
parametrized pdfs from eq. (6) (orange lines) for the log-mixing ratios
$x_{i}$ for Planet 17 in the database. We observe that when the concentration
of the corresponding absorber is relatively large ($x_{i}\gtrsim-6$), the
absorber is well detected and the posterior distribution resembles a Gaussian
centered near the true $x_{i}$ value, as shown in the first, third and fourth
panel of Fig. 2. If, on the other hand, the concentration is rather small (or
alternatively, if the absorber is rather weak), the presence of this absorber
is not detected and the posterior distribution is uniform up to some detection
threshold $m_{i}$ (see the fifth panel in Fig. 2). In intermediate cases like
the one in the second panel, the posterior distribution can be approximated
with a linear sum of those two shapes. The parameters of the ansatz (6) were
fit to the training data by minimizing the squared difference between the
cumulative distributions of the parametrized distribution and the provided
ground truth distributions on 1000 points. The minimization was done using the
scipy minimize function with the L-BFGS-B method [3, 17, 15].

Figure 2: Comparison of the given ground truth distributions (blue histograms)
and the fitted parametrized pdfs from eq. (6) (orange lines) for the base 10
logarithms of the chemical mixing ratios for Planet 17 in the database. The
ground truth second quartiles (50-th percentiles) in this case were -6.46,
-10.31, -5.01, -5.91 and -9.39, respectively.

We noticed that in contrast to the chemical abundances, for the planet radius
$R_{p}$ and the planet temperature $T$, the corresponding posterior
distributions in the ground truth training data exhibited a relatively high
degree of correlation. To reflect this fact, we used a different ansatz,
namely, a bivariate normal distribution

$f_{R_{p}T}\left(R_{p},T;\mu,\sigma_{R_{p}},\sigma_{T},\rho\right)=\frac{\exp\left(-\frac{1}{2}\left(\left[R_{p},T\right]-\mu\right)^{T}\Sigma^{-1}\left(\left[R_{p},T\right]-\mu\right)\right)}{2\pi|\Sigma|},$
(7)

where $\mu=\left[\mu_{R_{p}},\mu_{T}\right]$ contains the respective means and

$\Sigma=\left(\begin{array}[]{cc}\sigma_{R_{p}}^{2}&\sigma_{R_{p}}\sigma_{T}\rho\\\
\sigma_{R_{p}}\sigma_{T}\rho&\sigma_{T}^{2}\end{array}\right)$

is defined in terms of the respective standard deviations $\sigma_{R_{p}}$ and
$\sigma_{T}$ and the correlation coefficient $\rho$. The correlation
coefficient was taken as $\rho=-0.7$ and the remaining four parameters of the
ansatz (7) were then fit to the ground truth training data by computing the
covariance matrix with the NumPy package [8].

Figure 3: Probability distributions for the planet radius ($x$-axis) and the
planet temperature ($y$-axis) for five different planets in the database. The
blue points are sampled from the given ground truth distributions, while the
orange points are sampled from the respective fitted parameterized
distributions (6).

The parametrization of the posterior distributions was tested in two ways.
First, a qualitative comparison of the two distributions was performed by
visual inspection; a few representative examples are shown in Fig. 3, and they
all exhibit good agreement. A second quantitative test involved computing the
two-sample Kolmogorov–Smirnov (KS) test scores between the ground truth
distributions and the respective parametrized ansatz. (The KS score is the
largest absolute difference between the cumulative distribution functions of
the two distributions under consideration.) The corresponding average scores
are shown in Table 1 and are observed to be small, which indicates good
agreement, since a lower KS score implies a better fit.

Table 1: Average two-sample KS test scores between the ground truth distributions and the parametrized distributions (6) and (7).  Parameters | $R_{p}$ | T | $H_{2}O$ | $CO_{2}$ | $CO$ | $CH_{4}$ | $NH_{3}$
---|---|---|---|---|---|---|---
Test Score | .036 | .042 | .034 | .038 | .042 | .035 | .033

###  3.3 Model architecture

Three separate ML models were used to obtain final predictions. Two of those
models were used to obtain the four trainable ansatz parameters in (7) for the
planet radius $R_{p}$ and temperature $T$. These two models were composed of
fully connected neural networks. The final activation function was a sigmoid
and its output was converted to the respective ansatz parameter by a linear
transformation involving the range and minimum value for each parameter.

For the ML model predicting the chemical abundances, we used a different
architecture shown in Fig. 4. The model used several fully connected neural
networks arranged into four modules represented with the blue boxes in the
figure. As indicated with the $\times$ symbol, the outputs of the modifying
layer modules were used to multiplicatively reweight data from a previous
stage. The model’s final activation function was a sigmoid. The model outputs
20 values which get reshaped into a $(4,5)$ matrix. In order to convert the
sigmoid output into a meaningful value for the respective parameter, each
column was multiplied elementwise with the vector $[1,11,11,-6]$, then the
vector $[0,-12,-12,6]$ was added to the product. This linear transformation
keeps the prediction for $\alpha_{i}$ unchanged, but maps the predictions for
$m_{i}$ and $\mu_{i}$ between $-12$ and $-1$ and the prediction for
$\sigma_{i}$ between $0$ and $6$. These scaled values were then used as the
predicted ansatz parameters, where each column contains the
$[\alpha_{i},m_{i},\mu_{i},\sigma_{i}]$ set of parameters for the log-mixing
ratio of the $i$-th chemical absorber.

Figure 4: The model architecture used for the chemical model. LR represents
the Leaky ReLU activation function. The $\times$ symbol indicates an
elementwise multiplication of two data sets with the same size.

###  3.4 Loss function

The evaluation metric for the data challenge involved the two-sample
Kolmogorov-Smirnov test. Therefore, we chose our loss function to mimic the KS
distance between the parametrized ground truth ansatz and our predicted
distribution. For the planet radius and temperature, both of those
distributions are Gaussian, and in that case the KS distance can be computed
analytically in terms of the respective means and standard deviations.

Since the ansatz (6) for the chemical abundances was more complicated, it was
not possible to obtain an analytic expression for the KS distance. Instead, we
used the following approximation

$L(y,\hat{y})=\sum_{i=0}^{5}1.5|\alpha_{i}-\hat{\alpha}_{i}|+\\\
|(1-\alpha_{i})(m_{i}-\hat{m}_{i})|+\alpha_{i}\sqrt{\left(\mu_{i}-\hat{\mu}_{i}\right)^{2}+\left(\sigma_{i}-\hat{\sigma}_{i}\right)^{2}},$
(8)

where the first (second) term enforces that the mean of the Gaussian (uniform)
component of the posterior ansatz (6) was predicted correctly, while the last
term is proportional to the earth mover’s distance between the respective
Gaussian components only.111At the time of the competition, the code
implementing the last term in (8) contained a typo which has been fixed since.

###  3.5 Training

Figure 5: Schematic illustration of the semi-supervised approach used for the
temperature ML model.

The models were constructed, trained and tested using the PyTorch package
[14]. The training was performed with the ADAM optimizer [9]. For each model,
we used different learning rates throughout the three different stages of
training, as well as different number of epochs for each stage. The training
of the $R_{p}$ model proceeded over 70 epochs, divided into stages of 20, 20
and 30, with learning rates $0.0001$, $0.00025$, and $0.0001$, respectively.
The training of the temperature model was similar, with learning rates of
$0.0005$, $0.0005$, and $0.0001$, respectively. Finally, the chemical
composition model was trained over 50 epochs, split into stages of 20, 25 and
5, with the same learning rates as the temperature model.

Figure 6: Evolution of the training loss (orange lines) and the validation
loss (blue lines) with the number of epochs on the horizontal axis, for the
three ML models: for the planet radius $R_{p}$ (top left panel), for the
atmospheric temperature $T$ (top right panel) and for the chemical abundances
(bottom panel). Each model was trained on 80% of the available training data
and then validated on the remaining 20%.

In order to take advantage of the large amount of unlabelled training data
available, we adopted the three-step semi-supervised approach illustrated in
Fig. 5. In the first step, we develop a secondary ML model whose purpose is to
provide pseudolabels for the unlabelled data. This model is trained on the
available labelled data. More importantly, among its inputs, it additionally
includes the provided true values for the forward model parameters ($R_{p}$,
$T$, and $x_{i}$) used for the generation of the training spectra with the
TauRex forward model. In the second step, this trained “pseudo-labeler” is
applied on the unlabelled data to predict labels for it. In the final third
step, the primary ML model described in Section 3.3 above is trained on a
weighted mixture of labelled and pseudo-labelled data, where the contribution
of the latter is suppressed by a factor $\beta$. Note that the primary ML
model does not make any use of the forward model parameters, since they are
not available for the competition test data.

For the training, we used a train-test split of 4:1 (80% - 20%). The evolution
of the respective losses with the number of epochs for each of the three
models is shown in Fig. 6.

##  4 Results

Table 2: Two-sample KS test scores on the validation set for four different model configurations. Model configuration | Scores
---|---
No |  SSL |  Train |  Test | $~{}~{}\beta~{}~{}$ | $R_{p}$ | T | $H_{2}O$ | $CO_{2}$ | $CO$ | $CH_{4}$ | $NH_{3}$
I | No | 80% | 20% | — | 0.5064 | 0.3129 | 0.1780 | 0.1558 | 0.1284 | 0.1807 | 0.1739
II | Yes | 80% | 20% | 5 | 0.5064 | 0.2142 | 0.1782 | 0.1558 | 0.1284 | 0.1806 | 0.1741
III | Yes | 80% | 20% | 1 | 0.5064 | 0.2096 | 0.1778 | 0.1558 | 0.1284 | 0.1809 | 0.1741
IV | Yes | 100% | 20% | 5 | 0.5030 | 0.1606 | 0.1485 | 0.1129 | 0.1004 | 0.1410 | 0.1588
Figure 7: Histograms of two-sample KS test scores of the planets in the
validation set for each of the seven target parameters.

Different hyperparameter configurations were tested and the optimal
configuration was selected based on the results on the validation set. Several
representative model configurations are shown in Table 2, together with the
corresponding average two-sample KS test scores on the validation set for each
of the seven target parameters. Comparing the results for model configurations
I and II in the table, we note that the semi-supervised learning (SSL)
approach of Fig. 5 made a notable improvement only in the case of the
temperature ML model. Therefore, for simplicity, the chemical and the planet
radius models used only labeled data for training. We checked that different
reweightings of the pseudo-labelled training data (model configuration III is
one such choice) did not significantly impact the results either.

The final models were trained with the selected optimal hyperparameter
configurations using all of the training data (model configuration IV in Table
2). The resulting distributions of two-sample KS test scores by parameter are
shown in Fig. 7.

##  5 Analysis

Figure 8: Comparison of the ground truth posterior distributions for $R_{p}$
(blue histograms) with the model’s prediction (orange histograms) for several
representative planets.

The predictions from the ML model configuration IV in Table 2 were entered in
the 2023 Ariel Machine Learning Data Challenge and ended up being among the
winning solutions. In the process, some interesting lessons have been learned.

The difficulty in reproducing the posterior distributions for the planet
radius $R_{p}$. The results shown in Table 2 and Figure 7 reveal that while
the posterior distributions for six of the seven target parameters were
reproduced quite well, the model was struggling with the case of the planet
radius $R_{p}$, where the KS distance did not drop below 0.5. The reason for
this is illustrated in Figure 8, which compares the ground truth posterior
distributions for $R_{p}$ (blue histograms) with the model’s prediction
(orange histograms) for several representative planets in the data. One can
observe that while the averages of the two distributions (ground truth and
model prediction) are consistent, the standard deviations are very different,
and the prediction (orange lines) is always much wider than the ground truth.
This actually explains why the KS distance is of order 0.5: this is the
asymptotic answer for two distributions with similar means and very different
variances. The extreme narrowness of the ground truth distributions for
$R_{p}$ poses a significant challenge for the ML model. For example, whenever
the model’s prediction for the mean is outside the ground truth distribution,
the best the model can do is to inflate the standard deviation and thus ensure
some overlap between the two distributions. This phenomenon is precisely what
is seen in Figure 8.

We also note that in many instances, the forward model value of $R_{p}$ used
to generate the spectrum lies completely outside the ground truth posterior
distribution, indicating that the Bayesian retrieval procedure was too
overconfident with regards to the $R_{p}$ parameter. This motivates further
investigations, e.g., applying ML for retrieving the forward model parameters
directly via regression techniques.

The role of the sampling distribution used for creating the training dataset.
Recently, the impact of the choice of sampling distribution on the information
content in the training database has been under intense discussion [7].
Interestingly, we note that the average scores for the chemical abundances
seem to be correlated to the sampling range used for each chemical absorber.
For example, the log-mixing ratios for $H_{2}O$, $NH_{3}$ and $CH_{4}$ share
the same sampling range (from -9 to -3) and correspondingly, have similar mean
scores ($\approx 0.18$). On the other hand, $CO_{2}$ was sampled from -9 to
-4, and its mean score is better, on the order of 0.15. Lastly, $CO$ was
sampled from -6 to -3, and has the best score, on the order of 0.13. While
there may be other factors at play as well, these results suggest that the
retrieval outcome may depend not only on the choice of function for the
sampling distribution, but also on the sampling range of parameters when
creating the training and testing datasets.

The benefit from semi-supervised learning in enlarging the effective training
dataset. Semi-supervised techniques allow for leveraging all of the available
training data (labelled and unlabelled). They are especially useful when
labelling the data is prohibitively costly or simply not possible (for
example, the competition rules did not allow using Taurex to label the
unlabelled data). In this year’s Ariel data challenge, the majority of the
available training data was unlabelled, which motivated the semi-supervised
approach described in Figure 5. As discussed in Section 4, this noticeably
improved the score for the planet temperature, which was not handled as well
by the regular supervised method. On the other hand, the supervised approach
already worked quite well for the chemical abundances, and the addition of
pseudo-labelled training data led to marginal improvements. Finally, in the
case of the planet radius, the limiting factor was not the lack of training
data, as already discussed above.

Comparing Table 2 to Table 1, we can see that our scores are not close to the
optimal score which would occur if the ansatz parameters were predicted
perfectly. Therefore, we believe that the approach described here, while
successful in this year’s competition, can be perfected further, e.g., by
including more features motivated by domain knowledge, by considering a more
general ansatz allowing correlations among the chemical abundances, improved
model expressiveness and loss functions. We look forward to next year’s Ariel
Machine Learning Data Challenge.

####  5.0.1 Acknowledgements

We thank the organizers of the Ariel 2023 Data Challenge for designing and
conducting such an exciting and insightful competition and for continuous
support and encouragement throughout. We are also grateful to the anonymous
referees for valuable comments and constructive suggestions on the manuscript.

## References

  * [1] Al-Refaie, A.F., Changeat, Q., Waldmann, I.P., Tinetti, G.: Taurex 3: A fast, dynamic, and extendable framework for retrievals. The Astrophysical Journal 917(1), 37 (aug 2021). https://doi.org/10.3847/1538-4357/ac0252, https://dx.doi.org/10.3847/1538-4357/ac0252
  * [2] Ariel machine learning data challenge (2023), https://www.ariel-datachallenge.space/ [Accessed: July 24 2023]
  * [3] Byrd, R.H., Lu, P., Nocedal, J., Zhu, C.: A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing 16(5), 1190–1208 (1995). https://doi.org/10.1137/0916069, https://doi.org/10.1137/0916069
  * [4] Changeat, Q., Yip, K.H.: ESA-Ariel Data Challenge NeurIPS 2022: Introduction to exo-atmospheric studies and presentation of the Ariel Big Challenge (ABC) Database. arXiv e-prints arXiv:2206.14633 (Jun 2022)
  * [5] Feroz, F., Hobson, M.P.: Multimodal nested sampling: an efficient and robust alternative to markov chain monte carlo methods for astronomical data analyses. Monthly Notices of the Royal Astronomical Society 384(2), 449–463 (Jan 2008). https://doi.org/10.1111/j.1365-2966.2007.12353.x, http://dx.doi.org/10.1111/j.1365-2966.2007.12353.x
  * [6] Feroz, F., Hobson, M.P., Cameron, E., Pettitt, A.N.: Importance nested sampling and the multinest algorithm. The Open Journal of Astrophysics 2(1) (Nov 2019). https://doi.org/10.21105/astro.1306.2144, http://dx.doi.org/10.21105/astro.1306.2144
  * [7] Fisher, C., Heng, K.: How Do We Optimally Sample Model Grids of Exoplanet Spectra? The Astrophysical Journal 934(1), 31 (Jul 2022). https://doi.org/10.3847/1538-4357/ac7801
  * [8] Harris, C.R., Millman, K.J., van der Walt, S.J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N.J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M.H., Brett, M., Haldane, A., del Río, J.F., Wiebe, M., Peterson, P., Gérard-Marchant, P., Sheppard, K., Reddy, T., Weckesser, W., Abbasi, H., Gohlke, C., Oliphant, T.E.: Array programming with NumPy. Nature 585(7825), 357–362 (Sep 2020). https://doi.org/10.1038/s41586-020-2649-2, https://doi.org/10.1038/s41586-020-2649-2
  * [9] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization (2017)
  * [10] Matchev, K.T., Matcheva, K., Roman, A.: Analytical Modeling of Exoplanet Transit Spectroscopy with Dimensional Analysis and Symbolic Regression. The Astrophysical Journal 930(1), 33 (May 2022). https://doi.org/10.3847/1538-4357/ac610c
  * [11] Matchev, K.T., Matcheva, K., Roman, A.: Transverse Vector Decomposition Method for Analytical Inversion of Exoplanet Transit Spectra. The Astrophysical Journal 939(2), 95 (Nov 2022). https://doi.org/10.3847/1538-4357/ac82f3
  * [12] Matchev, K.T., Matcheva, K., Roman, A.: Unsupervised Machine Learning for Exploratory Data Analysis of Exoplanet Transmission Spectra. The Planetary Sciences Journal 3(9), 205 (Sep 2022). https://doi.org/10.3847/PSJ/ac880b
  * [13] Nikolaou, N., Waldmann, I.P., Tsiaras, A., Morvan, M., Edwards, B., Hou Yip, K., Tinetti, G., Sarkar, S., Dawson, J.M., Borisov, V., Kasneci, G., Petkovic, M., Stepisnik, T., Al-Ubaidi, T., Bailey, R.L., Granitzer, M., Julka, S., Kern, R., Ofner, P., Wagner, S., Heppe, L., Bunse, M., Morik, K.: Lessons Learned from the 1st ARIEL Machine Learning Challenge: Correcting Transiting Exoplanet Light Curves for Stellar Spots. arXiv e-prints arXiv:2010.15996 (Oct 2020)
  * [14] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc. (2019), http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * [15] Virtanen, P., Gommers, R., Oliphant, T.E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S.J., Brett, M., Wilson, J., Millman, K.J., Mayorov, N., Nelson, A.R.J., Jones, E., Kern, R., Larson, E., Carey, C.J., Polat, İ., Feng, Y., Moore, E.W., VanderPlas, J., Laxalde, D., Perktold, J., Cimrman, R., Henriksen, I., Quintero, E.A., Harris, C.R., Archibald, A.M., Ribeiro, A.H., Pedregosa, F., van Mulbregt, P., SciPy 1.0 Contributors: SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods 17, 261–272 (2020). https://doi.org/10.1038/s41592-019-0686-2
  * [16] Yip, K.H., Waldmann, I.P., Changeat, Q., Morvan, M., Al-Refaie, A.F., Edwards, B., Nikolaou, N., Tsiaras, A., Alves de Oliveira, C., Lagage, P.O., Jenner, C., Y-K. Cho, J., Thiyagalingam, J., Tinetti, G.: ESA-Ariel Data Challenge NeurIPS 2022: Inferring Physical Properties of Exoplanets From Next-Generation Telescopes. arXiv e-prints arXiv:2206.14642 (Jun 2022)
  * [17] Zhu, C., Byrd, R.H., Lu, P., Nocedal, J.: Algorithm 778: L-bfgs-b: Fortran subroutines for large-scale bound-constrained optimization. ACM Trans. Math. Softw. 23(4), 550–560 (dec 1997). https://doi.org/10.1145/279232.279236, https://doi.org/10.1145/279232.279236


#  Improving the Question Answering Quality using Answer Candidate Filtering
based on Natural-Language Features

1st Aleksandr Gashkov${}^{\text{*}}$ Humanitarian Faculty
Perm National Research Polytechnic University
Perm, Russia
<EMAIL_ADDRESS>2nd Aleksandr Perevalov${}^{\text{*}}$ Computer Science
and Languages
Anhalt University of Applied Sciences
Köthen (Anhalt), Germany
<EMAIL_ADDRESS>3rd Maria Eltsova  Humanitarian Faculty
Perm National Research Polytechnic University
Perm, Russia
<EMAIL_ADDRESS>4th Andreas Both ${}^{\text{*}}$ corresponding authors
Computer Science and Languages Technology Innovation Unit Anhalt University of
Applied Sciences DATEV eG Köthen (Anhalt), Germany Nuremberg, Germany
<EMAIL_ADDRESS><EMAIL_ADDRESS>
(June 2021)

###### Abstract

Software with natural-language user interfaces has an ever-increasing
importance. However, the quality of the included Question Answering (QA)
functionality is still not sufficient regarding the number of questions that
are answered correctly.

In our work, we address the research problem of how the QA quality of a given
system can be improved just by evaluating the natural-language input (i.e.,
the user’s question) and output (i.e., the system’s answer).

Our main contribution is an approach capable of identifying wrong answers
provided by a QA system. Hence, filtering incorrect answers from a list of
answer candidates is leading to a highly improved QA quality. In particular,
our approach has shown its potential while removing in many cases the majority
of incorrect answers, which increases the QA quality significantly in
comparison to the non-filtered output of a system.

###### Index Terms:

question answering, answer validation, answer filtering, answer ranking,
improving question answering quality, natural language processing, English
language

##  I Introduction

Figure 1: Overview of the general research idea TABLE I: Answer Validation
publications of the last decade [1]

Ref. | Year | Languages | Datasets | Methods | Evaluation score
---|---|---|---|---|---
[2] | 2010 | English, Spanish | ResPubliQA [3] | EAT, NER, Acronym Checking |  65% Accuracy (English) 57% Accuracy (Spanish)
[4] | 2010 | Spanish | CLEF 06 [5] | RTE | 53% Accuracy
[6] | 2011 | German | CLEF 11 [7] | Rule-set | 44% Accuracy
[8] | 2011 | French | Web | Decision Tree Combination | 53% MRR
[9] | 2012 | German | CLEF-QA [10] | LogAnswer Framework | 61% Correct Top Rank
[11] | 2013 | English | AVE 08 [12] | RTE |  58% Precision 22% F-Score
[13] | 2013 | English | CLEF 11 [7] | RTE | 45% Precision
[14] | 2013 | Russian | ROMIP [15] | RTE | 70.4% F-Score
[16] | 2015 | English | Sem-Eval 2015 [17] | Word vectors + Classifier |  62.35% Accuracy 46.07% F-Score
[18] | 2018 | English | SQuAD-T [18] | GloVe + GRU | 71.24% F-score
[19] | 2019 | English | SQuAD 2.0 [20] | RTE + ELMo + Verifier | 74.2% F-score

Question Answering (QA) aims to provide precise answers to questions
formulated in a natural language (NL). There are many different approaches for
implementing QA systems (e.g., AskNow [21], QAnswer [22], DrQA [23]) focusing
on specific paradigms and technologies. The two main directions of QA are Open
Domain Question Answering (ODQA) and Knowledge Graph Question Answering
(KGQA).

The common data flow in QA systems is from a given input (question) to an
output (answer). Typically, both are given using NL, s.t., the accessibility
of such a system for humans is high. In this work, we tackle the problem of
answer validation (AV) considering a KGQA system as a black box. Thus, the
concrete approach for computing the answer is hidden (i.e., not visible and
changeable) and only textual representation of question and answer are
provided for further analyses.

Here, we propose an approach for improving QA quality by filtering out
incorrect answer candidates. The approach is built on the assumption that a
considered KGQA system provides a list of query candidates (e.g., written in
SPARQL) that will be used to retrieve the answers from a knowledge graph (KG).
In this scenario, the provided query candidates are executed and, therefore,
it is decided whether they will retrieve the correct answer or not (i.e., the
incorrect answer candidates should be eliminated) – we call this process
answer validation (AV) or filtering. Consequently, each query candidate must
be transformed to an answer candidate in NL (i.e., verbalized), in this
regard, our AV experiments are performed on 3 levels (cf., Figure 1) where the
verbalizations of query candidates (i.e., NL form of a query) are ($A_{1}$)
provided using well-formed NL (written by a human as a baseline), ($A_{2}$)
computed using Natural Language Generation (NLG) considering the contained
facts, and ($A_{3}$) computed using a bag-of-labels approach of available
entities.

In this paper, we follow our long-term research agenda of improving the
overall quality of KGQA systems following a domain-agnostic approach that is
not limited to just a single class of KGQA systems. Therefore, while having
limited access to internal data structures of a KGQA system, the NL form of
the questions and answers gains importance. To show the significance of our
approach, we not only consider AV module quality (i.e., F1 Score) but also its
impact on the end-to-end QA quality (i.e., Precision@k, NDCG@k).

In this paper, we address the following research questions considering the
task of filtering NL answer candidates:

  1. RQ1

Is it possible to improve the QA quality while filtering answers just by their
NL representation?

  2. RQ2

What QA quality is achievable while filtering the well-formed NL
representations of the answer candidates (cf., Approach $A_{1}$)?

  3. RQ3

What QA quality is achievable while having automatically generated NL answer
candidates (cf., Approach $A_{2}$ and Approach $A_{3}$)?

We conclude our main contributions as follows:

  * •

We demonstrate and evaluate two answer verbalization techniques ($A_{2}$ and
$A_{3}$) based on provided SPARQL query candidates.

  * •

We propose and validate a system-agnostic approach to filter a set of answer
candidates using just its NL representation.

  * •

The experimental results show that the proposed approach significantly
improves the QA quality.

This paper is structured as follows: after presenting the related work
(Section II), we give an overview of our approach in Section III. Section IV
presents the used components and datasets. Our experiments are described in
Section V followed by their evaluation (Section VI). Section VII concludes and
outlines future work.

##  II Related Work

Techniques that tackle the task of validating the answer were applied mainly
in ODQA where systems are often required to rank huge amounts of candidate
answers [24], e.g., the incorrect answer candidates in form of textual
paragraphs have to be eliminated by the AV module. In our work, we also use a
textual representation to validate answer candidates while using KGQA systems.
Thus, the ODQA field is related to our research questions.

The comprehensive list of the AV approaches was presented in [1]. Based on
this we show the most related work in the Answer Validation field of the past
decade in Table I.

In [2] Answer Validation is applied to an Information Retrieval system. The
validation process is performed on the basis of Expected Answer Type, Named
Entities Presence, and Acronym Checking (only if a question is about an
acronym). The authors mention that sometimes AV module is “too strict”, i.e.,
it removes also correct answers.

Other publications (e.g., [4, 11, 13, 14]) are based on the idea of
recognizing the textual entailment (RTE) using the output of several QA
systems, universal networking language, lexical similarity, and dependency
parsing algorithms accordingly. All mentioned papers used textual datasets
(containing a pair of one question and one answer).

Babych et al. [6] utilized a rule-based decision algorithm that includes:
syntactic analysis, predicate-argument relation analysis, and semantic
relation analysis. In [8] Decision Tree Combinations are used that are based
on the extracted textual features. Additionally, the system filters answer
candidates (i.e., potential answers to a given question) using Named Entities.
The authors of [9] applied the so-called LogAnswer Framework which
incorporates case-based reasoning.

[16, 18] and [19] applied supervised learning methods. The first utilized the
following features: lexical, n-grams, bad-answer specific, named entities,
term-frequency vectors, and word2vec [25]. In contrast, [18, 19] used a deep
neural network (DNN) model and only GloVe [26] and ELMo [27] features
correspondingly.

To conclude we see how AV approaches changed in time from rule-based to DNNs.
Additionally, we recognize that Answer Validation was used mainly in ODQA
systems while – to the best of our knowledge – no work related to KGQA exists.

##  III Approach

We assume here that a KGQA system provides a ranked list of potential answers;
we call them _answer candidates_. The first element of the list is reflecting
the answer the QA system would show to the user. However, any answer in the
list might be correct (i.e., it is considered to correctly answering the given
question), we denote such an answer candidate as _correct answer_). Note that
there may be multiple candidate answers in the ranked list that are considered
to represent the correct answer. The other answer candidates are representing
potential answers to the given question where the system has decided that they
are not correctly answering the given question (i.e., they are considered to
be _incorrect answers_ w.r.t. the given question).

To evaluate our research idea, we utilize a text classification model trained
on NL representations of correct and incorrect question-answer pairs, i.e.,
the goal is to predict if a corresponding answer is correct for a given
question using their NL representations (cf., Figure 1).

We use the following three settings to establish experiments where a
verbalization of an answer is compared to the corresponding NL question:
$A_{1}$ a well-formed answer text, $A_{2}$ an NL representation of each answer
candidate, and $A_{3}$ a bag-of-labels representation of each answer
candidate. These three approaches allow a distinct validation of the
capabilities of our approach while validating it from a “perfect” to a
“clumsy” textual answer representation. The AV classifier is trained w.r.t.
each verbalization technique, and then the results are compared. The AV
classifier is used to filter incorrect answer candidates, s.t., the QA quality
before and after the filtering process is measured.

For $A_{1}$ a dataset is available (cf., Section IV). For $A_{2}$ and $A_{3}$
we require an automatically executable process to generate the textual
representations.

In the following, we will describe the different approaches.

###  III-A Approach 1 – Well-formed NL Answers

The first approach ($A_{1}$) should use well-formed NL answers to reflect a
high-quality answer verbalization. For example, a well-formed answer to the
question “What was the cause of death of John Kennedy?” can be “John Kennedy
was assassinated.”.

Therefore, it has high demands w.r.t. the textual answer quality and is hard
to automate. For this reason, we will use the existing dataset VANiLLa (cf.,
Section IV-A) containing well-formed NL answers to establish a baseline for
our evaluations.

###  III-B Approach 2 – Automatically generated NL answers

The second approach ($A_{2}$) has less strict requirements rather than the
first one to enable an automatic approach. We will use an NLG method to
generate the answer verbalization. We assume here that a KGQA system (like
QAnswer, cf., Section IV-B) provides a ranked list of potential answers in the
form of SPARQL queries (i.e., a ranked list of _query candidates_). Hence, the
SPARQL queries are usable to generate the verbalizations with existing NLG
tools (e.g., Triple2NL, cf., Section IV-C). Regarding the example question
(see above), the following SPARQL can be proposed (see Listing 1)111In this
example, the SPARQL query retrieves the correct answer regarding the given
question. .

⬇

# question:

# What was the cause of death of John Kennedy?

# query:

PREFIX dbr: <http://dbpedia.org/resource/>

PREFIX dbo: <http://dbpedia.org/ontology/>

SELECT ?answer WHERE {

dbr:John_F._Kennedy dbo:deathCause ?answer .

}

# the result is dbr:Assassination_of_John_F._Kennedy

Listing 1: Retrieving answer URI via SPARQL from the DBpedia KG.

Providing the query and the result (RDF resource), we expect to receive the
textual output from an NLG tool. Given the example, a possible verbalization
might be “The John F. Kennedy’s death cause is Assassination of John F.
Kennedy”222The title of dbr:dbr:Assassination_of_John_F._Kennedy is
“Assassination of John F. Kennedy”.

###  III-C Approach 3 – Answer verbalization based on bag-of-labels

The third approach ($A_{3}$) manifests a number of labels to represent a
simple verbalization of each answer candidate. The labels are retrieved from
the resources, properties, and results mentioned in a SPARQL query candidate
and its results (cf., the previous subsection). The representation of such an
answer is readable, however, it can barely be described as an NL sentence.

For example, considering the example in Listing 1, the following resources are
present in the SPARQL query and it’s result: dbr:John_F._Kennedy,
dbo:deathCause, and dbr:Assassination_of_John_F._Kennedy. After that, all the
English labels of each resource are retrieved and concatenated (i.e., a single
string for each answer candidate). In this example, the following
verbalization of the answer candidate is obtained “John F. Kennedy death cause
Assassination of John F. Kennedy”.

##  IV Materials and Methods

In this section, we describe the components used to manifest the experimental
environment.

###  IV-A The VANiLLa Dataset

Based on our preliminary research, it was decided to use the VANiLLa dataset
[28] that contains NL representation for both questions and answers in
contrast to many other known datasets (cf., [29]333The comparison [29] reveals
the lack of datasets with verbalized answers; however, the existing datasets
(e.g., VQuAnDa and ParaQA) with NL representation of an answer in the full
form are relatively small: only 5000 question-answer pairs.). It contains
107,166 examples with an 80% (train) – 20% (test) split of questions and
answers in a full textual form. The dataset was built on top of the Wikidata
knowledge graph. There are six fields for each instance of the VANiLLa
dataset: question_id, question, answer, answer_sentence,
question_entity_label, question_relation. Obviously, the dataset represents
only the _correct question-answer pairs_ , which does not allow us to train
the AV classifier properly. To generate the data for the _incorrect question-
answer pairs_ (i.e., negative sampling), we created a set of the corresponding
question-answer pairs by having randomly paired a question and an answer from
different records of the VANiLLa dataset. The method is precisely introduced
in [30].

###  IV-B The KGQA system QAnswer

Our study requires a KGQA system to be utilized in order to prove the approach
validity with actual (correct and incorrect) answer candidates. Therefore, we
selected the well-known QAnswer system [22] which represents the current
state-of-the-art in KGQA and satisfies the requirements for $A_{2}$ and
$A_{3}$. QAnswer provides an API to ask a question and receive the
corresponding ranked query candidate list. It supports questions over several
knowledge bases including Wikidata. The SPARQL queries are constructed by
traversing the KG and discovering how the concepts and relations that are
mentioned in the question are arranged. To the best of our knowledge, there is
no other KGQA system available that satisfies our requirements and provides a
state-of-the-art QA quality. [31] compares the results of DeepPavlov’s and
QAnswer’s top-1 results on RuBQ dataset where QAnswer outperforms DeepPavlov
in terms of most metrics.

###  IV-C Triple-based NLG using Triple2NL

While following our approach $A_{2}$, we require automatically generated NL
representations for query candidates. However, there are quite a few state-of-
the-art open-source toolkits that introduce an API or a module for programming
language.

As the used QA system (QAnswer, cf., the previous subsection) responds with a
list of query candidates, a tool matching most of our demands is Triple2NL. It
represents an integrated system444It provides a REST API, which can be
deployed locally, but it is not able to generate text from more than one
triple. generating a complete NL representation for RDF and SPARQL. The
results (cf., [32, 33]) demonstrate that Triple2NL generates comprehensible
texts. Triple2NL’s approach is based on a bottom-up process to verbalization
and exploits some rules. To create a textual representation, Triple2NL tries
to get the label or the name of all triple elements from DBpedia. If no label
nor name is found, the last part of URI is taken as an NL representation. This
allows us to generate text for any KG by submitting labels instead of URIs.

###  IV-D BERT Model as Answer Validation Classifier

Modern text classification methods utilize large sets of unstructured data to
pre-train a model. In recent years, transformer-based models are holding a
significant part in the whole NLP industry and research community. Such models
as ELMo [27], ULM-FIT [34], XLNet [35], and BERT [36] represent the current
state-of-the-art. The well-known BERT still stays as the landmark model
showing one of the best results in many downstream tasks (cf., [37]).

In this work, we used the bert-base-cased model555Available online at
https://huggingface.co/bert-base-cased. the next sentence prediction (NSP)
setting as the input consists of a textual tuple question-answer.

##  V Experimental Setup

###  V-A Answer Validation Classifier

The AV classifier model (see Section IV-D) was trained and evaluated according
to each answer verbalization approach separately (see Section III). Each
experiment was executed ten times using different random seed values for the
dataset’s shuffle to ensure validity. We used such well-known metrics for
classification as Precision, Recall, and F1 Score. Given the obtained metric
results, we calculated standard deviation (std) which is denoted as “$\pm$” in
our results.

The generated dataset consists of the mixed correct and incorrect subsets and
contains 66,632 records from the original training subset of VANiLLa (the
original test subset was used in the next experiment described in Section
V-B). The train/test split was done randomly in every experiment with the
corresponding ratio 67%/33%. We evaluated both balanced and unbalanced (50
incorrect to 1 correct tuple) dataset’s settings to see the corresponding
effect.

###  V-B Evaluation of Answer Validation’s impact on Question Answering
Quality

To demonstrate the impact of AV on a KGQA system’s quality, we used the
following approach (see Figure 2).

Figure 2: Flowchart of the QA quality comparison process. The quality is
compared between filtered and non-filtered answer candidate sets

The KGQA system was evaluated on the original test subset of VANiLLa
containing 21,360 records, but only 8,955 of them have unique questions. We
decided to send each question to KGQA only once as the response is stable
(i.e., not changing inside one session). A considered KGQA system (here:
QAnswer) processed the test dataset and the obtained ordered lists of query
candidates were cached. As the query candidate was presented in a form of
SPARQL, we were able to apply $A_{2}$ and $A_{3}$ to validate the candidates.
Using one of the selected verbalization approaches, the answer candidates were
being filtered by the AV classifier while the order of candidates stayed the
same (i.e., we removed such answer candidates from the list). Finally, the QA
quality was measured and compared based on the non-filtered and filtered
answer candidate sets. We used well-known metrics such as Precision@k (P@k),
Normalized Discounted Cumulative Gain@k (NDCG@k) with $k\in\\{1,5\\}$ as QA
quality measures [38].

##  VI Results and Discussion

###  VI-A Experiment 1: High quality NL questions and answers ($A_{1}$)

The first experiment is intended to evaluate the capabilities of the AV
component on well-formed answer representations which are retrieved from the
VANiLLa dataset. The results of the experiment are presented in Table II.

TABLE II: Detailed results for Experiment 1 F1 Score | Precision | Recall
---|---|---
balanced data (1 to 1)
0.9968 $\pm$ 0.0089 |  0.9968 $\pm$ 0.0100 |  0.9968 $\pm$ 0.0098
unbalanced data (50 to 1)
0.9838 $\pm$ 0.0276 | 0.9918 $\pm$ 0.0251 | 0.9761 $\pm$ 0.0454

The first experiment’s results are strong and fulfill the quality
requirements. They demonstrate that detecting incorrect question-answer pairs
using only text is possible, and high quality is achievable. Although, using
the unbalanced setting for training negatively affects mostly the recall which
is the most important metric for the intended goal. Hence, assuming NL answers
of very high quality are provided by a QA system, then our approach should be
capable of identifying (and therefore, filtering) incorrect answers.

###  VI-B Experiment 2: NLG of limited quality

In this setting, we are evaluating artificially generated NL answers. There
are computed automatically from SPARQL queries and the corresponding results.

To our best knowledge, there is no KGQA system available that is providing an
API to produce full-fledged NL answers. Consequently, it is required to
generate answer verbalization from the available information (i.e., SPARQL
query candidates).

Generating artificial answers in a three-step process includes (1) providing a
question in textual form to the KGQA system, (2) sending the computed SPARQL
query answer candidates to Wikidata, and (3) generating NL representation from
the obtained list of the query candidates and Wikidata response.

The first step is straightforward: the question provided to the KGQA system.
The system returns a set of query candidates. A typical SPARQL query consists
of one or more triples including at least one variable (cf., the following
example).

⬇

PREFIX wd: <http://www.wikidata.org/entity/>

PREFIX wdt: <http://www.wikidata.org/prop/direct/>

SELECT DISTINCT ?o2 WHERE {

?s1 ?p1 wd:Q57747377 .

?s1 wdt:P21 ?o2 .

} LIMIT 1000

Listing 2: An example SPARQL query over Wikidata.

The second step extracts all variables’ values from the query’s result set as
all triples’ positions must be filled to generate a text. Hence, we change
SELECT DISTINCT ?o2 WHERE to SELECT DISTINCT * WHERE and execute a query on
Wikidata. For the given example it returns:

⬇

[

{

"s1": "wd:Q16027703",

"p1": "wdt:P735",

"o2": "wd:Q6581097"

},

{

"s1": "wd:Q2976815",

"p1": "wdt:P735",

"o2": "wd:Q6581097"

}

]

Listing 3: Result set of the SPARQL query of Listing 2 with SELECT DISTINCT *
WHERE. The RDF vocabulary prefixes are ommited.

The third step is generating the NL representation. The SPARQL request is
split into triples and all variables are substituted with their values. Each
triple is converted into a text individually and then, if there is more than
one triple, joined with the word “and”. For example, the first set of values
gives a sentence: “Claude-Nicolas Le Cat is given name Claude-Nicolas and
Claude-Nicolas Le Cat’s sex or gender is male.” Although the sentence looks
not fluent, it serves our goal well, which is clear from experimental results.

The experiment was carried out on balanced and unbalanced datasets. The
results are shown in Table III.

TABLE III: Detailed results for Experiment 2 F1 Score | Precision | Recall
---|---|---
balanced data (1 to 1)
0.9982 $\pm$ 0.0011 |  0.9980 $\pm$ 0.0020 |  0.9983 $\pm$ 0.0008
unbalanced data (50 to 1)
0.9631 $\pm$ 0.0170 | 0.9321 $\pm$ 0.0317 | 0.9968 $\pm$ 0.0053

The obtained results demonstrate that the difference between the well-formed
textual answers and the generated answers quality metrics is not significant.
However, in this experiment, the precision drops significantly for the
unbalanced set.

###  VI-C Experiment 3: Bag-of-labels representation

In this experimental setting, we evaluate bag-of-labels answer verbalizations
computed from entities and relations that are available in SPARQL query and
its result set. The evaluation results of the AV classifier according to
$A_{3}$ are shown in Table IV.

TABLE IV: Detailed results for Experiment 3 F1 Score | Precision | Recall
---|---|---
balanced data (1 to 1)
0.9613 $\pm$ 0.0029 |  0.9355 $\pm$ 0.0109 |  0.9886 $\pm$ 0.0089
unbalanced data (50 to 1)
0.9205 $\pm$ 0.0570 | 0.9289 $\pm$ 0.0520 | 0.9170 $\pm$ 0.0870

In comparison to the balanced dataset setting, usage of unbalanced data (50
incorrect to 1 correct) leads to a quality drop. Additionally, the
classification results demonstrate less robustness of the model while paying
attention to standard deviation (as in the previous experiment). These results
show that it is still possible to filter incorrect answers just by having the
entity labels of the answer candidate set.

###  VI-D Filtering Answers for a Question Answering Process

In this subsection, the impact of AV on QA quality is demonstrated. We
conducted the corresponding experiments using $A_{2}$ and $A_{3}$, the results
are shown in Table V.

TABLE V: Comparison of QA quality before and after filtering by our AV classifiers P@1 = NDCG@1 | P@5 | NDCG@5
---|---|---
Before AV | After AV | Before AV | After AV | Before AV | After AV
0.2476 | $A_{2}$ | 0.1036 | $A_{2}$ | 0.3249 | $A_{2}$
0.4251 | 0.1368 | 0.4698
$A_{3}$ | $A_{3}$ | $A_{3}$
0.2948 | 0.1183 | 0.3787

The results provide strong evidence that the QA quality was significantly
increased while using both approaches in comparison to not-validated answer
candidate sets. While considering P@1 (NDCG@1), the results were improved from
0.2476 to 0.4251 (i.e., by 71.7%) while using $A_{2}$ and to 0.2948 (i.e., by
19.1%) while using $A_{3}$. The corresponding results w.r.t. the P@5 are as
follows: from 0.1036 to 0.1368 (i.e., by 32.0%) while using $A_{2}$ and to
0.1183 (i.e., by 14.2%) while using $A_{3}$. Finally, the results for NDCG@5
are: from 0.3249 to 0.4698 (i.e., by 44.6%) while using $A_{2}$ and to 0.3787
(i.e., by 16.5%) while using $A_{3}$.

The $A_{2}$ outperforms $A_{3}$ which intuitively comes from the fact that the
$A_{2}$ produces more fluent answer verbalization rather than $A_{3}$. Hence,
the AV classifier is able to capture more semantics and therefore, distinguish
between correct and incorrect question-answer tuples. The $A_{1}$ obviously
could not be evaluated as it would require a huge amount of manual work or a
fine-tuned NLG module. However, we assume that it would produce even higher
quality.

It is worth emphasizing that the AV filtering removed the majority of the
answer candidates. For example, in the case of $A_{2}$ 57 out of 60 candidates
were removed on average, while in the case of $A_{3}$ – 44 out of 60.
Consequently, the position of correct candidates in the list goes up because
of filtering out the incorrect candidates.

###  VI-E Limitations

Notwithstanding the promising results described in the previous subsection,
our work has several limitations. The generalizability of the study is limited
by the used dataset (VANiLLa), KGQA system (QAnswer), NLG tool (Triple2NL),
and model for AV classifier (BERT). The source data for $A_{2}$ (SPARQL
queries) was constrained by removing COUNT, MAX, LIMIT, ORDER and FILTER
clauses as Triple2NL can only verbalize simple RDF sequences.

In the subject-predicate-object structure, VANiLLa defines resource URIs
solely for the predicate, in contrast, only labels are given for the subjects
and the objects of the sentences. Hence, it becomes difficult to determine
whether an answer candidate is correct or not without any ambiguity w.r.t. to
the uniqueness of labels in Wikidata. For example, there are 6172 resources
with the label “Correction” from the VANiLLa test subset. While it is hard to
distinguish automatically entities with the same label for a given question,
we considered any entity with a matching label as relevant for the current
question.

There is another limitation in the VANiLLa dataset’s structure. Some questions
imply many correct answers, but only one of them is given as a reference in
the dataset. In this case, we believe that if an object’s (or subject’s) label
and predicate correspond to the VANiLLa record, then the answer is correct,
regardless of the number of entities.

Obviously, these compromises might influence the quality (i.e., there is an
opportunity to improve our approach).

###  VI-F Summary

Despite several limitations described in the previous subsection, the answer
validation approach demonstrated its effectiveness by eliminating incorrect
answer candidates based only on NL representations of questions and possible
answers.

The AV classifier on well-defined answer verbalizations ($A_{1}$) as well as
on the automatically generated answer verbalizations ($A_{2}$, $A_{3}$)
demonstrated a reasonable F1 Score – 0.9968, 0.9982, and 0.9613 respectively
for balanced data setup. The unbalanced data setup leads to the significant
decrease in the classification quality. Therefore, the AV classifier should be
trained on a balanced dataset to avoid a high false-negative rate while
filtering answer candidates.

Given the comparison results before and after answer filtering with AV
classifier, we were enabled to significantly improve the KGQA system’s
quality. The maximal improvement was achieved using $A_{2}$: +71.7%
(P@1/NDCG@1), +32.0% (P@5), and +44.6% (NDCG@5). Although, $A_{3}$ also
demonstrated reasonable quality improvement.

While answering RQ1, the AV classifier is capable of eliminating wrong answers
and, therefore, moving correct answers to the top of the result list. The AV
classifier is capable of distinguishing between correct and incorrect answers
based just on automatically generated verbalizations obtained within $A_{2}$
and $A_{3}$ (RQ3). We could not fully answer RQ2 as the manual NL generation
for each query candidate requires huge labor costs. Therefore, extrapolating
classification results of $A_{1}$ (Table II) on QA quality comparison
experiments (Table V), we assume that the possible outcome of $A_{1}$ may even
outperform $A_{2}$ and $A_{3}$.

##  VII Conclusion

In this paper, we presented an approach to filter computed NL answer
candidates within a KGQA process. We have purposely limited ourselves to use
only NL features, in particular, labels of the entities, properties, and
concepts, s.t., our approach is not tightly integrated into existing KGQA
systems. Hence, our approach is a _generalized method for optimizing KGQA
systems_. For proving this, we defined 3 research questions that were answered
by our experiments. As we have used the state-of-the-art KGQA system QAnswer,
we consider our results to be very solid, as we were capable of improving the
quality even for this well-established system. Hence, we can conclude that our
approach should be applicable to any QA system that provides additional
information about the answer candidates.

Our results show a huge impact w.r.t. the VANiLLa dataset. In the executed
experiments, we were capable of removing incorrect answer candidates (from the
ranked list of SPARQL queries), s.t., the result improved by 71.7% and 44.6%
(w.r.t. P@1 and NDCG@5).

In the future, we will integrate our approach into existing QA systems that
are using NL input and output, s.t., an additional QA quality improvement is
achieved. It would also be possible to use the results of our classification
as an additional feature for the core of a QA engine, s.t., the quality of
answer candidates might be improved on a lower system level. In particular,
there are QA systems that provide high performance while at the same time not
interpreting the question’s semantics precisely. Such systems might benefit
heavily from our approach.

## References

  * [1] A. Ns, A. Azhari, and A. Sari, “Survey on answer validation for Indonesian question answering system (IQAS),” _International Journal of Intelligent Systems and Applications_ , vol. 10, pp. 68–78, 04 2018.
  * [2] A. Rodrigo, J. Pérez-Iglesias, A. Peñas, G. Garrido, and L. Araujo, “A question answering system based on information retrieval and validation,” in _CLEF 2010 LABs and Workshops, Notebook Papers_ , 2010.
  * [3] J. Pérez-Iglesias, G. Garrido, Á. Rodrigo, L. Araujo, and A. Peñas, “Information retrieval baselines for the respubliqa task,” in _CLEF_ , 2009\.
  * [4] A. Téllez-Valero, M. M. y Gómez, L. Pineda, and A. Peñas, “Towards multi-stream question answering using answer validation,” _Informatica (Slovenia)_ , vol. 34, pp. 45–54, 2010.
  * [5] B. Magnini, D. Giampiccolo, P. Forner, C. Ayache, V. Jijkoun, P. Osenova, A. Peñas, P. Rocha, B. Sacaleanu, and R. Sutcliffe, “Overview of the clef 2006 multilingual question answering track,” in _Evaluation of Multilingual and Multi-modal Information Retrieval_ , C. Peters, P. Clough, F. C. Gey, J. Karlgren, B. Magnini, D. W. Oard, M. de Rijke, and M. Stempfhuber, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2007, pp. 223–256.
  * [6] S. Babych, A. Henn, J. Pawellek, and S. Padó, “Dependency-based answer validation for German,” in _CLEF 2011 Labs and Workshop, Notebook Papers, 19-22 September 2011, Amsterdam, The Netherlands_ , ser. CEUR Workshop Proceedings, V. Petras, P. Forner, and P. D. Clough, Eds., vol. 1177\. CEUR-WS.org, 2011.
  * [7] L. Vanderwende, “Answering and questioning for machine reading,” in _Machine Reading, Papers from the 2007 AAAI Spring Symposium_ , 2007, pp. 91–96.
  * [8] A. Grappy, B. Grau, M. Falco, A. Ligozat, I. Robba, and A. Vilnat, “Selecting answers to questions from web documents by a robust validation process,” in _2011 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology_ , vol. 1, 2011, pp. 55–62.
  * [9] I. Glöckner and K. Weis, “An integrated machine learning and case-based reasoning approach to answer validation,” in _2012 11th International Conference on Machine Learning and Applications_ , vol. 1, 2012, pp. 494–499.
  * [10] I. Glöckner and B. Pelzer, “The LogAnswer Project at CLEF 2009,” in _Working Notes for CLEF 2009 Workshop co-located with the 13th European Conference on Digital Libraries (ECDL 2009) , Corfù, Greece, September 30 - October 2, 2009_ , ser. CEUR Workshop Proceedings, C. Peters and N. Ferro, Eds., vol. 1175. CEUR-WS.org, 2009.
  * [11] P. Pakray, U. Barman, S. Bandyopadhyay, and A. Gelbukh, “Semantic answer validation using universal networking language,” in _International Journal of Computer Science and Information Technologies_ , vol. 3 (4), 2012.
  * [12] Á. Rodrigo, A. Peñas, and F. Verdejo, “Overview of the answer validation exercise 2008,” in _Evaluating Systems for Multilingual and Multimodal Information Access_ , C. Peters, T. Deselaers, N. Ferro, J. Gonzalo, G. J. F. Jones, M. Kurimo, T. Mandl, A. Peñas, and V. Petras, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2009, pp. 296–313.
  * [13] H. Gómez-Adorno, D. Pinto, and D. Vilariño, “A question answering system for reading comprehension tests,” in _Pattern Recognition_ , J. A. Carrasco-Ochoa, J. F. Martínez-Trinidad, J. S. Rodríguez, and G. S. di Baja, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp. 354–363.
  * [14] A. Solovyev, _Dependency-Based Algorithms for Answer Validation Task in Russian Question Answering_. Springer, Berlin, Heidelberg, 01 2013, vol. 8105, pp. 199–212.
  * [15] ——, “Who is to blame and where the dog is buried? method of answers validations based on fuzzy matching of semantic graphs in question answering system.” in _ROMIP 2011_ , Kazan, Russia, 2011, pp. 125–141.
  * [16] I. Zamanov, M. Kraeva, N. Hateva, I. Yovcheva, I. Nikolova, and G. Angelova, “Voltron: A hybrid system for answer validation based on lexical and distance features,” in _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_. Denver, Colorado: Association for Computational Linguistics, Jun. 2015, pp. 242–246.
  * [17] P. Nakov, L. Màrquez, W. Magdy, A. Moschitti, J. Glass, and B. Randeree, “SemEval-2015 task 3: Answer selection in community question answering,” in _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_. Denver, Colorado: Association for Computational Linguistics, Jun. 2015, pp. 269–281.
  * [18] C. Tan, F. Wei, Q. Zhou, N. Yang, W. Lv, and M. Zhou, “I know there is no answer: Modeling answer validation for machine reading comprehension,” in _Natural Language Processing and Chinese Computing_ , M. Zhang, V. Ng, D. Zhao, S. Li, and H. Zan, Eds. Cham: Springer International Publishing, 2018, pp. 85–97.
  * [19] M. Hu, F. Wei, Y. xing Peng, Z. Huang, N. Yang, and M. Zhou, “Read + verify: Machine reading comprehension with unanswerable questions,” _ArXiv_ , vol. abs/1808.05759, 2019.
  * [20] P. Rajpurkar, R. Jia, and P. Liang, “Know what you don’t know: Unanswerable questions for SQuAD,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 784–789.
  * [21] M. Dubey, S. Dasgupta, A. Sharma, K. Höffner, and J. Lehmann, “AskNow: A framework for natural language query formalization in SPARQL,” in _The Semantic Web. Latest Advances and New Domains - 13th International Conference, ESWC 2016, Heraklion, Crete, Greece, May 29 - June 2, 2016, Proceedings_ , ser. Lecture Notes in Computer Science, H. Sack, E. Blomqvist, M. d’Aquin, C. Ghidini, S. P. Ponzetto, and C. Lange, Eds., vol. 9678. Springer, 2016, pp. 300–316.
  * [22] D. Diefenbach, A. Both, K. Singh, and P. Maret, “Towards a question answering system over the semantic web,” _Semantic Web_ , vol. 11, pp. 421–439, 2020\.
  * [23] D. Chen, A. Fisch, J. Weston, and A. Bordes, “Reading Wikipedia to answer open-domain questions,” in _Association for Computational Linguistics (ACL)_ , 2017.
  * [24] B. Magnini, M. Negri, R. Prevete, and H. Tanev, “Is it the right answer? exploiting web redundancy for answer validation,” in _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_. Philadelphia, Pennsylvania, USA: Association for Computational Linguistics, 2002, pp. 425–432.
  * [25] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in _Advances in Neural Information Processing Systems 26 (NIPS 2013)_ , 2013\.
  * [26] J. Pennington, R. Socher, and C. D. Manning, “GloVe: Global vectors for word representation,” in _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2014.
  * [27] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” in _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , 2018.
  * [28] D. Biswas, M. Dubey, M. R. A. H. Rony, and J. Lehmann, “VANiLLa: Verbalized answers in natural language at large scale,” _CoRR_ , vol. abs/2105.11407, 2021.
  * [29] E. Kacupaj, B. Banerjee, K. Singh, and J. Lehmann, “ParaQA: A question answering dataset with paraphrase responses for single-turn conversation,” in _European Semantic Web Conference_. Springer, 2021, pp. 598–613.
  * [30] A. Both, A. Gashkov, and M. Eltsova, “Similarity detection of natural-language questions and answers using the VANiLLa dataset,” _Journal of Physics: Conference Series_ , vol. 1886, no. 1, p. 012017, April 2021.
  * [31] V. Korablinov and P. Braslavski, “RuBQ: A Russian dataset for question answering over Wikidata,” in _International Semantic Web Conference_. Springer, 2020, pp. 97–110.
  * [32] A.-C. N. Ngomo, D. Moussallem, and L. Bühmann, “A holistic natural language generation framework for the semantic web,” _arXiv preprint arXiv:1911.01248_ , 2019.
  * [33] T. Ferreira, C. Gardent, N. Ilinykh, C. van der Lee, S. Mille, D. Moussallem, and A. Shimorina, “The 2020 bilingual, bi-directional WebNLG+ shared task overview and evaluation results (WebNLG+ 2020),” in _Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+)_ , 2020.
  * [34] R. S. Howard J, “Universal language model fine-tuning for text classification,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2018.
  * [35] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, “XLNet: Generalized autoregressive pretraining for language understanding,” in _33rd Conference on Neural Information Processing Systems NeurIPS_ , 2019.
  * [36] J. Devlin, M.-W. Chang, K. Lee, and K. N. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _ArXiv e-prints_ , 2018.
  * [37] C. Sun, X. Qiu, Y. Xu, and X. Huang, “How to fine-tune BERT for text classification?” _ArXiv_ , vol. abs/1905.05583, 2019.
  * [38] H. Schütze, C. D. Manning, and P. Raghavan, _Introduction to information retrieval_. Cambridge University Press Cambridge, 2008, vol. 39.


# Increasing the price of a university degree does not significantly affect
enrolment if income contingent loans are available: evidence from HECS in
Australia

Fabio I. Martinenghi<EMAIL_ADDRESS>[ School of Economics,
University of New South Wales, Australia.  Australian Research Council Centre
of Excellence for Children and Families over the Life Course, Australia.

###### Abstract

I provide evidence that, when income-contingent loans are available, student
enrolment in university courses is not significantly affected by large
increases in the price of those courses. I use publicly available domestic
enrolment data from Australia. I study whether large increases in the price of
higher education for selected disciplines in Australia in 2009 and in 2012 was
associated with changes in their enrolment growth. I find that large increases
in the price of a course did not lead to significant changes in their
enrolment growth for that course.

JEL classification: I21, I23, I28.

###### keywords:

Educational finance , Income contingent loans , Demand for higher education

url]https://fabitmart.github.io/

##  1 Introduction

In Australia, the federal government provides financial aid for its
undergraduate domestic students (hereafter referred to as “students”) in
several ways. A complete exposition of how this system works can be found in
Chapman, (2006) and a summary in Chapman and Leigh, (2009). I will only
outline the essential features of this system. Under the Higher Education
Support Act 2003 (The Parliament of Australia,, 2003), the government pays
their degrees upfront and in full. Part of this government payment is a
subsidy, while the remainder is a loan. This latter part is called “student
contribution” and it is the total amount that students will pay for their
university degree. The student contribution is repaid via the Higher Education
Contribution Scheme (HECS). Under this scheme, students will gradually repay
the government only after they earn a sufficiently high income. In short, the
government gives prospective students an income-contingent loan111On income-
contingent loans, see for instance (Stiglitz et al.,, 2014) and (Britton et
al.,, 2019).. Moreover, HECS is progressive so that graduates with higher
incomes will pay the loan back at higher rates.

The federal government sets the maximum student contribution that Australian
universities can charge for each discipline, or, equivalently, the highest
amount that students will have to pay for studying a certain discipline. Each
discipline is assigned to one of three bands, which are associated with
different maximum student contributions. The Higher Education Support Act 2003
(The Parliament of Australia,, 2003) introduced a fourth band, the “National
Priorities”. This band is assigned to those disciplines that the Commonwealth
government is seeking to promote, including by reducing their maximum student
contribution. Between 2005-2009, Nursing and Education were declared national
priorities. Between 2009-2012, all the Natural and Physical Sciences (which
include Mathematical Sciences) were declared national priorities.

After 2009, when Nursing and Education were no longer considered a national
priority, their maximum student contributions increased by approximately 25%,
from $4249 to $5310. After 2012, when the Natural and Physical Sciences were
no longer considered a national priority, their maximum student contributions
increased by approximately 85%, from $4520 to $8363.

The HECS system was designed to make higher education financially accessible
to every Australian. The income-contingent nature of the loan is a feature
aimed at helping students with the cost of their degrees. It is possible that
the HECS system has relieved students from funding concerns to the point that
they have become insensitive to changes in their student contribution. If this
were true, it would imply that under a system like HECS, where income-
contingent loans are available to students, a government cannot effectively
use university fees (maximum student contribution) as a policy instrument to
boost demand for certain disciplines. In other words, changing the price of a
degree to encourage or discourage enrolment in it would not work.

In this note, I test whether the event of removing a discipline from the
National Priority band – hence increasing its maximum student contribution –
is associated which a change in the student enrolment for that discipline. The
change in student enrolment is measured as a change in the slope of the
student enrolment trend after the event. I find that moving a discipline from
the National Priority band had no significant impact on student enrolment.

##  2 Methods

###  2.1 Data

I collect and combine publicly available data on commencing domestic
undergraduate students and on the Annual Course Contribution Value (ACCVAL)
tables between 2004-2018. The ACCVAL tables provide information on maximum
student contribution by discipline. Years 2004 and 2005 have been dropped from
the analysis due to inconsistencies in how cross-institutional undergraduate
students were categorised. This categorisation differs from the rest of the
sample and creates uncertainty around the yearly number of commencing
undergraduate students. The dataset and the R code to reproduce it222This
includes an automated download of all the data and their manipulation. are
available on my website (link).

###  2.2 Empirical framework

While commencing students data is available for all disciplines, the sample
used for estimation does not include disciplines that were never in the
National Priority band. Given this data availability, an obvious candidate
methodology is difference-in-differences (DID). However, the parallel trends
assumption does not seem to hold when comparing the time series of the
treatment and candidate control groups (not shown). Different disciplines
often display markedly different trends which—in absence of richer and more
disaggregated data—will lead to biased estimates. Moreover, the “no spillover
effects” assumption does not seem to hold either. This is because, by design,
the disciplines chosen for the control group would have to be similar to the
ones in the treatment group. This very similarity implies that a student might
respond to an increase in the price for a degree in the treatment group by
choosing another similar degree in the control group. This would violate the
“no spillover effects” assumption.

Instead of a difference-indifferences approach, I choose one close to an event
study. I define the event as “discipline $i$ is removed from the National
Priority band (or, equivalently, it experiences an increase in student
contribution)”. There are two instances of this event: one in 2009 and a
second in 2012, as detailed above. Furthermore, I restrict the sample to (i)
the ever-treated disciplines, i.e. the disciplines that were in the National
Priority band at least once; (ii) a time window spanning between 3 years
before each event and 5 years after. The size of the window is determined by
how many periods are available around both instances of the event. I use this
sample to test whether, under an income-contingent loan regime as HECS,
drastically increasing the maximum student contribution for a discipline—i.e.
the cost to study it—is followed by significant changes in enrolment trends in
that discipline.

###  2.3 Empirical specification

To test this hypothesis, I estimate the following fixed effect equation for
discipline $i$ and time $t$:

$ln(Y_{i,t})=\alpha_{i}+\beta_{1}T_{t}+\beta_{2}D_{i,t}+\delta
D_{i,t}*T_{t}+\epsilon_{i,t}$ (1)

where $ln(Y_{i,t})$ is the natural logarithm of the number of commencing
students by discipline and year, $T_{t}$ is a yearly event time variable333An
event time variable is a variable equal to the number of periods before/after
the event. It takes negative values before the event, positive values after
the event and it equals to zero in the year of the event., $D_{i,t}$ takes a
value of one after the discipline is no longer in the National Priority band
and zero otherwise and $D_{i,t}*T_{t}$ is their interaction term. The
coefficient of interest is $\delta$, which can be interpreted as the yearly
percentage change in enrolments following a sharp increase in student
contribution.

##  3 Results

###  3.1 Graphical analysis

Figure 1 shows the number of commencing students enrolled by disciplines and
year. The disciplines shown are those which have been in the National Priority
band at least once. Panel (a) shows the number of yearly commencing students
who benefited from the National Priority band between 2005 and 2009 - where
the year 2009 is marked by a red vertical line. The National Priority band
between these years included disciplines related to Nursing and Education.
Panel (b) shows the number of yearly commencing students who benefited from
the National Priority band between 2009 and 2012 – 2012 also being marked in
red. The National Priority band during these years included all Physical and
Natural Sciences, plus Mathematics.

An inspection of Figure 1 suggests that commencing students were rather
insensitive to the large changes in student contribution marked by the end of
the National Priority programmes. In other words, the demand for those
disciplines seems inelastic to changes in price. The only discipline not
displaying an uninterrupted linear trend is Teacher Education. The number of
Teacher Education commencing students peaks in 2012, after which a decreasing
trend starts. This change in trend occurs several years after the change in
student contribution, hence the two are unlikely to be related.

(a) Nursing and Education

(b) Physical and Natural Sciences

Figure 1: Termination of National Priority

###  3.2 Regression results

Column 1 in Table 1 shows the estimates for Equation 1. As anticipated by the
graphical analysis, on average there is no significant change in the trend of
enrolments after a sharp increase in student contribution ($\delta=-0.06$ with
a p-value of 0.16). In Column 2, Equation 1 is estimated in the restricted
case where $\beta_{2}=0$. In other words, in Column 2 the intercept is not
allowed to vary after the event. Column 3 and 4 estimate the same equations as
columns 1 and 2, respectively, but dropping the field “Other Education” from
the sample. While the coefficient of interest is never significantly different
from zero, Columns 3 and 4 show that “Other Education” is responsible for half
of the size of the coefficient of interest, $\delta$. When “Other Education”
is dropped, $\delta$ reduces to less than $0.04$ (in both Columns 3 and 4) and
the p-values double to $0.32$ (also in both columns). The leverage of “Other
Education” can also be noticed upon inspecting Figure 2, which visualises the
sample after trimming and centering the outcome variable, $ln(Y_{i,t})$.

Table 1: Changes in enrolments after an increase in student contribution

| (1) | (2) | (3) | (4)
---|---|---|---|---
T | 0.0523 | 0.0632∗ | 0.0496 | 0.0591
| [-0.01,0.11] | [0.00,0.13] | [-0.02,0.12] | [-0.01,0.13]
| (0.03) | (0.03) | (0.03) | (0.03)
Event | 0.0651∗∗ |  | 0.0574∗∗ |
| [0.03,0.10] |  | [0.02,0.09] |
| (0.02) |  | (0.02) |
Event$\times$T | -0.0616 | -0.0616 | -0.0363 | -0.0363
| [-0.15,0.03] | [-0.15,0.03] | [-0.12,0.04] | [-0.12,0.04]
| (0.04) | (0.04) | (0.03) | (0.03)
Constant | 8.206∗∗∗ | 8.231∗∗∗ | 8.446∗∗∗ | 8.468∗∗∗
| [8.10,8.31] | [8.12,8.34] | [8.37,8.52] | [8.39,8.55]
| (0.05) | (0.05) | (0.03) | (0.04)
Obs. | 90 | 90 | 81 | 81
95% confidence intervals in brackets, robust s.e. in parentheses
∗ $p<0.05$, ∗∗ $p<0.01$, ∗∗∗ $p<0.001$

  * 1.

Note. Column 1 estimates of Equation 1. A linear model with discipline fixed
effects is fit to data on the logarithm of yearly commencing students by
discipline. The specification controls for a linear event time trend, which is
interacted with the treatment dummy variable. A discipline is defined as
treated after it is no longer in the National Priority band. Column 2
restricts Equation 1 to the case of no change in the intercept after the event
(i.e. $\beta_{2}=0$). Columns 3 and 4 estimate the same equations as column 1
and 2 respectively but excluding “Other education”. This is done in order to
show how “Other education” alone doubles the size of the (negative)
coefficient.

Table 2: Testing $\delta=0$ using 90% confidence intervals

| (1) | (2) | (3) | (4)
---|---|---|---|---
Event$\times$T | -0.0616 | -0.0616 | -0.0363 | -0.0363
| [-0.14,0.01] | [-0.13,0.01] | [-0.10,0.03] | [-0.10,0.03]
Obs. | 90 | 90 | 81 | 81
90% confidence intervals in brackets
∗ $p<0.05$, ∗∗ $p<0.01$, ∗∗∗ $p<0.001$

  * 1.

Note. I construct and inspect 90% confidence intervals for $\delta$, using
robust standard errors. This is for testing whether $\delta=0$. The model
specifications are identical to the ones associated with Table 1 and are
identically ordered. I am again unable to reject the null hypothesis of
$\delta=0$.

As an additional test of $\delta=0$ and following (Steiger,, 2004), I
construct and inspect the 90% confidence interval for $\delta$ (Table 2). The
model specifications are identical to the ones associated with Table 1. In
each specification, zero is included within the 90% confidence interval. In
Columns (3) and (4), whose specifications exclude the outlier discipline, the
confidence interval is even more centered around zero.

##  4 Discussion

These results are consistent with the hypothesis that income-contingent loans,
such as the one featuring in the HECS system, make the demand for a discipline
inelastic to price increases. Even when the price for studying a discipline
(the student contribution) increases sharply, the trend in new enrolments does
not significantly decrease on average. Therefore, HECS seems to be meeting its
goal of allowing students to make decisions about their higher education
independently its price.

At the same time, these results suggest that if students have access to
income-contingent loans, they will not respond to policies that increase the
price of a discipline to disincentivise them from choosing it. In light of the
evidence provided, the economic argument supporting this kind of policies
seems weak, leaving a justification based on moral grounds as the only avenue
for a government to support them444(Chapman,, 1997) calls this issue “the
issue of whether or not it is appropriate for the government to charge
individuals based on what the expected direct benefits on average from a
course are”..

Figure 2: Event window

  * 1.

Note. This figure shows the centered log-transformation of commencing students
by year and discipline. These disciplines were all at least once in the
National Priority band. Event time zero is the time at which the event “being
removed from the National Priority band” occurs. The colour coding is
identical to Figure 1’s, please refer to that legend.

## Declaration of Competing Interest

I have no conflicting interest to declare

## Acknowledgements

I thank Andrew Norton for his help in navigating the Australian education
policy world and for sharing his data, which allowed me to start working on
this project quickly. I thank Oliver Maclaren, Daniël Lakens and Guy Prochilo
for providing guidance and references on how to provide evidence of absence of
an effect rather than an absence of evidence.

## References

  * Britton et al., (2019) Britton, J., van der Erve, L., and Higgins, T. (2019).  Income contingent student loan design: Lessons from around the world.  Economics of Education Review, 71:65 – 82.  Higher Education Financing: Student Loans.
  * Chapman, (1997) Chapman, B. (1997).  Conceptual Issues and the Australian Experience With Income Contingent Charges for Higher Education.  The Economic Journal, 107(442):738–751.
  * Chapman, (2006) Chapman, B. (2006).  Government Managing Risk: Income contingent loans for social and economic progress, volume 40.  Routledge, London.
  * Chapman and Leigh, (2009) Chapman, B. and Leigh, A. (2009).  Do very high tax rates induce bunching? implications for the design of income contingent loan schemes*.  Economic Record, 85(270):276–289.
  * Steiger, (2004) Steiger, J. (2004).  Beyond the f test: Effect size confidence intervals and tests of close fit in the analysis of variance and contrast analysis.  Psychological methods, 9:164–82.
  * Stiglitz et al., (2014) Stiglitz, J. E., Higgins, T., and Chapman, B. (2014).  Income Contingent Loans: Theory, Practice and Prospects.  Palgrave Macmillan.
  * The Parliament of Australia, (2003) The Parliament of Australia (2003).  Higher education support act 2003.
https://www.legislation.gov.au/comlaw/Legislation/ActCompilation1.nsf/0/91FD9474C87B50A0CA256F85000DEF57/$file/HigherEducationSupport2003WD02.pdf.