# Harnessing expressive capacity of Machine Learning modeling to represent
complex coupling of Earth’s auroral space weather regimes
††thanks: With support from DARPA under Department of the Interior Heliosphere
to Earth’s Atmosphere Rendering via Excellent Artificial Intelligence Training
(HEARTBEAT) Award D19AC00009 to Georgia Institute of Technology with subaward
to ASTRA as well as acknowledging R. McGranaghan’s NASA Early Career
Investigator Program grant #80NSSC21K0622.

Jack Ziegler  Atmospheric Space and Technology Research Associates (ASTRA)
Louisville, CO, USA
<EMAIL_ADDRESS>Ryan M. Mcgranaghan  ASTRA
Louisville, CO, USA
<EMAIL_ADDRESS>
###### Abstract

We develop multiple Deep Learning (DL) models that advance the state-of-the-
art predictions of the global auroral particle precipitation. We use
observations from low Earth orbiting spacecraft of the electron energy flux to
develop a model that improves global nowcasts (predictions at the time of
observation) of the accelerated particles. Multiple Machine Learning (ML)
modeling approaches are compared, including a novel multi-task model, models
with tail- and distribution-based loss functions, and a spatio-temporally
sparse 2D-convolutional model. We detail the data preparation process as well
as the model development that will be illustrative for many similar time
series global regression problems in space weather and across domains. Our ML
improvements are three-fold: 1) loss function engineering; 2) multi-task
learning; and 3) transforming the task from time series prediction to spatio-
temporal prediction. Notably, the ML models improve prediction of the extreme
events, historically obstinate to accurate specification and indicate that
increased expressive capacity provided by ML innovation can address grand
challenges in the science of space weather.

###### Index Terms:

Deep Learning, regression, space weather, loss function, modeling the tail,
nowcasting, inverse convolution, sparse data, spatio-temporal separation,
auroral physics

##  I Introduction

‘Space weather’ refers to the tangible effects that solar energy has on our
technologies, space- and ground-based infrastructure, and our daily lives [1].
It is a wide-reaching science that spans from the sun to the Earth’s surface,
requiring the integration and sophisticated representation of heterogeneous
and complex data. A key to understanding the weather in outer space is how
regions between the Sun and the Earth’s surface are connected. One challenging
space weather phenomena to model is the coupling of energy from the solar wind
into the magnetosphere and down into the Earth’s upper atmosphere (100–1,000
km altitude). Particles moving along magnetic field lines ‘precipitate’ into
this region, carrying energy and momentum which drive space weather and a host
of societal impacts such as satellite damage or destruction and disruptions to
communication systems.

Predicting auroral particle precipitation is a grand challenge in space
weather because it is a complex result of driving parameters from the sun,
solar wind, and Earth space environment. We need prediction capabilities for
the global spatio-temporal patterns of precipitation, a problem that has been
obstinate to empirical approaches. In this work we advance the state of
precipitation prediction by applying cutting-edge techniques from the field of
ML.

The specific problem formulation is to use observations of the solar wind (the
carrier of solar energy throughout interplanetary space), the magnetosphere,
and the Earth’s space environment to predict the spatio-temporal pattern of
total electron energy flux as observed by the Defense Meteorological Satellite
Program (DMSP) satellites. These fluxes represent electrons that come from the
magnetosphere and move along magnetic field lines into the upper atmosphere
(100-1000 km altitude). When they impact molecules (e.g., our atmosphere) they
cause ionizations and visible emissions – the aurora.

Auroral particle precipitation is a critical input to global circulation
models (e.g., [2]), which describes the effects that the energy deposited by
electrons and ions has. This causes ionizations that change the ionosphere and
affect radio communications, heating the thermosphere leading to enhanced
neutral density. This changes the orbits of all objects within the upper
atmosphere and the coupling to the entire Earth system. Our models use DMSP
data dating back to 1987. DMSP spacecraft carry a particle detector (SSJ)
which is sensitive to particles with characteristic energies between 30 eV and
30 keV. Figure 1 shows polar plots that each have three satellites,
illustrating their coverages over different years [3].

In this work, we first treat each satellite measurement independently and then
use all measurements available during a small window as simultaneous to
construct multi-spatial observations at a given time step.

Figure 1: Combining NH and SH observations from multiple satellites to create
a dataset that nearly covers all local time (0 to 24 hour time zones) [4].

Our models achieve $>50\%$ reduction in errors from the current state-of-the-
art model, The Oval Variation, Assessment, Tracking, Intensity, and Online
Nowcasting Prime (OVATION Prime) model [5, 6]. These new models better capture
the dynamic changes of the auroral electron energy flux, and provide evidence
that it can capably reconstruct mesoscale phenomena.

The key contributions of this work are:

  1. 1.

Advancing the state-of-the-art ML applied to auroral particle precipitation;

  2. 2.

Assessment of the efficacy of increasing ML model expressive capacity (loss
function engineering, multi-task learning, and transforming time series
prediction to spatio-temporal prediction);

  3. 3.

New lessons learned to advance ML research through application to novel
science data.

Item 2) is of important significance to other ML modeling problems dealing
with sparse 2D temporal data, a common problem for any model using satellite
sensor data or data from any sensor that is following 1-D traces on a 2D
domain.

##  II Related Work

Existing models of particle precipitation do not adequately capture the
mesoscale (100-400 km scale) spatial features and take advantage of only a
small number of the important driving parameters. Typically, in the upper
atmosphere, simple statistical and empirical patterns of the particle
precipitation coarsely parameterized by small subsets of the available
information are used to quantify the complex processes [7].

Particle precipitation models such as the widely used [8] model are driven
only by the planetary Kp index whose time resolution is three hours and
prevents the model from capturing changes that occur on a shorter time-scales.
More recently, models have embraced the need to incorporate more information
and to utilize different sets of organizing parameters as a cornerstone to
advancing our understanding and ability to predict particle precipitation.
Lacking from the [8] and related empirical models is a realistic
representation of the temporal dependence between solar wind input and
particle precipitation response. It has also become apparent that increasing
the spatial resolution of the models reveals physically meaningful mesoscale
behavior (defined in this work to be 100–400 km, 1-3∘ magnetic latitude (MLAT)
width).

The series of OVATION, later becoming OVATION Prime (OVP) models [9, 5, 10,
11, 12] utilized solar wind parameters, making the implicit assumption that
these parameters are the cause and the observed electron precipitation the
effect. Moreover, data from geospace (e.g., geomagnetic indices) and from LEO
satellites assumes that these parameters provide information of the
simultaneous and global electron precipitation distribution (e.g., ’state
descriptors’ [13]). OVP was a significant advance, however is limited by a
reliance on relatively simple relationships to organizing parameters and
spatial binning. The success and shortcomings of OVP give credence to the
importance of organizing parameters and representation of driver-response
relationship. [14] took the first step to address these gaps and applied ML to
create the current state-of-the-art model, PrecipNet. They showed that rich
information exists in the spectrum of Heliophysics observational data and that
careful representation of this information with ML can lead to profound
progress. That work lays the foundation for the advance we present in this
paper.

##  III Advancing the Frontier of ML for Auroral Particle Precipitation

PrecipNet and related work represent the state-of-the-art for ML applied to
auroral particle precipitation. Against this backdrop, we raise the question,
“Does increasing the expressive capacity/complexity of the ML algorithm lead
to demonstrable improvements in ML prediction for physical problems?” We
explore four new areas of the ML algorithm search space:

  1. 1.

Multi-task learning models

  2. 2.

‘Tail of the distribution’-specific custom loss functions

  3. 3.

Sample distribution weighted loss function

  4. 4.

Novel 2D, inverse convolutional[15] (Conv2D) model for sparse spatial-temporal
training data

In addition, we explore combinations of the four areas. [14] outlined a
consistent and rigorous framework for assessing space weather models and we
adopt their evaluation formalism (Section 6) to justify our quantifiable
improvements.

###  III-A Input Parameters

Our models utilize the time-dependent NASA space weather data
https://omniweb.gsfc.nasa.gov/OMNIWeb as inputs. The solar wind parameters and
magnetic field values are primary examples. All of the details of our dataset
and python scripts for data preparation are provided for reproducibility and
extensibility https://github.com/rmcgranaghan/HEARTBEAT GIT repo. The focus of
this work is on model improvements and comparisons and therefore the inputs
and outputs are the same as in [14]. Inputs are time series and the target to
be predicted represents both 2D spatial and temporal variations. For any time
step, only spatially sparse target observations exist (at only one or a few
spatial locations from different simultaneous observing satellites). Thus, we
must develop relationships between the input features and the particle
precipitation response across many years to construct a dataset that is
representative of both the spatial and temporal variations.

  * •

Inputs: Global solar wind observations and geomagnetic indices at five-min.
cadence;

  * •

Target Output: Total electron energy flux at any given location (nowcast),
MLat and MLT (Magnetic Local Time (zone)), with (altitude variations ignored).

We pose the problem as regression, predicting the magnitude of the total
electron energy flux using both instantaneous and time history observations.
Note that a DMSP spacecraft observation is only available at one location at
any given time. There may be multiple satellites providing observations at
some but not all time steps.

The solar wind input variables are collected by satellites orbiting between
the Sun and the Earth and quantify the state of the solar energy that drives
space weather. The geomagnetic indices are proxies for space weather activity
based on magnetometer observations at Earth’s surface and aggregated into
global indices that quantify the near-Earth space environment (like a
hurricane is described as e.g., category 1 - 5).

The ML target is the total electron energy flux. Our withheld validation set
consists of all DMSP satellite data from the F16 spacecraft in the year 2010,
more than 50000 data points when using one-min. resolution. Taking a
consecutive year of data for validation, rather than randomly sampling the
full database, prevents influence by autocorrelations in the data.
Model Inputs: 148 features total (full list described in [14]).

  * •

Spatial and temporal inputs: Magnetic Latitude (MLAT), Magnetic local time
(MLT) (which can be seen in Figure 1: rings are increments of MLAT from the
pole (90∘) and MLT is around the dial (the sun, 12 noon, is off to the top of
the figures)

  * •

Global variables’ time history inputs:
Instantaneous (referenced to the target observation time): 0, -5, -10, and -15
min.
Averaged: -30 and -45 min. and -1, -3, -5, and -6 hrs.
Geomagnetic indices: AE, AL, AU, F107, SymH
Solar wind-dependent and -derived parameters:
Bx, By, Bz, Vsw, Psw, Vx, PC, Newell [16] and Borovsky [17] Coupling Functions

Model output target: Total Electron Energy Flux at time 0. Validation points
are at points where satellites are but the models are easily evaluated at all
spatial point at one time step for weather nowcasting.

By “global” inputs we mean that they represent a parameter for the whole Earth
and do not have a particular spatial location, just a temporal variation. We
explore more parsimonious input feature sets (less than the full 148
features), but those results are beyond the scope of this short paper. Also,
note that the spatial inputs, MLAT and MLT, are in a reference frame that does
not rotate with the earth. The local time has the range of 0 to 24 hours and
has 12 noon facing the sun.

###  III-B Data Challenges

Auroral particle precipitation and space weather create numerous data
challenges that advance to novel ML methods:

  * •

Target Transformation: The electron energy fluxes [eV/cm2/s] vary over six
orders of magnitude and require a log transformation.

  * •

Unrealistic fluxes: We remove target values that are deemed non-physical due
to magnitude (justified by physical understanding of auroral particle
precipitation). These were determined to be those exceeding the 99.995th
percentile $(>7.3710^{13}eV/{cm}^{2}/sr/s)$.

  * •

Imbalanced data: In space weather, like with terrestrial weather, the active
periods are much less frequent than quiet times. Large fluxes tend to occur
during storms, so our target variable exhibits a large right “tail”.

  * •

Multimodal target variable distribution: Target points composed of multiple
“modes” of skewed Gaussian-like distributions (see Figure 2).

The one-sec. resolution target data is available but currently too large for
training efficiently. Our PrecipNet-based models are able to train using the
full one-min. resolution DMSP data in the order of 10 min. with a 12 GB Nvidia
2080Ti GPU. The Conv2D model trains in a few hours. Preparing the combined
dataset, cleaned OMNI (independent variables) and DMSP (dependent variables),
is the most computationally inefficient process, demanding 128 GB of data to
be downloaded and filtered to a lower 1-min Cadence, as well as the removal of
outliers, and the calculation of moving averages of the input variables and
derived quantities.

Figure 2: Showing scaled comparisons of training and validation data
histograms. Note the smoothness of the training distribution due to more data
points and the difference in the left peak or most common values of the
target.

When viewing a histogram of the logarithm of the target flux, Figure 2,
multiple modes in the distribution are observed. There is evidence that these
modes correspond to unique auroral spatial regions: sub-auroral (below the
auroral oval), auroral, and in the polar cap (poleward of the auroral oval).
The validation data distribution is also shown for reference, and its
differing low flux mode (for the unique year 2010) poses a modeling challenge.

###  III-C Modeling Challenges

This research responds to several modeling challenges:

  * •

Deconvolving spatial and temporal responses to inputs

  * •

Extreme event predictions of rare space weather storms

  * •

Evaluating 2D spatial performance of model

  * •

Designing loss functions / metrics prioritizing large fluxes

  * •

Predicting mesoscale variations and structures

Our models need to deconvolve temporal and spatial responses as there is only
one time-step for each 2D data sample’s position in the orbit, at times there
are no more than 3 satellites in orbit at once.

##  IV Model descriptions

###  IV-A Baseline Model

We use PrecipNet[14] as our baseline model against which to demonstrate
improvement. The architecture of PrecipNet is detailed in Figure 3.

Figure 3: PrecipNet Baseline Model

Each model we present builds from this baseline architecture. Three updates
are tested: application of custom loss functions, use of a final layer for
sequential classification and regression (multi-task model), and addition of a
final 2D inverse/transposed convolutional layer (the ‘Conv2D’ model). This
baseline model uses the mean square error (MSE) loss and therefore has been
optimized to have the lowest MSE over all training data. However, using MSE on
imbalanced data leads to sub-optimal accuracy for predicting rare high flux
events. Using the log 10 scale transformation helps with balancing the data
but not enough.

###  IV-B Multitask Auroral Region Model

To separate the physically distinct processes that produce the multi-modal
nature of the auroral flux distribution, we developed a ’multi-task’ auroral
boundary model that first classifies by auroral region and then regresses the
flux value in each region. We chose a three-class classification task to
separate sub-auroral, auroral, and polar preciptitation regions that are known
to have distinct precipitation characteristics. The 2010-2014 subset (12
satellite years) of the DMSP data includes observations of the auroral
boundary transition regions based on spatial gradients of the electron flux
[18]. We used these transitions to create a classification model of the
auroral regions, one representative result is shown in Figure 4. The result of
this region estimation model can be fed into the regression model for electron
flux as an additional input or used to separate the data to train three
models. It was also used as a classification result for training and
validation model, as was the case of the final variation.

Figure 4: Auroral region classification model result for the northern
hemisphere and top down view of the earth centered at the magnetic north pole
from 45 to 90 degrees MLAT and the sun off to the top of the figure, showing
the high activity ’red’ auroral region, ’yellow’ moderate activity polar
region, and a ’black’ low activity equatorial region.

Using the auroral region classifier (90% accurate classification over 45 to 90
degree MLat paths), as one experiment, the data was separated and three
separate models were trained on each region. The table in Figure 4 show that
the MSE for the polar region (yellow) is the largest and the (black)
Equatorial the lowest. This shows that the model is most accurate at
predicting the black, low-activity sub-auroral region. Surprisingly, the
auroral region (red) (which contains the most dynamic precipitation) is only
20% higher in error while the polar region error is 50% higher. This means
that the model is more accurate at representing the flux near the outer (lower
MLat) auroral boundary than the inner. This could also mean that it is
possible that there is not enough input information to describe the polar
region flux.

Combining MSE and categorical cross-entropy: Our final multi-task model
predicts the auroral region type and the total electron energy flux
simultaneously with a loss function that has two penalty terms: categorical
cross entropy and MSE. This model has two output branches, one side with three
softmax values for predicting the class, and the other side which predicts the
flux quantity for each region. Note that in the loss function, only the
model’s predicted region value (one of the three regression values) is used in
the MSE contribution. These models were inspired from the following blog post:
Anchors and multi-bin loss for multi-modal target regression.

$\displaystyle\textrm{Loss}=$
$\displaystyle\textrm{Mean}\big{(}(y_{\textrm{flux-true}}-y_{\textrm{flux-
pred}})^{2}\big{)}$ (1)
$\displaystyle+\;\textrm{C.C.L.}\big{(}y_{\textrm{class-
true}},y_{c\textrm{class-pred}}\big{)}$

where C.C.L. represents the categorical crossentropy loss (as defined by the
Tensorflow Keras library).

###  IV-C Custom Tail Loss Model

In order to better capture the extreme values we adopt a similar approach to
[19] where a logic-based function only activates in the tail regions of an
unbalanced distribution of training data. The result is a “tail-weighted loss”
function which is useful to capture extreme values of a distribution.

$\displaystyle\textrm{Loss}=$
$\displaystyle{(y_{\textrm{true}}-y_{\textrm{pred}}})^{2}(1+a)\;$ (2)
$\displaystyle\mathbf{\textrm{for}}\;y_{\textrm{true}}>y_{r}\;\mathbf{\&}\;y_{\textrm{pred}}<y_{r}$

Our variant of this loss function in [19] was found to increase the accuracy
in the tail region without significantly decreasing the accuracy in the lower
flux region, demonstrated in Table 9. For example a 26% decrease in error is
observed in the 90th precentile, while the accuracy at the upper half of the
50th percentile reveals no loss of accuracy. This function is a sum of the MSE
and a special term which is only $>$ 0 in particular circumstances where for a
threshold, the $y_{\textrm{true}}$ value is above and $y_{\textrm{pred}}$ is
below. This additional term has the effect of penalizing the neural network
when a true high value has a prediction below a predetermined threshold. If
the true value is also below this threshold or if both the true and predicted
value are over the threshold, then there is no penalty. This threshold $y_{r}$
is chosen, where $y$ is the target and $r$ signifies the “right side” of the
tail. This value is within the tail region on the right side of the histogram.
When this condition is true, an additional loss penalty is added to the loss,
proportional by the factor “$a$” to the MSE,
$(y_{\textrm{true}}-y_{\textrm{pred}})^{2}$. In our Tail Loss results,
empirical exploration led to the use of a sum of multiple $a$ and $y_{r}$
terms which we selected from points on the tail of the training distribution
that were underrepresented by the original PrecipNet with just MSE loss. The
results in this work used five $(a,y_{r})_{i}$ pairs of (2.5,12), (5,12.5),
(10,13), (10,13.25), (10,13.5), where the units of $y_{r}$ are log10
[$eV/{cm}^{2}/ster/s$]. The values of ’$a$’ were found empirically and the
target values where selected by seeing where on the right tail of the
predicted training distribution (shown in Figure 2) the target range was
insufficiently represented.

###  IV-D Distribution Re-balancing Loss Model

The Tail Loss model required empirically determining the ’$a$’ and ’$y_{r}$’
values. With a different approach we instead use sample weighting based on a
distribution of the sample targets. We take the inverse of the frequency of
occurrence (in specific bins) of the testing target data to specify the sample
weights. Those weights are multiplied by the MSE contribution to the loss:

$\displaystyle\textrm{bins}=\min(\log_{10}(y_{\textrm{train}}),...,\max(\log_{10}(y_{\textrm{train}}),$
(3)
$\displaystyle\textrm{weight}_{n}=\frac{1}{[\textrm{hist}(\textrm{bin}^{i}_{n})+1]\cdot
n_{\textrm{total}}}$

A 50-bin histogram used to construct the distribution-based loss is shown in
Figure 5. The Dist Loss approach rebalances the data in a similar fashion to
that of classification models. The analogy is that each bin is viewed as a
class even though we are not using categorical loss. This approach is also
applicable to any left or right skewed distribution.

Figure 5: Histogram and weights: Each sample is weighted inversely by the
bin’s value in which it resides

###  IV-E Inverse Two-Dimensional Convolutional Model

All of the models discussed thus far predict one spatial value per one model
evaluation. This is advantageous for keeping the model simple and allows for
fast training and testing of specific spacial target points. It trains the
model to learn 2D features, but in a way that it is indirect and hidden from
the model designer. To create a global map from these models we must evaluate
them over all desired MLAT-MLT grid points. There is evidence that a 2D
convolutional and/or inverse convolutional approach (a review in McCann et
al.[15] ) can provide significant improvement regarding revealing 2D
structures and allowing more complex features to emerge. However in our case,
only a sparse spatial representation of the MLAT-MLT “image pixels” are
available. To create 2D training samples, we combine all concurrent (up to
three in orbit on any given year) DMSP satellite data values at each
measurement time step. Our previous models only use one satellite data point
at a time and do not couple the same time step data. The input data time
cadence is five min. and the target data are available at one-sec.
resolution.Therefore, it is an accurate assumption to assume that all spatial
locations of data $\pm$ 2.5 min. are at the ”same” five-min. training-sample
time step. With these assumptions, and a discretization of the MLAT-MLT space
by a 128 $\times$ 128 grid, multiple pixel traces of the satellites’ paths
create known target data points.

In the model training we use the custom loss function from Eq 4, where $t_{i}$
denotes a time range. The square error value of those points for which
$y_{true}$ exists in the time range are only used in the summations. The
visual in Figure 6 shows a situation where DMSP F16 and F17 both provide
multiple points in colored MLAT-MLT bins

$\textrm{Loss}=\sum_{t}\sum_{y_{\textrm{true}}(t_{i})\exists}({y_{\textrm{true}}(t_{i})-y_{\textrm{pred}}(t_{i})})^{2}$
(4)

Figure 6: Showing a possible loss function configuration of a 2D training
target at a single group of combined timestamps when measured data exists for
two DMSP satellites. Only the non-zero values of $y_{\textrm{true}}$ within a
$\pm$ 2.5 min. window are used in the loss evaluation.

Our network architecture’s layers are depicted in Figure 7, where it is shown
that for the first half of the model, the original PrecipNet layers are used
up to the 256 node dense layer. Instead of using the 1024 node layer, two
inverse convolutional layers are used (tf.keras.Conv2D_transpose). These
function as normal Conv2D layers, but in reverse, and are therefore also
referred to as Con2D_transpose layers. After these layers a dropout layer and
a normal Conv2D follow, where the periodic boundary condition of the MLT grids
is applied with an arbitrary 3 cell overlap. The final Conv2D layer creates
the 128x128 sized output. More that 128x128 resolution is possible, with
training time longer than a day.

Figure 7: Conv2D Model that takes in no spatial information and predicts the
entire northern hemisphere at once. Spatial information is used to select the
sparse cells in the 128x128 grid that are used in the loss function.

##  V Model Comparisons

We evaluate the qualitative characteristics exposed by combining different
model architectures and loss functions.

###  V-A Custom Loss function improvements

MSE regression loss causes large inaccuracies in the high flux tail of the
distribution. Therefore, the capability of the Tail Loss model must be
measured by improvements in this extreme tail region. In the C) and D) plots
of Figure 8 the Tail Loss function term significantly decreased prediction
error for flux values greater than 12 log10 [$eV/{cm}^{2}/ster/s$], a result
of 26% lower as shown in Table 9. Note that transforming out of the log scale,
this error difference changes from 250 to 1000% error. The significance of the
accuracy improvement is better visualized by looking at the non-transformed
target value histogram (Figure 8A and B), where the model predictions (green)
drastically improve the tail specification. Note that this does not come at
the expense of the lower target values at which accuracy is maintained.

Figure 8: Histograms and errors over ”bin-averaged” validation errors. Notice
that after a value of 11 log10 [$eV/{cm}^{2}/ster/s$] the Tail Loss model has
increased accuracy for modeling the physical processes within the the auroral
region during space weather storms.

Figure 9: Showing the reduction in error over the right tail region for the
90th, 95th, and 99th percentile, representing the solar storm events to be
predicted.

This improvement is even more significant in the 95th and 99th, which
represent the most extreme and potentially most impactful space weather
storms. This Tail Loss method is a significant improvement to the more
commonly known weighted loss which shifts all predictions to the left or right
equally.

###  V-B Distribution-based loss improvements

In Figure 10, the distribution loss model is compared to the Tail Loss model.
The distribution loss (Dist Loss) considers the imbalance of data across all
target data ranges (not just the right-side extreme values), and produces a
corresponding reduction in error at both sides of the distribution. Compared
to the PrecipNet baseline model, the right tail distribution and errors are
improved by both loss functions, however the Tail Loss model achieves the
largest overall accuracy improvement. The Dist Loss results improve over
PrecipNet at a log10 target value of 12 and greater, however, the Tail Loss
results improve beginning at an order of magnitude lower flux magnitudes
(log10 value of 11). It is possible that because the fact that the Dist Loss
model is learning both the left and right tails (or more specifically
imbalanced data as a whole), the right tail learning is suboptimal. The Dist
Loss could be further optimized by customizing the total number of bins used,
the applicable bin range for the weights, and the minimum bin size. These
optimizations along with the use of non-uniform bin sizes and inclusion of the
logic-based approach are the focus of ongoing work.

Figure 10: Distribution-loss and tail-loss model histograms and errors.

###  V-C Two-Dimensional Inverse Convolutional Model Results

The Conv2D model has two major potential enhancements: the ability to couple
multiple 2D points at one time step and the ability to directly keep track of
each training sample on the 2D hemisphere. This model treats the lower and
higher MLat values differently because of the ability of the convolutional
kernels (the filters) to keep track of the MLat-MLT regions over which they
are applied. The result is a small but noticeable improvement in the upper
tail region as well as a lowering and widening of the main left peak as shown
in Figure 11 . We also tested this approach with the Tail Loss and Dist Loss
functions, however, Figure 11 shows that the custom loss functions fail to
improve the 2D Conv model. This may be due to the sensitivity of the discrete
kernel-based 2D model to the interaction of weights. For example, the model
places emphasis on the spatially-gridded locations of samples, potentially
counteracting focus placed on samples from the tail of the distribution. These
results also suggest that the sample weights of the inverse 2D Conv model are
also imbalanced spatially with different target ranges, which could be
accounted for in the design of the loss function.

Figure 11: 2D inverse Conv2D result histograms.

###  V-D Auroral Activity Northern Hemisphere Comparisons

All models are capable of predicting the full northern hemisphere and this
prediction is an important component of evaluation [14]. We create these maps
for all models presented in this work in Figures 12, 13, and 14 for time steps
that occur during distinct space weather activity (January 25, 2000 18:30 UT,
January 25th, 2010 18:30, and March 18, 2010 at 02:00 UT). Northern hemisphere
predictions are shown and compared to imagery data from the Polar spacecraft.
The Polar visible imaging assembly (VIS instrument) provides auroral imagery
data that are a qualitative validation dataset, of sorts, over which to
compare morphologies of the models, shown in Figure 12. On each of the global
map plots we also compare to the ‘state-of-the-practice’ model, OVATION. This,
in general, over-predicts the auroral boundaries as well as over-predicting
the low range of the fluxes near the sub-auroral regions. Comparing all
models, the Tail Loss model without the Conv2D variation performs the best in
the auroral region, particularly with obtaining the peak flux predictions in
the auroral region while preserving the auroral region shape. However, the
Conv2d models predict finer scale spatial features and qualitative comparison
to Polar VIS indicates that these are likely physical. We further investigate
their physical validity by plotting time series data available at these times
for the events in 2010 (Figures 13 and 14 bottom plots). These time series
plots are from the F16 satellite path at which measurements were made every
second. In Figure 13, there are two flux peaks around (log10) magnitude 11 to
12 and these correspond to passing through each side of the auroral oval. Our
models show less variation than the one-sec. data (demonstrative of
”mesoscale” structures) because only 60-second cadence data was used for
training and validation. Also the Conv2D model results for the time series
appear to have ”steps” and this is due to the fixed spatial resolution of
128x128 for the Conv2D model output.

Figure 12: Solar Storm Comparison: Polar VIS experiment results for 1/25/2000
at 18:30. There is no DMSP measurement data at this time step, The upper right
plot is blank to serve as visual reference for the labels of the polar plots.
The red and yellow colors of the Polar VIS data are from radiance of photons
on the day side, not from collisions of charged particles with the atmosphere,
and therefore only the black and purple colors are to be compared.

We cannot yet verify the accuracy of these Conv2D model spatial variations
over the full 2D predictions, but the variations are physically and
qualitatively realistic. Future work will be to use additional space-based
imagery data (such as additional Polar VIS data and DMSP SSUSI images) to
verify the complete maps.

Figure 13: Low activity: 1/25/2010 at 18:30. The validation data are the
60-second cadence subset (black dots on 2D Baseline), used for training,
stopping criterion, and MSE error evaluation on 2010 F16 validation data.

In Figure 12, a qualitative comparison with Polar VIS is shown, however,
because it only shows strong fluxes, it is seen that all models improve in the
auroral oval. Even though these Polar VIS qualitative comparisons are
possible, there are additional complications to do quantitative comparisons,
and indeed direct apples-to-apples quantitative comparisons is not possible.
In terms of predicting the finer spatial scale phenomenon, the one-sec. data
measurement cadence of DMSP is useful, however, these data has a seemingly
random component that is difficult to deconvolve from a 1D spatio-temporal
trace into a 2D spatial map. With the input cadence having a five-min.
cadence, spatial variations will be more reasonable to model than temporal
ones. Future refinements of our models will continue to focus on the non-
random spatial phenomenon shown in the one-sec. traces.

##  VI Conclusions

Increasing our ML model expressive capacity yields improved predictions of
extreme events and reveals smaller scale phenomenon. A custom Tail Loss
function improves accuracy in the 90th percentile of the right tail of the
distribution by 26% (log10 scale) on average without sacrificing accuracy at
lower values of the distribution. Additionally, we lay the foundation for a
novel convolutional approach that combines data from multiple simultaneous
satellite measurements and utilizes the power of Conv2D kernels. Our approach
is new and we lay out possibilities for data, loss function, and modeling
approaches. A simple but computationally-expensive approach for improving
smaller spatial scale identification is running at higher resolution with the
Conv2D model and using one-sec. resolution target data as we are currently
training this model with 60-sec. sub-sampled data samples. The Conv2D
modelling approach is already showing new smaller-scale spatial predictions,
the accuracy of which could improve with more kernels, layers, and grid
resolution.

Figure 14: High activity: 3/18/2010 at 2:00. The validation data are the
60-sec. cadence subset (black dots on 2D Baseline), used for training,
stopping criterion, and MSE error evaluation on 2010 F16 validation data.

## Acknowledgment

We thank the NASA OMNI initiative, and those who provided the Defense
Meteorological Satellite Program (DMSP) data over the several decade lifetime
of these spacecraft.

## References

  * [1] C. J. Schrijver _et al._ , “Understanding space weather to shield society: A global road map for 2015–2025 commissioned by cospar and ilws,” _Advances in Space Research_ , vol. 55, no. 12, pp. 2745 – 2807, 2015.
  * [2] L. Scherliess, I. Tsagouri, E. Yizengaw, S. Bruinsma, J. S. Shim, A. Coster, and J. M. Retterer, “The international community coordinated modeling center space weather modeling capabilities assessment: Overview of ionosphere/thermosphere activities,” _Space Weather_ , vol. 17, no. 4, pp. 527–538, 2019.
  * [3] R. McGranaghan _et al._ , “Modes of high-latitude auroral conductance variability derived from dmsp energetic electron precipitation observations: Empirical orthogonal function analysis,” _Journal of Geophysical Research: Space Physics_ , vol. 120, no. 12, pp. 11,013–11,031, 2015.
  * [4] R. M. McGranaghan, “Determining global ionospheric conductivity in the satellite and data assimilation age and assessing its influence on the magnetosphere-ionosphere-thermosphere system,” phd thesis, University of Colorado Boulder, 2016.
  * [5] P. T. Newell, T. Sotirelis, and S. Wing, “Diffuse, monoenergetic, and broadband aurora: The global precipitation budget,” _Journal of Geophysical Research: Space Physics_ , vol. 114, no. A9, 2009.
  * [6] J. L. Machol, J. C. Green, R. J. Redmon, R. A. Viereck, and P. T. Newell, “Evaluation of OVATION Prime as a forecast model for visible aurorae,” _Space Weather_ , vol. 10, no. 3, 2012.
  * [7] P. T. Newell, K. Liou, Y. Zhang, T. S. Sotirelis, L. J. Paxton, and E. J. , _Auroral Precipitation Models and Space Weather_. American Geophysical Union (AGU), 2015, ch. 18, pp. 275–290.
  * [8] D. A. Hardy, M. S. Gussenhoven, and E. Holeman, “A statistical model of auroral electron precipitation,” _Journal of Geophysical Research: Space Physics_ , vol. 90, no. A5, pp. 4229–4248, 1985.
  * [9] P. T. Newell, T. Sotirelis, J. M. Ruohoniemi, J. F. Carbary, K. Liou, J. P. Skura, C.-I. Meng, C. Deehr, D. Wilkinson, and F. J. Rich, “OVATION: Oval variation, assessment, tracking, intensity, and online nowcasting,” _Annales Geophysicae_ , vol. 20, no. 7, pp. 1039–1047, 2002.
  * [10] P. T. Newell, T. Sotirelis, and S. Wing, “Seasonal variations in diffuse, monoenergetic, and broadband aurora,” _Journal of Geophysical Research: Space Physics_ , vol. 115, no. A3, 2010.
  * [11] P. T. Newell, T. Sotirelis, K. Liou, A. R. Lee, S. Wing, J. Green, and R. Redmon, “Predictive ability of four auroral precipitation models as evaluated using polar uvi global images,” _Space Weather_ , vol. 8, no. 12, 2010.
  * [12] P. T. Newell, A. R. Lee, K. Liou, S.-I. Ohtani, T. Sotirelis, and S. Wing, “Substorm cycle dependence of various types of aurora,” _Journal of Geophysical Research: Space Physics_ , vol. 115, no. A9, 2010.
  * [13] J. W. Gjerloev, C. L. Waters, and R. J. Barnes, “Deriving global convection maps from SuperDARN measurements,” _Journal of Geophysical Research: Space Physics_ , vol. 123, no. 4, pp. 2902–2915, 2018.
  * [14] R. M. McGranaghan, J. Ziegler _et al._ , “Toward a next generation particle precipitation model: Mesoscale prediction through machine learning (a case study and framework for progress),” _Space Weather_ , vol. 19, no. 6, 2021.
  * [15] M. T. McCann, K. H. Jin, and M. Unser, “Convolutional neural networks for inverse problems in imaging: A review,” _IEEE Signal Processing Magazine_ , vol. 34, no. 6, p. 85–95, Nov 2017.
  * [16] P. T. Newell, T. Sotirelis, K. Liou, C.-I. Meng, and F. J. Rich, “A nearly universal solar wind-magnetosphere coupling function inferred from 10 magnetospheric state variables,” _Journal of Geophysical Research: Space Physics_ , vol. 112, no. A1, 2007.
  * [17] J. E. Borovsky and J. A. Valdivia, “The earth’s magnetosphere: A systems science overview and assessment,” _Surveys in Geophysics_ , vol. 39, no. 5, p. 817–859, 2018.
  * [18] L. M. Kilcommons _et al._ , “A new dmsp magnetometer and auroral boundary data set and estimates of field-aligned currents in dynamic auroral boundary coordinates,” _Journal of Geophysical Research: Space Physics_ , vol. 122, no. 8, pp. 9068–9079, 2017.
  * [19] D. Qi and A. J. Majda, “Using machine learning to predict extreme events in complex systems,” _Proceedings of the National Academy of Sciences_ , vol. 117, no. 1, pp. 52–59, 2020.


# Passing the Brazilian OAB Exam: data preparation and some
experiments††thanks: The authors would like to thank João Alberto de Oliveira
Lima for introducing us to the LexML resources, and Peter Bryant for his
careful review of the article.

Pedro Delfino FGV Direito Rio and FGV/EMAp Bruno Cuconato FGV/EMAp Edward
Hermann Haeusler PUC-Rio Alexandre Rademaker IBM Research and FGV/EMAp

###### Abstract

In Brazil, all legal professionals must demonstrate their knowledge of the law
and its application by passing the OAB exams, the national bar exams. The OAB
exams therefore provide an excellent benchmark for the performance of legal
information systems since passing the exam would arguably signal that the
system has acquired capacity of legal reasoning comparable to that of a human
lawyer. This article describes the construction of a new data set and some
preliminary experiments on it, treating the problem of finding the
justification for the answers to questions. The results provide a baseline
performance measure against which to evaluate future improvements. We discuss
the reasons to the poor performance and propose next steps.

##  1 Introduction

The “Ordem dos Advogados do Brasil” (OAB) is the professional body of lawyers
in Brazil. Among other responsibilities, the institution is responsible for
the regulation of the legal profession in the Brazilian jurisdiction. One of
the key ways of regulating the legal practice is through the “Exame Unificado
da OAB” (unified Bar examination). Only those who have been approved on this
exam are allowed to work as practising attorneys in the country. In this way,
the OAB exam is similar to the US Bar Exam.

Thus, the OAB exam provides an excellent benchmark for the performance of a
system attempting to reason about the law. That is, passing the OAB exam would
signal that the system has acquired the most important aspects of legal
knowledge, up to a level comparable to a human lawyer.

This paper reports our effort in the construction of the data set and some
preliminary experiments with the collected data. We obtained the official data
from previous exams and their answer keys from http://oab.fgv.br/. As our
first contribution, we collected the PDF files, extracted the text from them,
and cleaned up the results, producing machine-readable data. 111All data files
are freely available at http://github.com/own-pt/oab-exams. In Section 2 we
describe the data.

Question answering is an automatic way of determining the right answer to a
question presented in natural language form [12]. An ideal legal question
answering system would take a question in natural language and a corpus of all
legal documents in a given jurisdiction, and would return both a correct
answer and its legal foundation, i.e., which sections of which norms provide
support for the answer. Since such a system is far from our current
capabilities, we started with a simpler task.

In [7] the authors report a textual entailment study on this US Bar exam
material. In the experiment, the authors treat the relationship between the
question and the multiple-choice answers as a form of textual entailment.
Answering a multiple choice legal exam is a more feasible challenge, although
it is still a daunting project without restrictions on the input form, such as
preprocessing natural language questions to make them more intelligible to the
computer (see Section 2) or restricting the legal domain. That is the reason
we have chosen to restrict the domain to a single section of the OAB exams:
Ethics, one which is governed by only a few legal norms.

We have conducted three experiments in question answering (Section 4). In the
first experiment, we tried to find the right answer between the multiple-
choice alternatives. The last two were in shallow question answering (SQA), a
form of question answering where a system does not provide answers directly,
but rather retrieves documents that justify the already provided answer. We
have adapted the methodology described in [13, 14] to answer multiple-choice
exams instead of closed-ended answers.

To be able to provide the right justification (the article of a law) for each
question, we needed to have the text of the laws available. This is a
particular challenge in the legal domain, as normative instruments are not
readily available in a uniform format, suitable for being consumed by a
computer program. Fortunately, using resources provided by the LexML Brasil
project, we were able to collect and convert to XML format all the normative
documents we needed, which were initially provided in HTML and PDF (Section
3).

##  2 The OAB Exams

Before 2010, Brazilian bar exams were regional – each state in the country had
its own exam. Only in 2010 were the exams nationally unified. In order to be
approved, candidates need to be approved in two stages. The first phase
consists of multiple choice questions. The second phase involves free-text
questions. The first phase has 80 multiple choice questions and each question
has 4 options, ranging from letters A to D. Candidates are asked to choose the
correct alternative and in order to be approved, candidates need at least a
50% performance.

Every year, there are 3 applications of the exam in the country, which are
currently being conducted by FGV 222http://fgvprojetos.fgv.br/node/135.
Concerning exam statistics, the first phase is responsible for eliminating the
majority of the candidates. Historically, the exam has a global 80% failure
rate. The most recent exam, whose multiple choice stage was held on July 2017,
had the highest failure rate: 86% of the candidates failed.

Since 2012, the exams have revealed a pattern for which areas of Law the
examination board focuses on and in which order the questions appear on the
exam. Traditionally, the first 10 questions are about Ethics. Despite the
name, this area is not about classic texts on morals from Aristotle or
Socrates. In the context of the exam Ethics means questions about the rights,
the duties and the responsibilities of the lawyer. This is the simplest part
of the exam with respect to the legal foundation of the questions. Almost all
the questions on Ethics are based on the Federal law 8906 from 1994, which is
a relatively short (89 articles) and well designed normative document. A minor
part of the questions on Ethics is related to two other norms: (i)
“Regulamento Geral da OAB” (OAB General Regulation, 169 articles) and (ii)
“Código de Ética da OAB” (OAB ethics code, 66 articles). It is important to
note that these two norms are neither legislative nor executive norms. Indeed,
they are norms created by OAB itself. OAB’s prerogative to do so is assured by
8906 federal law.

Beyond the first 10 questions about Ethics, the remaining questions appear in
the order of areas presented in Table 1, starting with Ethics and ending with
taxes. The committee responsible for the exam may change this pattern at any
time, since, in this aspect, they are not bound by the edict of the exam.

area | # | (%) | area | # | (%)
---|---|---|---|---|---
Ethics | 10 | 65 | Constitutional Law | 7 | 42
Consumer’s Law | 2 | 56 | Civil Procedures | 6 | 40
Children’s Law | 2 | 54 | Philosophy | 2 | 40
Criminal Procedures | 5 | 47 | Labor’s Law Proc. | 6 | 40
Regulatory Law | 6 | 47 | Criminal Law | 6 | 38
Human Rights | 3 | 47 | International Law | 2 | 37
Civil Law | 7 | 44 | Business Law | 5 | 33
Environmental | 2 | 43 | Taxes | 4 | 42
Labor’s Law | 5 | 42 |  |  |
Table 1: Questions per subject area and their performance rates

In addition to the high frequency rate, the Ethics questions also have a high
performance rate among candidates, while Business Law has the lowest
performance rate. In the third column of Table 1, we reproduce some
descriptive statistics from [1]. This column shows the average performance on
the different areas covered by the exam.

We obtained the previous exam files from the FGV website
333http://oab.fgv.br/.. After downloading exam files, we converted them from
PDF to text using Apache Tika 444https://tika.apache.org/.. Finally, we
manually revised the obtained text, and introduced markup to signal the
beginning of questions, alternatives and some meta data about them. We have
chosen a simple set of markup to facilitate the final parser that converts the
exam files to XML. This data cleanup was not easy, as it was necessary to deal
with errors in the encoding of some files, missing words, hyphenated words and
to remove irrelevant parts like footnotes, the instructions page etc. In order
to make this research reproducible, all the data is available at our public
repository on GitHub. 555http://www.github.com/own-pt/oab-exams/.

The final data comprises 22 exams totaling 1820 questions. A range of issues
on the texts of the questions of the exams was identified. Many of the
problems are similar to the ones found in the BAR exams and described by [7].
For instance, some questions do not contain an introductory paragraph defining
a context situation for the question. Instead of that, they have only meta
comments (e.g. “assume that…” and “which of the following alternative is
correct?”) followed by the choices. Some questions are in a negative form,
asking the examinee to select the wrong option or providing a statement in the
negative form such as “The collective security order cannot be filed by…”.
Moreover, some questions explicitly mention the law under consideration,
others do not. Many questions present a sentence fragment and ask for the best
complement among the alternatives, also exposed as incomplete sentences.
Nevertheless, we identify some similarities between questions of the same
subject area.

We sampled 30 questions on Ethics for analysis (from the 210 questions in all
exams) and one of the authors manually identified the articles in the laws
that justify the answer, creating our golden data set. The key finding was
that, usually, one article on the federal law 8906 was enough to justify the
answer to the questions (15 questions). Less often, the justification was not
in the law 8906, but rather in OAB General Regulation (3 questions), or on the
OAB Ethics Code (8 questions). Three questions were justified by two articles
in law 8906, and another in jurisprudence from the Superior Court of Justice
about an article from the law 8906.

##  3 The LexML Brasil project

For the experiments reported in Section 5 and the further experiments that we
plan to conduct, we needed three norms in a machine readable format: Federal
law 8906 from 1994, the “Regulamento Geral da OAB” (OAB general regulation)
and the “Código de Ética da OAB” (OAB ethics code). Moreover, we needed the
documents in a format that preserved the original internal structure of the
documents, i.e., the sections, articles, and paragraphs.

The LexML [4] is a joint initiative of the Civil Law legal system countries
seeking to establish open standards for the interchange, identification and
structuring of legislative and court information. One of the initial goals of
the initiative, later abandoned, was the standardization of a single document
format (called LexML) for encoding normative documents of all participating
countries. Currently only Brazil has developed a XML schema called “LexML
Brasil”. The remain participants migrated to Akoma Ntoso [15] and EUR-Lex
666http://eur-lex.europa.eu..

The LexML project has a public repository at https://github.com/lexml. As the
repository shows, the project is active but it does not have vast
documentation. One useful tool developed by the LexML Brazil team is a parser
of legal documents 777https://github.com/lexml/lexml-parser-projeto-lei, still
in beta. The software receives as input a DOCX file (a common format for
Brazilian legal documents) with the norm and outputs it in XML format, using
the tags and the structure according to the conventions of the LexML schema.
We had to make minor modifications in the three documents before submitting
them to the parser; the XML files produced are available in our repository.

##  4 The Experiments Setup

The original idea for the experiment is described in [13], and it runs as
follows: one collects the legal norms in a corpus and preprocesses them
performing tasks such as converting text to lower case, eliminating
punctuation and numbers and, optionally, removing stop-words. After that, the
articles of the norms are represented as TF-IDF vectors in a Vector Space
Model (VSM) [3].

In a VSM, every document $d$ is represented by a vector whose size is the
vocabulary size of the corpus $D$. When using TF-IDF weighting, the value of
each component $t$ of the vector corresponding to $d$ is given by Equation 1.

$\mathrm{TFIDF}_{t,d}=\frac{f_{t,d}}{\sum_{t^{\prime}\in
d}f_{t^{\prime},d}}\log\left(\frac{|D|}{|d\in D:t\in d|}\right)\\\ $ (1)

where $f_{t,d}$ is the number of times term $t$ appears in document $d$.

A directed graph is then created, with a node for each article of a norm in
the corpus. When provided a question-answer pair, our system preprocesses the
question statement and the alternatives in the same way as it does to the
articles in the base graph. It turns them into TF-IDF vectors using IDF values
from the document corpus.888This means that if a term occurs in the question
statement or alternative but not on the legal norm corpus, its IDF value is 0.
The statement node is connected to every article node, and each article node
is then connected to every alternative node. In this we differ from [13], as
we have no need for heuristic rules for splitting the question provided.

The edges are given weights whose value is the inverse cosine similarity
between the connected nodes’ TF-IDF vectors. The system then calculates the
shortest path between question statement and answer item using Dijkstra’s
algorithm, and returns the article that connects them as the answer
justification. Unlike [13] our graph structure does not allow for more than
one node connecting statement and alternative, as we knew from previous legal
analysis that questions were usually justified by a single article.

The intuition behind such a method is that the more similar two nodes are, the
lesser is the distance between them; as a document that answers a given query
is presupposed similar to the question, it makes sense to retrieve the article
in the shortest path between the statement and the alternative as a
justification for the answer.

Figure 1: Graph layout of the SQA system. If a $A$ is the number of article
nodes, we then have $5A$ edges, as we have one statement and four
alternatives.

##  5 The Results

We have taken the method described in Section 4 as a baseline for further
improvements, and we have conducted three experiments using it. Our golden
answer set was manually created by one of the authors (Section 2). It consists
of a table that includes 30 questions from eleven different editions of the
OAB exam. Each line of the table represents one question, with fields for the
year/edition of the exam where it appears, the question number, the URN
(uniform resource locator) of the legal norm and its article(s) which support
the answer.999This document is also available from our public repository.

In our first experiment we had an ambitious objective: we had our system
receive a question statement and its multiple alternatives, and we wanted it
to retrieve the right answer along with its justification in the legal norm.
When given the question and its alternatives, the system would add them to the
base graph composed by the respective legal norm’s articles, connecting them
to the article nodes and the choices, but not to each other. The system would
return the shortest paths between the question statement and its alternatives,
and the presumed justification would be the article connecting the statement
and the closest item to it. The system’s performance against this task was not
impressive: although it chose the correct alternative 10 times, it only
provided the correct justification for 8 of these.

Analyzing the system’s output paints a more nuanced picture, however. In some
cases, the system would find the correct justification article for the correct
answer, but would pick as its putative answer another (incorrect) item,
because it had a shorter path. Other times, it would not be capable of
deciding between two (or more) answer items, as they all had a shortest path
of the same distance. The following exam question is a sample case where this
statistical approach to question answering is defective:

> The young adults Rodrigo (30-year-old), and Bibiana (35-year-old), who are
> properly enrolled in an OAB section […] Considering the situation described,
> choose the correct alternative:
>
> A) Only Bibiana meets the eligibility criteria for the roles.
>
> B) Only Rodrigo meets the eligibility criteria for the roles.
>
> […]
>
> (2016 OAB exam, 19th edition, question 7)

As one can see, these two options differ by only one word (the names of the
fictional lawyers), and both are unlikely to be in the text of the legal
norms, which means that they do not affect the calculation of similarity. This
gives us the same distance between the question statement and these two answer
choices, and the system is incapable of choosing between them. A similar
situation arises when one answer item makes a statement and another item
denies this statement:

> [question statement]
>
> A) does not compel him to pay the agreed upon legal fees.
>
> B) does compel him to pay the agreed upon legal fees.
>
> […]
>
> (2015 OAB exam, 18th edition, question 1)

In a question like this a system can only systematically report a correct
answer if it has a higher-level understanding of the texts at hand: no bag-of-
words model will suffice. In Section 6 we begin exploring logical approaches
to question answering, which may offer a solution to this problem.

As our first experiment demonstrated that our simple system could not reliably
pick the correct answer among four alternatives, we turned our attention from
pure question answering to shallow question answering, where our system would
only have to provide the correct legal basis for the correct answer provided
along with the question.

In our second experiment we read the three XML files corresponding to the
norms in the Legal Ethics domain, and built separate base graphs from their
articles. For each question in our golden set, we added its statement and its
correct answer to the base graph built from the norm which justifies the
answer. The sole task of our system in this case is to the determine which
article from the norm it was given that justifies the answer. In this simpler
form, performance was not bad: the system retrieved the correct article in 21
out of 30 question-answer pairs.

In our third experiment, we tried to see if our system could provide the
correct article (from the appropriate legal norm) without us telling it at
which norm it should look. Following this idea, we have taken the articles
from all norms and built a single base graph. For each question in our golden
set, we again added its statement and correct answer as nodes connected to all
article nodes in the graph (see Figure 1) , and then calculated the shortest
path between them to retrieve the system’s putative justification to the
question-answer pair. The system now had to retrieve the correct article among
articles from all norms – which, being in the same legal domain, had similar
wordings and topics – therefore increasing the difficulty of the task. Despite
this, its performance did not plunge: it scored the right article in 18 out of
the 30 question-answer pairs.

Although the results from the SQA experiments were not bad, they are not very
encouraging. We would like to have an autonomous system that could pass the
multiple-choice OAB exam, not an assistant system that could show us where the
answer keys are legally grounded.

##  6 A possible logic-based approach

One of the key observations that emerge from the results in Section 5 is the
importance of logical reasoning for our final goal of constructing a system to
pass the OAB exam with a full understanding of the questions and laws.

For the future, we aim to investigate how to enrich the data with lexical
information and syntactic dependencies as an intermediary step toward a
semantic representation of the questions and laws statements. Nevertheless, we
have to decide what should be an adequate logic language to represent laws and
the deep semantics from the text statements. Since the adequacy of a logic
language can be evaluated even before a procedure to obtain logic expressions
from natural language texts is developed, this section presents some
preliminary discussion about one possible logic, the $i\mathcal{ALC}$ logic,
which is a logic specifically developed to formalize laws and to reason
regarding them [9]. Here we sketch the ideas of how $i\mathcal{ALC}$ can be
used in the further experiments with the OAB exams.

In [9, 8] we discuss how Kelsen’s [11] pure theory of law points out a
framework that takes into account the legal knowledge forming a collection of
individual, legally valid statements. Thus, each legally valid statement may
be seen as an inhabitant among the many individual laws of the represented
legal system. The natural precedence existing between individual legal
statements can be taken as a pre-order relation on the legal statements. For
example, “Ana is liable” precedes “Ana signed a selling contract”, since in
order to be valid the parts involved in the contract must be liable.

The legal principle that rules the stability of the law implies that the
precedence of individual laws preserves properties (decisions, conditions of
applicability, adequate fora, etc) regarding them. From these jurisprudence
considerations we have that intuitionistic logic is the best choice for
reasoning on laws within a non-deontic approach. In [9, 8] it is shown, by
means of an example, how the intuitionistic negation is better than its
classic counterpart for reasoning with conflict of laws.

In the presence of this natural precedence order between legally valid
statements, the intuitionistic interpretation of subsumption between concepts
$A$ and $B$ ($A\sqsubseteq B$) reflects more adequately the structure of
existing legal systems than its classical interpretation counterpart. The
classical interpretation of $A\sqsubseteq B$ says that any legally valid
statement satisfying concept $A$ also satisfies concept $B$, no matter how
these concepts are related to the precedence relation. For example, in the
three tiers involved in the U.S. legal system (District court, Appellate court
and Supreme court) the legally valid statements are strongly related by
natural precedence order. If $E$ is taken to be a concept meaning
Environmental crimes and $G$ means Grave crimes, then $E\subseteq G$ may not
hold at all. A district court would not judge a situation of an oil-company
that spoiled a beach in town as a grave crime. Note that at the federal level,
the spoiling of natural resources is a grave crime. When considering the
intuitionistic interpretation the subsumption $E\subseteq G$ holds, for any
valid legal statement superseding “oil-company spoiled the coast and is
sentenced to provide a compensation to the county” is superseded by a federal,
valid legal statement classified as a grave crime. In the Supreme court, the
oil-company would be sentenced for committing a grave crime.

$\mathcal{ALC}$ [2] is the basic description logic $\mathcal{AL}$ (for
Attributive Language) plus the notion of complement $\mathcal{C}$ (negation)
of arbitrary concepts. $\mathcal{ALC}$ is a fragment of classical first-order
logic having only unary and binary relation symbols. $i\mathcal{ALC}$ is
closed related to many known intuitionistic modal and hybrid logics, as is
discussed in [10] and in [5].

To illustrate the use of $i\mathcal{ALC}$ for reasoning over the OAB exams
questions, let us consider the translated question below:

> Three friends graduated in a Law School in the same class: Luana, Leonardo,
> and Bruno. Luana, 35 years old, was already a manager in a bank when she
> graduated. Leonardo, 30 years, is mayor of the municipality of Pontal.
> Bruno, 28 years old, is a military policeman in the same municipality. The
> three want to practice law in the private sector. Considering the
> incompatibilities and impediments to practice, please select the correct
> answer.
>
>
> A) Luana is not prohibited from practicing law because she is an employee of
> a private institution, so there are no impediments or incompatibilities.
>
> B) Bruno, like all other civil servants, is only prohibited from practicing
> the law against the government agency that remunerates him.
>
> C) The three graduates, Luana, Leonardo, and Bruno, have functions
> incompatible with legal practice. They are therefore prohibited from
> exercising private practice. (CORRECT)
>
> D) Leonardo is banned from practicing law only against or in favor of legal
> entities from the public sector, public companies, mixed-capital companies,
> public foundations, parastatal entities or concessionaire corporations or
> public service licensees.
>
> (2016 OAB exam, 19th edition, question 4)

The justification of the answer to this question is obtained in the law 8906,
article 28. 101010The complete text of the law can be found at
http://bit.ly/29gZc83 The relevant fragments of this article, translated into
English, are:

> Legal practice is incompatible, even for self-defense, with the following
> activities:
>
> I - head of the Executive and members of the Bureau of the Legislative
> Branch and their legal substitutes; […]
>
> V - occupants of positions or functions linked directly or indirectly the
> police activity of any nature; […]
>
> VIII - occupiers of management positions in financial institutions,
> including private ones. […]

In $i\mathcal{ALC}$, the law 8906 is formalized as a concept defined as the
intersection of the concepts from its articles, that is, $Law_{8906}\equiv
Art_{1}\sqcap\ldots\sqcap Art_{28}\sqcap\ldots\sqcap Art_{87}$. Article 28 in
turn is also further formalized as the intersection of the concepts from its
paragraphs, $Art_{28}\equiv P_{1}\sqcap P_{2}\ldots$. The paragraph VIII is
formalized by the two concepts $Lawyer\sqsubseteq\neg Financial$ and
$Financial\sqsubseteq\neg Lawyer$. Paragraph V is formalized by
$Lawyer\sqsubseteq\neg Police$ and $Police\sqsubseteq Lawyer$. Finally,
paragraph I by $Lawyer\sqsubseteq\neg ChiefCouncil$ and
$ChiefCouncil\sqsubseteq\neg Lawyer$. The $Lawyer$ concept can be read as the
set of valid legal statements (VLS) about lawyers. That is, each concept can
be thought as the set of VLSs where it holds.

From the statements of the question, we have the hypotheses $lual\colon
Laywer$ (Luana acts as lawyer), $leoal\colon Lawyer$ (Leonardo acts as lawyer)
and $bal\colon Lawyer$ (Bruno acts as lawyer). Using the deductive system for
$i\mathcal{ALC}$ first presented in [8], we can prove that Luana, Bruno and
Leonardo can not act as lawyers. The proof below shows the Luana case, and
similar deductions can be constructed for the others.

$lual\colon Police$ $Police\sqsubseteq\neg Lawyer$ $lual\colon\neg Lawyer$
$[lual\colon Lawyer]$ $lual\colon\bot$ $\neg(lual\colon Lawyer)$

##  7 Conclusion and Future Works

We presented a new data set with all Brazilian bar exams and their answer keys
jointly with three Brazilian norms in LexML format. Furthermore, we also
presented some preliminary experiments with the goal of constructing a system
to pass in the OAB exam. We obtained reasonable results considering the
simplicity of the methods employed.

For the next steps, many other experiment setups can be tested. For instance,
we can construct the TF-IDF vectors using lemmas of the words, possibly
increasing the similarities. We can also add edges between articles,
considering that 10% of our golden set includes more than one article as
justification. We also plan to use the OpenWordnet-PT [6], properly expanded
with terms of the legal domain, following results from [17]. In another
direction, we need to increase the size of the golden set. Using crowd-
sourcing websites to obtain more justifications from humans is a possibility.

Many different proposals for data standards for the representation of laws are
available. Why has a single standard not been largely adopted yet? We aim to
explore the best candidate for encoding, possibly in collaboration with the
LexML team, the remaining normative documents that we will need to cover all
areas of the OAB exams.

Finally, the results of the experiments presented here clearly show that we
need ‘deep’ linguistic processing to capture the meaning of natural language
utterances in representations suitable for performing inferences. That will
require the use of a combination of linguistic and statistical processing
methods, possibly using results from [16]. The final objective is to obtain
formal representations, encoded in $i\mathcal{ALC}$ or another variant, from
the texts ready for formal reasoning.

## References

  * [1] Flávia Alfenas Amorim and Gabriel Dib Tebechrani Neto.  Exame da ordem em números.  Technical report, Fundação Getulio Vargas and Conselho Federal da Ordem dos Advogados do Brasil, 2016.  http://hdl.handle.net/10438/18493.
  * [2] Franz Baader, Diego Calvanese, Deborah McGuinness, Daniele Nardi, and Peter Patel-Schneider, editors.  The Description Logic Handbook: theory, implementation, and applications.  Cambridge University Press, 2003.
  * [3] D Manning Christopher, Raghavan Prabhakar, and Schütze Hinrich.  Introduction to information retrieval.  An Introduction To Information Retrieval, 151:177, 2008.
  * [4] João Alberto de Oliveira Lima and Fernando Ciciliati.  LexML brasil: versão 1.0.  available at http://projeto.lexml.gov.br/, December 2008.
  * [5] Valeria de Paiva, Edward Hermann Hausler, and Alexandre Rademaker.  Constructive description logic: Hybrid-style.  In Proc. HyLo’2010, 2010.
  * [6] Valeria de Paiva, Alexandre Rademaker, and Gerard de Melo.  OpenWordNet-PT: An Open Brazilian WordNet for Reasoning.  In Proceedings of 24th International Conference on Computational Linguistics, COLING (Demo Paper), 2012.
  * [7] Biralatei Fawei, Adam Z Wyner, and Jeff Pan.  Passing a USA national bar exam: a first corpus for experimentation.  In Language Resources and Evaluation, pages 3373–3378, 2016.
  * [8] Edward Hermann Haeusler, Valéria de Paiva, and Alexandre Rademaker.  Intuitionistic logic and legal ontologies.  In Proc. JURIX 2010, pages 155–158. IOS Press, 2010.
  * [9] Edward Hermann Haeusler, Valeria De Paiva, and Alexandre Rademaker.  Using intuitionistic logic as a basis for legal ontologies.  Informatica e Diritto, 1(2), 2010.  also published at Proceedings of the 4th Workshop on Legal Ontologies and Artificial Intelligence Techniques, Fiesole, Florence, Italy.
  * [10] Edward Hermann Haeusler and Alexandre Rademaker.  An intuitionisticaly based description logic.  Technical report, Arxiv, 2014.
  * [11] Hans Kelsen.  General theory of norms.  Oxford Univ. Press, USA, 1991.
  * [12] Ruslan Mitkov.  The Oxford handbook of computational linguistics.  Oxford University Press, 2005.
  * [13] Alfredo Monroy, Hiram Calvo, and Alexander Gelbukh.  Using graphs for shallow question answering on legal documents.  In MICAI 2008: Advances in Artificial Intelligence: 7th Mexican International Conference on Artificial Intelligence, Atizapán de Zaragoza, Mexico, October 27-31, 2008 Proceedings, pages 165–173. Springer Berlin Heidelberg, Berlin, Heidelberg, 2008.
  * [14] Alfredo Monroy, Hiram Calvo, and Alexander Gelbukh.  NLP for shallow question answering of legal documents using graphs.  Computational Linguistics and Intelligent Text Processing, pages 498–508, 2009.
  * [15] Monica Palmirani, Roger Sperberg, Grant Vergottini, and Fabio Vitali.  Akoma ntoso version 1.0 part 1: XML vocabulary.  Technical report, OASIS Committee Specificatio, 2015.
  * [16] Paulo Quaresma and Irene Rodrigues.  A question-answering system for portuguese juridical documents.  In 10th international conference on Artificial intelligence and law, pages 256–257. ACM, 2005.
  * [17] Maria Teresa Sagri, Daniela Tiscornia, and Francesca Bertagna.  Jur-WordNet.  In Petr Sojka, Karel Pala, Pavel Smrz, Christiane Fellbaum, and Piek Vossen, editors, Global Wordnet Conference, 2004.