2021

[3]Yang Yu

1]School of Computer Science, Fudan University, Shanghai, 200433, China

2]Department of Computer Science, Johns Hopkins University, 3910 Keswick Road,
Baltimore, 21211, MD, United States

3]Institute for Interdisciplinary Information Sciences, Tsinghua University,
Beijing, 610101, China

# Knowledge is Power: Understanding Causality Makes Legal Judgment Prediction
Models More Generalizable and Robust

Haotian Chen<EMAIL_ADDRESS>Lingwei Zhang<EMAIL_ADDRESS>Yiran Liu
<EMAIL_ADDRESS>Fanchao Chen<EMAIL_ADDRESS><EMAIL_ADDRESS>[  [  [

###### Abstract

Legal Judgment Prediction (LJP), aiming to predict a judgment based on fact
descriptions according to rule of law, serves as legal assistance to mitigate
the great work burden of limited legal practitioners. Most existing methods
apply various large-scale pre-trained language models (PLMs) finetuned in LJP
tasks to obtain consistent improvements. However, we discover the fact that
the state-of-the-art (SOTA) model makes judgment predictions according to
irrelevant (or non-casual) information. The violation of rule of law not only
weakens the robustness and generalization ability of models but also results
in severe social problems like discrimination. In this paper, we use causal
structural models (SCMs) to theoretically analyze how LJP models learn to make
decisions and why they can succeed in passing the traditional testing paradigm
without learning causality. According to our analysis, we provide two
solutions intervening on data and model by causality, respectively. In detail,
we first distinguish non-causal information by applying the open information
extraction (OIE) technique. Then, we propose a method named the Causal
Information Enhanced SAmpling Method (CIESAM) to eliminate the non-causal
information from data. To validate our theoretical analysis, we further
propose another method using our proposed Causality-Aware Self-Attention
Mechanism (CASAM) to guide the model to learn the underlying causality
knowledge in legal texts. The confidence of CASAM in learning causal
information is higher than that of CIESAM. The extensive experimental results
show that both our proposed methods achieve state-of-the-art (SOTA)
performance on three commonly used legal-specific datasets. The stronger
performance of CASAM further demonstrates that causality is the key to the
robustness and generalization ability of models.

##  1 Introduction

Understanding why is critical for Legal Judgment prediction (LJP) models,
which determines whether the legal artificial intelligence (legal AI) yields
to the rule of law or to the rule of correlation. LJP is a crucial task of
legal AI. A LJP model aims at assisting the legal practice by predicting the
judgment of a certain case (e.g., charge, term of penalty, or law article)
according to the corresponding case fact descriptions zhao2022legal ;
cui2022survey . In contrast to other natural-language processing tasks, the
LJP model must correctly learn the reasons behind each case rather than only
make predictions. Rule of law defines the uniform principles and protocols for
all judgments in a country. Every judgment must have a clear reasoning process
that can cite back to the rules in the laws. Therefore, there exists a stable
and uniform common ground-truth knowledge underlying all judgment cases. If a
LJP model is legitimated to be adopted for legal practice, it has to learn the
common ground-truth knowledge. Otherwise, the judgment prediction is invalid
even if it is accurate when fitting the historical cases. To sum up, only by
learning rules of law represented by common ground causality, can models
perform better, achieve more robustness, and be trustworthy.

Previous methods on legal text processing are based on rules
lawlor1963computers , statistical methods nagel1960weighting ;
keown1980mathematical , or machine learning methods katz2012quantitative ;
aletras2016predicting , which suffers from being sensitive to noises and
lacking generalization ability w.r.t. other law domains. Recently, the rapid
development of large-scale pre-trained language models (PLMs) based on
transformers significantly benefits this area cui2022survey . Some of the PLMs
including BERT devlin2018bert  are further pre-trained on legal corpora, such
as Legal-BERT chalkidis2020legalbert , exhibiting the SOTA performance on
legal text processing benchmarks (e.g., LexGLUE) zheng2021when ;
chalkidis2022lexgluea .

However, we found clues that the commonly adopted state-of-art LJP model
chalkidis2020legalbert  misunderstands the data and learns the spurious
correlations. The current LJP model can make right predictions according to
irrelevant reasons, which has not been reported yet. In Figure 1, we provide
an example, where the prediction of Legal-BERT is reversed by a small change
that does not cause an essential semantic change. Further, we discovered that
the most important keywords deciding the model predictions mainly concentrated
on punctuation marks and function words. A large number of predictions only
rely on less than $5\%$ of words from the fact descriptions rather than
considering the whole text as shown in Figure 1. All the evidence indicates
that the current LJP models learn spurious correlations or shortcuts
geirhos2020shortcut  rather than the common ground-truth knowledge about the
rules of the laws.

Figure 1: An example of reversed prediction caused by character substitution.

In this paper, we reveal the principles for tackling the fatal problem. We use
structural causal models (SCMs) to theoretically analyze the mechanism of
learning models in the LJP task and then argue that three factors hamper the
development of the current LJP models. Specifically, our analysis derives that
the main challenge of learning models in this task is to infer the unique
causal mechanism from its generated training data. Significant development
rarely occurs in tackling this challenge due to three reasons. First, the
learning models themselves neglect the use of human knowledge, which largely
increases the uncertainty for them to infer causal relationships from the
training data. Second, each case in the training data suffers from the problem
of data imbalance, where high-frequency words often co-occur with other words
and thus lead to the spurious correlation issue. Third, our current evaluation
methods focus on measuring average error across a held-out test set,
neglecting the fact that the test set and training set are identically
independently distributed (i.i.d.): Models just need to greedily absorb all
correlations that happen to be predictive in the test set even if they are not
causal relationships.

To address the issues, we provide two methods focusing on the improvement of
training data and the architecture of models, respectively. From the
perspective of data, we focus on mitigating the disturbance from non-causal
information by removing them from the training data and thus propose a causal
information-enhanced sampling method (CIESAM). From the perspective of model
architecture, we aim to prevent the PLM from learning non-causal information
by restricting the interaction (represented by attention weights) between
causal and non-causal words. Specifically, we propose a causality-aware self-
attention mechanism (CASAM) to reallocate the attention weights throughout the
overall transformer encoder, which leads the PLM to pay more attention to
causal information. The extensive experimental results show that both of our
proposed methods perform better on generalization and robustness than the
baseline models and achieve new SOTA performance on the three commonly used
legal prediction datasets. Additionally, CASAM performs better than CIESAM as
it learns from the unique ground-truth causal relationship.

##  2 Problem Settings

In this section, we propose a structural causal model to analyze the mechanism
of learning models in the LJP task and then point out three factors that
impede them from learning the rule of law.

###  2.1 Structural Causal Model

Here, we propose a structural causal model (SCM) to explain the underlying
causal relationships in the LJP task. The SCM pearl2009causal  represents the
causal relations between variables by a directed acyclic graph (DAG). It
denotes the random variables as nodes while their causal relationships as the
directed edges. Literature pearl2009causal  demonstrated that the same DAG
also captures the conditional correlations between the same set of random
variables. For example, we denote the fact that $X$ directly causes $Y$ by
$X\rightarrow Y$. If $X$ is the common cause of both $Y$ and $Z$, the latter
two variables are independent given $X$.

The rules of law define a basket of causal relations between the facts of the
criminal cases and the associated judgments. Because all judges have to follow
the rules, the judgment cases in the same country must have a stable
relationship between the facts and the judgments no matter the variance of
judges. Therefore, we call those pieces of information about the facts
deciding the judgments as the causal information $C$. While a judge prepares
the case description $T$, she has to explain the relationship between the
facts and the judgment $Y$. Beyond, $T$ is also contingent on other features
denoted by $N$, such as the grammar principle and individual writing patterns.
The features of $N$ do not influence the judgment $Y$ in the ground truth and
are denoted as non-causal information. Due to the grammar requirements and
other reasons, $N$ and $C$ can be correlated with each other. We model the
above causal relation in Figure 2 named the current paradigm of learning
models. According to the rules of law, $N$ has to be independent of $Y$. For
example, for a fact description $T$ expressed by Bob, a 47 years old European
white male, robbed Alice of her car., a court will make the judgment that Bob
commits a robbery, regardless of the gender, race, region, or age of Bob. In
this case, the event description Bob robbed Alice of her car includes the
causal information $C$ while the demographic information of Bob, as well as
the usage of the function words, are non-causal information $N$. The usage of
the function words, which is part of $N$, can correlate with the texts
including $C$.

Figure 2: Structural causal model of the LJP task. The rule of law generates
the data in LJP, and models are expected to accurately estimate the
correlations between variables (the unique causal graph in purple). There are
various potential results that are all optimal and possible to be learned by
models. Three problems prevent models from learning the purple causal graph.
Our proposed CIESAM filters the 4 yellow graphs and CASAM filters 10 graphs
including the yellow and blue ones.

###  2.2 Three Unsolved Problems

In this section, we first explain how models can learn causality and then
introduce three unsolved problems that prevent learning models from learning
causality.

####  2.2.1 How to Learn Causality

Models learn the correlation relationships among variables, which can be
generated in either of the three ways: causality, confounding, and data
selection bias cui2022stable . Only the correlations generated by causality
are what we expect the models to learn from. For example, in Figure 2, the
correlation effect between $C$ and $Y$ learned by models equals the causal
effect, which faithfully reflects their intrinsic dependency. Correlations
generated in two other ways are often referred to as spurious correlations,
which bring the following two challenges and hamper models from learning
causality.

####  2.2.2 Data Imbalance

The first problem is data imbalance (i.e., selection bias). Usually, the raw
data of case descriptions compose an imbalanced dataset of $C$, $N$, and $Y$.
We use data-driven methods to train different models (estimators) for learning
the correlation relationships between input variables and output variables
cui2022stable . Consequently, the learning model can incorrectly fit the
correlations between those variables. For instance, male criminals can be
penalized harder than female ones in robbery cases because the males are
stronger and thus can cause more serious damage. The random sampling process
involves more cases where men cause serious damage. The data imbalance can
lead the LJP model to misunderstand the relationship between gender (non-
causal information $N$) and penalty decision (prediction $Y$) in robbery
cases, thus causing the learning model to overfit the correlation between
gender and penalty to improve its performance on the test set.

####  2.2.3 Lack of Human Knowledge

The second problem is the imbalance and entanglement between causal and non-
causal information, which makes models unable to distinguish the correlations
generated by causal information. Specifically, current LJP models are trained
to self-explore $P(Y\mid T)$, the correlation between $Y$ and $T$, rather than
directly learning $P(Y\mid C)$ due to the difficulty of comprehensively
figuring out $C$. The learning process can lead the LJP model to learn the
incorrect causal relation between $C$, $N$, and $Y$. Therefore, extra
information that can reduce the information entropy, such as causal
information or human knowledge, is of great urgency. Our logic basics are as
follows.

Multiple Potential Estimating Results. In the LJP task, judges recognize the
causal relationship between the causal information in fact descriptions and
their final prediction (i.e., $C\rightarrow Y$) in each case for fairness. The
causal relationship between C and Y manifests the rule of law and is
considered stable across all cases. Since models are not able to distinguish
causal relationships, they are expected to accurately learn the correlation
between $C$ and $Y$ from the training data to form the unique ground truth
solution to LJP: The learned correlation between $C$ and $Y$ will be causal
relation if there are no other spurious correlations in the training data.
However, existing LJP models suffer from learning the ground truth causal
relation. They are delicately designed to fit $P(Y\mid T)$, which generates
spurious correlations between both “$C$ and $N$” and “$N$ and $Y$” in
situations where $T$ is given as shown in Figure 2. Consequently, there exist
multiple potential learning results, which depend on the optimization process.
The optimization process can also be random if the optimizer is developed from
stochastic gradient descent. We list all of the potential decision rules
(causality) in Figure 2 that are possible to be learned by models and
presented by parameters.

All Estimating Results Can be Optimal. From the 25 potential learning results
in Figure 2, 11 results are optimal w.r.t. the training objective: They can
experimentally (which is demonstrated by baseline models) and theoretically
minimize the loss function to zero. The underlying reason is that most learned
spurious correlations will establish in the testing set as the training set
and test set are identically and independently distributed (sampled from the
same distribution). In this setting, models greedily learn all correlations
including spurious correlations to improve their performance with the cost of
their generalization ability and robustness. We select the best-performed one
while neglecting its other ability. To solve the issue, we communicate with
legal experts to propose several legal-specific attacks for evaluation, the
corresponding experiments and details are shown in Section 4.3 and Section
4.5.

Only One of Them is Our Need. Among those correlations, only the correlation
generated by $C\rightarrow Y$ is the ground truth solution that a model ought
to learn. However, learning the correlation generated by $C\rightarrow Y$
becomes a random event $R$, whose probability can be determined by many
factors, including training data and model.

####  2.2.4 Incomplete Testing Data

The third problem is that the current testing data is not comprehensive. In
the LJP task, the training data and testing data are assumed to be identically
and independently distributed (i.i.d.): They are often sampled from the same
distribution (e.g., a court like the European Court of Human Rights or a
region like European). However, the i.i.d. assumption can easily be violated
in real cases. Recent research shen2021outofdistribution ; wiles2022a  reports
that learning models become vulnerable when exposed to test data with
distributional shifts, which indicates that evaluating the out-of-distribution
(OOD) generalization ability of models in LJP is of critical significance. The
data for testing such an ability is neglected, which misleads the model
selection and makes the learned spurious correlations undiscovered and even
predictive in the current testing data: spurious correlations in training data
can also be established in the testing data due to i.i.d. assumption.

###  2.3 Impact of Unsolved Problems

The above explanation manifests the dynamic that the spurious correlation
error weakens the generalization capability of LJP models. Beyond the causal
information, the non-causal information and the texts vary by judges and
cases. For instance, the text of a legal case is contingent on the judge who
is assigned to write the case description. The writing patterns of judges
affect the functional relation between $N$ and $T$. Thus, if the LJP model
incorrectly learns the spurious correlations between $N$ and $Y$ in the
training set, the model can perform poorly in another set when the two sets
have different writing patterns as well as different functional relations
between $N$ and $Y$. However, due to rule of law, the relation between $C$ and
$T$ is stable, if the LJP model correctly understands the reason for a
judgment, the model can uniformly perform well in various data sets
cui2022stable .

Learning spurious correlations can also make the LJP model less robust. If the
LJP model learns the spurious correlation, varying $N$ can lead to a change in
the prediction results. However, in the ground truth, $N$’s variation must not
affect $Y$ due to the principle of rule of law.

###  2.4 Our Solutions

The first problem is widely noticed by the AI community and tackled through
various methods, including data augmentation and weighted approaches. However,
the second and third problem is hardly noticed in the LJP task. The second
problem can not be solved by adopting the same methods used in the first
problem. Specifically, the LJP model considers $N$ as another cause of $Y$
rather than the non-causal information. While parts of $N$ play the same role
in both training and testing sets, the cross-validation can fail to exclude
the overfitting effects. For instance, the grammar rules play the same role in
all case descriptions. Learning models can overfit the relation between the
information about grammar rules and the legal judgments for improving
prediction accuracy. In most situations (reflected by word frequency), the
model makes predictions according to punctuation marks and function words.

Since only the purple causal graph in Figure 2 satisfies the rule of law, we
focus on increasing its probability to be learned through data and model: two
of the most crucial factors that largely affect a deep model. We achieve this
goal by excluding the possibility of other potential estimates. Note that due
to the precision of our adopted OIE tools (for distinguishing $N$ from $C$),
we cannot perfectly implement exclusion in experiments. Instead, we can still
approach our goal by reducing the possibility of other potential estimates.
Specifically, we focus on preventing models from learning the spurious
correlation between $N$ and $Y$. We consider two methods as follows.

Intervention on Data. If we assume that the training data only comprise causal
information $C$ labeled with $Y$, machine learning methods can only learn the
correlation between $C$ and $Y$ (i.e., the causal relationship between $C$ and
$Y$). However, the assumption can hardly be satisfied due to two reasons: (1)
the causal information $C$ can be described by natural language in infinite
ways, which is infeasible for the data collector to sample all of them in the
training data; (2) non-causal information $N$ such as grammar and writing
style can be inevitably involved into the training data. The former results in
knowledge deficiency in the training data while the latter decreases the
probability of the target estimate of parameters (occurrence of $R$).
Therefore, to offset the decline of $P(R|X)$, we propose CIESAM for making
attempt to filter the non-causal information.

Intervention on Model. We can also opt to improve the understanding ability of
models, making them able to distinguish and avoid learning spurious
correlations. For example, a straightforward linear model can avoid the
disturbance of non-causal information if it knows the legal knowledge: It can
set the coefficient in front of non-causal variables to zero regardless of
their amount in the training data. To this end, we propose CASAM for infusing
learning models with causal information and knowledge.

Complete Testing Data by Legal-Specific Attacks. We complete the testing data
by proposing legal-specific attacks to bring distributional shifts into the
data. More details are shown in Section 4.3.

##  3 Methodology

According to the above analysis, it is necessary to mitigate the spurious
correlation error in the LJP model. Otherwise, the LJP model predicts the
judgment according to spurious correlations and violates the rules of law.
Consequently, the LJP model is not trustworthy even if it has a good
prediction performance.

Directly eliminating the spurious correlation error is a challenge. To fully
prevent models from learning spurious correlations, there are two
straightforward options: 1) directly conditioning on $C$ or 2) removing $N$ to
block its effect of misleading the training. However, there lacks a method of
filtering out $C$ from the judgment case descriptions. In the case
descriptions, causal and non-causal information are entangled to form complete
semantic expression. It is hard to clarify and separate the causal information
from the non-causal part in a case description. Further, the data imbalance is
also hard to be corrected due to the nature of language such as the Zipf’s law
reed2001pareto . For instance, the methods of reweighting the samples such as
propensity-score weighting are the major methods of correcting the data
imbalance. However, it is hard to apply the reweighting methods to adjust the
data imbalance existing in the fact descriptions.

In this paper, we propose two methods of lowering the proportion of $N$ in the
training data for LJP model and mitigating the spurious correlation error. The
first method named causal information enhanced sampling method (CIESAM) is
inspired by CATT yang2021causal , whose main idea is to control the training
samples (or sampling process) for learning. The second method named causality-
aware self-attention mechanism (CASAM) focuses on intervening in the
computation process, rendering causal information to get noticed within and
throughout the learning model. Both methods are based on the technology of
open-information extraction (OIE). We notice that OIE tools are able to
capture the minimal context that mostly maintains the content
stanovsky2018supervised . It is possible to adopt OIE as a filter to separate
the context that mainly includes the causal information of the legal texts
from those that mainly include the non-causal information. Further, the open-
source coreference method merges the context that possesses the same semantic
clark2016deep .

Figure 3: Overview of our framework.

To sum up, the overall framework can be divided into two steps. In the first
step, we adopt the OIE and open-source coreference methods to refine the
dataset and mitigate the data imbalance in legal texts. We first perform open
information extraction (OIE) on input legal texts to discard the context that
contains a high proportion of non-causal information. Then, we graphically
structure the extracted pieces of information shown in Figure 3. In the
extracted information (knowledge) graph, the nodes denote the subjects,
objects, and predicates while the edges are dependencies. The nodes possessing
the same semantic meaning will be merged into one by the open-source
coreference model. During the process of constructing graphs, redundant non-
causal information is further reduced by merging. Meanwhile, documents are
substantially compressed to focus on core information. The above data
processing lowers the proportion of $N$ in the legal-case texts, thereby
mitigating the spurious correlation between $N$ and $Y$. In the second step,
we apply the knowledge to intervene in the learning process in two ways. In
the rest of this section, we provide the detail of our methods.

###  3.1 Graph Construction by OIE.

In this section, we detail the process of extracting graph structures from
text, which aims to discard and merge redundant non-causal information. First,
we apply coreference resolution clark2016deep  and open information extraction
stanovsky2018supervised  tools to identify the corresponding mentions or
pronouns of each entity, and then extract relational triplets from sentences.
In our constructed graph, we represent subjects and objects as nodes, which
are connected by predicates as directed edges. Second, the nodes will be
merged to reduce redundant non-causal information if they have similar names
or meanings, which is identified by TF-IDF overlap and coreference resolution
tools, respectively. Finally, as to the subsequent newly extracted triplets,
we also calculate the TF-IDF overlap between the existing triplets and the new
one. If the value is higher than our predefined threshold, we rule out the new
triplet to reduce information replication.

###  3.2 CASAM

We introduce our proposed CASAM in this section. CASAM partly inherits the
architecture of the transformer vaswani2017attention  or Legal-BERT
chalkidis2020legalbert  encoder which consists of L stacking blocks. Each
block comprises a feed-forward network, residual connection, layer
normalization, and a causal attention module. Given a fact description $D$, we
obtain its embedding matrix $\mathbf{X}\in\mathbb{R}^{N\times d}$ according to
the embedding layer of a transformer encoder, where $N$ and $d$ denote the
sequence length and the dimension of hidden layers, respectively. Following
the transformer encoder, CASAM maps $\mathbf{X}$ to query, key, and value
matrices in each block by
$\mathbf{Q}=\mathbf{X}\mathbf{W}_{q},\mathbf{K}=\mathbf{X}\mathbf{W}_{k},\mathbf{V}=\mathbf{X}\mathbf{W}_{v},$
where $\mathbf{W}_{q},\mathbf{W}_{k},\mathbf{W}_{v}\in\mathbb{R}^{d\times k}$
are model parameters the in $l$-th block, $l$ is omitted in the equation for
brevity.

Different from the widely adopted self-attention mechanism which considers all
words to correlate with each other, our proposed causal attention module
performs causal intervention between each word pair. The former provides
abundant correlations represented by unsupervised attention weights for models
to explore, neglecting the fact that learning methods will greedily absorb all
correlations (including spurious correlations) found in data to minimize their
training error, which leads to spurious correlation error. The latter tries to
discern the potential causal relationships and block non-causal information to
prevent learning spurious correlations. Specifically, our proposed CASAM first
derives an adjacency matrix $\mathbf{A}$ according to a certain graph
$\mathcal{G}$ constructed by the aforementioned open information extraction
(OIE) tool. The entries $\mathbf{A}_{ij}$ tabulate the binary variable
identifying whether the combination of $i$-th word and $j$-th word will
causally affect the final judgment. Then, based on the original attention
weights calculated by $\mathbf{S}=\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d}}$,
the new attention weights are derived by,

$\mathbf{S}^{\prime}=\alpha\mathbf{S}+(1-\alpha)\mathbf{S}\odot\mathbf{A},$
(1)

where $\odot$ denotes the element-wise multiplication between matrices and
$\alpha$ is a hyperparameter ranging from $0$ to $1$, which is adjusted
according to the accuracy of an OIE tool: the more accurate the OIE tool, the
higher the $\alpha$. The output $\mathbf{Y}$ of each causal attention module
is derived by,

$\mathbf{Y}=\operatorname{softmax}(\mathbf{S}^{\prime})\mathbf{V}.$ (2)

We input $\mathbf{Y}$, the output of the final causal attention layer
considered as the representation of a fact description $D$, into a linear
layer followed by a sigmoid function to obtain the final predictions.

###  3.3 CIESAM

CIESAM incorporates the processed data and the raw data for balancing the
spurious correlation mitigation and information loss minimization. To avoid
the conflict and inconsistency that occurred in encoding heterogeneous
information through Transformer shao2020graph , we perform graph linearization
to model the extracted graphs as sequences. Existing methods of converting
graphs into sequences can roughly be divided into two categories: training
graph-to-sequence models wei2021graphtosequence  based on graph transformer
cai2020graph  or heterogeneous graph transformer yao2020heterogeneous , and
graph linearization fan2019using ; pasunuru2021efficientlya  methods which
artificially designing some rules to store graphs in a structured sequence.
The latter faithfully express a graph and the former introduce noises in their
generated sequences. To avoid introducing new non-causal information which may
induce new spurious correlations, we adopt a graph linearization method, which
is considered more suitable. Specifically, we first obtain the weights of
nodes and edges by counting how many times their corresponding phrases appear
in a document. Second, we perform graph traversal in a breadth-first manner
according to the weights. Finally, the resulting sequences of graph traversal
are adopted as the linearized graphs and encoded as $\mathbf{X}_{g}$. The
final output of CIESAM is derived by,

$\mathbf{Y}^{\prime}=\operatorname{LegalBERT}\left(\beta\mathbf{X}+\left(1-\beta\right)\mathbf{X_{g}}\right),$
(3)

where $\beta$ is the hyperparameter ranging from $0$ to $1$. Its value is
positively correlated with the accuracy of an OIE tool. Similar to CASAM,
$\mathbf{Y}^{\prime}$ is input into a linear layer followed by a sigmoid
function to predict the final judgment.

###  3.4 Model Selection

Traditionally, the training data and validation data are i.i.d. and the latter
are adopted to monitor the training process for selecting the best-performed
version of a learning model. According to our analysis in Section 2.2.4, the
selection can be biased as the evaluation of the generalization ability of
models is incomplete: lacking the evaluation of OOD performance. To solve the
issue, we complete the validation data by our proposed legal-specific attacks
to evaluate both the robustness and generalization ability of models.
Different from previous methods, we aim to select the most robust and
generalizable version of a learning model during the training process.

##  4 Experiments

###  4.1 Datasets

ECtHR Task A & B. The European Court of Human Rights (ECtHR) dataset
chalkidis2019neural  is the only publicly available human-annotated LJP
dataset in English, consisting of approximately 11,000 cases from the ECtHR
database. In each case, allegations are written as fact descriptions, the
judgment results — about which of human rights provisions legislated by
European Convention of Human Rights (ECHR) does the current state breach — are
recorded as the label. All cases are chronologically categorized as training
set (9k, 2001-2016), development set (1k, 2016-2017), and test set (1k,
2017-2019). Each case can either violate single, multiple, or none of the
given legal articles. For each model, the input is fact descriptions of a
case, and the output is the judgment, represented by a set of violated
articles. In Task A, the violated articles are considered by the court. In
Task B, the violated articles are put forward by the applicants.

LEDGAR. LEDGAR (Labeled EDGAR) tuggener2020ledgar  is a dataset for contract
provision classification. Considering the underlying common legal text
classification techniques, we conduct experiments on the dataset not only for
a more comprehensive evaluation, but also to test the generalization ability
of models. In LEDGAR, the contract provisions are crawled from the U.S.
Securities and Exchange Commission (SEC) website and are available from an
Electronic Data Gathering, Analysis, and Retrieval (EDGAR) system on the
website. Nearly 850k contract provisions from 12.5k categories are included in
the originally proposed LEDGAR. Following the legal language understanding
benchmark LexGLUEchalkidis2022lexgluea , we use 80k contract provisions
labeled with 100 most frequent categories from the original dataset. The
dataset is chronologically split into a training set (60k, 2016-2017), a
development set (10k, 2018), and a test set (10k, 2019).

###  4.2 Baselines

TFIDF+SVM combines the Term Frequency and Inverse Document Frequency technique
with a linear Support Vector Machine.
BERT devlin2019bert  is a pre-trained transformer model used to predict masked
language and the next sentence.
RoBERTa liu2019roberta  is also a transformer-based model, it uses dynamic
masking and uses larger training corpora in the pre-training stage compared
with BERT.
DeBERTa he2021deberta  computes attention using disentangled matrices and it
applies an enhanced mask decoder. For the finetuning task, it proposes a new
adversarial training technique.
Longformer beltagy2020longformer  uses sparse attention mechanism to make the
model suitable for long sequence of language.
BigBird zaheer2020big  is also a transformer-based language model, it uses
local, global, and random attention to get better performance on long
sequences of language.
CaseLaw-BERT zheng2021when  is a law case oriented BERT model. This model is
based on a BERT model and trained with law case data.
Legal-BERT chalkidis2020legalbert  model is similar to the CaseLaw-BERT model,
they are both trained based on BERT. Legal-BERT is trained with legal corpora,
contracts, law cases, and other law-related documents.

The backbone of our proposed model is based on Legal-BERT in this paper, our
model can be easily extended to other backbones in future work.

###  4.3 Experimental Settings

Implementation Details. Our experiment is based on PyTorch and Hugging Face
Transformer wolf2020transformersa . At the graph construction stage, co-
reference resolution predictor clark2016deep  and OIE predictor
stanovsky2018supervised  are used to extract graph relationships and construct
the graph. Later, we use breadth-first search to get the linearized graph
text. We apply the pre-trained Legal BERT transformer from Hugging face to be
our encoder. With the original fact descriptions and the corresponding graph
text, we use two Legal BERT encoders to get the embeddings. The learning rate
is $1e-4$ and the optimizer is AdamW. Following previous work
chalkidis2022lexgluea , we evaluate the performance (e.g., generalization
ability) of models by $\mu$-$F_{1}$ and $m$-$F_{1}$ scores.

Evaluated Attacks. According to the suggestions provided by experts in the
legal domain, we consider several types of attacks for thorough robustness
evaluation. In each type of the following attacks written in bold, we make a
distinct perturbation in the given fact description that will not change the
judgment from the perspective of the experts. For those attacks written in
italics, the perturbation will not change the judgment in most circumstances
according to the experts. We provide descriptions of all types of attacks: (1)
functional word attacks. We adopt the token ‘[mask]’ as a substitute for a
functional word; (2) word-level attacks, which mask a single word; (3)
sequence number attacks, which remove the sequence number in front of the
given description; (4) dot attacks after sequence number. We remove the dot
after a sequence number; (5) punctuation mark attacks, which mask a
punctuation mark; (6) auxiliary verb attacks, which mask an auxiliary verb;
(7) article attacks, which mask an article before a noun; (8) preposition
attacks. We attack prepositions except the preposition ‘of’ (which may
indicate the ownership relationship), the preposition ‘for’ (which may
represent whether someone does something on purpose), and those prepositions
that locate between numbers.

To evaluate the robustness of models, we adopt certified ratio (CR) to measure
the percentage of consistent predictions (unchanged predictions) under a
perturbation (wrong predictions are also included) and $1-\text{CR}$ as the
success rate (SR) of attack gurel2022knowledge .

Method | ECtHR(A) | ECtHR(B) | LEDGAR
---|---|---|---
$\mu$-$F_{1}$ |  $m$-$F_{1}$ |  $\mu$-$F_{1}$ |  $m$-$F_{1}$ |  $\mu$-$F_{1}$ |  $m$-$F_{1}$
TFIDF+SVM* | 64.5 | 51.7 | 74.6 | 65.1 | 87.2 | 82.4
BERT | 71.1 | 61.2 | 79.2 | 72.1 | 88.0 | 82.1
RoBERTa | 72.0 | 65.6 | 77.6 | 70.9 | 87.6 | 81.3
DeBERTa | 71.5 | 66.7 | 80.2 | 73.1 | 88.1 | 82.9
Longformer | 71.0 | 62.1 | 79.7 | 71.9 | 87.7 | 81.3
BigBird | 69.8 | 59.7 | 78.1 | 68.5 | 87.1 | 80.8
CaseLaw-BERT | 71.6 | 65.5 | 78.6 | 71.9 | 88.0 | 81.8
Legal-BERT | 72.3 | 66.0 | 80.6 | 75.2 | 88.2 | 81.9
CIESAM | 73.4 | 67.4 | 81.0 | 76.7 | 88.7 | 83.6
CASAM | 73.8 | 68.5 | 81.4 | 76.0 | 88.7 | 83.5
Table 1: Overall experimental results. The signal ‘*’ denotes that the results
of the corresponding models are quoted from LexGLUE chalkidis2022lexgluea .

Attribution Method. Current feature attribution methods can be roughly divided
into three categories: gradient-based methods which calculate a score for each
input feature by gradients springenberg2015striving ; li2016visualizinga ;
simonyan2019deep , reference-based methods which consider the difference
between a predefined “reference” and the output of a model as the attribution
score ribeiro2016why ; shrikumar2017learning ; sundararajan2017axiomatic , and
erasure-based methods which measure the change of model prediction as the
attribution score after removing the target feature zeiler2014visualizinga ;
li2016visualizinga ; feng2018pathologies ; chen2020generating . We adopt an
erasure-based method li2020bertattack  due to its simplicity and faithfulness.
Specifically, if a fact description
$D=[d_{1},\dots,d_{i-1},d_{i},d_{i+1},\dots]$ is input into a certain model,
and the corresponding output prediction score on the ground truth label $y$ is
$o_{y}(D)$, then the attribution value on $d_{i}$ is written by,

$F_{y}(d_{i})=o_{y}(D)-o_{y}(D^{\prime}),$ (4)

where $D^{\prime}=[d_{1},\dots,d_{i-1},[\operatorname{MASK}],d_{i+1},\dots]$.
Erasure-based methods directly satisfy the way of evaluating an AI judger by
rule of law: Will the judgment change if the causal elements get erased or
changed from the fact descriptions? Will the AI judger consistently stick to
rule of law in any circumstances (e.g., changes in irrelevant information)?

Attack | Method | ECtHR(A) | ECtHR(B) | LEDGAR
---|---|---|---|---
CR | SR | CR | SR | CR | SR
Functional word | Bert | 99.37 | 0.63 | 99.32 | 0.68 | 93.76 | 6.24
RoBerta | 99.29 | 0.71 | 99.06 | 0.94 | 95.81 | 4.19
| Longformer | 91.46 | 8.54 | 93.63 | 6.37 | 96.48 | 3.52
| Bigbird | 96.11 | 3.89 | 95.82 | 4.18 | 96.39 | 3.61
| CaseLaw-Bert | 99.59 | 0.41 | 99.57 | 0.43 | 94.33 | 6.67
| Legal Bert | 99.78 | 0.22 | 99.69 | 0.31 | 94.81 | 5.19
| CIESAM | 99.69 | 0.31 | 99.75 | 0.25 | 96.54 | 3.46
| CASAM | 99.89 | 0.11 | 99.88 | 0.12 | 96.17 | 3.83
Word-level | Bert | 99.27 | 0.73 | 99.23 | 0.77 | 84.00 | 16.00
RoBerta | 99.22 | 0.78 | 98.96 | 1.04 | 79.10 | 20.90
| Longformer | 81.89 | 18.11 | 86.07 | 13.93 | 78.91 | 21.09
| Bigbird | 83.35 | 16.65 | 85.63 | 14.37 | 81.61 | 18.39
| CaseLaw-Bert | 99.45 | 0.55 | 99.42 | 0.58 | 86.06 | 13.94
| Legal Bert | 99.66 | 0.34 | 99.53 | 0.47 | 88.73 | 11.27
| CIESAM | 99.54 | 0.46 | 99.64 | 0.36 | 90.91 | 9.09
| CASAM | 99.75 | 0.25 | 99.79 | 0.21 | 89.65 | 10.35
Seq. Num. | Bert | 99.38 | 0.62 | 99.37 | 0.63 | - | -
RoBerta | 99.73 | 0.27 | 99.48 | 0.52 | - | -
| CaseLaw-Bert | 99.58 | 0.42 | 99.47 | 0.53 | - | -
| Legal Bert | 99.79 | 0.21 | 99.67 | 0.33 | - | -
| CIESAM | 99.74 | 0.26 | 99.70 | 0.30 | - | -
| CASAM | 99.79 | 0.21 | 99.87 | 0.13 | - | -
Dot after Seq. Num. | Bert | 99.35 | 0.65 | 99.42 | 0.58 | - | -
RoBerta | 99.43 | 0.57 | 99.33 | 0.67 | - | -
| CaseLaw-Bert | 99.73 | 0.27 | 99.70 | 0.30 | - | -
| Legal Bert | 99.83 | 0.17 | 99.81 | 0.19 | - | -
| CIESAM | 99.75 | 0.25 | 99.82 | 0.18 | - | -
| CASAM | 99.83 | 0.17 | 99.89 | 0.11 | - | -
Punctuation mark | Bert | 99.44 | 0.56 | 99.37 | 0.63 | 96.45 | 3.55
RoBerta | 99.30 | 0.70 | 99.09 | 0.91 | 95.97 | 4.03
| Longformer | 91.59 | 8.41 | 93.58 | 6.42 | 96.79 | 3.21
| Bigbird | 96.37 | 3.63 | 96.01 | 3.99 | 96.70 | 3.30
| CaseLaw-Bert | 99.68 | 0.32 | 99.67 | 0.33 | 97.47 | 2.53
| Legal Bert | 99.82 | 0.18 | 99.64 | 0.36 | 96.76 | 3.24
| CIESAM | 99.72 | 0.28 | 99.80 | 0.2 | 95.68 | 4.32
| CASAM | 99.83 | 0.17 | 99.87 | 0.13 | 98.08 | 1.92
Auxiliary Verb | Bert | 99.54 | 0.46 | 99.21 | 0.79 | 98.22 | 1.78
RoBerta | 99.02 | 0.98 | 99.02 | 0.98 | 70.00 | 30.00
| Longformer | 99.10 | 0.90 | 98.20 | 1.80 | 90.00 | 10.00
| Bigbird | 99.27 | 0.73 | 99.27 | 0.73 | 90.00 | 10.00
| CaseLaw-Bert | 99.63 | 0.37 | 99.67 | 0.37 | 99.05 | 0.95
| Legal Bert | 99.74 | 0.26 | 99.65 | 0.35 | 98.48 | 1.52
| CIESAM | 99.70 | 0.30 | 99.71 | 0.29 | 92.64 | 7.36
| CASAM | 99.88 | 0.12 | 99.89 | 0.11 | 98.89 | 1.11
Article | Bert | 99.35 | 0.65 | 99.32 | 0.68 | 98.45 | 1.55
RoBerta | 99.00 | 1.00 | 98.9 | 1.01 | 98.44 | 1.36
| Longformer | 96.51 | 3.49 | 97.50 | 2.50 | 98.05 | 1.95
| Bigbird | 98.96 | 1.04 | 99.16 | 0.84 | 97.37 | 2.63
| CaseLaw-Bert | 99.64 | 0.36 | 99.65 | 0.35 | 98.54 | 1.46
| Legal Bert | 99.79 | 0.21 | 99.77 | 0.23 | 98.44 | 1.56
| CIESAM | 99.71 | 0.29 | 99.75 | 0.25 | 95.92 | 4.08
| CASAM | 99.87 | 0.13 | 99.91 | 0.09 | 99.17 | 0.83
Preposition | Bert | 99.38 | 0.62 | 99.33 | 0.67 | 93.84 | 6.16
RoBerta | 99.29 | 0.71 | 99.07 | 0.93 | 95.78 | 4.22
| Longformer | 91.49 | 8.51 | 93.62 | 6.38 | 96.44 | 3.56
| Bigbird | 96.17 | 3.83 | 95.83 | 4.17 | 96.36 | 3.64
| CaseLaw-Bert | 99.59 | 0.41 | 99.55 | 0.45 | 94.39 | 5.61
| Legal Bert | 99.78 | 0.22 | 99.68 | 0.32 | 94.88 | 5.12
| CIESAM | 99.70 | 0.30 | 99.71 | 0.29 | 96.47 | 3.53
| CASAM | 99.85 | 0.15 | 99.88 | 0.12 | 96.26 | 3.74

Table 2: Results of robustness evaluation on the test sets of three benchmark
datasets. Seq. Num. denotes sequence number.

###  4.4 Main Results

The generalization ability (performance) evaluation results of baselines and
our model are shown in Table 1. We can observe that the performance of our
framework significantly outperforms the SOTA baseline methods, achieving a new
SOTA performance on all three benchmark datasets. Note that our framework is
based on the Legal-BERT backbone. Compared with Legal-BERT, CIESAM yields
performance gains of $3.4\%$/$3.4\%$ of $\mu$/$m$-F1 scores in ECtHR Task A,
$0.6\%$/$2.0\%$ of $\mu$/$m$-F1 scores in ECtHR Task B, and $0.5\%$/$0.6\%$ of
$\mu$/$m$-F1 scores in LEDGAR, while CASAM yields gains of $3.8\%$/$4.5\%$ of
$\mu$/$m$-F1 scores in ECtHR Task A, $1.0\%$/$1.3\%$ of $\mu$/$m$-F1 scores in
ECtHR Task B, and $0.5\%$/$0.5\%$ of $\mu$/$m$-F1 scores in LEDGAR. The
experimental results in Table 1 indicate that, with the guidance of our
theoretical analysis, both of our methods effectively improve the performance
of Legal-BERT: They block $N$ to reduce the spurious correlation error, which
leads the model to learn the underlying ground-truth knowledge and thereby
enhancing the generalization ability of the model.

###  4.5 Results of Robustness Evaluation

We evaluate the robustness of models against diverse attacks. As shown in
Table 2, the robustness of our proposed CASAM is significantly stronger than
its backbone on the three datasets under all kinds of attacks. Without any
modification, the original Legal-BERT exhibits poor robustness, especially on
ECtHR Task A, which is labeled with real-world judgments from the court.
Changes in the irrelevant information in fact descriptions will eventually
render the Legal-BERT judger altering at least $14.40\%$ of its judgments,
which terribly hurts its robustness and the trust in it. Such kinds of
mistakes caused by the spurious correlation error impede the deployment of AI
judgers in real-world applications. Our proposed framework significantly
mitigates the underlying error and thus enhances the robustness of models.
Especially, both of our methods surpass their Legal-BERT backbone by at least
$14\%$ of both the certified ratio and success rate on ECtHR Task A. In the
two legal judgment prediction tasks, both of our proposed CIESAM and CASAM
achieve the certified ratio over $99\%$, which indicates that they get
extremely close to the standard of being trustworthy under diverse attacks
proposed by experts in the legal domain.

We can observe that CASAM and CIESAM achieve close performance on judgment
prediction tasks. We posit the underlying reason: despite their different
implementations, they satisfy the common theoretical background. CASAM
intervenes in the architecture of the model, breaking the correlation between
$N$ and $C$ in the training procedure, thereby preventing $N$ from correlating
with $Y$, while CIESAM directly controls the input data and focuses on
discarding $N$, thus preventing it from correlating with any other variables.
As we mentioned in Section 2.2.1 that only the correlations generated by
causality are what we expect the models to learn from, both methods focus on
removing other kinds of correlations and only reserving those generated by
causality (learning $P(Y\mid C)$).

Despite the significant robustness improvement under all kinds of attacks, we
explain the reason why the evaluation results (the CR and SR are $91.42\%$ and
$8.58\%$, respectively) of our proposed methods under word-level attacks are
largely different from other attacks on LEDGAR. Although legal judgment
prediction and legal text classification often share common techniques, the
underlying decision rules of the two task is different. Different from LJP
where a judger is required to both perform legal reasoning and consider all of
the circumstances in a case for a just judgment, we rely on fewer words in
legal text classification. For example, if we notice the word ‘vegetables’,
‘fruits’, or ‘agriculture’ in a legal file, we know it probably belongs to the
‘agriculture’ category. If we mask these words, it will even be difficult for
humans to classify the file. Under word-level attacks, these words will
inevitably be masked, leading to distinct evaluation results.

###  4.6 Analysis and Discussion

In this section, we take a step further toward characterizing the spurious
correlation error in the context of the LJP task. We also shed some light on
the underlying reasons why our proposed methods achieve stronger
generalization ability and robustness.

Figure 4: Visualization of selection bias.

Visualization of Selection Bias. To characterize the selection bias, we
investigate the ECtHR Task A dataset as an example and analysis to what extent
the Legal-BERT is affected by the bias. First, we use a feature attribution
method to obtain the top $5\%$ words that are considered most crucial by
Legal-BERT when making a judgment prediction in the test set. Second, we count
the frequency of each word in the top $5\%$ words and in the training set of
ECtHR Task A, respectively. As shown in Figure 4, we can observe three
phenomena: (1) there is an obvious word frequency bias in the training set of
ECtHR Task A; (2) the same kind of bias occurs in the top $5\%$ crucial words
considered by Legal-BERT; (3) the two kinds of frequency exhibit a common
distribution. The first phenomenon, exhibiting a severe bias in the training
set, can lead learning models to suffer from selection bias, which is
demonstrated by the causal structural model in Figure 2 and instantiated by
the second phenomenon. The third phenomenon indicates the fact that, without
any intervention, learning models will faithfully learn the bias distribution
in the training data, which is undesirable for all heuristic learning methods.
If the bias brought by a training set is correlated with gender, race, or
geography, learning models will even trigger severe social problems.

Figure 5: Results on ECtHR Task A.

Effect of Debiasing. To investigate the reason why our proposed methods
possess better generalization ability and robustness, we visualize their
decision rules of predicting judgment in the test set of ECtHR A and B. We
count the frequency of each word occurring in the top $5\%$ words, which are
considered most crucial by Legal-BERT and our proposed CASAM, respectively.
The results are shown in Figure 5 and Figure 6. Our observations are
summarized as follows.

Figure 6: Results on ECtHR Task B.

(1) Without any adjustments in training data or the architecture of the model,
Legal-BERT significantly correlates non-causal information with the judgments.
It predicts judgments through those words that hardly possess any semantic
meanings. The spurious correlations render Legal-BERT vulnerable to attacks
and impede its deployment in real-world legal scenarios: The changes in non-
causal information like writing style (frequently or rarely use these function
words) can even affect the predictions of Legal-BERT. (2) After our
intervention in the architecture of Legal-BERT, it significantly decouples the
non-causal information (e.g., the punctuation marks and function words) and
final predictions, which presents the effectiveness of reducing the
possibility of potential estimates shown in Figure 2. (3) Our intervention
makes Legal-BERT learn new causal information (e.g., content words that indeed
affect the predictions), especially in Figure 6, which indicates that our
proposed methods succeed in learning causal information (the ground-truth
estimate) for predicting by reducing the possibility of other potential
estimates. This explains why our proposed methods achieve both SOTA
generalization ability and robustness.

Note that CASAM can still be aware of non-causal information in some
situations shown in Figure 6 due to the precision of OIE tools, resulting in
the fact that $N$ still cannot be precisely distinguished from $C$. It hampers
more performance gains of our proposed methods. We leave the improvement of
OIE tools for future work.

###  4.7 Case Study

Figure 7 exhibits the case study of our proposed model compared with Legal-
BERT. After merging and discarding the redundant non-causal information, we
retain the causal information to aid the prediction. Specifically, more than
learning the spurious correlation between the token ‘[’ and articles 5, 9, 10,
11, and 14 as shown in Figure 1, Legal-BERT learns the spurious correlation
between ‘the’ and article 5 in this case, which results in predicting judgment
according to non-causal information. Perturbations like missing the token ‘[’
or ‘the’ in fact descriptions can frequently happen in real-world applications
due to the writing habits of a legal assistant. They surprisingly confuse
Legal-BERT to change a prediction from “no crime” and “Violated article 3” to
“violating articles 5, 9, 10, 11, and 14” and “violated articles 3 and 5”,
respectively. The severe spurious correlation error impedes the real-world
application of legal AI. Under the guidance of our proposed causal model, we
merge and discard non-causal information in our proposed method based on
Legal-BERT, which largely mitigates the spurious correlation and learns the
correlations that represent causal relations. Our proposed method is able to
make predictions according to the relevant causal information “detention”,
thereby leading to a right prediction (enhancing generalization ability) and
avoiding focusing on the non-causal information such as “the” (improving
robustness).

Figure 7: Case study

##  5 Conclusion

In this paper, we investigate the decision rule of the legal-specific PLM in
legal AI. We exhibit the potential problems of the decision rules caused by
spurious correlation error, and propose a structural causal model to
theoretically analyze the underlying mechanism. Under the guidance of our
analysis, we propose a method to simultaneously reduce non-causal information
and retain causal information in the given fact descriptions. The experimental
results indicate that spurious correlations between non-causal information and
predictions largely damage the generalization ability and robustness of legal
AI. We appeal to future work to take the spurious correlation error into
consideration for improving the overall performance of legal AI.

## References

  *   * (1) Zhao, Q., Gao, T., Zhou, S., Li, D., Wen, Y.: Legal Judgment Prediction via Heterogeneous Graphs and Knowledge of Law Articles. Applied Sciences 12(5), 2531 (2022). https://doi.org/10.3390/app12052531
  * (2) Cui, J., Shen, X., Nie, F., Wang, Z., Wang, J., Chen, Y.: A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges. arXiv (2022)
  * (3) Lawlor, R.C.: What computers can do: Analysis and prediction of judicial decisions. American Bar Association Journal, 337–344 (1963)
  * (4) Nagel, S.: WEIGHTING VARIABLES IN JUDICIAL PREDICTION. MULL: Modern Uses of Logic in Law 2(3), 93–97 (1960) 29760840
  * (5) Keown, R.: Mathematical models for legal prediction. Computer/lj 2, 829 (1980)
  * (6) Katz, D.M.: Quantitative legal prediction-or-how i learned to stop worrying and start preparing for the data-driven future of the legal services industry. Emory LJ 62, 909 (2012)
  * (7) Aletras, N., Tsarapatsanis, D., Preoţiuc-Pietro, D., Lampos, V.: Predicting judicial decisions of the European Court of Human Rights: A Natural Language Processing perspective. PeerJ Computer Science 2, 93 (2016). https://doi.org/10.7717/peerj-cs.93
  * (8) Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.N.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186 (2018)
  * (9) Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., Androutsopoulos, I.: LEGAL-BERT: The Muppets straight out of Law School. In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2898–2904. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.findings-emnlp.261
  * (10) Zheng, L., Guha, N., Anderson, B.R., Henderson, P., Ho, D.E.: When does pretraining help?: Assessing self-supervised learning for law and the CaseHOLD dataset of 53,000+ legal holdings. In: Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, pp. 159–168. ACM, São Paulo Brazil (2021). https://doi.org/10.1145/3462757.3466088
  * (11) Chalkidis, I., Jana, A., Hartung, D., Bommarito, M., Androutsopoulos, I., Katz, D., Aletras, N.: LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4310–4330. Association for Computational Linguistics, Dublin, Ireland (2022). https://doi.org/10.18653/v1/2022.acl-long.297
  * (12) Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., Wichmann, F.A.: Shortcut learning in deep neural networks. Nature Machine Intelligence 2(11), 665–673 (2020). https://doi.org/10.1038/s42256-020-00257-z
  * (13) Pearl, J.: Causal inference in statistics: An overview. Statistics Surveys 3(none) (2009). https://doi.org/10.1214/09-SS057
  * (14) Cui, P., Athey, S.: Stable learning establishes some common ground between causal inference and machine learning. Nature Machine Intelligence 4(2), 110–115 (2022). https://doi.org/10.1038/s42256-022-00445-z
  * (15) Shen, Z., Liu, J., He, Y., Zhang, X., Xu, R., Yu, H., Cui, P.: Towards Out-Of-Distribution Generalization: A Survey. arXiv (2021)
  * (16) Wiles, O., Gowal, S., Stimberg, F., Rebuffi, S.-A., Ktena, I., Dvijotham, K.D., Cemgil, A.T.: A fine-grained analysis on distribution shift. In: International Conference on Learning Representations (2022)
  * (17) Reed, W.J.: The Pareto, Zipf and other power laws. Economics Letters 74(1), 15–19 (2001). https://doi.org/10.1016/S0165-1765(01)00524-9
  * (18) Yang, X., Zhang, H., Qi, G., Cai, J.: Causal Attention for Vision-Language Tasks. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9842–9852. IEEE, ??? (2021). https://doi.org/10.1109/CVPR46437.2021.00972
  * (19) Stanovsky, G., Michael, J., Zettlemoyer, L., Dagan, I.: Supervised Open Information Extraction. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 885–895. Association for Computational Linguistics, New Orleans, Louisiana (2018). https://doi.org/10.18653/v1/N18-1081
  * (20) Clark, K., Manning, C.D.: Deep Reinforcement Learning for Mention-Ranking Coreference Models. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2256–2262. Association for Computational Linguistics, Austin, Texas (2016). https://doi.org/10.18653/v1/D16-1245
  * (21) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is All you Need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, vol. 30, pp. 5998–6008 (2017)
  * (22) Shao, N., Cui, Y., Liu, T., Wang, S., Hu, G.: Is Graph Structure Necessary for Multi-hop Question Answering? In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7187–7192 (2020). https://doi.org/10.18653/v1/2020.emnlp-main.583
  * (23) Wei, P., Zhao, J., Mao, W.: A Graph-to-Sequence Learning Framework for Summarizing Opinionated Texts. IEEE/ACM Transactions on Audio, Speech, and Language Processing 29, 1650–1660 (2021). https://doi.org/10.1109/TASLP.2021.3071667
  * (24) Cai, D., Lam, W.: Graph Transformer for Graph-to-Sequence Learning. Proceedings of the AAAI Conference on Artificial Intelligence 34(05), 7464–7471 (2020). https://doi.org/10.1609/aaai.v34i05.6243
  * (25) Yao, S., Wang, T., Wan, X.: Heterogeneous Graph Transformer for Graph-to-Sequence Learning. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7145–7154. Association for Computational Linguistics, Online (2020). https://doi.org/%****␣sn-article.bbl␣Line␣450␣****10.18653/v1/2020.acl-main.640
  * (26) Fan, A., Gardent, C., Braud, C., Bordes, A.: Using Local Knowledge Graph Construction to Scale Seq2Seq Models to Multi-Document Inputs. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4184–4194. Association for Computational Linguistics, Hong Kong, China (2019). https://doi.org/10.18653/v1/D19-1428
  * (27) Pasunuru, R., Liu, M., Bansal, M., Ravi, S., Dreyer, M.: Efficiently Summarizing Text and Graph Encodings of Multi-Document Clusters. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4768–4779. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.naacl-main.380
  * (28) Chalkidis, I., Androutsopoulos, I., Aletras, N.: Neural Legal Judgment Prediction in English. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4317–4323. Association for Computational Linguistics, Florence, Italy (2019). https://doi.org/10.18653/v1/P19-1424
  * (29) Tuggener, D., von Däniken, P., Peetz, T., Cieliebak, M.: LEDGAR: A large-scale multi-label corpus for text classification of legal provisions in contracts. In: Proceedings of the 12th Language Resources and Evaluation Conference, pp. 1235–1241. European Language Resources Association, Marseille, France (2020)
  * (30) Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Proceedings of the 2019 Conference of the North, pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota (2019). https://doi.org/10.18653/v1/N19-1423
  * (31) Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019) arxiv:1907.11692
  * (32) He, P., Liu, X., Gao, J., Chen, W.: DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION. In: International Conference on Learning Representations (2021)
  * (33) Beltagy, I., Peters, M.E., Cohan, A.: Longformer: The Long-Document Transformer. ArXiv (2020). https://doi.org/10.48550/ARXIV.2004.05150
  * (34) Zaheer, M., Guruganesh, G., Dubey, K.A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., Ahmed, A.: Big bird: Transformers for longer sequences. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems, vol. 33, pp. 17283–17297. Curran Associates, Inc., ??? (2020)
  * (35) Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., Rush, A.: Transformers: State-of-the-Art Natural Language Processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.emnlp-demos.6
  * (36) Gürel, N.M., Qi, X., Rimanic, L., Zhang, C., Li, B.: Knowledge Enhanced Machine Learning Pipeline against Diverse Adversarial Attacks. arXiv (2022)
  * (37) Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: The all convolutional net. In: ICLR (Workshop Track) (2015)
  * (38) Li, J., Chen, X., Hovy, E., Jurafsky, D.: Visualizing and Understanding Neural Models in NLP. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 681–691. Association for Computational Linguistics, San Diego, California (2016). https://doi.org/10.18653/v1/N16-1082
  * (39) Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: Visualising image classification models and saliency maps. Proceedings of the International Conference on Learning Representations (ICLR), pp. 1–8. ICLR, ??? (2014)
  * (40) Ribeiro, M.T., Singh, S., Guestrin, C.: ”Why Should I Trust You?”: Explaining the Predictions of Any Classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144. ACM, San Francisco California USA (2016). https://doi.org/10.1145/2939672.2939778
  * (41) Shrikumar, A., Greenside, P., Kundaje, A.: Learning important features through propagating activation differences. In: Precup, D., Teh, Y.W. (eds.) Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 70, pp. 3145–3153. PMLR, ??? (2017)
  * (42) Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: Precup, D., Teh, Y.W. (eds.) Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 70, pp. 3319–3328. PMLR, ??? (2017)
  * (43) Zeiler, M.D., Fergus, R.: Visualizing and Understanding Convolutional Networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Computer Vision – ECCV 2014 vol. 8689, pp. 818–833. Springer International Publishing, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_53
  * (44) Feng, S., Wallace, E., Grissom II, A., Iyyer, M., Rodriguez, P., Boyd-Graber, J.: Pathologies of Neural Models Make Interpretations Difficult. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3719–3728. Association for Computational Linguistics, Brussels, Belgium (2018). https://doi.org/10.18653/v1/D18-1407
  * (45) Chen, H., Zheng, G., Ji, Y.: Generating Hierarchical Explanations on Text Classification via Feature Interaction Detection. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5578–5593. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.acl-main.494
  * (46) Li, L., Ma, R., Guo, Q., Xue, X., Qiu, X.: BERT-ATTACK: Adversarial Attack Against BERT Using BERT. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6193–6202. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.emnlp-main.500


# Accelerating deep neural networks for efficient scene understanding in
automotive cyber-physical systems

Stavros Nousias1, Erion Vasilis Pikoulis1,2, Christos Mavrokefalidis1,2, Aris
S. Lalos1 1Industrial Systems Institute, Athena Research Center, Patras
Science Park, Greece
2Computer Engineering and Informatics Dept., University of Patras, Greece
Emails<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Automotive Cyber-Physical Systems (ACPS) have attracted a significant amount
of interest in the past few decades, while one of the most critical operations
in these systems is the perception of the environment. Deep learning and,
especially, the use of Deep Neural Networks (DNNs) provides impressive results
in analyzing and understanding complex and dynamic scenes from visual data.
The prediction horizons for those perception systems are very short and
inference must often be performed in real time, stressing the need of
transforming the original large pre-trained networks into new smaller models,
by utilizing Model Compression and Acceleration (MCA) techniques. Our goal in
this work is to investigate best practices for appropriately applying novel
weight sharing techniques, optimizing the available variables and the training
procedures towards the significant acceleration of widely adopted DNNs.
Extensive evaluation studies carried out using various state-of-the-art DNN
models in object detection and tracking experiments, provide details about the
type of errors that manifest after the application of weight sharing
techniques, resulting in significant acceleration gains with negligible
accuracy losses.

##  I Introduction

In recent years, Cyber-Physical Systems (CPSs) play an important role in
modern technology [1], by interconnecting computational and physical
resources. CPSs are realized by embedded computers and communication networks
that govern physical actuators that operate in the physical world, while
receiving inputs from sensors, thus creating a smart control loop capable of
adaptation, autonomy, and improved efficiency. CPSs have impacted almost all
aspects of our daily life connected with, for instance, transportation
systems, health-care devices, household appliances, electrical power grids,
oil and natural gas distribution and many more. Specifically, in the field of
Intelligent Transportation System (ITS) [2], the use of CPSs can lead to
comprehensive systems that combine advanced technologies with conventional
transportation infrastructures, improving the performance of transportation
systems, enhancing travel security, fuel economy and, ultimately, enhancing
the travel experience of road users.

One of the most essential operations executed at the ITS CPSs to enable the
aforementioned benefits is the perception and understanding of dynamic and
complex environments from multi-modal sensor data. The critical nature of the
perception ability [3] in safety functions for autonomous driving is self-
evident: a deviation of, for example, 30 $cm$ in the estimated lateral
position of the autonomous vehicle can make all the difference between a
“correct” and an “incorrect” (and, potentially, life-threatening) maneuver
initiation. One of the major challenges to be addressed, regards the highly
dynamic behavior of road users (e.g., pedestrians, cyclists, cars), which can
change their motion style in an instance, or start/stop moving abruptly.
Consequently, prediction horizons for active perception systems are typically
short; even so, small performance improvements can produce tangible benefits.
For example, accident analyses [4] show that being able to initiate emergency
braking 0.16 $s$ (i.e. five frames at 33 Hz) earlier, at a time to collision
of 0.66 $s$, reduces the chance of incurring injury requiring a hospital stay
from 50% to 35%, given an initial vehicle speed of 50 $km/h$. The
aforementioned facts clearly indicate the need for fast and effective scene
understanding solutions including, among others, image classification, object
detection, object tracking and semantic segmentation.

Here, we focus on DNN-based object detection and, in particular, on the
application of MCA techniques on high-performance, pre-trained detectors.
Employing MCA techniques can be critical for the efficient execution of the
relevant deep models on embedded devices that are deployed on autonomous
vehicles. In the following, first, the positioning of the paper is provided
through the description of the relevant bibliography and its contribution.
Then, the MCA techniques and the object detectors that are adopted for this
study, are briefly described. Finally, before concluding the paper, a thorough
experimental evaluation of the MCA impact on the behaviour of the adopted
models is presented.

##  II Relevant bibliography and contribution

Object detection has been evolved considerably since the appearance of deep
convolutional neural networks [5]. Nowadays, there are two main branches of
proposed techniques. In the first one, the object detectors, using two stages,
generate region proposals which are subsequently classified in the categories
that are determined by the application at hand (e.g., vehicles, cyclists and
pedestrians, in the case of autonomous driving). Some important,
representative, high performance examples of this first branch are Faster
R-CNN [6], Region-based Fully Convolutional Network (R-FCN) [7], Feature
Pyramid Network (FPN) [8] and Mask R-CNN [9]. In the second branch, object
detection is cast to a single-stage, regression-like task with the aim to
provide directly both the locations and the categories of the detected
objects. Notable examples, here, are Single Shot MultiBox Detector (SSD) [10],
SqueezeDet [11], YOLOv3 [12] and EfficientDet [13].

Although two-stage detectors demonstrate better performance than the single-
stage counterparts, the latter have lower computational and storage
requirements which leads, generally, to faster inference time [14]. In
autonomous driving, Advanced Driver Assistance Systems (ADAS) rely on embedded
systems with limited resources. ADAS is responsible of executing various
machine learning tasks, including object detection, meaning that efficient
implementations that take into account those limitations are critical [15]. To
this end, single-stage detectors have been particularly studied for autonomous
driving by either proposing specialized, compact deep models (e.g., [16],
SqueezeDet [11], SA-YOLOv3 [17], Mini-YOLOv3 [18]) or applying MCA techniques
[19] to existing, pre-trained models (e.g.,[20] [21], [22], Efficient YOLO
[23], ICME 2020 Competition [24]).

The study and development of methodologies and algorithms for the compression
and acceleration of high-performing, yet highly resource demanding deep
models, has been a very active area of research in recent years ([19], [25],
[26], [27]). Three main groups of works have have appeared in the literature.
In the first one, the proposed techniques transform the models by removing /
pruning parts of the neural network (e.g., parameters, connections, channels,
etc. [28], [29]). In the second group, the representation of the involved
parameters is limited via scalar, vector and product quantization, allowing
the so-called weight sharing ([30], [31]). Finally, in the third group, the
involved parameters are transformed via appropriate tensor / matrix
decompositions that impose low-rankness, sparsity, etc., [32].

Most of the MCA techniques that have been applied for the problem of object
recognition (as the ones mentioned above), belong either to pruning or scalar
quantization, which currently are supported by toolboxes like the TensorFlow
Model Optimization Toolkit. Here, moving a step further, we focus on more
elaborate and high-performing MCA techniques that belong to weight sharing
[33], [34] and study their impact on the performance of object detection for
autonomous driving. The contributions of the paper are as follows:

  * •

Two weight sharing techniques are employed for the compression / acceleration
of two object detection deep models that are based on the well-known ResNet50
and on SqueezeNet DNNs [11].

  * •

An analysis is provided on the error types that manifest after the application
of weight sharing techniques.

  * •

The results obtained on the KITTI dataset using the selected DNN models,
reveal acceleration gains of up to $70\%$ with negligible accuracy loss.

##  III Weight sharing via Product Quantization

Generally speaking, the linear operation carried out by the convolutional
layers can be viewed as involving the computation of dot-products between
input and kernel vectors lying in an $N$-dimensional space, with $N$ being the
number of input/kernel channels.

Product quantization first partitions the original $N$-dimensional vector
space into a number of subspaces and subsequently performs vector quantization
in each of them. Specifically, it approximates (represents) the original sub-
vectors lying in each subspace using a codebook of limited size. In doing so,
product quantization approximates the original dot-products between the input
and kernel sub-vectors, by the ones between the input and the
representatives/codewords (whose number is much smaller), hence the great
potential for acceleration.

Conventionally, vector quantization is achieved by clustering the sub-vectors
using the popular $k$-means algorithm (essentially treating them as data
points lying in the corresponding subspace), and employing the cluster
centroids as the desired codewords. However, a recently proposed approach that
treats the problem from a Dictionary Learning perspective, has shown very
promising results [34], achieving up to 100 $\%$ (or, $2\times$) acceleration
gain over conventional techniques, on state-of-the-art pre-trained models
(VGG, ResNet, SqueezeNet) from the ImageNet competition.

The new approach presented in [34] enables the use of a codebook that is
several times larger than the ones obtained via $k$-means-based approaches
(for the same target acceleration), which leads to considerable improvement
regarding the incurred the quantization error. This is achieved by imposing a
special structure to the learned codewords, using a Dictionary-Learning
framework.

More specifically, let us define the conventional approximation scheme
(referred to as VQ hereafter) regarding the kernel sub-vectors of a particular
subspace, as follows:

$\mathbf{W}\approx\mathbf{C}\mathbf{\Gamma},$ (1)

where $\mathbf{W}$, $\mathbf{C}$, denote the matrices holding the original
sub-vectors, and the codewords (cluster centroids), respectively, in their
columns, while each column of $\mathbf{\Gamma}$ is essentially a one-hot
encoding of the codewords in $\mathbf{C}$ (i.e. one element is equal to $1$
and all others are equal to $0$). Thus, according to (1), each original sub-
vector (column of $\mathbf{W}$) is approximated by exactly one of the $K_{vq}$
codewords (columns of $\mathbf{C}$).

On the other hand, the newly proposed approach (referred to as DL hereafter)
is based on a different codebook structure, namely:

$\mathbf{W}\approx\mathbf{D}\mathbf{\Lambda}\mathbf{\Gamma},$ (2)

where $\mathbf{W}$ and $\mathbf{\Gamma}$ are as in (1), while $\mathbf{D}$ and
$\mathbf{\Lambda}$ denote the dictionary of normalized atoms, and the matrix
of sparse coefficients, respectively. Specifically, each column of
$\mathbf{\Lambda}$ contains $\rho$ non-zero elements, with $\rho$ being the
sparsity level. Thus, according to the DL-based approximation defined in (2),
each of the codewords contained in codebook $\mathbf{D}\mathbf{\Lambda}$, is
obtained as a linear combination of $\rho$ atoms from $\mathbf{D}$. The
codebook size (columns of $\mathbf{\Lambda}$) in this approximation is denoted
as $K_{dl}$ while the dictionary size (columns of $\mathbf{D}$) as $L_{dl}$,
with $L_{dl}<K_{dl}$.

The main advantage of the DL approach lies in its ability to employ codebooks
that are much larger in size than the ones used by the VQ approach, for the
same target acceleration. This owes to the linearity of the involved dot-
products, and comes as a direct consequence of the special structure of DL-
based codebook, namely its decomposition into a dense dictionary $\mathbf{D}$
and a sparse matrix $\mathbf{\Lambda}$. To be more specific, this endows it
with the ability to increase the size of $\mathbf{\Lambda}$ (thus increasing
the codebook size) while at the same time limiting the dictionary size (thus
restricting the number of dense dot-products). Equivalently, this results in
significant acceleration gains (compared to the VQ approach) for the same
quantization error, as shown in [34].

##  IV Application on widely adopted DNN models

Two deep detection network architectures, namely SqueezeDet and
Resnet50ConvDet, were employed for the evaluation of the presented weight
sharing approach. They are fully convolutional detection networks presented by
Wu et al. [11], consisting of a feature-extraction part that extracts high
dimensional feature maps for the input image, and ConvDet, a convolutional
layer to locate objects and predict their class. For the derivation of the
final detection, the output is filtered based on a confidence index also
extracted by the ConvDet layer. Figure 1 presents the overall architecture of
the deep networks, the convolutional volume kernel shapes and the feature
tensor shapes.

As it can be observed from Fig. 1(a), the feature-extraction (convolutional)
part of SqueezeDet is based on SqueezeNet [35], which is a fully convolutional
neural network that employs a special architecture that drastically reduces
its size while still remaining within the state-of-the-art performance
territory. Its building block is the “fire” module that consists of a
“squeeze” $1\times 1$ convolutional layer with the purpose of reducing the
number of input channels, followed by $1\times 1$ and $3\times 3$ “expand”
convolutional layers that are connected in parallel to the “squeezed” output.
SqueezeNet consists of $8$ such modules connected in series.

On the other hand, the backbone of ResNetDet is based on the convolutional
layers of ResNet50 [36], whose building block consists of three layers,
stacked one over the other, as depicted in Fig. 1(b). The three layers are
$1\times 1$, $3\times 3$, $1\times 1$ convolutions. The $1\times 1$
convolution layers are responsible for reducing and then restoring the
dimensions. The $3\times 3$ layer is left as a bottleneck with smaller
input/output dimensions. The convolutional part of ResNetDet consists of $13$
such blocks.

(a)

(b)

Figure 1: Architectures of the employed detector networks (a) SqueeezeDet, and
(b) ResNetDet. The convolutional layers highlighted by the red frames
constitute the target layers in our acceleration experiments. $B$ is the batch
size, $H$ the height and $W$ the width of a volume kernel. $C_{L-1}$ is the
number of channels of the previous layer.

##  V Experimental evaluation

###  V-A Training

Both networks were trained with the KITTI odometry dataset [37] consisting of
7477 color traffic scenes images of $1242\times 375$ pixels. Three classes are
taken into account, namely, cyclists, pedestrians and cars which were manually
annotated with bounding boxes containing the objects in the scene. A
significant observation regarding the dataset is that not all objects of the
same class are labeled in each and every image. Such a fact plays a role in
the evaluation of the detection outcome as our analysis will reveal. The
dataset was split in a $80\%,20\%$ for training and validation, respectively,
resulting in $N_{tr}=5980$ training examples and $N_{val}=1497$ validation
examples.

For the training of the SqueezeDet architecture, Stochastic Gradient Descent
(SGD) was employed with the following values for the hyperparameters
(determined via experimentation); batch size $B=8$, learning rate
$LR=10^{-4}$, with a weight decay rate $D_{W}=10^{-4}$, a learning rate decay
rate of $D_{LR}=2*LR/N_{e}$, number of steps $N_{s}=3\times N_{tr}$ and a
dropout rate of $50\%$, over a total of $N_{e}=300$ epochs. Training and
testing took place in an NVIDIA GeForce GTX 1080 graphics card with 8GB VRAM
and compute capability $6.1$ in a Intel(R) Core(TM) i7-4790 CPU @ 3.60Hz based
system with 32GB of RAM.

Likewise, for Resnet50ConvDet, we also employed SGD with hyperparameter values
as in the case of SqueezeDet. Training and evaluation of Resnet50ConvDet took
place in an NVIDIA GeForce Geforce RTX 2080 with 16GB VRAM and compute
capability $7.5$ in a Intel(R) Core(TM) i7-4790 CPU @ 3.60Hz based system with
16GB of RAM.

In all cases, training took place with a data augmentation scheme where the
bounding boxes drift by $k_{x}*150$ pixels across the x-axis and $k_{y}*150$
pixels across the y-axis, where $k_{x},k_{y}\sim U(0,1)$. A $50\%$ probability
is also assumed to flip the object.

###  V-B Acceleration scheme

In our experiment, we apply the rival techniques to the two detection models
in a “full-model” acceleration scenario. It involves accelerating multiple (or
all) convolutional layers of the original models and measuring the achieved
performance of the accelerated networks. The reported acceleration ratios are
defined as the ratio of original vs accelerated computational complexities,
measured by required multiply-accumulate (MAC) operations.

The full-model acceleration of the involved networks is achieved by following
the progressive strategy proposed in [33], whereby the individual layers are
quantized sequentially in stages, having the original network as a starting
point. The quantization operation is followed by a fine-tuning step involving
the remaining original layers, after each stage. Fine-tuning and performance
evaluation, are based on the training and validation datasets from KITTI,
respectively, as previously explained.

#### Accelerating SqueezeDet

The feature-extraction part of SqueezeDet, namely SqueezeNet, is responsible
for roughly $83\%$ of the total $5.3\times 10^{9}$ MAC operations required.
Since SqueezeNet is specifically designed for efficiency, and in order to
maintain a good balance between acceleration and performance, in our
experiments we only targeted the “expand” layers of the network, as shown in
Fig. 1(a). Acceleration was performed in $8$ acceleration stages, with each
stage involving a particular “expand” modules (followed by fine-tuning). Using
acceleration ratios of $\alpha=8$, $10$, $12$, and $20$ on the targeted
layers, an acceleration of the SqueezeNet part by $72\%$, $74\%$, $75\%$, and
$78\%$, respectively, and a total model acceleration by $59\%$, $60\%$,
$62\%$, and $65\%$, respectively, were achieved.

#### Accelerating ResNetDet

The feature extraction part of ResNetDet is responsible for roughly $81\%$ of
the total $3.5\times 10^{10}$ MACs required by the network. Following the
network’s architecture, in our experiments with ResNetDet, we accelerated its
convolutional (feature-extraction) blocks in a one-block-per-stage fashion
leading to $13$ total acceleration stages. Using acceleration ratios of
$\alpha=8$, $10$, $12$, and $20$ on the targeted layers (see Fig. 1(b)), an
acceleration of the feature-extraction part by $84\%$, $86\%$, $88\%$, and
$92\%$, respectively, and a total model acceleration by $67\%$, $69\%$,
$71\%$, and $74\%$, respectively, were achieved.

###  V-C Metrics

For each detection, the Intersection Over Union (IOU) score is computed as the
ratio of area of intersection to the area of union between the predicted and
ground-truth bounding boxes. A true positive occurs when IOU$>0.5$ and the
predicted class is the same as the ground-truth class. A false positive occurs
when IOU$<0.5$ or a different class is detected, meaning that unmatched
bounding boxes are taken as false positives for a given class. Precision,
recall and mean average precision (mAP) are subsequently calculated according
to [38].

###  V-D Results

The progressive, stage-wise acceleration results for the employed networks,
using both the VQ and the DL acceleration techniques for various acceleration
ratios, are shown in Fig. 2. The rightmost point in every plot depicts the
performance of the “fully” accelerated network, i.e., after all targeted
convolutional layers have been accelerated. At each point, the performance of
the detectors was assessed based on the achieved mean average precision (mAP)
and recall.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 2: Performance evaluation and comparison of DL vs VQ acceleration
techniques on ResNetDet (top row) and SqueezeDet (bottom row).

As a general comment, the results presented in Fig. 2 reveal a very promising
performance by the employed weight-sharing techniques, and, especially so, for
the DL-based one, whose application results in significantly accelerated
detectors, with limited loss of their detection capabilities, as expressed by
both the mAP and recall values. We stress at this point that the obtained
acceleration gains can be further enhanced by better configuring the MCA
methodology, so that it is tailored to the specific architecture of the deep
model to be transformed. This involves, for instance, careful selection of the
layers to be quantized, the amount of compression/acceleration ratio per layer
(based on the sensitivity of the layer), etc. Moreover, comparatively
speaking, the DL-based technique managed to generally outperform its rival in
our experiments, as highlighted by the plots presented in Fig. 2, for an
acceleration ratio of $a=10$ (Fig. 2(a)$\&$(c), and (e)$\&$(g), for ResNetDet,
and SqueezeDet, respectively).

Application instances of the accelerated versus the original networks using
examples from the KITI dataset are shown in Fig. 3, respectively.

(a) Open street junction

(b) Narrow street

Figure 3: Application of accelerated vs original SqueezeDet models, using
examples from the KITTI dataset. Green rectangles correspond to ground truth
boxes, while red rectangles to predictions. The confidence scores are also
shown in red letters. Yellow rectangles in (a) and yellow dot in (b) highlight
the most obvious performance degradation of the accelerated networks, as
compared to the original one.

###  V-E Error-type analysis

For a better insight on the obtained results, we performed an in-depth
analysis of the error-types of the employed detectors. For this analysis, we
examined 64 images containing the groundtruth annotation and the detection
outcome and classified the errors into seven categories; a) object located but
not labeled in dataset, b) object located buy bounding box not in place
($IOU<0.5$), c) object located but overlapping double bounding box appeared,
d) non existent object located, e) object not located due to occlusion, f)
object not located at all, and g) mirrored object (i.e., on glass surface),
object located but in wrong class. Furthermore, we manually classified the 64
images into clear scenes with sufficient light and no occlusions, and messy
scenes with many objects some of them being occluded. The motivation behind
this perspective is that the detector correctly detects an object but it is
assumed as an error or the detector correctly misses an object (i.e.,
occlusion) but it is assumed as an error since it was originally annotated in
the dataset. As we can observe $50\%$ of the errors in the examined images,
are objects that were actually found but either they were not annotated or
there was a bounding box issue. The results of this qualitative analysis are
summarized by the bar-charts shown in Fig. 4.

Figure 4: Error type analysis using manually evaluated examples.

(a)

(b)

(c)

(d)

Figure 5: Examples of error types. Green boxes refer to groundtruth data,
while red boxes to detections.

##  VI Conclusions

This work investigates the acceleration benefits of weight sharing methods in
deep learning based scene analysis for automotive CPSs. Best practices for
optimizing the available variables and the training procedures are described
based on extensive evaluations on the KITTI dataset. The presented results
provide details about the type of errors that manifest, resulting in
significant acceleration gains with negligible accuracy losses. By inspecting
the error analysis it can be easily seen that most of the errors are
attributed to annotation uncertainties. A more thorough investigation that
utilizes also synthetic datasets generated from the CARLA autonomous driving
simulator is currently under investigation and it is expected to alleviate the
impact of the uncertainties to the training and validation errors, providing
additional space for acceleration gains.

## Acknowledgement

This paper has received funding from H2020 project CPSoSaware (No 873718) and
the DEEP-EVIoT - Deep embedded vision using sparse convolutional neural
networks project (MIS 5038640) implemented under the Action for the Strategic
Development on the Research and Technological Sector, co-financed by national
funds through the Operational programme of Western Greece 2014-2020 and
European Union funds (European Regional Development Fund).

## References

  * [1] S. Zanero, “Cyber-physical systems,” _Computer_ , vol. 50, no. 4, pp. 14–16, 2017.
  * [2] J. Zhang, F. Wang, K. Wang, W. Lin, X. Xu, and C. Chen, “Data-driven intelligent transportation systems: A survey,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 12, no. 4, pp. 1624–1639, 2011.
  * [3] J. Zhang and D. Tao, “Empowering things with intelligence: A survey of the progress, challenges, and opportunities in artificial intelligence of things,” _IEEE Internet of Things Journal_ , 2020.
  * [4] J. Lenard, R. Danton, M. Avery, A. Weekes, D. Zuby, and M. Kühn, “Typical pedestrian accident scenarios for the testing of autonomous emergency braking systems,” in _Enhanced Safety of Vehicles (ESV) Conference, Washington, USA_ , 2011.
  * [5] Z.-Q. Zhao, P. Zheng, S.-t. Xu, and X. Wu, “Object detection with deep learning: A review,” _IEEE transactions on neural networks and learning systems_ , vol. 30, no. 11, pp. 3212–3232, 2019.
  * [6] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: towards real-time object detection with region proposal networks,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 6, pp. 1137–1149, 2016\.
  * [7] J. Dai, Y. Li, K. He, and J. Sun, “R-fcn: Object detection via region-based fully convolutional networks,” _arXiv preprint arXiv:1605.06409_ , 2016.
  * [8] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2117–2125.
  * [9] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2961–2969.
  * [10] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in _European conference on computer vision_. Springer, 2016, pp. 21–37.
  * [11] B. Wu, F. Iandola, P. H. Jin, and K. Keutzer, “Squeezedet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2017, pp. 129–137.
  * [12] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” _arXiv preprint arXiv:1804.02767_ , 2018.
  * [13] M. Tan, R. Pang, and Q. V. Le, “Efficientdet: Scalable and efficient object detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 10 781–10 790.
  * [14] E. Yurtsever, J. Lambert, A. Carballo, and K. Takeda, “A survey of autonomous driving: Common practices and emerging technologies,” _IEEE Access_ , vol. 8, pp. 58 443–58 469, 2020.
  * [15] J. Borrego-Carazo, D. Castells-Rufas, E. Biempica, and J. Carrabina, “Resource-constrained machine learning for adas: A systematic review,” _IEEE Access_ , vol. 8, pp. 40 573–40 598, 2020.
  * [16] A. Kozlov and D. Osokin, “Development of real-time adas object detector for deployment on cpu,” in _Proceedings of SAI Intelligent Systems Conference_. Springer, 2019, pp. 740–750.
  * [17] D. Tian, C. Lin, J. Zhou, X. Duan, Y. Cao, D. Zhao, and D. Cao, “Sa-yolov3: An efficient and accurate object detector using self-attention mechanism for autonomous driving,” _IEEE Transactions on Intelligent Transportation Systems_ , 2020.
  * [18] Q.-C. Mao, H.-M. Sun, Y.-B. Liu, and R.-S. Jia, “Mini-yolov3: real-time object detector for embedded applications,” _IEEE Access_ , vol. 7, pp. 133 529–133 538, 2019.
  * [19] L. Deng, G. Li, S. Han, L. Shi, and Y. Xie, “Model compression and hardware acceleration for neural networks: A comprehensive survey,” _Proceedings of the IEEE_ , vol. 108, no. 4, pp. 485–532, 2020.
  * [20] N. Krittayanawach and P. Vateekul, “Robust compression technique for yolov3 on real-time vehicle detection,” in _2019 11th International Conference on Information Technology and Electrical Engineering (ICITEE)_. IEEE, 2019, pp. 1–6.
  * [21] J. Xu, Y. Nie, P. Wang, and A. M. López, “Training a binary weight object detector by knowledge transfer for autonomous driving,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 2379–2384.
  * [22] H.-H. Nguyen, D. N.-N. Tran, and J. W. Jeon, “Towards real-time vehicle detection on edge devices with nvidia jetson tx2,” in _2020 IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia)_. IEEE, 2020, pp. 1–4.
  * [23] Z. Wang, J. Zhang, Z. Zhao, and F. Su, “Efficient yolo: A lightweight model for embedded deep learning object detection,” in _2020 IEEE International Conference on Multimedia & Expo Workshops (ICMEW)_. IEEE, 2020, pp. 1–6.
  * [24] C.-C. Tsai, Y.-H. Yang, H.-W. Lin, B.-X. Wu, E. C. Chang, H. Y. Liu, J.-S. Lai, P. Y. Chen, J.-J. Lin, J. S. Chang, _et al._ , “The 2020 embedded deep learning object detection model compression competition for traffic in asian countries,” in _2020 IEEE International Conference on Multimedia & Expo Workshops (ICMEW)_. IEEE, 2020, pp. 1–6.
  * [25] V. Sze, Y.-H. Chen, T.-J. Yang, and J. S. Emer, “Efficient processing of deep neural networks: A tutorial and survey,” _Proceedings of the IEEE_ , vol. 105, no. 12, pp. 2295–2329, Dec. 2017.
  * [26] Y. Cheng, D. Wang, P. Zhou, and T. Zhang, “Model compression and acceleration for deep neural networks: The principles, progress, and challenges,” _IEEE Sig. Proc. Mag._ , vol. 35, pp. 126–136, 2018.
  * [27] Q. Zhang, M. Zhang, T. Chen, Z. Sun, Y. Ma, and B. Yu, “Recent advances in convolutional neural network acceleration,” _Neurocomputing_ , vol. 323, pp. 37–51, 2019.
  * [28] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,” _arXiv:1608.08710_ , 2016.
  * [29] S. Lin, R. Ji, Y. Li, C. Deng, and X. Li, “Toward compact convnets via structure-sparsity regularized filter pruning,” _IEEE transactions on neural networks and learning systems_ , 2019.
  * [30] S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, “Deep learning with limited numerical precision,” in _International Conference on Machine Learning_ , 2015, pp. 1737–1746.
  * [31] Y. Gong, L. Liu, M. Yang, and L. Bourdev, “Compressing deep convolutional networks using vector quantization,” _arXiv:1412.6115_ , 2014.
  * [32] S. Bhattacharya and N. D. Lane, “Sparsification and separation of deep learning layers for constrained resource inference on wearables,” in _Proceedings of the 14th ACM Conference on Embedded Network Sensor Systems CD-ROM_ , 2016, pp. 176–189.
  * [33] J. Cheng, J. Wu, C. Leng, Y. Wang, and Q. Hu, “Quantized cnn: A unified approach to accelerate and compress convolutional networks,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 29, no. 10, pp. 4730–4743, 2018.
  * [34] E.-V. Pikoulis, C. Mavrokefalidis, and A. S. Lalos, “A new clustering-based technique for the acceleration of deep convolutional networks,” in _Proceedings of the IEEE International Conference on Machine Learning and Applications (ICMLA)_ , 2020.
  * [35] F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer, “Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size,” _arXiv preprint arXiv:1602.07360_ , 2016.
  * [36] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [37] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “The kitti vision benchmark suite,” _URL http://www. cvlibs. net/datasets/kitti_ , vol. 2, 2015.
  * [38] D. M. Powers, “Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation,” _arXiv preprint arXiv:2010.16061_ , 2020.