# Learning to Rank when Grades Matter

Le Yan, Zhen Qin, Gil Shamir, Dong Lin, Xuanhui Wang, Mike Bendersky
Google1600 Amphitheater PkwyMountain ViewCA 94043 lyyanle, zhenqin, gshamir,
dongl, xuanhui<EMAIL_ADDRESS>
###### Abstract.

Graded labels are ubiquitous in real-world learning-to-rank applications,
especially in human rated relevance data. Traditional learning-to-rank
techniques aim to optimize the ranked order of documents. They typically,
however, ignore predicting actual grades. This prevents them from being
adopted in applications where grades matter, such as filtering out “poor”
documents. Achieving both good ranking performance and good grade prediction
performance is still an under-explored problem. Existing research either
focuses only on ranking performance by not calibrating model outputs, or
treats grades as numerical values, assuming labels are on a linear scale and
failing to leverage the ordinal grade information. In this paper, we conduct a
rigorous study of learning to rank with grades, where both ranking performance
and grade prediction performance are important. We provide a formal discussion
on how to perform ranking with non-scalar predictions for grades, and propose
a multiobjective formulation to jointly optimize both ranking and grade
predictions. In experiments, we verify on several public datasets that our
methods are able to push the Pareto frontier of the tradeoff between ranking
and grade prediction performance, showing the benefit of leveraging ordinal
grade information.

Learning to Rank; Ordinal Regression; Multiobjective Optimization

††ccs: Information systems Information retrieval

##  1\. Introduction

Learning to rank (LTR) with graded labels is ubiquitous in real-world
applications. For example, in traditional LTR datasets such as Web30K, human
raters rate each query-document pair from “irrelevant” (graded as 0) to
“perfectly relevant” (graded as 4). Grades are _ordinal_ , i.e., represented
by discrete numbers with a natural order, but not necessarily preserving
numerical relations. For example, grade 4 is not necessarily twice as relevant
as grade 2. Traditional LTR work focuses on ranking performance or treats
grades as numerical values (Yan et al., 2022), ignoring potential non-
linearity of the grading scale. Predicting actual grades is traditionally
treated as a classification problem, which has not been given much attention
in the LTR literature (Li et al., 2007), and that usually ignores the order of
the grades. Unlike classical LTR work, we consider the problem in which both
ranking performance and grade prediction performance, measured by ranking
metrics and classification accuracy, respectively, are both important. We
argue that achieving good performance on both fronts delivers a better user
facing experience via optimal ranking _and_ capabilities such as filtering out
“poor” documents with certain grades. For example, user could choose to show
just perfectly relevant results or any relevant results when grade predictions
are available.

In the sequel, we present a rigorous study of LTR with graded labels. We
formally demonstrate ranking with non-scalar predictions for grades. Based on
ordinal prediction aggregation, we propose a multiobjective formulation that
directly trades-off ranking and grade prediction. We conduct an extensive
experimental study on 3 public LTR datasets, comparing with state-of-art
ranking methods, and ranking-agnostic classification methods. Experimental
results show interesting trade-off behaviors of different methods. Our
proposed methods are able to push the Pareto frontier of ranking and grade
prediction performances.

##  2\. Related Works

LTR has been widely studied with focus on designing losses and optimization
methods to improve ranking performance. Several notable losses include
Pairwise Logistic (Burges et al., 2005) (also called RankNet) and ListNet (Xia
et al., 2008). Subsequent work included multiple perspectives to optimize
ranking metrics. These include LambdaRank (Burges, 2010), SoftNDCG (Taylor et
al., 2008), SmoothNDCG (Chapelle and Wu, 2010), ApproxNDCG (Qin et al., 2010;
Bruch et al., 2019) and GumbelApproxNDCG (Bruch et al., 2020), among many
others. A recent work (LambdaLoss (Jagerman et al., 2022)) used ideas from
LambdaRank to develop a theoretically sound framework for neural optimization
of ranking metrics.

Ranking methods studied in the LTR literature focus on improving ordering, but
not on prediction accuracy of the actual labels (or grades). Previous work (Li
et al., 2007) studied if accurate label predictions could lead to good
ranking, but not directly optimizing both objectives. Multi-objective setting
has been well studied in Gradient Boosting Decision Trees (Chapelle et al.,
2010; Svore et al., 2011; Carmel et al., 2020), but little attention has been
paid to the two objectives we are considering. Calibrated LTR, where model
predictions are anchored to concrete meanings, has also drawn some attention
due to its practical value (Tagami et al., 2013; Chaudhuri et al., 2017; Yan
et al., 2022). However, existing work treats grades as real values, assuming
that grade values are on a linear scale. This is inaccurate for many
applications where the grades are ordinal and discrete, but not linear. To the
best of our knowledge, our work is the first to formally study and demonstrate
benefit on various tasks of learning to rank with graded labels when
prediction of the labels matter.

##  3\. Problem Formulation

We consider a ranking dataset with graded documents,

$\mathcal{D}=\\{\\{q,\\{{\bf
x}_{i},y_{i}\\}|i\in\mathcal{D}_{q}\\}|q\in\mathcal{Q}\\}.$

Dataset $\mathcal{D}$ consists of queries $q\in\mathcal{Q}$, each associated
with a set of candidate documents $\mathcal{D}_{q}$. Document $i$ is featured
by ${\bf x}_{i}$ and graded label $y_{i}$. Without loss of generality, we
assume $y_{i}\in\\{0,1,...,L-1\\}$ for $L$ possible ordinal classes. The
ordinal relevance relation aligns with the integer order. The graded labels in
the setting play two aligned roles: (1) they define the ordinal categories
that a document appears in a query; and (2) presenting the list of documents
in descending order of the grades optimizes ranking performance.

Conventionally, optimization focuses on one of two objectives: (1) to predict
the correct category of each query document pair; or (2) to exploit the
correct ranking regardless of the category predictions. Ideally, as the
perfect ranking can be achieved by sorting the grades, i.e., perfect category
predictions indicate perfect ranking, optimizing (1) is sufficient to reach
(2). In practice, however, directly optimizing (2) usually leads to much
better ranking performance. In this work, we consider a formulation to jointly
optimize the two objectives.

###  3.1. Ordinal grade prediction

We aim to predict correct graded labels for query-document pairs.

#### Mean squared error.

A naive and straight-forward way is to cast ordinal classes as real values and
then apply linear regression. We consider a parametric model that predicts a
real value for each query-document pair minimizing mean squared error between
the model prediction $f_{\theta}({\bf x}_{i})$ and the graded label $y_{i}$,

(1) $\mathcal{L}^{\rm MSE}(\mathcal{D})=\sum_{i}\left(f_{\theta}({\bf
x}_{i})-y_{i}\right)^{2}.$

The model converges to the expected $y_{i}$, and we can pick the grade that
minimizes the distance to the model’s prediction,

(2) $\hat{y}_{i}={\rm argmin}_{l=0,...,L-1}|l-f_{\theta}({\bf x}_{i})|.$

An implicit assumption is that the grade scale is well calibrated. Thus,
differences in relevance are equal if differences in labels are equal.
However, this may not be the case for every graded dataset.

#### Multi-class cross entropy.

Making predictions of graded categories can be seen as a multi-class
classification problem, and the presumption above is no longer needed. The
model predictions, ${\bf f}_{\theta}({\bf x}_{i})$, with $L$ logits for $L$
grades, can be transformed to normalized probabilities with a softmax
function,

(3) $p(y_{i}=l|{\bf x}_{i})=\frac{\exp({f}^{l}_{\theta}({\bf
x}_{i}))}{\sum_{j}\exp({f}^{j}_{\theta}({\bf x}_{i}))}.$

The superscript $l$ labels the $l$-th component of the predictions. The model
is trained to minimize cross-entropy loss,

(4) $\mathcal{L}^{\rm
CE}(\mathcal{D})=-\sum_{i}\sum_{l=0}^{L-1}\mathbb{I}(y_{i}=l)\ln(p(y_{i}=l|{\bf
x}_{i})),$

where $\mathbb{I}(y_{i}=l)$ is the indicator function of item $i$ taking label
$l$. Given the model predicted probabilities of each ordinal category, we
naturally use the label maximizing the corresponding probability as the
predicted ordinal grade,

(5) $\hat{y}_{i}={\rm argmax}_{l=0,...,L-1}p(y_{i}=l|{\bf x}_{i}).$

The multi-class cross entropy approach ignores the ordinal relation of grades,
which could possibly be leveraged in training. For example, if a document is
not likely in grade $l$ or higher, then it is less likely in grade $l+1$ or
higher. Ordinal regression methods have been applied to leverage this
relation.

#### Univariate ordinal regression.

Univariate ordinal regression leverages ordinal relations by mapping ordinal
grades into consecutive regions on the real axis. $L-1$ variables $\phi_{1}$,
$\phi_{2}$, $...$, $\phi_{L-1}$, constrained to $\phi_{l}\leq\phi_{m}$ iff
$l<m$, are trained as class boundaries for the full dataset (or slices of it).
Together with $\phi_{0}=-\infty$ and $\phi_{L}=\infty$, the $L+1$ boundaries
partition the real axis into $L$ consecutive regions. A model learns a per-
item shift $f_{\theta}(\mathbf{x}_{i})$ for the grid of boundaries. Fitting
the shifted boundaries to an infinite support _probability density function
(PDF)_ renders the integral over each region as the class probability (where
integrating from $-\infty$ to a shifted boundary gives the _cumulative density
function (CDF)_ of an item up to some label class). Fitting a _logistic PDF_
gives probability,

(6) $p(y_{i}\geq l|{\bf x}_{i})=\frac{1}{1+\exp(-[f_{\theta}({\bf
x}_{i})-\phi_{l}])}$

for item $i$ belonging to class $l$ or greater. Thus,

(7) $p(y_{i}=l|{\bf x}_{i})=p(y_{i}\geq l|{\bf x}_{i})-p(y_{i}\geq l+1|{\bf
x}_{i})\\\ =\frac{1}{1+\exp(-[f_{\theta}({\bf
x}_{i})-\phi_{l}])}-\frac{1}{1+\exp(-[f_{\theta}({\bf x}_{i})-\phi_{l+1}])}$

is the probability of $i$ taking label $l$. With the probability in Eq. (7),
the model $f_{\theta}$ and boundaries $\\{\phi_{l}\\}$ are trained to minimize
the cross entropy loss in Eq. (4).

#### Multivariate ordinal regression.

Multivariate ordinal regression, see also in Ref. (Pobrotyn et al., 2020),
leverages the ordinal relations by dividing the $L$-level ordinals into $L-1$
successive binary classifications, which learn $L-1$ values
$f^{l}_{\theta}({\bf x}_{i})$, each with logistic regression, giving

(8) $p(y_{i}\geq l|{\bf x}_{i})=\frac{1}{1+\exp(-f^{l}_{\theta}({\bf
x}_{i}))},\quad{\rm for}\ l=1,2,...,L-1,$

with $p(y_{i}\geq 0|{\bf x}_{i})=1$ and $p(y_{i}\geq L|{\bf x}_{i})=0$. Then,

(9) $p(y_{i}=l|{\bf x}_{i})=\frac{1}{1+\exp(-f^{l}_{\theta}({\bf
x}_{i}))}-\frac{1}{1+\exp(-f^{l+1}_{\theta}({\bf x}_{i}))}.$

The multivariate ordinal regression trains the model to minimize the sum of
the $L-1$ consecutive logistic losses,

(10) $\mathcal{L}^{\rm
Ord}(\mathcal{D})=-\sum_{i}\sum_{l=1}^{L-1}\left[\mathbb{I}(y_{i}\geq
l)\ln(p(y_{i}\geq l|{\bf x}_{i}))\right.\\\
\left.+\mathbb{I}(y_{i}<l)\ln(1-p(y_{i}\geq l|{\bf x}_{i}))\right].$

Both univariate and multivariate ordinal methods could predict the grade using
max probability, as in Eq. (5).

Table 1. The statistics of the three largest public benchmark datasets for LTR
models.

| #features | #queries | avg. | grade ratio (%)
---|---|---|---|---
|  | training | validation | test | #docs | 0 | 1 | 2 | 3 | 4
Web30K | 136 | 18,919 | 6,306 | 6,306 | 119 | 51.4 | 32.5 | 13.4 | 1.9 | 0.8
Yahoo | 700 | 19,944 | 2,994 | 6,983 | 24 | 26.1 | 35.9 | 28.5 | 7.6 | 1.9
Istella | 220 | 20,901 | 2,318 | 9,799 | 316 | 96.3 | 0.8 | 1.3 | 0.9 | 0.7

###  3.2. Ranking prediction

LTR methods usually care only about the ranking of documents in the same list,
and can be insensitive to the absolute values of predictions. In the most
popular state-of-the-art ranking methods, as introduced below, the model
predicts a ranking score, $s_{i}=f_{\theta}({\bf x}_{i})\in\mathbb{R}$, for
each query-document pair, and the documents in the same query list are then
ranked by sorting their scores.

#### Lambda loss.

As ranking performance is usually measured by ranking metrics, some methods
directly optimize these metrics or corresponding surrogates. The Lambda loss
(Burges, 2010; Jagerman et al., 2022) is an example, where we reweight the
gradient of each pair in a pairwise logistic loss (Burges et al., 2005) by the
difference between the ranking metric to its value when flipping the pair. To
optimize the _Normalized Discounted Cumulative Gain (NDCG)_ metric (Järvelin
and Kekäläinen, 2002), we apply

(11) $\mathcal{L}^{\rm
Lambda}(\mathcal{D})=-\sum_{q\in\mathcal{Q}}\sum_{i,j\in\mathcal{D}_{q}:y_{i}>y_{j}}\Delta_{i,j}\ln\frac{1}{1+\exp(-(s_{i}-s_{j}))},$

where $\Delta_{ij}$ is the LambdaWeight as defined in Eq. (11) of (Jagerman et
al., 2022).

##  4\. Methods

The main challenge to balance the two roles of the graded labels is to align
grade prediction methods and ranking methods.

#### Ranking score of grade prediction methods.

Compared with mean squared error and univariate ordinal methods, where we can
directly leverage the scalar predictions as the ranking scores,
$s_{i}=f_{\theta}({\bf x}_{i})$, it is less straightforward to determine
ranking scores for the multivariate multi-class cross entropy and ordinal
methods. The multivariate output corresponds to well-defined probabilities as
in Eqs. (3) and (8), but contains only part of the information for ranking. A
single output scalar is insufficient for ranking. We thus propose to use the
expected grade predictions in these methods as the ranking scores, which align
with sorting by grades. Following Eq. (3), for multi-class cross entropy
method, we have

(12) $s_{i}=\mathbb{E}[y_{i}]=\sum_{l=0}^{L-1}lp(y_{i}=l|{\bf
x}_{i})=\sum_{l=0}^{L-1}l\frac{\exp(f_{\theta}^{l}({\bf
x}_{i}))}{\sum_{j}\exp(f_{\theta}^{j}({\bf x}_{i}))}.$

Following Eq. (8), assuming equally spaced consecutive label values, for the
multivariate ordinal method, we have

(13) $s_{i}=\mathbb{E}[y_{i}]=\sum_{l=1}^{L-1}[l-(l-1)]p(y_{i}\geq l|{\bf
x}_{i})=\sum_{l=1}^{L-1}\frac{1}{1+\exp(-f_{\theta}^{l}({\bf x}_{i}))}.$

#### Multiobjective methods

Given the ranking score from the ordinal predictions in Eqs. (12) and (13), we
can also extend the multiobjective setting to multi-class cross entropy and
multivariate ordinal methods, with a total loss,

(14) $\mathcal{L}^{\rm MultiObj}(\mathcal{D})=(1-\alpha)\mathcal{L}^{\rm
Ord}(\mathcal{D};f_{\theta})+\alpha\mathcal{L}^{\rm Rank}(\mathcal{D};s),$

where the ranking score function $s$ is defined by the grade prediction
function $f_{\theta}$, and $\alpha$ gives the relative weight on the ranking
method.

##  5\. Experiments

###  5.1. Experimental Setup

We study the problem with three large public learning-to-rank datasets, Web30K
(Qin and Liu, 2013), Yahoo (Chapelle and Chang, 2011), and Istella (Dato et
al., 2016). The statistics of the datasets used are summarized in Table 1.

#### Comparing Methods

The focus of this paper is on the loss function, thus all compared methods on
each dataset share the same model architecture, containing three layers with
1024, 512, 256 hidden units, implemented with a public learning to rank
library: TensorFlow Ranking 111https://github.com/tensorflow/ranking. In
addition, we apply the log1p input transformations, batch normalization, and
dropout (Qin et al., 2021). Hyperparameters including learning rate, batch
normalization momentum, dropout rate, and rank loss weight $\alpha$ are tuned
for each method when applicable to the validation set.

As summarized in Table 2, we study the naive methods (MSL, MCCE, UniOrd, and
Ordinal) that train models to directly predict relevance grades, compared with
the SOTA ranking methods (Lambda, Softmax, USoft, Gumbel), as well as the
multiobjective methods allowing us to optimize both grade prediction accuracy
and ranking simultaneously.

Table 2. Compared methods.

Method |  Description
---|---
MSL |  Mean squared error loss method in Eq. (1).
MCCE |  Multi-class classification in Eq. (4).
UniOrd |  Univariate Ordinal regression in Eq. (7).
Ordinal |  Vanilla multivariate Ordinal regression in Eq. (8).
Lambda (Jagerman et al., 2022) |  LambdaLoss@1 method optimizing NDCG metric in Eq. (11).
MSL (Lambda) (Yan et al., 2022) |  Multiobjective method combining MSL and Lambda in Eq. (14).
MCCE (Lambda) |  Multiobjective method combining MCCE and Lambda in Eqs. (12) and (14).
UniOrd (Lambda) |  Multiobjective method combining UniOrd and Lambda in Eq. (14).
Ordinal (Lambda) |  Multiobjective method combining Ordinal and Lambda in Eqs. (13) and (14).

Table 3. Comparisons on classification and ranking for three LTR datasets.
Bold numbers are the best in each column. Up arrow “$\uparrow$” and down arrow
“$\downarrow$” indicate statistical significance with p-value=0.01 of better
and worse ACC/NDCG performance than the multiobjective baseline “MSL
(Lambda)”, respectively. The results of multiobjective methods in the table
correspond to the ones of optimal balance of ACC and NDCG@10, as the bold
markers in Figure 1.

| Web30K | Yahoo | Istella
---|---|---|---
Method | CE | MSE | ACC | NDCG@10 |  | CE | MSE | ACC | NDCG@10 | CE | MSE | ACC | NDCG@10 |
MSL | 12.348 | 0.5414 | 0.5531↑ | 0.5002↓ |  | 13.570 | 0.5781 | 0.5089 | 0.7720 | 1.8310 | 0.1166 | 0.9337↓ | 0.7120↓ |
MCCE | 0.9035 | 0.5384 | 0.6018↑ | 0.5028↓ |  | 1.0531 | 0.5736 | 0.5260↑ | 0.7722 | 0.1236 | 0.1085 | 0.9611↑ | 0.7111↓ |
UniOrd | 0.9202 | 1.5899 | 0.5953↑ | 0.4953↓ |  | 1.0916 | 1.6029 | 0.5155↑ | 0.7692↓ | 0.1276 | 112.39 | 0.9612↑ | 0.7151↓ |
Ordinal | 0.9066 | 0.5405 | 0.6013↑ | 0.5053 |  | 1.0628 | 0.5763 | 0.5235↑ | 0.7698↓ | 0.1252 | 0.1093 | 0.9616↑ | 0.7123↓ |
Lambda (Jagerman et al., 2022) | 0.9444 | 1.8466 | 0.5709↑ | 0.5057 |  | 1.4078 | 3.8040 | 0.2993↓ | 0.7716 | 0.1577 | 544.92 | 0.9578↑ | 0.7310↑ |
MSL (Lambda) (Yan et al., 2022) | 12.543 | 0.5566 | 0.5460 | 0.5054 |  | 13.591 | 0.5781 | 0.5081 | 0.7726 | 1.6886 | 0.1215 | 0.9389 | 0.7251 |
MCCE (Lambda) | 0.9027 | 0.5377 | 0.6030↑ | 0.5107↑ |  | 1.0604 | 0.5736 | 0.5232↑ | 0.7734 | 0.1328 | 0.1206 | 0.9605↑ | 0.7288↑ |
UniOrd (Lambda) | 0.9280 | 1.5973 | 0.5877↑ | 0.5073 |  | 1.1109 | 1.5923 | 0.5040 | 0.7721 | 0.1422 | 319.43 | 0.9581↑ | 0.7320↑ |
Ordinal (Lambda) | 0.9056 | 0.5394 | 0.6006↑ | 0.5100↑ |  | 1.0650 | 0.5758 | 0.5225↑ | 0.7743↑ | 0.1365 | 0.1242 | 0.9593↑ | 0.7298↑ |

(a) Web30K (b) Yahoo (c) Istella

Figure 1. Tradeoffs of methods on classification accuracy (ACC) versus
NDCG@10. Lines correspond to the Pareto fronts of different grade prediction
objectives in the multiobjective setting, labeled in the legend. The results
best balancing ACC and NDCG@10, marked in bold, are chosen to represent the
MultiObj method in Table 3.

#### Metrics

To quantify the methods on both grade prediction accuracy and ranking, we
consider metrics in both categories. For ranking performance, we measure NDCG
metrics (Järvelin and Kekäläinen, 2002), which we try to maximize.
Specifically, we use NDCG@10, which scores the top 10 positions. For grade
prediction performance, we want to minimize cross entropy (CE) in Eq. (4) and
the mean square error (MSE) in Eq. (1), and to maximize the classification
accuracy (ACC). The grade prediction metrics CE and ACC depend on predictions
of grade probabilities. These are not defined by ranking methods that predict
a single score. To evaluate such metrics for ranking methods, we convert
ranking scores to grade probabilities by introducing ordinal boundaries
$\phi_{l}$, as those used for univariate ordinal regression Eqs. (6) and (7).
The boundaries $\phi_{l}$ are trained to optimize cross entropy in Eq. (4)
with fixed model parameters $\theta$.

###  5.2. Results and Discussion

The main results are summarized in Table 3. We can make the following
observations: (i) In terms of grade prediction performance, MCCE and Ordinal
are strong baselines: they show the best competitive CE and ACC performance,
which they are directly optimized for. In addition, by predicting the expected
grade value using Eqs. (12) and (13), they also give competitive MSE. (ii)
More interestingly, on Web30K, multiobjective setting combining MCCE objective
and Lambda objective shows the best CE, MSE, and ACC. This demonstrates that
the ranking objective is synergetic to the grade prediction with MCCE on this
dataset. (iii) Similarly, the best ranking NDCG is approached by one of the
proposed multiobjective method on each dataset: MCCE (Lambda) on Web30K,
Ordinal (Lambda) on Yahoo, and UniOrd (Lambda) on Istella. These best values
are statistically significantly better than the state-of-the-art ranking
baselines, which also indicates a synergetic interaction of two objectives on
the ranking task. (iv) On contrary, the traditional mutliobjective method
combining MSL and Lambda show inferior grade prediction performance to MSL
only and inferior ranking performance to Lambda (except on Yahoo). This
implies no synergy between MSL and ranking losses.

We further analyze the behaviors of the methods in terms of their trade-offs
between the ranking performance (measured by NDCG@10) and the grade prediction
performance (measured by ACC). The results are shown in Figure 1. For each of
the multiobjective methods, we can probe multiple points by varying $\alpha$,
and we connect the Pareto frontiers for each combination of a grade prediction
method and the Lambda method. From the tradeoff plot, we observe: (i) The
grade prediction objective and the ranking objective are not simply trading
off with each other, but can work collaboratively in certain range of rank
weight $\alpha$; (ii) Proposed combinations of grade prediction objective
(MCCE, Ordinal, and UniOrd) and ranking objective probe different Pareto
frontiers on different datasets, and are consistently better than a simple
combination of MSL and Lambda. These behaviors provide guidance to
practitioners: Depending on the dataset, practitioners can bias towards one of
multiobjective methods and tune the ranking objective weight $\alpha$ for the
best balance of grade prediction and ranking.

As this work focuses on the neural network models, whether these observations
could be extended to GBDT models needs further study. But we foresee no
constraints to limit the generalization.

##  6\. Conclusion

We provided a rigorous study of learning to rank with graded labels when
grades matter, which has practical values but is less explored in the
literature. We studied several existing classification and state-of-the-art
ranking methods, and proposed several methods by addressing challenges of
performing learning to rank with the goal of also accurately predicting
ordinal grades. Experiments show that grade prediction and ranking can have
synergetic interaction, allowing us to push the Pareto frontier in the ranking
and grade prediction trade-off.

## References

  * (1)
  * Bruch et al. (2020) Sebastian Bruch, Shuguang Han, Michael Bendersky, and Marc Najork. 2020.  A stochastic treatment of learning to rank scoring functions. In _Proceedings of the 13th international conference on web search and data mining_. 61–69.
  * Bruch et al. (2019) Sebastian Bruch, Masrour Zoghi, Michael Bendersky, and Marc Najork. 2019.  Revisiting approximate metric optimization in the age of deep neural networks. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1241–1244.
  * Burges et al. (2005) Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005.  Learning to rank using gradient descent. In _Proceedings of the 22nd international conference on Machine learning_. 89–96.
  * Burges (2010) Christopher JC Burges. 2010\.  From ranknet to lambdarank to lambdamart: An overview.  _Learning_ 11, 23-581 (2010), 81.
  * Carmel et al. (2020) David Carmel, Elad Haramaty, Arnon Lazerson, and Liane Lewin-Eytan. 2020.  Multi-objective ranking optimization for product search using stochastic label aggregation. In _Proceedings of The Web Conference 2020_. 373–383.
  * Chapelle and Chang (2011) Olivier Chapelle and Yi Chang. 2011.  Yahoo! learning to rank challenge overview. In _Proceedings of the Learning to Rank Challenge_. 1–24.
  * Chapelle et al. (2010) Olivier Chapelle, Pannagadatta Shivaswamy, Srinivas Vadrevu, Kilian Weinberger, Ya Zhang, and Belle Tseng. 2010.  Multi-task learning for boosting with application to web search ranking. In _Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining_. 1189–1198.
  * Chapelle and Wu (2010) Olivier Chapelle and Mingrui Wu. 2010.  Gradient descent optimization of smoothed information retrieval metrics.  _Information retrieval_ 13, 3 (2010), 216–235.
  * Chaudhuri et al. (2017) Sougata Chaudhuri, Abraham Bagherjeiran, and James Liu. 2017\.  Ranking and Calibrating Click-Attributed Purchases in Performance Display Advertising. In _2017 AdKDD & TargetAd_. 7:1–7:6.
  * Dato et al. (2016) Domenico Dato, Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Nicola Tonellotto, and Rossano Venturini. 2016\.  Fast ranking with additive ensembles of oblivious and non-oblivious regression trees.  _ACM Transactions on Information Systems_ 35, 2, Article 15 (2016).
  * Jagerman et al. (2022) Rolf Jagerman, Zhen Qin, Xuanhui Wang, Michael Bendersky, and Marc Najork. 2022.  On Optimizing Top-K Metrics for Neural Ranking Models. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2303–2307.
  * Järvelin and Kekäläinen (2002) Kalervo Järvelin and Jaana Kekäläinen. 2002.  Cumulated gain-based evaluation of IR techniques.  _ACM Transactions on Information Systems (TOIS)_ 20, 4 (2002), 422–446.
  * Li et al. (2007) Ping Li, Qiang Wu, and Christopher Burges. 2007.  Mcrank: Learning to rank using multiple classification and gradient boosting.  _Advances in neural information processing systems_ 20 (2007).
  * Pobrotyn et al. (2020) Przemysław Pobrotyn, Tomasz Bartczak, Mikołaj Synowiec, Radosław Białobrzeski, and Jarosław Bojar. 2020.  Context-aware learning to rank with self-attention.  _arXiv preprint arXiv:2005.10084_ (2020).
  * Qin and Liu (2013) Tao Qin and Tie-Yan Liu. 2013.  Introducing LETOR 4.0 datasets.  _arXiv preprint arXiv:1306.2597_ (2013).
  * Qin et al. (2010) Tao Qin, Tie-Yan Liu, and Hang Li. 2010.  A general approximation framework for direct optimization of information retrieval measures.  _Information retrieval_ 13, 4 (2010), 375–397.
  * Qin et al. (2021) Zhen Qin, Le Yan, Honglei Zhuang, Yi Tay, Rama Kumar Pasumarthi, Xuanhui Wang, Michael Bendersky, and Marc Najork. 2021\.  Are Neural Rankers still Outperformed by Gradient Boosted Decision Trees?. In _International Conference on Learning Representations_.
  * Svore et al. (2011) Krysta M Svore, Maksims N Volkovs, and Christopher JC Burges. 2011\.  Learning to rank with multiple objective functions. In _Proceedings of the 20th international conference on World wide web_. 367–376.
  * Tagami et al. (2013) Yukihiro Tagami, Shingo Ono, Koji Yamamoto, Koji Tsukamoto, and Akira Tajima. 2013.  CTR Prediction for Contextual Advertising: Learning-to-Rank Approach. In _Proceedings of the 7th International Workshop on Data Mining for Online Advertising_. Article 4, 8 pages.
  * Taylor et al. (2008) Michael Taylor, John Guiver, Stephen Robertson, and Tom Minka. 2008.  Softrank: optimizing non-smooth rank metrics. In _Proceedings of the 2008 International Conference on Web Search and Data Mining_. 77–86.
  * Xia et al. (2008) Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. 2008.  Listwise approach to learning to rank: theory and algorithm. In _Proceedings of the 25th international conference on Machine learning_. 1192–1199.
  * Yan et al. (2022) Le Yan, Zhen Qin, Xuanhui Wang, Michael Bendersky, and Marc Najork. 2022.  Scale Calibration of Deep Ranking Models. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_. 4300–4309.


* [Van02] John Andrew Vano.  A Nash-Moser implicit function theorem with Whitney regularity and applications.  ProQuest LLC, Ann Arbor, MI, 2002.  Thesis (Ph.D.)–The University of Texas at Austin.
  * [Win01] Arthur T. Winfree.  The geometry of biological time, volume 12 of Interdisciplinary Applied Mathematics.  Springer-Verlag, New York, second edition, 2001.
  * [Win75] A. T. Winfree.  Patterns of phase compromise in biological cycles.  J. Math. Biol., 1(1):73–95, 1974/75.
  * [WY02] Qiudong Wang and Lai-Sang Young.  From invariant curves to strange attractors.  Comm. Math. Phys., 225(2):275–304, 2002.
  * [WY03] Qiudong Wang and Lai-Sang Young.  Strange attractors in periodically-kicked limit cycles and Hopf bifurcations.  Comm. Math. Phys., 240(3):509–529, 2003.
  * [YdlL21] Yian Yao and Rafael de la Llave.  Computing the invariant circle and the foliation by stable manifolds for 2-d maps by the parameterization method: Numerical implementation and results.  2021\.  Manuscript in progress.
  * [ZdlL18] Lei Zhang and Rafael de la Llave.  Transition state theory with quasi-periodic forcing.  Commun. Nonlinear Sci. Numer. Simul., 62:229–243, 2018.
  * [Zeh75] E. Zehnder.  Generalized implicit function theorems with applications to some small divisor problems. I.  Comm. Pure Appl. Math., 28:91–140, 1975.


11institutetext: Wipro Limited, Salt Lake, Kolkata, India 22institutetext:
Department of Computer Science and Engineering, Indian Institute of
Technology, Jodhpur, Rajasthan, India
22email<EMAIL_ADDRESS><EMAIL_ADDRESS>33institutetext:
Department of Computer Science and Engineering, Indian Institute of
Technology, Kharagpur, West Bengal, India
33email<EMAIL_ADDRESS>
# TransDocAnalyser: A Framework for Offline Semi-structured Handwritten
Document Analysis in the Legal Domain

Sagar Chakraborty and Gaurav Harit and Saptarshi Ghosh 11222233

###### Abstract

State-of-the-art offline Optical Character Recognition (OCR) frameworks
perform poorly on semi-structured handwritten domain-specific documents due to
their inability to localize and label form fields with domain-specific
semantics. Existing techniques for semi-structured document analysis have
primarily used datasets comprising invoices, purchase orders, receipts, and
identity-card documents for benchmarking. In this work, we build the first
semi-structured document analysis dataset in the legal domain by collecting a
large number of First Information Report (FIR) documents from several police
stations in India. This dataset, which we call the FIR dataset, is more
challenging than most existing document analysis datasets, since it combines a
wide variety of handwritten text with printed text. We also propose an end-to-
end framework for offline processing of handwritten semi-structured documents,
and benchmark it on our novel FIR dataset. Our framework used Encoder-Decoder
architecture for localizing and labelling the form fields and for recognizing
the handwritten content. The encoder consists of Faster-RCNN and Vision
Transformers. Further the Transformer-based decoder architecture is trained
with a domain-specific tokenizer. We also propose a post-correction method to
handle recognition errors pertaining to the domain-specific terms. Our
proposed framework achieves state-of-the-art results on the FIR dataset
outperforming several existing models.

###### Keywords:

Semi-structured document Offline handwriting recognition Legal document
analysis Vision Transformer FIR dataset

##  1 Introduction

Semi-Structured documents are widely used in many different industries. Recent
advancement in digitization has increased the demand for analysis of scanned
or mobile-captured semi-structured documents. Many recent works have used
different deep learning techniques to solve some of the critical problems in
processing and layout analysis of semi-structured documents [34, 16, 23].
Semi-structured documents consist of printed, handwritten, or hybrid (both
printed and handwritten) text forms. In particular, hybrid documents (see
Figure 1) are more complex to analyze since they require segregation of
printed and handwritten text and subsequent recognition. With recent
advancements, the OCR accuracy has improved for printed text; however,
recognition of handwritten characters is still a challenge due to variations
in writing style and layout.

Figure 1: Examples of First Information Report (FIR) documents from different
police stations in India. The FIR dataset developed in this paper consists of
a wide variety of such semi-structured FIR documents containing both printed
and handwritten text.

Earlier works have focused on techniques for layout analysis, named-entity
recognition, offline handwriting recognition, etc., but sufficient work has
not been done on developing an end-to-end framework for processing semi-
structured documents. A general end-to-end framework can be easily fine-tuned
for domain-specific requirements. In this paper we present the first framework
for semi-structured document analysis applied to legal documents.

There have been many works on legal documents, such as on case document
summarization [6], relevant statute identification from legal facts [31],
pretraining language models on legal text [32] and so on. But almost all prior
research in the legal domain has focused on textual data, and not on document
images. In particular, the challenges involved in document processing and
layout analysis of legal documents is unattended, even though these tasks have
become important due to the increasing availability of scanned/photographed
legal documents.

In this work, we build the first dataset for semi-structured document analysis
in the legal domain. To this end, we focus on First Information Report (FIR)
documents from India. An FIR is usually prepared by police stations in some
South Asian countries when they first get a complaint by the victim of a crime
(or someone on behalf of the
victim).111https://en.wikipedia.org/wiki/First_information_report An FIR
usually contains a lot of details such as the date, time, place, and details
of the incident, the names of the person(s) involved, a list of the statutes
(written laws, e.g., those set by the Constitution of a country) that might
have been violated by the incident, and so on. The FIRs are usually written on
a printed form, where the fields are filled in by hand by police officials
(see examples in Figure 1). It is estimated that more than 6 million FIRs are
filed every year across thousands of police stations in various states in
India. Such high volumes lead to inconsistent practices in-terms of
handwriting, layout structure, scanning procedure, scan quality, etc., and
introduce huge noise in the digital copies of these documents.

Our target fields of interest while processing FIR documents are the
handwritten entries (e.g., name of the complainant, the statutes violated)
which are challenging to identify due to the wide variation in handwriting. To
form the dataset, which we call the FIR dataset, we created the meta-data for
the target fields by collecting the actual text values from the police
databases, and also annotated the documents with layout positions of the
target fields. The FIR dataset is made publicly available at
https://github.com/LegalDocumentProcessing/FIR_Dataset_ICDAR2023.

The FIR dataset is particularly challenging since its documents are of mixed
type, with both printed and handwritten text. Traditional OCR identifies
blocks of text strings in documents and recognizes the text from images by
parsing from left to right [19]. NLP techniques like named-entity recognition
(NER), which uses raw text to find the target fields, cannot be applied
easily, since traditional OCRs do not work well in recognition of mixed
documents with handwritten and printed characters occurring together. Another
drawback of traditional OCRs in this context is their inability to recognise
domain-specific words due to their general language-based vocabulary. In this
work, we propose a novel framework for analysing such domain-specific semi-
structured documents. The contributions of the proposed framework as follows:

  1. 1.

We use a FastRCNN + Vision Transformer-based encoder trained for target field
localization and classification. We also deploy a BERT-based text decoder that
is fine-tuned to incorporate legal domain-specific vocabulary.

  2. 2.

We use a domain-specific pretrained language model [32] to improve the
recognition of domain-specific text (legal statutes, Indian names, etc.). This
idea of using a domain-specific language model along with OCR is novel, and
has a wider applicability over other domains (e.g., finance, healthcare, etc)
where this technique can be used to achieve improved recognition from domain-
specific documents.

  3. 3.

We improve the character error rate (CER) by reducing the ambiguities in OCR
through a novel domain-specific post-correction step. Using domain knowledge,
we created a database for each target field (such as Indian names, Indian
statutes, etc.) to replace the ambiguous words from OCR having low confidence
using a combination of TF-IDF vectorizer and K-Nearest Neighbour classifier.
This novel post-correction method to handle recognition errors pertaining to
proper nouns, enables our proposed framework to outperform state-of-the-art
OCR models by large margins.

To summarize, in this work we build the first legal domain-specific dataset
for semi-structured document analysis. We also develop a framework to localise
the handwritten target fields, and fine-tune a transformer-based OCR (TrOCR)
to extract handwritten text. We further develop post-correction techniques to
improve the character error rate. To our knowledge, the combination of Faster-
RCNN and TrOCR with other components, such as Vision Transformer and legal
domain-specific tokenizers, to create an end-to-end framework for processing
offline handwritten semi-structured documents is novel, and can be useful for
analysis of similar documents in other domains as well.

##  2 Related Work

We briefly survey four types of prior works related to our work – (i) related
datasets, (ii) works addressing target field localization and classification,
(iii) handwritten character recognition, and (iv) works on post-OCR correction
methods

Related Datasets: There exist several popular datasets for semi-structured
document analysis. FUNSD [22] is a very popular dataset for information
extraction and layout analysis. FUNSD dataset is a subset of RVL-CDIP dataset
[17], and contains 199 annotated financial forms. The SROIE dataset [21]
contains 1,000 annotated receipts having 4 different entities, and is used for
receipt recognition and information extraction tasks. The CloudSCan Invoice
dataset [29] is a custom dataset for invoice information extraction. The
dataset contained 8 entities in printed text.

Note that no such dataset exists in the legal domain, and our FIR dataset is
the first of its kind. Also, the existing datasets contain only printed text,
while the dataset we build contains a mixture of printed and hand-written text
(see Table 2 for a detailed comparison of the various datasets).

Localization and Labelling of field components: Rule-based information
extraction methods (such as the method developed by Kempf et al. [10] and many
other methods) could be useful when documents are of high quality and do not
contain handwritten characters. But when document layouts involve huge
variations, noise and handwritten characters, keyword-based approaches fail to
provide good results. Template-based approaches also fail due to scanning
errors and layout variability [36, 1, 2].

Srivastava et al. [12] developed a graph-based deep network for predicting the
associations between field labels and field values in handwritten form images.
They considered forms in which the field label comprises printed text and
field value can be handwritten text; this is similar to what we have in the
FIR dataset developed in this work. To perform association between the target
field labels and values, they formed a graphical representation of the textual
scripts using their associated layout position.

In this work, we tried to remove the dependency on OCR of previous works [12]
by using layout information of images to learn the positions of target fields
and extract the image patches using state-of-the-art object detection models
such as [37, 35, 33].

Zhu et. al. [37] proposed attention modules that only attend to a small set of
key sampling points around a reference, which can achieve better performance
than baseline model [8] with 10$\times$ less training epochs. Tan et. al. [35]
used weighted bi-directional feature pyramid network (BiFPN), which allows
easy and fast multi-scale feature fusion. Ren et al [33] proposed an improved
version of their earlier work [14] provides comparative performances with [37,
35] with lower latency and computational resources on FIR dataset. Hence, we
use Faster RCNN model in this framework for localization and classification of
the field component.

Handwritten Character Recognition: Offline handwriting recognition has been a
long standing research interest. The works [3, 4, 5] presented novel features
based on structural features of the strokes and their spatial relations with a
character, as visible from different viewing directions on a 2D plane.
Diesendruck et al. [11] used Word Spotting to directly recognise handwritten
text from images. The conventional text recognition task is usually framed as
an encoder-decoder problem where the traditional methods[19] leveraged CNN-
based [24] encoder for image understanding and LSTM-based [20] decoder for
text recognition.

Chowdhury et al. [9] combined a deep convolutional network with a recurrent
Encoder-Decoder network to map an image to a sequence of characters
corresponding to the text present in the image. Michael, Johannes et al. [28]
proposed a sequence-to-sequence model combining a convolutional neural network
(as a generic feature extractor) with a recurrent neural network to encode
both the visual information, as well as the temporal context between
characters in the input image. Further, Li et al. [25] used for the first time
an end-to-end Transformer-based encoder-decoder OCR model for handwritten text
recognition and achieved SOTA results. The model [25] is convolution-free
unlike previous methods, and does not rely on any complex pre/post-processing
steps. The present work leverages this work and extends its application in
legal domain.

Post-OCR correction: Rectification of errors in the recognised text from the
OCR would require extensive training which is computation heavy. Further,
post-OCR error correction requires a large amount of annotated data which may
not always be available. After the introduction of the Attention mechanism and
BERT model, many works have been done to improve the results of the OCR using
language model based post-correction techniques. However, Neural Machine
Translation based approaches as used by Duong et al. [13] are not useful in
the case of form text due to the lack of adequate context and neighbouring
words. We extend the idea used in the work of Trstenjak et al. [7] where they
used edit distance and cosine similarity to find the matching words. In this
paper we used K-nearest neighbour with edit distance to find best matches for
the words predicted with low confidence score by the OCR.

##  3 The FIR Dataset

First Information Report (FIR) documents contain details about incidents of
cognisable offence, that are written at police stations based on a complaint.
FIRs are usually filed by a police official filling up a printed form; hence
the documents contain both printed and handwritten text. In this work, we
focus on FIR documents written at police stations in India. Though the FIR
forms used across different Indian states mostly have a common set of fields,
there are some differences in their layout (see examples in Fig. 1). To
diversify the dataset, we included FIR documents from the databases of various
police stations across several Indian states – West
Bengal222http://bidhannagarcitypolice.gov.in/fir_record.php,
Rajasthan333https://home.rajasthan.gov.in/content/homeportal/en.html,
Sikkim444https://police.sikkim.gov.in/visitor/fir,
Tripura555https://tripurapolice.gov.in/west/fir-copies and
Nagaland666https://police.nagaland.gov.in/fir-2/.

As stated earlier, an FIR contains many fields including the name of the
complainant, names of suspected/alleged persons, statutes that may have been
violated, date and location of the incident, and so on. In this work, we
selected four target fields from FIR documents for the data annotation and
recognition task – (1) Year (the year in which the complaint is being
recorded), (2) Complainant’s name (name of the person who lodged the
complaint), (3) Police Station (name of the police station that is responsible
for investigating the particular incident), and (4) Statutes (Indian laws that
have potentially been violated in the reported incident; these laws give a
good indication of the type of the crime). We selected these four target
fields because we were able to collect the gold standard for these four fields
from some of the police databases. Also, digitizing these four fields would
enable various societal analysis, such as analysis of the nature of crimes in
different police stations, temporal variations in crimes, and so on.

Figure 2: Sample of various entities present in First Information Reports with
different writing styles, distortions and scales.

Annotations: We manually analysed more than 1,300 FIR documents belonging to
different states, regions, police stations, etc. We found that FIR documents
from the same region / police station tend to have the similar layout and form
structure. Hence we selected a subset of 375 FIR documents with reasonably
varying layouts / form structure, so that this subset covers most of the
different variations. These 375 documents were manually annotated. Annotations
were done on these documents using LabelMe annotation
tool777https://github.com/wkentaro/labelme to mark the bounding boxes of the
target fields.

Figure 3: Examples of ground truth annotations for two of the entities shown
in Figure 2

Figure 2 shows some samples of various entities present in our dataset, and
Figure 3 shows examples of ground truth annotations for two of the entities in
Figure 2. In the ground truth, each bounding box has four co-ordinates
(X_left, X_width, Y_right, Y_height) which describe the position of the
rectangle containing the field value for each target field.

Train-test split: During the annotation of our dataset, we identified 79
different types of large scale variations, layout distortions/deformations,
which we split into training and testing sets. We divided our dataset (of 375
document images) such that 300 images are included in the training set and the
other 75 images are used as the test set. During training, we used 30% of
training dataset as a validation set. Table 1 shows the bifurcation statistics
for training and test sets.

Table 1: FIR Dataset statistics Split | Images | Layout | Words | Labels
---|---|---|---|---
Training | 300 | 61 | 1,830 | 1,230
Testing | 75 | 18 | 457 | 307

Preprocessing the images: For Faster-RCNN we resized the document images to a
size of 1180 $\times$ 740, and used the bounding boxes and label names to
train the model to predict and classify the bounding boxes. We convert the
dataset into IAM Dataset format [27] to fine-tune the transformer OCR.

Novelty of the FIR dataset: We compare our FIR dataset888
https://github.com/LegalDocumentProcessing/FIR_Dataset_ICDAR2023 with other
datasets for semi-structure document analysis in Table 2. The FIR dataset
contains both printed and handwritten information which makes it unique and
complex compared to several other datasets. Additionally, the FIR dataset is
the first dataset for semi-structured document analysis in the legal domain.

Table 2: Comparison of the FIR dataset with other similar datasets Dataset | Category | #Images | Text Type | #Entites
---|---|---|---|---
|  |  | Printed | Handwritten |
FUNSD [22] | Form | 199 |  | x | 4
SROIE [21] | Receipt | 1000 |  | x | 4
Cloud Invoice [29] | Invoice | 326571 |  | x | 8
FIR (Ours) | Form | 375 |  |  | 4

##  4 The TransDocAnalyser Framework

We now present TransDocAnalyser, a framework for offline processing of
handwritten semi-structured documents, by adopting Faster-RCNN and
Transformer-based encoder-decoder architecture, with post-correction to
improve performance.

###  4.1 The Faster-RCNN architecture

Faster-RCNN [33] is a popular object detection algorithm that has been adopted
in many real-world applications. It builds upon the earlier R-CNN [15] and
Fast R-CNN [33] architectures. We pass the input images through the Faster-
RCNN network to get the domain-specific field associations and extract the
image patches from the documents.

Our modified Faster-RCNN architecture consists of three main components (as
schematically shown in Figure 4)– (1) Backbone Network , (2) Region Proposal
Network (RPN), and (3) ROI Heads as detailed below.

Figure 4: Modified Faster-RCNN based architecture for target field
localization and labelling

(1) Backbone Network: ResNet-based backbone network is used to extract multi-
scaled feature maps from the input – that are named as P2, P3, P4 , P8 and so
on – which are scaled as 1/4th, 1/8th, 1/16th and so on. This backbone network
is FPN-based (Feature Pyramid network) [26] which is multi-scale object
detector invariant to the object size.

(2) Region Proposal Network (RPN): Detects ROI (regions of interest) along
with a confidence score, from the multi-scale feature maps generated by the
backbone network. A fixed-size kernel is used for region pooling. The regions
detected by the RPN are called proposal boxes.

(3) ROI Heads: The input to the box head comprises (i) the feature maps
generated by a Fully Connected Network (FCN), (ii) the proposed boxes which
come from the RPN. These are 1,000 boxes with their predicted labels. Box head
uses the bounding boxes proposed by the RPN to crop and prepare the feature
maps. (iii) ground truth bounding boxes from the annotated training datasets.
The ROI pooling uses the proposed boxes detected by RPN, crops the rectangular
areas of the feature maps, and feeds them into the head networks. Using Box
head and mask head together in Faster-RCNN network, inspired by He et al. [18]
improves the overall performance.

During training, the box head makes use of the ground truth boxes to
accelerate the training. The mask head provides the final predicted bounding
boxes and confidence scores during the training. At the time of inference the
head network uses non-maximum suppression (NMS) algorithm to remove the
overlapping boxes and selects the top-k results as the predicted output based
on thresholds on their confidence score and intersection over union (IOU).

Figure 5: TrOCR architecture with custom enhancements. The Text decoder uses a
domain-specific InLegalBert [32] based tokenizer. OCR predictions go for post-
correction if the confidence score is less than the threshold. We convert the
OCR prediction into a TF-IDF vector and search in the domain-specific field
database to find the Nearest Match.

###  4.2 The TrOCR architecture

Once the localized images are generated for a target field (e.g., complainant
name) by Faster-RCNN, the image patches are then flattened and sent to the
Vision Transformer (ViT) based encoder model. We use TrOCR [25] as the
backbone model for our finetuning (see Figure 5). TrOCR [25] is a Transformer-
based OCR model which consists of a pretrained vision Transformer encoder and
a pretrained text decoder. The ViT encoder is trained on the IAM handwritten
dataset, which we fine-tune on our FIR dataset. We use the output patches from
the Faster-RCNN network as input to the ViT encoder, and fine-tune it to
generate features. As we are providing the raw image patches received from
Faster-RCNN into the ViT encoder, we did not apply any pre-processing or
layout enhancement technique to improve the quality of the localised images.
On the contrary, we put the noisy localised images cropped from the form
fields directly, which learns to suppress noise features by training.

We also replace the default text decoder (RoBERTa) with the Indian legal-
domain specific BERT based text decoder InLegalBERT [32] as shown in Fig. 5.
InLegalBert [32] is pre-trained with a huge corpus of about 5.4 million Indian
Legal documents, including court judgements of the Indian Supreme Court and
other higher courts of India, and various Central Government Acts.

To recognize characters in the cropped image patches, the images are first
resized into square boxes of size 384 × 384 pixels and then flattened into a
sequence of patches, which are then encoded by ViT into high-level
representations and decoded by InLegalBERT into corresponding characters step-
by-step.

We evaluate and penalise the model based on the Character Error Rate (CER).
CER calculation is based on the concept of Levenshtein distance, where we
count the minimum number of character-level operations required to transform
the ground truth text into the predicted OCR output. CER is computed as
$CER=(S+D+I)/N$ where $S$ is the number of substitutions, $D$ is the number of
deletions, $I$ is the number of Insertions, and $N$ is the number of
characters in the reference text.

###  4.3 KNN-based OCR Correction

For each predicted word from OCR, if the confidence score is less than a
threshold 0.7, we consider the OCR output to be ambiguous for that particular
word. In such cases, the predicted word goes through a post-correction step
which we describe now (see Figure 6).

Table 3: Excerpts from field-specific databases used to prepare TF-IDF vectorized records for KNN search. All databases contain India-specific entries. Names | Surnames | Police Stations | Statutes / Acts
---|---|---|---
Anamul | Haque | Baguiati | IPC (Indian Penal Code)
Shyam | Das | Airport | D.M. Act (Disaster Management Act)
Barnali | Pramanik | Newtown | D.C. Act (Drug and Cosmetics Act)
Rasida | Begam | Saltlake | NDPS Act

For each target field, we create a database of relevant values and terms
(which could be written in the field) from various sources available on the
Web. Table 3 shows a very small subset of some of the field-specific databases
such as Indian names, Indian surnames, Indian statutes (Acts and Sections),
etc. We converted each database into a set of TF-IDF vectors (see Figure 6).
Here TF-IDF stands for Term Frequency times Inverse Document Frequency. The
TF-IDF scores are computed using n-grams of groups of letters. In our work we
used $n=3$ (trigrams) for generating the TF-IDF vectors for OCR predicted
words as well as for the entities in the databases.

Figure 6: Term Frequency and Inverse Document frequency (TF-IDF) Vectorizer
based K-Nearest Neighbour model for post-correction on OCR output

For a given OCR output, based on the associated field name which is already
available from the field classification by Faster-RCNN, we used the K-Nearest
Neighbour (KNN) classifier to select the appropriate vectorized database. KNN
returns best matches with a confidence score based on the distance between the
search vector (OCR output) and the vectors in the chosen database. If the
confidence score returned by KNN is greater than $0.9$, then the OCR predicted
word gets replaced with the word predicted by the K-Nearest Neighbour search.

##  5 Experimental settings

We ran all experiments on a Tesla T4 GPU with CUDA version 11.2. We used CUDA
enabled Torch framework 1.8.0.

In the first stage of the TransDocAnalyser framework, we trained the Faster
RCNN from scratch using the annotated dataset (the training set). Table 4
shows the settings used for training the Faster-RCNN model. Prior to the
training, input images are resized in 1180 $\times$ 740\. For memory
optimization, we run the model in two steps, first for 1500 iteration and then
for 1000 iteration on the stored model. We tried batch sizes (BS) of 16, 32
and 64, and finalized BS as 64 because of the improvement in performance and
training time. We used the trained model Faster-RCNN model to detect and crop
out the bounding boxes of each label from the original document (as shown in
Fig. 2) and created our dataset to fine-tune the ViT encoder.

We also created a metadata file mapping each cropped image (as shown in Fig.
2) with its corresponding text as described in [27] to fine-tune the decoder.

Table 4: Faster-RCNN model training parameters Base Model | Base Weights | Learning Rate | Epoch # | # of Class | IMS/batch | Image Size
---|---|---|---|---|---|---
ResNet 50 | Mask RCNN | 0.00025 | 2500 | 4 | 4 | 1180 $\times$ 740
Table 5: Tranformer OCR (TrOCR) parameters used for model fine-tuning Feature Extractor | Tokenizer | Max Len | N-gram | Penalty | # of Beam | Optimizer
---|---|---|---|---|---|---
google-vit-patch16-384 | InLegalBERT | 32 | 3 | 2.0 | 4 | AdamW

Table 5 shows the parameter settings used for fine-tuning the TrOCR model.
Image patches are resized to 384 $\times$ 384 dimension to fine-tune ViT
encoder. In the TrOCR model configuration, we replaced the tokenizer and
decoder settings based on InLegalBert. We tried with batch size (BS) of 2, 4,
8, 16, 32, 64, and BS = 8 provided the best result on the validation set. We
fine-tuned the Encoder and Decoder of the OCR for 40 epochs and obtained the
final results.

The KNN-based OCR correction module used n-grams with $n=1,2,3,4$ to generate
the TF-IDF vectors of the field-specific databases. Using $n=3$ (trigrams) and
KNN with $K=1$ provided the best results.

Table 6: Performance of field labelling on the FIR dataset (validation set and test set). Re: Recall, Pr: Precision, F1: F1-score, mAP: mean average precision. Results on dataset | Target field | Faster R-CNN
---|---|---
|  | $\mathbf{Re}\uparrow$ | $\mathbf{Pr}\uparrow$ | $\mathbf{F1}\uparrow$ | $\mathbf{mAP}\uparrow$
Validation | Year | $0.98$ | $0.96$ | $0.97$ | $0.97$
| Statute | $0.85$ | $0.82$ | $0.83$ | $0.84$
| Police Station | $0.96$ | $0.90$ | $0.93$ | $0.93$
| Complainant Name | $0.84$ | $0.76$ | $0.80$ | $0.77$
Test | Year | $0.97$ | $0.96$ | $0.97$ | $0.96$
| Statute | $0.84$ | $0.87$ | $0.86$ | $0.80$
| Police Station | $0.93$ | $0.88$ | $0.91$ | $0.91$
| Complainant Name | $0.80$ | $0.81$ | $0.81$ | $0.74$

##  6 Results

In this section, we present the results of the proposed framework
TransDocAnalyser in three stages – (i) The performance of Faster-RCNN on
localization and labelling of the target fields (Table 6); (ii) Sample of OCR
results with Confidence Scores (Table 7); and (iii) Comparison of the
performance of the proposed framework with existing OCR methods (Table 8).

Table 6 shows the results of field label detection using Faster-RCNN on both
test and validation sets of the FIR dataset. The performance is reported in
terms of Recall (Re), Precision (Pr), F1 (harmonic mean of Recall and
Precision) and mean Average Precision (mAP). For the localization and
labelling, a prediction is considered correct if both the IOU (with the ground
truth) and the confidence threshold are higher than $0.5$. The results show
that our model is performing well, with the best and worst results for the
fields ‘Year’ (F1 = 0.97) and ‘Name’ (F1 = 0.8) respectively. This variation
in the results is intuitive, since names have a lot more variation than the
year.

Figure 7: Examples of localization and labelling of target fields by Faster-
RCNN. The predicted bounding boxes are highlighted in green on the images. The
associated class labels are highlighted in red.

Figure 7 shows examples of outputs of Faster-RCNN on some documents from the
test set of the FIR dataset. The predicted bounding boxes are highlighted in
green rectangles, and the predicted class names are marked in red on top of
each bounding box.

Table 7: Finetuned (TrOCR) predictions on the generated image patches shown below Image Patches | OCR Results | Confidence Score
---|---|---
| 2019 | 0.89
| Lian Min Thang | 0.77
| Nscbi Airport | 0.79
| Amar  Prakesh | 0.63
| 379 | 0.96

The output of Faster-RCNN provides bounding boxes and field names for each
image, using which image patches are generated and sent to the Encoder-Decoder
architecture. Table 7 shows some examples of image patches and the finetuned
TrOCR predictions for those image patches. It is seen that the name “Amar
Prakash” is predicted as ‘Amar Prakesh” with confidence score below a
threshold of 0.7 (which was decided empirically). As the prediction confidence
is below the threshold, this output goes to the post-correction method
proposed in this work.

Table 8 compares the final performance of our proposed framework
TransDocAnalyser, and compares our model with Google-Tesseract and Microsoft-
TrOCR for handwritten recognition on proposed FIR dataset.999We initially
compared Tesseract with TrOCR-Base, and found TrOCR to perform much better.
Hence subsequent experiments were done with TrOCR only. The performances are
reported in terms of Character Error Rate (CER), Word Error Rate (WER), and
BLEU scores [30]. Lower values of CER and WER indicate better performance,
while higher BLEU scores are better.

Table 8: Benchmarking state-of-the-art TrOCR and our proposed framework TransDocAnalyser on the FIR dataset (best values in boldface) OCR models | Target Field | Evaluation Metrics
---|---|---
|  | $\mathbf{CER}\downarrow$ | $\mathbf{WER}\downarrow$ | $\mathbf{BLEU}\uparrow$
Tesseract-OCR | Year | $0.78$ | $0.75$ | $0.14$
| Statute | $0.89$ | $0.83$ | $0.12$
| Police Station | $0.91$ | $0.89$ | $0.10$
| Complainant Name | $0.96$ | $0.87$ | $0.9$
TrOCR-Base | Year | $0.38$ | $0.32$ | $0.72$
| Statute | $0.42$ | $0.38$ | $0.68$
| Police Station | $0.50$ | $0.44$ | $0.62$
| Complainant Name | $0.62$ | $0.56$ | $0.56$
TrOCR-Large | Year | $0.33$ | $0.32$ | $0.75$
| Statute | $0.34$ | $0.33$ | $0.73$
| Police Station | $0.36$ | $0.38$ | $0.65$
| Complainant Name | $0.51$ | $0.50$ | $0.57$
TrOCR-InLegalBert | Year | $0.17$ | $0.17$ | $0.84$
| Statute | $0.19$ | $0.21$ | $0.92$
| Police Station | $0.31$ | $0.26$ | $0.78$
| Complainant Name | $0.45$ | $0.39$ | $0.72$
TransDocAnalyser (proposed) | Year | $\mathbf{0.09}$ | $\mathbf{0.02}$ | $\mathbf{0.96}$
| Statute | $\mathbf{0.11}$ | $\mathbf{0.10}$ | $\mathbf{0.93}$
| Police Station | $\mathbf{0.18}$ | $\mathbf{0.20}$ | $\mathbf{0.83}$
| Complainant Name | $\mathbf{0.24}$ | $\mathbf{0.21}$ | $\mathbf{0.78}$

We achieve state-of-the-art results using the proposed TransDocAnalyser
framework which outperforms the other models with quite a good margin (see
Table 8). While the TrOCR + InLegalBert model also performed well, our
proposed framework TransDocAnalyser (consisting of vision transformer-based
encoder, InLegalBert tokenizer and KNN-based post-correction) achieved the
best results across all the four target fields of the FIR dataset.

##  7 Conclusion

In this work, we (i) developed the first dataset for semi-structured
handwritten document analysis in the legal domain, and (ii) proposed a novel
framework for offline analysis of semi-structured handwritten documents in a
particular domain. Our proposed TransDocAnalyser framework including Faster-
RCNN, TrOCR, a domain-specific language model/tokenizer, and KNN-based post-
correction outperformed existing OCRs.

We hope that the FIR dataset developed in this work will enable further
research on legal document analysis which is gaining importance world-wide and
specially in developing countries. We also believe that the TransDocAnalyser
framework can be easily extended to semi-structured handwritten document
analysis in other domains as well, with a little fine-tuning.

Acknowledgement: This work is partially supported by research grants from
Wipro Limited (www.wipro.com) and IIT Jodhpur (www.iitj.ac.in).

## References

  * [1] Amano, A., Asada, N.: Complex table form analysis using graph grammar. In: Proceedings of the 5th International Workshop on Document Analysis Systems V. p. 283–286. Springer-Verlag, Berlin, Heidelberg (2002)
  * [2] Amano, A., Asada, N., Mukunoki, M., Aoyama, M.: Table form document analysis based on the document structure grammar. International Journal on Document Analysis And Recognition (IJDAR) (2006)
  * [3] Bag, S., Harit, G.: A medial axis based thinning strategy and structural feature extraction of character images. In: Proceedings of IEEE International Conference on Image Processing. pp. 2173–2176 (2010)
  * [4] Bag, S., Harit, G.: An improved contour-based thinning method for character images. Pattern Recogn. Lett. 32(14), 1836–1842 (oct 2011)
  * [5] Bag, S., Harit, G.: Topographic feature extraction for bengali and hindi character images. Signal & Image Processing : An International Journal 2 (07 2011)
  * [6] Bhattacharya, P., Hiware, K., Rajgaria, S., Pochhi, N., Ghosh, K., Ghosh, S.: A comparative study of summarization algorithms applied to legal case judgments. In: Proceedings of European Conference on Information Retrieval (ECIR). pp. 413–428 (2019)
  * [7] Bruno, T., Sasa, M., Donko, D.: KNN with TF-IDF based framework for text categorization. Procedia Engineering 69, 1356–1364 (2014)
  * [8] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Computer Vision – ECCV 2020\. pp. 213–229. Springer International Publishing (2020)
  * [9] Chowdhury, A., Vig, L.: An efficient end-to-end neural model for handwritten text recognition. In: British Machine Vision Conference (2018)
  * [10] Constum, T., Kempf, N., Paquet, T., Tranouez, P., Chatelain, C., Brée, S., Merveille, F.: Recognition and information extraction in historical handwritten tables: Toward understanding early 20th century paris census. In: Proceedings of IAPR Workshop on Document Analysis Systems (DAS). p. 143–157 (2022)
  * [11] Diesendruck, L., Marini, L., Kooper, R., Kejriwal, M., McHenry, K.: A framework to access handwritten information within large digitized paper collections. In: Proceedings of IEEE International Conference on E-Science. pp. 1–10 (10 2012)
  * [12] Divya, S., Gaurav, H.: Associating field components in heterogeneous handwritten form images using graph autoencoder. In: 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW). vol. 5, pp. 41–46 (2019)
  * [13] Duong, Q., Hämäläinen, M., Hengchen, S.: An unsupervised method for OCR post-correction and spelling normalisation for Finnish. In: Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa). pp. 240–248 (2021)
  * [14] Girshick, R.: Fast r-cnn. In: Proceedings of IEEE International Conference on Computer Vision (ICCV). pp. 1440–1448 (2015)
  * [15] Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. pp. 580–587 (2014)
  * [16] Ha, H.T., Medved’, M., Nevěřilová, Z., Horák, A.: Recognition of ocr invoice metadata block types. In: Proceedings of Text, Speech, and Dialogue. pp. 304–312 (2018)
  * [17] Harley, A.W., Ufkes, A., Derpanis, K.G.: Evaluation of deep convolutional nets for document image classification and retrieval. In: 2015 13th International Conference on Document Analysis and Recognition (ICDAR). pp. 991–995 (2015)
  * [18] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of IEEE International Conference on Computer Vision (ICCV). pp. 2980–2988 (2017)
  * [19] Hegghammer, T.: OCR with Tesseract, Amazon Textract, and Google Document AI: a benchmarking experiment. Journal of Computational Social Science 5(1), 861–882 (May 2022)
  * [20] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (nov 1997)
  * [21] Huang, Z., Chen, K., He, J., Bai, X., Karatzas, D., Lu, S., Jawahar, C.V.: Competition on scanned receipt ocr and information extraction. In: Proceedings of International Conference on Document Analysis and Recognition (ICDAR). pp. 1516–1520 (2019)
  * [22] Jaume, G., Kemal Ekenel, H., Thiran, J.P.: Funsd: A dataset for form understanding in noisy scanned documents. In: 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW). vol. 2, pp. 1–6 (2019)
  * [23] Kim, G., Hong, T., Yim, M., Nam, J., Park, J., Yim, J., Hwang, W., Yun, S., Han, D., Park, S.: Ocr-free document understanding transformer. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision – ECCV 2022. pp. 498–517. Springer Nature Switzerland (2022)
  * [24] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C., Bottou, L., Weinberger, K. (eds.) Advances in Neural Information Processing Systems. vol. 25. Curran Associates, Inc. (2012)
  * [25] Li, M., Lv, T., Chen, J., Cui, L., Lu, Y., Florencio, D., Zhang, C., Li, Z., Wei, F.: TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models. In: Proceedings of AAAI (2023)
  * [26] Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 936–944 (2017)
  * [27] Marti, U.V., Bunke, H.: The iam-database: An english sentence database for offline handwriting recognition. International Journal on Document Analysis and Recognition 5, 39–46 (11 2002)
  * [28] Michael, J., Labahn, R., Grüning, T., Zöllner, J.: Evaluating sequence-to-sequence models for handwritten text recognition. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 1286–1293 (2019)
  * [29] Palm, R.B., Winther, O., Laws, F.: Cloudscan - a configuration-free invoice analysis system using recurrent neural networks. In: Proceedings of IAPR International Conference on Document Analysis and Recognition (ICDAR). pp. 406–413 (2017)
  * [30] Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: A Method for Automatic Evaluation of Machine Translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. p. 311–318 (2002)
  * [31] Paul, S., Goyal, P., Ghosh, S.: LeSICiN: A Heterogeneous Graph-Based Approach for Automatic Legal Statute Identification from Indian Legal Documents. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36 (2022)
  * [32] Paul, S., Mandal, A., Goyal, P., Ghosh, S.: Pre-trained Language Models for the Legal Domain: A Case Study on Indian Law. In: Proceedings of the International Conference on Artificial Intelligence and Law (ICAIL) (2023)
  * [33] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Proceedings of the International Conference on Neural Information Processing Systems - Volume 1. p. 91–99. MIT Press (2015)
  * [34] Subramani, N., Matton, A., Greaves, M., Lam, A.: A survey of deep learning approaches for OCR and document understanding. CoRR abs/2011.13534 (2020)
  * [35] Tan, M., Pang, R., Le, Q.V.: Efficientdet: Scalable and efficient object detection. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 10778–10787 (2020)
  * [36] Watanabe, T., Luo, Q., Sugie, N.: Layout recognition of multi-kinds of table-form documents. IEEE Trans. Pattern Anal. Mach. Intell. 17(4), 432–445 (apr 1995)
  * [37] Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable DETR: Deformable Transformers for End-to-End Object Detection. In: Proceedings of International Conference on Learning Representations (ICLR) (2021)