# We Need to Talk About Classification Evaluation Metrics in NLP

Peter Vickers
University of Sheffield
<EMAIL_ADDRESS>
&Loïc Barrault
Meta AI France
<EMAIL_ADDRESS>
Emilio Monti
Amazon U.K.
<EMAIL_ADDRESS>
&Nikolaos Aletras
University of Sheffield
<EMAIL_ADDRESS>
###### Abstract

In Natural Language Processing (NLP) classification tasks such as topic
categorisation and sentiment analysis, model generalizability is generally
measured with standard metrics such as Accuracy, F-Measure, or AUC-ROC. The
diversity of metrics, and the arbitrariness of their application suggest that
there is no agreement within NLP on a single best metric to use. This lack
suggests there has not been sufficient examination of the underlying
heuristics which each metric encodes. To address this we compare several
standard classification metrics with more ‘exotic’ metrics and demonstrate
that a random-guess normalised Informedness metric is a parsimonious baseline
for task performance. To show how important the choice of metric is, we
perform extensive experiments on a wide range of NLP tasks including a
synthetic scenario, natural language understanding, question answering and
machine translation. Across these tasks we use a superset of metrics to rank
models and find that Informedness best captures the ideal model
characteristics. Finally, we release a Python implementation of Informedness
following the SciKitLearn classifier format.111Code and documentation is
available at https://github.com/petervickers/aacl-informedness

##  1 Introduction

Some of the most widely used classification metrics for measuring classifier
performance in NLP tasks are Accuracy, F1-Measure and the Area Under the Curve
- Receiver Operating Characteristics (AUC-ROC). For example, seven out of nine
tasks of popular NLP benchmark GLUE (Wang et al., 2018) use either Accuracy or
F1.

Such metrics reduce the full collection of true classes $y$ and predicted
classes $\hat{y}$ to a single scalar value. For instance accuracy, the most
common classification metric, is equal to the proportion of predicted classes
which match true classes. Whilst capturing all the qualities of a classifier
in any single scalar value is rather impossible (Chicco et al., 2021), the
quality of the heuristic rule (Valverde-Albacete et al., 2013) influences both
the overall ranking of models and the intra-task understanding of model
capability.

It is difficult to evaluate true model ability with Accuracy due to the
‘Accuracy Paradox’ (Ben-David, 2007): simply guessing the most common class
can reward a score equal to that class’s prevalence in the test set. We expand
this paradox into two phenomena: (1) the reward given to models that predict
more classes which appear more often (are more prevalent) (Lafferty et al.,
2001); and (2) the probabilistic lower bound for accuracy being much greater
than zero for random guessing models in most realistic scenarios, a phenomenon
we term _baseline credit_ (Youden, 1950).

F1-Measure (Manning and Schütze, 1999) is the harmonic mean of precision and
recall and so represents a balance of two desirable characteristics of
classifiers. F1 is defined against a single class, and so within even a binary
classification case its value changes if the classes are reversed.
Additionally, the weighting of precision and recall is a function of the model
itself (Hand and Christen, 2018), making it a poor metric for ranking models.
In order to handle the multi-class case, macro- and micro- averaging
strategies have been proposed. In the single-label case we consider, micro
averaging is reduced to Accuracy, whilst macro-averaging is equivalent to
averaging the F1 score across all classes. Therefore, F1-Macro retains both
the biases of F1 in the single class case and introduces a further heuristic
in weighting all classes equally regardless of class prevalence.

An alternative to the F-Measure, the Receiver Operating Characteristic (ROC)
curve visually presents the trade-off between Recall and Precision as a
function of the decision threshold. The Area Underneath the ROC Curve (AUC) is
a metric which integrates the ROC curve to return a scalar value. As Hand
(2009) has shown, AUC is effectively applying a cost function dependent on the
False Positive Rate of the specific classifier, so systems cannot be compared
if they have different False Positive Rates.

In this paper we perform an extensive empirical analysis of various
classification metrics in synthetic and real settings. We advocate for using
Informedness, an unbiased and cognitively plausible multi-class classification
metric (Powers, 2003, 2013) for comparing classification performance of
different models instead of common metrics such as accuracy and F1. This
metric avoids crediting modes exhibiting guessing or bias which distort the
comparability of mainstream classification models. Informedness reports the
proportion of the time a classifier makes an informed decision; that is, a
decision better than bias exploitation strategies. Finally, it allows
comparison between tasks of different bias or complexity, and negates the need
for dataset re-balancing to ‘fit the metric’.

Our main contributions are as follows:

  * •

A definition of Informedness as a classification metric suited to NLP
applications

  * •

Synthetic and real task comparisons of Informedness against an extensive list
of classification metrics

  * •

An in-depth analysis on how the use of different metrics can affect model
ranking and within task understanding of model capabilities

  * •

Python implementation of Informedness and Normalised Information Transfer to
encourage further study within the community

##  2 Classification Evaluation Metrics

We begin by defining various classification metrics and discussing their
strengths and limitations.

Metrics operate over a set of classifications, where a true class $y$ and a
predicted class $\hat{y}$ are the two elements in each classification. Both
$y$ and $\hat{y}$ are indications of a class from out of a set of classes $C$.
The full classification output

$\left\\{\begin{array}[]{c}\\{y=C_{0},\hat{y}=C_{0}\\},\\\
\\{y=C_{1},\hat{y}=C_{0}\\},\\\ \\{y=C_{1},\hat{y}=C_{1}\\},\\\
\vdots{}\end{array}\right\\}$

is unwieldy, so a metric is used to reduce the set more compact form,
typically a single scalar value. First, the set of classifications may be
considered as a Confusion Matrix (or contingency table), which is an
$N\times{}N$ matrix with the columns by convention indicating the true class
and the rows indicating the predicted class. Cells are assigned the number of
classification events for the given actual and predicted class. In most NLP
cases, creating a classification matrix is a non-destructive operation as the
only information lost is the order of the classifications.

As part of our definitions, we introduce the per-class contingency table:

| Class of Interest c | Other Class | Real Class
---|---|---|---
Class of Interest c | TPc | FPc |
Other Class | FNc | TNc |
Predicted Class |  |  |

Table 1: Classification Contingency Table

We define this table for a class of interest c. In the binary case, this would
be one of two classes and hence two tables could be created, each the 180°
rotation of the other. In the multi-class case, there will be $c$ such
matrices.

From this table we also introduce Class Prevalence: the proportion of all
samples which have a given real class, and Class Bias: the proportion of all
samples which have a given predicted class. Prevalence is
(TP+FN)/(TP+FN+TN+FN). Prediction Bias is (TP+FP)/(TP+FN+TN+FN).

Since an $N\times{}N$ is considered too complex to compare models, a further
simplification is often used to produce a single scalar value. As this
reduction is an information-destructive operation (Chicco et al., 2021), the
heuristic rule (Valverde-Albacete et al., 2013) which the metric applies to
obtain a single value will determine what that metric considers be a ‘good’
model.

#### Accuracy:

It is defined as the proportion of correctly identified samples out of a total
set of evaluation samples. Accuracy encodes the heuristic that the best model
will have the most correctly predicted instances. This prior allows for the
‘accuracy paradox’ where an uninformed model may guess the most common class
artificially overestimating the generalizability score.

$\displaystyle\textrm{Accuracy}=\frac{1}{S}\sum_{c=0}^{C}TP_{c}$ (1)

where $C$ is the number of classes, $TP_{c}$ is the number True Positives for
class $c$ and $S$ is the total number of samples.

#### Balanced Accuracy:

This is a variant designed to counteract the class-frequency-weighted nature
of accuracy (Brodersen et al., 2010). As shown by Chicco et al. (2021), the
binary case is equivalent to a re-scaled Informedness (see below).

#### F-Measure:

This metric is defined as the geometric mean of the Precision and Recall of a
binary classifier.

$\displaystyle\textrm{F1-Macro}=\frac{1}{C}\sum_{c=0}^{C}\frac{TP_{c}}{TP_{c}+\displaystyle\frac{1}{2}(FP_{c}+FN_{c})}$
(2)

where $TP_{c},FP_{c},FN_{c}$ denote True Positives, False Positives and False
Negatives for each class $c$. In the multi-class case (3+ classes), those are
computed for each class in turn. F1-Macro encodes the heuristic that the
average of F1-Measure for all classes is a good representation of model
performance. However, this has no intuitive interpretation. Additionally, as
the number of negative samples increases, the number of samples which are
misclassified as positive will also increase. As F1 is independent of the
total number of samples, it ignores this important component of model
assessment. F-Measure may be generalised to multi-class classification through
micro or macro averaging. Micro-averaging sums the True Positives, False
Positives, and False Negatives when calculating Precision and Recall, and is
equivalent to accuracy in the uni-label case. Macro-averaging takes the
arithmetic mean over Precision and Recall for every class.

#### Kappa:

This is a family of metrics which calculate the inter-annotator reliability
between annotators, rather than the performance of a classifier on a task.
However, they account for the probability of chance agreement. Given
annotators $a_{0},a_{1}$ they take the general form:

$k=\frac{\text{Accuracy}(a_{0},a_{1})-\text{Chance
Agreement}(a_{0},a_{1})}{1-\text{Chance Agreement}(a_{0},a_{1})}$ (3)

Kappa metrics differ in how they estimate from how the chance agreement is
calculated (Cohen, 1960). It is possible to use Kappa as a metric for
classification systems by defining the system and the true labels as
annotators (Ben-David, 2007). However, Powers (2012) has shown that Kappa is
unfair to models in cases where the rates of true classes and predicted
classes are unequal.

#### Informedness:

This metric treats classification evaluation as an ‘odds game’, where a model
with no predictive capability is unable to gain any credit through either
label bias or baseline credit. It was first proposed in the binary case as
Youden’s J-statistic (Youden, 1950) and was generalised to the multi-class
case in Powers (2003). Informedness is defined as the proportion of samples
for which the model guesses better than random chance. The expected value of a
model which is always correct is 1, and the expected value of a model which
predicts correctly x% of the time, and guesses from the prevalence 100-x% of
the time is x.

For a class with an empirical probability (prevalence) of $p(y=c)$, the gain
(or loss) $i$ for a single prediction is computed as:

$i(y,\hat{y})=\begin{cases}\displaystyle\frac{1}{p(y=c)}\mathrm{\hskip
31.2982pt~{}if~{}}\hat{y}=c\\\\[15.05624pt]
\displaystyle-\frac{1}{1-p(y=c)}\mathrm{~{}~{}~{}if~{}}\hat{y}\neq{}c\end{cases}$
(4)

where $p(y=c)$ is the empirical probability of class $c$, calculated from the
test set. Scores are aggregated across the whole classification set as:

$\textrm{I}=\sum_{c=0}^{C}\frac{p(\hat{y}=c)}{N}\sum_{y}\mathbbm{1}(y=c)i(y,\hat{y})$
(5)

Where $\mathbbm{1}(y=c)$ is an indicator function which takes 1 when $y=c$ and
0 otherwise.

#### Mathew’s Correlation Coefficient (MCC):

MCC is a measure of the correlation of the predicted classes $\hat{y}$ with
the true classes $y$. Whilst its definition ensures that random guessing will
score 0, for any model better than random guessing, it will not report the
possibility of random chance. MCC is dependent on the relative frequencies of
classes in the test set, which makes comparison between models evaluated on
different datasets impossible (Chicco et al., 2021). Formally, MCC is defined
as:

$\textrm{MCC}=\frac{\textrm{Cov}(\hat{y},y)}{\sigma_{\hat{y}}\cdot{}\sigma_{y}}$
(6)

#### Normalized Information Transfer (NIT):

This information-theoretic measure reports the degree to which the classifier
reduces the uncertainty of the input distribution by considering the
information transfer through the classifier. It was introduced by Valverde-
Albacete et al. (2013). Formally, NIT is defined as:

$\textrm{NIT}=2^{\textrm{MI}_{\hat{y},y}-H_{U_{y}}}$ (7)

Where $\textrm{MI}_{\hat{y},y}$ is the Mutual Information of the Real and
Predicted Classes, whilst $H_{U_{y}}$ is the Entropy of the Real Classes if
they come from a uniform distribution.

As with Informedness, NIT considers prevalence, forcing classifiers to add
Shannon Information, that is, to correctly classify samples, in order to
increase the metric score.

(a) Binary classification case

(b) Multi-class classification case

Figure 1: Accuracy, Balanced Accuracy, F1-Macro, Informedness, MCC, and NIT of
the same binary (top) or multi-class (bottom) classifier as a function of the
class distribution and the model’s prediction capability from 0% (Random
Guess) to 100% (Perfect).

##  3 Experiment 1: Metric Evaluation on a Toy Setting

We first compare the metrics outlined in Section 2 on a toy setting, aiming to
unveil the main differences between them. We assume a simulated model as
follows:

  * •

First, we sample from a uniform distribution [0,1] and then pick the correct
label if the sample is smaller than model predictive power;

  * •

Otherwise, we randomly sample from the class-prevalence weighted output
distribution.

  * •

We score a simulated model with a fixed probability of making a correct
classification

We believe this is an acceptable representation of how a reasonably designed
and trained neural network would behave.

Figure 1 shows the performance of a binary (top) and multi-class (bottom)
classifier as a function of the class distribution and the model’s predictive
capacity from random guess to perfect.

In the binary case, we first observe that Accuracy becomes more distored as as
the prevalence of either class increases. On the other hand, Balanced Accuracy
and F1-Macro score are robust against prevalence, but are susceptible to
random chance exploitation. Surprisingly, the NIT is superficially similar to
accuracy. This can be explained by the fact that when one class is far more
probable than the others, the Mutual Information between a random distribution
sampled from the same prior is high.

In both binary and multi-class cases MCC-Macro appears to behave exactly as
Informedness. This only holds in the case where the classification ability of
the model is constant across classes (Chicco et al., 2021). We simulate model
ability as a function of prevalence, so our figures do not capture this
dynamic of the MCC-Macro. However, we do show that in this case Informedness
correctly identifies the underlying probability of the model making an
informed decision.

##  4 Experiment 2: Metric Evaluation on Natural Language Understanding Tasks

| Single Sentence | Similarity and Paraphrase | Natural Language Inference |
---|---|---|---|---
Model (Metric) | CoLA | SST-2 | MRPC | QQP | STS-B | MNLI-M | MNLI-MM | QNLI | RTE | WNLI | All
DistillBERT (Acc.) | 79.7 | 90.5 | 84.2 | 77.4 | 51.8 | 81.4 | 81.6 | 88.6 | 57.6 | 56.3 | 74.9
DistillBERT (Inform.) | 57.0 | 81.0 | 69.4 | 77.4 | 41.6 | 72.1 | 72.5 | 77.2 | 14.7 | -43.1 | 52.0
Random Guess (Acc.) | 58.1 | 51.4 | 56.7 | 53.5 | 18.3 | 33.5 | 33.6 | 50.0 | 49.9 | 51.8 | 45.7
Random Guess (Inform.) | 01.2 | 02.8 | -01.1 | 00.0 | 01.0 | 00.1 | 00.5 | 00.0 | -00.3 | 02.0 | 00.6
$\Delta$ Accuracy | 21.6 | 39.1 | 27.5 | 23.9 | 33.5 | 47.9 | 48.0 | 38.6 | 07.7 | 04.5 | 29.2
$\Delta$ Informedness | 55.8 | 78.2 | 70.5 | 77.4 | 40.6 | 72.0 | 72.0 | 77.2 | 15.0 | -45.1 | 51.4

Table 2: GLUE Results. See Wang et al. (2018) for tasks details and evaluation
metrics. All values are scaled by 100. ‘All’ is a uniform weighted mean of the
individual metric scores as in https://gluebenchmark.com/leaderboard.

Next, we compare metrics across a range of NLU tasks and show that the metric
choice affects the model ranking. First, we test on the GLUE Multi-Task
Natural Language Understanding Benchmark. GLUE is a suite of nine NLP tasks
representing a range of domains, biases, and difficulties (Wang et al., 2018).
Interestingly the GLUE employs different metrics across tasks, i.e. Accuracy,
MCC, Pearson Correlation and Spearman’s Correlation. MCC is a discretised
version of the Pearson correlation and Spearman’s Correlation is the Pearson
Correlation calculated on the Rank transformation of the values. To make the
continuous $[0,5]$ STS-B task values tractable for classification metrics, we
discretize into $[0,5]\cap\mathbb{Z}$ by rounding to the nearest integer.

We experiment with following two approaches:

  * •

Random Guess: A ‘most likely’ guesser, which chooses the most common class
from training;

  * •

DistilBERT: We also finetune DistilBERT (Sanh et al., 2019) for five epochs on
each sub-task.

Table 2 shows model performance across models, metrics and tasks. For the sake
of clarity, the last two lines show the difference between DistilBERT and
Random Guess scores. The ‘All’ column is a uniform-weighted mean of the metric
scores across the GLUE tasks. In the case of informedness, it represents the
average probability of an informed decision across all nine tasks. The use of
Informedness across the GLUE tasks allows for direct comparison with the
knowledge that bias is discounted.

First, we note that sampling classes according to their prior probability (see
Guess rows) produces high accuracy scores for many tasks whilst Informedness
remains very close to 0. This fact makes it clear that Informedness provides a
more interpretable metric when it comes to evaluating model capability. For
all tasks, we observe a lower Informedness than Accuracy. This is expected due
to the properties of the metrics shown in Figure 1. For unbalanced tasks
(CoLA, MRPC, WNLI), the gap between accuracy and Informedness is increased as
Informedness removes the label bias gain. In the three-class tasks (MNLI-M and
MNLI-MM), the delta between accuracy and Informedness is reduced but still
pronounced.

WNLI is the most interesting result. DistilBERT accuracy (56.3) is a small
amount (4.5) larger than random guessing which suggests a weakly predictive
model. However, Informedness is strongly negative (-43.1), which suggests that
the model is underperforming the prior class distribution to a large degree.
We hypothesise this is because the WNLI task is adversarial. We quote the GLUE
authors: ‘Due to a data quirk, the development set is adversarial: hypotheses
are sometimes shared between training and development examples, so if a model
memorizes the training examples, they will predict the wrong label on
corresponding development set example.’ (Wang et al., 2018) Here accuracy
suggests a weak model, whilst Informedness reports the real behaviour.

Another advantage of Informedness is the possibility of direct comparison
between tasks with varying bias (e.g. CoLA and SST-2) and varying classes
(e.g. CoLA and MNLI) without the need to correct for prevalence. Because MCC
gives each class equal weight, it cannot be used to compare across tasks with
varying class distributions (Chicco et al., 2021). Informedness and NIT
support comparison between tasks, but NIT may be confusing for task comparison
as it awards credit for guessing.

##  5 Experiment 3: Metric Evaluation in Visual Question Answering

Visual Question Answering (VQA) is the task of answering a question about an
image and is often cast as a classification task which requires selecting a
correct answer from a large set of candidate classes (Antol et al., 2015). Due
to the real-world imbalances (for instance, more tables are made of wood than
marble), VQA datasets have high tendencies to inherent biases, making accuracy
a poor metric to use.

In this work, we consider two VQA datasets: (1) GQA (Hudson and Manning, 2019)
and (2) KVQA (Shah et al., 2019)

(a) Unbalanced dataset

(b) Balanced dataset

Figure 2: Metrics on GQA Unbalanced (left) and Balanced (right) validation
splits. Error-bars show the standard deviation across five runs. Numbers after
the question category are (question count) and [answer class entropy].

###  5.1 GQA

We select GQA for the high variance in class count and prevalence across
question types. It provides ‘unbalanced’ and ‘balanced’ versions. ‘Unbalanced’
is the default dataset and features a strong prevalence skew due to real world
biases towards certain classes. ‘Balanced’ is a resampled version of dataset
where the class distributions have been resampled to reduce the class
imbalance.

With GQA, we perform an intra-dataset comparison. Such a comparison is a
common step in model and dataset analysis when researchers wish to compare the
relative capabilities of a model on different sub-tasks. We provide a model
with a predictable behaviour by simulating a 50% probability of choosing the
correct answer and a 50% probability of sampling from the class prevalence
within a question type. For clarity, we only examine the low-frequency
categories ‘company’, ‘dir’ and ‘typeChoose’ and the high-frequency categories
‘relO’, ‘exist’, and ‘existRel’. Results for a representative sub-set of the
question types are shown in Figure 2. Refer to Appendix A for the full dataset
results.

| Dataset | Metric
---|---|---
Question type | Classes | Entropy | Accuracy | F1-Macro | Informedness | MCC-Macro | NIT
1-Hop | 5336 | 7.4 | 66.9 | 10.8 | 64.6 | 10.8 | 25.8
1-Hop Count. | 5 | 1.1 | 79.3 | 38.9 | 58.1 | 31.5 | 58.1
1-Hop Subtr. | 66 | 4.1 | 26.5 | 03.0 | 18.8 | 02.9 | 17.3
Boolean | 2 | 1.0 | 94.9 | 63.2 | 89.7 | 89.7 | 81.9
Comparison | 11 | 2.1 | 91.1 | 37.0 | 90.2 | 47.3 | 84.9
Counting | 9 | 2.1 | 80.9 | 56.1 | 75.4 | 56.2 | 61.2
Intersect. | 2 | 1.0 | 79.5 | 78.5 | 56.3 | 59.5 | 62.1
Multi-Ent. | 81 | 3.2 | 78.0 | 10.8 | 76.1 | 12.0 | 56.5
Multi-Hop | 119 | 3.6 | 87.9 | 34.8 | 87.0 | 43.9 | 68.9
Multi-Relat. | 4104 | 6.8 | 75.4 | 11.7 | 73.7 | 12.1 | 38.1
Spatial | 1260 | 10.0 | 19.9 | 07.4 | 18.6 | 09.2 | 16.3
Subtract. | 93 | 5.9 | 39.8 | 36.6 | 45.9 | 34.3 | 08.6

Table 3: Model performance on KVQA across metrics.

First, we have many cases where Accuracy, Balanced Accuracy and F1-Macro are
75% on binary questions. This baseline credit makes it hard to compare between
model performance, which is calibrated to be uniform, across dataset sub-
tasks. Practically, we are not able to use Accuracy, F1-Macro, or NIT to look
at ‘typeChoose’ questions and see if the model is as strong as on ‘existRel’.
Meanwhile, MCC-Macro and Informedness converge on the correct value (0.5) even
with the 46 samples in ‘dir’ question type. The ‘dir’ case demonstrates how
the deletion of samples to create a more uniform prevalence is not required
with sophisticated metrics. That is, Informedness and MCC are closer to the
true value for ‘dir’ with the unbalanced sample than with the balanced one.
Meanwhile, the balanced dataset has only a minor effect on accuracy and
F1-score, with ‘dir’ and ‘typeChoose’ questions being slightly closer to an
unbiased score. This reinforces our hypothesis that dataset balancing is not
the correct approach to evaluation.

For the questions with many samples (‘relO’, ‘exist’, and ‘existRel’), all
metrics have low variance. For ‘exist’, and ‘existRel’, F1-Macro and Accuracy
converge on 0.75, which reflects correctly predicting a binary task half the
time, and randomly guessing the other half. For the ‘relO’ question class,
Accuracy and F1-Macro tend to the true proportion of the time the model is
predicting the correct answer, but this can be attributed to the higher
entropy for this class of questions. The same behaviour can be observed for
additional question types in Appendix A.

These experiments show that that Informedness automatically accounts for
prevalence imbalance and provides a better assessment of the model capability.
Whilst MCC appears similar, it over-punishes classifiers which have variable
per-class performance Chicco et al. (2021), which we do not believe is in line
with desired characterises of classifiers in NLP.

###  5.2 KVQA

Having established metric characteristics through controlling model
performance, we now move to model evaluation in the wild. First, the KVQA
dataset (Shah et al., 2019) provides multiple question type attributes for
each question. The task requires reasoning over retrieved knowledge graph
facts as well as arithmetical operations. For modelling, we select ‘REUNITER’,
a simple yet effective transformer based model (Vickers et al., 2021), and re-
evaluate it with informedness.

We are interested in this case for the opportunity to have a metric which
allows comparison within a dataset between subsections with different class
distributions. We present results across unbiased metrics Informedness, MCC-
Macro and NIT (Powers, 2003; Chicco et al., 2021; Valverde-Albacete and
Peláez-Moreno, 2014) along with accuracy grouped by question type in Table 3.

The ‘1-Hop’ category is a superset of many question types requiring a single
KG fact to answer. This question type is scored very differently across all
metrics but the difference between Informedness (64.6%), MCC-Macro (10.8%) and
NIT (25.8%) is especially striking given the agreement between Informedness
and NIT in the synthetic case from Section 5.1. This range indicates the model
is doing well in general: if it were guessing from a prior, it would have an
Informedness of zero. The difference can be explained by the different
dynamics of Informedness and MCC raised above. The model is much better than
random chance at predicting certain popular classes, but struggles with low-
frequency obscure classes. This is supported by a high accuracy at the same
time as a low F1-Macro (12.9). In this case, F1-Macro, MCC, and NIT harshly
and unfairly penalize the model.

Looking at the ‘Intersection’ type, we see the opposite behaviour. Accuracy
and F1-Macro are all fairly high (78.5 and above) while Informedness is rather
low (56.3). This means that Accuracy and F1-Macro exaggerate the predictive
power of the model for this type of question. The similar score of MCC-Macro
(59.5%) to Informedness indicates that the model has even performance across
classes.

Interestingly, accuracy reports that the model is poor at ‘subtraction’
questions, which Informedness is much higher (45.9). We hypothesise this is
because (1) transformer models are not good at arithmetic without extensive
task-specific pretraining and (2) the high number of output labels will have
lower baseline credit.

Through the use of Informedness, we come to a different conclusion of the
relative strengths of the model. We find that the model has better
mathematical ability than accuracy indicated, whilst the ability to reason
over intersectional facts is much poorer than accuracy reports. For example,
this could lead to focus on improving this sub-task in the future.

Meanwhile, we have the issue that both Informedness and NIT are proposed as
suitable metrics for reporting the cross-task capability of different
classifiers, but they report divergent scores and sub-task rankings. This is
because both metrics target different criteria: NIT the transmission of
information from the true labels to the predicted labels, and Informedness the
probability of an informed decision. We propose that Informedness is a more
intuitive measure for NLP, and refer to Section 3 for a toy example
demonstration.

##  6 Experiment 4: Metric Evaluation on Formality Control for Spoken Language
Translation

| Off-the-shelf MT | Formality-aware MT
---|---|---
Accuracy | 50.0 | 95.4
Balanced Accuracy | 50.0 | 95.3
F1-Macro | 49.2 | 95.4
Informedness | 00.0 | 91.8

Table 4: Metric scores on Formality Control for Spoken Language Translation
(En-De) between off-the-shelf and formality-aware MT systems.

In the last set of experiments, we consider a contextual task involving
machine translation (MT). The Special Task on Formality Control for Spoken
Language Translation (Anastasopoulos et al., 2022) evaluates an MT model to
correctly express the desired formality (either formal or informal) in its
translation hypotheses. Focusing on the English-to-German language pair, we
use the winning system proposed by Vincent et al. (2022). The model is trained
to recognise a formality token to generate adequate translations, and an off-
the-shelf formality-unaware MT model on the test set provided by the
organisers. We report accuracy, Balanced accuracy, F1-Macro and Informedness
on the English-to-German test set.

Table 4 displays metric scores between off-the-shelf and formality-aware MT
systems. We see that the model with no knowledge of the formality is still
able to achieve accuracy and F1-score of around 0.5, which seems to mean that
the model is able to correctly produce a translation with correct formality
50% of the time. Meanwhile, Informedness drops to zero. As the dataset is
balanced, this is a product of Informedness removing baseline credit making it
a more suitable choice as an evaluation metric.

Overall, Informedness provides a better and more interpretable measure of the
system capability to model the task. This demonstrates that Informedness can
be used as an effective tool for comparing two different systems.

##  7 Discussion

###  7.1 Limitations of current metrics

The results obtained across all experiments highlight that widely-used metrics
(e.g. Accuracy, F1-Macro) for classification evaluation in NLP feature biases
which suggest higher performance than either intuitive reasoning or
information theory support. Importantly, this bias makes comparing classifiers
across tasks with different class distributions impossible.

Additionally, through the analysis of a real model on the KVQA task, we showed
that traditional metrics are not suited to intra-dataset analysis when
evaluating a single model’s performance across various sub-tasks. This is
highly problematic, as knowing if a model is better at a particular sub-task
such as the sub-tasks of addition or syntactic parsing is crucial for model
analysis.

###  7.2 Improving Evaluation of Classification Tasks in NLP

Across all experiments, we found that Informedness better captures model
generalizability than all other metrics. Given this finding and the main
limitation of popular metrics such as Accuracy and F1 across different NLP
tasks, we encourage the community and practitioners to consider reporting
Informedness alongside metrics such as Accuracy and F1 in future experiments
and analyses.222For a discussion of the limitations of Informedness, see
Limitations section.

##  8 Conclusion

We have presented an extensive empirical analysis of various classification
metrics across a wide range of tasks including NLU, VQA and MT with controlled
formality. Our experiments demonstrated that the use of a class-invariant
metric, Informedness, allows for a fairer ranking and understanding of model
generalization capacity.

Whilst we find that Informedness is the most intuitive metric, we also found
that it is also the fairest in driving inter and intra-model comparisons.

Finally, we provide sklearn.metrics style implementations of both NIT and
Informedness, previously unavailable in Python

We hope that our work is the first step towards rethinking the way NLP
classification systems are evaluated in the future and will raise awareness to
the community.

## Limitations

Informedness cannot fully represent all of the characteristics of a
classification system within a single scalar value. It assumes that the
distribution of classes in the training and test set are identical. This
assumption is used to determine the loss and gain for a particular class
according to the distribution in the test set. However, we allow for train
class distributions to be passed to our implementation of Informedness.

In this work, we further assume that an uninformed model will reproduce the
training distribution. In the case that models are poorly parameterised, or
the testing set is very small, this may not be the case. This could lead to
models which are not using the input data to have Informedness scores other
than zero. Likewise, systems which use strategies such as ‘guess the most
common’ may have Informedness scores other than zero.

Informedness is sensitive to the number of evaluation samples, which may
result in less stable estimation of model’s performance in situations with low
numbers ($<50$) of examples. We consider that all metrics are subject to this
and that it is reasonable to expect that evaluation is performed on sizeable
test sets.

## Acknowledgements

This work was supported by the Centre for Doctoral Training in Speech and
Language Technologies (SLT) and their Applications funded by UK Research and
Innovation [grant number EP/S023062/1].

## References

  * Anastasopoulos et al. (2022) Antonios Anastasopoulos, Loïc Barrault, Luisa Bentivogli, Marcely Zanon Boito, Ondřej Bojar, Roldano Cattoni, Anna Currey, Georgiana Dinu, Kevin Duh, Maha Elbayad, Clara Emmanuel, Yannick Estève, Marcello Federico, Christian Federmann, Souhir Gahbiche, Hongyu Gong, Roman Grundkiewicz, Barry Haddow, Benjamin Hsu, Dávid Javorský, Vĕra Kloudová, Surafel Lakew, Xutai Ma, Prashant Mathur, Paul McNamee, Kenton Murray, Maria Nǎdejde, Satoshi Nakamura, Matteo Negri, Jan Niehues, Xing Niu, John Ortega, Juan Pino, Elizabeth Salesky, Jiatong Shi, Matthias Sperber, Sebastian Stüker, Katsuhito Sudoh, Marco Turchi, Yogesh Virkar, Alexander Waibel, Changhan Wang, and Shinji Watanabe. 2022.  Findings of the IWSLT 2022 evaluation campaign.  In _Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022)_ , pages 98–157, Dublin, Ireland (in-person and online). Association for Computational Linguistics.
  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015.  Vqa: Visual question answering.  _Proceedings of the IEEE International Conference on Computer Vision_ , 2015 Inter:2425–2433.
  * Ben-David (2007) Arie Ben-David. 2007.  A lot of randomness is hiding in accuracy.  _Engineering Applications of Artificial Intelligence_ , 20:875–885.
  * Brodersen et al. (2010) Kay H. Brodersen, Cheng Soon Ong, Klaas E. Stephan, and Joachim M. Buhmann. 2010.  The balanced accuracy and its posterior distribution.  _Proceedings - International Conference on Pattern Recognition_ , pages 3121–3124.
  * Chicco et al. (2021) Davide Chicco, Niklas Tötsch, and Giuseppe Jurman. 2021.  The matthews correlation coefficient (mcc) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation.  _BioData Mining_ , 14:1–22.
  * Cohen (1960) Jacob Cohen. 1960.  A coefficient of agreement for nominal scales.  _Educational and Psychological Measurement_ , 20:37–46.
  * Hand and Christen (2018) David Hand and Peter Christen. 2018.  A note on using the f-measure for evaluating record linkage algorithms.  _Statistics and Computing_ , 28:539–547.
  * Hand (2009) David J Hand. 2009.  Measuring classifier performance: A coherent alternative to the area under the roc curve.  _Mach. Learn._ , 77:103–123.
  * Hudson and Manning (2019) D. A. Hudson and C. D. Manning. 2019.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6693–6702, Los Alamitos, CA, USA. IEEE Computer Society.
  * Lafferty et al. (2001) John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001.  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.  In _Proceedings of the Eighteenth International Conference on Machine Learning_ , ICML ’01, page 282–289, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
  * Manning and Schütze (1999) Christopher D Manning and Hinrich Schütze. 1999.  _Foundations of Statistical Natural Language Processing_.  The MIT Press.
  * Powers (2003) David Powers. 2003.  Recall and precision versus the bookmaker.  _International Conference on Cognitive Science (ICCS)_ , pages 529–534.
  * Powers (2012) David M W Powers. 2012.  The problem with kappa.  _European Chapter of the Association for Computational Linguistics_ , 13:345–355.
  * Powers (2013) David M. W. Powers. 2013.  A computationally and cognitively plausible model of supervised and unsupervised learning.  In _Proceedings of the 6th International Conference on Advances in Brain Inspired Cognitive Systems_ , BICS’13, page 145–156, Berlin, Heidelberg. Springer-Verlag.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _ArXiv_ , abs/1910.0.
  * Shah et al. (2019) Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. 2019.  Kvqa: Knowledge-aware visual question answering.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 33:8876–8884.
  * Valverde-Albacete and Peláez-Moreno (2014) Francisco J. Valverde-Albacete and Carmen Peláez-Moreno. 2014.  100% classification accuracy considered harmful: The normalized information transfer factor explains the accuracy paradox.  _PLOS ONE_ , 9:e84217.
  * Valverde-Albacete et al. (2013) Francisco José Valverde-Albacete, Jorge Carrillo de Albornoz, and Carmen Peláez-Moreno. 2013.  A proposal for new evaluation metrics and result visualization technique for sentiment analysis tasks.  _Information Access Evaluation. Multilinguality, Multimodality, and Visualization_ , 8138:41–52.
  * Vickers et al. (2021) Peter Vickers, Nikolaos Aletras, Emilio Monti, and Loïc Barrault. 2021.  In factuality: Efficient integration of relevant facts for visual question answering.  _59th Annual Meeting of the Association for Computational Linguistics_ , pages 468–475.
  * Vincent et al. (2022) Sebastian Vincent, Loïc Barrault, and Carolina Scarton. 2022.  Controlling formality in low-resource NMT with domain adaptation and re-ranking: SLT-CDT-UoS at IWSLT2022.  In _Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022)_ , pages 341–350, Dublin, Ireland (in-person and online). Association for Computational Linguistics.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Youden (1950) W J Youden. 1950.  Index for rating diagnostic tests.  _Cancer_ , 3:32–35.

##  Appendix A GQA Full Comparison

Figure 3: Metrics on GQA Unbalanced. Questions are grouped by reasoning type
annotation on the X axis and sorted by count. X axis labels gives the
reasoning type, the number of samples, and the entropy of the answer class
distribution Figure 4: Metrics on GQA Balanced. Questions are grouped by
reasoning type annotation on the X axis and sorted by count in GQA Unbalanced
for comparison. X axis labels gives the reasoning type, the number of samples,
and the entropy of the answer class distribution


# SuperOCR: A Conversion from Optical Character Recognition
to Image Captioning

Baohua Sun, Michael Lin, Hao Sha, Lin Yang
Gyrfalcon Technology Inc.
1900 McCarthy Blvd Suite 208, Milpitas, CA, 95035
<EMAIL_ADDRESS>
###### Abstract

Optical Character Recognition (OCR) has many real world applications. The
existing methods normally detect where the characters are, and then recognize
the character for each detected location. Thus the accuracy of characters
recognition is impacted by the performance of characters detection. In this
paper, we propose a method for recognizing characters without detecting the
location of each character. This is done by converting the OCR task into an
image captioning task. One advantage of the proposed method is that the
labeled bounding boxes for the characters are not needed during training. The
experimental results show the proposed method outperforms the existing methods
on both the license plate recognition and the watermeter character recognition
tasks. The proposed method is also deployed into a low-power (300mW) CNN
accelerator chip connected to a Raspberry Pi 3 for on-device applications.

##  1 Introduction

Optical Characters Recognition (OCR) has many real-world applications.
Depending on the application scenarios, there are mainly two types of OCR
tasks. The first type of OCR task is the separated single character
recognition, such as MNIST [11] and CASIA handwritten Chinese character
recognition [13], which are generally treated as a classification problem; the
second OCR task is to recognize a string of characters at random locations
with various backgrounds in the image, such as Street View House Number
recognition (SVHN) [17] and CASIA handwritten Chinese text recognition [13].
Specially, the number of characters are fixed in some application scenarios,
such as license plate recognition [34] and watermeter characters recognition
[35]. In this paper, we focus on the second type of OCR task with fixed number
of characters.

Current methods [10, 9, 25, 22, 21, 16] for the second type of OCR task
normally detect each character first as an intermediate result, and then
recognize the character in each detected location. For real-world
applications, such as license plate recognition and watermeter numbers
recognition, people are only interested in the characters reading, instead of
the locations of the characters. Recent research [34, 12] also study the end-
to-end method to simultaneously detect and recognize the characters. However,
besides the characters recognition module, the characters detection module is
still designed within the DNN model.

There are problems with the current methods for both the training phase and
the inference phase. During the training phase, the labeled location
information, which is usually given in the format of bounding boxes, are
required in order to train the model. However, for the majority of real-world
applications, the interested output for the OCR tasks are the readings of the
characters in the image, instead of where these characters are located. These
location information during inference phase are only used as intermediate
result, and will be of no use once the characters recognition is completed.
However, due to the design of current methods, the location labels are mandate
in order to train the model, which unnecessarrily increase the cost of
obtaining the labeled traing data. In fact, labeling the bounding boxes of the
characters is much heavier work than only labeling the readings of the
characters. In real-world application scenarios, the boundary of the location
may also be rotated [34]. Therefore, some extra efforts for labeling the
characters location in quadrilateral areas are also needed in addition to the
bounding boxes, which adds more costs for labeling. As a result, the existing
methods are very expensive to train. For a task only interested in the
characters reading, these human labors could have been saved if the algorithm
design requires no characters location information for training.

During the inference phase, the detection module also has to calculate the
location of the characters in order for the recognition module to work on,
which results in an extra computation power consumption. The existing methods
are usually deployed on GPUs which consumes large volumes of power. In spite
of recent years development and wide availability of low-power CNN
accelerators [30, 28], the existing methods are hard to be fully deployed into
the edge devices. This is because the detection module usually has a different
network architecture from that of the image recognition module, which also
makes the inference difficult to be deployed on a single accelerator chip to
accelerate the heterogeneous network. One practical compromise is to partially
deploy the DNN network on the accelerator chip, and execute the rest of the
network on the host processor, e.g. CPU. However, this is not the most power-
efficient solution for OCR on the edge device, since the whole network is not
fully accelerated, and the power consumption on the host processor is still
very high. The ideal implementation should be the entire OCR DNN network
loaded into the low-power accelerator, and the host processor acting only as a
controller to send the input image and receive the output characters readings.

This paper proposes a method named SuperOCR to address the above mentioned
problems by eliminating the need of detection module in the OCR network. The
key idea is to convert the OCR task into an image captioning task.

The OCR task shares some common features with the image captioning task. Image
captioning takes an image as input, and outputs the text describing the
contents in the image [7, 1, 5, 8, 15, 20, 33, 36, 38]. Recent work of the
SuperCaptioning method [31] combines the multi-modal information of image and
text together, and uses a CNN model to predict the image caption iteratively
until a stopping criteria is met. This method is inspired by the recent
success of Super Characters method [29] in Natural Language Processing (NLP)
tasks, and borrows the concept of two-dimensional embedding to project the
text as part of an image. The SuperCaptioning method requires no knowledge of
the objects location, and only uses a CNN model for image recognition in each
iteration. Similar to the OCR tasks addressed in this paper, the image
captioning task also takes an image as input, and outputs a string of
characters or words.

In spite of these similarities, the OCR task and the image captioning task are
different in some ways. First, the answer for an OCR task is normally
objective and unique. However, this is not neccessary for the image captioning
task, which is more subjective and non-exclusive. An image can have multiple
correct captions given by different people from different perspectives of the
image; Second, the focus of an OCR task could be a small area located in a
large background, e.g. recognizing the house numbers in a street view image.
While the caption for an image may either pay attention to a small area or
describes only the general scenary; Third, some OCR applications have fixed
number of characters to be recognized, such as watermeter character
recognition and license plate recognition. However, the image captioning task
normally has no such constraint. These differences bring challenges when
treating the OCR problem as an Image Captioning problem.

The proposed SuperOCR method casts the OCR problem as a constrained image
captioning problem, and modifies the SuperCaptioning method to adapt to the
differences between these two problems. The contribution of this paper is
three-folded. First, to the best of our knowledge, this is the first paper to
address the OCR problem by converting it into an image captioning problem;
Second, similar to the Super Captioning method which requires no detection
module, the SuperOCR method recognizes the characters in random locations
without detecting the locations of the characters. This also lowers the cost
of building the OCR training set because it saves the labor for the labeled
bounding boxes of the characters; Third, the proposed method is also deployed
on a Raspberry Pi 3 with a 300mw low-power CNN accelerator chip. Experimental
results on car license plate recognition dataset CCPD [34] shows our method
obtains an accuracy of 98.7% on the CCPD-Base dataset, while the best existing
method performance is 98.5%. The significant improved accuracy is observed on
more challenging tasks of CCPD-Rotate(95.6% over 90.8%), CCPD-Weather(94.7%
over 87.9%), and CCPD-Tilt(96.0% over 92.5%). Another experiment on a
watermeter characters recognition dataset [35] also shows our method improves
the accuracy from 89.6% to 90.5%.

Figure 1: Illustration of the proposed SuperOCR method. The OCR task is
converted into an image captioning task, by restricting the image description
as the readings of the interested characters found in the image.

##  2 Proposed Method: SuperOCR

In this paper, we focus on the second type of OCR task which has a fixed
number of characters. The proposed SueprOCR method treats the OCR task as an
image captioning problem, and the characters are predicted iteratively by a
trained CNN model. This is very similar to the SuperCaptioning [31] method for
generating the image descriptions.

Fig. 1 illustrates the proposed SuperOCR method during the inference phase,
using the Chinese license plate recognition as an example. The input image is
first resized to occupy the upper portion of a new image called SuperOCR
image, and the lower portion of the SuperOCR image is reserved blank for the
characters to be predicted. The SuperOCR image is then fed into a trained CNN
model to predict the first character. The first predicted charater will then
be drawn onto the bottom left portion to update the SuperOCR image, as shown
in Fig. 1. Again, the updated SuperOCR image will be fed into the same CNN
model to predict the next character, and the predicted next character will be
drawn on the bottom portion of the SuperOCR image, next to the last predicted
chacter. This process repeats for the same number of iterations as the given
number of characters.

It can be seen that the proposed SuperOCR method predicts the characters in a
sequential manner. During each iteration, the prediction is an image
classificaiton problem. The number of classes is determined by the possible
values of the characters. This is similar to the vocabulary of words in the
image captioning tasks.

In order to train the CNN model in the SuperOCR method, the training dataset
containing the SuperOCR images needs to be generated from the labeled OCR
dataset. Only the labeled characters readings are utilized during the SuperOCR
image generation, while the locations of the characters are not needed.

###  2.1 Generate the Labeled SuperOCR Images

In this paper, we only focus on the OCR task which has a fixed number of
characters to be recognized in each input image. For an input image $I$, its
ground truth label $Str$ is composed of a string of $N$ characters to
represent the characters reading. The proposed SuperOCR method generates $N$
SuperOCR images for the input image $I$. For the $n$th character in $Str$,
where $n\in[1,N]$, the $n$th SuperOCR image is generated. This SuperOCR image
has the resized $I$ on the upper portion, and also embeds the $(n-1)$
characters before the $n$th character on the lower portion. For the special
case of $n=1$, the generated SuperOCR image only has the resized $I$ on the
upper portion, but has nothing on the bottom portion. The generated SuperOCR
image is labeled as one instance for the class of the $n$th character.

One important design option during generation is the font size of the
characters in the SuperOCR image. Research in Super Characters [29] method has
shown the impact of characters size on the classification accuracy. It shows
that larger font size is beneficial to classification accuracy. For
applications like watermeter character recognition and license plate
recognition, the number of characters to predict is usually a fixed number. In
order to make the best use of the space on the SuperOCR image, the bottom
portion of the SupuerOCR image only holds $(N-1)$ characters, instead of $N$.
The saved space for the $N$th character will be distributed into the other
$(N-1)$ characters, in order to increase the font size of each character for
better classification accuracy. For example, a license plate with totally 7
characters will have a design of maximum 6 characters at the lower portion of
the SuperOCR image, as shown in Fig. 1. The 7th SuperOCR image will be labeled
as an instance of the class “8”.

Method | Resolution | Base | Rotate | Tilt | Weather | DB | FN | Challenge
---|---|---|---|---|---|---|---|---
|  | (100k) | (10k) | (10k) | (10k) | (20k) | (20k) | (10k)
SSD300 [14]+HC | 480x480 | 98.3 | 88.4 | 91.5 | 87.3 | 96.6 | 95.9 | 83.8%
YOLO9000 [18]+HC | 480x480 | 98.1 | 84.5 | 88.5 | 87.0 | 96.0 | 88.2 | 80.5%
FasterRCNN [19]+HC | 480x480 | 97.2 | 82.9 | 87.3 | 85.5 | 94.4 | 90.9 | 76.3%
TE2E [12] | 480x480 | 97.8 | 87.9 | 92.1 | 86.8 | 94.8 | 94.5 | 81.2%
RPnet [34] | 480x480 | 98.5 | 90.8 | 92.5 | 87.9 | 96.9 | 94.3 |  85.1%
SuperOCR | 310x310 | 98.7 | 95.6 | 96.0 | 94.7 | 94.6 | 83.6 | 76.9%
Table 1: Recognition results on license plate recognition. SuperOCR obtains
the highest accuracy for subsets of Base, Rotate, Tilt, and Weather from CCPD
datasets, while using a smaller input resolution of 331x331 than the other
methods.

###  2.2 Train with Generated SuperOCR Images

The generated SuperOCR training dataset is the same as a typical dataset for
the purpose of image classification. The successful CNN network architectures
for image classifications could be applied to this dataset through transfer
learning. Depending on where the trained model will be deployed, the design
choices for model architectures are different.

If the trained model is to be deployed on general purpose hardwares, such as
GPU or CPU, there will be a wide range of CNN model architectures to choose
from. One efficient way is to fine-tune pretrained CNN models on ImageNet
dataset [2], such as PolyNet [39], EfficientNet [32], and etc.

If the target hardware to implement the CNN models are CNN ASIC (Application
Specific Integrated Circuits) chips [30, 28], the supported model architures
will be limited. In this case, the available model architectures includes VGG
[26], ResNet [4], MobileNet [6], and so on.

###  2.3 Inference with the Trained CNN Model

During the inference phase, the trained CNN model is used to recognize the
next character iteratively. In each iteration, the updated SuperOCR image is
fed into the same CNN model. The entire characters reading is the sequential
combination of all the recognized characters.

For OCR applications on edge devices, power consumption is strictly
constrained. One advantage of the proposed SuperOCR method is being able to be
deployed on a low-power CNN Domain-Specific Accelerator (CNN-DSA) chip [30,
28] connected to a low-end CPU, such as the Raspberry Pi (RBP). The CNN-DSA
works as a coprocessor, which takes the heavy-lifting computation for the CNN
inference. The RBP works as the host to generate the SuperOCR images, sending
the image to the CNN-DSA, receiving the prediction for the next character, and
returning the the sequencial combination of the characters as the final OCR
result.

##  3 Experimental Results

The proposed SuperOCR method are applied to two application scenarios. One is
the license plate recognition, the other is the watermeter characters
recognition. Both applications have the fixed number of characters to be
recognized, and the locations of the characters are random in each image.

###  3.1 License Plate Recognition

In this experiment, CCPD [34] dataset is used. CCPD is a large and
comprehensive dataset for car license plate recognition. It has more than 250k
unique car images taken and mannually labeled by more than 800 Parking Fee
Collector (PFC) from a roadside parking company. The images are taken by a
handheld POS (Point-of-Sale) machine, so the location of the characters are
random distributed in the image. In each image, the car license plate contains
seven characters, including a Chinese character, a letter, and five
numbers/letters. Images of uncontrolled conditions are also included to test
the robustness of OCR models, such as rotation, distortion, illumination, snow
or fog weather, and so on.

####  3.1.1 Experimental Setup

The 250k unique images of CCPD data are split into train, validation, and test
as done in [34]. The total 200k images of the CCPD-Base subset are split into
Base-Train (100k) and Base-Val (100k). The paper [34] mentions that the CCPD-
Base is divided into two equal parts, but it does not describe the details of
how to divide. The corresponding code in its github
111https://github.com/detectRecog/CCPD also does not release the details. So
we randomly split CCPD-Base into two halves as training and validation
respectively. The other subsets of CCPD are also used for evaluation as done
in the CCPD paper, including CCPD-DB(Dark or Bright illuminations, 20k), CCPD-
FN(Far or Near distance, 20k), CCPD-Rotate(Rotated, 10k), CCPD-Tilt(Tilted,
10k), CCPD-Weather(Image taken in a rainy, snowy or foggy day, 10k), and CCPD-
Challenge(Challenging OCR images, 10k).

There are totally 65 unique characters in the license plates of all the
images. The PolyNet [39] model pretrained on ImageNet is used to finetune on
these 65 classes in this experiment. The pretrained PolyNet model has a
relatively larger input resolution of 331x331 than the majority of the
pretrained models of 224x224. However, it is still smaller than the input
resolution of 480x480 for models used in [34]. The 331x331 input image has
less than 70% in length for each side of the 480x480 image, and less than 50%
of the area of pixels.

The SuperOCR image in this experiment is designed in two steps. First, the
original input image is resized to occupy the upper potion of the SuperOCR
image. In this experiment, the size of 331x305 is designed for the resized
image, in order to best reserve the original image information, and at the
same time leave abundent space for embedding the predicted characters. Second,
only the first six characters on the license plate are embedded on the
SuperOCR image, in order for a larger font size. The size of the generated
SuperOCR dataset is seven times as big as the original dataset in CCPD.

A single 1080Ti (11GB memory, 250W) GPU is used, with batch size of 5 for
training. Base learning rate is set as 1e-4, with a dropping rate of 0.99
every 5000 iterations. The weight decay is 7e-4 and momentum is 0.9. No data
augumentation is used for training the SuperOCR model. On the contary, RPnet
[34] augments the training data by randomly sampling four times on each image
to increase the training set by five times, which is helpful to training. Thus
the future work could include data augumentation and larger memory and
multiple GPUs to further boost the accuracy of SuperOCR model.

####  3.1.2 Recognition Result

Table 1 shows the OCR accuracy of the proposed SuperOCR method on the subsets
of the CCPD dataset, including validation dataset (Base 100k), Rotate, Tilt,
Weather, DB (Dark Bright), FN (Far Near), and Challenge. The recognition
result of existing methods are also listed, including SSD300 [14]+HC, YOLO9000
[18]+HC, FasterRCNN [19]+HC, TE2E [12], and RPnet [34]. For a fair comparison,
the resolution of the model input image is also listed in the second column.
The resolution with the smallest input resolution is highlighted in bold font.
For the other columns, the highest accuracy is highlighted. The result of
other existing methods are borrowed from [34]. Note that [34] defines that the
recoginition is correct if and only if IoU of the license plate detection is
greater than 0.6 and all characters in the license plate are correctly
recognized. On the contrary, since there is no detection module in the
proposed SuperOCR method, the correctness of a recognition does not depend on
the IoU of the detection result. When calculating the recognition accuracy of
the proposed SuperOCR method, a recognition is defined as correct if all
characters in the license plate are correctly recognized.

From the table we can see that the proposed SuperOCR method obtains slightly
better result of 98.7% on Base(100k) evaluation dataset than the previous
state-of-the-art of 98.5%. The Base dataset is relatively easy to recognize
since the images are taken under controled scenarios. However, under some
uncontroled scenarios, the recognition results given by the proposed SuperOCR
method are significantly much better than the existing methods. For example,
for images with rotated license plate, the accuracy of the proposed SuperOCR
method is 95.6%, which is 4.8% more accurate than the best existing method.
For tilted images of license plate, the accuracy is 96.0%, which is 3.5%
better than the best existing method. For images taken in snowy or foggy
weather, 6.8% accuracy improvement is observed by using the SuperOCR method,
i.e. improving the accuracy from the previous state-of-the-art 87.9% to 94.7%.
For the most challenging images, as well as the dark and bright images, the
proposed SuperOCR still obtains better accuracy than FasterRCNN+HC (Holistic-
CNN [27]). The SuperOCR method performs worst on the far and near images,
which is mainly caused by the smaller input resolution of 331x331 and the even
smaller original image on the SuperOCR image. In the SuperOCR image, the
original image only occupies part of the whole image of 331x331, thus the
license plate is shrinked too much compared with the 480x480 input image. As a
result, the license plate characters are even more difficult to recognize,
especially for the images taken in a far distance. Alternating to a model with
larger input size will help to improve the performance of recognition on
images taken in a far distance.

Figure 2: An example of SuperOCR method for watermeter number recognition. The
input image may be tilted and/or blured. There are 20 possible classes for
each digit, including the regular ten classes for digits from 0 through 9, and
also the intermediate ten classes. For example, the state between 0 and 1 is
the 10th class with a reading of “0.5”, and the state between 6 and 7 is the
16th class with a reading of “6.5”. The final recognition result of this
example is “01816.5”.

###  3.2 Watermeter Characters Recognition

In this experiment, the Watetmeter Number Recognition dataset (WNR) in [35] is
used, which is downloadable at its corresponding github
222https://github.com/jiarenyf/Water-Meter-Number-DataSet. The WNR dataset is
released by Deep Leaning and Visual Computing Lab of South China University of
Technology for the research of watermeter number recognition. It consists of
two parts. The first part contains 1000 clean images, and the second part
contains 5000 noisy images with variant illumination, refraction, and
occulation. The sizes of these images are different, with the width in range
[201, 418], and height in range [37, 111]. This results in a relatively large
range of aspective ratio in [2.933, 6.619]. In each image, there are five
characters representing the reading from a mechanical watermeter. Besides the
regular 10 classes for digits of 0 through 9, the “mid-states” which
representing the states in-between two consecutive digits are also counted as
another 10 classes. Thus each character can be one of the 20 classes.

####  3.2.1 Experimental Setup

Following the convention in [35], a label $C$ in the range of [10, 19] denotes
a intermediate state, while the range of [0, 9] denotes a regular state. For
example, $C$=10 denotes the intermediate state between 0 and 1, while $C$=11
denotes the intermediate state between 1 and 2, and so on. If one of the
character is an intermediate state, the final watermeter reading of the entire
five characters will depend on the position of this intermediate state
character. Specifictly, if this intermediate state character is at the end of
the watermeter reading, the final reading will be interpreted as $C-9.5$.
Otherwise, it will be interpreted as $C-10$.

As done in [35], the same 1000 of the 5000 noisy images are used for testing,
and the remaining 4000 of the 5000 noisy images, plus all the 1000 clean
images are used for training. In our experiment we only use the original
images without any data augumentation.

Fig. 2 shows the SuperOCR method for this WNR dataset. The size of SuperOCR
image used for this experiment is 224x224. In each image, the top 224x150
pixels are designed to hold the resized original WNR image, and the bottom of
the SuperOCR image is reserved for the first 4 characters. Similar to the
above license plate recognition experiment, the last character will save the
pixel area for the other characters to have a larger font size. Thus the last
character is not be printed on the SuperOCR image.

####  3.2.2 Implementation on Edge Devices

For IoT (Internet of Things) applications like watermeter characters
recognition, the desired hardware implementations are edge devices with low
power and low cost. We implement the SuperOCR method on a CNN-DSA chip [30]
connected to a Raspberry Pi. The Raspberry Pi, which can also be replaced by
other low-end MCUs, works as a host to generate the SuperOCR image and send it
to the CNN-DSA chip. And the CNN-DSA chip works as the coprocessor to execute
the CNN model. A quantized model Gnet18 [23] is used in this exeriment. Gnet18
is a modified ResNet18 model with all shortcut removed in order to fit into
the CNN accelerator [30, 28]. After the CNN layers, the FC layers are
implemented on the host for the output prediction. The heavy-lifting work of
CNN computations are done on the CNN-DSA chip, and the computational power for
the FC layers required by the host CPU is negligible.

####  3.2.3 Recognition Result

Method | LCR% | AR%
---|---|---
ResNet18 | 81.70 | 95.92
CharNet | 84.10 | 96.64
CRNN [24] | 86.10 | 97.04
ConvS2S[3] | 85.80 | 97.28
FCSRN[35] | 85.60 | 96.98
SRN[35] | 89.60 | 97.82
SuperOCR(CNN-DSA) | 90.50 |  97.98%
Table 2: Accuracy comparison on watermeter dataset. The results of the first
six methods are given by [35], where AR is defined as the Accuracy Rate, and
LCR is defined as the Line Correct Rate to denote the accuracy for the whole
line of characters. The SuperOCR implementation on the CNN-DSA implementation
uses the CNN accelerator chip with 300mW power consumption.

The recognition result on WNR dataset is shown in Table 2. The first six rows
of result are given by [35], and the last row shows the result given by the
SuperOCR method implemented on the low-power CNN-DSA chip. It can be seen that
the proposed SuperOCR overperforms all the other existing methods on both LCR
and AR metrics. LCR (Line Correct Rate) and AR (Accuracy Rate) are defined in
[37]. The LCR calculate the accuracy at the image level, i.e. an image will be
counted as correct if and only if all the characters in this image is correct.
On the contrary, AR calculate the accuracy at the character level.

For realworld applications, accuracy is not the only criteria for the system
performance. Power and cost are also the very important factors when comparing
different solutions. The proposed SuperOCR method is implemented on the low-
power and low-cost edge-device to show its feasibility as a practical method
in a real system. The other existing methods in Table 2 are not able to be
directly deployed into a CNN-DSA chip. If they are deployed on a 1080Ti GPU,
the power consumption for inference is around 90W with a maximum of 2.5GB
memory footprint. As a comparison, the proposed SuperOCR method implemented on
the CNN-DSA chip only consumes 300mW power, which is 300x power saving than
the GPU device. The on-chip memory of the CNN-DSA is only 9MB.

##  4 Conclusion

In this paper, we propose the SuperOCR method to recognize the characters
without any detection module. This is done by converting the OCR task into an
image captioning task, and borrows the concept of two-dimensional embedding
from the Super Characters method applied in Natural Language Processing tasks.
This is the first work of casting the OCR problem into an image captioning
problem. The proposed method could be fine-tuned from a CNN model pretrained
on large image recognition dataset. Experimental results shows the proposed
method outperforms other existing methods. In addition, we also deploy the
method on Raspberry Pi 3 with a low-power CNN accelerator for edge
applications.

## References

  * [1] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang.  Bottom-up and top-down attention for image captioning and visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077–6086, 2018.
  * [2] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [3] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin.  Convolutional sequence to sequence learning.  In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1243–1252. JMLR. org, 2017.
  * [4] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [5] MD Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, and Hamid Laga.  A comprehensive survey of deep learning for image captioning.  ACM Computing Surveys (CSUR), 51(6):1–36, 2019.
  * [6] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [7] Justin Johnson, Andrej Karpathy, and Li Fei-Fei.  Densecap: Fully convolutional localization networks for dense captioning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4565–4574, 2016.
  * [8] Andrej Karpathy and Li Fei-Fei.  Deep visual-semantic alignments for generating image descriptions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128–3137, 2015.
  * [9] Rayson Laroca, Evair Severo, Luiz A Zanlorensi, Luiz S Oliveira, Gabriel Resende Gonçalves, William Robson Schwartz, and David Menotti.  A robust real-time automatic license plate recognition based on the yolo detector.  In 2018 International Joint Conference on Neural Networks (IJCNN), pages 1–10. IEEE, 2018.
  * [10] Rayson Laroca, Luiz A Zanlorensi, Gabriel R Gonçalves, Eduardo Todt, William Robson Schwartz, and David Menotti.  An efficient and layout-independent automatic license plate recognition system based on the yolo detector.  arXiv preprint arXiv:1909.01754, 2019.
  * [11] Yann LeCun.  The mnist database of handwritten digits.  http://yann. lecun. com/exdb/mnist/, 1998.
  * [12] Hui Li, Peng Wang, and Chunhua Shen.  Toward end-to-end car license plate detection and recognition with deep neural networks.  IEEE Transactions on Intelligent Transportation Systems, 20(3):1126–1136, 2018.
  * [13] Cheng-Lin Liu, Fei Yin, Da-Han Wang, and Qiu-Feng Wang.  Casia online and offline chinese handwriting databases.  In 2011 International Conference on Document Analysis and Recognition, pages 37–41. IEEE, 2011.
  * [14] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  Ssd: Single shot multibox detector.  In European conference on computer vision, pages 21–37. Springer, 2016.
  * [15] Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher.  Knowing when to look: Adaptive attention via a visual sentinel for image captioning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 375–383, 2017.
  * [16] Syed Zain Masood, Guang Shu, Afshin Dehghan, and Enrique G Ortiz.  License plate detection and recognition using deeply learned convolutional neural networks.  arXiv preprint arXiv:1703.07330, 2017.
  * [17] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  2011\.
  * [18] Joseph Redmon and Ali Farhadi.  Yolo9000: better, faster, stronger.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7263–7271, 2017.
  * [19] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In Advances in neural information processing systems, pages 91–99, 2015.
  * [20] Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel.  Self-critical sequence training for image captioning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7008–7024, 2017.
  * [21] Syed Tahir Hussain Rizvi, Denis Patti, Tomas Björklund, Gianpiero Cabodi, and Gianluca Francini.  Deep classifiers-based license plate detection, localization and recognition on gpu-powered mobile platform.  Future Internet, 9(4):66, 2017.
  * [22] Zied Selmi, Mohamed Ben Halima, and Adel M Alimi.  Deep learning system for automatic license plate detection and recognition.  In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 1, pages 1132–1138. IEEE, 2017.
  * [23] Hao Sha, Baohua Sun, Nicholas Yi, Wenhan Zhang, and Lin Yang.  On-device chatbot system using superchat method on raspberry pi and CNN domain specific accelerator.  In Albert Bifet, Michele Berlingerio, João Gama, Jesse Read, and Ana Rita Nogueira, editors, Proceedings of the 8th International Workshop on Big Data, IoT Streams and Heterogeneous Source Mining: Algorithms, Systems, Programming Models and Applications co-located with 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2019), Anchorage, Alaska, August 4-8, 2019, volume 2579 of CEUR Workshop Proceedings. CEUR-WS.org, 2019.
  * [24] Baoguang Shi, Xiang Bai, and Cong Yao.  An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition.  IEEE transactions on pattern analysis and machine intelligence, 39(11):2298–2304, 2016.
  * [25] Sergio Montazzolli Silva and Claudio Rosito Jung.  Real-time brazilian license plate detection and recognition using deep convolutional neural networks.  In 2017 30th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), pages 55–62. IEEE, 2017.
  * [26] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [27] Jakub Špaňhel, Jakub Sochor, Roman Juránek, Adam Herout, Lukáš Maršík, and Pavel Zemčík.  Holistic recognition of low quality license plates by cnn using track annotated data.  In 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pages 1–6. IEEE, 2017.
  * [28] Baohua Sun, Daniel Liu, Leo Yu, Jay Li, Helen Liu, Wenhan Zhang, and Terry Torng.  Mram co-designed processing-in-memory cnn accelerator for mobile and iot applications.  arXiv preprint arXiv:1811.12179, 2018.
  * [29] Baohua Sun, Lin Yang, Patrick Dong, Wenhan Zhang, Jason Dong, and Charles Young.  Super characters: A conversion from sentiment classification to image classification.  In Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 309–315, 2018.
  * [30] Baohua Sun, Lin Yang, Patrick Dong, Wenhan Zhang, Jason Dong, and Charles Young.  Ultra power-efficient cnn domain specific accelerator with 9.3 tops/watt for mobile and embedded applications.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1677–1685, 2018.
  * [31] Baohua Sun, Lin Yang, Michael Lin, Charles Young, Patrick Dong, Wenhan Zhang, and Jason Dong.  Supercaptioning: Image captioning using two-dimensional word embedding.  arXiv preprint arXiv:1905.10515, 2019.
  * [32] Mingxing Tan and Quoc V Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  arXiv preprint arXiv:1905.11946, 2019.
  * [33] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan.  Show and tell: Lessons learned from the 2015 mscoco image captioning challenge.  IEEE transactions on pattern analysis and machine intelligence, 39(4):652–663, 2016.
  * [34] Zhenbo Xu, Wei Yang, Ajin Meng, Nanxue Lu, Huan Huang, Changchun Ying, and Liusheng Huang.  Towards end-to-end license plate detection and recognition: A large dataset and baseline.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 255–271, 2018.
  * [35] Fan Yang, Lianwen Jin, Songxuan Lai, Xue Gao, and Zhaohai Li.  Fully convolutional sequence recognition network for water meter number reading.  IEEE Access, 7:11679–11687, 2019.
  * [36] Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, and Tao Mei.  Boosting image captioning with attributes.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4894–4902, 2017.
  * [37] Fei Yin, Qiu-Feng Wang, Xu-Yao Zhang, and Cheng-Lin Liu.  Icdar 2013 chinese handwriting recognition competition.  In 2013 12th International Conference on Document Analysis and Recognition, pages 1464–1470. IEEE, 2013.
  * [38] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo.  Image captioning with semantic attention.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4651–4659, 2016.
  * [39] Xingcheng Zhang, Zhizhong Li, Chen Change Loy, and Dahua Lin.  Polynet: A pursuit of structural diversity in very deep networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 718–726, 2017.


symmetry energy. Within the parabolic approximation, the symmetry energy at
saturation is indeed given by

$J_{0}=\left.\dfrac{E}{A}\right|_{n_{0},\beta=1}-\left.\dfrac{E}{A}\right|_{n_{0},\beta=0}=\left.\dfrac{E}{A}\right|_{n_{0},\beta=1}+B_{0}\>.$
(111)

Then, constraining the value of $J_{0}$ implies a constraint on $E/A$ of PNM
at $n_{0}$. Taking into account Eq. (31), the condition above writes

$\mu^{\ast}_{n,0}+V_{n,0}-\left.\dfrac{P_{n}}{n_{0}}\right|_{n_{0},\beta=1}=m_{\mathrm{nuc}}+J_{0}-B_{0}$
(112)

with the effective chemical potential

$\mu_{n,0}^{\ast}=\sqrt{k_{n,0}^{2}+\left(m_{n,0}^{\ast}\right)^{2}}$ (113)

of the neutron at the saturation density $n_{0}$. The effective mass of the
neutron $m_{n,0}^{\ast}=m_{\mathrm{nuc}}-\Gamma_{\sigma}n_{n,0}^{(s)}$ has to
be determined self-consistently with the scalar density $n_{n,0}^{(s)}$,
defined in Eq. (34), using the Fermi momentum
$k_{n,0}=\left(3\pi^{2}n_{0}\right)^{1/3}$ of the neutron. Since there are no
deuterons in PNM, the vector potential of the neutron is given by

$\displaystyle V_{n,0}$ $\displaystyle=$ $\displaystyle
C_{\omega,0}n_{\omega}+C_{\rho,0}n_{\rho}$
$\displaystyle+\frac{1}{2}\left(C_{\omega,0}^{\prime}n_{\omega}^{2}+C_{\rho,0}^{\prime}n_{\rho}^{2}-C_{\sigma,0}^{\prime}n_{\sigma}^{2}\right)$

and the pressure assumes the simple form

$\displaystyle P_{n}(n_{0})$ $\displaystyle=$
$\displaystyle\frac{1}{4}\left(\mu_{n,0}^{\ast}n_{n,0}^{(v)}-m_{n,0}^{\ast}n_{n,0}^{(s)}\right)$
$\displaystyle+\frac{1}{2}\left[D_{\omega,0}n_{\omega}^{2}+D_{\rho,0}n_{\rho}^{2}-D_{\sigma,0}n_{\sigma}^{2}\right]$

with $n_{\omega}=n_{\rho}=n_{n,0}^{(v)}=n_{0}$ and $n_{\sigma}=n_{n,0}^{(s)}$.
The rescaled couplings $C_{\omega,0}$, $C_{\sigma,0}$,
$C_{\omega,0}^{\prime}$, $C_{\sigma,0}^{\prime}$, $D_{\omega,0}$, and
$D_{\sigma,0}$, c.f., Eq. (39), at saturation are already known and thus
$C_{\rho,0}$ can be deduced from

$\displaystyle C_{\rho,0}$ $\displaystyle=$
$\displaystyle\frac{2}{n_{0}}\left[m_{\mathrm{nuc}}+J_{0}-B_{0}-\frac{3}{4}\mu^{\ast}_{n,0}\right.$
$\displaystyle\left.-\frac{1}{4}m_{n,0}^{\ast}\frac{n_{n,0}^{(s)}}{n_{0}}-\frac{1}{2}C_{\omega,0}n_{\omega}-\frac{1}{2n_{0}}C_{\sigma,0}n_{\sigma}^{2}\right]$

and, finally,

$\Gamma_{\rho,0}=m_{\rho}\sqrt{C_{\rho,0}}$ (117)

for the coupling of the $\rho$ meson at saturation.

##  Appendix D Analytical expressions for the mass shift parameters

The following analytical expressions permit to calculate the parameters $c$
and $d$ appearing in Eq. (58):

$\displaystyle c$ $\displaystyle=$ $\displaystyle\left[1-\tanh(e)\right]^{-1}$
$\displaystyle\left[\Delta m_{d,0}-\dfrac{a}{1+b}-f\tanh(g)\right]$
$\displaystyle\eta$ $\displaystyle=$
$\displaystyle\left[1-\tanh(e)\right]^{-1}$
$\displaystyle\left\\{\dfrac{\partial\Delta m_{d}}{\partial
n_{b}}\right|_{n_{0}}n_{0}-\dfrac{a}{(1+b)^{2}}+\dfrac{ce}{\cosh^{2}(e)}$
$\displaystyle\left.-f\gamma\tanh(g)-\dfrac{fg}{\cosh^{2}(g)}-c[1-\tanh(e)]\right\\}$

where the values for $a$, $b$, $e$, $f$ and $g$ are determined as explained in
Section IV.4, while $\Delta m_{d,0}$ and $\left.\dfrac{\partial\Delta
m_{d}}{\partial n_{b}}\right|_{n_{0}}$ indicate the mass shift and its density
slope at saturation density $n_{0}$.

## References

  * Shen  _et al._ (2011) G. Shen, C. Horowitz, and S. Teige, Phys. Rev. C 83, 035802 (2011).
  * Hempel  _et al._ (2012) M. Hempel, T. Fischer, J. Schaffner-Bielich, and M. Liebendörfer, Astrophys. J. 748, 70 (2012).
  * Lattimer and Prakash (2016) J. M. Lattimer and M. Prakash, Phys. Rep. 621, 127 (2016).
  * Abbott  _et al._ (2018) B. P. Abbott _et al._ , Phys. Rev. Lett. 121, 161101 (2018).
  * Klähn  _et al._ (2006) T. Klähn _et al._ , Phys. Rev. C 74, 035802 (2006).
  * Zheng  _et al._ (2017) H. Zheng, S. Burrello, M. Colonna, and V. Baran, Phys. Lett. B 769, 424 (2017).
  * Zheng  _et al._ (2018) H. Zheng, S. Burrello, M. Colonna, D. Lacroix, and G. Scamps, Phys. Rev. C 98, 024622 (2018).
  * Oertel  _et al._ (2017) M. Oertel, M. Hempel, T. Klähn, and S. Typel, Rev. Mod. Phys. 89, 015007 (2017).
  * Akmal  _et al._ (1998) A. Akmal, V. R. Pandharipande, and D. G. Ravenhall, Phys. Rev. C 58, 1804 (1998), arXiv:nucl-th/9804027 .
  * Carlson  _et al._ (2015) J. Carlson, S. Gandolfi, F. Pederiva, S. C. Pieper, R. Schiavilla, K. E. Schmidt, and R. B. Wiringa, Rev. Mod. Phys. 87, 1067 (2015), arXiv:1412.3081 [nucl-th] .
  * Epelbaum  _et al._ (2009) E. Epelbaum, H.-W. Hammer, and U.-G. Meißner, Rev. Mod. Phys. 81, 1773 (2009).
  * Hammer  _et al._ (2020) H.-W. Hammer, S. König, and U. Van Kolck, Rev. Mod. Phys. 92, 025004 (2020).
  * Furnstahl  _et al._ (2021) R. J. Furnstahl, H. W. Hammer, and A. Schwenk, Few Body Syst. 62, 72 (2021), arXiv:2107.00413 [nucl-th] .
  * Typel (2014) S. Typel, Phys. Rev. C 89, 064321 (2014).
  * Tanaka  _et al._ (2021) J. Tanaka _et al._ , Science 371, 260 (2021).
  * Raduta and Gulminelli (2010) A. R. Raduta and F. Gulminelli, Phys. Rev. C 82, 065801 (2010).
  * Bender  _et al._ (2003) M. Bender, P.-H. Heenen, and P.-G. Reinhard, Rev. Mod. Phys. 75, 121 (2003).
  * Dutra  _et al._ (2012) M. Dutra, O. Loureno, J. S. Martins, A. Delfino, J. R. Stone, and P. Stevenson, Phys. Rev. C 85, 035201 (2012).
  * Sellahewa and Rios (2014) R. Sellahewa and A. Rios, Phys. Rev. C 90, 054327 (2014).
  * Dutra  _et al._ (2014) M. Dutra _et al._ , Phys. Rev. C 90, 055203 (2014).
  * Furnstahl, R. J. (2020) Furnstahl, R. J., Eur. Phys. J. A 56, 85 (2020).
  * Grasso (2019) M. Grasso, Prog. Part. Nucl. Phys. 106, 256 (2019).
  * Marino  _et al._ (2021) F. Marino, C. Barbieri, A. Carbone, G. Colò, A. Lovato, F. Pederiva, X. Roca-Maza, and E. Vigezzi, Phys. Rev. C 104, 024315 (2021).
  * Yang  _et al._ (2016) C. J. Yang, M. Grasso, and D. Lacroix, Phys. Rev. C 94, 031301 (2016).
  * Grasso  _et al._ (2017) M. Grasso, D. Lacroix, and C. J. Yang, Phys. Rev. C 95, 054327 (2017).
  * Burrello  _et al._ (2021) S. Burrello, J. Bonnard, and M. Grasso, Phys. Rev. C 103, 064317 (2021).
  * Burrello and Grasso (2022) S. Burrello and M. Grasso, Eur. Phys. J. A 58, 1 (2022).
  * Yang  _et al._ (2017) C.-J. Yang, M. Grasso, and D. Lacroix, Phys. Rev. C 96, 034318 (2017).
  * Burrello  _et al._ (2020) S. Burrello, M. Grasso, and C.-J. Yang, Phys. Lett. B 811, 135938 (2020).
  * Typel  _et al._ (2010) S. Typel, G. Röpke, T. Klähn, D. Blaschke, and H. H. Wolter, Phys. Rev. C 81, 015803 (2010).
  * Burrello  _et al._ (2015) S. Burrello, F. Gulminelli, F. Aymard, M. Colonna, and A. R. Raduta, Phys. Rev. C 92, 055804 (2015).
  * Hempel and Schaffner-Bielich (2010) M. Hempel and J. Schaffner-Bielich, Nucl. Phys. A 837, 210 (2010).
  * Hempel  _et al._ (2011) M. Hempel, J. Schaffner-Bielich, S. Typel, and G. Röpke, Phys. Rev. C 84, 055804 (2011).
  * Sagun  _et al._ (2014) V. Sagun, A. Ivanytskyi, K. Bugaev, and I. Mishustin, Nucl. Phys. A 924, 24 (2014).
  * Röpke  _et al._ (1982a) G. Röpke, L. Münchow, and H. Schulz, Phys. Lett. B 110, 21 (1982a).
  * Röpke  _et al._ (1982b) G. Röpke, L. Münchow, and H. Schulz, Nucl. Phys. A 379, 536 (1982b).
  * Typel (2020) S. Typel, Eur. Phys. J. Special Top. 229, 3433 (2020).
  * Pais and Typel (2017a) H. Pais and S. Typel, in  _Nuclear particle correlations and cluster physics_, edited by W.-U. Schroeder (World Scientific, Singapore, 2017) pp. 95–133.
  * Fomin  _et al._ (2012) N. Fomin _et al._ , Phys. Rev. Lett. 108, 092502 (2012).
  * Hen  _et al._ (2017) O. Hen, G. A. Miller, E. Piasetzky, and L. B. Weinstein, Rev. Mod. Phys. 89, 045002 (2017).
  * Arrington  _et al._ (2012) J. Arrington, D. Higinbotham, G. Rosner, and M. Sargsian, Prog. Part. Nucl. Phys. 67, 898 (2012).
  * degli Atti (2015) C. C. degli Atti, Phys. Rep. 590, 1 (2015).
  * Egiyan  _et al._ (2006) K. S. Egiyan _et al._ , Phys. Rev. Lett. 96, 082501 (2006).
  * Subedi  _et al._ (2008) R. Subedi _et al._ , Science 320, 1476 (2008).
  * Hen  _et al._ (2014) O. Hen _et al._ , Science 346, 614 (2014).
  * Duer  _et al._ (2018) M. Duer _et al._ (CLAS Collaboration), Nature 560, 617 (2018).
  * Duer  _et al._ (2019) M. Duer _et al._ (CLAS Collaboration), Phys. Rev. Lett. 122, 172502 (2019).
  * Schmookler  _et al._ (2019) B. Schmookler _et al._ , Nature 566, 354 (2019).
  * Rios  _et al._ (2014) A. Rios, A. Polls, and W. Dickhoff, Phys. Rev. C 89, 044303 (2014).
  * Yang  _et al._ (2019) Z. Yang, X. Shang, G. Yong, W. Zuo, and Y. Gao, Phys. Rev. C 100, 054325 (2019).
  * Li and Schulze (2016) Z.-H. Li and H.-J. Schulze, Phys. Rev. C 94, 024322 (2016).
  * Cai and Li (2016) B.-J. Cai and B.-A. Li, Phys. Rev. C 93, 014619 (2016).
  * Li  _et al._ (2018) B.-A. Li, B.-J. Cai, L.-W. Chen, and J. Xu, Prog. Part. Nucl. Phys. 99, 29 (2018).
  * Li  _et al._ (2008) B.-A. Li, L.-W. Chen, and C. M. Ko, Phys. Rep. 464, 113 (2008).
  * Russotto  _et al._ (2016) P. Russotto _et al._ , Phys. Rev. C 94, 034608 (2016).
  * Zheng  _et al._ (2016) H. Zheng, S. Burrello, M. Colonna, and V. Baran, Phys. Rev. C 94, 014313 (2016).
  * Burrello  _et al._ (2019a) S. Burrello, M. Colonna, G. Colò, D. Lacroix, X. Roca-Maza, G. Scamps, and H. Zheng, Phys. Rev. C 99, 054314 (2019a).
  * Burrello  _et al._ (2019b) S. Burrello, M. Colonna, and H. Zheng, Front. Phys. 7, 53 (2019b).
  * Weinstein  _et al._ (2011) L. B. Weinstein, E. Piasetzky, D. W. Higinbotham, J. Gomez, O. Hen, and R. Shneor, Phys. Rev. Lett. 106, 052301 (2011).
  * Van Cuyck  _et al._ (2016) T. Van Cuyck, N. Jachowicz, R. González-Jiménez, M. Martini, V. Pandey, J. Ryckebusch, and N. Van Dessel, Phys. Rev. C 94, 024611 (2016).
  * Guo  _et al._ (2021) W.-M. Guo, B.-A. Li, and G.-C. Yong, Phys. Rev. C 104, 034603 (2021).
  * Lu  _et al._ (2021a) H. Lu, Z. Ren, and D. Bai, Nucl. Phys. A 1011, 122200 (2021a).
  * Typel and Wolter (1999) S. Typel and H. Wolter, Nucl. Phys. A 656, 331 (1999).
  * Buss  _et al._ (2012) O. Buss, T. Gaitanos, K. Gallmeister, H. van Hees, M. Kaskulov, O. Lalakulich, A. Larionov, T. Leitner, J. Weil, and U. Mosel, Phys. Rep. 512, 1 (2012).
  * Shahrbaf  _et al._ (2022) M. Shahrbaf, D. Blaschke, S. Typel, G. R. Farrar, and D. E. Alvarez-Castillo, (2022), arXiv:2202.00652 [nucl-th] .
  * Pais and Typel (2017b) H. Pais and S. Typel, “Comparison of equation of state models with different cluster dissolution mechanisms,” in  _Nucl. Part. Correl. Clust. Phys._ (World Scientific, 2017) Chap. 4, pp. 95–132.
  * Typel and Alvear Terrero (2020) S. Typel and D. Alvear Terrero, Eur. Phys. J. A 56, 160 (2020).
  * Ferreira and Providencia (2012) M. Ferreira and C. Providencia, Phys. Rev. C 85, 055811 (2012), arXiv:1206.0139 [nucl-th] .
  * Pais  _et al._ (2018) H. Pais, F. Gulminelli, C. Providência, and G. Röpke, Phys. Rev. C 97, 045805 (2018).
  * Pais  _et al._ (2019) H. Pais, F. Gulminelli, C. Providência, and G. Röpke, Phys. Rev. C 99, 055806 (2019).
  * Qin  _et al._ (2012) L. Qin _et al._ , Phys. Rev. Lett. 108, 172701 (2012).
  * Pais  _et al._ (2020a) H. Pais _et al._ , Phys. Rev. Lett. 125, 012701 (2020a).
  * Pais  _et al._ (2020b) H. Pais _et al._ , J. Phys. G: Nucl. Part. Phys. 47, 105204 (2020b).
  * Bougault  _et al._ (2020) R. Bougault _et al._ , J. Phys. G: Nucl. Part. Phys. 47, 025103 (2020).
  * Custódio  _et al._ (2020) T. Custódio, A. Falcão, H. Pais, C. Providência, F. Gulminelli, and G. Röpke, Eur. Phys. J. A 56, 1 (2020).
  * Röpke (2009) G. Röpke, Phys. Rev. C 79, 014002 (2009).
  * Röpke (2011) G. Röpke, Nucl. Phys. A 867, 66 (2011).
  * Röpke (2015) G. Röpke, Phys. Rev. C 92, 054001 (2015).
  * Typel (2018) S. Typel, J. Phys. G 45, 114001 (2018).
  * Wang  _et al._ (2012) M. Wang, G. Audi, A. Wapstra, F. Kondev, M. MacCormick, X. Xu, and B. Pfeiffer, Chin. Phys. C 36, 1603 (2012).
  * Typel  _et al._ (2014) S. Typel, H. H. Wolter, G. Röpke, and D. Blaschke, Eur. Phys. J. A 50, 17 (2014), arXiv:1309.6934 [nucl-th] .
  * Blaizot (1980) J. P. Blaizot, Phys. Rept. 64, 171 (1980).
  * Youngblood  _et al._ (1999) D. H. Youngblood, H. L. Clark, and Y. W. Lui, Phys. Rev. Lett. 82, 691 (1999).
  * Shlomo  _et al._ (2006) S. Shlomo, V. Kolomietz, and G. Colò, Eur. Phys. J. A 30, 23 (2006).
  * Stone  _et al._ (2014) J. R. Stone, N. J. Stone, and S. A. Moszkowski, Phys. Rev. C 89, 044316 (2014), arXiv:1404.0744 [nucl-th] .
  * Burrello  _et al._ (2014) S. Burrello, M. Colonna, and F. Matera, Phys. Rev. C 89, 057604 (2014).
  * Burrello  _et al._ (2016) S. Burrello, M. Colonna, and F. Matera, Phys. Rev. C 94, 012801 (2016).
  * Li  _et al._ (2021) B.-A. Li, B.-J. Cai, W.-J. Xie, and N.-B. Zhang, Universe 7, 182 (2021), arXiv:2105.04629 [nucl-th] .
  * Lu  _et al._ (2021b) H. Lu, Z. Ren, and D. Bai, Nucl. Phys. A 1011, 122200 (2021b).