# Human-level COVID-19 Diagnosis from Low-dose CT Scans Using a Two-stage
Time-distributed Capsule Network

Parnian Afshar  Concordia Institute for Information Systems Engineering
(CIISE), Concordia University, Montreal, Canada  Department of Electrical and
Computer Engineering, University of Toronto, Toronto, Canada  Moezedin Javad
Rafiee  Department of Medicine and Diagnostic Radiology, McGill University
Health Center-Research Institute, Montreal, QC, Canada  MD  Department of
Medicine and Diagnostic Radiology, McGill University Health Center-Research
Institute, Montreal, QC, Canada  Farnoosh Naderkhani  Concordia Institute for
Information Systems Engineering (CIISE), Concordia University, Montreal,
Canada  Shahin Heidarian  Department of Electrical and Computer Engineering,
Concordia University, Montreal, QC, Canada  Nastaran Enshaei  Concordia
Institute for Information Systems Engineering (CIISE), Concordia University,
Montreal, Canada  Anastasia Oikonomou  Department of Medical Imaging,
Sunnybrook Health Sciences Centre, University of Toronto, Toronto, Canada  MD
Department of Medical Imaging, Sunnybrook Health Sciences Centre, University
of Toronto, Toronto, Canada  Faranak Babaki Fard  Faculty of Medicine,
University of Montreal, Montreal, QC, Canada  MD  Faculty of Medicine,
University of Montreal, Montreal, QC, Canada  Reut Anconina  Department of
Medical Imaging, Sunnybrook Health Sciences Centre, University of Toronto,
Toronto, Canada  MD  Department of Medical Imaging, Sunnybrook Health Sciences
Centre, University of Toronto, Toronto, Canada  Keyvan Farahani  Center for
Biomedical Informatics and Information Technology, National Cancer Institute
(NCI), Rockville, MD, USA.  Konstantinos N. Plataniotis  Department of
Electrical and Computer Engineering, University of Toronto, Toronto, Canada
Arash Mohammadi  Concordia Institute for Information Systems Engineering
(CIISE), Concordia University, Montreal, Canada  Corresponding author: Arash
Mohammadi<EMAIL_ADDRESS>
###### Abstract

Reverse transcription-polymerase chain reaction (RT-PCR) is currently the gold
standard in COVID-19 diagnosis. It can, however, take days to provide the
diagnosis, and false negative rate is relatively high. Imaging, in particular
chest computed tomography (CT), can assist with diagnosis and assessment of
this disease. Nevertheless, it is shown that standard dose CT scan gives
significant radiation burden to patients, especially those in need of multiple
scans. In this study, we consider low-dose and ultra-low-dose (LDCT and ULDCT)
scan protocols that reduce the radiation exposure close to that of a single
X-Ray, while maintaining an acceptable resolution for diagnosis purposes.
Since thoracic radiology expertise may not be widely available during the
pandemic, we develop an Artificial Intelligence (AI)-based framework using a
collected dataset of LDCT/ULDCT scans, to study the hypothesis that the AI
model can provide human-level performance. The AI model uses a two stage
capsule network architecture and can rapidly classify COVID-19, community
acquired pneumonia (CAP), and normal cases, using LDCT/ULDCT scans. Based on a
cross validation, the AI model achieves COVID-19 sensitivity of $89.5\%\pm
0.11$, CAP sensitivity of $95\%\pm 0.11$, normal cases sensitivity
(specificity) of $85.7\%\pm 0.16$, and accuracy of $90\%\pm 0.06$. By
incorporating clinical data (demographic and symptoms), the performance
further improves to COVID-19 sensitivity of $94.3\%\pm 0.05$, CAP sensitivity
of $96.7\%\pm 0.07$, normal cases sensitivity (specificity) of $91\%\pm 0.09$
, and accuracy of $94.1\%\pm 0.03$. The proposed AI model achieves human-level
diagnosis based on the LDCT/ULDCT scans with reduced radiation exposure. We
believe that the proposed AI model has the potential to assist the
radiologists to accurately and promptly diagnose COVID-19 infection and help
control the transmission chain during the pandemic.

## Introduction

Since the beginning of the coronavirus disease (COVID-19) outbreak in December
2019 in Wuhan, China, a global healthcare crisis has emerged [1]. Real-time
reverse transcription-polymerase chain reaction (RT-PCR) is currently
considered as the gold standard method in COVID-19 diagnosis. RT-PCR is,
however, prone to a number of limitations, i.e., besides being time consuming,
it is associated with high false-negative rate in different clinical samples
[2]. Due to high sensitivity and rapid access, chest computed tomography (CT)
scan has been the main imaging modality for diagnosis, prognostic assessment,
and detection of complications of COVID-19 [3, 4]. The most common
manifestations of COVID-19 pneumonia in chest CT scan are multifocal ground-
glass opacities (GGO) with or without consolidative areas, predominantly
having peripheral, lower-lobes, and posterior anatomic distribution [3, 4]. CT
scan can contribute to assessing the complications, extent of COVID-19
involvement, and risk of intensive care unit (ICU) admission.

The main concern of widespread use of CT scan as a screening tool for
suspected patients during the outbreak is the radiation exposure. In some
scenarios, severely symptomatic patients will need multiple chest CT scans
during the course of their disease. The cumulative effect of these multiple
exposures can significantly increase the radiation dose. Studies [5] have
shown that the projected radiation to body organs during chest CT scan is
highest in thyroid, lung, breast, and esophagus. Due to their longer life
expectancy, higher dose-effective breast tissue and cell proliferation [6, 7],
children and young women are more vulnerable to radiation exposure damage with
increased risk of radiation-following malignancy. As low as reasonably
achievable (ALARA) [8] rule states that whenever radiation is expected, the
exposure should be kept at the minimum achievable level such that the
resulting scan still provides reasonable resolution.

Diagnostic accuracy of Low and Ultra-low-dose CT scan (LDCT and ULDCT) in
detection and follow-up of pulmonary nodule and other lung pathologies has
been previously established [9]. The radiation dose associated with standard
chest CT is estimated at 7 mSv, which is reduced to 1-1.5 mSv with LDCT
methods and as low as 0.3 mSv with ULDCT ones. The advantage of the low dose
protocols is the reduction of radiation dose by more than 80%. Recent studies
[10] have shown that DNA double-strand breaks and chromosome damage increased
in patients undergoing a standard-dose CT scan while no effect on human DNA
was demonstrated in patients undergoing low-dose CT scan. LDCT and ULDCT have
shown significant accuracy in the detection of GGOs and consolidation in
patients with pneumonia [11]. Since GGO and consolidation are the most common
CT findings of COVID-19, recently, replacing standard CT scan with LDCT and
ULDCT has been recommended [12] as a solution to decrease radiation exposure
in COVID-19 patients. In a retrospective study [13], LDCT with iterative
reconstruction (IR) demonstrated sensitivity, specificity, positive predictive
value, negative predictive value, and accuracy of about 90% in the diagnosis
of COVID-19. In conjunction with other clinical findings, LDCT and ULDCT can
potentially replace standard-dose for the evaluation of patients, in
particular pregnant and young women, and pediatric populations, to decrease
radiation exposure [14].

While rapid detection of positive COVID-19 cases is of utmost importance,
physicians and healthcare personnel are overwhelmed with increasing number of
patients in need of immediate care and treatment. In other words, expert
thoracic radiologists may not be available at all times to diagnose positive
cases in a timely fashion, leading to not only delays in treatment, but also
further transmission of the virus by patients who are not promptly isolated.
Recent studies [15, 16] have demonstrated the capabilities of artificial
intelligence (AI) models in achieving human-level performance in detection of
lung nodules from LDCT. Motivated by this, in this study we aim at developing
an AI model based on deep learning architectures to distinguish COVID-19 from
community acquired pneumonia (CAP) and normal cases, using LDCT and ULDCT. To
the best of our knowledge, Reference [17] is the only study considering LDCT
in AI-based COVID-19 analysis, by simulating standard dose scans from low dose
ones. The aforementioned study, however, uses synthesized data, i.e., it does
not use real LDCT/ULDCT data from COVID-19 individuals, and does not deal with
the disease diagnosis, which is the main focus of our research. We hypothesize
that AI can achieve a human-level performance in diagnosing COVID-19 based on
LDCT and ULDCT scans.

We developed a two-stage deep learning model, shown in Fig. 1, built upon the
capsule network architecture [18, 19], which takes segmented lung regions as
inputs. The first stage of the proposed model is a capsule network responsible
of detecting slices with evidence of infection (caused by COVID-19 or CAP).
Following a prior research [20], 10 most probable slices with infection, along
with their infectious probability, are provided as input to the second stage,
which analyzes all the candidates in parallel, making the decision by
aggregating the outcome of each single candidate. The final output is the
probability of the underlying case representing COVID-19, CAP, or normal
classes. While the first stage model is trained on slice-level labels provided
by an experienced radiologist, the second stage is trained on patient-level
labels. Expanding the proposed deep learning model, we incorporated clinical
data, i.e., the output of the deep learning model is merged with clinical data
and fed to a multi-layer perceptron (MLP), shown in Fig. 2, to make the final
diagnosis.

Figure 1: The proposed 2 stage deep learning model for COVID-19 diagnosis
using LDCT/ULDCT. At the first stage, CT slices go through a capsule network,
one by one, to detect those with evidence of infection. At the second stage,
10 most probable slices with infection detected in the previous stage go
through a time-distributed capsule network, output of which determines the
probability of COVID-19, CAP, and normal, after applying a global max pooling.
Figure 2: The MLP model combining the output of the two-stage deep learning
model with the clinical data. Clinical data includes demographic
characteristics and 5 common COVID-19 and CAP symptoms. Four sets of fully
connected layers determine the final output.

## Results

Table 1: Characteristics of the collected dataset and CAP cases adopted from
Reference [21]. In total, the study includes 220 patients

. SD stands for standard deviation, M stands for male and F stands for female.
COVID-19 CAP Normal P-value: COVID-19 vs. rest P-value: CAP vs. rest P-value:
Normal vs. rest Sex M: 58.6% F: 41.4% M: 58% F: 42% M: 39.3% F: 60.7% 0.7386
0.1848 0.0314 Age in Years (Mean $\pm$ SD) $49.53\pm 15.5$ $57.78\pm 21.94$
$40.18\pm 15.37$ 0.7283 0.0003 0.0002 Weight in Kg (Mean $\pm$ SD) $80.75\pm
14.84$ $67.38\pm 12.96$ $75.91\pm 14.52$ 0.0001 0.0000 0.6881 Dyspnea $26.9\%$
$18\%$ $45\%$ 0.8932 0.0091 0.0425 Cough $31.7\%$ $53\%$ $33.93\%$ 0.1480
0.0160 0.4916 Fever $14.4\%$ $36\%$ $9\%$ 0.5130 0.0242 0.0589 Chest Pain
$7\%$ $0\%$ $10.7\%$ 0.7571 0.9999 0.6439 Fatigue $10.5\%$ $0\%$ $1.7\%$
0.0107 0.9999 0.2681

We collected a dataset of LDCT and ULDCT scans of 104 COVID-19 positive cases,
and 56 normal cases, collected in October 2020, December 2020, and January
2021, Babak Imaging Center, Tehran, Iran. Diagnosis of 36.5% of the COVID-19
cases (38 cases) is confirmed with the RT-PCR test. The rest are specified by
taking the consensus between 3 experienced thoracic radiologists (M.J.R.,
F.B.F., and A.O.), who labeled the dataset by taking the imaging findings,
clinical characteristics (symptoms and history), and epidemiology into
account. The three radiologists reached an agreement of 95.6%. They also
scored the severity of the COVID-19 cases between 1 and 4, based on the
percentage of the lung involvement. Four positive COVID-19 cases do not reveal
any related imaging findings. As we did not have access to LDCT scans of CAP
patients, we combined this dataset with 60 standard-dose volumetric CT scans
[21]. Therefore, we ended up with a total of 220 patients. The dataset
characteristics are shown in Table 1. P-values are obtained using logistic
regression, by considering three binary scenarios of COVID-19 versus CAP and
normal, CAP versus COVID-19 and normal, and normal versus COVID-19 and CAP.
For ease of access, here we translate the first row of the table: "58.6% of
the COVID-19 patients are men, 58% of the CAP cases are men, 39.3% of the
normal cases are men, Sex has a p-value of 0.7386 when distinguishing COVID-19
from the other two classes, it has a p-value of 0.1848 when distinguishing CAP
from the other two classes, and a p-value of 0.0314 when distinguishing normal
from other two classes". Finally, a fourth experienced thoracic radiologist
(R.A.), blind to the ground-truth, labeled the collected dataset to compare
the performance of the AI model with a human expert. The radiologist was first
provided with only the CT scans, and then the clinical data.

To decrease bias towards a specific test set, we adopted a 10-fold cross
validation approach to assess the performance of the radiologist and the AI
model, based on two scenarios of using CT scans only, and incorporating the
clinical data. The dataset is randomly split into 10 equal size test sets,
leading to 10 sets each including 22 cases. We made sure that each set
contained 10% of the COVID-19, CAP, and normal cases, leading to 10 or 11
COVID-19, 6 CAP, and 5 or 6 normal cases in each test set. The AI model is
trained 10 times, setting one of the test sets aside and using the rest for
training. Averaging over the 10 folds, the slice-level classifier in the first
stage achieved accuracy of $89.88\%$, sensitivity of $88.24\%$, and
specificity of $92.01\%$, in detecting the slices with infection.

Table 2: Performance of the AI model and the radiologist blind to the labels, using only CT scans. | COVID-19 Sensitivity | CAP Sensitivity | Normal Sensitivity | Accuracy |  |
---|---|---|---|---|---|---
Fold | AI | Radiologist | AI | Radiologist | AI | Radiologist | AI | Radiologist | Accuracy P-value | COVID-19 AUC
1 | $\frac{10}{11}$ | $\frac{10}{11}$ | $\frac{6}{6}$ | $\frac{5}{6}$ | $\frac{4}{5}$ | $\frac{5}{5}$ | $\frac{20}{22}$ | $\frac{20}{22}$ | $1$ | $0.95$
2 | $\frac{10}{10}$ | $\frac{10}{10}$ | $\frac{6}{6}$ | $\frac{6}{6}$ | $\frac{6}{6}$ | $\frac{6}{6}$ | $\frac{22}{22}$ | $\frac{22}{22}$ | $1$ | $1$
3 | $\frac{10}{11}$ | $\frac{9}{11}$ | $\frac{6}{6}$ | $\frac{4}{6}$ | $\frac{3}{5}$ | $\frac{5}{5}$ | $\frac{19}{22}$ | $\frac{18}{22}$ | $1$ | $0.91$
4 | $\frac{10}{10}$ | $\frac{10}{10}$ | $\frac{4}{6}$ | $\frac{6}{6}$ | $\frac{5}{6}$ | $\frac{6}{6}$ | $\frac{19}{22}$ | $\frac{22}{22}$ | $0.25$ | $0.99$
5 | $\frac{8}{11}$ | $\frac{10}{11}$ | $\frac{5}{6}$ | $\frac{5}{6}$ | $\frac{5}{5}$ | $\frac{5}{5}$ | $\frac{18}{22}$ | $\frac{20}{22}$ | $0.5$ | $0.9$
6 | $\frac{10}{11}$ | $\frac{10}{11}$ | $\frac{6}{6}$ | $\frac{5}{6}$ | $\frac{5}{5}$ | $\frac{5}{5}$ | $\frac{21}{22}$ | $\frac{20}{22}$ | $1$ | $0.98$
7 | $\frac{8}{10}$ | $\frac{10}{10}$ | $\frac{6}{6}$ | $\frac{5}{6}$ | $\frac{5}{6}$ | $\frac{6}{6}$ | $\frac{19}{22}$ | $\frac{21}{22}$ | $0.625$ | $0.96$
8 | $\frac{7}{10}$ | $\frac{8}{10}$ | $\frac{6}{6}$ | $\frac{5}{6}$ | $\frac{5}{6}$ | $\frac{6}{6}$ | $\frac{18}{22}$ | $\frac{19}{22}$ | $1$ | $0.95$
9 | $\frac{10}{10}$ | $\frac{6}{10}$ | $\frac{6}{6}$ | $\frac{6}{6}$ | $\frac{5}{6}$ | $\frac{6}{6}$ | $\frac{21}{22}$ | $\frac{18}{22}$ | $0.375$ | $1$
10 | $\frac{10}{10}$ | $\frac{10}{10}$ | $\frac{6}{6}$ | $\frac{6}{6}$ | $\frac{5}{6}$ | $\frac{6}{6}$ | $\frac{21}{22}$ | $\frac{22}{22}$ | $1$ | $1$
Total | $89.5\%\pm 0.11$ | $89.4\%\pm 0.12$ | $95\%\pm 0.11$ | $88.33\%\pm 0.11$ | $85.7\%\pm 0.16$ | $100\%$ | $90\%\pm 0.06$ | $91.8\%\pm 0.07$ | - | $0.96\pm 0.03$

Figure 3: ROC curve for COVID-19 diagnosis (vs CAP and normal) using the
proposed deep learning model and CT scans only.

Using only CT scans, we evaluated the developed deep learning model and
compared it with the fourth thoracic radiologist, as shown in Table 2.
Averaging over all the 10 folds, AI model achieves COVID-19 sensitivity of
$89.5\%\pm 0.11$, CAP sensitivity of $95\%\pm 0.11$, normal sensitivity
(specificity) of $85.7\%\pm 0.16$, and accuracy of $90\%\pm 0.06$. The
radiologist, on the other hand, achieves COVID-19 sensitivity of $89.4\%\pm
0.12$, CAP sensitivity of $88.33\%\pm 0.11$, normal sensitivity (specificity)
of $100\%$, and accuracy of $91.8\%\pm 0.07$. We tested the hypothesis of the
AI model and radiologist having the same performance, in term of accuracy,
using a McNemar [22] test with the significance level of 0.05, leading to
P-values over the significance level for all the 10 folds. The lower
specificity of the AI model conforms the non-specific COVID-19 findings [23].
COVID-19 sensitivity versus one minus specificity is plotted in the receiver
operating characteristics (ROC) curve, shown in Fig. 3. Area under the curve
(AUC) is $0.96\pm 0.03$.

Figure 4: Grad-CAM visualization of one CAP slice. This figure shows that the
proposed AI model is paying attention to relevant locations of the image.
Figure 5: Grad-CAM visualization of two COVID-19 slices. This figure shows
that the proposed AI model is paying attention to relevant locations of the
image.

Based on the CT scans only, we analyzed the misclassified COVID-19 cases
through all folds (11 cases in total), and studied their relation with the
disease severity, coming to the conclusion that 4 out of 11 cases, did not
have any related imaging findings, 5 were scored 1 by the three radiologists,
one was scored 2, and only one case was scored at 3, which means the developed
model is less likely to misclassify severe cases. Neither the developed model
nor the experienced radiologist was able to detect the 4 COVID-19 cases
without imaging findings, using CT scans only. Furthermore, since the CAP
patients come from a different cohort and scanned with a standard dose, we
visualized the model’s output for CAP cases, one of which is shown in Fig. 4,
using Grad-CAM localization technique. This figure shows that the model is
paying more attention to disease-related regions of the image, rather than
dose-related ones. We performed the same localization technique on two slices
with infection of the same COVID-19 patient, shown in Fig. 5.

Table 3: Performance of the AI model and the radiologist blind to the labels, using both CT scans and clinical data. | COVID-19 Sensitivity | CAP Sensitivity | Normal Sensitivity | Accuracy |  |
---|---|---|---|---|---|---
Fold | AI | Radiologist | AI | Radiologist | AI | Radiologist | AI | Radiologist | Accuracy P-value | COVID-19 AUC
1 | $\frac{11}{11}$ | $\frac{10}{11}$ | $\frac{5}{6}$ | $\frac{6}{6}$ | $\frac{5}{5}$ | $\frac{5}{5}$ | $\frac{21}{22}$ | $\frac{21}{22}$ | $1$ | $0.99$
2 | $\frac{10}{10}$ | $\frac{10}{10}$ | $\frac{6}{6}$ | $\frac{6}{6}$ | $\frac{6}{6}$ | $\frac{6}{6}$ | $\frac{22}{22}$ | $\frac{22}{22}$ | $1$ | $1$
3 | $\frac{10}{11}$ | $\frac{10}{11}$ | $\frac{6}{6}$ | $\frac{6}{6}$ | $\frac{4}{5}$ | $\frac{5}{5}$ | $\frac{20}{22}$ | $\frac{21}{22}$ | $1$ | $0.91$
4 | $\frac{9}{10}$ | $\frac{10}{10}$ | $\frac{5}{6}$ | $\frac{6}{6}$ | $\frac{6}{6}$ | $\frac{6}{6}$ | $\frac{20}{22}$ | $\frac{22}{22}$ | $0.5$ | $0.98$
5 | $\frac{10}{11}$ | $\frac{10}{11}$ | $\frac{6}{6}$ | $\frac{5}{6}$ | $\frac{5}{5}$ | $\frac{5}{5}$ | $\frac{21}{22}$ | $\frac{20}{22}$ | $1$ | $0.97$
6 | $\frac{10}{11}$ | $\frac{10}{11}$ | $\frac{6}{6}$ | $\frac{5}{6}$ | $\frac{4}{5}$ | $\frac{5}{5}$ | $\frac{20}{22}$ | $\frac{20}{22}$ | $1$ | $0.99$
7 | $\frac{9}{10}$ | $\frac{10}{10}$ | $\frac{6}{6}$ | $\frac{5}{6}$ | $\frac{5}{6}$ | $\frac{6}{6}$ | $\frac{20}{22}$ | $\frac{21}{22}$ | $1$ | $1$
8 | $\frac{9}{10}$ | $\frac{10}{10}$ | $\frac{6}{6}$ | $\frac{5}{6}$ | $\frac{6}{6}$ | $\frac{6}{6}$ | $\frac{21}{22}$ | $\frac{21}{22}$ | $1$ | $0.98$
9 | $\frac{10}{10}$ | $\frac{8}{10}$ | $\frac{6}{6}$ | $\frac{6}{6}$ | $\frac{5}{6}$ | $\frac{6}{6}$ | $\frac{21}{22}$ | $\frac{20}{22}$ | $1$ | $0.95$
10 | $\frac{10}{10}$ | $\frac{10}{10}$ | $\frac{6}{6}$ | $\frac{6}{6}$ | $\frac{5}{6}$ | $\frac{6}{6}$ | $\frac{21}{22}$ | $\frac{22}{22}$ | $1$ | $0.99$
Total | $94.3\%\pm 0.05$ | $94.4\%\pm 0.05$ | $96.7\%\pm 0.07$ | $93.3\%\pm 0.08$ | $91\%\pm 0.09$ | $100\%$ | $94.1\%\pm 0.03$ | $95.4\%\pm 0.03$ | - | $0.96\pm 0.03$

Figure 6: ROC curve for COVID-19 diagnosis (vs CAP and normal) using the
proposed deep learning model and both CT scans and clinical data.

Using both CT scans and clinical data, we evaluated the developed deep
learning model and compared it with the radiologist, as shown in Table 3.
Averaging over all the 10 folds, AI model achieves COVID-19 sensitivity of
$94.3\%\pm 0.05$, CAP sensitivity of $96.7\%\pm 0.07$, normal sensitivity
(specificity) of $91\%\pm 0.09$ , and accuracy of $94.1\%\pm 0.03$. The
radiologist, on the other hand, achieves COVID-19 sensitivity of $94.4\%\pm
0.05$, CAP sensitivity of $93.3\%\pm 0.08$, normal sensitivity (specificity)
of $100\%$, and accuracy of $95.4\%\pm 0.03$. We tested the hypothesis of the
AI model and radiologist having the same performance, using LDCT and clinical
data, in terms of accuracy, leading to P-values over the significance level
for all the 10 folds. COVID-19 sensitivity versus one minus specificity is
plotted in the receiver operating characteristics (ROC) curve, shown in Fig.
6. Area under the curve (AUC) is $0.96\pm 0.03$.

Based on using both CT scans and clinical data, we analyzed the misclassified
COVID-19 cases through all folds (6 cases in total), and studied their
relation with the disease severity, coming to the conclusion that 3 out of 6
cases, did not have any related imaging findings, one was scored 1 by the
three radiologists, and two cases were scored at 3. Incorporating the clinical
data, the AI model can detect one of the four positive COVID-19 cases, without
having related imaging findings, whereas the radiologist did not detect any of
them.

Finally, we tested the developed AI model, incorporating LDCTs and clinical
data, on an extra set of 100 positive COVID-19 patients, whose diagnosis are
confirmed with RT-PCR test and are collected in a different time interval
(narrow validation). These patients were not included in any of the 10 folds
and are completely unseen to the model and radiologist. While 68 out of 100
cases have imaging findings, 32 do not reveal any related manifestations. Male
cases constitute 53% of the total cases, and age average is $46.16$ with a
standard deviation of $14.07$. The AI model correctly identifies all the 68
positive cases having imaging findings, whereas it detects only 3 of those not
having related findings. Radiologist, on the other hand, correctly classifies
64 out of 68 patients having imaging findings as COVID-19 and classifies 4 as
CAP. None of the cases without imaging findings are identified by the
radiologist. The p-value between the AI model and radiologist’s sensitivity is
0.01.

## Discussion

Figure 7: The dataset consists of LDCTs and ULDCTs accompanied with clinical
data and ground truth. Two scenarios are considered. First, only images are
used for training and comparison with the radiologist. Second, both images and
clinical data are utilized.

Although LDCT and ULDCT can reveal COVID-19 related findings and reduce the
potential radiation-related harms, an accurate diagnosis requires full
investigation by radiologists, which may not be possible during the outbreak.
Based on our experiments, the proposed capsule network-based AI model has the
potential to rapidly distinguish COVID-19 cases from CAP and normal ones with
a human-level performance using LDCT and ULDCT, having a radiation dose of a
single X-ray image. In other words, with minimal radiation, the developed AI
system can assist the radiologists and contribute to controlling the chain of
COVID-19 transmission.

To validate the proposed AI model, we considered two scenarios, as shown in
Fig. 7. In the first scenario, the AI model is fed with only the images and
compared with the radiologist blind to both ground truth and clinical data.
Although this strategy does not follow routine clinical practice, the goal was
to investigate the diagnostic potentials of the images without incorporating
clinical data. In the second scenario, both images and clinical data are fed
to the AI model which is accordingly compared to the radiologist blind only to
the ground truth. We showed that by incorporating the clinical data, COVID-19
sensitivity increases by 4.8%, CAP sensitivity increases by 1.7%, and normal
sensitivity and accuracy increase by 5.3% and 4.1%, respectively.

It is worth noting that although the incorporated CAP cases are extracted from
a different cohort, the motivation is to investigate the capability of the AI
model to distinguish COVID-19 from CAP, as these two have overlapping chest CT
findings [24]. The fact that the CAP cases are screened using a standard-dose
CT rather than LDCT and ULDCT can be considered as a limitation of our study,
since low-dose screening is associated with less detection capabilities [25].
Nevertheless it is a common practice to construct datasets of CT scans
acquired using different radiation doses [26], in order to develop models
generalized on larger datasets. Furthermore, although our COVID-19 and normal
cases are scanned with either LDCT or ULDCT, the image quality of the latter
can be counterbalanced using reconstruction techniques [27].

Our study has some other limitations. First, the dataset is collected from a
single centre, and experiments are required to verify its performance on data
from external institutes, as it is critical to investigate if the model
generalizes to diverse population [28, 29]. Vulnerability to data shifts, and
bias against underrepresented population [28] are also crucial to address
before the AI model can be put into practice. It is worth mentioning that as
the extra set of 100 positive COVID-19 patients are collected in a disjoint
time interval from the original set, it can act as a narrow validation [29].
It is, however, collected from the same institute and thus does not account
for broad validation. It is also of high interest to explore domain validation
for COVID-19 diagnosis, where test set comes from different variants. Second,
the sample size is relatively small. Verifying the model’s performance on
larger multi-centre datasets is the goal of our upcoming studies. The capsule
network used in our study, is capable of handling small datasets compared to
conventional models and due to fewer trainable parameters it is less prone to
over-fitting, however, larger datasets can still improve the performance of
the model. We also aim at expanding the proposed AI model to predict the
disease severity besides the diagnosis. Moreover, although as shown in Figs. 4
and 5 visualization of the AI model’s output shows it is paying attention to
relevant regions, more research is required to increase its explainability.
Low performance on COVID-19 cases without imaging finding is another
limitation of the developed model.

In conclusion, we believe the developed AI model achieves human-level
performance by incorporating LDCT/ULDCT and clinical data, having the
advantage of reducing the risks related to radiation exposure. This model can
act as a decision support system for radiologists and help with controlling
the transmission chain. As our developed AI model is not intended to be a
primary diagnostic tool, we aim at testing the model alongside a thoracic
radiologist to assess its performance as a decision support tool rather than a
stand-alone system.

## Methods

This study is conducted following the policy certification number 30013394 of
Ethical acceptability for secondary use of medical data approved by Concordia
University, Montreal, Canada. Informed consent is obtained from all the
patients.

### LDCT/ULDCT Dataset

The LDCT/ULDCT dataset consists of volumetric Low-dose chest CT scans of 104
COVID-19 positive cases, and 56 healthy cases, collected in October 2020,
December 2020, and January 2021, Babak Imaging Center, Tehran, Iran. 36.5% of
the COVID-19 cases (38 cases) are confirmed with positive RT-PCR. Diagnosis
for the rest of the cases is obtained by consensus between three experienced
radiologists (M.J.R., F.B.F., and A.O.) with 95.6% agreement. The three
radiologists have considered the following three main criteria when labeling
the dataset:

  * •

Imaging findings including GGOs, consolidation pattern, crazy paving,
bilateral and multifocal lung involvement, peripheral distribution, and lower
lobe predominance of findings;

  * •

Clinical findings including symptoms and history, and;

  * •

Epidemiology

The CT slices of the confirmed COVID-19 cases are then labeled by the first
radiologists as having evidence of infection or not. Furthermore, all the
three radiologists have scored the severity of the COVID-19 cases, by
assigning a number between 1 and 4, where 1 is a mild and 4 is a severe case.
The final severity score is the average over the scores from three
radiologists, rounded to the nearest integer. Severity is determined based on
the percentage of the lung involvement, as follows:

  * •

1: 1%-24%

  * •

2: 25%-49%

  * •

3: 50%-74%

  * •

4: $\geq$75%

Male and Female cases form 52% and 48% of the LDCT dataset, respectively, with
the minimum age of 14 and maximum of 78. Male dominance is common in many
COVID-19 datasets [30], partly because men are more vulnerable to COVID-19
[31]. Furthermore, no correlation between gender and CT finding is found out
[32]. It is worth mentioning that to comply with the DICOM supplement 142
(Clinical Trial De-identification Profiles) [33], we have de-identified all
the CT studies.

The volumetric LDCT and ULDCT scans are obtained from a SIEMENS SOMATOM Scope
scanner. All scans are in the axial view and reconstructed into $512\times
512$ images using the Filtered Back Projection method [34]. The radiation dose
in standard chest CT scans is estimated at $7mSv$, which is reduced to
$1\textendash 1.5mSv$ in LDCT scans and as low as $0.3mSv$ in the ULDCT ones.
For patients with $>60kg$ body weight LDCT images are acquired using the $mAs$
value of $20$, $kVp$ of $110v$, and the slice thickness of $2mm$, whereas for
patients with the body weight of less than $60kg$ the ULDCT images are
obtained with $15mAs$.

As we did not have access to LDCT/ULDCT for CAP cases, we used a set of
standard-dose volumetric chest CT scans of 60 patients [21], collected before
the start of pandemic from April 2018 to December 2019. This set contains 35
male and 25 female cases, with mean age of 57.7 and standard deviation of
21.7. The slices of the CAP set are also analyzed by the first radiologist to
identify slices with evidence of infection. CAP images are acquired using tube
current of $94-500mA$, $kVp$ of $110-120v$, and the slice thickness of $2mm$,
using SIEMENS SOMATOM Scope scanner.

As shown in Table 1, all cases are accompanied by demographic and clinical
data, i.e., sex, age, weight, and presence or absence of 5 symptoms of cough,
fever, dyspnea, chest pain, and fatigue. We compared the performance of the
proposed AI model with a fourth experienced radiologist (R.A.) who was blind
to the labels, and classified the standard-dose and LDCT/ULDCT as COVID-19,
CAP, and normal, first by means of the CT scans only, and then by
incorporating the clinical data.

We also included an extra set of 100 positive COVID-19 patients, confirmed
with positive RT-PCR. Male cases constitute 53% of the total cases, and age
average is $46.16$ with a standard deviation of $14.07$. This set was
collected in April 2021.

### Data Preprocessing

We used a pre-trained U-Net-based lung segmentation model [35], referred to as
“U-net (R231CovidWeb)", to segment lung regions and discard irrelevant
information. This segmentation model is fine-tuned on COVID-19 images, which
increases its performance and reliability for the problem at hand.
Consequently, all images are downsampled from $512\times 512$ to $256\times
256$.

### Two Stage Deep Learning Model

The proposed two stage deep learning model, shown in Fig. 1, consists of two
consecutive capsule networks [18, 19], which are advantageous over commonly
used convolutional neural networks (CNNs) in handling the spatial relations
between image instances. The segmented chest CT scans are the inputs to the
first stage, which identifies images with evidence of infection. The slice
with infection could be related to a CAP or COVID-19 patient. 10 most probable
slices with infection are then selected as inputs to the second stage, which
consists of time-distributed capsule networks, referring to processing slices
at the same time through the same model. In this stage, classification
probabilities generated from individual slices go through a global max pooling
operation to make the final decision. Next, the two stages are explained in
more details.

#### Stage 1: Capsule Network

Capsule networks are relatively new AI architectures proposed to overcome some
key shortcomings of traditional deep neural networks and provide more
informative features. The key to Capsule networks’ richer feature
representation is the use of vectors (collection of neurons referred to as a
Capsule) instead of scalars (single neurons). In other words, Capsules are
groups of neurons acting as one unit, which is activated depending on the
probability that a specific entity exists in the input. Capsule networks
consist of layers of these Capsules stacked together to form a deep neural
network and learn discriminative features from the input data. While
conventional deep learning solutions are incapable of conveying information
about the relative correlations between the extracted features, Capsule
networks can address this issue (via their routing by agreement mechanism) and
better model existing correlations inside the network. Through the routing by
agreement process, capsules in a lower layer try to predict the output of the
capsules in the next layer, and predictions are given priorities based on
their correctness. The amplitude of the capsule vectors in the last layer
represents the probability that the input image belongs to a specific target
class. Another key advantage of Capsule networks is their ability to collect
more detailed information with a smaller number of trainable parameters. This
in turn results in achieving better performance with a reduced number of input
data, making them the ideal AI model for the problem at hand.

The first stage of the proposed AI framework is responsible for identifying
the slices demonstrating infection (caused by COVID-19 or CAP) in a series of
CT images corresponding to a patient. The first stage will provide a subset of
candidate slices to be analyzed in the next stage, which focuses only on the
disease type. To train the first stage, we used 2D CT images and their
corresponding label (infectious vs non-infectious) to construct a slice-level
classifier whose output determines the probability of the input image
belonging to a specific target class (infectious vs non-infectious). We then
extracted 10 slices with the highest infection probability for each patient to
be used as the input of the second stage. Given the specific characteristics
of the COVID-19 disease manifestation, which include multi-focal GGOs,
predominantly in peripheral, lower-lobes, and posterior anatomic areas of the
lung, we have adopted a capsule network-based classifier instead of the
conventional CNN-based classifiers. As demonstrated in our previous studies
[19], capsule networks are highly capable of capturing spatial relations
between the components in medical images using small datasets and fewer
parameters compared to their counterparts, which is of utmost importance in
the case of COVID-19 disease.

The architecture of the first stage initiates with a stack of four
convolutional layers, one pooling layer, and one batch normalization layer
which are augmented by two shortcut connections to deliver shallow features to
the deeper layers of the model. These layers are then followed by a stack of
three capsule layers to generate the final output, which is the probability of
the input image belonging to the related target class. It is worth noting that
in the first stage, we are dealing with an imbalanced dataset with more number
of slices without the evidence of infection. To cope with this imbalanced
dataset, we have modified the loss function in the training step and
considered a higher penalty for the errors in the slices demonstrating
infection.

#### Stage 2: Time-distributed Capsule Network

The second stage of the proposed AI framework is a time-distributed capsule
network that takes the 10 candidates from the previous stage as inputs. These
images are processed in parallel through capsule networks with the same
architecture sharing all the trainable weights. These capsule networks consist
of three convolutional layers, one batch normalization and one max pooling
layer. The output of the last convolutional layer is reshaped to form the
primary capsules, which then go through two capsule layers. The final capsule
layer for each candidate corresponds to the three classes of COVID-19, CAP,
and normal. To take into account the probability of the candidate slice being
infected, COVID-19 and CAP classes are multiplied by the infectious
probability generated by the first stage. The normal class is also multiplied
by one minus the infectious probability. At the end, a global max pooling
operation is applied to the outputs of the capsule networks corresponding to
candidate slices, to make the final decision.

We trained the second stage time-distributed capsule network with an Adam
optimizer with learning rate of $1e^{-4}$, batch size of 8, and 150 epochs.
Similar to the first stage, we used a modified margin loss function to
consider more penalty for the minority class. Margin loss is the original loss
function for capsule networks introduced in Reference [18].

### Incorporating the Clinical Data

After training the two-stage deep learning model, output probabilities of the
three classes (COVID-19, CAP, normal) are concatenated with the 8 clinical
data (demographic and symptoms, i.e., sex, age, weight, and presence or
absence of 5 symptoms of cough, fever, dyspnea, chest pain, and fatigue) and
fed to a multi-layer perceptron (MLP) model, shown in Fig. 2. This model has 4
fully-connected layers with 64 neurons, where each layer is followed by batch
normalization. The last layer includes 3 neurons with a “Softmax" activation
function. We trained the MLP model with a cross-entropy loss and Adam
optimizer with the learning rate of $1e^{-4}$, batch size of 16, and 500
epochs.

### Grad-CAM Visualization

We utilized the Grad-CAM localization mapping method [36] to provide a deep
insight into the intermediate layers and identify what components in a CT
image have obtained the most attention by the model. The Grad-CAM method
extracts the spatial information, which is preserved by the convolutional
layers and specifies the parts in the image having the most contribution to
the final prediction. More specifically, the Grad-CAM method generates a
localization heatmap corresponding to each layer and the target class to
determine the locations to which the model paid the most attention. This
localization heatmap is derived by a weighted average of all feature maps in
the convolutional layer followed by a Rectified Linear Unit (ReLU) activation
function.

### Statistical Analysis

K-fold cross-validation [37] is a statistical approach to assess the
performance of a model on an unseen dataset. According to this approach, the
original dataset is randomly split into $K$ equal number of samples, and
through $K$ iterations of training and testing, each of the $K$ sets is set
aside for testing, and the rest is used for training. This approach is used in
this study to evaluate the performance of the AI model as well as the
radiologist blind to the labels, where $K$ is set to 10. Performance of each
fold is reported, along with the mean and standard deviation over all the
folds. Furthermore, in each $K$ fold, 30% of the training set is used for the
validation of the associated model, according to which the most optimal model
is selected and tested on the test set. McNemar [22] test, with the
significance level of 0.05, is used to test the hypothesis of human and AI
model having the same performance. Logistic regression is applied to assess
the significance of clinical factors in three binary classification scenarios,
i.e., COVID-19 versus CAP and normal, CAP versus COVID-19 and normal, and
normal versus COVID-19 and CAP.

### Data Availability

Data is publicly available at “https://ieee-dataport.org/open-
access/covid-19-low-dose-and-ultra-low-dose-ct-scans".

### Code Availability

Codes are publicly available at “https://github.com/ParnianA/LDCT-COVID".

### Acknowledgements

This work was partially supported by the Natural Sciences & Engineering
Research Council (NSERC) of Canada through RGPIN-2016-04988.

### Author Contributions

P.A. and Sh.H. implemented the AI models. P.A., M.J.R., and A.M. drafted the
manuscript. M.J.R. collected the dataset and was the first radiologist to
label the patients. A.O. and F.B.F. were the two other radiologists to label
the dataset. R.A. was the forth radiologist with whom the AI model was
compared. F.N. and N.E. performed the statistical analysis. K.N.P. and K.F.
supervised the validation of the model. A.M. supervised the study. All authors
revised the manuscript.

### Competing Interest

The authors declare no competing interests.

## References

  * [1] Wu, Z. & McGoogan, J.  Characteristics of and important lessons from the coronavirus disease 2019 (covid-19) outbreak in china: Summary of a report of 72314 cases from the chinese center for disease control and prevention.  _JAMA_ 323, 1239–1242 (2020).
  * [2] Fang, Y. _et al._ Coronavirus disease 2019 (covid-19): a systematic review of imaging findings in 919 patients.  _Radiology_ 296, E115–E117 (2020).
  * [3] Salehi, S., Abedi, A., Balakrishnan, S. & Gholamrezanezhad, A.  Sensitivity of chest ct for covid-19: Comparison to rt-pcr.  _American Journal of Roentgenology_ 215, 87–93 (2020).
  * [4] Rafiee, M., Babaki Fard, F., Samimi, K., Rasti, H. & Pressaco, J.  Spontaneous pneumothorax and pneumomediastinum as a rare complication of covid-19 pneumonia: Report of 6 cases.  _Radiology case reports_ 16, 687–692 (2021).
  * [5] Miglioretti, D. _et al._ The use of computed tomography in pediatrics and the associated radiation exposure and estimated cancer risk.  _JAMA Pediatr_ 167, 700–707 (2013).
  * [6] Fearon, T. & Vucich, J.  Pediatric patient exposures from ct examinations: Ge ct/t 9800 scanner.  _AJR_ 144, 805–809 (1985).
  * [7] Pearce, M. _et al._ Radiation exposure from ct scans in childhood and subsequent risk of leukaemia and brain tumours: a retrospective cohort study.  _Lancet_ 380, 499–505 (2012).
  * [8] Bell, D. & Gerstmair, A.  As low as reasonably achievable (alara).
  * [9] Taekker, M., Kristjansdottir, B., Ole, G., Laursen, C. & Pietersen, P.  Diagnostic accuracy of lowdose and ultra-low-dose ct in detection of chest pathology: a systematic review.  _Clinical Imaging_ 74, 139–148 (2021).
  * [10] Sakane, H. _et al._ Biological effects of lowdose chest ct on chromosomal dna.  _Radiology_ 295, 439–445 (2020).
  * [11] Park, J. _et al._ The usefulness of low-dose ct scan in elderly patients with suspected acute lower respiratory infection in the emergency room.  _The British Journal of Radiology_ 89 (2016).
  * [12] Schulze-Hagen, M. _et al._ Low-dose chest ct for the diagnosis of covid-19—a systematic, prospective comparison with pcr.  _Dtsch Arztebl Int_ 117, 389–395 (2020).
  * [13] Dangis, A. _et al._ Accuracy and reproducibility of low-dose submillisievert chest ct for the diagnosis of covid-19.  _Radiol Cardiothorac Imaging_ 2, e200196 (2020).
  * [14] Tofighi, S., Najafi, S., Johnston, S. & Gholamrezanezhad, A.  Low-dose ct in covid-19 outbreak: radiation safety, image wisely, and image gently pledge.  _Emergency Radiology_ 27, 601–605 (2020).
  * [15] Ardila, D. _et al._ End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography.  _Nature Medicine_ 25, 954–961 (2019).
  * [16] Espinoza, J. L. & Dong, L. T.  Artificial intelligence tools for refining lung cancer screening.  _Journal of clinical medicine_ 9, 3860 (2020).
  * [17] Shiri, I. _et al._ Ultra-low-dose chest ct imaging of covid-19 patients using a deep residual neural network.  _European Radiology_ 31, 1420–1431 (2021).
  * [18] Sabour, S., Frosst, N. & Hinton, G. E.  Dynamic routing between capsules.  _Proceedings of the 31st International Conference on Neural Information Processing Systems_ 3859–3869 (2017).
  * [19] Afshar, P. _et al._ 3d-mcn: A 3d multi-scale capsule network for lung nodule malignancy prediction.  _Scientific Reports_ 10, 7948 (2020).
  * [20] Mei, X. _et al._ Artificial intelligence–enabled rapid diagnosis of patients with covid-19.  _Nature Medicine_ 26, 1224–1228 (2020).
  * [21] Afshar, P. _et al._ Covid-ct-md, covid-19 computed tomography scan dataset applicable in machine learning and deep learning.  _Scientific Data_ 8 (2021).
  * [22] McNemar, Q.  Psychological statistics.  _Wiley_ (1962).
  * [23] Prokop, M. _et al._ Co-rads: A categorical ct assessment scheme for patients suspected of having covid-19—definition and evaluation.  _Radiology_ 296, E97–E104 (2020).
  * [24] Altmayer, S. _et al._ Comparison of the computed tomography findings in covid-19 and other viral pneumonia in immunocompetent adults: a systematic review and meta-analysis.  _European Radiology_ 1–12 (2020).
  * [25] Jonas, D. _et al._ Screening for lung cancer with low-dose computed tomography: Updated evidence report and systematic review for the us preventive services task force.  _JAMA_ 325, 971–987 (2021).
  * [26] Armato, S. _et al._ The lung image database consortium (lidc) and image database resource initiative (idri): A completed reference database of lung nodules on ct scans.  _Medical Physics_ 38, 915–931 (2011).
  * [27] Finance, J. _et al._ Low dose chest ct and lung ultrasound for the diagnosis and management of covid-19.  _Journal of Clinical Medicine_ 10 (2021).
  * [28] Wu, E. _et al._ How medical ai devices are evaluated: limitations and recommendations from an analysis of fda approvals.  _Nature Medicine_ 27, 576–584 (2021).
  * [29] Kleppe, A. _et al._ Designing deep learning studies in cancer diagnostics.  _Nature Reviews Cancer_ 21, 199–211 (2021).
  * [30] Sun, Z., Zhang, N., Li, Y. & Xu, X.  A systematic review of chest imaging findings in covid-19.  _Quantitative imaging in medicine and surgery_ 10, 1058–1079 (2020).
  * [31] Bwire, G.  Coronavirus: Why men are more vulnerable to covid-19 than women?  _SN Comprehensive Clinical Medicine_ 2, 874–876, DOI: https://doi.org/10.1007/s42399-020-00341-w (2020).
  * [32] Francone, M. _et al._ Chest ct score in covid-19 patients: correlation with disease severity and short-term prognosis.  _European Radiology_ 30, 6808–6817 (2020).
  * [33] Committee, D. S., Group, W., Trials, C. & Text, F.  Supplement 142: Clinical trial de-identification profiles.  _DICOM Standard_ 1–44 (2011).
  * [34] Pontana, F. _et al._ Chest computed tomography using iterative reconstruction vs filtered back projection (part 1): evaluation of image noise reduction in 32 patients.  _European Radiology_ 21, 627–635 (2011).
  * [35] Hofmanninger, J. _et al._ Automatic lung segmentation in routine imaging is primarily a data diversity problem, not a methodology problem.  _European Radiology Experimental_ 4, 50 (2020).
  * [36] Selvaraju, R. R. _et al._ Grad-cam: Visual explanations from deep networks via gradient-based localization.  _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ (2017).
  * [37] Nilashi, M., Ibrahim, O. & Ahani, A.  Accuracy improvement for predicting parkinson’s disease progression.  _Scientific Reports_ 6 (2016).


# Minimum Bayes’ Risk Decoding for System Combination of Grammatical Error
Correction Systems

Vyas Raina
ALTA Institute
Department of Engineering
University of Cambridge
<EMAIL_ADDRESS>
&Mark Gales
ALTA Institute
Department of Engineering
University of Cambridge
<EMAIL_ADDRESS>

###### Abstract

For sequence-to-sequence tasks it is challenging to combine individual system
outputs. Further, there is also often a mismatch between the decoding
criterion and the one used for assessment. Minimum Bayes’ Risk (MBR) decoding
can be used to combine system outputs in a manner that encourages better
alignment with the final assessment criterion. This paper examines MBR
decoding for Grammatical Error Correction (GEC) systems, where performance is
usually evaluated in terms of edits and an associated F-score. Hence, we
propose a novel MBR loss function directly linked to this form of criterion.
Furthermore, an approach to expand the possible set of candidate sentences is
described. This builds on a current max-voting combination scheme, as well as
individual edit-level selection. Experiments on three popular GEC datasets and
with state-of-the-art GEC systems demonstrate the efficacy of the proposed MBR
approach. Additionally, the paper highlights how varying reward metrics within
the MBR decoding framework can provide control over precision, recall, and the
F-score in combined GEC systems. 111Code available at:
https://github.com/rainavyas/mbr_gec

##  1 Introduction

Ensembling, the combination of system outputs, is a powerful technique in deep
learning, exploiting diverse model capabilities for robust predictions. Though
numerous methodologies exist for system combination (Ganaie et al., 2021),
when there is only access to model outputs, many methods are inapplicable and
thus the simplest method becomes the averaging of model outputs. However, for
sequence-to-sequence (seq2seq) systems, such as summarization, machine
translation, and grammatical error correction (GEC), output averaging is less
straightforward. A further challenge with seq2seq tasks is the mismatch
between the decoding and assessment criteria. Kumar and Byrne (2004) proposed
the utilization of Minimum Bayes’ Risk (MBR) decoding as a means to select an
output that minimizes the theoretical risk according to a designated reward
metric. We propose a novel variant of MBR decoding for GEC to allow for system
combination and give better alignment with the assessment criteria.

The nature of a GEC task permits the use of MBR decoding within the
"edit"-space. Each output sequence can be represented as a set of "edits"
required to transform the input sequence into the output. Consequently, the
selection of a single output sequence for GEC can be achieved through MBR
decoding with a reward function defined on the set of edits, aligned with the
edit-based F-score typically used in GEC assessment criteria. Beyond
selection, an additional technique known as max-voting (Tarnavskyi et al.,
2022) can be employed to combine different sets of edits. We propose an
enhancement to the performance achieved through max-voting by treating the
output sequences obtained from the combination as additional candidates for
MBR decoding. Further, with a greedy MBR decoding algorithm, we explore the
edit space to identify other candidate edit sets. Through experiments on three
popular GEC datasets and use of state of the art GEC systems (Grammarly’s
GECToR (Omelianchuk et al., 2020)), we demonstrate that our MBR decoding
approach in the edit space consistently leads to significant performance
gains. Further, we also show that by selecting different reward metrics as
part of the MBR decoding approach we can provide explicit control over
precision, recall and the overall F-score used to assess GEC systems.

##  2 Related Work

Grammatical Error Correction: Early GEC systems using hand-crafted rules
(Naber, 2003) were replaced by encoder-decoder architectures, using for
example Recurrent Neural Networks (Cho et al., 2014). Today, many state of the
art GEC systems use Transformer-based (Vaswani et al., 2017) encoder-decoder
architectures to perform the sequence-to-sequence GEC task Kaneko et al.
(2020); Chen et al. (2020); Kiyono et al. (2019); Lichtarge et al. (2020);
Stahlberg and Kumar (2020). However, LaserTagger (Malmi et al., 2019), the PIE
model (Awasthi et al., 2019) and Grammarly’s GECToR (Omelianchuk et al., 2020)
are all able to achieve competitive performance using a sequence-to-edit
structure for the overall sequence-to-sequence task, where a token can be
tagged with edit operations. Once a set of tags have been defined, the edit
operations can be applied to the input sequence to generate the grammatically
correct output sequence. The GECToR system is particularly efficient at
inference as it uses a Transformer encoder followed by softmax over linear
layers for edit tag prediction, which is significantly faster than standard
sequence-to-sequence GEC system decoders. Further, Wu et al. (2023)
demonstrated that GECToR performs better than the most recent generative large
language models, e.g. ChatGPT (Brown et al., 2020), which tend to over-
correct, compromising on recall performance. Hence this work uses the GECToR
model as its base GEC architecture.

System Combination for seqseq systems: Individual deep learning systems for
classification tasks can be combined in many ways: stacking Wolpert (1992),
negative correlation learning Liu and Yao (1999), max-voter schemes Ju et al.
(2018); Simonyan and Zisserman (2014) or probability averaging He et al.
(2016); Raina et al. (2020); Szegedy et al. (2015). However, for generative
language tasks such as GEC, where the output is a sequence of tokens, many
traditional ensembling approaches are inapplicable. Sequence-level ensembling
approaches, however, can address this by averaging conditional token level
probabilities of multiple systems Sennrich et al. (2015); Freitag et al.
(2017); Malinin and Gales (2021); Fathullah et al. (2021). However, this
approach requires identical member architectures as well as access to the
output probabilities of the predicted tokens. With the rising trend of limited
black box access to large language models (e.g. ChatGPT Liu et al. (2023)),
system combination methods that only require the generated output sequences
have practical benefit.

With access to only the output sequences from individual seq2seq systems, it
is challenging to combine them into a single output. For automatic speech
recognition, Sim et al. (2007) select a single output using a simple Minimum
Bayes’ Risk (MBR) decoding approach (Kumar and Byrne, 2004), where the aim is
effectively to select the most average/representative output sequence.
Similarly Manakul et al. (2023) use MBR to combine sequences for clinical
document summarization. The MBR approach has also recently been applied to
machine translation (Rosti et al., 2007a, b; Freitag et al., 2022; Müller and
Sennrich, 2021; Zhang et al., 2022). For GEC systems, Tarnavskyi et al. (2022)
propose a max voting scheme, where only edits predicted by the majority of
individual systems are retained. We further improve GEC performance by
applying MBR decoding to a sequence selection set augmented with sequences
from max voting. We further enrich this selection space with a greedy search
over edits.

##  3 Output Sequence Combination for GEC

A Grammatical Error Correction (GEC) system predicts a grammatically correct
output sequence $\mathbf{y}$ from an input sequence, $\mathbf{x}$. With
multiple different GEC system output sequence predictions,
$\mathcal{Y}=\\{\mathbf{y}_{1},\ldots,\mathbf{y}_{N}\\}$, for the same input
sequence, $\mathbf{x}$, it is challenging to combine them into a single, best
sequence. It is useful to consider the edit-space, where a set of edits,
$\mathbf{e}_{n}(\mathbf{x},\mathbf{y}_{n})=\\{e_{1},\ldots,e_{|\mathbf{e}_{n}|}\\}$
can be used to represent each predicted output sequence, $\mathbf{y}_{n}$
222Given an input sequence $\mathbf{x}$ and an output sequence $\mathbf{y}$ it
is simple to create an edit set, using tools such as ERRANT (Bryant et al.,
2017).. A single edit in the edit set can be defined fully by an input token
in $\mathbf{x}$ and an edit operation to apply (insertion, deletion or
substitution). This section describes how Minimum Bayes’ Risk decoding can be
used in the edit-space to combine the different output sequences in
$\mathcal{Y}$.

###  3.1 MBR decoding for GEC

MBR decoding aims to select the most representative output sequence,
$\mathbf{y}^{*}\in\mathcal{Y}$. For GEC, we aim to maximise a reward score
$\mathcal{R}$ in the edit-space that encourages better alignment with the
final assessment metric,

$\mathbf{y}^{*}=\operatorname*{arg\,max}_{{\mathbf{y}}\in\mathcal{Y}}\left\\{\mathbb{E}_{p(\tilde{\mathbf{y}}|x)}[\mathcal{R}(\tilde{\mathbf{e}}(\mathbf{x},\tilde{\mathbf{y}}),\mathbf{e}(\mathbf{x},\mathbf{y}))]\right\\},$
(1)

where the reward score, $\mathcal{R}(\tilde{\mathbf{e}},\mathbf{e})$, views
$\tilde{\mathbf{e}}$ as reference edits and e as the hypothesis/predicted
edits. In practice, it is difficult to meaningfully estimate the posterior
distribution, $p(\tilde{\mathbf{y}}|x)$ for each output sequence. Hence, we
consider only similarly performing systems’ output sequences,
$\mathcal{Y}^{(\text{c})}\in\mathcal{Y}$ to calculate the expectation of the
reward and so we approximate each of these sequences as equiprobable,

$\mathbf{y}^{*}\approx\operatorname*{arg\,max}_{{\mathbf{y}}\in\mathcal{Y}^{(\text{s})}}\left\\{\frac{1}{|\mathcal{Y}^{\text{(c)}}|}\sum_{\tilde{\mathbf{y}}\in\mathcal{Y}^{\text{(c)}}}\mathcal{R}(\tilde{\mathbf{e}}(\mathbf{x},\tilde{\mathbf{y}}),\mathbf{e}(\mathbf{x},\mathbf{y}))\right\\},$
(2)

where $\mathcal{Y}^{(\text{s})}$ represents the set of possible output
sequences we want to select from.

###  3.2 MBR decoding with edit voting

Inspired by Tarnavskyi et al. (2022) the different edit sets,
$\\{\mathbf{e_{1}},\ldots,\mathbf{e}_{N}\\}$ associated with the different
output sequences, can be combined to create a single edit set,
$\mathbf{e}^{(m)}$ containing all the individual edits present in at least $m$
of the edit sets (i.e. $m$ votes). This new combined edit set represents a new
combined output sequence, $\mathbf{y}^{(m)}$. The MBR decoding approach of
Equation 1 can now be applied by simply including the combined sequence in the
set of sequences to select from, such that
$\mathbf{y}^{(m)}\in\mathcal{Y}^{(\text{s})}$. Note that the voting scheme can
generate a maximum of $N$ different combined sequences, with
$\mathbf{e}^{(1)}$ being the union of all edit sets and $\mathbf{e}^{(N)}$ the
intersection. Hence the selection space of sequences
$\mathcal{Y}^{(\text{s})}$ can be made richer with an extra $N$ sequences.

###  3.3 Greedy MBR decoding for edit selection

Instead of augmenting the selection set $\mathcal{Y}^{(\text{s})}$ with only a
few sequences, it is useful to consider all possible edit sets. However, it is
computationally infeasible to consider every possible edit set. Hence, this
work proposes a practical, greedy method to increase the richness of the
selection set. The minimal edit set is arguably the intersection of all edit
sets, $\mathbf{e}^{(N)}$. In contrast the set of possible edits is given by
the union set, $\mathbf{e}^{(1)}$. Hence, we can insert individual edits one
by one from the union set to the intersection set. Every new edit insertion
into the existing edit set represents a new output sequence $\mathbf{y}$ (that
can be added to $\mathcal{Y}^{(\text{s})}$). However, we only retain the edit
insertions that give a new output sequence that increases the MBR expected
reward,
$\frac{1}{|\mathcal{Y}^{\text{(c)}}|}\sum_{\tilde{\mathbf{y}}\in\mathcal{Y}^{\text{(c)}}}\mathcal{R}(\tilde{\mathbf{e}}(\mathbf{x},\tilde{\mathbf{y}}),\mathbf{e}(\mathbf{x},\mathbf{y}))$
from Equation 2. This way we can efficiently search a richer selection set,
$\mathcal{Y}^{(\text{s})}$ of output sequences to find the best combined
output sequence $\mathbf{y}^{*}$.

###  3.4 MBR reward score

Equation 1 uses a reward score $\mathcal{R}(\tilde{\mathbf{e}},\mathbf{e})$ to
perform MBR decoding. Careful selection of the reward score allows for control
over the desired metric to optimise. We can for example aim to combine systems
in a manner that encourages better edit recall,

$\mathcal{R}^{\text{(rec)}}(\tilde{\mathbf{e}},\mathbf{e})=\frac{|\tilde{\mathbf{e}}\cap\mathbf{e}|}{|\tilde{\mathbf{e}}|}.$
(3)

Conversely, it may be desirable to have a system with high precision,

$\mathcal{R}^{\text{(prec)}}(\tilde{\mathbf{e}},\mathbf{e})=\frac{|\tilde{\mathbf{e}}\cap\mathbf{e}|}{|\mathbf{e}|}.$
(4)

However, it is usually desirable to have a GEC system with a good combination
of precision and recall, as measured by a F-k score,

$\mathcal{R}^{\text{(f\\{k\\})}}(\tilde{\mathbf{e}},\mathbf{e})=\frac{(1+k^{2})|\tilde{\mathbf{e}}\cap\mathbf{e}|}{|\tilde{\mathbf{e}}|k+|\mathbf{e}|}.$
(5)

As the precision is more important than recall for GEC systems, this work
aligns the reward metric with the F0.5 score. The Jaccard Similarity reward
metric is also explored as an alternative in Appendix A.

##  4 Experiments

###  4.1 Experimental setup

We evaluate performance of the combined systems on three popular grammatical
error correction corpora. First Certificate in English (FCE) corpus
Yannakoudakis et al. (2011) is a subset of Cambridge Learner Corpus OpenCLC
(2019) made up of written examinations for general and business English of
candidates from 86 different mother tongues, consisting of 2,720 test
sentences. Building Education Applications 2019 (BEA-19) Bryant et al. (2019)
offers a test set of 4477 sentences, sourced from essays written by native and
non-native English students. Conference on Computational Natural Language
Learning 2014 (CoNLL-14) Ng et al. (2014) test set consists of 1312 sentences
sourced from 50 essays written by 25 non-native English speakers. Three
different state of the art GECToR models are used as the individual systems to
be combined 333GECToR model Weights:
https://github.com/grammarly/gector#pretrained-models. Each system uses a
different Transformer encoder (bert (b), roberta (r) or xlnet (x)). Table 1
gives the performance of these individual systems 444GEC performance for CoNLL
and FCE is measured using the ERRANT tool (Bryant et al., 2017). Note that
CoNLL is often evaluated with a different scorer in other papers. BEA is
evaluated using the online submission portal:
https://codalab.lisn.upsaclay.fr/competitions/4057.

Model | conll | bea | fce
---|---|---|---
b | 56.15$\left(\begin{subarray}{c}61.75\\\ 41.19\end{subarray}\right)$ | 65.41$\left(\begin{subarray}{c}67.33\\\ 58.71\end{subarray}\right)$ | 49.66$\left(\begin{subarray}{c}54.47\\\ 36.68\end{subarray}\right)$
r | 56.82$\left(\begin{subarray}{c}61.99\\\ 42.59\end{subarray}\right)$ | 68.21$\left(\begin{subarray}{c}70.21\\\ 61.21\end{subarray}\right)$ | 49.86$\left(\begin{subarray}{c}53.47\\\ 39.28\end{subarray}\right)$
x | 56.77$\left(\begin{subarray}{c}61.74\\\ 42.95\end{subarray}\right)$ | 68.00$\left(\begin{subarray}{c}69.89\\\ 61.36\end{subarray}\right)$ | 50.52$\left(\begin{subarray}{c}53.49\\\ 41.00\end{subarray}\right)$
Table 1: F0.5 and (precision, recall) performance for individual GECToR
systems

###  4.2 Results

MBR decoding (Equation 2) is applied in the edit-space for the three
individual GECToR systems’ outputs (b,r,x). Here, as the systems have similar
performance (equiprobable posterior assumption valid), we let the selection
set and the set of sequences to calculate the expected reward be the same
$\mathcal{Y}^{(\text{s})}=\mathcal{Y}^{(\text{c})}=\\{b,r,x\\}$. Table 2
compares the different reward functions, $\mathcal{R}$, when applying MBR
decoding. Selection with precision (Equation 6) and F0.5 (Equation 5) oriented
reward metrics give a significant increase in performance over the individual
systems in Table 1. Although the recall reward (Equation 3) does not increase
F0.5 performance, it does significantly increase recall performance. This
demonstrates that a simple application of MBR decoding can be used to combine
individual systems to improve performance and selection of the reward function
gives specific control over precision and recall of the combined system.

Reward | conll | bea | fce
---|---|---|---
$\mathcal{R}^{\text{(rec)}}$ | 55.13$\left(\begin{subarray}{c}57.76\\\ 46.66\end{subarray}\right)$ | 64.67$\left(\begin{subarray}{c}64.59\\\ 64.99\end{subarray}\right)$ | 48.74$\left(\begin{subarray}{c}50.12\\\ 43.90\end{subarray}\right)$
$\mathcal{R}^{\text{(prec)}}$ | 59.78$\left(\begin{subarray}{c}69.38\\\ 34.48\end{subarray}\right)$ | 70.87$\left(\begin{subarray}{c}75.93\\\ 55.96\end{subarray}\right)$ | 52.35$\left(\begin{subarray}{c}60.13\\\ 34.50\end{subarray}\right)$
$\mathcal{R}^{\text{(f05)}}$ | 59.71$\left(\begin{subarray}{c}66.42\\\ 42.53\end{subarray}\right)$ | 69.95$\left(\begin{subarray}{c}72.95\\\ 60.07\end{subarray}\right)$ | 52.05$\left(\begin{subarray}{c}56.61\\\ 39.36\end{subarray}\right)$
Table 2: MBR with
$\mathcal{Y}^{(\text{c})}=\mathcal{Y}^{(\text{s})}=\\{b,r,x\\}$.

Section 3.2 describes how MBR decoding can be applied to systems combined by a
voting scheme in the edit space. Table 3 shows the performance of systems
combined with voting, where an individual edit requires $m$ votes (from b,r or
x edit system predictions) to be included in the combined edit set,
$\mathbf{e}^{(m)}$ to form the single combined sequence $\mathbf{y}^{(m)}$.
Note here that $\mathbf{e}^{(1)}$ is the union set and $\mathbf{e}^{(3)}$ is
the intersection and so these sequences encourage either a higher recall or
precision respectively. Table 4 shows the impact of MBR decoding where all the
separate voting sets ($\mathbf{y}^{(1)},\mathbf{y}^{(2)},\mathbf{y}^{(3)}$)
are included in the selection set,
$\mathcal{Y}^{(\text{s})}=\\{b,r,x,\mathbf{y}^{(1)},\mathbf{y}^{(2)},\mathbf{y}^{(3)}\\}$.
Note that we maintain the same set of sequences for the expected reward
calculation, $\mathcal{Y}^{(\text{s})}=\\{b,r,x\\}$ to ensure the equiprobable
posterior assumption holds 555Experiments with an alternative set of sequences
for $\mathcal{Y}^{(\text{c})}$ are in Appendix C. It is evident that a richer
selection set allows for even greater improvements in model performance for
precision and F0.5 reward MBR decoding.

System | conll | bea | fce
---|---|---|---
$\mathbf{y}^{(1)}$ | 47.13$\left(\begin{subarray}{c}48.02\\\ 43.89\end{subarray}\right)$ | 55.94$\left(\begin{subarray}{c}55.03\\\ 59.91\end{subarray}\right)$ | 41.76$\left(\begin{subarray}{c}42.38\\\ 39.43\end{subarray}\right)$
$\mathbf{y}^{(2)}$ | 60.58$\left(\begin{subarray}{c}68.41\\\ 41.54\end{subarray}\right)$ | 71.82$\left(\begin{subarray}{c}75.86\\\ 59.22\end{subarray}\right)$ | 52.73$\left(\begin{subarray}{c}58.16\\\ 38.38\end{subarray}\right)$
$\mathbf{y}^{(3)}$ | 59.60$\left(\begin{subarray}{c}77.30\\\ 31.10\end{subarray}\right)$ | 72.96$\left(\begin{subarray}{c}84.32\\\ 47.41\end{subarray}\right)$ | 52.50$\left(\begin{subarray}{c}67.05\\\ 28.01\end{subarray}\right)$
Table 3: Voting combination, $\mathbf{y}^{(m)}$ ($m$ votes). Reward | conll | bea | fce
---|---|---|---
$\mathcal{R}^{\text{(rec)}}$ | 53.99$\left(\begin{subarray}{c}55.99\\\ 47.23\end{subarray}\right)$ | 63.81$\left(\begin{subarray}{c}63.47\\\ 65.25\end{subarray}\right)$ | 48.18$\left(\begin{subarray}{c}49.29\\\ 44.20\end{subarray}\right)$
$\mathcal{R}^{\text{(prec)}}$ | 60.24$\left(\begin{subarray}{c}76.59\\\ 32.50\end{subarray}\right)$ | 73.42$\left(\begin{subarray}{c}83.40\\\ 49.66\end{subarray}\right)$ | 53.51$\left(\begin{subarray}{c}66.74\\\ 29.85\end{subarray}\right)$
$\mathcal{R}^{\text{(f05)}}$ | 60.43$\left(\begin{subarray}{c}67.94\\\ 41.90\end{subarray}\right)$ | 70.84$\left(\begin{subarray}{c}74.48\\\ 59.25\end{subarray}\right)$ | 52.71$\left(\begin{subarray}{c}57.83\\\ 38.92\end{subarray}\right)$
Table 4: MBR with $\mathcal{Y}^{(\text{c})}=\\{b,r,x\\}$ and
$\mathcal{Y}^{(\text{s})}=\\{b,r,x,\mathbf{y}^{(1)},\mathbf{y}^{(2)},\mathbf{y}^{(3)}\\}$.

Finally, as described in Section 3.3, MBR decoding can be performed over a
richer edit selection space by greedily adding individual edits to the
intersection edit set, $\mathbf{e}^{(3)}$ from the union edit set,
$\mathbf{e}^{(1)}$. Experiments revealed (Appendix B) that allowing for all
edits to be included from the union set can significantly increase the risk of
poor insertions, compromising performance. Hence, instead we only consider
edits from $\mathbf{e}^{(2)}$ to be added to the intersection set
$\mathbf{e}^{(3)}$. Table 5 demonstrates that MBR decoding over this richer
set of sequences can give better performance (CoNLL) than MBR with voting, but
does not always give the best performance (BEA and FCE have better performance
in Table 4). This is perhaps because the expected reward over the individual
systems (b,r,x) is not necessarily perfectly aligned with the final F0.5 score
relative to the true reference edits used in evaluation and thus over-
optimisation of the selection set for MBR decoding does not help performance
for some datasets.

Reward | conll | bea | fce
---|---|---|---
$\mathcal{R}^{\text{(rec)}}$ | 61.06$\left(\begin{subarray}{c}69.50\\\ 41.11\end{subarray}\right)$ | 72.20$\left(\begin{subarray}{c}76.87\\\ 58.08\end{subarray}\right)$ | 52.94$\left(\begin{subarray}{c}58.87\\\ 37.73\end{subarray}\right)$
$\mathcal{R}^{\text{(prec)}}$ | 59.65$\left(\begin{subarray}{c}76.58\\\ 30.79\end{subarray}\right)$ | 72.76$\left(\begin{subarray}{c}84.18\\\ 47.16\end{subarray}\right)$ | 52.62$\left(\begin{subarray}{c}67.61\\\ 27.89\end{subarray}\right)$
$\mathcal{R}^{\text{(f05)}}$ | 61.08$\left(\begin{subarray}{c}69.71\\\ 40.85\end{subarray}\right)$ | 72.34$\left(\begin{subarray}{c}77.16\\\ 57.19\end{subarray}\right)$ | 53.00$\left(\begin{subarray}{c}59.07\\\ 37.57\end{subarray}\right)$
Table 5: MBR with $\mathcal{Y}^{(\text{c})}=\\{b,r,x\\}$ and greedy search for
$\mathcal{Y}^{(\text{s})}$.

##  5 Conclusions

The combination of sequence-to-sequence grammatical error correction (GEC)
systems is challenging. There is also often a mismatch between the decoding
criterion and assessment criterion used for GEC systems. This work
demonstrates that a novel Minimum Bayes’ Risk (MBR) decoding approach within
the edit-space can give an effective system combination method that aligns
better with the assessment criteria. We further showed that enhancing the
selection space to encompass sequences formulated by max-voting over
individual edits can further improve system performance. Moreover, the
employment of a greedy search strategy, guided by an MBR reward function, can
result in performance gains for the combined system. Crucially, the choice of
a reward function in the MBR framework gives users the ability to optimize
desired characteristics of the combined GEC system, such as precision, recall
or the F-score.

##  6 Limitations

This work explored how MBR decoding can be used to combine individual GEC
systems, as well as align the combined system’s performance to the edit-based
F-score used to assess GEC systems. Experiments were performed with
Grammarly’s GECToR based systems. It would be useful to extend these
experiments to other state of the art GEC systems. Although these other
systems are not as efficient as GECToR due to the use of an auto-regressive
Transformer decoder (as opposed to GECToR’s encoder only structure), it is
still meaningful to understand how these systems react to MBR decoding used
for system combination. This is particularly relevant as generative large
language models are increasingly used for standard natural language tasks.

##  7 Ethics Statement

This work reports on an efficient method to combine individual GEC system
outputs in a manner that better aligns with assessment and improve
performance. There are no perceived ethical risks associated with this work.

##  8 Acknowledgements

This paper reports on research supported by Cambridge University Press &
Assessment (CUP&A), a department of The Chancellor, Masters, and Scholars of
the University of Cambridge.

## References

  * Awasthi et al. (2019) Abhijeet Awasthi, Sunita Sarawagi, Rasna Goyal, Sabyasachi Ghosh, and Vihari Piratla. 2019.  Parallel iterative edit models for local sequence transduction.  _CoRR_ , abs/1910.02893.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  _CoRR_ , abs/2005.14165.
  * Bryant et al. (2019) Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019\.  The BEA-2019 shared task on grammatical error correction.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 52–75, Florence, Italy. Association for Computational Linguistics.
  * Bryant et al. (2017) Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017.  Automatic annotation and evaluation of error types for grammatical error correction.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 793–805, Vancouver, Canada. Association for Computational Linguistics.
  * Chen et al. (2020) Mengyun Chen, Tao Ge, Xingxing Zhang, Furu Wei, and Ming Zhou. 2020.  Improving the efficiency of grammatical error correction with erroneous span detection and correction.  _CoRR_ , abs/2010.03260.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using RNN encoder-decoder for statistical machine translation.  _CoRR_ , abs/1406.1078.
  * Fathullah et al. (2021) Yassir Fathullah, Mark J.F. Gales, and Andrey Malinin. 2021.  Ensemble distillation approaches for grammatical error correction.  In _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 2745–2749.
  * Freitag et al. (2017) Markus Freitag, Yaser Al-Onaizan, and Baskaran Sankaran. 2017.  Ensemble distillation for neural machine translation.  _arXiv preprint arXiv:1702.01802_.
  * Freitag et al. (2022) Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. 2022.  High quality rather than high model probability: Minimum Bayes risk decoding with neural metrics.  _Transactions of the Association for Computational Linguistics_ , 10:811–825.
  * Ganaie et al. (2021) Mudasir A. Ganaie, Minghui Hu, Mohammad Tanveer, and Ponnuthurai N. Suganthan. 2021\.  Ensemble deep learning: A review.  _CoRR_ , abs/2104.02395.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778.
  * Ju et al. (2018) Cheng Ju, Aurélien Bibaut, and Mark van der Laan. 2018.  The relative performance of ensemble methods with deep convolutional neural networks for image classification.  _Journal of Applied Statistics_ , 45(15):2800–2818.
  * Kaneko et al. (2020) Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, and Kentaro Inui. 2020.  Encoder-decoder models can benefit from pre-trained masked language models in grammatical error correction.  _CoRR_ , abs/2005.00987.
  * Kiyono et al. (2019) Shun Kiyono, Jun Suzuki, Masato Mita, Tomoya Mizumoto, and Kentaro Inui. 2019.  An empirical study of incorporating pseudo data into grammatical error correction.  _CoRR_ , abs/1909.00502.
  * Kumar and Byrne (2004) Shankar Kumar and William Byrne. 2004.  Minimum Bayes-risk decoding for statistical machine translation.  In _Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004_ , pages 169–176, Boston, Massachusetts, USA. Association for Computational Linguistics.
  * Lichtarge et al. (2020) Jared Lichtarge, Chris Alberti, and Shankar Kumar. 2020.  Data weighted training strategies for grammatical error correction.  _CoRR_ , abs/2008.02976.
  * Liu et al. (2023) Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, Zihao Wu, Dajiang Zhu, Xiang Li, Ning Qiang, Dingang Shen, Tianming Liu, and Bao Ge. 2023.  Summary of chatgpt/gpt-4 research and perspective towards the future of large language models.
  * Liu and Yao (1999) Yong Liu and Xin Yao. 1999.  Ensemble learning via negative correlation.  _Neural networks_ , 12(10):1399–1404.
  * Malinin and Gales (2021) Andrey Malinin and Mark Gales. 2021.  Uncertainty estimation in autoregressive structured prediction.  In _International Conference on Learning Representations_.
  * Malmi et al. (2019) Eric Malmi, Sebastian Krause, Sascha Rothe, Daniil Mirylenka, and Aliaksei Severyn. 2019.  Encode, tag, realize: High-precision text editing.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5054–5065, Hong Kong, China. Association for Computational Linguistics.
  * Manakul et al. (2023) Potsawee Manakul, Yassir Fathullah, Adian Liusie, Vyas Raina, Vatsal Raina, and Mark Gales. 2023.  Cued at probsum 2023: Hierarchical ensemble of summarization models.
  * Müller and Sennrich (2021) Mathias Müller and Rico Sennrich. 2021.  Understanding the properties of minimum Bayes risk decoding in neural machine translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 259–272, Online. Association for Computational Linguistics.
  * Naber (2003) Daniel Naber. 2003.  A rule-based style and grammar checker.
  * Ng et al. (2014) Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014.  The CoNLL-2014 shared task on grammatical error correction.  In _Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task_ , pages 1–14, Baltimore, Maryland. Association for Computational Linguistics.
  * Omelianchuk et al. (2020) Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem N. Chernodub, and Oleksandr Skurzhanskyi. 2020.  Gector - grammatical error correction: Tag, not rewrite.  _CoRR_ , abs/2005.12592.
  * OpenCLC (2019) OpenCLC. 2019.  Open cambridge learner english corpus.
  * Raina et al. (2020) Vyas Raina, Mark J.F. Gales, and Kate M. Knill. 2020.  Universal adversarial attacks on spoken language assessment systems.  In _Interspeech 2020_. ISCA.
  * Rosti et al. (2007a) Antti-Veikko Rosti, Necip Fazil Ayan, Bing Xiang, Spyros Matsoukas, Richard Schwartz, and Bonnie Dorr. 2007a.  Combining outputs from multiple machine translation systems.  In _Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference_ , pages 228–235, Rochester, New York. Association for Computational Linguistics.
  * Rosti et al. (2007b) Antti-Veikko Rosti, Spyros Matsoukas, and Richard Schwartz. 2007b.  Improved word-level system combination for machine translation.  In _Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics_ , pages 312–319, Prague, Czech Republic. Association for Computational Linguistics.
  * Sennrich et al. (2015) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015.  Improving neural machine translation models with monolingual data.  _arXiv preprint arXiv:1511.06709_.
  * Sim et al. (2007) K. C. Sim, W. J. Byrne, M. J. F. Gales, H. Sahbi, and P. C. Woodland. 2007.  Consensus network decoding for statistical machine translation system combination.  In _2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP ’07_ , volume 4, pages IV–105–IV–108.
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_.
  * Stahlberg and Kumar (2020) Felix Stahlberg and Shankar Kumar. 2020.  Seq2Edits: Sequence transduction using span-level edit operations.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5147–5159, Online. Association for Computational Linguistics.
  * Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015.  Going deeper with convolutions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1–9.
  * Tarnavskyi et al. (2022) Maksym Tarnavskyi, Artem Chernodub, and Kostiantyn Omelianchuk. 2022.  Ensembling and knowledge distilling of large sequence taggers for grammatical error correction.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3842–3852, Dublin, Ireland. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Wolpert (1992) David H Wolpert. 1992.  Stacked generalization.  _Neural networks_ , 5(2):241–259.
  * Wu et al. (2023) Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael Lyu. 2023.  Chatgpt or grammarly? evaluating chatgpt on grammatical error correction benchmark.
  * Yannakoudakis et al. (2011) Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011.  A new dataset and method for automatically grading ESOL texts.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 180–189, Portland, Oregon, USA. Association for Computational Linguistics.
  * Zhang et al. (2022) Yidan Zhang, Yu Wan, Dayiheng Liu, Baosong Yang, and Zhenan He. 2022.  Rmbr: A regularized minimum bayes risk reranking framework for machine translation.

##  Appendix A Jaccard Similarity as MBR Reward

Section 3.4 proposes three different reward functions, $\mathcal{R}$ to guide
the MBR decoding process (Equation 1) to better align with desired assessment
criteria. Here, we consider the Jaccard Similarity as an alternative reward
function that can combine precision and recall properties,

$\mathcal{R}^{\text{(jac)}}(\tilde{\mathbf{e}},\mathbf{e})=\frac{|\tilde{\mathbf{e}}\cap\mathbf{e}|}{|\tilde{\mathbf{e}}\cup\mathbf{e}|}.$
(6)

Table 6 gives the performance of systems combined with MBR decoding using the
Jaccard reward function.

$\mathcal{Y}^{(\text{s})}$ | conll | bea | fce
---|---|---|---
$\\{b,r,x\\}$ | 59.22$\left(\begin{subarray}{c}65.39\\\ 43.01\end{subarray}\right)$ | 69.15$\left(\begin{subarray}{c}71.55\\\ 60.96\end{subarray}\right)$ | 51.56$\left(\begin{subarray}{c}55.71\\\ 39.73\end{subarray}\right)$
$\\{b,r,x,\mathbf{y}^{(1)},$ $\mathbf{y}^{(2)},\mathbf{y}^{(3)}\\}$ | 59.21$\left(\begin{subarray}{c}65.50\\\ 42.76\end{subarray}\right)$ | 69.26$\left(\begin{subarray}{c}71.87\\\ 60.48\end{subarray}\right)$ | 51.63$\left(\begin{subarray}{c}55.96\\\ 39.45\end{subarray}\right)$
Greedy | 60.94$\left(\begin{subarray}{c}69.84\\\ 40.37\end{subarray}\right)$ | 72.37$\left(\begin{subarray}{c}77.29\\\ 57.68\end{subarray}\right)$ | 52.92$\left(\begin{subarray}{c}59.09\\\ 37.33\end{subarray}\right)$
Table 6: GEC system performance with Jaccard Similarity reward function for
MBR decoding. In all settings,
$\mathcal{Y}^{(\text{s})}=\mathcal{Y}^{(\text{c})}=\\{b,r,x\\}$

Comparing to results in the main experiments (Section 4.2), it can be seen
that using the Jaccard similarity reward gives similar behaviour but slightly
worse performance than the F0.5 reward function used for MBR decoding. This is
perhaps expected because both metrics encourage good precision and recall, but
the final GEC systems are assessed using the F0.5 score. Hence, the Jaccard
similarity reward offers a worse approximation to the final assessment metric
than an explicit F0.5 reward in MBR decoding.

##  Appendix B Greedy MBR Decoding Selection Space

Section 3.3 describes an approach where MBR decoding can be used to greedily
search over an edit space between the intersection edit set,
$\mathbf{e}^{(3)}$ and the union edit set, $\mathbf{e}^{(1)}$ to find a
combined edit set that as per the expected reward in the MBR algorithm should
give better performance. Results in the main paper in Table 5 search the edit
space between the intersection set, $\mathbf{e}^{(3)}$ and $\mathbf{e}^{(2)}$.
Table 7 shows that it is sensible to not continue searching for all edits in
the union set, $\mathbf{e}^{(1)}$, as searching the entire space compromises
performance. This is perhaps due to the increased noise added into the system
by potentially including spurious edits from the union set.

Reward | conll | bea | fce
---|---|---|---
$\mathcal{R}^{\text{(rec)}}$ | 53.22$\left(\begin{subarray}{c}56.48\\\ 43.24\end{subarray}\right)$ | 63.82$\left(\begin{subarray}{c}65.34\\\ 58.38\end{subarray}\right)$ | 47.73$\left(\begin{subarray}{c}50.77\\\ 38.52\end{subarray}\right)$
$\mathcal{R}^{\text{(prec)}}$ | 59.03$\left(\begin{subarray}{c}76.58\\\ 30.79\end{subarray}\right)$ | 72.58$\left(\begin{subarray}{c}84.27\\\ 46.67\end{subarray}\right)$ | 52.56$\left(\begin{subarray}{c}67.71\\\ 27.73\end{subarray}\right)$
$\mathcal{R}^{\text{(f05)}}$ | 58.42$\left(\begin{subarray}{c}67.33\\\ 38.19\end{subarray}\right)$ | 69.66$\left(\begin{subarray}{c}75.05\\\ 54.11\end{subarray}\right)$ | 50.42$\left(\begin{subarray}{c}57.33\\\ 34.02\end{subarray}\right)$
Table 7: Greedy MBR decoding performance with edit search from intersection
edit set, $\mathbf{e}^{(3)}$ to the union edit set, $\mathbf{e}^{(1)}$. By
considering all possible edits in the union set, we can reduce performance.
Hence in the main paper we limit edits to be between $\mathbf{e}^{(2)}$ and
$\mathbf{e}^{(3)}$.

##  Appendix C Alternative Expected Reward Set, $\mathcal{Y}^{(\text{c})}$

Equation 1 for MBR decoding can be simplified to equation 2, where we make the
assumption that every sequence $\mathbf{y}\in\mathcal{Y}^{(\text{c})}$ used to
calculate the expected reward is equiprobable (i.e. the posterior distribution
is the same). We justify this assumption in the main paper by considering only
similarly performing systems to form the set of sequences over which the
expected reward is calculated: $\mathcal{Y}^{(\text{c})}=\\{b,r,x\\}$. It is
interesting consider a situation where we violate/test this equiprobable
posterior assumption by considering different possible sequence sets for
$\mathcal{Y}^{(\text{c})}$ and observing the impact on performance after MBR
decoding system combination. Table 8 reports the performance of MBR decoding
with different output sequence sets, $\mathcal{Y}^{(\text{c})}$ used to
calculate the expected reward. In comparison to the equivalent results in the
main paper in Table 2, it is evident that a deviation from
$\mathcal{Y}^{(\text{c})}=\\{b,r,x\\}$ does not compromises performance. This
demonstrates that it is possible to diverge from the similar performing system
constraint to validate the equiprobable posterior assumption to generate good
combined systems using MBR decoding.

Reward | conll | bea | fce
---|---|---|---
$\mathcal{R}^{\text{(prec)}}$ | 59.83$\left(\begin{subarray}{c}69.43\\\ 38.52\end{subarray}\right)$ | 70.81$\left(\begin{subarray}{c}75.81\\\ 56.02\end{subarray}\right)$ | 52.40$\left(\begin{subarray}{c}60.15\\\ 34.57\end{subarray}\right)$
$\mathcal{R}^{\text{(f05)}}$ | 59.96$\left(\begin{subarray}{c}67.76\\\ 41.07\end{subarray}\right)$ | 70.44$\left(\begin{subarray}{c}74.21\\\ 58.55\end{subarray}\right)$ | 52.09$\left(\begin{subarray}{c}57.89\\\ 37.19\end{subarray}\right)$
Table 8: Impact of changing the set of sequences, $\mathcal{Y}^{(\text{c})}$
used to calculate the expected reward when using MBR decoding for system
combination. We let
$\mathcal{Y}^{(\text{c})}=\\{b,r,x,\mathbf{y}^{(1)},\mathbf{y}^{(2)},\mathbf{y}^{(3)}\\}$.
In all settings we maintain the same selection set,
$\mathcal{Y}^{(\text{s})}=\\{b,r,x\\}$.