# JuriBERT: A Masked-Language Model Adaptation for French Legal Text

Stella Douka
École Polytechnique
AUEB
Hadi Abdine
École Polytechnique
Michalis Vazirgiannis
École Polytechnique
AUEB
Rajaa El Hamdani
HEC Paris
David Restrepo Amariles
HEC Paris

###### Abstract

Language models have proven to be very useful when adapted to specific
domains. Nonetheless, little research has been done on the adaptation of
domain-specific BERT models in the French language. In this paper, we focus on
creating a language model adapted to French legal text with the goal of
helping law professionals. We conclude that some specific tasks do not benefit
from generic language models pre-trained on large amounts of data. We explore
the use of smaller architectures in domain-specific sub-languages and their
benefits for French legal text. We prove that domain-specific pre-trained
models can perform better than their equivalent generalised ones in the legal
domain. Finally, we release JuriBERT, a new set of BERT models adapted to the
French legal domain.

##  1 Introduction

Domain-specific language models have evolved the way we learn and use text
representations in natural language processing. Instead of using general
purpose pre-trained models that are highly skewed towards generic language, we
can now pre-train models that better meet our needs and are highly adapted to
specific domains, like medicine and law. In order to achieve that, models are
trained on large scale raw text data, which is a computationally expensive
step, and then are used in many downstream evaluation tasks, achieving state-
of-the-art results in multiple explored domains.

The majority of domain-specific language models so far are applied to the
English language. Abdine et al. (2021) published French word vectors from
large scale generic web content that surpassed previous non pre-trained word
embeddings. Furthermore, Martin et al. (2020) introduced CamemBERT, a
monolingual language model for French, that is used for generic everyday text,
and proved its superiority in comparison with other multilingual models. In
the meantime, domain-specific language models for French are in lack. There is
an even greater shortage when it comes to the legal field. Sulea et al. (2017)
mentioned the importance of using state-of-the-art technologies to support law
professionals and provide them with guidance and orientation. Given this need,
we introduce JuriBERT, a new set of BERT models pre-trained on French legal
text. We explore the use of smaller models architecturally when we are dealing
with very specific sub-languages, like French legal text. Thus, we publicly
release JuriBERT111You can find the models in
http://master2-bigdata.polytechnique.fr/FrenchLinguisticResources/resources#juribert
in 4 different sizes online.

##  2 Related Work

Previous work on domain-specific text data has indicated the importance of
creating domain-specific language models. These models are either adaptations
of existing generalised models, for example Bert Base by Devlin et al. (2019)
trained on general purpose English corpora, or pre-trained from scratch on new
data. In both cases, domain-specific text corpora are used to adjust the model
to the peculiarities of each domain.

A remarkable example of adapting language models is the research done by Lee
et al. (2019) who introduced BioBERT, a domain-specific language
representation model pre-trained on large scale biomedical corpora. BioBERT
outperformed BERT and other previous models on many biomedical text mining
tasks and showed that pre-training on specific biomedical corpora improves
performance in the field. Similar results were presented by Beltagy et al.
(2019) that introduced SciBERT and showed that pre-training on scientific-
related corpus improves performance in multiple domains, and by Yang et al.
(2020) who showed that FinBERT, pre-trained on financial communication
corpora, can outperform BERT on three financial sentiment classification
tasks.

Moving on to the legal domain, Bambroo and Awasthi (2021) worked on LegalDB, a
DistilBERT model Sanh et al. (2019) pre-trained on English legal-domain
specific corpora. LegalDB outperformed BERT at legal document classification.
Elwany et al. (2019) also proved that pre-training BERT can improve
classification tasks in the legal domain and showed that acquiring large scale
English legal corpora can provide a major advantage in legal-related tasks
such as contract classification. Furthermore, Chalkidis et al. (2020)
introduced LegalBERT, a family of English BERT models, that outperformed BERT
on a variety of datasets in text classification and sequence tagging. Their
work also showed that an architecturally large model may not be necessary when
dealing with domain-specific sub-languages. A representative example is Legal-
BERT-Small that is highly competitive with larger versions of LegalBert. We
intent to further explore this theory with even smaller models.

Despite the increasing use of domain-specific models, we have mainly been
limited to the English language. On the contrary, in the French language,
little work has been done on the application of text classification methods to
support law professionals, with the exception of Sulea et al. (2017) that
managed to achieve state-of-the-art results in three legal-domain
classification tasks. It is also worth mentioning Garneau et al. (2021) who
introduced CriminelBART, a fine-tuned version of BARThez Eddine et al. (2020).
CriminelBART is specialised in criminal law by using French Canadian legal
judgments. All in all, no previous work has adapted a BERT model in the legal
domain using French legal text.

##  3 Downstream Evaluation Tasks

In order to evaluate our models we will be using two legal text classification
tasks provided by the Court of Cassation, the highest court of the French
judicial order.

The subject of the first task is assigning the Court’s Claimant’s pleadings,
"mèmoires ampliatifs" in French, to a chamber and a section of the Court. This
leads to a multi-class classification task with 8 different imbalanced
classes. In Table 1 we can see the eight classes that correspond to the
different chambers and sections of the Court, as well as their support in the
data. The classes represent 4 chambers: the first civil chamber (C1) that
deals with topics like Civil Contract Law and Consumer Law, the second civil
chamber (C2) with topics like Insurance Law and Traffic accidents, the third
civil chamber (C3) dealing with Real property and Construction Law among other
topics and the Commercial, Economic and Financial Chamber (CO) for Commercial
Law, Banking and Credit Law and others. Each chamber has two or more sections
dealing with different topics.

The second task is to classify the Claiment’s pleadings to a set of 151
subjects, "matières" as stated in French. Figure 5 in appendix shows the
support of the matières in the data. As we can see in Figure 1 the 10
recessive matières have between 7 to 1 examples in our dataset. We decided to
remove the last 3 matières as they have less than 3 examples and therefore it
is not possible to split them in train, test and development sets.

Class | Support
---|---
CO | 28 198
C1_Section1 | 14 650
C1_Section2 | 16 730
C2_Section1 | 11 525
C2_Section2 | 9 975
C2_Section3 | 13 736
C3_Section1 | 16 176
C3_Section2 | 12 282
Table 1: Chambers and Sections of the Court of Cassation and data support
Figure 1: 10 Recessive Matieres

##  4 JuriBERT

We introduce a new set of BERT models pre-trained from scratch in legal-domain
specific corpora. We train our models on the Masked Language Modeling (MLM)
task. This means that given an input text sequence we mask tokens with 15%
probability and the model is then trained to predict these masked tokens. We
follow the example of Chalkidis et al. (2020) and choose to train
significantly even smaller models, including Bert-Tiny and Bert-Mini. The
architectural details of the models we pre-trained are presented in Table 2.
We also choose to further pre-train CamemBERT Base on French legal text in
order to better explore the impact of using domain-specific corpora in pre-
training.

### Training Data

For the pre-training we used two different French legal text datasets. The
first dataset contains data crawled222We used Heritrix, a crawler that
respects the robots.txt exclusion directives and META nofollow tags. See
https://github.com/internetarchive/heritrix3 from the
Légifrance333https://www.legifrance.gouv.fr/ website and consists of raw
French Legal text. The Légifrance text is then cleaned from non French
characters. We also use the Court’s decisions and the Claimant’s pleadings
from the Court of Cassation that consists of 123361 long documents from
different court cases. All personal and private information, including names
and organizations, has been removed from the documents for the privacy of the
stakeholders. The combined datasets provide us with a collection of raw French
legal text of size 6.3 GB that we will use to pre-train our models.

### Legal Tokenizer

In order to pre-train a new BERT model from scratch we need a new Tokenizer.
We trained a ByteLevelBPE Tokenizer with newly created vocabulary from the
training corpus. The vocabulary is restricted to 32,000 tokens in order to be
comparable to the CamemBERT model from Martin et al. (2020) and minimum token
frequency of 2. We used a RobertaTokenizer as a template to include all the
necessary special tokens for a Masked Language Model. Our new Legal Tokenizer
encodes the data using 512-sized embeddings.

### JuriBERT

For the pre-training of the JuriBERT Model we used both the crawled Légifrance
data and the Pleadings Dataset, thus creating a 6.3GB collection of legal
texts. The encoded corpus was then used to pre-train a BERT model from
scratch. Our model was pre-trained in 4 different architectures. As a result
we have JuriBERT Tiny with 2 layers, 128 hidden units and 2 attention heads
(6M parameters), JuriBERT Mini with 4 layers, 256 hidden units and 4 attention
heads (15M parameters), JuriBERT Small with 6 layers, 512 hidden units and 8
attention heads (42M parameters) and JuriBERT Base with 12 layers, 768 hidden
units and 12 attention heads (110M parameters). JuriBERT Base uses the exact
same architecture as CamemBERT Base.

### Task Specific JuriBERT

We also pre-trained a task-specific model that is expected to perform better
in the classification of the Claiment’s pleadings. For that we used only the
Pleadings Dataset for the pre-training that is a 4GB collection of legal
documents. The task-specific JuriBERT model for the Cour de Cassation task was
pre-trained in 2 architectures, JuriBERT Tiny (L=2, H=128, A=2) and JuriBERT
Mini (L=4, H=256, A=4).

### JuriBERT-FP

Apart from pre-training from scratch we decided to also further pre-train
CamemBERT Base on the training data. Our goal is to compare its performance
with the original JuriBERT model to further explore the impact of using
specific-domain corpora during pre-training. JuriBERT-FP uses the same
architecture as CamemBERT Base and JuriBERT Base.

##  5 Methods

#### Pre-training Details

All the models were pre-trained for 1M steps. A learning rate of $1e-4$ was
used along with an Adam optimizer ($\beta_{1}$=0.9, $\beta_{2}$=0.999) with
weight decay of 0.1 and a linear scheduler with 10,000 warm-up steps. All the
models were pre-trained with batch size of 8 expect for JuriBERT Base and
JuriBERT-FP that used batches of size 4. For the pre-training we used an
Nvidia GTX 1080Ti GPU.

Model | Architecture | Params
---|---|---
JuriBERT Tiny | L=2, H=128, A=2 | 6M
JuriBERT Mini | L=4, H=256, A=4 | 15M
JuriBERT Small | L=6, H=512, A=8 | 42M
JuriBERT Base | L=12, H=768, A=12 | 110M
JuriBERT-FP | L=12, H=768, A=12 | 110M
Table 2: Architectural comparison of JuriBERT models Model | Pre-training Corpora
---|---
CamemBERT | 138GB
BARThez | 66GB
JuriBERT | 6.3GB
JuriBERT-FP | 6.3GB
Task JuriBERT | 4GB
Table 3: Size of pre-training corpora used by different models

#### Fine-tuning Details

Our models were fine-tuned on the downstream evaluation task using the same
classification head as Devlin et al. (2019) that consists of a Dense layer
with tanh function followed by a Dense layer with softmax activation function
and Dropout layers with fixed dropout rate of 0.1. We applied grid-search to
the learning rate on a range of {$2e-5,3e-5,4e-5,5e-5$}. We used an Adam
optimizer along with a linear scheduler that provided the training with 100k
warm-up steps. We train for a maximum of 30 epochs with patience of 2 epochs
on the early stopping callback and checkpoints for the best model. For the
classification we use only the paragraphs starting with ’ALORS QUE’ from the
Pleadings Dataset, as they include all the important information for the
correct chamber and section. This was suggested by a lawyer from the Court of
Cassation as the average size of a mèmoire ampliatif is extremely big, from 10
to 30 pages long. By using the ’ALORS QUE’ paragraphs we have text sequences
with average size of 800 tokens. For the chambers and sections classification
task we split the data in 14% development and 16% test data. For the matières
classification we split the data in 17% development and 14% test data and
stratify in order to have all classes represented in each subset. Both tasks
use a fixed batch size of 4. For the fine-tuning we used an Nvidia GTX 1080Ti
GPU.

##  6 Results

The results on the downstream evaluation tasks are presented in Tables 4 and
5. We compare our models with two CamemBERT versions, Base and Large, and with
BARThez, a sequence-to-sequence model dedicated to the French language.
CamemBERT has been pre-trained on 138GB of French raw text from the OSCAR
corpus. Despite the difference in pre-training corpora size, with our model
using only 6.3GB of legal text, JuriBERT Small managed to outperform both
CamemBERT Base and CamemBERT Large. This further proves the importance of
domain-specific language models in natural language processing and transfer
learning. Despite our expectations, the performance of JuriBERT Base does not
exceed the performance of its smaller equivalent models. We attribute this
peculiarity in the usage of smaller batch sizes when pre-training JuriBERT
Base and also the fact that larger models usually need more computational
resources and more time and data in order to converge.

JuriBERT Small also outperforms BARThez on the chambers and sections
evaluation task, which is pre-trained on 66GB of French raw text and usually
used for generative tasks. On the matières classification task BARThez is the
dominant model with JuriBERT Small being second. We infer that the complexity
of the second task benefits more from the robustness and size of BARThez than
from the specific-domain nature of JuriBERT.

Comparing our models with the same architectures, it becomes apparent that all
task-specific JuriBERT models perform better than their equivalent domain-
specific JuriBERT models besides using only 4GB of pre-training data. The
results confirm that a BERT model pre-trained from scratch only on the corpus
that is then used for fine-tuning can perform better than a domain-specific
one on the same task as we expected.

JuriBERT-FP outperforms JuriBERT Base and achieves similar results with
CamemBERT Base on the chambers and sections classification task. This shows
that further pre-training a general purpose language model can have better
results than training from scratch. However, it did not manage to outperform
JuriBERT Small in both tasks, which can be attributed to the smaller batch
size used during pre-training and to the size of the model as mentioned before
for JuriBERT Base. Unfortunately, there are no smaller versions of CamemBERT
available to further test this theory. On the matières classification task,
JuriBERT-FP still outperforms JuriBERT Base. On the contrary, it performs
worse than CamemBERT Base. Along with the state-of-the-art results of BARThez,
this leads us to believe that in order to achieve better results in more
complex tasks JuriBERT models require more pre-training corpora.

All in all, JuriBERT Small achieves equivalent results with previous larger
generic language models with an accuracy of 83.95% on the first task and
71.80% on the second task on the test data. JuriBERT Small, JuriBERT Mini and
even JuriBERT Tiny all outperform JuriBERT Base, proving that smaller models
architecturally can achieve comparable, if not better, results when we are
training on very domain-specific data. A larger model, not only requires more
resources to be trained, but is also not as efficient as its smaller
equivalents. This is of major importance for researchers with limited
resources available. Furthermore, JuriBERT-FP achieves better results than
JuriBERT Base in both tasks. This leads us to infer that pre-training from an
existing language model can be a major advantage, as opposed to randomly
initialising our weights.

Model | Lrate | Dev | Test
---|---|---|---
CamemBERT Base | $2e-5$ | 82.75 | 83.22
CamemBERT Large | $2e-5$ | 79.69 | 79.91
BARThez | $3e-5$ | 83.70 | 83.49
JuriBERT Tiny | $3e-5$ | 82.00 | 81.58
JuriBERT Mini | $3e-5$ | 83.08 | 82.62
JuriBERT Small | $3e-5$ | 83.86 | 83.95
JuriBERT Base | $3e-5$ | 82.26 | 82.51
Task JuriBERT Tiny | $4e-5$ | 81.91 | 81.59
Task JuriBERT Mini | $4e-5$ | 82.75 | 82.66
JuriBERT-FP | $2e-5$ | 83.07 | 83.28
Table 4: Accuracy of models on the chambers and sections classification task Model | Lrate | Dev | Test
---|---|---|---
CamemBERT Base | $3e-5$ | 71.64 | 71.66
BARThez | $2e-5$ | 72.17 | 72.09
JuriBERT Small | $2e-5$ | 71.67 | 71.80
JuriBERT Base | $3e-5$ | 70.28 | 70.38
JuriBERT-FP | $2e-5$ | 70.99 | 71.21
Table 5: Accuracy of models on the matieres classification task

##  7 Limitations

As we mentioned before both JuriBERT Base and JuriBERT-FP have been pre-
trained using smaller batch sizes than the other models due to limited
resources. We acknowledge that this may have affected their performance
compared to the other models. However, we believe that their lower performance
can also be attributed to their size as larger models are computationally
heavier and thus require more resources to converge.

Acquiring large scale legal corpora, especially for a language other than
English, has proven to be challenging due to their confidential nature. For
this reason, JuriBERT models were fine-tuned on two downstream evaluation
tasks that contain data from the pre-training dataset collection. Further
testing shall be required in order to validate the performance of our models
on different tasks.

The differences in performance between the generic language models and the
newly created JuriBERT models are very small. More specifically, only JuriBERT
Small manages to outperform CamemBERT Base and Barthez with a difference in
accuracy of 0.73%. We attribute this limitation in the use of much less pre-
training data. However we emphasize that JuriBERT manages to achieve similar
results despite the difference in pre-training corpora size. Thus, we expect
JuriBERT to achieve better results in the future provided that we further pre-
train with more data.

##  8 Conclusions and Future Work

We introduce a new set of domain-specific Bert Models pre-trained from scratch
on French legal text. We conclude that our task is very specific and as a
result it does not benefit from general purpose models like CamemBERT. We also
show the superiority of much smaller models when training on very specific
sub-languages like legal text. It becomes apparent that large architectures
may in fact not be necessary when the targeted sub-language is very specific.
This is important for researchers with lower resources available, as smaller
models are fine-tuned a lot faster on the downstream tasks. Furthermore, we
show that a BERT model pre-trained from scratch on task-specific data and then
fine-tuned on this very task can perform better than a domain-specific model
that has been pre-trained on a lot more data. We point out of course that a
domain-specific model can outperform a task-specific one on other tasks and is
generally preferred when we need a multi-purpose BERT model with many
applications in the French legal domain. In future work, we plan to further
explore the potential of JuriBERT in other tasks and as a result prove its
superiority over the task-specific one.

## Acknowledgements

This research was supported by the ANR chair AML/HELAS (ANR-CHIA-0020-01).

## References

  * Abdine et al. (2021) Hadi Abdine, Christos Xypolopoulos, Moussa Kamal Eddine, and Michalis Vazirgiannis. 2021.  Evaluation of word embeddings from large-scale french web content.  _CoRR_ , abs/2105.01990.
  * Bambroo and Awasthi (2021) Purbid Bambroo and Aditi Awasthi. 2021.  LegalDB: Long DistilBERT for legal document classification.  In _2021 International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies (ICAECT)_ , pages 1–4.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A pretrained language model for scientific text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3615–3620, Hong Kong, China. Association for Computational Linguistics.
  * Chalkidis et al. (2020) Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020.  LEGAL-BERT: The muppets straight out of law school.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2898–2904, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Eddine et al. (2020) Moussa Kamal Eddine, Antoine J.-P. Tixier, and Michalis Vazirgiannis. 2020.  BARThez: a skilled pretrained French sequence-to-sequence model.  _CoRR_ , abs/2010.12321.
  * Elwany et al. (2019) Emad Elwany, Dave Moore, and Gaurav Oberoi. 2019.  BERT goes to law school: Quantifying the competitive advantage of access to large legal corpora in contract understanding.  _CoRR_ , abs/1911.00473.
  * Garneau et al. (2021) Nicolas Garneau, Eve Gaumond, Luc Lamontagne, and Pierre-Luc Déziel. 2021.  CriminelBart: A French Canadian legal language model specialized in criminal law.  In _Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law_ , ICAIL ’21, page 256–257, New York, NY, USA. Association for Computing Machinery.
  * Lee et al. (2019) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _CoRR_ , abs/1901.08746.
  * Martin et al. (2020) Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020.  CamemBERT: a tasty French language model.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7203–7219, Online. Association for Computational Linguistics.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _CoRR_ , abs/1910.01108.
  * Sulea et al. (2017) Octavia-Maria Sulea, Marcos Zampieri, Shervin Malmasi, Mihaela Vela, Liviu P. Dinu, and Josef van Genabith. 2017.  Exploring the use of text classification in the legal domain.  _CoRR_ , abs/1710.09306.
  * Yang et al. (2020) Yi Yang, Mark Christopher Siy Uy, and Allen Huang. 2020.  Finbert: A pretrained language model for financial communications.  _CoRR_ , abs/2006.08097.

##  Appendix A Appendix

Figure 2: Accuracy, Precision, Recall and F1-Score of JuriBERT Small on the
chambers and sections classification task on the test dataset. The graph
contains all 8 classes. Figure 3: Confusion Matrix of JuriBERT Small on the
chambers and sections classification task on the test dataset. The graph
includes accuracy and error rate for each class. Figure 4: Sample of Accuracy,
Precision, Recall and F1-Score of JuriBERT Small on the matieres
classification task on the test dataset. The graph contains 28 classes and the
overall accuracy of all 148 classes. Figure 5: Support of the 151 Matieres in
the Court of Cassation data.


# A study of CNN capacity applied to Left Ventricle segmentation in cardiac
MRI††thanks: This preprint has not undergone peer review (when applicable) or
any post-submission improvements or corrections. The Version of Record of this
article is published in SN Computer Science, and is available online at
https://doi.org/10.1007/s42979-021-00897-x.

Marcelo Arruda,  Daniel Lima,  José Krieger,  Marco Gutierrez
Biomedical Informatics Lab
Heart Institute (InCor/HC.FMUSP)
Avenida Doutor Eneas Carvalho de Aguiar 44, Cerqueira César, 05403-904, Sao
Paulo, SP, BR Corresponding author email<EMAIL_ADDRESS>
(May 10, 2021)

###### Abstract

CNN (Convolutional Neural Network) models have been successfully used for
segmentation of the left ventricle (LV) in cardiac MRI (Magnetic Resonance
Imaging), providing clinical measurements. In practice, two questions arise
with deployment of CNNs: 1) when is it better to use a shallow model instead
of a deeper one? 2) how the size of a dataset might change the network
performance? We propose a framework to answer them, by experimenting with deep
and shallow versions of three U-Net families, trained from scratch in six
subsets varying from 100 to 10,000 images, different network sizes, learning
rates and regularization values. 1620 models were evaluated using 5-fold
cross-validation by loss and DICE. The results indicate that: sample size
affects performance more than architecture or hyper-parameters; in small
samples the performance is more sensitive to hyper-parameters than
architecture; the performance difference between shallow and deeper networks
is not the same across families.

_Keywords_ deep learning $\cdot$ model selection $\cdot$ medical imaging
$\cdot$ magnetic resonance imaging

##  1 Introduction

The use of deep learning has been growing and becoming very relevant for some
types of medical image analysis. Deep learning includes CNN (Convolutional
Neural Network) models, which have been successfully used for segmentation of
the left ventricle (LV) in cardiac MRI (Magnetic Resonance Imaging) by Chen et
al. (2020). The myocardium segmentation (Figure 1) is the basis for other
steps that provide measurements of ventricular function and myocardial
viability (such as ejection fraction and ventricular mass), and is used for
diagnosis of heart diseases, e.g. coronary artery disease Aziz et al. (2013).
One of the most successful CNN architectures for medical image segmentation is
the U-Net Ronneberger et al. (2015). It has been successfully applied in
several medical imaging analyses such as pancreas detection Oktay et al.
(2018) and prostate segmentation Ghavami et al. (2019). Figure 1 shows an
example of LV segmentation in cardiac MRI.

Figure 1: Cardiac MRI images are acquired in two different axis: short-axis
(a-c) and long-axis (d). In (a) we see a cardiac short-axis MRI of heart,
centred in the heart, and cropped to frame the right and left ventricles; in
(b) the left ventricle myocardium is highlighted, and (c) only shows the left
ventricle myocardium segmentation mask. Images (a-c) are reproduced from the
dataset analysed in Nikolov et al. (2018) and (d) is reproduced from the Atlas
of Human Cardiac Anatomy website Spencer et al. (2013).

One of the limitations of deep learning approaches in many medical
applications is the need for large labeled datasets, which are not always
available. Creating labelled datasets requires clinical specialists for visual
inspection and manual marking of images, which is costly and takes significant
time. By considering this limitation, the U-Net architecture was designed for
fast convergence and sufficiently accurate segmentation in relatively small
datasets Oktay et al. (2018). U-Net is a CNN architecture with many possible
variations such as different neural blocks, block width and network depth, so
it is possible to balance the performance according to the images under study.

However, with the deployment of CNNs in medical practice, another two
questions arise: 1) when is it better to use a small and shallow model instead
of a larger and deeper one? 2) how the number of samples in a dataset might
change the network performance? These questions motivate our work, which
provides a framework of analysing the CNN capacity and dataset sizes for
medical applications, and demonstrate its applicability to LV segmentation in
cardiac MRI.

### Our Contributions

In order to grasp how dataset size affects U-Net performance in segmentation,
this paper describe a framework where we train 18 different U-Nets of
different sizes (number of network parameters and depth) and families: VGG
Simonyan and Zisserman (2015), ResNet He et al. (2016) and EfficientNet Tan
and Le (2019). These U-Net variations were trained from scratch in six dataset
sizes (suppose patients submitted to MRI study), and also with variations of
hyper-parameters learning rate and L2 regularization. They were evaluated with
5-fold cross-validation by the loss and DICE index, and each fold was tested
in a separate holdout set. We then analyse the variables and results in a
factorial study using GAM (Generalized Additive Model), ANOVA (Analysis of
Variance) and HSD (Honest Significant Difference) tests. In summary, our
objectives in this paper are:

  * •

Observe the relationship between prediction error, dataset size and network
size;

  * •

Identify hyper-parameter effects on performance by linear models, ANOVA and
HSD;

  * •

Provide a framework to check when the choice for the network to be used in an
application might change due to the number of images available in a training
dataset.

This paper is organized as follows: Section 2 (Related work) reviews the base
CNN architectures for image segmentation and related studies of neural network
capacity; Section 3 (Materials and Methods) present the dataset preparation,
the neural networks, hyper-parameters, linear models, ANOVA and HSD tests;
Section 4 (Experiments) present the experiments, metrics, results and
discussion; and Section 5 (Conclusion) summarizes our goals, the proposed
framework, achievements and ideas for future investigations.

##  2 Related work

In this section we review related literature in the two main aspects of our
framework: biomedical image segmentation (2.1), and how to evaluate neural
network capacity (2.2).

###  2.1 Biomedical Segmentation with U-Net and its backbones

U-Net is an encoder-decoder CNN architecture for biomedical image
segmentation, which encodes the input image in a sequence of four down-scaling
layers (blocks), then up-scales the encoded tensors, concatenates them to the
previous input and combines the results in the reversed order, forming a
network that resembles an “U”. It is possible to change its backbone by
replacing the original convolution blocks for well-known blocks from other
CNNs, e.g. residual blocks Zhao et al. (2017), while keeping the U shape
encoder decoder structure. The U-Net is trained by minimizing the cross-
entropy between the class probabilities of each pixel in the output image and
the segmentation mask given as class labels. The success of U-Net led to the
proposal of backbone innovations, such as newer versions of the skip
connections Chaurasia and Culurciello (2017); Zhou et al. (2018). Other
encoder-decoder architectures have also been proposed, some keeping the
general U-net structure, and others with novel structures such as feature
pyramids Lin et al. (2017) and pooling pyramids Zhao et al. (2017). The
standard U-Net backbone is based on VGG Simonyan and Zisserman (2015), but we
extended our analysis by considering two more state-of-the-art backbones:
ResNet He et al. (2016) and EfficientNet Tan and Le (2019).

###  2.2 Evaluating Neural Network Capacity

Given that there are several U-Net architectures available, making an
appropriate choice is challenging and can impact the results of developing
applications. These reflect in two critical aspects of medical studies
planning: a) estimating the size of the sample needed for developing an
effective solution (both in terms of images and patients) and b) selecting
appropriate models, because different medical applications solve problems of
different complexities, which thus require different network models of
adequate capacity. However, there is no established method for calculating
network capacity nor a consolidated understanding of how it relates to CNN
depth and to the number of parameters Bartlett et al. (2017); Neyshabur et al.
(2019). Addressing these aspects in a clinical study is not trivial. While
bigger networks might be prone to over-fit in small datasets, smaller networks
might not have the required capacity for the challenge at hand. A further
complication, at least for LV segmentation, is the fact that the relationship
between U-Net, dataset sizes and generalization error is not well understood.
Recent literature Benkendorf and Hawkins (2020); Shahinfar et al. (2020) have
been investigating sample size issues, so in our approach we extend this
investigation to hyper-parameters and network architecture. Next we present
our methodology, then experiments, results and conclusion.

##  3 Materials and Methods

In this section we present the dataset characteristics, preparation and
splitting (Section 3.1, the U-Net CNN architecture and backbones based on VGG,
ResNet and EfficientNet (Section 3.2), hyper-parameter optimization (Section
3.3), and the linear models and statistical tests used in the analysis
(Section 3.4).

###  3.1 Dataset preparation

All of the datasets we created are sub-samples of the publicly available Left
Ventricle Segmentation Challenge (LVSC) dataset Nikolov et al. (2018). The
LVSC training set has cardiac cine-MRI pictures of 100 patients, with diverse
number of slices (8–24), slice thickness (6–8mm), slice gaps (2–4mm), number
of phases in the cardiac cycle (18–35), image sizes (138×192 to 512×512
pixels) and acquisition devices (GE, Phillips or Siemens MRI scanner systems
with 1.5 or 3.0T). All images were loaded into float32 arrays, had the pixel
values rescaled to [0,1], resized to 256×256 pixels with area and bi-cubic
interpolation respectively for downscaling and up-scaling, and zero padding to
maintain aspect ratio.

Due to anatomical and positional differences, patient id is a major source of
variation, so we sub-sampled 100 images from each patient. In order to
stratify samples in the whole cardiac volume, we selected an approximately
equal number of images from each slice, regularly spaced throughout the
cardiac cycle (images $\lfloor i\times phases\times
slices/100\rfloor~{}\mathrm{for}~{}i~{}\mathrm{in}~{}[1,100]$). This resulted
in 10,000 images (100 patients × 100 images per patient). Then, an 80-20%
holdout group split was performed for development (training + validation) and
test. Afterwards, five smaller datasets were sub-sampled from the bigger
dataset (10k), with 200, 500, 1000, 2500, and 5000 images. For each of those
datasets, the development set was sub-sampled from the initial development set
with 8000 images, while the test set was sub-sampled from the holdout test set
with 2000 images. Both the holdout and the sub-sampling were performed by
randomly selecting of patients, such that a given patient had either all of
her/his images in one set only. In each execution the development set is again
split in 5-folds of equal sizes for cross-validation: each fold is taken for
validation using a model trained on the remaining four folds. Next we present
the neural networks, hyper-parameters and linear models and statistical tests.

###  3.2 Neural Networks

U-Net Ronneberger et al. (2015) implements the downscaling path similarly to
the VGG network Simonyan and Zisserman (2015), using a sequence of 2D
convolutions, ReLU activations and max-poolings. We also tested U-Net with
modern backbones based on ResNet He et al. (2016) and EfficientNet Tan and Le
(2019). ResNets divide the pathway in an input identity map path (a short-cut
connection), and a residual path, that learns a residual function (which model
the residuals from the inputs). By learning the residuals instead of the main
signal, more convolution blocks can be stacked to build deeper neural networks
without compromising their training. EfficientNets are based on mobile
inverted bottleneck convolution blocksSandler et al. (2018) with
squeeze–excitation optimizationHu et al. (2018) and a loss function that
balances accuracy and speed, by adjusting network depth, width and resolution
hyper-parameters Tan and Le (2019); Tan et al. (2019).

To study network size we selected small and medium-large sizes for each
backbone: EfficientNets B0 and B5; ResNets 18 and 50, with approximately the
same numbers of parameters to the chosen EfficientNets; and VGG 16 and 19, the
two most common VGG sizes in literature. EfficientNet B0 has 5.3 million
parameters and achieved 76.3% top-one accuracy in the ImageNet classification
task, while B5 has 30 million parameters and achieved 83.3% accuracy in that
same task. This is a bigger gain for scaling up than for other families of
networks Tan and Le (2019). Another aspect that impacts performance and
convergence of neural networks is the choice of hyper-parameters, presented
next.

###  3.3 Hyper-parameter optimization

Given that training strategies might differ regarding dataset and network
size, we also chose to analyse some optimization parameters. However, as the
time to compute all hyper-parameter space is combinatorial, only some values
were selected. For each model and dataset combination, we tested different
initial learning rates ($10^{-2}$, $10^{-3}$ and $10^{-4}$) and different L2
regularization values ($10^{-2}$, $10^{-4}$ and $10^{-6}$). The initial
learning rate balances training speed and convergence stability. The
regularization has a further implication in model capacity, as it balances
accuracy and over-fitting by constricting network weights. In order to
understand the effect of hyper-parameters in combination to dataset and
network size, we analyse these variables with linear models, ANOVA and HSD
tests of next section.

###  3.4 Linear Models, ANOVA and HSD

To understand the effect of CNN choices on the performance, we develop a
linear regression model based on the dataset size, model architecture, model
size, and hyper-parameters as input variables. Linear regression models are
statistical approaches to analyse the dependence of a response variable $Y$ on
the linear combination of input variables $X=x_{1},x_{2},...,x_{n}$. These
models follow the general formula $Y=aX+b+\epsilon$, where $a$ corresponds to
weights for each variable $x_{i}$, and $b$ is a bias term (intercept), and
$\epsilon$ is the error of the model. The parameters $a$ and $b$ are commonly
adjusted by ordinary least squares regression (OLS). ANOVA for regression
models tests the statistical significance of input variables (groups), with
respect to the sums of squares (SS) of regressed variables and observed data.
The HSD test was developed by Tukey for pairwise comparison of multiple group
means, and is used as a post hoc adjustment Tabachnick and Fidell (2007).

Generalized linear models (GLM) extend linear models by applying a _link_
function that transforms the predicted distribution, e.g.
$\log(Y)=aX+b\rightarrow\hat{Y}=\exp(aX+b)$ in the case of _log link_.
Generalized additive models (GAM) further extends GLM by applying
transformations in each variable, e.g.
$g(E[Y])=f_{1}(x_{1})+f_{2}(x_{2})+...+f_{n}(x_{n})+b$, and discovering the
most appropriate functions $f_{i}$ for each input variable Hastie and
Tibshirani (1990). The linear models are evaluated by Akaike Information
Criterion (AIC), according to the maximum likelihood of estimates and number
of parameters of the model Akaike (1974). We tested several linear models and
settled our analysis with $DICE\sim
ls+ls:Family+\log(Dataset)+\log(lr):ls:Family+\log(reg)$, which is presented
in the next section along experiments, metrics, results and discussion.

##  4 Experiments

In this section we present the experimental configuration (4.1), evaluation
metrics (4.2), results (4.3) and discussion (4.4). In Section 4.3 (Results) we
present the experimental results, report the DICE index and BCE Loss of all
models, compare their differences, and analyse the model’s dependences on the
chosen parameters by linear regression models, ANOVA and HSD tests. In Section
4.4 (Discussion) we comment our findings in relation to literature and place
general considerations.

###  4.1 Experimental configuration

For each of the 324 different combinations of variables (model, dataset,
initial learning rate, L2 regularization), the model was trained from scratch
using 5-fold cross-validation. The development dataset was divided into 5
equally-sized subsets (folds) and each one is used for validation while the
remaining four are used to fit the model. For each fold, every combination of
parameters is trained for 50 epochs or less. We stopped training when there
was no improvement in the validation loss after 5 epochs (early stopping).
This results in 1620 training/evaluation runs, corresponding to 5 folds × 324
different conditions.

After all runs, we computed the metrics and fitted a linear regression model
to the experiment parameters and results, then performed statistical ANOVA and
HSD tests to assess significant differences. All experiments were performed in
a Foxconn HPC M100-NHI with eight NVIDIA V100-SXM2-16GB GPUs, using the
software we implemented with Python, PyDicom, Tensorflow, SegmentationModels,
Slurm and StatsModels. Next we present the evaluation metrics, then results
and discussion.

###  4.2 Evaluation metrics

To evaluate the quality of each model, we use two metrics: BCE (Binary Cross-
Entropy) loss and DICE index Dice (1945). These are computed on the validation
set of each training run, and also separately on the held-out test set from
the corresponding dataset. We then compare all configurations across datasets
and models, considering the best hyper-parameters for each configuration. BCE
and DICE formulas are given below:

$\displaystyle BCE$ $\displaystyle=-\frac{1}{N}\sum
y\cdot\log(\hat{y})+(1-y)\cdot\log(1-\hat{y})$ $\displaystyle DICE$
$\displaystyle=\frac{2TP}{2TP+FP+FN}=2\frac{\sum
y\cdot\lfloor\hat{y}\rceil}{\sum y+\lfloor\hat{y}\rceil}$

Where $TP,FP,FN$ are true positives, false positives and false negatives,
respectively, $N$ is the number of pixels in the dataset, $y$ are the binary
labels for all pixels, $\hat{y}$ are the corresponding predictions and
$\lfloor\cdot\rceil$ denotes rounding. BCE is a positive value which gives the
amount of entropy mismatch in the predictions, the lower the better. DICE
range from 0% (no correspondence) to 100% (images are perfectly equal) and is
equivalent to F1-score. In the next section we present the results of the
experiments, our analyses and discussion.

###  4.3 Results

All networks performed badly on the smallest dataset (due to insufficient
training cases) and improved gradually when fed with more data. The network
performed similarly well on the larger datasets (Figure 2). However, the
networks behaved very differently in the middle sizes (between 400 and 2000
samples), both in absolute performance and variability of the metrics. Loss
function behaved similarly, showing a similar improvement related to
increasing dataset size. Although, we opted to focus this report on DICE index
because it is easier to interpret, has close relation to the task, and is
common in literature.

Figure 2: Evaluation of models: the left panel shows DICE index (upper) and
BCE Loss (lower) on the test set for all 1,620 network models, in all datasets
and all folds. Each dataset (x-axis) contains six medians and IQRs for B0, B5,
R18, R50, V16 and V19, respectively, corresponding to EfficientNets B0 and B5,
ResNets 18 and 50 and VGGs 16 and 19. All networks performed worse in the
smaller datasets and similarly well on the larger datasets. We observe mixed
results with a gradual performance increase in the middle ground between 400
and 4,000 images. The right panel shows Mean DICE scores across datasets and
models. The error bars are 95% confidence intervals calculated using HSD test.
The results of all models are directly comparable, and overlapping intervals
should be interpreted as the corresponding null hypothesis (no statistical
difference). Figure 3: Predictions of DICE index for each model in each
dataset size given by the linear regression model (large X marks connected
with lines), and the observed values for the corresponding DICE data (small +,
x and arrow marks).

After all runs, we fitted a linear regression model (Table 1) to the
experiment parameters and results, then performed statistical ANOVA and HSD
tests to assess significant differences (Table 2 and Figure 2). The variables
$Family,ls,lr$ and $reg$ correspond to the network architecture family (VGG,
ResNet or EfficientNet), model version size (Long or Short, $ls$), initial
learning rate ($lr$) and L2 regularization parameters ($reg$), respectively.
The nesting of $Family$ within model version size ($ls/Family=ls+ls:Family$)
in the regression formula is a way to access a rough effect of model size
while not treating different architectures (long and short versions) as
equivalents.

We also fitted a linear model with a similar formula and using a log link
function instead of the log transformations on predictors: $\log(DICE)\sim
ls+ls:Family+Dataset+lr:ls:Family+reg$, but it had a bigger AIC value of -582.
According to the results, the dataset size had overall larger absolute
coefficients (Table 1) and more significant effect (Tables 1 and 2) in DICE
than architecture, initial learning rates or L2 regularization values in the
linear model. HSD test was applied to the mean DICE and confidence intervals
(Figure 2). EfficientNets had consistently higher DICE scores up to 2000
images. ResNets and VGGs required more data to achieve a comparable level of
performance, but slightly surpassed EfficientNets in the two largest datasets.
In the next section we discuss these results in relation to other evidence in
literature, and place general considerations.

Term | coef. | z | p-value
---|---|---|---
Intercept | -0.5810 | -178.383 | 0.0000
$ls[T.short]$ | 0.0016 | 0.039 | 0.9686
$ls[long]:Family[T.ResNet]$ | -0.0936 | -23.476 | 0.0189
$ls[short]:Family[T.ResNet]$ | -0.1107 | -27.771 | 0.0055
$ls[long]:Family[T.VGG]$ | -0.3903 | -97.863 | 0.0000
$ls[short]:Family[T.VGG]$ | -0.3237 | -81.173 | 0.0000
$\log(Dataset)$ | 0.1627 | 787.068 | 0.0000
$\log(lr):ls[long]:Family$ | 0.0158 | 40.202 | 0.0001
$\log(lr):ls[short]:Family[EfficientNet]$ | 0.0193 | 49.085 | 0.0000
$\log(lr):ls[long]:Family[ResNet]$ | 0.0130 | 32.955 | 0.0010
$\log(lr):ls[short]:Family[ResNet]$ | 0.0111 | 28.156 | 0.0049
$\log(lr):ls[long]:Family[VGG]$ | -0.0281 | -71.467 | 0.0000
$\log(lr):ls[short]:Family[VGG]$ | -0.0193 | -49.051 | 0.0000
$\log(reg)$ | -0.0003 | -0.383 | 0.7021

Model: $DICE\sim ls+ls:Family+log(Dataset)+log(lr):ls:Family+log(reg)$

Table 1: Coefficients and p-values for the linear model, with AIC: -2213. Term | SS | F | p-value | $\eta^{2}_{p}$
---|---|---|---|---
$ls$ | 0.0145 | 0.9802 | 0.3223 | 0.0001
$ls:Family$ | 1.7284 | 29.1844 |  <10-22 | 0.0145
$\log(Dataset)$ | 91.7177 | 6194.7601 |  <10-30 | 0.7693
$\log(lr):ls:Family$ | 1.9866 | 22.3631 |  <10-24 | 0.0167
$\log(reg)$ | 0.0022 | 0.1463 | 0.7021 | 0.0000
Residuals | 23.7779 | - | - | 0.1994
Table 2: ANOVA table for linear model variables and their interaction.
$\eta^{2}_{p}$ is the effect size, measuring the proportion of SS explained by
each term, while $Residuals$ measure the unknown SS. The terms $ls:Family$,
$\log(Dataset)$ and $\log(lr):ls:Family$ are responsible for 80% of DICE index
variance in the trained models. We observe that gross performance increment
comes from larger datasets, while learning rates and network size give small
but significant increments depending on the network architecture ($Family$).

###  4.4 Discussion

Apparently, smaller models (EfficientNet-B0, VGG16 and ResNet18) are not too
small for LV segmentation tasks, and were able to achieve good levels of
performance when given sufficient data. One could expect that different models
would reach different plateaus. But we observed in all families that, as the
training dataset size increases, the gain in DICE performance of deeper
networks vanishes as the performance of shallower models levels up.
Considering each network family separately, the network size effect is greater
for the EfficientNets than for the ResNets and VGGs (Table 1 coefficients). As
in the case of ImageNet classification, the gain of scaling up is bigger for
the EfficientNets than in other families. We noticed that in all dataset sizes
B5 performed significantly better than B0 (Figure 2), showing that it did not
over-fit more easily with the increased number of parameters, even considering
that B0 reached comparable performance with more data.

These results contradict the common sense that bigger models would suffer more
from over-fitting in smaller datasets, decreasing their performance, and that
they would gain more performance as more data was made available Martin and
Mahoney (2018). Our findings with EfficientNet support the idea of Neyshabur
et al. (2019) that bigger models does not necessarily over-fit, but this
pattern does not hold for other families. Considering the ResNets, the deeper
ResNet50 starts with a DICE performance below the shallow counterpart
(ResNet18), but improved faster and caught up with 4000 samples. We observed
that VGGs never differed significantly, probably because of their smaller
differences in depth and number of parameters, and they also have high
sensitivity to hyper-parameters. VGG16 had a particular tipping point from the
worst results in the 800 images dataset to scoring near the top contenders on
the next dataset with 2000 images.

A general consideration in the intermediate dataset sizes is that all models
were relatively more sensitive to hyper-parameters than to architecture, as
their performances had higher variance than when more data was made available.
This might affect sample size and model choice, as with 2000 images
EfficientNet B5 performed significantly better than the ones from other
families, but with 8000 images all models had no practical difference. Thus,
in scenarios where bigger datasets are unavailable and acquisition is costly,
it might be possible to use smaller data samples with deeper networks and
careful optimization. When a bigger dataset size is available, neither model
selection or optimization provides much performance gain, different approaches
such as changing pre-processing or loss function might be required. However,
as in the demonstrated case, any method should be tested in a range of
training dataset sizes in order to guarantee its performance. In the next
section we conclude this paper, summarizing our goals, experiments, results
and future investigations.

##  5 Conclusion

In this work we described a framework to check when the choice for the network
to be used in an application might change due to the number of images
available in a training dataset. We demonstrated an application of this
framework for LV segmentation in cardiac MRI images using the LSVC dataset. In
our experiments we observed the relationship between prediction error, dataset
size and network size; and also identified hyper-parameter effects on network
performance, by modelling with linear model $DICE\sim
ls+ls:Family+\log(Dataset)+\log(lr):ls:Family+\log(reg)$ and executing and
ANOVA of its interaction terms. The results indicated that: sample size
affected performance more than architecture or hyper-parameters; in small
samples the performance was more sensitive to hyper-parameters than
architecture; the performance difference between shallow and deeper networks
was not the same across families.

Further investigation and extension of these experiments will focus on
analysing the regularization effects’ patterns regarding network size. Also,
we plan to use similar strategies in Neural Architecture Search (NAS) and
Automatic Deep Learning (AutoDL) to discover better segmentation networks for
evolving and multi-centric medical imaging datasets.

## Acknowledgement

We thank Zerbini Foundation and FoxConn for supporting this work; A. Marco, C.
Graves, D. Cardenas, J. R. Ferreira Jr. and R. F. Pereira for discussing ideas
about experiments; R. Moreno and M. Rebelo for initial manuscript revision.

## References

  * Akaike [1974] H. Akaike.  A new look at the statistical model identification.  _IEEE Trans. on Autom. Control_ , 19(6):716–723, 1974.
  * Aziz et al. [2013] Y. F. A. Aziz, F. Fadzli, R. Rizal Azman, F. Mohamed Sani, A. Vijayananthan, and M. Nazri.  State of the heart: Cmr in coronary artery disease.  _Current Medical Imaging_ , 9(3):201–213, 2013\.
  * Bartlett et al. [2017] P. L. Bartlett, D. J. Foster, and M. Telgarsky.  Spectrally-normalized margin bounds for neural networks.  In _Proc. NIPS_ , pages 6241–6250, 2017.
  * Benkendorf and Hawkins [2020] D. J. Benkendorf and C. P. Hawkins.  Effects of sample size and network depth on a deep learning approach to species distribution modeling.  _Ecological Informatics_ , 60:101137, 2020.
  * Chaurasia and Culurciello [2017] A. Chaurasia and E. Culurciello.  Linknet: Exploiting encoder representations for efficient semantic segmentation.  In _Proc. IEEE VCIP_ , pages 1–4, 2017.
  * Chen et al. [2020] C. Chen, C. Qin, H. Qiu, G. Tarroni, J. Duan, W. Bai, and D. Rueckert.  Deep learning for cardiac image segmentation: A review.  _Frontiers in Cardiovascular Medicine_ , 7:25, 2020.
  * Dice [1945] L. R. Dice.  Measures of the amount of ecologic association between species.  _Ecology_ , 26(3):297–302, 1945.
  * Ghavami et al. [2019] N. Ghavami, Y. Hu, E. Gibson, E. Bonmati, M. Emberton, C. M. Moore, and D. C. Barratt.  Automatic segmentation of prostate mri using convolutional neural networks: Investigating the impact of network architecture on the accuracy of volume measurement and mri-ultrasound registration.  _Medical image analysis_ , 58:101558, 2019.
  * Hastie and Tibshirani [1990] T. J. Hastie and R. J. Tibshirani.  _Generalized additive models_ , volume 43.  CRC press, 1990.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _Proc. IEEE CVPR_ , pages 770–778, 2016.
  * Hu et al. [2018] J. Hu, L. Shen, and G. Sun.  Squeeze-and-excitation networks.  In _Proc. IEEE CVPR_ , pages 7132–7141, 2018.
  * Lin et al. [2017] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie.  Feature pyramid networks for object detection.  In _Proc. IEEE CVPR_ , pages 2117–2125, 2017.
  * Martin and Mahoney [2018] C. H. Martin and M. W. Mahoney.  Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior.  _arXiv preprint arXiv:1710.09553_ , 2018.
  * Neyshabur et al. [2019] B. Neyshabur, Z. Li, S. Bhojanapalli, Y. LeCun, and N. Srebro.  Towards understanding the role of over-parametrization in generalization of neural networks.  In _Proc. ICLR_ , 2019.
  * Nikolov et al. [2018] S. Nikolov, S. Blackwell, R. Mendes, J. De Fauw, C. Meyer, C. Hughes, H. Askham, B. Romera-Paredes, A. Karthikesalingam, C. Chu, et al.  Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapy.  _arXiv preprint arXiv:1809.04430_ , 2018.
  * Oktay et al. [2018] O. Oktay, J. Schlemper, L. L. Folgoc, M. Lee, M. Heinrich, K. Misawa, K. Mori, S. McDonagh, N. Y. Hammerla, B. Kainz, et al.  Attention u-net: Learning where to look for the pancreas.  In _Proc. Conf. MIDL–Medical Imaging with Deep Learning_ , 2018.
  * Ronneberger et al. [2015] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Proc. MICCAI_ , pages 234–241. Springer, 2015.
  * Sandler et al. [2018] M. Sandler, A. G. Howard, M. Zhu, A. Zhmoginov, and L. Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In _Proc. IEEE CVPR_ , pages 4510–4520, 2018.  doi:10.1109/CVPR.2018.00474.
  * Shahinfar et al. [2020] S. Shahinfar, P. Meek, and G. Falzon.  “how many images do i need?” understanding how sample size per class affects deep learning model performance metrics for balanced designs in autonomous wildlife monitoring.  _Ecological Informatics_ , 57:101085, 2020.
  * Simonyan and Zisserman [2015] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  In _Proc. ICLR_ , 2015.
  * Spencer et al. [2013] J. H. Spencer, J. L. Quill, M. G. Bateman, M. D. Eggen, S. A. Howard, R. P. Goff, B. T. Howard, S. G. Quallich, and P. A. Iaizzo.  The benefits of the atlas of human cardiac anatomy website for the design of cardiac devices.  _Expert Review of Medical Devices_ , 10(6):729–734, 2013.  doi:10.1586/17434440.2013.843449.
  * Tabachnick and Fidell [2007] B. G. Tabachnick and L. S. Fidell.  _Experimental designs using ANOVA_.  Thomson/Brooks/Cole Belmont, CA, 2007.
  * Tan and Le [2019] M. Tan and Q. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In _Proc. ICML_ , pages 6105–6114. PMLR, 2019.
  * Tan et al. [2019] M. Tan, B. Chen, R. Pang, V. Vasudevan, M. Sandler, A. Howard, and Q. V. Le.  Mnasnet: Platform-aware neural architecture search for mobile.  In _Proc. IEEE/CVF CVPR_ , pages 2820–2828, 2019.
  * Zhao et al. [2017] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia.  Pyramid scene parsing network.  In _Proc. IEEE CVPR_ , pages 2881–2890, 2017.
  * Zhou et al. [2018] Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang.  Unet++: A nested u-net architecture for medical image segmentation.  In _Deep learning in medical image analysis and multimodal learning for clinical decision support_ , pages 3–11. Springer, 2018.