# DoReMi: First glance at a universal OMR dataset
††thanks: Funded by UKRI, EPSRC and Steinberg.

1stElona Shatri  Centre for Digital Music
Queen Mary University of London
London, UK
<EMAIL_ADDRESS>2nd György Fazekas  Centre for Digital Music
Queen Mary University of London
London, UK
<EMAIL_ADDRESS>
###### Abstract

The main challenges of Optical Music Recognition (OMR) come from the nature of
written music, its complexity and the difficulty of finding an appropriate
data representation. This paper provides a first look at DoReMi, an OMR
dataset that addresses these challenges, and a baseline object detection model
to assess its utility. Researchers often approach OMR following a set of small
stages, given that existing data often do not satisfy broader research. We
examine the possibility of changing this tendency by presenting more metadata.
Our approach complements existing research; hence DoReMi allows harmonisation
with two existing datasets, DeepScores and MUSCIMA++. DoReMi was generated
using a music notation software and includes over 6400 printed sheet music
images with accompanying metadata useful in OMR research. Our dataset provides
OMR metadata, MIDI, MEI, MusicXML and PNG files, each aiding a different stage
of OMR. We obtain 64% mean average precision (mAP) in object detection using
half of the data. Further work includes re-iterating through the creation
process to satisfy custom OMR models. While we do not assume to have solved
the main challenges in OMR, this dataset opens a new course of discussions
that would ultimately aid that goal.

###### Index Terms:

optical music recognition, deep learning, dataset, sheet music

##  I Introduction

Despite improvements in music notation software, writing music on paper or
distributing music in print is still very common. However, written music needs
to be digitised for further editing, preparation for printing, sharing or
auditioning on a computer, or creating playable musical demos. The process of
manually digitising scores is slow and tiresome. New work is not the only
subject of this process. The abundance of undigitised sheet music in archives
and libraries is an essential motivation of OMR as well. Scores are often
scanned and become part of library archives which advances their accessibility
only partially. For example, the search of scanned music is limited to
metadata and excludes musical content or patterns. Moreover, scanned scores do
not allow for plagiarism check. An automatic process of converting handwritten
scores, prints and scans to documents that a machine can read and interpret is
the essence of Optical Music Recognition (OMR). OMR research typically divides
the problem into distinct stages. Conventionally, four main stages are
considered, starting with image pre-processing, followed by the detection of
primitive musical objects, reconstructing musically meaningful composites, and
finally, encoding using a machine-readable format [24]. By progressing one
stage at a time, inconsistencies were created, mainly in the datasets used by
researchers. Often, new work does not align with current work creating issues
in evaluation and comparison.

Recently, researchers have been working on bringing their datasets together,
namely MUSCIMA++ [4] and DeepScores [9], to improve compatibility. MUSCIMA++
contains handwritten music, while DeepScores is a typeset music dataset, both
of which can be used in object detection. Harmonising them aids the goal of
creating a reference dataset. With the motivation to support this goal and
compare and use these datasets in future experiments, DoReMi is easily
harmonised with MUSCIMA++ and DeepScores. Another limitation of several
existing OMR datasets is that only one research stage is supported primarily,
while other stages, such as reconstruction, are typically understudied. DoReMi
addresses this issue by also providing semantic information of the written
music. Given that DoReMi was generated using music notation software, we could
retrieve musical information that otherwise would be impossible to obtain.
Such data is available for playable notes (also grace notes), stave lines,
clefs, slurs and ties. Depending on the nature of the element, different
degrees of semantic information is provided. This data is integrated with
visual data, such as bounding boxes and pixel information of each element. We
expect this to facilitate further research in end-to-end deep learning systems
in OMR, which are quickly becoming state-of-the-art. Furthermore, DoReMi
includes different types of representations, including MusicXML, MIDI, MEI and
PNG images of sheet music alongside OMR metadata.

##  II Handwritten and Typeset OMR Datasets

Most of the existing OMR datasets target different approaches, stages and
objectives within OMR research. Differences between datasets primarily lie in
the annotations that accompany images of sheet music. There are also
variations in the type of notation, such as CWMN or mensural notation, in
primary focus, and differences in the printing type, i.e., typeset or
handwritten scores. A list of the main datasets, engraving type, number of
symbols and A4 images, formats provided and usage is given in Table I.
Conditional on the final objective of the task, there are also different music
representations used in datasets. For instance, we only need the MIDI file as
the ground truth to retrieve a replayable-only file. If the graphical elements
and an editable score are needed, other more complex formats such as MusicXML
and MEI are essential. The use case determines the optimal music
representation. MEI and MusicXML are score focused, while MIDI focuses on the
musical content and synthesiser control.

HOMUS (Handwritten Online Musical Symbols) [6] takes the approach of recording
pen-based (online) compositions. They present the strokes drawn by pen and the
image generated after drawing the symbols. The authors then propose a baseline
on the recognition of these two modalities. To conclude the recognition
accuracy, they use Nearest Neighbor (NN) technique and Hidden Markov Models
(HMM).

CVC-MUSCIMA [8] is the root dataset of MUSCIMA++. CVC-MUSCIMA was originally
designed to perform stave line removal. A total of 50 different musicians were
asked to write 20 pages of identical sheet music, having the same pen and
style. This dataset creates a link between low-level such as noteheads and
higher-level symbols such as key and time signatures and is mainly used and
best suited for musical object detection. Baseline experiments for object
detection using MUSCIMA++ use detection algorithms such as Faster R-CNN [3],
Single Shot Detectors [22] and DeepWaterShed Detectors [23]. A version of
MUSCIMA++ containing annotations for measures and staves only exists for bar
measure detection.

Another dataset that assists in the detection stage is DeepScores [9] which is
the largest OMR dataset that contains annotated images of typeset scores used
for object classification, detection and segmentation. There is variation
provided by rendering the sheets using five different fonts. This dataset
assists work in recognising tiny objects in large images. A new version of
DeepScores has detailed annotations, increased number of annotated symbols,
while also providing oriented bounding boxes for the symbols, a higher level
of rhythm and pitch information that includes onset beat for all symbols and
line position for noteheads and finally compatibility with MUSCIMA++ dataset
[19].

Printed Images of Music Staves (PrIMuS) [10] is one of the few datasets that
satisfy use cases in training end-to-end object recognition models. It
includes 87,678 real-music sequences of notes, typically the first ones, in
five different formats: MIDI, PNG, MEI, semantic and an encoding that contains
the symbols and their positions, disregarding their musical meaning. Another
version Camera-PrIMuS [11] includes images with distortion to simulate real-
world imperfections.

Universal Music Symbol Collection [7] is a dataset that collects and combines
symbols from HOMUS, MUSCIMA++, Audiveris OMR dataset, the Printed Music
Symbols dataset, OpemOMR dataset and two sets from the group of Rebelo et al.
[15] and Fornes et al. [16] that can be used to train classifiers. The symbols
amount to 74,000 handwritten and 16,000 printed symbols. The objective was to
create a universal, harmonised dataset that could assist in building a written
music classifiers.

Limitations of each dataset lie in their differences in data types, formats,
and narrow objectives. DoReMi goes one step further by allowing harmonisation
with MUSCIMA++ and DeepScores while adding semantic and graphical information
about the symbols. Furthermore, DoReMi uses five file formats, XML (with
positions), MusicXML, PNG, MEI and MIDI, with complementary information. PNG
images are complemented by metadata in an XML metadata file. MEI, MusicXML and
MIDI all representing possible encoded outputs of OMR, depending on the task.

TABLE I: Comparison of major OMR datasets published to date Dataset |  Engraving |  Symbols |  Images |  Classes |  Format |  Usage
---|---|---|---|---|---|---
DoReMi |  Typeset |  911771 |  6432 |  94 |  ML metadata, images, MIDI, MEI, MusicXML |  Object Detection, Reconstruction and Encoding, End-to-end
Handwritten Online Musical Symbols (HOMUS) [6] |  Handwritten |  15200 |  - |  32 |  Text-File |  Symbol Classification (online + offline)
Universal Music Symbol Collection [7] |  Typeset + Handwritten |  90000 |  - |  79 |  Images |  Symbol Classification (offline)
MUSCIMA ++ [4] |  Handwritten |  91255 |  140 |  110 |  Images, Measure Annotations, MuNG |  Symbol Classification, Object Detection and Measure Recognition
DeepScores [9, 19] |  Typeset |  100m |  255,386 |  135 |  Images, XML |  Symbol Classification, Object Detection, Semantic Segmentation
PrIMuS [11] |  Typeset |  87678 |  - |  - |  Images, MEI, simplified encoding, agnostic encoding |  End-to-End Recognition
Capitan collection [20] |  Handwritten |  - |  10230 |  30 |  Images, Text-File |  Symbol Classification
Bounding Box Annotations of Musical Measures [21] |  Typeset |  940 |  24,329 |  - |  Images |  CSV, plain JSON and COCO

##  III Dataset Description

In this section, we present DoReMi, an OMR dataset with typeset symbols. This
dataset is designed to be compatible with MUSCIMA++ and DeepScores and serve
as a reference for research in other stages of OMR. Notably, it helps research
in the reconstruction stage of music semantics such as notes pitch, duration,
beats and their relations to other elements in the score. Furthermore, we
expect DoReMi to aid work towards an end-to-end OMR system in non-monophonic
scores.

Music used to generate this dataset comes from a software test set provided by
the Dorico team 111Dorico is a music notation software -
https://new.steinberg.net/dorico. This test set includes a wider number of
objects, classes and various cases of notations not normally seen in real-
world music. About 600 files of the underlying material used to generate this
dataset is copyright protected; therefore, we only include openly
distributable scores in the final published dataset
222https://github.com/steinbergmedia/DoReMi/releases. However, pre-trained
models trained in the whole dataset will also be published. DoReMi includes
around 6432 images of sheet music with nearly a million annotated objects
which is $\frac{1}{50}$th the size of DeepScores and 42 times the size of
MUSCIMA++. Each object on the page is annotated with category labels from 94
different classes. However, there is an emphasised class imbalance; stems and
noteheads make up half of the annotated objects in the dataset. We also
provide prepared subsets fulfilling different requirements on the number of
pages, number of classes and the number of staves. Most of the images include
one system per page; depending on the number of voices, they have one or more
staves per page.

Following the organisation of MUSCIMA++, DoReMi has an OMR metadata file which
includes bounding boxes of each element: top, left, width and height. It also
includes the pixel mask for each element giving each object’s pixels inside
the bounding box. Additionally, DoReMi provides the relationships between
primitives. It vaguely follows the Music Notation Graph (MUNG), which creates
a graph representation of music notations. Inlinks and outlinks reference back
and forth to the ID of the objects they are related to—for instance, a
notehead half outlinks to a stem or a slur or both of them. Conversely, the
stem inlinks to the notehead half.

As opposed to other existing datasets, DoReMi provides semantic information on
playable notes and interpretive elements, see Appendix A. Certain (playable)
objects are also annotated with a Dorico event ID which is a unique event
identifier that provides additional information on how some objects are
linked. For instance, notes like noteheads have information on the duration
beats, onset beats, pitch octave, midi pitch code, normalised pitch step and
an event ID. For elements such as clefs, our dataset provides an event ID,
clef type, clef hotspot, clef required stave lines and clef stave position.
Clef hotspot identifies the midi pitch that clef denotes, i.e. for treble clef
is G4, as that is the pitch of the second stave line from the bottom. Clef
required stave lines shows how many stave lines the clef needs. Time
signatures include the event ID and its description, for example, 3/2 (h,
1+1+1) 333Beat division: h means half note, 1 + 1 + 1 means 3 equal beat,
1,2,3. Flags, if they are part of grace notes, have a boolean value set to
True. Slurs and ties have their event IDs, while barlines, rests, accidentals,
augmentation dots, stems do not have such information. Beams do not have their
event ID; instead, they have a list of the event IDs their respective
noteheads have. Other types of data given are Dorico project files, MIDI
files, PNGs, MusicXML and MEI. PNG files are binarised and provided with a
resolution of 300 DPI and dimensions of 2475x3504 pixels. There is a
possibility of creating lower or higher resolution images depending on
limitations in computational expense. One OMR XML metadata file may be
pointing back to multiple images. Each image has a reference page ID in the
XML file. MIDI files included can be used as ground truth to OMR tasks where
MIDI is the desired output. MusicXML [17] and MEI [14] are two file formats
that we desire to output after encoding the reconstructed information. They
were conceived for two different reasons. MusicXML was first proposed as a
file format to ease digital sheet exchange in the music publishing industry
[18]. MEI was born in the music research world to aid the storage of diverse
music manuscripts. Both share similarities in the objects they encode and
their file format being XML. MEI, beyond functionality in notation and page
layout, also encodes information about the notation in a more structured and
semantic way. In other words, MusicXML was designed for software rendering,
while MEI captures more music semantics [24].

##  IV Baseline Experiments

TABLE II: Object detection baseline results in DoReMi and MUSCIMA++ Meta-Architecture | Feature Extractor | Classes | Training steps | Data% | mAP (%)
---|---|---|---|---|---

DoReMi |  |  |  |  |
Faster R-CNN | Inception-ResNet-v2 | 71 | 80K | 90% | 57.5067
Faster R-CNN | Inception-ResNet-v2 with MUSCIMA++ | 71 | 120K | 90% | 64.8614
Faster R-CNN | ResNet50 | 71 | 120K | 90% | 63.4910
Faster R-CNN | ResNet101 | 71 | 148K | 90% | 26.994

MUSCIMA++ |  |  |  |  |
Faster R-CNN | Inception-ResNet-v2 | 110 | 80K | 100% | 82.4

Object detection is a crucial stage of optical music recognition. It is
concerned with localising and identifying objects of certain classes in a
sheet music image. Using the DoReMi dataset, we propose a baseline in object
detection to assess the benefits brought forward by the richer data in this
task. Based on previous work from Pacha et al. [2, 6, 9], we use Faster R-CNNs
as our central architecture. We also compare the results with those reproduced
using MUSCIMA++ using the same architecture.

Detecting objects in sheet music is considered more challenging than many
general-purpose computer vision tasks, given that the number of tiny objects
is very high. To detect such objects, we need to localise objects that assist
classification. This localisation is defined by four values
$b_{x},b_{y},b_{h}$ and $b_{w}$, while the first two terms determine the
centre of the bounding box that isolates that object, the latter ones provide
its height and width. The network also provides a class name for
classification purposes. When objects to be detected are not overlapping, it
is reasonably easy to obtain and interpret their bounding boxes. When two
objects overlap in the same grid cell, having more than one midpoint in a
cell, the network needs to know which object to predict. This overlap of
objects is often seen in sheet music. To deal with overlaps, anchor boxes can
be used. Objects are assigned to the respective grid cell and an anchor box
for that grid cell with the highest Intersection over Union (IoU). We get two
predicted bounding boxes for each of the grid cells then omit low probability
object predictions. Subsequently, for each class in our dataset, we run non-
max suppression to generate predictions. Non-max suppression eliminates the
bounding boxes that have a low probability, retaining the bounding box with
the highest probability.

Fast R-CNN [3] serves as a base for almost all proceeding work in object
detection. Fast R-CNNs use selective search to generate region proposals,
which is expensive. After region CNNs were introduced, Faster R-CNN became the
state-of-the-art approach in object detection [3]. Most of the work subsequent
to it follows a similar architecture by adding other valuable blocks such as
Mask R-CNNs, which also output the object masks indicating the pixels where
the object is in the bounding box. Faster R-CNNs are very similar to Fast
R-CNN, with the most significant difference being how the region proposals are
considered and CNNs are run on those, which cut the run time. Faster R-CNNs
introduce region proposal network (RPN), which enables sharing full-image
convolution features with the detection network [2]. RPN is a fully
convolutional layer, trained to generate good region proposals used by the
rest of the network.

The experiments II are performed in a subset of the DoReMi dataset consisting
of 5832 A4 images of scores. These images are of the same dimension throughout
the dataset with the limitation of one system per page. These systems very
often consist of multiple staves. The maximum number of staves per page is
six. Moreover, files that had less frequent objects were disregarded for this
set of experiment.

###  IV-A Feature Extractors

We first apply a convolutional feature extractor in all input images, so the
highest-level features are retrieved. Based on the number of parameters and
layer types, the processing time is highly affected. We show three types of
feature extractors and their training time for reference. We use open source
Inception Resnets (v2) [13] and Resnet50 and Resnet101 [12]. All use
Tensorflow implementations. Inception Resnets (v2) is a blended extractor with
164 layers which benefits from optimisations of the residual connections and
efficiency of Inception units. It replaces the filter concatenation stage of
the Inception architecture with residual connections. Resnet50 is another
feature extractor with a depth of 50 layers, meaning that it is considerably
lighter than Inception Resnet (v2), hence faster. Resnet101 belongs to the
same residual connection family with Resnet50 with more depth.

###  IV-B Results

A common technique to boost detection results is to feed the data into a pre-
trained network using the image representations in intermediate layers. We use
the respective pre-trained models in the general COCO dataset for each
extractor, except for a Faster R-CNN meta-architecture that uses Inception
Resnet (v2) with a pre-trained model on MUSCIMA++. This model provides the
best mAP score, which can be a result of the same domain pre-training.
Resnet50 performs very well, yielding a mAP of 63% as shown in Table from
where we can also see that Resnet50 produce better results. Furthermore
training time using Resnet50 extractors is half of that using Inception Resnet
(v2) II.

##  V Discussions

Baseline work in object detection with deep learning uses pre-trained models
such as Faster R-CNN, Fast R-CNN, SSD [1], with some fine-tuning during
training. However, while such pre-trained models can detect musical objects,
classes that are not well-represented pose a significant challenge. Creating
custom models for OMR is one of the main challenges for both the detection and
note assembly stage, given the graphical and structured nature of music. Such
models would improve the state-of-the-art work in OMR, but it would further
facilitate and trigger new research paths. The DoReMi dataset is designed in a
way that it can be harmonised with Deepscore and MUSCIMA++, which allows
various engravings and takes steps towards creating standardised, universal
OMR datasets. The design can further help to standardise evaluation at
different stages of OMR and facilitate end-to-end approaches in the field. The
dataset design follows the mung (Music Notation Graph), where nodes represent
primitives such as noteheads, stems, beams, and their relations being stored.
The DoReMi dataset is the first step towards overcoming some of the challenges
mentioned above. DoReMi in baseline experiments does not perform as good as
MUSCIMA++, resulting from MUSCIMA++ images being cut by staff, meaning the
number of objects is often smaller and of a larger dimension. While this
dataset aims to fill in existing gaps in OMR research, it is also prepared
with a view to investigate a more extensive research question, i.e., whether
deep learning can assist the OMR research field by providing data in richer
structure, so common tasks considered in isolation may be gradually integrated
into more complex architectures. What sets DoReMi apart from other existing
datasets primarily is data richness. Given that this dataset was generated
using Dorico, we could obtain more musical information and different data
types. Finally, the pipeline developed for assembling this dataset allows us
to extend the dataset, as well as iterate and re-design the structure if
needed based on the course of new research.

##  Appendix A Dataset Samples and Individual Class Attributes

Figure 1: Snippet of three different nodes in OMR XML data files TABLE III: List of 94 classes showing frequency of appearance for each class and their data attributes Class | Freq | DE id | clef type | clef hs | clef RSL | clef SP | staff id | DE ids | grace note | DB | OB | PO | MPC | NPS | TSD | text
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
accidentalDoubleFlat | 244 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
accidentalDoubleSharp | 330 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
accidentalFlat | 12705 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
accidentalKomaFlat | 5 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
accidentalKomaSharp | 5 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
accidentalNatural | 11137 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
accidentalQuarterToneFlatStein | 162 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
accidentalQuarterToneSharpStein | 191 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
accidentalSharp | 12908 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
accidentalThreeQuarterTonesFlatZimmermann | 1 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
accidentalThreeQuarterTonesSharpStein | 27 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
accidentalTripleFlat | 1 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
accidentalTripleSharp | 1 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
articAccentAbove | 1477 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
articAccentBelow | 1916 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
articMarcatoAbove | 278 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
articMarcatoBelow | 34 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
articStaccatissimoAbove | 536 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
articStaccatissimoBelow | 414 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
articStaccatoAbove | 5018 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
articStaccatoBelow | 6108 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
articTenutoAbove | 823 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
articTenutoBelow | 735 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
augmentationDot | 2762 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
barline | 28142 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
beam | 52539 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
cClef | 1161 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
dynamicFF | 242 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
dynamicFFF | 88 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
dynamicFFFF | 4 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
dynamicForte | 777 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
dynamicFortePiano | 60 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
dynamicForzando | 35 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
dynamicMF | 1803 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
dynamicMP | 185 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
dynamicPiano | 3987 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
dynamicPP | 1807 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
dynamicPPP | 93 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
dynamicPPPP | 13 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
dynamicRinforzando2 | 5 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
dynamicSforzato | 236 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
dynamicSforzatoFF | 6 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
dynamicText | 110 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
fClef | 3543 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
flag16thDown | 430 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
flag16thUp | 18245 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
flag32ndDown | 31 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
flag32ndUp | 7264 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
flag64thUp | 10 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
flag8thDown | 3672 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
flag8thUp | 8113 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
gClef | 12278 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
gradualDynamic | 7084 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
kStaffLine | 160426 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
mensuralNoteheadMinimaWhite | 9 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
noteheadBlack | 247741 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
noteheadDiamondWhole | 8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
noteheadDoubleWholeSquare | 3 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
noteheadHalf | 8648 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
noteheadTriangleUpBlack | 27 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
noteheadTriangleUpHalf | 3 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
noteheadWhole | 1502 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
noteheadXBlack | 81 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
ornamentMordent | 7 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
ornamentTrill | 58 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
ornamentTurn | 6 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
rest | 4 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
rest16th | 29411 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
rest32nd | 10218 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
rest64th | 4 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
rest8th | 24124 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
restHalf | 1376 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
restQuarter | 13223 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
restWhole | 14382 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
slur | 13928 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
stem | 227889 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
systemicBarline | 2078 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
tie | 8626 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
timeSig1 | 2 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
timeSig2 | 1084 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
timeSig3 | 1203 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
timeSig4 | 2674 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
timeSig5 | 507 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
timeSig6 | 246 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
timeSig7 | 125 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
timeSig8 | 1661 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
timeSig9 | 79 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
timeSigCommon | 206 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
timeSigCutCommon | 113 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
timeSignatureComponent | 443 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
tupletBracket | 4970 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
tupletText | 4900 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
unpitchedPercussionClef1 | 223 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
wiggleTrill | 45 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

## Acknowledgement

The authors acknowledge the support of the AI and Music CDT, funded by UKRI
and EPSRC under grant agreement no. EP/S022694/1 and our industry partner
Steinberg Media Technologies GmbH for their continuous support especially
during the dataset creation.

## References

  * [1] A. Pacha, K. Choi, B. Coüasnon, Y. Ricquebourg, R. Zanibbi and H. Eidenberger, ”Handwritten Music Object Detection: Open Issues and Baseline Results,” 2018 13th IAPR International Workshop on Document Analysis Systems (DAS), Vienna, Austria, 2018, pp. 163-168, doi: 10.1109/DAS.2018.51.
  * [2] R. Girshick, ”Fast R-CNN,” 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, 2015, pp. 1440-1448, doi: 10.1109/ICCV.2015.169.
  * [3] Ren, S., He, K., Girshick, R., and Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint arXiv:1506.01497.
  * [4] J. Hajič and P. Pecina, ”The MUSCIMA++ Dataset for Handwritten Optical Music Recognition,” 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), Kyoto, Japan, 2017, pp. 39-46, doi: 10.1109/ICDAR.2017.16.
  * [5] J. Calvo-Zaragoza, J. Hajič Jr., and A. Pacha. 2020. Understanding Optical Music Recognition. ACM Comput. Surv. 53, 4, Article 77 (September 2020), 35 pages. DOI:https://doi.org/10.1145/3397499
  * [6] J. Calvo-Zaragoza and J. Oncina, ”Recognition of Pen-Based Music Notation: The HOMUS Dataset,” 2014 22nd International Conference on Pattern Recognition, Stockholm, Sweden, 2014, pp. 3038-3043, doi: 10.1109/ICPR.2014.524.
  * [7] A. Pacha and H. Eidenberger, ”Towards a Universal Music Symbol Classifier,” 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), Kyoto, Japan, 2017, pp. 35-36, doi:
  * [8] Fornés, A., Dutta, A., Gordo, A., and Lladós, J. (2012). CVC-MUSCIMA: a ground truth of handwritten music score images for writer identification and staff removal. International Journal on Document Analysis and Recognition (IJDAR), 15(3), 243-251.
  * [9] Tuggener, L., Elezi, I., Schmidhuber, J., Pelillo, M., Stadelmann, T. (2018, August). DeepScores-a dataset for segmentation, detection and classification of tiny objects. In 2018 24th International Conference on Pattern Recognition (ICPR) (pp. 3704-3709). IEEE.
  * [10] Calvo-Zaragoza, J., Rizo, D. (2018). End-to-end neural optical music recognition of monophonic scores. Applied Sciences, 8(4), 606.
  * [11] Calvo-Zaragoza, J., and Rizo, D. (2018, September). Camera-PrIMuS: Neural End-to-End Optical Music Recognition on Realistic Monophonic Scores. In ISMIR (pp. 248-255).
  * [12] He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).
  * [13] C. Szegedy, S. Ioffe, and V. Vanhoucke. Inception-v4, inception-resnet and the impact of residual connections on learning. arXiv preprint arXiv:1602.07261, 2016.
  * [14] Roland, P. (2002, September). The music encoding initiative (MEI). In Proceedings of the First International Conference on Musical Applications Using XML (Vol. 1060, pp. 55-59).
  * [15] A. Rebelo, G. Capela, and J. S. Cardoso, “Optical recognition of music symbols” International Journal on Document Analysis and Recognition (IJDAR), vol. 13, no. 1, pp. 19–31, 2010.
  * [16] A. Fornes, J. Llados, and G. Sanchez, ”Old Handwritten Musical Symbol Classification by a Dynamic Time Warping Based Method”. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, pp. 51–60.
  * [17] Good, M. (2001, December). MusicXML: An internet-friendly format for sheet music. In Xml conference and expo (pp. 03-04).
  * [18] Cunningham, S. (2004). Suitability of musicxml as a format for computer music notation and interchange. In Proceedings of IADIS Applied Computing 2004 International Conference, Lisbon, Portugal.
  * [19] Tuggener, L., Satyawan, Y. P., Pacha, A., Schmidhuber, J., and Stadelmann, T. (2020). The DeepScoresV2 dataset and benchmark for music object detection. In 25th International Conference on Pattern Recognition 2020 (ICPR’20), Online, 10-15 January 2021. IAPR.
  * [20] Calvo-Zaragoza, J., Rizo, D., and Quereda, J. M. I. (2016, August). Two (Note) Heads Are Better Than One: Pen-Based Multimodal Interaction with Music Scores. In ISMIR (pp. 509-514).
  * [21] Frank Zalkow, Angel Villar Corrales, TJ Tsai, Vlora Arifi-Müller, and Meinard Müller: “Tools for Semi-Automatic Bounding Box Annotation of Musical Measures in Sheet Music”. Late Breaking/Demo at the 20th International Society for Music Information Retrieval, Delft, The Netherlands, 2019.
  * [22] Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., and Berg, A. C. (2016, October). Ssd: Single shot multibox detector. In European conference on computer vision (pp. 21-37). Springer, Cham.
  * [23] Tuggener, L., Elezi, I., Schmidhuber, J., and Stadelmann, T. (2018). Deep watershed detector for music object recognition. arXiv preprint arXiv:1805.10548.
  * [24] Shatri, E. and Fazekas, G., “Optical music recognition: State of the art and major challenges,” in Proceedings of the International Conference on Technologies for Music Notation and Representation TENOR’20/21, R. Gottfried, G. Hajdu, J. Sello, A. Anatrini, and J. MacCallum, Eds. Hamburg, Germany: Hamburg University for Music and Theater, 2020, pp. 175–184.


11institutetext: Government College of Engineering and Ceramic Technology
11email<EMAIL_ADDRESS>
# AKHCRNet: Bengali handwritten character recognition using deep learning

Roy  Akash

###### Abstract

I propose a state of the art deep neural architectural solution for
handwritten character recognition for Bengali alphabets, compound characters
as well as numerical digits that achieves state-of-the-art accuracy 96.8% in
just 11 epochs. Similar work has been done before by Chatterjee, Swagato, et
al.[1] but they achieved 96.12% accuracy in about 47 epochs. The deep neural
architecture used in that paper was fairly large considering the inclusion of
the weights of the ResNet 50 model which is a 50 layer Residual Network. This
proposed model achieves higher accuracy as compared to any previous work & in
a little number of epochs. ResNet50 is a good model trained on the ImageNet
dataset, but I propose an HCR network that is trained from the scratch on
Bengali characters without the ”Ensemble Learning” that can outperform
previous architectures.

###### Keywords:

Deep Learning Image Recognition Convolutional Networks Handwritten character
recognition

##  1 Introduction

###  1.1 Motivation

Bengali is one of the official languages of the Republic of India and the
official language of Bangladesh. About 230 million people speak and write
Bengali as a native language around the world. So recognition of Bengali
character is an important problem needed to be solved which could potentially
help solving other noble applications like optical character recognition,
handwritten character recognition and even word recognition.

But solving Bengali character recognition is much more tough than the English
counterpart because

  * •

There are far more characters in Bengali alphabets,

  * •

Bengali has conjunct consonants in it’s alphabets which are harder to classify
because it consists of two distinct Bengali characters. Any learning algorithm
could simply classify those characters as one or the other,

  * •

Many Bengali characters resembles each other closely maybe differentiated by
small lines and dots, for example 1st and the 2nd vowels in Bengali alphabets
are differentiated by one vertical line.

The problem of English character recognition may have been solved but the
Bengali counterpart still have not achieved the human level accuracy.

Although there is a substantial amount of work that has been done on this
topic as detailed in the previously done section improvements still can be
achieved as the benchmark has not yet reached human-level accuracy or
surpassed that. Using recent advancements both in the usage of convolution
kernels in Deep Learning and also in model training procedures (like hyper-
parameter tuning, data augmentation, etc.) can increase the accuracy of HCR
Models. Although scientists who most recently worked on this topic used the
transfer learning from the ResNet50 model achieved 96.12% accuracy, I used a
sophisticated architecture and trained from scratch & got a better result. The
BanglaLekha-Isolated Dataset as described in datasets section was used for
evaluation mainly because it has a large sample size and large variance and
also because the output classes are balanced. The dataset has a total of 84
characters, which has 10 numerals, 50 basic characters, and 24 most common
conjunct-consonants.

###  1.2 Previous works

Several other people laid the foundation for Bengali handwritten character
recognition. In those, work Ray and Chatterjee did the first significant
work[2] using nearest neighbour classifiers. After that, many more researchers
did other methods, improving the performance of Bangla Handwritten Character
Recognition (HCR) like Hasant et al.[3] proposed an HCR capable of classifying
both printed and handwritten characters by using Discrete Cosine Transform
method over the input image and Hidden Markov Model (HMM) for character
classification. Wen et al.[4] proposed a Bangla numerals recognition method
using Principal Component Analysis and Support Vector Machines, but those are
using traditional machine learning methods. In Hassan and Khan’s work, they
used the KNN algorithm where features were extracted using local binary
patterns. Das et al.[5] proposed a feature set representation for Bangla
handwritten character recognition. It was done by hand engineering 8 distance
features, 24 shadow features, 84 quad-tree based longest run features, and 16
centroid features. Their accuracy was about 85.40% on a 50 character class
dataset as hand engineering was difficult for the Conjunct-Consonants in
Bengali characters. Rumman Rashid Chowdhury used CNN based classifier[11] and
gained over 98% accuracy on 10 neumericals and 91.12% accuracy on 50 basic
Bengali character. In the mentioned methods, the main concern is that they
used many handcrafted features extracted for the small dataset, and also those
didn’t achieved the human level accuracy. So these methods are in no means
achieved the human level accuracy in handwritten character recognition. As
convolutional neural networks eliminated the need for feature extraction,
it’ll be much more economical to make the HCR network to learn the features on
it’s own.

##  2 Dataset

The dataset I’m going to be using for this problem is the BanglaLekha Isolated
Dataset[12]. There are many open datasets that could be used for this
classification problem but this dataset is chosen because of the followings:

  * •

This dataset has over 166,000 examples of Bengali characters. Neural networks
can learn better with huge amounts of data.

  * •

Dataset is made with male as well as female native Bengali speakers of
different states of Bangladesh. So the dataset is not biased towards a
specific gender category.

  * •

Their age ranging from 4-27. So the dataset is not biased to a certain age
groups

  * •

This dataset also introduces handwritten data from the people of special
ability. So the dataset is not biased to a people of special abilities.

  * •

This dataset also doesn’t have a class imbalance problem (More or less same
amount of examples for each category).

But the data has some blank images or images that are not taken properly or
mislabeled data due to human error. So I have manually gone through every
directory of 84 classes and removed some of the mislabeled data the dataset.
This process is much important as I am using an algorithm to generate the
validation dataset (data the neural network hasn’t seen yet) taken at random
from the directory. If any of those mislabeled or blank images get into the
validation set the neural network will never know and this will generate
avoidable errors in the predictions. But in the training dataset, if 1-2
images get blank or mislabeled error, it doesn’t matter much. But due to sheer
amount of data and unavailability of man power I only managed to remove
mislabeled data from 20 categories.

Dataset preparation goes like this: Out of 166K (after cleaning mislabeled and
blank images) I’ve used 28% of data randomly chosen from the 84 classes, so no
human bias for selection of validation data is present and the rest of the
images are used for training.

This is how the dataset is structured:

Dataset | Total Images Used | Percentage of total data
---|---|---
Training | 118394 | 72%
Validation | 45983 | 28%

So the validation data is selected by computer at random before the training
and the neural network doesn’t see the validation data during training, this
way we can safely measure the accuracy of the network. The accuracy measuring
algorithm used here is described below in the architecture section. In the
next section it’s shown how data is processed before going into the network.

##  3 Image data pre-processing

The very recent work that has been done by Chatterjee, Swagato, et al.[1]
achieved the best result yet for a training of 47 epochs at 96.12% and the 97%
accuracy achieved by Saha and Saha et. al used ensemble CNN architecture run
for 500 epochs. Individually, their accuracy was 95.67% and 92.43%.

Let’s look deeper at what Chatterjee, Swagato, et al. did. They used ResNet50
model, which is pre-trained on the ImageNet dataset. For a hand-written
classification task I think that they’ve used transfer learning from a model
that is specifically not trained to classify handwritten characters. I think
that ResNet 50 model’s CNN learnt to detect several features that is not quite
useful for this task of classifying hand written characters. If I’d to use a
model, I’d use models that maybe trained previously on similar handwritten
datasets like the MNIST dataset[8], I think that would converge much faster
and could get better results.

Secondly they used images of size sometimes height of 200 px, width of 200 px,
color depth of 3 to comply with ResNet architecture, other times 128 by 128,
but in general these are huge image size. I got rid of the unnecessary color
channels. I used gray-scale image as an input shape because fundamentally you
wouldn’t care whether your bengali handwritten character is colorful or not in
order to classify them. This made less information but the ’same’ useful
information going into the network, so the training will converge sooner than
that of K Chatterjee et.al’s training duration which is 47 epochs.

Next step of pre-processing is to decrease the amount of information coming in
neural network. I propose instead of using 200 pixels for width and height
that is 40,000 parameters we should be using 32 pixels for width and height
that is 1024 parameters. Let’s have a look at the data and see if those image
resizing processing actually hurts the data and make them unrecognizable for
the network? Each pixel value of the images is also normalized between 0-1 to
converge faster, and this also makes the neural network easier to train.

I’m using the idea of bi-linear interpolation technique when resizing the
input images from the original size, that goes like this:

When we are given an image suppose that we want to find the value of the
unknown function f at a specific point in image (x, y). It is assumed that we
know the value of function f at the four points being

$Q_{11}=(x_{1},y_{1}),$ $Q_{12}=(x_{1},y_{2}),$ $Q_{21}=(x_{2},y_{1}),$
$Q_{22}=(x_{2},y_{2}).$

We first do linear interpolation in the x-direction. This gives us the
following:

$f(x,y_{1})\approx\frac{x_{2}-x}{x_{2}-x_{1}}f(Q_{11})+\frac{x-x_{1}}{x_{2}-x_{1}}f(Q_{21})$
(1)

$f(x,y_{2})\approx\frac{x_{2}-x}{x_{2}-x_{1}}f(Q_{12})+\frac{x-x_{1}}{x_{2}-x_{1}}f(Q_{22})$
(2)

Then to the y direction:

$f(x,y)\approx\frac{y_{2}-y}{y_{2}-y_{1}}f(x,y_{1})+\frac{y-y_{1}}{y_{2}-y_{1}}f(x,y_{2})$
(3)
$=\frac{y_{2}-y}{y_{2}-y_{1}}\left(\frac{x_{2}-x}{x_{2}-x_{1}}f(Q_{11})+\frac{x-x_{1}}{x_{2}-x_{1}}f(Q_{21})\right)+\frac{y-y_{1}}{y_{2}-y_{1}}\left(\frac{x_{2}-x}{x_{2}-x_{1}}f(Q_{12})+\frac{x-x_{1}}{x_{2}-x_{1}}f(Q_{22})\right)$

Using this bi-linear interpolation technique for image resizing you can see
the result here:

(a) Image Size (171, 170)

(b) Image Size (32, 32)

Figure 1: Class 1 Image before resize and after

So we can clearly see that our images are still recognizable and so we can get
away with 1024 pixels, which will not going to hurt the deep learning model
for less data in an image. Let’s see more images resized to (32, 32, 1) and
how those looks:

(a) Image Size (245, 227)

(b) Image Size (32, 32)

Figure 2: Class 80 image resized before & after

##  4 Architecture

The data is coming into the network as (32, 32) gray-scale image, so I have
added a (32, 32, 1) layer as the input layer. For the next two layers I’ve
used the (5, 5) convolutions and 32 filters of those, this combination worked
great against a single (3, 3) convolution. These convolution layers have
”same” padding as we have very little information on the images and we don’t
want to lose those during convolutions. Throughout the entire deep neural
architecture, the ReLU activation is used. Where

$ReLU(z)=max(0,z)$

Next, a Batch-Normalization is used, this makes our model’s weights of deeper
layers more robust to changes than to the first few layers. Next, I’ve added a
Max Pooling Layer.

I’ve added a block of (1, 1) convolution with 128 filters followed by one (3,
3), one (5, 5) convolution with 128 filters separately and one (1, 1)
convolution and one ”same” padded max-pooling layer followed by a (3, 3)
convolution. All convolution layers mentioned here have ”same” padding.
Without the inclusion of this block the architecture converged to a 96.38%
accuracy in 20 epochs, which still has the better validation accuracy than
that of previous state of the art architecture, but with the inclusion of this
block validation accuracy peaked 96.80% in just 11 epochs.

The total of 4 outputs from the previous block has been concatenated into a
(16, 16, 448) layer and a ReLU activation is applied on top of it. Following
this there is another two sets of convolution-maxpooling-batch norm-maxpooling
blocks. In these two blocks the convolutions has 256 and 512 filters
respectively. Same padding and ReLU activation is used throughout and the data
is flattened out in the next layer.

Following this I’ve added fully connected layer which is 4 layer deep and each
layer has 1024, 512, 256, 128 number of hidden units respectively and the
output layer has 84 units for the 84 classes.

Figure 3: Layer by layer image data flow in AKHCRNet Model

The last layer has 84 hidden units with Softmax activation applied to it to
comply with the 84 predictable classes in the Bengali dataset. The last few
hidden layers are penalized with L2 Regularization method to decrease
overfitting, Cortes, Corinna et. al [6]. The regularizers used in the last few
hidden layers are kernel regularizer, as weights in the network is a very high
dimensional parameter-vector, but the bias is just a single real number. so
almost all the parameters are in weights rather than in the bias. So I think
adding a bias regularizer won’t make that much of a difference. So it’s useful
to only use the kernel regularizer in this problem.

For a L layer deep network the loss function J is the following:

$J(w^{[1]},b^{[2]},...,w^{[L]},b^{[L]})$

Where W[i], b[i] is the i-th layers weight vector and bias respectively. The
function J is a softmax cross-entropy loss also known as categorical cross-
entropy loss shown below. Softmax function:

$f(s)_{i}=\frac{e^{s_{i}}}{\sum_{j}^{C}e^{s_{j}}}$ (4)

Categorical cross-entropy loss:

$CCE=-\sum_{i}^{C}t_{i}\log(f(s)_{i})$ (5)

In this specific case of Multi-Class classification the labels are one-hot, so
only the positive class Cp keeps its position in the loss. There is only one
element of the target vector t which is not zero ti = tp. So discarding the
elements of the summation which are zero due to target labels, we can say:

$CCE=-\log\left(\frac{e^{s_{p}}}{\sum_{j}^{C}e^{s_{j}}}\right)$

Here ti and si are the ground truth and the CNN prediction score for each
class i in all classes C, An softmax activation is applied.

To make understanding clear from now on the categorical loss will be
represented as

$L(\hat{y},y)$

where y hat is the predicted label from softmax output unit for a given ground
truth y. After adding the L2 regularization the loss function J has become:

$J(w^{[1]},b^{[2]},...,w^{[L]},b^{[L]})=\frac{1}{m}\sum_{1}^{n}L(\hat{y},y)+\frac{\lambda}{2m}\sum_{1}^{L}{\left\|W^{[L]}\right\|}^{2}$

The extra term

$\frac{\lambda}{2m}\sum_{1}^{L}{\left\|W^{[L]}\right\|}^{2}$

is the L2 weight regularizer. W is a weight vector of dimensions (n[L-1],
n[L]) where n[L-1] and n[L] is the number of hidden unit in the L-1 th and Lth
layer of the deep network. So the weight decay is now applied so that
overfitting doesn’t occur. The reason behind this deep network is to ensure
not to under-fit the network. I used a 7-10 layer deep network, it was getting
90% to 92% accurate in 20 epochs. So adding more layers has made the
architecture more prone to learning intricate features that it hasn’t learnt
before.

As this is turning out to be a very deep network I also added a dropout with
the rate of 50% in the second hidden layer. Experiments are made and it is
seen that between 20-80% dropout rate of 50% worked the best for this problem.

##  5 Training, Learning rate scheduling, Optimization

The Adam Optimizer algorithm is used for this architecture, with no change to
the default hyper-parameters

$\alpha=1e-3$ $\beta_{1}=0.9$ $\beta_{2}=0.999$ $\epsilon=1e-8$

and instead of using a automated learning rate scheduler I’ve manually tweaked
the learning rate epoch by epoch. Here is how that goes:

Epochs | Set Learning rate | Accuracy achieved | Cross Entropy Loss
---|---|---|---
First 5 Epochs | 0.001 | 95.11% | 0.30081
Next 3 Epochs | 0.0001 | 96.79% | 0.22147
Last 3 Epochs | 0.00004 | 96.80% | 0.21612

The result after 11 epochs is shown in the Result Section.

##  6 Result

In the following figure Figure-4 epoch by epoch validation loss and accuracy
is shown. Clearly no sign of overfitting can be seen, the overall accuracy is
96.80% in 11 epochs which is a monumental jump from previous 500 epoch CNN
ensemble training by Saha and Saha et al. and 47 epoch training by K
Chatterjee et al.

(a) Validation loss over time

(b) Validation accuracy over time

Figure 4: validation accuracy and loss with each passing iteration.

Below is the confusion matrix of 84 classes of Bengali character predictions.
You can also see all the precision, recall, f1-score for each category in the
following link. https://bit.ly/akhcrnetreport-csv

Figure 5: Confusion Matrix

Here Precision is the ratio of correctly predicted positive observations in
Bengali characters to the total positive predictions. The question that this
metric answer is of all characters that labeled as some label, how many of
them is actually that label? The high precision always relates to the low
false positive rate. We have got about 0.97 (avg over 84 class examples)
precision which is pretty good for a hand character recognizer. The report for
each class can be seen following the link https://bit.ly/akhcrnetreport-csv.

$precision=\frac{TP}{(TP+FP)}$

Recall (Sensitivity) - Recall is the ratio of correctly predicted positive
observations to the all observations in actual class - yes. The question
recall answers is: Of all the characters that truly that label, how many did
we label? We have got recall of 0.97 which is good for this model.

$recall=\frac{TP}{(TP+FN)}$

F1 score - F1 Score is calculated by the weighted average of precision and
recall. Therefore, this score takes into account both false positives and
false negatives. Intuitively it is not as easy to understand as accuracy, but
f1-score can be more useful than accuracy, as we have an some-what uneven
class distribution. Accuracy works best when the false positives and false
negatives have the similar cost. If the cost of false positives and false
negatives are very different, it’s better to look at both Precision and
Recall. In our case, F1 score is 0.97 (avg). The f1-score for each class can
be seen following the link

https://bit.ly/akhcrnetreport-csv.

$f1-score=\frac{2*(Recall*Precision)}{(Recall+Precision)}$

###  6.1 Discussion

I call this architecture AKHCRNet which is a fancy way of saying Hand
Character recognizer Network by by Akash. This architecture out-performs every
other proposed Bengali classifier algorithms. Let’s see how other researchers
works did back then:

Researchers and their work |  Total Classes | Architecture | Accuracy | Epochs
---|---|---|---|---
Purkayastha et al.[7] | 50 | Vanilla CNN | 89.01% |
Rahmen et al.[9] | 50 | CNN | 85.96% |
Alif et al.[10] | 84 |  | Modified ResNet-18
---
with dropout layer of
20% rate
95.10% |
Rumman Rashid Chowdhury et al.[11] |  | 50 Characters
---
\+ 10 Numbers
Custom CNN |  | 91.13 % (50 Char)
---
\+ 98.42% (Numbers)
92
Majid and Smith[13] | 50 |  | Non CNN
---
Architecture
96.80% |
Saha and Saha[14] | 84 | Ensemble CNN |  | 97.21%
---
(95.67% &
96.12%)
500
Chatterjee, Swagato, et al.[1] | 84 |  | Transfer Learning on
---
ResNet-50
96.12% | 47
Proposed Work | 84 |  | Custom 32-Layer Deep
---
Network, AKHCRNet
96.80% | 11

The Saha and Saha paper did the ensemble of CNN, with two CNN architecture,
each of which had the accuracy 95.67% and 92.43% respectively, but AKHCRNet
model was able to achieve the accuracy of 96.80%. The code and the weight file
is available for download here:

https://www.github.com/theroyakash/AKHCRNet.

##  7 The Conclusion and Future Scope

AKHCRNet model was able to achieve the accuracy of 96.80% on the BanglaLekha-
Isolated Dataset of 166K images, Without ensembling, without transfer learning
and with a minimal number of epochs our proposed solution achieves state of
the art result on Bengali Handwritten character recognition.

AKHCRNet’s custom very deep neural architecture trains on a hard and large
dataset from scratch without transfer learning and achieves a 96.8% accuracy
with just 11 epochs. The success of this architecture could make possible
other indic (Indic is an adjective that refers to the Indo-Aryan languages)
character classification easier.

## References

  * [1] Chatterjee, Swagato, et al. ”Bengali handwritten character classification using transfer learning on deep convolutional neural network.” arXiv preprint arXiv:1902.11133 (2019).
  * [2] A. K. Ray and B. Chatterjee, “Design of a Nearest Neighbour Classifier System for Bengali Character Recognition,” IETE Journal of Research, vol. 30, no. 6, pp. 226–229, Nov. 1984.
  * [3] M. A. Hasnat, S. M. Habib, and M. Khan, “A high performance domain specific OCR for Bangla script,” in Novel Algorithms and Techniques In Telecommunications, Automation and Industrial Electronics, Springer, 2008, pp. 174–178.
  * [4] Y. Wen, Y. Lu, and P. Shi, “Handwritten Bangla Numeral Recognition System and Its Application to Postal Automation,” Pattern Recogn., vol. 40, no. 1, pp. 99–107, Jan. 2007
  * [5] Das, Nibaran, et al. ”An improved feature descriptor for recognition of handwritten Bangla alphabet.” arXiv preprint arXiv:1501.05497 (2015).
  * [6] Cortes, Corinna, Mehryar Mohri, and Afshin Rostamizadeh. ”L2 regularization for learning kernels.” arXiv preprint arXiv:1205.2653 (2012).
  * [7] B. Purkaystha, T. Datta, and M. S. Islam, “Bengali handwritten character recognition using deep convolutional neural network,” in 2017 20th International Conference of Computer and Information Technology (ICCIT). IEEE, 2017, pp. 1–5
  * [8] LeCun, Y., Cortes, C.: MNIST handwritten digit database. (2010).
  * [9] M. M. Rahman, M. Akhand, S. Islam, P. C. Shill, M. Rahman et al., “Bangla handwritten character recognition using convolutional neural network,” International Journal of Image, Graphics and Signal Pro- cessing (IJIGSP), vol. 7, no. 8, pp. 42–49, 2015
  * [10] M. A. R. Alif, S. Ahmed, and M. A. Hasan, “Isolated Bangla handwritten character recognition with convolutional neural network,” in 2017 20th International Conference of Computer and Information Technology (ICCIT), 2017, pp. 1–6.
  * [11] Chowdhury, Rumman and Hossain, Mohammad and Islam, Raihan and Andersson, Karl and Hossain, Sazzad. Bangla Handwritten Character Recognition using Convolutional Neural Network with Data Augmentation. 10.1109/ICIEV.2019.8858545.
  * [12] M. Biswas et al., “BanglaLekha-Isolated: A multi-purpose comprehensive dataset of Handwritten Bangla Isolated characters,” Data in Brief, vol. 12, pp. 103–107, Jun. 2017
  * [13] N. Majid and E. H. B. Smith, “Introducing the Boise State Bangla Handwriting Dataset and an Efficient Offline Recognizer of Isolated Bangla Characters,” in 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), 2018, pp. 380–385.
  * [14] S. Saha and N. Saha, “A Lightning fast approach to classify Bangla Handwritten Characters and Numerals using newly structured Deep Neural Network,” Procedia Computer Science, vol. 132, pp. 1760–1770, 2018.


$M^{c}_{\textup{M-DSM}}=\bordermatrix{&\Gamma&\textup{M}_{1}&\textup{M}_{2}&\textup{M}_{2}\cr\Gamma&0_{4\times
4}&\blacksquare&\blacksquare&\blacksquare\cr\textup{M}_{1}&\blacksquare&0_{4\times
4}&0_{4\times 4}&0_{4\times 4}\cr\textup{M}_{2}&\blacksquare&0_{4\times
4}&0_{4\times 4}&0_{4\times 4}\cr\textup{M}_{3}&\blacksquare&0_{4\times
4}&0_{4\times 4}&0_{4\times 4}\cr},\;\;\;$ (378)

This type of mass matrix has an emergent sub-lattice symmetry where

$U_{SL}^{\dagger}M^{c}_{\textup{M-DSM}}U_{\textup{SL}}=-M^{c}_{\textup{M-DSM}}$
(379)

with

$U_{\textup{SL}}=\bordermatrix{&\Gamma&\textup{M}_{1}&\textup{M}_{2}&\textup{M}_{2}\cr\Gamma&1_{4\times
4}&0_{4\times 4}&0_{4\times 4}&0_{4\times 4}\cr\textup{M}_{1}&0_{4\times
4}&-1_{4\times 4}&0_{4\times 4}&0_{4\times 4}\cr\textup{M}_{2}&0_{4\times
4}&0_{4\times 4}&-1_{4\times 4}&0_{4\times 4}\cr\textup{M}_{3}&0_{4\times
4}&0_{4\times 4}&0_{4\times 4}&-1_{4\times 4}\cr}$ (380)

which guarantees that there are at least 8 zero eigenvalues leading to (at
least) 8 gapless modes for any such mass. Note that the gapless modes will be
a linear combination of spinors from all the valleys and, in particular will
depend on the non-zero entries denoted by $\blacksquare$. Such semimetals are
dubbed ”M-Dirac semimetals (M-DSM)”. In Table 4, the entries 19 and 21 are of
this type.

##  Appendix K Spinless fermions on a honeycomb lattice with $\pi$-flux at
$1/4$-th filling

Figure 15: Honeycomb lattice with $\pi$-flux. (Left) Unit cell consisting of 8
sites adopted for the analysis. Fermions hop to nearest neighbors where red
links shown have a hopping amplitude with a negative sign. (Right) Band
structure showing four bands, each of which is two-fold degenerate. The light
blue plane shows the quarter-filled chemical potential.

In this section, we revisit the physics of spinless fermions hopping on a
honeycomb lattice with a $\pi$-flux through each of the honeycomb plaquettes.
We adopt the unit cell shown in Fig. 15, and choose a more convenient gauge
for obtaining the $\pi$-flux (this enables an efficient implementation of
lattice symmetries). We obtain four bands, each of which is two-fold
degenerate. At quarter filling, we obtain two Dirac cones located at the
$\Gamma$ point of the hexagonal Brillouin zone as shown in Fig. 14. The low
energy physics is described by a four-component spinor $\chi_{\pi}$ with a
Lagrangian density similar to Eq. (371) with,

$\gamma^{0}_{\pi}=M_{03},\gamma^{1}_{\pi}=-M_{02},\gamma^{2}_{\pi}=M_{01}$
(381)

where

$M_{\mu\nu}=\sigma_{\mu}\sigma_{\nu}.$ (382)

where $\sigma_{\mu}$ are Pauli matrices defined just below Eq. (373). The
system has an emergent global SU(2) symmetry generated by
$M_{i0},i\in\\{1,2,3\\}$ which is the analog of the chiral symmetry discussed
near Eq. (30).

A fermion bilinear of the form

$-i\bar{\chi}_{\pi}\gamma^{0}_{\pi}M_{\mu 3}\chi_{\pi}$ (383)

described by a mass matrix $M_{\mu 3},\mu\in\\{0,1,2,3\\}$ which anticommutes
with $\gamma^{1}_{\pi}$ and $\gamma^{2}_{\pi}$, gaps out the Dirac fermions.
The space of these mass matrices can be reduced in the irreps of the space
group, resulting the phases described below.

###  K.1 Integer Chern insulator

This mass transforms as a one-dimensional irrep, breaking time reversal and
reflection symmetries of the lattice while preserving all the proper
rotational symmetries of the hexagonal lattice

$\mathcal{A}_{2g}{}^{o}$ 1 $M_{03}$

resulting in a mass term

$\Delta_{\textup{ICI}}=-i\langle\bar{\chi}_{\pi}\chi_{\pi}\rangle.$ (384)

It is clear that the SU(4) symmetric ICI found in Eq. (72) is a “larger
dimensional” realization of such a phase with a larger value of the Chern-
Simons level.

###  K.2 Stripy density waves

The remainder of the three masses organize as a triplet under the space group
symmetries, preserving time reversal but breaking rotational and translational
symmetries.

$\mathcal{T}_{1g}{}^{e}$ 1
$\frac{1}{\sqrt{2}}M_{13}+\frac{1}{\sqrt{6}}M_{23}-\frac{1}{\sqrt{3}}M_{33}$ 2
$-\frac{1}{\sqrt{2}}M_{13}+\frac{1}{\sqrt{6}}M_{23}-\frac{1}{\sqrt{3}}M_{33}$
3 $\sqrt{\frac{2}{3}}M_{23}+\frac{1}{\sqrt{3}}M_{33}$

The three components of the masses correspond to the fermion bilinears

$\begin{split}-i\langle\bar{\chi}_{\pi}\left(\frac{1}{\sqrt{2}}M_{10}+\frac{1}{\sqrt{6}}M_{20}-\frac{1}{\sqrt{3}}M_{30}\right)\chi_{\pi}\rangle,\\\
-i\langle\bar{\chi}_{\pi}\left(-\frac{1}{\sqrt{2}}M_{10}+\frac{1}{\sqrt{6}}M_{20}-\frac{1}{\sqrt{3}}M_{30}\right)\chi_{\pi}\rangle,\\\
-i\langle\bar{\chi}_{\pi}\left(\sqrt{\frac{2}{3}}M_{20}+\frac{1}{\sqrt{3}}M_{30}\right)\chi_{\pi}\rangle\end{split}$
(385)

any one of which produces a stripy density wave similar to that shown in Fig.
6. The mass matrices that appear here are orthogonal linear combinations of
the chiral symmetry generators discussed just below Eq. (381). Indeed, it is
evident that the chiral masses shown in Eq. (76) correspond to this case.

##  Appendix L A model with $j=1/2$ spins

In this section, we construct a model on a honeycomb lattice where the spin-
orbit coupling is realized in a $j=1/2$ system, i. e., as a system with
spin-$1/2$ degrees of freedom. Although this model is not directly motivated
by a material system, it is nevertheless useful to study, in a simpler
setting, the conceptual underpinnings of how spin-orbit coupling produces
interesting new phenomena. The model is defined using Fig. 2 where each
lattice site has two $j=1/2$ orbitals. The hopping Hamiltonian is same as that
in Eq. (9), with the key difference that
$U_{\mathbf{r}\mathbf{r}^{\prime}}=\\{\tau_{x},\tau_{y},\tau_{z}\\}$
($\tau_{i}$ here are the Pauli matrices acting on the $j$-1/2 space)
respectively when $\mathbf{r}\mathbf{r}^{\prime}$ is the $x,y,z$ type link
shown in Fig. 2. This system has the following microscopic symmetries among
those listed in table. 5 and time reversal,

  1. 1.

Lattice translations as in ${\bf T}_{1}$ and ${\bf T}_{2}$

  2. 2.

${\bf C}_{3}$ rotations

  3. 3.

$\bm{\sigma}_{d}$ dihedral reflection

  4. 4.

Time reversal $\mathbb{T}$, with $\mathbb{T}^{2}=-1$

The interesting aspect of this model is that by carrying out transformations
similar to those discussed in appendix D, one can arrive at a system with a
$\pi$-flux through each honeycomb plaquette, and a global SU(2) flavor
symmetry. In other words, this model is the “SU(2) version” of the SU(4) model
discussed in the main text.

We continue to discuss this model in the global basis. The band structure of
this model is identical to that shown in Fig. 14, the difference being that
each band is non-degenerate. At quarter filling, the physics can be described
by four Dirac cones, one each located at
$\Gamma,\textup{M}_{1},\textup{M}_{2},\textup{M}_{3}$. We get a Dirac action
similar to Eq. (371), with $\chi_{g}$ as in Eq. (370) where $\chi_{g\nu}$,
$\nu\in\\{\Gamma,\textup{M}_{1},\textup{M}_{2},\textup{M}_{3}\\}$ are
2-component spinors. The Dirac gamma matrices are

$\gamma^{0}_{g}=M_{003},\gamma^{1}_{g}=M_{002},\gamma^{2}_{g}=-M_{001}$ (386)

and

$M_{\mu\nu\rho}=\sigma_{\mu}\sigma_{\nu}\sigma_{\tau}$ (387)

where Pauli matrices $\sigma_{\mu}$ are described just below Eq. (373)

The symmetry transformations are described by

$\begin{split}\Omega_{{\bf
C}_{3}}=&\frac{1}{8}\left(M_{000}+i\sqrt{3}M_{003}-M_{010}-i\sqrt{3}M_{013}\right.\\\
&+iM_{020}-\sqrt{3}M_{023}+M_{030}+i\sqrt{3}M_{033}-M_{100}\\\
&-i\sqrt{3}M_{103}+M_{110}+i\sqrt{3}M_{113}+iM_{120}\\\
&-\sqrt{3}M_{123}+M_{130}+i\sqrt{3}M_{133}-iM_{200}\\\
&+\sqrt{3}M_{203}-iM_{210}+\sqrt{3}M_{213}+M_{220}\\\
&+i\sqrt{3}M_{223}+iM_{230}-\sqrt{3}M_{233}+M_{300}\\\
&+i\sqrt{3}M_{303}+M_{310}+i\sqrt{3}M_{313}-iM_{320}\\\
&\left.+\sqrt{3}M_{323}+M_{330}+i\sqrt{3}M_{333}\right)\\\
\Omega_{\bm{\sigma}_{d}}=&\displaystyle{\frac{1}{2}\left(-M_{032}+M_{112}+M_{222}-M_{302}\right)}\\\
\Omega_{{\bf T}_{1}}=&\displaystyle{M_{300}}\\\ \Omega_{{\bf
T}_{2}}=&\displaystyle{M_{030}}\\\
\Omega_{\mathbb{T}}=&\displaystyle{iM_{002}}\end{split}$ (388)

Again, we see that the spinors $\chi_{g\Gamma}$ (see Eq. (376)) form an
invariant subspace under the action of the symmetries above. One, therefore,
expects to obtain semimetallic phases when the mass matrices of the type
$M_{\mu\nu 3},\mu,\nu\in\\{0,1,2,3\\}$ that gap out (see Eq. (375)) the Dirac
Lagrangian Eq. (371) defined by Eq. (386) are resolved into irreducible
components. Below we briefly describe seven irreducible masses and the
resulting phases obtained by such an analysis.

###  L.1 Chiral masses

####  L.1.1 Integer Chern insulator

${\bf C}_{3}$ $\bm{\sigma}_{d}$ ${\bf T}_{1}$ ${\bf T}_{2}$ $\mathbb{T}$ ✓ ✗ ✓
✓ ✗ $\mathcal{A}^{o}$ 1 $M_{003}$

This mass is SU(2) symmetric and produces a fully gapped state. Viewed on the
lattice, it produces spin-independent second neighbour hoppings akin to the
Haldane honeycomb model as in Fig. 5 with an effective Chern-Simons action
described by Eq. (73).

####  L.1.2 Stripy Density Wave Phase

${\bf C}_{3}$ $\bm{\sigma}_{d}$ ${\bf T}_{1}$ ${\bf T}_{2}$ $\mathbb{T}$ ✗ ✗ ✗
✗ ✓ $\mathcal{T}^{\text{e}}$ 1 $M_{123}$ 2 $-M_{203}$ 3 $-M_{323}$

This is again an SU(2) invariant mass that results in a stripy density wave
similar to the SU(4) invariant case found in Eq. (76).

The two cases described above exhaust the chiral masses.

###  L.2 SU(2) Flavor masses

####  L.2.1 Quantum dipolar Hall mass

${\bf C}_{3}$ $\bm{\sigma}_{d}$ ${\bf T}_{1}$ ${\bf T}_{2}$ $\mathbb{T}$ ✗ ✗ ✗
✗ ✓ $\mathcal{T}^{e}$ 1 $M_{213}$ 2 $-M_{233}$ 3 $-M_{023}$

This mass produces spin-dependent second neighbour hopping that produces a
uniform SU(2) flux in a second-neighbor triangle, gapping out the system. The
phase with this mass is described by a mutual Chern-Simons theory like Eq.
(83) resulting in dipole-filtered edge states and is analogous to the phase
discussed in section. VII.1.2.

###  L.3 Mixed masses

####  L.3.1 Ferromagnetic insulator

${\bf C}_{3}$ $\bm{\sigma}_{d}$ ${\bf T}_{1}$ ${\bf T}_{2}$ $\mathbb{T}$ ✓ ✗ ✓
✓ ✗ $\mathcal{A}^{o}$ 1
$\frac{M_{033}}{\sqrt{3}}+\frac{M_{303}}{\sqrt{3}}+\frac{M_{333}}{\sqrt{3}}$

This mass manifests as a uniform magnetic field in the direction perpendicular
to the honeycomb and fully gap out the Dirac fermions. The analogous state
correponding to this in the SU(4) case is discussed in section. VIII.1.2.

####  L.3.2 Ferromagnetic semimetal – $\Gamma$-Dirac Semimetal

${\bf C}_{3}$ $\bm{\sigma}_{d}$ ${\bf T}_{1}$ ${\bf T}_{2}$ $\mathbb{T}$ ✗ ✗ ✓
✓ ✗ $\mathcal{E}^{{o}}$ 1 $\frac{M_{033}}{\sqrt{2}}-\frac{M_{333}}{\sqrt{2}}$
2
$\frac{M_{033}}{\sqrt{6}}-\sqrt{\frac{2}{3}}M_{303}+\frac{M_{333}}{\sqrt{6}}$

The doublet mass produces a local magnetic field in the plane of the honeycomb
lattice and the components rotate into each other under lattice symmetries;
this phase is analogous to the ferro spin-octupolar semimetallic phase
discussed in section. VIII.2.5. For any generic linear combination of the
masses, the Dirac cone at the $\Gamma$ point remains ungapped, while for
special linear combinations of the two masses, there is one additional gapless
mode as discussed in section. VIII.2.5. This is a semimetallic phase of
$\Gamma$-DSM type.

####  L.3.3 Stripy spin density wave semimetal (M-Dirac Semimetal)

${\bf C}_{3}$ $\bm{\sigma}_{d}$ ${\bf T}_{1}$ ${\bf T}_{2}$ $\mathbb{T}$ ✗ ✗ ✗
✗ ✗ $\mathcal{T}^{{o}}$ 1 $\frac{M_{113}}{\sqrt{2}}-\frac{M_{223}}{\sqrt{2}}$
2 $-\frac{M_{103}}{\sqrt{2}}-\frac{M_{133}}{\sqrt{2}}$ 3
$-\frac{M_{013}}{\sqrt{2}}-\frac{M_{313}}{\sqrt{2}}$

This mass produces a spin density wave of the stripy kind similar to that
discussed in Eq. (172). Interestingly, this produces a semimetallic phase of
the M-DSM kind, precisely as discussed for the SU(4) case in Eq. (172).

####  L.3.4 Stripy spin density wave semimetal – ($\Gamma$-Dirac Semimetal)

${\bf C}_{3}$ $\bm{\sigma}_{d}$ ${\bf T}_{1}$ ${\bf T}_{2}$ $\mathbb{T}$ ✗ ✗ ✗
✗ ✗ $\text{T}_{\text{u}}{}^{\text{o}}$ 1
$\frac{M_{113}}{\sqrt{2}}+\frac{M_{223}}{\sqrt{2}}$ 2
$\frac{M_{133}}{\sqrt{2}}-\frac{M_{103}}{\sqrt{2}}$ 3
$\frac{M_{313}}{\sqrt{2}}-\frac{M_{013}}{\sqrt{2}}$

This is a triplet mass that produces a stripy magnetic field; the key
difference between the one just discussed above, is that this possesses an
isolated Dirac cone, where a single Dirac cone at $\Gamma$ is always left
ungapped. This is similar to the SU(4) case discussed in Eq. (VIII.2.2) that
produces a stripy spin-octupolar density wave. For the SU(2) case one has a
stripy density wave. Again, just as in the case discussed in Eq. (VIII.2.2),
there are special linear combinations of the masses that obtain additional
gapless modes.

## References

  * Vafek and Vishwanath (2014) O. Vafek and A. Vishwanath, Annual Review of Condensed Matter Physics 5, 83 (2014).
  * Goerbig and Montambaux (2017) M. Goerbig and G. Montambaux, “Dirac fermions in condensed matter and beyond,” in  _Dirac Matter_, edited by B. Duplantier, V. Rivasseau, and J.-N. Fuchs (Springer International Publishing, Cham, 2017) pp. 25–53.
  * Wallace (1947) P. R. Wallace, Phys. Rev. 71, 622 (1947).
  * Novoselov  _et al._ (2005) K. S. Novoselov, A. K. Geim, S. V. Morozov, D. Jiang, M. I. Katsnelson, I. Grigorieva, S. Dubonos, and a. Firsov, Nature 438, 197 (2005).
  * Zhang  _et al._ (2005) Y. Zhang, Y.-W. Tan, H. L. Stormer, and P. Kim, Nature 438, 201 (2005).
  * Geim (2009) A. K. Geim, Science 324, 1530 (2009).
  * Castro Neto  _et al._ (2009) A. H. Castro Neto, F. Guinea, N. M. R. Peres, K. S. Novoselov, and A. K. Geim, Rev. Mod. Phys. 81, 109 (2009).
  * Goerbig (2011) M. O. Goerbig, Rev. Mod. Phys. 83, 1193 (2011).
  * Das Sarma  _et al._ (2011) S. Das Sarma, S. Adam, E. H. Hwang, and E. Rossi, Rev. Mod. Phys. 83, 407 (2011).
  * Novoselov (2011) K. S. Novoselov, Rev. Mod. Phys. 83, 837 (2011).
  * Geim (2011) A. K. Geim, Rev. Mod. Phys. 83, 851 (2011).
  * Wehling  _et al._ (2014) T. Wehling, A. Black-Schaffer, and A. Balatsky, Advances in Physics 63, 1 (2014).
  * Young and Kane (2015) S. M. Young and C. L. Kane, Phys. Rev. Lett. 115, 126803 (2015).
  * Hirata  _et al._ (2021) M. Hirata, A. Kobayashi, C. Berthier, and K. Kanoda, Reports on Progress in Physics 84, 036502 (2021).
  * Kobayashi  _et al._ (2007) A. Kobayashi, S. Katayama, Y. Suzumura, and H. Fukuyama, Journal of the Physical Society of Japan 76, 034711 (2007).
  * Goerbig  _et al._ (2008) M. O. Goerbig, J.-N. Fuchs, G. Montambaux, and F. Piéchon, Phys. Rev. B 78, 045415 (2008).
  * Wollman  _et al._ (1995) D. A. Wollman, D. J. Van Harlingen, J. Giapintzakis, and D. M. Ginsberg, Phys. Rev. Lett. 74, 797 (1995).
  * Kirtley  _et al._ (1995) J. Kirtley, C. Tsuei, J. Sun, C. Chi, L. S. Yu-Jahnes, A. Gupta, M. Rupp, and M. Ketchen, Nature 373, 225 (1995).
  * Volovik (1993) G. Volovik, Jetp Letters 58, 469 (1993).
  * Wang and MacDonald (1995) Y. Wang and A. H. MacDonald, Phys. Rev. B 52, R3876 (1995).
  * Balents  _et al._ (1998) L. Balents, M. P. Fisher, and C. Nayak, International Journal of Modern Physics B 12, 1033 (1998).
  * Wan  _et al._ (2011) X. Wan, A. M. Turner, A. Vishwanath, and S. Y. Savrasov, Physical Review B 83, 205101 (2011).
  * Young  _et al._ (2012) S. M. Young, S. Zaheer, J. C. Y. Teo, C. L. Kane, E. J. Mele, and A. M. Rappe, Phys. Rev. Lett. 108, 140405 (2012).
  * Moore and Balents (2007) J. E. Moore and L. Balents, Phys. Rev. B 75, 121306 (2007).
  * Roy (2009) R. Roy, Phys. Rev. B 79, 195322 (2009).
  * Fu  _et al._ (2007) L. Fu, C. L. Kane, and E. J. Mele, Phys. Rev. Lett. 98, 106803 (2007).
  * Hasan and Kane (2010) M. Z. Hasan and C. L. Kane, Rev. Mod. Phys. 82, 3045 (2010).
  * Qi and Zhang (2011) X.-L. Qi and S.-C. Zhang, Rev. Mod. Phys. 83, 1057 (2011).
  * Chen  _et al._ (2009) Y. Chen, J. G. Analytis, J.-H. Chu, Z. Liu, S.-K. Mo, X.-L. Qi, H. Zhang, D. Lu, X. Dai, Z. Fang, _et al._ , Science 325, 178 (2009).
  * Pacile  _et al._ (2008) D. Pacile, J. Meyer, Ç. Ö. Girit, and A. Zettl, Applied Physics Letters 92, 133107 (2008).
  * Li  _et al._ (2009) G. Li, A. Luican, and E. Y. Andrei, Physical review letters 102, 176804 (2009).
  * Nair  _et al._ (2008) R. R. Nair, P. Blake, A. N. Grigorenko, K. S. Novoselov, T. J. Booth, T. Stauber, N. M. Peres, and A. K. Geim, Science 320, 1308 (2008).
  * Li  _et al._ (2008) Z. Li, E. A. Henriksen, Z. Jiang, Z. Hao, M. C. Martin, P. Kim, H. L. Stormer, and D. N. Basov, Nature physics 4, 532 (2008).
  * Levy  _et al._ (2010) N. Levy, S. Burke, K. Meaker, M. Panlasigui, A. Zettl, F. Guinea, A. C. Neto, and M. F. Crommie, Science 329, 544 (2010).
  * Checkelsky  _et al._ (2009) J. G. Checkelsky, L. Li, and N. P. Ong, Phys. Rev. B 79, 115434 (2009).
  * Bardarson and Moore (2013) J. H. Bardarson and J. E. Moore, Reports on Progress in Physics 76, 056501 (2013).
  * Hermele  _et al._ (2005) M. Hermele, T. Senthil, and M. P. A. Fisher, Phys. Rev. B 72, 104404 (2005).
  * Affleck and Marston (1988) I. Affleck and J. B. Marston, Physical Review B 37, 3774 (1988).
  * Marston and Affleck (1989) J. B. Marston and I. Affleck, Physical Review B 39, 11538 (1989).
  * Rantner and Wen (2001) W. Rantner and X.-G. Wen, Phys. Rev. Lett. 86, 3871 (2001).
  * Wen (2002) X.-G. Wen, Physical Review B 65, 165113 (2002).
  * Hermele  _et al._ (2004) M. Hermele, T. Senthil, M. P. Fisher, P. A. Lee, N. Nagaosa, and X.-G. Wen, Physical Review B 70, 214437 (2004).
  * Franz and Tešanović (2001) M. Franz and Z. Tešanović, Phys. Rev. Lett. 87, 257003 (2001).
  * Rantner and Wen (2002) W. Rantner and X.-G. Wen, Phys. Rev. B 66, 144501 (2002).
  * Son (2015) D. T. Son, Phys. Rev. X 5, 031027 (2015).
  * Boyack  _et al._ (2021) R. Boyack, H. Yerzhakov, and J. Maciejko, The European Physical Journal Special Topics 230, 979 (2021).
  * Zhou  _et al._ (2007) S. Y. Zhou, G.-H. Gweon, A. Fedorov, d. First, PN, W. De Heer, D.-H. Lee, F. Guinea, A. Castro Neto, and A. Lanzara, Nature materials 6, 770 (2007).
  * Note (1) This phase can have other gapless mode such as Goldstone boson if a continuous symmetry is broken.
  * Ryu  _et al._ (2009) S. Ryu, C. Mudry, C.-Y. Hou, and C. Chamon, Physical Review B 80, 205319 (2009).
  * Herbut  _et al._ (2009a) I. F. Herbut, V. Juričić, and B. Roy, Phys. Rev. B 79, 085116 (2009a).
  * Tanaka and Hu (2005) A. Tanaka and X. Hu, Phys. Rev. Lett. 95, 036402 (2005).
  * Gross and Neveu (1974) D. J. Gross and A. Neveu, Phys. Rev. D 10, 3235 (1974).
  * Zinn-Justin (1991) J. Zinn-Justin, Nuclear Physics B 367, 105 (1991).
  * Sachdev (1999) S. Sachdev, Physics world 12, 33 (1999).
  * Herbut (2006) I. F. Herbut, Phys. Rev. Lett. 97, 146401 (2006).
  * Senthil  _et al._ (2004) T. Senthil, A. Vishwanath, L. Balents, S. Sachdev, and M. P. Fisher, Science 303, 1490 (2004).
  * Senthil and Fisher (2006) T. Senthil and M. P. A. Fisher, Phys. Rev. B 74, 064405 (2006).
  * Kane and Mele (2005) C. L. Kane and E. J. Mele, Phys. Rev. Lett. 95, 226801 (2005).
  * Haldane (1988) F. D. M. Haldane, Phys. Rev. Lett. 61, 2015 (1988).
  * Witczak-Krempa  _et al._ (2013) W. Witczak-Krempa, G. Chen, Y. B. Kim, and L. Balents, arXiv preprint arXiv:1305.2193 (2013).
  * Raghu  _et al._ (2008) S. Raghu, X.-L. Qi, C. Honerkamp, and S.-C. Zhang, Phys. Rev. Lett. 100, 156401 (2008).
  * Grover and Senthil (2008) T. Grover and T. Senthil, Phys. Rev. Lett. 100, 156804 (2008).
  * Herbut  _et al._ (2009b) I. F. Herbut, V. Juričić, and O. Vafek, Phys. Rev. B 80, 075432 (2009b).
  * Seifert  _et al._ (2020) U. F. P. Seifert, X.-Y. Dong, S. Chulliparambil, M. Vojta, H.-H. Tu, and L. Janssen, Phys. Rev. Lett. 125, 257202 (2020).
  * Liu  _et al._ (2022) Z. H. Liu, M. Vojta, F. F. Assaad, and L. Janssen, Phys. Rev. Lett. 128, 087201 (2022).
  * Bi and Senthil (2019) Z. Bi and T. Senthil, Phys. Rev. X 9, 021034 (2019).
  * Abanov and Wiegmann (2000) A. Abanov and P. B. Wiegmann, Nuclear Physics B 570, 685 (2000).
  * Singh and Gegenwart (2010) Y. Singh and P. Gegenwart, Phys. Rev. B 82, 064412 (2010).
  * Liu  _et al._ (2011) X. Liu, T. Berlijn, W.-G. Yin, W. Ku, A. Tsvelik, Y.-J. Kim, H. Gretarsson, Y. Singh, P. Gegenwart, and J. P. Hill, Phys. Rev. B 83, 220403 (2011).
  * Singh  _et al._ (2012) Y. Singh, S. Manni, J. Reuther, T. Berlijn, R. Thomale, W. Ku, S. Trebst, and P. Gegenwart, Phys. Rev. Lett. 108, 127203 (2012).
  * Banerjee  _et al._ (2016) A. Banerjee, C. Bridges, J.-Q. Yan, A. Aczel, L. Li, M. Stone, G. Granroth, M. Lumsden, Y. Yiu, J. Knolle, _et al._ , Nature materials 15, 733 (2016).
  * Sandilands  _et al._ (2015) L. J. Sandilands, Y. Tian, K. W. Plumb, Y.-J. Kim, and K. S. Burch, Physical review letters 114, 147201 (2015).
  * Plumb  _et al._ (2014) K. Plumb, J. Clancy, L. Sandilands, V. V. Shankar, Y. Hu, K. Burch, H.-Y. Kee, and Y.-J. Kim, Physical Review B 90, 041112 (2014).
  * Choi  _et al._ (2012) S. Choi, R. Coldea, A. Kolmogorov, T. Lancaster, I. Mazin, S. Blundell, P. Radaelli, Y. Singh, P. Gegenwart, K. Choi, _et al._ , Physical review letters 108, 127204 (2012).
  * Swaroop and Flengas (1964a) B. Swaroop and S. Flengas, Canadian Journal of Physics 42, 1886 (1964a).
  * Swaroop and Flengas (1964b) B. Swaroop and S. Flengas, Canadian Journal of Chemistry 42, 1495 (1964b).
  * Yamada  _et al._ (2018) M. G. Yamada, M. Oshikawa, and G. Jackeli, Physical Review Letters 121, 097201 (2018).
  * Ushakov  _et al._ (2020) A. V. Ushakov, I. Solovyev, and S. V. Streltsov, JETP Letters 112, 642 (2020).
  * Kaplan (2009) D. B. Kaplan, Modern Perspectives in Lattice QCD: Quantum Field Theory and High Performance Computing. Proceedings, International School, 93rd Session , 223 (2009).
  * Semenoff (1984) G. W. Semenoff, Phys. Rev. Lett. 53, 2449 (1984).
  * Moshe and Zinn-Justin (2003) M. Moshe and J. Zinn-Justin, Physics Reports 385, 69 (2003).
  * Romhányi  _et al._ (2017) J. Romhányi, L. Balents, and G. Jackeli, Phys. Rev. Lett. 118, 217202 (2017).
  * Kim  _et al._ (2008) B. J. Kim, H. Jin, S. J. Moon, J.-Y. Kim, B.-G. Park, C. S. Leem, J. Yu, T. W. Noh, C. Kim, S.-J. Oh, J.-H. Park, V. Durairaj, G. Cao, and E. Rotenberg, Phys. Rev. Lett. 101, 076402 (2008).
  * Kitaev (2006) A. Kitaev, Annals of Physics 321, 2 (2006).
  * Nanda  _et al._ (2020) A. Nanda, K. Dhochak, and S. Bhattacharjee, Phys. Rev. B 102, 235124 (2020).
  * Bercx  _et al._ (2014) M. Bercx, M. Hohenadler, and F. F. Assaad, Phys. Rev. B 90, 075140 (2014).
  * (87) A. Chaubey, B. Mandal, S. Bhattacharjee, and V. B. Shenoy, Unpublished.
  * Hou  _et al._ (2007) C.-Y. Hou, C. Chamon, and C. Mudry, Phys. Rev. Lett. 98, 186809 (2007).
  * Qi  _et al._ (2006) X.-L. Qi, Y.-S. Wu, and S.-C. Zhang, Phys. Rev. B 74, 085308 (2006).
  * Thouless  _et al._ (1982) D. J. Thouless, M. Kohmoto, M. P. Nightingale, and M. den Nijs, Phys. Rev. Lett. 49, 405 (1982).
  * Callan Jr and Harvey (1985) C. G. Callan Jr and J. A. Harvey, Nuclear Physics B 250, 427 (1985).
  * Berg  _et al._ (2009) E. Berg, E. Fradkin, and S. A. Kivelson, Nature Physics 5, 830 (2009).
  * Senthil and Fisher (2000) T. Senthil and M. P. Fisher, Physical Review B 62, 7850 (2000).
  * Chamon  _et al._ (2008) C. Chamon, C.-Y. Hou, R. Jackiw, C. Mudry, S.-Y. Pi, and A. P. Schnyder, Phys. Rev. Lett. 100, 110405 (2008).
  * Ran  _et al._ (2008) Y. Ran, A. Vishwanath, and D.-H. Lee, arXiv preprint arXiv:0806.2321 (2008).
  * Sbaih  _et al._ (2013) M. A. Sbaih, M. K. Srour, M. Hamada, and H. Fayad, Electronic Journal of Theoretical Physics 28 (2013).
  * Unukovych and Sotnikov (2021) V. Unukovych and A. Sotnikov, Phys. Rev. B 104, 245106 (2021).
  * Yamada  _et al._ (2021) M. G. Yamada, M. Oshikawa, and G. Jackeli, Phys. Rev. B 104, 224436 (2021).
  * Lajkó  _et al._ (2017) M. Lajkó, K. Wamer, F. Mila, and I. Affleck, Nuclear Physics B 924, 508 (2017).
  * Lieb  _et al._ (1961) E. Lieb, T. Schultz, and D. Mattis, Annals of Physics 16, 407 (1961).
  * Affleck and Lieb (1986) I. Affleck and E. H. Lieb, Letters in Mathematical Physics 12, 57 (1986).
  * Affleck (1988) I. Affleck, Physical Review B 37, 5186 (1988).
  * Oshikawa (2000) M. Oshikawa, Phys. Rev. Lett. 84, 1535 (2000).
  * Hastings (2005) M. Hastings, Europhysics Letters 70, 824 (2005).
  * Hasegawa  _et al._ (2006) Y. Hasegawa, R. Konno, H. Nakano, and M. Kohmoto, Phys. Rev. B 74, 033413 (2006).
  * Corboz  _et al._ (2012) P. Corboz, M. Lajkó, A. M. Läuchli, K. Penc, and F. Mila, Phys. Rev. X 2, 041013 (2012).
  * Calvera and Wang (2021) V. Calvera and C. Wang, arXiv preprint arXiv:2103.13405 (2021).
  * Florens and Georges (2004) S. Florens and A. Georges, Phys. Rev. B 70, 035114 (2004).
  * Murakami  _et al._ (2004) S. Murakami, N. Nagosa, and S.-C. Zhang, Phys. Rev. B 69, 235206 (2004).