# Universal image segmentation for optical identification of 2D materials

Randy M. Sterbentz  Department of Physics and Astronomy, University of Nevada
Las Vegas, Las Vegas, Nevada 89154, USA  Kristine L. Haley  Department of
Physics and Astronomy, University of Nevada Las Vegas, Las Vegas, Nevada
89154, USA  Joshua O. Island  Department of Physics and Astronomy, University
of Nevada Las Vegas, Las Vegas, Nevada 89154, USA<EMAIL_ADDRESS>
###### Abstract

Machine learning methods are changing the way data is analyzed. One of the
most powerful and widespread applications of these techniques is in image
segmentation wherein disparate objects of a digital image are partitioned and
classified. Here we present an image segmentation program incorporating a
series of unsupervised clustering algorithms for the automatic thickness
identification of two-dimensional materials from digital optical microscopy
images. The program identifies mono- and few-layer flakes of a variety of
materials on both opaque and transparent substrates with a pixel accuracy of
roughly 95%. Contrasting with previous attempts, application generality is
achieved through preservation and analysis of all three digital color channels
and Gaussian mixture model fits to arbitrarily shaped data clusters. Our
results provide a facile implementation of data clustering for the universal,
automatic identification of two-dimensional materials exfoliated onto any
substrate.

##  1 Introduction

Image segmentation has become an essential technique in fields from medical
imaging[1, 2, 3] and autonomous driving[4] to robotic perception[5] and image
compression[6, 7, 8]. Through unsupervised segmentation of large data sets,
trained algorithms can recognize and predict elements of new images. An
appealing application of image segmentation is in the thickness identification
of two-dimensional (2D) materials from their digital optical microscopy
images. Current flake detection methods rely heavily on identification by
trained researchers, a human-learning process, in which flakes are identified
by their contrast difference on a substrate after significant trial and error.
Automatic thickness identification would relieve this tedious, time-consuming
screening process and possibly improve identification accuracy.

The easiest implementation of image segmentation for 2D materials is by
thresholding. This is performed by analyzing image contrast, from reflectance
or transmittance for example, and partitioning regions of an image based on
contrast level difference. This technique has been widely and successfully
employed in the identification and characterization of exfoliated 2D
materials[9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]. Thresholding techniques,
while easily implemented, suffer from inaccuracy when contrast differences
become relatively small, for example a single layer of graphene on a
silicon/silicon dioxide (Si/SiO2) substrate, and can be highly dependent on
precise experimental conditions hindering universal application.

Recently, a variety of more advanced machine learning techniques have emerged
to automate and improve the process of identifying exfoliated 2D materials[20,
21, 22, 23, 24, 25, 26, 27, 28, 29, 30]. These include techniques based on
neural networks and data clustering but have been primarily applied to opaque
substrates and almost entirely to standard Si/SiO2. Transparent substrates are
commonly used for exfoliation or experiments on 2D materials[31, 32, 33, 34,
35, 11]. A method that can be applied to identify the thickness of any
material on any substrate is highly desirable.

Here we present an open source program[36] written in Python 3 to
automatically identify the thickness of exfoliated 2D flakes which can be
universally applied to different materials and substrates. We combine three
well-established clustering techniques to form a training script to segment
layers of a flake, manually label the layers, and then use that training to
test thicknesses of other flakes. The program presents roughly a 95% pixel
accuracy for graphene and transition metal dichalcogenides on silicon/silicon
dioxide and polydimethylsiloxane (PDMS) substrates. Importantly, no change to
the program’s adjustable parameters are needed to identify different materials
on different substrates, allowing simple and universal application to any
material/substrate combination.

##  2 Overview of the program

An overview of the program applied to graphene on Si/SiO2 is shown in Figure
1. The training stage begins with a set of optical microscopy images cropped
to few-layer flakes whose thicknesses are determined using optical contrast
methods[9]. Figure 1(a) shows a cropped optical image of a few-layer graphene
flake on a Si/SiO2 (300 nm SiO2) substrate with each layer thickness labeled
(1-4 layers). A scatter plot of the red, green, and blue (RGB) channel values
(normalized to a range 0-1) for each pixel in the image is shown in Figure
1(b). The data points have been colored according to their RGB values. At this
stage, the scatter plot shows only broad features that can be generally
associated with the substrate (pink) and few-layer graphene (purple) but no
clear correspondence to individual thicknesses can be made. The raw image is
preprocessed using a bilateral filter to reduce noise and a background
normalization using a planar fit. The result, after compression to roughly
10,000 total pixels, is shown in Figure 1(c). Preprocessing reveals the
individual clusters of data in the scatter plot (Figure 1(d)) associated with
the substrate and flake layers.

The location and distribution of these clusters in RGB space are found using a
series of unsupervised clustering techniques summarized here and detailed
further below. The centers are first located using mean shift and density-
based spatial clustering. Once the centers are identified, fit characteristics
such as the weight, mean position, and distribution of each cluster are found
using a Gaussian mixture model. An image of the result of the fitting
algorithm is shown in Figure 1(e). The pixels in the color plot have been
colored according to the fit results and the scatter plot (Figure 1(f)) shows
these fit assignments in more detail. Once the cluster characteristics are
extrapolated for several training images, a master catalogue is created that
ties the fit clusters to the predetermined flake thicknesses in our training
images.

To determined accuracy of the training, we test the master catalogue on a set
of images with identified thicknesses. An example of the testing results for
graphene on Si/SiO2 is shown in Figures 1(g-j). In the testing stage,
untrained optical images (Figure 1(g-h)) are first preprocessed using the same
procedure as in training but without cropping. The preprocessed images are
then checked against the master catalogue for flake thickness assignment given
the pixel location in RGB space. Figure 1(i) shows the result (cropped to show
detail of flake of interest) with layer thicknesses identified by the color
bar. The associated scatter plot in Figure 1(j) shows the corresponding
clusters and layer assignments. In the following section we detail the
implemenation of the clustering algorithms before presenting results of our
program performed on other materials and substrates.

Figure 1:  Overview of the program composed of training and testing stages.
(a-b) Raw optical image of a few-layer graphene flake on 300 nm of SiO2 (a)
and its corresponding scatter plot of each pixel in RGB color space (b). The
scatter plot data points are colored to match their RGB value. (c-d) The
result after preprocessing the image in panel (a). The preprocessing reveals
well-defined clusters in the associated scatter plot (d). (e-f) A color plot
of pixel-cluster association (e) and corresponding scatter plot (f). Each
pixel is colored based on its most probable data cluster identity. (g-h) Raw
optical image of few-layer graphene flake with unknown thickness (g) used for
testing and its corresponding RGB scatter plot (h). (i-j) Crop of panel (g)
around the flake of interest (i) and corresponding scatter plot after the
testing stage (j).

##  3 Unsupervised clustering algorithms

Our training script incorporates three clustering algorithms to identify the
center of the data clusters and fit their distributions. Without explicitly
knowing the number of clusters (layers) in the image, the script begins with
an unsupervised method of determining the seed number. We use mean shift[37,
38, 39] and density-based[40, 41] algorithms to first find these cluster
centers which are then fed to a Gaussian mixture model for fitting arbitrary
ellipsoidal distributions.

Mean shift is an unsupervised machine learning algorithm that locates centers
of high data density. The algorithm begins by populating color space with an
array of points, referred to as “mean points” ($\vec{\rho}_{k}$). Figure 2(a)
shows the same data as in Figure 1(d) but with red closed circles indicating
the initial positions of the equally spaced mean points (an $8\times 8\times
8$ array). The next step groups all data points within a defined radius
($\epsilon$) of each mean point together. We define $\epsilon$ to be just
large enough to overlap with its nearest neighbors. The average location of
the data pixels within $\epsilon$ of a given mean point becomes the new
position of that mean point ($\vec{\rho}_{k}^{\prime}$) after one iteration of
the algorithm. This is calculated by:

$\vec{\rho}_{k}^{\prime}=\frac{1}{M}\sum_{i}{\vec{x}_{i}}\textrm{ for
}|\vec{x}_{i}-\vec{\rho}_{k}|<\epsilon,$ (1)

where $\vec{x}_{i}$ is the position of each data point and $M$ is the total
number of data points within $\epsilon$ of $\vec{\rho}_{k}$. In this way, each
mean point gradually shifts towards higher densities of data. Figure 2(b)
shows one iteration of the algorithm. Several points have moved to their new
mean positions according to the data within $\epsilon$ of each
$\vec{\rho}_{k}$.

Mean shift is computationally slow, having to calculate the distance between
every data point ($\approx$10,000 pixels) and every mean point (initially
512). To increase efficiency, mean points that have no data within $\epsilon$
after the first cycle, and thus make no contribution towards a data cluster,
are deleted. Additionally, mean points may approach their local maximum at
different rates. Per mean point, as soon as the number of data points within
their radius starts to decrease, they are turned off and no longer involved in
future calculations. Figure 2(c) shows the final state of the algorithm where
all mean points have converged to their local density maxima. After this,
outliers are removed before moving to the next algorithm.

Once mean shift is complete, several mean points will themselves be clustered
in color space and some mean points will have converged to outliers. Due to
the ellipsoidal shape of the clusters in RGB space after preprocessing, the
mean points will tend to lie along lines. An efficient algorithm for grouping
these lines is Density-Based Spatial Clustering of Applications with Noise
(DBSCAN)[40, 41]. DBSCAN groups data together by following the trajectory of
nearby points. The algorithm starts by “visiting” a random mean point. A
radius (we find $\epsilon/2$ works well) around it is checked for other mean
points. If none are found, the starting point is labeled an outlier. If there
are neighbors, they are grouped together. One of the other points in this
group is visited next, checking the same radius around itself to find new
points to add to the group. This repeats until no new points are added to the
group and every point within the group has been visited. Once the group is
finished, a new group starts at a randomly chosen mean point and the process
repeats. The centers of each group are found by averaging their respective
mean points. Figure 2(d) shows the result after running the DBSCAN algorithm
on the mean points in Figure 2(c).

Figure 2:  Mean shift and DBSCAN clustering for identification of cluster
centers. (a) The same data as in Figure 1(d) with red closed circles showing
the initial positions of the mean points in the mean shift algorithm. (b)
Scatter plot after one cycle of mean shift; outlier mean points have been
deleted and others have moved towards their local density maxima. (c) The
final state of the mean points after they have converged to their maxima. (d)
The RGB pixels plotted with the identified cluster centers (colored closed
circles) after the DBSCAN algorithm.

The combination of mean shift and DBSCAN presents an unsupervised method of
determining how many clusters are in a given image and their centers.
Following this step, this information can be used to seed a more powerful
clustering technique for data with ellipsoidal distributions. The popular
K-means clustering technique[42, 43, 44] for example is undesirable here as it
assumes spherical clusters. Instead, we use a multivariate Gaussian Mixture
Model (GMM)[45, 46, 47, 48, 49, 2] that allows fitting of data with arbitrary
normal distributions. This expands application of the program by automatically
handling new materials and substrate combinations that may have different
cluster distributions in RGB space.

In the GMM, each fitting ellipsoid has three characteristics developing
throughout the process: the weight ($\phi_{k}$, defining the number of data
points near ellipsoid $k$), the centroid ($\vec{\mu}_{k}$, defining the mean
of the data points belonging to ellipsoid $k$), and the covariance matrix
($\Sigma_{k}$, defining the shape and orientation of ellipsoid $k$ in RGB
space). These characteristics are used to calculate the probability
$\gamma_{ik}$ of a data point ${\vec{x}_{i}}$ belonging to ellipsoid $k$. This
probability is given by:

$\gamma_{ik}=\frac{\phi_{k}\mathcal{N}(\vec{x}_{i},\vec{\mu}_{k},\Sigma_{k})}{\sum_{j=1}^{K}\phi_{j}\mathcal{N}(\vec{x}_{i},\vec{\mu}_{j},\Sigma_{j})},$
(2)

where $K$ is the total number of clusters and $\mathcal{N}$ is the three-
variable (for 3-dimensional RGB space) Gaussian distribution given by:

$\mathcal{N}(\vec{x}_{i},\vec{\mu}_{k},\Sigma_{k})=[(2\pi)^{3}|\Sigma_{k}|e^{(\vec{x}_{i}-\vec{\mu}_{k})^{T}\Sigma_{k}^{-1}(\vec{x}_{i}-\vec{\mu}_{k})}]^{-\frac{1}{2}}$
(3)

The weights, means, and covariance matrices used in these relations are
calculated through:

$\phi_{k}=\frac{1}{N}\sum_{i=1}^{N}\gamma_{ik},$ (4)

$\vec{\mu}_{k}=\frac{\Sigma^{N}_{i=1}\gamma_{ik}\vec{x}_{i}}{\Sigma^{N}_{i=1}\gamma_{ik}},$
(5)

$\Sigma_{k}=\frac{\sum_{i=1}^{N}\gamma_{ik}|\vec{x}_{i}-\vec{\mu}_{k}|^{2}}{\sum_{i=1}^{N}\gamma_{ik}}.$
(6)

First we initialize each of the fitting ellipsoids by setting all initial
weights to $1/K$. The centroids are taken directly from the results of DBSCAN
$\vec{\mu}=\vec{\rho}$. The covariance matrices are initialized from the
centroids using Equation 6 with $\gamma_{ik}=1$. Figure 3(a) shows the
initialization of the fitting ellipsoids for our example few-layer graphene
data set from Figure 1(d) and Figure 2. The ellipsoids have been scaled to a
95% confidence level.

An unsupervised machine learning algorithm, referred to as expectation-
maximization (EM), is used to further optimize the ellipsoid parameters and
fit the data. The expectation step determines $\gamma_{ik}$ based on the
initialized weights, centroids, and covariance matrices calculated above. The
maximization step uses these probabilities to re-calculate each cluster’s
weight, centroid, and covariance matrix. These two steps iterate and gradually
the ellipsoid parameters converge. Figure 3(b) shows the algorithm results
after 2 cycles and Figure 3(c) shows the results after 30 cycles. After 30
cycles, the ellipsoids resemble the distributions of the data with several
small tight ellipsoids corresponding to the substrate and 1-4 layers of
graphene, and two larger ellipsoids (purple and blue) accounting for noise.
The max change of all cluster’s weights between maximization steps
($\Delta\phi_{k}<0.0001$) is used to define convergence and end the algorithm.
Figure 3(d) shows the results of the algorithm after convergence (total 61
cycles) for this data set. Note that the large purple and blue ellipsoids are
a product of over-fitting the data (fitting 7 ellipsoids to 5 data clusters).
These ellipsoids do not contribute to the master catalogue, but are important
for fitting data points associated with thicker layers ($>4$) and outliers.
The over fitting also allows the primary ellipsoids to confine themselves to
the core of their data clusters. Once convergence has been reached, only
ellipsoids that fit well to known layer thicknesses are added to a catalogue.

The training process is repeated for multiple flakes of the same material and
substrate (we trained around 5 for each material/substrate combination),
saving their ellipsoid characteristics into the same catalogue. A master
catalogue is then created by averaging together the characteristics of
ellipsoids with like-thickness. This master catalogue is the tool with which
we can test other images to determine their flake layer thicknesses (Figure
1(i-j)).

Figure 3:  Cluster fitting with a Gaussian mixture model (GMM). The scatter
plot from Figure 1(d) is superimposed with 95% confidence ellipsoids based on
the fit characteristics of the GMM-EM algorithm. (a) The initialized
ellipsoids show little correspondence to the underlying data. (b) After two
cycles of expectation-maximization, the ellipsoids better resemble the data
clusters. (c) After 30 cycles, some ellipsoids have nearly converged on their
data clusters. (d) The convergence condition is reached after 61 cycles.

##  4 General application to other materials and substrates

Our script can be universally applied to the identification of other 2D
material thicknesses on opaque and transparent substrates. This generality is
achieved by analyzing all three dimensions of the color-space data and fitting
the resulting clusters of arbitrary shape with our GMM-EM algorithm.
Importantly, no change in the adjustable parameters ($\epsilon$ or GMM
convergence) are required for the following results.

Figure 4 displays the power of this generality by identifying the layer
thickness of two additional materials, molybdenum disulfide (MoS2) and
molybdenum diselenide (MoSe2), on opaque (Si/SiO2) and transparent
(polydimethylsiloxane (PDMS)) substrates. MoS2 on Si/SiO2 (Figure 4(a-e))
presents clusters very similar to those of graphene on Si/SiO2 but they are
separated further in RGB space (Figure 4(b)). Further training for this
material/substrate combination would improve our testing results which only
identify layer thicknesses of 1 and 2 (Figure 4(d-e)). From the covariance
matrices we note that while all the data clusters are technically triaxial
ellipsoids (none of the semi-axes are equal), the clusters for materials on
Si/SiO2 are roughly prolate spheroids with one axis (blue) an order of
magnitude larger than the other two semi-axes (red and green).

MoS2 on PDMS (Figure 4(f-j)) presents clusters again extending along the blue
axis, though not as strongly as materials on Si/SiO2. The clusters are
similarly well-separated in RGB space as they are for MoS2 on Si/SiO2. Testing
for this set identifies monolayer, bilayer and trilayer thicknesses (Figure
4(j)). Finally, MoSe2 on PDMS presents the most spherical ellipsoids of our
investigation still slightly extending along blue, (Figure 4(k-o)) and mono-
through trilayer thicknesses are easily identified (Figure 4(o)).

##  5 Discussion

Our investigation focuses on the development of a program that can be
universally applied to different 2D materials and substrates. This requirement
invariably introduces computation time when compared with other recent
segmentation methods[22, 50]. The training time, for example, reported in Ref.
[50] for the entire program is roughly 31 hours. Computation times for the
training stage of our program depend on the image composition. A single layer
image can take about ten minutes but images with multilayer flakes (more
clusters) take as long as 5 hours. Our program results here are from training
sets of roughly 10 images corresponding to about 10 hour computation time.
However, this is a single event time cost because once the master catalogue is
trained for a particular material and substrate combination, it can then be
used repeatedly in the testing step, which is more efficient.

Image testing requires roughly one minute to identify layer thicknesses of new
images. Computation time is sufficiently short for testing because image
pixels are simply compared with the master catalogue. This time would allow
in-situ identification of flakes from images taken by human inspection of a
substrate. The time spent scanning between images can take several minutes.
The time may also be sufficient for an automated scanning system such as that
presented in ref. [51]. Improvements in computation time may be sought through
further image compression or possibly reducing the testing step to two
dimensions of the three-dimensional RGB space, possibly blue and either green
or red, similar to algorithms presented in ref. [23]. Although, the dropped
color dimension would have to be identified for a particular
material/substrate combination.

For each material/substrate combination investigated in this study, the pixel
accuracy was determined by creating a ground truth image and comparing it,
pixel-by-pixel, with the testing images (see Figure S1 in the supplemental
materials for details). Pixel accuracy was slightly better for materials on
PDMS but overall the program achieves an average accuracy of 95% for the
materials and substrates investigated in this study. This pixel accuracy is
comparable to that achieved in studies based on much larger training sets.
Ref. [50] reports pixel accuracy of 97% from a training set of 917 images.
Based on these results, normalized confusion matrices for each combination
were calculated showing the individual layer accuracy as well. Finally, we
note that a clear advantage to our approach is the simplicity of our program
which relies on well-known and proven clustering techniques with relatively
high pixel accuracy from small training sets.

##  6 Conclusion

Summarizing, we have presented a code for the automatic identification of
flake thicknesses that can be universally applied to a variety of 2D materials
and substrates. The algorithm analyzes data clusters in RGB space of
preprocessed optical microscopy images. It can accurately identify mono- and
few-layer thicknesses with a pixel accuracy of 95%. We anticipate the program
will be of use for a wide variety of materials and substrates for the
continued interest and investigation into the properties and characteristics
of 2D materials.

##  7 Acknowledgements

The authors thank Jeffery Cloninger for technical support and Najme S. Taghavi
and Dr. Andres Castellanos-Gomez for optical images of materials on PDMS.

##  8 Author contributions statement

J.O.I. conceived the project. R.M.S. coded the program with guidance from
J.O.I.. R.M.S. and K.L.H. provided optical images of 2D materials. All authors
wrote and reviewed the manuscript.

Figure 4:  General thickness identification of 2D materials on opaque and
transparent substrates. (a-c) An example training process for MoS2 flakes
exfoliated onto Si/SiO2 substrates. (a) Raw optical image before
preprocessing. (b) RGB scatter plot of the identified clusters with pixels
colored according to the cluster they belong to. (c) Reconstructed image of
the MoS2 flake in (a) after the training step. (d-e) Testing process for MoS2
on Si/SiO2. (d) Raw optical image of an MoS2 flake on Si/SiO2. (e) Layer
identification after testing the image in (d). (f-h) An example training
process for MoS2/PDMS. (i-j) An example testing process for MoS2/PDMS. (k-m)
An example training process for MoSe2/PDMS. (n-o) An example testing process
for MoSe2/PDMS.

##  9 References

## References

  * [1] Clarke, L. P. _et al._ MRI segmentation: Methods and applications.  _Magnetic Resonance Imaging_ 13, 343–368, DOI: 10.1016/0730-725X(94)00124-L (1995).
  * [2] Ji, Z. _et al._ Fuzzy Local Gaussian Mixture Model for Brain MR Image Segmentation.  _IEEE Transactions on Information Technology in Biomedicine_ 16, 339–347, DOI: 10.1109/TITB.2012.2185852 (2012).
  * [3] Forouzanfar, M., Forghani, N. & Teshnehlab, M.  Parameter optimization of improved fuzzy c-means clustering algorithm for brain MR image segmentation.  _Engineering Applications of Artificial Intelligence_ 23, 160–168, DOI: 10.1016/j.engappai.2009.10.002 (2010).
  * [4] Feng, D. _et al._ Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges.  _IEEE Transactions on Intelligent Transportation Systems_ 1–20, DOI: 10.1109/TITS.2020.2972974 (2020).
  * [5] Schwarz, M., Milan, A., Periyasamy, A. S. & Behnke, S.  RGB-D object detection and semantic segmentation for autonomous manipulation in clutter.  _The International Journal of Robotics Research_ 37, 437–451, DOI: 10.1177/0278364917713117 (2018).
  * [6] Ratakonda, K. & Ahuja, N.  Lossless image compression with multiscale segmentation.  _IEEE Transactions on Image Processing_ 11, 1228–1237, DOI: 10.1109/TIP.2002.804528 (2002).
  * [7] Sanderson, H. & Crebbin, G.  Image segmentation for compression of images and image sequences.  _IEE Proceedings - Vision, Image and Signal Processing_ 142, 15–21, DOI: 10.1049/ip-vis:19951681 (1995).
  * [8] Shen, L. & Rangayyan, R.  A segmentation-based lossless image coding method for high-resolution medical image compression.  _IEEE Transactions on Medical Imaging_ 16, 301–307, DOI: 10.1109/42.585764 (1997).
  * [9] Li, H. _et al._ Rapid and Reliable Thickness Identification of Two-Dimensional Nanosheets Using Optical Microscopy.  _ACS Nano_ 7, 10344–10353, DOI: 10.1021/nn4047474 (2013).
  * [10] Bing, D. _et al._ Optical contrast for identifying the thickness of two-dimensional materials.  _Optics Communications_ 406, 128–138, DOI: 10.1016/j.optcom.2017.06.012 (2018).
  * [11] Taghavi, N. S. _et al._ Thickness determination of MoS2, MoSe2, WS2 and WSe2 on transparent stamps used for deterministic transfer of 2D materials.  _Nano Research_ 12, 1691–1695, DOI: 10.1007/s12274-019-2424-6 (2019).
  * [12] Ni, Z. H. _et al._ Graphene Thickness Determination Using Reflection and Contrast Spectroscopy.  _Nano Letters_ 7, 2758–2763, DOI: 10.1021/nl071254m (2007).
  * [13] Jung, I. _et al._ Simple Approach for High-Contrast Optical Imaging and Characterization of Graphene-Based Sheets.  _Nano Letters_ 7, 3569–3575, DOI: 10.1021/nl0714177 (2007).
  * [14] Wang, X., Zhao, M. & Nolte, D. D.  Optical contrast and clarity of graphene on an arbitrary substrate.  _Applied Physics Letters_ 95, 081102, DOI: 10.1063/1.3212735 (2009).
  * [15] Zhang, H. _et al._ Optical thickness identification of transition metal dichalcogenide nanosheets on transparent substrates.  _Nanotechnology_ 28, 164001, DOI: 10.1088/1361-6528/aa6133 (2017).
  * [16] Yu, Y. _et al._ Investigation of multilayer domains in large-scale CVD monolayer graphene by optical imaging.  _Journal of Semiconductors_ 38, 033003, DOI: 10.1088/1674-4926/38/3/033003 (2017).
  * [17] Wang, Y. Y. _et al._ Thickness identification of two-dimensional materials by optical imaging.  _Nanotechnology_ 23, 495713, DOI: 10.1088/0957-4484/23/49/495713 (2012).
  * [18] Rubio-Bollinger, G. _et al._ Enhanced Visibility of MoS2, MoSe2, WSe2 and Black Phosphorus: Making Optical Identification of 2D Semiconductors Easier.  _Electronics_ 4, 847–856, DOI: 10.3390/electronics4040847 (2015).  1511.05474.
  * [19] Jessen, B. S. _et al._ Quantitative optical mapping of two-dimensional materials.  _Scientific Reports_ 8, 6381, DOI: 10.1038/s41598-018-23922-1 (2018).
  * [20] Yang, J. & Yao, H.  Automated identification and characterization of two-dimensional materials via machine learning-based processing of optical microscope images.  _Extreme Mechanics Letters_ 39, 100771, DOI: 10.1016/j.eml.2020.100771 (2020).
  * [21] Wu, B., Wang, L. & Gao, Z.  A two-dimensional material recognition image algorithm based on deep learning.  In _2019 International Conference on Information Technology and Computer Application (ITCA)_ , 247–252, DOI: 10.1109/ITCA49981.2019.00061 (2019).
  * [22] Masubuchi, S. _et al._ Deep-learning-based image segmentation integrated with optical microscopy for automatically searching for two-dimensional materials.  _npj 2D Materials and Applications_ 4, 1–9, DOI: 10.1038/s41699-020-0137-z (2020).
  * [23] Masubuchi, S. & Machida, T.  Classifying optical microscope images of exfoliated graphene flakes by data-driven machine learning.  _npj 2D Materials and Applications_ 3, 1–7, DOI: 10.1038/s41699-018-0084-0 (2019).
  * [24] Lin, X. _et al._ Intelligent identification of two-dimensional nanostructures by machine-learning optical microscopy.  _Nano Research_ 11, 6316–6324, DOI: 10.1007/s12274-018-2155-0 (2018).
  * [25] Li, Y. _et al._ Rapid identification of two-dimensional materials via machine learning assisted optic microscopy.  _Journal of Materiomics_ 5, 413–421, DOI: 10.1016/j.jmat.2019.03.003 (2019).
  * [26] Greplova, E. _et al._ Fully Automated Identification of Two-Dimensional Material Samples.  _Physical Review Applied_ 13, 064017, DOI: 10.1103/PhysRevApplied.13.064017 (2020).
  * [27] Cellini, F., Lavini, F., Berger, C., de Heer, W. & Riedo, E.  Layer dependence of graphene-diamene phase transition in epitaxial and exfoliated few-layer graphene using machine learning.  _2D Materials_ 6, 035043, DOI: 10.1088/2053-1583/ab1b9f (2019).
  * [28] Hu, X., Qiu, C. & Liu, D.  Rapid thin-layer ws 2 detection based on monochromatic illumination photographs.  _Nano Research_ 1–6 (2020).
  * [29] Dong, X. _et al._ 3d deep learning enables accurate layer mapping of 2d materials.  _ACS nano_ (2021).
  * [30] Aleithan, S. H. & Mahmoud-Ghoneim, D.  Toward automated classification of monolayer versus few-layer nanomaterials using texture analysis and neural networks.  _Scientific Reports_ 10, 1–8 (2020).
  * [31] Castellanos-Gomez, A. _et al._ Deterministic transfer of two-dimensional materials by all-dry viscoelastic stamping.  _2D Materials_ 1, 011002, DOI: 10.1088/2053-1583/1/1/011002 (2014).
  * [32] Island, J. O. _et al._ Precise and reversible band gap tuning in single-layer MoSe2 by uniaxial strain.  _Nanoscale_ 8, 2589–2593, DOI: 10.1039/C5NR08219F (2016).
  * [33] Lippert, S. _et al._ Influence of the substrate material on the optical properties of tungsten diselenide monolayers.  _2D Materials_ 4, 025045, DOI: 10.1088/2053-1583/aa5b21 (2017).
  * [34] Nguyen, D. C. _et al._ Visibility of hexagonal boron nitride on transparent substrates.  _Nanotechnology_ 31, 195701, DOI: 10.1088/1361-6528/ab6bf4 (2020).
  * [35] Chu, H. _et al._ Linear Magnetoelectric Phase in Ultrathin ${\mathrm{}}MnPS{}{}_{3}$ Probed by Optical Second Harmonic Generation.  _Physical Review Letters_ 124, 027601, DOI: 10.1103/PhysRevLett.124.027601 (2020).
  * [36] Sterbentz, R. M.  Universal image segmentation.  https://github.com/islandlab-unlv/Universal-image-segementation (2020).
  * [37] Comaniciu, D. & Meer, P.  Mean shift: A robust approach toward feature space analysis.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 24, 603–619, DOI: 10.1109/34.1000236 (2002).
  * [38] Zhou, Y.-m., Jiang, S.-y. & Yin, M.-l.  A Region-Based Image Segmentation Method with Mean-Shift Clustering Algorithm.  In _2008 Fifth International Conference on Fuzzy Systems and Knowledge Discovery_ , vol. 2, 366–370, DOI: 10.1109/FSKD.2008.363 (2008).
  * [39] Zhou, H., Wang, X. & Schaefer, G.  Mean Shift and Its Application in Image Segmentation.  In Kwaśnicka, H. & Jain, L. C. (eds.) _Innovations in Intelligent Image Analysis_ , Studies in Computational Intelligence, 291–312, DOI: 10.1007/978-3-642-17934-1_13 (Springer, Berlin, Heidelberg, 2011).
  * [40] Ester, M., Kriegel, H.-P., Sander, J. & Xu, X.  A density-based algorithm for discovering clusters in large spatial databases with noise.  In _Proceedings of the Second International Conference on Knowledge Discovery and Data Mining_ , KDD’96, 226–231 (AAAI Press, Portland, Oregon, 1996).
  * [41] Xu, R. & Wunsch, D.  Survey of clustering algorithms.  _IEEE Transactions on Neural Networks_ 16, 645–678, DOI: 10.1109/TNN.2005.845141 (2005).
  * [42] Celebi, M. E., Kingravi, H. A. & Vela, P. A.  A comparative study of efficient initialization methods for the k-means clustering algorithm.  _Expert Systems with Applications_ 40, 200–210, DOI: 10.1016/j.eswa.2012.07.021 (2013).
  * [43] Dhanachandra, N., Manglem, K. & Chanu, Y. J.  Image Segmentation Using K -means Clustering Algorithm and Subtractive Clustering Algorithm.  _Procedia Computer Science_ 54, 764–771, DOI: 10.1016/j.procs.2015.06.090 (2015).
  * [44] Dhanachandra, N. & Chanu, Y. J.  A Survey on Image Segmentation Methods using Clustering Techniques.  _European Journal of Engineering Research and Science_ 2, 15–20, DOI: 10.24018/ejers.2017.2.1.237 (2017).
  * [45] Gupta, L. & Sortrakul, T.  A gaussian-mixture-based image segmentation algorithm.  _Pattern Recognition_ 31, 315–325, DOI: 10.1016/S0031-3203(97)00045-9 (1998).
  * [46] Nguyen, T. M. & Wu, Q. M. J.  Dirichlet Gaussian mixture model: Application to image segmentation.  _Image and Vision Computing_ 29, 818–828, DOI: 10.1016/j.imavis.2011.09.001 (2011).
  * [47] Permuter, H., Francos, J. & Jermyn, I.  A study of Gaussian mixture models of color and texture features for image classification and segmentation.  _Pattern Recognition_ 39, 695–706, DOI: 10.1016/j.patcog.2005.10.028 (2006).
  * [48] Ribeiro, H. L. & Gonzaga, A.  Hand Image Segmentation in Video Sequence by GMM: A comparative analysis.  In _2006 19th Brazilian Symposium on Computer Graphics and Image Processing_ , 357–364, DOI: 10.1109/SIBGRAPI.2006.23 (2006).
  * [49] Santosh, D., Venkatesh, P. N., Poornesh, P., Rao, L. N. & Kumar, N.  Tracking Multiple Moving Objects Using Gaussian Mixture Model (2013).
  * [50] Han, B. _et al._ Deep-Learning-Enabled Fast Optical Identification and Characterization of 2D Materials.  _Advanced Materials_ n/a, 2000953, DOI: 10.1002/adma.202000953 (2020).
  * [51] Masubuchi, S. _et al._ Autonomous robotic searching and assembly of two-dimensional crystals to build van der Waals superlattices.  _Nature Communications_ 9, 1413, DOI: 10.1038/s41467-018-03723-w (2018).

##  10 Supplemental Materials: Universal image segmentation for optical
identification of 2D materials

##  1 Python code and packages

The software was written in Python 3.7.3 and includes the following packages:
scipy==1.5.2, matplotlib==3.0.3, numpy==1.16.2, and opencv-python==4.2.0.34.
The full Python code is available at https://github.com/islandlab-
unlv/Universal-image-segementation.

##  2 Pixel accuracy and confusion matrices

In order to measure the fidelity of our program, we created confusion matrices
by comparing the testing results (predictions) with manually recolored images
based on layer thickness (ground truths). Figure S1(a) shows a ground truth
image for a graphene flake on Si/SiO2 and Figure S1(b) shows the testing
result using the same colorscale. Figure S1(c) shows the raw counts for the
confusion matrix of graphene on Si/SiO2. Element $ij$ in the confusion matrix
is the number of pixels predicted to be $i$-layer that are known to be
$j$-layer. The diagonal thus is where the program predicted the layer
thickness correctly. We calculate the pixel accuracy by taking the ratio of
the trace of the matrix with the sum of all elements in the matrix. For
graphene on SiO2, we calculate a pixel accuracy of 96.7%. For the other test
samples in Figure S1, we calculate the following pixel accuracies: MoS2/SiO2
in Figure S1(e-h) (94.7%), MoS2/PDMS in Figure S1(i-l) (98.0%), MoSe2/PDMS in
Figure S1(m-p) (96.7%).

Normalizing a confusion matrix by the true labels elucidates how well the
program can predict each layer thickness. Columns 0-2 of Figure S1(d), for
example, state we can successfully identify 99.0% of the Si/SiO2 substrate,
97.3% of the monolayer graphene regions, and 94.3% of the bilayer graphene
regions. It appears to falter with tri- and quadlayer identification, but this
is a result of the necessary preprocessing on the test image. Despite
qualities of bilateral filtering as an edge-preserving filter, we always had
some blurring at the flake-thickness boundaries while applying it. As a
result, narrow strips of a layer thickness could be blurred away, as is the
case with the thin stripes of trilayer present in the graphene/SiO2 test image
(light grey stripes at the bottom left of panels (a) and (b) of Figure S1).
These confusion matrices were created off of a single test image per
material/substrate tested, and are beholden to the biases that arise from
those individual samples. With finer tuning of the bilateral filter and a
larger test set, this blurring should be mitigated.

Figure S1:  Pixel accuracy testing. (a-d) Ground truth (a) and prediction (b)
images from testing for a graphene flake on a Si/SiO2 substrate. Raw pixel
count confusion matrix for graphene on SiO2 (c), and the same confusion matrix
normalized by column (normalized to true label) (d). (e-h) Ground truth (e)
and prediction (f) images from testing for a MoS2 flake on a Si/SiO2
substrate. Raw pixel count confusion matrix for MoS2 on SiO2 (g), and the same
confusion matrix normalized by column (normalized to true label) (h). (i-l)
Ground truth (i) and prediction (j) images from testing for a MoS2 flake on a
PDMS substrate. Raw pixel count confusion matrix for MoS2 on PDMS (k), and the
same confusion matrix normalized by column (normalized to true label) (l).
(m-p) Ground truth (m) and prediction (n) images from testing for a MoSe2
flake on a PDMS substrate. Raw pixel count confusion matrix for MoSe2 on PDMS
(o), and the same confusion matrix normalized by column (normalized to true
label) (p).


# Contextualized Automatic Speech Recognition
with Attention-Based Bias Phrase Boosted Beam Search

###### Abstract

End-to-end (E2E) automatic speech recognition (ASR) methods exhibit remarkable
performance. However, since the performance of such methods is intrinsically
linked to the context present in the training data, E2E-ASR methods do not
perform as desired for unseen user contexts (e.g., technical terms, personal
names, and playlists). Thus, E2E-ASR methods must be easily contextualized by
the user or developer. This paper proposes an attention-based contextual
biasing method that can be customized using an editable phrase list (referred
to as a bias list). The proposed method can be trained effectively by
combining a bias phrase index loss and special tokens to detect the bias
phrases in the input speech data. In addition, to improve the
contextualization performance during inference further, we propose a bias
phrase boosted (BPB) beam search algorithm based on the bias phrase index
probability. Experimental results demonstrate that the proposed method
consistently improves the word error rate and the character error rate of the
target phrases in the bias list on both the Librispeech-960 (English) and our
in-house (Japanese) dataset, respectively.

Index Terms—  speech recognition, attention, contextualization, biasing, beam
search

##  1 Introduction

End-to-end (E2E) automatic speech recognition (ASR) [1, 2] methods directly
convert acoustic feature sequences to token sequences without requiring the
multiple components used in conventional ASR systems, such as acoustic models
(AM) and language models (LM). Various E2E-ASR methods have been proposed
previously, including connectionist temporal classification (CTC) [3],
recurrent neural network transducer (RNN-T) [4], attention mechanism [5, 6],
and their various hybrid systems [7, 8, 9]. Since the effectiveness of E2E-ASR
methods is inherently related to the context in the training data, performance
expectations may not be satisfied consistently for the given user context. For
example, personal names and technical terms tend to be important keywords in
different contexts, but such terms may not appear frequently in the available
training data, which would result in poor recognition accuracy. It is
impractical to train a model for all contexts during training; thus, the user
or developer should be able to contextualize the model easily without
training.

A typical approach to this problem is shallow fusion using an external LM [10,
11, 12, 13, 14]. For example, [10, 11, 12] used a weighted finite state
transducer (WFST) to construct an in-class LM to facilitate contextualization
for the target named entities. Neural LM fusion methods have been also
proposed [13, 14]. The LM fusion technique attempts to enhance accuracy by
combining an E2E-ASR model with an external neural LM and then rescoring the
hypotheses generated by the E2E-ASR model. However, whether employing WFST or
neural LMs, training an external LM requires additional training steps.

Thus, several methods have been proposed that do not require retraining. These
methods include knowledge graph modeling [15] for recognizing out-of-
vocabulary named entities, contextual spelling correction [15] using an
editable phrase list, and named entity aware ASR model [16] that recognize
specific named entities based on phoneme similarity. However, these methods
have limitations, such as requiring a speech synthesis (TTS) model for
training and not being able to handle words other than predefined target named
entities.

Deep biasing methods[17, 18, 19, 20] provide an alternative approach to
realize effective contextualization without requiring retraining processes and
TTS models. In such methods, the E2E-ASR model can be contextualized using an
editable phrase list, which is referred to as a bias list in this paper. Most
deep biasing methods implement a cross-attention layer between the bias list
and input sequences to recognize the bias phrases correctly. However, it has
been observed that simply adding a cross-attention layer for the bias list is
not effective [21]. Thus, [21, 22] introduced an additional branch designed to
detect bias phrases, which indirectly helps to update the parameters of the
cross-attention layer through an auxiliary loss. In contrast, [23, 24]
introduced an auxiliary loss function directly on the cross-attention layer
(referred to as bias phrase index loss and will be described in Section 3.2),
which detects to the bias phrase index. While this approach allows for a
direct parameter update of the cross-attention layer, it cannot distinguish
whether the output tokens come from the bias list or not. In addition, [23]
requires two-stage training using a pretrained ASR model, which is time
consuming.

This paper proposes a deep biasing method that employs both an auxiliary loss
directly on the cross-attention layer, termed as bias phrase index loss, and
special tokens for bias phrases to realize more effective bias phrase
detection. Unlike conventional indirect methods [21, 22], our method
facilitates the effective training of the cross-attention layer through the
bias phrase index loss. Additionally, our technique departs from current
methods [23] by introducing special tokens for bias phrases. This allows the
model to focus on the bias phrases more effectively, eliminating the need for
a two-stage training process. Furthermore, we propose a bias phrase boosted
(BPB) beam search algorithm that integrates the bias phrase index probability
during inference, augmenting the performance in bias phrase recognition. The
main contributions of this study are as follows:

  * •

We propose a deep biasing model that utilizes both bias phrase index loss and
special tokens for the bias phrases.

  * •

We propose a bias phrase boosted (BPB) beam search algorithm to further
improve the performance for the target phrases.

  * •

We demonstrate that the proposed method is effective for both the
Librispeech-960 and our in-house Japanese dataset.

##  2 Attention-based encoder-decoder ASR

This section describes an attention-based encoder-decoder system that consists
of an audio encoder and an attention-based decoder, which are extended to the
proposed method.

###  2.1 Audio encoder

The audio encoder comprises two convolutional layers, a linear projection
layer, and $M_{\text{a}}$ Conformer blocks [25]. The audio encoder transforms
an audio feature sequence $\bm{X}$ to $T$ length hidden state vectors
$\bm{H}=[\bm{h}_{1},...,\bm{h}_{T}]\in\mathbb{R}^{T\times d}$ where $d$
represents the dimension as follows:

$\displaystyle\bm{H}$ $\displaystyle=\mathrm{AudioEnc}(\bm{X}).$ (1)

###  2.2 Attention-based decoder

The posterior probability is formulated as follows:

$P_{\text{att}}(\bm{y}\mid\bm{X})=\prod_{s=1}^{S}P\left(y_{s}\mid\bm{y}_{0:s-1},\bm{X}\right),$
(2)

where $s$ and $S$ represent the token index and the total number of tokens,
respectively. Given $\bm{H}$ generated by the audio encoder in Eq. (1) and the
previous token sequence $\bm{y}_{0:s-1}$, the attention-based decoder
recursively estimates the next token $y_{s}$ as follows:

$\displaystyle
P(y_{s}|\bm{y}_{0:s-1},\bm{X})=\mathrm{AttnDec}(\bm{y}_{0:s-1},\bm{H}).$ (3)

The attention-based decoder comprises an embedding layer with a positional
encoding layer, $M_{\text{d}}$ Transformer blocks, and a linear layer. Each
Transformer block has a multiheaded self-attention layer, a cross-attention
layer (i.e., audio attention), and a linear layer with layer normalization
(LN) layers and residual connections. Here, the audio attention layer
including the LN is formulated as follows:

$\bm{U}^{{}^{\prime}}=\mathrm{Softmax}\left(\frac{\mathrm{LN}(\bm{U})\bm{H}^{T}}{\sqrt{d}}\right)\bm{H}+\bm{U},$
(4)

where $\bm{U}$ and $\bm{U}^{{}^{\prime}}$ represent the input and output of
the audio attention layer, respectively. In addition, the hybrid CTC/attention
model [7] includes a CTC decoder. The attention-based decoder will be extended
to the proposed bias decoder in Section 3.2.

Fig. 1: Overall architecture of the proposed method, including the audio
encoder, bias encoder, and bias decoder. The BPB beam search algorithm is used
during inference.

##  3 Proposed deep biasing method

Figure 1 shows the overall architecture of the proposed method, which
comprises the audio encoder, bias encoder, and bias decoder. These components
are described in the following subsections.

###  3.1 Bias encoder

The bias encoder comprises an embedding layer with a positional encoding
layer, $M_{\text{e}}$ Transformer blocks, a mean pooling layer, and a bias
list $\bm{B}=\\{\bm{b}_{0},\bm{b}_{1},\cdots,\bm{b}_{N}$}, where $n$ and
$\bm{b}_{n}$ represent the bias phrase index and the token sequence of the
$n$-th bias phrase (e.g., “play a song”), respectively. Here, $\bm{b}_{0}$ is
a dummy phrase which means “no-bias”. After applying zero padding based on the
max token length $L_{\text{max}}$ in the bias list $\bm{B}$, the embedding
layer and the Transformer blocks extract a set of token-level feature
sequences, $\bm{G}\in\mathbb{R}^{(N+1)\times L_{\text{max}}\times d}$ as
follows:

$\displaystyle\bm{G}=\mathrm{Transformer}(\mathrm{Embedding}(\bm{B})).$ (5)

Then, mean pooling is performed to extract a phrase-level feature sequence,
$\bm{V}=[\bm{v}_{0},\bm{v}_{1},\cdots,\bm{v}_{N}]\in\mathbb{R}^{(N+1)\times
d}$, as follows:

$\displaystyle\bm{V}=\mathrm{MeanPool}(\bm{G}).$ (6)

###  3.2 Bias decoder

The bias decoder is an extension of the attention-based decoder described in
Section 2.2, where an additional cross-attention layer (i.e., bias attention)
is introduced to each Transformer block, as shown in Figure 1. Unlike Eq. (2),
the posterior probability is formulated using the bias list $\bm{B}$ as
follows:

$P_{\text{batt}}(\bm{y}\mid\bm{X},\bm{B})=\prod_{s=1}^{S}P\left(y_{s}\mid\bm{y}_{0:s-1},\bm{X},\bm{B}\right).$
(7)

Given $\bm{H}$, $\bm{V}$ in Eqs. (1), (6), and $\bm{y}_{0:s-1}$, the bias
decoder estimates the next token $y_{s}$ recursively, unlike Eq. (3), as
follows:

$\displaystyle
P\left(y_{s}\mid\bm{y}_{0:s-1},\bm{X},\bm{B}\right)=\mathrm{BiasDec}(\bm{y}_{0:s-1},\bm{H},\bm{V}).$
(8)

In the Transformer block of the bias decoder, the bias attention layer
including the LN is formulated as follows:

$\bm{U}^{{}^{\prime\prime}}=\mathrm{Softmax}\left(\frac{\mathrm{LN}(\bm{U}^{{}^{\prime}})\bm{V}^{T}}{\sqrt{d}}\right)\bm{V}+\bm{U}^{{}^{\prime}}.$
(9)

In addition, the bias attention layer estimates the bias phrase index sequence
$\bm{\hat{n}}=[\hat{n}_{1},\hat{n}_{2},\cdots,\hat{n}_{S}]$ as follows:

$P_{\text{bidx}}(\bm{\hat{n}}\mid\bm{X},\bm{B})=\prod_{s=1}^{S}P\left(\hat{n}_{s}\mid\bm{y}_{0:s-1},\bm{X},\bm{B}\right),$
(10)
$P\left(\hat{n}_{s}\mid\bm{y}_{0:s-1},\bm{X},\bm{B}\right)=\mathrm{Softmax}\left(\frac{\mathrm{LN}(\bm{u}^{{}^{\prime}}_{s})\bm{V}^{T}}{\sqrt{d}}\right),$
(11)

where $\bm{u}^{{}^{\prime}}_{s}$ denotes the $s$-th feature vector of
$\bm{U}^{{}^{\prime}}=[\bm{u}^{{}^{\prime}}_{0},\bm{u}^{{}^{\prime}}_{1},\cdots,\bm{u}^{{}^{\prime}}_{S}]$.
For example, if a bias phrase, “play a song” with a bias index of 2 (Figure 1)
is detected in a complete utterance, “I play a song today”, the bias phrase
index sequence $\bm{\hat{n}}$ = [0, 2, 2, 2, 0]. Model parameters are
optimized using the cross entropy losses as follows:

$L_{\text{batt}}=\mathrm{CrossEntropy}(\bm{y}_{\text{gt}},P_{\text{batt}}(\bm{y}\mid\bm{X},\bm{B})),$
(12)
$L_{\text{bidx}}=\mathrm{CrossEntropy}(\bm{\hat{n}}_{\text{gt}},P_{\text{bidx}}(\bm{\hat{n}}\mid\bm{X},\bm{B})),$
(13)

where $\bm{y_{\text{gt}}}$ and $\bm{\hat{n}}_{\text{gt}}$ represent the one-
hot vector sequences of the reference transcription and the reference bias
phrase index including the no-bias option. Here, we refer to $L_{\text{bidx}}$
as bias phrase index loss, respectively.

###  3.3 Training

During the training process, a bias list $\bm{B}$ is created randomly from the
corresponding reference transcriptions for each batch. Specifically, 0 to
$N_{\text{utt}}$ bias phrases of 2 to $L_{\text{max}}$ token lengths are
extracted uniformly for each utterance, resulting in a total of $N$ bias
phrases ($N_{\text{utt}}\times n_{\text{batch}}$). After the bias list
$\bm{B}$ is extracted randomly, special tokens ($<$sob$>$/$<$eob$>$) are
inserted before and after the extracted phrases in the reference transcription
to distinguish whether the output tokens come from the bias list or not. The
proposed method is optimized via multitask learning using the weighted sum of
losses, as expressed in Eqs. (12), (13), and the CTC loss ($L_{\text{ctc}}$):

$L=\lambda_{\text{ctc}}L_{\text{ctc}}+\lambda_{\text{batt}}L_{\text{batt}}+\lambda_{\text{bidx}}L_{\text{bidx}},$
(14)

where $\lambda_{\text{ctc}}$, $\lambda_{\text{batt}}$, and
$\lambda_{\text{bidx}}$ represent the training weights.

###  3.4 BPB beam search algorithm

We also propose a bias phrase boosted (BPB) beam search algorithm that
exploits the bias phrase probability as described in Algorithm 1. The bias
decoder calculates the token probability $\bm{p}_{\text{new}}$ including the
special tokens, $<$sob$>$/$<$eob$>$, using Eq. (8) (line 5). We then estimate
the bias phrase index $\hat{n}_{s}$ using Eq. (11) and the argmax function
(line 6). Here, the number of bias phrases $N$ in the bias list $\bm{B}$ can
increase significantly during inference, which would reduce the peak value
after applying the softmax function in Eq. (9). Thus, Eq. (9) is approximated
using the top $k_{\text{score}}$ pruning as follows:

$\bm{U}^{{}^{\prime\prime}}=\mathrm{Softmax}\left(\mathrm{Top\\_k_{score}}\left(\frac{\mathrm{LN}(\bm{U^{{}^{\prime}}})\bm{V}^{T}}{\sqrt{d}}\right)\right)\bm{V}+\bm{U^{{}^{\prime}}}.$
(15)

Then, if $\hat{n}_{s}$ = 0 (i.e., “no-bias”), the token probabilities for the
special tokens $\bm{p}_{\text{new}}$[sob] and $\bm{p}_{\text{new}}$[eob] are
penalized based on the weight $\alpha_{\text{pen}}$ (line 8, 9), otherwise,
the corresponding token probabilities are increased according to the weight
$\alpha_{\text{bonus}}$ (line 11 - 13). For example, if the detected bias
phrase is “play a song”, the token probabilities for “play”, “a”, and “song”
are increased with $\alpha_{\text{bonus}}$. Based on the boosted probabilities
$\bm{p}_{\text{new}}$, the top $k_{\text{beam}}$ pruning is performed as in
the conventional beam search [7].

##  4 Experiment

###  4.1 Experimental setup

The input features are 80-dimensional Mel filterbanks with a window size of
512 samples and a hop length of 160 samples. Then, SpecAugment [26] is
applied. The audio encoder has two convolutional layers with a stride of two
for downsampling, a 256-dimensional linear projection layer, and 12 Conformer
blocks with 1024 linear units. The bias encoder and the bias decoder have
three Transformer blocks with 1024 linear units and six Transformer layers
with 2048 units, respectively. The attention layers in the audio encoder, the
bias encoder, and the bias decoder are 4-multihead attentions with a
dimension, $d$, of 256. During the training process, a bias list $\bm{B}$ is
created randomly for each batch with $N_{\text{utt}}$ = 2 and $L_{\text{max}}$
= 10 described in Section 3.3. In this experiment, the bias list $\bm{B}$ has
a total of $N$ = 50 to 200 bias phrases within a batch. The training weights
$\lambda_{\text{ctc}}$, $\lambda_{\text{batt}}$, and $\lambda_{\text{bidx}}$
(described in Eq. (14)) are set to 0.3, 0.7, and 1.0, respectively. The
proposed model is trained for 150 epochs at a learning rate of 0.0015 with
15,000 warmup steps using the Adam optimizer. During the decoding process, the
hyper parameters of $k_{\text{beam}}$, $k_{\text{score}}$,
$\alpha_{\text{bonus}}$, and $\alpha_{\text{pen}}$ (Section 3.4) are set to
20, 50, 1.0 and 10.0, respectively.

The Librispeech corpus (960 h, 100 h) [27] is used to evaluate the proposed
method using ESPnet as the E2E-ASR toolkit [28]. The proposed method is
evaluated in terms of word error rate (WER), bias phrase WER (B-WER), and
unbiased phrase WER (U-WER) [29]. Note that insertion errors are counted
toward B-WER if the inserted phrases are present in the bias list; otherwise,
insertion errors are counted toward the U-WER. The goal of the proposed method
is to improve the B-WER with a slight degradation in the U-WER and overall
WER.

###  4.2 Preliminary analysis of the proposed techniques

Table 1: Preliminary analysis on the Librispeech-100 test-clean.

ID | Model | WER | U-WER | B-WER
---|---|---|---|---
A1 |  Baseline [7] | 8.59 | 5.87 | 30.71
B1 | Bias decoder | 8.11 | 5.43 | 29.89
B2 | B1 + bias phrase index loss | 7.53 | 5.27 | 25.92
B3 |  B2 + $<$sob$>$/$<$eob$>$ tokens | 6.93 | 4.96 | 23.00
B4 | B3 + BPB beam search | 5.92 | 5.00 | 17.93

(a) Without bias phrase index loss

(b) With bias phrase index loss

Fig. 2: Effect of the bias phrase index loss. The horizontal and vertical axes
show token index $s$ and bias phrases in $\bm{B}$, respectively. Table 2: Main
WER results obtained on Librispeech-960 data (U-WER/B-WER). Bold values
indicate cases where the proposed method outperformed the baselines, and
underlined values represent the best results.

|  $N$ = 0 (no-bias) |  $N$ = 100 |  $N$ = 500 |  $N$ = 1000
---|---|---|---|---
Model | test-clean | test-other | test-clean | test-other | test-clean | test-other | test-clean | test-other
Baseline [7] | 3.56 | 7.55 | 3.56 | 7.55 | 3.56 | 7.55 | 3.56 | 7.55
|  (2.6/11.7) |  (5.6/24.8) | (2.6/11.7) | (5.6/24.8) |  (2.6/11.7) | (5.6/24.8) |  (2.6/11.7) |  (5.6/24.8)
CPPNet [21] | 4.29 | 9.16 | 3.40 | 7.77 | 3.68 | 8.31 | 3.81 | 8.75
|  (2.6/18.3) | (5.9/37.5) | (2.6/10.4) | (6.0/23.0) | (2.8/10.9) | (6.5/24.3) | (2.9/11.4) | (6.9/25.3)
Proposed | 5.81 | 9.17 | 2.94 | 6.21 | 3.24 | 6.56 | 4.07 | 7.60
w/o BPB | (4.8/13.7) | (6.8/30.1) |  (2.5/6.5) |  (5.4/13.1) |  (2.7/7.9) |  (5.5/15.9) |  (3.4/9.7) |  (6.4/18.6)
Proposed | 5.05 | 8.81 | 2.75 | 5.60 | 3.21 | 6.28 | 3.47 | 7.34
w/ BPB | (3.9/14.1) | (6.6/27.9) |  (2.3/6.0) |  (4.9/12.0) |  (2.7/7.0) |  (5.5/13.5) |  (3.0/7.7) |  (6.4/15.8)

Firstly, we verify the effect of the proposed techniques on the
Librispeech-100 as a preliminary experiment. Table 1 shows the effect of the
bias phrase index loss, $L_{\text{bidx}}$ described in Eq. (13), the special
tokens for the bias phrases ($<$sob$>$/$<$eob$>$), and the BPB beam search on
the Librispeech-100 test-clean evaluation set with a bias list size of $N$ =
100. Comparing with the baseline (the hybrid CTC/attention model [7]), simply
introducing the bias attention layer does not improve the performance (A1 vs.
B1), whereas the bias phrase index loss improves the B-WER significantly,
which results in an improvement to the overall WER (B1 vs. B2). Figure 2 shows
the visualization results of the bias phrase index probabilities described in
Eq. (11). The bias phrase index probabilities are estimated correctly by
introducing the bias phrase index loss, $L_{\text{bidx}}$ in Eq. (13). In
addition, introducing the special tokens ($<$sob$>$/$<$eob$>$) further
improves the B-WER (B2 vs. B3). Furthermore, the BPB beam search technique
significantly improves the B-WER with a slight degradation in U-WER (B3 vs.
B4).

###  4.3 Main results

Table 2 shows the results obtained by the proposed method on the
Librispeech-960 data for different bias list sizes $N$. Baseline is the hybrid
CTC/attention model [7]. When the bias list size $N$ = 100, the proposed
method improves the B-WER, which in turn significantly improves the U-WER and
WER. In addition, the proposed BPB beam search technique further improves the
B-WER without degrading the overall WER and U-WER. The B-WER and U-WER tend to
deteriorate as the number of bias phrases $N$ increased; however, the proposed
BPB beam search technique is particularly effective in terms of suppressing
the deterioration of the B-WER. As a result, the proposed method outperforms
the baseline in terms of both WER and B-WER. Although the proposed method
underperforms the baseline when no bias phrases are used ($N$ = 0), we do not
consider it as a critical issue because the users usually register important
keywords for them.

###  4.4 Analysis of the BPB beam search algorithm

Figure 3 shows the effect of the decoding weight $\alpha_{\text{bonus}}$ of
the BPB beam search on the Librispeech-960 test-other with a bias list size of
$N$ = 100. Although, even without using the proposed BPB beam search
technique, the proposed method improves the B-WER as described in Section 4.3,
the BPB beam search technique further improves the B-WER. When the decoding
weight $\alpha_{\text{bonus}}>1.5$, the B-WER, U-WER, and the overall WER
deteriorate. The B-WER, U-WER, and the overall WER are the best at
$\alpha_{\text{bonus}}$ = 1.0.

Fig. 3: Effect of the decoding weight $\alpha_{\text{bonus}}$ of the BPB beam
search on Librispeech-960.

Figure 4 illustrates the inference results from three distinct approaches: the
baseline method, our proposed method excluding the BPB beam search technique,
and our proposed method incorporating the BPB beam search technique. Here,
bolded face represents the bias phrases, and words in red and blue represent
incorrectly and correctly recognized words, respectively. Even without the BPB
beam search technique, the proposed method reduces the misrecognition of the
bias phrases compared to the baseline; however, some bias phrases are not
correctly recognized even when the correct bias phrase index is estimated. In
contrast, the proposed BPB beam search technique recognizes the bias phrases
more correctly.

Fig. 4: Typical example. Bolded faces, red and blue faces represent the bias
phrases, incorrectly and correctly recognized, respectively.

###  4.5 Validation on Japanese dataset

We also validate the proposed method on our in-house dataset containing 93
hours of Japanese speech data, including meeting and morning assembly
scenarios, the Corpus of Spontaneous Japanese (581 h) [30], and 181 hours of
Japanese speech in the database developed by the Advanced Telecommunications
Research Institute International [31] with the same experimental setup
described in Section 4.1. Table 3 shows the evaluation results obtained on the
in-house dataset when $N$ = 203 phrases, such as personal names and technical
terms, are registered in the bias list $\bm{B}$. The proposed method
significantly improves the B-CER with a slight degradation in the U-CER. Thus,
the proposed method is effective for both English and Japanese languages.

Table 3: Experimental results on our in-house Japanese dataset. Model | CER | U-CER | B-CER
---|---|---|---
Baseline [7] | 9.85 | 8.17 | 22.32
Proposed ($N$=203) | 9.78 | 9.16 | 14.54
Proposed w/ BPB ($N$=203) | 9.67 | 9.20 | 13.16

##  5 Conclusion

This study introduces a deep biasing model incorporating bias phrase index
loss and specialized tokens for bias phrases. Additionally, the BPB beam
search technique is employed, leveraging bias phrase index probabilities to
enhance accuracy. Experimental results demonstrate that our model enhances
both WER and B-WER performances. Notably, the BPB beam search boosts B-WER
performance with minimal impact on overall WER, evident in both English and
Japanese datasets.

## References

  * [1] Rohit Prabhavalkar, Takaaki Hori, Tara N Sainath, Ralf Schlüter, and Shinji Watanabe,  “End-to-end speech recognition: A survey,”  arXiv preprint arXiv:2303.03329, 2023.
  * [2] Jinyu Li et al.,  “Recent advances in end-to-end automatic speech recognition,”  APSIPA Transactions on Signal and Information Processing, vol. 11, no. 1.
  * [3] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber,  “Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,”  in Proc. ICML, 2006, pp. 369–376.
  * [4] Alex Graves,  “Sequence transduction with recurrent neural networks,”  in Proc. ICML, 2012.
  * [5] Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio,  “Attention-based models for speech recognition,”  Advances in neural information processing systems, vol. 28, 2015\.
  * [6] William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals,  “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,”  in Proc. ICASSP, 2016, pp. 4960–4964.
  * [7] Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi,  “Hybrid ctc/attention architecture for end-to-end speech recognition,”  IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240–1253, 2017.
  * [8] Tara N Sainath et al.,  “Two-pass end-to-end speech recognition,”  arXiv preprint arXiv:1908.10992, 2019.
  * [9] Yui Sudo, Muhammad Shakeel, Brian Yan, Jiatong Shi, and Shinji Watanabe,  “4D ASR: Joint modeling of CTC, attention, transducer, and mask-predict decoders,”  in Proc. Interspeech, 2023, pp. 3312–3316.
  * [10] Rongqing Huang, Ossama Abdel-Hamid, Xinwei Li, and Gunnar Evermann,  “Class lm and word mapping for contextual biasing in end-to-end asr,”  in Proc. Interspeech, 2020, pp. 4348–4351.
  * [11] Ian Williams, Anjuli Kannan, Petar Aleksic, David Rybach, and Tara Sainath,  “Contextual speech recognition in end-to-end neural network systems using beam search,”  in Proc. Interspeech, 2018.
  * [12] Atsushi Kojima,  “A study of biasing technical terms in medical speech recognition using weighted finite-state transducer,”  Journal of the Acoustical Society of Japan, vol. 43, pp. 66–68, 2022.
  * [13] Anjuli Kannan, Yonghui Wu, Patrick Nguyen, Tara N Sainath, Zhijeng Chen, and Rohit Prabhavalkar,  “An analysis of incorporating an external language model into a sequence-to-sequence model,”  in Proc. ICASSP, 2018, pp. 5824–5828.
  * [14] Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates,  “Cold fusion: training seq2seq models together with language models,”  in Proc. Interspeech, 2018, pp. 387–391.
  * [15] Xiaoqiang Wang et al.,  “Towards contextual spelling correction for customization of end-to-end speech recognition systems,”  IEEE Trans. Audio, Speech, Lang. Process., vol. 30, pp. 3089–3097, 2022.
  * [16] Yui Sudo, Kazuya Hata, and Kazuhiro Nakadai,  “Retraining-free customized asr for enharmonic words based on a named-entity-aware model and phoneme similarity estimation,”  in Proc. Interspeech, 2023, pp. 3312–3316.
  * [17] Golan Pundak, Tara N Sainath, Rohit Prabhavalkar, Anjuli Kannan, and Ding Zhao,  “Deep context: End-to-end contextual speech recognition,”  in Proc. SLT, 2018, pp. 418–425.
  * [18] Mahaveer Jain, Gil Keren, Jay Mahadeokar, and Yatharth Saraf,  “Contextual rnn-t for open domain asr,”  in Proc. Interspeech, 2020, pp. 11–15.
  * [19] Antoine Bruguier, Rohit Prabhavalkar, Golan Pundak, and Tara N Sainath,  “Phoebe: Pronunciation-aware contextualization for end-to-end speech recognition,”  in Proc. ICASSP, 2019, pp. 6171–6175.
  * [20] Saket Dingliwal, Monica Sunkara, Srikanth Ronanki, Jeff Farris, Katrin Kirchhoff, and Sravan Bodapati,  “Personalization of ctc speech recognition models,”  in Proc. SLT, 2023, pp. 302–309.
  * [21] Kaixun Huang, Ao Zhang, Zhanheng Yang, Pengcheng Guo, Bingshen Mu, Tianyi Xu, and Lei Xie,  “Contextualized End-to-End Speech Recognition with Contextual Phrase Prediction Network,”  in Proc. Interspeech, 2023, pp. 4933–4937.
  * [22] Minglun Han, Linhao Dong, Zhenlin Liang, Meng Cai, Shiyu Zhou, Zejun Ma, and Bo Xu,  “Improving end-to-end contextual speech recognition with fine-grained contextual knowledge selection,”  in Proc. ICASSP, 2022, pp. 491–495.
  * [23] Christian Huber, Juan Hussain, Sebastian Stüker, and Alexander Waibel,  “Instant one-shot word-learning for context-specific neural sequence-to-sequence speech recognition,”  in Proc. ASRU, 2021, pp. 1–7.
  * [24] Shilin Zhou, Zhenghua Li, Yu Hong, Min Zhang, Zhefeng Wang, and Baoxing Huai,  “Copyne: Better contextual asr by copying named entities,”  arXiv preprint arXiv:2305.12839, 2023.
  * [25] Anmol Gulati et al.,  “Conformer: Convolution-augmented transformer for speech recognition,”  in Proc. Interspeech, 2020, pp. 5036–5040.
  * [26] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le,  “SpecAugment: A simple data augmentation method for automatic speech recognition,”  in Proc. Interspeech, 2019, pp. 2613–2617.
  * [27] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: an asr corpus based on public domain audio books,”  in Proc. ICASSP, 2015, pp. 5206–5210.
  * [28] Shinji Watanabe et al.,  “ESPnet: End-to-end speech processing toolkit,”  in Proc. Interspeech, 2018, pp. 2207–2211.
  * [29] Duc Le, Jain, et al.,  “Contextualized streaming end-to-end speech recognition with trie-based deep biasing and shallow fusion,”  in Proc. Interspeech, 2021, pp. 1772–1776.
  * [30] Kikuo Maekawa,  “Corpus of spontaneous Japanese: Its design and evaluation,”  in ISCA & IEEE Workshop on Spontaneous Speech Processing and Recognition, 2003.
  * [31] Akira Kurematsu et al.,  “Atr japanese speech database as a tool of speech recognition and synthesis,”  Speech Communication, vol. 9, no. 4, pp. 357–363, 1990.