[2,3,4]Stéphane Verger

[1,2,5]Magnus Andersson

1]Department of Physics, Umeå University, Umeå, 90187, Sweden

2]Integrated Science Lab, Department of Physics, Umeå University, Umeå, 90187,
Sweden

3]Umeå Plant Science Centre (UPSC), Department of Forest Genetics and Plant
Physiology, Swedish University of Agricultural Sciences, Umeå, 90183, Sweden

4]Umeå Plant Science Centre (UPSC), Department of Plant Physiology Umeå
University, Umeå, 90187, Sweden

5]Umeå Centre for Microbial Research (UCMR) Umeå University, Umeå, 90187,
Sweden

# Segmentation and Characterization of Macerated Fibers and Vessels Using Deep
Learning

Saqib Qamar<EMAIL_ADDRESS>Abu Imran Baba<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>[  [  [  [  [

###### Abstract

Purpose: Wood comprises different cell types, such as fibers and vessels,
defining its properties. Studying their shape, size, and arrangement in
microscopic images is crucial for understanding wood samples. Typically, this
involves macerating (soaking) samples in a solution to separate cells, then
spreading them on slides for imaging with a microscope that covers a wide
area, capturing thousands of cells. However, these cells often cluster and
overlap in images, making the segmentation difficult and time-consuming using
standard image-processing methods. Results: In this work, we develop an
automatic deep learning segmentation approach that utilizes the one-stage
YOLOv8 model for fast and accurate fiber and vessel segmentation and
characterization in microscopy images. The model can analyze 32640 x 25920
pixels images and demonstrate effective cell detection and segmentation,
achieving a mAP0.5-0.95 of 78 %. To assess the model’s robustness, we examined
fibers from a genetically modified tree line known for longer fibers. The
outcomes were comparable to previous manual measurements. Additionally, we
created a user-friendly web application for image analysis and provided the
code for use on Google Colab. Conclusion: By leveraging YOLOv8’s advances,
this work provides a deep learning solution to enable efficient quantification
and analysis of wood cells suitable for practical applications.

###### keywords:

Instance segmentation, YOLO, composites, soaking, optical microscopy

##  1 Introduction

Fiber cells are the main constituent of wood. These fibers provide the wood
with mechanical strength to resist external sources of mechanical stress such
as wind. They are also crucial to allow and support the growth in the height
of the trees [1]. But more importantly, they have a high economic importance
as they are the basic constituent of most of the wood derived products. Wood
fibers are extracted by pulping, which separates these cells into independent
fiber cells. They can then be reassembled as paper, and fibers are
increasingly used in applications such as in bio-composites, smart papers, and
new packaging and materials. In addition to fibers, wood comprises vessels,
that together with the fibers generally determine the characteristics of wood
[2]. Nevertheless, one key factor for wood quality is the fiber length that
can serve in fiber-based products. Yet, we still know very little about how
these fibers develop and elongate and the genetic determinants that regulate
fiber length. With this knowledge, fiber properties could be improved to
produce better and stronger wood for construction and fiber-based products in
more sustainable production systems. The demand for renewable fiber-based
products is constantly increasing as we move away from fossil resources-based
products, thus we need to establish better tools to study fiber length in a
very robust and high-throughput manner.

A classical method to study wood fiber length consists of macerating (soaking)
wood samples in a solution that separates individual cells [3]. Cells
suspended in liquid solutions can be transferred onto microscopy slides for
examination. Once prepared, these slides can be observed using a standard
light microscope. The images captured can then be analyzed using image
processing software like ImageJ, which allows for the manual measurement of
individual fiber lengths [4]. This task is, however, highly time-consuming and
prone to user bias and errors during the manual measurement step. Overall,
this highly limits the throughput of this type of analysis. Alternatively, it
is possible to use so-called fiber analyzers. These machines allow the high
throughput image acquisition of fibers floating in a constantly stirred
solution, generating high-speed and unbiased measurements [5]. However, the
resolution of data acquisition is often limited, and it can be difficult to
differentiate individual fibers from clumps of fibers that have not been
properly detached from each other. While this may not be a limitation for
industrial applications, it can become limiting when it comes to accurately
studying the biology of fibers and their individual length, width, and other
shape descriptors. This type of equipment can also be very costly and is not
widely available to many research labs contrary to light microscopes, even
those equipped with a motorized stage. A more appropriate solution for
fundamental research application may thus be to make use of commonly available
light microscopes equipped with motorized stages to acquire large fields of
view of wood macerates mounted between slides and cover-slips. This can then
be coupled with new image processing technologies to automate the time-
consuming task of identifying and measuring fiber cells in those high-
resolution images.

While most recent light microscopes are equipped with motorized stages and the
capacity to automatically generate large stitched images, the remaining
limiting step is the image processing for the detection (segmentation) and
shape analysis of the fibers. Several studies have developed image processing
methods to analyze fibers from wood section images [6, 7, 8, 9, 10, 11, 12].
However, most of these methods are only adapted for images from cross-sections
of wood samples. These samples are typically prepared in a transverse
orientation. Although this method provides crucial information about certain
aspects of cell arrangement and wood density, it does not directly yield
information about the length of fiber cells. Those images are also less
challenging for classical image segmentation tasks, as individual cells in the
images do not overlap. As such, the image can be divided into regions (e.g.,
individual cells and background) where each pixel is only assigned to one cell
or region. On the other hand, 2D images obtained from wood macerates contain
many fibers that frequently overlap. In turn, many pixels in the image can
belong to more than one cell of interest. This is challenging because most
existing segmentation algorithms, such as thresholding, watershed, and even
most machine learning and deep learning approaches, do not handle this type of
case when solving segmentation tasks.

Deep learning is a type of machine learning that has been very successful in
image and language tasks [13, 14, 15, 16, 17, 18]. Deep learning uses neural
networks called deep neural networks (DNNs) that are trained on large amounts
of data to perform a task directly without needing human-designed features
[19]. Convolutional neural networks (CNNs) are a type of deep neural network
well-suited for processing images [20]. CNNs apply a series of filters to the
input to automatically learn and extract useful features from the images [21].
This removes the need for humans to design feature extractors manually. CNNs
have been widely used for image segmentation [22, 23], including for
segmenting cells in microscope images in biomedicine [24, 25]. Because in our
images of wood macerates, cells often overlap, we need a neural network
architecture that allows object detection and that could thus fully identify
individually crossing and overlapping cells rather than simply splitting the
image into regions. This makes instance segmentation the best approach since
it identifies each individual cell and assigns pixels to the appropriate cell
instance. Popular existing instance segmentation methods such as Faster R-CNN
[26], Mask R-CNN [27], and RetinaNet [28] have been successfully applied to
cellular image segmentation [29, 30, 31]. These methods use deep convolutional
neural networks (CNNs) like VGG [32] and ResNet [33] to extract features from
the input images. They rely on anchors, which are predefined bounding boxes of
certain shapes and sizes tiled across the image, to detect multiple
overlapping cells. Multiple anchors are assigned to each pixel. Based on the
extracted features, the network then predicts the probability that each anchor
contains a cell. Non-maximum suppression (NMS) [34] selects the anchors with
the highest probabilities as the final cell detections. However, these two-
stage methods are slow for inference. One-stage methods such as Single Shot
MultiBox Detector (SSD) [35] and You Only Look Once (YOLO) series streamline
object detection by simultaneously predicting object locations and class
probabilities in a single pass. This approach significantly enhances the speed
of inference compared to multi-stage processes. Among one-stage models, YOLO
offers a good balance of speed and accuracy for real-time applications. Given
that wood images often feature numerous overlapping cells, we propose that the
YOLO algorithm is particularly well-suited for addressing this challenge.

In this paper, we develop a YOLOv8 [36] based model to segment and classify
fiber and vessel cells from 2D microscopy images. We initially created a new
dataset consisting of wood macerate samples and their images. This dataset
includes 3850 images, which encompass 28 358 individual fibers and 1502
individual vessels. Each of these was manually annotated to retrain the YOLOv8
model. Subsequently, we demonstrate that the retrained YOLOv8 model achieves
rapid inference speeds while maintaining high accuracy in identifying and
classifying individual vessels and fibers. Furthermore, the model proves
highly efficient even when processing large tiled images (32 640 x 25 920). To
facilitate the use of this retrained model, we developed a browser-based
interface. This interface, accessible after local installation, enables users
to upload images via drag and drop. The system then provides measurements of
the fibers and vessels, including their length, width, and area. Additionally,
users can access the training and prediction code run on Google Colab [37] in
the GitHub repository [38]. Lastly, to validate the effectiveness of our new
methods, we applied them to a well-known poplar transgenic line, which is
recognized for having distinctly longer wood fibers than its wild-type
relative. This approach aimed to corroborate previous manual measurements of
fiber length differences in these lines.

Figure 1: Schematic of the YOLOv8 architecture for fiber and vessel
segmentation. The model contains a Feature Extractor for feature extraction,
Feature Fusion for feature aggregation, Prediction Head for predicting the
objects’ bounding boxes, classes, and masks. The loss component is used to
optimize the model performance. An input image is passed through the network,
which performs classification, detection, and segmentation jointly. This
enables the delineation of individual cells even when overlapping, as shown in
the prediction output.

##  2 Image capturing method

###  2.1 Sample preparation

Stem samples from approximately 3-month-old greenhouse-grown hybrid aspen
(Populus tremula L. × tremuloides Michx.) clone T89 were collected between
internodes above 10 centimeters from the soil. Initially, the bark was removed
from these stem segments and then the equal size of the xylem region was
trimmed from these wood samples starting (2-3 mm) after the cambium region to
macerate them into individual vessels and fiber cells. Maceration was
performed on xylem wood samples by immersing them in maceration solution (30%
Hydrogen peroxide: glacial acetic acid, 2:1 v/v) and heating at 90°C with
periodic shaking for 5 hours, as previously described in [3]. The macerated
solution was then sedimented by low-speed centrifugation (1500 rpm), washed a
few times with water, and imaged immediately with a microscope.

3-month-old greenhouse transgenic trees overexpressing the GA20ox1 gene
¨Arabidopsis gibberellin 20-oxidase¨ (Ara GA20ox1 Line 1A) [5] were used with
their respective controls as T89. The stem segments from 3-3 trees each from
the overexpressing line and wild type T89 were sampled, macerated, stained,
and finally imaged with similar conditions or parameters as described above.
Four 361-tile stitched images (containing fibers and vessels) were obtained
from the trimmed stem samples of each of these trees for both the control and
overexpression line.

###  2.2 Imaging

A few drops from the obtained macerated solution were first stained with the
safranin solution (1 %) that stains lignified tissues in xylem cells. Then,
the samples were imaged on a glass slide using a Leica DMi8 inverted
microscope in brightfield mode with transmitted white light (Leica
Microsystems, Germany). Individual images were acquired using a 10X objective
lens and DFC7000T color camera with a pixel size of 4.54 µm and a resolution
of 1920 x 1440. Tile images made of 19 x 19 (361) individual images were
acquired with the Navigator function of the microscope using a 10 % overlap.
Tile images were merged by the LAS X software (Leica Microsystems, Germany).

###  2.3 Statistics

We used a t-test statistical method to validate whether the trained model
outputs consistent results across different image groups. Specifically, we
tested for scale invariance by running the model on different-sized crops from
the same large images. We also compared model predictions on an overexpression
line GA20ox 1A and wildtype T89 as control. The t-statistic is calculated as
$t=\frac{(\bar{X}_{1}-\bar{X}_{2})}{\text{SED}}$, where $\bar{X}_{1}$ and
$\bar{X}_{2}$ are the sample means and SED is the standard error of the
difference between the means. We employed Scipy’s $ttest_{i}nd()$ to
automatically compare two independent data samples, assessing significant mean
differences based on t-test assumptions.

##  3 Computational Method

###  3.1 Images annotation and dataset preparation

To identify fibers and vessels in microscope images, we created a dataset of
over 1300 individual images showing these structures in different shapes and
sizes. The images are 1920 x 1440 pixels in size. We carefully outlined the
fibers and vessels in each image using the VGG Image Annotator [39]. This
created a ground truth or guide to the actual fibers and vessels in the
images. The outlines were saved as JSON files, which store the coordinates of
the polygons drawn around each object, which can be seen in Fig. S1. We
resized the large images into smaller 1024 x 1024 pixel images to avoid
running out of memory on our graphics card. We also used data augmentation
techniques to increase the number of training images. For example, by applying
transformations like rotating, scaling, and flipping the original images, we
created more variety in the dataset. This also helps the machine learning
model learn robust features that apply to new images. From 1300 original
images, we generated 3850 augmented images with around 29 861 annotated
objects.

To train the YOLO model, we randomly split the dataset into training (85%) and
validation (15%) sets. The training data is used to update the model’s
parameters. The validation data is used to evaluate the model during training
but not to update parameters. This split helps prevent overfitting and ensures
the model generalizes well.

###  3.2 Deep learning approach

We used the YOLOv8-seg deep learning method, which is a variant of YOLOv8
architecture designed explicitly, for instance, segmentation tasks [36].
During the model training process, we utilized the pre-trained YOLOv8 model on
the COCO val2017 dataset as a starting point. The architecture of the YOLOv8
algorithm consists of four main components: feature extractor, feature fusion,
prediction head, and loss function. The components are shown in Fig. 1 and
with more details in Fig. S2. Here, we look into the design concepts of each
architecture module.

The feature extractor is the first part of the model and is responsible for
extracting features at different stages from the input image. The output
features from the different stages have different spatial resolutions. The
earlier stages of the feature extractor network extract low-level features
such as edges and corners. The later stages extract high-level features such
as object shapes and parts. The feature extractor down-samples the input image
because it extracts features at later stages.

The feature fusion module combines the output features from different stages
of the feature extractor network to form a unified representation of the
image. Deep neural networks capture increasingly detailed features as the
network becomes deeper, which improves object prediction. However, as the
network depth increases, the object localization accuracy for detecting small
objects decreases owing to excessive convolution operations, resulting in the
loss of important information. To address this tradeoff, the feature fusion
module incorporates a multi-scale fusion of features using architectures such
as a Feature Pyramid Network (FPN) and Path Aggregation Network (PAN). The
feature fusion module performs different operations to extract higher-level
features from the input features and consolidate the outputs from various
stages of the feature extractor into a single representation. This unified
representation enhances the object detection and segmentation performance.

The prediction-head module transforms the encoded image features into usable
predictions for object detection and segmentation. This makes the final
predictions based on the consolidated representation from the feature-fusion
module. The head module combines features from earlier modules and leverages
them to predict bounding boxes, classes, and masks for the objects in the
image. By dividing the prediction into specialized branches, it efficiently
performed classification, localization, and masking in a coordinated manner.
The prediction head is the final component that outputs the actual detections
and segmentations after processing using the complete YOLOv8 architecture.

The loss function in YOLOv8 measures how accurately the model detects and
segments objects. It compares the model predictions with the ground-truth
labels. The loss function is used to train the model to improve its
performance. YOLOv8 has separate branches for classification, bounding-box
regression, and masking. For classification and masking, the cross-entropy
loss was used to minimize errors. Bounding box detection uses two losses:
distributed focal loss (DFL) and CIoU Loss. These consider the aspect ratio
between the predicted and ground-truth boxes. The overall loss is the sum of
the losses from different branches. A lower loss indicates that the model is
more accurate at detecting, classifying, and segmenting objects. Loss guides
model training to improve these areas. The detailed layer-wise architecture is
illustrated in Fig. S2.

###  3.3 Measuring the morphology of objects

To accurately measure the morphological traits of objects detected in images,
we utilize a combination of deep learning-based instance segmentation
prediction and traditional image processing techniques. Specifically, we
leverage the YOLOv8 model to identify and localize objects of interest within
images, predicting bounding boxes and contours around each instance. However,
the predicted bounding boxes may not tightly envelop the objects. To enable
more precise morphological measurement, we integrate functions from the OpenCV
computer vision library to further analyze the object contours [40].

Once YOLOv8 predicts axis-aligned bounding boxes and contours of detected
objects, we calculate a minimum area bounding rectangle that encloses all the
contour points using OpenCV’s cv2.minAreaRect() built-in function. This
rotated rectangle fits the object contour much more tightly than the original
bounding box. We extract the coordinates of the four corners of this rectangle
using cv2.boxPoints(). this gives us a precise measurement of the object’s
length and width based on the rectangle side lengths. The longer side is taken
as the length of the object, while the shorter side corresponds to the width.
Pixel distances are converted to real-world units by multiplying with a scale
factor depending on the microscopy resolution.

In addition to length and width, we leverage OpenCV’s cv2.contourArea() method
to accurately measure the area enclosed within the detected contours. This
gives a more precise estimate of object area than simply using the bounding
box area. The object detection model’s bounding boxes provide a good initial
localization, but contour analysis gives significantly more accurate
morphological measurements. A schematic diagram showing the concept is shown
in Fig. 2, where each grid cell represents a pixel point. The corner points of
the rotated rectangle that fits with the minimum area on detected contours of
objects are denoted as P1(x1, y1), P2(x2, y2), P3(x3, y3), and P4(x4, y4). $a$
and $b$ refer to the number of pixels in the detection box’s length and width,
respectively, calculated using Eq. 1 and 2. The object’s length is determined
by selecting the highest value between the two parameters $a$ and $b$.
Finally, the average width is calculated by dividing the object area by the
length.

$a=\sqrt{(x1-x2)^{2}+(y1-y2)^{2}}.$ (1)

$b=\sqrt{(x2-x3)^{2}+(y2-y3)^{2}}.$ (2)

Figure 2: Example of a segmented fiber and the corresponding image analysis
that automatically measures the object’s morphological traits.  Table 1: The
table compares YOLOv8m-seg, YOLOv8l-seg, and YOLOv8x-seg models trained on our
generated dataset using both SGD and Adam optimizers.

Model | Optimizer | Best Epoch | mAP 50 |  | mAP 50-95
---
| Speed
---
GPU V100 (ms)
YOLOv8m-seg | SGD | 466 | 0.93 | 0.73 | 7.9
| Adam | 486 | 0.95 | 0.75 | 8.0
YOLOv8l-seg | SGD | 452 | 0.93 | 0.75 | 16.2
| Adam | 505 | 0.94 | 0.76 | 15.7
YOLOv8x-seg | SGD | 576 | 0.93 | 0.76 | 21.4
| Adam | 591 | 0.95 | 0.78 | 20.9

##  4 Experimental setup

###  4.1 YOLOv8 model retraining and hyperparameter tuning with wood macerate
dataset

During the training process of all models, we utilized pre-trained YOLOv8
models on the COCO val2017 dataset as a starting point. To train the YOLO
model, we use 3350 images for training and 500 images for validation. The
training data is used to update the model’s parameters. The validation data is
used to evaluate the model during training but not to update parameters. We
have used YOLOv8-m, l, and x in our experiment. YOLOv8-m uses a medium-sized
feature extractor and more feature fusion levels, YOLOv8-l utilizes a larger
feature extractor and most feature fusion levels compared to YOLOv8-m, and
YOLOv8-x uses an even larger feature extractor and more feature fusion
compared to YOLOv8-l. We find the convergence level and best optimizer for
these models during training. Based on experimental data from Ultralytics, we
observed that YOLOv5 training required 300 epochs, while YOLOv8 training
increased the number of epochs to 600. Initially, we set the number of epochs
to 600 and incorporated a patience value of 50. This means that if no
noticeable improvement occurred after waiting for 50 epochs, the training
would terminate early. However, during the training of YOLOv8m, we found that
the model reached its best performance at epoch 486 and training stopped early
at epoch 534.

We chose hyperparameters for model training as suggested in reference [41].
The selection of an appropriate optimization algorithm is crucial when
training YOLOv8-seg. The optimizer determines how the model parameters are
updated during training to minimize the loss function. For small custom
datasets, the Adam (Adaptive Moment Estimation) optimizer is recommended,
while the SGD (Stochastic Gradient Descent) optimizer tends to perform better
on larger datasets. Consequently, we trained YOLOv8-seg models separately
using the Adam and SGD optimizers. The results of comparing the effects of
these two optimizers on model training are presented in Table 1.

We opted for the Adam optimizer with a weight decay of 5x$10^{-4}$ and an
initial learning rate of 1x$10^{-3}$. Furthermore, we set the input image size
to 1024 and trained the different models using a TITAN V100 16GB with a batch
size of 8. The models were trained on Python 3.8 and PyTorch 1.10.0. We
recommend using Python 3.7 or higher and PyTorch 1.7 or higher for training.
It is important to note that utilizing GPUs with more significant memory and
higher computational power can significantly enhance the speed of model
training.

###  4.2 Evaluation metrics

To evaluate the performance of the models, we use four metrics: precision,
recall, mean average precision (mAP), and F1-score. These metrics are commonly
used for object detection and segmentation tasks, and they are calculated
based on the number of true positive (TP), true negative (TN), false positive
(FP), and false negative (FN) predictions made by the model.

####  4.2.1 Intersection over Union (IoU)

To assess the accuracy of object detection algorithms we use the Intersection
over Union (IoU) metric. It measures how well the algorithm’s predicted
bounding box overlaps with the ground truth bounding box. To calculate IoU, we
compare the area of overlap between the predicted bounding box (referred to as
the ”candidate” box) and the ground truth bounding box (referred to as the
”ground truth” box) to their combined area. The formula for IoU is:

$IoU=\frac{Area(C)\cap Area(G)}{Area(C)\cup Area(G)}$

Here, $C$ represents the candidate bounding box, and $G$ represents the ground
truth bounding box where the actual object is located. The IoU value ranges
from 0 to 1, with a higher value indicating a better algorithm performance. A
higher IoU signifies a smaller difference between the candidate and ground
truth bounding boxes, which implies a more accurate object detection result.

####  4.2.2 Precision-Recall Curve

The Precision-Recall Curve (P-R Curve) is a graphical representation that
illustrates the relationship between the recall and precision metrics. In this
curve, the x-axis represents the recall values, while the y-axis represents
the precision values. Each point on the curve corresponds to a specific
threshold value used in the algorithm. The points are connected to form a
curve that provides insights into the trade-off between recall and precision.
To calculate the recall (R) and precision (P) values for each point on the
curve, we use the following equations:

$Precision=\frac{TP}{(TP+FP)}$ $Recall=\frac{TP}{(TP+FN)}$

By analyzing the P-R Curve, we can gain insights into the trade-off between
precision and recall at different threshold values, helping us understand the
performance of an algorithm in segmentation tasks.

####  4.2.3 F1-score

With the F1-score we can assess the accuracy of the models, providing a
balanced evaluation by considering both precision and recall. It represents
the harmonic mean of precision and recall and is defined as,

$F1-score=\frac{2PR}{(P+R)}=\frac{2TP}{(2TP+FP+FN)}.$

In short, the F1-score combines both precision and recall giving a single
value that represents the algorithm’s overall accuracy. A higher F1-score
indicates a better algorithm performance in achieving both high precision and
high recall simultaneously.

####  4.2.4 Mean Average Precision (mAP)

Mean Average Precision (mAP) is a widely adopted evaluation metric for
assessing the performance of object detection algorithms across multiple
classes. In this paper, we considered mAP 50 and mAP 50-95, in which, mAP 50
calculates the average precision for all classes at an IoU threshold of 0.5
while mAP 50-95 computes the average precision for all classes over a range of
IoU thresholds from 0.5 to 0.95, with a step size of 0.05. This variation of
mAP offers a more comprehensive evaluation by considering a wider range of IoU
thresholds.

##  5 Result and Discussion

###  5.1 Model Selection

In this work, we have used the pre-trained yolov8 for fiber and vessel
detection as well as segmentation. We have used 3 YOLO variants m, l, and x
with our custom dataset. YOLOv8-m used a medium-sized feature extractor,
YOLOv8-l utilized a larger feature extractor compared to YOLOv8-m, and
YOLOv8-x used an even larger feature extractor and more feature fusion
compared to YOLOv8-l. The quantitative results for precision, recall, F1
score<EMAIL_ADDRESS>and<EMAIL_ADDRESS>values of the three YOLOv8 models in fiber
and vessel segmentation are presented in Table 2. Among these models,
YOLOv8m-seg demonstrated the highest precision (0.97), recall (0.91), and F1
score (0.94), while performing relatively lower in mAP: 0.5-0.95 (0.75).
YOLOv8l-seg outperformed YOLOv8m-seg only in mAP: 0.5-0.95 (0.76).

Based on these findings, the YOLOv8x-seg model was identified as the optimal
choice for fiber and vessel detection and segmentation, exhibiting superior
performance across the evaluation metrics of<EMAIL_ADDRESS>and<EMAIL_ADDRESS>Consequently, the YOLOv8x-seg model was selected for further analysis,
specifically in estimating fiber and vessel length, width, and area. This
selection ensures a consistent and focused evaluation of the model’s practical
application, aligning with the study’s objectives.

Table 2: The table shows results for detecting fibers and vessels using YOLOv8
models. PARAM(M) is a trainable parameter in the model.

Model | Precision | Recall | F1-Score | mAP 50 | mAP 50-95 |  | Param (M)
---
YOLOv8m-seg | 0.97 | 0.91 | 0.94 | 0.95 | 0.75 | 27.2
YOLOv8l-seg | 0.96 | 0.91 | 0.94 | 0.94 | 0.76 | 45.9
YOLOv8x-seg | 0.97 | 0.90 | 0.93 | 0.95 | 0.78 | 70.1

###  5.2 Model Performance

We thoroughly evaluated model performance for the tasks of detecting and
segmenting fiber and vessel objects in images. We examined precision-recall
curves to understand the trade-off between precision and recall for detection.
For segmentation, we focused on how precisely the model delineates the
boundaries and segments of the detected objects. Additionally, we analyzed
F1-confidence curves to understand the relationship between F1 scores and
model confidence levels for detection and segmentation. Examining the
F1-confidence curves provided insights into how precision and recall were
balanced across varying confidence thresholds. Evaluating the model’s
performance step-by-step is important for assessing its effectiveness.

Figure 3 depicts the behavior of the selected yolov8x-seg model used for
object detection and segmentation of fiber and vessel objects. The precision-
recall curve in Fig. 3(a) represents the trade-off between precision and
recall for the detection task. At a threshold of 0.5, the mean average
precision (mAP) values were 0.951 for fiber and 0.959 for vessel detection.
The overall mAP of 0.955 for all classes combined indicated the model’s
overall performance in object detection.

Figure 3(b) shows the F1-confidence curve, which illustrates the relationship
between the F1 score and the model’s confidence. At a confidence threshold of
0, the F1-score was 0.95 for both fiber and vessel classes. This score
represents the balance between precision and recall. A higher F1-score
indicates better model performance in terms of both precision and recall.
These findings provide valuable insights into the model’s performance and help
assess its suitability for detecting fiber and vessel objects in images.

Moving on to the segmentation task, the model achieved values of 0.942 and
0.959 at a threshold of 0.5 for fiber and vessel segmentation, respectively.
The overall mAP of 0.95 for all combined classes at the same threshold is
shown in Fig. 3(c), highlighting the model’s effectiveness in segmenting
objects. Figure 3(d) displays the F1-confidence curve for the segmentation
task, where the model attained an F1-score of 0.94 at a confidence threshold
of 0. This score represents the trade-off between precision and recall for
both fiber and vessel classes.

Figure 3: These plots showcase the precision-recall and F1-confidence curves
used to evaluate the performance of YOLOv8x in detecting and segmenting fibers
and vessels. The model demonstrates strong mAP and F1-score across thresholds,
confirming its effectiveness in object detection and segmentation tasks.

We conclude that for fiber and vessel detection and segmentation, the
YOLOv8x-seg model performed best compared to other models in terms of mAP. The
YOLOv8m-seg model had the lowest mAP value specifically for fiber detection
and segmentation. It is worth noting that the task of segmenting fibers and
vessels is quite challenging due to the overlap of objects in the images.

###  5.3 Qualitative Results

Figure 4 shows fiber and vessel detection and segmentation examples achieved
using the YOLOv8x-seg model. These examples highlight the model’s ability to
accurately identify and outline fiber and vessel structures in the images. The
visual results obtained from the model not only demonstrate its potential
within the scope of our study but also provide valuable insight into its
practical application for fiber and vessel analysis.

We highlight that we conducted experiments using various background images.
Still, the model successfully detected and segmented the fiber and vessel
structures despite the differences in the image backgrounds. This highlights
the effectiveness of the model, as it can perform well regardless of the
background variations. Such robustness is particularly valuable in fiber and
vessel segmentation, where images with diverse backgrounds are encountered.

Figure 4: The images depict YOLOv8 effectively detecting and segmenting
variously shaped and sized fibers and vessels against diverse backgrounds,
showcasing its robust performance across different scenarios.

Based on the findings of this study, we conclude that the YOLOv8 model family
effectively detects and segments fibers and vessels in microscopy images.
Despite achieving high accuracy in fiber detection and segmentation, the
YOLOv8 model still made some mistakes by generating false positives and false
negatives in certain cases. We highly examples of this in Fig. 5 (a), where
the model failed to identify a fiber segment, as highlighted by the yellow
ellipse. This failure to detect the fiber was likely due to limitations in the
training of the model. Similarly, in Fig. 5(b), the ellipse region highlights
where YOLOv8 failed to identify the fiber and vessel. This is mainly because
the number of samples used in this study for training the model is limited. To
tackle this issue, Adar et al. [42] suggested that training the model with a
larger dataset containing more input features can greatly enhance its ability
to generalize and perform well on new and unseen data. A larger dataset would
enable the model to capture the subtle differences in fiber and vessel
structures in various background images. Additionally, a larger dataset can
help to mitigate the risk of overfitting, where the model becomes too
specialized to the training dataset and performs poorly on new samples.

Figure 5: The detection of fibers and vessels encountered some challenges, as
shown in these examples where (a) a fiber was not detected, and (b) both a
fiber and a vessel were not detected.

###  5.4 Quantifying fibers and vessel in microscopy images

In order to complete the task of fiber and vessel detection and segmentation,
we implemented standard measurements to describe the shape of the objects
using the model output. Specifically, we evaluated how well our model
performed with images of different sizes. We separated the images into two
categories based on size. The first type measured 1920 x 1440 pixels and was
used to train the model. The second type was a large image of 33 384 x 25 112
pixels stitched from 361 small 1920 x 1440 images. The model first identified
the locations of fibers and vessels in the image and then measured the
morphological traits of the detected objects using the method described in
subsection 3.3. For both image types, we used the trained model to measure
characteristics like the length, width and area of the fibers and vessels
detected. The total detected items in two large images are 1678 fibers and 117
vessels.

Figure 6: Boxplots depict fibers’ and vessels’ length, width, and area
measurements from two large images of 33,384 x 25,112 pixels. Fibers range
from 250-450 $\mu m$ long, 20-30 $\mu m$ wide, and 6000-11 000 $\mu m^{2}$ in
area. Vessels are 220-270 $\mu m$ long, 42-68 $\mu m$ wide, and 10 000-22 000
$\mu m^{2}$ in area. The box plots show the 1st and 3rd quartiles (box
limits), mean values (cross sign), and median values (solid line).

Figure 6 shows box plots summarizing the measurements of the detected fibers
and vessels. For most fibers, the length ranged from 250-450 $\mu m$, the
width ranged from 20-30 $\mu m$, and the area ranged from 6000-11 000 $\mu
m^{2}$. The average fiber length was 350 $\mu m$, the average width was 24
$\mu m$, and the average area was 8,561 $\mu m^{2}$.

For most vessels, the length ranged from 220-270 $\mu m$, the average width
ranged from 42-68 $\mu m$, and the area ranged from 10 000-22 000 $\mu m^{2}$.
The average vessel length was 280 $\mu m$, the average width was 56 $\mu m$,
and the average area was 16 354 $\mu m^{2}$. In the supplementary materials,
scattered and histogram plots offer a detailed visualization of the
distribution and relationship between the length, width, and area in the
image, as depicted in Fig. S3.

Further, testing on different image sizes allowed us to check if the model
performance stayed consistent regardless of size changes. The results show
that the model’s effectiveness is independent of image dimensions. This scale-
invariance makes the model more robust and can analyze images of varying sizes
which are encountered in real applications. This property is crucial for real-
world problems, as different image dimensions will inevitably be encountered
in practice. To evaluate scale invariance and object detection and
segmentation abilities, including overlapped objects, 20 small images of size
1920 x 1440 pixels were randomly selected, containing 137 fibers and 20
vessels. Measurements were compared to those from the large 33 384 x 25 112
pixel image.

We found the model extracted aligned average dimensions for fibers across
these two image sizes in which overlapped fibers were also detected. This
includes a length of $\approx$ 350 $\mu m$, width of $\approx$ 25 $\mu m$, and
area of $\approx$ 9000 $\mu m^{2}$ for the fibers. For fiber length, width,
and area, the p-values were 0.964, 0.998, and 0.975, respectively, indicating
no statistically significant difference between small and large image
measurements. However, we noted a variability when comparing the vessel
measurements with a lower average length of 248 $\mu m$ and, higher average
width of 65 $\mu m$. The vessel area averages were closer between image sizes
at 16 556 $\mu m^{2}$ on small images and 16 892 $\mu m^{2}$ on large images.
The p-values for vessel length, width, and area were 0.299, 0.124, and 0.413,
respectively. These results illustrate the scale invariance performance of the
model across different image dimensions. In the supplementary material, we
also provide the analysis of 20 small-sized images (1920 x 1440), see Fig. S4
and two mid-sized images consisting of 99 tiles (8275 x 7250), see Fig. S5,
using box plots, scatter plots, and histogram plots to assess the robustness
of the model.

In conclusion, the capability to reliably extract metrics for fibers and
vessels including overlapped at different scales highlights this model’s
strength in handling overlapping objects and scale invariance. These abilities
are crucial for real-world applications requiring robust detection and
segmentation alongside invariant performance across image sizes.

###  5.5 Comparison of GA20ox 1A line and wildtype T89

To further evaluate our approach for research applications, we tested it on a
new dataset consisting of 12 large images of samples taken from the T89
wildtype tree and 12 large images from the transgenic line overexpressing the
GA20ox1 gene. This transgenic line was previously reported to show
approximately 10 % increase in fiber length compared to the wildtype T89 [5].
In total, the model identified 5651 fibers in the 12 T89 images with an
average length of 305.26 $\pm$ 111 $\mu m$ and for the GA20 line it identified
5782 fibers with an average of 354.30 $\pm$ 78 $\mu m$, see Fig. 7. A student
t-test indicated that the samples were highly significant different, as
evidenced by a p-value less than 0.0001. These results show an approximately
15 % length increase for the GA20ox line compared to the wildtype, in line
with previously reported results.

Figure 7: Violin plots that depict fibers’ length measurements from large
images of 33 384 x 25 112 pixels from T89 and GA20ox line. The envelope shows
the distribution, the thick center lines represent the 1st and 3rd quartiles
(black line limits), and median values (center yellow dots).

###  5.6 GUI application

We have developed a desktop application using Python’s Flask framework,
designed to simplify the process of uploading and analyzing images with our
automated algorithm. The application’s workflow is shown in Fig. S6. Users can
select an image from a gallery or upload one via drag-and-drop. Once an image
is uploaded, the application predicts the presence of fibers and vessels. It
also quantifies each detected object, with results available in a downloadable
data file. This file includes details like object type, length, width, and
area. When installed on a local computer with an i9 10th Gen processor and
32GB of RAM, the application processes a 1024 x 1024 image in an average of
140 milliseconds.

####  5.6.1 Github resources

Detailed instructions for installation of the developed code and instructions
for implementation are found in the supporting information. Additionally,
users can find instructions on how to retrain and test the model using Google
colab in the GitHub resources [38]. The ’Train$\\_$custom$\\_$data.ipynb’ file
can be used to retrain the model; simply follow the instructions provided in
the file. The ’prediction$\\_$file.ipynb’ file is used to test the model on
your custom dataset.

##  6 Conclusion

This paper introduces a deep learning solution using YOLOv8 to automatically
analyze and quantify wood fibers and vessels in challenging microscope images,
offering high-throughput capabilities. To achieve this, we trained multiple
YOLOv8 models on diverse wood image datasets and evaluated their performance
in detecting and segmenting fibers and vessels. The most robust model was
chosen. The model can consistently and reliably extract essential cell metrics
across different image scales, such as length, width, and area. The model’s
consistent metric extraction underscores its strong practical applicability.
We also created a web application pipeline that is useful in practical
situations. Users can then upload images for automatic cell counting and
extraction. Thus, we conclude that this study introduces an innovative high-
throughput method for analyzing wood cells in densely populated 2D microscopy
images, even when cells are partially obscured.

Supplementary information Additional information and user guides are included
in the supporting information file.

## Declarations

### Funding

The project was funded by Kempestiftelserna (JCK-2129.3) and the National
Academic Infrastructure for Supercomputing in Sweden (NAISS) at Umeå,
partially funded by Vetenskasrådet (2022-06725). The authors acknowledge the
facilities and technical assistance of the Umeå Plant Science Centre
Microscopy facility and the plant growth facility. This work was also
supported by grants from the Wallenberg Foundation (KAW 2016.0341 and KAW
2016.0352), VINNOVA (2016–00504), and the Novo Nordisk Foundation
(NNF21OC0067282) to S.V. We also thank Bio4Energy for supporting this work.

### Declaration of competing interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

### Consent for publication

The authors declare that they have no competing interests.

### Availability of data and code

All the code in this project was developed using python and different public
libraries as defined in the supporting information. The Google Colab project
is found here [37] and the code can be downloaded at a public GitHub
repository here [38].

### Authors’ contributions

Saqib Qamar: Data curation, Software, Methodology, Validation, Formal
analysis. Abu Imran Baba: Investigation, Validation. Stéphane Verge:
Conceptualization, Resources, Supervision. Magnus Andersson:
Conceptualization, Resources, Supervision, Funding acquisition, Project
administration. All authors contributed to Writing, review and editing.

### Ethics approval and consent to participate

No applicable

## References

  *   * [1] T. Gorshkova, N. Brutch, B. Chabbert, M. Deyholos, T. Hayashi, S. Lev-Yadun, E.J. Mellerowicz, C. Morvan, G. Neutelings, G. Pilate, Plant fiber formation: state of the art, recent and expected progress, and open questions.  Critical Reviews in Plant Sciences 31(3), 201–228 (2012)
  * [2] K. Wilson, D.J.B. White, et al., _The anatomy of wood: its diversity and variability._ (Stobart & Son Ltd., 1986)
  * [3] A. Siedlecka, S. Wiklund, M.A. Péronne, F. Micheli, J. Leśniewska, I. Sethson, U. Edlund, L. Richard, B. Sundberg, E.J. Mellerowicz, Pectin methyl esterase inhibits intrusive and symplastic cell growth in developing wood cells of populus.  Plant physiology 146(2), 554 (2008)
  * [4] C. Schneider, W. Rasband, K.W. Eliceiri, Nih image to imagej: 25 years of image analysis.  Nature methods 9(7), 671–675 (2012)
  * [5] M. Eriksson, M. Israelsson, O. Olsson, T. Moritz, Increased gibberellin biosynthesis in transgenic trees promotes growth, biomass production and xylem fiber length.  Nature biotechnology 18(7), 784–788 (2000)
  * [6] P. Kennel, G. Subsol, M. Guéroult, P. Borianne, _Automatic identification of cell files in light microscopic images of conifer wood_ , in _2010 2nd international conference on image processing theory, tools and applications_ (IEEE, 2010), pp. 98–103
  * [7] M. Aronsson, _On 3D fibre measurements of digitized paper_ , vol. 254 (2002)
  * [8] M. Donoser, H. Riemenschneider, H. Bischof, _Efficient partial shape matching of outer contours_ , in _Computer Vision–ACCV 2009: 9th Asian Conference on Computer Vision, Xi’an, September 23-27, 2009, Revised Selected Papers, Part I 9_ (Springer, 2010), pp. 281–292
  * [9] C. Lu, L.J. Latecki, N. Adluru, X. Yang, H. Ling, _Shape guided contour grouping with particle filters_ , in _2009 IEEE 12th International Conference on Computer Vision_ (IEEE, 2009), pp. 2288–2295
  * [10] S. Pan, M. Kudo, Recognition of wood porosity based on direction insensitive feature sets.  Trans. Mach. Learn. Data Min. 5(1), 45–62 (2012)
  * [11] G. Brunel, P. Borianne, G. Subsol, M. Jaeger, Y. Caraglio, Automatic identification and characterization of radial files in light microscopy images of wood.  Annals of botany 114(4), 829–840 (2014)
  * [12] H. Boztoprak, M. Ergun, Determination of vessel and fibers in hardwoods.  Gaziosmanpasa Journal of Scientific Research 6(2), 87–96 (2017)
  * [13] A. Voulodimos, N. Doulamis, A. Doulamis, E. Protopapadakis, et al., Deep learning for computer vision: A brief review.  Computational intelligence and neuroscience 2018 (2018)
  * [14] Z.Q. Zhao, P. Zheng, S.t. Xu, X. Wu, Object detection with deep learning: A review.  IEEE transactions on neural networks and learning systems 30(11), 3212–3232 (2019)
  * [15] E. Cambria, B. White, Jumping nlp curves: A review of natural language processing research.  IEEE Computational intelligence magazine 9(2), 48–57 (2014)
  * [16] M. Reichstein, G. Camps-Valls, B. Stevens, M. Jung, J. Denzler, N. Carvalhais, f. Prabhat, Deep learning and process understanding for data-driven earth system science.  Nature 566(7743), 195–204 (2019)
  * [17] S. Qamar, H. Jin, R. Zheng, P. Ahmad, M. Usama, A variant form of 3d-unet for infant brain segmentation.  Future Generation Computer Systems 108, 613–623 (2020)
  * [18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need.  Advances in neural information processing systems 30 (2017)
  * [19] Y. LeCun, Y. Bengio, G. Hinton, Deep learning.  nature 521(7553), 436–444 (2015)
  * [20] A. Biem, Neural networks: A review.  Data classification: Algorithms and applications pp. 205–244 (2014)
  * [21] S. Qamar, R. Öberg, D. Malyshev, M. Andersson, A hybrid cnn-random forest algorithm for bacterial spore segmentation and classification in tem images.  Scientific Reports 13(1), 18758 (2023)
  * [22] J. Long, E. Shelhamer, T. Darrell, _Fully convolutional networks for semantic segmentation_ , in _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ (2015), pp. 3431–3440.  10.1109/CVPR.2015.7298965
  * [23] V. Badrinarayanan, A. Kendall, R. Cipolla, Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  IEEE transactions on pattern analysis and machine intelligence 39(12), 2481–2495 (2017)
  * [24] O. Ronneberger, P. Fischer, T. Brox, _U-net: Convolutional networks for biomedical image segmentation_ , in _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ (Springer, 2015), pp. 234–241
  * [25] H. Fu, Y. Xu, S. Lin, D.W. Kee Wong, J. Liu, _Deepvessel: Retinal vessel segmentation via deep learning and conditional random field_ , in _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19_ (Springer, 2016), pp. 132–139
  * [26] S. Ren, K. He, R. Girshick, J. Sun, Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems 28 (2015)
  * [27] K. He, G. Gkioxari, P. Dollár, R. Girshick, _Mask r-cnn_ , in _Proceedings of the IEEE international conference on computer vision_ (2017), pp. 2961–2969
  * [28] T. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, S. Belongie, _Feature pyramid networks for object detection_ , in _Proceedings of the IEEE conference on computer vision and pattern recognition_ (2017), pp. 2117–2125
  * [29] J. Johnson, Adapting mask-rcnn for automatic nucleus segmentation.  arXiv preprint arXiv:1805.00500 (2018)
  * [30] H. Tsai, J. Gajda, T.F. Sloan, A. Rares, A.Q. Shen, Usiigaci: Instance-aware cell tracking in stain-free phase contrast microscopy enabled by machine learning.  SoftwareX 9, 230–237 (2019)
  * [31] R. Hollandi, A. Szkalisity, T. Toth, E. Tasnadi, C. Molnar, B. Mathe, I. Grexa, J. Molnar, A. Balind, M. Gorbe, et al., A deep learning framework for nucleus segmentation using image style transfer.  Biorxiv p. 580605 (2019)
  * [32] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition. arxiv 1409.1556 (09 2014).  URL https://arxiv. org/abs/1409.1556. Accessed: February (2020)
  * [33] K. He, X. Zhang, S. Ren, J. Sun, _Deep residual learning for image recognition_ , in _Proceedings of the IEEE conference on computer vision and pattern recognition_ (2016), pp. 770–778
  * [34] A. Neubeck, L. Van Gool, _Efficient non-maximum suppression_ , in _18th international conference on pattern recognition (ICPR’06)_ , vol. 3 (IEEE, 2006), pp. 850–855
  * [35] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.Y. Fu, A.C. Berg, _Ssd: Single shot multibox detector_ , in _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14_ (Springer, 2016), pp. 21–37
  * [36] G. Jocher, A. Chaurasia, J. Qiu.  Ultralytics yolov8 (2023).  URL https://github.com/ultralytics/ultralytics
  * [37] E. Bisong, _Google Colaboratory_ (Apress, Berkeley, CA, 2019), pp. 59–64
  * [38] S. Qamar.  fiberseg (2023).  URL https://github.com/sqbqamar/fiberseg
  * [39] A. Dutta, A. Zisserman.  The via annotation software for images, audio and video (2019)
  * [40] G. Bradski, The OpenCV Library.  Dr. Dobb’s Journal of Software Tools (2000)
  * [41] J. Glenn.  Yolov5 release v6. 1 (2022)
  * [42] M. Frid, E. Klang, M. Amitai, J. Goldberger, H. Greenspan, _Synthetic data augmentation using GAN for improved liver lesion classification_ , in _2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018)_ (2018), pp. 289–293.  10.1109/ISBI.2018.8363576


11institutetext:  1 Department of Automation and Applied Informatics
Budapest University of Technology and Economics2 Institute for Computer
Science and Control

# The Role of Interpretable Patterns in Deep Learning for Morphology

Judit Ács1,2 András Kornai2

###### Abstract

We examine the role of character patterns in three tasks: morphological
analysis, lemmatization and copy. We use a modified version of the standard
sequence-to-sequence model, where the encoder is a pattern matching network.
Each pattern scores all possible N character long subwords (substrings) on the
source side, and the highest scoring subword’s score is used to initialize the
decoder as well as the input to the attention mechanism. This method allows
learning which subwords of the input are important for generating the output.
By training the models on the same source but different target, we can compare
what subwords are important for different tasks and how they relate to each
other. We define a similarity metric, a generalized form of the Jaccard
similarity, and assign a similarity score to each pair of the three tasks that
work on the same source but may differ in target. We examine how these three
tasks are related to each other in 12 languages. Our code is publicly
available.111https://github.com/juditacs/deep-morphology

##  1 Introduction

Deep neural networks are successful at various morphological tasks as
exemplified in the yearly SIGMORPHON Shared Task[1, 2, 3]. However these
neural networks operate with continuous representations and weights which is
in stark contrast with traditional, and hugely successful, rule-based
morphology. There have been attempts to add rule-based and discrete elements
to these models through various inductive biases[4].

In this paper we tackle two morphological tasks and the copy task as a control
with an interpretable model, SoPa. Soft Patterns[5] or SoPa is a finite-state
machine parameterized with a neural network, that learns linear patterns of
predefined size. The patterns may contain epsilon transitions and self-loops
but otherwise are linear. _Soft_ refers to the fact that the patterns are
intended to learn abstract representations that may have multiple surface
representations, which SoPa can learn in an end-to-end fashion. We call these
surface representations _subwords_ , while the abstract patterns, _patterns_
throughout the paper. An important upside of SoPa is that interpretable
patterns can be extracted from each sample. [5] shows that SoPa is able to
retrieve meaningful word-level patterns for sentiment analysis. Each pattern
is matched against every possible subword and the highest scoring subword is
recovered via a differentiable dynamic program, a variant of the forward
algorithm. We apply this model as the encoder of a sequence-to-sequence or
_seq2seq_ 222also called encoder-decoder model model[6], and add an LSTM[7]
decoder. We initialize the decoder’s hidden state with the final scores of
each SoPa pattern and we also apply Luong’s attention[8] on the intermediate
outputs generated by SoPa. We call this model SoPa Seq2seq. We compare each
setup to a sequence-to-sequence with a bidirectional LSTM encoder,
unidirectional LSTM decoder and Luong’s attention.

We show that SoPa Seq2seq is often competitive with the LSTM baseline while
also interpretable by design. SoPa Seq2seq is especially good at morphological
analysis, often surpassing the LSTM baseline, which confirm our linguistic
intuition namely that subword patterns are useful for extracting morphological
information.

We also compare these models using a generalized form of Jaccard-similarity
and we find that some trends coincide with linguistic intuition.

##  2 Data

Universal Morphology or UniMorph is project that aims to improve how NLP
handles languages with complex morphology.333https://unimorph.github.io/
Specified in [9], UniMorph has been used to annotate 350 languages from the
English edition of Wiktionary444https://en.wiktionary.org/. Wiktionary
contains inflection tables that list inflected forms of a word. Part of the
UniMorph project is converting these tables into _(lemma, inflected form,
tags)_ triplets such as _(ablak, ablakban, N IN+ESS SG)_. The first tag is the
part-of-speech which is limited to the main open classes (nouns, verbs and
adjectives) in most languages, `IN+ESS` is the inessive case and `SG` denotes
singular.

###  2.1 Data sampling

Our goal is to sample 10000 training, 2000 development and 2000 test examples.
We retrieved 109 UniMorph repositories (109 languages) but only 57 languages
have at least 14000 samples, the lowest possible number for our purposes. We
first prefilter the languages and assign them to languages families and genus
using the World Atlas of Languages or WALS555https://wals.info/. WALS does not
contain extinct, constructed or liturgical languages, and we do not
incorporate these in our dataset. Out of the 109 languages, 19 have no WALS
entry. 29 languages have large enough UniMorph datasets that allow obtaining
10000/2000/2000 samples.666Albanian has only 1982 test samples but we wanted
to include it as a language isolate from the Indo-European family. Table 1
summarizes the dataset.

Language | Family | Genus | sample | lemma | paradigm | alphabet | F/L | POS
---|---|---|---|---|---|---|---|---
Arabic | Afro-Asiatic | Semitic | 138k | 4007 | 196 | 45 | 26.3 | NVA
Turkish | Altaic | Turkic | 213k | 3017 | 186 | 46 | 54.7 | NVA
Quechua | Hokan | Yuman | 178k | 1003 | 553 | 22 | 146.8 | NVA
Albanian | Indo-European | Albanian | 14k | 587 | 59 | 27 | 17.4 | NV
Armenian | Indo-European | Armenian | 259k | 6991 | 134 | 46 | 35.3 | NVA
Latvian | Indo-European | Baltic | 129k | 7238 | 78 | 34 | 10.3 | NVA
Lithuanian | Indo-European | Baltic | 33k | 1391 | 139 | 56 | 20.1 | NVA
Irish | Indo-European | Celtic | 45k | 7299 | 53 | 53 | 3.3 | NVA
Danish | Indo-European | Germanic | 25k | 3190 | 14 | 44 | 7.7 | NV
German | Indo-European | Germanic | 171k | 15032 | 37 | 63 | 4.5 | NV
English | Indo-European | Germanic | 115k | 22765 | 5 | 65 | 4.0 | V
Icelandic | Indo-European | Germanic | 76k | 4774 | 44 | 54 | 10.9 | NV
Greek | Indo-European | Greek | 147k | 11872 | 118 | 76 | 6.5 | NVA
Kurdish | Indo-European | Iranian | 203k | 14143 | 128 | 61 | 14.3 | NVA
Asturian | Indo-European | Romance | 29k | 436 | 223 | 32 | 49.5 | NVA
Catalan | Indo-European | Romance | 81k | 1547 | 53 | 35 | 40.6 | V
French | Indo-European | Romance | 358k | 7528 | 48 | 44 | 35.3 | V
Bulgarian | Indo-European | Slavic | 54k | 2413 | 95 | 31 | 18.9 | NVA
Czech | Indo-European | Slavic | 109k | 5113 | 147 | 62 | 10.0 | NVA
Slovenian | Indo-European | Slavic | 59k | 2533 | 94 | 56 | 8.9 | NVA
Georgian | Kartvelian | Kartvelian | 74k | 3777 | 109 | 33 | 17.5 | NVA
Adyghe | NW Caucasian | NW Caucasian | 20k | 1635 | 30 | 40 | 11.9 | NA
Zulu | Niger-Congo | Bantoid | 49k | 566 | 249 | 46 | 57.2 | NVA
Khaling | Sino-Tibetan | Mahakiranti | 156k | 591 | 432 | 32 | 91.5 | V
Estonian | Uralic | Finnic | 27k | 886 | 64 | 26 | 28.0 | NV
Finnish | Uralic | Finnic | 1M | 57165 | 97 | 50 | 27.1 | NVA
Livvi | Uralic | Finnic | 63k | 15295 | 104 | 55 | 4.0 | NVA
Northern Sami | Uralic | Saami | 62k | 2103 | 80 | 31 | 25.9 | NVA
Hungarian | Uralic | Ugric | 517k | 14883 | 93 | 53 | 34.1 | NV
Table 1:  Dataset statistics. The languages are sorted by language family. F/L
refers to the form-per-lemma ratio. POS indicates which part of speech are
present in the dataset out of the nouns, verbs and adjectives.

##  3 Tasks

We train both kinds of seq2seq models on three tasks: morphological analysis
(abbreviated as _morphological analysis_), _lemmatization_ , and _copy_ or
autoencoder. The source sequence is the inflected form of the word in all
three tasks, while the target sequence is a list of morphosyntactic tags for
morphological analysis, the lemma for lemmatization and the same as the source
side for copy. Table 2 shows examples for the three tasks.

Inflected words and lemmas are treated as a sequence of characters but tags
are treated as standalone symbols. We share the vocabulary and the embedding
between the source and target side when training for copy and lemmatization
but we use separate vocabularies for morphological analysis.

Language | Task | Source |  Target
---|---|---|---
Hungarian | morphological analysis | vásároljanak |  V SBJV PRS INDF 3 PL
Hungarian | morphological analysis | lepkékben |  N IN+ESS PL
English | morphological analysis | hugging |  V V.PTCP PRS
French | morphological analysis | désinstalleriez |  V COND 2 PL
Hungarian | lemmatization | vásároljanak |  vásárol
Hungarian | lemmatization | lepkékben |  lepke
English | lemmatization | hugging |  hug
French | lemmatization | désinstalleriez |  désinstaller
Hungarian | copy | vásároljanak |  vásároljanak
Hungarian | copy | lepkékben |  lepkékben
English | copy | hugging |  hugging
French | copy | désinstalleriez |  désinstalleriez
Table 2: Dataset examples.

##  4 Models

We train two kinds of sequence-to-sequence models which only differ in the
choice of the encoder. Both models first pass the input through an embedding.
We train the embeddings from randomly initialized values and do not use
pretrained embeddings. We use character embeddings with 50 dimensions for
character inputs and outputs and tag embeddings with 20 dimensions for
morphological tags (only for morphological analysis). The embeddings are
shared between the encoder and decoder for lemmatization and copy, since both
the source and the target sequences are characters. The output of the source
embedding is the input to the encoder module which is a SoPa with 120 patterns
in SoPa Seq2seq case and a bidirectional LSTM in the baseline. The decoder
later attends on the intermediary outputs of these modules. The final hidden
state of the encoder module is used to initialize the decoder. The decoder
side of these models is identical in both setups, an LSTM with Luong’s
attention. All LSTMs have 64 hidden cells and a single layer.

The size of SoPa patterns (3, 4, and 5 in our case) define the number of
forward arcs that a pattern has. These may contain epsilon steps and self
loops but an epsilon or a self loops is always followed by a main transition
(consuming an actual symbol). This means that a 3 long pattern may contain one
epsilon and one main transition, two epsilons or two main transitions. Any
main transition may be preceded by a self loop. The pattern size includes the
start state and the end state. In our experiments we used 3, 4, and 5 long
patterns, 40 patterns of each length.

Most of the training details are also identical. We train with batch size 64,
and we use early stopping if the development loss and accuracy stop improving
for 5 epochs. We maximize the number of epochs in 200 but this is never
reached. We save the best model based on development accuracy. We use the Adam
optimizer with 0.001 learning rate for all experiments.

SoPa is more difficult to train than LSTMs, so we decay the learning rate by
0.5 if the development loss does not decrease for 4 epochs.

##  5 Model similarity

We define a similarity metric between two SoPa Seq2seq models measured on
datasets that share their source side. The target side may differ. The three
tasks introduced in Section 3, all take inflected word forms as their source
sequence, which allows computing our similarity metric between each pair of
tasks.

SoPa works with a predefined number of patterns and tries matching each
pattern on any subword of the input with a particular length. The highest
scoring subword is used in the final source representation. We take the
highest scoring $T=10$ patterns for each input and compare the subwords that
resulted in these scores. The metric is defined as the average similarity over
the dataset $D$:

$\text{Sim}(M_{1},M_{2},D)=\frac{1}{|D|}\sum_{d\in D}S(M_{1}(d),M_{2}(d)),$
(1)

where $M_{1}$ and $M_{2}$ are the models, and $S$ is the similarity of the two
representations generated by the encoder side of the models on sample $d$,
defined as:

$S(M_{1}(d),M_{2}(d))=\frac{1}{2T}(\sum_{p_{i}\in P_{1}}\max_{p_{j}\in
P_{2}}J(p_{i},p_{j})+\sum_{p_{j}\in P_{2}}\max_{p_{i}\in
P_{1}}J(p_{i},p_{j})),$ (2)

where $T$ is a predefined number of highest scoring patterns on that sample
(10 in our experiments), $P_{1}$ is the set of $T$ highest scoring patterns of
$M_{1}$, $P_{2}$ is the set of $T$ highest scoring patterns of $M_{2}$ and $J$
is the Jaccard similarity of two subwords defined as the proportion of
overlapping symbols by the union of all symbols. Jaccard similarity is 0 if
there is no overlap and is 1 when the subwords are the same. For each sample,
we first choose the highest scoring $T$ patterns from each model, we denote
these sets of patterns as $P_{1}$ and $P_{2}$. Then we find the subwords
corresponding to these patterns. We compute the pairwise Jaccard similarities
between every element of $P_{1}$ and $P_{2}$. Then for each pattern, we find
the most similar pattern from the other model. The average of these scores is
the similarity of the two models on that sample (see Eq. 2) and the average
over all samples (see Eq. 1) is the similarity of two models on dataset $D$.
This metric is symmetric and it ranges from 0 to 1. Table 3 shows a small
example of this similarity on the word _ablakban_.

|  ^ablakban$ | ^ablakban$ | ^ablakban$ | ^ablakban | Max
---|---|---|---|---|---
^ablakban$ | 0 | 0.2 | 1 | 0.75 | 1
^ablakban$ | 0 | 0.5 | 0.5 | 0.75 | 0.75
^ablakban$ | 0 | 0.5 | 0 | 0.167 | 0.5
^ablakban$ | 0 | 0.75 | 0.167 | 0.333 | 0.75
Max | 0 | 0.75 | 1 | 0.75 | J=0.6875
Table 3:  Simlarity (Eq. 2) between two models $M_{1}$ and $M_{2}$ on one
sample using the 4 highest scoring subwords ($T=4$) with the subwords
underlined. Rows correspond to the highest scoring subwords from $M_{1}$ (ban,
kba, lak, kban), while columns correspond to the subwords from $M_{2}$ (^ab,
akb, ban, lakb). A Jaccard similarity matrix (with position information) is
constructed. The final similarity is the mean maximum of every row and every
column of the matrix.

##  6 Results and analysis

We first show that SoPa Seq2seq is competitive with the LSTM Seq2seq baseline,
especially for morphological analysis. An output is considered accurate if it
fully matches the reference and we do not consider partial matching. Some
languages prove to be too difficult for the models, which may be due to the
lack of context that is often needed for morphological analysis and
orthographic changes often present in lemmatization. We continue our analysis
on languages where each of the three tasks are performed by SoPa ‘reasonably
well’, which we set to 40% accuracy or higher on the development set. This
leaves us with 12 languages. The reason we set a lower limit to accuracy is
that we have no reason to believe that a bad model’s representation is useful
for the task. Fig. 1 shows the test accuracy in these languages. Lemmatization
is consistently the most difficult task for SoPa, while SoPa is on pair with
LSTM Seq2seq in morphological analysis, sometimes outperforming it. We
attribute this result to the fact that a morphological tag often corresponds
to a single morpheme, usually with a few possible surface realizations that
SoPa’s ‘soft’ patterns can pick up on. On the other hand lemmatization and
copy require regenerating much of the input which is more difficult from an
inherently summarized representation such as the one SoPa generates.

Figure 1:  Accuracy of SoPa Seq2seq models on each language and task. Figure
2:  Model similarity between all task pairs by language. Higher similarity
indicates that two models handle the same source in a more similar way.

We continue by computing the pairwise similarity value defined in Eq. 1
between the three tasks. Higher values indicate that SoPa finds similar
patterns valuable for generating the output. Fig. 2 shows the pairwise
similarity of models trained for the three tasks. We only compute these
similarities on samples where the output of _both_ models are correct
(generally 40-60% of the test samples).

Lemmatization and morphological analysis are the least similar in almost every
language. This is not surprising considering that lemmatization is the task of
discarding information that morphological analysis needs to correctly tag.
Quechua is the only exception from this trend which could be explained by the
very rich inflectional morphology (especially at the type-level) that results
in lemmas being significantly shorter than inflected forms. This means that
copy needs to memorize a lot more of the source word than lemmatization.

Another trend we observe, is that copy and morphological analysis are more
similar than copy and lemmatization in languages with rich inflectional
morphology such as Armenian, Hungarian, Kurdish and Turkish and the opposite
is true in fusional and morphologically poor languages such as Danish and
English. Georgian seems to be an exception and further exploration is an
exciting research direction.

Finally we demonstrate SoPa’s interpretability by extracting the most
frequently matched subwords in each language and task. Table 4 lists the most
common subwords in English, French and Hungarian in each task. It should be
noted that these subwords are very short because we used 3, 4 and 5 long
patterns that match 2, 3 and 4 characters not including self loops and short
patterns simply occur more frequently.

language | task | subwords
---|---|---
English | copy | ed,e$,ed$,es,in,at,re,s$,te,ri
English | lemmatization | at,g$,er,in,ng,iz,s$,en,ize,es
English | morphological_analysis | d$,s$,e$,es$,$,ed,ed$,o,ng,g$
French | copy | s$,ss,is,as,ie,ai,z$,nt,ns,en
French | lemmatization | er,s$,t$,nt,ie,ns,ra,is,ri,^d
French | morphological_analysis | s$,t$,z$,nt$,ez$,e$,ai,er,ns$,es$
Hungarian | copy | l$,n$,k$,sz,t$,nk$,kk,el,ok,na
Hungarian | lemmatization | sz,t$,k$,l$,ta,tá,^k,n$,kb,ró
Hungarian | morphological_analysis | l$,t$,n$,k$,ek,a$,$,g$,á$,ak$
Table 4:  Top subwords extracted from English, French and Hungarian. ^and $
denote word start and end respectively.

##  7 Conclusion

We presented an application of Soft Patterns – a finite state automaton
parameterized by a neural network – as the encoder of a sequence-to-sequence
model. We show that it is competitive with the popular LSTM encoder on
character-level copy and morphological tagging, while providing interpretable
patterns.

We analyzed the behavior of SoPa encoders on morphological analysis,
lemmatization and copy by computing the average Jaccard similarity between the
patterns extracted from the source side. We found two trends that coincide
with linguistic intuition. One is that lemmatization and morphological
analysis require patterns that match less similar subwords than the other two
task pairs. The other one is that copy and morphological analysis are more
similar in languages with rich inflectional morphology.

## Acknowledgments

Work partially supported by 2018-1.2.1-NKP-00008: Exploring the Mathematical
Foundations of Artificial Intelligence; and National Research, Development and
Innovation Office grant NKFIH #120145 ‘Deep Learning of Morphological
Structure’. We thank Roy Schwartz for his help in understanding the inner
mechanics of SoPa.

## References

  * [1] Cotterell, R., Kirov, C., Sylak-Glassman, J., Yarowsky, D., Eisner, J., Hulden, M.:  The SIGMORPHON 2016 shared task—morphological reinflection.  In: Proceedings of the 2016 Meeting of SIGMORPHON, Berlin, Germany, Association for Computational Linguistics (August 2016)
  * [2] Cotterell, R., Kirov, C., Sylak-Glassman, J., Walther, G., Vylomova, E., Xia, P., Faruqui, M., Kübler, S., Yarowsky, D., Eisner, J., Hulden, M.:  The CoNLL-SIGMORPHON 2017 shared task: Universal morphological reinflection in 52 languages.  In: Proceedings of the CoNLL-SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, Vancouver, Canada, Association for Computational Linguistics (August 2017)
  * [3] Cotterell, R., Kirov, C., Sylak-Glassman, J., Walther, G., Vylomova, E., McCarthy, A.D., Kann, K., Mielke, S., Nicolai, G., Silfverberg, M., Yarowsky, D., Eisner, J., Hulden, M.:  The CoNLL–SIGMORPHON 2018 shared task: Universal morphological reinflection.  In: Proceedings of the CoNLL–SIGMORPHON 2018 Shared Task: Universal Morphological Reinflection, Brussels, Belgium, Association for Computational Linguistics (October 2018)
  * [4] Aharoni, R., Goldberg, Y.:  Morphological inflection generation with hard monotonic attention.  arXiv preprint arXiv:1611.01487 (2016)
  * [5] Schwartz, R., Thomson, S., Smith, N.A.:  SoPa: Bridging CNNs, RNNs, and Weighted Finite-State Machines.  In: Proc. 56th ACL Annual Meeting, Melbourne, Australia (2018) 295–305
  * [6] Sutskever, I., Vinyals, O., Le, Q.V.:  Sequence to sequence learning with neural networks.  In: Proc. NIPS, Montreal, CA (2014) 3104–3112
  * [7] Hochreiter, S., Schmidhuber, J.:  Long short-term memory.  Neural Computation 9(8) (11 1997) 1735–1780
  * [8] Luong, T., Pham, H., Manning, C.D.:  Effective approaches to attention-based neural machine translation.  In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics (2015) 1412–1421
  * [9] Sylak-Glassman, J.:  The composition and use of the universal morphological feature schema (unimorph schema).  Technical report (2016)