# DUCPS: Deep Unfolding the Cauchy Proximal Splitting Algorithm for B-Lines
Quantification in Lung Ultrasound Images

Tianqi Yang, Oktay Karakuş, , Nantheera Anantrasirichai, , Marco Allinovi and
Alin Achim Tianqi Yang, Nantheera Anantrasirichai and Alin Achim are with
Visual Information Laboratory, University of Bristol, Bristol, BS8 1UB UK
<EMAIL_ADDRESS><EMAIL_ADDRESS>alin.achim@bristol.ac.uk).Oktay Karakuş is with School of Computer Science and
Informatics, Cardiff University, Cardiff, CF24 4AG UK (e-mail:
KarakusO@cardiff.ac.uk).Marco Allinovi is with Nephrology, Dialysis and
Transplantation, Careggi University Hospital, Florence, Italy (e-mail:
marco.allinovi@gmail.com).

###### Abstract

The identification of artefacts, particularly B-lines, in lung ultrasound
(LUS), is crucial for assisting clinical diagnosis, prompting the development
of innovative methodologies. While the Cauchy proximal splitting (CPS)
algorithm has demonstrated effective performance in B-line detection, the
process is slow and has limited generalization. This paper addresses these
issues with a novel unsupervised deep unfolding network structure (DUCPS). The
framework utilizes deep unfolding procedures to merge traditional model-based
techniques with deep learning approaches. By unfolding the CPS algorithm into
a deep network, DUCPS enables the parameters in the optimization algorithm to
be learnable, thus enhancing generalization performance and facilitating rapid
convergence. We conducted entirely unsupervised training using the
Neighbor2Neighbor (N2N) and the Structural Similarity Index Measure (SSIM)
losses. When combined with an improved line identification method proposed in
this paper, state-of-the-art performance is achieved, with the recall and F2
score reaching 0.70 and 0.64, respectively. Notably, DUCPS significantly
improves computational efficiency eliminating the need for extensive data
labeling, representing a notable advancement over both traditional algorithms
and existing deep learning approaches.

###### Index Terms:

Deep unfolding, Cauchy proximal splitting, lung ultrasound, line detection,
inverse problem

##  I Introduction

Lung ultrasound (LUS) has emerged as a highly effective diagnostic tool for a
wide range of pulmonary conditions. Over recent years, there has been a
significant advancement in techniques for analyzing lung ultrasound images,
rendering it an invaluable asset in medical diagnostics [1]. One of the key
features in LUS are the so-called B-lines, which manifest as laser-like,
vertical, hyperechoic reverberation artifacts originating from the pleural
line [2].

The motivation behind research into B-line detection is multifaceted. Firstly,
the accurate identification and quantification of B-lines can significantly
improve the early diagnosis and management of various lung conditions,
reducing the need for more invasive procedures like CT scans or bronchoscopy,
which are often more expensive and less readily available. Early and precise
detection of pulmonary issues through B-lines can lead to timely and targeted
therapeutic interventions, potentially reducing morbidity and mortality
associated with diseases such as pneumonia and pulmonary edema. Studies have
demonstrated their correlation with the volume of extravascular lung water [3,
4, 5]. Moreover, enhancing the automated detection of B-lines through advanced
imaging algorithms holds the promise of making LUS more accessible and easier
to use for clinicians who may not have specialized training in ultrasound
imaging. This democratization of lung ultrasound technology can be
particularly beneficial in resource-limited settings or during large-scale
health crises, such as the COVID-19 pandemic, where rapid and widespread
pulmonary assessment is crucial. Additionally, understanding the nuances of
B-line formation and their exact correlation with different pathological
states can deepen our overall knowledge of pulmonary physiology and
pathophysiology. For instance, in chronic kidney disease, fluid overload and
pulmonary congestion are common complications that can be effectively
monitored through B-line detection, providing critical insights for managing
such patients and preventing further complications. This, in turn, can inspire
the development of new therapeutic approaches and preventive measures for
lung-related illnesses. In essence, the ongoing research and development in
B-line detection aim not only to enhance diagnostic accuracy and efficiency
but also to broaden the applicability of LUS, ensuring that more patients
benefit from this powerful diagnostic tool across various healthcare settings.

Various techniques have been utilized in previous endeavors to automatically
detect B-lines. Brattain et al. [6] employed hand-crafted image processing
techniques such as polar reformatting and thresholding. Moshavegh et al. [7]
employed a random walk method to delineate the pleural line and then applied
an alternate sequential filtration to identify B-lines, excluding the upper
pleural region. Anantrasirichai et al. [8, 9] proposed line detection as an
inverse problem, wherein a B-mode LUS image is transformed into a
representation of radius and orientation in the Radon domain. Subsequently,
they solved the inverse problem using the alternating direction method of
multipliers (ADMM) [10]. Farahi et al. [11] then developed an algorithm based
on the use of Wavelet to denoise and Radon transforms to extract LUS features.
Further contributions to model-based methods were later proposed by Karakus
et. al. [12] for assessing COVID-19 patients, achieving improved performance
by regularizing the solution with the Cauchy proximal splitting (CPS)
algorithm [13]. While model-based methods do not rely on learning mappings
from data directly, data is often employed to estimate a small number of
parameters. However, simplistic models frequently struggle to capture the
nuances of high-dimensional complex data and dynamic variations.

With the emergence of deep learning, convolutional neural networks (CNNs) have
become key in robust B-line detection. CNNs can learn complex mappings that
are difficult to manually design. Van Sloun and Demi [14] used a CNN with
gradient-based class activation mapping (Grad-CAM) [15] to detect B-lines in
LUS frames. Roy et al. [16] expanded on this with a spatial transformer
network (STN) for weakly supervised localization of pathological patterns and
scoring COVID-19 severity. Improved performance was obtained by adding domain
knowledge into deep neural networks [17]. Mason et al. achieved better
segmentation performance through domain adaptation, exploring B-line
segmentation in LUS frames of COVID-19 and pneumonia patients. Several
methodologies utilize both spatial and temporal information, including the use
of 3-D filters [18] and spatio-temporal attention [19]. To address the
challenge of obtaining expert annotations, Zhao et al. [20] used simulated LUS
images to train CNNs for B-line artifact segmentation, while our previous
study [21] explored contrastive learning for unsupervised B-line
representation.

The limitations present in model-based methods and opaque deep learning
systems have prompted the emergence of integrated techniques. These techniques
aim to mitigate the drawbacks of each approach while capitalizing on their
respective strengths. Model-based techniques often struggle with capturing
complex patterns and may require extensive domain knowledge for accurate
modeling, while deep learning systems lack interpretability and can be
challenging to implement without labeled data. Merging these techniques
combines the robustness and interpretability of traditional signal processing
with the flexibility of deep learning, yielding a more comprehensive approach.
This integration enhances performance, facilitates better analysis, and
improves understanding of underlying data for specific tasks. In the
influential study by Gregor and LeCun [22], a promising approach known as
algorithm unfolding (or unrolling) was introduced, facilitating the linkage of
iterative algorithms like sparse coding techniques with neural networks. This
research has inspired further studies in different areas of biomedical
research, such as compressive sensing [23], computed tomography [24, 25], and
ultrasonography [26].

In this work, we propose a novel framework for B-line detection that leverages
the advantages of model-based methods, eliminating the need for labeled data,
and the fast inference capabilities of deep learning to enable real-time
operation. This extends our previous work [27], where we adopt deep unfolding
to achieve rapid convergence by training a feedforward neural network to
approximate iterative algorithms. Here, we model the lines in noisy LUS images
via a Radon transform and reconstruct the lines employing the Cauchy prior.
Then, we unfold the CPS algorithm for solving the inverse problem for line
detection in LUS images, aiming to address a drawback inherent in its manually
predefined parameter configuration, which may limit its ability to generalize
effectively. The architecture of the deep unfolded CPS (DUCPS) algorithm is
shown in Fig.1. A morphological line identification procedure is subsequently
used to localize horizontal (pleural line and A-lines) and vertical (B-lines)
line artifacts. The proposed method outpaces traditional model-based methods
by a significant margin, boasting speeds up to 10 times faster than PUI [9].
In comparison to state-of-the-art neural network detection, it demonstrates
reproducibility and explainability, making it ideal for real-time B-line
detection.

To summarize, the main contributions of this work are:

  1. 1.

We propose an unsupervised deep unfolding network structure (DUCPS) designed
to solve the inverse problem of model-based line detection in LUS images.
Specifically, we unfold the Cauchy proximal splitting (CPS) algorithm into a
deep network, enabling the parameters in the optimization algorithm to be
learnable.

  2. 2.

We train the network with the Neighbor2Neighbor loss, considering its
effectiveness in handling noisy images. This approach significantly improves
precision, recall, and F-scores compared to DUBLINE [27];

  3. 3.

We introduce a morphological approach to identify B-lines in LUS images by
restricting the search area and detecting local maxima in the Radon domain,
thereby distinguishing B-lines from other line artifacts efficiently.

  4. 4.

We design a scoring method to evaluate B-line detection accuracy, which offers
precise feedback on detection positioning relative to the target area.

The remainder of this paper is organized as follows: we present the proposed
DUCPS algorithm and the line artifact identification method in detail in
Section II. Section III introduces the method designed for evaluating our
method. In Section IV, we describe the data used in the study, and show the
experimental results as well as the comparison with other methods. Finally,
Section V concludes this article with remarks and future work directions.

##  II Methodology

Figure 1: Diagrammatic representation of DUCPS. The iteration step (yellow
block) is executed a number of times corresponding to the network layers. Each
iteration $k$ depends on algorithm parameters, which are transferred into
network parameters $W$ and $\mu$ (coloured in red) which are learned from
training datasets through end-to-end training.

###  II-A Line artefact model

An ultrasound image can be represented as the result of applying an inverse
Radon transform to a set of lines in a forward model, as

$y=R^{-1}x+n,$ (1)

where the observed ultrasound image, denoted as $y$, corresponds to a set of
lines $x$ characterised by a distance $r$ from the center of $y$ and an
orientation $\omega$ from the horizontal axis of the image. Represented by $R$
and $R^{-1}$, the Radon transform and its inverse are involved and are used as
in [28], while $n$ signifies the additive noise. In a noise-free scenario, the
Radon transform is typically defined as shown in (2), where $\delta(\cdot)$
denotes the Dirac-delta function.

$x=\int_{\mathbb{R}^{2}}y(i,j)\delta(r-icos\omega-jsin\omega)didj.$ (2)

The Cauchy distribution is recognized as a distinctive member within the
$\alpha$-stable distribution family, acknowledged for its heavy-tailed nature
and its tendency to encourage sparsity [29], in particular in transform domain
representations of ultrasound images [30]. The closed-form probability density
function of the Cauchy distribution is given by

$p(x)\propto\frac{\gamma}{\gamma^{2}+x^{2}},$ (3)

where $\gamma$ serves as the scale (or dispersion) parameter, governing the
extent of distribution spread. Therefore, utilizing maximum a posteriori
estimation, the reconstruction of $x$ can be accomplished using the subsequent
equation, taking into account the model described in (1) and employing the
Cauchy prior.

$\hat{x}=\arg\min_{x}\left\\{\frac{\left\|y-R^{-1}x\right\|_{2}^{2}}{2\sigma^{2}}-\sum_{i,j}\log{\frac{\gamma}{\gamma^{2}+x_{i,j}^{2}}}\right\\}$
(4)

###  II-B Deep Unfolded CPS

Deep Unfolding in Model-Driven Neural Networks: Deep unfolding is a method
that integrates iterative algorithms with neural network techniques to
efficiently so lve optimization problems [31]. As shown in Fig.1, this
approach takes an iterative optimization algorithm with the maximum number of
iterations, unfolds its structure, and incorporates trainable parameters.
These parameters are then optimized using deep learning techniques including
loss functions, optimizers, and backpropagation. The resultant unfolded
algorithm, enhanced with learned parameters, can be applied to B-line
detection in LUS images.

In more detail, the problem in (4) can be solved by employing the CPS
algorithm [12] which computes the proximal operator of the Cauchy-based
penalty and employs the forward-backward (FB) algorithm for optimization. By
leveraging the well-defined iterative mathematical formulation of CPS, we
unfold the algorithm into a neural network architecture. The unfolding process
proposed below will allow the network parameters to be trained end-to-end
using gradient-based optimization. The inputs to the network are the initial
states, including the original image $y$ in the spatial domain and its
corresponding radon transform $x$, and the output is the prediction $\hat{x}$
after a certain number of iterations.

Neural network formulation: To unfold CPS, iteration (5) and (6) can be
reformulated into a single network layer as depicted in Fig.1. This layer
encompasses a sequence of analytical operations, including matrix-vector
multiplication, summation, and the Cauchy proximal operator, which are
analogous to the operations within a neural network. A diagrammatic
representation of one iteration step elucidates its similarity to a single
network layer. Executing CPS $k$ times can be interpreted as cascading $k$
such layers, thereby forming a $k$-layer neural network. In each layer, the
optimisation can be broken up into a forward gradient step using the data
fidelity term with the form of
$(\left\|y-R^{-1}x\right\|_{2}^{2})/2\sigma^{2}$, and a backward step using
the Cauchy-based penalty. In this way, the trained network corresponds exactly
to the original algorithm.

In the forward gradient step, the intermediate variable $z^{k}$ is updated by

$\displaystyle z^{k}$ $\displaystyle=x^{k}-\mu(R^{-1})^{T}(R^{-1}x^{k}-y),$
(5) $\displaystyle=Wx^{k}+\mu(R^{-1})^{T}y,$

where $\mu$ refers to the step size, and $(R^{-1})^{T}$ serves as the forward
Radon projection [32]. $W$ and $\mu$ are trainable matrices with positive
elements. $W$ is initialized as a constant matrix that has the same size as
$x$, with elements nearly equal to 1, while $\mu$ is set to be a single-
element matrix with a value close to 0 so that $z^{0}\approx x^{0}$. In this
paper, we use $1-10^{-5}$ and $10^{-5}$ to initialise $W$ and $\mu$
respectively. The $R$ and $R^{-1}$ are constructed by Radon_transformation
functions111https://github.com/drgHannah/Radon-Transformation/tree/main.

In the backward step, $x^{k}$ is updated by

$x^{k+1}=prox_{Cauchy}^{\mu}(z^{k})$ (6)

where the Cauchy proximal operator is defined as

$prox_{Cauchy}^{\mu}(z)=\arg\min_{u}\left\\{\frac{\left\|z-u\right\|_{2}^{2}}{2\mu}-\log{\frac{\gamma}{\gamma^{2}+u^{2}}}\right\\}.$
(7)

As suggested in [13], obtaining the solution to this minimization problem
involves differentiating (7) with respect to $u$, and setting the resulting
expression to zero. Consequently, we arrive at the cubic function given in (8)
and the solution to this function can be obtained through Cardano’s method as
first shown in [33].

$u^{3}-zu^{2}+(\gamma^{2}+2\mu)u-z\gamma^{2}=0$ (8)

To comply with the CPS algorithm, we set the number of layers to meet the
stopping criteria when

$\displaystyle\frac{\left\|x^{k}-x^{k-1}\right\|}{\left\|x^{k-1}\right\|}<10^{-3}$

It is noteworthy that the number of layers $k$ in the trained network can be
significantly lower—by an order of magnitude—than the number of iterations
needed for the original CPS algorithm to converge for a new input. This
substantial reduction markedly enhances optimisation efficiency.

Loss function: After unfolding CPS into a network, which we prefer to refer to
as DUCPS, the network is trained through back-propagation using real datasets
to optimise the parameters $W$ and $\mu$. Considering that we aim to train the
network in an unsupervised way, we employ the Neighbor2Neighbor (N2N) training
scheme [34] as it can train the network from a single image and is suitable
for noisy input. For detailed discussions, we refer the reader to Section 3
and Section 4 of [34]. The training framework is depicted in Fig. 2. A pair of
sub-sampled images $(g_{1}(y),g_{2}(y))$ are generated from a neighbor sub-
sampler $G$ [34] to generate noisy image pairs from single noisy images $Ry$
for training. The loss function includes a reconstruction term $L_{rec}$,
which is computed between the network output and the noisy target, and a
regularisation term $L_{reg}$ that measures the difference of the ground-truth
pixel values between the sub-sampled noisy image pair.

$\displaystyle Loss_{N2N}=L_{rec}+\alpha
L_{reg}=\left\|f_{\theta}(g_{1}(Ry))-g_{2}(Ry)\right\|_{2}^{2}$ (9)
$\displaystyle+\alpha\cdot\left\|f_{\theta}(g_{1}(Ry))-g_{2}(Ry)-(g_{1}(f_{\theta}(Ry))-g_{2}(f_{\theta}(Ry)))\right\|_{2}^{2},$

where $f_{\theta}(\cdot)$ refers to DUCPS with $\theta$ representing the
network parameters. $\alpha$ is a hyper-parameter that controls the strength
of the regularisation term.

Figure 2: The overview of Neighbor2Neighbor training scheme [34]. The neighbor
sub-samplers that appear twice are the same.

We also investigate the algorithm performance when training with a common
loss, i.e. the Structural Similarity Index Measure (SSIM) loss, used in
computer vision tasks and also in our previous work [27]. We trained the
network to maximize the SSIM [35] between the intermediate variable $z^{k}$
and the input image. Incorporating the forward step in the FB algorithm
achieves the purpose of data fitting, thereby defining the loss function as
follows:

$\displaystyle Loss_{SSIM}=(1-SSIM(R^{-1}(z^{k}),y)).$ (10)

###  II-C Line identification

The line identification process is implemented after acquiring the restored
image $\hat{x}$ using the local maxima technique. The steps are illustrated in
Fig. 3 and described below.

Figure 3: The process of identifying line artifacts. Searching areas for the
pleural line (red), A-line (blue), and B-line (green) are masked in the Radon
domain. The detected pleural line is displayed in red, the A-line is in blue,
and the B-lines are depicted in green.

It is crucial to set different searching areas to distinguish line artefacts
considering that they possess characteristic structures in the spatial domain.
This includes limiting the searching angles and upper and lower borders in the
Radon domain. For pleural line and A-lines, we limit the searching angle
within the range $\Omega_{h}\in\left[90^{\circ}\pm 20^{\circ}\right]$
($0^{\circ}$ originates from the $x$ axis, which indicates a horizontal line)
as they are generally horizontal. Similarly, the B-lines are detected at
$\Omega_{v}\in\left[\pm 10^{\circ}\right]$.

Step 1. Given an LUS image $y$ having a height of $H$ pixels, the borders of
pleural line searching are defined as

Pleural line upper border $\displaystyle=\frac{1}{3}H,$ (11) Pleural line
lower border $\displaystyle=\frac{1}{4}H.$ (12)

This setting accounts for the fact that the pleural line represents the
junction of the visceral and parietal pleura, while also helping to mitigate
the influence of the appearance of the chest wall and intercostal muscles in
the upper area of the image. Then, the provided search area is scanned to
detect the local peaks of the reconstructed Radon information. We select the
local peak with the maximum intensity as the detected pleural line, as it is
typically the brightest line within the corresponding search area in the
spatial domain.

Step 2. The area above the detected pleural line in the spatial image is
removed to facilitate the detection of remaining artifacts. By comparing the
Radon images in Step 1 and Step 3, we can observe that the Radon information
becomes less noisy after the removal of redundant tissue, revealing clearer
local peak information for subsequent steps. The detections will be performed
based on the Radon transform of the dimmed image.

Step 3a. On the one hand, A-lines are equidistant physiological horizontal
lines below the pleural line, and the distance between them is equal to the
distance between skin and pleural line. Therefore, the searching area for
A-lines are limited as

A-line upper border $\displaystyle=\frac{1}{2}H-\frac{3}{2}H_{p},$ (13) A-line
lower border $\displaystyle=\frac{1}{2}H,$ (14)

where $H_{p}$ denotes the distance from the detected pleural line and the
center of $y$.

On the other hand, as A-lines may appear less distinct or weaker in intensity
at greater depths compared to shallower depths in LUS images, we set a
threshold to validate the candidate peak points and select the brightest as
the detected A-line. The intensity of the validated A-line ($I_{A}$) should
satisfy

$\displaystyle I_{A}\geq\lambda I_{p},$ (15)

where $\lambda\in\left[0,1\right]$ is a proportional parameter and $I_{p}$
refers to the intensity of the detected pleural line.

Step 3b. In the case of using a linear transducer, possible locations of
vertical artefacts can span across the entire width ($W$) of the image, and
therefore candidate B-lines are detected in the Radon domain within the border

B-line upper and lower borders $\displaystyle=\frac{1}{2}W.$ (16)

Step 4. Note that B-lines originate from the pleural line and should extend to
the bottom of the ultrasound image, thus verification is needed to eliminate
the Z-lines - these artefacts do not erase A-lines and have no use for
diagnosis. Specifically, We discard the candidate B-lines that intersect with
A-lines in the spatial domain, and the remaining ones are identified as the
final detected B-lines.

##  III Scoring rules

To statistically evaluate the proposed method, we design rules for scoring the
detected B-lines. As sketched in Fig. 4, the yellow bounding box (bbox)
represents the ground truth where a B-line is located, and the green line
indicates the location of detection. The width of the bbox is divided into ten
regions on each side of a center line. The closer the green line is to the
center line, the higher the score it receives, with scores ranging from 1 to
0.1 in decrements of 0.1 as the distance from the center line increases. Each
region closest to the center line scores 1, decreasing by 0.1 for each
subsequent region moving away from the center. Any green lines falling outside
the bbox receive a score of 0, indicating no accuracy in detection within the
target area. This scoring system provides a symmetrical and graded assessment
of detection accuracy, offering precise feedback on the positioning of
detections relative to the target area.

Figure 4: An example of scoring the detected B-line. In this case, the green
line is scored 0.8.

##  IV Results and discussion

###  IV-A Dataset

The video clips were obtained from 45 patients from the Nephrology, Dialysis
and Transplantation Department, Careggi University Hospital, Florence, Italy.
LUS evaluations were performed whilst patients attended for regular
hemodialysis, using an ultrasound machine (MyLab Class C-Esaote®, Genoa,
Italy) with a 6–18 MHz linear probe. The data was saved in DICOM format.

Data preprocessing includes extracting the frames from the video clips,
cropping the region that only contains lung information, and finally padding
all the images to the size of 480$\times$600 pixels (height $\times$ width).

We randomly split the dataset into three groups (Group1, Group2, and Group3)
at the patient level. Each of the groups contains data from 15 patients. By
partitioning the dataset into 3 subsets, we will evaluate the proposed method
in a 3-fold manner. This helps prevent information leakage between groups and
iteratively uses each subset for testing while the remaining subsets are used
for training. The training set in each fold is formed by randomly selecting
2000 images from two of the groups (for example Group1 and Group2). Then we
randomly select 8 images from each patient in the remaining group (for example
Group3), and a physician with long-term expertise in LUS provides the
annotation of the line artefacts as the ground truth. The annotated images are
subsequently divided into two sets which are used for pre-trained model (for
example YOLOv7 [36] and FasterRCNN [37]) fine-tuning and algorithm testing
respectively. The final test set contains in total 240 images. The detailed
data splitting strategy is illustrated in Fig. 5. The procedure is implemented
twice to enhance result stability and increase the reliability and credibility
of the experiment by assessing consistency, minimizing chance influences, and
aiding in issue identification.

Figure 5: Illustration of setting up the dataset. Data random splitting is
implemented twice, each of which is designed to perform a 3-fold validation.

###  IV-B Experimental setup

We evaluated the B-line detection performance among several methods including
PUI [9], CPS [12], DUBLINE [27], YOLOv7 [36] and FasterRCNN [37]. For model-
based methods, we implement the experiment using MATLAB R2023a with 64-bit OS
12th Gen Intel® CoreTM i7-12700 2.10 GHz. Regarding the deep unfolding
technique, the networks are all implemented in PyTorch and are trained for 20
epochs on Nvidia Geforce RTX 3090. The number of layers is set to 10 and 7 for
DUBLINE and DUCPS respectively. The initial learning rate is set to $10^{-4}$
for both methods and a scheduler is used to adjust the learning rate by 0.5
after 5 epochs. In the case of training with N2N loss, we set $\alpha=1$ as
suggested in [34]. YOLOv7 and FasterRCNN are fine-tuned and then tested on the
same GPU. It’s important to note that while the networks are loaded onto the
GPU for training, the computations related to the methods are performed on the
same CPU setup utilized for model-based methods.

###  IV-C Results Analysis

Our assessment is performed in terms of true positives (TP), false positives
(FP), false negatives (FN), precision, recall, and the F-scores. We pay more
attention to the F2 score as it is crucial to focus on minimizing FNs in
medical contexts, where detecting all artefacts is paramount to prevent
misdiagnoses and ensure timely treatment. Specifically, we use the threshold
of 0.5 so as to calculate the values based on the scoring rules proposed in
Section III. Detections that are scored above the threshold are counted as TP.
For YOLOv7 and FasterRCNN, we summarised the results with the confidence
threshold being 0.5. The statistical analysis is presented in Table I.
Besides, we compared the performance of employing different line
identification schemes (where the line identification method proposed in this
paper is denoted as LIM) and different loss functions (SSIM and N2N).

By comparing the results of the original PUI algorithm and a version of it
obtained by replacing the line identification part with LIM (referred to as
PUI_LIM thereafter), it can be demonstrated that our proposed line
identification strategy improves overall detection performance when employing
the same optimization techniques, particularly enhancing the recall with an
average increase of 10.44% compared to [9].

Within the same line identification scheme, deep unfolding techniques have
been instrumental in improving recall. Specifically, both DUBLINE_SSIM+LIM and
DUCPS_SSIM+LIM, designed upon traditional optimization algorithms, have
increased recall by 9.28% and 10.21% compared to PUI_LIM and CPS+LIM
respectively, consequently resulting in higher F2 scores. It is worth noting
that the initial design of CPS was intended for handling curvilinear probe
images. Although we adjusted parameters to adapt to the dataset in this study,
CPS still exhibited low precision in the experiments. However, DUCPS enables
the training of parameters within the optimization procedure, significantly
enhancing the algorithm’s generalization ability and boosting precision by
13.96%.

To find out the impact of different loss functions on experimental results, we
can observe DUCPS_SSIM+LIM and DUCPS_N2N+LIM. Using the N2N loss led to the
best performance across various matrices in model-based methods. While
improving the average precision to 47.28%, it also raised recall to 70.13%.
Additionally, both F1 and F2 scores reached their highest values among all
methods, achieving 0.5633 and 0.6381, respectively. We also noticed that due
to the sub-sampling technique utilized in the N2N loss, the network training
requires less memory compared to using SSIM loss, resulting in faster training
speeds.

Examining the performance of deep learning neural networks on the test data,
we notice that while achieving relatively high scores in some folds, such as
YOLOv7 reaching 100% precision in Fold 3 of the first random splitting, the
proportion of FN is excessively high. A similar situation is observed in the
testing of FasterRCNN in Random2-Fold2. In clinical settings, this could
result in artefacts undetected, potentially causing patients to miss the
window for treatment. Additionally, in some tests, neither YOLOv7 nor
FasterRCNN predicted any artifacts with confidence scores higher than 0.5. The
same scenario occurred when testing other detection networks, including
CascadeRCNN [38], CenterNet [39], DETR [40], and RetinaNet [41]. This may be
due to the fine-tuning dataset being too small, leading the neural networks to
not learn sufficient artefact features, highlighting the importance of dataset
and annotation quality for deep learning neural networks. In contrast, DUCPS
demonstrates robustness across all tests and does not require data annotation.

TABLE I: Line detection performance (Threshold = 0.5). Data random splitting
is executed twice, with each instance tailored for conducting a 3-fold
validation. The best average performances are in bold. The best performance in
an individual fold is indicated by an underlined italics.

|  |  | Precision $\uparrow$ | Recall $\uparrow$ | F1 $\uparrow$ | F2 $\uparrow$ | Num_Bline | Num_Detection | TP $\uparrow$ | FP $\downarrow$ | FN $\downarrow$
---|---|---|---|---|---|---|---|---|---|---|---
|  | Fold1 | 0.53 | 0.47 | 0.50 | 0.49 | 99 | 88 | 47 | 41 | 52
|  | Fold2 | 0.52 | 0.53 | 0.52 | 0.53 | 109 | 112 | 58 | 54 | 51
| Random1 | Fold3 | 0.34 | 0.42 | 0.38 | 0.40 | 85 | 106 | 36 | 70 | 49
|  | Fold1 | 0.52 | 0.49 | 0.51 | 0.50 | 136 | 128 | 67 | 61 | 69
|  | Fold2 | 0.49 | 0.44 | 0.47 | 0.45 | 91 | 81 | 40 | 41 | 51
| Random2 | Fold3 | 0.35 | 0.52 | 0.42 | 0.47 | 66 | 97 | 34 | 63 | 32
PUI [9] | average |  | 0.46 | 0.48 | 0.47 | 0.47 |  |  |  |  |
|  | Fold1 | 0.44 | 0.48 | 0.46 | 0.48 | 99 | 108 | 48 | 60 | 51
|  | Fold2 | 0.51 | 0.61 | 0.55 | 0.58 | 109 | 129 | 66 | 63 | 43
| Random1 | Fold3 | 0.46 | 0.65 | 0.54 | 0.60 | 85 | 119 | 55 | 64 | 30
|  | Fold1 | 0.57 | 0.54 | 0.55 | 0.54 | 136 | 129 | 73 | 56 | 63
|  | Fold2 | 0.49 | 0.59 | 0.53 | 0.57 | 91 | 111 | 54 | 57 | 37
| Random2 | Fold3 | 0.36 | 0.64 | 0.46 | 0.55 | 66 | 116 | 42 | 74 | 24
PUI_LIM | average |  | 0.47 | 0.58 | 0.52 | 0.55 |  |  |  |  |
|  | Fold1 | 0.39 | 0.72 | 0.51 | 0.62 | 99 | 180 | 71 | 109 | 28
|  | Fold2 | 0.27 | 0.53 | 0.36 | 0.44 | 109 | 217 | 58 | 159 | 51
| Random1 | Fold3 | 0.28 | 0.55 | 0.37 | 0.46 | 85 | 168 | 47 | 121 | 38
|  | Fold1 | 0.42 | 0.69 | 0.52 | 0.61 | 136 | 225 | 94 | 131 | 42
|  | Fold2 | 0.23 | 0.45 | 0.30 | 0.38 | 91 | 178 | 41 | 137 | 50
| Random2 | Fold3 | 0.25 | 0.61 | 0.35 | 0.47 | 66 | 162 | 40 | 122 | 26
CPS [12]+LIM | average |  | 0.31 | 0.59 | 0.40 | 0.50 |  |  |  |  |
|  | Fold1 | 0.41 | 0.71 | 0.52 | 0.62 | 99 | 172 | 70 | 102 | 29
|  | Fold2 | 0.42 | 0.61 | 0.50 | 0.56 | 109 | 160 | 67 | 93 | 42
| Random1 | Fold3 | 0.32 | 0.67 | 0.44 | 0.55 | 85 | 176 | 57 | 119 | 28
|  | Fold1 | 0.46 | 0.62 | 0.53 | 0.58 | 136 | 184 | 84 | 100 | 52
|  | Fold2 | 0.34 | 0.60 | 0.43 | 0.52 | 91 | 164 | 55 | 109 | 36
| Random2 | Fold3 | 0.32 | 0.67 | 0.43 | 0.55 | 66 | 137 | 44 | 93 | 22
DUBLINE [27]_SSIM+LIM | average |  | 0.38 | 0.65 | 0.47 | 0.56 |  |  |  |  |
|  | Fold1 | 0.81 | 0.25 | 0.38 | 0.29 | 99 | 31 | 25 | 6 | 74
|  | Fold2 | NaN | 0.00 | NaN | NaN | 109 | 0 | 0 | 0 | 109
| Random1 | Fold3 |  1.00 | 0.08 | 0.15 | 0.10 | 85 | 7 | 7 | 0 | 78
|  | Fold1 | 0.00 | 0.00 | NaN | NaN | 136 | 31 | 0 | 31 | 136
|  | Fold2 | 0.86 | 0.73 |  0.79 |  0.75 | 91 | 77 | 66 | 11 | 25
| Random2 | Fold3 | NaN | 0.00 | NaN | NaN | 66 | 0 | 0 | 0 | 66
YOLOv7 [36] | average |  | 0.67 | 0.27 | 0.33 | 0.29 |  |  |  |  |
|  | Fold1 | 0.33 | 0.52 | 0.40 | 0.46 | 99 | 155 | 51 | 104 | 48
|  | Fold2 | 0.52 | 0.13 | 0.21 | 0.15 | 109 | 27 | 14 | 13 | 95
| Random1 | Fold3 | NaN | 0.00 | NaN | NaN | 85 | 0 | 0 | 0 | 78
|  | Fold1 | NaN | 0.00 | NaN | NaN | 136 | 0 | 0 | 0 | 136
|  | Fold2 | 0.94 | 0.33 | 0.49 | 0.38 | 91 | 32 | 30 | 2 | 61
| Random2 | Fold3 | NaN | 0.00 | NaN | NaN | 66 | 0 | 0 | 0 | 66
FasterRCNN [37] | average |  | 0.60 | 0.32 | 0.37 | 0.33 |  |  |  |  |
|  | Fold1 | 0.47 | 0.71 | 0.56 | 0.64 | 99 | 149 | 70 | 79 | 29
|  | Fold2 | 0.49 | 0.68 | 0.57 | 0.63 | 109 | 152 | 74 | 78 | 35
| Random1 | Fold3 | 0.40 | 0.71 | 0.51 | 0.61 | 85 | 151 | 60 | 91 | 25
|  | Fold1 | 0.54 | 0.76 | 0.63 | 0.70 | 136 | 191 | 103 | 88 | 33
|  | Fold2 | 0.41 | 0.57 | 0.48 | 0.53 | 91 | 126 | 52 | 74 | 39
| Random2 | Fold3 | 0.37 | 0.74 | 0.49 | 0.62 | 66 | 133 | 49 | 84 | 17
DUCPS_SSIM+LIM (ours) | average |  | 0.45 | 0.69 | 0.54 | 0.62 |  |  |  |  |
|  | Fold1 | 0.51 |  0.78 | 0.62 | 0.70 | 99 | 151 | 77 | 74 | 22
|  | Fold2 | 0.51 | 0.65 | 0.57 | 0.62 | 109 | 139 | 71 | 68 | 38
| Random1 | Fold3 | 0.42 | 0.68 | 0.52 | 0.61 | 85 | 138 | 58 | 80 | 27
|  | Fold1 | 0.56 | 0.76 | 0.65 | 0.71 | 136 | 186 | 104 | 82 | 32
|  | Fold2 | 0.42 | 0.60 | 0.50 | 0.56 | 91 | 130 | 55 | 75 | 36
| Random2 | Fold3 | 0.41 | 0.73 | 0.53 | 0.63 | 66 | 116 | 48 | 68 | 18
DUCPS_N2N+LIM (ours) | average |  | 0.47 | 0.70 | 0.56 | 0.64 |  |  |  |  |

Fig. 6 shows the line artefact detection results for different methods.
Through visualization of the results, we observe that compared to PUI_LIM, our
proposed approach reduces the probability of missed detections, thereby
decreasing FNs, particularly in scenarios with high B-line density (as shown
in row 1 and row 3). While CPS[12] and DUBLINE_SSIM [27]+LIM also detect more
B-lines, our method exhibits fewer FPs, as evident in row 2 and row 3.
Compared to the detection results of deep learning neural networks, the
stability of the proposed method is highlighted. For instance, YOLOv7 [36]
often fails to confidently predict the presence of B-Lines in many instances
(e.g., row 1, row 2, and row 3), while the results of FasterRCNN [37] show
greater variability, ranging from failure to detect B-lines (row 2) to
detecting excessive artifacts (row 4). Additionally, although there are
accurate B-line localisations in some cases, there are frequent occurrences of
overlapping predicted positions, and many predicted B-lines do not meet the
condition of originating from the pleural line (row 1, row 3, and row 4). It
is worth noting that the proposed method still has limited ability to
distinguish genuine B-lines from some vertical artifacts, often misidentifying
bright artifacts as B-lines, as shown in row 4.

Figure 6: Line detection results for various frames from different patients.
Ground truth is annotated by yellow bounding boxes. For model-based methods
and deep unfolding methods, pleural lines are depicted as red lines, and
detected B-lines are shown as green lines. For YOLOv7 and FasterRCNN, detected
B-lines are shown in blue and orange boxes, respectively.

Table II illustrates the average runtime of different optimization methods on
the test dataset. Although deep unfolding methods run slowly utilizing CPU
only, it is evident that employing deep unfolding significantly enhances
algorithmic optimization efficiency with the assistance of GPU. When trained
with SSIM loss, DUBLINE_SSIM and DUCPS_SSIM exhibit runtimes of 0.1079
seconds/frame and 0.1194 seconds/frame, respectively, representing an
efficiency improvement of at least 30 times compared to PUI [9]. It is worth
noting that in CPS [12], situations may arise where the iteration count
exceeds the maximum without convergence; such cases are not included in the
statistical results presented in the table. Nevertheless, the execution speed
of deep unfolding surpasses CPS by over tenfold. With N2N loss, although the
optimization time slightly increases due to the higher complexity of the loss
function compared to SSIM, the processing speed per frame still remains below
1 second.

TABLE II:  Average optimization speed on the test set (seconds/frame).

CPU only | PUI [9]∗ | CPS [12]∗∗ | DUBLINE_SSIM [27] | DUCPS_SSIM | DUCPS_N2N
---|---|---|---|---|---
4.1443 | 1.3239 | 15.4906 | 15.8665 | 15.9449
GPU | YOLOv7 [36] | FasterRCNN [37] | DUBLINE_SSIM [27] | DUCPS_SSIM | DUCPS_N2N
0.0423 | 0.0924 | 0.1079 | 0.1194 | 0.2542

∗ Excluding the line identification part.
∗∗ Excluding data that exceeds the maximum iteration count.

##  V Conclusion

The detection of artefacts in LUS, particularly B-lines, is gaining increasing
attention in clinical practice due to their significant diagnostic value. In
this work, we introduce a novel and explainable framework called DUCPS for
B-line detection, leveraging the technique of deep unfolding to combine the
strengths of traditional model-based methods and deep learning approaches. By
unfolding the CPS into a multi-layer network, the parameters in the
optimization algorithm become learnable, improving the algorithm’s
generalization performance and achieving rapid convergence. Subsequently, we
train the network in a totally unsupervised manner, achieving the best
performance across all metrics when using the N2N loss function. When
utilizing the SSIM loss function, the algorithm exhibits the highest
optimization efficiency. Overall, the proposed method enhances the
computational efficiency of traditional algorithms while achieving robust
detection without the need for extensive data labeling.

The limitation of this work stems from its relatively low precision. While
high recall effectively avoids missed detections, the increased FPs can
contribute to the workload of repeated validation in clinical diagnosis, which
is an aspect requiring further attention in future work.

## References

  * [1] T. Yang, O. Karakuş, N. Anantrasirichai, and A. Achim, “Current advances in computational lung ultrasound imaging: a review,” _IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control_ , 2022.
  * [2] D. A. Lichtenstein, “Lung ultrasound in the critically ill,” _Annals of Intensive Care_ , vol. 4, no. 1, pp. 1–12, 2014.
  * [3] V. E. Noble, A. F. Murray, R. Capp, M. H. Sylvia-Reardon, D. J. Steele, and A. Liteplo, “Ultrasound assessment for extravascular lung water in patients undergoing hemodialysis: time course for resolution,” _Chest_ , vol. 135, no. 6, pp. 1433–1439, 2009.
  * [4] M. Allinovi, M. Saleem, P. Romagnani, P. Nazerian, and W. Hayes, “Lung ultrasound: a novel technique for detecting fluid overload in children on dialysis,” _Nephrology Dialysis Transplantation_ , vol. 32, no. 3, pp. 541–547, 2017.
  * [5] Q. Fu, Z. Chen, J. Fan, C. Ling, X. Wang, X. Liu, and Y. Shen, “Lung ultrasound methods for assessing fluid volume change and monitoring dry weight in pediatric hemodialysis patients,” _Pediatric Nephrology_ , vol. 36, no. 4, pp. 969–976, 2021.
  * [6] L. J. Brattain, B. A. Telfer, A. S. Liteplo, and V. E. Noble, “Automated B-line scoring on thoracic sonography,” _Journal of Ultrasound in Medicine_ , vol. 32, no. 12, pp. 2185–2190, 2013.
  * [7] R. Moshavegh, K. L. Hansen, H. Møller-Sørensen, M. B. Nielsen, and J. A. Jensen, “Automatic detection of B-lines in $invivo$ lung ultrasound,” _IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control_ , vol. 66, no. 2, pp. 309–317, 2019.
  * [8] N. Anantrasirichai, M. Allinovi, W. Hayes, and A. Achim, “Automatic B-line detection in paediatric lung ultrasound,” in _2016 IEEE International Ultrasonics Symposium (IUS)_. IEEE, 2016, pp. 1–4.
  * [9] N. Anantrasirichai, W. Hayes, M. Allinovi, D. Bull, and A. Achim, “Line detection as an inverse problem: application to lung ultrasound imaging,” _IEEE Transactions on Medical Imaging_ , vol. 36, no. 10, pp. 2045–2056, 2017.
  * [10] S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein _et al._ , “Distributed optimization and statistical learning via the alternating direction method of multipliers,” _Foundations and Trends® in Machine learning_ , vol. 3, no. 1, pp. 1–122, 2011.
  * [11] M. Farahi, J. Aranda, H. Habibian, and A. Casals, “Automatic feature detection in lung ultrasound images using wavelet and radon transforms,” _arXiv preprint arXiv:2306.12780_ , 2023.
  * [12] O. Karakuş, N. Anantrasirichai, A. Aguersif, S. Silva, A. Basarab, and A. Achim, “Detection of line artifacts in lung ultrasound images of COVID-19 patients via nonconvex regularization,” _IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control_ , pp. 2218–2229, 2020.
  * [13] O. Karakuş, P. Mayo, and A. Achim, “Convergence guarantees for non-convex optimisation with Cauchy-based penalties,” _IEEE Transactions on Signal Processing_ , vol. 68, pp. 6159–6170, 2020.
  * [14] R. J. Van Sloun and L. Demi, “Localizing B-lines in lung ultrasonography by weakly supervised deep learning, in-vivo results,” _IEEE Journal of Biomedical and Health Informatics_ , pp. 957–964, 2019.
  * [15] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-CAM: Visual explanations from deep networks via gradient-based localization,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 618–626.
  * [16] S. Roy, W. Menapace, S. Oei, B. Luijten, E. Fini, C. Saltori, I. Huijben, N. Chennakeshava, F. Mento, A. Sentelli, E. Peschiera, R. Trevisan, G. Maschietto, E. Torri, R. Inchingolo, A. Smargiassi, G. Soldati, P. Rota, A. Passerini, R. J. G. van Sloun, E. Ricci, and L. Demi, “Deep learning for classification and localization of COVID-19 markers in point-of-care lung ultrasound,” _IEEE Transactions on Medical Imaging_ , vol. 39, no. 8, pp. 2676–2687, 2020.
  * [17] O. Frank, N. Schipper, M. Vaturi, G. Soldati, A. Smargiassi, R. Inchingolo, E. Torri, T. Perrone, F. Mento, L. Demi, M. Galun, Y. C. Eldar, and S. Bagon, “Integrating domain knowledge into deep networks for lung ultrasound with applications to COVID-19,” _IEEE Transactions on Medical Imaging_ , vol. 41, no. 3, pp. 571–581, 2021.
  * [18] C. Baloescu, G. Toporek, S. Kim, K. McNamara, R. Liu, M. M. Shaw, R. L. McNamara, B. I. Raju, and C. L. Moore, “Automated lung ultrasound B-line assessment using a deep learning algorithm,” _IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control_ , pp. 2312–2320, 2020.
  * [19] H. Kerdegari, N. T. H. Phung, A. McBride, L. Pisani, H. V. Nguyen, T. B. Duong, R. Razavi, L. Thwaites, S. Yacoub, A. Gomez, and V. Consortium, “B-line detection and localization in lung ultrasound videos using spatiotemporal attention,” _Applied Sciences_ , p. 11697, 2021.
  * [20] L. Zhao, T. C. Fong, and M. A. L. Bell, “COVID-19 feature detection with deep neural networks trained on simulated lung ultrasound B-mode images,” in _2022 IEEE International Ultrasonics Symposium (IUS)_. IEEE, 2022, pp. 1–3.
  * [21] T. Yang, N. Anantrasirichai, O. Karakuş, M. Allinovi, and A. Achim, “A semi-supervised learning approach for B-line detection in lung ultrasound images,” _2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI)_ , pp. 1–5, 2022.
  * [22] K. Gregor and Y. LeCun, “Learning fast approximations of sparse coding,” in _Proceedings of the 27th International Conference on International Conference on Machine Learning_ , 2010.
  * [23] Y. Yang, J. Sun, H. Li, and Z. Xu, “ADMM-CSNet: A deep learning approach for image compressive sensing,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 42, no. 3, pp. 521–538, 2018.
  * [24] J. Adler and O. Öktem, “Learned Primal-Dual reconstruction,” _IEEE Transactions on Medical Imaging_ , vol. 37, no. 6, pp. 1322–1332, 2018.
  * [25] E. Kang, W. Chang, J. Yoo, and J. C. Ye, “Deep convolutional framelet denosing for low-dose CT via wavelet residual network,” _IEEE Transactions on Medical Imaging_ , vol. 37, no. 6, pp. 1358–1369, 2018.
  * [26] R. J. Van Sloun, R. Cohen, and Y. C. Eldar, “Deep learning in ultrasound imaging,” _Proceedings of the IEEE_ , vol. 108, no. 1, pp. 11–29, 2019.
  * [27] T. Yang, N. Anantrasirichai, O. Karakuş, M. Allinovi, H. C. Koydemir, and A. Achim, “DUBLINE: A deep unfolding network for b-line detection in lung ultrasound images,” _arXiv preprint arXiv:2311.06672_ , 2023.
  * [28] B. T. Kelley and V. K. Madisetti, “The fast discrete radon transform. i. theory,” _IEEE Transactions on Image Processing_ , vol. 2, no. 3, pp. 382–400, 1993.
  * [29] A. Achim and E. E. Kuruoglu, “Image denoising using bivariate $\alpha$-stable distributions in the complex wavelet domain,” _IEEE Signal Proc. Letters_ , vol. 12, pp. 17–20, Jan. 2005.
  * [30] A. Achim, A. Bezerianos, and P. Tsakalides, “Novel Bayesian multiscale method for speckle removal in medical ultrasound images,” _IEEE Trans. Med. Imag._ , vol. 20, pp. 772–783, Aug. 2001.
  * [31] A. Balatsoukas-Stimming and C. Studer, “Deep unfolding for communications systems: A survey and some new directions,” in _2019 IEEE International Workshop on Signal Processing Systems (SiPS)_ , 2019, pp. 266–271.
  * [32] L. Z. Gengsheng, _Medical Image Reconstruction: A Conceptual Tutorial_. Higher Education Press, 2010.
  * [33] A. Loza, D. Bull, N. Canagarajah, and A. Achim, “Non-Gaussian model-based fusion of noisy images in the wavelet domain,” _Computer Vision and Image Understanding_ , vol. 114, no. 1, pp. 54–65, 2010.
  * [34] T. Huang, S. Li, X. Jia, H. Lu, and J. Liu, “Neighbor2Neighbor: Self-supervised denoising from single noisy images,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 14 781–14 790.
  * [35] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE Transactions on Image Processing_ , pp. 600–612, 2004.
  * [36] C.-Y. Wang, A. Bochkovskiy, and H.-Y. M. Liao, “YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023, pp. 7464–7475.
  * [37] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” _Advances in Neural Information Processing Systems_ , vol. 28, 2015.
  * [38] Z. Cai and N. Vasconcelos, “Cascade R-CNN: Delving into high quality object detection,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 6154–6162.
  * [39] K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, and Q. Tian, “CenterNet: Keypoint triplets for object detection,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 6568–6577.
  * [40] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _European Conference on Computer Vision (ECCV)_. Springer, 2020, pp. 213–229.
  * [41] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollar, “Focal loss for dense object detection,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , Oct 2017.


11institutetext: Universitat Politècnica de València, Spain
11email: {adepau<EMAIL_ADDRESS>
22institutetext: Escola Politécnica da Universidade de São Paulo
22email<EMAIL_ADDRESS>

# Sexism Prediction in Spanish and English Tweets Using Monolingual and
Multilingual BERT and Ensemble Models

Angel Felipe Magnossão de Paula 11 0000-0001-8575-5012 Roberto Fray da Silva
22 0000-0002-9792-0553 Ipek Baris Schlicht 11 0000-0002-5037-2203

###### Abstract

The popularity of social media has created problems such as hate speech and
sexism. The identification and classification of sexism in social media are
very relevant tasks, as they would allow building a healthier social
environment. Nevertheless, these tasks are considerably challenging. This work
proposes a system to use multilingual and monolingual BERT and data points
translation and ensemble strategies for sexism identification and
classification in English and Spanish. It was conducted in the context of the
sEXism Identification in Social neTworks shared 2021 (EXIST 2021) task,
proposed by the Iberian Languages Evaluation Forum (IberLEF). The proposed
system and its main components are described, and an in-depth hyperparameters
analysis is conducted. The main results observed were: (i) the system obtained
better results than the baseline model (multilingual BERT); (ii) ensemble
models obtained better results than monolingual models; and (iii) an ensemble
model considering all individual models and the best standardized values
obtained the best accuracies and F1-scores for both tasks. This work obtained
first place in both tasks at EXIST, with the highest accuracies (0.780 for
task 1 and 0.658 for task 2) and F1-scores (F1-binary of 0.780 for task 1 and
F1-macro of 0.579 for task 2).

###### Keywords:

Sexism identification Sexism classification BERT Deep learning.

††footnotetext: IberLEF 2021, September 2021, Málaga, Spain.
Copyright © 2021 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).

##  1 Introduction

The emergence of social networks and microblogs has created a new medium for
people to express themselves, providing freedom of speech and the possibility
for quickly spreading opinions, news, and information [17, 14]. This has
impacted considerably on peoples’ lives, by increasing access to all kinds of
information. Nevertheless, a small part of the users employs those media for
spreading hate messages, increasing the impacts of racism, sexism, and others
types of prejudices and hate speech [32, 10].

One crucial problem faced by the different stakeholders related to social
media platforms is detecting hate speech [10, 35, 6, 5], both in general and
issue-specific forms. Also, some types of hate speech tend to be more
challenging to identify, as they present characteristics such as irony or
sarcasm, among others [35, 6, 23]. Sexism is a type of toxic language that
could be used both as hate speech and, in a much more subtle way, as sarcasm.
Sexism is related to all kinds of behaviors and content that aim to spread
prejudice against women, reduce their importance in society, or behave
aggressively or offensively [27, 6, 23]. There are several forms of sexism,
and identifying them in social media messages is a fundamental challenge among
the various natural language processing (NLP) tasks [10, 35, 27, 6, 23, 5,
26].

The detection of sexism can be broken into two main tasks: (i) sexism
identification, which aims to identify if a message or post containing sexist
contents (regardless of the type of sexism contained in it); and (ii) sexism
classification, which aims to classify the type of sexism contained in a given
sexist message or post [6, 27, 26, 11]. Both are very relevant, and the second
task is dependent on the first, as it needs posts that are confirmed as sexist
as inputs for the different classification models. Additionally, the
difficulty of using data-driven models may increase for languages that are
more complex or that have fewer resources available, such as high-quality word
embeddings, pre-trained language-specific models, task-specific lexicons,
among others.

To advance the state-of-the-art knowledge in both sexism identification and
classification on social media messages, the Iberian Languages Evaluation
Forum (IberLEF) proposed the sEXism Identification in Social neTworks shared
2021 (EXIST 2021) shared task. For the rest of this work, this challenge will
be referred to as EXIST shared task. The main goal of the IberLEF forum is to
promote scientific advances towards innovative solutions for detecting sexism
on social media platforms [19]. For this reason, the 2021 shared task provided
datasets in English and Spanish for both tasks labeled by experts, following
state-of-the-art data collection and labeling procedures [19]. Those datasets
are expected to become benchmarks for state-of-the-art research on sexism
identification and classification on social media messages.

Therefore, a relevant gap in the literature is to develop data-driven models
that better identify and classify sexist content on social media messages,
considering the implementation in different languages. This would: (i) advance
both the knowledge on the use of artificial intelligent models for data-driven
sexism identification and detection; (ii) provide a better methodology for
identifying and classifying sexist content, which is highly relevant for
identifying unacceptable user behavior; and (iii) address the problem of
generalizing the model throughout different languages. Related to this gap, it
is vital to observe that identifying online sexism can be considerably
challenging because posts may have several forms: they may sound hateful and
offensive, or friendly and funny, misleading the current classifying models
used for this task [27].

State of the art systems for addressing those tasks for multiple languages
uses the Bidirectional Encoder Representations from Transformers (BERT)
multilingual model, an NLP model that uses transformers and is pre-trained on
a comprehensive text corpora [31, 25, 21, 20, 16]. This model is trained on
datasets of multiple languages, but it is not language-specific. The pre-
trained models are then fine-tuned on task-specific datasets on the target
language.

The main goal of this work is to propose and evaluate a system to identify and
classify sexist content in social media messages in multiple languages, using
the EXIST 2021 shared task dataset [19] for implementation and evaluation. The
official shared task metrics were used: accuracy for task 1 (sexism detection)
and F1-macro for task 2 (sexism classification). However, we also implemented
other relevant metrics for NLP tasks to better evaluate the different models
in relation to the state of the art baseline model, the multilingual BERT:
precision and recall.

The three main research questions that are going to be addressed in this work
are: (i) does the use of monolingual BERT models provides better results than
the multilingual BERT model to identify and classify sexist content on social
media messages in English and Spanish?; (ii) does the use of an ensemble
strategy improves the results of the individual models?; and (iii) does the
results differ between the English and Spanish languages? Besides answering
those three questions, this work also conducts an in-depth analysis of the
main hyperparameters for the implemented models for both languages.

The main contribution of this work is to propose and evaluate the sexism
identification and classification system in multiple languages considering
different components: monolingual BERT models, multilingual BERT, data points
translation, and different ensemble strategies. We also explore the main
hyperparameters of the implemented models in-depth, comparing the final models
with the state-of-the-art BERT multilingual model. This work obtained first
place in both sexism identification and classification tasks at the EXIST
shared task [19].

This work is organized in the following sections: section 2 describes the main
concepts and models used for sexism prediction in social media messages;
section 3 contains the main steps of the methodology used; section 4 describes
the proposed system for addressing both sexism identification and
classification; section 5 contains the main results of the system’s
implementation on the EXIST shared task dataset; section 6 contains a
discussion of relevant topics on the system’s use, modification, and potential
improvements; and section 7 concludes this paper.

##  2 Sexism identification and classification using artificial intelligence
models

The works by [23, 27, 6] explore in-depth the impacts of the different types
of sexism on social media platforms, describing several important classes of
sexism. As sexism is an important type of hate speech, we also refer the
reader to the works by [5, 35] for excellent reviews on identifying and
classifying the different forms of hate speech. The main concepts observed in
those works were considered in our approach.

This work addresses two very relevant tasks: (i) sexism identification in
natural language texts; and (ii) classification of types of sexism in natural
language texts. Some examples of works that addressed the first task are [21,
18, 24]. Examples that addressed the second task are [10, 16, 29].

It is essential to observe that the second task is considerably more complex
because different languages can be used in the different classes (as well as
the traditional problems related to social media messages: abbreviations,
emojis, misspellings, memes, among others).

Although there are a variety of different models and strategies used for
sexism detection and classification, the most traditionally used models are:
support vector machines (SVM), convolutional neural networks (CNN), long
short-term neural networks (LSTM), and BiLSTM [23, 13, 27, 11, 6, 35]. In the
last years, the BERT has been widely used [23, 13, 27, 11, 6, 35]. This model
(and its variations) have presented the best results on those tasks, as
observed in the works by [16, 31, 21, 25].

The NLP literature addresses several identification and classification tasks
related to extracting and evaluating opinions from natural language texts. In
general, those tasks are addressed using three main approaches [15, 30, 22]:
(i) lexical-based, in which specific dictionaries (lists of words with
corresponding values on important dimensions for the task) are used to
classify the input text; (ii) statistical learning or machine learning-based,
in which machine learning and deep learning models are used, generally with
word embeddings or bag of words models, to classify the text; and (iii)
hybrid, in which both lexicons and machine learning models are used.

However, it is essential to note that: (i) lexical-based systems are not able
to learn (and could be improved by using a deep learning model, such as BERT);
(ii) deep learning models, especially the multilingual BERT, are state-of-the-
art on sexism identification and classification [31, 25, 21, 20, 16]; (iii)
lexicons tend to be language-specific, making it more challenging to apply the
solutions for multiple languages; and (iv) few works use BERT with domain-
specific lexicons for sexism identification and classification. One of those
lexicons that are highly relevant in this context is the Hurtlex lexicon [3].
This was used in the works by [24, 16], among several others.

The BERT model was proposed by [9] and can be described as a language learning
model aimed at providing a general structure that can be further refined with
fine-tuning on specific tasks and domains. Its main objective is to learn the
main features and semantics of a language, based on semi-supervised learning
on a vast text corpora (such as the BookCorpus and the Wikipedia database) [9,
28, 1]. Its architecture and training workflow is composed of three main
components: transformers (which is an advanced deep learning model),
bidirectional training, and use of encoder representations [9, 28, 1].

In this work, we use the multilingual BERT [9], the English version of the
model [9], and the Spanish version of the model, called BETO [4]. For an in-
depth analysis of how the BERT model works, we refer the reader to [28]. For
an in-depth comparison with multilingual BERT with other models, as well as an
in-depth description of how they work, we refer the reader to [34].

However, very few works in the literature consider dealing with datasets with
multiple languages. This work addressed this gap by proposing a system that
contains multiple models and ensemble strategies.

This paper aims to fulfill the gap of evaluating monolingual and multilingual
BERT models for identifying and classifying sexism in texts in multiple
languages.

##  3 Methodology

The methodology used in this work was composed of six steps. Figure 1
illustrates the strategy used to tackle each of the tasks. For task 1 (sexism
identification), the classification models considered two labels: 0 (non-
sexist) and 1 (sexist). For task 2 (sexism classification), the tweets labeled
as non-sexist (from task 1) were eliminated. Then, the classification models
were used to predict the following sexism categories on the remaining tweets:
ideological and inequality; stereotyping and dominance; objectification;
sexual violence; and misogyny and non-sexual violence. For a thorough
description of those classes, we refer to the EXIST shared task at IberLEF
2021 [19], which developed and labeled the dataset that was used in this
research.

Figure 1: Illustration of the two main tasks evaluated in this research.

The steps of the methodology were:

1\. Data collection: we used the dataset developed for the EXIST shared task
at IberLEF 2021 [19]. This dataset contained labeled data from two social
media platforms: Twitter and Gab. For an in-depth description of this dataset,
we refer the reader to Section 5 of this work;

2\. Data processing: for both tasks, we used the following processing
techniques: separation of the dataset between languages (English and Spanish),
tokenization, lemmatization, and elimination of stop words. These are widely
used in the literature for the implementation of machine learning models on
NLP tasks, such as hate speech detection, sexism identification, sentiment
analysis, among others [31, 25, 21, 20, 16, 22]. There was no need to
eliminate data points from the datasets, as the shared task organizers had
already thoroughly curated them. The training subset was then divided into
training (80%) and validation (20%) for cross-validation purposes.
Additionally, one of the training strategies used for some of the models
implemented involved translating the social media messages from one language
to the other (for example, from English to Spanish to train a Spanish language
model). This strategy doubled the number of data points available for the
single language models (even if part of the meaning may have been lost during
the translation process). The googletrans (https://github.com/ssut/py-
googletrans) library was used for the translation process;

3\. Exploratory data analysis: in this step, an exploratory analysis of the
dataset was conducted to understand better the different class distributions
on both tasks throughout the training dataset. No data imbalance problems were
observed;

4\. Model implementation and hyperparameters analysis: in this research, we
implemented the following models: (i) the BERT Multilingual model or mBERT [9]
(named M1 in this research); (ii) single language models (one for English and
one for Spanish, named M2-English and M2-Spanish); (iii) single language
models with translated data points (one for English and one for Spanish, named
M3-English and M3-Spanish); and (iv) ensemble models (used only for the test
subset). All the implementations were conducted with the Hugging Face BERT
implementation library (https://huggingface.co/transformers/index.html) [33],
with 10-fold cross-validation on the training stage. A thorough
hyperparameters analysis was conducted, considering the following
hyperparameters and values: output BERT type (hidden or pooler), batch size
(32 and 64), learning rate (0.00002, 0.00003, and 0.00005), and number of
epochs (1 to 8). Following the official metrics of the EXIST 2021 shared task,
accuracy was used as the quality metric for model training on task 1, and
F1-macro was used on task 2. Besides this metric, an analysis of model
overfitting was conducted for each model, based on charts that contained the
models’ accuracies on the different epochs;

5\. Final models implementation: the final models and model ensembles were
built using the best hyperparameters identified in Step 4. They were then
trained on the whole training datasets (training plus validation subsets).
Table 1 contains all the final models implemented: (i) M1: multilingual model;
(ii) separated single language models without translation on the training
datasets (M2, composed of M2-English and M2-Spanish) and with translation on
the training datasets (M3, composed of M3-English and M3-Spanish); (iii)
English single language model with translation only on the test subset (M4)
and on training and test subsets (M5), both derived from the M3-English model;
(iv) Spanish single language model with translation only on the test subset
(M6) and on training and test subsets to Spanish (M7), both derived from the
M3-Spanish model; (v) ensembles considering only the best models: E1 (majority
vote), E2 (higher unstandardized value), and E3 (higher standardized value);
and (vi) ensembles considering all the models: E4 (majority vote), E5 (higher
unstandardized value), and E6 (higher standardized value);

Table 1: Final models implemented and their characteristics Model | Multilingual | Translation of data points | Observations
---|---|---|---
Training | Test
M1 | X |  |  | Baseline model
M2 |  |  |  | Separated single language models
M3 |  | X |  | Separated single language models
M4 |  |  | X | English single language model
M5 |  | X | X | English single language model
M6 |  |  | X | Spanish single language model
M7 |  | X | X | Spanish single language model
E1 | Depends on the individual models used | Majority vote, best models
E2 | Depends on the individual models used | Unstandardized value, best models
E3 | Depends on the individual models used | Standardized value, best models
E4 | Depends on the individual models used | Majority vote, all models
E5 | Depends on the individual models used | Unstandardized value, all models
E6 | Depends on the individual models used | Standardized value, all models

6\. Models comparison: the final comparison of all models was then conducted
on the test subsets. The official metrics for the EXIST shared task at IberLEF
2021 [19] were considered the quality metrics for both sexism identification
and classification tasks. For task 1, we evaluated the accuracy, precision,
recall, and F1-binary metrics. For task 2, we evaluated the accuracy,
precision, recall, and F1-macro metrics). Additionally, an analysis of a
sample of correctly and incorrectly classified data points on the test set was
conducted, aiming to better understand each model’s main strengths,
weaknesses, and opportunities for future improvements. Lastly, the best model
was chosen.

The implementation was done using Python on a Google Collaboratory Pro
(https://colab.research.google.com/) TPU , with the following technical
specifications: Intel(R) Xeon(R) CPU @ 2.30GHz CPU, 26GB of RAM, and TPU v2.
The code implemented is available on an open Github repository
(https://github.com/AngelFelipeMP/BERT-tweets-sexims-classification). Section
5 presents the main results of the exploratory data analysis and the model
implementations.

##  4 Proposed system: components and implementation

The proposed system considered two separate workflows: training and testing.
The objective of the training workflow was to fine-tune the pre-trained BERT
models. It considered three options: (i) using a multilingual BERT model
(illustrated in Figure 2), which was also considered our baseline, since it is
the state of the art for multilingual NLP classification tasks; (ii) using
monolingual BERT models without data points translation (illustrated in Figure
3); and (iii) using monolingual BERT models with data points translation
(illustrated in Figure 3).

Figure 2: Workflow for training the Multilingual model (M1). Figure 3:
Workflow for training the monolingual models (M2-English and M2-Spanish) and
the translated languages models (M3-English and M3-Spanish), considering the
English models as an example.

It is essential to observe that those three options considered 10-fold cross-
validation on training to identify the best hyperparameter values for each
model. The result from the first option was the M1 model. The results from the
second option were the models M2-English and M2-Spanish, which would be used
as components of the M2 model. The results from the third option were the
models M3-English and M3-Spanish, which would be used as components of the M3
model.

Figure 4 illustrates the test workflow. The objective of this workflow was to
use the previously tested models as components for the final models, their
training on the whole training dataset (training plus validation subsets), and
testing on the test subset. This workflow also introduces the six ensemble
models implemented, considering different model configurations and rules for
generating the models. It is vital to observe that this model could be easily
expanded for other languages, quality metrics, and data sources.

Figure 4: Workflow for testing the final models on the test subset.

##  5 Results

This section contains the main research results and is divided into three
subsections: 5.1 contains a description of the dataset used; 5.2 contains the
main results and observations related to the hyperparameters analysis; and 5.3
contains the final models’ comparison on the test subset, considering four
metrics: accuracy, precision, recall, and F1-score (F1-binary for task 1 and
F1-macro for task 2).

###  5.1 Description of the EXIST 2021 shared task dataset

The dataset from EXIST 2021 shared task at IberLEF 2021 [19] was used in this
work. This dataset contained labeled data from two social media platforms: (i)
Twitter, with 6,977 tweets for training and 3,386 tweets for testing (both
subsets equally distributed between English and Spanish); and (ii) Gab, with
492 gabs in English and 490 gabs in Spanish (used only for testing purposes).
It is important to note that Gab is an uncensored social media website with
considerably fewer users than Twitter.

It is vital to observe that the labeling procedure adopted by the shared task
organizers considered both experts and crowdsourcing labeling (considering a
specific procedure developed by experts in this domain). The dataset
distribution was balanced on the training and test subsets. For a thorough
description of the dataset, we refer the readers to IberLEF 2021 [19].

The five classes that were used in this work for the sexism classification
task (also referred to by the organizers of the dataset as sexism
categorization) are the ones provided by the EXIST challenge dataset[19].
These classes contain, as described by [19, 27]:

  * •

Ideological and inequality: texts that affirm that the feminist movement
deserves no credits, rejects the existence of inequality between genders, or
claims that men are oppressed gender;

  * •

Stereotyping and dominance: texts that claim that women are inappropriate for
specific tasks, suitable only for specific roles, or that men are superior to
women;

  * •

Objectification: texts that claim that women should have certain physical
qualities or that separate women from their dignity and personal aspects;

  * •

Sexual violence: texts that contain sexual suggestions or sexual harassment;

  * •

Misogyny and non-sexual violence: texts that express different forms of hatred
and violence towards women.

###  5.2 Hyperparameters analysis

Due to the considerable difference between the identification and
classification tasks, this section will analyze both separately and then
conclude with a comparison of the best hyperparameters values for all models
for both tasks. The results observed in this subsection can be used as a guide
for further implementations of BERT models for sexism identification and
classification, considering multiple languages.

Table 2 illustrates the best hyperparameters values for task 1 for each of the
five models implemented on the training step of the proposed system (M1,
M2-Eng, M2-Sp, M3-Eng, and M3-Sp), as well as their associated quality metrics
on the validation subset.

Table 2: Best hyperparameters values, accuracy, precision, recall and
F1-binary for the different models on the validation subset of the training
dataset for Task 1 - Sexism identification.

Lang. | Model | Best hyperp. values | Acc. | Prec. | Rec. | F1b
---|---|---|---|---|---|---
Multi | M1 | OB:pooler / Lr:0.00005 / Bs:32 / Ne:7 | 0.774 | 0.749 | 0.808 | 0.774
English | M2-Eng | OB:hidden / Lr:0.00005 / Bs:32 / Ne:5 | 0.782 | 0.764 | 0.788 | 0.768
M3-Eng | OB:pooler / Lr:0.00005 / Bs:64 / Ne:6 | 0.765 | 0.748 | 0.778 | 0.759
Spanish | M2-Sp | OB:hidden / Lr:0.00005 / Bs:32 / Ne:8 | 0.790 | 0.780 | 0.795 | 0.783
M3-Sp | OB:hidden / Lr:0.00005 / Bs:32 / Ne:6 | 0.775 | 0.756 | 0.795 | 0.771

Legend: Lang.: model language; OB: output BERT type; Lr: learning rate; Bs:
batch size; Ne: number of epochs; Acc.: accuracy; Prec.: precision; Rec.:
recall; F1b: F1-binary.

Based on the analysis of Table 2, it is important to observe that: (i) the
M2-Eng (monolingual without translation) presented better results for the
English language; (ii) the M2-Sp (monolingual without translation) presented
better results for the Spanish language; (iii) the F1-binary for the Spanish
language models (M2-Sp and M3-Sp) presented better results than for the
English language models (M2-Eng and M3-Eng); (iv) most of the models presented
better results by using the hidden output BERT type; (v) all models presented
a learning rate of 0.00005; (vi) most models presented better results with a
batch size of 32; and (vii) most models presented better results with 6 or
more epochs. We have focused the analysis and model choice considering the
accuracy, as it was the official metric for the EXIST shared task.

Table 3 illustrates the best hyperparameters values for task 2 for each of the
five models implemented on the training step of the proposed system and their
quality metrics on the validation subset. It is possible to observe that: (i)
similar to task 1, the M1 model (multilingual) did not present better results
than any of the languages; (ii) the M3-Eng (monolingual with translation)
presented the best results for the English language; (iii) the M3-Sp
(monolingual with translation) presented better results for the Spanish
language; (iv) most of the models presented better results by using the hidden
output BERT type; (v) like in task 1, all models presented better results by
using a learning rate of 0.00005; (vi) most models presented better results
with a batch size of 32; and (vii) most models presented better results with 7
or 8 epochs.

Table 3: Best hyperparameters values, accuracy, precision, recall and F1-macro
for the different models on the validation subset of the training dataset for
Task 2 - Sexism classification.

Lang. | Model | Best hyperp. values | Acc. | Prec. | Rec. | F1m
---|---|---|---|---|---|---
Multi | M1-Multi | OB:pooler / Lr:0.00005 / Bs:32 / Ne:8 | 0.636 | 0.632 | 0.624 | 0.604
English | M2-Eng | OB:hidden / Lr:0.00005 / Bs:32 / Ne:8 | 0.661 | 0.647 | 0.633 | 0.610
M3-Eng | OB:hidden / Lr:0.00005 / Bs:32 / Ne:5 | 0.661 | 0.660 | 0.652 | 0.632
Spanish | M2-Sp | OB:hidden / Lr:0.00005 / Bs:32 / Ne:8 | 0.682 | 0.656 | 0.670 | 0.628
M3-Sp | OB:hidden / Lr:0.00005 / Bs:64 / Ne:7 | 0.656 | 0.653 | 0.650 | 0.630

Legend: Lang.: model language; OB: output BERT type; Lr: learning rate; Bs:
batch size; Ne: number of epochs; Acc.: accuracy; Prec.: precision; Rec.:
recall; F1m: F1-macro.

Lastly, Table 4 contains a cross-model and cross-language analysis of the
results on the validation subset. It presents the hyperparameter values of the
best models in each category (M1, M2-Eng, M2-Sp, M3-Eng, and M3-Sp), as a
percentage of the total number of models, for each task. For example, on the
first cell, it is possible to observe that, for the output BERT type on task
1, 60% of the final models contained a hidden output BERT, while 40% used the
pooler type. Based on an analysis of this table, it is possible to conclude
that: (i) for both tasks, the hidden output BERT type provided the best
results; (ii) higher learning rates (0.00005) presented the best results for
both tasks; (iii) the best batch size for both tasks was 32; and (iv) the best
number of epochs differed among tasks, probably due to their different nature.

Table 4: Percentage of best performing models for each hyperparameter value for both tasks and languages, considering the highest accuracy for task 1 and F1-macro for task 2 on the validation subset. Hyperparameter |  | Hyperparameter values and percentage
---
of best performing models
Task 1 | Task 2
Output BERT type |  | Hidden: 60%
---
Pooler: 40%
| Hidden: 80%
---
Pooler: 20%
Learning rate |  | 0.00002: 0%
---
0.00003: 0%
0.00005: 100%
| 0.00002: 0%
---
0.00003: 0%
0.00005: 100%
Batch size |  | 32: 80%
---
64: 20%
| 32: 80%
---
64: 20%
Number epochs |  | <= 6: 60%
---
7: 20%
8: 20%
| <= 6: 20%
---
7: 20%
8: 60%

###  5.3 Final models comparison

Table 5 contains the results of the thirteen final models on the test subset
for both tasks, considering the following metrics: accuracy, precision,
recall, and F1-score (F1-binary for task 1 and F1-macro for task 2). For both
tasks, it is vital to observe that: (i) the ensemble models presented a better
F1-score than the monolingual models and the multilingual model; and (ii) the
E6 (ensemble model considering all individual models and the best standardized
values) obtained the best accuracy and F1-score.

Table 5: Accuracy, precision, recall and F1-score of the final models on the
test subset for tasks 1 and 2.

Model | Task 1 - Sexism identification | Task 2 - Sexism classification
---|---|---
Acc. | Prec. | Rec. | F1b | Acc. | Prec. | Rec. | F1m
M1 | 0.761 | 0.739 | 0.784 | 0.761 | 0.621 | 0.617 | 0.621 | 0.611
M2 | 0.774 | 0.749 | 0.803 | 0.775 | 0.688 | 0.677 | 0.674 | 0.675
M3 | 0.782 | 0.773 | 0.778 | 0.775 | 0.676 | 0.661 | 0.663 | 0.658
M4 | 0.732 | 0.737 | 0.693 | 0.715 | 0.612 | 0.607 | 0.597 | 0.595
M5 | 0.766 | 0.773 | 0.730 | 0.751 | 0.661 | 0.648 | 0.657 | 0.649
M6 | 0.735 | 0.739 | 0.699 | 0.718 | 0.639 | 0.629 | 0.621 | 0.624
M7 | 0.753 | 0.734 | 0.769 | 0.751 | 0.642 | 0.625 | 0.627 | 0.624
E1 | 0.784 | 0.797 | 0.744 | 0.769 | 0.669 | 0.683 | 0.666 | 0.653
E2 | 0.786 | 0.777 | 0.791 | 0.781 | 0.686 | 0.673 | 0.677 | 0.674
E3 | 0.784 | 0.773 | 0.784 | 0.779 | 0.682 | 0.669 | 0.674 | 0.670
E4 | 0.790 | 0.781 | 0.787 | 0.784 | 0.661 | 0.673 | 0.656 | 0.645
E5 | 0.785 | 0.767 | 0.799 | 0.782 | 0.701 | 0.687 | 0.690 | 0.687
E6 | 0.789 | 0.776 | 0.794 | 0.785 | 0.703 | 0.690 | 0.692 | 0.689

Legend: Acc.: accuracy; Prec.: precision; Rec.: recall; F1b: F1b-binary; F1m:
F1-macro. M1 is the baseline (BERT Multilingual).

For task 1, it is essential to observe on Table 5 that: (i) the baseline model
(M1) presented better accuracy results than the M4, M6, and M7 models; (ii)
the E4 model presented results that were comparable to the E6 model in terms
of accuracy, both being considered the best models for this task; (iii) the E1
model presented the best precision; and (iv) the M2 model presented the best
recall. For task 2, it can be observed that: (i) with an exception for M4, all
models presented a better F1-macro than the M1 model, indicating that the use
of monolingual may provide significantly better results than multilingual
models for sexism classification; and (ii) the E6 model presented the best
results for all metrics, indicating that it outperformed all other models for
this task.

Table 6 presents a comparison of the best individual and ensemble models for
tasks 1 and 2, considering the two official metrics of the EXIST shared task:
accuracy and F1-score. Considering the differences of the F1-scores of each
model and the best model, it is possible to conclude that: (i) the differences
are significantly higher for the task of sexism classification; (ii) the
baseline model (M1) obtained the worst F1-score among those models (around 3%
lower for task 1 and 11% for task 2 in comparison to the E6 model); (iii) the
baseline model (M1) obtained the worst accuracy for both tasks; and (iv)
although the E4 model presented similar results for task 1, it observed a
6.39% lower F1-score in comparison to the E6 model for task 2. The analysis of
the models’ accuracies lead to the same conclusions.

It is also possible to observe that the same overall conclusions hold for the
accuracy metric.

Table 6:  Comparison of best individual and ensemble models for tasks 1 and 2
considering accuracy and F1-score.

Model | Task 1 - Sexism identification | Task 2 - Sexism classification
---|---|---
Acc. | Diff E6 | F1b | Diff E6 | Acc. | Diff E6 | F1m | Diff E6
M1 | 0.761 | -3.55% | 0.761 | -3.06% | 0.621 | -11.66% | 0.611 | -11.32%
M2 | 0.774 | -1.90% | 0.775 | -1.27% | 0.688 | -2.13% | 0.675 | -2.03%
M3 | 0.782 | -0.89% | 0.775 | -1.27% | 0.676 | -3.84% | 0.658 | -4.50%
E4 | 0.790 | 0.13% | 0.784 | -0.13% | 0.661 | -5.97% | 0.645 | -6.39%
E6 | 0.789 | 0.00% | 0.785 | 0.00% | 0.703 | 0.00% | 0.689 | 0.00%

Legend: Acc.: accuracy; Prec.: precision; Rec.: recall; F1b: F1b-binary; F1m:
F1-macro. M1 is the baseline (BERT Multilingual). Diff E6 is the difference
between that model’s metric and the same metric for the E6 model (best overall
model for both tasks).

Our approach ranked first in both sexism identification and classification
tasks at EXIST, with the highest accuracies (0.780 for task 1 and 0.658 for
task 2) and F1-scores (F1-binary of 0.780 for task 1 and F1-macro of 0.579 for
task 2), considering the E6 model. We also observed that ensemble models
provide a better generalization.

##  6 Discussions

This section briefly explores several important aspects related to the system
proposed in this work and its results, encompassing the following topics:
implementation aspects, system design, use of ensembles, system adaptation for
other languages, results obtained concerning the literature, impacts of the
different system components, and the use of the proposed system in real
scenarios.

It is vital to note that the system proposed in this work can be extended
using additional components with few adaptations to the code. Some additional
interesting components to explore are lexicons (both generalist, such as Vader
[12] and domain-specific, such as Hurtlex [3]), word embeddings, and transfer
learning (via training on multiple datasets). Additional models could also be
implemented to improve feature engineering (such as unsupervised learning
models) or improve prediction quality (such as different weak models used in
an ensemble strategy).

Although ensemble models are relatively common in other domains, such as price
prediction [2] and sentiment analysis [7], they are not widely spread on
sexism identification and classification, as this is a new task. In general,
if the behavior of the weak models can capture different aspects of the task,
an ensemble strategy could improve the final prediction results [2, 8]. In
this work, we evaluated several simple average ensemble strategies. However,
an in-depth analysis of more complex ensemble strategies with the proposed
system could be conducted in future works. As was observed in this work, the
use of ensembles can significantly improve the results obtained by the
individual models.

Another important aspect is related to adapting the proposed system for other
languages. Concerning this aspect, it is vital to separate the languages into
two main groups: (i) languages with individual BERT models already
implemented; and (ii) languages that currently have no widely accepted
individual BERT model implemented. In the first group, the system allows for
easy implementation with minimal coding needed: the only components needed are
the BERT individual model pre-trained on that language and the task-specific
dataset for fine-tuning and testing.

In the second group, it is necessary to train a language-specific BERT model
before using the proposed system. This task demands a considerable amount of
computation power and resources, demanding processing clusters and large text
corpora in the target language (such as the Wikipedia text database). However,
adaptations can be implemented in the proposed system to use different models
that are easier to implement and demand fewer data, such as recurrent neural
networks or convolutional neural networks with language-specific word
embeddings or lexicons. The ensemble component can then be used, considering
the BERT multilanguage (if it encompasses the target language) and the
implemented models.

Lastly, the proposed system can be implemented and used in real case scenarios
to improve sexism identification on social media platforms. After the
hyperparameters and final models are chosen, as described and explored in this
work, the prediction process is considerably fast. The system has the
potential to be implemented as a separate service on the social media
platform, analyzing the content published by its users and pointing out sexist
messages and their respective classes (considering the five classes studied in
this work).

##  7 Conclusion and future work

As was explored throughout this work, a widespread problem on social networks
and microblogs is the misuse of these tools to spread toxic language and
sexist content. Identifying and detecting sexism in those media is
considerably challenging, especially in a scenario with multiple languages.
This paper explored the fine-tuning of multilingual and monolingual BERT
models for English and Spanish and the use of different ensemble
configurations to identify and classify sexism on tweets and gabs. The dataset
used was provided by the EXIST shared task, which contained two tasks: sexism
identification and sexism classification.

The proposed system in this research considered the use of fine-tuning of pre-
trained BERT models and the translation of the training dataset (to increase
the number of data points used by the model for learning) and ensemble models
with different characteristics. Our central hypothesis was that using this
system would provide better results than the traditional use of the
multilingual BERT model. Our results have shown that the use of ensembles
provided better results for both tasks, primarily the ensemble that considered
all trained models and the higher standardized label values as the final
predictions. This model obtained significantly better results than the
baseline multilingual BERT model, with an F1-score around 3% higher for the
sexism identification task and 11% higher for the sexism classification task.
Those results and models and the in-depth hyperparameters analysis that was
conducted can be used as a guide for future research on both tasks.

Future works are related to: (i) conducting an analysis considering additional
datasets; (ii) implementing additional models; (iii) implementing different
ensemble configurations; (iv) implementing unsupervised models for feature
engineering; (v) analyzing the impacts on the models’ results of using
lexicons (both general and domain-specific) as features; (vi) analyzing the
impacts on the models’ results of using word embeddings as features; and (vii)
implementing and evaluating the use of deep reinforcement learning to improve
the models’ results, especially on the sexism classification problem.

## References

  * [1] Acheampong, F.A., Nunoo-Mensah, H., Chen, W.: Transformer models for text-based emotion detection: a review of bert-based approaches. Artificial Intelligence Review pp. 1–41 (2021)
  * [2] Ballings, M., Poel, D.V.D., Hespeels, N., Gryp, R.: Evaluating multiple classifiers for stock price direction prediction. Expert Systems with Applications 42(20), 7046–7056 (2015). https://doi.org/10.1016/j.eswa.2015.05.013
  * [3] Bassignana, E., Basile, V., Patti, V.: Hurtlex: A multilingual lexicon of words to hurt. In: 5th Italian Conference on Computational Linguistics, CLiC-it 2018\. vol. 2253, pp. 1–6. CEUR-WS (2018)
  * [4] Cañete, J., Chaperon, G., Fuentes, R., Ho, J.H., Kang, H., Pérez, J.: Spanish pre-trained bert model and evaluation data. In: PML4DC at ICLR 2020 (2020)
  * [5] Chetty, N., Alathur, S.: Hate speech review in the context of online social networks. Aggression and violent behavior 40, 108–118 (2018)
  * [6] Chiril, P., Moriceau, V., Benamara, F., Mari, A., Origgi, G., Coulomb-Gully, M.: An annotated corpus for sexism detection in french tweets. In: Proceedings of The 12th Language Resources and Evaluation Conference. pp. 1397–1403 (2020)
  * [7] Da Silva, N.F., Hruschka, E.R., Hruschka Jr, E.R.: Tweet sentiment analysis with classifier ensembles. Decision Support Systems 66, 170–179 (2014)
  * [8] Dean, N.E., y Piontti, A.P., Madewell, Z.J., Cummings, D.A.T., Hitchings, M.D.T., Joshi, K., Kahn, R., Vespignani, A., Halloran, M.E., Longini Jr, I.M.: Ensemble forecast modeling for the design of COVID-19 vaccine efficacy trials. Vaccine 38(46), 7213–7216 (2020)
  * [9] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  * [10] Founta, A., Djouvas, C., Chatzakou, D., Leontiadis, I., Blackburn, J., Stringhini, G., Vakali, A., Sirivianos, M., Kourtellis, N.: Large scale crowdsourcing and characterization of twitter abusive behavior. In: Proceedings of the International AAAI Conference on Web and Social Media. vol. 12 (2018)
  * [11] Frenda, S., Ghanem, B., Montes-y Gómez, M., Rosso, P.: Online hate speech against women: Automatic identification of misogyny and sexism on twitter. Journal of Intelligent & Fuzzy Systems 36(5), 4743–4752 (2019)
  * [12] Hutto, C., Gilbert, E.: Vader: A parsimonious rule-based model for sentiment analysis of social media text. In: Proceedings of the International AAAI Conference on Web and Social Media. vol. 8 (2014)
  * [13] Istaiteh, O., Al-Omoush, R., Tedmori, S.: Racist and sexist hate speech detection: Literature review. In: 2020 International Conference on Intelligent Data Science Technologies and Applications (IDSTA). pp. 95–99. IEEE (2020)
  * [14] Jang, J.W., Park, Y.G., Hur, S.I., An, Y.J.: Study on the impact of activity-based flexible office characteristics on the employees’ innovative behavioral intention. In: International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing. pp. 87–103. Springer (2021)
  * [15] Johnman, M., Vanstone, B.J., Gepp, A.: Predicting FTSE 100 returns and volatility using sentiment analysis. Accounting & Finance 58, 253–274 (2018). https://doi.org/10.1111/acfi.12373
  * [16] Koufakou, A., Pamungkas, E.W., Basile, V., Patti, V.: Hurtbert: Incorporating lexical features with bert for the detection of abusive language. In: Proceedings of the Fourth Workshop on Online Abuse and Harms. pp. 34–43 (2020)
  * [17] Lopes, A.R.: The impact of social media on social movements: The new opportunity and mobilizing structure. Journal of Political Science Research 4(1), 1–23 (2014)
  * [18] Lynn, T., Endo, P.T., Rosati, P., Silva, I., Santos, G.L., Ging, D.: A comparison of machine learning approaches for detecting misogynistic speech in urban dictionary. In: 2019 International Conference on Cyber Situational Awareness, Data Analytics And Assessment (Cyber SA). pp. 1–8. IEEE (2019)
  * [19] Montes, M., Rosso, P., Gonzalo, J., Aragón, E., Agerri, R., Álvarez Carmona, M. ., Mellado, E. ., Carrillo-de Albornoz, J., Chiruzzo, L., Freitas, L., Adorno, H.G., Gutiérrez, Y., Zafra, S.M.J., Lima, S., Plaza-de Arco, F.M., Taulé, M.e.: Proceedings of the iberian languages evaluation forum (iberlef 2021). In: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2021). CEUR Workshop Proceedings (2021)
  * [20] Mozafari, M., Farahbakhsh, R., Crespi, N.: A bert-based transfer learning approach for hate speech detection in online social media. In: International Conference on Complex Networks and Their Applications. pp. 928–940. Springer (2019)
  * [21] Mozafari, M., Farahbakhsh, R., Crespi, N.: Hate speech detection and racial bias mitigation in social media based on bert model. PloS one 15(8), e0237861 (2020)
  * [22] Nassirtoussi, A.K., Aghabozorgi, S., Wah, T.Y., Ngo, D.C.L.: Text mining for market prediction : a systematic review. Expert Systems with Applications 41(16), 7653–7670 (2014). https://doi.org/10.1016/j.eswa.2014.06.009
  * [23] Pamungkas, E.W., Basile, V., Patti, V.: Misogyny detection in twitter: a multilingual and cross-domain study. Information Processing & Management 57(6), 102360 (2020)
  * [24] Pamungkas, E.W., Cignarella, A.T., Basile, V., Patti, V., et al.: Automatic identification of misogyny in english and italian tweets at evalita 2018 with a multilingual hate lexicon. In: Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian (EVALITA 2018). vol. 2263, pp. 1–6. CEUR-WS (2018)
  * [25] Pavlopoulos, J., Thain, N., Dixon, L., Androutsopoulos, I.: Convai at semeval-2019 task 6: Offensive language identification and categorization with perspective and bert. In: Proceedings of the 13th international Workshop on Semantic Evaluation. pp. 571–576 (2019)
  * [26] Poletto, F., Basile, V., Sanguinetti, M., Bosco, C., Patti, V.: Resources and benchmark corpora for hate speech detection: a systematic review. Language Resources and Evaluation pp. 1–47 (2020)
  * [27] Rodríguez-Sánchez, F., Carrillo-de Albornoz, J., Plaza, L.: Automatic classification of sexism in social networks: An empirical study on twitter data. IEEE Access 8, 219563–219576 (2020)
  * [28] Rogers, A., Kovaleva, O., Rumshisky, A.: A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics 8, 842–866 (2020)
  * [29] Sharifirad, S., Jacovi, A., Univesity, I.B.I., Matwin, S.: Learning and understanding different categories of sexism using convolutional neural network’s filters. In: Proceedings of the 2019 Workshop on Widening NLP. pp. 21–23 (2019)
  * [30] Sohangir, S., Wang, D., Pomeranets, A., Khoshgoftaar, T.M.: Big data : deep Learning for financial sentiment analysis. Journal of Big Data 5(3), 1–25 (2018). https://doi.org/10.1186/s40537-017-0111-6
  * [31] Sohn, H., Lee, H.: Mc-bert4hate: Hate speech detection using multi-channel bert for different languages and translations. In: 2019 International Conference on Data Mining Workshops (ICDMW). pp. 551–559. IEEE (2019)
  * [32] Wani, M.A., Agarwal, N., Bours, P.: Impact of unreliable content on social media users during covid-19 and stance detection system. Electronics 10(1), 5 (2021)
  * [33] Wolf, T., Chaumond, J., Debut, L., Sanh, V., Delangue, C., Moi, A., Cistac, P., Funtowicz, M., Davison, J., Shleifer, S., et al.: Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pp. 38–45 (2020)
  * [34] Wu, S., Dredze, M.: Beto, bentz, becas: The surprising cross-lingual effectiveness of bert. arXiv preprint arXiv:1904.09077 (2019)
  * [35] Yin, W., Zubiaga, A.: Towards generalisable hate speech detection: a review on obstacles and solutions. arXiv preprint arXiv:2102.08886 (2021)