# Anomaly Detection in Automated Fibre Placement: Learning with Data
Limitations

Assef Ghamisi
Department of Electrical and Computer Engineering
University of Victoria
3800 Finnerty Rd.
Victoria BC, V8P 5C2, Canada
<EMAIL_ADDRESS>Todd Charter††footnotemark:
Department of Electrical and Computer Engineering
University of Victoria
3800 Finnerty Rd.
Victoria BC, V8P 5C2, Canada
<EMAIL_ADDRESS>Li Ji
LlamaZOO Interactive Inc.
1019 Wharf St. #200
Victoria BC, Canada
<EMAIL_ADDRESS>Maxime Rivard
National Research Council Canada
1200 Montreal Road
Ottawa ON, K1A 0R6, Canada
<EMAIL_ADDRESS>Gil Lund
Fives Lund LLC
13536 Beacon Coal Mine Rd S
Seattle WA, 98178, USA
<EMAIL_ADDRESS>Homayoun Najjaran
Department of Mechanical Engineering
University of Victoria
3800 Finnerty Rd.
Victoria BC, V8P 5C2, Canada
<EMAIL_ADDRESS>These authors contributed equally to this work.Corresponding
Author

###### Abstract

Conventional defect detection systems in Automated Fibre Placement (AFP)
typically rely on end-to-end supervised learning, necessitating a substantial
number of labelled defective samples for effective training. However, the
scarcity of such labelled data poses a challenge. To overcome this limitation,
we present a comprehensive framework for defect detection and localization in
Automated Fibre Placement. Our approach combines unsupervised deep learning
and classical computer vision algorithms, eliminating the need for labelled
data or manufacturing defect samples. It efficiently detects various surface
issues while requiring fewer images of composite parts for training. Our
framework employs an innovative sample extraction method leveraging AFP’s
inherent symmetry to expand the dataset. By inputting a depth map of the fibre
layup surface, we extract local samples aligned with each composite strip
(tow). These samples are processed through an autoencoder, trained on normal
samples for precise reconstructions, highlighting anomalies through
reconstruction errors. Aggregated values form an anomaly map for insightful
visualization. The framework employs blob detection on this map to locate
manufacturing defects. The experimental findings reveal that despite training
the autoencoder with a limited number of images, our proposed method exhibits
satisfactory detection accuracy and accurately identifies defect locations.
Our framework demonstrates comparable performance to existing methods, while
also offering the advantage of detecting all types of anomalies without
relying on an extensive labelled dataset of defects.

_Keywords_ Automated Fibre Placement, Anomaly Detection, Computer Vision,
Unsupervised Learning, Convolutional Autoencoder

##  1 Introduction

Automated Fibre Placement (AFP) is an advanced composite manufacturing method
for forming strong and lightweight components from strips of reinforced fibres
known as tows. It is commonly used in quality-critical industries such as
aerospace, where quality inspection and assurance are paramount [1, 2]. Most
existing inspection techniques are implemented via manual human examination
strip by strip, which is time-consuming, and thus a major production
bottleneck [3]. To address this problem, recent research seeks to automate
defect detection by using artificial intelligence (AI), computer vision (CV)
and deep learning (DL) methodologies, reducing manual effort in AFP
inspection, and expediting the production [4].

As a widely adopted methodology, supervised learning with explicit labelling
has been applied for inspection in AFP and similar industries. These methods
leverage convolutional neural networks (CNNs) to train from extensive labelled
datasets of surface imaging of manufactured parts. These surface images take a
variety of forms, including photographs [5], thermal images [6], and depth
maps from many types of profilometry sensors [7, 8].

Sacco et al. (2020) [7] review the applications of machine learning in
composite manufacturing processes and present a case study of state-of-the-art
inspection software for AFP processes. The presented inspection method uses a
deep convolutional neural network for semantic segmentation to classify
defects on a per-pixel basis. They use about 800 scans which is relatively a
large dataset in this domain, yet the results show their method often misses
some defects. Object detection is a well-developed subfield of computer vision
in which models learn to recognize specific objects from a large dataset of
labelled bounding boxes. Zhang et al. (2022) [8] offer an alternate approach
using object detection. This work implements a modified YOLOv5 network, which
is a popular and commercially available object detection model. With a large
dataset of 3000 images containing five different defect types, their proposed
model demonstrated effective performance in detecting those five defect types.

To achieve better real-time inspection, Meister et al. [9] evaluate the use of
convolutional and recurrent neural network architecture for analyzing laser-
scanned surfaces line by line as 1D signals. The different network structures
are assessed on both real and synthetic datasets, demonstrating sufficient
performance. Through experimentation, the authors evaluate the effects of
training and testing on differing data types (real or synthetic), realizing
that deviations between the training and testing domain have a greater
potential to impact the results of their proposed 1D analysis methodology.

These supervised learning methods, however, can be impractical in industrial
projects because they require large, unambiguously labelled training datasets
which are not typically available. There are three key reasons for the lack of
labelled training data. First, collecting real-world data from production
machines is expensive and disruptive to existing production schedules. Second,
defects and anomalies in real-world production are rare. To collect enough
defect and anomaly samples for the models to learn from, one must collect a
very large amount of data, adding to the training cost. Third, real-world
defects can take many different forms, and there is no universally accepted
standard of how a human inspector should delineate and record anomalies and
defects, not to mention how to label them for machine learning [10].
Industrial practitioners of AFP manufacturing typically rely on
organizational-specific standards and individual professional practices to
identify and correct AFP anomalies. These separate standards and practices
cannot be easily translated into a well-defined labelling strategy for
labelling training sets.

One solution is to create synthetic datasets which can be utilized for
training supervised models. Using AI models for synthetic dataset creation has
been explored in many applications with varying success. To address the
limited defect data in AFP, [11] compares different data synthesis techniques
for generating defect data useful to AFP inspection applications. The paper
compares synthetic image datasets generated by various GAN-based models, even
implementing a CNN-based defect classifier for analysis. However, there is a
lack of comparison of the generated datasets to real-world data regarding
image diversity and realism [11]. In a similar problem, for the task of
machine fault detection where faulty data is scarce and normal data is
abundant, [12] implemented a Conditional-GAN for generating fault data in
different conditions from normal data samples. This paper provides a more in-
depth analysis comparing generated data to real ground truth fault data,
showcasing that generated feature distributions are similar to those of real
faults. Such data generation approaches have demonstrated effectiveness,
though they still require sufficiently large and representative datasets, and
cannot generalize to unseen defects or anomalies.

Circumventing the need for large datasets with labelled defects, unsupervised
anomaly detection methods focus on learning the high-level representations of
non-defective, normal data to identify outlying anomalies. In AFP
manufacturing, normal data is typically well-defined thanks to the simple,
invariant structure of the tows (narrow strips of composite material) and
limited layup manners used. In this study, we utilize the non-defective
samples, which constitute the majority of any real-world AFP datasets, to
train a classifier capable of discerning normal and abnormal composite
structures.

Autoencoders, known for their capability of reconstructing input data, have
emerged as potent tools for detecting anomalies within images [13]. An
autoencoder works by learning to encode normal input samples into a lower
dimensional latent vector that can be decoded to reconstruct the original
sample. The reconstruction is compared with the original, and a reconstruction
error metric is calculated. When an abnormal sample is provided, the
reconstruction errors will be high since the autoencoder was never trained
with similar images. A threshold is then applied to the reconstruction errors
to determine if the sample is normal, with a higher error indicating the
sample is more likely to be abnormal or defective.

There is a lack of research applying these methods in the AFP industry,
however, the approach has demonstrated success in other similar defect
detection tasks. Ulger et al. (2021) [14] employ convolutional and variational
autoencoders (CAE and VAE) for solder joint defect detection. Reconstruction
errors guide classification, applying a threshold to differentiate between
normal and abnormal inputs. Tsai et al. (2021) [15] also employ CAE and VAE
for textured surface defect detection, favouring CAE in a Receiver Operator
Characteristic (ROC) analysis. The proposed anomaly detector is tested on
various textured and patterned surface types, including wood, liquid crystal
displays, and fibreglass. Additionally, Chow et al. (2020) [16] implement a
CAE for concrete defect detection, introducing a window-based approach for
high-resolution images. Their window-based implementation enables pixel-wise
anomaly maps to provide localization and contextual understanding of
anomalies.

We propose a comprehensive framework for anomaly detection in AFP based on the
autoencoder methodology. Compared to the existing methods that require a large
labelled dataset including manufacturing defects, our approach is compatible
with a small training dataset of normal samples. Our autoencoder-based anomaly
detector uses data collected from the AFP setup shown in Figure 1(a). The
autoencoder is trained on a collection of local samples taken from depth
images of the composite carbon fibre surfaces. The depth images are obtained
with an Optical Coherence Tomography (OCT) sensor installed on the AFP layup
head, which captures high-resolution point clouds of the layup tow surfaces
[17, 18]. A picture of the AFP head with the OCT sensor installed on it is
provided in Figure 1(b).

To simplify and enhance efficiency in processing information, the 3D point
clouds are converted into 2D depth maps. Since the point cloud is a
measurement of the surface elevation, it can be projected onto a 2D depth map
without the loss of information. These depth maps are presented as grayscale
images, where the brightness of each pixel corresponds to the surface
elevation on the composite part. Fig. 2 provides various representations of a
sample composite part, highlighting that defects are less discernible in the
photograph due to reflections and low visual contrast. However, defects become
more evident in the depth map, facilitating defect detection.

(a)

(b)

Figure 1: The industrial AFP setup is shown in two separate views. On the left
is an overall view of the fibre placement machine (a), and on the right is a
close-up shot of the robotic tool applying carbon fibre tows. The OCT sensor
is visible to the upper left of the roller.

(a)

(b)

(c)

Figure 2: Different representations of a composite part manufactured with an
AFP machine. Above (a) is a 3D point cloud measured using OCT Technology. The
bottom left (b) shows the depth map generated from the 3D point cloud, and a
real photograph of the same composite part is shown in the bottom right (c)
for comparison.

The work offers the following contributions.

  1. 1.

We introduce a novel, end-to-end framework for anomaly detection and
localization in Automated Fibre Placement, circumventing the data limitations
in this industry. Our proposed framework has several advantages compared to
the existing methods. It detects all types of anomalies, without the need for
manual data labeling or defect samples. Also, it works with a small number of
composite images.

  2. 2.

Another major contribution of this work is an efficient data extraction
methodology that can convert a limited number of composite images into a large
dataset of local samples. Utilizing classical computer vision algorithms to
detect the boundaries of composite tapes, this method generates a dataset that
exploits the inherent symmetry of AFP composite materials.

  3. 3.

We design and validate an autoencoder with the optimal size of the latent
domain that can identify the best distinctive features to differentiate
between normal and defective samples.

  4. 4.

The proposed framework generates a map representing the local anomaly score of
the AFP-manufacture parts and visualizes this map on the original composite
scan. This visual representation serves as a valuable tool for AFP
technicians, aiding them in the identification and resolution of anomalies
within the composite structure.

The organization of this manuscript is as follows: Section 2 describes, the
whole procedure of anomaly detection, including data prepossessing, training
of the AI model, and implementation details. In Section 3, the evaluation
results of both the anomaly detector and the localization system are
presented. Finally, concluding remarks of this work are provided in Section 4.

##  2 Methodology

Figure 3 summarizes our anomaly detection framework. First, a composite scan
is processed and local samples are extracted from the images. Then, the
trained autoencoder generates an anomaly map, used to detect and locate the
defects in the image.

Figure 3: An overview of the defect detection process shows the necessary
steps.

###  2.1 Data Preparation

The raw depth maps contain impulse artifacts, also known as salt and pepper
noise, which can be detrimental. To remove the noise, we used a median filter
with the kernel size $3\times 3$ applied to the whole depth image [19].
Compared to a Gaussian filter with a small radius (a low-pass filter) [20], a
median filter has less risk of losing high-frequency features.

Another data preparation step is needed because different raw depth-map images
have inconsistent ranges of values depending on the distance of the laser
origin to the composite surface. This effect is commonly caused when the OCT
sensor is mounted to a fixed location behind the AFP head while scanning a
contoured surface. These variations can cause undesired behaviour in the
defect detection methodology. To address this, all the images undergo a min-
max normalization so that the minimum depth value is mapped to zero and the
maximum value is mapped to one. By applying this linear transformation, the
visual contrast of the images is improved while keeping the original depth
ratio. The normalization function is provided in Eq. 1 in which $z_{i,j}$ is
the original depth value, $p_{i,j}$ is the normalized pixel value, and $Z$ is
the whole depth map matrix.

$p_{i,j}=\frac{z_{i,j}-\min(Z)}{\max(Z)-\min(Z)}$ (1)

###  2.2 Local Sample Extraction

The training dataset used in this work is composed of depth maps from 42 non-
defective composite surfaces. Developing an effective end-to-end network for
defect detection using such a limited number of scans presents a significant
challenge. However, we address this issue by leveraging the consistent
uniformity along the composite tows and extracting cropped windows of the
scans to form a dataset with many more localized samples. This is possible
under the assumption that each cropped section conforms to a similar
distribution, given that defect-free tows should exhibit minimal to no
disparity across their segments. Consequently, the analysis of smaller regions
allows us to employ a more compact neural network to learn from a broader
spectrum of local samples rather than relying on a larger network to process
full scans. Moreover, by extracting localized samples along the tows, our
network gains exposure to a greater variation of tow structure.

One of the most basic methods to detect local objects in an image is to move a
window on the image and classify the smaller region inside the window [21].
This method is known as sliding window in computer vision literature and has
its own limitations. For example, the scale of the object may vary depending
on how close the object is to the camera. Consequently, multiple sizes of
sliding windows are required which can be computationally expensive. In our
current dataset, on the other hand, most of the defects are localized to one
tow, and therefore approximately the same relative size and there is no wide
variation in perspective or orientation of the objects. Consequently, only one
scale of sliding windows is sufficient for this use case. This also helps to
keep computation complexity relatively low for this approach. Besides, there
is preliminary knowledge of the composite part scans, like the number of tows
and the general direction they follow. This enables a customized sliding
window method that makes use of the known information. Moreover, the depth
maps generated from our OCT scans have a specific structure. For example, all
tows are placed straight and horizontal in the images, the number of tows is
known, and their width is also known. To incorporate this predetermined
knowledge, a line detector algorithm based on Hough Transform [22, 23] detects
the vertical and horizontal edges of the tows. After detecting the boundaries
of the tows, the center of them (centerlines) are calculated by averaging each
two consecutive horizontal lines, bounded within the detected vertical lines.
This process of centerline detection is illustrated in Figure 4. Finding the
center of the tows makes it possible to directly focus on the regions that are
candidates for defect instead of scanning the whole image. In other words, it
creates a skeleton that directs and constrains the region of interest. This
can reduce additional effort on the classifier side.

(a)

(b)

Figure 4: The centerline detection procedure contains two main steps:
detecting horizontal and vertical lines (a) and estimating tow centerlines
from the detected lines (b).

Based on the detected centerlines, a square window slides across the tows to
extract cropped regions. We select a window size of $32\times 32$ pixels to
cover approximately $1.5$ times the width of composite tapes. In this
implementation, a stride of $8$ pixels is used to move the window and sample
the information cropped inside. This combination of window size and stride
allows enough overlap between the nearby samples while keeping the samples
sufficiently distinctive. Some of the extracted samples are presented in
Figure 5. At the time of inference, each window that is detected as an anomaly
is considered to contain a defect, while the windows that are not anomalies
are assumed to have normal tow structures. The next sections explain the
approach to distinguishing between normal and abnormal samples.

Figure 5: A dataset is created from cropped sections of the depth maps, using
the sliding window method. Normal samples are shown on top and abnormal
samples are shown below.

###  2.3 Anomaly Detection

As mentioned in Section 1, autoencoders have shown great success at
identifying anomalies in images. An autoencoder is an unsupervised learning
model that reconstructs the given input by learning to minimize the error
between the input and reconstructed output. They do this by encoding the input
to a vector of latent features, also known as the bottleneck, and then
decoding those latent features to reconstruct the input. Convolutional
Autoencoders (CAEs) are a group of autoencoders that use convolution layers in
their network structure. Convolutional neural networks (CNN) are more popular
for image-based autoencoders than basic fully-connected networks. This is
because CNNs incorporate receptive fields using kernels that maintain the
spatial relationships of the data. CNNs are also computationally efficient
with sparse connectivity of neurons.

If only normal samples are used to train the autoencoder, it will be able to
reconstruct similar normal samples accurately, and the reconstruction results
for abnormal samples will be poor. Therefore, reconstruction error can be used
as an indicator of how anomalous each input is. For inference, each cropped
window of a composite material depth map is fed into the trained autoencoder.
The reconstruction error of each window is then used as an anomaly score to
create an anomaly map for the entire image. Reconstruction error of a window
centred at $(x,y)$ is calculated using Eq. 2 in which $p_{i,j}$ and
$\hat{p_{i,j}}$ are the pixel value of the input and reconstructed output,
respectively, and $b$ is half of the size of each window.

$m_{x,y}=\frac{1}{(2b)^{2}}\sum_{i=x-b}^{x+b-1}\sum_{j=y-b}^{y+b-1}[p_{i,j}-\hat{p}_{i,j}]^{2}$
(2)

In this work, a CAE is designed and used as the anomaly detector. The design
incorporates symmetric encoder and decoder structures shown in figures 6(a)
and 6(b) respectively. For training the model, mean squared error is employed
as the loss function. Although the full method uses continuous valued anomaly
maps to identify the defects rather than a binary prediction, binary
classification can still be useful in validating the model performance. For
this, a threshold parameter is introduced to classify samples based on their
reconstruction error. To select the threshold value, a Receiver Operating
Characteristic (ROC) curve is applied. The ROC curve plots the true positive
rate against the false positive rate while varying the threshold value. In an
ideal case, the selected threshold would give a true positive rate of 1 and a
false positive rate of 0. In the ROC plot, this corresponds to the upper left
corner and hence the best threshold value is selected from the curve at the
point closest to that corner.

(a) Encoder

(b) Decoder

Figure 6: A graphic depicting the network structure of the proposed
autoencoder. Above is the encoder structure (a), and below is the decoder
structure (b).

###  2.4 Defect Localization

The anomaly detection generates an array of anomaly scores for each tow, which
can be considered as a 1D digital signal. Any area of this signal with a
concentration of high values indicates the presence of a defect. In Computer
Vision literature, these areas are called blobs [24, 25]. For detecting the
blobs, we use the Difference of Gaussian (DoG) method [26]. In this approach,
the signal ($f(x)$) is filtered using Gaussian kernels with increasing values
for standard deviations ($\sigma$) as described in Eq. 3. Then, the
subtractions of each two successively filtered signals are calculated. The
local maxima of $g(x,\sigma)$ represents the blobs. In such maxima points, $x$
and $\sigma$ correspond to the location and characteristic scale (size) of the
blob, respectively.

$g(\sigma,x)={\sigma}^{2}\frac{\partial n_{\sigma}^{2}}{\partial x^{2}}\ast
f(x)$ (3)

For each defect, two parameters are detected, radius and center. With this
information the detected blobs can be transferred from anomaly map to image
space.

##  3 Results and Discussion

The performance of the anomaly detection and localization system depends on
two factors. First is the number of samples the anomaly detector is correctly
classifying as normal or abnormal. Second is the size and location accuracy of
the predicted defects. This section evaluates these two aspects using a test
dataset with an additional two composite surfaces containing defects.

###  3.1 Anomaly Detection

The network structure proposed in Figure 6 is implemented using three
different latent dimensions 2, 16, and 128 for comparison. Each network is
trained with a dataset consisting of 27406 only normal samples. An Adam
optimizer is employed with an MSE loss function to train the network. The
batch size is set to 128. Each autoencoder undergoes training for 50 epochs,
completing in under 5 minutes on a computer with the following specifications:

  * •

Processor (CPU): Intel(R) Xeon(R) E5-1607 v4 @ 3.10GHz

  * •

Graphics (GPU): NVIDIA GeForce GTX 1080

  * •

Memory (RAM): 32.0 GB

The curves in Figure 7 demonstrate the training losses of each autoencoder.
Comparing the training loss curves shows that the models’ reconstruction
ability improves with higher dimensional latent space. The curves also show
that the models are learning relatively quickly with tiny improvement in the
later epochs.

Figure 7: The training MSE losses of the three autoencoders are plotted in
comparison over 50 epochs.

Reconstruction results for the autoencoders are demonstrated in Figure 8. The
original samples are randomly selected from normal and abnormal classes in
test set. These results clearly show the improved reconstruction performance
with a higher dimensional latent space. It shows that the autoencoder with a
128-dimensional latent vector is able to produce good reconstructions for both
normal and abnormal samples. The 16-dimensional autoencoder, on the other
hand, produces relatively good reconstructions for normal samples and poorer
reconstructions of defect samples. This is ideal for the classification method
to distinguish anomalies. Finally, the autoencoder with only a 2-dimensional
latent space is unable to make good reconstructions for any of the input
samples.

Figure 8: The resulting reconstructions from the autoencoders with various
latent sizes are compared for both normal and abnormal test samples.

Figure 9 shows comparisons of the reconstruction error. In figure 9(a) the
distributions of mean square error are shown for the 16-dimensional
autoencoder on the training set and test set. Note that the training set only
includes normal samples whereas the test set contains both normal and abnormal
samples, separated accordingly. As the figure suggests, the normal samples
have a similar distribution in both the training and test sets. On the other
hand, the abnormal samples have a generally higher MSE with a slight overlap
on normal sample distribution. In an ideal case, if there were no overlap
between these two distributions we could find a perfect threshold as the
decision boundary to classify the samples into normal and abnormal categories.
With existing overlap, however, an ROC curve can help to select the decision
boundary that makes the best trade-off between true and false positive rates.
Taking a closer look at the difference between the three autoencoders, Figure
9(b) shows boxplots of the MSE for normal and abnormal samples in the test
set, using different number of latent features. Here it shows how separable
the two classes are based on reconstruction error alone. For the 2-dimensional
autoencoder, the interquartile ranges are separable, but there is a
significant overlap when considering the whiskers. For the 16-dimensional
autoencoder, the separation is greatly improved with minimal overlap between
the whiskers. The 128-dimensional autoencoder, however, does not show
significant separation and would be impossible to accurately classify the two
classes based on MSE alone.

(a) Comparing Test and Training

(b) Comparing Different Latent Sizes

Figure 9: Distributions of MSE for different input types and latent sizes are
presented for comparison.

Figure 10 shows the ROC curves for each of the three autoencoders and the
selected best threshold points shown as stars. These results further
demonstrate that classification performance does not correspond with
reconstruction performance, as the 2D and 128D autoencoders’ ROC curves have
worse classification performance than the 16D model. The best-performing model
is the autoencoder with a 16-dimensional latent space, achieving a high true
positive rate with a low false positive rate.

Figure 10: ROC curves of the test set are plotted for the three autoencoder
classifiers.

Classification results of each autoencoder with the selected threshold values
found by the ROC curves are summarised in Table 1. The table reports
precision, recall, F1 score, area under the ROC curve (AUC), and selected
threshold for each classifier.

Table 1: Test Metrics of Convolutional Autoencoder Latent Dim. | AUC | Best Threshold | Precision | Recall | F1-Score
---|---|---|---|---|---
128 | 0.978 | 3.86 | 0.918 | 0.929 | 0.923
16 | 0.999 | 13.69 | 0.992 | 0.976 | 0.984
2 | 0.942 | 42.08 | 0.922 | 0.810 | 0.862

Figure 11 shows the effect of the latent vector size on the performance of the
model. Figure 11(a) suggests that larger latent dimensions will produce lower
reconstruction errors. However, an accurate classification model does not
require the lowest reconstruction error, but a moderate reconstruction that
leads to better classification performance. In Figure 11(b) the best latent
dim is found by calculating the minimum AUC of the ROC curve while varying the
latent dim.

(a) MSE

(b) AUC of ROC

Figure 11: MSE and AUC of ROC are plotted for autoencoder models with latent
dimensions varied from 1 to 128.

Figure 12 shows the classification confusion matrix while using the optimal
latent size. The off-diagonal values in this matrix are low which shows that
most of the samples from both normal and abnormal classes are classified
correctly.

Figure 12: The confusion matrix demonstrates the evaluation of the best model.

###  3.2 Defect Localization

Figure 13 illustrates the results of the anomaly detector on a 2D depth map.
The colour of each point indicates the normalized MSE for reconstructing a
small window around the point with the anomaly detector. As can be seen, the
defective areas have a large density of points with a higher MSE. This
information can be used to detect these areas while ignoring individual
outlier values.

Figure 13: An anomaly map is generated from the MSE of individual cropped
windows.

In Figure 14 the process of detecting the defects from the anomaly map is
illustrated. The elevation in each curve represents the MSE value for one tow
(represented by colour in the previous figure). The arrows show the detected
blobs after applying the Derivative of Gaussians method. It is observed that
only the areas with an extended length of high MSE values are detected as
blobs.

Figure 14: Anomaly scores are visualized as 1D signals for blob detection.

Figure 15 shows the final output of the computer vision pipeline, comparing
the annotated defect bounding boxes (ground truth) with the predicted bounding
boxes.

Figure 15: Predicted bounding boxes are displayed on the original depth map in
comparison with the ground truth bounding boxes. Also, the values of
Intersection over Union are displayed.

###  3.3 Qualitative Comparison

The proposed framework is unique and to the best of our knowledge, no other
studies have implemented an end-to-end unsupervised defect detection method
for AFP inspection. Unfortunately, there are no publicly available datasets in
this domain to serve as a benchmark for AFP inspection tasks. Additionally,
the dataset used in this work is insufficient for training supervised learning
models, which constitute the majority of current studies in this field. For
these reasons, an explicit quantitative comparison of our method with other
state-of-the-art approaches is not possible. However, a qualitative comparison
of the most relevant studies is presented in Table 2, showcasing the
advantages of our framework.

In regard to other defect detection methods, the main advantages of our
proposed approach stem from the unsupervised learning process which enables
learning with data limitations. Foremost, our method detects all types of
surface anomalies in AFP, whereas existing methods are limited to specific
defect types. Additionally, unlike other methods, ours does not require
labelling which is time-consuming and prone to errors. Moreover, our proposed
framework works with fewer composite scans and it does not need any samples of
defects.

Some methods implement semantic segmentation which requires explicit pixel-
wise labelling. This is not necessarily needed, as our method provides
sufficient localization with bounding boxes and anomaly maps. Unlike other
methods, ours does not classify the defects, however, a separate
classification module could easily be integrated using the detected bounding
boxes. Besides, in an industry where the majority of the inspected parts are
non-defective, directly detecting the defective parts can reduce most of the
effort.

Table 2: A Comparison of Existing Learning-based AFP Defect Detection Methods with the Proposed Framework Reference |  Detection |  Classification |  Localization |  Segmentation |  Detects All Abnormalities |  Labelling not Required |  Defects not Required |  Dataset Size | Results
---|---|---|---|---|---|---|---|---|---
Classification | Localization
Meister and Wermes (2023) [9] | ✓ | ✓ | - | - | - | - | - | 469 | Acc1: 95.12% | -
Zhang et al. (2022) [8] | ✓ | ✓ | ✓ | - | - | - | - | 3,000 | - | mAP4: 93.1%
Tang et al. (2022) [27] | ✓ | ✓ | ✓ | ✓ | - | - | - | 43 | Pr2: 84.4%, Re3: 77.1% | mIoU5: 0.776
Sacco et al. (2020) [7] | ✓ | ✓ | ✓ | ✓ | - | - | - | 800 | Average Acc: 49.1% | -
Schmidt et al. (2019) [6] | ✓ | ✓ | - | - | - | - | - | 12,000 | Acc: ¿92% | -
This work | ✓ | - | ✓ | - | ✓ | ✓ | ✓ | 44 | Acc: 98.7% | mIoU: 0.708
1: Accuracy, 2: Precision, 3: Recall, 4: mean Average Precision, 5: mean
Intersection over Union

##  4 Conclusions

This paper introduces a practical and novel method for the inspection of
composite materials manufactured by Automated Fibre Placement (AFP). The AFP
process is susceptible to various types of defects which can significantly
impact the final product’s quality, necessitating thorough inspection of the
composite parts. Manual human inspection has traditionally been employed for
this purpose, but it is time-consuming, labour-intensive, and prone to human
errors. To enhance the efficiency, accuracy and reliability of AFP, the
development of an automated inspection system is crucial. Current inspection
procedures mostly utilize profilometry technologies like laser scanning,
thermal imaging, and optical sensors to generate visual measurements of the
part’s surface. The data used in this work is obtained from a laser scanner
that operates based on OCT technology, though the framework presented is
general and can be adapted to other types of profilometry data.

In AFP inspection, robust and generalized supervised learning methods are
infeasible due to limitations in available labelled data. Anomaly detection
methods, on the other hand, can circumvent this challenge by focusing on
learning the structure of normal samples to identify any abnormalities. The
proposed computer vision framework detects individual tows in AFP composites
and creates a dataset of sub-images by sliding a window along the center of
each tow. The extracted data is then used to train an autoencoder designed to
detect anomalies. Using the same sliding window procedure, the autoencoder
produces anomaly scores for local regions of the composite part. These scores
are aggregated to form an anomaly map of the full image. This anomaly map can
then be used as an explicit indication tool by an operator. We further process
this anomaly map using a 1D blob detection algorithm to generate bounding
boxes around defects.

Compared to other state-of-the-art automated inspection methods in AFP, our
approach offers several advantages. Since the autoencoder learns the inherent
structure of normal tows, it is capable of detecting all anomalies, unlike
other methods which can only detect defects specific to their training data.
Furthermore, our suggested framework operates with fewer composite scans and
eliminates the requirement for defect samples. For verification purposes the
autoencoder is evaluated in a classification task, achieving over 98%
classification accuracy. Additionally, the overall framework is implemented on
a set of test samples where bounding boxes generated by the method achieve an
Intersection over Union (IoU) of 0.708. This demonstrates sufficient accuracy
in the localization of detected defects.

This paper outlines a novel defect detection approach for AFP and emphasizes
its practicality, particularly in addressing data limitations. There are
several potential directions for future research to enhance the inspection
system’s capabilities and extend its relevance to broader domains. To improve
dataset quality and quantity, we suggest investigating data engineering
techniques like data augmentation and synthetic data generation. Additionally,
while the current system identifies anomalies, it lacks the ability to
classify specific defect types. To address this, we recommend utilizing the
generated bounding boxes to collect training data for developing a
classification model. We also recommend adapting our framework for use in
industries that share similar tape-by-tape structures. This can lead to
enhancements in defect identification and quality assurance across different
sectors.

## Statements and Declarations

### Acknowledgements

We would like to acknowledge the financial support of LlamaZOO Interactive
Inc. and Natural Sciences and Engineering Research Council (NSERC) Canada
under the Alliance Grant ALLRP 567583 - 21. In addition, we would like to
recognize the research collaboration of LlamaZOO Interactive Inc., the
National Research Council of Canada (NRC), and Fives Lund LLC.

### Conflict of Interest

The authors declare that they have no conflicting interests or financial
motives related to the work in this article.

### Data Availability

The data generated and analyzed in this paper is proprietary and due to its
commercialization potential, is not made publicly available. The information
is protected to maintain its commercial value. We regret any inconvenience
this may cause and appreciate your understanding of the importance of
preserving its proprietary nature.

## References

  * [1] M. Palardy-Sim, M. Rivard, S. Roy, G. Lamouche, C. Padioleau, A. Beauchesne, L.-G. Dicaire, D. Lévesque, J. Boisvert, M.-A. Octeau, and others, “Next Generation Inspection Solution for Automated Fibre Placement,” in The Fourth International Symposium on Automated Composites Manufacturing, p. 64, 2019.
  * [2] B. Böckl, A. Wedel, A. Misik, and K. Drechsler, “Effects of defects in automated fiber placement laminates and its correlation to automated optical inspection results,” Journal of Reinforced Plastics and Composites, vol. 42, pp. 3–16, Jan. 2023.  Publisher: SAGE Publications Ltd STM.
  * [3] S. Meister, M. A. M. Wermes, J. Stüve, and R. M. Groves, “Review of image segmentation techniques for layup defect detection in the Automated Fiber Placement process,” Journal of Intelligent Manufacturing, vol. 32, pp. 2099–2119, Dec. 2021.
  * [4] A. Brasington, C. Sacco, J. Halbritter, R. Wehbe, and R. Harik, “Automated fiber placement: A review of history, current technologies, and future paths forward,” Composites Part C: Open Access, vol. 6, p. 100182, Oct. 2021.
  * [5] M. Zemzemoglu and M. Unel, “Design and Implementation of a Vision Based In-Situ Defect Detection System of Automated Fiber Placement Process,” in 2022 IEEE 20th International Conference on Industrial Informatics (INDIN), (Perth, Australia), pp. 393–398, IEEE, July 2022.
  * [6] C. Schmidt, T. Hocke, and B. Denkena, “Deep learning-based classification of production defects in automated-fiber-placement processes,” Production Engineering, vol. 13, pp. 501–509, June 2019.
  * [7] C. Sacco, A. Baz Radwan, A. Anderson, R. Harik, and E. Gregory, “Machine learning in composites manufacturing: A case study of Automated Fiber Placement inspection,” Composite Structures, vol. 250, p. 112514, Oct. 2020.
  * [8] Y. Zhang, W. Wang, Q. Liu, Z. Guo, and Y. Ji, “Research on Defect Detection in Automated Fiber Placement Processes Based on a Multi-Scale Detector,” Electronics, vol. 11, p. 3757, Jan. 2022.  Number: 22 Publisher: Multidisciplinary Digital Publishing Institute.
  * [9] S. Meister and M. Wermes, “Performance evaluation of CNN and R-CNN based line by line analysis algorithms for fibre placement defect classification,” Production Engineering, vol. 17, pp. 391–406, June 2023.
  * [10] F. Heinecke and C. Willberg, “Manufacturing-Induced Imperfections in Composite Parts Manufactured via Automated Fiber Placement,” Journal of Composites Science, vol. 3, p. 56, June 2019.  Number: 2 Publisher: Multidisciplinary Digital Publishing Institute.
  * [11] S. Meister, N. Möller, J. Stüve, and R. M. Groves, “Synthetic image data augmentation for fibre layup inspection processes: Techniques to enhance the data set,” Journal of Intelligent Manufacturing, vol. 32, pp. 1767–1789, Aug. 2021.
  * [12] M. Ahang, M. Jalayer, A. Shojaeinasab, O. Ogunfowora, T. Charter, and H. Najjaran, “Synthesizing Rolling Bearing Fault Samples in New Conditions: A Framework Based on a Modified CGAN,” Sensors, vol. 22, no. 14, 2022.
  * [13] J. E. D. Albuquerque Filho, L. C. P. Brandão, B. J. T. Fernandes, and A. M. A. Maciel, “A Review of Neural Networks for Anomaly Detection,” IEEE Access, vol. 10, pp. 112342–112367, 2022.  Conference Name: IEEE Access.
  * [14] F. Ulger, S. E. Yuksel, and A. Yilmaz, “Anomaly Detection for Solder Joints Using $\beta$-VAE,” IEEE Transactions on Components, Packaging and Manufacturing Technology, vol. 11, pp. 2214–2221, Dec. 2021.  Conference Name: IEEE Transactions on Components, Packaging and Manufacturing Technology.
  * [15] D.-M. Tsai and P.-H. Jen, “Autoencoder-based anomaly detection for surface defect inspection,” Advanced Engineering Informatics, vol. 48, p. 101272, Apr. 2021.
  * [16] J. K. Chow, Z. Su, J. Wu, P. S. Tan, X. Mao, and Y. H. Wang, “Anomaly detection of defects on concrete structures with the convolutional autoencoder,” Advanced Engineering Informatics, vol. 45, p. 101105, Aug. 2020.
  * [17] M. Rivard, M. Palardy-Sim, G. Lamouche, S. Roy, C. Padioleau, A. Beauchesne, D. Levesque, F. Boismenu, L. G. Dicaire, J. Boisvert, S. Peters, J. Chen, M. A. Octeau, J. B. Robles, F. Ferland, M. Tanguay, J. Hissett, D. Swope, S. Albers, R. Harper, K. Wright, B. Buhrkuhl, M. Klakken, G. Lund, and A. Yousefpour, “Enabling responsive real-time inspection of the automated fiber placement process,” International SAMPE Technical Conference, vol. 2020-June, 2020.
  * [18] M. Palardy-Sim, M. Rivard, G. Lamouche, S. Roy, C. Padioleau, A. Beauchesne, D. Levesque, L.-G. Dicaire, J. Boisvert, S. Peters, and others, “Advances in a Next Generation Measurement & Inspection System for Automated Fibre Placement,” in Proceedings of the Manufacturing & Processing Technologies Conference in the Composites and Advanced Materials Expo (CAMX), Anaheim, CA, USA, pp. 23–26, 2019\.
  * [19] J. Azzeh, B. Zahran, and Z. Alqadi, “Salt and pepper noise: Effects and removal,” JOIV: International Journal on Informatics Visualization, vol. 2, no. 4, pp. 252–256, 2018.
  * [20] X. Deng, Y. Ma, and M. Dong, “A new adaptive filtering method for removing salt and pepper noise based on multilayered PCNN,” Pattern Recognition Letters, vol. 79, pp. 8–17, Aug. 2016.
  * [21] N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 1, pp. 886–893 vol. 1, 2005.
  * [22] P. V. Hough, “Machine analysis of bubble chamber pictures,” in Proc. of the International Conference on High Energy Accelerators and Instrumentation, Sept. 1959, pp. 554–556, 1959.
  * [23] R. O. Duda and P. E. Hart, “Use of the Hough transformation to detect lines and curves in pictures,” Communications of the ACM, vol. 15, pp. 11–15, Jan. 1972.
  * [24] A. Danker and A. Rosenfeld, “Blob Detection by Relaxation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. PAMI-3, pp. 79–92, 1981.
  * [25] H. Kong, H. C. Akakin, and S. E. Sarma, “A generalized Laplacian of Gaussian filter for blob detection and its applications,” IEEE transactions on cybernetics, vol. 43, no. 6, pp. 1719–1733, 2013.  Publisher: IEEE.
  * [26] D. G. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints,” International Journal of Computer Vision 2004 60:2, vol. 60, pp. 91–110, Nov. 2004.  Publisher: Springer.
  * [27] Y. Tang, Q. Wang, L. Cheng, J. Li, and Y. Ke, “An in-process inspection method integrating deep learning and classical algorithm for automated fiber placement,” Composite Structures, vol. 300, p. 116051, 2022.


# Change Summarization of Diachronic Scholarly Paper Collections by Semantic
Evolution Analysis

Naman Paharia  IIT Kharagpur
Kharagpur, India
<EMAIL_ADDRESS>Muhammad Syafiq Mohd Pozi  SOC, Universiti Utara
Malaysia
<EMAIL_ADDRESS>IIR4.0, Universiti Kebangsaan Malaysia Adam Jatowt
University of Innsbruck
Innsbruck, Tirol, Austria
<EMAIL_ADDRESS>
###### Abstract

The amount of scholarly data has been increasing dramatically over the last
years. For newcomers to a particular science domain (e.g., IR, physics, NLP)
it is often difficult to spot larger trends and to position the latest
research in the context of prior scientific achievements and breakthroughs.
Similarly, researchers in the history of science are interested in tools that
allow them to analyze and visualize changes in particular scientific domains.
Temporal summarization and related methods should be then useful for making
sense of large volumes of scientific discourse data aggregated over time. We
demonstrate a novel approach to analyze the collections of research papers
published over longer time periods to provide a high level overview of
important semantic changes that occurred over the progress of time. Our
approach is based on comparing word semantic representations over time and
aims to support users in better understanding of large domain-focused archives
of scholarly publications. As an example dataset we use the ACL Anthology
Reference Corpus that spans from 1979 to 2015 and contains 22,878 scholarly
articles.

###### Index Terms:

temporal mining, summarization, visualization

##  I Introduction

In this paper, the problem we focus on is how to analyze and visualize
evolution of scholarly research based on large corpora such as one consisting
of computer science scholarly publications, over a period of time. As such
datasets typically cover tens of thousands or more articles it is difficult to
understand important changes that occurred in the target scientific domain
over time. We propose a novel change-oriented summarization approach based on
word semantic evolution to provide information on key changes that occurred in
the data over time. As an underlying data source we use the Association for
Computational Linguistics (ACL) dataset, and specifically, the Parcit
Structured XML111https://acl-arc.comp.nus.edu.sg/archives/acl-
arc-160301-parscit version of the ACL Anthology Reference Corpus [2]. By
analyzing this data we provide answers to questions about what concepts are
highly changing and what are their directions of changes.

A particular research direction that is currently gaining interest in the text
summarization field is how to incorporate time factor into automatic text
summarization models. Examples can been seen in news [13, 19, 16], social
networks [10], and scholarly articles [12]. Several strategies have been
proposed to understand the significance of individual topics [18], to learn
how multidisciplinary research area is founded [17], or to illustrate the
evolution of scientific communities [4]. A common approach is by frequency
analysis [7], or using topic modelling, so that the correlations between link
citations or the sets of co-occurring words are uncovered [5, 6]. The problem
with those quantitative approaches however, is that, the semantic perspective
is usually not considered explicitly during the modelling task. Recently a
common way of identifying whether a word has changed its meaning over time is
by measuring how much its embedding vector has changed through tracking shifts
in the word’s distributional neighborhood [11, 14, 9, 15]. However, the
diachronic word embedding approaches typically focus on individual words
aiming to characterize their evolution. On the other hand, in this research,
we aim for a collection-level approach which would allow understanding major
changes in the entire document collection over time.

##  II Method

###  II-A Data Collection and Preprocessing

The ACL Anthology Reference Corpus consists of conference and journal papers
in natural language processing and computational linguistics research domains
which are included within ACL Anthology. The corpus contains 22,878 papers
published during a span of 36 years from 1979 to 2015. The distribution of the
number of research papers per year is non-uniform, with increasing numbers of
papers in the recent years. In this work, we use the body text and the year of
the publication date of each paper to analyse how specific terminology drifted
from 1979 to 2015.

The dataset was divided into 5 non-overlapping time frames: 1979-1995,
1996-2000, 2001-2005, 2006-2010, and 2011-2015. The skewness in data
distribution was considered while making this divisions so the time frames
have more or less similar number of documents. A vocabulary, $V$, was created
using the complete dataset after filtering it for stopwords and numbers and
extracting nouns and noun phrases. The vocabulary was then filtered to remove
words with frequency smaller than 100 as well as to remove words which
occurred only in a single timeframe since we focus on the changes of high
significance. Finally, words with length less than 3 were discarded.

###  II-B Text Representation

For representing terms, we used a BERT-based model, called SciBERT [1] with 12
attention layers and the hidden layer of size 768 which was pretrained on
1.14M papers from Semantic Scholar222https://www.semanticscholar.org/
resulting in a corpus size of 3.17B tokens. SciBERT was then fine-tuned on the
entire ACL corpus to represent semantic meanings of words in each time frame.

###  II-C Representing Semantic Word Change

Analysing topical changes in scientific corpora as well as detecting emerging
trends have been traditionally done by frequency analysis or topic modelling.
A different approach, which we propose, is to compare semantic meanings of
words, using the contextualized embeddings. Comparing average embeddings of
the same word in different time frames will give a representation of the
change of a word. In particular, the cosine similarity between the average
embedding of a target word in the first time frame (1979-1995) and the average
embeddings of this word in the last time frame (2011-2015) will represent the
overall semantic change for the word such that the lower the similarity value,
the higher the semantic change expected. Tab. I shows words with the lowest
and the highest similarity values in our dataset.

TABLE I: Terms with the smallest and highest cosine similarity between the last and first time unit. Terms | Smallest Similarity | Terms | Highest Similarity
---|---|---|---
seventh | 0.6420 | must | 0.9975
fell | 0.6728 | example | 0.9973
plan | 0.7319 | words | 0.9972
mann | 0.7648 | corpus | 0.9970
entail | 0.7849 | node | 0.9970
west | 0.7982 | japanese | 0.9969
copyright | 0.7996 | probability | 0.9969
maxim | 0.8000 | terminal | 0.9968
prop | 0.8143 | threshold | 0.9968
jack | 0.8190 | complex | 0.9966
stanford | 0.8212 | automaton | 0.9966

Table 1 suggests that cosine similarity alone on the level of individual terms
does not suffice for summarizing semantic change in the entire corpus. The
results for high cosine similarity are rather intuitive, but the results
corresponding to the low values of similarity tend to deviate from our
expectation. These words underwent largest semantic movement yet they seem to
carry little significance for our task, neither offer any valid
interpretation. Named entities (mann, jack, stanford), numbers (seventh, one),
or words like copyright, west, program and volume seem to be coincidentally
used in different contexts in different time frames.

Thus, to recognize and understand key semantic shifts on the level of entire
collection, we explore the possibility of grouping words with related meaning
drifts. For each word $w_{i}\epsilon V$ we define a difference vector
$\vec{d_{w_{i}}}$ representing the difference of its average embedding in the
first time frame $\vec{u_{w^{1}_{i}}}$ and the one in last time frame
$\vec{u_{w^{2}_{i}}}$.

$\small\vec{d_{w_{i}}}=\vec{u_{w^{1}_{i}}}-\vec{u_{w^{2}_{i}}}$ (1)

This difference vector captures the information representing the meaning
change of a term from the first to the last time frame. To better understand
this concept, let’s take an example of the term machine. Its difference vector
will point to words which the term machine moved away from, i.e. mechanical
machine, factory and so on, towards the context uses such as algorithm,
learning. The representation of this semantic change from 1979 to 2015 will be
embedded in the difference vector.

To explore the drifts of target words we compute the cosine similarity values
between the difference vector of $w_{i}$ ($\vec{d_{w_{i}}}$) and other words
in vocabulary ($\vec{u_{w^{1}_{j}}}$) to find the concepts from where the
target word $w_{i}$ moved away (lowest similarity) and the words that $w_{i}$
moved towards (highest similarity). In Table II we show the obtained top words
with the lowest and highest similarity values for few selected target words
$w_{i}$. For example, we can observe there that the word merit in 1980s meant
the quality of being worthy or praised, yet it drifted towards the concept of
evaluation in ML models in 2011-2015. Similarly, the word miner in 1980s
represents a person working in mines but it is related to data extraction and
information retrieval in the recent time.

TABLE II: Word movement pattern Terms | Diverted From | Moved To
---|---|---
web |  stein, fin, ray, ink, fell |  document, advertising, words, text, citation, sentiment
merit |  worthy, worth, warrant, wish, sake, permit |  confidence, performance, improvement, accuracy, delta, slope
intercept |  attack, kill, shoot, combat, fight |  beginning, logarithm, effects, latitude, centroid
machine |  workstation, coordinator, factory, constitution, graphic |  bayes, learning, reinforcement, radial, regression
miner |  work, mining, father, husband |  induction, information, extraction, statistics, retrieval
reinforcement |  justification, proliferation, viability, uniformity |  machine, descent, bayes, markov, radial
activation |  elaboration, instantiation, initiation, manifestation |  radial, loss, hinge, regularization, gradient

###  II-D Change-Oriented Term Grouping

For summarizing the semantic changes in the entire corpus, we need to analyse
the relative movements of multiple words in aggregate. As exposed above, some
words have meaningful semantic drift, while some are subject to random or no
change. Grouping words by the patterns of their semantic drift should help to
remove random and meaningless semantic changes (such as some examples in Table
I) and will allow to find important changes. In other words, we look for
relations between the movement patterns of different words. This relation can
be positive, meaning that it characterizes a group of words that converge in
terms of their semantics by moving towards the same point in the semantic
vector space, even though at first the words may have no or have only limited
shared semantics. Alternatively, the relation can be negative, for the words
that divert apart from each other.

To summarize semantic changes, we cluster words which come from different
semantic areas (have different meanings) but which ”converge” together over
time. We do this by using words’ difference vectors (Eq. 1) as the criterion
for word similarity. As the number of clusters is not available beforehand and
can not be easily estimated, we use Affinity Propagation (AP) Algorithm [8].
In AP clustering algorithm one does not need to specify the initial cluster
centers as the method automatically finds a subset of exemplar points which
can best describe the data groups. Since the clustering involves words’
difference vectors, the words in a cluster could have different initial
semantic meaning yet they all converge together or move towards the same
semantic meaning. Example words like Architecture, Input, Parameter, Weight,
Dimensions are unrelated to each other and have different meanings in the
initial time period (1985-1995) but based on the subset of the scientific
corpus spanning 2011-2015, these words became related to deep learning models.

To further filter the results, we discarded clusters with less than 5 words
and removed words within the first quartile of magnitude of their difference
vectors ($\vec{d_{w_{i}}}$) to ensure that only words with a significant
semantic change are contained in the resulting clusters.

###  II-E Cluster Ranking

Clusters produced in the above change-oriented clustering represent the
temporal movement of words’ semantics. The output of Affinity Propagation
results in 340 clusters. We then need to determine the cluster quality to
detect most important and informative clusters. For this we exploit an
observable property of the clusters that, as the words converge together to
the similar semantics, they appear together more frequently in the sentences
than before. Thus for each cluster, we calculate the frequency counts of the
occurrences of this cluster’s members in the same sentences in each analyzed
time frame. In other words, for each cluster we determine the number of
sentences having high containment of the cluster’s words in both the first and
the last time frame. Effectively, we construct the discrete probability
distribution of sentences over the number of cluster words occurring in the
same sentences in a given time frame.

To clarify, let X be the set of records in x, where x is the set of all the
sentences in a time frame and Y be the integer representing the number of
words common in a sentence and cluster. Thus for a cluster i, the probability
of sentences having n words in common with the cluster (or $P_{i}(X\cap y=n)$)
will be the number of sentences with n common words divided by the total
number of sentences in a time frame. Considering the number of cluster members
in the same sentence to be utmost 10, the discrete probability distribution
for the frequency count of a cluster will be:

$y_{i}=[P_{i}(X\cap y=1),P_{i}(X\cap y=2),...,P_{i}(X\cap y=10)]$ (2)

In particular, Wasserstein Distance (called also Earth Mover’s Distance or
EMD) is used to calculate the distance between these two distributions. Given
two random distributions, EMD can be conceptually portrayed as the task of
taking a mass of earth (one distribution) to spread it in space understood as
a collection of holes (another distribution) in the same space. EMD measures
the least amount of work needed to fill the holes with earth. Thus for a given
cluster, the higher the work needed, the higher the semantic importance and
novelty of that cluster. Note that EMD is good for our case as there are
ordinal relations between units of the distribution. For example, sentences
with 4 words from the same cluster are more important than sentences with 3
words which in turn are more important than sentences with 2 words. If for a
cluster there are many sentences containing large numbers of its words (for
example, sentences with 4 or more words from the cluster) it means this
cluster is quite coherent. EMD in our case is used for measuring the
difference in the cluster’s coherence at the last and the first time frames.
Suppose there is a cluster for which there were fewer sentences with high
number of its member words in the first time unit compared to last, in this
case, EMD will be high and the cluster will be judged as important since it
represents words that converged to each other over time. EMD is defined to
minimize the following equation:

$\small WORK(U,V,F)=\sum_{i=1}^{m}\sum_{j=1}^{n}d_{i,j}f_{i,j}$ (3)

where $d_{ij}=d(y_{i},y_{j})$ is the ground distance between $y_{i}$ and
$y_{j}$, and $F=[f_{ij}]$ such that $f_{ij}$ is the flow that needs to be
determined between $y_{i}$ and $y_{j}$. In this work, variables in Eq. 3 are:

$y_{i}$ | $\;:\;$ |  discrete probability distribution for $i_{th}$ cluster in $1^{st}$ time frame.
---|---|---
$y_{j}$ | $\;:\;$ |  discrete probability distribution for $j_{th}$ cluster in $5^{th}$ time frame.
$U$ | $\;:\;$ |  a matrix of $y_{i}$ for all i in $1^{st}$ time frame.
$V$ | $\;:\;$ |  a matrix of $y_{j}$ for all j in $5^{th}$ time frame.

The top-ranked clusters by this metric are given in Table III.

TABLE III: Top-ranked clusters using Earth Mover’s Distance $Value$ | Terms
---|---
2317.15 |  data, set, label, split, tuning, subset, class, fold, generalization, validation
2068.34 |  standard, test, real, deployment, trial
1737.55 |  model, variable, fit, space, interpolation, bayes, predictor, cue, cache, novelty
1611.03 |  integer, factor, permutation, discount, inflation, multiplicity
1376.89 |  submission, pipeline, evaluation, setting, official, configuration, setup, backup
1374.19 |  result, system, classifier, tie, approach, counterpart, detector

As it can be observed from the table, most of the top clusters (cluster 1, 3,
5 and 6) consist of the words which drifted towards a general context of data-
science and machine learning, as expected for the computational linguistics
corpus.

##  III Experimental Analysis

We perform a pilot study of the proposed approach using the ACL Anthology
corpus and LDA-based baseline. Latent Dirichlet Allocation (LDA) [3] is a
generative probabilistic model for collections of discrete data (e.g., text
corpora) with the goal to map all the documents to topics such that the words
in each latent topic tend to co-occur with each other. In our experiment we
utilize LDA333https://radimrehurek.com/gensim/models/ldamodel.html to monitor
the change of topic importance from the time frame 1 to 5 and we compare the
results with the ones by the proposed change-oriented clustering. LDA model
was trained on the combined dataset of documents from the time frame 1 & 5,
with fine-tuned hyper-parameters such as the number of topics equal to 100
with 20 passes through corpus along with filtering words that occur in less
than 30 and more than 75% of the documents (other parameters had default
values). LDA outputs a document-topic matrix representing the importance of
the topics per each document. The topics were then sorted in the increasing
order of the difference between their importance scores in time frame 1 and
ones in time frame 5. The top 10 topics that gained most importance (i.e.,
become more dominant) were then selected as the results.

###  III-A Results

Table IV shows the comparison between the cluster scores for EMD-based
semantic change-oriented clustering which is our proposed model vs. Latent
Dirichlet Allocation based ranking. 3 expert reviewers who have worked in NLP
field for at least 5 years, were asked to score the top 10 clusters returned
by each clustering method on the basis of their quality using 0-5 Likert scale
(0 means meaningless results while 5 indicates the highest quality results).
The average score from the judges was taken as the final score for each
cluster. Table IV shows the results for the top 1, 3, 5 and 10 clusters from
both the rankings. Change-oriented clustering outperforms LDA baseline in all
cases with an overall score improvement of 16%. Additionally, a consistent
trend of the decreasing scores when moving from the top-1 to top-10 signifies
the effectiveness of the proposed EMD based ranking.

TABLE IV: Experimental results. Terms | Top 1 | Top 3 | Top 5 | Top 10
---|---|---|---|---
EMD | 4.66 | 3.88 | 3.5 | 3.76
LDA | 2.66 | 2.33 | 2.77 | 2.96

##  IV Conclusion

We propose a novel method to identify changing concepts from a large
scientific document corpora that are representative for semantic evolution of
this corpora. Our approach can help in better understanding scientific
document collections and the important changes that are latent in them over
time. The approach we developed is based on analyzing temporal changes in
semantics of terms and on quantifying as well as aggregating their drifting
patterns.

In the future, we will find representative sentences for each important
cluster to provide a constrastive type summary of the semantic drift
underlying the cluster and to better represent each output cluster.

## Acknowledgements

We are grateful for the support from the Malaysia Ministry of Higher
Education444FRGS/1/2019/ICT02/UUM/02/2 and the free use of
Rescale555https://techagainstcovid.com/ resources.

## References

  * [1] Iz Beltagy, Kyle Lo and Arman Cohan  “SciBERT: A pretrained language model for scientific text”  In _arXiv preprint arXiv:1903.10676_ , 2019
  * [2] Steven Bird et al.  “The acl anthology reference corpus: A reference dataset for bibliographic research in computational linguistics”  In _EUROPEAN LANGUAGE RESOURCES ASSOC-ELRA_ , 2008
  * [3] David M Blei, Andrew Y Ng and Michael I Jordan  “Latent dirichlet allocation”  In _the Journal of machine Learning research_ 3 JMLR. org, 2003
  * [4] Kevin W Boyack and Richard Klavans  “Creation of a highly detailed, dynamic, global model and map of science”  In _Journal of the Association for Information Science and Technology_ 65.4 Wiley Online Library, 2014
  * [5] Ying Ding  “Scientific collaboration and endorsement: Network analysis of coauthorship and citation networks”  In _Journal of informetrics_ 5.1 Elsevier, 2011
  * [6] Kun Dong et al.  “An integrated method for interdisciplinary topic identification and prediction: a case study on information science and library science”  In _Scientometrics_ 115.2 Springer, 2018
  * [7] M. Färber, C. Nishioka and A. Jatowt  “ScholarSight: Visualizing Temporal Trends of Scientific Concepts”  In _2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL)_ , 2019
  * [8] Brendan J. Frey and Delbert Dueck  “Clustering by passing messages between data points”  In _Science_ 315, 2007
  * [9] William L Hamilton, Jure Leskovec and Dan Jurafsky  “Diachronic word embeddings reveal statistical laws of semantic change”  In _arXiv preprint arXiv:1605.09096_ , 2016
  * [10] M Kishore Kumar and Sanampudi Suresh Kumar  “Temporal summarization for online news articles”  In _2017 IEEE International Conference on Power, Control, Signals and Instrumentation Engineering (ICPCSI)_ , 2017  IEEE
  * [11] Andrey Kutuzov, Lilja Øvrelid, Terrence Szymanski and Erik Velldal  “Diachronic word embeddings and semantic shifts: a survey”  In _arXiv preprint arXiv:1806.03537_ , 2018
  * [12] Muhammad Syafiq Mohd Pozi, Adam Jatowt and Yukiko Kawai  “Temporal Summarization of Scholarly Paper Collections by Semantic Change Estimation: Case Study of CORD-19 Dataset”  In _Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020_ , 2020
  * [13] Guy D Rosin and Kira Radinsky  “Generating Timelines by Modeling Semantic Change”  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , 2019, pp. 186–195
  * [14] Sandeep Soni, Kristina Lerman and Jacob Eisenstein  “Follow the leader: Documents on the leading edge of semantic change get more citations”  In _Journal of the Association for Information Science and Technology_ Wiley Online Library, 2020
  * [15] Nina Tahmasebi, Lars Borin and Adam Jatowt  “Survey of Computational Approaches to Lexical Semantic Change”, 2019  arXiv:1811.06278 [cs.CL]
  * [16] Chengyu Wang, Xiaofeng He and Aoying Zhou  “Event phase oriented news summarization”  In _World Wide Web_ 21.4 Springer, 2018
  * [17] Jian Xu et al.  “Understanding the formation of interdisciplinary research from the perspective of keyword evolution: A case study on joint attention”  In _Scientometrics_ 117.2 Springer, 2018
  * [18] Erjia Yan  “Research dynamics, impact, and dissemination: A topic-level analysis”  In _JASIST_ 66.11 Wiley Online Library, 2015
  * [19] Yi Yu et al.  “Multi-TimeLine Summarization (MTLS): Improving Timeline Summarization by Generating Multiple Summaries”  In _Proceedings of ACL-IJCNLP_ , 2021