# Image registration based automated lesion correspondence pipeline for
longitudinal CT data

Subrata Mukherjee1, Thibaud Coroller2, Craig Wang3, Ravi K. Samala1, Tingting
Hu1, Didem Gokcay1,
Nicholas Petrick1, Berkman Sahiner1, Qian Cao1  1U.S. Food and Drug
Administration,
Center for Devices and Radiological Health, Office of Science and Engineering
Labs,
Division of Imaging, Diagnostics, and Software Reliability, Silver Spring, MD
2Novartis Pharmaceutical Corporation, New Jersey, USA  3Novartis AG,
Switzerland

###### Abstract

Patients diagnosed with metastatic breast cancer (mBC) typically undergo
several radiographic assessments during their treatment. Accurately detecting
and monitoring individual lesions over time, is crucial for making informed
clinical decisions. mBC often involves multiple metastatic lesions in
different organs, therefore it is imperative to accurately track and assess
these lesions to gain a comprehensive understanding of the disease’s response
to treatment. Computerized analysis methods that rely on lesion-level tracking
have often used manual matching of corresponding lesions, a time-consuming
process that is prone to errors. This paper introduces an automated lesion
correspondence algorithm designed to precisely track both targets’ lesions
(mBC lesions that are actively monitored and tracked) and non-targets’ lesions
(abnormalities that are not currently of primary focus) in longitudinal data.
As part of a collaboration between U. S. Food and Drug Administration and
Novartis Pharmaceuticals, we demonstrate the applicability of our algorithm on
the anonymized data from two Phase III trials, MONALEESA-3 and MONALEESA-7.
The dataset contains imaging data of patients for different follow-up
timepoints and the radiologist annotations for the patients enrolled in the
trials. Target and non-target lesions are annotated by either one or two
groups of radiologists. To facilitate accurate tracking, we have developed a
registration-assisted lesion correspondence algorithm. The algorithm employs a
sequential two-step pipeline: (a) Firstly, an adaptive Hungarian algorithm is
used to establish correspondence among lesions within a single volumetric
image series which have been annotated by multiple radiologists at a specific
timepoint. (b) Secondly, after establishing correspondence and assigning
unique names to the lesions, three-dimensional (3D) rigid registration is
applied to various image series at the same timepoint. Registration is
followed by ongoing lesion correspondence based on the adaptive Hungarian
algorithm and updating lesion names for accurate tracking. This iterative
algorithm is then applied to all timestamps and extended across multiple
timepoints for a given patient to ensure precise temporal tracking of targets
and non-targets. Validation of our automated lesion correspondence algorithm
is performed through triaxial plots based on axial, sagittal, and coronal
views, confirming its efficacy in matching lesions.

###### Index Terms:

Volumetric registration, adaptive Hungarian, lesion correspondence.

##  I Introduction

This paper presents work from a collaboration between the U. S. Food and Drug
Administration and Novartis, where we retrospectively utilize anonymized data
from Phase III clinical trials, MONALEESA-2, 3 and 7, to investigate novel
prognostic and predictive factors for metastatic breast cancer patients [1,
2]. The clinical trials were originally conducted to evaluate the efficacy and
safety of ribociclib as an investigational drug in combination with endocrine
therapy leading to the approval of ribociclib as a targeted therapy for
metastatic breast cancer (mBC), specifically mBC characterized by hormone
receptor-positive (HR+) and human epidermal growth factor receptor 2-negative
(HER2-) status [3, 4, 5]. Metastases are a leading cause of cancer-related
mortality [6], and requires longitudinal imaging to track metastases for
effective patient management. The imaging subcomponent of our larger
collaboration aims to employ CT scan images at various time points to evaluate
tumor progression and guide treatment decisions. The CT images in our dataset
were read by either one or two radiologists where they annotated metastatic
lesions as part of a blinded independent central review process.

Computerized analysis of this longitudinal CT data, annotated by different
radiologists, presents challenges, including the appearance of new lesions,
the disappearance of responding lesions, differnt annotations by different
radiologists and across timepoints, and variations in the field of view (FOV)
due to different scan series. Manual sorting of lesions and correspondences
acoss longitudinal CTs is highly labor and time intensive. Additionally,
metastatic cancers may exhibit diverse responses across multiple lesions [7,
8, 9]. Imaging can vary over time due to different patient alignment with
respect to the scanner leading to different image scan series and thereby
different field of views. The anatomy of patients may also change between time
points. Moreover, annotations by different radiologists often leads to
discrepancies in lesions and lesion naming, making manual matching and
tracking of lesions challenging. Therefore, we developed an automated lesion
correspondence algorithm capable of effectively tracking distinct lesions
throughout the longitudinal imaging study, while addressing the aforementioned
complexities and challenges.

Previous literature [10, 11, 12] have addressed lesion matching through
registration. However, registration approaches alone do not fully capture the
complexities of our mBC datasets. Our contribution in this work lies in
addressing the following key challenges: establishing lesion correspondence
despite annotation discrepancies among radiologists, handling multiple
timepoints as well as different image scan series within each time point, and
tracking non-target lesions in addition to the RECIST [13] target lesions. Our
proposed algorithm utilizes the Hungarian (also named Kuhn-Munkres) algorithm
[14, 15], along with a distance threshold and 3-dimensional (3D) rigid
registration [16], in a sequential manner. The algorithm dynamically updates
the correspondence assignment process, ensuring consistent naming for matched
lesions and new names for unmatched ones. We employ a combinatorial
optimization technique to address the assignment challenge involving lesion
centroid coordinates from distinct radiologists working on the same scan
series at a specific timepoint. This task is carried out independently for
both targets and non-targets, considering each lesion as a vertex and their
respective pairings as edges (details in the section III). This graph-based
correspondence approach is systematically applied to scans evaluated by two
different groups of radiologists. Subsequently, we perform volumetric
registration across different scan series originating from the same timepoint.
Following this, the correspondence algorithm is reapplied, leading to updates
in the nomenclature of targets and non-targets. This iterative process is
carried out for each timepoint and extended to multiple timepoints. Depending
on the number of image series and timepoints, volumetric registration may need
to be run multiple times. We employed rigid registration, which offers faster
results compared to other deformable volumetric registration techniques [17].

The rest of the paper is organized as follows: In section II, we describe our
study population and the variabilities in the MONALESSA dataset. In Section
III, we outline our correspondence and registration algorithms. Section IV
shows the results of the algorithm in the form of triaxial plots on a sample
of patients. Finally, Section V concludes the preliminary testing, and
discusses potential future work.

##  II CT Imaging Data

The effective sample from MONALEESA-3 and MONALEESA-7 encompass approximately
300 patients. Tumor evaluations occur initially at baseline (screening),
followed by every 8 weeks during the first 18 months, and then every 12 weeks
thereafter, until disease progression, patient passing, or withdrawal from the
study, whichever comes first. We specifically focused on the first two
timepoints, namely Screening and Week 8 in this study. Target lesions were
annotated by either a radiologist from one radiologist group, or two
independent radiologists from two groups, adhering to the Response Evaluation
Criteria in Solid Tumors (RECIST) criteria v1.1 [13]. The same radiologists
also provided two-dimensional (2D) segmentations of the RECIST target lesions.
RECIST non-target lesions were annotated, but not segmented. Each volumetric
scan consists of a three dimensional (3D) image with dimensions of
approximately $512\times 512\times\sim 100$ voxels.

Figure 1: (a) Overview of the complex architecture in our MONALEESA datasets
reflecting the discrepancy in annotation across radiologists, (b) a possible
scenario for data availability throughout the patient’s participation in the
clinical trial. Here ’T’ represents targets and ’NT’ represents non-targets.

Figure 1(a) shows the complex structure of our MONALEESA dataset where an
arbitrary patient is typically inspected at multiple timepoints. At each
timepoint, there can be one or multiple CT scan series with each CT series
annotated by either one or two radiologists families ($R_{1}$ and $R_{2}$ are
two different families annotating lesions as shown in figure 1(a)). Figure
1(b) reflects a possible scenario of data availability during a typical
patient journey in the trial.

A problem that arises in the analysis of the dataset is that there is no
reliable lesion correspondence among the different radiologists. Figure 1(a)
illustrates such discrepancies where one radiologist family (R1) may designate
one lesion (A) as target 1 (T1) whereas another radiologist family (R2) may
annotate the same lesion as target 2 (T2). This leads to a swapping of lesion
correspondence among radiologists (as shown in right hand side figure 1(a)).
Furthermore, as we progress to later weeks, new lesions may appear, or
existing lesions may disappear. Consequently, there is a clear need to
establish unique lesion correspondences of lesions with and across radiologist
and withing and across timepoints. Figure 2 illustrates a situation in which
one radiologist (R1) designates one lesion as ’T1’ during the screening, while
another radiologist annotates a different lesion as ’T1’ for the same
timepoint.

Figure 2: Example with two radiologist families marking the same lesion. Left
column - in top zoomed axial and bottom axial orientation, radiologist (R1)
marked one target lesion as T1. right column- in top zoomed axial and bottom
axial orientation, radiologist (R2) annaoates a different target as T1.

Registration is needed as imaging across different positions of a patient
leads to different scan series, each capturing a different range and section
of the anatomy. Figure 3 highlights the necessity for performing registration,
as there can be different image scan series with varying anatomical sections
at a given follow-up timepoint. Figure 3 shows three different scan series
with markedly different sections of a patient’s anatomy at Week 8.

Figure 3: Example with different anatomical sections at the same timepoint.
(a) Axial orientation (in top) and coronal orientation (in bottom) of three
different scan series at the Week 8 timepoint or a subject. Thsi variaotion in
the series shows the need for registration because of the different anatomy
scanned.

These scenarios underscore the importance of developing a registration-
assisted lesion correspondence algorithm for accurate tracking and unique
naming of both targets and non-targets.

##  III Automated lesion matching algorithm

We describe the Registration based Automated Lesion Matching and
Correspondence (RALMAC) algorithm here. Initially, the lesion matching
algorithm is applied to establish correspondence between lesion centroids, for
both target and non-target lesions, from two separate radiologists for the
same image series at a specific timepoint. Subsequently, the lesion matching
algorithm is employed to determine correspondence across different series
within a specific timepoint and across various timepoints.

Automated Lesion Matching and Correspondence Algorithm. This algorithm is
rooted in the Kuhn-Munkres algorithm [14, 15] popularly known as the Hungarian
algorithm. It employs combinatorial optimization to efficiently match lesion
centroids based on their spatial coordinates. The objective is to find one-to-
one correspondence between two sets of lesions by minimizing a cost function
based on the Euclidean distance between lesion centroids in the two sets. For
a particular image series, the two sets correspond to data frames containing
lesion annotations from two different radiologists. In the case of different
image series, the sets represent data frames from different scan series after
registration. The algorithm’s working principle is illustrated in Figure 4.
and finds correspondence separately among target and non-target lesions. The
centroids of the target lesions are marked in red, while those of non-target
lesions are marked in black (in figure 4). The Hungarian algorithm operates on
the distances between all lesion centroids in these groups.

The algorithm treats the correspondence problem as a bipartite graph, denoted
as $G=\\{R_{1},R_{2},E\\}$ as illustrated in figure 4. Each partition of the
graph (a and b in figure 4) $R_{1}=\\{r_{1,1},\ldots,r_{1,i}\\}$ represents
target centroids (in red in figure 4) of one radiologist / one image series
whereas $R_{2}=\\{r_{2,1},\ldots,r_{2,j}\\}$ represents target centroids (in
red in figure 4) of another radiologist / another image series. The same holds
for the non-target groups (marked in black in figure 4). The edge E indicates
a match between the two sets (as marked by dotted line between matches).
Unmatched lesions (lesion 5 and 2 in figure 4) are assigned new names. To
enhance matching accuracy, an additional distance threshold is applied. Lesion
pairs with distances exceeding this threshold are considered unmatched, even
if the Hungarian algorithm suggests a correspondence based on the cost
function. The distance threshold is 40 mm for the target lesions and $50$ mm
for the non-target lesions. This scenario is observed between lesion 3 and 4
in figure 4 as the Hungarian alorithm suggest the lesions are matched but our
distance threshold defines them as unmatched and assigns new names. After
establishing correspondence, the algorithm assigns names to target lesions as
$\\{G_{1},G_{2},\ldots,G_{j}\\}$ and to non-target lesions as
$\\{NG_{1},NG_{2},\ldots,NG_{i}\\}$ with j and i unique target and non-target
lesions after matching.

Figure 4: Schematic showing unique lesion matching concept. Target lesions are
shown in red and non-target lesions in black. The matched lesions are
inidcated by circles whereas unmatched lesions are indicated by squares.

For an example, the algorithm first finds correspondence among radiologist
families in the scan series at Screening ( top left plot of figure 5).
Then,repeats this process for the Week 8 scan series (top right plot).
Finally, after volumetric registration across timepoints (Screening and Week
8), the algorithm establishes correspondence between lesions from Screening
and Week 8 (shown in bottom plot of figure 5) .

Figure 5: Top left upper plot shows the application of correspondence
algorithm to match lesions across radiologists at screening. Top right plot
shows the application of correspondence between radiologists at Week 8. Bottom
plot shows the correspondence across the timepoints post registration.Lesion
marked by radiologist 1 are in blue and those marked by radiologist 2 are in
red (for the top two plots). In the bottom plot, lesions at ’Screening’ time
point are in blue and those at time point ’Week 8’ are in red.

Volumetric Rigid Registration. Volumetric (3D) rigid registration is a pivotal
process in medical imaging analysis, facilitating the alignment of two 3D
image datasets to enable spatial correspondence. This technique is
instrumental in a multitude of clinical applications, such as monitoring
disease progression, formulating treatment strategies, and conducting
comparative analyses of anatomical structures across different time points or
between patients. The Simple Image Toolkit (SITK) [16] provides a robust
framework for implementing 3D rigid registration, which we applied here to
align different image series from a specific time stamp or between varying
stages such as screening and a subsequent 8-week follow-up.

Steps of the SITK based rigid registration is depicted in figure 6. In rigid
registration, the image series designated as the ’fixed’ image serves as the
benchmark, while the ’moving’ image series is transformed to align with the
fixed one. When performing registration across different timepoints,
typically, the image at the screening phase is kept fixed, and subsequent
images, like those from Week 8, are adjusted accordingly.

Prior to registration, both fixed and moving images undergo preprocessing,
including binary thresholding and connected component analysis. This step
isolates the pertinent anatomical features while eliminating extraneous
elements such as air pockets in the images. The rigid transformation, denoted
by $T(S)$ maps position $\mathbf{S}=[x,y,z]^{T}$ in fixed image (A) to
corresponding position in moving image (B). Transformation T(X) has six
degrees of freedom: 3 translations $(t_{1},t_{2},t_{3})$ and three rotation
angles $(\alpha_{1},\alpha_{2},\alpha_{3})$. A 3D rigid transformation which
first rotates position and then applies a translation is given by:
$T(S)=R(S)+[t_{1}\;t_{2}\;t_{3}]^{T}$

where the rotation matrix $R$ is shown in equation 1:

$\begin{split}R=&\begin{bmatrix}1&0&0\\\
0&\cos(\alpha_{1})&\sin(\alpha_{1})\\\
0&-\sin(\alpha_{1})&\cos(\alpha_{1})\end{bmatrix}\\\
&\begin{bmatrix}\cos(\alpha_{2})&0&-\sin(\alpha_{2})\\\ 0&1&0\\\
\sin(\alpha_{2})&0&\cos(\alpha_{2})\end{bmatrix}\\\
&\begin{bmatrix}\cos(\alpha_{3})&\sin(\alpha_{3})&0\\\
-\sin(\alpha_{3})&\cos(\alpha_{3})&0\\\ 0&0&1\end{bmatrix}\end{split}$ (1)

Figure 6: Schematic of the volumetric rigid registration framework based on
the transformation approach adopted in the paper.

The registration process starts with an initial alignment based on the
centroids and principal axes of both images to achieve a foundational
correspondence between the physical contents. Throughout the rigid
registration workflow, depicted in figure 6, an optimizer, a similarity
metric, and an interpolator are used to refine the image alignment through an
iterative process. Mattes Mutual Information [18], a metric quantifying the
dependency between pixel intensities of corresponding images, is employed for
image matching, utilizing a 50-bin histogram for estimation. To compute the
metric, a $10\%$ random sampling of points is used. The registration model
integrates linear interpolation and a gradient descent optimizer, with a
learning rate of $0.1$, a $1000$ iteration count, and a convergence threshold
of $1\times 10^{-10}$. A multi-resolution approach is adopted, altering image
resolution throughout the registration to enhance robustness and accuracy. The
shrink factor array $[2,1,1]$ scales down the image size initially by half,
maintaining the original size in subsequent levels. Gaussian smoothing,
determined by the ’smoothing sigmas’ parameter with values $[4,2,1]$, reduces
noise and variations across the levels, thereby mitigating local optima during
the registration process. Upon reaching convergence, both the transformation
matrix and the registered image are saved.

Figure 7 (left plot) displays this initial alignment, between Screening and
Week 8 image series for subject 1 of Table I. A triaxial representation of the
overlay between the registered and fixed images is subsequently presented. The
congruence of the images is evident in the axial, sagittal, and coronal views,
indicating a successful volumetric registration with a substantial overlap of
the registered and fixed images, affirming the effectiveness of the applied
registration process. Next, an inverse transformation is done on the obtained
transformation matrix and this inverse transform is applied on the lesion
coordinates (spatial coordinates of lesion centroids) of the moving (here Week
8) image series. Finally lesion correspondence is conducted as before between
the fixed image lesion coordinates and transformed moving image lesion
coordinates.

Figure 7: Overlay of the moving and fixed image after initialization (left),
registered (Week 8 image after registration) and Screening (fixed) image
showing the effectiveness of 3D rigid registration across three
orientations.The slice number denotes the specific slice chosen within each
orientation, while alpha represents the parameter controlling transparency.

##  IV Results and Discussion

We established correspondence between lesions across different timepoints,
radiologists, and image scan series. For each individual patient, identified
by their unique subject ID, we manually validated the correspondence and
matching of lesions. This was accomplished through visual inspection of
triaxial plots showing the lesion locations (see figure 8 and 9).
Visualization ensures that the distinctiveness of each lesion is maintained,
contingent on the lesion residing within the same region in the triaxial
plots. We have adopted an organized approach in our triaxial plots: each
column portrays a specific orientation of the lesion, while the rows represent
differences in the data such as different timepoints, radiologist
interpretations, or image scan series in which the lesion was detected.

An example of our algorithm’s performance is evident in the case of the
example subject. We validate our RALMAC algorithm’s output focusing
specifically on target lesions, highlighted in red for clarity. As depicted in
Figure 8, three distinct target lesions, denoted as ${G_{1},G_{2}},G_{3}$ are
identified. The consistency in the spatial positioning of the target lesion
‘$G_{1}$’ in the triaxial plots effectively underscores its uniqueness across
Screening and Week 8 (see first two rows). Similarly, the lesion ‘$G_{2}$’,
’$G_{3}$’ demonstrates the algorithm’s precision, maintaining lesion
correspondence across multiple timepoints and radiologist assessments (see
rows 3 to 6 for $G_{2}$ and rows 7-8 for $G_{3}$).

Figure 8: Visualization of the lesion correspondence algorithm performance in
identifying and tracking unique target lesions (target centroids shown in red)
for subject 1 of Table I

For a different subject, (figure 9) our RALMAC algorithm tracked a single
unique target lesion, ‘$G_{1}$’ across different image series and timepoints.
The uniformity of the lesion’s location across the triaxial plots indicates
our algorithm’s robust ability to accurately establish lesion correspondence.

Figure 9: Visualization of the lesion correspondence algorithm performance in
identifying and tracking a target lesions (target centroids shown in red) for
subject 5 of Table I

We looked at the performance of the proposed algorithm based on screening and
week 8’s CT scan data from 25 randomly chosen patients in the dataset. In
Table I, we report the number of target lesions for each patient that were
reported by two radiologists (column R1 and R2 of table I) as well as the
number of misaligned target lesions between them. In the misaligned (M/X/N
stands for Misaligned/Missing/ New) column, we not only sum the number of
target lesions that had different nomenclature in the two radiologist’s
markings but also target lesions that were reported by one and missed by the
other. Figure 10 show that there is a significant fraction of misalignment
between the two radiologists in the data. In columns 2, 3 and 4 of the table
I, we respectively report the number of unique targe lesions that are reported
by our algorithm (U), the number of target lesions that are missed by our
algorithm (MI) and the number of target lesions that are falsely reported by
our algorithm (F).

To better conceptualize what we mean by missing and falsely reported targets,
we provide examples below:

  * •

MI: Assume that only one target $(G_{1})$ is identified by the radiologists on
Screening, and two different targets were identified at Week 8. If the
algorithm matches both target lesions seen on Week 8 to $G_{1}$, then this
constitutes a missing target.

  * •

F: Assume that a particular lesion $(G_{1})$ is identified by the radiologists
on both Screening and Week 8 . If the algorithm fails to match the lesion seen
on Week 8 to $G_{1}$ and instead calls it a new target, then this constitutes
a false report.

Thus, the true number of target lesions for each patient is $U+MI-F$. We see
that our proposed algorithm fails for 2 out of 25 or 8% of the patients. In
both cases, it overestimates the number of target lesions. As such our
algorithm estimated a total of 65 unique targets lesions when the true number
of target lesions was 62, leading to an overestimation rate of 4.84%.

On closer introspection, we found that the overestimation of target lesions by
the proposed algorithm was mainly due to a failure of aligning initially
misaligned targets. The algorithm failed to recognize the misaligned targets
as targets which are in close proximity and should have been merged and
labeled as a single target. Overall, the algorithm worked well in labeling
targets and never under-counted.

TABLE I: Evaluation of the correspondence algorithm on 25 random patients in
the dataset.

Subject Algorithm Radiologist U MI F R1 R2 Align M/X/N 1 3 0 0 2 2 1 2 2 2 0 0
2 2 2 0 3 6 0 0 5 3 1 5 4 3 0 0 2 2 1 2 5 1 0 0 1 0 0 1 6 1 0 0 1 0 0 1 7 2 0
0 2 0 0 2 8 1 0 0 1 0 0 1 9 2 0 0 2 2 2 0 10 4 0 0 2 2 0 4 11 2 0 0 2 2 2 0 12
6 0 1 5 2 2 3 13 4 0 2 2 0 0 2 14 2 0 0 2 1 1 1 15 1 0 0 1 1 1 0 16 2 0 0 2 0
0 2 17 4 0 0 4 2 2 2 18 2 0 0 2 1 1 1 19 3 0 0 2 2 1 2 20 2 0 0 1 1 0 2 21 1 0
0 1 0 0 1 22 3 0 0 3 2 2 1 23 2 0 0 2 2 2 0 24 3 0 0 1 3 1 2 25 3 0 0 3 2 1 2

Figure 10: Plot showing the % of misaligned target lesions reported by
radiologists (in black) and the misclassification rate of the algorithm (in
red) across the $25$ patients. The size of the dots are proportional to true
number of target lesions.

##  V Conclusion and Future Work

This paper introduces a novel registration-based automated lesion
correspondence and matching (RALMAC) algorithm which was shown to works well
in tracking target lesions and non-target lesions in diverse CT imaging
datasets based on our preliminary assessment of 25 patients. We plan to make
our implementation publicly available as a CDRH regulatory science tool [19].
As part of our future work, we aim to assess the efficacy of the algorithm in
lesion tracking across all follow-up timepoints (till the end of follow up or
an event). The establishment of lesion correspondence across the timepoints
will be instrumental for the next phase of our overall study that aims to
predict patient response to therapy based on longitudinal imaging.

## Acknowledgment

This work is supported by the FDA Office of Women’s Health. Subrata
Mukherjee’s appointment was supported by the Research Participation Program at
the U.S. Food and Drug Administration administered by the Oak Ridge Institute
for Science and Education through an interagency agreement between the U.S.
Department of Energy and the U.S. Food and Drug Administration.

## References

  * [1] T. Coroller, B. Sahiner, A. Amatya, A. Gossmann, K. Karagiannis, C. Moloney, R. K. Samala, L. Santana-Quintero, N. Solovieff, C. Wang, et al., “Methodology for good machine learning with multi-omics data,” Clinical Pharmacology & Therapeutics, 2023.
  * [2] T. P. Coroller, B. Sahiner, A. Amatya, A. Gossman, K. Karagiannis, R. K. Samala, L. Santana-Quintero, N. Solovieff, C. Wang, L. Amiri-Kordestani, et al., “Multi-omics investigation on the prognostic and predictive factors in metastatic breast cancer using data from phase iii ribociclib clinical trials: A statistical and machine learning analysis plan,” medRxiv, pp. 2023–08, 2023.
  * [3] D. Tripathy, S.-A. Im, M. Colleoni, F. Franke, A. Bardia, N. Harbeck, S. A. Hurvitz, L. Chow, J. Sohn, K. S. Lee, et al., “Ribociclib plus endocrine therapy for premenopausal women with hormone-receptor-positive, advanced breast cancer (monaleesa-7): a randomised phase 3 trial,” The Lancet Oncology, vol. 19, no. 7, pp. 904–915, 2018.
  * [4] D. J. Slamon, P. Neven, S. Chia, P. A. Fasching, M. De Laurentiis, S.-A. Im, K. Petrakova, G. V. Bianchi, F. J. Esteva, M. Martín, et al., “Phase iii randomized study of ribociclib and fulvestrant in hormone receptor–positive, human epidermal growth factor receptor 2–negative advanced breast cancer: Monaleesa-3,” Journal of Clinical Oncology, vol. 36, no. 24, pp. 2465–2472, 2018.
  * [5] G. N. Hortobagyi, S. M. Stemmer, H. A. Burris, Y.-S. Yap, G. S. Sonke, S. Paluch-Shimon, M. Campone, K. L. Blackwell, F. André, E. P. Winer, et al., “Ribociclib as first-line therapy for hr-positive, advanced breast cancer,” New England journal of medicine, vol. 375, no. 18, pp. 1738–1748, 2016.
  * [6] C. L. Chaffer and R. A. Weinberg, “A perspective on cancer cell metastasis,” science, vol. 331, no. 6024, pp. 1559–1564, 2011.
  * [7] O. Humbert and D. Chardin, “Dissociated response in metastatic cancer: an atypical pattern brought into the spotlight with immunotherapy,” Frontiers in Oncology, vol. 10, p. 566297, 2020.
  * [8] C. Wang and Y. Shen, “Study on the distribution features of bone metastases in prostate cancer,” Nuclear medicine communications, vol. 33, no. 4, pp. 379–383, 2012.
  * [9] D. T. Huff, V. Santoro-Fernandes, S. Chen, M. Chen, C. Kashuk, A. J. Weisman, R. Jeraj, and T. G. Perk, “Performance of an automated registration-based method for longitudinal lesion matching and comparison to inter-reader variability,” Physics in Medicine & Biology, vol. 68, no. 17, p. 175031, 2023.
  * [10] Z. An, H. Ma, L. Liu, Y. Wang, H. Lu, C. Zhou, R. Xiong, and J. Hu, “Robust orthogonal-view 2-d/3-d rigid registration for minimally invasive surgery,” Micromachines, vol. 12, no. 7, p. 844, 2021.
  * [11] J. Xu, H. Greenspan, S. Napel, and D. L. Rubin, “Automated temporal tracking and segmentation of lymphoma on serial ct examinations,” Medical physics, vol. 38, no. 11, pp. 5879–5886, 2011.
  * [12] V. Santoro-Fernandes, D. Huff, M. L. Scarpelli, T. G. Perk, M. R. Albertini, S. Perlman, S. S. Yip, and R. Jeraj, “Development and validation of a longitudinal soft-tissue metastatic lesion matching algorithm,” Physics in Medicine & Biology, vol. 66, no. 15, p. 155017, 2021.
  * [13] D. Muenzel, H.-P. Engels, M. Bruegel, V. Kehl, E. Rummeny, and S. Metz, “Intra-and inter-observer variability in measurement of target lesions: implication on response evaluation according to recist 1.1,” Radiology and oncology, vol. 46, no. 1, pp. 8–18, 2012.
  * [14] J. Munkres, “Algorithms for the assignment and transportation problems,” Journal of the society for industrial and applied mathematics, vol. 5, no. 1, pp. 32–38, 1957.
  * [15] G. A. Mills-Tettey, A. Stentz, and M. B. Dias, “The dynamic hungarian algorithm for the assignment problem with changing costs,” Robotics Institute, Pittsburgh, PA, Tech. Rep. CMU-RI-TR-07-27, 2007.
  * [16] K. Marstal, F. Berendsen, M. Staring, and S. Klein, “Simpleelastix: A user-friendly, multi-lingual library for medical image registration,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 134–142, 2016.
  * [17] J. Larrey-Ruiz and J. Morales-Sánchez, “Optimal parameters selection for non-parametric image registration methods,” in International Conference on Advanced Concepts for Intelligent Vision Systems, pp. 564–575, Springer, 2006.
  * [18] E. Neri, D. Caramella, C. Bartolozzi, et al., “Image processing in radiology,” Medical Radiology. Diagnostic Imaging. Springer, Berlin, 2008.
  * [19] CDRH, “Catalog of regulatory science tools to help assess new medical devices,” 2023.  https://www.fda.gov/medical-devices/science-and-research-medical-devices/catalog-regulatory-science-tools-help-assess-new-medical-devices Accessed: Feb. 15th, 2024.


#  A Novel Patent Similarity Measurement Methodology: Semantic Distance and
Technological Distance

Yongmin Yoo  Cheonkam Jeong  Sanguk Gim  Junwon Lee  Zachary Schimke  Deaho
Seo  Department of Computer Science, University of Auckland, Auckland, New
Zealand  Department of Linguistics, University of Arizona, Arizona, Tucson,
United States  SR Universe, Seoul, Republic of Korea  Royal Melbourne
Institute of Technology, Melbourne, Australia  James E. Rogers College of Law,
University of Arizona, Tucson, Arizona, United States  School of Information,
Yonsei University, Seoul, Republic of Korea

###### Abstract

Measuring similarity between patents is an essential step to ensure novelty of
innovation. However, a large number of methods of measuring the similarity
between patents still rely on manual classification of patents by experts.
Another body of research has proposed automated methods; nevertheless, most of
it solely focuses on the semantic similarity of patents. In order to tackle
these limitations, we propose a hybrid method for automatically measuring the
similarity between patents, considering both semantic and technological
similarities. We measure the semantic similarity based on patent texts using
BERT, calculate the technological similarity with IPC codes using Jaccard
similarity, and perform hybridization by assigning weights to the two
similarity methods. Our evaluation result demonstrates that the proposed
method outperforms the baseline that considers the semantic similarity only.

###### keywords:

Patent Similarity, Semantic Distance, Technological Distance, Technological
Management, Natural Language Processing

##  1 Introduction

Since the advent of the 4th revolution, technology has developed rapidly. Many
new innovations and strategies have emerged every day. In this fast stream, a
need to measure novelty of them in a quick but accurate manner is arising.
Measuring similarity between patents has been used as an effective way to
gauge novelty, and accordingly many methods to measure the patent similarity
have been proposed, for instance Arts et al., (2018) [1]. However, none of the
previous methods fulfills both reliability and immediacy. Thus, we propose a
novel method to measure the patent similarity.

A patent document consists of a title, an abstract, a description, and claims,
along with administrative parts of the patent, such as the patent number,
inventor(s), IPC codes, CPC codes, and filing date. As a patent document is
full of technical terminologies, or not jargon-free, it is difficult for
laypersons to understand it properly. Furthermore, as a patent document
provides every detail of the patent, it is challenging to grasp the gist of it
quickly. This may impede innovation processes in that it is costly to figure
out whether what one would like to invent can be granted, especially in a
rapidly changing world. In order to ease the processes, a large body of
research has proposed measurements for the patent similarity.

There are several sources with which the patent similarity can be measured:
Similarity of purposes of patents, that of application areas of patents,
textual similarity of patents, bibliographic similarity, etc. as defined in
Moehrle, (2010) [2]. In their seminal work, Arts et al., (2018) proposed a
keyword-based method using texts of patents to measure the patent similarity
[1]. On the one hand, Rodriguez et al., (2015) devised a similarity
measurement method by utilizing direct and indirect co-citation links between
patents [3]. In a similar vein, Lai and Wu, (2005) built a patent
classification system based on a co-citation analysis of bibliometrics [4].
Not only with patent texts and bibliography, linguistic structures have been
also considered to be another source for the patent similarity measurement;
for example, Wang et al., (2019) presented a similarity method based on
semantic information of patent texts [5].

Another body of research focuses more on automating the measurement process by
utilizing machine learning and/or deep learning, in particular Natural
Language Processing (NLP), algorithms. Younge et al., (2016) built a single
vector space model that measures the patent similarity with the full technical
description in the patent document [6]. Lu et al., (2020) adopted a deep
learning-based approach and demonstrated that their model outperforms previous
text representation and classification models [7]. Several works proposed
neural network-based models, for instance Convolutional Neural Networks
(CNN)-based models (He & Lin, 2015 [8]; Peng et al., 2020 [9]). Still others
adopted transformer-based approaches; for example, Lee and Hsiang (2019)
applied Bidirectional Encoder Representations from Transformers (BERT; Devlin
et al., 2018 [10]) to patent texts [11].

It is true that all the previous models have their own benefits. However, none
of them satisfies both reliability and immediacy, which are of primary
importance in today’s tech-savvy world. They mostly focused on semantic
information of patent tests either manually or automatically. In order to
tackle these issues, we propose a hybrid method to measure the patent
similarity. To be specific, we consider both semantic and technological
aspects of patents to enhance reliability, and build an automatic method to
measure the patent similarity quickly. In order to achieve these, we measure
semantic similarity between patents based on titles and abstracts, which we
call Semantic Distance (SD), and technological similarity between them based
on IPC codes, which we call Technological Distance (TD). In building a model,
we consider utilizing the state-of-the-art NLP techniques for similarity
measurements (Garg & Ramakrishnan, 2020 [12]; González-Carvajal & Garrido-
Merchán, 2020 [13]; Yoo et al., 2020 [14], among others).

The contributions of this research are in the respects of brevity,
comprehensiveness, and versatility as follows.

  * 1.

Brevity: The proposed model is simpler than the previous models in terms of
model complexity, but with higher-level of performance, as we use minimal but
essential information.

  * 2.

Comprehensiveness: Recently issued patents are much more complex to analyze
than those in the past in that they encompass various sub-technologies. This
tendency necessitates a comprehensive model to measure the patent similarity.
The proposed model is more comprehensive than the previous models as it
incorporates both semantic and technological similarities.

  * 3.

Versatility: The proposed model is effective to gauge novelty, but it can be
used for other purposes, such as trend analysis (e.g., Choi & Hwang, 2014
[15]).

This research is composed as follows. Section 2 provides literature review.
Section 3 explicates the proposed model with the main components of it in a
detailed manner. Section 4 describes the experimental results. Section 5
provides discussion, and Section 6 draws a conclusion.

##  2 Literature research

The topics into which previous research delved are mainly in relation to the
patent similarity methods and document classification of the patent. They are
explicated in this section.

###  2.1 Patent similarity

Previously, Moehlre, (2010) posited two levels of patent similarity types with
respect to the content and form [2]. According to them, the content-level
similarity can be captured based on structural similarity of patents, such as
purposes of the inventions, application areas of them, etc. while the form-
level similarity based on textual similarity of patents, such as similarity
between patent texts, bibliographic similarity, etc. Among the possible
sources for patent similarity measurements, Arts et al., (2018) devised a
keyword-based method using patent abstracts to measure the textual similarity
of patents [1]. They extracted keywords from patent abstracts, established a
Jaccard-based similarity method, and verified validation by an expert panel. A
body of research proposed similarity measurement methods by utilizing
bibliographic information. Lai and Wu, (2005) established a patent
classification system based on a co-citation analysis of bibliometrics [4].
Similarly, Rodriguez et al., (2015) measured pairwise patent similarity by
using both direct and indirect co-citation links between patents [3]. Recent
studies have proposed similarity measurement methods considering linguistic
nature of patent texts, specifically Subject-Action-Object structures (Wang et
al., 2019 [5]), both semantic and syntactic structures (Yang et al., 2021
[16]), and various word senses (Ahmad & Faisal, 2022 [17]).

On the other hand, a large body of research has focused more on methodological
aspects by adopting machine learning and deep learning, in particular NLP,
techniques. Cascini and Zini, (2008) devised a clustering algorithm that
estimates the patent similarity by considering hierarchical and functional
interactions between patents [18]. Vector space models were also applied to
the patent analysis. Younge et al.,(2016) developed a single vector space-
based model that measures continuous similarity distance between patent pairs
in an automatic manner [6]. In a similar vein, Feng, (2020) devised a
similarity measurement method via vector space representations of patent
abstract using Document Vectors (Doc2Vec) [19]. In addition to these machine
learning-based techniques, NLP-based ones were applied to the patent analysis.
Noh and Lee, (2015) applied text mining to patent analysis through keyword
selection and processing strategies [20]; similarly, Joung and Kim, (2017)
adopted a keyword-based approach for technology planning [21]. Recently, BERT-
based methods have been applied to the patent analysis, for example Lee &
Hsiang, 2019 [11].

###  2.2 Document similarity

Since patents are documents in natural languages, we introduce previous
research on how to measure the pairwise document similarity in natural
language. Kadhim, (2019) explained a canonical workflow for the text
classification by providing examples of supervised machine learning models,
such as K-nearest neighbors [22]. More modern research has focused on
improving the model performance. For instance, Farouk, (2020) devised a
comprehensive method to measure the sentence similarity that combines both
structure and word-based similarities and exploits sentence semantic
structure, specifically discourse representation structure (DRS) [23].
Similarly, Lan, (2022) proposed a hybrid method that incorporates the term
similarity weighted tree data structure and the semantic similarity
information of HowNet (Dong & Dong, 2020 [24]) in order to improve low
accuracy of the text similarity caused by TF-IDF (term frequency-inverse
document frequency) and tackle the curse of dimensionality issue [25].
Transformer-based models also have been applied to the text classification.
González-Carvajal and Garrido-Merchán, (2020) demonstrated superiority of BERT
to the traditional TF-IDF embedding method on text classification with machine
learning-based models [13]. Furthermore, Garg and Ramakrishman, (2020) devised
a BERT-based method to generate adversarial examples for text classification
to make text classification models more robust [12]. In addition, Viji et al.,
(2022) implemented a Bidirectional Long Short-Term Memory (Bi-LSTM) model with
weighted word embeddings through BERT fine-tuning, and corroborated the Bi-
LSTM model with BERT in NLP application tasks, such as measuring text
similarity in questions or pairs of documents [26].

##  3 Methodology

Our method considers two aspects. First, we calculate the Semantic Distance,
which compares the contents of the two patents. Second, we calculate the
Technical Distance, which considers the technological aspects of the two
patents. Our model properly hybridizes and weights the two aspects.

###  3.1 Semantic distance

Figure 1: Model architecture for the semantic distance measurement

We measure the semantic similarity between two patents using a BERT model, one
of the top performing models in a large number of natural language processing
tasks. Among multiple variants of the BERT models, Sentence-BERT (Reimers &
Gurevych, 2019 [27]) is adopted to calculate the semantic distance between
them. First, we feed the whole abstracts with titles to BERT, by which a set
of output vectors per patent is derived, as illustrated in Figure 1. Second,
we calculate the mean of each set of output vectors using Equation (1). Then,
we calculate the cosine similarity of the two mean vectors (i.e., vectorA,
vectorB) through Equation (2)

$\displaystyle\begin{array}[]{@{}l}\begin{array}[]{l}vector_{A}=\displaystyle\frac{\sum_{i=1}^{n}{\left(Patent_{A}\right)}_{i}}{n}\\\
vector_{B}=\displaystyle\frac{\sum_{i=1}^{n}{\left(Patent_{B}\right)}_{i}}{n}\end{array}\end{array}$
(1)

$\displaystyle\begin{array}[]{@{}l}\cos_{A,B}\;=\;\displaystyle\frac{A\cdot
B}{\left|A\right|\left|B\right|}=\displaystyle\frac{\sum_{i=1}^{n}A_{i}\times
B_{i}}{\sqrt{\sum_{i=1}^{n}(A_{i})^{2}}\times\sqrt{\sum_{i=1}^{n}(B_{i})^{2}}}\end{array}$
(2)

Finally, the semantic distance SDA,B is derived by normalization of the cosine
similarity value using Equation (3)

$\displaystyle\begin{array}[]{@{}l}SD_{A,B}=\displaystyle\frac{\left(cos_{A,B}+1\right)}{2}\end{array}$
(3)

###  3.2 Technological distance

Figure 2: Model architecture for the technological distance measurement

We measure the technological similarity of patents with IPC codes in patent
documents. To be specific, we measure to what extent two patents are similar
in terms of technology by calculating the overlapping portion of the IPC codes
of them. IPC codes encode section, class, sub-class, main group, and sub-group
hierarchically. Since it is rare for IPC codes to match perfectly, we measure
the technological distance TD based on the three-depth structure, consisting
of section, class, and sub-class.

First, we calculate how many techniques are overlapped between patents using
Equation (4) and obtain the total number of techniques between them using
Equation (5). We then calculate a Jaccard similarity with IntersectionA,B and
UnionA,Bas illustrated inEquation (6), which is defined as TDA,B. The workflow
of calculating TD is provided in Figure 2.

$\displaystyle\begin{array}[]{@{}l}Intersection{}_{A,B}=Patent_{A}\cap
Patent_{B}\end{array}$ (4)

$\displaystyle\begin{array}[]{@{}l}Union_{A,B}=Patent_{A}\cup
Patent_{B}\end{array}$ (5)

$\displaystyle\begin{array}[]{@{}l}TD_{A,B}=\displaystyle\frac{Intersection_{A,B}}{Union_{A,B}}\end{array}$
(6)

###  3.3 Hybrid similarity

We propose a hybrid similarity method, SDTD, using both SD and TD defined in
3.2. To be specific, we give TD weight to SD and then multiply (TD \+ 1) to
SD. Thereafter, we normalize the multiplied value by dividing it by 2, in
order to prevent TD value from being too large and SDTD value from exceeding
1, as illustrated in Equation (7).

$\displaystyle\begin{array}[]{@{}l}SDTD_{A,B}=\\{\left(TD+1\right)\ast
SD\\}/2\end{array}$ (7)

##  4 Experiment

In this section, we provide a data evaluation method and experiment results.
In training the models, we use Python version 3.10.9 and Pytorch version
1.13.1 from Seoul, Korea. The computer specifications used in the experiment
are as follows: 4 GeForce RTX 2080Ti and 16 Intel(R) Core(TM) i7-9800X CPU @
3.80GHz.

###  4.1 Data

Google patent is a platform where patents from 2016 to 2020 granted by United
States Patent and Trademark Office are provided. Of them, Artificial
Intelligence Patent Dataset (Giczy et al., 2019 [28]) are adopted for this
research. Using Google patent, we randomly extract 420 pairs patents from 2019
to 2020, with which we perform expert validation.

The expert panel consists of three scientists, each of whom has expertise in
Data Analytics, Data Mining, and Artificial Intelligence, respectively. Before
asking them to rate the patents, we set a guideline for rating, following
advice by the author Zachery Schimke, a Juris Doctor candidate, who
specializes in Patent at the University of Arizona. To be specific, we adopt a
seven-scale rating scheme; it consists of the five odd numbers between 0 and
10, to which 0 and 10 are also added as exceptional cases. Then, we establish
five criteria: Area, Task or Purpose, Methodology, and Application. Depending
on the number of the overlapping criteria, rating scores are given, as
described inTable 1.

Table 1: Criteria for patent rating |
---|---
Score |  Criterion
0 |  Totally different
1 |  1 similar property
3 |  2 similar properties
5 |  3 similar properties
7 |  3 similar properties & 1 somewhat similar property
9 |  4 similar properties, but not the same
10 |  Exactly same

Following the guideline, each rater assesses how semantically similar two
patents are solely based on patent abstracts with titles. In order to ensure
reliability, we winnow out the data based on a variance analysis and then ask
the author Zachery Schimkem, a law expert, to rate them. As described inTable
2, we calculate Distance and then set Threshold as 8. If Distance is larger
than Threshold, we separately store the data for Law expert rating; otherwise,
we calculate the mean score.

Table 2: Data rating flow
---
$\begin{array}[]{l}\boldsymbol{L}\boldsymbol{a}\boldsymbol{w}\boldsymbol{\;}\boldsymbol{e}\boldsymbol{x}\boldsymbol{p}\boldsymbol{e}\boldsymbol{r}\boldsymbol{t}\boldsymbol{\;}\boldsymbol{r}\boldsymbol{a}\boldsymbol{t}\boldsymbol{i}\boldsymbol{n}\boldsymbol{g}\boldsymbol{:}\\\
Asking\;a\;law\;expert\;to\;rate\;patents\\\
if\;scores\;rated\;by\;the\;expert\;panel\\\ have\;huge\;discrepancies\\\ \\\
\begin{array}[]{l}\mu\;=\;(r_{1}+r_{2}+r_{3})/3\\\ Distance\;=\;(\mu-
r_{1})^{2}+(\mu-r_{2})^{2}+(\mu-r_{3})^{2}\\\ Threshold\;=\;8\\\
\boldsymbol{i}\boldsymbol{f}\;Distance\;\geq\;Threshold:\\\
\;\;\;\;return\;Law\;expert\;rating\\\
\boldsymbol{e}\boldsymbol{l}\boldsymbol{s}\boldsymbol{e}\boldsymbol{:}\\\
\;\;\;\;Score\;=\;(\;r_{1}+r_{2}+r_{3})/3\end{array}\end{array}$

Out of 420 pairs of the patents, 85 pairs turn out to be the cases for Law
expert rating. After obtaining the 85 pairs rated by the law expert, we
combine them with the remaining data, totaling 420 pairs. The mean score of
the data is 1.844 with the standard deviation of 1.882.

For the analysis of the technical similarity, we utilize IPC codes in the
patent documents. As described in 3.2., we adopt the three-depth structure,
meaning that we use the code up to sub-class. For instance, if a patent
consists of G06F40/30, G06F40/40, and G06F40/56 codes, we regard the patent as
having one IPC code, rather than three. Accordingly, we preprocess the codes.
The total number of patents is 840, ranging from 1 to 6. The mean number of
IPC codes per patent is 1.602 with a standard deviation of 0.812, as
illustrated in Figure 3.

Figure 3: Frequency of patents depending on the number of IPC codes per patent

###  4.2 Evaluation metrics

We use Pearson’s correlation coefficient and Spearman’s one as indicators to
evaluate the experimental results. Pearson’s correlation coefficient is a
measure of the linear correlation between two variables. Pearson’s correlation
coefficient is calculated byEquation (8).

$\displaystyle\begin{array}[]{@{}l}r=\displaystyle\frac{n\left(\sum
xy\right)-\left(\sum x\right)\left(\sum y\right)}{\sqrt{[n\sum x^{2}-(\sum
x)^{2}]\ast[n\sum y^{2}-(\sum y){}^{2}]\;}}\end{array}$ (8)

On the other hand, the Spearman’s correlation coefficient is a method of
calculating the correlation coefficient using the rank of two values rather
than the actual value to calculate the correlation coefficient. Spearman’s
correlation coefficient is obtained byEquation (9).

$\displaystyle\begin{array}[]{@{}l}r=1-\displaystyle\frac{6\sum
D^{2}}{n(n-1)}\end{array}$ (9)

###  4.3 Result

In this section, we report the experiment results. As illustrated in Table 3,
the proposed model shows the best performance among the five models based on
the Pearson’s and Spearman’s correlation coefficients, followed by the BERT
model. This verifies that the proposed model considering both semantic and
technological similarities outperforms any other models that incorporate the
semantic similarity only.

Table 3: Model performance |  |
---|---|---
Model |  Pearson |  Spearman
CNN |  0.074951 |  0.035229
LSTM |  0.05409 |  0.05967
Bi-LSTM |  0.14646 |  0.185534
BERT |  0.507742 |  0.542219
Proposed model |  0.56935 |  0.644496

##  5 Discussion

We propose a hybrid method to measure the patent similarity in terms of both
semantic and technical aspects. The experiment result demonstrates that the
proposed model demonstrates the best performance among the five models in
Table 3. This champions the necessity of incorporating both semantic and
technological similarities in measuring the patent similarity. In this
section, we discuss applications of the proposed model and possible future
works.

The proposed method can be an effective starting-off tools for invention,
especially at the beginning stage of it, by solving a big list of similar
problems. By the virtue of this property, recommendation systems for patents
can be built. In addition, many applied studies can be conducted using the
proposed method; for example, it can respond to patent trolls that have
recently become a hot topic. Moreover, plagiarism between patents can be
semantically and technically analyzed through the proposed similarity
measurement. It is also possible to identify technology trends by drawing a
patent road map based on similar patents.

Although the proposed model is the best model among the five models, ways to
improve the model performance merit discussion. With regard to the semantic
similarity, we apply a vanilla BERT to patent abstracts with titles based on
the length of patent abstracts. Still, we can improve the model performance by
using variants of BERT. For instance, we can apply D2SBERT (Yoo et al., 2023
[29]) to longer documents that may contain more information. Moreover, we can
use a more lightweight BERT model, like DistilBERT (Sanh et al., 2019 [30]),
for distribution. Regarding the technical similarity, we can improve the model
performance by incorporating CPC codes, as well as IPC codes, and/or the
bibliographic information of the patent. All of these are left for future
work.

##  6 Conclusion

Measuring the patent similarity is a crucial step for innovation. In a rapidly
changing world, it should be performed in an accurate, but fast manner. In
order to improve reliability and immediacy, we devise a hybrid method to
measure the patent similarity considering both semantic and technological
similarities between patents. For this, we establish a BERT-based model that
measures both semantic similarity with patent abstracts including titles and
technological similarity with IPC codes in the patent documents. The
experiment result verifies the effectiveness of the proposed method. The
proposed model features sublimity in that it satisfies brevity,
comprehensiveness, and versatility, but this can be even better by conducting
the possible future works discussed.

## Declaration of competing interest

The authors declare no conflict of interest.

## Funding

This work was supported in part by the Business Agency, South Korea under
Grant CY220049.

## Acknowledgements

The authors acknowledge the helpful comments and suggestions by the anonymous
reviewers and the guest editor.
Author Contributions are as follows: Conceptualization, Y.Y.; methodology,
Y.Y; validation, S.G., C.J., Y.Y.; formal analysis, S.G., Y.Y; dataset, C.J.,
Z.S.; Experiment, Y.Y, S.G; writing—original draft preparation, Y.Y., J.L;
writing—review and editing, C.J., Y.Y; visualization, S.G., Y.Y; supervision,
Y.Y, D.S; All authors have read and agreed to the published version of the
manuscript.

## References

  * [1] S. Arts, B. Cassiman, J. C. Gomez, Text matching to measure patent similarity, Strategic Management Journal 39 (1) (2018) 62–84.
  * [2] M. Moehrle, Measures for textual patent similarities: a guided way to select appropriate approaches, Scientometrics 85 (1) (2010) 95–109.
  * [3] A. Rodriguez, B. Kim, M. Turkoz, J.-M. Lee, B.-Y. Coh, M. K. Jeong, New multi-stage similarity measure for calculation of pairwise patent similarity in a patent citation network, Scientometrics 103 (2015) 565–581.
  * [4] K.-K. Lai, S.-J. Wu, Using the patent co-citation approach to establish a new patent classification system, Information processing & management 41 (2) (2005) 313–330.
  * [5] X. Wang, H. Ren, Y. Chen, Y. Liu, Y. Qiao, Y. Huang, Measuring patent similarity with sao semantic analysis, Scientometrics 121 (2019) 1–23.
  * [6] K. A. Younge, J. M. Kuhn, Patent-to-patent similarity: A vector space model, Available at SSRN 2709238 (2016).
  * [7] Y. Lu, X. Xiong, W. Zhang, J. Liu, R. Zhao, Research on classification and similarity of patent citation based on deep learning, Scientometrics 123 (2020) 813–839.
  * [8] H. He, K. Gimpel, J. Lin, Multi-perspective sentence similarity modeling with convolutional neural networks, in: Proceedings of the 2015 conference on empirical methods in natural language processing, 2015, pp. 1576–1586.
  * [9] S. Peng, H. Cui, N. Xie, S. Li, J. Zhang, X. Li, Enhanced-rcnn: an efficient method for learning sentence similarity, in: Proceedings of The Web Conference 2020, 2020, pp. 2500–2506.
  * [10] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018).
  * [11] J.-S. Lee, J. Hsiang, Patentbert: Patent classification with fine-tuning a pre-trained bert model, arXiv preprint arXiv:1906.02124 (2019).
  * [12] S. Garg, G. Ramakrishnan, Bae: Bert-based adversarial examples for text classification, arXiv preprint arXiv:2004.01970 (2020).
  * [13] S. González-Carvajal, E. C. Garrido-Merchán, Comparing bert against traditional machine learning text classification, arXiv preprint arXiv:2005.13012 (2020).
  * [14] Y. Yoo, T.-S. Heo, Y. Park, K. Kim, A novel hybrid methodology of measuring sentence similarity, Symmetry 13 (8) (2021) 1442.
  * [15] J. Choi, Y.-S. Hwang, Patent keyword network analysis for improving technology development efficiency, Technological Forecasting and Social Change 83 (2014) 170–182.
  * [16] J. Yang, Y. Li, C. Gao, Y. Zhang, Measuring the short text similarity based on semantic and syntactic information, Future Generation Computer Systems 114 (2021) 169–180.
  * [17] F. Ahmad, M. Faisal, A novel hybrid methodology for computing semantic similarity between sentences through various word senses, International Journal of Cognitive Computing in Engineering 3 (2022) 58–77.
  * [18] G. Cascini, M. Zini, Measuring patent similarity by comparing inventions functional trees, in: Computer-Aided Innovation (CAI) IFIP 20th World Computer Congress, Proceedings of the Second Topical Session on Computer-Aided Innovation, WG 5.4/TC 5 Computer-Aided Innovation, September 7-10, 2008, Milano, Italy, Springer, 2008, pp. 31–42.
  * [19] S. Feng, The proximity of ideas: An analysis of patent text using machine learning, PloS one 15 (7) (2020) e0234880.
  * [20] H. Noh, Y. Jo, S. Lee, Keyword selection and processing strategy for applying text mining to patent analysis, Expert Systems with Applications 42 (9) (2015) 4348–4360.
  * [21] J. Joung, K. Kim, Monitoring emerging technologies for technology planning using technical keyword based analysis from patent data, Technological Forecasting and Social Change 114 (2017) 281–292.
  * [22] A. I. Kadhim, Survey on supervised machine learning techniques for automatic text classification, Artificial Intelligence Review 52 (1) (2019) 273–292.
  * [23] M. Farouk, Measuring text similarity based on structure and word embedding, Cognitive Systems Research 63 (2020) 1–10.
  * [24] Z. Dong, Q. Dong, Hownet-a hybrid language and knowledge resource, in: International conference on natural language processing and knowledge engineering, 2003. Proceedings. 2003, IEEE, 2003, pp. 820–824.
  * [25] F. Lan, Research on text similarity measurement hybrid algorithm with term semantic information and tf-idf method, Advances in Multimedia 2022 (2022).
  * [26] D. Viji, S. Revathy, A hybrid approach of weighted fine-tuned bert extraction with deep siamese bi–lstm model for semantic text similarity identification, Multimedia Tools and Applications 81 (5) (2022) 6131–6157.
  * [27] N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, arXiv preprint arXiv:1908.10084 (2019).
  * [28] A. V. Giczy, N. A. Pairolero, A. A. Toole, Identifying artificial intelligence (ai) invention: A novel ai patent dataset, The Journal of Technology Transfer 47 (2) (2022) 476–505.
  * [29] Y. Yoo, T.-S. Heo, D. Lim, D. Seo, Multi label classification of artificial intelligence related patents using modified d2sbert and sentence attention mechanism, arXiv preprint arXiv:2303.03165 (2023).
  * [30] V. Sanh, L. Debut, J. Chaumond, T. Wolf, Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, arXiv preprint arXiv:1910.01108 (2019).

## Author biography

Yongmin Yoo was born in 1991 in Busan, Republic of Korea. He received a
master’s degree in Industrial Engineering from Inha university, Korea. He is
currently a PhD candidate in University of Auckland, New Zealand. He has
experience working as a Natural Language Processing researcher at NHN, one of
the biggest companies in Korea. His research interests are in Technological
Management, Data Mining, especially Text Mining, Deep Learning, Machine
Learning and Natural Language Processing.

Cheonkam Jeong was born in 1990 in Seoul, Republic of Korea. She earned both
B.A. and M.A. degrees in English Linguistics from Hankuk (Korea) University of
Foreign Studies in Seoul. She is currently a PhD candidate in Linguistics at
the University of Arizona in the United States. She also holds a M.S. degree
in Human Language Technology, which was earned en route to her PhD. She has
experience working as a Natural Language Processing Engineer and Speech
Specialist for several projects, one of which is sponsored by the U.S. Defense
Advanced Research Projects Agency. Her research interests lie in Natural
Language Processing and Automatic Speech Recognition/Synthesis.

Sanguk Gim was born in 1991 in Changwon, Republic of Korea. He received a
bachelor’s degree in Psychology from Kyungnam University. He worked as a Data
Analyst at NHN Diquest. He currently works at SR Universe as a Deep Learning
Engineer.

His research interests are in deep learning, machine learning and natural
language processing.

Junwon Lee was born in 1988 in Gangwon, Republic of Korea. He received a
bachelor’s degree in Business Administration from Kookmin University. He is
currently studying for a master’s degree in Data Science at Royal Melbourne
Institute of Technology, Melbourne, Australia. He has experience working as a
Field Incident Statistical Data Analyst at Group Renault, one of the biggest
vehicle companies. His research interests include Data Mining, more
specifically Text mining.

Zachary Schimke was born in California in the United States. He earned his
B.S. in Mechanical Engineering from California State University Northridge. He
is currently a J.D. candidate at the University of Arizona James E. Rogers
College of Law. He has been a member of the School’s Mock Trial and Patent
Moot court teams. He is interested in Intellectual Property as well as Patent
Prosecution and Litigation.

Deaho Seo was born in 1991 in Seoul, Republic of Korea. He received a B.S
degree in Information Systems and a M.S degree in Industrial Engineering from
Hanyang University. He is a Ph.D candidate in School of Information, at Yonsei
University. He has experience working as a Big Data rSsearcher at the Korea
Advanced Institute of Science and Technology and the Korea Electronics
Technology Institute. Currently, he is serving as the CEO of Elesther. His
research interests include Text Mining, Vision-based Anomaly Detection, Start
Factory, and E-commerce solution. He received the Best Young Entrepreneur
Award from the Ministry of SMEs and Startups in Korea. He has published 8
books related to artificial intelligence and big data.