# Synergizing Data Imputation and Electronic Health Records for Advancing
Prostate Cancer Research: Challenges, and Practical Applications.

Abderrahim O. Batouche1,2,3,∗, Eugen Czeizler2,3,∗, Miika Koskinen4, Tuomas
Mirtti2,5 and Antti S. Rannikko2,6
1Doctoral Programme in Computer Science, University of Helsinki, Helsinki,
Finland
2Research Program in Systems Oncology, University of Helsinki, Helsinki,
Finland
3ICAN-Digital Precision Cancer Medicine Flagship, Helsinki, Finland
4HUS Helsinki University Hospital, Helsinki, Finland
5Department of Pathology, University of Helsinki and Helsinki University
Hospital, Helsinki, Finland
6Department of Urology, University of Helsinki and Helsinki University
Hospital, Helsinki, Finland
∗These authors contributed equally to this work.
{abderrahim.batouche<EMAIL_ADDRESS>{miika.koskinen,
tuomas.mirtti<EMAIL_ADDRESS>

###### Abstract

The presence of detailed clinical information in electronic health record
(EHR) systems presents promising prospects for enhancing patient care through
automated retrieval techniques. Nevertheless, it is widely acknowledged that
accessing data within EHRs is hindered by various methodological challenges.
Specifically, the clinical notes stored in EHRs are composed in a narrative
form, making them prone to ambiguous formulations and highly unstructured data
presentations, while structured reports commonly suffer from missing and/or
erroneous data entries. This inherent complexity poses significant challenges
when attempting automated large-scale medical knowledge extraction tasks,
necessitating the application of advanced tools, such as natural language
processing (NLP), as well as data audit techniques. This work aims to address
these obstacles by creating and validating a novel pipeline designed to
extract relevant data pertaining to prostate cancer patients. The objective is
to exploit the inherent redundancies available within the integrated
structured and unstructured data entries within EHRs in order to generate
comprehensive and reliable medical databases, ready to be used in advanced
research studies. Additionally, the study explores potential opportunities
arising from these data, offering valuable prospects for advancing research in
prostate cancer.

##  1 INTRODUCTION

Prostate cancer (PCa) is a prevalent disease known for its indolent nature,
often characterised by slow development and protracted progression over time
[Institute, 2023, Society, 2023]. As such, one specific challenge in
performing medical research pertaining to PCa is dealing with incomplete
medical records and missing data, e.g., as a result of city relocation or
disease follow-up across different health providers. This, in turn, can hinder
the results of ongoing research studies analysing the effectiveness of
diagnoses and various treatment planning approaches [Holmes et al., 2021].
Ultimately, this can affect clinical decision-making and the patient’s well-
being.

To overcome the limitations of incomplete and/or erroneous data, Electronic
Health Records (EHRs) mining has emerged as a crucial approach in medical
research as well as within clinical practice[Yadav et al., 2018]. EHRs mining
leverages advanced data analytic and artificial intelligence (AI) approaches
to extract valuable insights from vast amounts of patient data[Ajmal et al.,
2023, Javaid et al., 2022]. By identifying patterns, trends, and risk factors
associated with prostate cancer, EHRs mining facilitates the early detection
of advanced diseases and the personalisation of treatment strategies [Knighton
et al., 2016, Seneviratne et al., 2018, Henkel et al., 2022]. However,
challenges such as missing data and data security must be addressed to ensure
patient information remains complete, confidential, and secure. Additionally,
the lack of interoperability between different EHR systems poses hurdles in
data sharing and aggregation, limiting the full potential of mining for both
prostate cancer research, as well as for general improvement of patient care
[De La Torre-Díez et al., 2013]. Overcoming these issues and promoting
standardised data collection practices and protocols will be pivotal in
advancing the field of PCa treatment through EHRs mining[Herp et al., 2023],
as well as the overall medical research in general.

In our work, we have designed and developed a data preprocessing pipeline that
can leverage routinely collected information from our EHRs (HUS Datalake
[Oscar Bruck, 2023, Tietoevry.com, 2023, Misukka, 2022]) to efficiently and
accurately retrieve and consolidate clinicians’ work on PCa treatment
analysis. Using Microsoft Azure machine learning studio and batches from HUS
datalake that are available at the HUS Acamedic environment (a secure scalable
data analytics platform developed for medical research [kuorttinen, 2023]), we
developed an EHR mining pipeline using Python libraries to read, process, and
provide curated data for further research applications.

One of the key clinical inputs exhibiting missing entries within the EHR of a
significant number of PCa patients is the occurrence of curative treatment,
i.e., radical prostatectomy (RP) or radiation treatment (RT). Since imputation
of such missing data is inevitable, we had to use a different approach to
uncover these lost data entries. Using routinely collected values of the
prostate-specific antigen (PSA) lab measurements, we were able to successfully
identify and even classify curative PCa treatments. To our knowledge, this is
the first attempt to approach the inference of EHR missing treatment records
through PSA time series data. Our approach enabled us to enhance our EHR by
incorporating approximately 2.8 thousand new curative treatment events,
marking a notable 27% growth compared to the treatment events available
beforehand. The explanation for this relatively large increase is multi-
folded. Some patients might have been treated outside the (Helsinki and
Uusima) district unit whose database our study is based upon. Others might
have been treated within private practice units, which again are not covered
by our database. Finally, we can assume that a proportion of these missing
treatment events are due to human error in correctly recording them within the
EHR.

Another key clinical information (as well as key surrogate measurement within
medical research analysis) which is most of the times not directly recorded
within EHRs, either in structured or non-structured format, is the time
instant when PCa patients are classified as having a biochemical recurrence
(BCR). After primary cancer treatment, BCR is achieved when the PSA level in
the blood surpasses a certain threshold, thus indicating that the disease may
be returning or progressing. Thus, BCR status is an important indicator both
clinically, as it signs that further monitoring or treatment may be needed to
manage the condition [Stephenson et al., 2006, Artibani et al., 2018], and
from a (medical) data analysis perspective, as it is a surrogate for PCa
mortality [Zhao et al., 2022, Artibani et al., 2018]. By following the PSA
measurements as well as all EHR-available PCa treatment records we were able
to effectively determine (and report) the status and timing of BCR for all PCa
patients.

##  2 METHODS

###  2.1 Data source

Our pipeline starts by identifying patients of interest within a large
academic EHR system (Figure 1). We used the Finnprostate dataset, which is a
large patient registry study combining Finnish national healthcare data with
local hospital data (n=700,000) of men suspected of having PCa (PSA measured)
or diagnosed with PCa. From Finnprostate, we gathered a HUS (Hospital District
of Helsinki and Uusimaa) sub-cohort of men (n=326,796) having comprehensive
patient information regarding out-patient clinic and hospital visits as well
as data regarding laboratory tests, medication prescriptions, radiological,
pathological, and surgical reports, as well as comorbidities covering the
years 1993 to 2019. The above data is embedded within the regional HUS
Acamedic datalake.

Figure 1: Data preprocessing pipeline for Prostate Cancer research data.

Medical research commonly encounters missing data. Despite this prevalence, it
is nowadays generally accepted to perform various data analysis tasks on
partially incomplete records, as long as the missing values are not
substantial, and the analysis methods themselves can cope with specific
uncertainties. Moreover, the use of advanced imputation techniques such as
maximum likelihood [Wald, 1949], multiple imputation[Schafer, 1999], or
Bayesian methods [Kong et al., 1994] have a good track record in addressing
many of the missing data entries. However, certain complex missing data
records, such as the moment and type of a deployed treatment, or the first
diagnostic biopsy of a tumour and its aggressiveness, are very hard to be
addressed by any of the available computational imputation methods.

In our data processing work (yellow box, Figure 1), imputation was reinforced
with customised algorithms that rely on clinical guidelines, experts’
interpretations, as well as the intrinsic information redundancy available
within EHR, in order to retrieve the missing data. All created algorithms are
described in Table 1.

###  2.2 Missing curative treatments detection

The Treatment Detection Algorithm (DTX_algo) plays a pivotal role in enhancing
our data quality by identifying and incorporating missing curative treatment
records (in Algorithm 4). The algorithm takes all patient’s $data$ as an input
and returns a list of missing curative treatments.

The Significant PSA Drop Algorithm (SIGDROP) constitutes the initial phase of
DTX, meticulously tracking a patient’s PSA values subsequent to their
diagnostic biopsy (Algorithm 1). The algorithm takes PSA measurements of
patient $i$, and returns, if any:

  * •

$drop\\_date$: The date of the PSA drop, which is the highest (maximum) point
from where a significant PSA drop starts; is subsequently considered as a
treatment date.

  * •

$nadir\\_date$: The date of the PSA nadir, which is the lowest (minimum) point
to where the significant drop reached.

  * •

$PSA_{min}$: The minimum values (at the time $nadir\\_date$); this value is
used to classify the drop into RP or RT.

The algorithm’s operation commences with the pursuit of the maximum PSA value
($PSA_{max}$, line 3-4), followed by an endeavour to identify the minimum
value within the ensuing $\delta\leq$ 12-month period (lines 5-32). Upon
successful identification of a decreasing value, at lines 15-16, the algorithm
calculates $\alpha$, which is the drop percentage that undergoes rigorous
testing to ascertain its adherence to predetermined significance conditions
(line 17). This process is indispensable in establishing the genuineness of
the observed drop and confirming its clinical significance.

Table 1: Summary of Algorithms Algorithm Name | Input | Output | Complexity |  Short Description
---|---|---|---|---
SIGDROP | $PSA_{i}$ |  $drop\\_date$, $nadir\\_date$, $PSA_{min}$ | $O(M)$ |  Detects significant PSA drop and related dates.
DTX | $PATIENTS\\_LIST$ | $L$ | $O(M*N)$ |  Detects missing treatments based on PSA data.
CRT | $p_{i}$ | $d_{m1}$ | $O(1)$ |  Detects Clinical Relapse after RT
CRP | $p_{i}$ | $d_{m1}$ | $O(1)$ |  Detects Clinical Relapse after RP
PRP | $p_{i}$ | $d_{m2}$ | $O(N)$ |  Detects PSA Relapse after RP
PRT | $p_{i}$ | $d_{m2}$ | $O(N)$ |  Detects PSA Relapse after RT
DBCR | $TREATED\\_PATIENTS$ | $L_{bcr}$ | $O(M*N)$ |  Main algorithm for BCR detection.
\- RP=Radical prostatectomy, RT=Radiation therapy, BCR=Biochemical recurrence. |
\- In M*N M is the number of PSA measurments and N is the number of patients |

0: $PSA_{i}$

0: $drop\\_date$, $nadir\\_date$, $PSA_{min}$

1: $M\leftarrow size(PSA_{i})$

2: if $M\geq 0$ then

3: $PSA_{max}\leftarrow PSA_{i}[1]$

4: $date\\_PSA_{max}\leftarrow getDate(PSA_{max})$

5: for $j=1$ to $M-1$ do

6: $e\leftarrow PSA_{i}[j]-PSA_{i}[j+1]$

7: $\delta\leftarrow date\\_PSA_{next}-date\\_PSA_{max}$

8: if $e\leq 0$ then

9: $date\\_PSA_{next}\leftarrow getDate(PSA_{i}[j+1])$

10: if ($PSA_{max}<PSA_{i}[j+1]$) or $\delta>12m$ then

11: $PSA_{max}\leftarrow PSA_{i}[j+1]$

12: $date\\_PSA_{max}\leftarrow getDate(PSA_{max})$

13: end if

14: else

15: $\beta\leftarrow PSA_{max}-PSA_{i}[j+1]$

16: $\alpha\leftarrow\frac{\beta}{PSA_{max}}$

17: if ($\alpha\geq 0.75$ and $\beta\geq 3$) or ($\alpha\geq 0.5$ and
$\beta\geq 4$) then

18: $PSA_{min}\leftarrow PSA_{i}[j+1]$

19: else

20: if $\delta>12m$ then

21: $PSA_{max}\leftarrow PSA_{i}[j+1]$

22: $date\\_PSA_{max}\leftarrow getDate(PSA_{max})$

23: else

24: $\gamma\leftarrow date\\_PSA[j+2]-date\\_PSA_{max}$

25: if $j+2\leq M$ and $\gamma>12$ then

26: $PSA_{max}\leftarrow PSA_{i}[j+1]$

27: $date\\_PSA_{max}\leftarrow getDate(PSA_{max})$

28: end if

29: end if

30: end if

31: end if

32: end for

33: end if

34: if $PSA_{min}$ exists then

35: $drop\\_date\leftarrow get\\_date(PSA_{max})$

36: $nadir\\_date\leftarrow get\\_date(PSA_{min})$

37: return $drop\\_date,nadir\\_date,PSA_{min}$

38: end if

39: return $NULL$

Algorithm 1 SIGDROP - Significant PSA drop detection

Having validated the drop as significant, and (line 6) with no EHR-recorded
curative treatment between the date of drop ($d_{max}$) and the date of the
nadir ($d_{min}$), DTX proceeds to collate all such identified drops,
systematically categorising them into two distinct treatment modalities:
radiation therapy (RT) and radical prostatectomy (RP) (Algorithm 2 line 7-10).
This classification not only facilitates comprehensive treatment record
augmentation but also provides valuable missing insights into the patient’s
therapeutic journey.

###  2.3 Biochemical recurrence detection

Biochemical recurrence (BCR) serves as a crucial indicator for PCa mortality.
However, its availability in EHRs is not always guaranteed. In such cases,
various methods can be employed to retrieve and impute this information. Our
Detect Biochemical Recurrence (DBCR_algo) Algorithm is specifically designed
to analyse data from treated patients, identifying potential relapses and
categorising patients as either having experienced a BCR or not (Algorithm 7).
To achieve this outcome, DBCR utilises four (04) distinct functions, each
tailored to a specific task.

Clinical guidelines governing PSA relapse are stringent and clearly defined
[Van Den Broeck et al., 2020], and these guidelines are meticulously
integrated into the PRP and PRT functions (Algorithms 5-6).

  * •

PSA-based relapse after radical prostatectomy $PRP(p_{i})$: this function uses
the European Association of Urology (EAU) guidelines [Van Den Broeck et al.,
2020] to detect whether a PSA-based relapse occurred after radical
prostatectomy. If an ultrasensitive PSA [Shen et al., 2005] measurement
$psa_{j}$ was taken for patient $p_{i}$ then we take this into consideration
to define the maximum threshold (lines 3-7).

  * •

PSA-based relapse after radiation therapy $PRT(p_{i})$: this function is also
using the EAU guidelines [Van Den Broeck et al., 2020] to detect whether a
PSA-based relapse occurred after radiation therapy. The algorithm searches for
the first increase of 2 PSA units from a nadir value.

Going beyond this, our novel BCR detection method is not solely reliant on PSA
relapse; instead, it incorporates expert knowledge and translates it into a
new tool for detecting BCR based on secondary treatments (Figures 2 and 3).
The CRP and CRT functions (Algorithms 2-3) have been developed to identify
possible relapses that may have been missed (after an RP or an RT primary
treatment, respectively) either due to the absence of PSA tests or because the
curating doctor decided on a secondary treatment before the PSA value has
actually crossed the EAU-guideline threshold. The exact approaches used to
define clinical relapse after RP and RT primary treatments are described in
Figure 2 and Figure 3, respectively.

Figure 2: BCR definition after radical prostatectomy. Figure 3: BCR definition
after radiation therapy.

0: $p_{i}$

0: $d_{m1}$

1: $L\leftarrow[\;]$

2: if $lastRTDate(p_{i})>firstRPDate(p_{i})$ then

3: if $lastRTDate(p_{i})-firstRPDate(p_{i})>1yr$ then

4: $L\leftarrow L+firstRTDateAfterOneYear(p_{i})$

5: end if

6: if $hasHTCT(p_{i})$ and $lastHTCTDate(p_{i})>firstRPDate(p_{i})$ then

7: if $lastHTCTDate(p_{i})-firstRPDate(p_{i})\geq 2yr$ then

8: $L\leftarrow L+firstHTCTDateAfterOneYear(p_{i})$

9: end if

10: end if

11: else

12: if $hasHTCT(p_{i})$ and $lastHTCTDate(p_{i})>firstRPDate(p_{i})$ then

13: $L\leftarrow L+firstHTCTDateAfterRp(p_{i})$

14: end if

15: end if

16: $d_{m1}\leftarrow getMin(L)$

17: return $d_{m1}$

Algorithm 2 CRP - Clinical Relapse after RP

0: $p_{i}$

0: $d_{m1}$

1: $L\leftarrow[\;]$

2: if $hasRP(p_{i})$ and $lastRPDate(p_{i})>firstRTDate(p_{i})$ then

3: $L\leftarrow L+firstRpDateAfterRt(p_{i})$

4: end if

5: if $hasSecondRT(p_{i})$ and $secondRTDate(p_{i})-firstRTDate(p_{i})>1yr$
then

6: $L\leftarrow L+secondRTDate(p_{i})$

7: end if

8: if $hasHTCT(p_{i})$ and $firstHTCTDate(p_{i})-firstRTDate(p_{i})\geq 6m$
then

9: $L\leftarrow L+firstHTCTDate(p_{i})$

10: end if

11: if $hasHTCT(p_{i})$ and $firstHTCTDate(p_{i})-firstRTDate(p_{i})>3yr$ then

12: $L\leftarrow L+firstHTCTDate(p_{i})$

13: end if

14: $d_{m1}\leftarrow getMin(L)$

15: return $d_{m1}$

Algorithm 3 CRT - Clinical Relapse after RT

The DBCR Algorithm then uses all the outputs of the above functions, namely
the dates ($d_{1},d_{2}.d_{3},d_{4}$) of possible BCR occurences, and selects
the earliest date (if exists) as the date of biochemical recurrence for
patient $p_{i}$ (Algorithm 7 lines 7-10).

###  2.4 Evaluation

Retrieving missing data is of utmost importance in the pre-processing of EHR
data for critical and sensitive applications. Additionally, assessing the
quality of imputed data holds significant value as it provides insights into
the effectiveness of the methods and algorithms employed. In our study, data
evaluation involves a two-tier validation process. The first level (a.k.a.
’step-1’ evaluation) employs automated tests, where we verify the accuracy of
our algorithms by taking records without missing treatment data, applying the
imputation algorithm, and subsequently scrutinising the outcomes. The second
level (a.k.a. the ’step-2’ evaluation) entails expert validation, wherein a
random selection of imputed data is manually inspected by domain experts,
ensuring its correctness.

##  3 RESULTS

###  3.1 Curated database

The initial phase of this work was to explore the HUS datalake and extract the
most accurate and comprehensive data suitable for subsequent medical research
applications. As a result, we successfully created a structured and curated
database that contains crucial patient information, as defined in Table 2.

Table 2: The curated data tables Data | Number of rows (%) | Number of Patients (%)
---|---|---
T1: Pathology | 23,393 | 12,277
GG1 | 6618 (28) | 3652 (30)
GG2 | 6383 (27) | 3313 (27)
GG3 | 4747 (20) | 2503 (20)
GG4 | 2310 (10) | 1195 (10)
GG5 | 3335 (14) | 1614 (13)
T2: Treatment | 40,369 | 9800
RP | 2743 (7) | 2742
RT | 18,254 (45) | 7248
HT | 15,804 (39) | 4088
CT | 3568 (9) | 514
T3: PSA | 1,424,440 | 238,399
T4: MRI | 20,103 | 15,807
T5: Medications | 13,837,600 | 290,055
GG1–GG5 = Gleason grade group 1–5 (associated to each pathological entry)
RP=Radical prostatectomy, RT=Radiation therapy,
HT=Hormonal therapy, CT=Chemotherapy.

###  3.2 Treatments data

Table 3: Evaluation of DTX algorithm performance - | Available CTx | Estimated CTx | Correct estimated CTx | New estimated CTx
---|---|---|---|---
- | DB | DTX | DTX $\cap$ DB | True-Class | False-Class | DTX $\setminus$ DB
PID | 7563 | 9725 | 6962 (92%) | 6294 (90%) | 668 (10%) | 2763 (+27%)
PID-RP | 2495 | 2722 | 2233 (90%) | 1929 (86%) | 304 (14%) | 0489 (+16%)
PID-RT | 5068 | 7003 | 4729 (93%) | 4365 (92%) | 364 (08%) | 2274 (+31%)

Following the data curation and structuring, we have implemented the DTX
algorithm in order to detect and impute the missing curative treatment data.
As a result, our database now incorporates n=2763 new PCa-related treatment
records, representing a 27% increase compared to the original data found in
the HUS datalake. The number of patients with RP has increased by 16% (n=489),
while the number of those with RT has increased by 31% (n=2274).

In Table 3 we present the results of ’step-1’ DTX performance evaluation,
i.e., estimated vs. known (EHR-available) treatment records. We record an
imputation performance of 92% (n=6962) correct estimated curative treatments,
i.e., treatments estimated using the DTX algorithm that are also found in the
existing database. Out of these, 90% (n=6294) are correctly classified as RP
or RT, whereas 10% (n=669) are wrongly classified. RP classification was 86%
correct, whereas RT classification reached 92%.

The ’step-2’ evaluation of the DTX algorithm was performed vs. manual
validation by domain experts, where the experts were using the entire
collection of unstructured reports associated with the test subjects in order
to uncover their treatment history. The ’step-2’ evaluation started by
sampling 40 random patients, i.e., 20 random RP + 20 random RT, that were
detected by the algorithm as having curative treatments (CTx), however this
treatment did not appear within the EHR (DTX $\setminus$ DB in Table 3). The
results of this manual validation are summarised in Table 4. Only one patient
from the RP group was unverifiable (no data = treatment cannot be confirmed),
while five RT patients had the same situation. In addition, 95% RP patients
were confirmed to have a curative PCa treatment, and 60% RT patients were
confirmed. In total 79% of the sampled patients (whose treatments were not
recorded within EHR) were confirmed to have PCa curative treatment.

Table 4: Manual validation for DTX algorithm performance - | Sample | Unverifiable | Verifiable | True CTx | All True CTx
---|---|---|---|---|---
PID-RP | 20 | 01 | 19 (95%) | 18 (95%) | 27 (79%)
PID-RT | 20 | 05 | 15 (75%) | 09 (60%)

###  3.3 BCR data

Our DBCR algorithm successfully identified 2851 patients (Figure 4-5) who
developed a biochemical recurrence after a curative treatment (27% of the
treated patients). Among them, 70% (n=2007) were detected using the PRP and
PRT methods in accordance with the EAU guidelines [Van Den Broeck et al.,
2020], while 30% (n=844) were identified using our new algorithms (CRP, CRT),
which might have otherwise gone unnoticed.

Figure 4: BCR detected data: The distribution of time to relapse. Figure 5:
BCR detected data: The time to relapse by Gleason grade group.

###  3.4 Applications

After cleaning the data and improving its quality, we have successfully
carried out multiple applications. The primary application involved developing
a data visualisation tool, enabling clinicians and researchers to visualise
the trajectory of PCa patients, including their PSA values, treatments,
pathological results, medical prescriptions, and others (Figure 6).

Figure 6: Helsinki BCR system enabling access to patients’ trajectory and the
BCR prediction models.

Additionally, we have investigated the potential grade inflation in PCa
Gleason grade groups during the MRI era; the research focused on patients in
Gleason grade groups 1 and 2. The hypothesis of this work was that some
patients in grade group 1 in the pre-MRI era are nowadays, in the MRI era,
classified and treated as grade group 2 patients. With enough evidence, the
work proved the hypothesis which will open serious discussions to reassess
current risk stratification tools and clinical decision-making. Updating
guidelines on cancer grading and treatments is crucial to be aligned with the
precision of modern MRI technology.

Furthermore, we are utilising the curated EHR data to train machine learning
models to predict biochemical recurrence within the following 3-to-10 years
from initial curative treatment. Knowing that prostate cancer is a slow-
developing cancer, BCR is one of the most important and accurate surrogates to
prostate cancer mortality. Therefore, predicting BCR would have a significant
impact on treatment decisions and treatment planning. Our (preliminary)
trained models achieved good performance (Accuracy=0.93, AUC=0.93,
Precision=0.88) on an internal validation. The models are trained on n=5262
patients who have had PCa curative treatment.

##  4 DISCUSSION

Ensuring high data quality is essential when building effective AI models and
conducting significant statistical analyses. This importance is particularly
heightened in clinical research and applications where decisions may directly
impact patients’ lives. Electronic Health Records (EHR), like HUS Datalake,
play a critical role in this process, making it imperative to develop robust
exploration methods to harness the available data.

In our work, we explored, curated, and augmented bio-medical data from within
Finnish healthcare records, with a specific focus on prostate cancer patients.
By establishing a new mining framework and developing novel analysis
algorithms, we successfully consolidated our data, enabling us to conduct
meaningful and impactful medical research. One of our approaches was to use
the time series data on patients’ PSA levels, a subset of medical data which
is typically well collected and curated within EHR, in order to infer the
existence, and the type, of EHR missing curative treatment events. To our
knowledge, this is the first time PSA time series data were used in this way,
although, in [Bettencourt-Silva et al., 2015], the authors employed a similar
approach in order to generate a completeness score for the overall data
quality of the cohort. Based on this approach, we were able to consolidate our
EHR by adding approx. $2.8$k new curative treatment events, representing a
$27\%$ increase from the EHR-available treatment events.

Another important outcome of our mining framework was documenting the status
and timing of our PCa patients’ BCR. Differently than in previous EHR mining
frameworks for PCa medical data, see e.g. [Park et al., 2021b, Park et al.,
2021a], we define BCR-status based on both PSA-level measurements (after
primary curative treatment, i.e., radiation therapy –RT– or radical
prostatectomy –RP–) as well as based on secondary curative and adjuvant
therapies, i.e., PCa related hormonal- and/or chemotherapy. This approach
takes into consideration the clinical reality that sometimes, curating doctors
decide on secondary therapies before the PSA level crosses the threshold
established by current EAU guidelines as the BCR level. Using this approach,
we accurately captured an additional 844 BCR events (representing a $42\%$
increase from PSA-only detected BCR events), which otherwise would either not
have been found at all or would have been given a significant later time-
stamp.

One important observation from our EHR data curation and analysis work is that
there exists a large amount of redundancy in these data sources. This is
particularly observable within the free text input written by doctors during
their medical checkups and/or lab, pathological, or imaging reports. On the
other hand, due to a multitude of factors, including human error, focusing on
only one particular type of data source at a time, such as lab results,
pathological reports, or even surgery records, one encounters a significant
amount of missing data entries. Thus it becomes both feasible and greatly
beneficial to use the data redundancy feature of EHR in order to ”recuperate”
these missing data entries. This is why, a ”data investigation” approach, such
as the one described in this manuscript, is more relevant than classical ”data
imputation” methods. Indeed, these latter approaches provide only average-like
behaviours and also are completely inefficient in detecting missing events,
such as a radiation treatment event altogether missing from within the EHR.

Strongly connected to the above reasoning, one could not overlook the
potential impact the use of Large Language Models (LLM) could have in
detecting and augmenting the existing EHR data [Thirunavukarasu et al., 2023].
Such models could be employed to extract (from the free text provided by
doctors) relevant information such as missing events, e.g. treatments
performed in different clinics, cities, or even countries, or information that
is usually not structurally recorded within EHR, e.g., family history, use of
alcohol and tobacco products, general health status of the patient, etc.
During the current EHR data analysis no LLM was employed; however, the
approach is currently actively analysed for future usage within our models.

##  5 CONCLUSION

This work demonstrates the challenges of mining Finnish electronic health
records for prostate cancer (PCa) research, as well as the opportunities it
offers in gaining valuable insights. Our methodology, when applied to the HUS
datalake, enabled the detection of missing treatments and biochemical
recurrences (BCR), which led to a range of clinically relevant findings,
including patients’ timeline histories, the Gleason grade group inflation
finding, and the BCR classification models. The results of our framework
highlight the potential of EHR data mining to advance PCa research and guide
personalised patient care.

## ACKNOWLEDGEMENTS

This work was supported by grants from the Cancer Society Finland, the Academy
of Finland, Jane and Aatos Erkko Foundation, and State funding for university-
level health research. It is a joint effort of doctoral students, senior
researchers, and clinicians at the University of Helsinki and the University
Hospital of Helsinki.

## REFERENCES

  * Ajmal et al., 2023 Ajmal, S., Ahmed, A. A. I., and Jalota, C. (2023).  Natural Language Processing in Improving Information Retrieval and Knowledge Discovery in Healthcare Conversational Agents.  Journal of Artificial Intelligence and Machine Learning in Management, 7(1):34–47.
  * Artibani et al., 2018 Artibani, W., Porcaro, A. B., De Marco, V., Cerruto, M. A., and Siracusano, S. (2018).  Management of Biochemical Recurrence after Primary Curative Treatment for Prostate Cancer: A Review.  Urologia Internationalis, 100(3):251–262.
  * Bettencourt-Silva et al., 2015 Bettencourt-Silva, J. H., Clark, J., Cooper, C. S., Mills, R., Rayward-Smith, V. J., and de la Iglesia, B. (2015).  Building data-driven pathways from routinely collected hospital data: A case study on prostate cancer.  JMIR Med Inform, 3(3):e26.
  * De La Torre-Díez et al., 2013 De La Torre-Díez, I., González, S., and López-Coronado, M. (2013).  EHR Systems in the Spanish Public Health National System: The Lack of Interoperability between Primary and Specialty Care.  Journal of Medical Systems, 37(1):9914.
  * Henkel et al., 2022 Henkel, M., Horn, T., Leboutte, F., Trotsenko, P., Dugas, S. G., Sutter, S. U., Ficht, G., Engesser, C., Matthias, M., Stalder, A., Ebbing, J., Cornford, P., Seifert, H., Stieltjes, B., and Wetterauer, C. (2022).  Initial experience with AI Pathway Companion: Evaluation of dashboard-enhanced clinical decision making in prostate cancer screening.  PLOS ONE, 17(7):e0271183.
  * Herp et al., 2023 Herp, J., Braun, J.-M., Cantuaria, M. L., Tashk, A., Pedersen, T. B., Poulsen, M. H. A., Krogh, M., Nadimi, E. S., and Sheikh, S. P. (2023).  Modeling of electronic health records for time-variant event learning beyond bio-markers—a case study in prostate cancer.  IEEE Access, 11:50295–50309.
  * Holmes et al., 2021 Holmes, J. H., Beinlich, J., and Boland, M. R. (2021).  Why Is the Electronic Health Record So Challenging for Research and Clinical Care?  Methods of information in medicine, 60(1-02):32–48.
  * Institute, 2023 Institute, N. C. (2023).  The natural history of prostate cancer.
  * Javaid et al., 2022 Javaid, M., Haleem, A., Singh, R. P., Suman, R., and Rab, S. (2022).  Significance of machine learning in healthcare: Features, pillars and applications.  International Journal of Intelligent Networks, 3:58–73.
  * Knighton et al., 2016 Knighton, A. J., Belnap, T., Brunisholz, K., Huynh, K., and Bishoff, J. T. (2016).  Using Electronic Health Record Data to Identify Prostate Cancer Patients That May Qualify for Active Surveillance.  EGEMS (Washington, DC), 4(3):1220.
  * Kong et al., 1994 Kong, A., Liu, J. S., and Wong, W. H. (1994).  Sequential imputations and bayesian missing data problems.  Journal of the American Statistical Association, 89(425):278–288.
  * kuorttinen, 2023 kuorttinen, E. (2023).  HUS Acamedic - secure operating environment.
  * Misukka, 2022 Misukka, M. (2022).  Standardizing electronic health records in order to advance secondary use of hospital data lakes - A case study on HUS data lake.  Master’s thesis, Aalto University. School of Science.
  * Oscar Bruck, 2023 Oscar Bruck (2023).  HUS Datalake.
  * Park et al., 2021a Park, J., Rho, M. J., Moon, H. W., Kim, J., Lee, C., Kim, D., Kim, C.-S., Jeon, S. S., Kang, M., and Lee, J. Y. (2021a).  Dr. answer ai for prostate cancer: Predicting biochemical recurrence following radical prostatectomy.  Technology in Cancer Research & Treatment, 20.
  * Park et al., 2021b Park, J., Rho, M. J., Moon, H. W., Park, Y. H., Kim, C.-S., Jeon, S. S., Kang, M., and Lee, J. Y. (2021b).  Prostate cancer trajectory-map: clinical decision support system for prognosis management of radical prostatectomy.  Prostate International, 9(1):25–30.
  * Schafer, 1999 Schafer, J. L. (1999).  Multiple imputation: a primer.  Statistical Methods in Medical Research, 8(1):3–15.  PMID: 10347857.
  * Seneviratne et al., 2018 Seneviratne, M. G., Banda, J. M., Brooks, J. D., Shah, N. H., and Hernandez-Boussard, T. M. (2018).  Identifying Cases of Metastatic Prostate Cancer Using Machine Learning on Electronic Health Records.  AMIA … Annual Symposium proceedings. AMIA Symposium, 2018:1498–1504.
  * Shen et al., 2005 Shen, S., Lepor, H., Yaffee, R., and Taneja, S. S. (2005).  ULTRASENSITIVE SERUM PROSTATE SPECIFIC ANTIGEN NADIR ACCURATELY PREDICTS THE RISK OF EARLY RELAPSE AFTER RADICAL PROSTATECTOMY.  Journal of Urology, 173(3):777–780.
  * Society, 2023 Society, A. C. (2023).  Prostate cancer.
  * Stephenson et al., 2006 Stephenson, A. J., Kattan, M. W., Eastham, J. A., Dotan, Z. A., Bianco, F. J., Lilja, H., and Scardino, P. T. (2006).  Defining biochemical recurrence of prostate cancer after radical prostatectomy: A proposal for a standardized definition.  Journal of Clinical Oncology, 24(24):3973–3978.  PMID: 16921049.
  * Thirunavukarasu et al., 2023 Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., and Ting, D. S. W. (2023).  Large language models in medicine.  Nature Medicine, 29(8):1930–1940.
  * Tietoevry.com, 2023 Tietoevry.com (2023).  HUS facilitates clinical data exploitation through data lake.
  * Van Den Broeck et al., 2020 Van Den Broeck, T., Van Den Bergh, R. C., Briers, E., Cornford, P., Cumberbatch, M., Tilki, D., De Santis, M., Fanti, S., Fossati, N., Gillessen, S., Grummet, J. P., Henry, A. M., Lardas, M., Liew, M., Mason, M., Moris, L., Schoots, I. G., Van Der Kwast, T., Van Der Poel, H., Wiegel, T., Willemse, P.-P. M., Rouvière, O., Lam, T. B., and Mottet, N. (2020).  Biochemical Recurrence in Prostate Cancer: The European Association of Urology Prostate Cancer Guidelines Panel Recommendations.  European Urology Focus, 6(2):231–234.
  * Wald, 1949 Wald, A. (1949).  Note on the consistency of the maximum likelihood estimate.  The Annals of Mathematical Statistics, 20(4):595–601.
  * Yadav et al., 2018 Yadav, P., Steinbach, M., Kumar, V., and Simon, G. (2018).  Mining Electronic Health Records (EHRs): A Survey.  ACM Computing Surveys, 50(6):1–40.
  * Zhao et al., 2022 Zhao, Y., Tao, Z., Li, L., Zheng, J., and Chen, X. (2022).  Predicting biochemical-recurrence-free survival using a three-metabolic-gene risk score model in prostate cancer patients.  BMC Cancer, 22(1):239.

## APPENDIX

0: $PATIENTS\\_LIST$

0: $L$

1: $L\leftarrow[\;]$

2: for all $p_{i}$ in $PATIENTS\\_LIST$ do

3: $PSA_{i}\leftarrow getPsa(p_{i})$

4: $Tx_{i}\leftarrow getTreatments(p_{i})$

5: $(d_{max},d_{min},PSA_{min})\leftarrow SIGDROP(PSA_{i})$

6: if $TxExisits(d_{max},d_{min},PSA_{min},Tx_{i})=False$ then

7: if $PSA_{min}<0.1$ then

8: $tx\\_type\leftarrow$ ’RP’

9: else

10: $tx\\_type\leftarrow$ ’RT’

11: end if

12: $L\leftarrow L+(p_{i},tx\\_type,drop\\_date)$

13: end if

14: end for

15: return $L$

Algorithm 4 DTX - Missing treatments detection

0: $p_{i}$

0: $d_{m2}$

1: $PSA\leftarrow getPsaAfterRp(p_{i})$

2: for $psa_{j}$ in $PSA$ do

3: if $usp(psa_{j})=TRUE$ then

4: $th\leftarrow 0.2$

5: else

6: $th\leftarrow 0.4$

7: end if

8: if $psa_{j}>th$ then

9: $d_{m2}\leftarrow getDate(psa_{j})$

10: return $d_{m2}$

11: end if

12: end for

13: return NULL

Algorithm 5 PRP - PSA Relapse after RP

0: $p_{i}$

0: $d_{m2}$

1: $PSA\leftarrow getPsaAfterRt(p_{i})$

2: $nadir\leftarrow getMax(PSA)$

3: for $psa_{j}$ in $PSA$ do

4: if $nadir>psa_{j}$ then

5: $nadir\leftarrow psa_{j}$

6: end if

7: $inc\leftarrow psa_{j}-nadir$

8: if $inc>2$ then

9: $d_{m2}\leftarrow getDate(psa_{j})$

10: return $d_{m2}$

11: end if

12: end for

13: return $NULL$

Algorithm 6 PRT - PSA Relapse after RT

0: $TREATED\\_PATIENTS$

0: $L_{bcr}$

1: $L_{bcr}\leftarrow[\;]$

2: for all $p_{i}$ in $TREATED\\_PATIENTS$ do

3: $d_{1}\leftarrow PRP(p_{i})$

4: $d_{2}\leftarrow CRP(p_{i})$

5: $d_{3}\leftarrow PRT(p_{i})$

6: $d_{4}\leftarrow CRT(p_{i})$

7: if $allAreNULL(d_{1},d_{2},d_{3},d_{4})=FALSE$ then

8: $bcr\\_date\leftarrow getMin(d_{1},d_{2},d_{3},d_{4})$

9: $new\\_bcr\leftarrow(p_{i},bcr\\_date)$

10: $L_{bcr}\leftarrow L_{bcr}+new\\_bcr$

11: end if

12: end for

13: return $L_{bcr}$

Algorithm 7 DBCR


# Supervised Contrastive Learning for Interpretable Long-Form Document
Matching

Akshita Jha  Virginia Tech, Arlington, VAUSA<EMAIL_ADDRESS>,  Vineeth
Rakesh  InterDigital, CAUSA<EMAIL_ADDRESS>,  Jaideep Chandrashekar
InterDigital, CAUSA<EMAIL_ADDRESS>,  Adithya
Samavedhi  Virginia Tech, Arlington, VAUSA<EMAIL_ADDRESS>and  Chandan K.
Reddy  Virginia Tech, Arlington, VAUSA<EMAIL_ADDRESS>
(2022)

###### Abstract.

Recent advancements in deep learning techniques have transformed the area of
semantic text matching. However, most state-of-the-art models are designed to
operate with short documents such as tweets, user reviews, comments, etc.
These models have fundamental limitations when applied to long-form documents
such as scientific papers, legal documents, and patents. When handling such
long documents, there are three primary challenges: (i) the presence of
different contexts for the same word throughout the document, (ii) small
sections of contextually similar text between two documents, but dissimilar
text in the remaining parts (this defies the basic understanding of
”similarity”), and (iii) the coarse nature of a single global similarity
measure which fails to capture the heterogeneity of the document content. In
this paper, we describe CoLDE: Contrastive Long Document Encoder – a
transformer-based framework that addresses these challenges and allows for
interpretable comparisons of long documents. CoLDE uses unique positional
embeddings and a multi-headed chunkwise attention layer in conjunction with a
supervised contrastive learning framework to capture similarity at three
different levels: (i) high-level similarity scores between a pair of
documents, (ii) similarity scores between different sections within and across
documents, and (iii) similarity scores between different chunks in the same
document and across other documents. These fine-grained similarity scores aid
in better interpretability. We evaluate CoLDE on three long document datasets
namely, ACL Anthology publications, Wikipedia articles, and USPTO patents.
Besides outperforming the state-of-the-art methods on the document matching
task, CoLDE is also robust to changes in document length and text
perturbations and provides interpretable results. The code for the proposed
model is publicly available at https://github.com/InterDigitalInc/CoLDE.

semantic text matching, long documents, contrastive learning, attention,
embeddings, interpretability, transformer, BERT

††copyright: acmcopyright††journalyear: 2022††doi: ††conference: ACM TKDD;
May; 2022††booktitle: May ’22: ACM Transactions on Knowledge Discovery from
Data††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Information Systems
Document Representation††ccs: Information Systems Document Structure††ccs:
Computing Methodologies Information Extraction

##  1\. Introduction

Figure 1. An example illustrating the challenges in long document matching and
an overview of the proposed CoLDE framework. The Query document is compared
against two target documents Target 1 and Target 2. All documents
homogeneously discuss compression (albeit the context varies). CoLDE provides
a weighted similarity score between different chunks and sections within and
across the documents.

Semantic text matching (STM) is an important and challenging problem in the
field of information retrieval. Given a query, which could range from a few
words to a few pages, the objective of STM is to retrieve a set of documents
related to the query. While recent advancements in language modeling have
certainly produced promising results, their effectiveness is limited to the
following tasks: (a) matching short-queries to short-documents, or (b)
matching short-queries to long-documents. A few examples of short-document
matching include contextual text similarity (Kenter and De Rijke, 2015),
sentence matching (Hu et al., 2014), and question ranking (Amiri et al., 2016)
where one or both documents (i.e., the query and the target document) have
limited textual content. However, semantic text matching between two long
documents, which is a more challenging problem, has not been thoroughly
studied in the literature. Some important use cases for long-document matching
include ranking of research papers, comparison of patent documents, and
clustering of Wikipedia articles. In this paper, we exclusively focus on
semantic text matching for long documents. There are three main challenges
that arise while developing a model for long document matching.

Challenge 1. It is infeasible to learn the semantics of individual words,
phrases, and sentences by considering only a few lines of text, as longer
documents span several pages. In other words, learning a representative
embedding that encapsulates the context of an entire document is extremely
challenging. Further evidence of this is provided by the state-of-the-art
Transformer models, such as BERT (Devlin et al., 2019), which cannot handle
more than 512 tokens (or words) during a single feed-forward pass. Current
solutions for addressing these limitations include various pooling techniques
which have already proven to be ineffective (Reimers and Gurevych, 2019).

Challenge 2. Accounting for various levels of (dis)similarity between
different sections of text is another challenge. To understand this better,
let us consider a scenario where the end-user is interested in finding long
documents most similar to the given document (Query). For the sake of
simplicity, we consider only two possible target long documents (Target 1 and
Target 2) to choose from (see Figure 1). In the above three documents (Query,
Target 1, and Target 2), there are three levels of similarity: (a) A similar
boiler plate text about the popularity of deep learning that does not
contribute to the distinguishing content of the documents, (b) Semantically
similar words – Query has a ‘semantic match’ with both Target 1 and Target 2
(‘reducing the size’ is equivalent to the word ‘compression’), and (c)
Contextually similar content – Query is contextually similar only to Target 2.
The context is provided by statements such as ‘compressing neural networks’
unlike Target 1 which discusses ‘image compression’. Here, Query is most
similar to Target 2 but in a long-form document, such contextual information
cannot be readily learned merely from the initial few lines (or paragraphs) as
the distinguishing information might be presented later on in the long-form
document.

Challenge 3. The final challenge is interpretability. When comparing
documents, it is not sufficient to just provide a score that quantifies the
amount of similarity. It is also important to explain what makes the documents
(dis)similar.

Model Overview. To tackle the aforementioned challenges, we propose CoLDE:
Contrastive Long Document Encoder for matching long documents in an
interpretable manner. Intuitively, while learning document representations,
sections within the same document should be closer to each other and sections
across different documents should be farther apart in the latent embedding
space. Additionally, representations of documents that belong to the same
class should be closer to each other compared to the documents that belong to
a completely different class. With this intuition, we leverage the structure
of a long-form document along with its class label and build a supervised
contrastive learning framework (Khosla et al., 2020) for long document
matching. CoLDE divides a long document into different sections and uses
unique positional embeddings to capture the long document structure for added
interpretability. A key challenge in document matching is the coarse nature of
a single global similarity score. While some works do demonstrate how
attention scores can be used to provide insights into attention-based models
(Wiegreffe and Pinter, 2019; Yang et al., 2016b; Ghaeini et al., 2018), we
define interpretability as the weighted similarity scores between different
sections and different chunks of text in a document. These fine-grained
similarity scores can aid the end-users in understanding the reason as to why
the two documents being (dis)similar. We provide three levels of similarity
scores: (i) Similarity score between documents, (ii) Similarity scores between
sections within and across different documents, and (iii) Similarity scores
between different text chunks within and across documents. We use a
combination of contrastive loss and a multi-headed attention layer for
different text chunks to get this weighted similarity score. Figure 1 presents
the model overview. We demonstrate the performance of our model using three
different semi-structured long document datasets: (i) ACL Anthology Network
papers, (ii) Wikipedia articles, and (iii) Patent documents.

Recently, a few attempts have been made to encode long sequences of text using
transformers (Beltagy et al., 2020; Ding et al., 2020; Zaheer et al., 2020).
Our work is significantly different from these approaches since we primarily
focus on end-to-end long document matching. To the best of our knowledge, our
work is the first to utilize the long document structure for contrastive
learning in text and provide fine-grained analysis of similarity scores within
and across documents.

The primary contributions of this paper are as follows:

  * •

Propose a simple yet effective novel supervised contrastive learning model,
CoLDE: Contrastive Long Document Encoder, to compare long documents like
research papers, patent documents, and Wikipedia articles.

  * •

Develop new data augmentation techniques for text along with unique positional
embeddings to capture the long document structure and consequently learn
better representations.

  * •

Demonstrate the interpretability provided by the CoLDE model – by providing
high-level similarity score between documents, in addition to fine-grained
similarity score between different sections, and between different text chunks
within and across documents.

The rest of this paper is organized as follows. Section 2 describes the
related work. Section 3 presents the proposed framework, CoLDE, in detail. Our
experimental methodology is discussed in Section 4 and Section 5 concludes our
work.

##  2\. Related Work

Long Document Matching. Guo et al. (Guo et al., 2016) propose a joint deep
learning based architecture for ad-hoc retrieval when comparing documents.
Several works have also used convolutional networks (Hu et al., 2014; Pang et
al., 2016; Yu et al., 2018), with weighting mechanism (Yang et al., 2016a) to
generate a final query-document score. Mitra et al. (Mitra et al., 2017)
propose a combination model that uses local representations and distributed
representations for text matching. Yang et al. (Yang et al., 2016b) propose a
hierarchical attention network for document classification whereas Adhikari et
al. (Adhikari et al., 2019) use BERT for document classifcation. Jian et al.
(Jiang et al., 2019) propose a multi-depth attention based hierarchical
recurrent neural network (SMASH) for long-document matching. However, Yang et
al. (Yang et al., 2020) pre-train a transformer based hierarchical model
(SMITH) for text matching that outperforms them across multiple datasets.
While our proposed CoLDE model is also based on transformers, we provide fine-
grained similarity scores unlike the previous models that only classify
whether a document is relevant or irrelevant based on a (coarse) single
document level similarity score.

Long Document Encoding. Earlier works used Doc2Vec (Le and Mikolov, 2014;
Chen, 2017) for long document representation. A growing body of literature is
now examining the use of transformer based models for long document encoding
(Child et al., 2019; Ho et al., 2019; Kitaev et al., 2019; Liu and Lapata,
2019; Qiu et al., 2020). Longformer (Beltagy et al., 2020), Poolingformer
(Zhang et al., 2021), Performers (Choromanski et al., 2020), and Linear
Transformers (Katharopoulos et al., 2020) adapt transformers to use an
attention mechanism to aggregate information for long sequences. Big Bird
(Zaheer et al., 2020) proposes a sparse attention mechanism that reduces
BERT’s quadratic dependency on the sequence length to linear. CogLTX (Ding et
al., 2020) presents a framework that uses multi-step reasoning over key
sentences to overcome the insufficient long-range attentions in BERT by using
text blocks for rehearsal and decay. Transformer-XL (Dai et al., 2019) and
Compressive Tranformers (Rae et al., 2019) compress the transformers to use
attentive sequence over long text. Our work is significantly different from
these approaches since we primarily focus on end-to-end long document
matching.

Interpetability for Text. Researchers have devised approaches to measure the
feature importance in text (Ross et al., 2017; Sundararajan et al., 2017).
There is an increased interest in using causal frameworks for understanding
black-box predictions (Alvarez-Melis and Jaakkola, 2017). Some researchers
claim that attention is not explanation (Jain and Wallace, 2019). However,
Wiegreffe et al. (Wiegreffe and Pinter, 2019), Yang et al. (Yang et al.,
2016b), and Ghaeni et al. (Ghaeini et al., 2018) demonstrate how attention
scores can be used to provide insights into attention-based models. We use
multi-headed chunkwise attention in conjunction with supervised contrastive
loss to provide fine-grained similarity scores that help in interpreting the
final decision made by the model.

Contrastive Learning. Contrastive learning has seen tremendous success in
recent times. SimCLR (Chen et al., 2020) and Supervised SimCLR (Khosla et al.,
2020) use self-supervised and supervised contrastive learning frameworks,
respectively, to learn meaningful image representations that help in
downstream tasks. DeCLUTR (Giorgi et al., 2021) uses a self-supervised loss
function for learning universal sentence embeddings without labels. SimCSE
(Gao et al., 2021) uses dropout as a form of data augmentation for learning
sentence representations in a self-supervised as well as a supervised manner.
Luo et al. (Luo et al., 2021) use different semantic paraphrasing techniques
for contrastive augmentation to learn better sentence representations. All
these methods either focus on images or short-text. To the best of our
knowledge, we are the first ones to apply constrastive learning on the task of
long-form document matching.

##  3\. The Proposed COLDE Framework

Figure 2. Model architecture for CoLDE. The contrastive learning framework
consists of three primary components: (i) Data Augmentation, (ii) Data
Encoder, and (iii) Contrastive Loss Function. It takes as input long documents
and divides them into several sections. Each section is further split into
chunks of 512 tokens and enhanced with unique positional embeddings (Fig. 3)
before being given to the data encoder module. The contrastive loss function
in conjunction with multi-headed chunkwise attention provide fine-grained
similarity scores within and across sections and chunks. Figure 3.
Visualization of the proposed unique positional embeddings for a long-document
with two sections. In this example, each section has two chunks of 512 tokens
each. The embeddings are summed up together to form the augmented token
embeddings for BERT. They capture the structural information present in the
long document.

Inspired by the recent success of contrastive learning algorithms in the
computer vision domain (Chen et al., 2020; Khosla et al., 2020), we build a
simple yet effective contrastive learning framework for long-document
matching. Intuitively, CoLDE maximizes the similarity between latent
representations of sections within the same document and minimizes the
similarity between representations of sections from different documents.
Additionally, it ensures that representations of documents belonging to the
same class are closer to each other in the latent embedding space when
compared to the documents from a different class. CoLDE consists of three
primary components: (i) Data Augmentation, (ii) Data Encoder, and (iii)
Contrastive Loss Function. Figure 2 presents the complete architecture of the
proposed model.

Problem Statement. The problem being tackled in this paper is defined as
follows. Given a query document $s$, and a set of target documents $D_{T}$,
the goal is to estimate the semantic similarity $sim(.)$ between $s$ and a
target document $t\in D_{T}$ at three different levels: (i) Document level
$\hat{y_{d}}=sim(s,t)$ for every document pair, (ii) Section level
$\hat{y_{p}}=sim(s_{i},t_{j})$, where $s_{i}\in s$ and $t_{j}\in t$ for every
section pair, and (iii) Chunk level $\hat{y_{c}}=sim(c_{q_{s}},c_{q_{t}})$,
where $c_{q_{s}}\in s_{i}$ and $c_{q_{t}}\in t_{j}$ for every chunk pair. The
semantically similar target documents, sections, and chunks will have a higher
similarity score. The notation used throughout the paper and their definitions
have been given in Table 1.

Table 1. Table of Notations. Notation | Definitions
---|---
$r$ | Token ID
$q$ | Chunk ID
$p$ | Section ID
$x$ | Long document
$s$ | Query document
$t$ | Target document
$f(.)$ | Encoder module
$sim(.)$ | Similarity function
$Aug(.)$ | Data augmentation module
$\hat{y}_{d}$ | Document level similarity
$\hat{y}_{p}$ | Section level similarity
$\hat{y}_{c}$ | Chunk level similarity
$T_{r}$ |  Embedding for token $r$
$C_{p}$ |  Embedding for chunk $q$
$S_{p}$ |  Embedding for section $p$

###  3.1. Data Augmentation

Long documents, unlike short-text, follow an inherent structure and are
organized into sections and sub-sections (e.g., introduction, related work,
and methodology). The task of learning meaningful representations for long
documents, can be broken down into learning better representations for their
sections and sub-sections. To achieve this, we divide a long document into
different sections as follows: $\tilde{x_{i}},\tilde{x_{j}}=Aug(x)$ where
$Aug(.)$ is the augmentation module that takes a long document $x$ as input
and extracts different sections $\tilde{x_{i}}$ and $\tilde{x_{j}}$ from the
document. It should be highlighted that data augmentation is different from
data pre-processing as it takes a long document and creates more samples from
it. This is represented as Step 1 in Figure 2. These sections contain a subset
of the text and represent a subset of information present in the original long
document. One of the major limitations of BERT is its inability to handle more
than 512 tokens at a time, thus making it ineffective for encoding long
documents. To overcome this limitation, we divide a document section into
multiple chunks of non-overlapping 512 tokens. Additionally, we enhance these
input section chunks using unique positional embeddings (described below) to
encode the long document structure.

#### Unique Positional Embeddings

The token embeddings from BERT are combined with long document structure aware
positional embeddings before providing them as input to the BERT model. These
positional embeddings are explained below (shown in Fig. 3).

  * •

Section Embeddings: Section embeddings $S_{p}$ are unique values given to each
section where $p$ represents the section number. Since we divide a long
document into two sections, these embeddings take two possible values: $0$ and
$1$. Embeddings for section $\tilde{x}_{i}$ take the value of 0 and that for
section $\tilde{x}_{j}$ take the value of 1.

  * •

Chunk Embeddings: Since BERT can only handle 512 tokens at a time, each
section is further divided into several chunks of 512 tokens. Each of these
chunks is given a unique ID $C_{q}$ where $q$ represents the chunk number.

  * •

Token Embeddings: Token embeddings $T_{r}$ are the index $r$ of the token in
the input sequence. They are the same as the ones seen in the standard BERT
model.

These additional embeddings are summed up together and form the augmented
token embeddings $\tilde{T}_{pqr}$ given by:

$\tilde{T}_{pqr}=\sum_{p=0}^{n_{p}}\sum_{q=0}^{n_{c}}\sum_{r=0}^{512}(S_{p}+C_{pq}+T_{pqr})$

where $n_{p}$ is the total number of sections and $n_{c}$ is the total number
of chunks in a section. There are 512 tokens in each chunk. These augmented
token embeddings are given to the data encoder for further processing.

###  3.2. Data Encoder

CoLDE’s data encoder $f(.)$ is built over the BERT model (Devlin et al.,
2019). We use a pre-trained BERT model that is fine-tuned during training. The
augmented input chunks of 512 tokens are given as input to BERT. Zero-padding
is done for chunks smaller than 512 tokens. The encoded BERT chunk embeddings
are given to a bidirectional-LSTM (Bi-LSTM) (Graves and Schmidhuber, 2005)
layer for aggregation which uses a unique ‘multi-headed chunkwise attention’
component (described below).

#### Multi-headed Chunkwise Attention

To get chunk level similarity scores, we introduce a multi-headed chunkwise
attention layer. This is a self-attention layer (Vaswani et al., 2017) that
computes attention between the BERT embeddings of different chunks of 512
tokens within and across sections. As a consequence, we can obtain a fine-
grained understanding of the chunk that attends the most to other text chunks
in the document. The chunk with the highest attention weight w.r.t. a Query
chunk plays the most important part in computing the similarity score. We
compute the multi-headed chunkwise attention between sections by treating each
chunk in section $p$ as a Query $(Q)$. The chunks of section $p+1$ are treated
as Keys $(K)$ against which the attention is computed. The chunks of section
$p+1$ are also used in calculating the Values $(V)$. These vectors are of
dimension $d$. The weights for calculating the query $Q$, keys $K$, and values
$V$ are learned during training. The matrix of the outputs is computed as
follows: $Atention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d}})V$.

#### Dimensionality Reduction

The encoded BERT output of different chunks, along with the multi-headed
chunkwise attention, is given sequentially to a Bi-LSTM layer. The Bi-LSTM
layer is required for aggregation of the segmented chunk representations in
order to obtain an intermediate high-dimensional section representation
$\tilde{z}_{l}$ corresponding to section $\tilde{x}_{l}$, where $l$ is the
section number. This intermediate section representation
$\tilde{z}_{l}=f(\tilde{x}_{l})$ is further given to a projection layer
$Proj(.)$ for reducing the dimensionality which results in a final section-
level representation $z_{l}=Proj(\tilde{z}_{l})$. The final section
representations $z_{l}$ is used to compute the contrastive loss between
sections across query and target documents. The loss is then back-propagated
through the network during training.

###  3.3. Contrastive Loss Function

#### Supervised Contrastive Loss

Using the contrastive learning framework, we bring representations of sections
within the same documents closer and obtain a similarity score between
sections within and across documents. CoLDE also provides document level
similarity scores and ensures that documents belonging to the same class are
closer to each other in the latent embedding space. To achieve this, we use
supervised contrastive loss (Khosla et al., 2020) as our objective function.
We randomly sample a set of $N$ long documents and their corresponding labels
$\\{x_{k},y_{k}\\}_{k=1}^{N}$. This results in $2N$ data points since each
document is divided into two sections. In this paper, we refer to the set of
$N$ documents as a ‘batch’ and the set of $2N$ documents as an ‘augmented
batch’. The supervised contrastive loss function is described as follows:

$\mathcal{L}^{sup}=\sum_{i=1}^{2N}\sum_{j=1}^{2N}\frac{-1}{2N_{y_{i}}-1}\mathbbm{1}_{i\neq
j}\cdot\mathbbm{1}_{\tilde{y_{i}}=\tilde{y_{j}}}\cdot log\frac{exp(z_{i}\cdot
z_{j}/\tau)}{\sum_{k=1}^{2N}\mathbbm{1}_{i\neq k}\cdot exp(z_{i}\cdot
z_{k}/\tau)}$

In the above equation, $N$ is the batch size; $z_{l}=Proj(f(\tilde{x}_{l}))$
where $\tilde{x}_{l}$ is a section; $f(.)$ is the encoder; and $Proj(.)$ is
the projection layer. The three indicator functions: (i) $\mathbbm{1}_{i\neq
j}\in\\{0,1\\}$ evaluate to 1 iff $i\neq j$; (ii)
$\mathbbm{1}_{\tilde{y_{i}}=\tilde{y_{j}}}\in\\{0,1\\}$ evaluate to 1 when the
labels for two sections are the same; and (iii) $\mathbbm{1}_{i\neq
k}\in\\{0,1\\}$ evaluate to 1 iff $i\neq k$. The symbol $\cdot$ indicates
inner (dot) product; $\tau$ denotes a temperature parameter. The final loss is
summed across all the samples. The triplet loss, one of the widely used losses
for supervised training, is a special case of the contrastive loss when the
batch size $N=2$ and contains only one positive and one negative sample.

The numerator incorporates all positive sections in an augmented batch, i.e.,
every section with the same label in the augmented batch is treated as a
positive sample. The denominator, on the other hand, performs a summation over
the negative samples. To understand the positive and negative samples better,
let us consider a simple scenario where $N=3$. The batch has three documents:
$D_{1}$, $D_{2}$, and $D_{3}$ with labels $1$, $1$, and $0$, respectively.
Each document is split into two sections which results in an augmented batch
of $2N=6$. With respect to the source document $D_{1}$, sections of $D_{1}$
and $D_{2}$ are considered to be positive samples for each other. Sections
from $D_{3}$ are considered to be negative samples. Our loss function
encourages the encoder to output section representations for $D_{1}$ and
$D_{2}$ to be closer to each other in the latent embedding space. Although,
the above example only describes the binary class scenario, the loss also
generalizes well to a multi-class setting. In such a multi-class scenario, the
positive and negative samples are computed w.r.t. to the label of the source
document.

#### Negative Samples

For a batch size of $N$, all documents having the same label are treated as
positive samples and the others are considered to be negative samples w.r.t. a
source document. The negative samples belong to one of the three categories:
(i) hard negatives, (ii) semi-hard negatives, and (iii) easy negatives. These
three categories represent the distance between the representation of the
source document and the negative samples in the latent embedding space,
ranging from very close for hard negatives to very far for easy negatives.
Most models explicitly require these different kinds of negatives as inputs.
This is not the case for CoLDE. When using the supervised contrastive loss,
hard negative mining is implicitly performed by the model (Khosla et al.,
2020). The discriminatory power of the encoder increases with the increase in
the number of negative samples w.r.t. a source document, i.e., with increase
in batch size, the number of negative samples increase.

##  4\. Experiments

In this section, we empirically evaluate the proposed CoLDE framework by
studying the following research questions:

  * •

RQ1: Does CoLDE perform better than existing methods for downstream tasks such
as long document matching?

  * •

RQ2: How does the supervised contrastive loss function and the proposed multi-
headed attention layer aid in interpretability?

  * •

RQ3: How robust is the model to changes in batch size, document length, and
text perturbation?

  * •

RQ4: What is the impact of different components on the performance of CoLDE?

###  4.1. Methodology

####  4.1.1. Datasets

For our experiments, we consider three different long document datasets: (i)
ACL Anthology Network Corpus (AAN), (ii) Wikipedia (WIKI), and (iii) Patent
(PAT) datasets.

  * •

ACL Anthology Network Corpus
(AAN)111http://aan.how/download/#aanNetworkCorpus: The AAN corpus (Radev et
al., 2013) consists of 23,766 papers written by 18,862 authors in 373 venues
related to NLP and forms a citation network. Each paper is represented by a
node with directed edges connecting a paper (the parent node) to all its cited
papers (children nodes). Papers that have been cited by the parent paper are
treated as similar samples. We consider this broad notion of similarity which
is in line with the previous works in this area (Jiang et al., 2019; Yang et
al., 2020). For every similar sample, an irrelevant paper is randomly chosen
to create a balanced dataset. Sets of similar papers are given the same
labels. To prevent leakage of information and make the task more difficult,
the references and the abstract sections are removed. Papers without any
content are also removed. The dataset consists of 12,665 randomly selected
research papers split into two sections. On average, each section has 6
chunks.

  * •

Wikipedia (WIKI)222https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-
pages-articles.xml.bz2: We use the Wikipedia dump, and adopt a methodology
similar to the one proposed by Jiang et al. (Jiang et al., 2019) to process
this data. We create a dataset of similar Wikipedia articles by assuming that
similar articles have similar outgoing links. The Jaccard similarity between
the outgoing links of the source and the target articles is calculated. If the
Jaccard similarity $>0.5$, the documents are assumed to be similar, otherwise
they are considered dissimilar. Only articles with two or more similar
articles are selected. Sets of similar articles are given the same labels. The
dataset consists of 59,911 Wikipedia articles split into two sections. On
average, each section has 3 chunks.

  * •

Patent (PAT)333https://github.com/google/patents-public-data: This dataset
consists of 3,000 documents sampled from the publicly available USPTO patents.
A patent document is extremely long and primarily consists of (i) Abstract,
(ii) Claims, and (iii) Description sections. We make use of the Claims and the
Description sections for our experiments. On average, the Claims section has 3
chunks, and the Description section has 27 chunks. Three human annotators were
given pairs of documents and were asked to label them as similar or not based
on topics of relevance. They referred to the Abstract, Claims, and CPC
Codes444https://www.uspto.gov/web/patents/classification/cpc/html/cpc.html of
the patents to measure the similarity. The final document content similarity
label was based on majority vote.

We report the results of the baseline models on the above datasets.555Since
the pre-processed dataset used in (Jiang et al., 2019; Yang et al., 2020,
2016b) is not public, our results are not exactly aligned with theirs. We use
their publicly available code and fine-tune the model on our dataset.

####  4.1.2. Implementation Details

The proposed CoLDE model was implemented in PyTorch. For the purpose of our
experiments, we use the pre-trained $\text{BERT}_{\text{BASE}}$ model provided
by the Huggingface
library666https://huggingface.co/transformers/model_doc/bert.html. It has 12
Transformer blocks, a hidden size of 768 and 12 attention blocks. The BERT
model is fine-tuned during training. The Bi-LSTM output dimension is 512. The
projection layer reduces the dimension to 256. We use a scaled-dot product
attention function (Vaswani et al., 2017) with 4 heads as our multi-headed
chunkwise attention function. The learning rate is set to 5e-5. We use an Adam
optimizer with a weight decay of 0.01. The value for $\tau$ is fixed to 0.5
for all the experiments. The three datasets are split into 80-10-10 for train,
validation, and test sets, respectively. The best performance on the
validation set is achieved after training the model for 25 epochs on AAN,
after 35 epochs on WIKI, and after 200 epochs on the Patent dataset. We
experiment with different batch sizes of 50, 75, 100, and 200. The model
training on a batch size of 50 and 75 is done on one 16GB Tesla V100 GPU. For
a batch size of 100 and 200, we used one 48GB RTX-8000 GPU.

####  4.1.3. Baseline Experiments

We primarily compare our model with the following long document comparison
baselines.

  * •

DSSM (Huang et al., 2013): A siamese deep learning model with three fully-
connected feed-forward layers with cosine similarity as its objective
function.

  * •

ARC-I (Hu et al., 2014): A siamese based matching model with three layers of
1D convolutional neural network using cosine similarity.

  * •

Hierarchical Attention Networks (HAN) (Yang et al., 2016b) : A hierarchical
structure model consisting of two layers of RNN with attention mechanism using
cosine similarity.

  * •

Siamese-BERT (S-BERT) (Devlin et al., 2019): This is a siamese matching model
with BERT. To handle long document inputs, S-BERT only uses the first 512
tokens of each document. It uses a pre-trained BERT model which is fine-tuned
during training and cosine similarity.

  * •

Siamese Multi-depth Transformed based Hierarchical Encoder (SMITH) (Yang et
al., 2020): The current SOTA model for long-form document matching tasks. The
model uses a transformer-based hierarchical encoder to obtain sentence-level
and document-level representations for long-form documents.

  * •

Siamese Longformer (S-LONG) (Beltagy et al., 2020): A transformer-based model
for long sequences with an attention mechanism that scales linearly with
sequence length. It takes a maximum input length of 4096 tokens.

###  4.2. RQ1: Evaluation on Long Document Matching

We evaluate the performance of our model on the downstream task of long
document matching. The document matching problem can be treated as a
classification task777We use classification metrics instead of information-
retrieval metrics due to the limitations of the dataset which has very few
positive samples (2-3) for every document.. During evaluation, the model is
given a pair of documents and the task is to predict whether the two input
documents are similar or not. The training is done using the model
architecture described in Figure 2. However, during testing, we do not split
the document into sections and use only the Data Encoder module to get a
single representation for the query and the target documents, respectively. We
then use cosine distance to compute the similarity between these
representations. If the document level similarity score is above a certain
threshold, the documents are considered to be relevant; otherwise, they are
considered to be irrelevant. We report precision (P), recall (R), F1-score
(F1), and Accuracy (Acc) for this task in Table 2. The utility of the model
would depend on the specific use-case, but we observe that CoLDE outperforms
all the methods and achieves the best F1-scores and Accuracy on the document
matching task on all three datasets. We attribute this superior performance to
our new mechanism of exploiting the structure of long documents in conjunction
with the contrastive learning framework.

Table 2. Comparison of different methods on various long-document datasets: (i) AAN, (ii) WIKI, and (iii) PAT. Different metrics such as Precision (P), Recall (R), F1-Score (F1), and Accuracy (Acc) for document matching task are reported. The best result is highlighted in boldface and the second best is underlined. CoLDE’s improvements over other methods are statistically significant with $p<0.05$ as measured by Student’s t-test (whenever CoLDE’s performance is the best). We also present the results for the ablated variants of CoLDE. | AAN | WIKI | PAT
---|---|---|---
Model | P | R | F1 | Acc | P | R | F1 | Acc | P | R | F1 | Acc
DSSM | 0.676 | 0.716 | 0.711 | 0.686 | 0.736 | 0.830 | 0.780 | 0.788 | 0.538 | 0.538 | 0.538 | 0.521
ARC-I | 0.647 | 0.683 | 0.665 | 0.655 | 0.799 | 0.666 | 0.727 | 0.764 | 0.782 | 0.695 | 0.736 | 0.708
HAN | 0.641 | 0.743 | 0.688 | 0.664 | 0.676 | 0.954 | 0.791 | 0.748 | 0.521 | 0.521 | 0.521 | 0.538
S-BERT | 0.665 | 0.666 | 0.665 | 0.566 | 0.665 | 0.667 | 0.666 | 0.499 | 0.536 | 0.495 | 0.515 | 0.520
SMITH | 0.732 | 0.603 | 0.662 | 0.691 | 0.640 | 0.980 | 0.774 | 0.714 | 0.524 | 0.485 | 0.502 | 0.509
S-LONG | 0.633 | 0.723 | 0.675 | 0.699 | 0.794 | 0.733 | 0.762 | 0.752 | 0.848 | 0.533 | 0.655 | 0.521
CoLDE | 0.719 | 0.756 | 0.734 | 0.742 | 0.860 | 0.767 | 0.809 | 0.801 | 0.829 | 0.849 | 0.839 | 0.833
CoLDE${}_{\text{-Aug}}$ | 0.623 | 0.629 | 0.623 | 0.616 | 0.606 | 0.580 | 0.592 | 0.606 | 0.778 | 0.809 | 0.790 | 0.757
CoLDE${}_{\text{-LSTM}}$ | 0.718 | 0.701 | 0.709 | 0.706 | 0.780 | 0.754 | 0.766 | 0.786 | 0.795 | 0.843 | 0.818 | 0.792
CoLDE${}_{\text{-CL}}$ | 0.599 | 0.499 | 0.545 | 0.595 | 0.943 | 0.510 | 0.658 | 0.500 | 0.614 | 0.584 | 0.595 | 0.586

###  4.3. RQ2: Case Study on Model Interpretation

To demonstrate the interpretability provided by CoLDE, we select three
documents, $D_{1}$, $D_{2}$, and $D_{3}$ with the following properties.
$D_{1}$ cites $D_{2}$; hence, they are similar. $D_{3}$ is not cited by either
$D_{1}$ or $D_{2}$; hence, $D_{3}$ is different from other two documents.
Figure 4(c) shows the three levels of interpretability (in terms of similarity
scores) offered by CoLDE using the architecture describe in Figure 2.
Qualitative analysis using the text from these documents is shown in Figure 5.

(a) Level 1: Document Level

(b) Level 2: Section Level

(c) Level 3: Chunk Level

Figure 4. Three levels of interpretability provided by CoLDE: (a) Document
Level Similarity, (b) Section Level Similarity, and (c) Chunk Level
Similarity. We consider three research papers: $D1$, $D2$, and $D3$. $D1$ and
$D2$ are similar to each other. Document $D3$ is different from the other two.
$I1-I3$ correspond to the ‘Introduction’; $RW1-RW3$ correspond to the ‘Related
Work’ sections of documents $D1-D3$, respectively. Qualitative analysis for
Fig. 4(c) is shown in Fig. 5.

.

Figure 5. Qualitative Analysis: Documents $D_{1}$ and $D_{2}$ are similar. The
introduction chunk $I_{11}$ from $D_{1}$ is about ‘topic segmentation and
extractive summarisation’. The introduction chunk $I_{21}$ from $D_{2}$ is
about ‘text segmentation’; the related work chunk $RW_{21}$ from $D_{2}$
presents prior work in the area of ‘text segmentation’. The introduction chunk
$I_{12}$ from $D_{1}$ summarizes the author contributions. Document $D_{3}$ is
on question answering and is different from the other two documents. Darker
colored lines indicate higher similarity.

  * •

Level 1-Document Level Similarity: We obtain the high-level similarity score
using CoLDE. Figure 4(a) illustrates the level of similarity between the three
documents $D_{1}$, $D_{2}$, and $D_{3}$. A document is most similar to itself.
Documents $D_{1}$ and $D_{2}$ have a higher similarity score when compared to
document $D_{3}$. Qualitatively, $D_{1}$ and $D_{2}$ are about ‘automatic
segmentation of speech’ whereas $D_{3}$ is about ‘question answering’ (see
Figure 5).

  * •

Level 2-Section Level Similarity: While document-level similarity is certainly
useful to get a global overview, it does not demonstrate the reason behind the
(dis)similarity. Therefore, CoLDE goes a level deeper by providing section
level similarity. To explain this outcome, we split each document
$\\{D_{k}\\}_{1}^{3}$ into two sections $\\{(I_{k},RW_{k})\\}_{1}^{3}$, where
$I_{k}$ and $RW_{k}$ indicate the Introduction and Related Work of
$k^{\text{th}}$ document, respectively. Figure 4(b) shows how these are
(dis)similar. First, sections from the same document have a high similarity
score ($I_{1}$ and $RW_{1}$; $I_{2}$ and $RW_{2}$; $I_{3}$ and $RW_{3}$).
Second, since documents $D_{1}$ and $D_{2}$ are similar, their introductions
$I_{1}$ and $I_{2}$ are more similar compared to $I_{3}$ from $D_{3}$. This is
also true for sections $RW_{1}$ and $RW_{2}$, which are more similar to each
other compared to $RW_{3}$. Figure 5 shows some of the content from $I_{1}$,
$I_{2}$, and $RW_{2}$. The introductions $I_{1}$ and $I_{2}$ discuss
‘topic/text segmentation in dialogues and meetings’, and $RW_{2}$ presents the
related work in the area.

  * •

Level 3-Chunk Level Similarity: Besides providing the section-level
similarity, CoLDE also explains which segment (or chunk) of a section is
important. To illustrate this outcome, we divide $(I_{k},RW_{k})$ into $j$
chunks, which is denoted by $(I_{jk},RW_{jk})$. In our experiments, we set
$j=2$; in other words, each document is divided into 4 chunks. Figure 4(c)
shows some interesting traits of chunk-level interpretability between
documents $D_{1}$ and $D_{2}$. The introduction chunk $I_{11}$ from document
$D_{1}$ is similar to the first introduction chunk $I_{21}$ from $D_{2}$. In
Figure 5, we see that $I_{11}$ broadly discusses ‘the system components that
perform topic segmentation and extractive summarisation’, and $I_{21}$
discusses ‘text segmentation and its applications’. Additionally, the related
work chunk $RW_{21}$ from $D_{2}$ is very similar to both $I_{11}$ and
$I_{21}$ from documents $D_{1}$ and $D_{2}$, respectively. Qualitatively,
$RW_{21}$ presents prior work in the area of ‘text segmentation’. However, we
observe that the second introduction chunk $I_{12}$ from document $D_{1}$ has
very low similarity score with the above chunks ($I_{11}$, $I_{21}$, and
$RW_{21}$) and interestingly, $I_{12}$ is about ‘visual aids for
interpretability and a summary of the author contributions’.

CoLDE, therefore, provides fine-grained inter-document and intra-document
(dis)similarity scores at three different levels which makes the model more
interpretable.

###  4.4. RQ3: Robustness Analysis of CoLDE

(a) AAN

(b) WIKI

(c) PAT

Figure 6. Effect of batch size on CoLDE’s accuracy for document matching task
on the three datasets.

####  4.4.1. Effect of Batch Size on CoLDE

Recent works on contrastive learning in the computer vision domain (Chen et
al., 2020; Khosla et al., 2020; Park et al., 2020) have shown that the
discriminating ability of the model increases with increase in batch size
since there will be an increase in the number of informative negative samples.
We empirically validate this hypothesis and demonstrate that this holds true
for the long document matching task. Figure 6 shows the impact of batch size
on model accuracy when it is trained for the same number of epochs on three
datasets: (i) AAN, (ii) WIKI, and (iii) PAT. For the same number of epochs,
larger batch-size performs better for AAN and PAT datasets. However, because
of the presence of hard as well as easy negatives in the WIKI dataset, a
larger batch size will have many more easy negatives making the task more
challenging. Although, CoLDE does eventually converge to the optimal accuracy
of $\sim$80% for all batch sizes on the WIKI dataset, it takes longer for
bigger batch sizes than for smaller ones. Irrespective of the dataset, the
best performance of CoLDE steadily increases with the number of training
epochs.

(a) AAN

(b) WIKI

(c) PAT

Figure 7. Effect of document length on CoLDE’s accuracy for document matching
task on the three datasets.

####  4.4.2. Effect of Document Length on CoLDE

In order to study CoLDE’s robustness to varying document lengths, we limit the
maximum number of tokens in each section during training and observe the final
test accuracy on the document matching task. The model is given the entire
long document during evaluation. Figure 7 shows the model accuracy for the
maximum lengths of 1000, 4000 and 8000 tokens. We compare CoLDE’s performance
with other methods for similar document lengths. We observe that CoLDE
outperforms the baselines and its performance steadily increases with increase
in document length, emphasising the need for explicit long document matching
models.

Figure 8. Effect of text perturbation on AAN dataset for document matching
task.

####  4.4.3. Effect of Text Perturbation on CoLDE

We study the effect of text perturbation on the model performance for the AAN
dataset. We randomly shuffle the sections before applying different models and
measure their accuracy on the document comparison task. Figure 8 shows the
model performance for all the baseline methods. We observe a significant drop
in the model performance for BERT-based models (S-BERT, S-LONG, and SMITH),
whereas CoLDE’s performance remains unchanged. There is no significant change
in the accuracy for DSSM, ARC-I, and HAN. We observe that, although, other
transformer-based models are sensitive to text perturbation, CoLDE is robust.
The proposed unique positional embeddings used to enhance the tokens during
the data augmentation phase are long document aware, and hence, even after
shuffling the text sections, they retain the original long document structure
information, thus preserving the accuracy on the downstream task.

###  4.5. RQ4: Ablation Study of CoLDE

We study the effect of different components of CoLDE during training by
conducting ablation study experiments. The results of the ablation study on
the document matching task are shown in Table 2.

####  4.5.1. Effect of Data Augmentation

We remove the Data Augmentation module to study its impact on the overall
model performance. This model variant is termed as CoLDE${}_{\text{-Aug}}$. We
do not split a long document into different sections. We also do not add the
proposed positional embeddings to the input document. Instead, the entire long
document is split into chunks of 512 tokens and given to the data encoder
module. The objective function used is supervised contrastive loss. We observe
that there is a significant drop in the performance for document matching on
all three datasets. The data augmentation module helps in learning better
document representations by bringing the representations of sections within a
document closer. Moreover, sections of documents belonging to the same class
label are also brought closer to each other in the latent embedding space when
used in conjunction with the supervised contrastive loss. In its absence, we
lose out on learning meaningful document representations, in addition to
section level similarity.

####  4.5.2. Effect of LSTM and Multi-Headed Chunkwise Attention

BERT can only handle 512 tokens at a time. To overcome this limitation and get
section representations, we use a Bi-LSTM layer for aggregation. We study its
effectiveness by removing only the Bi-LSTM and the multi-headed chunkwise
attention layer from the data encoder module. We term this variant as
CoLDE${}_{\text{-LSTM}}$. The section representation is the ‘average’ of the
BERT output representations for all chunks of 512 tokens within a section.
This is then given to the projection layer for dimensionality reduction. The
objective function used is supervised contrastive loss. We observe that there
is a performance drop on the document matching task when not using the Bi-LSTM
layer for aggregation. Bi-LSTM layer effectively captures the sequential
nature of the chunks within a section which is lost when computing the average
of the BERT output representations for text chunks.

####  4.5.3. Effect of using Contrastive Loss

We conduct an ablation study by replacing the supervised constrastive loss
with cosine similarity. Since, we do not require documents to be split into
sections for computing the cosine similarity between them, removing the
contrastive loss function entails disabling the data augmentation module as
well. This variant is termed as CoLDE${}_{\text{-CL}}$. The input documents
are split into multiple chunks of 512 tokens and given to the data encoder
module. The final representations from this module are used to compute the
cosine similarity. From Table 2, we observe that there is a significant drop
in performance when using cosine similarity as opposed to using the supervised
constrastive loss as our objective function. Supervised contrastive loss helps
in providing fine-grained similarity scores and ensures that the latent
representations of sections within the same document are closer to each other.
Additionally, representations of documents belonging to the same class are
also closer in the latent embedding space.

####  4.5.4. Effect of Number of Input Sections

We study the effect of number of sections on the performance of CoLDE by
splitting the source and the target documents into sections ranging from 2 to
5, before being given as input to the model. CoLDE was robust to these changes
and the performance was consistent with the ones shown in Table 2 for
different input sections for all datasets.

##  5\. Conclusion

In this work, we introduced a contrastive learning framework CoLDE for long-
form document matching. CoLDE leverages the long document structure in
conjunction with supervised contrastive loss to provide fine-grained
similarity scores within and across documents at three different levels: (i)
Document Level, (ii) Section Level, and (iii) Chunk Level. The overall
framework consists of three primary components: (i) Data Augmentation, (ii)
Data Encoder, and (iii) Contrastive Loss Function. We showed that CoLDE
outperforms the existing state-of-the-art models for long document matching
and presented a detailed case study for model interpretation. We also
empirically demonstrated CoLDE’s robustness to batch size, document length,
and text perturbation compared to other methods. Moreover, we conducted an
ablation study to examine the effectiveness of various components of CoLDE. In
summary, we demonstrated that CoLDE’s contrastive learning approach is
effective for interpretable long-form document matching.

###### Acknowledgements.

This work was supported in part by the US National Science Foundation grant
IIS-1838730.

## References

  * (1)
  * Adhikari et al. (2019) Ashutosh Adhikari, Achyudh Ram, Raphael Tang, and Jimmy Lin. 2019\.  Docbert: Bert for document classification.  _arXiv preprint arXiv:1904.08398_ (2019).
  * Alvarez-Melis and Jaakkola (2017) David Alvarez-Melis and Tommi Jaakkola. 2017.  A causal framework for explaining the predictions of black-box sequence-to-sequence models. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. 412–421.
  * Amiri et al. (2016) Hadi Amiri, Philip Resnik, Jordan Boyd-Graber, and Hal Daumé III. 2016.  Learning text pair similarity with context-sensitive autoencoders. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 1882–1892.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020\.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_ (2020).
  * Chen (2017) Minmin Chen. 2017\.  Efficient Vector Representation for Documents through Corruption. In _ICLR (Poster)_.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020.  A simple framework for contrastive learning of visual representations. In _International conference on machine learning_. PMLR, 1597–1607.
  * Child et al. (2019) Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019\.  Generating long sequences with sparse transformers.  _arXiv preprint arXiv:1904.10509_ (2019).
  * Choromanski et al. (2020) Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. 2020\.  Rethinking Attention with Performers. In _International Conference on Learning Representations_.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019\.  Transformer-XL: Attentive Language Models beyond a Fixed-Length Context. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. 2978–2988.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. 4171–4186.
  * Ding et al. (2020) Ming Ding, Chang Zhou, Hongxia Yang, and Jie Tang. 2020\.  CogLTX: Applying BERT to Long Texts.  _Advances in Neural Information Processing Systems_ 33 (2020).
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple Contrastive Learning of Sentence Embeddings.  _arXiv preprint arXiv:2104.08821_ (2021).
  * Ghaeini et al. (2018) Reza Ghaeini, Xiaoli Fern, and Prasad Tadepalli. 2018\.  Interpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inference. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. 4952–4957.
  * Giorgi et al. (2021) John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. 2021\.  DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 879–895.
  * Graves and Schmidhuber (2005) Alex Graves and Jürgen Schmidhuber. 2005.  Framewise phoneme classification with bidirectional LSTM networks. In _Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005._ , Vol. 4. IEEE, 2047–2052.
  * Guo et al. (2016) Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. 2016\.  A deep relevance matching model for ad-hoc retrieval. In _Proceedings of the 25th ACM international on conference on information and knowledge management_. 55–64.
  * Ho et al. (2019) Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. 2019.  Axial attention in multidimensional transformers.  _arXiv preprint arXiv:1912.12180_ (2019).
  * Hu et al. (2014) Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014\.  Convolutional neural network architectures for matching natural language sentences.  _Advances in neural information processing systems_ 27 (2014), 2042–2050.
  * Huang et al. (2013) Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013\.  Learning deep structured semantic models for web search using clickthrough data. In _Proceedings of the 22nd ACM international conference on Information & Knowledge Management_. 2333–2338.
  * Jain and Wallace (2019) Sarthak Jain and Byron C Wallace. 2019.  Attention is not Explanation. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. 3543–3556.
  * Jiang et al. (2019) Jyun-Yu Jiang, Mingyang Zhang, Cheng Li, Michael Bendersky, Nadav Golbandi, and Marc Najork. 2019\.  Semantic text matching for long-form documents. In _The World Wide Web Conference_. 795–806.
  * Katharopoulos et al. (2020) Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020.  Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. In _International Conference on Machine Learning_. PMLR, 5156–5165.
  * Kenter and De Rijke (2015) Tom Kenter and Maarten De Rijke. 2015.  Short text similarity with word embeddings. In _Proceedings of the 24th ACM international on conference on information and knowledge management_. 1411–1420.
  * Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020.  Supervised Contrastive Learning.  _Advances in Neural Information Processing Systems_ 33 (2020).
  * Kitaev et al. (2019) Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2019\.  Reformer: The Efficient Transformer. In _International Conference on Learning Representations_.
  * Le and Mikolov (2014) Quoc Le and Tomas Mikolov. 2014.  Distributed representations of sentences and documents. In _International conference on machine learning_. PMLR, 1188–1196.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Hierarchical Transformers for Multi-Document Summarization. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 5070–5081.  https://doi.org/10.18653/v1/P19-1500
  * Luo et al. (2021) Dongsheng Luo, Wei Cheng, Jingchao Ni, Wenchao Yu, Xuchao Zhang, Bo Zong, Yanchi Liu, Zhengzhang Chen, Dongjin Song, Haifeng Chen, et al. 2021\.  Unsupervised Document Embedding via Contrastive Augmentation.  _arXiv preprint arXiv:2103.14542_ (2021).
  * Mitra et al. (2017) Bhaskar Mitra, Fernando Diaz, and Nick Craswell. 2017\.  Learning to match using local and distributed representations of text for web search. In _Proceedings of the 26th International Conference on World Wide Web_. 1291–1299.
  * Pang et al. (2016) Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016\.  Text matching as image recognition. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 30.
  * Park et al. (2020) Taesung Park, Alexei A Efros, Richard Zhang, and Jun-Yan Zhu. 2020.  Contrastive learning for unpaired image-to-image translation. In _European Conference on Computer Vision_. Springer, 319–345.
  * Qiu et al. (2020) Jiezhong Qiu, Hao Ma, Omer Levy, Wen-tau Yih, Sinong Wang, and Jie Tang. 2020\.  Blockwise Self-Attention for Long Document Understanding. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_. 2555–2565.
  * Radev et al. (2013) Dragomir R. Radev, Pradeep Muthukrishnan, Vahed Qazvinian, and Amjad Abu-Jbara. 2013.  The ACL anthology network corpus.  _Language Resources and Evaluation_ (2013), 1–26.  https://doi.org/10.1007/s10579-012-9211-2
  * Rae et al. (2019) Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. 2019\.  Compressive Transformers for Long-Range Sequence Modelling. In _International Conference on Learning Representations_.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 3973–3983.
  * Ross et al. (2017) Andrew Slavin Ross, Michael C Hughes, and Finale Doshi-Velez. 2017.  Right for the right reasons: training differentiable models by constraining their explanations. In _Proceedings of the 26th International Joint Conference on Artificial Intelligence_. 2662–2670.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks. In _International Conference on Machine Learning_. PMLR, 3319–3328.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _Proceedings of the 31st International Conference on Neural Information Processing Systems_. 6000–6010.
  * Wiegreffe and Pinter (2019) Sarah Wiegreffe and Yuval Pinter. 2019.  Attention is not not Explanation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 11–20.
  * Yang et al. (2016a) Liu Yang, Qingyao Ai, Jiafeng Guo, and W Bruce Croft. 2016a.  aNMM: Ranking short answer texts with attention-based neural matching model. In _Proceedings of the 25th ACM international on conference on information and knowledge management_. 287–296.
  * Yang et al. (2020) Liu Yang, Mingyang Zhang, Cheng Li, Michael Bendersky, and Marc Najork. 2020.  Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. 1725–1734.
  * Yang et al. (2016b) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016b.  Hierarchical Attention Networks for Document Classification. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, San Diego, California, 1480–1489.  https://doi.org/10.18653/v1/N16-1174
  * Yu et al. (2018) Jianfei Yu, Minghui Qiu, Jing Jiang, Jun Huang, Shuangyong Song, Wei Chu, and Haiqing Chen. 2018.  Modelling domain relationships for transfer learning on retrieval-based question answering systems in e-commerce. In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining_. 682–690.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020\.  Big bird: Transformers for longer sequences.  _Advances in Neural Information Processing Systems_ 33 (2020).
  * Zhang et al. (2021) Hang Zhang, Yeyun Gong, Yelong Shen, Weisheng Li, Jiancheng Lv, Nan Duan, and Weizhu Chen. 2021.  Poolingformer: Long Document Modeling with Pooling Attention. In _Proceedings of the 38th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 139)_ , Marina Meila and Tong Zhang (Eds.). PMLR, 12437–12446.  https://proceedings.mlr.press/v139/zhang21h.html