\[\do\\\do\]

The completeness (in terms of content) of financial
documents is a fundamental requirement for investment funds.
To ensure completeness, financial regulators have to spend a huge
amount of time for carefully checking every financial document based on the relevant
content requirements, which prescribe the information types to be
included in financial documents (e.g., the fund name, the description
of shares' issue conditions and procedures).
Although several techniques have been proposed to automatically detect
certain types of information in documents in various application
domains, they provide limited support to help
regulators automatically identify the text chunks related to financial
information types, due to the complexity of financial documents and
the diversity of the sentences characterizing  an information type.

In this paper, we propose , an artificial intelligence (AI)-based method for tracing
content requirements in financial documents.
Given a new financial document, first selects a set of candidate sentences for efficient information type identification.
Then, uses a combination of rule-based and data-centric
approaches, by leveraging  information retrieval (IR) and machine
learning (ML) techniques that analyze the words, sentences, and
contexts related to an information type,  to rank candidate sentences.
Finally, using a list of domain-specific indicator phrases related to
each information type, a heuristic-based selector, which considers both the
sentence ranking and the domain-specific phrases, determines a list of sentences corresponding to each information type.

We evaluated by assessing its effectiveness
in tracing financial content requirements in 100 real-world financial documents.
Experimental results show that is able to provide accurate identification
with average precision and recall values of  and , respectively. Furthermore,
can help regulators detect about 80% of missing information types in financial documents.

§ INTRODUCTION

In the financial market, each type of investment
fund, such as UCITS[UCITS (Undertakings for the Collective Investment in Transferable Securities) funds) refers to a regulatory framework that allows for the sale of cross-Europe
mutual funds.], is presented to clients through one or more financial documents (such as KIIDs — Key Investor Information Document — and prospectuses).
Before these documents are made publicly available,
they are submitted to national financial regulators, which check their
compliance with the content requirements prescribed by relevant national and international laws.
Content requirements specify the different information types that must be
included in documents; in this work we will consider the content
requirements of financial documents, also called financial
content requirements.  For example, the following article
of the Luxembourgish law [CSSF (The Commission de Surveillance du Secteur Financier), 2010]
“the frequency of the calculation of issue prices (should be
prescribes the inclusion of the information related to
“issue price” in the prospectuses of UCITS funds.

After the submission of a financial document,
agents of a financial regulator peruse the document and manually
identify the passages of text (e.g., sentences or paragraphs) related
to each mandated information type to ensure the document completeness,
since missing information may lead to substantial fines and cause legal problems and severe
investment losses when conducting activities on financial markets.
However, the manual identification of information types is a non-trivial task.
First, financial documents are lengthy with typically hundreds of
pages and more than 3000 sentences, and contains several tables and lists.
Considerable time could be spent even by simply going through the entire document.
Second, the language used in financial documents is highly nuanced
with sophisticated jargon; the meaning of a sentence could drastically change when only a specific word or the context is different.
Agents have to carefully read and analyze every sentence to avoid  any misunderstanding of the content.
Since the amount of manual work involved often leads to higher fund setup costs and longer time-to-market of investment funds,
it is important to develop approaches for automatically identifying
the passages of text related to the content requirements, which can then further
enable automated compliance checking techniques.

Mining financial data play a critical role in improving the quality of
financial services.  Existing studies focus on mining financial data
from both Web media (e.g., financial news and discussion
boards [Li et al., 2017]) and traditional financial documents (e.g.,
annual financial reports and 10-K [Li et al., 2016]).  In
contrast to these studies, we focus on the task of tracing content
requirements in financial documents, which is important to ensure the
completeness of financial documents.  Although tracing requirements in
documents have been widely studied in the areas of information
extraction and software
engineering [Cleland-Huang et al., 2014, Grishman, 2019], existing
approaches may not suit our task due
to the complexity, peculiarities, and the domain-specific vocabulary of financial documents.
In the field of information
extraction, the work typically focuses on extracting entities and
relations (e.g., named entities such as persons and locations) from natural language (NL)
documents [Li et al., 2020, Jiang and Zhai, 2007] instead of
identifying sentences related to financial content requirements.
Although recent advances in deep learning
make the accurate identification of sentences
possible [Devlin et al., 2019, Joshi et al., 2020],
the large training set required to train the underlying models [Zhang et al., 2021] is usually
unavailable in the financial area, due to the cost of annotating
thousands of financial documents by domain experts and the differences
among documents determined by national regulations.
In software
engineering, several studies [Cleland-Huang et al., 2010, Torre et al., 2020]
infer trace links between high-level NL
requirements (e.g., regulatory code) and low-level NL requirements
(e.g., privacy policies).
However, a
typical NL requirement is often explained with one or two sentences in
the regulatory text [Hayes et al., 2006]; in contrast, the
meaning of the same sentence in financial documents can be different
when it is used in different contexts, which is seldom the case for
SE requirements (where ambiguities are typically avoided).  Hence, we need to
design algorithms to trace financial content requirements while fully accounting for
the characteristics of financial documents.

In this paper, we present (Financial Information Type Identification), an AI-based method for tracing
financial content requirements.
The basic idea of is to learn the characteristics of sentences related to an information type combining both IR and ML techniques from a small set of labeled documents.
Given a new financial document, selects sentences for an information type based on the analysis results of IR and ML models.
Specifically, first preprocesses financial documents with typical natural language processing (NLP) techniques.
To conduct efficient analysis on thousands of sentences in a new financial document, a set of candidate sentences is retrieved by comparing the similarity between the related sentences in the labeled documents and every sentence in the new document.
For the candidate sentences, conducts a fine-grained analysis with IR and ML techniques.
Since the meaning of a sentence could drastically change when only a specific word or the context is different, uses an IR-based analysis to compare the words, sentences, and contexts between candidate sentences and the sentences related to an information type in the labeled documents.
In addition, we also mine and learn a set of features relevant to an information type with an ML-based statistical model.
According to the IR- and ML-based analysis, ranks each new sentence.
At last, uses a heuristic-based selector to select the final sentences.
We built a list of domain-specific phrases that are commonly used to explain an information type (e.g., financial jargon), as well as some excluded synonyms which are seldom used to express that information type according to domain experts' suggestions and the labeled documents.
By considering both sentence ranking and phrase lists,
identifies sentences for an information type from the candidate sentences.

We evaluated using the content requirements for UCITS prospectuses.
Three domain experts manually annotated sentences related to five representative information types for 100 UCITS prospectuses to form a dataset.
Experimental results show can accurately identify the
sentences for the five information types with average precision and recall values of  and , respectively; it significantly outperforms a keyword-search baseline by 27.7 pp (with pp = percentage points) in terms of F${_1}$-score, which simulates the process of locating information types by searching the list of domain-specific phrases in sentences.
Further, can help regulator's agents detect about 80% of missing information types.
Last, is effective even when the number of labeled documents
is limited. With more than 40 labeled documents, the precision value of is still higher than 70% for identifying most information types.

To summarize, the main contributions of this paper are:

* the first work, to the best of our knowledge, on
tracing content requirements in financial documents, which
is important for financial enterprises and regulators to
further enable automated compliance techniques;

* the approach, which addresses the problem of
automated information type identification: it combines IR and ML to conduct fine-grained analysis on the sentences related to each information type;

* an extensive evaluation on the effectiveness of .

The rest of the paper is organized as follows. Section <ref>
explains the characteristics of financial documents and their content requirements.
Section <ref> describes the core
algorithms of . Section <ref> reports on the evaluation
of . Section <ref> discusses related work. Section <ref>
concludes the paper and provides directions for future work.

§ BACKGROUND

§.§ Financial Documents

In the financial domain, every investment fund is required to provide informative financial documents.
These documents help the financial regulator and fund clients understand all relevant and critical information of an investment.
For example,  KIIDs describe the nature and key risks of the fund, while a prospectus provides details about an investment offering to the public.
Such financial documents mainly use natural language together
with auxiliary tables and mathematical formulae.
These documents are the key instruments to guarantee the compliance and controllability of an investment.

Financial documents for investment funds have several key characteristics.
First, they are lengthy with typically hundreds of pages. To minimize investment risks, financial documents must completely provide certain elements of information required by the regulator body for an investment fund (and its sub-funds). Based on 100 randomly selected UCITS prospectuses, our statistics show that a prospectus has on average 127 pages with more than 3000 sentences;
each information type can be related to diverse numbers of sentences (e.g., from 3 to 50 sentences) depending on the way to explain an information type by the investment company.
These characterisitcs make the document difficult to read and thoroughly analyze.
Second, financial documents may use similar sentences to explain different types of required information.
The meaning of a sentence could drastically change when only a specific word is different.
Even worse, the same sentence structure may also refer to different information types when the context of the sentence (e.g., the surrounding sentences) is different.
For example, the following sentences “The swing factor
may normally not exceed 3% of the net asset value of a sub-fund
…In such case, affected shareholders shall be informed as soon
as reasonably practicable thereafter …” refer to the
calculation method for issue price. In contrast, the sentences
“The dilution adjustment may normally not exceed 3% of
the net asset value. …In such case, affected shareholders shall
be informed as soon as reasonably practicable thereafter …”
refer to  the charges on issue price.
Hence, the regulator and clients have to carefully analyze the document to identify the sentences related to different types of information.

§.§ Content Requirements

round_rect = [rectangle, rounded corners, minimum width=1.5cm, minimum height=0.5 cm,text centered, draw=black, fill=white!30]
rect = [rectangle, minimum width=1.5cm, minimum height=0.8cm, text centered, text width=1.5cm, draw=black, fill=white!30]
text_node = [rectangle, minimum width=0.5cm, minimum height=0.35cm, text centered, text width=4cm, draw=white, fill= white!30]

[node distance=5mm, >=latex]
(page)[inner sep=0pt]
(locate) [draw=blue, minimum width=1.1cm, minimum height=1.1cm, line width=0.2mm, right = of page, xshift=-1.7cm, yshift=-0.4cm] ;
(ptext)[text_node, minimum width=0.2cm, below= of page, yshift=0.5cm] A page of financial document;

[fill=blue!10, ] (1.2,0.2) – (1.2,-1) – (1.5,-2.1) – (1.5,4.0)
– cycle;

(snippet)[inner sep=0pt, right=of page, yshift= 1cm]
(method) [text_node, minimum width=0.3cm, above= of snippet, yshift=-0.2cm, xshift=-2cm] Calculation method for issue price;
(issue) [text_node, minimum width=0.3cm, above= of snippet, yshift=-0.2cm, xshift=2cm] Issue conditions and procedures;
(point1)[draw=white, below = of method, yshift=0.2cm];
(point2)[draw=white, below = of issue, yshift=-3.3cm];
[arrow, ->, thick] (method)–(point1);
[arrow, ->, thick] (issue)–(point2);

A Snippet of Financial Document

Before presenting them to the public,
financial documents should be inspected by the regulator.
One of the main inspection tasks is to check the completeness of the documents.
In practice, the regulator defines content requirements for each type of financial document, specifying types of information that should be present within a document.
For example, for a UCITS prospectus submitted to the financial regulator in Luxembourg,
regulatory requirements stipulate about 124 information types, including for example the description of the calculation method for issue price and issue conditions and procedures (as presented in Fig. <ref>).
The regulator must analyze the prospectus to identify the sentences discussing each information type specified by content requirements.
Since missing information may cause legal problems and severe investment losses,
ensuring content completeness is usually the first and most fundamental procedure before looking into the details of the financial documents.
However, due to the length and peculiarities of the text in financial documents, this is usually a challenging task.

§ INFORMATION TYPE IDENTIFICATION

Financial document $d$ for analysis Information type $t$ A set of labeled documents $D^L$Number of candidate sentences for analysis $n_c$Related phrase list $\mathit{rlist}$ and unrelated phrase list $\mathit{ulist}$ Threshold $\theta$ for highly similar sentences

Sentences $S$ in $d$ related to $t$
Step 1: Pre-processing
$D^L, d \gets$ preProcessing$(D^L, d)$
Step 2: Identify candidate sentences
$S^L_{r_t}\gets$getRelatedSentences$(D^L, t)$
$S^L_{u_t}\gets$getUrrelatedSentences$(D^L, t)$
Candidates $S^d_{c_t}\gets$candidateSelection$(d, S^L_{r_t}, n_c)$
Step 3: Fine-grained sentence analysis
$s\in S^d_{c_t}$
$s.\mathit{group}, s.\mathit{avg\_s}, s.\mathit{max\_s}, s.\mathit{word}$ $\gets$calcIRSimilarity$(s, S^L_{r_t}, S^L_{u_t})$

Instances $I_\mathit{pos}, I_\mathit{neg} \gets$getInstances$(S^L_{r_t}, S^L_{u_t})$
Classifier $\mathit{cls}\gets$trainRandomForest$(I_\mathit{pos}, I_\mathit{neg})$
$s\in S^d_{c_t}$
$s.\mathit{prob}\gets$calcMLProbability$(cls, s)$

Step 4: Sentence selection
$n_r\gets$ calcAvgNumOfRelatedSentences$(S^L_{r_t})$
$S\gets$ Selector$(S^d_{c_t}, n_r, rlist, ulist, \theta)$
return $S$

In this section, we present our algorithm (: Financial Information Type Identification) to automatically identify the sentences related to an information type in financial documents.
Its pseudocode is shown in Algorithm <ref>.
takes as input a document $d$ for analysis, the information
type $t$ to identify (from the content requirements), labeled
documents $D^L$ (in which sentences related to $t$ have been annotated
by domain experts), and some auxiliary parameters; it returns a set of
sentences $S$ from $d$ related to $t$.

has four main steps:
pre-processing ( <ref>),
candidate sentence identification ( <ref>),
fine-grained sentence analysis ( <ref>), and
sentence selection ( <ref>).

It first pre-processes (line <ref>) the document
with a standard NLP pipeline (including sentence splitting,
tokenization, stop words removal, stemming, named entity recognition). Then, for an information type $t$, identifies a set of
candidate sentences in $d$ for fine-grained analysis
(lines <ref>–<ref>). Next,
analyzes the candidate sentences with information retrieval
(IR) and machine learning (ML); it assigns scores to each candidate
(lines <ref>–<ref>). Last, a
heuristic-based selector is applied to select the final sentences that
are most likely related to $t$ (line <ref>).

§.§ Pre-processing

Financial documents are typically available in PDF format. To ease their
manipulation, we convert them to a plain-text format using an
off-the-shelf converter PDFBox [1].
We then apply a standard NLP pipeline to preprocess the text.
The text is first split into sentences with Stanford CoreNLP [37].
Then, tokenization is applied to identify the words in a sentence.
We remove the stopwords [12] and convert each word into its root form with the Porter stemming algorithm [32].
In addition, the content of financial documents often includes named entities such
as numbers, person names, dates, and web addresses. To leverage the knowledge of these named entities, we perform named entity
recognition [37] on the input document to generalize
these named entities with their category names.

Once the above steps are completed, we obtain a list of preprocessed, simplified words and sentences from the financial documents.
For example, a sentence “Annex I takes effect from 1
January 2016” is processed to “Annex NUMBER take effect DATE”.

§.§ Candidate Sentence Identification

Although financial documents include thousands of sentences, a specific information type is usually addressed by less than 50 sentences.
Conducting fine-grained analysis on the entire document may therefore be
impractical on commodity hardware.
To solve this problem, tries to efficiently filter the majority of unrelated sentences in the pre-processed document $d$ and identify a small number of candidate sentences for further analysis.
The basic hypothesis for candidate sentence identification is that sentences in $d$ that are similar to existing sentences related to $t$ in the labeled documents $D^L$ may also be related to $t$.
Therefore, we calculate similarity between sentences in $d$ and sentences annotated as related to $t$ in $D^L$. We take the top-$n_c$ most similar sentences as candidates for fine-grained analysis.

Specifically, we collect the sentences annotated as related to $t$ in $D^L$.
We transform this group of sentences into a single vector (denoted as “group vector”) using a standard IR model, i.e., the bag-of-words model [Manning et al., 2008].
Given a corpus (e.g., documents in $D^L$), the bag-of-words model gets its vocabulary (i.e., the unique words) and represents a piece of text into a vector, where the length of the vector is equal to the size of the vocabulary.
In our context, each dimension of the group vector means a word in the vocabulary. If the group of related sentences does not contain a word, the value of the corresponding dimension is 0; otherwise, the value is computed by the TF-IDF (Term Frequency-Inverse Document Frequency) [Manning et al., 2008] of the word.
TF-IDF is defined as:

\begin{equation}\label{eq:tf-idf}
\text{TF-IDF}_{w, \mathit{text}}=f_{w, \mathit{text}}\times \log \frac{N}{n_w},
\end{equation}

where $f_{w, \mathit{text}}$ denotes the number of times that $w$ occurs in $\mathit{text}$ (e.g., the group of related sentences),
$N$ is the number of sentences in the corpus,
and $n_w$ is the number of sentences in the corpus that contain $w$.
TF-IDF based vectors assume that sentences can be represented with the frequently used and informative words, where TF (i.e., $f_{w, \mathit{text}}$) calculates the frequency of words and IDF (i.e., $\log \frac{N}{n_w}$) identifies informative words that are not used in almost all the sentences (e.g., “swing” and “dilution”).

In this step, we do not use more complex vectorization models (e.g., deep
learning based sentence embedding), as the bag-of-words model is
easy to deploy and understand; it does not require a large
domain-specific corpus (UCITS prospectuses in our case) to learn the embedding of domain-specific words and phrases, which is not always available.

Further, we vectorize each sentence in document $d$.
For each sentence, we collect its surrounding $n_\mathit{cxt}$ sentences,
which represent the context of the current sentence.
For example, if $n_\mathit{cxt}=1$, the context of a sentence includes its previous sentence and the next sentence.
We transform a sentence and its context into a single vector (hereafter called “context vector”).
We consider the context of a sentence because a single sentence may not contain enough information for our analysis.
The context of a sentence provides valuable information for understanding it.

Last, we compute the similarity between the group vector and the
context vector of each sentence in $d$ using cosine
similarity [Manning et al., 2008], defined as:

\begin{equation}\label{eq:cosine}
\mathit{sim}(\vec{v_1}, \vec{v_2})=\frac{\vec{v_1} \vec{v_2}}{\left| \vec{v_1} \right| \left| \vec{v_2} \right|},
\end{equation}

where $\vec{v_1} \vec{v_2}$ is the inner product of the two vectors and $\left| \vec{v_1} \right| \left| \vec{v_2} \right|$ is the product of the 2-norm for these vectors.
We then rank and select $n_c$ sentences in $d$ as candidates for further analysis.

In this study, the number of surrounding sentences $n_\mathit{cxt}$ is set to 1
and the number of candidate sentences $n_c$ is set to 200.
Our preliminary experiment shows that on average the candidate sentences contain 90% of sentences related to an information type, which can effectively filter 94.2% unrelated sentences and preserve the vast majority of related sentences.

§.§ Fine-grained Sentence Analysis

analyzes the relevance of candidate sentences in $d$ for an information type $t$ (denoted as $S^d_{c_t}$) with different techniques, including both information retrieval (IR) and machine learning (ML).
IR-based analysis calculates the text similarity between a sentence in $S^d_{c_t}$ and the sentences annotated as related to $t$ in $D^L$ (denoted as $S^L_{r_t}$).
It assumes that if a sentence is similar to the existing related sentences in $S^L_{r_t}$,
this sentence is more likely to be related to $t$.
IR-based analysis outputs similarity values for a sentence in $S^d_{c_t}$, which indicate the degree of text similarity of the candidate sentence with the sentences in $S^L_{r_t}$.
ML-based analysis mines a set of measurable properties of sentences (i.e., features) that can distinguish related sentences from unrelated ones.
It represents each sentence with a feature vector, where each dimension is a feature.
ML-based analysis uses the feature vectors of related and unrelated sentences in $D^L$ to train a statistical model.
For a sentence in $S^d_{c_t}$, the trained model outputs a probability value, which indicates the probability that the sentence is related to $t$.

IR- and ML-based techniques analyze sentences from different perspectives: the former calculates the text similarity and the latter trains statistical models with features. They may complement each other.

§.§.§ IR-based analysis

To perform a comprehensive comparison between sentences in $S^d_{c_t}$ and sentences in $S^L_{r_t}$,
calculates similarity at different granularity levels, including group similarity, sentence similarity, and word importance similarity.
Since financial documents may use similar sentences to explain different information types (as explained in section <ref>), we use different granularity levels to better identify the candidate sentences that are similar with the overall context of $S^L_{r_t}$, individual sentences in $S^L_{r_t}$, and the words specified to $t$ at the same time.

Group similarity compares the overall similarity between $S^L_{r_t}$ and every sentence in $S^d_{c_t}$.
It is calculated the same way as discussed in Section <ref>: the group similarity is the cosine similarity between the group vector of $S^L_{r_t}$ and the context vector of a candidate sentence $s$ (denoted as $s.\mathit{group}$).

Sentence similarity calculates the cosine similarity between a candidate sentence $s$ and each sentence in $S^L_{r_t}$ based on their context vectors.
Given $n$ sentences in $S^L_{r_t}$,
we can get $n$ sentence-level similarity values for $s$.
Based on these values,
calculates two sentence-level scores for $s$: the average and the maximum of the $n$ similarity values, denoted as $s.\mathit{avg\_s}$ and $s.\mathit{max\_s}$, respectively.

Word importance similarity analyzes the importance of a word for the information type $t$ based on the labeled documents $D^L$. The score of a sentence in $S^d_{c_t}$ is calculated  according to word importance.
This is automatically determined from three aspects;
precisely, we say that the importance of a word $w$ for $t$ is determined by the extent to which it satisfies the three following conditions:

* $w$ frequently appears in the related sentence set $S^L_{r_t}$;

* $w$ is only present in $S^L_{r_t}$;

* $w$ can be found in the related sentences of every labeled document.

The importance of a word $w$ is defined as:

\begin{equation}\label{eq:importance_word}
\begin{aligned}
\mathit{imp}(w) &= \mathit{freq_{avg}}(w) \times \mathit{spec}(w) \times \mathit{univ}(w) \\
&= \frac {1}{n} \sum_{r_i \in S^L_{r_t}} \frac{\mathit{freq}(r_i, w)}{\left|r_i \right|} \times \frac {\left| S^L_{r_t}(w)\right|}{\left| S^L(w)\right|} \times \frac {\left|D^L_{r_t}(w)\right|}{\left|D^L_{r_t}\right|}
\end{aligned}
\end{equation}

The first factor in the formula calculates the average frequency of $w$ in $S^L_{r_t}$,
where $n$ is the number of sentences in $S^L_{r_t}$,
$\mathit{freq}(r_i, w)$ counts the times that $w$ appears in the $i$th related sentence $r_i$,
and $\left|r_i \right|$ is the number of words in $r_i$.
The second factor considers the specificity of $w$ to $t$,
where $\left| S^L_{r_t}(w)\right|$ is the number of sentences in $S^L_{r_t}$ that contain $w$,
and $\left| S^L(w)\right|$ is the number of sentences in $D^L$ containing $w$.
Third, we analyze the universality of $w$ in the related sentences of $D^L$,
where $\left|D^L_{r_t}(w)\right|$ is the number of documents that have related sentences containing $w$,
and $\left|D^L_{r_t}\right|$ is the number of documents having related sentences.
Intuitively, the importance of a word $w$ is positive correlated with the three conditions:
if $w$ never appears in $S^L_{r_t}$, the values of these conditions are zero; in contrast, these values are 1 if $w$ is the only word in $S^L_{r_t}$ and it never appears in other sentences.
Hence, we multiply the values of three conditions to reflect the importance of $w$.

Based on $\mathit{imp}(w)$ for each word in $D^L$,
the word importance similarity for a sentence $s$ (denoted as $s.\mathit{word}$)
is calculated as $\frac {\sum_{w \in s} \mathit{imp}(w)}{\sum_{w \in S^L_{r_t}} \mathit{imp}(w)}$.
In this definition, $s.\mathit{word}$ is 0 if there is no overlapped word between $s$ and $S^L_{r_t}$, since the $\mathit{imp}(w)$ values of all words in $s$ are zero; otherwise $s$ is more similar with sentences in $S^L_{r_t}$ when it contains many words with high $\mathit{imp}(w)$ values.

§.§.§ ML-based analysis

ML-based analysis trains a statistical model with the sentences in $D^L$ and uses this model to predict the probability that a sentence is related to $t$.
It includes four main steps: training set preparation, feature engineering, model training, and prediction.

In this work, the training set is comprised of the sentences in labeled documents $D^L$.
The positive instances for training are the related sentences $S^L_{r_t}$ for an information type $t$.
Since the majority of sentences in $D^L$ are unrelated, to avoid biases
we perform under-sampling over the unrelated sentences $S^L_{u_t}$ to form the negative instances.
We sample a subset of $S^L_{u_t}$ which are similar to $S^L_{r_t}$ because these sentences are expected to be more difficult to distinguish from $S^L_{r_t}$; we use ML to learn the potential distinguishing criteria.
More specifically, we calculate the similarity between the group vector of $S^L_{r_t}$ and the context vector of each sentence in $S^L_{u_t}$;
we select the top-ranked unrelated sentences according to the size of $S^L_{r_t}$.
These sentences are textually similar with $S^L_{r_t}$.
Besides, a sentence in $S^L_{u_t}$ is also considered as a negative instance if it is the previous or the next sentence of a related sentence.
Since sentences close to each other usually discuss similar topics, ML may learn the reason that some sentences are annotated as unrelated.

Sentence Features for ML-based Analysis
every head row/.style=
before row=,
after row=,
every nth row=1before row=,
every last row/.style=
after row=,
columns/ID/.style =column name=ID,
columns/Details/.style =column name=Name (N), Type (T), Description (D), and Intuition (I), column type=p.9,
col sep=ampersand,
string type,

Regarding feature engineering, we constructed 25 features for model training. Table <ref> shows the name, type, and description of these features, and also their rationale.
F1–10 and F11–20 are 20 features related to the important words and phrases for an information type $t$.
F21–F25 calculate the length of a sentence and different types of named entities that could indicate the existence of an information type.
With these features, each sentence in the training set can be transformed into a feature vector, in which each dimension corresponds to a feature.
The label of the feature vector is 1 or 0, representing whether a sentence is related to $t$ or not.

We trained a random forest model for each information type with the corresponding feature vectors.
Random forest constructs a multitude of decision trees. Each decision tree is trained on a randomly selected subset of the training set.
We use random forest because it is known to better address over-fitting on small datasets [Friedman et al., 2008]; decision trees are also able to automatically identify the most discriminative features for an information type (i.e., feature selection).

During prediction,
we transform candidate sentences into feature vectors and feed each of them into the trained model for the information type $t$, thus obtaining the  probability of each sentence to be related to $t$ (denoted as $s.\mathit{prob}$).

§.§ Sentence Selection

Candidate sentences $S^d_{c_t}$
with $s.\mathit{group}$, $s.\mathit{avg\_s}$, $s.\mathit{max\_s}$, $s.\mathit{word}$, and $s.\mathit{prob}$ of each sentence Average number of related sentences per document $n_r$Related phrase list $\mathit{rlist}$ and unrelated phrase list $\mathit{ulist}$ Threshold $\theta$ for highly similar sentences

Sentences $S$ in $d$ related to $t$
${S^d_{c_t}}^\prime \gets $mergeSimilar$(S^d_{c_t}, \theta)$
$S \gets$selectBySimilarity$({S^d_{c_t}}^\prime, \theta)$
${S^d_{c_t}}^\prime\gets {S^d_{c_t}}^\prime \backslash S$
$s \in {S^d_{c_t}}^\prime$
$s.\mathit{score}\gets (s.\mathit{group} + s.\mathit{avg\_s} + s.\mathit{max\_s} + s.\mathit{word}+s.\mathit{ml})/5$

$i = 0$
$\lvert S \rvert<n_r $ and $ i < \lvert S^{rank}_{c_t} \rvert$
$S = S \cup {S^{rank}_{c_t}}[i++]$

$S\gets \emptyset$
$s_g \in S_g$
Boolean $rCheck\gets$hasWordsInList$(s_g, rlist)$
Boolean $uCheck\gets$ hasWordsInList$(s_g, ulist)$
$rCheck$ and $!uCheck$
$S = \{s_g\} \cup S$

return $S$

For an information type $t$, selects sentences from a document $d$ according to Algorithm <ref>.
The basic idea is to select sentences that are either highly similar with $S^L_{r_t}$ from at least one aspect (i.e., the overall level, the individual sentence level, or the important word level) or ranked higher by the comprehensive score decided by both IR- and ML-based analysis.
Combining with the knowledge from key-phrase lists, we can refine the selected sentences and accurately decide the final related sentence set.
The inputs include the candidate sentences $S^d_{c_t}$ and their
similarity and probability scores, the average number of related
sentences per labeled document $n_r$, a list of domain-specific related
phrases $\mathit{rlist}$ and unrelated phrases $\mathit{ulist}$, and
the auxiliary parameter $\theta$.
Notice that $\mathit{rlist}$ summarizes the phrases frequently used to express $t$; $\mathit{ulist}$ contains the phrases that are synonyms with related phrases but are seldomly used to express $t$.
Since domain experts usually use keyword search to help them find the possible location of related sentences, these lists can be manually constructed when deciding the criteria to annotate the training documents $D^L$.

Before sentence selection,
detects duplicate sentences in $S^d_{c_t}$ (line <ref>).
Two sentences are considered as duplicates if the similarity between their context vectors is larger than a threshold $\theta$.
Duplicate sentences are similar and usually express the same semantic meaning.
will either select or exclude them together.

first selects sentences by their similarity scores.
A sentence (and its duplicates) is selected if one of its similarity score ($s.\mathit{group}$, $s.\mathit{avg\_s}$, $s.\mathit{max\_s}$, or $s.\mathit{word}$) is greater than $\theta$,
because this sentence may express the same meaning as some related sentences in $D^L$.
For the unselected sentences, we calculate a sentence score for each sentence according to its similarity (from IR) and probability (from ML) scores  (lines <ref>–<ref>).
We rank sentences by their sentence scores and select the top-ranked sentences (and their duplicates) until the number of selected sentences reaches $n_r$ (lines <ref>–<ref>).

Last, we group the selected sentences based on their position in the document, because information types are usually addressed by several continuous sentences.
We put any two sentences into a group if the distance between them is
less than three sentences, since these sentences usually share the same context.
For example, we put sentences $s_i$ and $s_{i+2}$ into a group as they have the same context sentence $s_{i+1}$.
For each group of sentences, we check whether they contain domain-specific phrases in the related list $\mathit{rlist}$ or unrelated list $\mathit{ulist}$.
We annotate a group of sentences as related if they feature phrases in $\mathit{rlist}$ but no phrase in $\mathit{ulist}$ (lines <ref>–<ref>).

§ EVALUATION

In this section, we evaluate our approach
() for financial information type identification. First, we assess the
accuracy of in identifying the sentences related to an information type. Then, we evaluate the factors that impact the accuracy of , including different AI techniques and the size of the training set. Last, we analyze how helps inspect financial documents.
More specifically, we answer the following research questions:

* Can accurately identify the information types in financial documents?

* How do different AI techniques (i.e., IR and ML) affect the accuracy of ?

* What is the impact of the size of labeled documents on the accuracy of ?

* How can support the compliance analysis of financial documents?

§.§ Dataset and Settings

We evaluate with the content requirements for UCITS prospectuses, because UCITS is one of the most popular and representative investment regulatory frameworks, which has over 10 trillion of assets under management across the world [15].

We randomly collected 100 approved UCITS prospectuses from the official website of the regulator body in Luxembourg[CSSF approved prospectuses. <https://www.bourse.lu/home>] as our dataset.
All the documents were annotated by three domain experts. Due to the
time required for annotating documents,
the domain experts selected five representative information types for
evaluation, including disclaimer on periodical reports (T1),
calculation frequency for issue price (T2), calculation
method for issue price (T3), liquidation conditions and
procedure (T4), and issue conditions and procedure (T5).
They selected these information types by considering their complexity
and diversity. First, manually identifying
these information types is time-consuming, as one has to identify relevant
sentences among, on average, 3000 sentences (i.e., the
average length of a prospectus document). Second, these information
types are diverse in terms of the average number of related sentences
(i.e., 3.7 for T1, 8.0 for T2, 18.1 for T3, 16.1 for T4, and 47.3 for
T5). Overall, choosing this selection allowed us to
assess how identifies information types specified at
different levels of details (i.e., from a single sentence to several

The annotation was conducted in two phases. First, the domain experts
selected 50 documents from the dataset. They perused these documents
to define the detailed criteria for annotation (i.e., what types of
sentences/phrases should be related/unrelated to an information
type). During the second phase, domain experts annotated all 100
documents with the selected information types based on the established
annotation criteria. Each person annotated a disjoint subset of
documents individually. They then examined others' annotations and
discussed the possible inconsistency to achieve the final annotations.
The two phases took about three months, including one month for
defining annotation criteria and two months for the actual annotation.

The phrase lists $\mathit{rlist}$ and $\mathit{ulist}$  (see
<ref>) used for sentence selection were manually
built based on the phrases listed in the criteria. We extended the
phrases with synonyms occurring in the first 50 documents.
We set the parameter $\theta$ to 0.9; it was decided empirically by
evaluating the accuracy of using a range of values between 0
and 1 with a step of 0.1 on the first group of 50 documents.

We performed the experiments with a
computer running macOS 11.1 with a 2.30 GHz Intel Core
i9 processor and 32GB memory.

§.§ Accuracy of (RQ1)

To answer RQ1, we assessed the accuracy of in identifying sentences related to different information types.

§.§.§ Methodology

We evaluated using the annotated documents with $k$-fold cross-validation ($k=5$).
Since 50 annotated documents were used for phrase list construction and  parameter tuning, we kept them in the training set and only test on the remaining 50 documents.
In each fold, we selected 10 documents from the remaining 50 documents as the test set; the training set included the other 90 documents.
Given an information type $t$ and a test set,
we compared the sentences selected by with the ground
truth annotated by the domain experts.
We measured the accuracy of with
precision, recall, and F$_1$-score ($F_1$).
They are defined as
where true positives (TP) and false positives (FP) refer to sentences selected by which are related or not to  $t$, respectively.
False negatives (FN) refer to cases where misses a sentence related to  $t$.
F$_1$-score is
defined as
$F_1=\frac{2\times \mathit{Precision} \times \mathit{Recall}  }{\mathit{Precision}+\mathit{Recall}}$.

We compared with a keyword-search strategy (denoted as KW),
which is a common way for regulator agents to locate sentences.
Since we have constructed two phrase lists containing related phrases (i.e., $\mathit{rlist}$) and unrelated phrases (i.e., $\mathit{ulist}$), KW directly searched the candidate sentences with the phrase lists and selected the sentences that feature phrases in $\mathit{rlist}$ but no phrase in $\mathit{ulist}$ as related to an information type.

§.§.§ Results

Accuracy of Information Type Identification Algorithms
every head row/.style=
before row=2.5*ID   2cPrecision   2cRecall   2cF$_1$-score

after row=,
every last row/.style=
after row=,
every nth row=5before row=,
columns/ID/.style =column name=,
columns/P-KW/.style =column name=KW,
columns/P-FITI/.style=column name=,
columns/R-KW/.style =column name=KW,
columns/R-FITI/.style=column name=,
columns/F-KW/.style =column name=KW,
columns/F-FITI/.style =column name=,
col sep=comma,
string type,

Table <ref> shows the accuracy of the two algorithms for identifying information types.
identified a set of sentences related to each information
type with a precision value ranging from .
The recall value of ranges from , with an average recall value of .
The result means that could find, on average, more than 60% of sentences for an information type.

KW performed poorly compared to , identifying an average of
related sentences with an average precision value of
.  outperformed KW by 0.277 ( vs
) in terms of F$_1$-score.  Although keyword search is a
common activity for regulator agents, many false positives can be
returned due to the similarity of sentences in the same financial
document. Moreover, since the writing styles and the number of related sentences can be different for financial documents from different investment companies, regulator agents may not enumerate all keywords
for every related sentence, leading to a low recall value.  On the
contrary, could leverage these (possibly incomplete) keyword
lists to improve its accuracy in identifying information types.

When answering this RQ, it is important to remark that in the context
of (financial) document compliance checking, it is desirable to use
identification algorithms that yield a high precision value.  As
discussed at the end of section <ref>, after performing
candidate sentence identification, the top 200 candidate sentences
cover about 90% of the related sentences. Despite this high recall
value, the remaining 200 sentences are more difficult to distinguish;
users have to carefully read these sentences (or the associated
paragraphs) to find related sentences.  With an average
precision above 80%, can help users efficiently
locate the correct position of related sentences.

The answer to RQ1 is that identifies an average of 64.6% of relevant sentences for an information type, with an average precision value of , significantly outperforming the baseline based on keywords.

§.§ Impact of Different Components (RQ2)

selects related sentences according to the results from both
the IR- and the ML-based analyses (section <ref>). To answer RQ2, we assessed the impact of these two types of analysis on the
accuracy of .

§.§.§ Methodology

We implemented two variants of
(called $\text{\approach}_\mathit{IR}$ and $\text{\approach}_\mathit{ML}$) to assess the possible impact of each technique.
During sentence selection, $\text{\approach}_\mathit{IR}$ only selects
sentences based on the similarity values calculated in the IR-based
analysis, while $\text{\approach}_\mathit{ML}$ performs sentence
selection only relying on the probability calculated in the ML-based analysis.
To implement $\text{\approach}_\mathit{IR}$, we calculated the score of a sentence by averaging the four similarity values (i.e., $s.\mathit{group}$, $s.\mathit{avg\_s}$, $s.\mathit{max\_s}$, $s.\mathit{word}$) at line <ref> in Algorithm <ref>.
To implement $\text{\approach}_\mathit{ML}$, we disabled the function selectBySimilarity at line <ref> and assigned the score of a sentence as the probability value calculated from the ML-based analysis ($s.\mathit{ml}$).
We ran the vanilla version of (i.e., the one presented in
section <ref>) and the additional variants using the same settings
as in RQ1.

§.§.§ Results

[Precision of and its variants]

[ybar = .05cm,
bar width = 6pt,
xmin = 0,
xmax = 6,
ymax = 1,
enlarge x limits = value = .25,
xticklabels=T1, T2, T3, T4, T5,
ylabel near ticks,
legend pos=north west,

table [x=ID, y=FITI-IR-P, col sep=comma] data/rq-component.csv;$\text{\approach}_\mathit{IR}$;
table [x=ID, y=FITI-ML-P, col sep=comma] data/rq-component.csv;$\text{\approach}_\mathit{ML}$;
table [x=ID, y=FITI-P, col sep=comma] data/rq-component.csv;

[Recall of and its variants]

[ybar = .05cm,
bar width = 6pt,
xmin = 0,
xmax = 6,
ymax = 1,
enlarge x limits = value = .25,
xticklabels=T1, T2, T3, T4, T5,
ylabel near ticks,
legend pos=north west,

table [x=ID, y=FITI-IR-R, col sep=comma] data/rq-component.csv;$\text{\approach}_\mathit{IR}$;
table [x=ID, y=FITI-ML-R, col sep=comma] data/rq-component.csv;$\text{\approach}_\mathit{ML}$;
table [x=ID, y=FITI-R, col sep=comma] data/rq-component.csv;

[F$_1$-score of and its variants]

[ybar = .05cm,
bar width = 6pt,
xmin = 0,
xmax = 6,
ymax = 1,
enlarge x limits = value = .25,
xticklabels=T1, T2, T3, T4, T5,
ylabel near ticks,
legend pos=north west,

table [x=ID, y=FITI-IR-F, col sep=comma] data/rq-component.csv;$\text{\approach}_\mathit{IR}$;
table [x=ID, y=FITI-ML-F, col sep=comma] data/rq-component.csv;$\text{\approach}_\mathit{ML}$;
table [x=ID, y=FITI-F, col sep=comma] data/rq-component.csv;

Impact of IR-based and ML-based analyses for

As presented in Figure <ref>, IR- and ML-based analysis show different abilities in analyzing information types.
ML-based analysis tends to assign a high probability value to a small fraction of related sentences.
Hence, $\text{\approach}_\mathit{ML}$ achieves higher precision values for the majority of information types (i.e., T2 to T5) than $\text{\approach}_\mathit{IR}$.
The precision values of T2, T3, and T4 are also higher than those of (in Figure <ref>).
However, the recall of $\text{\approach}_\mathit{ML}$ is lower than both $\text{\approach}_\mathit{IR}$ and (in Figure <ref>).
As to $\text{\approach}_\mathit{IR}$, it tends to retrieve more related sentences than $\text{\approach}_\mathit{ML}$, leading to a high recall, though some false positives are included.

By integrating the two components, the accuracy in identifying information types is further improved: achieves a higher F$_1$-score than both $\text{\approach}_\mathit{IR}$ and $\text{\approach}_\mathit{ML}$ (in Figure <ref>).

The answer to RQ2 is that both IR-based and
the ML-based analyses contribute to improving the accuracy of
, complementing each other.

§.§ Impact of the Size of the Training Set (RQ3)

conducts information type identification relying on the  documents in the training set annotated by domain experts.
This RQ assesses the impact of the size of the training set on the accuracy of .

§.§.§ Methodology

We evaluated the accuracy of by varying the size of the training set from 1090 documents with a step of $10$.
Specifically, we randomly select a subset of the training set as a sampled training set.
We trained on each sampled training set and used the trained model to identify sentences in the test set for different information types.
We measured the accuracy of for different training set sizes, as when addressing RQ1.

§.§.§ Results

[Precision of ]

xlabel=Number of training documents,
ylabel near ticks,
xmin=0, xmax=100,
legend style=
legend columns=-1
grid style=dashed,
cycle list=
solid, every mark/.append style=solid, fill=red, mark=*
dotted, every mark/.append style=solid, fill=green, mark=square*
densely dotted, every mark/.append style=solid, fill=blue, mark=otimes*
loosely dotted, every mark/.append style=solid, fill=cyan, mark=triangle*
dashed, every mark/.append style=solid, fill=yellow,mark=diamond*
loosely dashed, every mark/.append style=solid, fill=olive,mark=*
densely dashed, every mark/.append style=solid, fill=orange,mark=square*
dashdotted, every mark/.append style=solid, fill=gray,mark=otimes*
dasdotdotted, every mark/.append style=solid, fill=brown,mark=star
densely dashdotted,every mark/.append style=solid, fill=teal,mark=diamond*

table [x=ID, y=T1, col sep=comma] data/rq-train-size-p.csv;T1;
table [x=ID, y=T2, col sep=comma] data/rq-train-size-p.csv;T2;
table [x=ID, y=T3, col sep=comma] data/rq-train-size-p.csv;
table [x=ID, y=T4, col sep=comma] data/rq-train-size-p.csv;
table [x=ID, y=T5, col sep=comma] data/rq-train-size-p.csv;

[Recall of ]

xlabel=Number of training documents,
ylabel near ticks,
xmin=0, xmax=100,
legend style=
legend columns=-1
grid style=dashed,
cycle list=
solid, every mark/.append style=solid, fill=red, mark=*
dotted, every mark/.append style=solid, fill=green, mark=square*
densely dotted, every mark/.append style=solid, fill=blue, mark=otimes*
loosely dotted, every mark/.append style=solid, fill=cyan, mark=triangle*
dashed, every mark/.append style=solid, fill=yellow,mark=diamond*

table [x=ID, y=T1, col sep=comma] data/rq-train-size-r.csv;T1;
table [x=ID, y=T2, col sep=comma] data/rq-train-size-r.csv;T2;
table [x=ID, y=T3, col sep=comma] data/rq-train-size-r.csv;
table [x=ID, y=T4, col sep=comma] data/rq-train-size-r.csv;
table [x=ID, y=T5, col sep=comma] data/rq-train-size-r.csv;

[F$_1$-score of ]

xlabel=Number of training documents,
ylabel near ticks,
xmin=0, xmax=100,
legend style=
legend columns=-1
grid style=dashed,
cycle list=
solid, every mark/.append style=solid, fill=red, mark=*
dotted, every mark/.append style=solid, fill=green, mark=square*
densely dotted, every mark/.append style=solid, fill=blue, mark=otimes*
loosely dotted, every mark/.append style=solid, fill=cyan, mark=triangle*
dashed, every mark/.append style=solid, fill=yellow,mark=diamond*

table [x=ID, y=T1, col sep=comma] data/rq-train-size-f.csv;T1;
table [x=ID, y=T2, col sep=comma] data/rq-train-size-f.csv;T2;
table [x=ID, y=T3, col sep=comma] data/rq-train-size-f.csv;
table [x=ID, y=T4, col sep=comma] data/rq-train-size-f.csv;
table [x=ID, y=T5, col sep=comma] data/rq-train-size-f.csv;

Impact of the size of the training set for

As shown in Figure <ref>,  precision
significant increases (i.e., in the case of T1) or becomes relatively stable
within a certain range (i.e., in the case of T2, T3, T4, and T5) as the size of the training set increases.
With more than 40 annotated documents, precision is stable, varying within a 10% range for all information types.
Further, we observe that the precision value is still higher than 70% for most information types (i.e., in the case of T2, T3, T4, and T5) when there are only 10 annotated documents.
This is because is able to identify a small fraction of sentences that are highly similar to the annotated related sentences in the training set.
However, as shown in Figure <ref>, the recall value of is low; for information types T2, T3, and T5, recall is around 40%,
As more documents are annotated, recall significantly increases, because may learn more about different expressions or wordings to explain an information type.
F$_1$-score also confirms the positive impact of increasing the size of the training set on .

The answer to RQ3 is that the accuracy of
improves as the size of the training set increases. requires a minimum size of 40 annotated documents to achieve high and stable precision.

§.§ for Financial Document Compliance Checking (RQ4)

To answer RQ4, we simulate the scenario of compliance checking for financial content requirements and analyze the accuracy of in this scenario.

§.§.§ Methodology

In our experiments, we assessed the accuracy of in
identifying information types with a set of approved prospectuses,
which satisfy all content requirements.  However, in actual compliance checking cases, regulator agents may also inspect
(unapproved) prospectuses that do not fulfill some content requirement
(i.e., they lack some specific information).  We simulated this
scenario to understand how can help regulator agents check
financial documents for compliance.

Since unapproved prospectuses are usually unavailable due to
confidentiality reasons, we simulated such prospectuses by removing
sentences related to an information type.  Specifically, we first
sampled 10 documents from the 50 testing documents as unapproved
prospectuses, to simulate the case where a small subset of documents is incomplete.  Second, we removed the pages containing sentences
related to the five information types.  We removed the whole page,
because when explaining the core content of an information type, submitters may also write
additional sentences near the related sentences (which are usually on the same page) to introduce the
context or background.  We assume that for missing information types,
submitters would also omit these sentences.  Additionally, we manually
checked the documents to remove sentences that may indicate the
existence of an information type. For example, for issue
condition and procedure (T5), a prospectus  contains the sentence
“shareholders should consult the Chapter How to Subscribe For
Shares” which determines the position of T5 in the text.  Lastly, we built a new test set containing 50 testing documents, ten of which are incomplete in terms of information types.

We ran on the new test set using the same cross-validation setting as in RQ1.
When reports that it could not find any sentence related to some information types, regulators are warned of missing information types.
We defined $\mathit{Precision}=\frac{|\mathit{TP}|}{|\mathit{TP}|+|\mathit{FP}|}$
where TP means correctly identifies a missing information type (e.g.,  no related sentence is recommended);
FP represents the case in which reports a missing information type but the document contains some related sentences;
FN corresponds to the case in which  recommends sentences for a missing information type.
We also calculate F$_1$-score according to precision and recall.

§.§.§ Results

Accuracy of in Detecting Missing Information Types
every head row/.style=
before row=,
after row=,
every last row/.style=
after row=,
every nth row=5before row=,
columns/ID/.style =column name=ID,
columns/P/.style =column name=Precision,
columns/R/.style =column name=Recall,
columns/F/.style =column name=F$_1$-score,
col sep=comma,
string type,

As shown in Table <ref>, achieves a precision ranging from  and a recall from  when detecting missing information types on the new test set.
For T1 to T4, identifies between 80% and 90% of the missing information types.
The recall value of T5 is low. We speculate that this is caused by the
high number of sentences (on average, 47.3) in the training set that are related to T5:
many of the annotated sentences may discuss the general background of T5 instead of its core investment information. When using such general sentences to analyze a new prospectus, could wrongly consider similar sentences in other locations as related; hence no warning of missing information types is reported.
Regarding precision, on average 84% of the reported missing information types are correct.

The answer to RQ4 is that detects 80% of the missing
information types with an average precision value of ,
which is a first significant step towards semi-automated financial documents compliance checking.

§.§ Practical Implications

We discuss how can help regulator agents manually inspect
financial documents.  Given a new financial document, can
analyze it and may warn regulator agents of possibly missing
information types.  Regulator agents could then browse the document to
confirm the warnings.  When suggests some sentences,
regulators can quickly read this small set of sentences (the set
usually contains less than 50 sentences, depending on the average
number of related sentences in labeled documents).  Thanks to its high
precision in identifying information types, in most cases can help regulator agents locate the right position of the sentences
related to a certain information type.
As part of future work, we plan to conduct a user study to analyze the
effect of on reducing the inspection time of financial documents.

§.§ Threats to Validity

There are two primary threats to validity in this work. The first relates to the generality of the study. To address this threat, we chose the content requirements  for UCITS prospectuses as it is a representative investment regulatory framework that has managed over 10 trillion of assets across the world for the past 30 years. We evaluated with 100 prospectuses from different investment companies. The difference in writing styles and document structures demonstrates the accuracy of even when analyzing very different financial documents.
Regarding generality, we selected only five representative
information types due to the time required to annotate documents and
build the training set.  To increase generality, we selected the information types by considering their complexity and diversity.
We remark that is easy to deploy for tracing other
information types. It requires domain experts to annotate the related
sentences for the information types in a set of financial documents
that will constitute the training set of the ML model and collect the key-phrase lists during the annotation.

The second threat relates to the process of creating the dataset.
The annotation of information types is a subjective process. Hence,
the annotation was conducted by three domain experts.
They annotated the ground truth independently and discussed possible inconsistencies to mitigate the subjectivity from a single domain expert.
Meanwhile, to answer RQ4, we simulated unapproved prospectuses  by
removing related sentences and any sentence indicating the existence of the information type; these documents may be different from real, unapproved prospectuses. However, the experiments show the accuracy of when identifying missing information types. In the future, we plan to use real unapproved prospectuses to evaluate .

§ RELATED WORK

Our approach is related to work done in the areas of mining financial
data and requirement traceability.

§.§ Mining financial data

Financial data (e.g., financial news, annual financial reports) play a
critical role in improving the quality of financial services and
minimizing the risks of financial activities (e.g., portfolio
selection [Zhang et al., 2020], stock trading strategy
analysis [Nuij et al., 2013], stock price movements prediction [Shi et al., 2018, Feng et al., 2021]).
Existing studies report that financial data from Web media (e.g.,
financial news and discussion boards) has become increasingly salient for analyzing stock markets [Li et al., 2017].
Arslan et al., 2021 and Fan et al., 2020 cluster and classify financial news to help analysts capture the core events in news.

Data mining has been applied not only to financial data from Web
media, but also to financial documents (e.g., annual financial
reports, 10-K). Li et al., 2016 extract financial tables from
annual financial reports, and automatically classify them into income
statements, balance sheets, and cash flow.  Mining financial tables
has also enabled activities like financial data
cross-checking [Li et al., 2020], key performance indicators
tracing [Brito et al., 2019], and financial fraud
detection [Craja et al., 2020, Ravisankar et al., 2011].

As to the analysis of NL sentences in financial
documents, Kumar et al., 2016 present a baseline system
AEFDT to identify financial named entities (e.g., amortization
expense, swing factor). Azzopardi et al., 2018 propose a
controlled NL to write financial statements for
financial service compliance checking. In contrast to these works, in this
paper we focus on the task of tracing content requirements in
financial documents, which is important for financial enterprises and
regulators to ensure the completeness of financial documents and further
enable automated compliance techniques.

§.§ Requirements traceability

Requirements traceability can be considered as a kind of information
extraction [Grishman, 2019], which extracts entities (e.g., named
entities [Li et al., 2020]), relations (the relationship between two
entities [Jiang and Zhai, 2007]), and events (e.g., knowledge about
incidents [Huang et al., 2019]) from the text.
The extracted information can be used for tasks like question answering [Rajpurkar et al., 2018] and knowledge aggregation [Zhang et al., 2015].
However, these tasks usually focus on analyzing small pieces of text (e.g., conversations, newsgroups, and weblogs) [Zhang et al., 2017, Rajpurkar et al., 2018, Weischedel et al., 2011].
Driven by the recent deep learning advances (such as
BERT [Devlin et al., 2019] and SpanBERT [Joshi et al., 2020]), the
effectiveness of information extraction  for analyzing complex
documents has also significantly improved.
However, applying these techniques to a different area (such as
the financial domain) always requires some sort of fine-tuning with domain-specific datasets.
Since fine-tuning is sometimes unstable on small datasets (with less
than 10k training samples) [Zhang et al., 2021], these
techniques cannot be applied in our context.  We indeed need to account for the limited
size of domain-specific datasets, bounded by the high cost of
annotations, which must be performed by domain experts and cannot be,
for example, crowd-sourced.

Requirements traceability has also been studied in the area of software engineering [Cleland-Huang et al., 2014],
where many requirement artifacts are written in NL [Boutkova and Houdek, 2011, Merten et al., 2016, Wang et al., 2020].
Regarding NL requirements, existing work infer trace links between high-level requirements (e.g., regulatory code) and low-level requirements (e.g., requirement specifications and privacy policies).
They usually recast the traceability task into an IR problem: taking high- or low-level requirements as queries to retrieve related or similar sentences from low-level requirements [Hayes et al., 2003].
IR techniques including latent semantic indexing, thesaurus, and relevance feedback have been investigated for this task [Hayes et al., 2006].
To address the term mismatch between high- and low-level requirements,
the domain ontology [Guo et al., 2017], word embedding [Torre et al., 2020], and indicator term mining [Cleland-Huang et al., 2010, Wang et al., 2020] methods
have been explored for better sentence matching.
In addition, for a certain type of artifact (e.g., privacy policies), the NL text can be visualized [Pullonen et al., 2019] or standardized with domain-specific languages [Caramujo et al., 2019] to improve its traceability.

In the aforementioned works, a typical NL requirement (either high or low-level) is usually one or two sentences in length [Hayes et al., 2006];
in contrast, in this work we have focused on complex NL artifacts (i.e., financial documents),
which have thousands of sentences.
Further, the same sentence may have different meanings when the context is different; this is not the case for many artifacts (e.g., privacy policies).
To address these unique challenges, we proposed to fully
consider the context, the content, and indicator words of each sentence
for better tracing financial content requirements.

§ CONCLUSION

In this paper, we proposed , an approach to automatically
identify content requirements in financial documents.
Our approach utilizes IR and ML to conduct analysis, at multiple levels of granularity, on financial documents to understand the context, semantics, and indicator terms of every sentence.
Furthermore, uses a heuristic-based sentence selector, which considers the knowledge of IR, ML, and domain-specific phrases to trace text spans related to the information types specified in the content requirements.
We evaluated by assessing its effectiveness in identifying information types based on 100 financial documents from different investment companies.
Evaluation results show that can accurately retrieve a large percentage of sentences related to  information types, with an average precision value of .
can thus effectively inform regulators about potentially missing information types and assist them in inspecting financial documents.
As part of future work, we plan to investigate the applicability of on other types of financial documents with different information types.

§ ACKNOWLEDGMENTS

This work has been supported by CSSF Luxembourg.

[1]
Apache PDFBox project.

[Arslan et al., 2021]
Yusuf Arslan, Kevin Allix, Lisa Veiber, Cedric Lothritz, Tegawendé F
Bissyandé, Jacques Klein, and Anne Goujon.
A comparison of pre-trained language models for multi-class text
classification in the financial domain.
In WWW'21 Companion, pages 260–268, 2021.

[Azzopardi et al., 2018]
Shaun Azzopardi, Christian Colombo, and Gordon J Pace.
A controlled natural language for financial services compliance
In Controlled Natural Language, pages 11–20. IOS Press, 2018.

[Boutkova and Houdek, 2011]
Ekaterina Boutkova and Frank Houdek.
Semi-automatic identification of features in requirement
In RE'11, pages 313–318. IEEE, 2011.

[Brito et al., 2019]
Eduardo Brito, Rafet Sifa, Christian Bauckhage, Rüdiger Loitz, Uwe
Lohmeier, and Christin Pünt.
A hybrid ai tool to extract key performance indicators from financial
reports for benchmarking.
In DocEng'19, pages 1–4, 2019.

[Caramujo et al., 2019]
João Caramujo, Alberto Rodrigues da Silva, Shaghayegh Monfared, André
Ribeiro, Pável Calado, and Travis Breaux.
Rsl-il4privacy: a domain-specific language for the rigorous
specification of privacy policies.
Requirements Engineering, 240 (1):0 1–26,

[Cleland-Huang et al., 2010]
Jane Cleland-Huang, Adam Czauderna, Marek Gibiec, and John Emenecker.
A machine learning approach for tracing regulatory codes to product
specific requirements.
In ICSE'10, pages 155–164, 2010.

[Cleland-Huang et al., 2014]
Jane Cleland-Huang, Orlena CZ Gotel, Jane Huffman Hayes, Patrick Mäder, and
Andrea Zisman.
Software traceability: trends and future directions.
In FOSE'14, pages 55–69. 2014.

[Craja et al., 2020]
Patricia Craja, Alisa Kim, and Stefan Lessmann.
Deep learning for detecting financial statement fraud.
Decision Support Systems, 139:0 113421, 2020.

[CSSF (The Commission de Surveillance du Secteur Financier), 2010]
CSSF (The Commission de Surveillance du Secteur Financier).
The Luxembourg law of 17 December 2010 relating to undertakings
for collective investment.
<https://www.cssf.lu/wp-content/uploads/L_171210_UCI.pdf>, 2010.

[Devlin et al., 2019]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
Bert: Pre-training of deep bidirectional transformers for language
In NAACL HLT'19, pages 4171–4186, 2019.

[12]
Damian Doyle.
Default english stopwords.

[Fan et al., 2020]
Mengzhen Fan, Dawei Cheng, Fangzhou Yang, Siqiang Luo, Yifeng Luo, Weining
Qian, and Aoying Zhou.
Fusing global domain information and local semantic information to
classify financial documents.
In CIKM'20, pages 2413–2420, 2020.

[Feng et al., 2021]
Fuli Feng, Moxin Li, Cheng Luo, Ritchie Ng, and Tat-Seng Chua.
Hybrid learning to rank for financial event ranking.
In SIGIR'21, pages 233–243, 2021.

[15]
Financial Times.
European ucits fund assets reach 10 tn landmark.

[Friedman et al., 2008]
Jerome Friedman, Trevor Hastie, Robert Tibshirani, et al.
The elements of statistical learning, pages 587–588.
Springer Series in Statistics, New York, second edition, 2008.

[Grishman, 2019]
Ralph Grishman.
Twenty-five years of information extraction.
Natural Language Engineering, 250 (6):0
677–692, 2019.

[Guo et al., 2017]
Jin Guo, Marek Gibiec, and Jane Cleland-Huang.
Tackling the term-mismatch problem in automated trace retrieval.
Empirical Software Engineering, 220 (3):0
1103–1142, 2017.

[Hayes et al., 2003]
Jane Huffman Hayes, Alex Dekhtyar, and James Osborne.
Improving requirements tracing via information retrieval.
In RE'03, pages 138–147. IEEE, 2003.

[Hayes et al., 2006]
Jane Huffman Hayes, Alex Dekhtyar, and Senthil Karthikeyan Sundaram.
Advancing candidate link generation for requirements tracing: The
study of methods.
Trans. on Software Engineering, 320 (1):0 4,

[Huang et al., 2019]
Yin Jou Huang, Jing Lu, Sadao Kurohashi, and Vincent Ng.
Improving event coreference resolution by learning argument
compatibility from unlabeled data.
In NAACL HLT'19, pages 785–795, 2019.

[Jiang and Zhai, 2007]
Jing Jiang and ChengXiang Zhai.
A systematic exploration of the feature space for relation
In NAACL HLT'07, pages 113–120, 2007.

[Joshi et al., 2020]
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer
Spanbert: Improving pre-training by representing and predicting
Trans. of the Association for Computational Linguistics,
8:0 64–77, 2020.

[Kumar et al., 2016]
Aman Kumar, Hassan Alam, Tina Werner, and Manan Vyas.
Experiments in candidate phrase selection for financial named entity
extraction-a demo.
In COLING'16 System Demonstrations, pages 45–48, 2016.

[Li et al., 2020]
Hongwei Li, Qingping Yang, Yixuan Cao, Jiaquan Yao, and Ping Luo.
Cracking tabular presentation diversity for automatic cross-checking
over numerical facts.
In SIGKDD'20, pages 2599–2607, 2020a.

[Li et al., 2020]
Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li.
A survey on deep learning for named entity recognition.
Trans. on Knowledge and Data Engineering, 2020b.

[Li et al., 2017]
Qing Li, Yan Chen, Jun Wang, Yuanzhu Chen, and Hsinchun Chen.
Web media and stock markets: A survey and future directions from a
big data perspective.
Trans. on Knowledge and Data Engineering, 300
(2):0 381–399, 2017.

[Li et al., 2016]
Quanzhi Li, Sameena Shah, and Rui Fang.
Table classification using both structure and content information: A
case study of financial documents.
In Big Data'16, pages 1778–1783. IEEE, 2016.

[Manning et al., 2008]
Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze.
Introduction to information retrieval.
Cambridge university press, 2008.

[Merten et al., 2016]
Thorsten Merten, Matúš Falis, Paul Hübner, Thomas Quirchmayr,
Simone Bürsner, and Barbara Paech.
Software feature request detection in issue tracking systems.
In RE'16, pages 166–175. IEEE, 2016.

[Nuij et al., 2013]
Wijnand Nuij, Viorel Milea, Frederik Hogenboom, Flavius Frasincar, and Uzay
An automated framework for incorporating news into stock trading
Trans. on Knowledge and Data Engineering, 260
(4):0 823–835, 2013.

[32]
Martin Porter.
The porter stemming algorithm.

[Pullonen et al., 2019]
Pille Pullonen, Jake Tom, Raimundas Matulevičius, and Aivo Toots.
Privacy-enhanced bpmn: Enabling data privacy analysis in business
processes models.
Software and Systems Modeling, 180 (6):0
3235–3264, 2019.

[Rajpurkar et al., 2018]
Pranav Rajpurkar, Robin Jia, and Percy Liang.
Know what you don’t know: Unanswerable questions for squad.
In ACL'18, pages 784–789, 2018.

[Ravisankar et al., 2011]
Pediredla Ravisankar, Vadlamani Ravi, G Raghava Rao, and Indranil Bose.
Detection of financial statement fraud and feature selection using
data mining techniques.
Decision Support Systems, 500 (2):0 491–500,

[Shi et al., 2018]
Lei Shi, Zhiyang Teng, Le Wang, Yue Zhang, and Alexander Binder.
Deepclue: Visual interpretation of text-based deep stock prediction.
Trans. on Knowledge and Data Engineering, 310
(6):0 1094–1108, 2018.

[37]
Stanford NLP Group.
Stanford nlp core.

[Torre et al., 2020]
Damiano Torre, Sallam Abualhaija, Mehrdad Sabetzadeh, Lionel Briand, Katrien
Baetens, Peter Goes, and Sylvie Forastier.
An ai-assisted approach for checking the completeness of privacy
policies against gdpr.
In RE'20, pages 136–146. IEEE, 2020.

[Wang et al., 2020]
Wentao Wang, Faryn Dumont, Nan Niu, and Glen Horton.
Detecting software security vulnerabilities via requirements
dependency analysis.
Trans. on Software Engineering, 2020.

[Weischedel et al., 2011]
Ralph Weischedel, Sameer Pradhan, Lance Ramshaw, Martha Palmer, Nianwen Xue,
Mitchell Marcus, Ann Taylor, Craig Greenberg, Eduard Hovy, Robert Belvin,
et al.
Ontonotes release 4.0.
LDC2011T03, Philadelphia, Penn.: Linguistic Data Consortium,

[Zhang et al., 2015]
Congle Zhang, Stephen Soderland, and Daniel S Weld.
Exploiting parallel news streams for unsupervised event extraction.
Trans. of the Association for Computational Linguistics,
3:0 117–129, 2015.

[Zhang et al., 2021]
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi.
Revisiting few-sample bert fine-tuning.
In ICLR'21, 2021.
to appear.

[Zhang et al., 2020]
Yifan Zhang, Peilin Zhao, Bin Li, Qingyao Wu, Junzhou Huang, and Mingkui Tan.
Cost-sensitive portfolio selection via deep reinforcement learning.
Trans. on Knowledge and Data Engineering, 2020.

[Zhang et al., 2017]
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D.
Position-aware attention and supervised data improve slot filling.
In EMNLP'17, pages 35–45, 2017.


$-+$ trajectories intersecting. The initial blue trajectory has now wound
twice, so the winding factor goes away and we get the original soliton data as
given in Eq (153). However, the green trajectory has wound once, hence we have
to correct its soliton data as given in Eq (153) with the winding factor. This
gives us the non-zero incoming $ij$ soliton data as

$\mu_{+-,-3}^{\text{in}}=\mu_{-+,-2}^{\text{in}}=-\mu_{+-,-4}^{\text{in}}=-\mu_{-+,-3}^{\text{in}}=1,$
(156)

and the non-zero incoming $ii$ soliton data is shown in Table 3. Using Eq
(149), we get the outgoing soliton data as shown in Table 4.

$n$ | $-+$ |  $+-$ | $++$ | $--$
---|---|---|---|---
$1$ | $0$ |  $0$ | $-1$ | $1$
$2$ | $1$ |  $0$ | $\frac{1}{2}$ | $-\frac{1}{2}$
$3$ | $-3$ |  $1$ | $-\frac{4}{3}$ | $\frac{4}{3}$
$4$ | $5$ |  $-3$ | $\frac{9}{4}$ | $-\frac{9}{4}$
$5$ | $-9$ |  $5$ | $-\frac{26}{5}$ | $\frac{26}{5}$
$6$ | $19$ |  $-9$ | $\frac{38}{3}$ | $-\frac{38}{3}$
Table 4: Outgoing soliton data $\mu_{ij,-n}^{\text{out}}$ at the fourth
intersection point along the $(++,-\mathbb{N})$ line

We remark that this is the final correction to soliton data with $n\leq 6$
along the $(++,-\mathbb{N})$ line. This is because all the other intersections
on the line are of type $(-+,-n_{1})$ and $(+-,-n_{2})$ such that
$n_{1}+n_{2}>6$. One can say that the soliton data “stabilises” after a
certain number of intersections, or equivalently, we say that the soliton data
stabilises as we go down the $(++,-\mathbb{N})$ line, in the order of
increasing $n$, ie. higher $n$ soliton data gets stabilised after lower $n$
soliton data.

In this way, we have found the soliton data explicitly for the full
$(++,-\mathbb{N})$ line in our universal network $\widetilde{\mathcal{W}}$
upto $n=6$.

## References

  * [1] Sibasish Banerjee, Pietro Longhi, and Mauricio Romo.  Exploring 5d BPS Spectra with Exponential Networks.  Annales Henri Poincare, 20(12):4055–4162, 2019.
  * [2] Sergio Cecotti and Cumrun Vafa.  On classification of N=2 supersymmetric theories.  Commun. Math. Phys., 158:569–644, 1993.
  * [3] Sergio Cecotti, Paul Fendley, Kenneth A. Intriligator, and Cumrun Vafa.  A New supersymmetric index.  Nucl. Phys. B, 386:405–452, 1992.
  * [4] Sergio Cecotti and Cumrun Vafa.  Topological antitopological fusion.  Nucl. Phys. B, 367:359–461, 1991.
  * [5] Sergio Cecotti, Davide Gaiotto, and Cumrun Vafa.  $tt^{*}$ geometry in 3 and 4 dimensions.  JHEP, 05:055, 2014.
  * [6] Albrecht Klemm, Wolfgang Lerche, Peter Mayr, Cumrun Vafa, and Nicholas P. Warner.  Selfdual strings and N=2 supersymmetric field theory.  Nucl. Phys. B, 477:746–766, 1996.
  * [7] Davide Gaiotto, Gregory W. Moore, and Andrew Neitzke.  Spectral networks.  Annales Henri Poincare, 14:1643–1731, 2013.
  * [8] Richard Eager, Sam Alexandre Selmani, and Johannes Walcher.  Exponential Networks and Representations of Quivers.  JHEP, 08:063, 2017.
  * [9] Kunal Gupta and Pietro Longhi.  To appear.
  * [10] Mathew Bullimore and Andrea Ferrari.  Twisted Hilbert Spaces of 3d Supersymmetric Gauge Theories.  JHEP, 08:018, 2018.
  * [11] Sibasish Banerjee, Mauricio Romo, Raphael Senghaas, and Johannes Walcher.  Exponential Networks for Linear Partitions.  3 2024.
  * [12] Hirosi Ooguri and Cumrun Vafa.  Knot invariants and topological strings.  Nucl. Phys. B, 577:419–438, 2000.
  * [13] Tudor Dimofte, Sergei Gukov, and Lotte Hollands.  Vortex Counting and Lagrangian 3-manifolds.  Lett. Math. Phys., 98:225–287, 2011.
  * [14] Mina Aganagic and Cumrun Vafa.  Mirror symmetry, D-branes and counting holomorphic discs.  12 2000.
  * [15] Mina Aganagic, Albrecht Klemm, and Cumrun Vafa.  Disk instantons, mirror symmetry and the duality web.  Z. Naturforsch. A, 57:1–28, 2002.
  * [16] Mina Aganagic, Tobias Ekholm, Lenhard Ng, and Cumrun Vafa.  Topological Strings, D-Model, and Knot Contact Homology.  Adv. Theor. Math. Phys., 18(4):827–956, 2014.
  * [17] Piotr Kucharski, Markus Reineke, Marko Stosic, and Piotr Sułkowski.  BPS states, knots and quivers.  Phys. Rev. D, 96(12):121902, 2017.
  * [18] Tobias Ekholm, Piotr Kucharski, and Pietro Longhi.  Physics and geometry of knots-quivers correspondence.  Commun. Math. Phys., 379(2):361–415, 2020.
  * [19] Kenneth Intriligator and Nathan Seiberg.  Aspects of 3d N=2 Chern-Simons-Matter Theories.  JHEP, 07:079, 2013.
  * [20] Gerald V. Dunne.  Aspects of Chern-Simons theory.  In Les Houches Summer School in Theoretical Physics, Session 69: Topological Aspects of Low-dimensional Systems, 7 1998.
  * [21] Tudor Dimofte and Sergei Gukov.  Chern-Simons Theory and S-duality.  JHEP, 05:109, 2013.
  * [22] Yutaka Yoshida and Katsuyuki Sugiyama.  Localization of three-dimensional $\mathcal{N}=2$ supersymmetric theories on $S^{1}\times D^{2}$.  PTEP, 2020(11):113B02, 2020.
  * [23] Christopher Beem, Tudor Dimofte, and Sara Pasquetti.  Holomorphic Blocks in Three Dimensions.  JHEP, 12:177, 2014.
  * [24] Tudor Dimofte, Davide Gaiotto, and Sergei Gukov.  Gauge Theories Labelled by Three-Manifolds.  Commun. Math. Phys., 325:367–419, 2014.
  * [25] Yuji Terashima and Masahito Yamazaki.  SL(2,R) Chern-Simons, Liouville, and Gauge Theory on Duality Walls.  JHEP, 08:135, 2011.
  * [26] Davide Gaiotto, Gregory W. Moore, and Andrew Neitzke.  Wall-crossing, Hitchin systems, and the WKB approximation.  Adv. Math., 234:239–403, 2013.
  * [27] Maxim Kontsevich and Yan Soibelman.  Stability structures, motivic Donaldson-Thomas invariants and cluster transformations.  11 2008.
  * [28] Davide Gaiotto, Gregory W. Moore, and Andrew Neitzke.  Wall-Crossing in Coupled 2d-4d Systems.  JHEP, 12:082, 2012.
  * [29] Sibasish Banerjee, Pietro Longhi, and Mauricio Romo.  A-branes, Foliations and Localization.  Annales Henri Poincare, 24(4):1077–1136, 2023.
  * [30] Alba Grassi, Qianyu Hao, and Andrew Neitzke.  Exponential Networks, WKB and Topological String.  SIGMA, 19:064, 2023.
  * [31] Murad Alim, Lotte Hollands, and Iván Tulli.  Quantum Curves, Resurgence and Exact WKB.  SIGMA, 19:009, 2023.
  * [32] J. M. F. Labastida and Marcos Marino.  Polynomial invariants for torus knots and topological strings.  Commun. Math. Phys., 217:423–449, 2001.
  * [33] J. M. F. Labastida, Marcos Marino, and Cumrun Vafa.  Knots, links and branes at large N.  JHEP, 11:007, 2000.
  * [34] Dmitry Galakhov, Pietro Longhi, and Gregory W. Moore.  Spectral Networks with Spin.  Commun. Math. Phys., 340(1):171–232, 2015.
  * [35] Ioana Coman, Pietro Longhi, and Jörg Teschner.  From quantum curves to topological string partition functions II.  4 2020.
  * [36] Online Encyclopedia of Integer Sequences.  Sequence A131868 https://oeis.org/A131868.
  * [37] M. Kontsevich R. Stanley Max Alekseyev, O. Gorodetsky.  https://mathoverflow.net/questions/195339/a-congruence-involving-binomial-coefficients, 2015\.