Automatic Detection of Relevant Information, Predictions and Forecasts]Automatic Detection of Relevant Information, Predictions and Forecasts in Financial News through Topic Modelling with Latent Dirichlet Allocation

[1]Silvia<EMAIL_ADDRESS>These authors contributed equally to this work.

1]Francisco de<EMAIL_ADDRESS>These authors contributed equally to this work.

1]Ana<EMAIL_ADDRESS>These authors contributed equally to this work.

1]Francisco J<EMAIL_ADDRESS>These authors contributed equally to this work.

1]Enrique<EMAIL_ADDRESS>These authors contributed equally to this work.

[1]Information Technologies Group, atlanTTic, University of Vigo, E.I. Telecomunicación, Campus Lagoas-Marcosende, Vigo, 36310, Spain

Financial news items are unstructured sources of information that can be mined to extract knowledge for market screening applications. They are typically written by market experts who describe stock market events within the context of social, economic and political change. Manual extraction of relevant information from the continuous stream of finance-related news is cumbersome and beyond the skills of many investors, who, at most, can follow a few sources and authors. Accordingly, we focus on the analysis of financial news to identify relevant text and, within that text, forecasts and predictions. We propose a novel Natural Language Processing (nlp) system to assist investors in the detection of relevant financial events in unstructured textual sources by considering both relevance and temporality at the discursive level. Firstly, we segment the text to group together closely related text. Secondly, we apply co-reference resolution to discover internal dependencies within segments. Finally, we perform relevant topic modelling with Latent Dirichlet Allocation (lda) to separate relevant from less relevant text and then analyse the relevant text using a Machine Learning-oriented temporal approach to identify predictions and speculative statements. Our solution outperformed a rule-based baseline system. We created an experimental data set composed of 2,158 financial news items that were manually labelled by nlp researchers to evaluate our solution. Inter-agreement Alpha-reliability and accuracy values, and rouge-l results endorse its potential as a valuable tool for busy investors. The rouge-l values for the identification of relevant text and predictions/forecasts were 0.662 and 0.982, respectively. To our knowledge, this is the first work to jointly consider relevance and temporality at the discursive level. It contributes to the transfer of human associative discourse capabilities to expert systems through the combination of multi-paragraph topic segmentation and co-reference resolution to separate author expression patterns, topic modelling with lda to detect relevant text, and discursive temporality analysis to identify forecasts and predictions within this text. Our solution may have compelling applications in the financial field, including the possibility of extracting relevant statements on investment strategies to analyse authors' reputations.

§ INTRODUCTION

§.§ Motivation

New efficient algorithms [Manogaran et al, 2018, Delić et al, 2019, Ma et al, 2020, Zuo et al, 2020] and the prolific sources of online information have boosted applied data analysis research. In this scenario, Natural Language Processing (nlp) techniques are being successfully applied to unstructured textual data [Guetterman et al, 2018, Zhang et al, 2019, Balyan et al, 2020, Lu et al, 2022], from the simplest approaches that use morphological information as input [Kumar et al, 2019] to more complex methodologies that take advantage of syntactic patterns and semantic relations [K. and Gupta, 2018].

Financial Knowledge Extraction (ke) is of particular interest. nlp techniques have been used to apply a wealth of market forecasting research to financial news, economic reports and financial expert comments [Xing et al, 2018]. Financial news describes relevant market events, their causes and their possible effects. Transferring human associative discourse capabilities [Lytos et al, 2019] from this type of content is challenging.

§.§ Financial knowledge extraction

Some representative examples of financial ke include information extraction from financial news for firm-based monitoring [Kelly and Ahmad, 2018]; analysis of financial risk such as volatility [Atkins et al, 2018] and Personal Finance Management applications [Isa et al, 2018], among other interesting use cases [Balyan et al, 2020]. Most of these ke systems engineer specific features of the content with their target in mind [Zhang et al, 2019].

It is well known that there is a strong relation between mass media news and stock market state [Cepoi, 2020, Swathi et al, 2022]. Previous research has shown that information published in media outlets or shared financial data in printed media, radio, television, and web sites is correlated with future stock market events [Loughran and McDonald, 2016]. Apart from providing valuable objective information in financial news, authors speculate about market events within political, social and cultural contexts. In these unstructured texts the discourse flows around certain key statements and predictions, and an automatic financial news analysis system should distinguish between less relevant data and predictions to gather knowledge to assist investors in decision making [Lutz et al, 2020].

§.§ Temporality at the discursive level

Temporal representation in texts and speculative statements in particular is based on semantic combinations of certain linguistic structures and elements [Mohamed and Oussalah, 2019]. However, the vast majority of works on temporality research at the discursive level have simply focused on verb tenses [Evers-Vermeul et al, 2017], ignoring their semantic context.

§.§ Research goal and main contribution

Our research is a case of financial News Analysis (na) [Kelly and Ahmad, 2018, Atkins et al, 2018, Isa et al, 2018] within the field of Intelligence Amplification, which has lately gained attention [Jang et al, 2019, Chau et al, 2019, Phi, 2020] as a means of enhancing the understanding and reasoning capabilities of automatic ke solutions and transferring human associative discourse capabilities to expert systems. Our case contributes to solving the problem of extracting relevant text from financial news and, within that relevant text, identifying forecasts and predictions. Our solution may be valuable in helping inexpert stockholders to process more financial news more efficiently.

To the best of our knowledge, this is the first study to propose an approach for the automatic detection of relevant events in financial na based on the joint consideration of relevance and temporality analysis at the discourse level.

§.§ Approach

Our approach comprises:

* Multi-paragraph topic segmentation and co-reference resolution to separate author expression patterns.

* Detection of relevant text through topic modelling with Latent Dirichlet Allocation (lda), outperforming a rule-based system.

* Identification of forecasts and predictions within relevant text using discursive temporality analysis and Machine Learning (ml).

We demonstrate the performance of these features using an experimental data set composed of news items from widely used financial sources. The final data set had 2,158 financial news items similar in size to or even larger than other studies in the literature [Li et al, 2018, Long et al, 2019, Al-Smadi et al, 2019, Zhang and Ghorbani, 2020, de Oliveira Carosia et al, 2021, Xie et al, 2021] that were manually labelled by nlp researchers.

§.§ Structure of the paper

The rest of this article is organised as follows. Section <ref> reviews related work on ke and na solutions. Section <ref> describes our automatic system for detecting relevant financial events based on nlp and ml techniques. Section <ref> presents the text corpus and numerical evaluation of our solution. Finally, Section <ref> concludes the article.

§ RELATED WORK

Stock market research is based on fundamental and technical approaches [Nti et al, 2020]. Fundamental approaches involve performing stock market forecasts using numerical data such as price variation. Technical approaches, in turn, focus on the temporal dimension of financial events. They apply trend modelling techniques to historical asset data forecasts.

Previous research on Data Mining and ke for stock market screening on textual data has considered financial news [Atkins et al, 2018, Carta et al, 2021], stockholder comments in blogs [Swathi et al, 2022] and social networks [Khan et al, 2022]. These systems apply nlp techniques [Xing et al, 2018] or ml models [Delić et al, 2019, Atkins et al, 2018], which may be supervised [Rustam et al, 2020], relying on automatic or manually annotated data sets, or unsupervised [Solorio-Fernández et al, 2020], taking into account the peculiarities of input data and descriptive patterns. The simplest approach consists of using a vector representation of the content and weighting the terms once meaningless elements, such as prepositions [García-Méndez et al, 2020], are removed. More complex approaches, like the one presented by De Arriba-Pérez et al. (2020) [De Arriba-Pérez et al, 2020], seek to identify syntactic and semantic patterns as key descriptors of financial news through lexica, grammar and name entity recognition techniques.

Traditional extraction methods for filtering relevant text in this context comprise manual[E.g., cascaded and non-deterministic finite state automatons, semantic information extraction solutions, etc.] and automatic[E.g., supervised and unsupervised ml methods, being the first more extended [Beliga et al, 2015].] pattern discovery approaches [Kaiser and Miksch, 2005]. The former require large knowledge bases, such as dictionaries and lexica, and rule sets. They tend to be constrained by specific application domains. Automatic approaches include simple statistical and more demanding, complex linguistic approaches, in addition to the previously mentioned ml solutions [Beliga et al, 2015]. tf-idf [Li et al, 2018] is remarkably simple, but it has been reported to under-perform on professional texts, as in our case. Alternative solutions combine the previous techniques with knowledge heuristics such as position, length and text format information. A more competitive solution is fuzzy logic for sentence scoring. However, this lacks adaptability and requires manual rule generation, which directly affects performance [Azhari and Kumar, 2017].

Unsupervised extraction is highly practical because it eliminates the burden of text tagging. Nevertheless, many ke solutions rely on supervised methodologies. Examples those of Gottipati S et al. (2018) [Gottipati et al, 2018], who designed an ml course improvement solution based on student feedback and compared its performance to a rule-based method; López-Úbeda et al. (2021) [López-Úbeda et al, 2021], who extracted relevant information from radiological reports; and Verneer et al. (2019) [Vermeer et al, 2019], who proposed a relevance detection system from social media messages (although they noted the great potential of lda as an alternative).

Among extraction solutions developed to detect relevant topics from news pieces (setting aside temporality analysis), Jacobs et al. (2018) [Jacobs et al, 2018] developed a supervised model for economic event extraction in English news using a sentence-level classification approach, as in our case; Oncharoen et al. (2018) [Oncharoen and Vateekul, 2018] applied the Open Information Extraction system to represent the news data as tuples (actor, action and object); Carta et al. (2021) [Carta et al, 2021] employed a real-time domain-specific clustering-based approach for event extraction in news; and Harb et al. (2008) [Harb et al, 2008] presented a linguistic-based opinion extraction system for blogs.

Assuming there is a direct causal relationship between financial news and asset prices [Cepoi, 2020], some authors have explored both ml and other sophisticated techniques such as deep learning to gather context-dependent information for stock market screening [Shilpa and Shambhavi, 2021]. Worthy of note in this respect is the Naive Bayes model by Atkins et al. (2018) [Atkins et al, 2018] for predicting stock market volatility, which employed as input word-topic correspondence feature vectors obtained with lda. Unlike our proposal, this model considered news content as a whole and did not differentiate non-relevant from relevant parts for their target application. Shilpa & Shambhavi (2021) [Shilpa and Shambhavi, 2021] presented a prediction framework based on sentiment analysis and stock market technical-indicator features. Temporality was not considered.

Prior work has addressed linguistic [Genç et al, 2020], template-based [Abu El-Qumsan and El-Halees, 2018] and statistical news summarisation approaches [Meena and Gopalani, 2020]. State-of-the-art summarisation systems may be extractive [Meena and Gopalani, 2020] or abstractive [Gupta and Gupta, 2019]. Extractive summaries, which are more akin to our goal, extract key sentences directly from the input text. These sentences are ranked by importance and selected if they pass a threshold. Query-focused and update summarisation approaches [Alhoshan and Altwaijry, 2020] also deserve consideration as they retrieve information tailored to a specific audience. The summarisation of online financial news in our work focuses on financial investors. The temporal dimension, expressed as discursive temporality, is crucial to us because relevant text in finance-related news may include in addition to factual information, speculations or predictions, whether quantitative or not. In further relation to summarisation, template-based systems on financial na [Barros et al, 2019] were limited in early research due to their computational load, the laborious task of defining the templates and their lack of flexibility.

ke solutions, and in particular, financial na systems, have not paid sufficient attention to temporal analysis. The vast majority simply use temporal references provided by timestamps or verb tenses. Evers-Vermeul et al. (2017) [Evers-Vermeul et al, 2017], for example, simply noted that as linguistic markers, verb tense suffixes express temporal order and coherence relations through text. Our work goes a step further by analysing sentence-level temporality through syntax and semantics, and detecting temporal elements, expressions and the patterns in which they are arranged.

Comparison with related works.

2*Authorship     2*Source     Relevance     Temporality     2*Technique

analysis     analysis

Atkins A et al. (2018) [Atkins et al, 2018]     2*News     2*     Time series data only     2*lda and ml

Oncharoen P et al. (2018) [Oncharoen and Vateekul, 2018]     3*News     3*     Temporal relations only     3*cnn and lstm

De Arriba-Pérez F et al. (2020) [De Arriba-Pérez et al, 2020]     Micro-blogging data     3*     3*     3*nlp and ml

Carta S et al. (2021) [Carta et al, 2021]     News and micro-blogging data     3*     3*     3*Clustering

Shilpa B et al. (2021) [Shilpa and Shambhavi, 2021]     2*News     2*     2*     2*nlp and ml

5*Our proposal     5*News     5*
5*     Multi-paragraph topic segmentation, co-reference, lda, discursive analysis and ml

Summing up, Table <ref> compares the most relevant work related to our proposal.
Our main contribution is the detection of relevant statements on financial news including forecasts and predictions. To do this, our system automatically groups related data and filters out background information. In brief, we present a novel technique combining lda analysis of automatically segmented news with temporality analysis at the discourse level. To our knowledge, this is the first na approach that jointly considers relevance and temporality at the discourse level.

§ SYSTEM ARCHITECTURE

In this section, we describe our system for the automatic detection of relevant financial events using nlp techniques and ke algorithms. Figure <ref> shows the scheme of the system. First, we segment the input text to group together closely related information. Then, we apply co-reference resolution to discover internal dependencies in the news content among key references to assets. The next stage is the tag processing stage, which consists of the detection, homogenisation and replacement of financial terms. This is followed by relevant topic modelling and temporal analysis to identify predictions and speculative statements, and ultimately provide investors with a synthesised version of financial news highlighting pertinent information, such as asset performance and forecasts/predictions as a summary.

Proposed system.

§.§ Multi-paragraph topic segmentation

The first stage of the multi-paragraph topic segmentation process applies the enhanced version of the TextTiling algorithm [He et al, 2020] to segment news content into subtopic paragraphs.

TextTiling exploits lexical co-occurrence and discourse distribution patterns to identify subtopic paragraphs within a text with reasonable precision from a human's perspective. The algorithm compares, in sequence, the similarity of adjacent text divisions of similar length. If the vocabulary in the first and second parts of the comparison differ, the division is considered a split point. We set a minimum text length of 500 characters to apply the algorithm. Otherwise, no segmentation is performed.

The rationale behind this first stage is the assumption that text segments must be coherent, self-contained information units. Explicit paragraphs of financial news are less useful in our case because, quite often, they just break up the text layout to facilitate readability. It is the informal, inner structure based on key statements and predictions about certain assets or stock markets that interests us.

Table <ref> shows an example of the original content of a financial news item. Table <ref> shows the same item after segmentation with TextTiling (for conciseness, we present just the first two segments). In this example, the first paragraph refers to the current state of the asset and the second describes its past performance. Logically, TextTiling does not always guarantee such a degree of coherence, but in our case, it contributes to the overall efficiency of our system as a building block.

Example of news entry.

Verizon Communications (nyse:vz) is proving to be a stable but undervalued company in the market today. In fact, vz stock is worth at least 55% more than its price today using an analysis of its dividend yield, its own P/E ratio history, and a comparison with its peers. The company reported “boring” earnings, according to Barron’s magazine for Q2 on July 24. But the magazine says “boring is good” in this market. They point out that the communications business is not a bad place to be in a pandemic. For example, Verizon said that its earnings on a non-GAAP adjusted basis was $1.18 per share. This was only 3% lower than a year ago. On an adjusted ebitda (earnings before interest, taxes, depreciation, and amortization) basis, its cash flow was down 5%. However, Verizon generated higher cash flow. For example, first-half 2020 cash flow from operations of $23.6 billion, an increase of $7.7 billion from first-half of 2019. This represents a huge increase of over 206%. Moreover, its free cash flow (fcf) in the first half was $13.7 billion, an increase of 74.1 percent year over year. But consider this. Verizon trades for a paltry 12.3 times this year’s expected earnings and just 12 times next year[...]

Example of news entry after applying multi-paragraph TextTiling segmentation.

ID     TextTiling segment
Verizon Communications (nyse:vz) is proving to be a stable but undervalued company in the market today. In fact, vz stock is worth at least 55% more than its price today using an analysis of its dividend yield, its own P/E ratio history, and a comparison with its peers.

The company reported “boring” earnings, according to Barron’s magazine for Q2 on July 24. But the magazine says “boring is good” in this market. They point out that the communications business is not a bad place to be in a pandemic. For example, Verizon said that its earnings on a non-GAAP adjusted basis was $1.18 per share[...]

§.§ Co-reference resolution

The purpose of co-reference resolution is to replace references with meaningful words, which improves the performance of the subsequent lda stage. Specifically, after segmenting the text, we use the Neural Network (nn) by Clark et al. (2016) [Clark and Manning, 2016] to generate high-dimensional vector representations for co-reference compatibility of cluster pairs. The nn is composed of two task-oriented sub-networks: a mention-pair encoder and a cluster-pair encoder, which create the distributed representations, and the cluster and mention-ranking models to score the pairs of clusters.

Table <ref> provides an example of a news entry after this procedure. Note how implicit asset references have been replaced by explicit ones, which is essential for the next stages.

Example of news entry before and after applying co-reference resolution.

Before co-reference resolution     After co-reference resolution
Verizon Communications (nyse:vz) is proving to be a stable but undervalued company in the market today. In fact, vz stock is worth at least 55% more than its price today using an analysis of its dividend yield, its own P/E ratio history, and a comparison with its peers.

Verizon Communications (nyse:vz) is proving to be a stable but undervalued company in the market today. In fact, vz stock is worth at least 55% more than vz stock price today using an analysis of vz stock dividend yield, vz stock own P/E ratio history, and a comparison with vz stock peers.

§.§ Tag processing: financial term detection, homogenisation and replacement

After text segmentation and co-reference resolution, the tag processing stage homogenises the input for the subsequent lda stage. First, asset identifiers are detected using our financial lexica[Available at <https://www.gti.uvigo.es/index.php/en/resources/14-resources-for-finance->
<knowledge-extraction>, October 2022.] on stock markets, tickers and currencies. In addition, we search for words such as company, enterprise, manufacturer and shareholder, which may refer to an asset. We then detect dependencies between these key terms. Next, we replace all references to stock markets, assets, asset abbreviations and currencies with the tags stock, ticker, ticker_abr and currency, respectively. Using the same lexica, we also replace financial terms and abbreviations with the tag fin_abr.

A search is also made for capitalised proper names in the news items. Initially, these names are checked in the above and replaced by fin_abr, ticker_abr and ticker tags, as appropriate, when there is a match. In the absence of a match, they are replaced by category tags taken from an entity recognition database.

We homogenise numerical values and dates, and group dates and times under the tag date and quantitative terms under the tag num using a Name Entity Recogniser (ner, see Section <ref>). Table <ref> provides a complete example of co-reference resolution and tag processing with detection of financial terms, homogenisation and replacement using the tools specified in Section <ref>.

Example of news content before and after applying financial term detection, homogenisation and replacement procedures.

Before     After
Verizon Communications (nyse:vz) is proving to be a stable but undervalued company in the market today. In fact, vz stock is worth at least 55% more than vz stock price today using an analysis of vz stock dividend yield, vz stock own P/E ratio history, and a comparison with vz stock peers.

ticker (stock:ticker_abr) is proving to be a stable but undervalued company in the market today. In fact, ticker_abr stock is worth at least num more than ticker_abr stock price today using an analysis of ticker_abr stock dividend yield, ticker_abr stock own fin_abr ratio history, and a comparison with ticker_abr stock peers.

§.§ Relevant text detection with lda topic modelling

lda [Jelodar et al, 2019] is an unsupervised algorithm that identifies different topics in a particular document, but it can be generalised to unknown documents if they belong to the same domain and share similar context and structure [Gupta and Katarya, 2021]. We use it to differentiate between relevant and less relevant information in segments (“documents” in this section) produced from financial news content. Our goal is thus to discover relevant information in financial news by separating it from non-relevant information. Note that this type of news has a rather characteristic structure where relevant information is often presented along with precise contextual data and expression patterns, unlike other conventional news items.

Therefore, we employ a Dirichlet distribution with two topics (Equation <ref>). The training algorithm iterates to minimise the number of topics per word and document.

\begin{equation}
\label{eq:lda}
\begin{split}
P(Z,W,\theta,\phi;\alpha,\beta) =
\prod_{j=1}^{M} P(\theta_j;\alpha)
\times
\prod_{i=1}^{2} P(\phi_i;\beta)
\times \prod_{t=1}^{N} P(Z_{j,t} \rvert \theta_j)
P(W_{j,t} \rvert \phi_{Z_{j,t}})
\end{split}
\end{equation}

*  is the set of target topics, two in this work.

*  is the set of words (once stop-words[Available at <bit.ly/3yzvXqJ>, October 2022.] are discarded), with size $N$.

*  is the size of the document collection.

* $\alpha$ and $\beta$ are the symmetric smoothing hyper-parameters to avoid discarding one of the topics due to zero intra-document or intra-corpus topic occurrences. Both hyper-parameters are initialised randomly. Specifically, $\alpha$ and $\beta$ are the topic-document and word-topic densities, respectively. Lower values of $\alpha$ and $\beta$ reduce the variability of topic assignment to specific documents and words.

* $P(\theta_j;\alpha)$ and $P(\phi_i;\beta)$ are the topic-documents and word-topics Dirichlet distributions, respectively.

* $P(Z_{j,t} \rvert \theta_j)$ and $P(W_{j,t} \rvert \phi_{Z_{j,t}})$ are the topic-documents and word-topics multinomial distributions, respectively.

During lda model training, by modifying $\alpha$ and $\beta$, ($i$) topics are assigned randomly to the words in each document, then, ($ii$) the algorithm iterates across the word-topic pairs in different documents generating new assignments and accepting them if they decrease the number of topics per word and document.

The algorithm converges when it finds a solution that minimises the number of intra-document topics and topics per word. Alternatively, an iteration limit can be set. Ultimately, the resulting assignment of words to topics can be used to define a criterion for detecting topics in new text. To this end, when a new sentence is presented to the algorithm, the step ($ii$) is repeated by also taking into account the words in the new sentence. The score of a sentence for a given topic is the number of words of that topic in the sentence divided by the length of the sentence in words. In principle, the sentence is considered to belong to the topic with the largest score. Note that in this estimation, the algorithm is started using the distribution with the best hyper-parameters $\alpha$ and $\beta$ when the training algorithm terminates.

The capability of the system to differentiate between relevant and non-relevant information in the resulting topics is related to two combined effects of data conditioning in previous stages. First, TextTiling groups text by the different expression patterns that the authors of financial news tend to use in relevant and non-relevant text. Second, co-reference resolution and tag processing create a higher density of certain tags in relevant text. For this reason, as a practical contribution, we defined a topic score $\rho$ that represents the density of significant tags stock, ticker, currency and fin_abr in financial news content, which is computed as the percentage of significant tags in a topic divided by the total number of tags in the whole data set. The topic with the highest $\rho$ value is considered relevant. Furthermore, to improve the precision of the lda algorithm in detecting relevant text (that is, its ability to avoid false positives) we introduced another practical contribution: an lda score threshold for accepting a sentence as relevant. It is computed as the minimum value of the configurable parameter $\delta$ and the mean value of the topic scores of the relevant sentences in the same segment.

Table <ref> shows an example of relevant information detection, together with some sentences on the relevant topic from the same segment and the corresponding lda classification scores. The mean value of the scores is 0.878[Note that this mean value is computed with all relevant sentences in the segment, only some of them are contained in Table <ref>.]. Thus, assuming that $\delta=0.8$, even though the third sentence belongs to the relevant topic, the system would consider it irrelevant.

Example of lda analysis of relevant information.

Text     lda score
c]@p10cm@ticker (stock:ticker_abr) is proving to be a stable but undervalued company in the market today.      0.847

c]@p10cm@In fact, ticker_abr stock is worth at least num more than ticker_abr stock price today using an analysis of ticker_abr stock dividend yield, ticker_abr stock own fin_abr ratio history, and a comparison with ticker_abr stock peers.     0.948

c]@p10cm@ticker reported “boring” earnings, according to Barron’s magazine for fin_abr on date.     0.571<0.8

c]@p10cm@For example, ticker said that ticker earnings on a non-fin_abr adjusted basis was num per share.     0.825

c]@p10cm@However, ticker generated higher cash flow.     0.870

c]@p10cm@Moreover, this free cash flow (fin_abr) in the first half was num billion, an increase of num percent year over year.     0.897

c]@p10cm@ticker trades for a paltry num times this year’s expected earnings and just num times next year.     0.934

§.§ Temporal analysis

Table <ref> shows the set of temporal features used to train the ml temporal analysis model. The focal point of this analysis is the use of verbs when referring to stock markets, assets and currencies, but, unlike previous works, we consider them to be part of the semantic context.

Temporal features in the ml temporal analysis model.

Name     Description

c]@p4cm@ {Pst,Prs,Fut}DepSub<EMAIL_ADDRESS>Number of {past,present,future} tense verbs from the dependency analysis when the asset is the subject of the clause.

c]@p4cm@ GlobalDepSub<EMAIL_ADDRESS>Global temporality by majority voting from the dependency analysis when the asset is the subject of the clause.

c]@p4cm@ {Pst,Prs,Fut}DepSubObj<EMAIL_ADDRESS>Number of {past,present,future} tense verbs from the dependency analysis when the asset is the subject or object of the clause.

c]@p4cm@ GlobalDepSubObj<EMAIL_ADDRESS>Global temporality by majority voting from the dependency analysis when the asset is the subject or object of the clause.

c]@p4cm@ {Pst,Prs,Fut}ProxSub<EMAIL_ADDRESS>Number of {past,present,future} tense verbs from the proximity analysis when the asset is the subject of the clause.

c]@p4cm@ GlobalProxSub<EMAIL_ADDRESS>Global temporality by majority voting from the proximity analysis when the asset is the subject of the clause.

c]@p4cm@ {Pst,Prs,Fut}ProxSubObj<EMAIL_ADDRESS>Number of {past,present,future} tense verbs from the proximity analysis when the asset is the subject or object of the clause.

c]@p4cm@ GlobalProxSubObj<EMAIL_ADDRESS>Global temporality by majority voting from the proximity analysis when the asset is the subject or object of the clause.

Within each relevant sentence, we perform a dependency analysis to link verbs to stock markets, assets and currencies and a proximity analysis based on the proximity between the verbs and the key elements identified. The system measures the distance of a term to the nearest verb as the number of intermediate words in both directions. For both analyses (dependency and proximity), we consider whether assets are the subjects or objects of their clauses. For each feature, we estimate the verb tense by majority voting among past, present and future tenses (in case of a tie, the future tense prevails).

Algorithms <ref> and <ref> describe the generation of the temporal features of a segment based on the dependency (Algorithm <ref>) and the proximity (Algorithm <ref>) analysis of the corresponding sentences. Both algorithms have linear time complexity O($n\cdot d$) owing to the d independent loops with a limited number of n elements (subject and object sentences in our case). In the particular scenario in which the number of subject sentences equals the number of object sentences, for each loop and each tense, time complexity is O($2\cdot3\cdot n$) for 2 analyses (subject and object) and 3 tenses (past, present, future). Table <ref> shows an example of the outcome. Note the financial terms and associated verbs. For this segment, for example, features  and  in Table <ref> are set to 1.

: Dependency analysis
[0]
{Pst,Prs,Fut}DepSub = {Pst:0,Prs:0,Fut:0}
{Pst,Prs,Fut}DepSubObj = {Pst:0,Prs:0,Fut:0}
DepSub_text = parsing_subject(text)
DepSubObj_text = parsing_subject_object(text)
tense in [Pst,Prs,Fut]
GlobalDepSub = majority({Pst,Prs,Fut}DepSub)
tense in [Pst,Prs,Fut]
GlobalDepSubObj = majority({Pst,Prs,Fut}DepSubObj)

: Proximity analysis
[0]
{Pst,Prs,Fut}ProxSub = {Pst:0,Prs:0,Fut:0}
{Pst,Prs,Fut}ProxSubObj = {Pst:0,Prs:0,Fut:0}
ProxSub_text = parsing_subject(text)
ProxSubObj_text = parsing_subject_object(text)
tense in [Pst,Prs,Fut]
GlobalProxSub = majority({Pst,Prs,Fut}ProxSub)
tense in [Pst,Prs,Fut]
GlobalProxSubObj = majority({Pst,Prs,Fut}ProxSubObj)

In addition to the temporal features, we also consider textual and numerical features in the ml temporal analysis model. The textual features are char-grams, word-grams and word tokens ($n$-grams within word boundaries), whose parameter ranges are selected by combinatorial searching. The two numerical features are the number of numerical values (excluding percentages) and the number of percentages in the news content.

After empirical tests with diverse ml algorithms, we chose a Linear Support Vector Classifier (svc) to estimate the temporality (past, present, future) of a segment (see our previous work [García-Méndez et al, 2022]). Before training the svc, we pre-processed the clauses in the financial news by converting text to lower case, and removing punctuation marks and non-Unicode characters such as accents and symbols.

Finally, Algorithm <ref> shows the logical flow of the proposed solution.

: Solution pipeline
[0]

segments = TextTiling(financial_news) %Multi-paragraph topic segmentation

s in segments

NeuralCoref(s) %Co-reference resolution

%Financial terms detection, homogenisation and replacement

keywords = ["company","enterprise","manufacturer","shareholder"]

topic_modelling = LDA(segments) %Relevant text detection with LDA topic modelling

highest_$\rho$ = 0

topic in topics

topic_$\rho$ = max(topic_modelling.filterTopic[topic].$\rho$)

topic_$\rho$ > highsest_$\rho$
relevant_topic = topic
highsest_$\rho$ = topic_$\rho$

relevant = topic_modelling.filterTopic[relevant_topic]

sentence in relevant

sentence.score < $\delta$

%Temporal analysis
s in segments

dependency = dependency_analysis(s)

proximity = proximity_analysis(s)

textual = compute_chargrams_wordgrams_tokens(s)

numerical = count_numerical_values_percentages(s)

§ EXPERIMENTAL RESULTS

In this section, we describe the experimental data set and the performance of our system for the detection of relevant financial events. Well-known state-of-the-art metrics were used for the evaluation: Alpha-reliability and accuracy [Krippendorff, 2018], and the Recall-Oriented Understudy for Gisting Evaluation (rouge) metric[Available at <https://github.com/pltrdy/rouge>, October 2022.] [Mohamed and Oussalah, 2019, Sanchez-Gomez et al, 2018, El-Kassas et al, 2020]. Comparisons of the proposed solution with a rule-based baseline and a supervised extraction approach are also provided.

§.§ Experimental setting

The experiments were performed on a computer with the following specifications:

* Operating system: Ubuntu 18.04.2 lts 64 bits

* Processor: IntelCore i9-9900K 3.60 GHz

* RAM: 32 GB DDR4

* Disk: 500 GB (7200 rpm SATA) + 256 GB SSD

Example of news entry after applying dependency and proximity analyses.

Dependency analysis     Proximity analysis
ticker (stock:ticker_abr) is proving to be a stable but undervalued company in the market today. In fact, ticker_abr stock is worth at least 55% more than ticker_abr stock price today using an analysis of ticker_abr stock dividend yield, ticker_abr stock own fin_abr ratio history, and a comparison with ticker_abr stock peers.
ticker reported “boring” earnings, according to Barron’s magazine for Q2 on July 24. But Barron’s magazine for Q2 says “boring is good” in this market. They point out that the communications business is not a bad place to be in a pandemic.
For example, ticker said that ticker earnings on a non-fin_abr adjusted basis was $1.18 per share. This was only 3% lower than a year ago. On an adjusted fin_abr (earnings before interest, taxes, depreciation, and amortization) basis, taxes, depreciation, and amortization) cash flow was down 5%.
However, ticker generated higher cash flow. For example, first-half 2020 cash flow from operations of $23.6 billion, an increase of $7.7 billion from first-half of 2019. This represents a huge increase of over 206%. Moreover, this free cash flow (ticker_abr) in the first half was $13.7 billion, an increase of 74.1 percent year over year.
But consider this. ticker trades for a paltry 12.3 times this year’s expected earnings and just 12 times next year.

ticker (stock:ticker_abr) is proving to be a stable but undervalued company in the market today. In fact, ticker_abr stock is worth at least 55% more than ticker_abr stock price today using an analysis of ticker_abr stock dividend yield, ticker_abr stock own fin_abr ratio history, and a comparison with ticker_abr stock peers.
ticker reported “boring” earnings, according to Barron’s magazine for Q2 on July 24. But Barron’s magazine for Q2 says “boring is good” in this market. They point out that the communications business is not a bad place to be in a pandemic.
For example, ticker said that ticker earnings on a non-fin_abr adjusted basis was $1.18 per share. This was only 3% lower than a year ago. On an adjusted fin_abr (earnings before interest, taxes, depreciation, and amortization) basis, taxes, depreciation, and amortization) cash flow was down 5%.
However, ticker generated higher cash flow. For example, first-half 2020 cash flow from operations of $23.6 billion, an increase of $7.7 billion from first-half of 2019. This represents a huge increase of over 206%. Moreover, this free cash flow (ticker_abr) in the first half was $13.7 billion, an increase of 74.1 percent year over year.
But consider this. ticker trades for a paltry 12.3 times this year’s expected earnings and just 12 times next year.

Regarding the implementation, we took the following decisions:

* Segmentation (Section <ref>): as previously mentioned, we applied the TextTiling algorithm[Available at <https://www.nltk.org/_modules/nltk/tokenize/texttiling.html>, October 2022.]. For this first stage, we used the default parameters of the classifier (see Listing <ref>). Stop-words were also removed.

*  is the pseudo-sentence size, 20 as default.

*  is the size in sentences of the block used in the comparison method, 10 as default.

*  is the method used for determining similarity scores,  as default.

*  is the list of stop-words used, nltk list as default.

*  is the width of the window used by the smoothing method, 2 as default.

*  is the number of smoothing passes, 1 as default.

* Co-reference detection (Section <ref>): we used the nn implementation of the NeuralCoref library[Available at <https://github.com/huggingface/neuralcoref>, October 2022.], which is a pipelined extension of the spaCy library[Available at <https://spacy.io>, October 2022.] to solve co-reference groups in blocks of text. Specifically, we used the pre-trained word embedding statistical model for English with its default features. Surprisingly, in our analysis, we noticed that NeuralCoref did not always detect the indispensable word stock. To circumvent this problem, following a trial and error process, we replaced stock with index before co-reference resolution.

[frame=single,caption=Configuration parameter ranges of the TextTiling algorithm., label=textiling_parameters]
W: 20
K: 10
similarity_method: block comparison
stopwords_list: nltk
smoothing_width: 2
smoothing_rounds: 1

* Tag processing (Section <ref>):
we employ the Freeling library[Available at <http://nlp.lsi.upc.edu/freeling/node/1>, October 2022.] to detect dependencies between terms. Capitalised proper names are detected with the spaCy[Available at <https://spacy.io>, October 2022.] library. If the names are not in our lexica, their categories are selected from spaCy EntityRecognizer tool[Available at <https://spacy.io/api/top-level#spacy.explain> and <https://github.com/explosion/spaCy/blob/master/
spacy/glossary.py>, October 2022.].
Depending on the category, the names are replaced by the following tags: money, person, norp (nationalities, religions or political groups), org (organisations, companies), product, event, and work of art (titles of artworks). Moreover, all elements recognised as loc, fac (buildings) and gpe (countries, cities, states) are grouped under the tag loc (locations).
Numerical values and dates are detected using EntityRecognizer. Terms recognised as date and time are grouped under the tag date, while those recognised as percent, cardinal and quantity are grouped under the tag num.

* Relevant text detection with lda topic modelling (Section <ref>): we employed the LdaMulticore module from the gensim Python library[Available at <https://pypi.org/project/gensim>, October 2022.]. Listing <ref> shows the configuration parameters used to train the model. Through repeated trials, we tuned the algorithm to 50 training ,  to  and  to . Finally, we set $\delta=0.8$.

*  is the number of latent topics to be extracted.

*  is the number of passes to be applied during training.

*  is a useful seed for reproducibility.

*  represents an a-priori belief about document-topic distribution, that is, prior to selection strategies. Its feasible values are: (i)  for symmetric document-topic distribution, (ii)  to use a fixed symmetric distribution of $1.0/num topics$, and (iii)  to use a fixed normalised asymmetric distribution of $1.0/(topic index + sqrt(num topics))$.

*  is an a-priori belief on topic-word distribution. It has the same feasible values as .

* Temporal analysis (Section <ref>): Freeling is used to tag assets as subjects or objects of their clauses to obtain the corresponding temporal features. We used the svc implementation from the Scikit-Learn Python library[Available at <https://scikit-learn.org/stable/supervised
_learning.html#supervised-learning>, October 2022.]. Regarding the parameterisation of the char-grams, word-grams and word tokens textual features, we applied a GridSearchCV[Available at <https://scikit-learn.org/stable/modules/
generated/sklearn.model_selection.GridSearchCV.html>, October 2022.] combinatorial search from Scikit-Learn within the ranges in our prior related work [García-Méndez et al, 2022].
The final choices were  = 0.30,  = 0,  = (2,4) and  = 10000.

*  and  are used to ignore terms with a lower (cut-off) and higher (corpus-specific stop words) document frequency than the given threshold, respectively.

*  indicates the lower and upper boundary for the extraction of word $n$-grams.

*  represents the number of features considered for the best split.

To select the best features for the temporal analysis across the whole set (temporal, textual and numerical features), we used SelectPercentile[Available at <https://scikit-learn.org/stable/modules/
feature_selection.html>, October 2022.] from Scikit-Learn with the $\chi^2$ score function and 80th percentile threshold. The hyper-parameters of the svc were tuned using GridSearchCV with 10-fold cross validation within the ranges in Listing <ref>. The optimal hyper-parameter values used were  = 0.001,  = balanced,  = ,  = 1500,  = ovr,  = l2,  = $10^{-9}$. Finally, the svc was evaluated by 10-fold cross validation using 600 sentences from financial news. This auxiliary data set is similar in size to other sets used in the literature [Li et al, 2018, Long et al, 2019, Al-Smadi et al, 2019, Zhang and Ghorbani, 2020, de Oliveira Carosia et al, 2021, Xie et al, 2021] and did not belong to the experimental data set described in Section <ref> that was used to detect relevant information, and was independently annotated. The svc attained 80.21% precision and 80.40% recall for the auxiliary set.

*  is the regularisation parameter.

*  is used to set the parameter  of the classes. If not given, all classes are assumed to have weight one. The  mode automatically adjusts weights in a manner that is inversely proportional to class frequencies in the input data.

*  represents the loss function.

*  is the hard limit on iterations, or -1 for no limit.

*  represents the one-vs-one scheme.

*  represents the penalty for the model.

*  represents the tolerance for the stopping criterion.

[frame=single,caption=Configuration parameter ranges for the lda model., label=ldamulticore_parameters]
numtopics: 2
passes: (25,50,75,100)
randomstate: 1
alpha|beta: (0.01,0.31,0.61,0.91,symmetric,asymmetric)

[frame=single,caption=Configuration parameter ranges for char-grams, word-grams and word tokens features., label=configuration_parameters]
mindf: (0, 0.002, 0.005, 0.008,0.01)
maxdf: (0.3,0.35,0.4,0.5,0.7,0.8,1)
ngramrange: ((1, 1),(1, 2),(1, 3),(1, 4),(2, 4))
maxfeatures: (10000,20000,30000,None)

[frame=single,caption=Configuration parameter ranges for svc., label=configuration_parameters_svn]
C: (0.0001, 0.0005, 0.001, 0.01)
classweight: [None,balanced]
loss: squared_hinge
maxiter: [500, 1000, 1500]
multiclass: ovr
penalty: l2
tol: (0.000000001, 0.00000001, 0.0000001, 0.000001)

§.§ Experimental data set

Our experimental data set was composed of 2,158 news pieces (average length of 27.98 sentences and 537.24 words). As previously mentioned, the pieces were automatically extracted with a script from popular and prestigious financial websites between 1st October 2018 and 1st October 2020. We filtered the news pieces to keep those that mentioned at least one of the stocks in our financial lexicagtiresources. For text processing purposes, double spaces, line breaks and tabs were replaced by a single space. Finally, we removed, urls, images and graphics and kept the date and author information.

Each entry in the resulting data set is composed of an identifier, a title, content, author information, source and date of publication. The entries are comparable in size to those described in previous ke [Li et al, 2018, Long et al, 2019, Al-Smadi et al, 2019, Zhang and Ghorbani, 2020, de Oliveira Carosia et al, 2021, Xie et al, 2021] and lda works [Park et al, 2019].

A brief descriptive analysis of the data set is given in Table <ref>.

Descriptive analysis of the experimental data set.

1cSource     1cAuthors     1cNews     1cAvg. chars     1cAvg. words

BenzingaAvailable at <https://www.benzinga.com>, October 2022.     152     400     3325.21     536.295

The Motley FoolAvailable at <https://www.fool.com>, October 2022.     75     298     3537.58     576.69

Markets InsiderAvailable at <https://markets.businessinsider.com>, October 2022.     13     223     5270.50     758.61

TipRanksAvailable at <https://www.tipranks.com>, October 2022.     4     210     1505.61     235.84

Business StandardAvailable at <https://www.business-standard.com>, October 2022.     22     198     2615.02     430.00

InvestorplaceAvailable at <https://investorplace.com>, October 2022.     42     158     4967.15     827.32

Iwatch MarketsAvailable at <https://iwatchmarkets.com>, October 2022.     2     154     555.51     95.24

Investing DailyAvailable at <https://www.investingdaily.com>, October 2022.     2     150     6319.58     1011.69

InvestopediaAvailable at <https://www.investopedia.com>, October 2022.     17     140     4462.16     703.70

GurufocusAvailable at <https://www.gurufocus.com>, October 2022.     20     49     4815.94     783.65

CNN BusinessAvailable at <https://edition.cnn.com/markets>, October 2022.     32     49     4507.49     745.49

Market WatchAvailable at <https://www.marketwatch.com>, October 2022.     28	    47     4352.68     721.57

Seeking AlphaAvailable at <https://seekingalpha.com>, October 2022.     13     33     783.67     124.45

CNBCAvailable at <https://www.cnbc.com>, October 2022.     4     31     1962.16     338.87

IOLAvailable at <https://www.iol.co.za>, October 2022.     6     18     1046.00     169.22
Total     15     431Without duplicates.     2158     3335.08     537.24

The texts were annotated by five nlp scientists from the atlanTTic Research Centre for Telecommunication Technologies at the University of Vigo. Manual annotations included relevant texts, asset identifiers and prediction/forecast texts. A number of guidelines were agreed on to enhance consistency (e.g., bold font for relevant text, italics for asset identifiers and underlining for prediction/forecast text). Table <ref> shows an example of an annotated news item from the experimental data set. We used the annotated asset identifiers to improve the content of our financial lexicagtiresources which increased by 3.95%.

Example of annotated news entry.

Verizon Communications (nyse:vz) is proving to be a stable but undervalued company in the market today. In fact, VZ stock is worth at least 55% more than its price today using an analysis of its dividend yield, its own P/E ratio history, and a comparison with its peers.
The company reported “boring” earnings, according to Barron’s magazine for Q2 on July 24. But the magazine says “boring is good” in this market. They point out that the communications business is not a bad place to be in a pandemic. For example, Verizon said that its earnings on a non-GAAP adjusted basis was $1.18 per share. This was only 3% lower than a year ago. On an adjusted ebitda (earnings before interest, taxes, depreciation, and amortization) basis, its cash flow was down 5%. However, Verizon generated higher cash flow. For example, first-half 2020 cash flow from operations of $23.6 billion, an increase of $7.7 billion from first-half of 2019. This represents a huge increase of over 206%. Moreover, its free cash flow (fcf) in the first half was $13.7 billion, an increase of 74.1 percent year over year. But consider this. Verizon trades for a paltry 12.3 times this year’s expected earnings and just 12 times next year[...]

§.§ Inter-agreement evaluation

We evaluated inter-annotator agreement using two well-known state-of-the-art metrics: Alpha-reliability and accuracy.

Table <ref> shows the coincidence matrix of relevance across all annotators. The two components in the diagonal show the number of news sentences on which all the annotators agreed, while the other two components show the cases on which at least one annotator disagreed. Tables <ref> and <ref> show the Alpha-reliability and accuracy coefficients by pairs of annotators. The mean values were 0.552 and 0.861, respectively. Previous works have considered an Alpha-reliability value above 0.41 to be acceptable [Rash et al, 2019, Seité et al, 2019, Salminen et al, 2019, Kilicoglu et al, 2021]. Inter-agreement accuracy was very high, at over 80%.

Coincidence matrix for relevant text annotation.

1l     Relevant     Context
Relevant     2752.5     1561.5

Context     1561.5     16584.5

Inter-agreement Alpha-reliability of relevant text annotation by pairs of annotators (An.).
1c     1cAn. 1     1cAn. 2     1cAn. 3     1cAn. 4     1cAn. 5
1cAn. 1     1c-     1c0.569     1c0.550     1c0.605     1c0.629

1cAn. 2     0.569     1c-     1c0.594     1c0.525     1c0.520

1cAn. 3     0.550     0.594     1c-     1c0.436     1c0.506

1cAn. 4     0.605     0.525     0.436     1c-     1c0.585

1cAn. 5     0.629     0.520     0.506     1c0.585     1c-

Inter-agreement accuracy of relevant text annotation by pairs of annotators (An.).
1c     An. 1     An. 2     An. 3     An. 4     An. 5
An. 1     -     0.862     0.844     0.891     0.895

An. 2     0.862     -     0.853     0.859     0.855

An. 3     0.844     0.853     -     0.818     0.838

An. 4     0.891     0.859     0.818     -     0.894

An. 5     0.895     0.855     0.838     0.894     -

§.§ Discussion of the results

Before applying our system for the automatic detection of relevant financial events, we first defined a simple rule-based system as a baseline. This system sets a relevance score by counting tickers, numbers and percentages and detecting future tenses using Freeling. As previously mentioned, relevant financial text has characteristic context data and expression patterns, and our goal was to determine whether a sophisticated technique such as lda would perform better than trivial rules.

The rule-based baseline approach is as follows. First, the text is segmented into sentences, and all references to stock markets, assets, asset abbreviations and currencies are replaced by the tags stock, ticker, ticker_abr and currency, respectively. As indicated in Section <ref>, financial terms and abbreviations are also replaced by the tag fin_abr. Freeling is applied to detect percentages and numerical values, which are replaced by the tag num. Sentences containing a future tense as detected by Freeling are considered to refer to the future. In brief, a sentence is considered relevant if it contains at least one financial tag (stock, ticker, ticker_abr, currency or fin_abr) and at least one num tag, and predictive if the main verb is in future tense.

Drawing from related work on more powerful supervised extraction strategies [Harb et al, 2008, Gottipati et al, 2018, Vermeer et al, 2019, López-Úbeda et al, 2021], we also applied a svc model[Note that a supervised approach was also used to estimate the temporality (past, present, future) of text in our proposal.] as a second comparison reference. The model was trained using manual annotations on relevant text, including predictions. Textual features were generated and hyper-parameter settings optimised as described in Section <ref>.

Next, we evaluated our system by checking it against the annotated segments. To do this, we employed rouge, a widely used set of metrics for evaluating automatic text extraction performance based on overlapping $n$-grams. We used rouge-l, which measures the longest common sub-sequence between the system output and the annotated news. This rouge variant has been applied as a string matching algorithm to compute the similarity between two texts [Gulden et al, 2019].

Tables <ref> and <ref> show the results obtained for the baseline systems and the proposed system. Even though the problem case is entirely novel (see Section <ref>), the results show that the application of sophisticated nlp techniques and ke algorithms such as those used in our solution results in improved extraction of relevance and temporality from financial news content.

Table <ref> shows the results for the detection of relevant information by the baseline systems and our system for the tags identified by the five annotators. Average values are also provided. In our tests, text was considered relevant when its score for a given topic doubled the score of the other topic[Note that there are only two topics under analysis in this work.]. The remaining text was considered to be less relevant or contextual information. The average rouge-l value across all annotators was 0.662, more than doubling the performance of the rule-based baseline approach. The manually intensive supervised extraction alternative was comparable to our unsupervised approach (in fact the former was often worse, depending on the annotator). This performance can be considered satisfactory in line with other works from the literature [Sanchez-Gomez et al, 2018, El-Kassas et al, 2020, Hark and Karcı, 2020, Alqaisi et al, 2020]. Table <ref> shows the results for the detection of relevant predictions/forecasts after the last svc classifier stage. The average rouge-l value in this case, 0.982, was excellent, with a significant improvement over the rule-based baseline reference of 0.713.

Counterintuitively, the performance of the rule-based baseline in Table <ref> is worse because this approach gives relevance to text that contains quantitative data even if it corresponds to merely contextual information, such as past states of assets and stock markets. It underperformed our proposed solution by an average of 51%. The level of agreement between our system and the annotators for the detection of predictions/forecasts (Table <ref>) was near perfect, with rouge values of more than 0.970 for all annotators. As expected, predictions and forecasts within relevant text are easier to detect than relevant text itself, explaining the lower rouge values in this second case, where the highest coefficient observed was 0.727 (for annotator 1).

rouge-l measures for the detection of relevant text, by annotator (An.) and average.
1c     1cAn. 1     1cAn. 2     1cAn. 3     1cAn. 4     1cAn. 5     1cAvg.

1c¡Rule-based baseline     1c0.257     1c0.390     1c0.333     1c0.390     1c0.246     1c0.323

1cSupervised system     1c0.655     1c0.582     1c0.615     1c0.643     1c0.574     1c0.614

1cProposed system     1c0.727     1c0.681     1c0.720     1c0.547     1c0.633     1c0.662

rouge-l measures the detection of relevant text with predictions/forecasts, by annotator (An.) and average.
1c     1cAn. 1     1cAn. 2     1cAn. 3     1cAn. 4     1cAn. 5     1cAvg.

1cRule-based baseline     1c0.748     1c0.716     1c0.661     1c0.715     1c0.723     1c0.713

1cSupervised system     1c0.984     1c0.905     1c0.906     1c0.938     1c0.945     1c0.936

1cProposed system     1c0.991     1c0.970     1c0.975     1c0.982     1c0.990     1c0.982

§.§ Application use case

Figure <ref> shows a news piece highlighted by our system. Relevant sentences are highlighted in blue, asset identifiers in pink and predictions/forecasts in green. Note the differences with the manual highlighting in Table <ref>. For example, human annotators might mark the sentence “In fact, VZ stock is worth at least 55% more than its price today” as a prediction. With our system, however, both forecasts and relevant, informative sentences are marked.

The dashboard at the bottom of Figure 2 summarises the results, showing the proportion of relevant segments and number of predictions and forecasts. The updated value of the financial asset, taken from Yahoo Finance[Available at <https://finance.yahoo.com>, October 2022.], is shown on the right.

Example of financial event detection.

§ CONCLUSIONS

Many valuable online financial news sources, such as economy journals and web pages (Motley Fool, InvestorDaily, etc.), contain opinions from experts describing relevant market events within sociological, political and/or cultural contexts.

The system proposed in this paper is designed to extract this relevant information, and in particular forecasts and predictions. To do this, it employs nlp techniques. It segments the text and applies lda analysis to filter out less relevant sentences, and then applies discursive temporality analysis to identify predictions and forecasts within the remaining relevant text. The result is a summary of relevant, easy-to-read information. We are not aware of any other ke systems that have applied a similar approach to resolve this problem.

To our knowledge and considering related work, our proposal is the first to jointly consider relevance and temporality at the discursive level. It contributes to transferring human associative discourse capabilities to expert systems by combining (i) multi-paragraph topic segmentation and co-reference resolution to separate author expression patterns, (ii) detection of relevant text through topic modelling with lda, and (iii) identification of forecasts and predictions within relevant text using discursive temporality analysis and ml.

We have created an experimental data set composed of 2,158 financial news items to evaluate our proposal. We have validated its annotation capacity by performing an inter-agreement analysis using Alpha-reliability and accuracy measures and evaluated its performance using the state-of-the-art rouge metric. The system attained rouge-l values of 0.662 and 0.982 for the detection of relevant data and predictions/forecasts, respectively. We also compared the performance of our system with a rule-based baseline system and a fully supervised system (which also performs supervised extraction of relevant text) to evaluate its competitiveness. It outperformed the rule-based system and was comparable to the fully supervised system, which unlike our solution requires manual annotation.

In future work, we plan to extend our research to Spanish and other languages to cover a broader community of investors. We will also evaluate the system in composite (multi-disciplinary) research domains.

§ DECLARATIONS

§.§ Acknowledgement

This version of the article has been accepted for publication, after peer review (when applicable) but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections. The Version of Record is available online at: https://doi.org/10.1007/s10489-023-04452-4

§.§ Funding

This work was partially supported by Xunta de Galicia grants ED481B-2021-118 and ED481B-2022-093, Spain. University of Vigo/CISUG covered the open access fee.

§.§ Competing Interests

The authors have no relevant financial or non-financial interests to disclose.

§.§ Ethics approval

Not applicable

§.§ Consent to participate

Not applicable

§.§ Consent for publication

Not applicable

§.§ Availability of data and material

The data sets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

§.§.§ Code availability

The code used in this work is not publicly available.

§.§.§ Authors' contributions

Silvia García-Méndez: Conceptualisation, Methodology, Software, Validation, Formal analysis, Investigation, Data Curation, Writing - original draft. Francisco de Arriba-Pérez: Conceptualisation, Methodology, Software, Validation, Formal analysis, Investigation, Data Curation, Writing - original draft. Ana Barros-Vila: Software, Validation, Investigation, Resources, Data Curation, Writing - review & editing. Francisco J. González-Castaño: Conceptualisation, Methodology, Data Curation, Writing - review & editing, Supervision. Enrique Costa-Montenegro: Methodology, Data Curation, Writing - review & editing.

[Manogaran et al, 2018]
Manogaran G, Varatharajan R, Lopez D, et al (2018) A new architecture of Internet of Things and big data ecosystem for secured smart healthcare monitoring and alerting system. Future Generation Computer Systems 82:375–387. 10.1016/j.future.2017.10.045

[Delić et al, 2019]
Delić V, Perić Z, Sečujski M, et al (2019) Speech Technology Progress Based on New Machine Learning Paradigm. Computational Intelligence and Neuroscience 2019:1–19. 10.1155/2019/4368036

[Ma et al, 2020]
Ma X, Fei Q, Qin H, et al (2020) A new efficient decision making algorithm based on interval-valued fuzzy soft set. Applied Intelligence 51(6):3226–3240. 10.1007/s10489-020-01915-w

[Zuo et al, 2020]
Zuo Y, Wu Y, Min G, et al (2020) An Intelligent Anomaly Detection Scheme for Micro-Services Architectures With Temporal and Spatial Data Analysis. IEEE Transactions on Cognitive Communications and Networking 6(2):548–561. 10.1109/TCCN.2020.2966615

[Guetterman et al, 2018]
Guetterman TC, Chang T, DeJonckheere M, et al (2018) Augmenting Qualitative Text Analysis with Natural Language Processing: Methodological Study. Journal of Medical Internet Research 20(6):e231. 10.2196/jmir.9702

[Zhang et al, 2019]
Zhang F, Fleyeh H, Wang X, et al (2019) Construction site accident analysis using text mining and natural language processing techniques. Automation in Construction 99:238–248. 10.1016/j.autcon.2018.12.016

[Balyan et al, 2020]
Balyan R, McCarthy KS, McNamara DS (2020) Applying Natural Language Processing and Hierarchical Machine Learning Approaches to Text Difficulty Classification. International Journal of Artificial Intelligence in Education 30(3):337–370. 10.1007/s40593-020-00201-7

[Lu et al, 2022]
Lu X, Deng Y, Sun T, et al (2022) MKPM: Multi keyword-pair matching for natural language sentences. Applied Intelligence 52(2):1878–1892. 10.1007/s10489-021-02306-5

[Kumar et al, 2019]
Kumar S, Kumar MA, Soman K (2019) Deep Learning Based Part-of-Speech Tagging for Malayalam Twitter Data (Special Issue: Deep Learning Techniques for Natural Language Processing). Journal of Intelligent Systems 28(3):423–435. 10.1515/jisys-2017-0520

[K. and Gupta, 2018]
K. V, Gupta D (2018) Unmasking text plagiarism using syntactic-semantic based natural language processing techniques: Comparisons, analysis and challenges. Information Processing & Management 54(3):408–432. 10.1016/j.ipm.2018.01.008

[Xing et al, 2018]
Xing FZ, Cambria E, Welsch RE (2018) Natural language based financial forecasting: a survey. Artificial Intelligence Review 50(1):49–73. 10.1007/s10462-017-9588-9

[Lytos et al, 2019]
Lytos A, Lagkas T, Sarigiannidis P, et al (2019) The evolution of argumentation mining: From models to social media and emerging tools. Information Processing & Management 56(6):102,055. 10.1016/j.ipm.2019.102055

[Kelly and Ahmad, 2018]
Kelly S, Ahmad K (2018) Estimating the impact of domain-specific news sentiment on financial assets. Knowledge-Based Systems 150:116–126. 10.1016/j.knosys.2018.03.004

[Atkins et al, 2018]
Atkins A, Niranjan M, Gerding E (2018) Financial news predicts stock market volatility better than close price. The Journal of Finance and Data Science 4(2):120–137. 10.1016/j.jfds.2018.02.002

[Isa et al, 2018]
Isa K, Rahman Ahmad A, Md. Yusoff R, et al (2018) NEWS Analysis Towards Youth Financial Competency Management. International Journal of Engineering & Technology 7(2.29):1151. 10.14419/ijet.v7i2.29.15146

[Zhang et al, 2019]
Zhang H, Boons F, Batista-Navarro R (2019) Whose story is it anyway? Automatic extraction of accounts from news articles. Information Processing & Management 56(5):1837–1848. 10.1016/j.ipm.2019.02.012

[Cepoi, 2020]
Cepoi CO (2020) Asymmetric dependence between stock market returns and news during COVID-19 financial turmoil. Finance Research Letters 36:101,658. 10.1016/j.frl.2020.101658

[Swathi et al, 2022]
Swathi T, Kasiviswanath N, Rao AA (2022) An optimal deep learning-based LSTM for stock price prediction using twitter sentiment analysis. Applied Intelligence pp 1—-14. 10.1007/s10489-022-03175-2

[Loughran and McDonald, 2016]
Loughran T, McDonald B (2016) Textual Analysis in Accounting and Finance: A Survey. Journal of Accounting Research 54(4):1187–1230. 10.1111/1475-679X.12123

[Lutz et al, 2020]
Lutz B, Pröllochs N, Neumann D (2020) Predicting sentence-level polarity labels of financial news using abnormal stock returns. Expert Systems with Applications 148:113,223. 10.1016/j.eswa.2020.113223

[Mohamed and Oussalah, 2019]
Mohamed M, Oussalah M (2019) SRL-ESA-TextSum: A text summarization approach based on semantic role labeling and explicit semantic analysis. Information Processing & Management 56(4):1356–1372. 10.1016/j.ipm.2019.04.003

[Evers-Vermeul et al, 2017]
Evers-Vermeul J, Hoek J, Scholman MC (2017) On temporality in discourse annotation: Theoretical and practical considerations. Dialogue and Discourse 8(2):1–20. 10.5087/dad.2017.201

[Jang et al, 2019]
Jang Y, Park CH, Seo YS (2019) Fake News Analysis Modeling Using Quote Retweet. Electronics 8(12):1377. 10.3390/electronics8121377

[Chau et al, 2019]
Chau JY, Reyes-Marcelino G, Burnett AC, et al (2019) Hyping health effects: a news analysis of the ‘new smoking' and the role of sitting. British Journal of Sports Medicine 53(16):1039–1040. 10.1136/bjsports-2018-099432

[Phi, 2020]
Phi GT (2020) Framing overtourism: a critical news media analysis. Current Issues in Tourism 23(17):2093–2097. 10.1080/13683500.2019.1618249

[Li et al, 2018]
Li Y, Pan Q, Wang S, et al (2018) A Generative Model for category text generation. Information Sciences 450:301–315. 10.1016/j.ins.2018.03.050

[Long et al, 2019]
Long W, Song L, Tian Y (2019) A new graphic kernel method of stock price trend prediction based on financial news semantic and structural similarity. Expert Systems with Applications 118:411–424. 10.1016/j.eswa.2018.10.008

[Al-Smadi et al, 2019]
Al-Smadi M, Al-Ayyoub M, Jararweh Y, et al (2019) Enhancing Aspect-Based Sentiment Analysis of Arabic Hotels' reviews using morphological, syntactic and semantic features. Information Processing & Management 56(2):308–319. 10.1016/j.ipm.2018.01.006

[Zhang and Ghorbani, 2020]
Zhang X, Ghorbani AA (2020) An overview of online fake news: Characterization, detection, and discussion. Information Processing & Management 57(2):102,025. 10.1016/j.ipm.2019.03.004

[de Oliveira Carosia et al, 2021]
de Oliveira Carosia AE, Coelho GP, da Silva AEA (2021) Investment strategies applied to the Brazilian stock market: A methodology based on Sentiment Analysis with deep learning. Expert Systems with Applications 184:115,470. 10.1016/j.eswa.2021.115470

[Xie et al, 2021]
Xie M, Ye Z, Pan G, et al (2021) Incomplete multi-view subspace clustering with adaptive instance-sample mapping and deep feature fusion. Applied Intelligence 51(8):5584–5597. 10.1007/s10489-020-02138-9

[Nti et al, 2020]
Nti IK, Adekoya AF, Weyori BA (2020) A systematic review of fundamental and technical analysis of stock market predictions. Artificial Intelligence Review 53(4):3007–3057. 10.1007/s10462-019-09754-z

[Carta et al, 2021]
Carta S, Corriga A, Ferreira A, et al (2021) A multi-layer and multi-ensemble stock trader using deep learning and deep reinforcement learning. Applied Intelligence 51(2):889–905. 10.1007/s10489-020-01839-5

[Khan et al, 2022]
Khan W, Ghazanfar MA, Azam MA, et al (2022) Stock market prediction using machine learning classifiers and social media, news. Journal of Ambient Intelligence and Humanized Computing 13(7):3433–3456. 10.1007/s12652-020-01839-w

[Rustam et al, 2020]
Rustam F, Reshi AA, Mehmood A, et al (2020) COVID-19 Future Forecasting Using Supervised Machine Learning Models. IEEE Access 8:101,489–101,499. 10.1109/ACCESS.2020.2997311

[Solorio-Fernández et al, 2020]
Solorio-Fernández S, Carrasco-Ochoa JA, Martínez-Trinidad JF (2020) A review of unsupervised feature selection methods. Artificial Intelligence Review 53(2):907–948. 10.1007/s10462-019-09682-y

[García-Méndez et al, 2020]
García-Méndez S, Fernández-Gavilanes M, Juncal-Martínez J, et al (2020) Identifying Banking Transaction Descriptions via Support Vector Machine Short-Text Classification Based on a Specialized Labelled Corpus. IEEE Access 8:61,642–61,655. 10.1109/ACCESS.2020.2983584

[De Arriba-Pérez et al, 2020]
De Arriba-Pérez F, García-Méndez S, Regueiro-Janeiro JA, et al (2020) Detection of Financial Opportunities in Micro-Blogging Data With a Stacked Classification System. IEEE Access 8:215,679–215,690. 10.1109/ACCESS.2020.3041084

[Beliga et al, 2015]
Beliga S, Meštrović A, Martinčić-Ipšić S (2015) An overview of graph-based keyword extraction methods and approaches. Journal of Information and Organizational Sciences 39(1):1–20

[Kaiser and Miksch, 2005]
Kaiser K, Miksch S (2005) Information Extraction. A survey. Tech. rep., Institute of Software Technology & Interactive Systems, Vienna University of Technology

[Li et al, 2018]
Li C, Guo J, Lu Y, et al (2018) LDA Meets Word2Vec. In: Proceedings of the The Web Conference. ACM Press, pp 1699–1706, 10.1145/3184558.3191629

[Azhari and Kumar, 2017]
Azhari M, Kumar YJ (2017) Improving text summarization using neuro-fuzzy approach. Journal of Information and Telecommunication 1(4):1–14. 10.1080/24751839.2017.1364040

[Gottipati et al, 2018]
Gottipati S, Shankararaman V, Lin JR (2018) Text analytics approach to extract course improvement suggestions from students' feedback. Research and Practice in Technology Enhanced Learning 13(1):6. 10.1186/s41039-018-0073-0

[López-Úbeda et al, 2021]
López-Úbeda P, Díaz-Galiano MC, Ureña-López LA, et al (2021) Pre-trained language models to extract information from radiological reports. In: CEUR Workshop Proceedings, vol 2936. CEUR

[Vermeer et al, 2019]
Vermeer SA, Araujo T, Bernritter SF, et al (2019) Seeing the wood for the trees: How machine learning can help firms in identifying relevant electronic word-of-mouth in social media. International Journal of Research in Marketing 36(3):492–508. 10.1016/j.ijresmar.2019.01.010

[Jacobs et al, 2018]
Jacobs G, Lefever E, Hoste V (2018) Economic Event Detection in Company-Specific News Text. In: Proceedings of the First Workshop on Economics and Natural Language Processing. Association for Computational Linguistics, pp 1–10, 10.18653/v1/W18-3101

[Oncharoen and Vateekul, 2018]
Oncharoen P, Vateekul P (2018) Deep Learning for Stock Market Prediction Using Event Embedding and Technical Indicators. In: Proceedings of the International Conference on Advanced Informatics: Concept Theory and Applications. IEEE, pp 19–24, 10.1109/ICAICTA.2018.8541310

[Carta et al, 2021]
Carta S, Consoli S, Piras L, et al (2021) Event detection in finance using hierarchical clustering algorithms on news and tweets. PeerJ Computer Science 7:e438. 10.7717/peerj-cs.438

[Harb et al, 2008]
Harb A, Plantié M, Dray G, et al (2008) Web opinion mining. In: Proceedings of the 5th international conference on Soft computing as transdisciplinary science and technology. ACM Press, p 211, 10.1145/1456223.1456269

[Shilpa and Shambhavi, 2021]
Shilpa B, Shambhavi B (2021) Combined deep learning classifiers for stock market prediction: integrating stock price and news sentiments. Kybernetes pp 1–26. 10.1108/K-06-2021-0457

[Genç et al, 2020]
Genç S, Akay D, Boran FE, et al (2020) Linguistic summarization of fuzzy social and economic networks: an application on the international trade network. Soft Computing 24(2):1511–1527. 10.1007/s00500-019-03982-9

[Abu El-Qumsan and El-Halees, 2018]
Abu El-Qumsan AY, El-Halees AM (2018) Template Based Medical Reports Summarization. International Journal of Computer Applications 179(17):47–55. 10.5120/ijca2018916301

[Meena and Gopalani, 2020]
Meena YK, Gopalani D (2020) Statistical Features for Extractive Automatic Text Summarization. In: Natural Language Processing: Concepts, Methodologies, Tools, and Applications. IGI Global, p 619–637, 10.4018/978-1-7998-0951-7.ch030

[Gupta and Gupta, 2019]
Gupta S, Gupta SK (2019) Abstractive summarization: An overview of the state of the art. Expert Systems with Applications 121:49–65. 10.1016/j.eswa.2018.12.011

[Alhoshan and Altwaijry, 2020]
Alhoshan M, Altwaijry N (2020) AUSS: An Arabic query-based update-summarization system. Journal of King Saud University Computer and Information Sciences 1:1319–1578. 10.1016/j.jksuci.2020.11.027

[Barros et al, 2019]
Barros C, Lloret E, Saquete E, et al (2019) NATSUM: Narrative abstractive summarization through cross-document timeline generation. Information Processing & Management 56(5):1775–1793. 10.1016/j.ipm.2019.02.010

[He et al, 2020]
He X, Wang J, Zhang Q, et al (2020) Improvement of Text Segmentation TextTiling Algorithm. Journal of Physics: Conference Series 1453:12,008–12,015. 10.1088/1742-6596/1453/1/012008

[Clark and Manning, 2016]
Clark K, Manning CD (2016) Improving Coreference Resolution by Learning Entity-Level Distributed Representations. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pp 643–653, 10.18653/v1/P16-1061, 1606.01323

[Jelodar et al, 2019]
Jelodar H, Wang Y, Yuan C, et al (2019) Latent Dirichlet allocation (LDA) and topic modeling: models, applications, a survey. Multimedia Tools and Applications 78(11):15,169–15,211. 10.1007/s11042-018-6894-4

[Gupta and Katarya, 2021]
Gupta A, Katarya R (2021) PAN-LDA: A latent Dirichlet allocation based novel feature extraction model for COVID-19 data using machine learning. Computers in Biology and Medicine 138:104,920. 10.1016/j.compbiomed.2021.104920

[García-Méndez et al, 2022]
García-Méndez S, de Arriba-Pérez F, Barros-Vila A, et al (2022) Detection of temporality at discourse level on financial news by combining Natural Language Processing and Machine Learning. Expert Systems with Applications 197:116,648. 10.1016/j.eswa.2022.116648

[Krippendorff, 2018]
Krippendorff K (2018) Content Analysis: An Introduction to its Methodology. SAGE Publications, 10.1177/1094428108324513

[Sanchez-Gomez et al, 2018]
Sanchez-Gomez JM, Vega-Rodríguez MA, Pérez CJ (2018) Extractive multi-document text summarization using a multi-objective artificial bee colony optimization approach. Knowledge-Based Systems 159:1–8. 10.1016/j.knosys.2017.11.029

[El-Kassas et al, 2020]
El-Kassas WS, Salama CR, Rafea AA, et al (2020) EdgeSumm: Graph-based framework for automatic text summarization. Information Processing & Management 57:102,264. 10.1016/j.ipm.2020.102264

[Park et al, 2019]
Park H, Park T, Lee YS (2019) Partially collapsed Gibbs sampling for latent Dirichlet allocation. Expert Systems with Applications 131:208–218. 10.1016/j.eswa.2019.04.028

[Rash et al, 2019]
Rash JA, Prkachin KM, Solomon PE, et al (2019) Assessing the efficacy of a manual‐based intervention for improving the detection of facial pain expression. European Journal of Pain 23(5):1006–1019. 10.1002/ejp.1369

[Seité et al, 2019]
Seité S, Khammari A, Benzaquen M, et al (2019) Development and accuracy of an artificial intelligence algorithm for acne grading from smartphone photographs. Experimental Dermatology 28(11):1252–1257. 10.1111/exd.14022

[Salminen et al, 2019]
Salminen J, Almerekhi H, Kamel AM, et al (2019) Online Hate Ratings Vary by Extremes. In: Proceedings of the 2019 Conference on Human Information Interaction and Retrieval. Association for Computational Linguistics, pp 213–217, 10.1145/3295750.3298954

[Kilicoglu et al, 2021]
Kilicoglu H, Rosemblat G, Hoang L, et al (2021) Toward assessing clinical trial publications for reporting transparency. Journal of Biomedical Informatics 116:103,717–103,727. 10.1016/j.jbi.2021.103717

[Gulden et al, 2019]
Gulden C, Kirchner M, Schüttler C, et al (2019) Extractive summarization of clinical trial descriptions. International Journal of Medical Informatics 129:114–121. 10.1016/j.ijmedinf.2019.05.019

[Hark and Karcı, 2020]
Hark C, Karcı A (2020) Karcı summarization: A simple and effective approach for automatic text summarization using Karcı entropy. Information Processing & Management 57(3):102,187. 10.1016/j.ipm.2019.102187

[Alqaisi et al, 2020]
Alqaisi R, Ghanem W, Qaroush A (2020) Extractive Multi-Document Arabic Text Summarization Using Evolutionary Multi-Objective Optimization With K-Medoid Clustering. IEEE Access 8:228,206–228,224. 10.1109/ACCESS.2020.3046494