Recent research efforts have explored the potential of leveraging natural language inference (NLI) techniques to enhance relation extraction (RE).
In this vein, we introduce , a novel adaptation method that harnesses NLI principles to enhance RE performance.
Our approach follows past works by verbalizing relation classes into class-indicative hypotheses, aligning a traditionally multi-class classification task to one of textual entailment. We introduce three key enhancements:
(1) Instead of labeling non-entailed premise-hypothesis pairs with the uninformative “neutral” entailment label, we introduce meta-class analysis, which provides additional context by analyzing overarching meta relationships between classes when assigning entailment labels;
(2) Feasible hypothesis filtering, which removes unlikely hypotheses from consideration based on pairs of entity types; and
(3) Group-based prediction selection, which further improves performance by selecting highly confident predictions.
is conceptually simple and empirically powerful, yielding significant improvements over conventional relation extraction techniques and other NLI formulations.
Our experimental results underscore the versatility of , demonstrating performance gains across both biomedical and general domains.

§ INTRODUCTION

Relation extraction (RE) is an NLP task that distills factual information from text by identifying relationships between entities in the form of fact triplets (e.g.,⟨head, relation, tail⟩) <cit.>. RE facilitates various downstream applications such as knowledge graph construction, question answering, and information retrieval  <cit.>; however, creating datasets for training RE models is costly and challenging, requiring annotators to identify entities and relations across large sections of text <cit.>.

Recent efforts have explored adapting the RE task to one of natural language inference (NLI), which enables leveraging relatively large NLI datasets to improve performance on an RE-adapted task <cit.>. RE-to-NLI works transform relation instances into premises paired with $m$ class-indicative hypotheses where $m$ is the number of relation classes in a dataset. A language model is trained to label premise-hypothesis pairs as entailed, contradicted, or neutral. We build on this line of work by introducing , a novel NLI adaptation method that improves RE performance by leveraging three key enhancements: meta-class analysis, automatic feasible hypothesis filtering, and group-based prediction selection.

Meta-class analysis: In past RE-to-NLI works, if a premise does not entail a hypothesis, the corresponding NLI target assigned is “neutral” <cit.>. However, this misses an implicit training signal we can gain by analyzing the semantics of a dataset's relation classes. When assigning NLI targets to adapted RE instances, we distinguish between task-based mutual exclusivity and definition-based mutual exclusivity, where task-based mutual exclusivity is an artifact of the single-class classification task inherent to a dataset. Each input instance is annotated with a single relation class, thereby arbitrarily making all classes mutually exclusive. In contrast, definition-based mutual exclusivity is derived from the relation classes' definitions. For example, within the BioRED dataset <cit.>, the “positive correlation” class is definitionally mutually exclusive and contradictory to the “negative correlation” class <cit.>. We call this method meta-class analysis and use it to determine the appropriate NLI targets for each premise-hypothesis pair. If two classes are definitionally mutually exclusive, the “contradict” label is applied to the premise-hypothesis pair, which contains the mutually exclusive class-indicative hypothesis, thereby injecting additional information about the meta-relationship between relation classes, allowing the model to exploit this information while learning relationship representations. Leveraging this insight, we can glean multiple informative training signals from a single relation instance when adapting the relation extraction task into the natural language inference task. We show through ablation experiments that adding meta-class analysis leads to significant performance gains on the RE task.

Feasible hypothesis filter: We introduce a feasible hypothesis filter that automatically removes infeasible hypotheses based on the head and tail entity types within a relation instance. We aggregate all valid head and tail entity type-pairs from the training data to construct the set of valid type-pairs corresponding to each relation class. We then leverage the set of valid type-pairs to automatically remove hypotheses that verbalize infeasible relationships. For instance, in the BioRED dataset <cit.>, it is impossible for a gene to “bind” to a disease (i.e., the “bind” label is never applied to a relation instance containing a gene-disease entity type pair). We, therefore, remove the “bind” hypothesis from all instances with gene-disease entity types. This filter improves training efficiency by reducing the number of NLI instances.

Group-based prediction selection: Group-based prediction selection exploits the feature of RE-to-NLI adaptation in that each relation instance is converted into premises paired with $m$ class-indicative hypotheses. When evaluating cases where the model predicts multiple “entail” labels for a single instance, we can select the most confident “entail” prediction and ignore other predictions. The model can naturally abstain from predicting a class by simply predicting “neutral” for any instance. Experimental results demonstrate that this group-based prediction selection method leads to additional performance gains.

as an RE-to-NLI adaptation method is technically domain agnostic; however, it is particularly well-suited for biomedical relation extraction where associations often have opposing classes such as “positively correlated” and “negatively correlated” <cit.> or “agonist” and “antagonist” <cit.> enabling a rich meta-class analysis. We also find that associations in biomedical RE are often type-dependent compared to general domain RE, making the feasible hypothesis filter more effective at trimming the set of feasible hypotheses. Still, we extend our evaluations beyond the biomedical domain to determine how fares on general domain relation extraction datasets. Notably, we observe improvements in both domains, reinforcing the effectiveness and versatility of .

We summarize the main contributions of this work as the following:

* We introduce a novel RE-to-NLI adaptation method, , and showcase its robustness and versatility in relation extraction datasets from general and biomedical domains.

* Through ablation experiments, we illustrate the effectiveness of components of .

* We openly provide all code, experimental settings, and datasets used to substantiate the claims made in this paper.[<https://github.com/coming_soon>]

Data flow used for . The original RE input instance (A) is converted into a premise where surface forms are masked with corresponding entity types (B). Each relation class is verbalized into a hypothesis (C), and a feasible hypothesis filter (D) removes infeasible hypotheses based on the pair of entity types. NLI targets are generated via meta-class analysis (E), which are the labels used to fine-tune an LLM via cross-entropy (F). Finally, we use softmax probabilities as a proxy for the model's confidence and select the most confident “entail” prediction among the group of predictions (G). Note, in this example, the model makes three predictions—one for each feasible hypothesis. The second “entail” prediction is incorrect but the group-based prediction module selects the first and correct “entail” prediction by assessing the model's confidence.

§ RELATED WORK

Traditionally, RE has been approached as a classification task, where input instances are classified as belonging to a relational class <cit.>. These methods have several drawbacks: they tend to generalize poorly <cit.>, and they heavily rely on relatively small and disjoint  RE datasets. To account for these drawbacks, recent works have proposed clever adaptation methods to recast RE into adjacent NLP tasks, such as a question-answering <cit.> and natural language inference <cit.>. Task adaptation presents an opportunity to leverage the relatively large datasets available for other tasks (e.g., SQuAD <cit.>, MultiNLI <cit.>, SNLI <cit.>, etc.), which can be particularly advantageous in the context of biomedical RE where datasets are often limited.

<cit.> recast RE into a question-answering task by associating relation instances with one or more natural-language questions, resulting in predicted spans denoting class indicative text. <cit.> adapts general domain relation extraction into an NLI task by using relation instances as premises where each premise is paired with a hypothesis generated by verbalizing a relation class. In doing so, they formulate a binary entailment task where they predict whether or not a premise entails the corresponding hypothesis.

<cit.> expands on <cit.> by incorporating a three-label classification objective where a model can predict “entail,” “contradict,” and “neutral” depending on the premise-hypothesis pair, bringing the task in line with a standard NLI formulation <cit.>. They manually generate hypothesis templates corresponding to each relation class in a dataset, and NLI targets are assigned based on the alignment of the premise-hypothesis pair. If the corresponding hypothesis is the verbalized version of the ground truth relation label, then “entail” is assigned as the NLI target for the instance. The “neutral” label is applied to positive class hypotheses which do not align with a given premise. The “contradict” label is applied in two cases: (1) if the premise is a positive relation instance (e.g., any class other than “no relation”), the “no-relation” class-indicative hypothesis is labeled as “contradict,” and (2) if the premise is a negative instance (e.g., “no relation”), then all other class-indicative hypotheses are labeled as “contradict.” <cit.> fine-tune a language model pre-trained on the MultiNLI <cit.> dataset to predict generated NLI targets. They observe impressive results in zero- and few-shot scenarios on TACRED <cit.>, a general domain, sentence-level RE dataset.

<cit.> explores cross-domain transfer learning, leveraging indirect supervision from general domain NLI datasets to improve biomedical RE-to-NLI adapted methods. Our work can be considered an extension of their proposed NBR method. However, we introduce a few key improvements: meta-class analysis, a feasible hypothesis filter, and group-based prediction selection. We also expand evaluations beyond sentence-level biomedical RE to include more challenging document-level RE <cit.>.

§ PROBLEM STATEMENT

Our problem is a hybridization of relation extraction and natural language inference; as such, we describe both tasks, as well as the adapted RE-to-NLI task in the following sections.

§.§ Relation extraction (RE)

RE takes inputs $\{x_1, x_2,\ldots, x_n\} \in X_{\textsc{RE}}$ where $X_{\textsc{RE}}$ is a corpus of sentences, paragraphs, or documents of size $n$ and $x_i$ is a singular instance containing an entity pair $e_{i_{1}}$ and $e_{i_{2}}$. Each input $x_i$ has a corresponding label $y_i$. Labels $\{y_1, y_2,\ldots, y_n\} = Y_{\textsc{RE}}$ belong to a set of $m$ relation classes $R = \{r_1, r_2,\ldots, r_m\}$. Relation extraction seeks to identify which class links the co-mentioned entities to form a fact triplet $\langle e_{i_{1}}, y_i, e_{i_{2}} \rangle$, or, semantically, ⟨head, relation, tail⟩.

§.§ Natural language inference (NLI)

NLI takes a premise $p_i \in P$ and a hypothesis $h_i \in H$, where $P$ and $H$ are the set of premises and hypotheses in a corpus, respectively, and seeks to determine whether the premise entails, contradicts, or is neutral to the respective hypothesis <cit.>.
Using $\hat{y}_i$ to represent an NLI label applied to the $i_{th}$ instance, $\hat{y}_i \in \{entail, contradict, neutral\}$, and a single NLI example can be expressed as $\langle p_i, \hat{y}_i, h_i \rangle$.

§.§ RE-to-NLI adaptation

RE-to-NLI adaptation converts relation extraction inputs and labels into premise-hypothesis pairs such that each input instance maps to $m = |R|$ premise-hypotheses pairs: $(x_i,y_i)\rightarrow\{(p_i, \hat{y}_j, h_j)\}_{j=1}^{|R|}$. We decompose RE-to-NLI adaptation into the following sub-steps:

* Premise generation, $x_i\rightarrow p_i$: Input instances $x_i \in X_{\textsc{RE}}$ directly become premises $p_i \in P^{|X_{\textsc{RE}}|}$ where $P$ is the collection of all premises generated from $X_{\textsc{RE}}$.

* Hypothesis generation, $H_{i} = \{h_j\}_{j=1}^{|R|}$: In the hypothesis generation step, a set of hypotheses $H_{i}$ paired with each premise $p_i$. This is achieved by first verbalizing relation classes in $R$ into a set of $m$ hypothesis templates $T=\{t_1, t_2,\ldots, t_m\}$. Each hypothesis template contains head and tail entity placeholders, which are replaced by the head and tail entities found in the corresponding premise $p_i$. The verbalizer function $f_{verbalizer}(\cdot)$ takes each hypothesis template and entity pair in premise $p_i$ to produce the set of hypotheses $H_i = \{f_{verbalizer}(t_j, e_{i_{1}}, e_{i_{2}})\}_{j=1}^{|R|}$.

* NLI target generation, $\hat{Y} = \{\hat{y}_i\}_{i=1}^{|X_\textsc{RE}| \times |R|}$: The set of NLI targets $\hat{Y}$ is generated via a function which takes the original instance label $y_i$ and the premise-hypothesis pair $f_{target}(y_i, p_i, h_j) \rightarrow \hat{y}_j$ where $\hat{y}_j = entail$ if verbalized class-indicative hypothesis $h_j$ aligns with the ground truth label $y_i$, and, depending on the adaptation method, $\hat{y}_j$ is assigned $neutral$ or $contradict$ for non-aligned hypotheses.

The adapted RE-to-NLI task is to correctly predict entailed premise-hypothesis pairs where each entailed pair has a 1-to-1 mapping to the original relation extraction class label.

§ METHODS

This section discusses each module used in , following left to right in Figure <ref>.

§.§ Generating Premise/hypothesis Pairs

Premise construction: Following <cit.>, a relation instance $x_i$ is transformed into a premise by replacing surface forms of the subject and object entities, $e_1$ and $e_2$, respectively, with their corresponding entity types, $e_{1_{type}}$ and $e_{2_{type}}$. Abstracting entity surface forms into entity types helps alleviate the long-tail nature of biomedical entities and encourages language models to learn from context instead of shallow heuristics <cit.>. The start and end spans of entity types are denoted with “$@$” and “$\$$,” respectively.

Hypothesis verbalizer: Past works have manually generated hypothesis templates for each relation class in a dataset which are then used, in turn, to generate hypotheses to pair with a given premise. A secondary contribution of is that we reduce this human effort by leveraging LLMs to automatically the set of hypothesis templates $\{t_1, t_2,\ldots,t_m\}\in T$, where $m$ corresponds to the number of relation classes in a dataset.[We use ChatGPT (GPT 3.5) via OpenAI's web interface.] We prompt the LLM to verbalize each relation class using natural language and placeholders for subject and object entities (see Appendix <ref> for more details). The subject and object placeholders within the hypothesis templates are replaced by the corresponding entity types, $e_{1_{type}}$ and $e_{2_{type}}$, found in the premise.

Feasible hypothesis filter:
There is an implicit multiplicative effect of adapting RE into an NLI task where each relationship instance produces $m$ class-indicative hypotheses resulting in $|X_{\textsc{RE}}| \times m$ premise-hypothesis pairs. We develop a “feasible hypothesis filter” to reduce this effect, automatically filtering out improbable hypotheses. To develop this filter, we aggregate valid sets of entity-type pairs by relationship classes across all training data:

$E_{valid}=\left\{r_1 \mapsto S_1, r_2 \mapsto S_2, \ldots, r_m \mapsto S_m\right\}$

where $r_j \in R$ for $j=1,2, \ldots, m$ and each $S_j$ is the set of tuples of entity-type pairs associated with all instances of relationship class $r_j$ from the training data.

Using this filter, we assess the feasibility of hypotheses given a pair of entity types:

$\hat{H}_i = \{h_j | (e_{1_{type}}, e_{2_{type}}) \in E_{valid}(r_{j})  \}_{j=1}^{|R|}$.

where $\hat{H}_i$ is a set of feasible hypotheses given the entity-type pair found in instance $i$, and  $\hat{H}_i \subset H$ where $H$ is the set of all possible hypotheses.

Since sets of feasible hypotheses are developed using the training data's relationships and entity-type pairs, the filter may remove valid hypotheses based on an entity-type pair and corresponding relation that exists only in the test set. For these instances, the entailed premise-hypothesis pair will not be presented to the model, leading to false negatives. However, in practice, we observe that this does not occur on the biomedical datasets we use for evaluation and should not occur as long as training data is sufficiently representative of the test data (i.e., the training data contains at least one relationship with a specific entity-type pair for every relation and entity-type pair found in the test set).

Meta-class analysis:
After applying the aforementioned feasible hypothesis filter, we leverage meta-class analysis to assign NLI targets, namely “entail,” “neutral,” and “contradict,” to the resultant premise-hypothesis pairs. To do this, we first construct definition-based mutually exclusive meta-relationships between relation classes. For example, in the ChemProt dataset, the “up regulator” class is, by definition, mutually exclusive to the “down regulator” class. For datasets with a negative class (e.g., “no relation”), the negative class is mutually exclusive to all positive classes and vice-versa.

With this analysis, we construct NLI labels in the following way:

* Entail: We assign the “entail” label to premise-hypothesis pairs in which the hypothesis aligns with the verbalized ground truth label $y_i$.

* Neutral: If the original relation instance expresses a positive relation class (i.e., any class other than the “no relation” class), then, from the meta-class analysis, all non-mutually exclusive positive class hypotheses that do not align with the ground-truth label $y_i$ are labeled as “neutral.”

* Contradict: Finally, the “contradict” label is assigned to all hypotheses that verbalize definitionally mutual exclusive relation classes. The “no relation” class is mutually exclusive to all positive classes.

See Appendix <ref> for tables showing how original relation labels map to NLI targets using meta-class analysis for each dataset.

LLM Fine-tuning:  With generated premise-hypothesis pairs, we train a discriminate language model, namely BioLinkBERTlarge <cit.>, to predict NLI targets. We concatenate premise-hypothesis pairs as the input to the language model and send the resultant representation of the special [CLS] token through a fully connected linear layer, which is trained using cross-entropy loss:

\begin{equation}\label{eq:cross_entropy}
\begin{aligned}
\mathcal{L}_{\mathrm{CE}}=-\sum_{i=1}^{m} y_{o,i} \cdot \log \left(p\left(y_{o,i}\right)\right)
\end{aligned}
\end{equation}

where $y$ is a binary indicator that is $1$ if and only if $i$ is the correct classification for observation $o$, $p(y_{o,i})$ is the Softmax probability that observation $o$ is of class $i$, and $m$ is the number of classes.

Group-based prediction selection: Given the multiplicative effect of adapting RE-to-NLI where one relation instance results in a group of up to $m$ premise-hypothesis pairs, we can employ a group selection method to select the most confident “entail” prediction. If the model predicts two or more entailed instances within a group of premise-hypothesis pairs, we use the softmax probability from Equation <ref> as a proxy for model confidence <cit.> and select the prediction with the highest confidence. We allow the model to naturally abstain from making a prediction by predicting “neutral” for all premise-hypothesis pairs in a group.

§ EXPERIMENTS

§.§ Datasets

We include a spread of experiments on various biomedical relation extraction datasets. BioRED is a document-level RE dataset featuring eight relation classes and four entity types: disease, variant, gene, and chemical <cit.>. BioRED also provides an orthogonal and binary “Novel” class, which annotates whether an instance expresses a novel finding. BC5CDR is a document-level RE dataset featuring binary relations between chemical and disease entities <cit.>. DDI13 is a drug-drug interaction dataset with four relation classes <cit.>, and ChemProt is a chemical-protein dataset featuring five relation classes <cit.>. GAD is a gene-disease dataset with binary relations <cit.>. We only include GAD in our main experiment for comparative purposes to past works. We believe that the GAD dataset should be retired from future works due to significant label accuracy issues, which the authors acknowledge.[<https://github.com/dmis-lab/biobert/issues/162>]

As mentioned in Section <ref>, our method is designed to leverage features of biomedical domain RE, namely the prevalence of definitional mutually exclusive classes and the importance of entity types vis-à-vis feasible relationships. However, we also seek to assess our method beyond the biomedical domain and extend our experiments to general domain datasets ReTACRED <cit.> and SemEval-2010 Task 8 <cit.>. ReTACRED is a re-annotated and improved version of TACRED <cit.> and features sentence-level instances and 40 relation classes—significantly more classes than any of the biomedical datasets we tested. SemEval-2010 is a sentence-level relation extraction dataset with ten relation classes.

§.§ Baselines

§.§.§ Traditional Multi-Class Classification

We select leading biomedical language models and train them to directly predict relation classes using cross-entropy. BioM-ALBERTxxlarge, BioM-BERTlarge, and BioM-ELECTRAlarge <cit.> are transformer architectures ALBERT <cit.>, BERT <cit.>, and ELECTRA <cit.>, respectively, adapted into the biomedical domain using a custom biomedical vocabulary and pre-training on PubMed abstracts <cit.> and PubMed Central articles <cit.>. BioMed RoBERTabase <cit.> features the RoBERTa architecture <cit.> adapted to the biomedical domain via continued pre-training on papers from the  Semantic Scholar Open Research Corpus <cit.>. PubMedBERTbase <cit.> is a BERT variant trained from scratch on PubMed abstracts with a custom biomedical vocabulary. BioLinkBERTlarge <cit.> is another BERT variant trained with two self-supervised objectives: masked language modeling and document relation prediction. BioLinkBERT and its general domain variety, LinkBERT, achieve impressive results on discriminative tasks in their respective domains.

§.§.§ NLI Adapted Models

NBR is a biomedical domain RE-to-NLI method that leverages BioLinkBERTlarge as a backbone language model. Like our method, NBR converts relation instances and labels into premise-hypothesis pairs. Key differences between NBR and our method are that NBR does not use meta-class analysis or feasible hypothesis filtering, and they leverage a ranking loss training objective to rank entailed premise-hypothesis pairs over non-entailed pairs.

The RE-to-NLI adaptation method used in is architecture-agnostic, so we also experiment with auto-regressive language models. We conduct the following experiment using identical data and methods to those discussed in Section <ref>; the only differences are the architecture and final training step.

We fine-tune Phi-2 and Phi-3 (3.8B).[We use the microsoft/phi-2 and microsoft/Phi-3-mini-4k-instruct checkpoints from Hugging Face.] For Phi-2 and Phi-3, we construct a sequence-to-sequence task and fine-tune the models to generate an NLI label “entail,” “contradict,” or “neutral” for each premise-hypothesis pair. Responses are aligned to ground truth labels using partial string matching by searching for the matches of the first three letters in each NLI label (e.g., “ent” $\rightarrow$ “entail,” “con” $\rightarrow$ “contradict,” “neu” $\rightarrow$ “neutral”). When a class cannot be matched, we assign “none,” which, during evaluation, is equivalent to “neutral.” For more information about training Phi-2 and Phi-3, see Appendix <ref>.

We also seek to assess the performance of large, frontier auto-regressive models, GPT 3.5 <cit.> and GPT 4 <cit.>,[Specifically, we use gpt-3.5-turbo-0125 and gpt-4-turbo-2024-04-09 via OpenAI's API.] leveraging few-shot, in-context learning. Responses from GPT 3.5 and GPT 4 sometimes include additional text. We use the partial string matching used for Phi-2 and Phi-3 to evaluate these responses. For more on the prompts we use to solicit predictions from GPT 3.5 and GPT 4, see Appendix <ref>.

For all NLI-adapted models, only entailed premise-hypothesis pairs map directly to the original RE training instance. Thus, we only keep NLI instances labeled or predicted as entailed when mapping instances back into the original RE labels for evaluation. This ensures a fair comparison across adapted and non-adapted methods.

§.§ General Domain Experiments

For our general domain experiments, we use DeBERTaV3large <cit.> and RoBERTa-MNLIlarge <cit.>. DeBERTaV3 is an improved version of BERT that replaces the mask language modeling pre-training objective with replaced token detection, a more sample-efficient pre-training task. RoBERTa-MNLIlarge is the RoBERTa architecture fine-tuned on the Multi-Genre Natural Language Inference (MNLI) corpus.[We use the FacebookAI/roberta-large-mnli checkpoint from Hugging Face.]

We also make slight modifications to the general domain version of . We use RoBERTa-MNLIlarge as the backbone language model, and we do not leverage surface-form abstraction for entity types (i.e., we leave the original entities as they appear in the text and do not replace them with their corresponding types). Entity surface form abstraction is a method developed for the long-tail nature of biomedical entities, and, in the general domain, entities have a less pronounced long-tail <cit.>. Also, some general domain RE datasets, such as SemEval-2010 Task 8, do not provide annotated entity type information.

§.§ Unified Task Experiments

Given that we adapt the task of RE into one of entailment, which leads to models predicting the same entail, neutral, contradict labels across disparate datasets, we naturally sought to investigate the potential of combining the relatively small and disjoint biomedical RE datasets into a single, unified task. We investigate two task-unification training methodologies: single-stage training and double-stage training. Single-stage training can be viewed as multi-task learning, where the model is trained simultaneously on multiple datasets and tested on a target dataset. Double-stage training can be viewed as an initial pre-training stage on all data except the target dataset, followed by fine-tuning and evaluating on the target dataset.

§ RESULTS

max width=1.0

Model	   	BC5CDR	   	BioRED	   	BioRED (novel)	   	ChemProt	    DDI13	   	GAD
7cTraditional Multi-class Classification
BioM-ALBERTxxlarge <cit.>     	0.679	   	0.668	   	0.863	   0.940	   	0.911	   	0.815

BioM-BERTlarge <cit.>     	0.681	      0.709	    0.904	   	0.934	   	0.917	   	0.795

BioM-ELECTRAlarge <cit.>	    0.687    	0.657	   	0.903	   	0.925	   	0.885	    0.830

BioMed RoBERTabase <cit.>	   	0.664	   	0.714	   	0.897	   	0.919	   	0.911	   	0.803

PubMedBERTbase <cit.>	       	0.651	   0.715    	0.891	   	0.923	   	0.916	   	0.803

BioLinkBERTlarge <cit.>	   	0.682	   	0.699	   	0.899	   	0.931	   0.917	   	0.806
7cNLI Adapted Models
NBR <cit.>   	0.679	   	0.543	   	0.664	   	0.883	   	0.846	   	0.831

Phi-2 <cit.>    	0.653	   0.715    	0.824	   	0.852	   	0.873	   	0.729

Phi-3 <cit.>           0.749            0.688           0.840           0.930              0.915       0.721

GPT 3.5$\dagger$	<cit.>           	0.282	   	0.470	   	0.594	   	0.494	   	0.386	   	0.548

GPT 4$\dagger$ <cit.>             	0.418	   	0.532	   	0.680	   	0.626	   	0.492	   	0.660
0.757	   	0.891	   	0.917	   	0.968	   	0.957	   	0.878

Micro F1 scores for traditional RE and NLI adapted methods. $\dagger$ Results from GPT 3.5 and GPT 4 are via in-context learning (see Appendix <ref> for details), whereas other models in the table were fine-tuned directly on the task.

We observe an interesting comparison between the BioLinkBERTlarge model and . Both experiments share the same backbone language model, yet the performance of our method is significantly higher, providing evidence of the effectiveness of adapting the RE task into one of textual entailment. We hypothesize that the boost in performance primarily comes from the additional data abstraction RE-to-NLI introduces by training the model to recognize entailed premise-hypothesis pairs instead of directly predicting suppositional classes. By combining RE-to-NLI adaptation with surface-form entity abstraction, the model is less prone to memorizing entities and shallow heuristics of relation classes; instead, it must understand the context and the natural language interplay between premise-hypothesis pairs that verbalize relation classes. Furthermore, the boost in performance between the NBR model and highlights the effectiveness of leveraging meta-class analysis, feasible hypothesis filtering, and group prediction selection.

Within the biomedical domain experiments, the NLI-adapted auto-regressive models underperform in general compared to the discriminative models. Predictably, the larger Phi-3 outperforms Phi-2 and fine-tuning smaller auto-regressive models outperforms larger models, GPT 3.5 and GPT 4, leveraging few-shot in-context learning. This aligns with findings from <cit.> that LLMs using in-context learning underperform relative to smaller, fine-tuned language models on information extraction tasks.

max width=.48

Model	   	ReTACRED	   	SemEval
3cTraditional Multi-class Classification
DeBERTaV3large <cit.>     0.809     0.807

RoBERTa-MNLIlarge <cit.>    	0.800	   	0.828
3cNLI Adapted Models
NBR <cit.>	   	0.875	   	0.826

Phi-2 <cit.>	   	0.862	   	0.855

Phi-3 <cit.>	   	0.880	   	0.871

GPT 3.5$\dagger$ <cit.>	   	0.306	   	0.340

GPT 4$\dagger$ <cit.>	   	0.565	   	0.616
0.943	   	0.902

Micro F1 scores from general domain RE experiments.

In the general domain, we observe better overall performance from auto-regressive architectures. The performance from Phi-3 approaches that of on both ReTACRED and SemEval-2010 Task 8 datasets. These results are promising for auto-regressive models, in general. We leave fine-tuning larger auto-regressive models to future work but expect additional gains to be made, potentially overtaking the smaller discriminative models.

§.§ Ablation Experiments

max width=.48

Model	   	BioRED	   	ChemProt    	ReTACRED
0.891	   	0.968    	0.943
(w/o Feasible Hypothesis Filter)	   	0.876	   	NA    	DNC

(w/o Meta-class Analysis) 	   	0.853	   	0.911    	0.916

(w/o Grouped Selection)	   	0.805	    0.950    	0.875

Micro F1 scores from ablation experiments which remove each proposed module within . Each module has a significant impact on performance. ChemProt is monolithic in its entity types found in each relation (chemicals and diseases), which prevents the use of the feasible hypothesis filter. On ReTACRED, we observe that without applying the feasible hypothesis filter, the model does not converge (DNC).

We conduct ablation experiments to better understand 's performance gains by removing modules and reporting the performance. Note that the performance of BioLinkBERTlarge in Table <ref> can be considered a type of ablation of that does not leverage NLI adaptation or any additional modules since uses BioLinkBERTlarge as its backbone language model. Note that for our ablations, we choose to examine the BioRED, ChemProt, and ReTACRED datasets due to the following: they feature more than two relation classes, and they contain one or more definition-based mutually exclusive relations as determined by our meta-class analysis.

* w/o Feasible Hypothesis Filter: We remove the feasible hypothesis filter, and, in doing so, each original relation instance is converted into $m$ premise-hypothesis pairs, with $m$ being the number of classes in a dataset. Removing the feasible hypothesis filter produced a moderate drop in performance on BioRED ($m=8$). Since the feasible hypothesis filter is based on entity type pairs, it is not available (NA) for datasets such as ChemProt, which only feature a single entity type pair (namely, chemical and gene) associated with every relation class. However, the feasible hypothesis filter is essential in model convergence when a dataset consists of many relation classes, such as ReTACRED ($m=40$). In our experiments on ReTACRED without the feasible hypothesis filter, the model did not converge (DNC), likely due to the overwhelming number of non-informative “neutral” premise-hypothesis pairs used in training.

* w/o Meta-class Analysis: Removing meta-class analysis and using “neutral” as the NLI target for all non-entailed premise-hypothesis pairs led to a considerable drop in performance, indicating the benefit of training the model with the additional training signal obtained via meta-class analysis. Note that in this ablation experiment, we maintain mutual exclusive NLI targets between positive and negative (“no relation”) classes.

max width=1.0

4cMulti-task Learning (single-stage)
2lEnsemble training data  $\rightarrow$ 	    	Test set	   	$\Delta$ F1
2lBC5CDR + BioRED + ChemProt + DDI13	   	BC5CDR	   	-0.049

2lBioRED + ChemProt + DDI13 + BC5CDR	   	BioRED	   	-0.031

2lChemProt + BioRED + DDI13 + BC5CDR	   	ChemProt	   	+0.012

2lDDI13 + BioRED + ChemProt + BC5CDR	   	DDI13	   	-0.007

4cContinued pre-training with supervised fine-tuning (double-stage)
1lPre-training corpus	$\rightarrow$    	1lFine-tuning	$\rightarrow$    	Test set	   	$\Delta$ F1
BioRED + ChemProt + DDI13	   	BC5CDR	   	BC5CDR	   	-0.005

ChemProt + DDI13 + BC5CDR	   	BioRED	   	BioRED	   	-0.014

BioRED + DDI13 + BC5CDR	   	ChemProt	   	ChemProt	   	-0.008

BioRED + ChemProt + BC5CDR	   	DDI13	   	DDI13	   	-0.011

Results from single-stage and double-stage task unification experiments. Delta micro F1 scores are relative to scores from Table <ref>. We do not observe signification performance improvements from our task unification experiments and leave further experimentation to future work.

* w/o Group prediction selection: To remove this module, we select all entailed predictions regardless of how many entail predictions are made within a group of premise-hypothesis pairs. Doing this removes the constraint imposed by the single-class classification task and allows the model to freely predict multiple classes for a single relation instance. This ablation experiment led to a drop in performance across ChemProt and ReTACRED but most significantly in BioRED, which we suspect results from the closeness in BioRED's “positively correlated” and “associated” relation classes. These classes were typically confused in an informal error analysis since “associated” can sometimes be considered a hypernym of “positively correlated,” leading the model to predict “entail” for both of the corresponding hypotheses.

§.§ Task Unification Results

Unfortunately, we did not observe a significant performance boost across our task-unification experiments (Table <ref>), potentially indicating that these biomedical datasets do not provide complementary information. Generally, the two-stage training is more effective than the single-stage training, but both fail to realize significant performance gains on the target datasets. We leave investigating other task-unification methods for future works.

§ CONCLUSION

The exploration of NLI techniques to enhance relation extraction has opened new avenues in natural language processing, and our study introduces as an advancement in this area. By adapting the RE task into an NLI framework and incorporating innovative strategies such as meta-class analysis, feasible hypothesis filtering, and group-based prediction selection, demonstrates remarkable improvements in RE performance. Our experiments, conducted across biomedical and general domain datasets, highlight the robustness and versatility of . By openly sharing our code, experimental settings, and datasets, we aim to facilitate further research and development in this promising intersection of NLI and RE, paving the way for more sophisticated and accurate information extraction systems in diverse domains.

§ LIMITATIONS

is not without its limitations. By verbalizing a hypothesis for each relation class, the training data is multiplied by the number of relation classes in the dataset, necessitating additional training resources. Our introduced module, the feasible hypothesis filter, relies heavily on accurate entity-type information. This information is crucial for the success of the adaptation process. However, the filtering process becomes ineffective if this information is unavailable or if numerous feasible hypotheses (e.g., 40+) exist for a given relation class and entity type pair. In these scenarios, the “entail” class becomes a minority class in a sea of “neutral” NLI instances, potentially causing the model to collapse to a trivial state of simply predicting “neutral” for every premise-hypothesis pair. Such a scenario would require the design of manually tuned sampling strategies or bespoke learning objectives to handle the overwhelming number of “neutral” premise-hypothesis pairs. We defer the exploration of such challenging settings to future research.

§ ETHICS STATEMENT

We do not anticipate any major ethical concerns; relation extraction is a fundamental problem in natural language processing. A minor consideration is the potential for introducing certain hidden biases into our results (i.e., performance regressions for some subset of the data despite overall performance gains). However, we did not observe any such issues in our experiments, and indeed these considerations seem low-risk for the specific datasets studied here because they are all published.

§ APPENDIX


# On Classifying the Effects of Policy Announcements on Volatility

Giampiero M. Gallo  Demetrio Lacava  Edoardo Otranto111Corresponding author
CRENoS and New York University in Florence, email<EMAIL_ADDRESS>University of Messina, email<EMAIL_ADDRESS>CRENoS and University of
Messina, email<EMAIL_ADDRESS>
###### Abstract

The financial turmoil surrounding the Great Recession called for unprecedented
intervention by Central Banks: unconventional policies affected various areas
in the economy, including stock market volatility. In order to evaluate such
effects, by including Markov Switching dynamics within a recent Multiplicative
Error Model, we propose a model–based classification of the dates of a Central
Bank’s announcements to distinguish the cases where the announcement implies
an increase or a decrease in volatility, or no effect. In detail, we propose
two smoothed probability–based classification methods, obtained as a
by–product of the model estimation, which provide very similar results to
those coming from a classical k–means clustering procedure. The application on
four Eurozone market volatility series shows a successful classification of
144 European Central Bank announcements.

###### keywords:

Markov switching model , Unconventional monetary policies , Stock market
volatility , Multiplicative Error Model , Smoothed Probabilities , Model–based
clustering JEL Codes: C32, C38, C58, E44, E52, E58

††journal: International Journal of Approximate Reasoning

##  1 Introduction

Since the onset of the Great Recession, many Central Banks resorted to
unconventional monetary policy in order to mitigate the consequences that the
crisis had on the real economy and financial markets as well. All the related
measures were introduced by means of monetary policy announcements; recent
literature focused on the consequences for the real economy (e.g. [1, 2]) and
for the financial markets, in particular their volatility [3, 4, 5], all
viewing the effect of the announcements as a constant factor in the models.
The actual strength of an announcement, however, turns out to be a consequence
of the conditions in which the measure is adopted, its wording, how it
constitutes a surprise relative to the consensus, how divergent expectations
are, and so on. The impact on financial markets, in particular on their
volatility, is a consequence of the immediate adjustment of the asset prices
to the new information, and the formation of new equilibria after the
announcement.

Within the class of Multiplicative Error Models (MEMs) by [6], the recent work
by [7] is a first attempt at measuring the unconventional policy effects as an
unobservable component of volatility, distinguishing the effects due to policy
implementation (as represented by a balance sheet–based continuous proxy
variable) from those on the announcement day (related to a dummy variable). In
what follows, we chose to modify their univariate Asymmetric Composite Model
(ACM – cf. also the previous contribution by [8]): here, rather than resorting
to a dummy variable, we take the view that effects of an announcement could be
measured in terms of changes in volatility level across two alternative
nonobservable regimes; to this end, we specify volatility dynamics as Markov
Switching (MS), without revealing the model when the announcement took place.
As a by–product, we suggest a classification rule for the underlying
announcements, according to whether they have a notable impact on volatility
via a change in regime or they induced a permanence in the same regime. Such a
classification is possible only thanks to the MS approach and would not be
reproduceable for other volatility models (notably, the plain Asymmetric
(A)MEM by [9], or the HAR by [10]).

In spite of it not being a customary classification method based on distance
measures, our approach is still within the large literature concerning
model–based clustering (for a recent update on the state of the art, cf. [11]
and convenient reviews of these methods are in [12]). Interestingly, next to
the whole and subsequence areas of analysis, [13] mention also the time-point
analysis strain of research, where we place our approach, aimed at detecting
both expected and unusual patterns through the identification of dynamic
changes in time series features.

By the same token, our proposal is clearly different from the model–based
techniques aimed at developing clustering in volatility of financial markets,
such as [14], [15], [16], [17], where the classification involves the full
time series and not individual observations. The same considerations can be
made by comparing our approach with [18], where an ACM model similar to the
one proposed in this paper is adopted, but, again, with the aim to classify
the similarities between entire financial time series.

Based on the estimation of our model on time series of realized volatility
(for four Eurozone stock indices, i.e., CAC40, DAX30, FTSEMIB and IBEX35), we
consider 144 announcements and we classify how such announcements had an
effect on the level of each market volatility: the groups we get are named
Plank (a neutral effect), Squat (a decrease in volatility) and Jump (an
increase in volatility). Built as a simple processing of the smoothed MS
probabilities of being in either a low or a high volatility regime, our
classification techniques are simple to implement and deliver results very
similar to a benchmark k–means clustering approach.

The paper is structured as follows. We detail our MS time-point analysis
approach to classification in Section 2, within which the model is presented
in Subsection 2.1 and the proposed classification procedures in Subsection
2.2. The empirical application is contained in Section 3, where we discuss
data features and the framework of events occurring in our sample period;
estimation results are discussed in Subsection 3.1 and the corresponding
classifications in Subsection 3.2. Finally, Section 4 contains some concluding
remarks.

##  2 A Markov Switching Approach to Classification

###  2.1 A Policy Analysis–oriented Modelling Approach

Volatility modelling exploits the availability of high frequency data,
favouring a decoupling of measurement and modelling with respect to the more
traditional approach based on GARCH models [19, 20]. The so–called Realized
Volatility (RV) is recognized to have better properties in measurement than
the outcome of the estimated GARCH–based conditional variance of returns [21].
As per forecasting, several conditional models are available for RV; one of
them is the MEM, proposed by [6], which takes volatility dynamics in terms of
the product of two positive time-varying factors, one representing its
conditional mean and the other a positive–valued disturbance. Several
improvements allow to capture stylized facts and to accommodate specific
cases; in particular the [9] specification introduces the asymmetric and
predetermined variable effects. In what follows, with an eye to capturing
policy effects, we extend the MS–AMEM by [22] to additively accomodate
policy–induced effects (component $\xi_{t,s_{t}}$) next to the volatility
dynamics (the base volatility component $\varsigma_{t}$):

$\begin{array}[]{l}RV_{t}=\mu_{t,s_{t}}\epsilon_{t},\,\epsilon_{t}|\mathbf{I}_{t-1}\sim
Gamma(\vartheta_{s_{t}},\frac{1}{\vartheta_{s_{t}}})\\\
\mu_{t,s_{t}}=\varsigma_{t}+\xi_{t,s_{t}}\\\ \varsigma_{t}=\omega+\alpha
RV_{t-1}+\beta\varsigma_{t-1}+\gamma D_{t-1}RV_{t-1}\\\
\xi_{t,s_{t}}=\varphi_{0}+\varphi_{1}s_{t}+\delta(E\left(x_{t}|\mathbf{I}_{t-1}\right)-\bar{x})+\psi\xi_{t-1,s_{t-1}}\end{array}$
(1)

As with any other MEM, the realized volatility at time $t$, $RV_{t}$, is seen
as the product of a conditional (on the past information set
$\mathbf{I}_{t-1}$) expectation term $\mu_{t}$ times a unit mean error term
$\epsilon_{t}$ following a Gamma distribution.111In suggesting the Asymmetric
MEM, [9] justify the adoption of a Gamma distribution in view of its
flexibility and the fact that it nests other notable distributions (such as
the exponential and the Chi-square). In a more general framework for MEMs,
[23] show that the first order conditions for a Gamma–based likelihood
function coincide with the objective function of a semiparametric GMM based
approach.

In our approach, in line with [7], the expected conditional volatility is
decomposed as the sum of $\varsigma_{t}$, evolving as a GARCH–like process
(with asymmetric effects tied to the negative sign of past returns captured by
a dummy variable, $D_{t-1}$), and a policy–specific and regime–dependent term
$\xi_{t,s_{t}}$, which follows an AR(1) model. The driving variable for this
dynamics is $x_{t}$ – a proxy for unconventional policy measures, entering the
model as the deviation of its conditional expectation from a long term mean
$\bar{x}$ – which accounts for the Central Banks’ balance sheet composition.

In the original ACM model by [7]222Actually, this model is derived from the
general framework in [8], where the mean of the conditional volatility is the
sum of two unobservable components: the particular ACM specification proposed
by [24] to model spillover effects in financial markets is adapted to
representing the base volatility and the unconventional policy effect
respectively. the last equation is:

$\begin{array}[]{c}\xi_{t}=\delta(E\left(x_{t}|\mathbf{I}_{t-1}\right)-\bar{x})+\varphi(\Lambda_{t}-\bar{\Lambda})+\psi\xi_{t-1},\end{array}$
(2)

where, the $\Lambda_{t}$ term is a dummy variable representing the effect of
the announcements and is taken as deviation from its long–term mean, also
here; the days of the announcements are not random variables, as they are put
in the calendar in advance by the ECB.

Contrary to the approach by [7], we are not considering the announcements
explicitly in the model via dummy variables; what constitutes the novelty in
the present econometric context is that we detect their presence when a change
in regime is attributable to a market volatility reaction to the announcement.
More in detail, we extend the ACM model by [7] in making the policy–specific
component $\xi_{t}$ become $\xi_{t,s_{t}}$, i.e., we consider a dichotomic
discrete latent variable $s_{t}=0,1$, representing the regime at time $t$.
When $s_{t}=0$, the time series is in a low volatility regime with intercept
$\varphi_{0}$, and increases by the term $\varphi_{1}\geq 0$ in the high
volatility regime ($s_{t}=1$). In practice, though, as the regime is not
observable, such an intercept turns out to be time–varying, as discussed
below. The dynamics of the state variable $s_{t}$ is driven by a first–order
Markov chain, that is:

$Pr(s_{t}=j|s_{t-1}=i,s_{t-2}\ldots)=Pr(s_{t}=j|s_{t-1}=i)=p_{ij}.$

Building also on the evidence and the discussion presented in [22], we obtain
the important result of expanding the flexibility of the model, since the
distribution of the error term now follows a mixture of two Gamma densities.

Positiveness ($\omega>0$, $\alpha,\beta,\gamma\geq 0$) and stationarity
($\alpha+\beta+\frac{\gamma}{2}<1$ and $\left|\psi\right|<1$) conditions
established in the case of the ACM are regime–independent and hence are valid
for the MS–ACM as well.

The Markov process assumption implies the dependence only on the current state
and implicitly assumes that the sojourn distribution is geometric. In
principle, we can assume a more general dependence on the duration of the
state or on $p$ lagged states, adopting semi–Markov processes and transition
distribution models. In the present framework, however, we rely on customary
assumption of market efficiency, based on the tenet that information is
instantly and completely incorporated into the current price. This theory is
supported by the empirical evidence with daily data and is consistent with the
econometric literature of MS models in the GARCH framework (see, for example,
[25, 26, 27]).

The likelihood function of the MS–ACM is obtained by means of the so called
Hamilton filter and smoother, as described in [28] (Ch. 22), adopting the
approximated solution proposed by [29] to solve the path dependence issue, a
computational problem due to the dependence of $\mu_{t}$ on all past values of
$s_{t}$. In fact, at the end of the recursive Hamilton filter, we would have
to keep track of all possible paths obtained by all the combinations of
regimes from the first time until the last one, with $2^{T}$ possible
different scenarios. To by–pass this problem, [29] proposes to collapse the 4
possible values of $\mu_{t}$ at time $t$, obtained at the end of each step of
the Hamilton filter, into $2$ values, averaging and weighing them with the
corresponding conditional probabilities $Pr[s_{t-1}=i,s_{t}=j|\mathbf{I}_{t}]$
(obtained in the same Hamilton filter step). This approximation is the most
popular solution, albeit not exact, causing a bias in the likelihood function;
in practice, though, the collapsing procedure does not involve substantial
errors in the final estimation: as shown by [22] through simulation
experiments, the approximated likelihood is satisfactory when the number of
observations is above 2000, an attainable goal with daily financial time
series.

As a recent contribution, [30] have generalized and improved the collapsing
procedure for MS–GARCH models, implementing optimal particle filters to
explicit the likelihood; the particle selection step is performed
deterministically and not stochastically, favoring the implementation and
reducing the computational effort. The authors compare their approximate
simple solution with the simulation–based approach of [31], showing a small
bias in practical situations. Moreover, this approach avoids the typical
problem of the particle filters, which provide likelihood estimates that are
discontinuous functions of the parameters. A recent simulated particle filter
approach was proposed by [32], as a modification to the classical setup,
obtaining estimated likelihood functions amenable to numerical optimization
and able to work also in presence of missing data. The comparison of these
different estimation procedures in the MS–AMEM framework is interesting, but
is beyond the scope of the present work.

The so–called smoothed probabilities $P[s_{t}|\mathbf{I}_{T}]$ provided by the
Hamilton smoother are used to make inference on the regime conditional on the
full information available, $\mathbf{I}_{T}$; a rule of thumb consists of
assigning the observation at time $t$ to regime 1 if $\hat{p}_{t}\equiv
P[s_{t}=1|\mathbf{I}_{T}]>0.5$, otherwise to regime 0. In our context, the
smoothed probabilities are further used to estimate the intercept in (1), as a
weighted average of $\hat{\varphi}_{0}$ and $\hat{\varphi}_{1}$:

$\hat{\varphi}_{t}=\hat{\varphi}_{0}(1-\hat{p}_{t})+(\hat{\varphi}_{0}+\hat{\varphi}_{1})\hat{p}_{t}$
(3)

where a hat indicates the QML estimate of the parameter. The interesting
feature of our model is that the intercept of $\xi_{t,s_{t}}$, the coefficient
$\hat{\varphi}_{t}$, is time–varying, providing the result that, differently
from Eq. (2), a change in regime will induce a change in the level of the
series with varying intensity, without feeding the model the dates of the
announcements. This is in contrast with a model relying on a dummy variable
for announcements which would force the effect on volatility dynamics to be
constant.

When we overlap the dates of the announcements to the estimated results, we
can thus monitor a possible change in the volatility level and its amplitude
at each date. As a consequence, we propose two different methods to classify
the policy announcements based on the changes of $\hat{\varphi}_{t}$ when $t$
is an announcement day, relative to the day before.

###  2.2 Classification of announcements

In formal terms, the $N$ dates of announcements (i.e., when $\Lambda_{t}=1$)
are selected within the overall time series and, for those, the values
$\hat{\varphi}_{t}-\hat{\varphi}_{t-1}$ are calculated. Notice that, from (3):

$\Delta\hat{\varphi}_{\tau}\equiv\hat{\varphi}_{\tau}-\hat{\varphi}_{\tau-1}=\hat{\varphi}_{1}(\hat{p}_{\tau}-\hat{p}_{\tau-1})\equiv\hat{\varphi}_{1}\Delta\hat{p}_{\tau}\quad\forall~{}\tau=t:\Lambda_{t}=1,$
(4)

that is, such announcement effects on the volatility level can be evaluated
through the variations in the smoothed probabilities directly. Thus, a first
form of classification of the $N$ announcement is to apply a clustering
algorithm to obtain groups with similar $\Delta\hat{p}_{t}$.

However, by the approach detailed in Eq.(4), one can notice that the
announcement effect (a large movement in the intercept) will be estimated to
be substantial, the larger the movement between the probability of being in
regime 1 at time $t-1$ and the corresponding one at time $t$. An alternative
to a clustering–based method can therefore be suggested, exploiting the
customary mapping of smoothed probabilities into a regime classification based
on the threshold $\hat{p}_{t}=0.5$.

A first smoothed probability–based classification (dubbed SP-level) can be
obtained directly from the position of the probabilities $\hat{p}_{t}$,
respectively, $\hat{p}_{t-1}$, relative to the threshold value $0.5$. In this
case we suggest 4 groups, with an immediate interpretation, in the following
way:

  1. 1.

No effect and low volatility – (low) Plank, if $\hat{p}_{t}\leq 0.5$ and
$\hat{p}_{t-1}\leq 0.5$;

  2. 2.

No effect and high volatility – (high) Plank, if $\hat{p}_{t}\geq 0.5$ and
$\hat{p}_{t-1}\geq 0.5$;

  3. 3.

Decrease in volatility – Squat, if $\hat{p}_{t}\leq 0.5$ and
$\hat{p}_{t-1}\geq 0.5$;

  4. 4.

Increase in volatility – Jump, if $\hat{p}_{t}\geq 0.5$ and $\hat{p}_{t-1}\leq
0.5$.

A second smoothed probability–based classification (called SP–diff, since it
is based on $\Delta\hat{p}_{t}$) gives three groups as follows:

  1. 1.

No effect – Plank, if $-0.5\leq\Delta\hat{p}_{t}\leq 0.5$; this group will
contain cases with a moderate effect, with or without regime change (i.e.
irrespective of whether the threshold is crossed), since subsequent
$\hat{p}_{t}$’s are close to one another;

  2. 2.

Decrease in volatility – Squat, if $\Delta\hat{p}_{t}<-0.5$; in this case the
volatility at time $t$ is attributed to regime 0 whereas the volatility at
time $t-1$ to regime 1, with a sharper change in the value of the probability
by more than 50%;

  3. 3.

Increase in volatility – Jump, if $\Delta\hat{p}_{t}>0.5$; in this case the
volatility at time $t$ is attributed to regime 1, whereas the volatility at
time $t-1$ to regime 0, and the change in the value of the probability is more
than 50%.

The diff–groups here are different from the level–groups, even if they have
the same label, since in this second approach we would classify as a Plank a
change in regime between $t-1$ and $t$, not accompanied by a relevant change
in $\hat{p}_{t}$’s. By both classifications, however, Squats and Jumps come
when the MS model produces a sharp mapping into regimes, and therefore should
give similar results.

If well designed, the three classifications should provide similar results,
and evidence will be provided below; both smoothed probability–based
classifications have the advantage of being immediately applicable each time
an announcement is provided.333Notice that our decoding is local, given that
the final clustering depends on the smoothed probabilities derived from the
Hamilton filter and Kim smoother.

To evaluate the degree of uncertainty attributable to these classification
methods based on the smoothed probabilities, we propose a new index. We note
that the sharpest assignment of the volatility to state 0, respectively, 1, is
obtained when the smoothed probability is exactly equal to 0 or 1. In
reference to our distinction between Plank (P), Squat (S) and Jump (J), such a
case implies that the difference of the contiguous probabilities would then be
equal to 0 in the case P, (-1) in the case S and (+1) in the case J.444Note
that the SP–level and SP–diff coincide, in such a case. As a consequence, we
suggest the following index:

$U=\frac{2}{N}\sum_{\tau=1}^{N}\left[\left|\Delta^{(P)}\hat{p}_{t}\right|+\left|\Delta^{(S)}\hat{p}_{t}+1\right|+\left|\Delta^{(J)}\hat{p}_{t}-1\right|\right],$
(5)

which varies between 0 (sharpest classification) and 1 (case of maximum
uncertainty).555Since the differences in the group P assume values between
-0.5 and 0.5, in group S between -1 and -0.5, in group J between 0.5 and 1, we
multiply the average in Eq. (5) by 2 to map the result into $[0,1]$.

##  3 An Empirical Application

We consider the annualized realized kernel volatility of four Eurozone stock
indices (CAC40, DAX30, FTSEMIB and IBEX35) as provided by the Oxford Man
Institute666Data are available at https://realized.oxford-
man.ox.ac.uk/data/download for the period from June 1, 2009 to December 31,
2019 (daily data, 2685 observations). Their profiles, shown in Figure 1,
behave similarly between one another, exhibiting bursts of market activity
occurring in correspondence with some events of relevance: two remarkable
peaks are recognizable in the first part of the sample, coinciding with the
flash crash on May 6, 2010 and with the Black Monday on August 8, 2011
(depicted as blue–dashed vertical lines in Figure 1). Similarly, some
volatility spikes correspond to monetary policy announcements (red–dashed
line); this is the case, for example, of the so–called conventional monetary
policy decisions concerning the interest rates (on August 2, 2012; November 7,
2013; December 4, 2014; December 3, 2015) and the unconventional policy
decision on March 10, 2016, when the Corporate and Public Sector Purchases
Programmes (CSPP and PSPP, respectively) were included in the Expanded Asset
Purchases Programme (EAPP). In addition, volatility clustering emerges quite
clearly throughout, with a long period of low volatility starting in July
2012, whereas short–lived periods of high volatility are observed at the
beginning of the sample, corresponding to the Greek sovereign debt crisis in
May 2010 and in the mid of 2011 when the sovereign debt crisis exploded
spilling over beyond the Eurozone.

We proxy for the implementation effects, the term
$E\left(x_{t}|\mathbf{I}_{t-1}\right)$ in Eq.(1), via the ratio of the amount
of securities held for unconventional policy purposes to that employed for
conventional policy measures777Data are obtained from the ECB website and
Datastream.. The conditional expectation of $x_{t}$ is estimated through the
ARIMA(4,1,1) model, according to a preliminary order identification procedure.
Finally, the list of monetary policy announcements consists of $N=144$ events,
constructed starting from the ECB press releases888Available at
https://www.ecb.europa.eu/press/pr/activities/mopo/html/index.en.html, each
setting $\Lambda_{t}=1$.

Figure 1: Realized Volatility series and relevant events (red and blue dashed-
lines) for financial markets. Sample Period: June 1, 2009 to December 31, 2019

In what follows, we chose to set the number of states to 2, after a
preliminary investigation among the alternatives. As it is well known, in the
MS world competing models cannot be compared with classical tests for the
problem of nuisance parameters present only under the alternative hypothesis
([33]). This notwithstanding, we have estimated 3-state MS models for each
market and compared them with the 2-state and the no MS cases, relying on the
information criteria as a tool for model selection ([34]); across all series,
the 2 state case is preferred in terms of BIC. This result is further
confirmed by the results of the 3–state case estimate, where the switching
coefficient $\varphi$ of the third state is not significant.999Results are
available among the supplemental material.

###  3.1 Estimation results

Estimation results of our MS–ACM are shown in Table 1 for the volatilities of
each financial index.101010Initializing ML estimation from 11 sets across a
rather wide range of random starting values for each model, we converge to the
same coefficient estimates, up to the third digit. On average, the
computational time of the full estimation procedure, using a GAUSS code
running on an I5–7th gen 2.5 Ghz processor, is approximately 2 minutes. They
support our choice to assume the constant of the $\xi_{t,st}$ component as
switching: in particular, the constant in the low volatility regime is equal
to zero across indices (a regime without monetary policy announcements),
whereas it increases remarkably in the high volatility regime. As a
consequence, we removed $\varphi_{0}$ from all models, and, for the same
reasons, the AR coefficient $\psi$. The unconventional policy proxy
significantly enters the model with a negative sign, as it is expected to
reduce the volatility level, with the strongest impact observed for the IBEX35
(-1.09) and the weakest occurring for the DAX30 (-0.44). As per the
probability coefficients, the probability of remaining in the low regime is
higher than that of the high volatility regime, leading to an average duration
of 1 day (calculated as $\frac{1}{1-p_{ii}},i=0,1$) in the high regime for all
markets, while the duration in the low regime ranges between 14 days (FTSEMIB)
and 53 days (DAX30).

Some model diagnostics reinforces the quality of our approach: in Table 2 we
show a few tests on the residuals. The Ljung–Box statistics, calculated for
lags 1, 5, 10, demonstrate that no substantial autocorrelation remains in the
residuals, which also show sample mean equal to 1, in line with the hypothesis
about the center of the Gamma distribution. Additionally, a Kolmogorov-Smirnov
test fails to reject the null hypothesis (even at a 10% significance level)
that the distribution is a mixture of two Gamma densities weighted by the
corresponding ergodic probabilities (derived from the estimated Markov Chain).
Furthermore, in order to compare the goodness of fit across alternative
models, we built an in-sample Model Confidence Set ([35]); the competing
models, next to our MS–ACM (Eq. 1), are the classical AMEM ([9]), the AMEMX
(an AMEM with $x_{t}$ added as a regressor), and the ACM (Eq. 2). The
procedure always selects our MS model as the best model and none of the
alternatives falls into the best set. Finally, as a further check, we
calculated the cross correlations at lag 1 between the residuals across
markets; the results show very low residual lagged cross correlations (all
below $0.1$ and most below $0.05$), pointing to the fact of no evidence of
dynamic interdependence. Evidently, this issue should be addressed within a
multivariate analysis, but this goes beyond the scope of this work.

In Figure 2, we reproduce the time–varying constant $\hat{\varphi}_{t}$,
superimposing the dates at which an announcement occurred (an information not
provided to the model). In line with an immediate absorption of the news, the
series jump in correspondence of monetary policy announcements, more
frequently so for the FTSEMIB and the IBEX35, than for the CAC40 and the
DAX30. In order to add readability to the results, we consider three
meaningful dates, in detail: August 4, 2011, when the ECB gave additional
details on a Longer Term Refinancing Operation (LTRO); August 2, 2012, when
the ECB communicated to the market that there would have been no changes of
interest rates, and December 3, 2015, when, conversely, only the interest rate
on deposit facility was decreased by 10bps. Note that these announcements
depressed returns in all cases, with the worst loss happening at the
unconventional policy announcement (August 4, 2011), between 4.41% (DAX30) and
6.9% (FTSEMIB). Interestingly, in the case of the two conventional policy
announcements, the estimated probability to be in the high regime is near 1;
on the day of the unconventional policy announcement (August 4, 2011), the
process appears to be in the low regime, an empirical support to the idea that
unconventional policies were successful in reducing stock market
volatility.111111This fact is also confirmed by the negative sign of the
$\delta$ coefficient in Table 1. Finally, we observe a reduction in
$\hat{\varphi}_{t}$ after the announcements, as evidence of the short
estimated permanence in the high volatility regime.

As a general remark, we find support for our motivation to address the very
stylized facts of RV within the MEM class: persistence, captured by an ARMA
representation of the dynamics; asymmetric response of volatility to the sign
of past returns, captured by a significant asymmetric term; lack of serial
correlation in the residuals, ascertained by the usual Ljung–Box test
statistics; the addition of MS features advanced here allows us to improve the
goodness of fit, as discussed before, and a different consideration of quiet
and turbulent periods.

Table 1: Estimation results (robust sandwich form of the standard errors [36] in parentheses) of four MS–ACMs relative to different Eurozone volatility series. Sample period: June 1, 2009 - December 31, 2019. | CAC40 | DAX30 | FTSEMIB | IBEX35
---|---|---|---|---
$\omega$ | 0.853 | 0.661 | 1.063 | 1.059
| (0.107) | (0.072) | (0.122) | (0.079)
$\alpha$ | 0.142 | 0.185 | 0.228 | 0.151
| (0.017) | (0.015) | (0.019) | (0.018)
$\beta$ | 0.732 | 0.722 | 0.649 | 0.732
| (0.024) | (0.019) | (0.024) | (0.019)
$\gamma$ | 0.112 | 0.082 | 0.080 | 0.086
| (0.011) | (0.009) | (0.019) | (0.011)
$\delta$ | -0.776 | -0.440 | -0.741 | -1.090
| (0.131) | (0.104) | (0.157) | (0.121)
$\varphi_{1}$ | 6.273 | 6.420 | 4.552 | 6.152
| (1.811) | (1.647) | (1.853) | (2.534)
$p_{00}$ | 0.964 | 0.981 | 0.928 | 0.943
| (0.025) | (0.017) | (0.027) | (0.036)
$p_{11}$ | 0.222 | 0.303 | 0.337 | 0.313
| (0.122) | (0.254) | (0.460) | (0.099)
$\theta_{0}$ | 8.852 | 11.248 | 15.034 | 11.992
| (0.379) | (0.509) | (1.375) | (0.756)
$\theta_{1}$ | 3.271 | 2.595 | 4.112 | 3.777
| (0.737) | (0.767) | (1.211) | (0.683)
Table 2: Residual diagnostics of four MS-ACMs relative to different Eurozone
volatility series: p-values of Ljung–Box statistics for different lags, mean
of residuals (standard deviation in parentheses) and Kolmogorov-Smirnov
statistic.

| CAC40 | DAX30 | FTSEMIB | IBEX35
---|---|---|---|---
Ljung–Box 1 | 0.711 | 0.161 | 0.403 | 0.220
Ljung–Box 5 | 0.263 | 0.121 | 0.449 | 0.087
Ljung–Box 10 | 0.364 | 0.276 | 0.639 | 0.340
mean | 0.994 | 0.998 | 0.996 | 0.994
| (0.343) | (0.321) | (0.292) | (0.303)
Kolmogorov-Smirnov | 0.019 | 0.015 | 0.016 | 0.015

The critical values of the Kolmogorov–Smirnov statistic are 0.024 at a
significance level of 0.10, 0.026 at a significance level of 0.05, 0.031 at a
significance level of 0.01.

Figure 2: $\hat{\varphi}_{t}$ as a weighted average resulting from the MS–ACM
(blue line) and monetary policy announcements (red dots). The period depicted
spans July 1, 2011 to December 31, 2015. Black dots correspond to three
meaningful announcements (details in the text): August 4, 2011, August 2,
2012, and December 3, 2015.

###  3.2 Classification Results

The estimated smoothed probabilities are used to classify the 144 dates of
announcements into the three categories: Plank, Squat, and Jump. We calculated
the variable $\Delta\hat{p}_{t}$ (see Eq.(3)) and apply a classical k–means
algorithm, minimizing the sum of squares from points to the assigned cluster
centers (the average of $\Delta\hat{p}_{t}$ within each group), to form three
groups. In the first column of Table 3 we show the number of dates belonging
to each group for each market. Most of the announcements do not cause a change
in volatility (Plank); however, a remarkable percentage of the announcements
cause a switching from the low to the high volatility for FTSEMIB and IBEX35,
that is 15.3% and 8.3%, respectively, whereas it is less than 5% for CAC40 and
DAX30. Some important announcements belonging to this category refer both to
conventional policies (for example, the decrease of interest rates established
on December 3, 2015, when they became negative) and unconventional policies
(e.g. the announcement on March 10, 2016, when the amount of securities
purchased within the implementation of the EAPP passes from € 60 to € 80
billion per month; the announcement on September 12, 2019, when the ECB
decided to run the EAPP as long as necessary).

Finally, a smaller percentage of the announcements caused a switching from the
high to the low volatility regimes: some examples are represented by the
details on the Covered Bond Purchases Program (CBPP) released on June 4, 2009
(for the CAC40), the Security Market Program (SMP) on May 10, 2010 (for the
FTSEMIB) and the announcement on June 8, 2017 concerning details on the EAPP
(for the IBEX35).

Our analysis confirms the results in [7] where the dummy representing the
announcements has a positive sign and is more significant for the FTSEMIB and
the IBEX35. The announcement effect seems to be more pronounced for the
volatility of the Italian and Spanish markets than for the French and German
ones, in line with the more stable performance of the latter during this
turbulent period.

In the last two columns of Table 3 we show the clustering obtained with the
two smoothed probability–based approaches (we merged the two Plank cases in
the SP–level classification, in order to make the results comparable with the
other classifications). The outcome seems quite similar across methods, with a
larger number of cases identified as Plank announcements relative to what the
k–means clustering delivers. Also, considering the SP–level approach, we can
notice that almost all Plank cases are identified when the regime was one of
low volatility at time $t-1$.

The centers of each group are around 0 for the Plank group across
classification methods. Some differences are present in the centers of the
other two categories, in particular between the k–means method and the two
smoothed probability–based approaches; FTSEMIB seems to be the one market with
less sharp classifications when derived from the three methods.

In Table 4 we report the values of $U$ for each classification method and each
series. In general, SP-diff presents the lowest $U$ values, but also SP-level
has a very similar performance.

A formal evaluation of the differences in the classifications obtained with
the three alternative methods can be conducted by means of the adjusted Rand
index [37, 38]. Such a method is generally used to compare the groups obtained
by a certain algorithm with respect to a benchmark clustering; in our case, we
use it to verify the similarity of clustering methods by taking possible pairs
in turn. The Rand index ranges in [0, 1], and takes on value 1 when the two
methods provide the same clustering, and value 0 in the case of maximum
difference between them. In Table 5 we show the values of this index across
volatility series; they are always larger than 0.9 (with the slight exception
of the comparison between k–means and SP–diff for the FTSEMIB, which is 0.85),
with value 1 in the case of the two smoothed probability–based methods for the
CAC40 and the DAX30. Given that the three alternative methods provide very
similar results, the smoothed probability–based solutions receive a good
support as an immediate by–product of the model estimation, not requiring
further statistical clustering algorithms.

Furthermore, to verify if the announcements are classified in a similar way,
this time across the four volatility series, we calculate the Rand index for
the same clustering method, but for different markets (Table 6); once again,
we get a strong agreement in the classification results, with Rand indices
always larger than 0.74, with a clear similarity between the CAC40 and the
DAX30. Moreover, it seems that the SP–diff method provides more similar
patterns among markets relative to the other two methods. By comparison, the
k–means provides lower values, but still high in terms of similarity.

We can conclude that the smoothed probability–based classifications confirm
their good performance in providing reliable results, also in view of their
very practical derivation, using the more formal k–means statistical
clustering as a benchmark classification.

Table 3: Classification of announcements for the Eurozone volatility series using three alternative algorithms. The numbers in parentheses are the centers of the corresponding group. In the group Plank of the SP–level classification, the number in square brackets represents the cases with high volatility both at $t$ and $t-1$ (High–Plank). Group | k–means | SP–level | SP–diff | k–means | SP–level | SP–diff
---|---|---|---|---|---|---
| CAC40 | DAX30
Plank | 132 | 136 [0] | 136 | 136 | 140 [1] | 140
| (0.010) | (0.004) | (0.004) | (0.007) | (0.005) | (0.005)
Squat | 5 | 1 | 1 | 3 | 0 | 0
| (-0.261) | (-0.561) | (-0.561) | (-0.198) | - | -
Jump | 7 | 7 | 7 | 5 | 4 | 4
| (0.704) | (0.704) | (0.704) | (0.664) | (0.733) | (0.733)
| FTSEMIB | IBEX35
Plank | 118 | 125 [2] | 133 | 123 | 131 [3] | 132
| (0.010) | (0.018) | (0.035) | (0.020) | (0.008) | (0.005)
Squat | 4 | 2 | 1 | 9 | 2 | 1
| (-0.460) | (-0.606) | (-0.726) | (-0.309) | (-0.498) | (-0.526)
Jump | 22 | 17 | 10 | 12 | 11 | 11
| (0.539) | (0.599) | (0.731) | (0.715) | (0.745) | (0.745)
Table 4: Evaluation of uncertainty by $U$ index for three alternative classification methods of four Eurozone volatility series. | k–means | SP–level | SP–diff
---|---|---|---
CAC40 | 0.124 | 0.088 | 0.088
DAX30 | 0.082 | 0.054 | 0.054
FTSEMIB | 0.244 | 0.210 | 0.192
IBEX35 | 0.207 | 0.156 | 0.154
Table 5: Adjusted Rand index between classification methods by volatility series. | k–means/ | k–means/ | SP–level/
---|---|---|---
| SP–level | SP–diff | SP–diff
CAC40 | 0.962 | 0.962 | 1
DAX30 | 0.962 | 0.962 | 1
FTSEMIB | 0.925 | 0.851 | 0.919
IBEX35 | 0.922 | 0.913 | 0.990
Table 6: Adjusted Rand index between pairs of volatility series by classification method. | k–means | SP–level | SP–diff
---|---|---|---
CAC40/DAX30 | 0.924 | 0.962 | 0.962
CAC40/FTSEMIB | 0.792 | 0.875 | 0.951
CAC40/IBEX35 | 0.854 | 0.913 | 0.923
DAX30/FTSEMIB | 0.812 | 0.859 | 0.933
DAX30/IBEX35 | 0.805 | 0.897 | 0.906
FTSEMIB/IBEX35 | 0.741 | 0.828 | 0.912

##  4 Concluding Remarks

In this paper we derive a novel Markov Switching Multiplicative Error Model to
include a component related to monetary policy actions: such a model extends
the recent MEM–class contribution by [7] which accommodates an additive
component related to volatility dynamics induced by policy measures. As a
relevant by–product, we advance a simple–to–obtain suggestion on how to map
the information on estimated volatility regimes to classify announcements of a
Central Bank in terms of their impact on volatility levels. Recent econometric
literature is producing great efforts in evaluating these transmission
mechanisms in real and financial economies, but generally they assume
different announcements as producing the same effect.

In the model we propose within a more realistic scenario, the announcements
are allowed to have different importance, and no prior classification is
imposed. With MS features, our model has the merit of extracting from
volatility an unobservable signal attributable to the unconventional policy
effects, with jumps in its intercept as a consequence of a policy
announcement: the estimated parameters allow us to derive a procedure to map
the variations in the switching intercept into groups interpretable as policy
effects on volatility.

We propose two smoothed probability–based classifications, one based on the
thumb rule of the classification derived from the mode of the regime (SP–level
method), and another based on the differences of the same smoothed
probabilities (we call it SP–diff). The smoothed probabilities are a
by–product of the Maximum Likelihood estimation, so they allow for an
immediate classification of the announcements. The application on four
European volatility series in the empirical application shows how the smoothed
probability–based classifications provide a very similar clustering with
respect to a statistical clustering algorithm (k–means). Such a classification
method based on the smoothed probabilities is extendable to all cases when a
MS model is a suitable representation of the data.

Given the time series framework, the procedure could be extended to be adapted
to the real–time classification of the announcements, when new observations
are available, and use the framework in a forecasting context, not examined
here. Given that the classification need to be reapplied to each new
announcement, it remains to be seen how robust the previous classification is
to the inclusion of new data or, for that matter, to outliers (see, for
example, [39]). Moreover, it could be interesting to verify whether such a
classification method is robust with respect to alternative time–varying
models, such as the smooth-transition MEM proposed in [22], and alternative
clustering methods.

An interesting extension121212We thank an anonymous referee to have suggested
this possibility. is to model the duration in a certain regimes as a function
of the announcements. One way to investigate this concern is to adopt a
time–varying transition probability (TVTP) matrix, where the probability to
stay in the same regime at time $t$ depends on the state at $t-1$ and the
dummy $\Lambda_{t}=1$ when there is an announcement at time $t$. As a
consequence, we are able to estimate the duration of a certain state in the
case of no announcement and in the case of announcement. We did estimate such
a TVTP MS–ACM and we indeed confirm that both probabilities, to stay in regime
0 and in regime 1, change in correspondence of the announcements, in the
direction of a switch from regime 0 to 1 or the permanence in regime 1 (the
coefficient for $p_{11}$ is, however, not significant, in view perhaps of the
short duration of regime 1). This model outperforms the standard model in
terms of Likelihood Ratio test, but, when used for classification purposes
(the main goal in this context), it introduces less sharp results relative to
the MS–ACM case, as it provides smoothed probabilities often close to 0.5. To
save space, these results are available in the supplemental material section.

## References

  * [1] G. Kapetanios, H. Mumtaz, I. Stevens, K. Theodoridis, Assessing the economy-wide effects of quantitative easing, The Economic Journal 122 (564) (2012) F316–F347.
  * [2] M. H. Pesaran, R. P. Smith, Counterfactual analysis in macroeconometrics: an empirical investigation into the effects of quantitative easing, Research in Economics 70 (2) (2016) 262–280.
  * [3] A. Shogbuyi, J. M. Steeley, The effect of quantitative easing on the variance and covariance of the UK and US equity markets, International Review of Financial Analysis 52 (2017) 281–291.
  * [4] D. Kenourgios, S. Papadamou, D. Dimitriou, Intraday exchange rate volatility transmissions across QE announcements, Finance Research Letters 14 (2015) 128–134.
  * [5] J. M. Steeley, A. Matyushkin, The effects of quantitative easing on the volatility of the gilt-edged market, International Review of Financial Analysis 37 (2015) 113–128.
  * [6] R. F. Engle, New frontiers for ARCH models, Journal of Applied Econometrics 17 (2002) 425–446.
  * [7] D. Lacava, G. M. Gallo, E. Otranto, Measuring the effects of unconventional policies on stock market volatility, CRENoS Working Paper: 2020_06 (2020).
  * [8] C. T. Brownlees, F. Cipollini, G. M. Gallo, Multiplicative Error Models, in: L. Bauwens, C. Hafner, S. Laurent (Eds.), Volatility Models and Their Applications, Wiley, 2012, pp. 223–247.
  * [9] R. F. Engle, G. M. Gallo, A multiple indicators model for volatility using intra-daily data, Journal of Econometrics 131 (2006) 3–27.
  * [10] F. Corsi, A simple approximate long-memory model of realized volatility, Journal of Financial Econometrics 7 (2009) 174–196.
  * [11] E. Maharaj, P. D’Urso, J. Caiado, Time Series Clustering and Classification, Chapman and Hall/CRC, 2019.
  * [12] T. W. Liao, Clustering of time series data—a survey, Pattern Recognition 38 (11) (2005) 1857–1874.
  * [13] S. Aghabozorgi, A. S. Shirkhorshidi, T. Y. Wah, Time-series clustering–a decade review, Information Systems 53 (2015) 16–38.
  * [14] J. Caiado, N. Crato, A GARCH–based method for clustering of financial time series: international stock markets evidence, in: C. Skiadas (Ed.), Recent Advances in Stochastic Modeling and Data Analysis, World Scientific Publishing, 2007, pp. 542–555.
  * [15] E. Otranto, Clustering heteroskedastic time series by model-based procedures, Computational Statistics and Data Analysis 52 (10) (2008) 4685–4698.
  * [16] G. De Luca, P. Zuccolotto, A tail dependence-based dissimilarity measure for financial time series clustering, Advances in Data Analysis and Classification 5 (4) (2011) 323–340.
  * [17] P. D’Urso, C. Cappelli, D. Di Lallo, R. Massari, Clustering of financial time series, Physica A: Statistical Mechanics and its Applications 392 (9) (2013) 2114 – 2129.
  * [18] E. Otranto, R. Gargano, Financial clustering in presence of dominant markets, Advances in Data Analysis and Classification 9 (3) (2015) 315–339.
  * [19] R. F. Engle, Autoregressive Conditional Heteroscedasticity with estimates of the variance of United Kingdom inflation, Econometrica 50 (1982) 987–1007.
  * [20] T. Bollerslev, Generalized Autoregressive Conditional Heteroskedasticity, Journal of Econometrics 31 (1986) 307–327.
  * [21] T. G. Andersen, T. Bollerslev, F. X. Diebold, P. Labys, The distribution of realized exchange rate volatility, Journal of the American Statistical Association 96 (2001) 42–45.
  * [22] G. M. Gallo, E. Otranto, Forecasting realized volatility with changing average levels, International Journal of Forecasting 31 (2015) 620–634.
  * [23] F. Cipollini, R. F. Engle, G. M. Gallo, Semiparametric vector MEM, Journal of Applied Econometrics 28 (2013) 1067–1086.
  * [24] E. Otranto, Capturing the spillover effect with multiplicative error models, Communications in Statistics-Theory and Methods 44 (15) (2015) 3173–3191.
  * [25] S. F. Gray, Modeling the conditional distribution of interest rates as a regime-switching process, Journal of Financial Economics 42 (1) (1996) 27–62.
  * [26] M. J. Dueker, Markov switching in GARCH processes and mean-reverting stock-market volatility, Journal of Business & Economic Statistics 15 (1) (1997) 26–34.
  * [27] F. Klaassen, Improving GARCH volatility forecasts with regime-switching GARCH, Empirical Economics 27 (2) (2002) 363–394.
  * [28] J. D. Hamilton, Time Series Analysis, Princeton University Press, 1994.
  * [29] C. Kim, Dynamic linear models with Markov switching, Journal of Econometrics 60 (1994) 1–22.
  * [30] M. Augustyniak, M. Boudreault, M. Morales, Maximum likelihood estimation of the markov-switching garch model based on a general collapsing procedure, Methodology and Computing in Applied Probability 20 (1) (2018) 165–188.
  * [31] M. Augustyniak, Maximum likelihood estimation of the Markov-switching GARCH model, Computational Statistics & Data Analysis 76 (2014) 61–75.
  * [32] D. C. Wee, F. Chen, W. T. Dunsmuir, Likelihood inference for Markov-switching GARCH (1,1) models using sequential Monte Carlo, Econometrics and Statistics (2020).
  * [33] B. E. Hansen, The likelihood ratio test under nonstandard conditions: testing the Markov switching model of GNP, Journal of Applied Econometrics 7 (S1) (1992) S61–S82.
  * [34] Z. Psaradakis, M. Sola, F. Spagnolo, N. Spagnolo, Selecting nonlinear time series models using information criteria, Journal of Time Series Analysis 30 (4) (2009) 369–394.
  * [35] P. R. Hansen, A. Lunde, J. M. Nason, The Model Confidence Set, Econometrica 79 (2) (2011) 453–497.
  * [36] H. White, Maximum likelihood estimation of misspecified models, Econometrica 50 (1982) 1–25.
  * [37] W. Rand, Objective criteria for the evaluation of clustering methods, Journal of the American Statistical Association 66 (1971) 846–850.
  * [38] L. Hubert, P. Arabie, Comparing partitions, Journal of Classification 2 (1985) 193–218.
  * [39] P. D’Urso, L. De Giovanni, R. Massari, GARCH-based robust clustering of time series, Fuzzy Sets and Systems 305 (2016) 1–28.