# Company classification using zero-shot learning

1st Maryan Rizinski  Faculty of Comp. Sci. and Eng.
Ss. Cyril and Methodius University
Skopje, North Macedonia
<EMAIL_ADDRESS>2nd Andrej Jankov  Faculty of Comp. Sci. and Eng.
Ss. Cyril and Methodius University
Skopje, North Macedonia
<EMAIL_ADDRESS>3nd Vignesh Sankaradas  Department of
Computer Science
Boston University, Metropolitan College
Boston, MA 02215, USA
<EMAIL_ADDRESS>4nd Eugene Pinsky  Department of Computer Science
Boston University, Metropolitan College
Boston, MA 02215, USA
<EMAIL_ADDRESS>5rd Igor Miskovski  Faculty of Comp. Sci. and Eng.
Ss. Cyril and Methodius University
Skopje, North Macedonia
<EMAIL_ADDRESS>6th Dimitar Trajanov  Faculty of Comp. Sci. and
Eng.
Ss. Cyril and Methodius University
Skopje, North Macedonia
<EMAIL_ADDRESS>
###### Abstract

In recent years, natural language processing (NLP) has become increasingly
important in a variety of business applications, including sentiment analysis,
text classification, and named entity recognition. In this paper, we propose
an approach for company classification using NLP and zero-shot learning. Our
method utilizes pre-trained transformer models to extract features from
company descriptions, and then applies zero-shot learning to classify
companies into relevant categories without the need for specific training data
for each category. We evaluate our approach on a dataset obtained through the
Wharton Research Data Services (WRDS), which comprises textual descriptions of
publicly traded companies. We demonstrate that the approach can streamline the
process of company classification, thereby reducing the time and resources
required in traditional approaches such as the Global Industry Classification
Standard (GICS). The results show that this method has potential for
automation of company classification, making it a promising avenue for future
research in this area.

###### Index Terms:

company classification, natural language processing, machine learning, zero-
shot learning, finance

##  I Introduction

In financial research, company classification is a popular approach that
involves grouping similar companies into categories or clusters [1, 2]. This
process of classifying companies into discrete categories has numerous
practical applications for financial researchers, analysts, decision-makers,
and investors. For example, it can help manage portfolio risk, facilitate
relative valuation, and enable peer-group comparisons [3]. Additionally, it
can aid in the analysis of the effects of corporate reorganizations, changes
in financial and investment policies, and the evaluation of the performance of
a specific company against a set of similar companies. Beyond the financial
sector, company classification can be useful in generating prospective leads
for sales and marketing teams, identifying new clients for insurance
companies, and pinpointing competitors for corporations. Investment banks and
venture capital firms can also benefit from company classification by
understanding the distribution of companies among different industries [4].

The task of company classification has traditionally relied on established
standards, such as the Standard Industrial Classification (SIC), the North
American Industry Classification System (NAICS), the Fama and French (FF)
model, and the Global Industry Classification Standard (GICS), among others.
Nonetheless, these methods have several limitations that hinder their
effectiveness. One significant challenge is the lack of interoperability
between different schemes, which results in classification inconsistencies due
to vendor-specific assignments. Furthermore, the creation of existing
standards is a laborious and time-consuming process that requires input from
human experts, making them prone to subjectivity.

As products and services become increasingly complex, updating classification
schemes becomes a challenging task. The dynamic market environment in which
companies operate causes frequent changes in their business, affecting their
industry affiliation. The existing classification standards are static and are
unable to keep up with the fast-changing environments. The current schemes
heavily rely on self-reporting and manual entry, resulting in slow, costly,
and ineffective updates when adapting to the changed business landscape. In
the absence of capabilities for real-time updates, these existing standards
may not be the optimal choice in various application settings, thereby
emphasizing the need to explore automation techniques.

The recent advancements in machine learning (ML) and natural language
processing (NLP) can be explored to address the limitations of the traditional
standards for company classification, and hold promises for reducing costs,
complexity, and manual labor. In particular, text classification using NLP
methods has achieved significant progress over the past decade. Large-scale
pre-trained transformer models have revolutionized the field of text
classification, enabling successful implementations across various application
domains such as machine translation, text summarization, and sentiment
analysis. These models witnessed successful deployments in systems that need
scalability and real-time analysis, making them also valuable to address the
problem of company classification.

TABLE I: GICS taxonomy illustrating the names of the four classification
levels, from broadest to narrowest, the number of categories for each
classification level and the number of digits used to represent each level.
GICS taxonomy
---
Level | Title |  | Number of
---
categories
Digits
Level 1 (broadest) | Sector | 11 | first 2 digits
Level 2 | Industry Group | 24 | first 4 digits
Level 3 | Industry | 64 | first 6 digits
Level 4 (narrowest) | Sub-industry | 139 | all 8 digits

In this paper, we explore the use of zero-shot classification using the model
valhalla/distilbart-mnli-12-3 on the Wharton Research Data Services (WRDS)
dataset. The WRDS dataset contains names and textual descriptions of 34,338
companies that are classified as per the GICS index. In the classification
experiment, we calculate standard metrics such as precision, recall, F1 score
and support for each classification category as well as for the overall model.
Our purpose is to evaluate the potential of zero-shot learning and use it as a
basis for future research with unsupervised and supervised learning.

The organization of the paper is as follows. Section II focuses on background
information and preliminaries. We review related work in the literature in
Section III. In section IV, we describe our NLP-based experiment and dataset
for company classification using zero-shot learning and discuss the obtained
results. Section V concludes the paper.

##  II Standards for company classification

###  II-A Definition of company classification

Company classification, also known as industry classification, involves
categorizing companies based on their business activities, industry, and other
relevant factors. The objective of this process is to group similar companies
together and distinguish them from others based on several comparison
parameters [5]. Homogeneity is a crucial criterion for selecting an industry
classification standard from the available options, and it is typically
evaluated using various approaches. For instance, [3] suggests using stock
return co-movement, while [1] recommends utilizing 12 fundamental variables.
By segmenting the market into partitions with distinct business and financial
characteristics [6, 7], these classifications provide a framework for
understanding the similarities and differences among companies. Ultimately,
the goal is to identify groups of businesses that engage in similar market
activities and have comparable market conditions.

###  II-B Mainstream standards

Industry classification standards are an essential tool for economic analysis,
financial research, and policy-making. Among the most prominent industry
classification systems are the Standard Industrial Classification (SIC), the
North American Industry Classification System (NAICS), Fama French (FF), and
the Global Industry Classification Standard (GICS). The SIC system,
established in the 1930s, is the oldest of the four and underwent periodic
revisions to adapt to changes in the economy. However, these efforts were
insufficient, prompting the governmental statistical agencies of the United
States, Canada and Mexico, to collaborate on a joint initiative to improve the
SIC system and create a more comprehensive and unified classification scheme
across North America. The result was the creation of the NAICS system in 1999.
The 2017 edition of the NAICS taxonomy partitions the North American economy
into 1057 industries with a six-digit code for each industry. The NAICS system
includes 20 sectors, 99 subsectors, and 311 industry groups [4].

The FF system was initially conceptualized by academic researchers in finance
as a means to investigate the industrial cost of capital [8]. FF achieves its
purpose by reclassifying the existing SIC codes and grouping companies into 48
distinct industry sectors. Despite its prevalence in academic research
concerning asset pricing, corporate finance, accounting, and economics, the FF
system has not gained much popularity within the financial industry. In
contrast, the Global Industry Classification Standard (GICS) was specifically
designed by Standard & Poor’s (S&P) and Morgan Stanley Capital International
(MSCI) to meet the needs of financial professionals, such as investment
managers and financial analysts. As shown in Table I, GICS employs an eight-
digit code to classify companies, and its structure is hierarchical,
encompassing 10 sectors subdivided into 24 industry groups, 64 industries, and
139 subindustries111A more detailed information about the GICS index is
available at https://www.msci.com/our-solutions/indexes/gics. The GICS scheme
classifies companies based on their business activity, sources of revenue and
earnings, as well as market perception concerning their primary lines of
business [3].

GICS has been shown to outperform other popular industry classification
systems, such as SIC, NAICS, and FF, in various comparison experiments [1, 9,
10, 11, 3]. This is due to its superior ability to capture industry
homogeneity, leading to more accurate industry classifications. Additionally,
the GICS index has been found to exhibit robust classification performance not
only in settings with large and well-known companies (e.g. S&P companies), but
also when applied to smaller and less-followed companies [12]. This strong
performance across a wide range of companies and industries makes the GICS
index an ideal candidate for zero-shot learning contexts, as utilized in this
paper.

Apart from the widely used mainstream classification schemes, there are
several accessible alternatives that may not be as popular among institutional
practitioners. The interested reader may refer to Bloomberg, Capital IQ
(available on finance.yahoo.com), Hoovers & First Research, Market Guide,
MarketLine, Morningstar, and Thomson Reuters (available on msn.money.com) [2].
Other schemes include the Thomson Reuters Business Classification (TRBC),
Industry Classification Benchmark (ICB), and International Standard Industrial
Classification of All Economic Activities (ISIC) [13].

TABLE II: Modified sector names of the GICS taxonomy using TF-IDF
preprocessing and removal of stop words with the purpose of increasing the F1
score in zero-shot classification. Sector names
---
Original GICS names | Names after TF-IDF
Energy |  | Oil, Natural Gas, Consumable Fuels
---
and Petroleum
Materials |  | Raw Materials, Mining, Minerals and
---
Metals (Gold, Silver and Copper)
Industrials | Industrials and Transportation
Consumer Discretionary |  | Non-Essential Goods, Retail
---
and E-Commerce
Consumer Staples |  | Food, Beverages and
---
Household Products
Health Care | Health Care
Financials | Banking and Lending
Information Technology | Software, Technology and Systems
Communication Services |  | Communications, Telecommunications,
---
Networking, Media and Entertainment
Utilities |  | Utilities, Energy Distribution and
---
Renewable Energy
Real Estate | Real Estate Properties

###  II-C Issues with the current standards

Despite the presence of diverse standards available for company classification
and their wide use, the existing classification schemes suffer from several
important limitations that deserve attention. Assigning companies to
industries is currently performed manually and is vendor-specific [2]. The
process is time-consuming, subjective, and prone to inaccuracies, as existing
classification schemes are often constructed and maintained by domain experts.
These schemes can also become quickly outdated due to market developments and
changes in products, technology, and business patterns. Therefore, relying
solely on human-aided classification is not optimal. In fact, even with domain
expertise and sufficient data, deciding which companies belong to an industry
is not straightforward.

Inconsistencies in the classification process across different data vendors
can pose issues in terms of accuracy and homogeneity, as highlighted by [1].
This is exemplified by the findings in [14], which analyzed manually assigned
SIC codes for companies and revealed a significant discrepancy between two
major data providers (Compustat and SRSP). Specifically, at the two-digit
level, approximately 36% of SIC classifications differed, while at the four-
digit level, almost 80% differed, as noted in [15].

The timely update of classification schemes to reflect changing business and
industry environments is another concern. Regardless of the underlying
standard used, the classification scheme should incorporate current and
frequently updated data. However, this poses a major challenge as adjusting
and updating classification schemes to reflect changes in company structure
and operations requires extensive human effort, which can be both time-
consuming and expensive [7].

With multitude of standards available, it is not not trivial to select an
appropriate industry classification scheme. The study in [16] highlights
several issues in this regard. A major concern is that the different data
sources used for classification exhibit mismatches even when applied within
the same classification system. As a result, the same company can be
classified in different partitions even under the same classification system
if different data sources are used for the classification.

##  III Related work

Although some studies have been conducted in recent years, the literature on
the application of NLP methods for industry classification remains limited
overall.

In [17], the authors investigate the effectiveness of deep learning models on
encyclopedic data from the English DBpedia Knowledge
Base222https://wiki.dbpedia.org. Specifically, the study evaluates the
performance of two popular models, Glove and ULMfit, against two baseline
models (one-hot unigram and one-hot bigram). The dataset used for the
experiments includes 300,000 textual descriptions of companies from DBpedia.
While the company descriptions are uniform in length, the dataset contains a
significant variation in industry representation. The results indicate that
the tested models perform well on the larger classes but exhibit a decline in
performance on the smaller classes. Importantly, the study does not use a
dataset that is considered a “gold standard”, and DBPedia lacks an established
industry taxonomy.

A relevant study, [13], uses the same dataset and experimental setup as [17],
but introduces BERT and XLNet models in addition to Glove and ULMfit. This
study compares the four models with the same baseline (one-hot unigram and
one-hot bigram) and finds that all algorithms perform acceptably on well-
represented classes, but experience decreased overall performance on less
represented classes. Although no algorithm stands out as the best for small
classes, XLNet and BERT demonstrate more stable performance overall, thanks to
their superior F1 scores. As noted in [13], there are currently no benchmark
datasets for industry classification. Previous studies rely on “Industry
Sector” data comprising 6,000 company descriptions collected from the web and
classified into 70 industry sectors, but they utilize algorithms that are
considered outdated in modern big data applications; these algorithms include
Naive Bayes, multinomial NB, Maximum Entropy classifier, Support Vector
Machine, and k-Nearest Neighbors.

The authors of [18] investigate the usefulness of text-based industry
classification using various word and document embedding techniques, in
conjunction with different clustering algorithms. Their approach is applied to
publicly traded companies in both the US and Chinese markets, and the results
are compared against the GICS index. The study utilizes advanced embedding
techniques such as BERT, but surprisingly, the results show that a simpler
technique, Latent Semantic Indexing (LSI), combined with k-means clustering,
outperforms BERT on two measures. This finding is remarkable because LSI, an
extension of conventional techniques such as Bag of Words (BoW) and Term
Frequency-Inverse Document Frequency (TF-IDF), is not commonly used in state-
of-the-art (SOTA) text classification. As a result, this study sheds new light
on the potential usefulness of LSI for text-based industry classification.

TABLE III: Distribution of companies in the WRDS dataset across various GICS
sectors. WRDS dataset
---
GICS sector | Number of companies
Energy | 2822
Materials | 3833
Industrials | 3934
Consumer Discretionary | 4662
Consumer Staples | 1433
Health Care | 4565
Financials | 5363
Information Technology | 5192
Communication Services | 1285
Utilities | 740
Real Estate | 509

A method for fine-tuning a pre-trained BERT model is proposed in [19], which
is then evaluated on two datasets consisting of US and Japanese company data.
The US dataset includes 2,462 annual reports from 2019 of companies listed on
the US stock market (Form 10-K documents), while the Japanese dataset contains
3,016 annual reports from 2018 of companies listed on the Tokyo Stock
Exchange. The paper’s objective is to explore the extent to which companies
with similar vector representations operate in comparable industries, as well
as to evaluate how effectively companies can be classified within a given
industry based solely on the industry name. The study compares BERT to two
baseline models, namely, BoW representation and skip-gram Word2Vec embedding,
and demonstrates BERT’s superior performance. The findings confirm the
effectiveness of the proposed approach and suggest the integration of
additional sources of business data, such as Price Earnings Ratio (PER) and
Price Book-value Ratio (PBR), to augment annual reports for the purpose of
industry classification.

Other relevant references include [7, 20, 4, 15], among others. [7] introduced
a new classification scheme called Business Text Industry Classification
(BTIC), which was built on a dataset of Form 10-K documents from S&P500
companies. Additionally, [20] proposed a Knowledge Graph Enriched BERT (KGEB)
model, which is capable of loading any pre-trained BERT model and fine-tuning
it for classification. KGEB enriches word representations with additional
knowledge through learning the graph structure in the underlying dataset. The
model was tested on a dataset of publicly listed companies on the Chinese
National Equities Exchange and Quotations (NEEQ). The authors of [4] employed
a deep neural network based on a multilayer perceptron architecture with four
fully-connected layers to predict the industries of novel companies. [15]
evaluated 28 classifiers based on four underlying Word2Vec models with varying
window sizes and different SVM kernels and logistic regression solvers. The
Word2Vec models were trained on a corpus of articles from the Guardian
newspaper with 600 million words. While the dataset is comprehensive, the
paper does not present a comparison with benchmark schemes such as GICS. The
use of Word2Vec is also somewhat outdated compared to SOTA deep learning
models.

##  IV Methodology and results

###  IV-A Dataset

We use the Wharton Research Data Services (WRDS) to create the dataset for
this research. WRDS is a web-based data management system that provides
researchers with access to a vast array of financial, economic, and marketing
data from different sources, including Compustat, CRSP, IBES, and others. WRDS
is a research platform that has been developed and maintained by the Wharton
School at the University of Pennsylvania to support researchers in their data-
driven research activities. The platform is used by academic researchers,
corporate professionals, and financial analysts to retrieve, manage, and
analyze large sets of data for their research projects. WRDS also provides a
suite of tools for data cleaning, analysis, and visualization to help users
get the most out of the data available on the platform. Using WRDS, we extract
the Compustat dataset which contains financial and market data on publicly
traded companies across the United States. The original dataset contains data
for 44,033 companies, including their names, descriptions, and classification
into sectors and industry groups as per the GICS taxonomy. We filter the
dataset due to the absence of GICS sector assignment for some entries. After
the filtering, the dataset consolidates a total of 34,338 entries (i.e.
companies). The distribution of the number of companies across the GICS
sectors is shown in Table III.

###  IV-B Experiment

We employ a zero-shot classification pipeline using the valhalla/distilbart-
mnli-12-3 model. We chose the valhalla/distilbart-mnli-12-3 model for its
popularity on Hugging Face, where it ranks as one of the top three models in
terms of number of downloads. To assess its effectiveness, we conducted a
comparative analysis with two other models, namely facebook/bart-large-mnli
and joeddav/xlm-roberta-large-xnli. Our evaluation showed that the
valhalla/distilbart-mnli-12-3 model performed slightly better than the other
two models.

The valhalla/distilbart-mnli-12-3 model belongs to the class of transformer
models which are a powerful type of neural network architecture that has been
widely adopted in NLP [21]. Transformer models are capable of modeling long-
range textual dependencies thereby effectively capturing relationships between
distant words in a sentence [22]. The transformer architecture includes an
attention mechanism which allows the model to selectively focus on relevant
parts of the input sequence. This enables the model to extract important
relationships between words and better capture the meaning of the input text
[23].

Zero-shot classification refers to the ability of a model to classify inputs
into multiple classes without requiring any training data [24]. Pre-trained
transformer models have shown potential in zero-shot classification tasks as
they have been trained on massive amounts of textual data, enabling them to
classify inputs into classes even if they have never seen examples of those
classes before. When needed, pre-trained models can be fine-tuned on specific
zero-shot classification tasks with only a small amount of training data,
allowing the models to adapt to the underlying task and improve the overall
performance (e.g. accuracy) on that task.

Figure 1: Confusion matrix of the zero-shot classification experiment using
the valhalla/distilbart-mnli-12-3 model on the WRDS dataset.

In our experiment, we utilized the valhalla/distilbart-mnli-12-3 model and
adopted the zero-shot classification technique. Specifically, we fed the model
with the company descriptions available in the WRDS dataset without performing
any fine-tuning of the model. As we employed the zero-shot learning approach,
we did not need to divide the dataset into train and test sets.

We obtained an F1 score of 0.56 using the original category names from the
GICS taxonomy. We aimed to boost the F1 score by modifying the sector names
with alternative labels that do not impede the model’s classification
performance. Specifically, we utilized TF-IDF vectorization to extract the top
30 most common words for each sector to obtain a more precise representation
of the sector names, with the goal of enhancing the accuracy of the zero-shot
classification model. This process involved pre-processing the company
descriptions using the NLTK333https://www.nltk.org/ library to identify all
verbs in the dataset and exclude them as stop words (since verbs are the most
frequently occurring words). In addition to default stop words, we also
excluded country names and certain abbreviations (e.g., ltd, llc) that occur
frequently but are not relevant to the companies. The original and modified
sector names are shown in Table II. This technique increases the F1 score to
0.64. However, the aforementioned change failed to result in any significant
improvement in the sectors that had the lowest F1 scores, namely Real Estate,
Consumer Staples, Consumer Discretionary, and Industrials.

The confusion matrix and classification report are presented on Figure 1 and
in Table V. As can be seen, the weighted F1 score obtained across the dataset
is 0.64. Highest individual F1 scores are obtained for Health Care and Oil &
Natural Gas with 0.84 and 0.81 F1 scores, followed by Banking & Lending and
Raw Minerals & Mining with 0.77 and 0.75 F1 scores respectively. Lowest F1
scores are noticed for Food, Beverages and Household Products with 0.30 F1
score, Real Estate with 0.39 F1 score, and Industrials and Transportation with
0.39 F1 score.

TABLE IV: Classification report for the valhalla/distilbart-mnli-12-3 model on the WRDS dataset with original sector names. | precision | recall | f1-score | support
---|---|---|---|---
Financials | 0.68 | 0.61 | 0.64 | 5363
Communication Services | 0.32 | 0.63 | 0.42 | 1285
Consumer Staples (Consumer Defensive) | 0.20 | 0.01 | 0.02 | 1433
Health Care | 0.83 | 0.84 | 0.83 | 4565
Industrials | 0.42 | 0.20 | 0.27 | 3934
Consumer Discretionary (Consumer Cyclical) | 0.41 | 0.46 | 0.43 | 4662
Energy | 0.56 | 0.91 | 0.69 | 2822
Materials | 0.54 | 0.65 | 0.59 | 3833
Real Estate | 0.29 | 0.89 | 0.44 | 509
Information Technology | 0.71 | 0.54 | 0.61 | 5192
Utilities | 0.44 | 0.25 | 0.32 | 740
accuracy |  |  | 0.56 | 34338
macro avg | 0.49 | 0.54 | 0.48 | 34338
weighted avg | 0.57 | 0.56 | 0.55 | 34338
TABLE V: Classification report for the valhalla/distilbart-mnli-12-3 model on the WRDS dataset with enhanced sector names. | precision | recall | f1-score | support
---|---|---|---|---
Banking and Lending | 0.84 | 0.71 | 0.77 | 5363
Communications, Telecommunications, Networking, Media and Entertainment | 0.39 | 0.62 | 0.48 | 1285
Food, Beverages and Household Products | 0.51 | 0.21 | 0.30 | 1433
Health Care | 0.80 | 0.89 | 0.84 | 4565
Industrials and Transportation | 0.57 | 0.29 | 0.39 | 3934
Non-Essential Goods, Retail and E-Commerce | 0.44 | 0.54 | 0.48 | 4662
Oil, Natural Gas, Consumable Fuels and Petroleum | 0.86 | 0.76 | 0.81 | 2822
Raw Materials, Mining, Minerals and Metals (Gold, Silver and Copper) | 0.86 | 0.67 | 0.75 | 3833
Real Estate Properties | 0.25 | 0.90 | 0.39 | 509
Software, Technology and Systems | 0.61 | 0.66 | 0.63 | 5192
Utilities, Energy Distribution and Renewable Energy | 0.47 | 0.80 | 0.59 | 740
accuracy |  |  | 0.64 | 34338
macro avg | 0.60 | 0.64 | 0.58 | 34338
weighted avg | 0.67 | 0.64 | 0.64 | 34338

##  V Conclusion

We highlight the growing importance of natural language processing (NLP) in
various business applications, including classification of companies. We
propose an approach for company classification using NLP and zero-shot
learning, which can streamline the process of categorizing companies and
potentially reduce the time and resources required for traditional approaches
such as the Global Industry Classification Standard (GICS). The proposed
approach utilizes pre-trained transformer models to extract features from
textual descriptions of companies and then applies zero-shot learning to
classify them into relevant categories without the need for specific training
data for each category. We evaluate the approach on the WRDS dataset and
demonstrate its effectiveness in classifying companies. The results show that
this method holds potential for automating the task of company classification,
which can benefit various industries, including finance, marketing, and
business intelligence, by providing a more efficient and cost-effective way of
categorizing companies. It can also help in identifying emerging trends and
patterns in the business world, which can be valuable for decision-making
processes.

## References

  * [1] S. Bhojraj, C. M. Lee, and D. K. Oler, “What’s my line? a comparison of industry classification schemes for capital market research,” _Journal of Accounting Research_ , vol. 41, no. 5, pp. 745–774, 2003.
  * [2] S. Lyocsa and T. Vyrost, “Industry classification: Review, hurdles and methodologies,” _Hurdles and Methodologies (September 30, 2009)_ , 2009.
  * [3] L. K. Chan, J. Lakonishok, and B. Swaminathan, “Industry classifications and return comovement,” _Financial Analysts Journal_ , vol. 63, no. 6, pp. 56–70, 2007.
  * [4] S. Wood, R. Muthyala, Y. Jin, Y. Qin, N. Rukadikar, A. Rai, and H. Gao, “Automated industry classification with deep learning,” in _2017 IEEE International Conference on Big Data (Big Data)_. IEEE, 2017, pp. 122–129.
  * [5] M. E. Porter and C. Strategy, “Techniques for analyzing industries and competitors,” _Competitive Strategy. New York: Free_ , 1980.
  * [6] R. L. Phillips and R. Ormsby, “Industry classification schemes: An analysis and review,” _Journal of Business & Finance Librarianship_, vol. 21, no. 1, pp. 1–25, 2016.
  * [7] H. Yang, H. J. Lee, S. Cho, and E. Cho, “Automatic classification of securities using hierarchical clustering of the 10-ks,” in _2016 IEEE International Conference on Big Data (Big Data)_. IEEE, 2016, pp. 3936–3943.
  * [8] E. F. Fama and K. R. French, “Industry costs of equity,” _Journal of financial economics_ , vol. 43, no. 2, pp. 153–193, 1997.
  * [9] C. O. Kile and M. E. Phillips, “Using industry classification codes to sample high-technology firms: Analysis and recommendations,” _Journal of Accounting, Auditing & Finance_, vol. 24, no. 1, pp. 35–58, 2009.
  * [10] K. Hrazdil and R. Zhang, “The importance of industry classification in estimating concentration ratios,” _Economics Letters_ , vol. 114, no. 2, pp. 224–227, 2012.
  * [11] L. Boni and K. L. Womack, “Analysts, industries, and price momentum,” _Journal of Financial and Quantitative analysis_ , vol. 41, no. 1, pp. 85–109, 2006.
  * [12] K. Hrazdil, K. Trottier, and R. Zhang, “A comparison of industry classification schemes: A large sample study,” _Economics Letters_ , vol. 118, no. 1, pp. 77–80, 2013.
  * [13] S. Slavov, A. Tagarev, N. Tulechki, and S. Boytcheva, “Company industry classification with neural and attention-based learning models,” in _2019 Big Data, Knowledge and Control Systems Engineering (BdKCSE)_. IEEE, 2019, pp. 1–7.
  * [14] K. M. Kahle and R. A. Walkling, “The impact of industry classifications on financial research,” _Journal of financial and quantitative analysis_ , vol. 31, no. 3, pp. 309–335, 1996.
  * [15] M. Lamby and D. Isemann, “Classifying companies by industry using word embeddings,” in _International Conference on Applications of Natural Language to Information Systems_. Springer, 2018, pp. 377–388.
  * [16] D. Katselas, B. K. Sidhu, and C. Yu, “Know your industry: The implications of using static gics classifications in financial research,” _Accounting & Finance_, vol. 59, no. 2, pp. 1131–1162, 2019.
  * [17] A. Tagarev, N. Tulechki, and S. Boytcheva, “Comparison of machine learning approaches for industry classification based on textual descriptions of companies,” in _Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)_ , 2019, pp. 1169–1175.
  * [18] J. He, K. Chen _et al._ , “Exploring machine learning techniques for text-based industry classification,” 2020.
  * [19] T. Ito, J. Camacho-Collados, H. Sakaji, and S. Schockaert, “Learning company embeddings from annual reports for fine-grained industry characterization,” in _Proceedings of the Second Workshop on Financial Technology and Natural Language Processing_ , 2020, pp. 27–33.
  * [20] S. Wang, Y. Pan, Z. Xu, B. Hu, and X. Wang, “Enriching bert with knowledge graph embedding for industry classification,” in _International Conference on Neural Information Processing_. Springer, 2021, pp. 709–717.
  * [21] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz _et al._ , “Transformers: State-of-the-art natural language processing,” in _Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations_ , 2020, pp. 38–45.
  * [22] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [23] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [24] P. K. Pushp and M. M. Srivastava, “Train once, test anywhere: Zero-shot learning for text classification,” _arXiv preprint arXiv:1712.05972_ , 2017\.


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

IberLEF 2023, September 2023, Jaén, Spain

<EMAIL_ADDRESS>] [1]

<EMAIL_ADDRESS>] [1]

<EMAIL_ADDRESS>] [1]

<EMAIL_ADDRESS>] [1]

[orcid=0000-0002-7810-2360<EMAIL_ADDRESS>url=https://researchportal.uc3m.es/display/inv25506, ]

[1]Corresponding author. [1]These authors contributed equally.

# A Framework for Identifying Depression on Social Media: MentalRiskES@IberLEF
2023

Simón Sánchez Viloria  Universidad Carlos III de Madrid (UC3M), Av.
Universidad, 30 (edificio Sabatini), 28911 Leganés (Madrid), Spain  Daniel
Peix del Río  Rubén Bermúdez Cabo  Guillermo Arturo Arrojo Fuentes  Isabel
Segura-Bedmar

(2023)

###### Abstract

This paper describes our participation in the MentalRiskES task at IberLEF
2023. The task involved predicting the likelihood of an individual
experiencing depression based on their social media activity. The dataset
consisted of conversations from 175 Telegram users, each labeled according to
their evidence of suffering from the disorder. We used a combination of
traditional machine learning and deep learning techniques to solve four
predictive subtasks: binary classification, simple regression, multiclass
classification, and multi-output regression. We approached this by training a
model to solve the multi-output regression case and then transforming the
predictions to work for the other three subtasks. We compare the performance
of two modeling approaches: fine-tuning a BERT-based model directly for the
task or using its embeddings as inputs to a linear regressor, with the latter
yielding better results. The code to reproduce our results can be found at:
https://github.com/simonsanvil/EarlyDepression-MentalRiskES

###### keywords:

Mental Health, Natural Language Processing, Depression, Social Media, Machine
Learning, Deep Learning, Transformers, Sentence Embeddings

##  1 Introduction

Mental health is a growing concern in our society. According to the World
Health Organization (WHO), 1 in 4 people will be affected by mental disorders
at some point in their lives [1]. In addition, the COVID-19 pandemic has had a
negative impact on the mental health of the general population, with an
increase in the number of people suffering from mental disorders [2]. Thus, it
is becoming increasingly important to evaluate the use of new technologies to
assess the risk of mental illness and the healthcare needs of the population
[3].

At the same time, social media platforms such as Telegram have become a
popular way for people to express their feeling and emotions. Telegram is a
free, end-to-end encrypted messaging service that allows users to send and
receive messages and media files in private chats or groups that can be
focused on particular topics and allow any user to observe or actively
participate. These characteristics make Telegram a suitable source for text-
mining [4].

With this context, an interesting approach is to use Natural Language
Processing (NLP) techniques to analyze the language used by people who suffer
from mental illness and discover patterns that can be used to identify them
and provide the necessary support. The MentalRiskES task at IberLEF 2023 [5]
aims to promote the development of NLP solutions specifically for Spanish-
speaking social media. They propose three main areas of focus for early-risk
detection: eating disorders (Task 1), depression (Task 2), and non-defined
disorders (Task 3).

In this work, we present our proposed solution to Task 2 of the 2023 edition
of MentalRiskES. This task involves evaluating the likelihood of a Telegram
user experiencing depression based on their comments within mental-health-
focused groups. The task is split into four predictive subtasks (2a, 2b, 2c,
2d) according to the type of output required. Our main contributions and
findings can be then summarized threefold:

  1. 1.

We conducted experiments using various language models based on BERT [6] to
solve the task. We found that a RoBERTa model [7] that had been previously
fine-tuned on a Spanish corpus to identify suicide behavior [8] tended to
yield the most accurate results. This suggests that fine-tuning for an
intermediate task can improve results for related tasks, which is supported by
existing literature [9, 10].

  2. 2.

Our approach to solving the task consisted of training only with the labels of
the regression subtasks (2b, 2d), as we deemed them the most informative.
Additionally, we show that you can use the labels of 2d to recover the labels
of the other three subtasks. The models trained to target task 2d achieved the
best results across all subtasks, even outperforming those that targeted 2b in
the simple regression metrics.

  3. 3.

We attempted two different predictive modeling approaches to solve the task
using the language model (LM) mentioned above. The first one involved
extracting the sentence embeddings of the messages of each user and using them
as features to train and evaluate classic linear and non-linear machine-
learning regressors. In the second one, we fine-tuned the LM directly for the
subtask. The first approach proved advantageous in terms of allowing for
quicker, more comprehensive experimentation and resulted in models that
achieved the best overall performance when evaluated on the test set.

The rest of the paper is organized as follows: In the next section, we analyze
the dataset used for the task (Section 2). Then, we describe in detail our
methodology for training and evaluating the models (Section 3). Finally, we
discuss the results obtained (Section 4) and present our conclusions and
future lines of work (Section 5).

##  2 Dataset Analysis

The dataset given for the task consisted of a total of 6,248 individual
messages from 175 Telegram users, each with a variable number of messages (see
figure 1). The annotation process consisted of labeling each user based on the
evidence from their conversation history of suffering from depression. Thus, a
total of 10 annotators were used for the tasks. Each was asked to assign one
of the following four labels to each user:

  * •

suffer+in favour: Indicates evidence (from text messages) of the user
suffering from depression but is also receptive/willing to help and overcome
it.

  * •

suffer+against: Indicates evidence of the user suffering from depression but
is against receiving or providing help to overcome it.

  * •

suffer+other: Indicates evidence of the user suffering from depression, but
there’s not enough information to assign them to any further category (against
or in favour)

  * •

control: Indicates no evidence of the user suffering from depression.

Figure 1: Plot of the count of users by the number of messages in the training
set and their assigned label. x-axis: number of messages per user. y-axis:
count of users with that amount of messages.

Furthermore, these labels were represented differently to support each of the
four subtasks of MentalRiskES: simple classification (task 2a), binary
regression (task 2b), multiclass classification (task 2c), and multi-output
regression (task 2d).

In the classification tasks (2a, 2c), the label assigned to each user was the
class that obtained the majority vote from the annotators, with the labels
being "1" for the "suffer" classes and "0" for the control in the case of task
2a. For the regression tasks (2b, 2d), the values of the labels were presented
as numeric probabilities in $[0,1]$ representing the confidence of the
respective class. They were calculated by adding the number of annotators who
gave the classification and dividing by 10 (the total number of annotators).
For task 2b, this was presented as one number representing the probability of
suffering from depression, while for task 2d, each subject label was presented
with four numbers representing the probability of each class. Appendix A shows
examples of how this data was given.

The following figure displays the label distribution for each task in the
training set. We can see how over 94 ($\sim$54%) of users were classified as
having depression. Furthermore, there is an imbalance in the labels for the
classification tasks due to the "suffer" label being divided into different
categories (leading to an over-representation of the "control" label).
Additionally, the "suffer+other" category is underrepresented when compared to
the other three.

Figure 2: Distribution of labels assigned to the users for each task. In tasks
a and d: 1 = evidence of suffering from depression. Task d: The sum of the
four labels adds up to 1.

##  3 Methodology

We proceeded to evaluate different techniques to solve each of the four
subtasks. Two main predictive-modeling approaches were explored: The first one
involved fine-tuning a pre-trained language model on each subtask and the
second was about training a standard ML regressor using sentence embeddings
encoded from the user’s messages as features. The following section describes
the steps taken for each approach, first describing how the data was pre-
processed and later explaining the training and evaluation process done for
each subtask.

###  3.1 Data Processing and Augmentation

Independent of the approach taken to train the models, the data was pre-
processed and augmented in the same way. The first thing we did was group all
the messages by the user they belonged to and concatenated them into a single
string, obtaining a total of 175 messages (one per user). This was done to
obtain a single representation of each user’s conversation history (from which
the labels were assigned) to be able to use it as input for the models.

To prepare for training, the data was split into training and validation sets,
leaving a random 26 (15%) users in the latter for stratified cross-validation,
where each set receives the same proportion of samples of each class [11]. The
stratification was done using the labels of task c to ensure equal
representation of the classes in both sets.

To increase the amount of data available for training and, at the same time,
attempt to model early detection (obtaining predictions early on in the
lifetime of the message history), we augmented the training set by adding
observations that only contained half of their messages. This was done by
first sorting the messages of each user in the training set by its date and
then only taking the first half, the resulting dataset was then appended to
the original training set to obtain a new one with twice the number of
observations to be used for training.

###  3.2 Solving all substaks by solving for regression

By the discussion in section 2, it should be clear to see that not all labels
of the subtasks give the same amount of information about the condition of the
subject and the likelihood of predicting it based on the available data.
Indeed, it’s clear that the probability values of task 2b give more
information about confidence in predicting depression than the simple binary
labels of task 2a. For the same reasons, the labels of task 2d are more
informative than those of task 2c as they give the full probability
distribution across the four classes.

Furthermore, we can show that it’s possible to use the multi-output regression
labels (2d) to recover the labels of the other three subtasks. To illustrate,
the multiclass classification labels of task 2c can be recovered by selecting
the class in the distribution that has the highest probability. Moreover, we
can obtain the labels of task 2a by simply converting these classes into
binary (1 for the "suffer" classes and 0 for all others). Lastly, the labels
of task 2b can be obtained by summing the probabilities of the "suffer"
classes in the distribution. We have confirmed this by applying these
modifications to the labels of the training set for task 2d and comparing them
to the original labels of the other three tasks.

This observation led us to consider using models that solve for more than one
subtask by only training it with the labels of task 2b or 2d. This allowed us
to reduce the number of models that had to be trained and focus on solving for
a single data modality (regression on $[0,1]$).

We approached simple regression in a standard way training models, training
models to minimize the Mean Squared Error between the output values and the
real ones. Additionally, we included the post-processing step of clipping the
output predictions of models of this type to the [0,1] range to ensure that
they were valid probabilities.

Multi-output regression using standard machine learning regression, on the
other hand, wasn’t as trivial as in the simple regression case. The models we
worked on didn’t support multi-output regression out of the box. The approach
we did involve training four regressors for each model, one for each class,
and then combining the predictions. We explored two methods for this: training
independent regressors or training them in a chain as explained by figure 3.
The full details of the process are described in appendix
LABEL:appendix:multi-output-regression.

Finally, similar to the simple regression case, the predictions of the multi-
output models were post-processed by dividing each of the four values by their
sum to obtain a vector whose values add up to one. That is,
$\hat{y}_{i}=\frac{\hat{y}_{i}}{\sum_{i}\hat{y}_{i}}$ for each $i$-th class.
This was done to ensure that the predictions were valid probability
distributions over the classes.

Figure 3: Graphical representation of the two methods used to implement multi-
output regression taken from [12]. In (a), the regressors are trained
independently with the same input, while in (b) they are trained in a chain
with the predictions of the previous ones being passed as features to the
next.

###  3.3 Modeling Approaches

####  3.3.1 Training a regressor with sentence embeddings

A sentence embedding is a semantically meaningful real-valued vector
representation of a sentence, obtained from the outputs of the hidden layers
of a language model. The properties of this representation are so that
sentences that express similar meanings are mapped (encoded) closer to each
other in the vector space [13].

In this way, the process of encoding text as numeric vectors can be used
directly to extract features for a classifier or regressor, which will try to
learn from the semantic information of these encodings to predict the label of
their corresponding messages. Note, however, that this approach requires the
need to have a pre-trained model to perform this encoding. Furthermore, it
assumes that the model will be good enough at capturing the semantic
information of the texts given as input, enough for the classifier/regressor
to learn from it.

Assuming that this is the case, this approach has the advantage that it is
much faster to train these kinds of regressors with regular CPUs, with the
most time-consuming part being obtaining the embeddings of the
training/evaluation messages, which only has to be done once. However, it is
necessary to evaluate different encoding models and different
classifiers/regressors (prediction models) to find the best combination for
the task at hand.

As such, we conducted experiments using different language models to find the
best encoding model. Particularly, we tested three different versions of BERT
[6] trained with different corpora in Spanish. These versions are described in
table 1. Additionally, we experimented with over 10 different regressors,
including Least Squares Linear regression [14], Random Forest [15], and
Gradient Boosting [16], among others. These models were chosen due to their
ease of implementation and the fact that they are commonly used in the
literature [17].

The process of training and evaluating these models proceeded then as follows:
First, the training set was encoded using the language model and the resulting
embeddings were used as features for a regressor. The regressor was then
trained using the labels of task 2d (the most informative ones) and the
resulting model was used to predict the labels of the validation set. The
predictions were then evaluated with the root mean squared error (RMSE). This
process was repeated for each combination of language model and regressor.

Appendix LABEL:appendix:embeddings contains the results of this experiment.
Based on that, roberta-suicide-es was deemed to be the best model for encoding
the texts. Additionally, appendix LABEL:appendix:embedding-regressor-eval
shows a detailed report of the evaluation of the best regression model with
these embeddings.

Model | Description
---|---
RoBERTa-base-bne [7] | RoBERTa model [18] trained with data from Spain’s National Library.
RoBERTa-suicide-es [8] | RoBERTa-base-bne fine-tuned for suicide detection.
BETO [19] | Variant of BERT [6] trained with Spanish corpora.
Table 1: Pre-trained BERT-based models used in our experiments.

####  3.3.2 Fine-tuning a Language Model for Regression

Apart from the approach mentioned above, we also experimented with the pure
Deep Learning (DL) approach of taking a language model and fine-tuning it with
the labels of the corresponding subtask. The model we fine-tuned was a version
of RoBERTa pre-trained for detecting suicidal behavior from texts in Spanish
[8]. We chose this model due to the fact of having been trained previously for
a task that shares similar characteristics to ours. Intermediate fine-tuning
has been proven to improve the results of downstream tasks by prior literature
[9, 10].

The HuggingFace Transformers [20] and Pytorch [21] libraries in Python were
utilized for loading the model weights and implementing the training loop. We
changed the head of the pre-trained model to a linear layer consisting of
output dimension 1 for simple regression or dimension 4 for multi-output
regression. The models were trained using an NVIDIA T4 GPU for a total of 30
epochs, where the weights of the pre-trained model remained fully frozen for
the first half and then were progressively unfrozen each epoch after that as
in [22].

Hyperparameters | Value
---|---
Optimizer | AdamW
Learning rate | $1e^{-5}$
Max Tokens | 1024
Num Epochs | 30
Batch Size | 1
Table 2: Hyperparameters for fine-tuning a RoBERTa model for regression tasks.

We used an Adam Optimizer with Mean-Squared Error (MSE) for the simple
regression models and a Cross-Entropy loss function for multi-regression
(since the labels consisted of numeric probabilities). Furthermore, since the
output for task 2d consisted of a probability distribution over the four
classes, we experimented with a custom loss function that adds a term to the
standard cross-entropy loss to penalize outputs whose sum is different from
one. However, this did not improve the results empirically as compared with
simply normalizing the outputs of the predictions after inference. The formula
of this loss is shown in equation 1. Other hyperparameters are shown in table
2.

$\mathcal{L}_{\text{custom}}=\mathcal{L}_{cross-
entropy}+\epsilon(1-\sum_{i\in[1,4]}\hat{y}_{i})^{2}$ (1)

In the equation above, $\hat{y}$ is the output of the model, $y$ is the target
label, $\epsilon$ is a hyperparameter that controls the weight of the penalty
term, and $y_{i}$ is the $i$-th element of the target label.

##  4 Results

Using the approaches mentioned in the prior section, we came up with different
models to solve the four subtasks of Task 2 of MentalRiskES. The results in
this section are obtained from selecting the best-performing models after
evaluating the different approaches and hyperparameters on the validation set.
The final predictions were obtained from a test set of messages from 149
subjects never observed during the training process and evaluated against the
task’s true labels.

In the tables below, we report the relevant metrics obtained for each subtask
and compare them against the ones obtained from baseline models provided by
the organizers of the competition. In particular, we report both absolute
metrics, obtained after observing all the messages of each subject, and early
detection metrics, obtained after incrementally observing the messages across
several rounds. Additionally, table 11 displays the inference-time
$\text{CO}_{2}$ emissions and energy consumption of each model, based on
computing their absolute predictions on the test set. These values were
estimated using the codecarbon python library [23].

For the absolute metrics, we show the accuracy, precision, recall, and F1
scores for the classification tasks (2a and 2c) and the root mean squared
error (RMSE) and coefficient of determination ($R^{2}$) for the regression
tasks (2b and 2d). The early detection metrics include the early-risk
detection metric (erde) computed after observing different rounds of messages
as well as other metrics (more details are provided in the competition
guidelines [5]).

The metrics are shown along with the name of the model used to obtain them.
The models are named as follows: [task name]_[model name]_[approach]. For
example, task2b_roberta-suicide-es_fine-tuning refers to the model trained
with the task 2b (binary classification) labels by fine-tuning the Roberta
model pre-trained for suicide detection. The "approach" can be either
embeddings or fine-tuning for the two approaches described in section 3.

Furthermore, all ML regressors trained with embeddings as features were Ridge
regressors, and all embeddings were obtained using roberta-suicide-es
encodings as this combination yielded the best results in the evaluation set.
The embeddings approaches for task 2d also include the multi-regression method
used (ind indicating that independent regressors were used and chain for
chained regressors).

###  4.1 Results for task 2a: binary classification

Table 3: Task A absolute Metric Results
Ranked by Macro F1. | accuracy | macro_precision | macro_recall | macro_f1
---|---|---|---|---
2d_roberta_embeddings_ind | 0.705 | 0.717 | 0.727 | 0.703
BaseLine - Roberta Large | 0.698 | 0.759 | 0.718 | 0.690
2d_roberta_embeddings_chain | 0.691 | 0.711 | 0.755 | 0.682
2b_roberta_embeddings | 0.691 | 0.713 | 0.764 | 0.681
2d_roberta-suicide-es_fine-tuning | 0.671 | 0.695 | 0.764 | 0.655
BaseLine - Deberta | 0.664 | 0.788 | 0.691 | 0.642
2b_roberta-suicide-es_fine-tuning | 0.638 | 0.663 | 0.735 | 0.616
BaseLine - Roberta Base | 0.631 | 0.744 | 0.658 | 0.605
Table 4: Task A early-detection Metric Results
Ranked by ERDE30. | erde30 | erde5 | latency_tp | latency_weighted_f1 | speed
---|---|---|---|---|---
2d_roberta-suicide-es_fine-tuning | 0.013 | 0.284 | 3.000 | 0.716 | 0.982
2b_roberta_embeddings | 0.020 | 0.286 | 3.000 | 0.725 | 0.982
2b_roberta-suicide-es_fine-tuning | 0.020 | 0.208 | 2.000 | 0.700 | 0.991
2d_roberta_embeddings_chain | 0.027 | 0.283 | 3.000 | 0.722 | 0.982
2d_roberta_embeddings_ind | 0.067 | 0.296 | 3.000 | 0.712 | 0.982
BaseLine - Deberta | 0.153 | 0.303 | 2.000 | 0.719 | 0.984
BaseLine - Roberta Large | 0.159 | 0.290 | 4.000 | 0.704 | 0.951
BaseLine - Roberta Base | 0.176 | 0.342 | 4.000 | 0.671 | 0.951

###  4.2 Results for task 2b: Simple Regression

Table 5: Task B absolute Metric Results
Ranked by RMSE. | RMSE | r2
---|---|---
2d_roberta_embeddings_chain | 0.241 | 0.591
2b_roberta_embeddings | 0.244 | 0.581
2d_roberta_embeddings_ind | 0.259 | 0.526
BaseLine - Roberta Base | 0.277 | 0.770
2d_roberta-suicide-es_fine-tuning | 0.304 | 0.349
2b_roberta-suicide-es_fine-tuning | 0.311 | 0.317
BaseLine - Deberta | 0.339 | 0.683
BaseLine - Roberta Large | 0.390 | 0.503
Table 6: Task B Ranking-based Results: Ranked by the p@30 metric | p@10 | p@20 | p@30 | p@5
---|---|---|---|---
BaseLine - Roberta Base | 0.800 | 0.700 | 0.567 | 0.600
BaseLine - Deberta | 0.600 | 0.550 | 0.567 | 0.800
BaseLine - Roberta Large | 0.500 | 0.550 | 0.567 | 0.400
2b_roberta-suicide-es_fine-tuning | 0.700 | 0.700 | 0.533 | 1.000
2b_roberta_embeddings | 0.800 | 0.450 | 0.367 | 0.800
2d_roberta_embeddings_ind | 0.700 | 0.350 | 0.233 | 0.600
2d_roberta_embeddings_chain | 0.200 | 0.150 | 0.133 | 0.000
2d_roberta-suicide-es_fine-tuning | 0.200 | 0.100 | 0.100 | 0.200

###  4.3 Results for task 2c: Multiclass Classification

Table 7: Task C absolute Metric Results
Ranked by Macro F1. | accuracy | macro_precision | macro_recall | macro_f1
---|---|---|---|---
2d_roberta-suicide-es_fine-tuning | 0.517 | 0.446 | 0.435 | 0.395
2d_roberta_embeddings_ind | 0.557 | 0.429 | 0.395 | 0.394
2d_roberta_embeddings_chain | 0.530 | 0.437 | 0.418 | 0.392
BaseLine - Roberta Large | 0.483 | 0.389 | 0.378 | 0.360
BaseLine - Deberta | 0.456 | 0.395 | 0.344 | 0.293
BaseLine - Roberta Base | 0.356 | 0.380 | 0.335 | 0.274
Table 8: Task C early-detection Metric Results:
Ranked by ERDE30. | erde30* | erde5 | latency_tp | latency_weighted_f1 | speed
---|---|---|---|---|---
2d_roberta_embeddings_chain | 0.157 | 0.284 | 3.000 | 0.718 | 0.982
2d_roberta-suicide-es_fine-tuning | 0.159 | 0.285 | 3.000 | 0.712 | 0.982
2d_roberta_embeddings_ind | 0.172 | 0.297 | 3.000 | 0.708 | 0.982
BaseLine - Deberta | 0.190 | 0.330 | 2.000 | 0.695 | 0.984
BaseLine - Roberta Base | 0.206 | 0.307 | 2.000 | 0.659 | 0.984
BaseLine - Roberta Large | 0.232 | 0.283 | 2.000 | 0.652 | 0.984

###  4.4 Results for task 2d: Multi-output Regression.

Table 9: Task D absolute Metric Results.
Ranked by mean RMSE. Labels are shortened as: sf = suffer+in favour, sa =
suffer+against, so = suffer+other, c =control

| rmse mean* | rmse sf | rmse sa | rmse so | rmse c | r2 mean | r2 sf | r2 sa | r2 so | r2 c
---|---|---|---|---|---|---|---|---|---|---
2d_roberta_embeddings_chain | 0.180 | 0.179 | 0.191 | 0.111 | 0.241 | 0.355 | 0.544 | 0.217 | 0.069 | 0.590
2d_roberta_embeddings_ind | 0.187 | 0.181 | 0.192 | 0.114 | 0.259 | 0.320 | 0.532 | 0.208 | 0.012 | 0.526
2d_roberta-suicide-es_fine-tuning | 0.222 | 0.212 | 0.230 | 0.143 | 0.304 | 0.006 | 0.358 | -0.144 | -0.538 | 0.349
BaseLine - Deberta | 0.232 | 0.246 | 0.250 | 0.125 | 0.306 | 0.484 | 0.661 | 0.295 | 0.260 | 0.721
BaseLine - Roberta Base | 0.410 | 0.547 | 0.272 | 0.235 | 0.585 | -0.145 | -0.496 | 0.355 | 0.185 | -0.624
BaseLine - Roberta Large | 0.437 | 0.682 | 0.312 | 0.158 | 0.598 | -0.209 | -0.678 | 0.890 | 0.059 | -0.306

Table 10: Task D ranking Metric Results: Ranked by p@30 | p@10 | p@20 | p@30 | p@5 | p@50
---|---|---|---|---|---
BaseLine - Deberta | 0.300 | 0.338 | 0.350 | 0.250 | 0.250
2d_roberta_embeddings_ind | 0.300 | 0.300 | 0.292 | 0.600 | 0.280
BaseLine - Roberta Large | 0.275 | 0.263 | 0.275 | 0.350 | 0.350
2d_roberta_embeddings_chain | 0.275 | 0.275 | 0.250 | 0.550 | 0.240
BaseLine - Roberta Base | 0.300 | 0.225 | 0.192 | 0.250 | 0.250
2d_roberta-suicide-es_fine-tuning | 0.075 | 0.113 | 0.167 | 0.150 | 0.145

###  4.5 Carbon Emissions

Table 11: Estimated $\text{CO}_{2}$ emissions of each model from predicting
all messages on the test set (absolute).
Estimations were obtained with the codecarbon python library [23] using a
Macbook Pro (2021) w/ M1 Pro and 16GB of RAM for inference. The models with
the lowest emissions are highlighted in bold.

| duration (secs) | emissions (kgCO2eq) | cpu_energy | ram_energy
---|---|---|---|---
model |  |  |  |
2b_roberta-suicide-es_fine-tuning | 5.74 | 1.56e-06 | 7.97e-06 | 2.53e-07
2d_roberta-suicide-es_fine-tuning | 7.393 | 2.00e-06 | 1.03e-05 | 2.58e-07
2b_roberta_embeddings | 23.287 | 6.70e-06 | 3.23e-05 | 2.94e-06
2d_roberta_embeddings_ind | 23.976 | 6.94e-06 | 3.33e-05 | 3.25e-06
2d_roberta_embeddings_chain | 23.721 | 7.14e-06 | 3.29e-05 | 4.63e-06

##  5 Conclusions

The results show that the approaches considered in this work were successful
at modeling each of the predictive subtasks, with at least one of our models
outperforming the baselines in most cases. We can make the following
observations:

  * •

The best-performing approach across all tasks seems to be the one that uses
the embeddings of the messages as input to a multi-output regression model
(task 2d). At least one model trained with this approach reached the top
ranking for tasks 2a, 2b, and 2d absolute ranking metrics and outperformed the
baseline absolute metrics across all tasks.

  * •

Most notably, the regression method that uses multi-output chained regressors
obtained the best metrics for task 2d across all models, outperforming the
fine-tuning approach by over 20% in the absolute metrics and reaching the
second highest spot in the early-risk metrics for this task.

  * •

Models trained for multi-output regression perform very well for binary
classification and simple regression tasks, even outperforming the models
trained for simple regression targets in their own subtask. This suggests that
using one model to solve for multiple targets was indeed a good approach to
this problem.

  * •

The models obtained with a pure DL approach from fine-tuning a RoBERTa model
are estimated to produce over 3-4x less emissions at inference time than the
hybrid approach from training linear regressors on sentence embeddings. This
gap is likely because the fine-tuning approach requires less computation at
inference time than the hybrid approach, which requires the computation of the
sentence embeddings before feeding them to multiple regressors, while the
fine-tuning approach is made in one forward pass.

Another finding we can conclude from these insights is that while our models
achieve great results in the absolute ranking metrics, they do not perform as
well for the metrics that assess early-risk performance. In our work, we did
not model explicitly for an early detection scenario; we only added
information about prior messages through data augmentation. This limitation
means our models may not perform as well in real-world situations where we aim
to detect signs of depression in a conversation early on.

Thus, it may be important to explore different training approaches to improve
the performance of early-risk detection. This might include directly employing
online learning to predict and update the model as new messages come in or
incorporating an ensemble of models to make independent decisions about a
message’s risk level and combining them for a final decision (as seen in
[23]). Additionally, we may also look into more efficient implementations of
the hybrid approach to minimize the disparity in emissions compared to pure DL
models. These improvements are crucial when considering the deployment of our
models in real-world situations and will be the focus of future work.

## References

  * World Health Organization [2001] World Health Organization, The World Health Report 2001: Mental Disorders affect one in four people, 2001\. URL: https://www.who.int/news/item/28-09-2001-the-world-health-report-2001-mental-disorders-affect-one-in-four-people.
  * Xiong et al. [2020] J. Xiong, O. Lipsitz, F. Nasri, L. M. W. Lui, H. Gill, L. Phan, D. Chen-Li, M. Iacobucci, R. Ho, A. Majeed, R. S. McIntyre,  Impact of COVID-19 pandemic on mental health in the general population: A systematic review,  Journal of Affective Disorders 277 (2020) 55–64. URL: https://www.sciencedirect.com/science/article/pii/S0165032720325891. doi:10.1016/j.jad.2020.08.001.
  * Losada et al. [2017] D. E. Losada, F. Crestani, J. Parapar,  erisk 2017: Clef lab on early risk prediction on the internet: experimental foundations,  in: Experimental IR Meets Multilinguality, Multimodality, and Interaction: 8th International Conference of the CLEF Association, CLEF 2017, Dublin, Ireland, September 11–14, 2017, Proceedings 8, Springer, 2017, pp. 346–360.
  * Dargahi Nobari et al. [2017] A. Dargahi Nobari, N. Reshadatmand, M. Neshati,  Analysis of Telegram, An Instant Messaging Service,  in: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM ’17, Association for Computing Machinery, New York, NY, USA, 2017, pp. 2035–2038. URL: https://dl.acm.org/doi/10.1145/3132847.3133132. doi:10.1145/3132847.3133132.
  * Mármol-Romero et al. [2023] A. M. Mármol-Romero, A. Moreno-Muñoz, F. M. Plaza-del Arco, M. T. Martín-Valdivia, L. A. Ureña-López, A. Montejo-Ráez,  Overview of MentalriskES at IberLEF 2023: Early Detection of Mental Disorders Risk in Spanish,  Procesamiento del Lenguaje Natural 71 (2023).
  * Devlin et al. [2019] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2019. URL: http://arxiv.org/abs/1810.04805. doi:10.48550/arXiv.1810.04805, arXiv:1810.04805 [cs].
  * Fandiño et al. [2022] A. G. Fandiño, J. A. Estapé, M. Pàmies, J. L. Palao, J. S. Ocampo, C. P. Carrino, C. A. Oller, C. R. Penagos, A. G. Agirre, M. Villegas,  Maria: Spanish language models,  Procesamiento del Lenguaje Natural 68 (2022). URL: https://upcommons.upc.edu/handle/2117/367156#.YyMTB4X9A-0.mendeley. doi:10.26342/2022-68-3.
  * Padial and Gómez [2023] D. L. Padial, D. Gómez, hackathon-somos-nlp-2023 - roberta-base-bne-finetuned-suicide-es· Hugging Face, 2023\. URL: https://huggingface.co/hackathon-somos-nlp-2023/roberta-base-bne-finetuned-suicide-es.
  * Phang et al. [2019] J. Phang, T. Févry, S. R. Bowman, Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks, 2019. URL: http://arxiv.org/abs/1811.01088. doi:10.48550/arXiv.1811.01088, arXiv:1811.01088 [cs].
  * Chang and Lu [2021] T.-Y. Chang, C.-J. Lu, Rethinking Why Intermediate-Task Fine-Tuning Works, 2021. URL: http://arxiv.org/abs/2108.11696, arXiv:2108.11696 [cs].
  * Kohavi [1995] R. Kohavi,  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection,  in: IJCAI’95: Proceedings of the 14th international joint conference on Artificial intelligence, 1995, pp. 1137–1143. URL: https://www.semanticscholar.org/paper/A-Study-of-Cross-Validation-and-Bootstrap-for-and-Kohavi/8c70a0a39a686bf80b76cb1b77f9eef156f6432d.
  * Antonenko and Read [2022] E. Antonenko, J. Read,  Multi-modal Ensembles of Regressor Chains for Multi-output Prediction,  in: T. Bouadi, E. Fromont, E. Hüllermeier (Eds.), Advances in Intelligent Data Analysis XX, Lecture Notes in Computer Science, Springer International Publishing, Cham, 2022, pp. 1–13. doi:10.1007/978-3-031-01333-1_1.
  * Perone et al. [2018] C. S. Perone, R. Silveira, T. S. Paula, Evaluation of sentence embeddings in downstream and linguistic probing tasks, 2018. URL: http://arxiv.org/abs/1806.06259, arXiv:1806.06259 [cs] version: 1.
  * Hoerl and Kennard [1970] A. E. Hoerl, R. W. Kennard,  Ridge Regression: Biased Estimation for Nonorthogonal Problems,  Technometrics 12 (1970) 55–67. URL: https://www.jstor.org/stable/1267351. doi:10.2307/1267351, publisher: [Taylor & Francis, Ltd., American Statistical Association, American Society for Quality].
  * Breiman [2001] L. Breiman,  Random Forests,  Machine Learning 45 (2001) 5–32. URL: https://doi.org/10.1023/A:1010933404324. doi:10.1023/A:1010933404324.
  * Friedman [2000] J. Friedman,  Greedy Function Approximation: A Gradient Boosting Machine,  The Annals of Statistics 29 (2000). doi:10.1214/aos/1013203451.
  * Pedregosa et al. [2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay,  Scikit-learn: Machine learning in Python,  Journal of Machine Learning Research 12 (2011) 2825–2830.
  * Liu et al. [2019] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, RoBERTa: A Robustly Optimized BERT Pretraining Approach, 2019. URL: http://arxiv.org/abs/1907.11692. doi:10.48550/arXiv.1907.11692, arXiv:1907.11692 [cs].
  * Cañete et al. [2020] J. Cañete, G. Chaperon, R. Fuentes, J.-H. Ho, H. Kang, J. Pérez,  Spanish pre-trained bert model and evaluation data,  in: PML4DC at ICLR 2020, 2020, pp. 1–10.
  * Wolf et al. [2020] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, A. Rush,  Transformers: State-of-the-Art Natural Language Processing,  in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Online, 2020, pp. 38–45. URL: https://aclanthology.org/2020.emnlp-demos.6. doi:10.18653/v1/2020.emnlp-demos.6.
  * Paszke et al. [2019] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala,  Pytorch: An imperative style, high-performance deep learning library,  in: Advances in Neural Information Processing Systems 32, Curran Associates, Inc., 2019, pp. 8024–8035. URL: http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf.
  * Liu et al. [2023] C. C. Liu, J. Pfeiffer, I. Vulić, I. Gurevych, Improving Generalization of Adapter-Based Cross-lingual Transfer with Scheduled Unfreezing, 2023\. URL: http://arxiv.org/abs/2301.05487, arXiv:2301.05487 [cs].
  * Schmidt et al. [2021] V. Schmidt, K. Goyal, A. Joshi, B. Feld, L. Conell, N. Laskaris, D. Blank, J. Wilson, S. Friedler, S. Luccioni,  Codecarbon: estimate and track carbon emissions from machine learning computing,  Cited on (2021) 20\.

##  Appendix A Dataset Examples

The data was given in JSON format after requesting the server. The following
examples are meant to show the structure of how the data was given and later
parsed.

⬇

[

{

"id_message": "1",

"message": "Me parece que es una buena idea, pero no estoy seguro",

"date": "2020-07-27 01:27:31"

},

{

"id_message": 2,

"message": "Buen dia a todos",

"date": "2020-07-27 02:03:28"

},

⬇

\end{minipage}

\end{center}

To complement the files described in the example above, the labels of each
subject were given in CSV format, where each row corresponded to one subject.
Four of these files were given, one for each task. Table \ref{tab:labels-
example} below shows some examples of this.

% \clearpage

\begin{table}[!h]

% \hspace*{-1.75cm}

% \vspace*{0.5cm}

% \centering

\scalebox{0.85}{

\begin{tabular}{llrlrrrr}

\toprule

{} & a\_label & b\_label & c\_label & d\_suffer\_in\_favour &
d\_suffer\_against & d\_suffer\_other & d\_control \\\

subject\_id & & & & & & & \\\

\midrule

subject101 & 1 & 0.9 & suffer+in favour & 0.7 & 0.1 & 0.1 & 0.1 \\\

subject104 & 1 & 0.7 & suffer+in favour & 0.4 & 0.0 & 0.3 & 0.3 \\\

subject106 & 1 & 1.0 & suffer+in favour & 0.5 & 0.5 & 0.0 & 0.0 \\\

subject108 & 1 & 0.5 & suffer+in favour & 0.4 & 0.1 & 0.0 & 0.5 \\\

subject109 & 0 & 0.1 & control & 0.0 & 0.0 & 0.1 & 0.9 \\\

\bottomrule

\end{tabular}

}

\caption{\textbf{Example of user labels for each task.} The letter in the
prefix of the column name indicates the label of the task (See section
\ref{sec:dataset-analysis}). For example, column \textit{a\_label} indicates
the label of the users for task 2a.}

\label{tab:labels-example}

\end{table}

\section{Evaluation of Embedding Models for Regression}
\label{appendix:embeddings}

Table \ref{tab:embeddings_rmse} shows the scores after evaluating with
different encodings for task 2d. The sentence embeddings were obtained after
concatenating the messages of each user into a single string. RMSE scores were
calculated as the mean of the results of 10 regressors trained with the
respective encoding as features. The best-performing embeddings were the ones
obtained with the RoBERTa model fine-tuned for suicide detection.

\begin{table}[h]

\scalebox{0.9}{

\begin{tabular}{lcccc|r}

\toprule

{} & suffer+in favour & suffer+against & suffer+other & control & mean \\\

encoder & & & & & \\\

\midrule

roberta-base-bne-suicide-es & 0.228 & 0.198 & 0.113 & 0.249 & 0.204 \\\

roberta-base-bne & 0.220 & 0.208 & 0.119 & 0.277 & 0.214 \\\

bert-base-spanish-wwm-cased (BETO) & 0.233 & 0.214 & 0.132 & 0.273 & 0.221 \\\

% all-mpnet-base-v2 & 0.276 & 0.222 & 0.120 & 0.365 & 0.261 \\\

\bottomrule

\end{tabular}

}

\caption{

Mean RMSEs scores after training 10 regressors with embeddings from different
transformers.

}

\label{tab:embeddings_rmse}

\end{table}

The difference in performance between \texttt{roberta-base-bne-suicide-es}
encodings and the other embeddings can be justified by the fact that we are
taking advantage of the information gained from the prior fine-tuning for
suicide detection of this model, which likely shares semantic similarities
with our data.

The implementation of the sentence embeddings was done using the HuggingFace
library. Before obtaining the encodings, the models were loaded from the
\href{https://huggingface.co/models}{HuggingFace Hub}. The reference and links
to the models in Hugging Face is included in \ref{tab:bert_models}.

% \clearpage

\section{Evaluation of Regression models trained with the Sentence Embeddings
Approach} \label{appendix:embedding-regressor-eval}

% \clearpage

% \subsection{Embeddings:}

Tables \ref{tab:eval_results_2b_embeddings} and \ref{tab:results_2d_chainpca}
report the regression metrics obtained on the validation set for various
estimators trained with the \texttt{roberta-base-bne-suicide-es} embeddings.
The results were obtained after evaluating over 10 estimators for the task,
selecting the 4-6 best-performing regressors, and doing grid-search
hyperparameter-tuning on them.

By these results, the Ridge Regression model (ridge), which implements Least
Squares Linear Regression with L2 regularization, seems to be the best
estimator for both tasks. The best-performing models of each task were then
trained with the entire data (train+validation sets) and the prediction of the
test set was obtained with them. These evaluations with these predictions are
the ones reported in section \ref{sec:results}.

\begin{table}[h]

\centering

\vspace*{-0.2cm}

\caption{Binary regression results in the evaluation set of various estimators
trained with embeddings for task 2b.

}

\label{tab:eval_results_2b_embeddings}

\begin{tabular}{lr|r}

\toprule

{} & rmse & r2 \\\

estimator & & \\\

\midrule

ridge & 0.254 & 0.489 \\\

ada & 0.257 & 0.480 \\\

lgbm & 0.259 & 0.471 \\\

svr & 0.260 & 0.467 \\\

rf & 0.264 & 0.448 \\\

mlp & 0.319 & 0.198 \\\

\bottomrule

\end{tabular}

\end{table}

\begin{table}[h]

\centering

\caption{Multi-output regression results in the evaluation set of various
estimators trained with embeddings for task 2d. Labels are shortened as:

sf = suffer+in favour, sa = suffer+against, so = suffer+other, c = control}

\label{tab:results_2d_chainpca}

% \hspace*{-0.1cm}

\scalebox{0.85}{

\begin{tabular}{lr|rrrrrrrrr}

\toprule

{} & RMSE mean & RMSE sf & RMSE sa & RMSE so & RMSE c & R2 mean & R2 sf & R2
sa & R2 so & R2 c \\\

estimator & & & & & & & & & & \\\

\midrule

ridge & 0.183 & 0.172 & 0.193 & 0.108 & 0.260 & 0.271 & 0.410 & 0.287 & -0.076
& 0.465 \\\

lr & 0.193 & 0.202 & 0.197 & 0.108 & 0.267 & 0.201 & 0.183 & 0.256 & -0.073 &
0.439 \\\

rf & 0.206 & 0.195 & 0.220 & 0.106 & 0.305 & 0.133 & 0.240 & 0.071 & -0.045 &
0.267 \\\

lgbm & 0.288 & 0.285 & 0.262 & 0.181 & 0.424 & -0.854 & -0.631 & -0.321 &
-2.043 & -0.421 \\\

\bottomrule

\end{tabular}

}

\end{table}

The estimators mentioned in the table above are implementations of common
regressors from Python’s Scikit-Learn library \cite{scikit-learn}. These
include: Ordinary ("lr") and Ridge ("ridge") Least Squares Regression, Ada-
Boost regression ("ada"), Light Gradient Boosting Machine ("lgbm"), Support
Vector Regression ("svr"), Random Forests ("rf"), and a Multi-Layer Perceptron
("mlp"). References of the implementations of these models can be found in the
\href{https://scikit-learn.org/stable/supervised_learning.html}{Scikit-Learn
documentation}.

\section{Multi-Output Regression with Independent Regressors and Regressor
Chains} \label{appendix:multi-output-regression}

For task 2d, we were required to obtain four values corresponding to a
probability distribution over the four classes (suffer+in favour,
suffer+against, suffer+other, control). In section \ref{sec:solving-for}, we
explained how this multi-output regression problem can be solved for the
sentence embeddings approach by training four regressors and then combining
their predictions using either the Independent Regressors or Regressor Chain
methods.

Here we explain how these methods work and were implemented. First of all, the
two methods can be summarized as follows, depending on how the regressors are
trained to obtain the probability distributions. Figure \ref{fig:multi-
regression-methods} shows a graphical representation of the two methods.

\begin{enumerate}

\item \textbf{Independent Regressors}: Each regressor is trained independently
with the labels of its corresponding class (e.g., the first regressor was
trained with the labels of the suffer+in favour class, the second with the
labels of the suffer+against class, and so on). The downside of this method is
that it doesn’t take into account the information of the other classes when
training each regressor, which is important as we know the labels are not
independent of each other (they must all sum to 1).

\item \textbf{Regressor Chain Method}: The regressors are trained in a chain,
where the first regressor is trained to predict the first class, and its
predictions are included in the features for the second regressor, and so on.
This method is useful when the labels of each class are not independent of
each other (like in our case), as it allows the regressors to learn from the
predictions of the previous ones. Since the order of the classes matters in
this method, we decided to put them in the order of most to least amount of
users annotated with that class: control, suffer+in favour, suffer+against,
suffer+other (see section \ref{sec:dataset-analysis}).

\end{enumerate}

To the second method, we can additionally add the option of applying Principal
Component Analysis (PCA) to reduce the dimensionality of the input embeddings
before training the models in the chain. Because the embeddings might have a
large dimensionality, this is done to make these models more likely to use the
information of the previous predictions.

Both methods were implemented with the Scikit-Learn library \cite{scikit-
learn} using the \texttt{MultiOutputRegressor} and \texttt{RegressorChain}
classes. The number of components to keep for PCA was chosen using based on
the percent of variance explained. The number of components was fine-tuned and
the best results was obtained with 40 components (85\% of variance).

% \begin{figure}[h]

% \centering

% \includegraphics[width=0.7\textwidth]{figures/multi-regression-methods.png}

% \caption{

% \mbox{

% Graphical representation of the two methods used to implement multi-output
regression. Figure taken from \cite{antonenko_multi-modal_2022}.

% % In the first one, the regressors are trained independently with the same
input, while in the second they are trained in a chain with the predictions of
the previous ones being passed as features to the next.

% }}

% \label{fig:multi-regression-methods2}

% \end{figure}

\end{document}

%%

%% End of file