Predicting ESG Scores from News Articles using Natural Language Processing

MASTER THESIS

Submitted in Partial Fulfillment of the Requirements for the Degree of

MASTER OF SCIENCE

in Information Systems

Prof. Adam Jatowt

Department of Computer Science

University of Innsbruck Digital Science Center

Submitted by:

Tanja Gaby Aue

Innsbruck, April 2022

###### Contents

  1. 1 Introduction
  2. 2 Overview of the ESG Topic
    1. 2.1 Definition and Underlying Concept
    2. 2.2 Regulatory Environment
    3. 2.3 ESG Rating Landscape
    4. 2.4 Relevance of ESG topics
  3. 3 Related Literature
    1. 3.1 Overview of ESG related Literature
    2. 3.2 ESG related Text Classification
    3. 3.3 Natural Language Processing
  4. 4 The Model
    1. 4.1 Data
      1. 4.1.1 Dataset Creation
      2. 4.1.2 Dataset Preparation
      3. 4.1.3 Dataset Description
    2. 4.2 Methodology
      1. 4.2.1 Derive Model Inputs
      2. 4.2.2 Rating Prediction
  5. 5 Results
    1. 5.1 Descriptive Analysis of Input Data
    2. 5.2 Evaluation of applied NLP Methods
      1. 5.2.1 ESG News classification
      2. 5.2.2 Sentiment Analysis
      3. 5.2.3 Cluster Analysis
    3. 5.3 Evaluation of Predicted ESG Rating
      1. 5.3.1 Comparison of Proposed Models
      2. 5.3.2 Evaluation by Market Capitalization
      3. 5.3.3 Input Ablation Study
    4. 5.4 Limitations
  6. 6 Contribution to existing Literature
  7. 7 Conclusion

###### List of Tables

  1. 1 Selected ESG issues by WhoCaresWins2004
  2. 2 Overview News Article Dataset
  3. 3 Overview of Asset4 ESG Scores
  4. 4 Selected Training Parameters for the Classification Model
  5. 5 CNN Model Parameters
  6. 6 Transformer Model Parameters
  7. 7 General Model Parameters
  8. 8 Overview of the Dataset per Market Capitalization
  9. 9 Classification Report
  10. 10 Rel-Noise Ratios grouped by Market Capitalization
  11. 11 Sum of Squared Errors (SSE) per clustering method
  12. 12 Model Results for the year 2020
  13. 13 Model Results for the year 2018
  14. 14 Model Results for the year 2019
  15. 15 Regression Model Results by Market Capitalization (2019)
  16. 16 Results Input Ablation Study
  17. 17 Overview of assigned Sentiment Labels
  18. 18 Model Results using basic k-means clustering (for the year 2019)

###### List of Figures

  1. 1 Regulatory Timeline
  2. 2 Overview ESG Prediction Model
  3. 3 Overview Layers Classification Model
  4. 4 Elbow Curves
  5. (a) Standard K-means
  6. (b) Spherical K-means
  7. 5 Basic CNN Model
  8. 6 Deep CNN Model
  9. 7 CNN including single Transformer Layer
  10. 8 Overview of Asset4 ESG Scores per Market Capitalization
  11. 9 DistilBERT Training and Validation
  12. (a) First Training Stage
  13. (b) Second Training Stage
  14. 10 Percentage of relevant articles per company
  15. (a) 2018
  16. (b) 2019
  17. (c) 2020
  18. 11 Overview of rel-noise ratios by month
  19. 12 Overview of pos-neg ratios by month
  20. 13 Overview of cluster size: Basic K-means
  21. 14 Distribution of Mean Average Difference  over 10 different train-test splits (2019)
  22. (a) Classification Models
  23. (b) Regression Models
  24. 15 Comparison of average ratings by Market Capitalization
  25. 16 Mean Difference by Market Capitalization (2019)

Predicting ESG Scores from News Articles using Natural Language Processing
Tanja Gaby Aue

ABSTRACT

Environmental, social and governance (ESG) engagement of companies moved into
the focus of public attention over recent years. With compulsory reporting
requirements being implemented and investors incorporating sustainability in
their investment decisions, the demand for a transparent and reliable ESG
rating is increasing. Therefore this paper analyzes news articles to derive
information about companies’ ESG activities. I build a model to test whether
representative ESG ratings can be predicted from news articles using state-of-
the-art natural language processing (NLP) techniques. A large news dataset for
about 3,000 US companies is created using the GDELT database. Asset4 ESG
ratings from Refinitiv provide a basis for comparison. Hereby, I find that
representative ESG ratings can be predicted from news articles. Comparing the
resulting ratings to the derived Asset4 ratings reveals interesting insights.

##  1 Introduction

Companies Environmental, Social and Governance (ESG) practices are
increasingly coming to the fore of the public, institutional investors and
politics. Investments in ESG compliant or sustainable assets gain in
popularity. For example in the US, assets invested in sustainable funds amount
to more than $330 billion as of September 2021 (cnnEvolutionESGinvest).
Another example are the assets under management by companies, which have
joined and signed the United Nation’s Principles for Responsible Investing
(PRI), these amount to 121 trillion US Dollars as of November 2021 (PRI).

The concept of ESG was first defined in the Global Compact Report of the
United Nations in 2004. However, its origin can be traced back to the 1970s,
when issues related to sustainability were raised more and more by Non-
Governmental Organizations (NGOs) (eccles2018exploring). In the last years,
various frameworks and initiatives defined standards for ESG compliance and
specified which information should be provided by companies. Widely used
frameworks include for example the Global Reporting Initiative (GRI), the
Sustainability Accounting Standards Board (SASB) and the Workforce Disclosure
Initiative (WDI) (forbesDemistifying). Recently, the US Securities and
Exchange Commission (SEC) announced to develop a framework for ESG disclosure
on their own. The goal is to provide a comprehensive framework which provides
the basis for comparable information (SecRegESG).

In line with the wide range of frameworks available, various agencies (e.g.
MSCI, Sustainalytics) provide company ESG ratings and include different
subcategories and measures in their analysis. As a result these ratings differ
significantly from provider to provider, emphasizing the complexity of the
assessment of environmental, social and governance topics. Due to the
divergence in ratings, stakeholders are faced with the challenge of choosing
the rating reflecting ESG compliance the best according to their view.

Identified Research Gap

Despite the different papers published in the last years related to CSR
(report) and ESG analysis, the assessment of firms’ ESG performance still
seems to be challenging. sokolov2021building and borms2021semi already
addressed the topic of ESG rating construction. However, sokolov2021building
focuse more on the classification of the content with respect to ESG
relevance. They propose approaches to construct indices based on their results
but do not apply these. While borms2021semi predict ESG scores using news
data, they do not leverage machine learning algorithms to do so but rather use
simpler aggregation functions. Additionally, the dataset they use is rather
small and their method requires human judgment. Therefore the goal of the
proposed thesis is to construct ESG ratings, which reflect the most recent
trends in ESG assessment by applying state-of-the art NLP techniques to a
large dataset of news articles. The criteria used will be outlined in detail
in order to provide full transparency. Creating these ratings without
including human judgment but fully automatically by leveraging NLP algorithms,
ensures that the predicted ratings are transparent and can be potentially
reconstructed by any stakeholder.

Overview of the Proposed Thesis

In the proposed study, an approach is tested, which generates ESG ratings for
companies exploiting recent advances in Natural Language Processing (NLP).
Predicting ESG ratings from news automatically, without human intervention,
could enable various stakeholders, like private investors and government
agencies, to monitor the ESG compliance of companies in a time and resource
efficient manner. More specific, the goal is to predict comprehensive ESG
ratings for companies from a large news dataset applying different state-of-
the-art NLP techniques and algorithms. Therefore ESG ratings for about 3,000
US American companies are derived from a third-party provider and a
corresponding news dataset is generated using the Global Database of Events,
Language, and Tone (GDELT). In the first step, articles are classified with
respect to their ESG relevance. The second analysis step focuses on
determining the articles sentiment and the third step clusters the articles
with respect to their semantics. In the fourth step, different models to
predict ESG ratings therefrom are adopted. In the last step the results are
evaluated and compared to the third-party ESG ratings.

The paper is organized as follows. The next section provides and overview of
the ESG topic and outlines its relevance as well as the regulatory
environment. The third section provides an overview of existing literature in
the area of ESG rating prediction and Natural Language Processing. The model
applied is described in detail in the fourth section. In the fifth section the
results are presented as well as the limitations of the model. The sixth
section discusses the results with respect to existing empirical findings and
the last section concludes.

##  2 Overview of the ESG Topic

###  2.1 Definition and Underlying Concept

In order to understand the purpose of company ESG ratings, it is necessary to
define what the term ESG encompasses and to differentiate it from related
terms. However, as the term captures a broad range of topics related to
Environmental, Social and Governance actions, definitions diverge depending on
the viewpoint and are mostly very general and wide-reaching.

The focus on environmental and social responsibility of companies and related
topics emerged in the 1970s, when NGOs increasingly made related issues public
(eccles2018exploring). Corporate Social Responsibility (CSR) is referred to as
companies engagement with societal and environmental topics and was the first
widely used term for the topic. What differentiates it from the term ESG is
that it does not explicitly include governance whereas ESG does. However, ESG
and CSR are often used interchangeable because indirectly governance related
topics are covered by CSR too (gillan2021firms).

ESG, an abbreviation for Environmental, Social and Governance, was first
mentioned as such in the ”Who Cares Wins” report published by the United
Nations (UN) and the Swiss Department for Foreign Affairs (WhoCaresWins2004).
The report with the title ”Recommendations by the financial industry to better
integrate environmental, social and governance issues in analysis, asset
management and securities brokerage” especially aims at raising awareness of
ESG related topics. It provides recommendations on how to integrate ESG issues
and potential countermeasures into business models. Their defined target group
is not limited to companies and investors but includes regulators, stock
exchanges, non-governmental organisations (NGOs) as well as analysts, advisors
and financial institutions alike. Following this report, the ”Who Cares Wins”
initiative kicked of, in order to develop some common ESG groundtruth and
identify how environmental, social and governance value created or destroy can
be measured (WhoCaresWins2006).

No specific definition of the term ESG and its three dimensions is provided
within these reports. Rather a sample of issues per dimension is listed, which
is likely to impact various companies. Table 1 shows the examples for the
three dimensions.

Table 1: Selected ESG issues by WhoCaresWins2004

Environmental Issues
---
\- Climate change and related risks
\- The need to reduce toxic releases and waste
\- New regulation expanding the boundaries of environmental liability with
regard to products and services
\- Increasing pressure by civil society to improve performance, transparency
and accountability,
leading to reputational risks if not managed properly
\- Emerging markets for environmental services and environment-friendly
products
Social Issues
\- Workplace health and safety
\- Community relations
\- Human rights issues at company and suppliers’ / contractors’ premises
\- Government and community relations in the context of operations in
developing countries
\- Increasing pressure by civil society to improve performance,
transparency and accountability, leading to reputational risks
Corporate Governance Issues
\- Board structure and accountability
\- Accounting and disclosure practices
\- Audit committee structure and independence of auditors
\- Executive compensation
\- Management of corruption and bribery issues

However, as mentioned by them, these can vary from company to company
depending on region, sector and other variants.

The following general definition of the term ESG is provided by the
International Finance Institute (IFC), which was one of several endorsing
institutions involved in the ”Who Cares Wins” initiative:

> ”A set of environmental, social and governance factors considered by
> companies when managing their operations, and investors when making
> investments, in respect of the risks, impacts and opportunities relating to
> but no limited to:
>
>   * •
>
> Environmental issues: potential or actual changes to the physical or natural
> environment (e.g. pollution, biodiversity impacts, carbon emissions, climate
> change, natural resource use)
>
>   * •
>
> Social issues: potential or actual changes on surrounding community and
> workers (e.g. health and safety, supply chain. Diversity and inclusion)
>
>   * •
>
> Governance issues: corporate governance structures and processes by which
> companies are directed and controlled (e.g. board structure and diversity,
> ethical conduct, risk management, disclosure and transparency), including
> the governance of key environmental and social policies and procedures.”
> (IFC_Guidebook)
>
>

The vague definition of the three dimensions, which captures both direct and
indirect impacts of companies operations further emphasizes the complexity of
narrowing down the topic. Actions and business practices can have far-reaching
consequences for the environment and society. Hence ESG is linked to various
sustainability and society themes, as will be analyzed further in the
following sections.

Similar to public institutions, academic literature provides quite abstract
definitions of the term ESG. For example, gillan2021firms state in their paper
”ESG refers to how corporations and investors integrate environmental, social
and governance concerns into their business model”. papoutsi2020does on the
other hand try to define sustainable practices with regards to the
environmental and social dimension:

>   * •
>
> ”Environmental practices involve resource conservation, reduction of waste
> consumption, and reductions in the use and production of dangerous
> substances.”
>
>   * •
>
> ”Social sustainability practices refer to companies’ accountability to
> society and encompass themes related to the elimination of poverty and
> diseases, equal and universal access to health services, universal access to
> education, and social welfare.”
>
>

Summing up, providing a universally valid definition of ESG is only possible
on a high level. Various indicators and impact factors can be identified for
each of the three dimensions, environmental, social and governance. Thus if a
more specific definition is needed, it has to be developed based on the
respective use case. Existing literature provides several examples of studies,
where either the concept ESG is narrowed down for their use case or a common
set of indicators is derived from frameworks, company publications and similar
(cf. papoutsi2020does; landrum2018identifying). For instance,
rahdari2015designing analyse various ESG related documents like frameworks,
guidelines and rating systems in order to derive a set of corporate
sustainability indicators.

###  2.2 Regulatory Environment

The issue of creating a common, general definition of what relates to and is
captured by the term ESG is well reflected in the scattered regulatory
environment. Various different standards and recommendations were published
since the 1990s, which provide disclosure guidelines for companies on the one
hand and tools for investors and stakeholders to evaluate companies ESG
activities on the other hand. In recent years the lack of standardization and
abundance of standards is recognized as a problem. First initiatives to
consolidate and converge frameworks are taken (cf. fitESGratingDiff;
IFC_Guidebook).

Figure 1 shown below provides an overview of the most important initiatives,
public institutions and frameworks related to the ESG topic.

Figure 1: Regulatory Timeline

The first important milestone in the area constitutes the establishment of the
Global Reporting Initiative (GRI) in 1997. It is one of the oldest
organizations in the area and provides the most extensive reporting framework.
Today, their standards are used by the majority of large companies worldwide
as a basis for ESG reporting. Two years later the AA 1000 Accountability
Principles were published. These principles are internationally accepted and
provide guidelines on how to react to sustainability challenges under the
aspect of long-term performance improvement (IFC_Guidebook). In the same year
the Organization for Economic Co-operation and Development (OECD) presented
the ”OECD Principle of Corporate Governance”, which elaborates important
elements of corporate governance and contains recommendations for governments
regarding regulation as well as best practices for businesses (cf.
OECD_Princ_1999). In 2000, two important projects have been launched, the UN
Global Compact and the Climate Disclosure Project (CDP). The CDP provides a
disclosure system to report and measure environmental and climate impact,
which is addresses to states, cities, companies and investors alike
(Reporting_Guide). By comparison, the UN Global Compact is a more business-
oriented initiative. It requests companies to commit to the implementation of
policies promoting, for example, the protection of human rights and the
environment. The overarching goal is to push companies towards supporting the
achievement of UN goals (IFC_Guidebook).

The next milestone, with respect to promoting sustainable practices and
related reporting, is the Accounting for Sustainability Project (A4S),
initiated by the Prince of Wales in 2004. The central aim of the project was
to integrate sustainability considerations into daily business decision. The
first principles, which focused on investors instead of companies or
regulators, were published in 2006 by the UNEP FI and the UN Global Compact.
The goal of the UN Principles for Responsible Investment (UN PRI) is to
provide guidelines for the integration of sustainability criteria into
investment analysis (PRI). In 2007, the Climate Disclosure Standards Board was
created. It is an international consortium of businesses and NGOs, providing a
framework on how to increase the precision of climate and social disclosures.
The UN Sustainable Stock Exchanges Initiative, launched in 2009, focused again
more on the investment side. It examines how stock exchanges and investors,
among others, can contribute to sustainable development (IFC_Guidebook).

In 2011, another important organization is founded, the Sustainability
Accounting Standards Board (SASB). Relying on research and market input, the
aim of the board is to identify financial material sustainability information
and how this can be communicated effectively between corporations and
investors (Reporting_Guide). Additionally the OECD published the OECD’s
Guidelines for Multinational Enterprises in 2011, which advice large
international companies on how to conduct business responsible.

Aiming at consolidating this complex landscape of various, diverging,
reporting frameworks outlined above, the International Integrated reporting
Framework was published in 2013 by the International Integrated Reporting
Council. It provides guidelines for integrating ESG reporting to financial
performance reporting (Reporting_Guide). Pursuing the same goal, streamlining
ESG and non-financial reporting, the European Commission launched the non-
financial reporting directive. It forces larger companies to disclose certain
non-financial information within the scope of financial reporting
(EU_directive). This is one of the first binding regulations for companies,
which includes requirements to disclose ESG related information. The earlier
frameworks released by several public institutions are usually non-binding
recommendations.

In 2015, the UN published the Sustainable Development Goals (SDGs), which is a
set of 17 goals with 167 targets. It addresses everyone, including
governments, companies, investors and regulators, to contribute to the
achievement of these goals (UNAgenda2030). Today, investors and companies
utilize the SGDs to evaluate their ESG performance and decisions. The Network
of Central Banks and Supervisors for Greening the Financial System (NGFS) is
founded in reply to the Paris ”One Planet Summit” in December 2017. The
purpose of the joint network of eight central banks is to enable the
achievement of the goals formulated in the Paris agreement. More specific, to
direct the financial system to enable sustainable innovations by targeted
investments (IFC_Guidebook).

Most recent developments comprise the formation of an IFRS working group and
the US Governance Improvement and Investor Protection Act in 2021. The IFRS
working group is targeted at reducing the fragmentation and facilitate the
convergence of standards and frameworks related to ESG and non-financial
reporting. A potential solution could be an international sustainability
reporting standard, prerequisites therefore are analysed by the working group.
The first requirements for compulsory reporting of ESG topics and metrics are
established in the United States by the Corporate Governance Improvement and
Investor Protection Act (SEC_reg_ESG). Additionally a Climate and ESG Task
Force is established by the US Securities and Exchange Commission, with the
objective of detecting misconduct related to environmental, social and
governance topics (SecPressRelease).

With new developments every year, the regulatory environment of ESG related
topics is still evolving. First initiatives to consolidate the diverse
standards and frameworks are taken. A higher degree of standardization is
expected in the next years and would be favorable for all stakeholders. As a
result, the ESG performance of companies could be estimated more reliable and
help to uncover corporations ”greenwashing” their operations.

###  2.3 ESG Rating Landscape

The goal of this study is to predict ESG ratings for companies. Comparable to
credit ratings, which measure the financial stability of companies, ESG
ratings are designed to measure companies performance with respect to
environmental and social responsibility as well as governance practices.
Whereas credit ratings do not show large deviations from provider to provider,
ESG ratings deviate substantially. It is not uncommon that the same company
receives a high rating from one provider and an average or low from another
(dimson2020divergent; berg2019aggregate). ESG ratings are provided by rating
agencies and financial institutions, for example MSCI, Bloomberg, Asset4
(Refinitiv) or Sustainalytics. Despite the increasing popularity of including
ESG metrics into investment analysis, as indicator for sustainability,
available metrics are still intransparent. Even though providers publish
information about their score construction methodology, crucial information
with respect to interpretability and comparability is missing. Depending on
the provider, data sources, data format and basic assumptions of what the
rating should represent vary. This in turn leads to significant differences in
the constructed ratings. Comparing the ratings of six popular providers,
berg2019aggregate identify the use of different attributes as well as how
these are measured as the two main sources of variation (explaining more than
90% thereof). The divergence in ratings and the lack of transparency poses a
problem for investors and makes the reliability of these metrics questionable
(avetisyan2017consolidation; widyawati2020systematic). Exemplary, it is
critized that MSCI’s ESG ratings do not measure the environmental
sustainability of the company but rather the exposure of the company to risks
originating from the environment, like water shortage (ESGmirage). For
investors, it thus takes substantial effort to understand what is incorporated
by the different ratings and which one reflects their position best
(li2020difference).

Focusing on the Asset4 ESG ratings provided by Refinitiv, which will be used
as benchmark for the proposed model, drempetic2020influence analyze the impact
of firm size on these ratings. Their results show a significant positive
correlation between the market capitalization (i.e. firm size) and the Asset4
ESG score. They argue that this finding is rather related to the resources
companies have to provide ESG data. Larger firms can allocate more resources,
which in turn leads to a higher data availability with respect to ESG topics.
For the proposed model, this implies that predicted ESG ratings should be
analyzed grouped by the size of the company to reveal whether news based ESG
ratings have the potential to treat smaller and larger companies more equally.

As a result, constructing ESG ratings based on the analysis of news articles
as proposed in this study, seems to be favorable with respect to increasing
transparency. Focusing on news articles instead of reports published by the
company itself, offers the advantage of relying on an independent information
source. In their own CSR or ESG reports, companies focus on the positive
achievements while trying to distract readers from issues. This raises
concerns with regards to ’greenwashing’, i.e. claiming to act sustainable by
over reporting or presenting misleading information to stakeholders and the
public although not actively engaging in sustainable practices
(fitESGratingDiff; azhar2019text).

###  2.4 Relevance of ESG topics

The growing interest in the ESG topic already indicates its importance. For
example, while last year, 2021, 80% of companies in the S&P500 published
sustainability reports, only 20% did so in 2011 (GA_sustain). ESG and related
topics are not only relevant to companies and investors but also to
governments and society. Viewing ESG from these three angels, helps to fully
understand its dimensions.

For companies, improving their ESG performance and reducing ESG related risks
has become an important part of long-term strategies. Especially negative ESG
news and misconduct poses both reputational and financial risks to
corporations. Results of financial performance analysis over recent years as
well as results from surveys emphasize the financial benefit of ESG and its
importance for long term value creation (cf. IFC_Guidebook;
forbesDemistifying). According to estimates of the Bank of America, S&P500
companies lost more than $600 billion of market value due to ESG issues over
the last seven years (BoFa_ESG_matters). With respect to risk management, the
importance of ESG increases from an reputational perspective too. Even
companies with a high value or reputation can not afford negative ESG
headlines as this potentially leads to brand damage, loss of customers and
lower investments. These risks are further increasing as the awareness with
regards to sustainability topics increases among investors, governments and
other stakeholders.

Investors and investment companies play a central role in this regards.
Allocating capital to the best performing ESG companies should not only be
considered from the perspective of potential higher returns but also from a
sustainability perspective. The UN estimates that up to $7 trillion per year
are necessary for the achievement of the Sustainable Development Goals (SDGs)
(IFC_Guidebook), which emphasizes the importance of directing capital to
sustainable projects and companies. Including ESG criteria in the investment
decision, is referred to as socially responsible investment (SRI). Despite the
increasing popularity of SRI in recent years, the questions of how to evaluate
the environmental, social and governance responsibility of a company remains
unsolved. In that respect, investors face the challenge of either selecting
one of the many frameworks to evaluate their investment options themselves or
of choosing out of available ESG ratings the one reflecting their opinion the
best. The lack of transparency of ESG ratings further complicates this
decision. Reliable ESG ratings, which are constructed transparent, would help
to resolve these issues and promote sustainable investments (cf. PRI;
forbesDemistifying).

The third dimension identifies the presence of ESG topics in discussions of
governments, public institutions and the society in general. The publication
of various standards and frameworks by public institutions (see Section 2.2)
to guide investors and companies on the one hand side and the emerging
compulsory regulation of ESG disclosure by policy makers on the other hand
side, emphasize the increasing attention towards environmental, social and
governance topics. Governmental agreements like the Paris Agreement and the
publication of the Sustainable Development Goals by the United Nations,
further move sustainability topics in the focus of public (cf. IFC_Guidebook;
SEC_reg_ESG).

Summing up, the popularity and relevance of ESG topics increased over the last
years because of the potential impact companies and investors have on the
future sustainable development and achievement of related goals. ESG ratings
provide two central contributions in this regards. First, the more accurate
they present actual ESG performance of companies rather then reward ”green
washing” the greater the incentive for companies to adapt their business
practices. Second, investors confidence, in choosing companies which are best-
performing with respect to ESG or expose the lowest risk related thereto, is
higher. This in turn enables them to actively invest and thereby support these
companies. Currently, confusion about metrics and their interpretation still
prevents some investors from considering ESG in their investment decision
(li2020difference).

##  3 Related Literature

###  3.1 Overview of ESG related Literature

The growing attention to ESG practices of companies and the trend of social
responsible investing in recent years arouse the interest of researchers in
the topic. The majority of literature dealing with ESG topics can be
classified into three streams. The largest stream studies the effect of
companies ESG compliance and ratings on their financial performance. ESG
investing or social responsible investing (SRI) can be identified as the
second main focus of academic literature. The third stream, which received
considerable attention of researches, analyses companies ESG compliance or ESG
activities in general.

friede2015esg review previous literature focusing on the relation between ESG
and the financial performance of companies, They identify about 2200 studies
related to the topic. According to them, most studies (about 90%) report a
positive relation between ESG and financial performance. Even though
controversies about the actual effect can be found in literature, they argue
that these are related to two issues. First, several partial reviews of
existing evidences in the area report deviating results. Second, there is a
confusion about company versus portfolio related findings. Portfolio studies
report a rather neutral effect of ESG on performance, however, measuring the
effect in a portfolio setting is more complex. Regardless thereof, company-
related studies reach the consent of a positive relation between ESG and
financial performance, which seems to be stable over time since the the mid
1990s (friede2015esg). gillan2021firms argue that findings are less clear as
concluded by friede2015esg. Nevertheless, the results of their review are in
line with (friede2015esg) findings. The majority of literature analysed shows
a positive relationship between ESG and performance too. The positive effect
seems to be higher for larger firms in developed countries as well as firms
which rely on advertisement (gillan2021firms). According to (gillan2021firms)
mixed and deviating results of different papers on the topic are due to
varying performance and ESG measures. Financial performance can be measured
for example by short or long-term stock returns, operating performance or
other metrics. ESG performance can be either measured aggregated, for example
by ESG ratings, or based on specific numbers like charitable donations.
Consequently, deviating effects of ESG on performance are not surprising due
to differences in what is actually measured. Additionally ESG ratings differ
substantial from provider to provider (cf. Section 2.3), which further impedes
the comparability (li2020difference). An interesting finding with respect to
the proposed analysis of news is reported by kruger2015corporate. Focusing on
the reaction of the market to ESG events, they report a stronger reaction to
negative compared to positive ESG events. While no significant effect for
positive events is identified by them, a significant negative reaction
especially to events with environmental and social impact is shown
(kruger2015corporate). This suggests that negative ESG news about a company
might have a greater impact on the rating of a company than positive news. The
goal of the proposed model is to predict ESG ratings, how these are correlated
to the financial performance of a company is not addressed. The focus rather
is on providing a reliable ESG rating, which reflects the ESG performance of a
company. Hence i will not go into further detail here.

Social responsible investment (SRI) has it origins back in the 1990s, when
investors started to screen companies for non-social or non-ethical behaviour
and excluded these from their investment portfolios. Since then it evolved
from excluding companies with ESG issues over choosing the best performing ESG
companies towards the more balanced approach of ESG ratings, which rank
companies with respect to their ESG performance on a certain scale e.g. 0 to
100 or A to F (similar to credit ratings) (haigh2004financial). While the
topic is addressed by academics from various angles, most important in the
context of ESG rating prediction are studies which relate SRI to ESG metrics.
Environmental, social and governance measures play an important role for SRI,
on the one hand side they are used to assess companies sustainability
performance and on the other hand side they facilitate the SRI market by
providing legitimacy and raising awareness (widyawati2020systematic). ESG
metrics are used by both, investors and researches, to quantify and compare
companies ESG or sustainability performance (cf. auer2016socially;
girerd2014dimensions). Thereby, the establishment of ESG ratings had a
significant positive effect on the growth SRI market. The ratings increase the
accessibility of SRI for investors while ESG agencies provide legitimacy by
interacting with various financial actors. However, the effect of ESG ratings
on the growth of the SRI market could be enhanced, if the transparency and
reliability of available ratings would increase and thereby simplify investors
choice of a rating matching their intentions (juravle2008identifying).

As mentioned in section 2.1., the terms Corporate Social Responsibility and
ESG are used interchangeable by practitioners, companies and researchers
alike. Same applies to similar terms like corporate sustainability,
environmental management and so forth (landrum2018identifying), which are all
interrelated with each other. Hence no clear distinction of literature
referring to one or the other can be made. Several studies discuss the
terminology in the beginning, determining one they use throughout their study
(cf. landrum2018identifying; gillan2021firms). Searching academic literature
databases for the term CSR, the majority of search results covers the
relationship between corporate social responsibility and financial
performance. Hence largest CSR and ESG literature stream can be treated as
one. For example, the literature review of friede2015esg, mentioned in the
paragraph focusing on the relation of ESG and financial performance, includes
both, studies which refer to ’CSR’ and ’ESG’. This further emphasizes the
interchangeable use of the terms. One difference that can be identified is
that studies which examine non-financial reports of companies mainly use the
term ’CSR’. The reasoning behind this is most probably related to the time
factor. The construct ’CSR’ emerged before ’ESG’, thus most companies non-
financial reports are published under the name ’Corporate Social
Responsibility Report’.

The third stream of literature, which attracted considerable attention,
provides interesting insights on the ESG activities and compliance of
companies using text mining and content analysis. Especially in this section
it is emphasized that insights from both, studies referring to CSR and ESG are
considered, several studies in this area analyze CSR reports.

rahdari2015designing address the issue of which sustainability indicators
represent the current view of guidelines, frameworks and rating systems the
best. Their aim is to provide a construct, which helps managers and investors
to report and evaluate a companies ESG performance. Focusing on the corporate
level, they extract sustainability indicators from the mentioned sources.
Therefore they select the best benchmarks among the categories and apply
several filters. For example, only indicators which are measurable
(qualitative or quantitative) are included. The resulting three dimensional
ESG framework is build based on 70 indicators and comprises 10 main criteria,
which can be further broken down into 30 sub-criteria. Climate change, product
safety and board composition are only three examples for sub-criteria. Similar
thereto, papoutsi2020does constructs sustainability indicators too. These
differ from the ones of rahdari2015designing in granularity as well as are
only provided for the two dimension environmental and social but not
governance. Using existing literature and reporting guidelines, they extract
39 environmental and 12 social indicators. Examples for environmental
indicators named by papoutsi2020does are ’reduce carbon footprint’, ’source
locally’ or ’use renewable energy’. Compared thereto, rahdari2015designing’s
indicators are formulated less precise, e.g. ’climate change’. This might be
resulting from the different perspective of the two papers. Whereas
rahdari2015designing aim to provide a general construct for various
stakeholders, papoutsi2020does use these indicators to score companies based
on the information disclosed in sustainability reports. Their aim is to
analyze whether disclosure in these reports indicates actual sustainability.
Therefore they exploit content analysis to score each indicator (from 0 to 3)
and use Bloomberg ESG ratings and Dow Jones sustainability indices as proxies.
According to their results, ESG measures can be explained significantly by the
information disclosed by companies. However, here the question needs to be
raised whether this should not be viewed the other way round. ESG ratings are
highly influenced by companies’ reports because most ESG rating providers rely
to a large extent on information disclosed in financial and non-financial
reports.

ning2021online examine the topic from a different perspective, they analyze
CSR reports of companies with the goal to determine the companies’
sustainability maturity. They identify typical topics discussed in CSR reports
using a latent dirichlet allocation (LDA) algorithm. LDA is a popular topic
modeling approach, which maps documents to topics based on word probabilities
within topics. Making use of the results, they model companies’ sustainability
maturity by applying a latent markov model (LMM) in the second step. The
maturity of a firm in their case refers to whether it reports or does not
report about the six categories economic, labor, product responsibility, human
rights, environment and social. Whereas their concept of measuring
sustainability maturity refers to the diversity of companies engagement in
related topics, (landrum2018identifying) define five stages of sustainability
based on the sensemaking of companies. According to their concept, companies
show weak sustainability, if they only engage in from a compliance perspective
necessary or internally beneficial CSR / ESG activities. The stage of strong
sustainability refers to companies, which try to mitigate the impact on the
environment and society influenced by their operations and aim at contributing
rather than posing harm to it (landrum2018identifying).

Analyzing the same information source, CSR reports, kiriu2020text use text
mining to evaluate companies’ ESG activities. They focus on the quantity and
specificity of ESG activities based on qualitative information provided in CSR
reports. Using word2vec word embeddings, they classify the words extracted
with respect to the three ESG dimensions, environmental, social and
governance. Based on the results they build a hierarchical word structure. To
assess the quantity of ESG activities, they use simply the number of words as
a proxy. The quality of ESG activities is measured by kiriu2020text by
assessing the divergence of words in the companies CSR reports. Running
regression analysis using Asset4 ESG scores as proxy for the quantitative
performance of a company, they find that high scores for quantity and
specificity of ESG activities imply a higher Asset4 ESG rating.

Instead of analyzing CSR reports, hence information provided by the company
itself, azhar2019text conduct a content analysis of news articles. Thereby
they show that companies’ CSR prominence can be assessed based on information
in the news. Using news as datasource mitigates issues like under- or
overreporting of ESG / CSR topics according to them. The media is seen as an
independent party, which watches and reports about firm’s activities from the
perspective of the public. The frequency of a company being mentioned
positively in an CSR related news articles is used as proxy to evaluate a
companies’ CSR prominence. Using the GDELT database, they derive a list of
articles, the returned URL list is crawled to get the articles’ content.
Keyword search is used by them to identify articles related to CSR and the
companies in their dataset (331 companies listed on Singapore Exchange).
Additionally they use network graphs, to detect collaborative networks of
companies. hisano2020prediction use news as information source too, arguing
similar to azhar2019text that these are less susceptible to manipulation. They
construct a heterogeneous information network with the goal to predict the
appearance of a company on an investment exclusion list due to ESG issues.
This refers to the ’worst-in-class’ approach often applied in investment
analysis, which refers to excluding the worst performing companies from
possible investment options. Thereby they report two interesting findings.
First, negative ESG related news lead to worse financial performance and
second, indicators from news perform quite well on predicting whether a
company will be on an investment exclusion list. The papers of azhar2019text
and hisano2020prediction provide valuable insights with respect to the
proposed model. Both papers provide evidence that news articles are a valid
source of company related ESG information and might even be favourable over
ESG / CSR reports due to their independence and commitment to public interest.

###  3.2 ESG related Text Classification

Most relevant to the planned study, is the stream of ESG-related literature
which analyzes and classifies texts with respect to their ESG relevance.
Different data sources as well as methods were evaluated by researchers. In
the following approaches of important papers with respect to the proposed
study are discussed.

Based on a news article dataset dealing with ESG controversies,
nugent2021detecting aim to improve the accuracy of text classification for
such topics leveraging domain-specific language models and data augmentation
approaches. Interesting with respect to the proposed thesis is their result
that further pre-training the baseline Bidirectional Encoder Representations
from Transformers (BERT) (cf. devlin2018bert) on a collection of financial and
economic news, outperforms the baseline BERT model for the ESG classification
task.

The approach of distant supervision of neural language models is pursued by
raman2020mapping. Using transcripts of earnings calls, they evaluate which
percentage of discussions in earnings calls focused on ESG topics over the
last 5 years. Therefore they rely on fine-tuning word embeddings of a
Transformer model using CSR reports. The idea is to train the algorithm on the
detection of ESG language thereby. In a first step, the sentences of the CSR
reports need to be labeled. To solve this efficiently, an unsupervised
mechanism is applied which identifies the labels based on vector-space
matching of popular indicators and the sentences’ word embeddings.

The papers mentioned developed models to detect ESG topics pursuing different
objectives, however, none of them aims to predict company ESG ratings. The
underlying idea of the proposed paper is to predict ESG ratings from news
data.

Searching existing literature for ESG rating prediction, two recent papers
need to be mentioned even though they use numerical data instead of text
analysis. damato2021esg use balance sheet items, more specific they build a
random forest algorithm to predict the ESG rating of a company based on the
variables year, net sales, EBIT (earnings before interest and tax), PE (price-
to-earning ratio), ROE (return on equity), DY (dividend yield) and the
company’s sector. According to their results, balance sheet items show a
predictive power on ESG ratings with sales being the most important variable.
chen2021social take a completely different approach. They rely on the
dimension specific Asset4 scores provided by Refinitiv. Applying a data
envelopment analysis (DEA) model they use quadratic an cubic terms instead of
the commonly used weighted average approach to calculate an combined ESG score
from the three sub-scores. Comparing portfolios constructed using the average
weighting and their approach, they show that performance can be improved
employing their method.

Focusing on previous literature applying text analysis and classification to
construct ESG ratings, the following two papers are the most important with
respect to the proposed model.

The recent paper of borms2021semi is highly relevant with respect to the
analysis in this paper. According to them, the ESG related information that
can be derived from news articles is of high value. They show that ESG indices
constructed from a Flemish-Dutch news dataset can predict negative adjustments
in ESG scores from the external provider Sustainalytics up to a few months in
advance. Applying a semi-supervised text mining approach they construct ESG
indices for 291 European companies from a dutch-written news corpus containing
365,319 articles after dropping irrelevant articles. In the first step of
their analysis, a set of keywords is generated for which the articles are
queried in the next step to assess their ESG relevance. Defining the seed
words for the three dimensions, environmental, social and governance, as well
as the additional negative sentiment dimension is done manually by the
researches making use of factors defined as important e.g. by Sustainalytics.
This set of keywords is then enlarged leveraging GloVe word embeddings of
their news corpus, prefiltered per dimension. The actual selection of keywords
is again done partly manual, for example irrelevant words are dropped. In
total their keyword set consists of nearly 600 words for the three ESG
dimensions and slightly more than 6,000 sentiment words. In order to aggregate
the information contained in the news corpus into indices, borms2021semi set
up a time series matrix for each entity which maps the filtered corpus to
numerical metadata variables, resulting in eleven frequency-based and 6
sentiment-adjusted indicators. The final step comprises aggregating these
matrices into time-series indicators.

Whereas borms2021semi apply rather basic NLP techniques, by using GloVe word
embeddings, sokolov2021building leverage a state-of-the-art NLP algorithm.
They use the Bidirectional Encoder Representations from Transformers (BERT) to
analyze social media data (Twitter tweets). sokolov2021building show that the
accuracy of classifying ESG relevant content can be improved using a slightly
modified standard BERT classifier. The modification corresponds to an
additional fully connected layer between the BERT embedding and the output
layers. For labeling the data, prior to model training, ground truth
generation was applied by sokolov2021building. More specific, after selecting
topics common to be discussed in social media, they prefiltered each of their
ESG categories using keywords and human review. Thereby they select 1,468 ESG
relevant tweets, labeled either positive or negative, out of the 6,000 tweets
initially derived from Twitter for the time period from June 16th to July 22nd
2019. The precision-recall curve (PRAUC) and the ROC-AUC, PR-AUC are used to
evaluate the final model. They propose an aggregation function for ESG index
construction. The daily index of a company is derived by dividing the sum of
each category’s predicted probabilities by the sum of input documents for the
day. Even thought they propose this method for ESG index construction, they do
not report the resulting indices or compare them to any benchmark. Summing up,
they show a promising approach for classifying social media data with respect
to ESG relevance. However, they only describe the methods to construct an
index but do not evaluate it. Further they state themselves that a robust
classifier would require a larger trusted dataset. Almost concurrently, three
of the authors of sokolov2021building published another paper, which
constructs ESG scores. With the goal to build ESG portfolios automatically,
sokolov2021weak first apply an approach of weak supervision to construct an
ESG classifier and aggregate the results to an ESG index. This is used in the
second step as input to a Black-Litterman model to build ESG portfolios,
however, this part is not relevant with respect to the proposed analysis.
Using New York Times (NYT) articles, they use author-curated tags provided by
NYT as weak labels for the ESG categories. Training a standard BERT classifier
based thereon, they classify their articles content with respect to ESG
relevance. Similar to sokolov2021building they fine-tune the BERT classifier
in two steps, in the first training stage only the classification layer is
trained. In the second the full network is fine-tuned. To evaluate the
performance they report ROC-AUC and PR-AUC scores as well as precision and
recall thresholds. They conclude that their results are promising but propose
various areas of improvement. This includes the use of smaller models like
DistilBERT or TinyBERT to reduce computational costs and mind resource issues.
In the proposed analysis a modified DistilBERT classifier will be used to
determine the articles ESG relevance.

The analysis in the proposed thesis is based on the idea of predicting ESG
ratings from news data like employed by sokolov2021building. However, compared
to the approach that will be applied in the proposed model, they did not
evaluate the sentiment and semantic of ESG related information. The resulting
time series will be used as input to train a deep learning model on the task
of predicting ESG rating. Hence they did not leverage the possibilities of
state-of-the-art natural language processing algorithms and time series
prediction. Furthermore the idea is to provide a fully automated approach,
which does not need any manual selection or similar as necessary in the method
applied by borms2021semi and sokolov2021building. Additionally the analysis
will be based on a larger dataset, both in terms of number of companies and
number of news articles.

###  3.3 Natural Language Processing

Three types of NLP tasks have to be solved in the following study to be able
to generate the input timeseries for the final prediction of the ESG rating.
First, the content of the news articles needs to classified with respect to
the three ESG dimensions, whether information about environmental, social or
governance topics is covered. This problem is commonly referred to as text
classification. Second, the sentiment of the labeled content has to be
determined, hence the task to be completed is sentiment analysis. Third, the
semantics are analysed, in order to derive a semantic representation of the
articles and the topics covered by them. The final task to be completed is the
prediction of the companies ESG ratings, utilizing the timeseries generated by
the three NLP analysis steps. Multivariate timeseries prediction or
classification is no NLP specific problem but rather a general machine
learning task addressed in various fields. The following section provides an
overview of the state-of-the-art in Natural Language Processing before
evaluating different algorithms proposed by literature to solve the four tasks
mentioned.

Generally, natural language processing (NLP) can be defined as the task of
automatically analyzing, interpreting and representing human language using
various computational techniques young2018recent. Recent trends in the area,
inspired by the advances in other fields, comprise leveraging deep learning
methods for NLP tasks. The concept of distributed representation, on which
word embeddings are based, lays the foundation for deep learning models. Word
embeddings are also called vector space models because words semantic similar
(e.g. happy and excited) to each other are placed close to each other in the
vector space. Hence they assume that words with similar contexts have a
similar meaning, which is stated by the distributional hypothesis (cf.
harris1954distributional). Two widely used models Word2Vec and GloVe, which
emerged in 2013 and 2014 respectively, are based on this assumption. With the
aim to improve the efficiency of word embedding construction using neural
networks, mikolov2013efficient proposed two new algorithms under the name
word2vec. The continuous bag-of-words model is designed to predict a word
based on its context while the skip-gram architecture works the other way
round. It takes a word and predicts its context. They further improve the
performance of the skip-gram model by proposing negative sampling, a simpler
alternative to the hierarchical softmax function (mikolov2013distributed). A
large benefit of these vector representations is that algebraic operations can
be performed therewith because distances between words, which are seen as
equally similar, are equal. For example the distance between the vectors of
’big’ and ’biggest’ is equal to the distance between the vectors of ’small’
and ’smallest’ mikolov2013efficient. Whereas word2vec leverages neural
networks with embedding layers, the GloVe model proposed by
pennington2014glove relys on word co-occurrence matrices to construct word
embeddings. The approach applied by pennington2014glove can be seen a as a
combination of the context-based model of Word2Vec with matrix factorization
techniques. More specifically, GloVe is an unsupervised model using global
log-bilinear regression. Instead of defining a certain local context window,
the model is based on a word co-occurence matrix which is constructed from the
whole corpus.

Common deep learning models used in natural language processing are
convolutional neural networks (CNNs), recursive neural networks (RNNs) and
long-short term memory networks (LSTMs), which represent a special type of
RNNs. Convolutional neural networks (CNNs) proved to be helpful for some
classification tasks as they perform well in mining semantic connections and
extracting position independent features. However, a main disadvantage of CNNs
is the inability to take into account long-term dependencies. By comparison,
RNNs can ’memorize’ information from previous steps and use it in the current
processing step, which presents a major advantage over CNNs (cf.
othan2019financial; young2018recent). Therefore RNNs are popular to be applied
to NLP tasks. RNNs process words or sentences in sequences, enabling to detect
context dependencies and the semantic meaning of a word in a given context.
However, which of both RNN or CNN performs better, can not be answered in
general but depends on the specific NLP task and its required semantics.
According to the comparison of (cf. yin2017comparative), it is especially
important whether the semantical meaning of whole sequences needs to be
understood or not. Long-short term memories (LSTMs), are a special type of
RNNs, which are particularly suitable for modelling long-term dependecies.
LSTMs allow for error back-propagation and thereby solve the problem of
exponential error growth of the activation function (cf. young2018recent).

With the goal to reach a higher degree of parallelization to overcome
constrains of sequential computation, vaswani2017attention introduced the
transformer architecture. The Transformer encoder-decoder architecture relies
entirely on multi-headed self attention, replacing sequence aligned recurrent
or convolutional neural network structures. The main idea of the Transformer
model is to exploit multi-headed self-attention for the computation of inputs
and outputs. Self-attention refers to deriving the representation of a
sequence by relating different positions within that sequence. Using multiple
heads allows to produce multiple outputs in parallel instead of sequentially.
This multi-headed self-attention mechanism allows to model large distance
dependencies, which is not possible using RNNs or CNNs due to memory
constraints when applying sequential processing (vaswani2017attention).

Most recent advances in the area propose the use of bidrectional encoder
representations, which represent a new generation of word embedding models.
The neural language model BERT or bidirectional encoder representations from
Transformers, proposed by devlin2018bert, leverages the Transformer
architecture. They are the first ones to apply a bidirectional approach, which
allows to compile context-related word embeddings considering both directions,
from left-to-right and right-to-left. The difference with respect to earlier
unidirectional models lies in applying a masked language model, which is build
to predict masked words based solely on their context (devlin2018bert).
Various researchers, like sokolov2021building and nugent2021detecting, used
BERT for NLP tasks and reported significant performance improvements over
other methods. othan2019financial compares classification results of BERT to
those of a CNN, RNN and LSTM and reports the best performance using BERT.

Text classification

The first NLP task is the classification of news articles as ESG relevant or
noice. As analyzed above, BERT seems to be the best performing algorithm for
such natural language processing tasks as of today. Therefore BERT will be
used to classify the texts. BERT is a supervised machine learning model,
requiring a prelabeled training dataset, however the dataset that will be used
is not labeled. Hence, the issue faced is how to classify the content of the
news article without having to label a large number of articles manually.
Different approaches are proposed by literature and will be discussed in the
following.

borms2021semi and sokolov2021building rely on extended keyword labeling.
Taking an initial set of ESG keywords, borms2021semi uses Glove word
embeddings to extend their keyword set. sokolov2021building select ten ESG
categories based on the MSCI ESG ratings, which are likely to be covered in
social media. A set of keyword is defined for each category. Using these
keywords, both filter their input dataset for texts containing the respective
keywords and review the category assignment manually. Relying on keywords to
assess the topical relevance of articles entails the risk of having an
incomplete keyword selection and thereby labeling relevant articles as
irrelevant. Especially with respect to the topic of ESG, this risk is present
because it is a multi-dimensional, extensive topic with no clear definition of
what should be included or excluded. Therefore, the keyword labeling approach
is only applied to compare the results of the proposed model with those
resulting from applying borms2021semi and sokolov2021building approaches.

An interesting approach to solve the problem of labeling a sufficient large
training set is the approach of weak or distant supervision exploited by
raman2020mapping and sokolov2021weak. As explained in Section 3.2,
sokolov2021weak use tags provided by the authors as weak labels to train a
BERT classifier. The method proposed by raman2020mapping is more complex. It
comprises 3-steps to classify the news articles with respect to their ESG
relevance. In a first step a corpus of highly ESG relevant language needs to
be labeled, on which a pre-trained language model will be fine-tuned in the
second step. CSR reports will be chosen as these discuss ESG relevant topics.
Pre-training a language model on topics discussed in these reports aims at
increasing the probability that these topics are classified as ESG relevant in
the news dataset too. In order to label the content of the CSR reports
sentence-wise, a pre-trained embedding vector containing ESG keywords is
created. It is used to calculate the cosine similarity between each keyword
and the sentences in corpus. The keywords have to be defined manually,
however, a rather small set is necessary as similar words do not need to be
defined separately but show a high cosine similarity in the embedding space
anyways (raman2020mapping). According to them, using pre-trained word
embedding vectors of keywords has the advantage over a simple keyword search
that it takes into account similar words and therefore reduces risk of missing
important keywords, which could result in classifying text erroneously as
irrelevant. With the labeled sentences at hand, raman2020mapping optimize a
pre-trained Transfomer language model by adding a customized output layer for
the classification task.

The application of a weak or distant supervision approach to label a training
set and classify the data showed promising results and seems to increase the
probability of assigning fitting labels. Thus an approach of weak-supervision
is chosen, in order to classify the news articles with respect to their ESG
relevance. More specifically, semantic similarity analysis will be used to
pre-label the dataset. This procedure is explained in detail in Section 4.2.1
and will be based on pretrained sequence embeddings as proposed by
reimers2019sentence. With the aim to provide a BERT version, which can be used
for tasks like semantic similarity search, they develop Sentence-BERT (SBERT).
SBERT represents a modified BERT network that is based on a siamese network
structure and adds a mean pooling operation of the CLS-token to derive fixed
sized sentence or sequence embeddings. While detecting similar sentence pairs
for 10,000 sentences using BERT would take about 65 hours because several
million combinations would have to be computed, SBERT is able to solve this
tasks in about 5 seconds according to reimers2019sentence. Further they report
improved performance for unsupervised tasks over the BERT model and other
sentence encoders algorithms like InferSent or SentEval.

Additionally, it needs to be analyzed which pre-trained BERT language model
should be selected. Various different pre-trained BERT models are proposed by
literature and are available in python via the Transformers library
(wolf2019huggingface). With respect to the proposed analysis, especially
variations of the classic BERT model either using sector specific datasets or
reducing model size and thereby increase efficiency are interesting.

Based on the classic BERT model, different sector specific variations are
proposed by researchers. This applies to the financial sector too. For example
liu2020finbert extended the training corpus used for the ”basic” BERT version,
consisting of English Wikipedia entries and books, by adding a financial
domain corpus, consisting of financial news articles, yahoo finance articles
and financial issue question answer pairs from the Reddit website. Similar,
araci2019finbert, desola2019finbert and yang2020finbert developed FinBERT
models trained on different financial corpora. Even though the topic of ESG
ratings is closely related to financial sector, it rather is a cross-sectional
topic. Hence general models instead of models specifically adopted to the
financial domain will be used. This has two reasons, on the one hand side
similar papers introduced in Section 3.2 apply the classic BERT model (cf.
borms2021semi; sokolov2021building). On the other hand side, a model trained
on financial language has a higher probability of classifying news related for
example to a rise in share prices as relevant. However, with respect to ESG
such articles should be classified as noise.

Focusing on general purpose models, different variations of BERT were
developed. Three popular examples are RoBERTa, ALBERT and DistilBERT. RoBERTa,
known as ’robustly optimized pretraining approach’, provides improved results
compared to BERT by modifying the training process. They train the model
longer, on larger mini-batches which consist of longer sequences
liu2019roberta. Whereas this approach increases training time and model size,
ALBERT and DistilBERT were developed with the goal to provide smaller and
faster models, which provide comparable performance to the original BERT
model. Two changes in the configuration are proposed by lan2019albert in order
to increase training speed and lower memory usage. In the ALBERT model the
embeddings matrix is split into two smaller matrices exploiting factorization
techniques. Additionally, parameter sharing is used to reduce the number of
parameters. Even though ALBERT has a smaller memory footprint, its
computational costs are comparable to the original BERT model because it uses
the same number of hidden layers (lan2019albert). Compared to RoBERTa and
ALBERT, which are both build on 12 hidden layers like the original BERT
architecture, DistilBERT uses only 6 hidden layers. The model proposed by
sanh2019distilbert relies on the concept of knowledge distillation, which
refers to replacing larger architectures by a ’lighter’ version. In the case
of DistilBERT, the initial BERT version is taken but the number of layers is
reduced by half as well as token embeddings and poolers are removed. It is
trained using the same corpus as the original BERT, a combination of the
English Wikipedia and the Toronto Book corpus. The models performance is
evaluated based on the ’General Language Understanding Evaluation’ (GLUE)
benchmark, which was introduced by wang2018glue to be able to compare
different models and architectures on a common benchmark. GLUE includes
various natural language understanding tasks like questions answering,
sentiment analysis and others. According to the results of sanh2019distilbert,
DistilBERT retains 97% of BERT’s language understanding performance while
being 60% faster.

Due to existing time, memory and computational constraints, especially under
consideration of the large dataset to be processed, the proposed text
classification algorithm is thus based on the significantly smaller and faster
DistilBERT model.

Sentiment Analysis

Sentiment analysis is also referred to as opinion mining, its objective is to
identify whether the opinion regarding the subject (e.g. company) in the
analysed context (e.g. ESG) is favorable (positive) or unfavourable (negative)
(salloum2020survey). Classifying texts into sentiment categories (e.g. 0 for
negative and 1 for positive) thereby enables to transform emotion into
machine-processable information. The sentiment of an article is an important
indicator that will be considered in the proposed model to predict a companies
ESG rating. The ESG news coverage of a company only shows how many articles
about a certain company are related to environmental, social and governance
topics. It does not provide information about whether an ESG related article
contains information in favor of the company’s ESG performance or rather
reflects an ESG issues. Therefore the sentiment of the articles will be
analysed.

Different approaches for sentiment analysis have been applied by researches,
evolving from lexicon based approaches to statistical methods, non-
contextualized word and document encoders as input to deep neural networks to
contextualized embeddings and transformer architectures. Due to the wide range
of models available, studies aiming at comparing their performance have been
published. mishev2020evaluation evaluate a wide range of sentiment analysis
models, including lexicon based approaches, Word2Vec and Glove word embeddings
combined with recurring and convolutional neural networks as well as various
Transformer architectures (e.g. BERT, DistilBERT, RoBERTa). Using two
different financial news datasets, they show that all Transformer models
outperform both lexicon based models as well as word and sentence embeddings
based deep neural network models. Comparing the different BERT architectures
with each other reveals that RoBERTa slightly outperforms the basic BERT
model. DistilBERT shows comparable performance to the larger BERT models while
being essentially smaller and faster (mishev2020evaluation). Similar results,
are provided by heitmann2020more. They compare the accuracy of their RoBERTa
based model to several other methods of sentiment analysis. Thereby they
provide evidence that other approaches, like the popular lexicon based Vader
algorithm or artificial neural networks, which are based for example on
word2vec word embeddings, perform significantly worse on average compared to
transformer based language models. The findings of mishev2020evaluation and
heitmann2020more argue in favor of using a BERT based model, instead of the
lexicon based approach employed by borms2021semi. In their paper sentiment
indicators for ESG news are derived by manually defining a sentiment lexicon,
which contains words associated with an sentiment polarity and is extended
using most similar words according to word embeddings.

Various researchers leveraged the BERT algorithm as well as its variants in
order to predict the sentiment of text data. An interesting approach,
combining FinBERT and a BiLSTM is presented by cheng2021sentiment. They use a
FinBERT version adapted to a financial corpus to optimize the word embeddings
and input these in a second step into a bidirectional LSTM model. Their
results imply that this approach helps to optimize the sentence-level
sentiment classification of financial comments with respect to a higher
accuracy. However, taking the same example used earlier of an article about a
rise in the sock price of a company, it can be seen that a pre-trained
financial sentiment analyzer produced with a larger probability unwanted
results. Rising stock prices are very positive from a financial point of view.
From an ESG perspective, a rise in stock price is irrelevant. Another example
would be higher expenses for research on sustainable production techniques.
Higher expense are seen rather negative from a financial point of view while
from an ESG perspective trying to increase production sustainability is very
positive.

The news articles are not prelabeled with respect to their sentiment.
Therefore, to solve the task of sentiment classification, a unsupervised,
pretrained algorithm has to be chosen. Various pre-trained sentiment analysis
models are provided via HuggingFace. For example a RoBERTa model pretrained on
millions of tweets or a DistilRoBERTa model pretrained on financial phrases.
However, as the dataset consists of news articles, a model has to be chosen
which is either pretrained on a large news dataset or on various different
datasets. As of now, no pretrained model on news in English language is
provided by HuggingFace. Therefore the best fitting model is the SieBERT model
proposed by heitmann2020more, which is fine-tuned on 15 different datasets out
of diverse sources, including social media and reviews. It is based on the
RoBERTa large model (liu2019roberta) and fine-tuned for the task of binary
sentiment analysis. Their results show that their model outperforms models
trained on a single text type when applied to a new dataset.

Semantic Analysis

Semantic analysis in NLP is used to identify the meaning of language with the
goal to interpret texts like humans would do. More specifically, it analyses
the context and text structure to identify the meaning of words and word
combinations which are ambiguous.

In the proposed model, semantic analysis will be used to cluster articles with
respect to their semantic similarity in order to identify articles discussing
similar topics as well how these evolve for a certain company over time.
Clustering is a popular method to group documents based on their semantics,
especially the k-means algorithm is used by various researches. The basic
k-means model identifies k clusters iteratively by looping trough the
following four steps (steinbach2000comparison):

  1. 1.

Random selection of k points in the vector space as the initial centroids.

  2. 2.

All points in the vector space are assigned to the closest centroid.

  3. 3.

Centroids are recomputed, i.e. moved to the center of the points assigned to
it.

  4. 4.

The distance between the centroid and its points is recalculated, Steps 2 and
3 are repeated until the distance does not change anymore (usually predefined
minimum change threshold).

An advantage of the k-means algorithm is that it is an unsupervised algorithm,
hence no labelled data is necessary. Therefore it is applicable to exploratory
classification into k groups. The number of groups k has to be defined
upfront, however, it can be identified by the elbow method or by calculating
the silhouette score. The elbow method refers to running the k-means algorithm
several times with different values for k (e.g. 2 to 10) and calculating ratio
of variance, the sum of squared errors (SSE), for each run. The method
utilizes visualization to determine the optimal k. Hence SSE is plotted
against k and the value of k for which the graph shows an ’elbow’ is chosen,
which refers to the point where no significant variance reduction is achieved
anymore by adding another cluster (cheng2021bert). Compared thereto, the
silhouette score can be calculated to mathematically identify the best value
for k. It measures the within cluster similarity compared to the distance to
other clusters. Ranging between -1 to +1, a high value implies that points
within one cluster lie close together while being well separated from points
of neighboring clusters.

The k-means algorithm is applied by various researchers to cluster documents.
For example, khan2019extractive apply k-means clustering in combination with
Term Frequency Inverse Document (TF-IDF) to summarize texts.
shah2018comparison compare various document representations techniques and
evaluate them using k-means clustering. Whereas most studies, especially
earlier ones, applied the TF-IDF method to represent texts numerical, more
recent studies leverage BERT sequence embeddings as text representations. TF-
IDF is referred to as a ’bag-of-words’ method, hence it does not consider the
context of a word. The importance of each word in TF-IDF depends on how often
it occurs in a certain document compared to in the overall
corpus(bafna2016document). Thus TF-IDF text representations are
uncontextualized, they neglect syntactic and semantic relationships. Compared
thereto, sequence embeddings derived from the BERT algorithm are
contextualized. These incorporate both the position and the context of a word
(cf. subakti2022performance; li2020text). For example, BERT gives the same
word a different embedding when being mentioned in a different context because
the same word can have a deviating meaning depending on the context.

li2020text and subakti2022performance, among others, use BERT embeddings in
combination with k-means to cluster documents. While li2020text focus on BERT
text representations to compare different weighting methods of sentence,
subakti2022performance evaluate the performance of TF-IDF versus BERT based on
four different clustering methods, including k-means. Their findings show that
using BERT embeddings outperforms TF-IDF for all four clustering methods
applied. Using various different evaluation metrics, the BERT based models
outperform the TF-IDF based in the majority of cases. While the above
mentioned papers apply the classical k-means algorithm, which is based on
euclidean distance as distance measure, park2019adc and li2020text propose a
modified version. Both papers examine the use of cosine similarity as distance
measure instead of euclidean distance. Cosine similarity presents a better
measure for contextualized text representations because it considers the
direction of the vectors instead of their magnitude. This allows to measure
the semantic similarity as the degree to which vectors of documents point in a
similar direction. cheng2021bert introduce this method of applying cosine
similarity as spherical k-means. They apply the spherical k-means method to
geospatial related literature in order to assign the documents to topics.

To conclude, the k-means algorithm is applied by various researchers to
cluster documents. Many apply it in combination with TF-IDF text
representations, however, more recent studies have shown that leveraging
contextualized representations of BERT improves the performance. Thus BERT
embeddings in combination with the k-means algorithm will be used to cluster
the articles. Both basic k-means and the modified spherical k-means algorithm
will be evaluated.

Multivariate Timeseries Analysis

The aim of the analysis is to predict representative ESG ratings for companies
based on information derived from news articles. Hence a fitting model for the
prediction of the rating needs to be identified. This represents a deep
learning task, which is not specific to the field of natural language
processing but faced in various other fields too.

First of all, the methods employed by borms2021semi and sokolov2021building
are considered, as they construct ESG indices in their studies too.
borms2021semi tracks the number of ESG keywords identified in each article as
well as the sentiment, which is based on sentiment keywords. Using a matrix
approach, this information is then aggregated to derive frequency based
indicators and sentiment indicators. They evaluate these indicators separately
by using them for stock screening (e.g. select the top 100 companies for
investment). However, they neither combine their frequency and sentiment
indicators nor compare them to ESG ratings from providers. With the goal to
predict a news based ESG rating, which is comparable to existing ESG ratings
from third parties, an aggregate ESG indicator needs to be constructed. Thus
their approach is not satisfying in terms of the objective. The method
proposed by sokolov2021building is rather simple. Analysing tweets, they only
take into account the predicted probabilities per category derived from the
BERT classifier. The probabilities are aggregated per category and day to
construct a daily ESG rating based on the collected tweets about the company
on that day. Their approach is applied as baseline for comparison. However,
with the goal to incorporate further information derived by sentiment and
semantic analysis, methods applied by other research fields are analysed with
respect to their applicability for the prediction of ESG ratings. The input
data for the ESG prediction is represented by three univariate or combined a
multivariate time series, thus time series prediction (classification or
regression) methods applied by researchers are examined in the following.

In the financial area, forecasting, for example, stock prices from text
documents using NLP gained in popularity in recent years. The predictive power
using natural language processing has increased due to advanced analysis
techniques xing2018natural. As the topic of ESG ratings is closely related to
the financial field, applied methods can reasonably be considered for the task
to of ESG rating prediction. Very often support vector regressions and
machines (SVMs) are used in the financial literature. However, SVMs only take
two-dimensional input not three-dimensional. Hence they are not applicable in
this case, as the multivariate input timeseries is three-dimensional.
Additionally some exploit convolutional neural networks (CNNs). These have two
advantages with respect to the task to be solved. First, they are able to
process three-dimensional data. Second, CNNs, originally intended for image
analysis, are commonly applied to extract important features from data. CNNs
are utilized by various researchers in different fields for time-series
prediction and classification and showed promising performance (cf.
livieris2020cnn; liu2018time). For example assaf2019mtex propose a special
convolutional neural network architecture for multivariate time series. It
consist of two stages, the first one is build using 2D convolutional layers
while the second is using a 1D convolutional layer as well as a flatten and a
final dense layer. They validate their model based on two different use cases,
to predict server outages and to predict the energy production of photovoltaic
power plants.

Another interesting approach in the field of time series classification is
applied by wang2021event. They leverage the transformer architecture to
estimate the time at which an event occurred. Using a set of documents, they
construct four timeseries capturing different information derived from the
texts, which are in turn aggregated to a multivariate time series. This
aggregated time series is used as input to the model, which exploits the
transformer self-attention to represent the sequence information and identify
important time steps. Additionally, they add two one-dimensional convolutional
layers in front of the Transformer to extract important features in the first
step. Similar thereto, wu2020deep propose a deep transformer model for time
series forecasting. They argue that the Transformer self-attention mechanism,
especially its ability to process the entire sequence of data, has an
advantage over sequential models. It is able to capture complex dynamics
inherent in time series data. In line therewith are the findings of
zerveas2021transformer. Implementing a transformer model for multivariate time
series representation learning, they show that this model can outperform other
time series classification and regression models.

Convolutional neural networks as well as transformer based models are
promising with respect to the ESG prediction task because both, sequence
information needs to be captured appropriately as well as important features
have to be identified. Thus different network architectures, incorporating
both CNNs and Transformer features, will be evaluated in the following
analysis. The architecture of the applied models is explained in detail in
Section 4.2.2.

##  4 The Model

###  4.1 Data

####  4.1.1 Dataset Creation

To provide meaningful results, Asset4 ESG ratings provided by Refinitiv are
used as reference point for the predictions. Only US American companies are
considered to allow focusing on news in English language, which facilitates
the use of state-of-the-art tools for language processing later on. As a
result, yearly ratings for 3,343 US companies could be retrieved. These 3,343
companies are not chosen using any selection criteria but ESG ratings are only
available for these US companies via Refinitiv.

Three different ratings are provided by Refinitiv, a basic ESG score, an ESG
controversies scores and a combined or controversy adjusted ESG score. All
three ratings can have values between 0 to 100, the higher the better the
company’s ESG performance. The basic ESG score mainly relies on information
published in annual and CSR / ESG reports of the company. They defined 10 main
themes, which are grouped into the three dimensions of ESG and are measured by
186 metrics:

Environmental: resource use, emissions, innovation

Social: workforce, human rights, community, product responsibility

Governance: management, shareholders, CSR strategy

The ESG controversies score is based on information from global media and is
the lower the more negative and controversial news about the company are
identified. The combined ESG score, represents the basic score, adjusted by
controversies. The lower the controversy score the higher the downward
adjustment of the combined score (RefinitivMethodology).

In the proposed model, ESG ratings will be predicted based on information from
news articles. Therefore the ESG combined scores are selected for comparison
of predicted values. The combined scores incorporate both, news signals and
overall considerations of ESG performance, providing a ’complete’ picture of
companies’ ESG activities.

Additionally to the ratings itself, the long and short version of the company
name as well as the market capitalization is derived from Refinitiv for each
company. Variants of the company name are used to filter for matching news
articles, which will be explained in the following paragraphs. The market
capitalization 111Market capitalization is calculated by multiplying the price
of a companies’ stock by the total number of outstanding shares. is referred
to as company size in the financial industry. It will be used in Section 5 to
analyse whether the precision of predictions varies with respect to company
size. Companies can be divided into three groups, small caps, mid caps and
large caps based on their market capitalization. Even though no official
thresholds exist, most common companies with a market capitalization greater
than $10 billion are considered as large caps, with values between $2 to $10
billion as mid caps and with values below $2 billion as small caps
(MarketCap_def).

To predict ESG ratings from news articles, a large dataset of news covering
ESG-related topics is necessary. With the goal to evaluate the approach for
3,343 US companies, the most promising data source is provided by the GDELT
project (cf. GDELT). It is a popular database, used by researchers across
various disciplines (cf. azhar2019text; wu2017forecasting).

The GDELT project provides a vast database covering a wide range of news
(print, web, broadcast, television) from most countries around the world in
more than 100 different languages. It is updated every 15 minutes. The Event
Database and the Global Knowledge Graph (GKG) are two examples of preprocessed
data sets available. For example, the GDELT Event Database categorizes news
into events and provides apart from the event identified (e.g. protest) and
the corresponding source more than 50 further attributes, like actors and
locations involved (cf. GDELT). There are various ways to access the data. The
raw datafiles can be downloaded from the GDELT website. This would be
extremely resource intensive as the files for one year exceed two terabytes.
The data sets are available in Google BiqQuery too, where it is possible to
query these and download only relevant data subsets. Analyzing the different
data sets in Google BiqQuery showed that collecting the data for the analysis
therefrom is not feasible either. Two issues were faced, first querying the
data per company did not provide satisfying results. For example, the event
database does not identify a certain company as ”Actor” for a certain event
but rather defined ”Actors” more general e.g. ”Business”. Second, even if a
way to effectively query the databases for a certain company could be
identified, this query would have to be executed manually for each of the
3,000 companies. This is related to a limitation in query size, a combined
query for various companies would exceed the free limit. Therefore the GDELT
Doc 2.0 API (cf. GDELTDocAPI) is used, which provides the option to use full
text search to identify news relevant for the desired topic from January 2017
on. More specifically, the most promising results for the proposed analysis
are retrieved using the GDELT 2.0 Doc API Client for
python222https://github.com/alex9smith/gdelt-doc-api. It is available via PyPi
and can be installed as python package. Using the GDELT Doc API Client, a list
of the top matching 250 articles for the filters and the time frame defined
can be retrieved. Different filters are available:

Start and end date: define the time frame.

Keyword: only articles containing the exact keyword entered in the filter are
returned.

Domain: defines the oulet(s) or webpage(s) (e.g. ”bbc.com”) of interest.

Country: search for news from specified countries.

Theme: results can be filtered for any of the GKG themes (e.g. ”terror”)333The
Global Knowledge Graph mentioned above, matches certain events and topics to a
theme. A list of themes is provided at the GDELT website:
http://data.gdeltproject.org/api/v2/guides/LOOKUP-GKGTHEMES.TXT..

Near: only articles in which the specified keywords (two or more) appear
within the defined distance, with respect to the number of words between them
(e.g. 10), are returned.

Repeat: only articles which mention a defined keyword (e.g. ”PayPal”) at least
a defined number of times (e.g. 2x) are returned.

Exploring these filter options, the goal is to derive the best matching
article data set, with respect to ESG-relevance, for each company. Testing the
different filter combinations revealed that the most promising combination is
the following:

Start and end date: first and last day of the respective month

Keywords: company name short, company name long, ”environmental”, ”social”,
”governance”

Repeat: 2x one-word representation of company name

The timeframe (start and end date) is set to one month, hence the top 250
matches per month for the defined filter criteria will be included in the
dataset. Choosing 250 articles per month provided the most promising results.
Explorative analysis showed that shorter timeframes like days or weeks
increase the number of unrelated articles in the match list disproportionate.
As a result, the initial dataset for each company contains at most 3,000
articles per year. It can contain less articles in case less matching articles
are found when querying the GDELT database. Therefore the number of articles
retrieved varies for different timeframes and companies. The chosen keyword
combination includes the three dimensions of ESG as well as a short and a long
version of the company name. For the example of the company Visa, ’Visa
Corporation’ would be the long and ’Visa’ the short name. This ensures that no
articles are missed due to different naming practices, some rather use an
abbreviation while others mention the complete name of a company. The third
filtering option used is the repeat filter. Requiring that the name of the
company is mentioned at least two times in each matching article, excludes
articles which mention the company only in passing. A similar approach is
applied by borms2021semi to ensure relevance of the articles. The repeat
filter technically only works using a single word, no combinations of two
words are allowed. Thus for some companies the most unique one-word
representation needs to be derived whereas for others like ’Paypal’ or
’Amazon’ the short company name consists only of one word already. For example
in the case of Coca Cola, ’Coca’ is chosen as one-word representation because
’Cola’ is often used for the drink ’Cola’ by other companies too.

The GDELT database does not return the articles itself when querying it but
some details about it. The following information is returned for each article:
URL, mobile URL, title, date and time the article was added to the GDELT
database, social image belonging to the article, domain, language and source
country.

In order to analyze the articles with respect to ESG topics using NLP, the
content of each article is necessary. Based on the list of URLs returned by
querying GDELT, this will be achieved using web crawling. Python offers the
package beautiful
soup444https://www.crummy.com/software/BeautifulSoup/bs4/doc/ to access, parse
and pull data out of HTML files. Thereby the title and the content of the
articles can be retrieved. However, not all articles can be accessed in this
manner, some webpages block web crawling, some require a user account and
other articles might not be available.

The above approach is used to create a comprehensive data set for the initial
set of 3,343 US companies. Ratings will be predicted for the years 2018, 2019
and 2020. Even though the GDELT database allows queries back to January 2017,
the dataset created for the year 2017 by webs crawling is not sufficient. The
main issue is the availability of articles, many articles listed in the GDELT
query results are no more available via the link provided.

####  4.1.2 Dataset Preparation

Deriving news articles from various different webpages using webscrapping has
the disadvantage that the results are rather noisy. The paragraphs drawn from
the raw HTML texts might include apart from the articles text, noisy
information like disclaimers, image captions and similar. Therefore the
content derived needs to be cleaned upfront, to ensure the input can be
handled appropriately by the NLP algorithms applied later on.

In a first step all articles with a length of less than 200 characters are
deleted. These are ’bad requests’, hence articles for which the webpage
requested returned an error message instead of the content. To ensure that no
articles, which only contain an error message remain in the dataset,
additionally a list of common error messages is created. The article’s content
is screened for these, deleting all articles which contain any error message.
Further all companies are dropped, for which not at least 5 articles could be
derived. This filtering step is applied to ensure the resulting rating is
based on a representative set of articles instead of one-sided information of
one or two articles (cf. borms2021semi).

The next step of cleaning is applied at paragraph level. The original HTML
texts were scrapped paragraph wise, by searching the HTML text for all ’p’
tags and adding these together as the article’s content. Hence some paragraphs
might be irrelevant, for example image captions or references to other
articles. To detect these, it is assumed that no variation of the companies
name is mentioned in irrelevant paragraphs. Thus the articles are screened per
paragraph, keeping only those which mention the name (either oneword, short or
long version) of the respective corporation at least once. The cleaning on
paragraph and article level, with the goal to retrieve the most important
information, is completed after this step.

The final cleaning step removes further noise and unwanted characters,
resulting from scrapping, within the articles content. All links, e-mail
addresses, date and time representations as well as Unicode characters and new
line representations are removed.

Common data preprocessing steps when preparing text data for the application
of machine learning algorithms are stemming, lemmatization and stops word
removal. However, in the case of the transformer model BERT, which is the
chosen algorithm to transform text into word embeddings, these steps are not
required. Byte-pair encoding is applied by BERT, which encodes words like
’destroying’ to ’destroy’ and ’##ing’. Hence valuable information would be
lost if verbs and nouns would be converted to their base form. Further, the
attention mechanism exploited by BERT does not focus on words which are common
in all data when finetuning. It rather considers the ones which will impact
the result, therefore stop word removal is not necessary.

No lowercasing of input text is conducted because the cased version of BERT is
used, which takes cased texts as input. Case information can contain valuable
information in this context. Company names start with an upper case letter and
for example ’Apple’ is associated with the company while with the lower cased
version the fruit ’apple’ is meant.

The BERT model takes a maximum number of 512 tokens per input sequence. With
an average length of 6,170 tokens per article after cleaning, it is not
possible to use the whole articles as input. Different options are available
on how to work with long texts and BERT. Some apply for example a sliding
window approach, however, thereof resulting long-range attentions can be
insufficient (ding2020cogltx). Another common approach is to keep only the
title and the first few sentences, which concatenated are treated as a summary
of the text. This approach is applied because it provides a feasible
representation in case of news articles. More specific, the title as well as
the first 5 sentences build the summary of each news article while the rest of
the article’s content is dropped. The resulting summary representations of the
articles have an average length of 622 tokens.

####  4.1.3 Dataset Description

Table 2 provides an overview of the article dataset before and after cleaning
and conversion into summaries. Initially 5,132 thousand articles could be
derived for the period from 2018 to 2020, using the GDELT database and
webcrawling. For example, the dataset for the year 2020 comprises 1,485
thousand articles for 3,059 US companies. On average 485 articles are derived
per company in 2020 whereas for the years 2018 and 2019 more articles were
derived per company, about 585 on average. The GDELT database collects
information of news from all around the world and thousands of different
sources. This is reflected in the dataset, the articles were derived from
30.966 different domains in 199 countries. As expected for US American
companies, the majority of news articles (72%) collected originate from the
United States. News out of the United Kingdom, India and Canada represent 7%,
3.4% and 2.5% of total articles, respectively. The top 7 source countries,
which comprise additionally to the mentioned ones Australia, Germany and
China, account for 90% of all articles. Looking at the distribution among
domains no major source can be identified, with the two largest domains,
prnewswire.com and news.yahoo.com, accounting for 7% and 4% of total articles,
respectively. Other source domains among the top 10, which account for 35% of
total articles, are for example: finanznachrichten.de, forbes.com and
dailymail.co.uk.

Table 2: Overview News Article Dataset

| Scrapped Data |  | Prepared Data
---|---|---|---
| 2018 | 2019 | 2020 |  | 2018 | 2019 | 2020
# of companies | 3,114 | 3,207 | 3,059 |  | 2,665 | 2,940 | 2,879
# of articles | 1,823,769 | 1,822,233 | 1,484,685 |  | 1,165,840 | 1,399,277 | 1,174,754
# of articles per company |  |  |  |  |  |  |
average | 586 | 585 | 485 |  | 435 | 473 | 406
min | 5 | 5 | 5 |  | 5 | 5 | 5
max | 2,624 | 2,796 | 2,835 |  | 1,984 | 2,154 | 2,005

After cleaning the data as described in Section 3.1.2, the dataset comprises
in total 3,725 thousand articles. 1,174 thousand articles for 2,879 companies
for the year 2020, 1,392 thousand articles for 2,940 companies and 1,159
thousand for 2,665 companies for the years 2019 and 2018, respectively. In
2018 449, in 2019 267 and in 2020 180 companies had to be dropped from the
dataset due to the following two reasons. Either because no Asset4 ESG score
is available for the company in the respective year or the companies set of
articles was smaller than 5 after removing noise and error messages. These
cleaning steps further reduced the average number of articles per company by
around 100 articles on average. It is reasonable that the number of companies
that had to be dropped is the highest for the year 2018 because the
probability of articles being no more available on the webpages of the news
providers increases the longer ago the articles was published.

Table 3: Overview of Asset4 ESG Scores

| Count | Average Score
---|---|---
Companies | 3,343 |
Companies | 3,343 |
Market Capitalization | 3,309 |
Scores 2018 | 2,842 | 35
Scores 2019 | 3,078 | 37
Scores 2020 | 3,075 | 38

Table 3 gives an overview of the number of Asset4 ESG scores that could be
derived from Refinitiv. Filtering for US companies, 3343 companies with ESG
data are returned. However, not for all companies scores are available in each
of the three years of interest. Additionally, the market capitalization value
is available for 3309 out of 3343 companies in the dataset. The average ESG
score of all companies is quite similar over the three years, ranging from 35
in 2018 to 38 in 2020.

###  4.2 Methodology

The aim of the proposed thesis is to examine whether meaningful ESG ratings
can be predicted without human intervention from news articles exploiting
Natural Language Processing. Therefore, the dataset needs to be analyzed to
derive meaningful input features for the prediction of the rating. In the
first step, articles are classified into ESG relevant and irrelevant articles.
Deriving the sentiment, separately for relevant and irrelevant articles per
company, is the second step. In third step, articles are clustered with
respect to their semantic similarity. Thereby, 8 timeseries are created, the
Article2ESG Relevance timeseries, the Article Sentiment timeseries as well as
6 semantic timeseries (one per cluster). These timeseries are created
separately for each company and each year. How these timeseries are
constructed in detail will be described in the following section. To predict
the news based ESG ratings, different neural network configurations, including
the Transformer architecture and convolutional layers, are applied to model
the input timeseries. An overview of the analysis steps is provided in Figure
2.

Figure 2: Overview ESG Prediction Model

####  4.2.1 Derive Model Inputs

Text Classification

To determine whether an article contains relevant information about companies
ESG performance and activities is a crucial step of the analysis. For the
rating prediction only ESG relevant information should be considered. The idea
is to solve the task of classifying exploiting BERT, a state-of-the-art NLP
algorithm. In order to fine-tune the BERT classifier for the specific task of
ESG classification, a set of labeled input data is necessary. As mention in
Section 2.1, clearly defining ESG relevance is rather difficult. This
represents a challenge for the proposed method as the dataset generated is
unlabeled, i.e. articles are not yet identified as ESG relevant or irrelevant.

Labeling a few thousand articles by hand would be very time-consuming and thus
is not feasible. However, to ensure the quality of the proposed model and the
predicted labels, a large training set is necessary. To solve this issue
efficiently the concept of weak supervision is applied. Weak supervision
refers to generating labeled training data programmatically instead of
labeling the data manually. This implies a two step process to classify the
articles with respect to their topical relevance. In a first step, a subset of
the data is pre-labeled by determining the articles’ semantic similarity to
texts defining each of the three ESG dimensions. The ESG definitions of the
Corporate Finance Institute 555The CFI is a highly regarded institution in the
financial profession providing for example trainings and certification
programs. The definitions can be found on their webpage:
https://corporatefinanceinstitute.com/resources/knowledge/other/esg-
environmental-social-governance/ are used therefore. The definitions of each
of the three dimensions, environmental, social and governance, are provided by
CFI paragraph wise, with 2, 10 and 5 paragraphs respectively. The most
intuitive might be to take the descriptions provided by Refinitiv for each of
their 10 category scores. However, these rather explain what is measured by
the respective score and are less likely to match the ESG information
mentioned in news and their semantics.

The news article subset, which is pre-labeled and used to train, validate and
test the classifier in the next step, is build from articles of 100 randomly
sampled companies. Pretrained text embeddings for the ESG definition sequences
and the news articles are constructed exploiting the sentence transformer
model (SBERT) provided by Huggingface (wolf2019huggingface;
reimers2019sentence) (see Section 3.3). More specific, the ’all-
distilroberta-v1’ sentence transformer is used, which maps the text sequences
to a 768 dimensional dense vector space 666Documentation:
https://www.sbert.net/docs/pretrained_models.html. This version of SBERT is
chosen due to resource constraints, it is significantly faster than the base
version of SBERT while comparable performance is provided for sentence
embeddings. The labeling of the data as ESG relevant or irrelevant is
conducted by calculating the pairwise cosine similarity between an article’s
content and each of the 10 ESG category descriptions. With possible cosine
similarity values between -1 to 1, all articles with a a similarity value
above 0.1 are labeled as relevant, all other articles are labeled as
irrelevant. This rather low threshold is chosen due to two reasons. First, the
ESG definitions are very general, hence setting the threshold too high would
be too selective. Especially in case of ESG, which is a very wide-reaching
topic. Second, different threshold are tested and 0.1 provides a distribution
of roughly 54% relevant and 46% noise while higher thresholds diminish the
number of relevant articles significantly. For example choosing 0.2 as
threshold, leads to a distribution of 17% relevant to 83% noise. A
distribution around 50:50 seems reasonable given the fact that the article set
per company was already selected by querying GDELT for the keywords
environmental, social and governance and therefore a certain topical relevance
can be assumed.

The second step of the process comprises fine-tuning a pretrained classifier
and predicting the labels for the whole dataset using the resulting model.
Different versions of BERT are available. For the given setting, the
pretrained general-purpose model DistilBERT is used. The model proposed by
sanh2019distilbert is available via HuggingFace and the Python transformers
library (wolf2019huggingface). DistilBERT is, as explained in Section 3.3., a
distilled version of the BERT model, which provides good performance while
being signigicantly faster. A general purpose model is chosen, to be able to
add a customized classification layer. The model implemented contains the
following layers:

Figure 3: Overview Layers Classification Model

Before the summaries of the news articles can be feed into the model, they
need to be tokenized and the two special tokens ’CLS’ (classifier) and ’SEP’
(seperator) have to be added to the text sequences. While the ’SEP’ token
signals the model the end of an input sequence, the ’CLS’ token is necessary
for classification tasks (devlin2018bert). Input sequences have to be of equal
length, therefore summaries shorter than the maximum length of 512 are padded
to the required length by adding zeros. Summaries longer than 512 are not
accepted by the model and thus need to be truncated, words at the end of the
sequence are dropped. To tokenize the articles’ summaries the
DistilBertTokenizer provided by Huggingface 777Documentation:
https://huggingface.co/docs/transformers/model_doc/distilbert is used
(wolf2019huggingface). It adds the special tokens automatically as well as
returns the input ids and attention mask for each summary. The tokenized
inputs are then feed into the DistilBERT model to derive the embedding
vectors. The applied default configuration of DistilBERT is based on 6 hidden
Transformer encoder layers with 12 attention heads each. The last hidden state
of the DistilBERT model outputs a 3-dimensiol tensor of size $[N,512,768]$.
With N equal to the number of input sequences, 512 equal to the number of
tokens per sequence and 768 to the number of hidden units. Since the objective
is to classify the summaries as ESG relevant or irrelevant, the last hidden
state of the DistilBERT model corresponding to the classifier token ’CLS’, is
selected as input to the following layers. It is represented by the first
position of the token sequence $[:,0,:]$ and can be seen as an aggregate
representation of the sequence (devlin2018bert). The derived 2-dimensional
vector is further processed by a dense layer of size 265 and a following
dropout layer with dropout rate 0.5. The dropout layer is implemented to
prevent overfitting. The rate of 0.5 indicates that a random selection of 50%
of input units is set to 0. The final dense layer of size 2 uses the softmax
activation function. It returns a list of two probabilities
$[p_{Noise},p_{ESG}]$ that represent the probability of an article belonging
to the respective category. This model architecture is inspired by
sokolov2021building, comparing it to other architectures adding for example an
additional Dense layer or a bidirectional LSTM layer between the embedding and
the dense layer did not improve the results.

The constructed DistilBERT classifier is fine-tuned in two steps. In the first
step only the customized classification layers are optimized, while in the
second step the whole network is fine-tuned using a lower learning rate (cf.
sokolov2021building; sokolov2021weak). The selected model parameters,
including early stopping criteria are presented in Table 4. Early stopping as
well as criteria for learning rate reduction are added in order to stop
training in case of overfitting. While the learning rate is reduced if five
consecutive training steps result in no improvement with respect to the
validation loss, training is stopped immediately if a whole epoch of training
does not reduce the validation loss. Both training steps are run for 4 epochs,
however, the second stage is stopped after 2 epochs for most model
configurations.

Table 4: Selected Training Parameters for the Classification Model

Parameter | Definition | Value
---|---|---
Learning Rate - Step 1 | Learning rate to fine-tune classifcation layer | 10-3
Learning Rate - Step 2 | Learning rate to fine-tune whole model | 10-4
Early Stopping Criteria | Training epochs without improvement before stopping | 1
Learning Rate Reduction Criteria | Training steps without improvement before reducing the learning rate by a factor of 0.1 | 5
Maximum Sequence Length | Maximum input sequence lenght for DistilBERT | 512
Batch Size | Batch size used for training and validation | 32
Loss Funcion | Model optimization based on minimizing: Sparse categorical cross-entropy loss
Evaluation Metric | Model performance evaluation metric: Accuracy

To mitigate the issue of class imbalance, stratified sampling is applied to
split the prelabeled dataset into train, validation and test data. Stratified
sampling refers to sampling group wise, hence the relative percentage of the
classes in the training, validation and test split is the same as for the
whole prelabeled article set.

Using the trained classification model, each article’s label is predicted. As
the model does not return the label but the categories probabilities, these
first need to be converted. Each article receives the label of the class with
the highest predicted probability. Additionally to the probabilities, the
trained embedding vectors of the articles’ summaries are returned. These will
be used for the further analysis.

The Article2ESG Relevance timeseries, which represent the results of the
classification analysis and will be used for the final prediction, is
calculated as follows. First the articles are grouped by month and company.
Having a set of articles per company and month, the monthly value for
Article2ESG Relevance is calculated according to the following formula:

$\displaystyle rel-noise_{m}=(Relevant_{m}/N_{m})-(Noise_{m}/N_{m})$ (1)

The monthly relevance to noise ratio, rel-noisem, is not a real ratio but
represented by the subtraction of the monthly noise from the monthly relevance
ratio. Relevantm is the number of relevant articles counted for the selected
company in the considered month m. Noisem represents the number of noise
articles respectively. Nm is total number of articles counted in month m. The
difference instead of the ratio is taken to mind issues of the noise ratio
being zero. If in a certain month m no noise articles are counted, the term
(Noisem / Nm) would be zero. However, mathematically dividing by zero is not
possible. Applying subtraction leads to a positive rel-noisem if the
percentage of ESG relevant articles in month m is larger than the percentage
of noise articles and to a negative value otherwise. The values range between
-1 (only noise articles) to 1 (only relevant articles). As a result, a
companies Article2ESG Relevance Timeseries contains 12 rel-noisem values, one
for each month of the considered year.

Sentiment Analysis

In a second step the sentiment of the labeled ESG relevant articles needs to
analysed. The sentiment of an article’s content, either positive or negative,
serves as indicator for the ESG performance. An article with positive
sentiment is treated as a sign for good ESG performance whereas an article
with negative sentiment is treated as a sign for the company facing an ESG
issue. The assumption that more positive news should lead to a higher ESG
rating while negative news should lead to a downgrade is self-explanatory.

Due to the fact that the article dataset is not prelabeled with any sentiment
label, the SieBERT model (heitmann2020more) (cf. Section 3.3) will be used to
predict the articles sentiment. It is available for use in Phyton via Hugging
Face’s sentiment analysis
pipeline888https://huggingface.co/docs/transformers/main_classes/pipelines.
This model was chosen as best fitting for evaluating a dataset of news in
English language, as explained in Section 3.3. Regardless thereof, an
additional validation of the models prediction is done manually by inserting
some simple ESG related statements to the model to verify the output is as
expected. For example, the following two statements were tested:

  * •

’The company kicked off a new employee benefit program.’
Probability: positive 0.998

  * •

’The company’s emission were increasing over the last year.’
Probability: negative 0.996

Running the SieBERT model, the sentiment label of each article’s summary is
returned, either positive (1) or negative (0), as well as the probability
score of the label. The label returned by the model is used to generate the
monthly Article Sentiment Timeseries per company. The sentiment timeseries is
calculated separately for the ESG relevant and irrelevant articles. Therefore,
the corpus is split into an ESG corpus, containing all articles which were
classified as ESG relevant in the first analysis step, and a noise corpus,
containing all articles classified as irrelevant or noise. The steps explained
in the following, are applied to both, the ESG and the noise corpus
separately, whereby two sentiment timeseries per company and year are derived.
The articles of each corpus are grouped by company and month, in order to
calculate a monthly sentiment ratio.

$\displaystyle pos-neg_{m}=(Positive_{m}/N_{m})-(Negative_{m}/N_{m})$ (2)

Similar to the rel-noisem ratio (cf. Equation (1)), the pos-negm ratio is not
a real ratio but calcuateld by deducting the negative ratio from the positive
ratio. The same reasoning applies. The positive and negative ratio in month m
is calculated by dividing the number of articles counted for the respective
sentiment (Positivem or Negativem) by the total number of articles (Nm) in the
corpus for a specific company in month m. The calculated monthly pos-negm
values represent whether the companies ESG performance was improving, in case
of a positive ratio or worsened, due to upcoming issues or similar, in case of
a negative ratio. As a results, the Article Sentiment Timeseries is build by
concatenating the monthly values for each company to a timeseries containing
12 values for the respective year.

Semantic Analysis

The third step of the proposed model, the semantic analysis, aims at grouping
the articles with respect to their semantics in order to identify the
evolution of topics discussed in the news. As outlined in Section 3.3, BERT
embeddings in combination with the k-means algorithm will be applied to
cluster the articles’ summaries.

In the first analysis step, the DistilBERT algorithm is fine-tuned on the task
of ESG classification. Instead of only predicting the label of each article,
additionally the fine-tuned embeddings of the summaries are returned by the
model. Thereby these task specific embeddings can be used as input to the
proposed clustering algorithm. The k-means clustering algorithm partitions the
dataset into k clusters by minimizing the following objective function J.

$\displaystyle J=\sum_{n=1}^{N}\sum_{k=1}^{K}r_{nk}\parallel
x_{n}-u_{k}\parallel\textsuperscript{2}$ (3)

To minimize the value of J, fitting values for rnk and uk need to be
identified. This is done by iteratively optimizing in the first step rnk and
in the second step uk until either a certain threshold or the defined maximum
number of iterations is reached. N represents the number of samples in the
dataset while K represents the number of clusters and has to be defined
upfront. The term rnk is either 0 or 1, depending on whether sample xn is a
member of cluster k (1) or not (0). The centroids of the clusters are
represented by uk and are initally set random. During the iterations each
sample xn is assigned to the closest centroid uk (cf. Equation (4)) in the
first step. In the second step, the centroids uk are moved so that the
distance between uk and its points xn is minimized (cf. Equation (5)).

$r_{nk}=\begin{cases}1&k=\text{ arg min\textsubscript{k} }\parallel
x_{n}-\mu_{k}\parallel\textsuperscript{2}\\\ 0&\text{ others }\par\end{cases}$
(4)

$\displaystyle u_{k}=\frac{\sum_{n=1}^{N}r_{nk}x_{n}}{\sum_{n=1}^{N}r_{nk}}$
(5)

Additionally to the above explained basic k-means algorithm, the spherical
k-means algorithm will be applied . It is specifically proposed for document
clustering with BERT to better account for semantic similarity (cf. Section
3.3). Therefore both algorithms and their impact on the final prediction will
be compared. Instead of minimizing the euclidean distance between two vectors,
the angle between the vectors is used to measure the similarity. The objective
of spherical k-means thus is to minimize the cosine dissimilarity between the
centroid uk and its points xn (cf. Equation 6).

$\displaystyle\sum_{n}(1-cos(u_{k},x_{n}))$ (6)

Due to the fact that the number of clusters K needs to be defined as input
parameter to the k-means algorithm, the elbow method is applied to identify an
appropriate number of clusters K. Figure 4 shows the elbow curve for the
standard k-means algorithm based on euclidean distance and the spherical
k-means algorithm based on cosine similarity as distance measure,
respectively.

(a) Standard K-means

(b) Spherical K-means

Figure 4: Elbow Curves

Looking at the graph depicted in 4(a), it seems that the benefit of adding
another cluster is proportionately smaller for values of k above 6. Hence the
’elbow’ is identified at k equal to 6, which is chosen as the number of
clusters K for the analysis. The same pattern can be identified in 4(b), here
K equal to 6 seems to be an appropriate number of clusters too. Utilizing the
fine-tuned embeddings and K equal to 6 as input, both algorithms, k-means and
spherical kmeans, are applied. As a result, each articles’ summary is assigned
to one out of 6 clusters.

To be able to include the results of the semantic analysis in the final
prediction of the ESG rating, one timeseries per cluster will be created.
Hence in total 6 Article Semantic timeseries will be created per company
according to the following steps. First, the dataset is grouped by cluster to
derive the total set of articles belonging to each cluster. In the next step
each cluster subset is grouped by company to derive the overall size of each
cluster per company. The monthly values of the timeseries of cluster K are
calculated by counting the number of articles per month belonging to the
respective cluster K. Hence the value of the timeseries for cluster K in month
m for a company with N articles refers to:

$\displaystyle size_{Km}=\sum_{n=1}^{N}\text{summaries\textsubscript{m}}$ (7)

The 12 monthly values per company and cluster are concatenated, to derive 6
Article Semantic timeseries per company. Thereby the monthly size of each of
the K clusters identified is tracked, representative for the main topics
identified and how dominant these are for a respective company.

####  4.2.2 Rating Prediction

The aim of the model is to predict ESG ratings, thus in the last step the ESG
ratings are predicted using the derived time series as input. Different
convolutional neural network and Transformer architectures will be compared to
identify the best fitting model. The following section describes the
constructed models in detail, including a basic CNN, a CNN with several
convolutional layer as well as a combination of CNN and Transformer layers.

The first and simplest model applied is a convolutional neural network,
consisting of the following layers:

Figure 5: Basic CNN Model

The eight constructed input time series, the Article2ESG relevance, the
Article Sentiment as well as the six Article Semantic time series (one per
cluster), constitute the input to the model. Before being feed into the model,
the timeseries are standardized using standard scaling, which refers to
deducting the mean of the values and diving it by the standard deviation. The
resulting standardized timeseries are then processed by a convolutional 1D
layer, followed by a maxpooling 1D layer. The convolutional layer applies
filters of defined size to the input matrix to retrieve a representative
feature matrix thereof. The maxpooling layer serves the purpose of
dimensionality reduction. As the name indicates, it returns a pooled feature
map, consisting of the the maximum value of each feature map to highlight the
most present features. The resulting output of these two layers is then
further processed by a flatten layer, which reshapes the output per sample
from two dimensional to one dimensional. Additionally a dense layer with a
ReLu activation function (cf. Equation 9) is added to introduce some non-
linearity to the model (chollet2021deep).

$ReLu(x)=\max(0,x)=\begin{cases}0&\text{ if }x<0\\\ x&\text{ if }\x\geq
0\par\end{cases}$ (8)

In the final classification layer, the softmax function is applied to return
the probability of each of the n classes. For each sample, the class with the
maximum probability is selected as final ESG rating.

Additionally to this rather simple convolutional neural network comprising
only a single convolutional layer, a deeper CNN is introduced to the input
timeseries. It includes three convolutional blocks as illustrated in Figure 6,
shown below.

Figure 6: Deep CNN Model

These three blocks are build of three layers each, a convolutional layer, a
batch normalization layer and a ReLu activation. This represents a common
architecture for convolutional blocks, applied for example by wang2021event.
Adding a batch normalization layer serves to standardize the layers’ input.
Several comparisons show that this helps to speed up the models training
(chollet2021deep). The three convolutional blocks are followed by a global max
pooling layer. Compared to the maxpooling applied in the first CNN version,
which returns one value per feature, the global max pooling takes the entire
feature map and returns a single value (i.e. the maximum value). Thereby the
global max pooling layer reduces the dimensionality so that no additional
flattening of the output is needed before being feed into the final
classification layer.

The chosen hyperparameters, which yielded the best results for the two CNN
models, are displayed in Table 5. While the basic CNN includes only one
convolutional layer with filter size 64 and kernel size 2, the deep CNN
comprises three convolutional layers with increasing filter size, from 32 to
128, and decreasing kernel size from 3 to 1.

Table 5: CNN Model Parameters

| Basic CNN | Deep CNN
---|---|---
Parameter | Block 1 | Block 1 | Block 2 | Block 3
Convolutional 1D layer |  |  |  |
Filter Size | 64 | 32 | 64 | 128
Kernel Size | 2 | 3 | 2 | 1
Padding | same | same | same | same
Dense Layer: Relu Activation | 265 | - | - | -

Besides convolutional neural networks, recently the Transformer architecture
is leveraged for time series classification and regression. Therefore two more
models are introduced, a basic model including only a single Transformer layer
and a deeper model with a multi layer Transformer encoder block.

Figure 7: CNN including single Transformer Layer

The basic model, which includes two convolutional blocks as well as a single
Transformer layer, is presented in Figure 7. The transformer encoder block,
consisting of a multi head attention, a dropout and two normalization layers,
is highlighted in gray. The layer normalization helps to stabilize the model.
Normalization is applied to the input, which refers to the output of the
second convolutional block, as well as to the output of the attention block.
The central layer is the multi-head attention layer, which applies the self-
attention mechanism as proposed by (vaswani2017attention). Further a dropout
layer is added after the attention layer, like in the original transformer
architecture. Like for the deep CNN, the output is transformed into class
predictions by applying global maxpooling and the softmax function. The deep
or multi layer transformer model, has a similar architecture as shown in
Figure 7. The difference is that the single Transformer block is replaced by
three consecutive transformer encoder blocks. Thereby a similar architecture
as proposed by wang2021event is leveraged. The transformer blocks are arranged
consecutively, hence the output of the previous transformer blocks is feed
into the following one.