date04022017

# Big Data Information and Nowcasting: Consumption and Investment from Bank
Transactions in Turkey

###### Abstract

We use the aggregate information from individual-to-firm and firm-to-firm in
Garanti BBVA Bank transactions to mimic domestic private demand. Particularly,
we replicate the quarterly national accounts aggregate consumption and
investment (gross fixed capital formation) and its bigger components
(Machinery and Equipment and Construction) in real time for the case of
Turkey. In order to validate the usefulness of the information derived from
these indicators we test the nowcasting ability of both indicators to nowcast
the Turkish GDP using different nowcasting models. The results are successful
and confirm the usefulness of Consumption and Investment Banking transactions
for nowcasting purposes. The value of the Big data information is more
relevant at the beginning of the nowcasting process, when the traditional hard
data information is scarce. This makes this information specially relevant for
those countries where statistical release lags are longer like the Emerging
Markets.

Ali B. Barlas (BBVA Research), Seda Guler Mert (BBVA Research), Berk Orkun Isa
(BBVA Research) Alvaro Ortiz (BBVA Research), Tomasa Rodrigo (BBVA Research),
Baris Soybilgen (Bilgi University) and Ege Yazgan (Bilgi University)

Keywords: Big Data; Dynamic Factor Model; BVAR; Machine Learning, Nowcasting.

##  1 Introduction

The economists normally use the information produced by National Statistical
Agencies or Central Banks (GDP, industrial production, unemployment, etc.) to
assess the state of the business cycle. While this information is consistently
designed to track the business cycle, it also has some shortcomings. One of
the important problems is that most of the key indicators are low frequency
and released with some time lag. In the case of some countries, the lag in
statistical releases can be considerable.

While there is some economic information available at high frequency (i.e
stock market prices, interest rates…) it is normally related to financial
conditions and expectations, which do not necessarily match the real
conditions of the economy. The need to react rapidly to the changing economic
conditions after the COVID-19 crisis has enhanced efforts to follow the
economy in ”real time” in several lines of analysis:

  * •

Focusing on alternative high frequency indicators: Some analysts have turned
their attention to the more advanced released indicators such as the soft data
surveys (i.e. Purchasing Manager Indexes or PMIs, Consumer confidence, etc.)
and other high frequency indicators like electricity production, chain store
sales released on a daily or weekly basis, respectively.

  * •

Developing higher frequency models: Central Banks have relied on the use of
traditional nowcasting methods but mixing quarterly or monthly variables with
higher frequency indicators (i.e. weekly, daily) to better capture the real
time component information. This has been the case of the Federal Reserve of
New York weekly economic index (Lewis et al., 2020), the Bundesbank Weekly
Activity Index (Eraslan and Götz, 2020) and the Central Bank of Portugal daily
economic index (Lourenco and Rua, 2020) among others.

  * •

Developing New Big Data Indicators: A new stream of work (Barlas et al., 2020;
Carvalho et al., 2020; Chetty et al., 2020) has focused on the use of
transaction data, company information or Google trends (Woloszko, 2020) to
capture the economic activity in real time. In this sense, the COVID-19
pandemic of 2020 has acted as a major stimulus for movement in this direction,
and in a short space of time an entire new literature has grown using these
indices.

In this working paper, we make several contributions. We extend the increasing
literature of electronic payments to an emerging economy as Turkey. We use the
information from individual-to-firm and firm-to-firm in Garanti BBVA111Garanti
BBVA is one of the top private deposits banks operating in Turkey with its
majority shareholder Banco Bilbao Vizcaya Argentaria (BBVA). BBVA is a
customer-centric global financial services group operating in several
countries Bank transactions to replicate the quarterly national accounts
aggregate consumption and investment (gross fixed capital formation) and its
bigger components (Machinery and Equipment and Construction). As a main
contribution to the existing literature, we extend the ability of banking
transactions to replicate investment by adding firm-to-firm-transactions to
the traditional individual-to-firm used for consumptions. While the literature
of Bank transactions to replicate consumption has been increasing rapidly, it
is difficult to find any empirical work using Bank Data transactions to
estimate investment flows. To our knowledge, only Barlas et al. (2020) has
focused in this analysis.222The real time investment information has several
advantages for analysts and policymakers. First, for the case of Turkey we
complete the domestic private demand by adding near a third of GDP to the
consumption share during the average of the last three years. Second,
investment is more volatile than consumption but with a special relevance in
the source of fluctuations in Developed and specially in Emerging Economies.
Third, some parts of investment as residential investment can have systemic
implications for the banking and financial systems, as the 2008-2009 crisis
revealed.

We also investigate the more efficient ways to introduce the Big Data
information in alternative nowcasting models in line with other recent works
(Babii et al., 2021). We use Dynamic Factor Models (DFM) and BVAR with some
Machine Learning models such as Linear Lasso regressions and non linear Random
Forest and Gradient Boost to test the out-of-sample nowcasting accuracy and
robustness of our Big Data proxies. We find that our big data proxies
significantly contribute the nowcasting performance. The contribution of the
big data information appears to be more relevant at the beginning of the
nowcasting process, when the traditional hard data information is relatively
scarce.

The increased availability of electronic payments data has spurred the recent
literature on real time economic activity, however most of the recent
empirical works have focused on Developed economies. In the US, Barnett et al.
(2016) derive an indicator-optimized augmented aggregator function over
monetary and credit card services using credit card transaction volumes. This
new indicator, inserted in a multivariate state space model, produces more
accurate nowcasting of the GDP compared to a benchmark model. Verbaan et al.
(2017) analyzes whether the use of debit card payments data improves the
accuracy of the nowcast and one quarter ahead forecast of Dutch private
household consumption. Baker et al. (2020b) and Olafsson and Pagel (2018),
which use data from ﬁnancial apps to track household spending and income.
Galbraith and Tkacz (2015) generate nowcast of the Canadian GDP and retail
sales using electronic payments data, including both debit card transaction
and cheques clearing through the banking system. In Portugal, Duarte et al.
(2017) obtain nowcast and one step ahead forecasts of Portuguese private
consumption by combining data from ATM and POS terminals. In Italy, Aprigliano
et al. (2017) check retail payment data to accurately forecast Italian GDP. In
Spain, Bodas et al. (2019) replicate a retail sales index through (PoS)
transactions data.

The COVID-19 pandemic of 2020 has acted as a major stimulus in this direction,
and in a short space of time an entire new literature has grown that uses
indices derived from transaction data to track the impact of virus spread and
lockdown. Again, most of the papers focused in the developed Economies.
Andersen et al. (2020) presented evidence of a sharp reduction of total card
spending during the early phase of the crisis in Denmark. Alexander (2020),
Baker et al. (2020a), Chetty et al. (2020), and Cox et al. (2020) focused in
the response of Card transactions to COVID mobility restrictions in the USA.
Bounie and Galbraith (2020) tracked detail responses of consumer transactions
in France. Chronopoulos et al. (2020) and Hacioglu et al. (2020) analyze the
response in UK. Only a couple of studies have extended this empirical work to
the Emerging Economies such as Barlas et al. (2020) for Turkey; Carvalho et
al. (2020) for Turkey, Mexico, Colombia, Peru, Argentina; Chen et al (2020)
for China. In this sense, we believe that our paper makes a significant
contribution to this literature by extending it to emerging markets, deriving
big data proxies for both consumption and investment flows and showing their
usefulness for nowcasting purposes.

The remainder of the paper is organized as follows. In the second section, we
describe the Big Data methodology to compute the private domestic demand for
the case of Turkey. Particularly, we describe how to mimic national account´s
consumption and investment demand from the firm-to-individuals and firm-to-
firm transactions included in Garanti BBVA Big Data. Once the indicators have
been developed and validated we describe their performance during the recent
times in Turkey. In the third and fourth sections we describe our nowcasting
methodology and check how the inclusion of the Big Data information
complements and adds extra information to the traditional nowcasting Dynamic
Factor Models, regularized and Machine Learning algorithms. Finally, in the
last section of the paper we conclude.

##  2 Consumption and Investment through a Banks’ Big Data: The role of
individual-to-firm and firm-to-firm transactions

The recent literature of using banking transactions to replicate the household
consumption has focused on the analysis of the Point of Sales (PoS) and the
credit and debit cards data. These transaction data can be obtained from the
anonymized information included in a Bank’s individual database (Carvalho et
al., 2020; Andersen et al., 2020) or from different sources compiled by some
specific companies (Chetty et al., 2020). In most of the works, individual´s
consumption is estimated through the individual-to-firm payments by PoS or
Card transactions exchanged by the goods and services consumed by households
while some authors such as Carvalho et al. (2021) have extended this
transactions to those covering the consumption of goods and services normally
paid through direct money transfers such as utilities, telephone and other
bills.

To extend investment demand expenditure we need to include firm-to-firm
transactions received by the firms that produce fixed assets as we explained
in Barlas et al. (2020). In this case, we assume that the money transfers from
firms to those firms manufacturing fixed assets are in exchange of these
investment goods. In the case of Dwellings investment, we need to include t
the individual-to-firm transactions reflecting the purchase of a house by
households.

The definition of gross fixed capital formation by the System of National
Accounts (SNA) is ”the resident producers’ net acquisitions (acquisitions
minus disposals) of fixed assets used in production for more than one year”.
The concept differentiates between: (1) dwellings; (2) other buildings and
structures, including major improvements to land; (3) machinery and equipment;
(4) weapons systems; (5) cultivated biological resources, e.g. trees and
livestock; (6) costs of ownership transfer on non-produced assets, like land,
contracts, leases and licenses; (7) R&D; (8) mineral exploration and
evaluation; (9) computer software and databases and (10) entertainment,
literary or artistic originals. Finally, these categories are grouped into
three big components: (1) Construction Investment; (2) Machinery & Equipment;
and (3) Other Investment including computer software, databases, research and
development, etc. In this paper, we track the first two components,
Construction and Machinery & Equipment, which in Turkey, accounts for nearly
90% of the total investment expenditures.

In this paper, we use the transactional database of Garanti BBVA, which
includes all monetary transactions between Garanti BBVA clients including
individuals and firms. In the case of consumption, we identify and filter out
the credit card and debit card transactions of the individuals. In this work,
we take into account inflows from individuals and firms to the firms producing
fixed investment assets in Machinery Investment and Construction (nearly 90%
of the total investment investment in the Turkish economy) and assume that
these transfers are in exchange of the demand for fixed capital formation.

###  2.1 Consumption through Individual-to-Firm transactions

In order to estimate the Big Data Consumption we use the Garanti BBVA
Database. During 2020, this database depicts 395.7 million filtered
transactions occurred between consumers and 1.67 million merchants spread
around Turkey. The information is geo-localized and include at least one
transaction data from every single city in Turkey. The aggregate nominal value
of those 395.7 transactions is equivalent to 299.9 billion TL (6% of
GDP)333Garanti BBVA Big Data database recorded roughly 469,000 daily
consumption transactions credit cards on average in 2019, whereas median
number of daily transactions was 478,000 in the same year. In 2020, average
daily number of transactions decrease to 373,000 with median of 409,000
transactions.

In order to compute the our Big Data Consumption Index, we include individual-
to-firm transactions including both goods and services sales. The Data
extraction process for these components involve a detailed processing and
filtering procedure. First, we only collects credit card and debit card
transactions data from certain channels on daily basis, namely (i) physical
transactions occurred at points-of-sale, (ii) e-commerce and (iii)
Mail/Telephone orders. Moreover, this filtering method only includes
transactions inside the 81 provinces in Turkey.

Once the information is extracted, the data is filtered to correct for
outliers and noise transactional data. Then we group the data by their
Merchant Category Codes (MCC) to classify the information into the
corresponding sectors (goods or services) and sub-sectors (airlines,
restaurants, tourism, etc.). We calculate year-on-year nominal growth rates of
(i) total sales transactions volume of goods, and (ii) total sales
transactions volume of services. Finally we compute the Real Big Data
Consumption index by deflating the nominal series with the Retail Sales
deflator and the Services Sub-Item of Consumer Price Index (CPI) released by
Turkish National Statistical office (TURKSTAT).

In order to limit the sample representative bias, we weight both indexes with
their respective time-varying shares in TURKSTAT Total Household Consumption
Data, and weighted growth rates of two indexes are combined in order to
construct the aggregate Garanti BBVA Consumption Index.

Figure 1: Big Data Consumption vs National Accounts (a) (March 2015 to March
2021,% YoY)

Source: Own Elaboration and Turkstat.

###  2.2 Including Firm-to-firm Transactions to track Investment

Our Big Data Investment Index is a synthetic indicator that aims to track
investment activity based on the daily money transfers or inflows (i.e. daily
purchases by means of payments) to the firms producing the fix assets.
Furthermore, the key assumption is that money transfers from all the firms and
individuals to those firms working in the investment-good producing sectors
are in exchange of the fixed assets they produce.

The total number of firms to monitor for investment expenditures in the BBVA
Big Data was nearly 367k during the year 2020. This includes 30.7k or 8% in
the Construction sector and 336.2K or 92% in the machinery group. The
aggregate number of transactions identified is 31.1 million which was worth of
1.8 trillion TL (36% of GDP).

Table 1 shows some comparative between Garanti BBVA database and the Company
Accounts Statistics 2009-2019 report published Jointly by the Central Bank of
Turkey and Turkstat.444The complete survey can be accesed at
http://www3.tcmb.gov.tr/sektor/2020/. This is the more complete study
published on companies in Turkey so far. Relative to this extense project the
information of Garanti BBVA database would cover near 24.5% of the sample with
slightly representation of machinery manufacturing firms (25.5%) than in
Construction(19.8%)

Table 1: Investment Firms Statistics: Garanti BBVA vs Central Bank of Turkey
(CBRT) & Turkstats Survey

|  | Garanti BBVA |  |  | CBRT-Turkstat |
---|---|---|---|---|---|---
Variable | Tot. | Machinery | Constr. | Tot. | Machinery | Constr.
Transactions(000s) | 24.6 | 22.3 | 2.3 |  |  |
Amount(US Bn) | 308 | 280 | 28 | 440 | 257 | 183
Firms(000s) | 179.7 | 156.5 | 23.2 | 730.2 | 614.4 | 115.8
Firms(% CBRT) | 24.6 | 25.5 | 19.8 |  |  |

  * •

Source: Garanti Bank and CBRT- Turkstat Survey.

In order to maintain robustness and data quality, these inflows are subject to
a set of criteria . The daily money transfers data include planned installment
payments, one-off payments, regular daily purchases and rest of transactions
in which the counter party is not the firm that receives the payment (i.e. the
transaction is not an internal transfer of funds). These inflows are filtered
according to mentioned set of criteria, which consists of several rules: (i)
The firm must be an active entity, that is, firms that went out of business
are automatically filtered out, (ii) city of the debtor party must be
identified, (iii) sector of the firm (agriculture, machinery, construction,
etc.) must be identified.

The inflows data fulfilling these criteria are then pooled and sorted by their
sectors according to NACE Rev. 2 classification. Investment good producing
sectors are then selected based on the sectoral distribution of Gross Fixed
Capital Formation in the Use Table (2012) released by TURKSTAT and the mapping
between Capital Goods and NACE codes in Main Industrial Groupings (MIGs)
classification. Based on this classification, the sectoral inflows are grouped
into Machinery and Construction sub-components which corresponds to around 90%
Investment expenditures in the official series.

In order to compute real investment we follow a similar process to the one
explained in the case of Big Data Consumption Index. Once we estimate the
nominal values for Machinery and Construction investment proxies, we deflate
their annual growth rates with the Domestic Producer Price Index (D-PPI) to
obtain the real growth rates. We also deflate the nominal series with the
corresponding producer price deflators, but in the absence of a complete set
of D-PPI individual deflators for all of the components and the fact that the
results do not change significantly, we opted to use the general D-PPI for all
of the components. The real year-on-year growth rates of two designated
sectors, namely machinery and construction, are derived for the purpose of
generating Garanti BBVA Big Data Construction Investment and Garanti BBVA Big
Data Machinery Investment indexes. Last but not least, two investment indexes
are once again weighted in consonance with their respective shares in TURKSTAT
Gross Fixed Capital Formation data and combined in the interest of generating
Garanti BBVA Big Data Investment Index.

Figure 2: Big Data and National Account Investment Aggregates (a) (March 2015
to March 2021, % YoY Nominal)

Source: Own Elaboration and Turkstat.

As we add the firm-to-firm money transfers to those sectors producing fixed
assets and the information is geo-localized, we can estimate the evolution of
fixed assets components and their geographical distribution.Thus, the Big Data
Investment Indicators developed in this work present the advantage to collect
information not only in real time but also in very granular way.

Figure 3 shows the evolution of the Big Data investment assets excluding other
investment. The different investment assets are grouped by two big aggregates:
machinery and transport and Construction. The heat map shows the evolution of
the yearly growth rate (three-month moving averages) from 2015 to 2021 (may).
The darker blue colors stand for the lowest 10th percentile while the lighter
blue colors represent the higher growth rates of the 90th percentile.

Figure 3: Big Data Investment Sectoral HeatMap (a) (% YoY Light Colours stand
for positive growth rates and Dark Colours for negative rates)

The period 2015-2021 different nature of shocks hitting the Turkish economy
during this period (2015 to 2021) makes Turkey an interesting case of study to
assess the impact of these investment shocks and its evolution over time on
sectoral basis. In this period, three important and different shocks hit the
economy (marked in orange): the failed coup of July of 2016, the financial
crisis during the summer of 2018 and the recent COVID-19 shock. The response
of the investment to these events has been somewhat different.

The response to the failed coup of the summer of 2016 (a political uncertainty
shock) was short-lived and concentrated on some specific activities (darker
colors), as shown in the non-homogeneous dark colors in the diffusion map. The
negative impact was mild and mainly observed in motor vehicles and
transportation fixed assets, while construction and machinery equipment was
not specifically affected and recovered very rapidly once the initial
political shock died out.

The response to the summer 2018 financial crisis (a full blown currency shock)
was more intense and homogeneous. After the sharp depreciation of the Turkish
lira (nearly 40%), the capital flows suddenly stopped and credits declined
rapidly for most activities. The shock to investment started just after the
financial crisis across the board for most of the sectors until the end of
2019 when credits started to grow after one-and-a-half years of rapid and deep
de-leveraging of the sector.

The response to COVID-19 (pandemic shock) has been somewhat in the middle of
the previous two shocks in terms of intensity, homogeneity and length. One
relevant aspect is that the temporary shock to investment look to be related
with the sudden but short sudden stop in the Global Value Changes(GVC) in
which Turkey is more active as metals and Automobiles (i.e. transportation).
The bigger and with longer lasting effects are concentrated in the investment
of big Civil Engineering projects which could be more intensively affected by
the uncertainty of the Pandemic.

Another important advantage of the Big Data financial transactions is that all
the transactions are geo-localized. This is important because allows to track
the investment activities not only by asset but also geographically.

Figure 4: Big Data Regional Investment Maps (a) (% YoY Light Colours stand for
positive growth rates and Dark Colours for negative rates)

Source: Own Elaboration.

The post Pandemic shock to the GVC is a good example of how this information
can help us to identify different types of shocks. Figure 4 shows how the
COVID-19 shock has affected the different investment assets (Total, Machinery
& Equipment and Construction) on a geographical basis. The Heat maps show the
yearly growth rates immediately before the mobility restrictions started to be
imposed in Europe (February 2020), the period including most of the mobility
restrictions (April-May 2020) and the ease of mobility restrictions and
recovery (July-August 2020).

The maps confirm some of our observations from the evolution of the fixed
assets performance showing that the negative effects on investment by the
COVID-19 shock have not been as homogeneous as the 2018 financial shock either
in sectoral nor in geographical terms. The key reason for this is that the
machinery investment response has been more differentiated and short lived
after the COVID-19 shock. An important result is that big cities such as
Istanbul or Ankara were not specially affected compared to other regions.
Meanwhile, as we move from the east to the west of the country, we observe
darker blues representative of contractions. This is not strange at all, as it
is precisely where the manufacturing industry is mainly located Akcigit et al.
(2019) and is consistent with the well-known regional east-west dualism in
Turkey Gezici et al. (2017). The maps also suggest that provinces specialized
in products such as metal and electrical equipment (Central-West), mostly
related to the automotive and durable consumer goods industries in the Global
Value Chain (GVC), experienced the sharpest temporal declines compared to the
textile industry in the GVC, which has an important presence also in the
central-south of the country. The fact that the shock has been less permanent
after COVID-19 is also relevant in geographical spillovers, as these GVCs are
important sources of spillovers to other industries.

The response of construction investment has been more homogeneous, but it is
also recovering faster so far than the 2018 financial crisis. Facing a more
negative performance than the machinery segment in the pre-COVID-19 period (as
the construction sector was experiencing some de-leveraging led by the
previous financial crisis), the initial response was homogeneous and amplified
the already weak situation. However, the situation from June onward started to
improve, at least in the big cities and the coast, but with some dark blue
areas in the middle of the country. Whether this is the result of different
allocation of credit or a different response of the macro-prudential policies
implemented by policymakers during COVID-19 is beyond this research.

###  2.3 Data included in the Nowcast Exercise

The data set used in our analysis include several variables at different
frequencies. We use quarterly chain-linked GDP series from TURKSTAT,
transformed in year-on-year growth rates, as the main target variable in our
nowcasting exercises. In terms of explanatory variables, we employ a total of
13 series representing broad categories of the Turkish economic activity
including hard data as the labour market, manufacturing and international
trade combined with financial data, soft data represented by corporate surveys
and big data activity indices including consumption and investment

We present the list of series used in Table 2 together with the vintages used
in the nowcasting exercises, their frequencies and applied transformations.
Since our main interest is to nowcast year-on-year GDP growth rate, for all
variables, we utilized raw, non-seasonally and calendar adjusted, form of the
series. All series are in real-form except for (nominal) total loan growth
trend, which is deflated by the headline CPI index released by TURKSTAT.

The data used in the nowcasting exercise is unbalanced in terms of different
length of the time series and the release lag structures. This will be
addressed with different techniques depending on the models that we employ in
the following sections. All variables except for the big data information are
publicly available.

Table 2: Detail of Variables Included in the Nowcasting Models

Variable | Type | Frequency | StartDate | Transformation
---|---|---|---|---
GDP | Hard | Quarterly | 2003 | YoY Growth
Industrial Production | Hard | Monthly | 2006 | YoY Growth
Auto Imports | Hard | Monthly | 2006 | YoY Growth
Auto Sales | Hard | Monthly | 2003 | YoY Growth
Auto Exports | Hard | Monthly | 2006 | YoY Growth
Non Metalllic Minerals | Hard | Monthly | 2006 | YoY Growth
Electricity Production | Hard | Daily | 2003 | YoY Growth
Number of Employed | Hard | Monthly | 2006 | YoY Growth
NUmber of Unemployed | Hard | Monthly | 2006 | YoY Growth
PMI | Soft | Monthly | 2006 | Level
Real Sector Confidence | Soft | Monthly | 2003 | Level
Loans (Credit) | Hard | Weekly | 2006 | Ann 13-week Growth
Big Data Consumption | Hard | Daily | 2015 | YoY Growth
Big Data Investment | Hard | Daily | 2015 | YoY Growth

  * •

Source: Own Elaboration

##  3 Nowcasting Methodology

In this section we describe our nowcasting methodology. We use linear and
nonlinear bridge equation models, dynamic factor models (DFM), and Bayesian
vector autoregressive models (BVAR) to nowcast year over year (YoY) GDP growth
rates. To build linear and nonlinear bridge equation models, we follow a
similar approach to Soybilgen and Yazgan (2021). The biggest difference
between our approach and Soybilgen and Yazgan (2021) is that we insert
variables directly into bridge equations whereas Soybilgen and Yazgan (2021)
first estimate dynamic factors and then use them in bridge equation models.
DFMs are estimated following Modugno et al. (2016) and Bańbura and Modugno
(2014) and BVAR is estimated following Ankargren and Yang (2019).

The DFM used in this study can deal with the missing data at the start of the
dataset, but we need to have a balanced dataset to estimate bridge equation
models and BVAR. As our dataset is highly unbalanced, we follow Stekhoven and
Bühlmann (2012) to fill out the missing data at the beginning of the dataset.

###  3.1 Bridge Equations

Let’s define
$x_{t_{m}}=(x_{1,t_{m}},x_{2,t_{m}},\dots,x_{n,t_{m}})^{\prime},t_{m}=1,2,\dots,T_{m}$
as $n$ monthly standardized explanatory variables. To construct bridge
equations, we convert monthly explanatory variables to quarterly ones,
$x_{t_{q}}=(x_{1,t_{q}},x_{2,t_{q}},\dots,x_{n,t_{q}})^{\prime},t_{q}=1,2,\dots,T_{q}$,
by taking simple averages of $x_{t_{m}}$. If there is any observation missing
at the end of the dataset for the reference quarter(s), we fill missing
observations using an autoregressive model whose lag is chosen by the Akaike
Information Criterion (AIC). Then, quarterly explanatory variables and
quarterly GDP growth rates are linked as follows:

$y_{t_{q}}=g(x_{t_{q}})+\varepsilon_{t_{q}},\\\ $ (1)

where $g()$ defines a linear or a nonlinear functional form. In this study, we
estimate equation 1 using ordinary least squares (OLS), random forests (RF),
and gradient boosted decision trees (GBM).

RF proposed by Breiman (2001) is an ensemble machine learning model based on
bagging (bootstrap aggregating) of decision trees. In RF, we first obtain $B$
bootstrapped training sets from original data and then fit a decision tree to
each bootstrapped training set while allowing only a random sample of
variables to be considered in each variable/split point for each terminal node
of a decision tree. Let $b=1,\dots,B$ denote the number of bootstrap
iterations. Then following Hastie et al. (2009), we predict QoQ GDP growth
rates using RF as:

  1. 1.

Obtain the bootstrapped data from the original data;

  2. 2.

Using the bootstrapped data obtained in the previous step, estimate a
regression tree, $\hat{g}_{RF}^{(b)}$, by considering just a fraction of
variables, $p$, at random from $n$ variables when determining the best
variable/split point for each terminal node of a decision tree;

  3. 3.

Repeat steps 1 and 2 $B$ times;

  4. 4.

Obtain predictions of GDP growth rates as
$\frac{1}{B}\sum_{b=1}^{B}\hat{g}_{RF}^{(b)}(x_{t_{q}+h_{q}})$.

GBM is another decision tree based ensemble machine learning model. The
difference between GBM and RF is that GBM turns weak learners into strong
learners in a sequential way instead of separately as in RF. After an initial
estimate, each tree is ﬁtted to the pseudo-residual, the gradient of the cost
function, of the previous estimate, and this ﬁtted tree is then used to update
the current estimate according to different learning rates for each region of
a decision tree. Let us define $m=1,\dots,M$ as the number of boosting
iterations, $\lambda$ as the learning parameter, and $L()$ as the loss
function. Then Following Friedman (2002) and Hastie et al. (2009), we predict
QoQ GDP growth rates using GBM as:

  1. 1.

Initialize
$g_{GBM}^{(0)}(\boldsymbol{x})=\displaystyle\operatorname*{argmin}_{\gamma}\sum_{t_{q}=1}^{T_{q}}L(y_{t_{q}},\gamma)$;

  2. 2.

Compute the gradient of the cost function,
$r_{t_{q},m}=\displaystyle\left[\frac{\partial
L\\!\left(y_{t_{q}},g_{GBM}^{(m-1)}(x_{t_{q}})\right)}{\partial
g_{GBM}^{(m-1)}(x_{t_{q}})}\right]$;

  3. 3.

Fit a decision tree to $r_{t_{q},m}$ giving terminal regions of the decision
tree, $R_{j,m},j=1,2,\dots,J_{m}$;

  4. 4.

For $j=1,2,\dots,J_{m}$, compute
$\gamma_{j,m}=\displaystyle\operatorname*{argmin}_{\gamma}\sum_{g_{t_{q}}\in
R_{j,m}}L\\!\left(y_{t_{q}},g_{GBM}^{(m-1)}(x_{t_{q}})+\gamma\right)$;

  5. 5.

Update
$g_{GBM}^{(m)}(\boldsymbol{x})=g_{GBM}^{(m-1)}(\boldsymbol{x})+\displaystyle\lambda\sum_{j=1}^{J_{m}}\gamma_{j,m}\mathbb{I}(\boldsymbol{x}\in
R_{m})$;

  6. 6.

Repeat steps 2, $\dots$, 5 $M$ times;

  7. 7.

Derive the final model
$g_{GBM}(\boldsymbol{x})=g_{GBM}^{(M)}(\boldsymbol{x})$;

  8. 8.

Obtain predictions of GDP growth rates as $g_{GBM}(x_{t_{q}+h_{q}})$.

For the linear bridge equations model, we obtain predictions of GDP growth
rates as $\hat{c}+\hat{\beta}x_{t_{q}+h_{q}}$ where $\hat{c}$ and
$\hat{\beta}$ are estimated OLS coefficients of equation 1.

###  3.2 Dynamic Factor Models

We model the DFM whose idiosyncratic components, $\epsilon_{i,t}$, follows an
AR(1) process as:

$\displaystyle x_{t_{m}}$ $\displaystyle=\Lambda f_{t_{m}}+\epsilon_{t_{m}};$
(2) $\displaystyle\epsilon_{t_{m}}$
$\displaystyle=\alpha\epsilon_{t_{m}-1}+v_{t_{m}};\quad v_{t_{m}}\sim
i.i.d.\;\mathcal{N}(0,\sigma^{2}),$ (3)

where $\Lambda$ is an $n$x$r$ vector containing factor loadings and $f_{t}$,
which is an $r$x$1$ vector of unobserved common factors, modelled as a
stationary vector autoregression process as follows:

$f_{t_{m}}=\varphi(L)f_{t_{m}-1}+\eta_{t_{m}};\quad\eta_{t_{m}}\sim
i.i.d.\;\mathcal{N}(0,R),$ (4)

where $\varphi(L)$ is an $r$x$r$ lag polynomial matrix and $\eta_{t_{m}}$ is
an $r$x$1$ vector of innovations.

To include quarterly GDP growth rates into the model, we use the approximation
of Giannone et al. (2013) and impose restrictions on the factor loadings as
follows:

$\displaystyle y_{t_{m}}^{Q}$
$\displaystyle=\bar{\Lambda}_{Q}[f_{t}^{{}^{\prime}}f_{t-1}^{{}^{\prime}}f_{t-2}^{{}^{\prime}}]+\bar{\epsilon}_{t_{m}}^{Q};$
(5) $\displaystyle\bar{\epsilon}_{t_{m}}^{Q}$
$\displaystyle=\alpha^{Q}\bar{\epsilon}_{t_{m}-1}^{Q}+\bar{v}_{t_{m}}^{Q};\quad\bar{v}_{t_{m}}^{Q}\sim
i.i.d.\;\mathcal{N}(0,\bar{\sigma}^{2}),$ (6)

where $y_{t_{m}}^{Q}$ denote a partially observed monthly counterpart of GDP
growth rates, in which the value of the quarterly series is assigned to the
third month of the respective quarter and
$\bar{\Lambda}_{Q}=[\Lambda_{Q}\Lambda_{Q}\Lambda_{Q}]$ is restricted factor
loadings. We estimate the DFM by following the procedure proposed by Bańbura
and Modugno (2014), which a modified version of the expectation-maximization
(EM) algorithm for maximum likelihood estimation. After casting equations 2-6
as state space form, Kalman filter and smoother allow us to extract the common
factors and generate projections for all of the variables in the model.

###  3.3 Bayesian Vector Autoregressive

Let’s assume that $x_{t_{m}}^{QM}=(x_{t_{m}},x_{t_{m}}^{Q})$ represent both
observed monthly variables, $x_{t_{m}}$, and unobserved monthly counterpart of
GDP growth rates, $x_{t_{m}}^{Q}$ and
$X_{t_{m}}^{QM}=(x_{t_{m}},y_{t_{m}}^{Q})$ represent observations. Similar to
the previous section, $y_{t_{m}}^{Q}$ denotes a partially observed monthly
counterpart of GDP growth rates that can only be observed in the third month
of the respective quarter and linked its unobserved monthly counterpart as
follows:

$y_{t_{m}}^{Q}=\frac{1}{3}(x_{t_{m}}^{Q}+x_{t_{m}-1}^{Q}+x_{t_{m}-2}^{Q}).$
(7)

We assume $x_{t_{m}}^{QM}$ follow a VAR(p) process as:

$x_{t_{m}}=\varphi(L)x_{t_{m}-1}+u_{t_{m}};\quad u_{t_{m}}\sim
i.i.d.\;\mathcal{N}(0,\Sigma),$ (8)

Following Schorfheide and Song (2015), Sebastian et al. (2020) and Ankargren
and Yang (2019), BVAR’s state-space form’s transition equation, which is the
companion form of the VAR(p) process, and the measurement equation are shown
as respectively:

$\displaystyle z_{t_{m}}$ $\displaystyle=\pi+\Pi
z_{t_{m}-1}+\zeta_{t_{m}};\quad\zeta_{t_{m}}\sim
i.i.d.\;\mathcal{N}(0,\Omega),$ (9) $\displaystyle X_{t_{m}}$
$\displaystyle=M_{t}\alpha z_{t_{m}}$ (10)

where
$z_{t_{m}}=(x_{t_{m}}^{\prime},x_{t_{m}-1}^{\prime},x_{t_{m}-p+1}^{\prime})^{\prime}$;
$\pi$, $\Pi$, and $\eta_{t_{m}}$ are the corresponding companion form
matrices; $M_{t}$ is a deterministic selection matrix; $\alpha$ is an
aggregation matrix.

We use the Minnesota prior for the autoregressive VAR coefficients and an
inverse Wishart prior for the error-covariance. Using a Gibbs sampler, we
generate draws from the posterior distributions and simulate future
trajectories of $X_{t}$ and calculate point forecasts of all variables. Using
Ankargren and Yang (2019), BVAR is estimated using Markov Chain Monte Carlo
(MCMC) and Gibbs sampling.

##  4 Nowcasting Performance

###  4.1 Main Results

In this study, we estimate our models between January 2016 and December 2020
using an expanding estimation window. We assume that each prediction is
computed at the end of the month and adjust announcement lag for each variable
accordingly as shown in Table A1. As vintage data is not available for Turkish
macro variables readily by any statistical agencies, we use a pseudo-real time
dataset that ignores historical data revisions.

In each month, we produce predictions for the current quarter. We also predict
the previous quarter if GDP of the previous quarter has not been announced
yet. As Turkish GDP is announced with more than two months delay, we produce
five predictions for each reference quarter.

We use mean absolute errors, $\textup{MAE}^{(i)}$, to evaluate the accuracy of
the $i$th nowcast produced by each model between 2016Q1 and 2020Q3 as follows:

$\textup{MAE}^{(i)}=(1/n)\sum_{t_{q}=2016Q1}^{2020Q3}|y_{t_{q}}-\hat{y}_{t_{q}}^{(i)}|;\quad
i=1,2,...,5.$ (11)

where $\hat{y}_{t_{q}}^{(i)}$ denotes the $i$th nowcast of a model and
$y_{t_{q}}$ represents the actual GDP growth rates.

Table 3 presents MAEs for linear/nonlinear bridge equation, BVAR and DF models
as well as the benchmark AR model. As can be seen from Table 3, all models
outperform the benchmark AR model in all periods. In first and second nowcasts
BVAR and DFM have the highest nowcasting accuracy and bridge equation models
perform poorly compared to DFM and BVAR models. Note that in first and second
nowcasts, we do not have any hard data available for the reference quarter,
hence most of the information content in the data relies on the big data
components. As a result, joint multivariate models that fill the missing part
of the dataset successfully outperform bridge equation models which rely on
auxiliary equations to fill out the missing data at the end of the data set.

Starting from third predictions, we see significant improvements both in
linear and nonlinear bridge equation models. For example, all bridge equation
models have lower MAEs than DFM in third nowcasts. In fourth and fifth
nowcasts, LM and DFM have the highest prediction accuracy, respectively.

Overall, there is no clear winner in terms of MAEs. DFM is the best behaved
model whose MAEs is steadily decreasing when informational content is
improved. BVAR results are highly volatile compared to DFM and the nowcasting
performances of nonlinear bridge equation models are lackluster compared to
their linear counterpart.

Table 3: MAEs of the models for successive nowcasting horizons between 2006Q1
and 2020Q3

|  AR |  DFM |  BVAR |  LM |  RF |  GBM
---|---|---|---|---|---|---
1st Nowcast |  3.71 |  1.92 |  1.77 |  3.46 |  2.60 |  3.13
2nd Nowcast |  3.71 |  1.85 |  2.29 |  3.07 |  2.32 |  2.55
3rd Nowcast |  3.80 |  1.72 |  1.52 |  1.70 |  1.53 |  1.71
4th Nowcast |  3.80 |  1.58 |  1.45 |  1.42 |  1.74 |  1.83
5th Nowcast |  3.80 |  1.38 |  1.64 |  1.46 |  1.65 |  1.49

  * •

Abbreviations: AR, the benchmark autoregressive model; DFM, the dynamic factor
model; BVAR, the Bayesian vector autoregressive model; LM, the linear bridge
equation model; RF, the random forest based bridge equation model; GBM, the
gradient tree boosted bridge equation model.

Turkey experienced several important economic and political downturns in this
period: the failed coup in 2016; the currency shock in 2018; the COVID-19
shock in 2020. Therefore, a visual inspection of the models’ predictions may
yield more information about the nowcasting performance of the models. Figure
5 presents nowcasts of the models for the successive nowcasting periods. In
first and second nowcasts, BVAR captures the downturn in 2016Q3, the jump in
2017Q3, and the COVID shock in 2020Q2 relatively well but overestimate the
recession between 2018Q4 and 2019Q3. In first and second nowcasts, DFM can not
capture volatile GDP growth rates in 2016Q3 and 2017Q3. Furthermore, DFM
underestimates GDP decline in 2020Q2. In other periods, DFM nowcasts GDP
growth rates quite successfully. In first and second nowcasts, both linear and
nonlinear bridge equation models perform quite poorly during 2017 and in
2020Q2. However, bridge equations perform much better in 2018 and 2019.
Especially, RF outperform all other models in 2018 and 2019. In third, fourth,
and fifth nowcasts, all models have relatively good nowcasting performance.
Even though all models predict negative growth rate in 2020Q2, GBM is the best
model that captures -10.3% GDP growth rate in 2020Q2.

Figure 5: Nowcasts Models performance and GDP (a) (March 2016 to September
2020,% YoY)

Abbreviations: DFM, the dynamic factor model; BVAR, the Bayesian vector
autoregressive model; LM, the linear bridge equation model; RF, the random
forest based bridge equation model; GBM, the gradient tree boosted bridge
equation model.

###  4.2 Nowcast Combination

Table 3 and Figure 5 show that there is no single best model that perform well
in all periods and some models like BVAR produce very volatile nowcasts.
Therefore, combining nowcasts of the models may provide better and more stable
nowcasts. We combine the prediction of each model to produce a final nowcast
as follows:

$\hat{Y}_{t_{q}}^{(i)}=\sum_{l=1}^{n}w_{t_{q},l}^{(i)}\hat{y}_{t_{q},l}^{(i)},\quad
l=1,2,\dots,L$ (12)

where $w_{t_{q},l}$ is the weight for the model $l$ for the $i$th nowcast;
$\hat{Y}_{t_{q}}^{(i)}$ shows the nowcast combination of the models for the
$i$th nowcast; $l=1,\dots,L$ is an index of all models. We use several types
of weights to combine nowcasts in our study. These are simple weights,
relative performance weights, and rank based weights.

First, we use simple averaging to calculate weights as follows:
$w_{t_{q},l}^{(i)}=1/L$. We also use the median forecast combination scheme.
Even though the equally weighted forecast combination often outperforms
sophisticated weighting techniques (Clemen, 1989; Hendry and Clements, 2004;
Huang and Lee, 2010; Stock and Watson, 2004), Genre et al. (2013) and
Soybilgen and Yazgan (2018) show that advanced combination schemes may
outperform equal weights in some cases.

Next, we calculate relative performance weights as:

$w_{t_{q},l}^{(i)}=\frac{(\text{MAE}_{t_{q},l}^{(i)})^{-1}}{\sum_{l=1}^{L}(\text{MAE}_{t_{q},l}^{(i)})^{-1}},$
(13)

where $w_{t_{q},l}^{(i)}$ donates MAE of the individual model $l$ for the
$i$th nowcast calculated at time $t_{q}$. We calculate MAE using the last two
year nowcast performance.

We also use rank based methods to compute weights as Timmermann (2006) argues
that this scheme is less sensitive to outliers compared to the relative
performance weight method. The rank based weights calculated as follows:

$w_{t_{q},l}^{(i)}=\frac{(\text{R}_{t_{q},l}^{(i)})^{-1}}{\sum_{l=1}^{L}(\text{R}_{t_{q},l}^{(i)})^{-1}},$
(14)

where $\text{R}_{t_{q},l}^{(i)}$ is the rank of the model $l$ for the $i$th
nowcast calculated at time $t_{q}$. Ranks are calculated according to MAEs
used in the relative performance method.

Table 4 presents MAEs for various nowcast combinations and Table 5 shows MAEs
for single models for the period of 2008Q2 and 2020Q3555We use previous two
year performances to calculate weights.. Simple and Median denote simple
averaging and the median nowcast, respectively. RPW and Rank denote nowcast
combinations calculated by relative performance weight and rank-based weights,
respectively.

Table 4: MAEs of nowcasting combinations for successive nowcasting horizons
between 2008Q2 and 2020Q3

|  Simple |  Median |  RPW |  Rank
---|---|---|---|---
1st Nowcast |  2.67 |  3.29 |  2.53 |  2.30
2nd Nowcast |  2.03 |  2.40 |  1.95 |  1.89
3rd Nowcast |  1.39 |  1.65 |  1.32 |  1.34
4th Nowcast |  1.44 |  1.43 |  1.44 |  1.45
5th Nowcast |  1.36 |  1.43 |  1.38 |  1.43

  * •

Abbreviations: Simple, nowcast combination using simple averaging; Median, the
median nowcast; RPW, nowcast combination using the relative performance
weight; Rank, nowcast combination using the rank based weight.

Table 5: MAEs of the individual models for successive nowcasting horizons
between 2008Q2 and 2020Q3

|  DFM |  BVAR |  LM |  RF |  GBM
---|---|---|---|---|---
1st Nowcast |  2.01 |  2.16 |  4.37 |  3.18 |  4.20
2nd Nowcast |  2.09 |  2.65 |  3.40 |  2.20 |  2.69
3rd Nowcast |  1.92 |  1.95 |  1.99 |  1.80 |  1.68
4th Nowcast |  1.59 |  1.57 |  1.22 |  2.06 |  1.88
5th Nowcast |  1.48 |  1.82 |  1.44 |  1.77 |  1.75

  * •

Abbreviations: DFM, the dynamic factor model; BVAR, the Bayesian vector
autoregressive model; LM, the linear bridge equation model; RF, the random
forest based bridge equation model; GBM, the gradient tree boosted bridge
equation model.

Results shows that rank based weighting method has the the highest nowcast
performance among all nowcast combination methods in first nowcasts, but it is
still worse than DFM. In second nowcasts, all nowcast combination schemes
except the median one outperform the best single model and Rank is still the
best one. In third nowcasts, all nowcast combination methods including the
median scheme beat the best single models. The performance difference between
the most of the nowcast combination methods and the single models are
substantial. In fourth and fifth nowcasts, the prediction performance of all
nowcast combination schemes become very similar.

Even though, nowcast combination schemes fail to outperform all single models
in all prediction horizons, nowcast combinations outperform most of single
models in many cases. Regarding the differences across weighting methods
advanced nowcast combination schemes outperform simple forecast combinations
in the first and second nowcasts where bridge equation models perform poorly.
In third, fourth, and fifth nowcasts where all models perform quite well, both
simple and advanced nowcast combination methods perform similarly.

###  4.3 Variable Selection with Lasso

If some of the variables in the dataset are not beneficial for the nowcasting
performance of models, pre-selection of variables may improve results of
models. In this part, we first select variables using a linear regression with
L1 regularization as known as Lasso regression, and then use those selected
variables in models.

In each period, we first estimate equation 1 with a Lasso regression and use
the variables with non-zero coefficients in the main nowcasting models. Table
6 present MAEs for linear/nonlinear bridge equation, BVAR and DF models whose
variables are first selected by Lasso regression.

Comparing the results with those of Table 3, pre-selection of variables by
Lasso increases nowcasting performance of the model with the exception of DFM
and GBM where it leads to a decrease in nowcasting performance. As DFM is
already a dimension reduction technique, an extra layer of variable selection
worsens its performance. Pre-selection of variables do not seem to help the
nowcasting performance of GBM either. On the contrary, selecting variables by
Lasso generally improve the nowcasting performance of BVAR, LM, and RF.
Overall, we see the biggest performance improvements in the linear bridge
equation model. LM becomes the best model in third, fourth and fifth nowcasts
with the help of Lasso regression.

Table 6: MAEs of the models with variable pre-selection process for successive
nowcasting horizons between 2006Q1 and 2020Q3

|  AR |  DFM |  BVAR |  LM |  RF |  GBM
---|---|---|---|---|---|---
1st Nowcast |  3.71 |  2.52 |  2.17 |  3.24 |  2.81 |  3.47
2nd Nowcast |  3.71 |  2.15 |  1.45 |  2.63 |  2.07 |  2.57
3rd Nowcast |  3.80 |  1.72 |  1.64 |  1.36 |  1.48 |  1.62
4th Nowcast |  3.80 |  1.73 |  1.38 |  1.28 |  1.73 |  1.76
5th Nowcast |  3.80 |  1.64 |  1.36 |  1.08 |  1.56 |  1.56

  * •

Abbreviations: AR, the benchmark autoregressive model; DFM, the dynamic factor
model; BVAR, the Bayesian vector autoregressive model; LM, the linear bridge
equation model; RF, the random forest based bridge equation model; GBM, the
gradient tree boosted bridge equation model.

This section can be concluded that DFM appears to be the best behaved model
whose MAEs is steadily decreasing when informational content is improved. It
is also fairly successful across different time periods but it experiences
some difficulties in capturing volatile growth periods. Nowcast combinations
outperform most of single models in many cases. Pre-selection of variables by
Lasso does not improve the performance DFM. LM appears to be the best model
after pre-selection.

##  5 The impact of Big Data on Nowcasting

In the previous section we have analyzed the general nowcasting performance
without considering any reference to the characteristics of the variables
included in the data set. In this section we will consider the role of big
data variables in nowcasting performance.

From the previous section we know that Lasso regression has helped to improve
the nowcasting performance. We first look at that how many times a specific
variable is selected by Lasso regression. We run Lasso regression 60 times
from 2006M01 to 2012M12. Table 7 shows the percentage of periods a variable is
chosen. Interestingly, variables related with car industry are never or rarely
selected. These variables are usually very volatile. Industrial production
series, real sector confidence index, PMI and total loans appear to be as the
most frequently chosen variables. The big data variables are chosen more than
50% of the time.

Table 7: Percentage of periods when a variable is chosen by Lasso regression

Name | Selection Ratio
---|---
IP | 100.0%
Car Imports | 0.0%
Ind. Production Non-Metallic Minerals | 98.3%
Car Total Sales | 1.7%
Electricity Demand | 48.3%
Number of Employed | 8.3%
Number of Unemployed | 15.0%
Car Exports | 0.0%
PMI | 98.3%
Total Loans 13week | 83.3%
Real Sector Confidence Index | 100.0%
Big Data Consumption | 55.0%
Big Data Investment | 68.3%

  * •

Source: Turkstat, Markitt, OSD and Own Elaboration

In Figure 6, we illustrate the dates (between 2016M01 and 2012M12) in which
the big data variables (consumption and/or investment) are chosen. Figure 6
shows that most of the time Lasso regression chooses at least one big data
variable. Interestingly starting from the COVID-19 shock, Lasso regression
starts to pick both of the big data variables.

Figure 6: Big Data Investment and Consumption variables selection by Lasso
Regression

Source: Own Elaboration

###  5.1 The big data impact on Nowcast Accuracy

We know analyze how our big data consumption and investment variables are
beneficial for nowcasting GDP in a timely manner. To accomplish this task,
first we present MAE differences (MAED) as follows:

$\textup{MAED}^{(i)}=\textup{MAE}^{(i)}-\textup{MAE}_{RD}^{(i)};\quad
i=1,2,...,5.$ (15)

where $\textup{MAE}_{RD}$ represents MAE of the models without big data. In
other words, we estimate models with the full dataset and models with a
reduced dataset that does not include big data variables. Then, we compare
their nowcasting performances.

Table 8 shows MAED of the models. According to Table 8, inclusion of big data
variables increase the nowcasting performance of all models in first nowcasts.
The nowcast performance improvement is especially large for BVAR and bridge
equation models. The big data variables provide significant improvement to LM
and RF in second nowcasts, but not to DFM. Starting from third nowcasts, big
data variables do not seem to exert any effects on the performance of the
models in a significant way. Note that the availability of hard data
concentrates usually to the third and fourth nowcast quarters. Hence in the
first two nowcast quarters big data plays a big role. However when hard data
become relatively abundant they become less important.

Table 8: MAEDs of the models for successive nowcasting horizons between 2006Q1
and 2020Q3

|  DFM |  BVAR |  LM |  RF |  GBM
---|---|---|---|---|---
1st Nowcast |  0.09 |  0.57 |  0.39 |  0.51 |  0.28
2nd Nowcast |  0.09 |  -0.60 |  0.26 |  0.22 |  0.03
3rd Nowcast |  0.07 |  -0.13 |  0.01 |  0.12 |  -0.04
4th Nowcast |  0.06 |  0.11 |  0.00 |  0.02 |  -0.29
5th Nowcast |  0.05 |  -0.01 |  0.06 |  -0.20 |  0.01

  * •

Abbreviations: DFM, the dynamic factor model; BVAR, the Bayesian vector
autoregressive model; LM, the linear bridge equation model; RF, the random
forest based bridge equation model; GBM, the gradient tree boosted bridge
equation model.

To further show the effect of big data variables on the nowcasting performance
of the models, we run the models on daily basis. We assume that big data
variables are released daily but other variables are announced at a specific
date as shown in Table A1. For the sake of simplicity, we assume that each
month consists of 30 days and calculate nowcasts for the reference quarter for
150 days until GDP is announced.

Instead of showing each model individually, we take simple averages of all
models’ nowcasts. This step reduces the volatility in nowcasts of individual
models and helps us to focus on the impact of variables on the nowcasting
performance of the models. Figure 7 shows the daily evaluation of MAEs during
150 days. As daily big data variables are highly volatile, we also show 7 day
moving averages of MAEs.

Figure 7: Daily MAEs of equally weighted nowcast combinations between 2006Q1
and 2020Q3

Source: Own Elaboration

It appears that big data variables improve the nowcasting performance of the
models for the first 45 days. Afterwards they become less important and do not
contribute the nowcast performance in a significant way. This further
strengthens our above conclusion that the big data variables are important
when hard data are rarely available. We see the biggest improvement in the
nowcasting performance of the models when the industrial production index’
first value for the reference quarter is released in the 73th day.

##  6 Conclusion

In this paper we have developed Consumption and Investment Big Data indexes
through individual-to-firm and firm-to firm transactions from a Big Data Bank
Data Base (Garanti BBVA).

The main purpose of the paper is to cross validate the information included in
these real time indexes and investigate how this information can be introduced
in an efficient way in the traditional nowcasting framework. For this purpose
we have estimated different individual and nowcasting combinations of
alternative models including Dynamic Factor Models, BVAR and Bridge equations
including Linear and Non Linear Machine Learning specifications as Random
Forest and Gradient Boosting.

The first relevant result is that the Financial Transactions contribute to
improve the traditional nowcasting models. Particularly, when some of the
traditional economic indicators are never or rarely selected, the Big Data
information from financial transactions will be useful more than 50% of the
time.

A second important result is that Financial Transaction information is more
relevant at the beginning of the nowcasting periods just when the hard data
information is scarce. Big Data information.It appears that big data variables
improve the nowcasting performance of the models for the first 45 days. This
conclusion strengthen our conclusion on big data variables are important when
hard data are rarely available and that this is a key result for Emerging
Markets where long lags in statistical releases are more important.

Besides the relevance of Big Data information we tested also the way to
introduce the information. The result show that while some of the alternative
produce good results in different nowcasting periods, the Dynamic Factor Model
(DFM) appears to be the best alternative model as its MAEs is steadily
decreasing when informational content is improved and looks fairly successful
across different time periods. Furthermore, nowcast combination schemes
outperform most of single models in many cases.

In sum, the paper show the relevance of the real time financial transaction to
enhance the traditional nowcasting models. Its relevance is remarked when the
amount of hard data information is scarce and can help to capture turning
points. These are important results as the developing of these information is
continuously increasing.

## References

  * Akcigit et al. (2019) Akcigit, U., Y. E. Akgunduz, S. M. Cilasun, E. Ozcan-Tok, and F. Yilmaz (2019).  Facts on business dynamism in turkey.  CBRT Working Paper No. 19/30.
  * Alexander (2020) Alexander, D; Karger, E. (2020).  Do stay-at-home orders cause people to stay at home? effects of stay-at-home orders on consumer behavior.  Federal Reserve Bank of Chicago Working Paper No. 2020-12.
  * Andersen et al. (2020) Andersen, A. L., E. T. Hansen, N. Johannesen, and A. Sheridan (2020).  Consumer responses to the covid-19 crisis: Evidence from bank account transaction data.  CEBI Working Paper Series No. 1820.
  * Ankargren and Yang (2019) Ankargren, S. and Y. Yang (2019).  Mixed-frequency bayesian var models in r: the mfbvar package.
  * Aprigliano et al. (2017) Aprigliano, V., G. Ardizzi, and L. Monteforte (2017).  Using the payment system data to forecast the italian gdp.  Bank of Italy Temi di Discussione Working Paper No. 1098.
  * Babii et al. (2021) Babii, A., E. Ghysels, and J. Striaukas (2021).  Machine learning time series regressions with an application to nowcasting.  Journal of Business & Economic Statistics, 1–23.
  * Baker et al. (2020a) Baker, S. R., R. A. Farrokhnia, S. Meyer, M. Pagel, and C. Yannelis (2020a).  How does household spending respond to an epidemic? consumption during the 2020 covid-19 pandemic.  NBER Working Paper No. 26949.
  * Baker et al. (2020b) Baker, S. R., R. A. Farrokhnia, S. Meyer, M. Pagel, and C. Yannelis (2020b).  Income, liquidity, and the consumption response to the 2020 economic stimulus payments.  NBER Working Paper No. 27097.
  * Bańbura and Modugno (2014) Bańbura, M. and M. Modugno (2014).  Maximum Likelihood Estimation of Factor Models on Datasets with Arbitrary Pattern of Missing Data.  Journal of Applied Econometrics 29(1), 133–160.
  * Barlas et al. (2020) Barlas, A. B., S. Güler, A. Ortiz, and T. Rodrigo (2020).  Investment in real time and High definition: A Big data Approach.  BBVA Research Working Papers No. 2013.
  * Barnett et al. (2016) Barnett, W., M. Chauvet, D. Leiva-Leon, and L. Su (2016).  Nowcasting nominal gdp with the credit-card augmented divisia monetary aggregates.  University of Kansas Working Papers Series in Theoretical and Applied Economics No. 201605.
  * Bodas et al. (2019) Bodas, D., J. R. García López, J. Murillo Arias, M. J. Pacce, T. Rodrigo López, J. d. D. Romero Palop, P. Ruiz de Aguirre, C. A. Ulloa Ariza, and H. Valero Lapaz (2019).  Measuring retail trade using card transactional data.  Technical report.
  * Bounie and Galbraith (2020) Bounie, D. Camara, Y. and J. Galbraith (2020).  Consumers’ mobility, expenditure and online- offline substitution response to covid-19: Evidence from french transaction data.  Hal working papers no. 02566443.
  * Breiman (2001) Breiman, L. (2001).  Random forests.  Machine learning 45(1), 5–32.
  * Carvalho et al. (2020) Carvalho, V. M., S. Hansen, A. Ortiz, J. R. Garcia, T. Rodrigo, S. Rodriguez Mora, and P. Ruiz de Aguirre (2020).  Tracking the covid-19 crisis with high resolution transaction data.  CEPR DP No 14642.
  * Carvalho et al. (2021) Carvalho, V. M., S. Hansen, A. Ortiz, T. Rodrigo, S. Rodriguez Mora, and P. Ruiz de Aguirre (2021).  National accounts in a world of naturally occurring data.  MiMEO.
  * Chetty et al. (2020) Chetty, R., J. Friedman, N. Hendren, M. Stepner, and T. O. I. Team (2020).  How did covid-19 and stabilization policies affect spending and employment? a new real-time economic tracker based on private sector data.  NBER WP No 27431.
  * Chronopoulos et al. (2020) Chronopoulos, D. K., M. Lukas, and J. O. Wilson (2020).  Consumer spending responses to the covid-19 pandemic: An assessment of great britain.  Working papers in responsible banking and finance no.20-012.
  * Clemen (1989) Clemen, R. T. (1989).  Combining Forecasts: A Review and Anotated Bibliography.  International journal of forecasting 5(4), 559–583.
  * Cox et al. (2020) Cox, N., P. Ganong, P. Noel, J. Vavra, A. Wong, D. Farrell, F. Greig, and E. Deadman (2020).  Initial impacts of the pandemic on consumer behavior: Evidence from linked income, spending, and savings data.  University of chicago, becker friedman institute for economics working paper.
  * Duarte et al. (2017) Duarte, C., P. M. Rodrigues, and A. Rua (2017).  A mixed frequency approach to the forecasting of private consumption with atm/pos data.  International Journal of Forecasting 33(1), 61–75.
  * Eraslan and Götz (2020) Eraslan, S. and T. Götz (2020).  An unconventional weekly economic activity index for germany.  Deutsche Bundesbank Technical Paper No. 02/202.
  * Friedman (2002) Friedman, J. (2002).  Stochastic gradient boosting.  Computational statistics & data analysis 38(4), 367–378.
  * Galbraith and Tkacz (2015) Galbraith, J. and G. Tkacz (2015).  Nowcasting gdp with electronic payments data.  European Central Bank Statistics Paper Series No. 10.
  * Genre et al. (2013) Genre, V., G. Kenny, A. Meyler, and A. Timmermann (2013).  Combining Expert Forecasts: Can Anything Beat the Simple Average?  International Journal of Forecasting 29(1), 108–121.
  * Gezici et al. (2017) Gezici, F., B. Y. Walsh, and S. M. Kacar (2017).  Regional and structural analysis of the manufacturing industry in turkey.  The Annals of Regional Science 59(1), 209–230.
  * Giannone et al. (2013) Giannone, D., S. M. Agrippino, and M. Modugno (2013).  Nowcasting china real gdp.  Technical report.
  * Hacioglu et al. (2020) Hacioglu, S., D. R. Känzig, and P. Surico (2020).  Consumption in the time of covid-19: Evidence from uk transaction data.  Cepr discussion papers no. 14733.
  * Hastie et al. (2009) Hastie, T., R. Tibshirani, and J. Friedman (2009).  The elements of statistical learning: data mining, inference, and prediction.  Springer Science & Business Media.
  * Hendry and Clements (2004) Hendry, D. F. and M. P. Clements (2004).  Pooling of Forecasts.  The Econometrics Journal 7(1), 1–31.
  * Huang and Lee (2010) Huang, H. and T.-H. Lee (2010).  To Combine Forecasts or to Combine Information?  Econometric Reviews 29(5-6), 534–570.
  * Lewis et al. (2020) Lewis, D. J., K. Mertens, J. H. Stock, and M. Trivedi (2020).  Measuring real activity using a weekly economic index.  New York FED WP No 920.
  * Lourenco and Rua (2020) Lourenco, N. and A. Rua (2020).  The dei: tracking economic activity daily during the lockdown.  Banco de Portugal Working Paper No 13.
  * Modugno et al. (2016) Modugno, M., B. Soybilgen, and E. Yazgan (2016).  Nowcasting turkish gdp and news decomposition.  International Journal of Forecasting 32(4), 1369–1384.
  * Olafsson and Pagel (2018) Olafsson, A. and M. Pagel (2018).  The liquid hand-to-mouth: Evidence from personal finance management software.  The Review of Financial Studies 31(11), 4398–4446.
  * Schorfheide and Song (2015) Schorfheide, F. and D. Song (2015).  Real-time forecasting with a mixed-frequency var.  Journal of Business & Economic Statistics 33(3), 366–380.
  * Sebastian et al. (2020) Sebastian, A., U. Måns, and Y. Yukai (2020).  A flexible mixed-frequency vector autoregression with a steady-state prior.  Journal of Time Series Econometrics 12(2).
  * Soybilgen and Yazgan (2018) Soybilgen, B. and E. Yazgan (2018).  Evaluating nowcasts of bridge equations with advanced combination schemes for the turkish unemployment rate.  Economic Modelling 72, 99–108.
  * Soybilgen and Yazgan (2021) Soybilgen, B. and E. Yazgan (2021).  Nowcasting us gdp using tree-based ensemble models and dynamic factors.  Computational Economics 57(1), 387–417.
  * Stekhoven and Bühlmann (2012) Stekhoven, D. J. and P. Bühlmann (2012).  Missforest—non-parametric missing value imputation for mixed-type data.  Bioinformatics 28(1), 112–118.
  * Stock and Watson (2004) Stock, J. and M. W. Watson (2004).  Combination forecasts of output growth in a seven-country data set.  Journal of Forecasting 23(6), 405–430.
  * Timmermann (2006) Timmermann, A. (2006).  Forecast combinations.  Handbook of economic forecasting, 135–196.
  * Verbaan et al. (2017) Verbaan, R., W. Bolt, and C. van der Cruijsen (2017).  Using debit card payments data for nowcasting dutch household consumption.  De Nederlandsche Bank Working Paper No. 571.
  * Woloszko (2020) Woloszko, N. (2020).  Tracking activity in real time with google trends.  OECD Economics Department Working Papers, No. 1634.

## Appendix

Table A.1: Announcement days and delays of the monthly variables

Name |  Announcement Lag in Months |  Announcement Day
---|---|---
Industrial Production (IP) | 2 | 13
Car Imports | 2 | 15
IP Non Metallic Minerals | 2 | 13
Car Sales | 2 | 15
Electricity Demand | 0 | 30
Number of Employed | 3 | 12
Number of Unemployed | 3 | 12
Car Exports | 2 | 15
Manufacturing PMI | 1 | 1
Total Loans 13week | 1 | 10
Real Sector Confidence Index | 0 | 26
Big Data Consumption | 0 | Daily
Big Data Investment | 0 | Daily

Source: Own Elaboration through Turkstat, OSD, Markitt, CBRT and own Big Data


11institutetext:  Technische Universität Berlin, Berlin, Germany
22institutetext: German Research Centre for Artificial Intelligence (DFKI),
Labor Berlin, Germany
22email: {salar.mohtaj $\mid$ babak.naderi $\mid$ sebastian.moeller}@tu-
berlin.de

# A Transfer Learning Based Model for Text Readability Assessment in German

Salar Mohtaj 1122 Babak Naderi 11 Sebastian Möller 1122
Faraz Maschhur 11 Chuyang Wu 11 Max Reinhard 11

###### Abstract

Text readability assessment has a wide range of applications for different
target people, from language learners to people with disabilities. The fast
pace of textual content production on the web makes it impossible to measure
text complexity without the benefit of machine learning and natural language
processing techniques. Although various research addressed the readability
assessment of English text in recent years, there is still room for
improvement of the models for other languages. In this paper, we proposed a
new model for text complexity assessment for German text based on transfer
learning. Our results show that the model outperforms more classical solutions
based on linguistic features extraction from input text. The best model is
based on the BERT pre-trained language model achieved the Root Mean Square
Error (RMSE) of 0.483.

###### Keywords:

Text readability Complexity Transfer learning Language model

##  1 Introduction

Text forms an integral part of exchanging information and interacting with the
world. Social media and web accelerated the textual content production
compared to the era before the web. In other words, along with the other types
of content (e.g., image and video), textual content has been increasing
drastically during past recent years. Text readability (in the following used
interchangeably with text complexity) is one of the factors which affects a
reader’s understanding of text [5].

A readability score is significant in informing readers about the difficulty
of a piece of text (e.g., a document) that they read. A readability score is
the mapping of a body of text to mathematical unit quantifying the degree of
readability. It is the basis of readability assessment. Readability assessment
has diverse use cases and applications, such as helping to people with
disabilities and also facilitate choosing of learning material for second
language learners [1].

Learning-based readability assessment refers to those approaches for assessing
complexity of a piece of text based by training machine learning (ML) models.
The main advantage of the learning-based techniques compared to traditional
readability formula (e.g., Flesch–Kincaid readability test [7] and Dale–Chall
readability formula [12]) is that they can be trained based on the target
group which subjected in the training data. Moreover, to make the the problem
of measuring readability easier to solve, traditional readability formulas
mainly focus on lexical and syntactic features (e.g., word length and sentence
length) [11]. On the other side, a range of more diverse features are
considered in learning-based approaches.

In this paper we proposed two models for automatically assessment of
readability of German text based on BERT transformer-based pre-trained
language model [6]. In our first model, we used GBERT [3] (German BERT) to
extract features from German text that are used as input for another Recurrent
Neural Network (RNN) on top. In the second model, we fine-tuned GBERT based on
the training data. We compared the obtained results with a baseline model
based on Random Forest method [2] that proposed in [15].

Our main contributions in this paper can be summarized as follow:

  * •

Compiling and open-sourcing111http://shorturl.at/dtD58 a test data that
contains 320 sentences from German Wikipedia articles that are scored by an
average number of 26 annotators in Level A2-B1.

  * •

Proposing a text readability assessment model for German text based on
transfer learning techniques.

The rest of the paper is organized as follow: Section 2 presents a number of
recent research on text readability and complexity assessment and related
tasks in NLP. The used dataset for training and testing the models are
thoroughly described in Section 3. The proposed models and the obtained
results are explained in Sections 4 and 5, respectively. Finally, discussions
and ideas for the future works are highlighted in section 6.

##  2 Related Work

In this section we briefly describe a number of recent research on automated
text readability analysis in German, as well as in English text.

In a classical machine learning based method, Xia et al. proposed a text
readability assessment system for second language learners [24]. To train a
supervised ML model, they extracted various types of features from input text
which includes lexico-semantic features, parse tree syntactic features,
language modeling features, and discourse-based features. The obtained results
from their experiments show the SVM classifier could achieve an accuracy of
0.80 and Pearson correlation of 0.9 on the WeeBit dataset [21].

In a deep learning based approach, Martinc et al. tested a set of neural
supervised and unsupervised architectures on three benchmark data in English
for the task of readability evaluation [11]. They also generated a new
Slovenian readability corpus from school book contents. As the unsupervised
approach, they tried to develop a new readability formula that can outperform
traditional readability formulas by relying on neural language model
statistics. Regarding the neural supervised model, they used BiLSTM,
Hierarchical attention networks (HAN), and transfer learning approaches to
train the models. Their obtained results show that the proposed unsupervised
readability measure is adaptable, robust, and transferable across languages.

Mohammadi et al. proposed a deep reinforcement learning model for readability
measuring [13]. They tested the proposed model on three English and Persian
readability corpora. They could skip the feature extraction phase that is
essential in the classical machine learning based models, by using the GloVe
embedding [17] and statistical language models. In order to extract features,
they fed the raw text into a Convolutional Neural Network (CNN) and extracted
beneficial features from a piece of text. The obtained results show that the
model could outperform the other approaches in Persian text readability
dataset and also could achieve competitive accuracy on English data (i.e., the
Weebit dataset [21]).

In a recent effort for classifying German text into different categories,
based on text difficulty for language learners, Szügyi et al. developed a
classifier based on linguistic features extracted from the texts [20]. They
extracted similar feature as [24], and also error measures and N-grams from
input text. To train and evaluate the proposed classification model, they
combined five different German readability assessment sources. Their results
show that morphological features plays the most important role in accurately
classifying the input text into the corresponding difficulty level. Moreover,
some of the syntactic and lexical features were also given a high weight by
the used ML algorithm.

Weiss et al. proposed another learning-based model for complexity modeling for
text readability across language [22]. They trained different ML models on
extracted features from German and English texts. They extracted 312 features
includes surface length, syntactic complexity, lexical complexity,
morphological complexity and so forth. Feature extraction and selection leads
to decreasing the number of features to 301 for model training. Among the ML
models that have been trained on the data, SVM could outperform the others and
achieve the best results. They achieved the F1 scores of 94.0, 89.6, and 92.6
for elementary, intermediate, and advanced texts, respectively.

To the best of our knowledge, the proposed model in this paper is the first
German model based on transferring knowledge from pre-trained language models
to the specific domain of readability evaluation. In the both proposed models
we skipped the time-consuming and resource-intensive process of feature
engineering.

##  3 Dataset

In this section we thoroughly describe the dataset. All the sentences in both
training and test dataset have been collected from Wikipedia articles. All
sentences have been evaluated by German learners in level A2-B1.

###  3.1 Training Data

To train the models for the experiments, we used TextComplexityDE dataset
[14]. TextComplexityDE is a text complexity detection dataset that is compiled
for the task of German readability assessment. It contains scores for
Complexity, Understandability, and Lexical Difficulty, for 1000 German
sentences. All the scores are in range of 1 to 7.

For this research we used the Complexity scores of 900 sentences to train the
models. The Complexity of each sentence in the training data is annotated by a
minimum number of 3 and maximum number of 18 annotators. The average number of
the opinion scores for each sentence in the data is 10 votes. The distribution
of Mean Opinion Score (MOS) for Complexity is presented in Figure 1. Also, the
distribution of the length of sentences (in character) in the training dataset
is depicted in Figure 2. As it is highlighted in the figure, most of the
sentences are 100 to 200 characters long, while there are a few sentences that
are longer than 400 characters.

Figure 1: The distribution of MOS values in the training data

Figure 2: The length distribution of sentences in the training data (in
characters)

Sentences contained in the training dataset were collected from German
Wikipedia articles 222http://de.wikipedia.org and the Leichte Sprache (Simple
language) dataset developed by Klaper et al. [9]. More details on the training
data is presented in [14].

###  3.2 Test Data

The ratings for the test dataset are collected in four different experiments.
For each experiment, 100 sentences were complied, 80 from 18 different
Wikipedia articles, and 20 sentences were shared between all experiments and
taken from the TextComplexityDE dataset. Participants are recruited through
online German learner groups in stoical media and also Language schools. For
online participants, there was a short mandatory listening and comprehensive
language test to make sure they have basic to intermediate knowledge of German
language. In addition, online sessions are divided to smaller test sessions in
which participant rated 11 sentences in one session. One out of the eleven
sentences was a gold standard question (i.e. a question which its answer is
obvious and known to the experimenter) which was used for removing submission
of participants who are not concentrated. We used a same 7-point Likert Scale
as it was used during creation of the TextComplexityDE dataset.

In the data cleansing step, all submission 1) with wrong answer to the gold
standard question, 2) which failed in language test, 3) with specific click
patterns (i.e. small variance) or those being too fast are removed. For each
sentence, Mean Opinion Score (MOS) is calculated by calculating the arithmetic
mean over the all ratings provided for that sentence. Using the 20 shared
sentences in each experiment, a first-order mapping function for MOS values
from each experiment to the MOS values of TextComplexityDE dataset are fitted.
It is to remove the well-known bias and gradient between different subjective
test. The final test dataset, includes 320 new sentences from 18 Wikipedia
articles which rated with minimum 16 participants.

##  4 Proposed Models

In this section we present our three proposed models for readability
assessment in German text. We start with the baseline model which developed
based on classical machine learning techniques, and then we continue with more
advanced models based on recently developed pre-trained language models.

###  4.1 Baseline Model

As the baseline, we developed a regression model based on Random Forest method
[2]. For this purpose, 73 linguistic features have been extracted, grouped in
traditional, lexical and morphological features. Feature engineering
approaches are employed to select more informative features.

After extracting linguistic features, we applied different feature selection
techniques such as omitting features with much number of missing values
(result in removing 32 features), removing features with high Pearson
correlation (result in removing 3 features) and recursive feature elimination
(result in removing 18 features). As a result, a total number of 20 features
are used to train the Random Forest model to predict readability score of
sentences.

More details about the baseline model, including the hyper-parameters and the
pre-processing steps that have been used in this model are presented in [15]

###  4.2 BERT Model

In this section we explain our models based on the BERT pre-trained language
model [6]. Our first model is based on extracting features from the BERT
model, in which we do the vectorization of input text by using BERT model. On
the other hand, the fine-tuning approach has been used in the second model to
slightly change the BERT model’s weights based on the data for the
experiments.

All the implementations in this section have been done using the HuggingFace
transformers library [23] and the Pytorch framework [16].

####  4.2.1 BERT for Feature Extraction

In this model we employed BERT to extract features from raw input text. For
this purpose, we fed the sentences into the German BERT model (we used gbert-
base333https://huggingface.co/deepset/gbert-base [3] model). The weights from
the last hidden layer are used as the corresponding vectors of the input
tokens. This way, the input tokens from the raw German sentence are converted
into vectors based on the context in which a token is represented.

The resulting embeddings from BERT are fed into multiple Gated Recurrent Units
(GRUs) [4] as depicted in Figure 3. To train the model the Adam optimizer [8]
with learning rate of 1e-3 has been used. Also, we used the batch size of 128
and dropout probability of 0.5 in the experiments.

Figure 3: The architecture of the proposed model based on BERT for feature
extraction

####  4.2.2 Fine-tuning BERT

The fine-tuning of pre-trained language models has shown promising results in
different NLP tasks in recent years [19, 18]. As a result, for the third
model, we developed a BERT based architecture to fine-tune BERT for the
readability assessment task.

Our fine-tuning based model follows the architecture for sequence
classification as it is described in [6]. The processes of predicting the MOS
value for an input sentence consists of the following steps:

  1. 1.

Tokenization: It includes splitting sentences into tokens and adding special
tokens related to the model, and also padding of the sentences.

  2. 2.

Passing the token ids through the twelve GBERT layers.

  3. 3.

Pooling: It includes the extraction of the last hidden state of the [CLS]
token. It passed through a dense linear layer by applying a Tanh activation.

  4. 4.

Regression: It includes a dropout layer and finally receiving the output float
value for the readability score.

We used gbert-base [3], for fine-tuning with AdamW optimizer [10]. Regarding
hyper-parameters, the model was fine-tuned in 3 epochs with learning rate of
5e-5 and the batch size of 16. The architecture of the model is presented in
Figure 4

Figure 4: The architecture of the proposed model based on BERT for fine-tuning

##  5 Evaluation of Results

###  5.1 Evaluation Metric

For evaluating the performance of the models, the Root Mean Square Error
(RMSE) metric, Pearson and Spearman correlation coefficient are used.

####  5.1.1 RMSE

shows the root of average squared difference between the estimated values
(readability scores) and the actual value, as presented in the following
equation. It is a common metric for regression analysis.

####  5.1.2 Pearson correlation

measures the linear correlation between the predicted values and the actual
ones for the sentences in the test set.

####  5.1.3 Spearman rank correlation

is correlation coefficient based on ranking the data and then calculating the
correlation on the ranked data.

###  5.2 Results

The obtained results by the three models on the test data are presented in
Table 1. As highlighted in the table, fine-tuning of the BERT model
outperforms the baseline and the feature extraction based model, in RMSE and
Pearson correlation metrics. Distribution of ratings and predicted values from
different models are illustrated in Figure 5.

Table 1: The obtained results (RMSE, Pearson and Spearman correlation coefficient) Model | RMSE | Pearson | Spearman
---|---|---|---
|  | correlation | correlation
Random Forest (baseline) | 0.813 | 0.551 | 0.581
BERT (Feature-Extraction) | 0.557 | 0.796 | 0.799
BERT (Fine-Tuning) | 0.483 | 0.86 | 0.844

On the other side, although the feature extraction model achieved promising
results, it is far less accurate than the fine-tuning approach in predicting
the readability of sentences. Finally, our baseline model achieved
approximately the same performance on the test data as the reported results in
[15] from the same model.

(a)

(b)

(c)

Figure 5: Distribution of predicted MOS values by a) Random Forrest
(Baseline), b) BERT (Feature Extraction), and c) BERT (Fine-Tuning)

##  6 Conclusion and Future Works

In this paper we proposed a transfer learning-based model for text readability
assessment in German text. We tested classical machine learning models based
on extracting features from text, and the transformer-based pre-trained
language models (e.g., BERT) on two separated test datasets. Our findings show
that fine-tuning the BERT model can outperform the other approaches.

As future work, different language models (e.g., XLNet and GPT-2) can be
tested on the dataset to better validate our findings in this paper. Moreover,
since the dataset includes Understandability and Lexical Difficulty scores in
addition to the Complexity, a multi-task learning experiment can be applied on
the data to measure the impact of the other scores on improving the overall
performance of the readability assessment.

## References

  * [1] Aluisio, S., Specia, L., Gasperin, C., Scarton, C.: Readability assessment for text simplification. In: Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications. pp. 1–9 (2010)
  * [2] Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)
  * [3] Chan, B., Schweter, S., Möller, T.: German’s next language model. In: Scott, D., Bel, N., Zong, C. (eds.) Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020. pp. 6788–6796. International Committee on Computational Linguistics (2020)
  * [4] Cho, K., van Merrienboer, B., Bahdanau, D., Bengio, Y.: On the properties of neural machine translation: Encoder-decoder approaches. In: Wu, D., Carpuat, M., Carreras, X., Vecchi, E.M. (eds.) Proceedings of SSST@EMNLP 2014, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, Doha, Qatar, 25 October 2014. pp. 103–111. Association for Computational Linguistics (2014)
  * [5] Dale, E., Chall, J.S.: The concept of readability. Elementary English 26(1), 19–26 (1949)
  * [6] Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Burstein, J., Doran, C., Solorio, T. (eds.) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers). pp. 4171–4186. Association for Computational Linguistics (2019)
  * [7] Kincaid, J.P., Fishburne Jr, R.P., Rogers, R.L., Chissom, B.S.: Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Tech. rep., Naval Technical Training Command Millington TN Research Branch (1975)
  * [8] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015)
  * [9] Klaper, D., Ebling, S., Volk, M.: Building a german/simple german parallel corpus for automatic text simplification. In: Williams, S., Siddharthan, A., Nenkova, A. (eds.) Proceedings of the Second Workshop on Predicting and Improving Text Readability for Target Reader Populations, PITR@ACL 2013, Sofia, Bulgaria, August 8, 2013. pp. 11–19. Association for Computational Linguistics (2013)
  * [10] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net (2019)
  * [11] Martinc, M., Pollak, S., Robnik-Sikonja, M.: Supervised and unsupervised neural approaches to text readability. Comput. Linguistics 47(1), 141–179 (2021)
  * [12] McClure, G.M.: Readability formulas: Useful or useless? IEEE transactions on professional communication (1), 12–15 (1987)
  * [13] Mohammadi, H., Khasteh, S.H.: Text as environment: A deep reinforcement learning text readability assessment model. CoRR abs/1912.05957 (2019)
  * [14] Naderi, B., Mohtaj, S., Ensikat, K., Möller, S.: Subjective assessment of text complexity: A dataset for german language. CoRR abs/1904.07733 (2019)
  * [15] Naderi, B., Mohtaj, S., Karan, K., Möller, S.: Automated text readability assessment for german language: A quality of experience approach. In: 11th International Conference on Quality of Multimedia Experience QoMEX 2019, Berlin, Germany, June 5-7, 2019. pp. 1–3. IEEE (2019)
  * [16] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E.Z., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library. In: Wallach, H.M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E.B., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada. pp. 8024–8035 (2019)
  * [17] Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: Moschitti, A., Pang, B., Daelemans, W. (eds.) Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL. pp. 1532–1543. ACL (2014)
  * [18] Prabhakar, A.A., Mohtaj, S., Möller, S.: Claim extraction from text using transfer learning. In: Bhattacharyya, P., Sharma, D.M., Sangal, R. (eds.) Proceedings of the 17th International Conference on Natural Language Processing, ICON 2020, Indian Institute of Technology Patna, Patna, India, December 18-21, 2020. pp. 297–302. NLP Association of India (NLPAI) (2020)
  * [19] Su, D., Xu, Y., Winata, G.I., Xu, P., Kim, H., Liu, Z., Fung, P.: Generalizing question answering system with pre-trained language model fine-tuning. In: Fisch, A., Talmor, A., Jia, R., Seo, M., Choi, E., Chen, D. (eds.) Proceedings of the 2nd Workshop on Machine Reading for Question Answering, MRQA@EMNLP 2019, Hong Kong, China, November 4, 2019. pp. 203–211. Association for Computational Linguistics (2019)
  * [20] Szügyi, E., Etler, S., Beaton, A., Stede, M.: Automated assessment of language proficiency on german data. In: Proceedings of the 15th Conference on Natural Language Processing, KONVENS 2019, Erlangen, Germany, October 9-11, 2019 (2019)
  * [21] Vajjala, S., Meurers, D.: On improving the accuracy of readability classification using insights from second language acquisition. In: Proceedings of the seventh workshop on building educational applications using NLP. pp. 163–173 (2012)
  * [22] Weiss, Z., Chen, X., Meurers, D.: Using broad linguistic complexity modeling for cross-lingual readability assessment. In: Proceedings of the 10th Workshop on NLP for Computer Assisted Language Learning. pp. 38–54. LiU Electronic Press, Online (May 2021)
  * [23] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T.L., Gugger, S., Drame, M., Lhoest, Q., Rush, A.M.: Transformers: State-of-the-art natural language processing. In: Liu, Q., Schlangen, D. (eds.) Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2020 - Demos, Online, November 16-20, 2020. pp. 38–45. Association for Computational Linguistics (2020)
  * [24] Xia, M., Kochmar, E., Briscoe, T.: Text readability assessment for second language learners. In: Tetreault, J.R., Burstein, J., Leacock, C., Yannakoudakis, H. (eds.) Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, BEA@NAACL-HLT 2016, June 16, 2016, San Diego, California, USA. pp. 12–22. The Association for Computer Linguistics (2016)