# Evidence of chaos and nonlinear dynamics in the Peruvian financial market

Alexis Rodriguez Carranza<EMAIL_ADDRESS>Marco A. P. Cabral<EMAIL_ADDRESS>Juan C. Ponte Bejarano<EMAIL_ADDRESS>Department of Sciences - Private
University of the North, Perú  Federal University of Rio de Janeiro, Brazil
Department of Sciences - Private University of the North, Perú

###### Abstract

Physicists experimentalists use a large number of observations of a
phenomenon, where are the unknown equations that describe it, in order to play
the dynamics and obtain information on their future behavior. In this article
we study the possibility of reproducing the dynamics of the phenomenon using
only a measurement scale. The Whitney immersion theorem ideas are presented
and generalization of Sauer for fractal sets to rebuild the asymptotic
behaviour of the phenomena and to investigate, chaotic behavior evidence in
the reproduced dynamics. The applications are made in the financial market
which are only known stock prices

###### keywords:

Caos; Dinâmica não lineal; Séries temporais.

, .

##  1 Introduction

For a study of the asymptotic behavior of solutions of a system, the area of
dynamical systems has developed a lot of tools , but in many phenomena as
financial markets, equations that model them are unknown and the only
available information is a temporal set of measures.

A time series is a function $s:I\\!\\!R\rightarrow I\\!\\!R$, and its image
could be considered as observations taken overtime in a particular phenomenon.
Those pieces of observations have information about the system and we are
ready to answer questions like does that serie have enough information to
rebuild the dynamics? If it is so, can we predestine its future behaviour?.
But there is one more general question. Will the financial market have a
random behaviour or not?. The chaos theory could give us an answer where
systems with an apparently ramdom behaviour, comes from a deterministic
system, it means a completely modeling system by equations. Chaotic system
complexity lies primarily in sensitivity to initial conditions, small
variations produce big changes.This sensitivity is quatified by Lyapunov
exponents. We will do our research in a time series of the stock prices in the
mining company MINSUR.SA to determine concrete evidence of chaos presence in
that series. Graphic(1) shows evolution of stock prices of the company.

Figure 1: Evolution of stock prices of the MINSUR company

##  2 Teoría do Caos

To show the use of chaos theory in time series, we consider the Lorenz system,
which is given by:

$\displaystyle\frac{dx}{dt}$ $\displaystyle=$ $\displaystyle 10(y-x)$
$\displaystyle\frac{dy}{dt}$ $\displaystyle=$ $\displaystyle x(b-z)-y$
$\displaystyle\frac{dz}{dt}$ $\displaystyle=$ $\displaystyle xy-cz$

The dynamics of system solutions is shown in the figure(2)

Figure 2: Sistema de Lorenz

It is well known Lorenz system is sensitive to small variations with initial
data, it means, solutions with very close initial data is not exact
exponentially[3]. Figure(3) shows such a behaviour in the first component of
solutions for very close data initial. The series in the figure(3) were
generated with initial data in points $(1,2,3)$ and
$(1.00001,2.00001,3.00001)$ in time with variation in mili seconds. Notice the
trajectories differ after 3 seconds, it means, small variations generate big
changes quickly. So that, chaos theory could explain some phenomena in the
financial market, where variations like speculations generate changes in a
short term. One of the goals in this work is to investigate chaotic behavior
on time series in the financial market precisely applied in the Peruvian
financial system

Figure 3: Component x of Lorenz system for close initial conditions

The main task to study the chaotic behavior in a time series is the rebuilding
of the dynamics it has. If a time series comes from a deterministic system, so
that it has dependence between its components given by the system of equations
that modeling and contains geometric information considering that trajectories
converge to an attractor which will be inmerse in some Euclidian space. Using
such information and The Whitney immersion theorem ideas in which a
dimensional compact manifold $n$ can be inmersed in $I\\!\\!R^{2n}$ and its
generalizations for fractal sets given for Tim Sauer[4], we will rebuild the
dynamics knowing the information of one of the components of Lorenz system
only.

Now we can see the probabilistic version of Theorem of Withney[4]

##  3 Main theorems

######  Theorem 1.

(Prevalent theorem of Whitney) If $A$ is a smooth compact manifold of
dimension $d$ contained on $I\\!\\!R^{k}$. Almost all smooth map,
$F:\\!\\!R^{k}\rightarrow\\!\\!R^{2d+1}$, is an immersion of $A$.

The theorem(1) says it is possible to rebuild the existing dynamics with
projections in Euclidian spaces. Despite we have the theorem of Whitney, we
still have a practical problem. We only can get a temporal observation of the
dynamics in the financial market, the evolution of the stock prices and we
will need $2d+1$ 1observations of the phenomenon. So, the obtained results are
not enough. Takens assumed this problem adding the contained dynamics in the
time series in the Theorem of Whitney. It is a projection via a function of
observation of the phase space where the dynamical system is developed for
$I\\!\\!R$. Therefore it contains information about the dynamics. For this, it
defines the delay cordinates, which only need a temporary observation.

######  Definition 2.

If $\Phi$ is a dynamics system over a manifold $A$, $T$ a positive integer
(called of delay), and $h:A\rightarrow\\!\\!R$ a smoth function. We defines
the map of delaying coordinates $F_{(\Phi,T,h)}:A\rightarrow\\!\\!R^{n+1}$

$F_{(\Phi,T,h)}(x)=(h(x),h(\Phi_{T}(x)),h(\Phi_{2T}(x)),\dots,h(\Phi_{nT}(x)))$

Taken[7] demonstrated a new version of theorem of Whitney for the delaying
coordinates

######  Theorem 3.

(Takens) $A$ is a dimensional compact manifold $m$,
$\\{\Phi_{k}\\}_{k\in\mathbb{Z}}$ a discrete dynamical system over $A$
generated by $F:A\rightarrow A$, and a function of classe $C^{2}$
$h:A\rightarrow I\\!\\!R$. is a generic characteristic that of map
$F_{(\Phi,h)}(x):A\rightarrow I\\!\\!R^{2m+1}$ defined by

$F_{(\Phi,h)}(x)=(h(x),h(\Phi_{k}(x)),h(\Phi_{2k}(x)),\dots,h(\Phi_{nk}(x)))$

is an immersion.

The final generalization used in this article was given by B.Hunt, T. Sauer
and J. Yorke[8], that is a fractal version of theorem of Whitney for the
delaying maps set with $A$ being a fractal set.

######  Theorem 4.

(Fractal Delay Embedding Prevalence Theorem) $\Phi$ A dynamics system over an
open subset $U$ of $I\\!\\!R^{k}$, and $A$ a compact subset of $U$ with box
dimension $d$. $n>2d$ an integer and $T>0$. Asume that $A$ contains only a
finite number of equilibrio points, it does not contains periodic orbits of
$\Phi$ of period $T$ o $2T$, a finit number of periodic orbits of $\Phi$ of
period $3T$, $4T,\dots,nT$, and these periodic orbits linearizations have
different eigenvalues. So for almost every smooth function(in the sense
prevalente) smooth function $h$ over $U$, the delay coordinates map
$F_{(\Phi,T,h)}:U\rightarrow\\!\\!R^{n}$ is injective over $A$

The theorem(4) does not provide an estimative about the smaller dimension for
which almost every delaying coordinates map is injective. However there are
numerical algorithms which allow calculate the immersion dimension and the
delaying time in the reconstruction.Following, we show the reconstruction of
the dynamics generated for the system of Lorenz using only the coordinate $x$
of the system

##  4 4\. Examples of reconstruction of the attractor using delay coordinates

we use the Lorenz’s attractor to show the technique of delaying coordinates.
The function of observation $h$, was the projection in the $x$ axis.

$\begin{array}[]{cccc}h:&I\\!\\!R^{3}&\rightarrow&I\\!\\!R\mbox{ }\\\
&(x,y,z)&\rightarrow&h(x,y,z)=x\end{array}$

The time series will be formed by $x$ coordinates of the trajectories that are
numerical solutions of the equation. Fixed the dimension, $n=3$, we change the
value of T $T$. According to Lorenz’s we use delaying times, $T=1$, $T=10$,
$T=17$, $T=90$. In the case $T=1$ points in the space are highly correlated
and the graphic is almost a straight line. At the other extreme, $T=90$,
points are not correlated and the gotten graphic does not represent the
reconstruction of the attractor. The optimal delay time was $T=17$.

Figure 4: reconstruction of attractor using proyections with time delay d=1,
10, 17 and 90 respectively

To determine the dimension where is rebuilt the dynamics, the technique of the
neighbours is used. The idea is about setting a dimension and calculate the
distance between close points, if the attractor does not have enough freedom,
so when we raise the dimension of the space, those next points get separeted
and are pointed as false neighbours. The procedure goes on until the
percentage of false neighbours is small. In the series formed with the first
components of Lorenz’s system, the number of false neighbours is almost zero
when the dimension of immersion is $3$, figure(5).

Figure 5: Dimension of inmersión

##  5 Identifying non linearity and chaos

First of all, we will give the definition of chaos according to Devaney[5].

######  Definition 5.

If $V$ is an interval. We say that $f:V\rightarrow V$ is chaotic over $V$ if:

  * 1.

$f$ has sensibility to initial conditions;

  * 2.

$f$ is transitive;

  * 3.

The periodic points are dense in $V$.

We say that $f$ has sensibility to initial conditions when the solutions with
initial conditions very close, diverge quickly.

######  Definition 6.

A map $f:[0,1]\rightarrow[0,1]$ has sensitive dependence in $x$ if the next
condition holds for some $\delta>0$: for each open I on $[0,1]$, containing
$x$, there is a $y\in I$ and $n\in I\\!\\!N$ such that

$|f^{n}(x)-f^{n}(y)|\geq\delta.$

######  Definition 7.

A map $f:[0,1]\rightarrow[0,1]$ is transitive if for each pair of subintervals
$I$ y $J$ de $[0,1]$ there is $n$ such that $f^{n}(I)\cap J\leq\phi$

Intuitively, $f$ is transitive implies the orbit of an interval $I_{0}$ is
dense on $[0,1]$.

Given chaos definition, we will investigate the existence of characteristics
$1)-3)$ in a time series. In this article we are interested in searching the
characteristic $1$ which is characterized by Lyapunov exponents.

It is important to know lineal systems can not be chaotic. We will identify no
lineal dependences on data. The most used techniques are: method of surrogates
data and the test BDS.
Kantz e Schreiber[18] recommend that before doing any non lineal analysis on a
data set, it is good practice to check if there is no linearity. Because the
linear systems are not chaotic and the linear stochastic processes can produce
very complex series which can be misunderstood as a product of chaotic system.
We presented the technique of surrogate data and the BDS test.

###  5.1 Surrogate data

A very useful technique used by physicist is surrogate generation of Surrogate
data, a procedure done by Theiler, Eubank, Longtin, Galdrikan y Farmer[24].
From the original data , it generates a set of random series so that, these
keep the linear properties of the original series (average, variance , Fourier
spectrum) but eliminating the possible nonlinear dependencies.Then an
indicator is evaluated, which is sensitive to nonlinear dependencies and try
to reject the null hypothesis, which states that data are obtained by a
stochastic linear process. If the null hypothesis is true, then the substitute
series procedure would not affect the indicator

The most used indicators are correlation dimension. It was used by Small et.
al.[13] and no linear prediction which was used by Kaplan[10].

We will illustrate the technique of substitute data applied in Lorenz series
and use the nonlinear prediction.

The election was due to deterministic systems the prediction on a short term
is possible, in contrast to Stochastic systems in which the prediction in a
short term is impossible . For the prediction nonlinear we will use the method
developed by Hegger, Kantz and Schreiber[9], where the value of predicting is
the average of “future” values. This state $n+k$, is defined by average of the
closed values. It is calculated by the following equation

$\overline{S}_{n+k}=\frac{1}{|U_{n}|}\sum_{S_{j}\in U_{n}}S_{j+k},$

Where $U_{n}$ is the neighborhood, $S_{j}$ are values within $U_{n}$ and
$S_{j+k}$ are the values at time $n+k$. held the prediction, this is compared
with the quadratic average error of prediction on future value of the original
serie with the surrogate series. If the prediction error of the series
original is smaller than all the surrogate series, then we reject the null
hypothesis. This graphic shows the results.

Figure 6:  In the graphic the proyection errors of Lorenz series and its
surrogate series

The projection errors of the original series were smaller than its
substitutes. Like this, we reject the null hypothesis.

The following technique is more about statistics.

##  6 BDS Test

It was developed by Brock, Dechert and Scheinkman[21], It’s based on the
correlation dimension to detect nonlinear structure in a time series.
Additionally this test can be used to test how good is the fit estimation
model

Given a time series $s_{1},s_{2},\dots,s_{N}$, Using the method of delay
coordinates, we have $M=N-(m-1)\tau$ vectors in $I\\!\\!R^{m}$,
$X_{i}=(s_{i},s_{i+\tau},\dots,s_{i+(m-1)\tau})$, where $\tau$ is the delay
time, $m$ is the immersion dimensión. The correlation dimension is given by
the formula,

$\displaystyle C(M,r)$ $\displaystyle=$
$\displaystyle\frac{1}{M(M-1)}\sum_{i=1}^{M}M_{X_{i}^{m}}(\epsilon)$
$\displaystyle=$ $\displaystyle\frac{1}{M(M-1)}\sum_{i\neq
j}\theta(\epsilon-||X_{i}-X_{j}||),$

Where $M_{X_{i}}(\epsilon)$ is the function point mass, which indicates the
probability where the points $X_{i},X_{j}$ are $\epsilon$ close to each other.

For practical and didactic issues we consider $\tau=1$. Now if $m=2$ we will
get

$\displaystyle X_{i}$ $\displaystyle=$ $\displaystyle(x_{i},x_{i+1}),$
$\displaystyle X_{j}$ $\displaystyle=$ $\displaystyle(x_{j},x_{j+1}).$

Then, $||X_{i}-X_{j}||\leq\epsilon$ implies $|x_{i}-x_{j}|\leq\epsilon$ and
$|x_{i+1}-x_{j+1}|\leq\epsilon$.

This allows to say that if the points $X_{i}$ e $X_{j}$ are closing to each
other, then, the points from series $x_{i}$ and $x_{j}$ as well. It happens
the same with points $x_{i+1}$ and $x_{j+1}$. Thus,

$\displaystyle M_{X_{i}}(\epsilon)$ $\displaystyle=$
$\displaystyle\mbox{P}(||X_{i}-X_{j}||\leq\epsilon)$ $\displaystyle=$
$\displaystyle\mbox{P}(|x_{i}-x_{j}|\leq\epsilon;|x_{i+1}-x_{j+1}|\leq\epsilon)$

Where $P$ indicates the probability. If the data $x_{1},x_{2},\dots,x_{N}$ are
IID(independent and identically distributed), then
$P(|x_{i}-x_{j}|\leq\epsilon)=P(|x_{i+1}-x_{j+1}|\leq\epsilon)$ and we have,

$\displaystyle M_{X_{i}}(\epsilon)$ $\displaystyle=$
$\displaystyle\mbox{P}(|x_{i}-x_{j}|\leq\epsilon)\mbox{P}(|x_{i+1}-x_{j+1}|\leq\epsilon)$
$\displaystyle=$ $\displaystyle\mbox{P}^{2}(|x_{i}-x_{j}|\leq\epsilon).$

Thus, in dimension $m$ we have

$M_{X_{i}}(\epsilon)=\mbox{P}^{m}(|x_{i}-x_{j}|\leq\epsilon)$

Then,

$\displaystyle C_{m}(M,\epsilon)$ $\displaystyle=$
$\displaystyle\frac{1}{M}\sum_{i=1}^{M}M_{X_{i}}(\epsilon)$ $\displaystyle=$
$\displaystyle\frac{1}{M}\sum_{i=1}^{M}P^{m}(|x_{1}-x_{2}|\leq\epsilon)$
$\displaystyle=$ $\displaystyle P^{m}(|x_{1}-x_{2}|\leq\epsilon).$

As
$P(|x_{1}-x_{2}|\leq\epsilon)=P(|x_{1}-x_{3}|\leq\epsilon)=\dots=P(|x_{i}-x_{j}|\leq\epsilon)=\dots$.
This suggests that the data is IID, then,

$C_{m}(M,\epsilon)=C_{1}^{m}(M,\epsilon),$

Where $C_{m}(M,\epsilon)$ is the integral of correlation in dimension $m$ and
$C_{1}(M,\epsilon)$ is the integral of correlation in dimension one.

In, Brock, Dechert y Scheinkman[21], statistical BDS test is defined by:

$\sqrt{M}\frac{C_{m}(M,\epsilon)-C_{1}^{m}(M,\epsilon)}{\sigma_{m}(\epsilon)}$

Where $\sigma_{m}(M,\epsilon)$ is the estimation of the standard asymptotic
error: $C_{m}(M,\epsilon)-C_{1}^{m}(M,\epsilon)$. They proved that:

$\sqrt{M}\frac{C_{m}(\epsilon)-C_{1}^{m}(\epsilon)}{\sigma_{m}(\epsilon)}\sim
N(0,1)$

Introducing to BDS test applied in Lorenz series and a series consisting of
random numbers.

###  6.1 BDS Test for Lorenz series

Dimension of immersion(m) | BDS
---|---
1 | 165.270466
2 | 447.623951
3 | 578.535241
4 | 772.617763
5 | 1078.467687
6 | 2338.679429
7 | 3593.653394
8 | 5640.618871

###  6.2 Test BDS for a series consisting of random numbers

Dimension of immersion(m) | BDS
---|---
1 | 0.560594
2 | 0.560594
3 | 0.425293
4 | 0.069765
5 | 0.116118
6 | 0.336857
7 | 0.633835
8 | 0.524584

To understand the results, let’s start talking about the degree of
significance of a hypothesis. To ask a null hypothesis, in the case of the BDS
test the null hypothesis is that the observations are independent and
identically distributed (IID), it can be make a mistake of rejecting the null
hypothesis being true. Suppose that the probability of committing this error
is $\alpha$, namely:

$\alpha=P(\mbox{rejeitar}\quad H_{0}|H_{0}\,\,\mbox{é}\quad\mbox{verdadeiro})$

$\alpha$ is called the degree of significance of this mistake. Now, in the BDS
test:

$\displaystyle H_{0}:C_{m}(M,\epsilon)$ $\displaystyle=$ $\displaystyle
C_{1}^{m}(M,\epsilon)$ $\displaystyle H_{a}:C_{m}(M,\epsilon)$
$\displaystyle\neq$ $\displaystyle C_{1}^{m}(M,\epsilon)$

By the test BDS, know that

$\sqrt{M}\frac{C_{m}(M,\epsilon)-C_{1}^{m}(M,\epsilon)}{\sigma_{m}(\epsilon)}\sim
N(0,1)$

Then,

$\quad\alpha=P(\mbox{rejeitar}\quad H_{0}|H_{0}\,\,\mbox{é}\mbox{verdadeiro})$
$\alpha=P(|\sqrt{M}\frac{C_{m}(M,\epsilon)-C_{1}^{m}(M,\epsilon)}{\sigma_{m}(\epsilon)}|>z_{c}|H_{0}\,\,\mbox{é}\quad\mbox{verdadeiro})$
$\alpha=P(|\sqrt{M}\frac{C_{m}(M,\epsilon)-C_{1}^{m}(M,\epsilon)}{\sigma_{m}(\epsilon)}|>z_{c}\sim
N(0,1))$
$\alpha=1-2P(0\leq\sqrt{M}\frac{C_{m}(M,\epsilon)-C_{1}^{m}(M,\epsilon)}{\sigma_{m}(\epsilon)}\leq
z_{c})$

So,

$P(0\leq\sqrt{M}\frac{C_{m}(M,\epsilon)-C_{1}^{m}(M,\epsilon)}{\sigma_{m}(\epsilon)}\leq
z_{c})=\frac{1-\alpha}{2}$

If we look at the tables of the normal distribution $N(0,1)$, we see that you
to a degree of signicancia of $\alpha=5\%$, $z_{c}=1.96$. As, we reject the
null hypothesis, with degree of signicancia of $5\%$, if
$|\sqrt{M}\frac{C_{m}(\epsilon)-C_{1m}(\epsilon)}{\sigma_{m}(\epsilon)}|>1.96$.
Noting again the tables we can observe that in the case of Lorenz series, we
reject the null hypothesis, the data are IID. Otherwise case series formed by
random numbers.

Detected non-linealidade in the time series, the next step is to search
sensitive dependence on initial conditions.

###  6.3 Lyapunov exponents

This section we obtain results that characterize the sensitive dependence on
initial conditions. The study of the attractor and the dynamics is performed
in the space reconstructed using delay coordinates.

This section we obtain results that characterize the sensitive dependence on
initial conditions. The study of the attractor and the dynamics is performed
in the space reconstructed using delay coordinates. Chaotic systems are,
generally, strange attractors which are generated by stretching and
contractions. Stretches are responsible for sensitivity on initial conditions
and are characterized by the exponential distance in finite time, paths with
initial conditions very next. Contractions allow the pathlines may not move
away indefinitely and that they remain confined to a bounded phase space
region. Distance speed is measured by the Lyapunov exponents.

In the case of a discrete dynamic defined by a map system
$f:\\!\\!R^{n}\rightarrow\\!\\!R^{n}$,

$T(x)=D_{x}f$

is the Jacobian matrix of $f$. then writing

$T(x)=T^{n}_{x}=T(f^{n-1}x),\dots,T(fx),T(x)$

The largest Lyapunov exponent is given by:

$\lambda_{1}=\lim_{n\rightarrow\infty}\frac{1}{n}|\\!|T^{n}_{x}u|\\!|$

for almost all vector $u$, for more details see[6]. As we can see the first
problem for the calculation of Lyapunov exponents is to obtain the matrizes
$T(x)=D_{x}f$. In the case of having only a time series the calculation of
Lyapunov exponent is even more complicated. Eckman y Ruelle[6],[16] proposed
the first algorithms based on the following ideas.

Consider a dynamic generated by the differential equation:

${}\frac{dx(t)}{dt}=F(x(t))$ (1)

In $I\\!\\!R^{n}$. If we do $u(t)=F(x(t))$ in the previous equality we get,

${}\frac{du(t)}{dt}=(D_{x(t)}F)(u(t)).$ (2)

The retrieved system is linear in $u$. Then, if the system(1) solutions are
$x(t)=f^{t}(x(0))$, then the solutions of the system(2) are
$u(t)=(D_{x(0)}f^{t})u(0)$. So the matrix, $T_{t}^{x}=D_{x}f^{t}$, can be
calculated integrating the equations (1). This idea for a linear system to
calculate the matrix, $T_{t}^{x}=D_{x}f^{t}$, was used by Eckman and
Ruelle[16] for estimating of Lyapunov exponents from a time series.

Consider a time series, $s_{1},s_{2},\dots,s_{n}$. Using delay coordinates,
obtain points $X_{1},X_{2},\dots,X_{m}$, in the space $m$-dimensional of
reconstruction. If we develop $f$ around point $x_{i}$, we would have that for
$x_{j}$ next to it,

$\displaystyle{}f(X_{j})$ $\displaystyle\approx$ $\displaystyle
f(X_{i})+Df_{X_{i}}(X_{j}-X_{i})$ $\displaystyle f(X_{j})-f(X_{i})$
$\displaystyle\approx$ $\displaystyle Df_{X_{i}}(X_{j}-X_{i})$ (3)

So, $T_{\tau}^{X(i)}=D_{X(i)}f^{\tau}$ is estimated by the best linear
approximation of the map that for points $x(j)$ next to point $X(i)$, takes
$X(j)-X(i)$ in points $f^{\tau}(X(j))-f^{\tau}(X(i))=X(j+\tau)-X(i+\tau)$
close. This process can be thought as the linear approximation of the
derivatives at each point in space lathe overhauled do. With the same ideas,
authors such as Wolf, Swift, Swinney Vastano[22], M. Sano, y. Sawada[20]
proposed an algorithm for the calculation of Lyapunov exponents.

Another factor to consider is that the Lyapunov exponents are defined in terms
of limits, which guarantees its invariance by smooth transformations. If we
think of a finite series, there is no guarantee that interference (noise)
data, lead to the same exponents. Rosenstein, Collins and De Luca[19] and
Holger Kantz[17] directly presented new techniques based on the idea that the
greatest exponent of Lyapunov measure the average speed of estrangement of the
paths with conditions next. The estimation of the maximum exponent of Lyapunov
using the algorithm of Rosenstein was made in the reconstructed attractor of
Lorenz. The value of $0.7973003$ was obtained.

Figure 7: Departure of paths with initial conditions close to Lorenz
attractor.

##  7 Applications

The series studied was the series of prices of the shares of the mining
company Minsur, counting on prices from $13/10/1993$ up to $26/11/2014$ with a
total of $21$ years of daily observations.

###  7.1 Identifying non-linearity and chaos

For the use of the surrogates are generated series substitute which preserve
linear properties of the original series, for example, the graph(8) shows
histograms of the original series as well as his surrogate series.

Figure 8: Histogram of the number of stocks the company MINSUR, as well as his
surrogate series prices

The next step is the calculation of the predictions of each of the series,
original and substitute, and comparing related errors. The figure(9) shows
that errors of prediction of the original series presents a relatively small
short-term prediction error and substitute series shows a high prediction
error from the first moment.

Therefore the null($H_{0}$: error of the original series prediction =
surrogate series prediction error)hypothesis is rejected since the original
series prediction error is less than all their substitute series prediction
error. Therefore, we reject the fact that the original series is generated by
a linear stochastic process.

Figure 9: Errors in the prediction of the original series and its surrogate
series.

On the other hand, the BDS test is an alternative more statistics to detect
non-linear units, perform the test for several dimensions of immersion. In the
chart are the values of the BDS statistics to dimensions of $1$ to $15$.

Dimension of immersion(m) | BDS
---|---
1 | 165.270466
2 | 195.653634
3 | 236.554950
4 | 296.226274
5 | 382.825922
6 | 508.150676
7 | 690.060422
8 | 955.415301
9 | 1344.764026
10 | 1919.802065
11 | 2774.198843
12 | 4050.599850
13 | 5967.524832
14 | 8860.578053
15 | 13419.054424

We see the table that $|BDS|>1.96$.Therefore, we reject the null hypothesis
(that the data are IID), with a degree of significance of $5\%$.

Identified the non-linearity in the series, the next step is the search for
evidence of chaotic dynamics.

###  7.2 The greatest exponent of Lyapunov

Finally, we calculate the greatest exponent of Lyapunov as an indicator of
chaos. The theorem ergodic justice Oseledec method Rosenstein and Kantz to
estimate the maximum exponent of Lyaunov by the formula(6.3).In practice it is
convenient to only calculate the maximum exponent of Lyapunov, Since the whole
spectrum depends on the dimension of immersion and colors more exponents than
actually exist, or less.

Figure 10: Departure of paths using the algorithm of Rosenstein and Kantz
respectively.

##  8 Conclusions

Using the analysis nonlinear and chaos theory applied in the series of prices
of the shares of the mining company MINSUR obtained the following results:

  1. 1.

The data contains non-linear dependence.

  2. 2.

We cannot conclude the existence of chaos in the series.

Even considering non-linear dependence in the data results to detect chaos are
not conables. Some authors[29] estimated the largest Lyapunov exponent using
only the first iteration of estrangement.

A job to follow is the search for presence of orbits periodically unstable,
using for this purpose, the recurrence maps or maps of Poincaré, as well as
the presence of a fractal attractor.

##  9 Acknowledgements

The authors are so grateful to the Department of mathematics applied of the
Universidad Federal de Rio de Janeiro, UFRJ, and the Department of Sciences of
the University private of the North, UPN, Trujillo-Perú, where this study was
developed.

## References

  * [1] HOLGER KANTZ, THOMAS SCHREIBER, Nonlinear Time Series Analysis. Cambridge University Press 2004.
  * [2] J.P. ECKMANN, D.RUELLE, Ergodic Theory of chaos and strange attractor, Reviews of Modern Physics, vol 57 No 3, Part I, 1985.
  * [3] MAÑE, R., On the dimension of the compact invariant sets of certain nonlinear maps. Dinamical Systems and turbulence. Warwick, Lecture Notes in Mathematics 898, pp 230-242..
  * [4] TIM SAUER, JAMES A. YORKE, MARTIN CASDAGLI, Embedology. Springer Netherlands, Journal of Statistical Physics, Pages 579-616.
  * [5] R.L. Devaney, An Introduction to Chaotic Dynamical System, Second ed., Addison-Wesley, Menlo Park California, 1989.
  * [6] J.P. Eckman, D. Ruelle(1985). Ergodic theory of chaos and strange attractors. The American Physical Society. Rev. Mod. Phys. 57, 617656
  * [7] Floris Takens(1981). Detecting strange attractors in turbulence. Springer Berlin, Book Series Lecture Notes in Mathematics, Volume 898/1981, Pages 366-381.
  * [8] Tim Sauer, James A. Yorke, Martin Casdagli(1992). Embedology. Springer Netherlands, Journal of Statistical Physics, Pages 579-616.
  * [9] Rainer Hegger, Holger Kantz, and Thomas Schreiber(1999). Practical implementation of nonlinear time series methods: The TISEAN package. Chaos 9, 413 (1999); doi:10.1063/1.166424.
  * [10] Kaplan, D.(2002). Making and using surrogate data. http://www.macalester.edu/~kaplan/barcelona/surrogatewrap.pdf.
  * [11] Andre M. Fraser, Harry. L. Swinney(1986). Independent coordinates for strange attractors from mutual information. Physical Review A (General Physics), Volume 33, Issue 2, pp.1134-1140
  * [12] Provenzale, A., Smith, L. A., Vio, R., Murante G.(1992). Distinguishing between low-dimensional dynamics and randomness in measured time series. Physica D, 58, 31.
  * [13] Michael Small, Dejin Yu, Jennifer Simonotto, Robert G Harrison, Neil Grubb, K.A.A Fox.(2002). Uncovering non-linear structure in human ECG recordings. Chaos, Solitons, Fractals Volume 13, Issue 8, Pages 1755-1762.
  * [14] C. E. Shannon(1948). A mathematical theory of communication. Bell System Technical Journal, vol. 27, pp. 379-423 and 623-656
  * [15] Thomas Schreiber, Andreas Schmitz(1999). Improved surrogate data for nonlinearity tests. Phys. Rev. Lett. 77, 635.
  * [16] J.P. Eckman, S. Oliffson Kamphorst, D. Ruelle, S. Ciliberto(1986). Lyapunov exponents from time series. Phys. Rev. A 34, 49714979
  * [17] Holger Kantz(1994). A robust method to estimate the maximal Lyapunov exponent of a time series. Physics Letters A, Volume 185, Issue 1, p. 77-87.
  * [18] Holger Kantz, Thomas Schreiber(1997). Nonlinear Time Series Analysis. Cambridge University Press.
  * [19] Michael T. Rosenstein, James J. Collins, Carlo J. De Luca(1993). A practical method for calculating largest Lyapunov exponents from small data sets. Physica D: Nonlinear Phenomena Volume 65, Issues 1-2, Pages 117-134
  * [20] M. Sano, Y. Sawada (1985). Measurement of the Lyapunov Spectrum from a Chaotic Time Series. APS Journals Phys. Rev. Lett. Volume 55, ssue 10 Phys. Rev. Lett. 55, 10821085
  * [21] W. A. Broock, J. A. Scheinkman, W. D. Dechert, B. LeBaron(1996). A test for independence based on the correlation dimension. Journals in its journal Econometric Reviews.Volume 15, Issue 3, Pages: 197-235
  * [22] Wolf, Alan; Swift, Jack B.; Swinney, Harry L.; Vastano, John A.(1984). Determining Lyapunov exponents from a time series. Physica D: Nonlinear Phenomena, Volume 16, Issue 3, p. 285-317.
  * [23] Scheinkman, Jose A.(1990). Nonlinearities in Economic Dynamics. Royal Economic Society, Economic Journal, Issue 400, Pages: 33-48.
  * [24] Theiler, James; Eubank, Stephen; Longtin, André; Galdrikian, Bryan; Doyne Farmer, J.(1992). Testing for nonlinearity in time series: the method of surrogate data. Physica D: Nonlinear Phenomena, Volume 58, Issue 1-4, p. 77-94.
  * [25] Thomas Schreiber, Andreas Schmitz.(1999). Improved surrogate data for nonlinearity tests. Phys. Rev. Lett. 77 (1996) 635.
  * [26] David A. Hsieh.(1991). Chaos and Nonlinear Dynamics: Application to Financial Markets. The Journal of Finance, Vol. 46, No. 5 (Dec., 1991), pp. 1839-1877.
  * [27] Thomas Willey.(1992). Testing for nonlinear dependence in daily stock indices. Journal of Economics and Business Volume 44, Issue 1, Pages 63-76.
  * [28] Apostolos Serletis, Periklis Gogas.(1997). Chaos in East European black market exchange rates. Research in Economics Volume 51, Issue 4, Pages 359-385.
  * [29] Michael Bask.(1996). Dimensions and Lyapunov Exponents from Exchange Rate Series. Chaos, Solirons and Fractals Vol. 7, No. 12, pp. 2199-2214.


# Multilingual Email Zoning

Bruno Jardim
Cleverly, Lisbon, Portugal
NOVA-IMS, Lisbon, Portugal
<EMAIL_ADDRESS>
&Ricardo Rei
NOVA-IMS, Lisbon, Portugal
Unbabel, Lisbon, Portugal
<EMAIL_ADDRESS>
Mariana S. C. Almeida
Cleverly, Lisbon, Portugal
<EMAIL_ADDRESS>

###### Abstract

The segmentation of emails into functional zones (also dubbed email zoning) is
a relevant preprocessing step for most NLP tasks that deal with emails.
However, despite the multilingual character of emails and their applications,
previous literature regarding email zoning corpora and systems was developed
essentially for English.

In this paper, we analyse the existing email zoning corpora and propose a new
multilingual benchmark composed of 625 emails in Portuguese, Spanish and
French. Moreover, we introduce Okapi, the first multilingual email
segmentation model based on a language agnostic sentence encoder. Besides
generalizing well for unseen languages, our model is competitive with current
English benchmarks, and reached new state-of-the-art performances for domain
adaptation tasks in English.

##  1 Introduction

Worldwide, email is a predominant means of social and business communication.
Its importance has attracted studies in areas of Machine Learning (ML) and
Natural Language Processing (NLP), impacting a wide range of applications,
from spam filtering Qaroush et al. (2012) to network analysis Christidis and
Losada (2019).

Figure 1: Okapi is composed of two building blocks: 1) a multilingual sentence
encoder (XLM-RoBERTa) to derive sentence embeddings; and 2) a segmentation
module that uses a BiLSTM with a CRF on top to classify each sentence into an
email zone.

The email body is commonly perceived as unstructured textual data with
multiple possible formats. However, it is possible to discern a level of
formal organization in the way most emails are formed. Different functional
parts can be identified such as greetings, signatures, quoted content, legal
disclaimers, etc. The segmentation of email text into zones, also known as
email zoning Lampert et al. (2009), has since become a prevalent preprocessing
task for a diversity of downstream applications, such as author profiling
Estival et al. (2007), request detection Lampert et al. (2010), uncover of
technical artifacts Bettenburg et al. (2011), automated template induction
Proskurnia et al. (2017), email classification Kocayusufoglu et al. (2019) or
automated email response suggestion Kannan et al. (2016); Chen et al. (2019).

Since email communication is a worldwide phenomenon, all previous applications
are in fact highly multilingual. Despite this, email zoning literature remains
English-centric and without a standardize zone taxonomy. To mitigate those
problems, we make the following research contributions:

  1. 1.

We discuss the existing zoning taxonomies and their limitations.

  2. 2.

We release Cleverly zoning corpus, the first multilingual corpus for email
zoning. This corpus consists of 625 emails in 3 languages rather than English
(Portuguese, Spanish and French), and encompasses 15 email zones as defined in
(Bevendorff et al., 2020)

  3. 3.

We introduce Okapi, a multilingual email segmentation system built on top of
XLM-RoBERTa Conneau et al. (2020) that can be easily extended to 100
languages.

To the best of our knowledge, Okapi is the first end-to-end multilingual
system exploring pre-trained transformer models Vaswani et al. (2017) to
perform email zoning. Besides having multilingual capabilities, Okapi is
competitive with existing approaches for English email zoning, and attained
state-of-the-art performance in domain adaptation tasks for English email
zoning.

The rest of the paper is organized as follows: Section 2 presents an overview
of the related literature. Section 3 provides a comprehensive review of
existing email zoning corpora, and introduces Cleverly zoning corpus, our new
multilingual email zoning corpus. Section 4 describes the Okapi model
architecture. Section 5 reports and discusses the results achieved. Finally,
Section 6 concludes the paper.

##  2 Literature Review

Authors | Source | emails | zones | Language
---|---|---|---|---
(Carvalho and Cohen, 2004)4 | 20 Newsgroup5 | 617 | 2 | English
(Estival et al., 2007)6 | Donated6 | 9,836 | 5 | English
(Lampert et al., 2009)7 | Enron8 | 400 | 3/9 | English
(Repke and Krestel, 2018)9 | Enron8 | 800 | 2/5 | English
| ASF11footnotemark: 1 | 500 | 2/5 | English
Bevendorff et al. (2020)1010footnotemark: 10 | Gmane3 | 3,033 | 15 | Multilingual*
| Enron8 | 300 | 15 | English
Ours | Gmane3 | 625 | 15 | Multilingual
Table 1: Summary of existing email zoning corpora. *Note that, although
Bevendorff et al. (2020)’s Gmane corpus is technically multilingual, it only
has 38 non-English test emails that are spread over 13 different languages.

Chen et al. (1999) were one of the pioneers in the topic of email
segmentation. Looking at linguist and geometrical patterns, their work focuses
on the identification of email signature. Similarly, Carvalho and Cohen (2004)
developed Jangada, a supervised learning system that classifies each line
using a Conditional Random Field (CRF) Lafferty et al. (2001) and a sequence-
aware perceptron Collins (2002), that identifies signature blocks and quoted
text from previous emails. Tang et al. (2005) proposed an email data cleansing
system based on a Support Vector Machine (SVM) Cortes and Vapnik (1995) model
that aimed at filtering the non-textual noisy content from emails
independently of downstream text mining applications, based on hand-coded
features.

Estival et al. (2007) were the first to introduce a general segmentation
schema for email text. Segmentation of emails is a crucial part on their work,
which aims at identifying the author’s basic demographic and psychometric
traits. In that work, the authors compared a range of ML algorithms together
with feature selection to classify email segments into five functional parts,
attaining improvements in the end task of auto profiling. Later, Lampert et
al. (2009) formally defined the functional parts as email zones, describing
the different segments inside email messages based on graphic, orthographic,
and lexical features. Lampert et al. (2009) also proposed Zebra, an email
zoning system based on a SVM. In a posterior work towards detecting emails
containing requests for action, Lampert et al. (2010) used Zebra to “zone”
emails, considering only the zones that had relevant patterns to increase the
accuracy of their request detection task.

As email zoning surpassed its original purpose of signature identification and
text cleansing into a more general task, Repke and Krestel (2018) extended its
utility to thread reconstruction. Inspired by Zebra Lampert et al. (2009), the
authors proposed Quagga Repke and Krestel (2018), a neural system with a
Convolutional Neural Network (CNN) LeCun et al. (1989) to produce sentence
representations followed by a Recurrent Neural Network (RNN) Elman (1990).
Quagga was trained and evaluated on English emails from both the Enron Klimt
and Yang (2004) corpus and the public mail archives of the Apache Software
Foundation (ASF)111http://mail-archives.apache.org/mod_mbox/, outperforming
Jangada and Zebra.

Until very recently, email zoning resorted to small samples of mailing lists
or newsgroup corpus and was limited to the English language. Bevendorff et al.
(2020) were the first to crawl email at scale, extracting 153 million emails
from the Gmane email-to-newsgroup gateway222https://news.gmane.io/ in
different languages such as English, Spanish, French and
Portuguese333https://webis.de/data.html?q=Webis-Gmane-19. The authors
annotated email zones for a subset of Gmane English emails and, due to the
idiosyncratic characteristics of the corpora, they developed a more fine-
grained zone classification schema with 15 zones. Moreover, Bevendorff et al.
(2020) introduced an email zoning system, named Chipmunk, that combines a
Bidirectional Gated Recurrent Unit (BiGRU) Cho et al. (2014) with a CNN. When
compared to other models in the literature, Chipmunk achieved better
performance.

##  3 Email Zoning Corpora

Several corpora and zoning schemes have been proposed in the literature under
different contexts. This section provides an overview of the existing corpora,
hoping to make it easier to develop and compare new email zoning methods in
the future.

Table 1 compiles the information of existing email zoning corpus. To the best
of our knowledge, Carvalho and Cohen (2004) released the first email zoning
corpus. The corpus consists of 617
emails444http://www.cs.cmu.edu/~vitor/codeAndData.html from the 20 Newsgroup
corpus555http://qwone.com/~jason/20Newsgroups/ identified with two zones:
signature and quotation. Despite the usefulness of identifying those zones for
email cleansing, this level of detail is still insufficient for a general
email segmentation.

Estival et al. (2007) released a corpus of 9,836 recruited respondents donated
email messages666available upon contact with the authors. and introduced a
wider annotation schema focusing on more email parts: author text, signature,
advertisement, quoted text, and reply lines. However, Estival et al. (2007)
still did not divide the email text into some other relevant zones, such as
greetings, closings nor identify attachments and code lines.

Lampert et al. (2009) were arguably the first to conceptualize the email
zoning task and fully define the characteristics of each identified zone, as
well as dividing the authored text into different zones. They annotated 400
English emails777http://zebra.thoughtlets.org/ from the Enron email corpus
database dump, identifying 3 email zones: sender, quoted conversation and
boilerplate zones, each containing a different set of sub-zones, within a
total of 9 sub-zones.

Repke and Krestel (2018) also resorted to the Enron
database888http://www.cs.cmu.edu/~enron/, annotating a total of 800
emails999https://github.com/HPI-Information-Systems/Quagga. Reconsidering the
task as thread reconstruction, they produced a new annotation schema,
considering a 2-level and a 5-level approach (the latter being a refinement of
the 2-level segmentation). Repke and Krestel (2018) also annotated 500 ASF
emails7 using both the 2-level and 5-level taxonomies. Their 5-level
annotation schema consists of segmenting emails into: body (typically
comprising $\sim\\!\\!80\%$ of the lines), header, signoff, signature and
greetings.

Bevendorff et al. (2020) introduced the Gmane corpus for email
zoning101010https://github.com/webis-de/acl20-crawling-mailing-lists. Even
though the corpus is composed of 31 languages, the annotated emails are mostly
in English, and their test set only contains a residual number of non-English
emails (38 emails covering 13 different languages), which is insufficient for
a consistent multilingual evaluation. Due to the richness of the Gmane
conversations on technical topics, Bevendorff et al. (2020) developed a more
fine grained classification schema, considering the segmentation of blocks of
code, log data and technical data. Whilst also preserving most of the common
zones introduced in previous works, they ended up with a total of 15 zones:
closing, inline headers, log data, MUA signature, paragraph, patch, personal
signature, quotation, quotation marker, raw code, salutation, section heading,
tabular, technical, visual separator. Following the same zone taxonomy they
also released a set of 300 English emails from the Enron database dump. In
both Enron and Gmane emails, the majority of the email segments belong to the
paragraph and quotation zones. This being said, Gmane has much more lines of
quotation than paragraph, while Enron is the other way around.

Overall, email zoning corpora show a great variability of zone taxonomies and
most works have introduced new zones to face the nature of each email source
or downstream task. The Enron database dump has been the most used source to
retrieve emails to build new corpus. On the other hand, the recent Gmane raw
dump of emails is multilingual and it contains various functional zones, which
opens the door to new challenges in email zoning and multilingual
methodologies.

###  3.1 Cleverly Zoning Corpus

| pt | es | fr
---|---|---|---
# zones | 15 | 14 | 14
# emails | 210 | 200 | 215
# lines | 12366 | 9824 | 6958
# lines / email | 58.9 | 49.1 | 32.4
# zones /email | 8.6 | 6.5 | 5.9
# unique zones / email | 5.8 | 5.1 | 4.9
Table 2: Some statistics of the Cleverly zoning corpus.

This section presents Cleverly zoning corpus, the first multilingual email
zoning corpus. To create the corpus, we searched the Gmane raw corpus
Bevendorff et al. (2020) for Portuguese (pt), Spanish (es) and French (fr)
emails. Then, following the classification schema proposed by Bevendorff et
al. (2020), we produced a total of 625 annotated emails.

Table 2 compiles a brief description of the email statistics for each of the
languages. While French is the language with more emails, Portuguese and
Spanish emails tend to be longer, resulting in a greater amount of lines and
an overall higher number of zones per email. The distribution of zones is
similar between the three languages, as detailed in Table 3.

Zone | pt (%) | es (%) | fr (%)
---|---|---|---
Quotation | 52.43 | 59.02 | 46.20
Paragraph | 16.33 | 17.36 | 27.61
MUA Sig. | 12.04 | 3.84 | 9.04
Personal Sig. | 3.93 | 4.47 | 2.00
Visual Sep. | 2.94 | 2.29 | 2,60
Quot. Mark. | 2.72 | 1.54 | 2.10
Closing | 2.63 | 2.00 | 3.73
Log Data | 1.04 | 3.79 | 1.82
Raw Code | 1.28 | 2.45 | 2.07
Inl. Head. | 2.96 | 0,82 | 1.33
Salutation | 0,96 | 0.81 | 1,35
Tabular | 0.32 | 0.42 | 0.27
Technical | 0.30 | 1.00 | 0.38
Patch | 0.02 | 0.20 | 0.02
Sec. Head. | 0.15 | 0.04 | 0.03
Table 3: Distribution, for each language, of the number of lines per zone in the Cleverly zoning corpus. The distributions were obtained by averaging statistics from both annotators. measure | pt | es | fr
---|---|---|---
accuracy | 0.93 | 0.92 | 0.96
$F_{1}$ A1A2 | 0.93 | 0.92 | 0.96
$F_{1}$ A2A1 | 0.94 | 0.92 | 0.96
$k$ | 0.90 | 0.87 | 0.94
Table 4: Inter-annotator agreement for each language in the Cleverly zoning
corpus, using Cohen’s kappa ($k$), accuracy and $F_{1}$ between annotators A1
and A2.

The annotation was carried out by two annotators. The first annotator was a
native Portuguese speaker and the second annotator a native Spanish speaker,
both with academical background in French and fluent in the third language.
Each email was annotated by both annotators using the
tagtog111111https://www.tagtog.net annotation tool.

Table 4 shows the inter-annotator agreement scores for each language using the
Cohen’s kappa coefficient ($k$) McHugh (2012), accuracy and $F_{1}$ of one
annotator versus the other. All annotations and required information to
compile the original emails are freely available at
https://github.com/cleverly-ai/multilingual-email-zoning.

##  4 Okapi Architecture

We propose Okapi, an email segmentation model composed of two building blocks:
a multilingual sentence encoder and a segmentation module. Figure 1 shows the
Okapi architecture.

###  4.1 Multilingual Sentence Encoder

To address the multilingual nature of emails we developed a language agnostic
sentence encoder that turns each email line into an embedding. Figure 2
illustrates the encoding process.

Figure 2: To derive a cross-lingual line embedding we use XLM-RoBERTa Conneau
et al. (2020) to extract word-level embeddings, and then apply average pooling
to the last 4 layers. This leads to a final 3072 features embedding.

Given an email line $x=\left[x_{0},x_{1},...,x_{n}\right]$, our encoder module
uses XLM-RoBERTa (base) Conneau et al. (2020) to produce an embedding
$\bm{e}_{j}^{(\ell)}$ for each token $x_{j}$ and each layer
$\ell\in\\{0,1,...,13\\}$. Since it has been shown that BERT-like models
capture within the network layers diverse linguistic information, and,
particularly, the last layers preserve most of the semantic information Tenney
et al. (2019), we keep, for each sentence, only the word embeddings from the
last 4 layers. Lastly, as in Reimers and Gurevych (2019), these word embedding
are turned into a 3072 sentence embedding $\bm{s_{k}}$ by averaging the
concatenation of the 4 word layer embeddings.

###  4.2 Segmentation Module

After passing each email line into the previous sentence encoder we get a
cross-lingual line embedding $\bm{s_{k}}$. After that, we pass all line
embeddings of an email into a Bidirectional Long Short-Term Memory (BiLSTM)
Graves and Schmidhuber (2005), with 1 layer and 64 hidden units, to derive
compact line representations that encompass information from the entire
structure of the email. Finally, as in Huang et al. (2015), we use a CRF
output layer to predict the zone of each line in the document. Preliminary
experiments showed that not using CRF either slightly deteriorates model
performance or does not have an impact on the results.

###  4.3 Training setup

During training, XLM-RoBERTa’s weights were kept frozen and only the BiLSTM
and CRF layers were updated. We experimented BiLSTM with $16,32,64,128,256$
and $512$ hidden units and more layers, but in the end, having a small
segmentation module, with $64$ hidden units and $1$ layer, generically yielded
the best performances in the validation splits. We used a dropout layer of
value 0.25 between the BiLSTM and the CRF, and the RMSprop optimizer with a
fixed learning rate of 0.001.

##  5 Results and Discussion

In this section, we analyse both multilingual and monolingual capabilities of
Okapi, considering various zoning corpora and annotation schemas.

###  5.1 Multilingual Email Zoning

zone | pt |  | es |  | fr
---|---|---|---|---|---
All | 0.91 |  | 0.93 |  | 0.93
Quotation | 0.99 |  | 0.99 |  | 0.99
Paragraph | 0.91 |  | 0.96 |  | 0.92
MUA Sig. | 0.95 |  | 0.82 |  | 0.91
Personal Sig. | 0.81 |  | 0.87 |  | 0.79
Visual Sep. | 0.92 |  | 0.90 |  | 0.96
Quot. Mark. | 0.55 |  | 0.97 |  | 0.97
Closing | 0.59 |  | 0.58 |  | 0.69
Log Data | 0.56 |  | 0.53 |  | 0.57
Raw Code | 0.54 |  | 0.74 |  | 0.84
Inl. Head. | 0.78 |  | 0.77 |  | 0.58
Salutation | 0.65 |  | 0.69 |  | 0.89
Tabular | 0.30 |  | 0.00 |  | 0.60
Technical | 0.67 |  | 0.56 |  | 0.48
Patch | 0.00 |  | 0.00 |  | 0.00
Sec. Head. | 0.34 |  | 0.00 |  | 0.00
Table 5: Multilingual zero-shot evaluation of Okapi, using Cleverly zoning
corpus. Global accuracy and recall of each zone, computed by averaging the
scores regarding both annotators.

We evaluate the multilingual capabilities of Okapi in a zero-shot fashion. For
that, we trained the model with the Gmane English corpus released by
Bevendorff et al. (2020), and tested it with the Cleverly multilingual corpus
that we annotated for Portuguese, Spanish and French.

Table 5 presents the performances of Okapi in our multilingual corpus for each
zone. Comparing with the typical performance of email zoning and the Gmane
corpus (see next Tables), Okapi achieves quite reasonable performances,
confirming its multilingual character. As expected, zone recall seems to be
dependent on the total number of lines per zone.

###  5.2 English Email Zoning

Model | Zones | Enron | ASF
---|---|---|---
Jangada | 2 | 0.88 | 0.97
Zebra | 2 | 0.25 | 0.18
Quagga | 2 | 0.98 | 0.98
Okapi | 2 | 0.99 | 0.99
Jangada | 5 | 0.85 | 0.91
Zebra | 5 | 0.24 | 0.20
Quagga | 5 | 0.93 | 0.95
Okapi | 5 | 0.96 | 0.95
Table 6: Email zoning accuracy of various models, for the corpus of Repke and Krestel (2018). Model | Zones | Gmane | Enron
---|---|---|---
Tang et al. (2005) | 15 | 0.80 | 0.73
Quagga | 15 | 0.94 | 0.83
Chipmunck | 15 | 0.96 | 0.88
Okapi | 15 | 0.96 | 0.88
Table 7: Zoning accuracy of various models, under the 15-level zoning schema
of Bevendorff et al. (2020).

Resorting to the numbers reported in the literature for email zoning, we
compared Okapi with existing monolingual methods using various English corpora
and zoning taxonomies. In particular, Table 6 compares Okapi with other zoning
systems on the corpora annotated by Repke and Krestel (2018) with 2 and 5
types of zones; and Table 7 shows the results obtained with the most recent
and fine-grained annotation schema with 15 zones proposed by Bevendorff et al.
(2020). For all those combination of corpora and zoning strategies, Okapi
achieved competitive, and sometimes better results when compared with state-
of-the-art methods for English email zoning, being simultaneously able to
perform well on different languages.

| Corpus | Accuracy | Accuracy
---|---|---|---
Model | Train/Test | 2 zones | 5 zones
Quagga | Enron/ASF | 0.94 | 0.86
Okapi | Enron/ASF | 0.98 | 0.93
Quagga | ASF/Enron | 0.86 | 0.80
Okapi | ASF/Enron | 0.97 | 0.88
Table 8: Comparison between Okapi and Quagga for domain adaptation,
considering Repke and Krestel (2018) 2 and 5 zoning schema.

Finally, we analyse how Okapi adapts to new domains. For that, Table 8 shows
the performance of both Okapi and Quagga Repke and Krestel (2018), when
evaluated in a different corpus then the one they were trained on. In these
experiments, Okapi clearly outperformed Quagga, indicating a superior ability
to generalize to unseen domains.

##  6 Conclusion

To overcome the English-centric email zoning literature we propose Okapi.
Besides having multilingual capabilities, the proposed model is competitive
with existing approaches for English email zoning, and attained state-of-the-
art performance in domain adaptation tasks of English email zoning.
Futhermore, to evaluate our model and to foster future research into
multilingual email zoning, we release Cleverly zoning corpus – a corpus with
625 emails annotated in Portuguese, Spanish and French.

##  7 Acknowledgments

This project has received funding from the European Union’s Horizon 2020
research and innovation program under grant agreement No 873904.

## References

  * Bettenburg et al. (2011) Nicolas Bettenburg, Bram Adams, Ahmed E. Hassan, and Michel Smidt. 2011.  A lightweight approach to uncover technical artifacts in unstructured data.  In _International Conference on Program Comprehension_ , pages 185–188, Los Alamitos, CA, USA. IEEE Computer Society.
  * Bevendorff et al. (2020) Janek Bevendorff, Khalid Al Khatib, Martin Potthast, and Benno Stein. 2020.  Crawling and preprocessing mailing lists at scale for dialog analysis.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1151–1158, Online. Association for Computational Linguistics.
  * Carvalho and Cohen (2004) Vitor R. Carvalho and William W. Cohen. 2004.  Learning to extract signature and reply lines from email.  In _CEAS - 2004 (Conference on Email and Anti-Spam)_ , Mountain View, CA, USA.
  * Chen et al. (1999) Hao Chen, Jianying Hu, and Richard W. Sproat. 1999.  Integrating geometrical and linguistic analysis for email signature block parsing.  _ACM Transactions on Information Systems_ , 17(4):343–366.
  * Chen et al. (2019) Mia Xu Chen, Benjamin N. Lee, Gagan Bansal, Yuan Cao, Shuyuan Zhang, Justin Lu, Jackie Tsay, Yinan Wang, Andrew M. Dai, Zhifeng Chen, Timothy Sohn, and Yonghui Wu. 2019.  Gmail smart compose: Real-time assisted writing.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD ’19, page 2287–2295, New York, NY, USA. Association for Computing Machinery.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using RNN encoder–decoder for statistical machine translation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1724–1734, Doha, Qatar. Association for Computational Linguistics.
  * Christidis and Losada (2019) Panayotis Christidis and Álvaro G. Losada. 2019.  Email based institutional network analysis: Applications and risks.  _The Social Sciences_ , 8(11):306.
  * Collins (2002) Michael Collins. 2002.  Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms.  In _Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing - Volume 10_ , EMNLP ’02, page 1–8, USA. Association for Computational Linguistics.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Cortes and Vapnik (1995) Corinna Cortes and Vladimir Vapnik. 1995.  Support-vector networks.  _Machine Learning_ , 20:273–297.
  * Elman (1990) Jeffrey L. Elman. 1990.  Finding structure in time.  _Cognitive Science_ , 14(2):179 – 211.
  * Estival et al. (2007) Dominique Estival, Tanja Gaustad, Son Bao Pham, Will Radford, and Ben Hutchinson. 2007.  Author profiling for english emails.  In _Proceedings of the 10th Conference of the Pacific Association for Computational Linguistics_ , pages 263–272.
  * Graves and Schmidhuber (2005) Alex Graves and Jürgen Schmidhuber. 2005.  Framewise phoneme classification with bidirectional LSTM and other neural network architectures.  _Neural Networks_ , 18(5):602–610.
  * Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015.  Bidirectional LSTM-CRF models for sequence tagging.  _CoRR_ , abs/1508.01991.
  * Kannan et al. (2016) Anjuli Kannan, Karol Kurach, Sujith Ravi, Tobias Kaufmann, Andrew Tomkins, Balint Miklos, Greg Corrado, Laszlo Lukacs, Marina Ganea, Peter Young, and Vivek Ramavajjala. 2016.  Smart reply: Automated response suggestion for email.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’16, page 955–964, New York, NY, USA. Association for Computing Machinery.
  * Klimt and Yang (2004) Bryan Klimt and Yiming Yang. 2004.  The enron corpus: A new dataset for email classification research.  In _Proceedings of the 15th European Conference on Machine Learning_ , ECML’04, page 217–226, Berlin, Heidelberg. Springer-Verlag.
  * Kocayusufoglu et al. (2019) Furkan Kocayusufoglu, Ying Sheng, Nguyen Vo, James Wendt, Qi Zhao, Sandeep Tata, and Marc Najork. 2019.  Riser: Learning better representations for richly structured emails.  In _The World Wide Web Conference_ , WWW ’19, page 886–895, New York, NY, USA. Association for Computing Machinery.
  * Lafferty et al. (2001) John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001.  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.  In _Proceedings of the Eighteenth International Conference on Machine Learning_ , ICML ’01, page 282–289, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
  * Lampert et al. (2009) Andrew Lampert, Robert Dale, and Cécile Paris. 2009.  Segmenting email message text into zones.  In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: A Meeting of SIGDAT, EMNLP 2009_ , pages 919–928. Association for Computational Linguistics (ACL).
  * Lampert et al. (2010) Andrew Lampert, Robert Dale, and Cecile Paris. 2010.  Detecting emails containing requests for action.  In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 984–992, Los Angeles, California. Association for Computational Linguistics.
  * LeCun et al. (1989) Yann LeCun, Bernhard E. Boser, John S. Denker, Donnie Henderson, Richard E. Howard, Wayne E. Hubbard, and Lawrence D. Jackel. 1989.  Backpropagation applied to handwritten zip code recognition.  _Neural Computation_ , 1(4):541–551.
  * McHugh (2012) M. McHugh. 2012.  Interrater reliability: the kappa statistic.  _Biochemia Medica_ , 22:276 – 282.
  * Proskurnia et al. (2017) Julia Proskurnia, Marc-Allen Cartright, Lluis Garcia-Pueyo, Ivo Krka, James B. Wendt, Tobias Kaufmann, and Balint Miklos. 2017.  Template induction over unstructured email corpora.  In _Proceedings of the 26th International Conference on World Wide Web_ , WWW ’17, page 1521–1530, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
  * Qaroush et al. (2012) Aziz Qaroush, Ismail M. Khater, and Mahdi Washaha. 2012.  Identifying spam e-mail based-on statistical header features and sender behavior.  In _Proceedings of the CUBE International Information Technology Conference_ , page 771–778, NY, USA. Association for Computing Machinery.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Repke and Krestel (2018) Tim Repke and Ralf Krestel. 2018.  Bringing back structure to free text email conversations with recurrent neural networks.  In _European Conference on Information Retrieval_ , pages 114–126. Springer.
  * Tang et al. (2005) Jie Tang, Hang Li, Yunbo Cao, and Zhaohui Tang. 2005.  Email data cleaning.  In _Proceedings of the Eleventh ACM SIGKDD International Conference on Knowledge Discovery in Data Mining_ , KDD ’05, page 489–498, New York, NY, USA. Association for Computing Machinery.
  * Tenney et al. (2019) Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019.  BERT rediscovers the classical NLP pipeline.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4593–4601, Florence, Italy. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30, pages 5998–6008. Curran Associates, Inc.