Approximation of the Gompertz function with a multilogistic function

Grzegorz Rza̧dkowski

Department of Finance and Risk Management, Warsaw University of Technology,
Narbutta 85, 02-524 Warsaw, Poland

e-mail<EMAIL_ADDRESS>
###### Abstract

The paper deals with the comparison of the Gompertz function and the logistic
function. We show that the Gompertz function can be approximated with high
accuracy by a sum of three logistic functions (multilogistic function). Two of
them are increasing and one is decreasing. We use second-order logistic
wavelets to estimate the parameters of the multilogistic function.

Keywords: Gompertz function, logistic function, logistic wavelet, scalogram.

2020 Mathematics Subject Classification: 42C40, 65T60, 11B83

##  1 Introduction

The Gompertz function is described by the following autonomous differential
equation of the first order

$x^{\prime}(t)=sx\log\frac{x_{sat}}{x}\quad\quad x(0)=x_{0}>0,$ (1)

with parameters $s-$growth rate and $x_{sat}-$saturation level (asymptote),
$0<x_{0}<x_{sat}$; $\log$ is the natural logarithm. After solving (1) we can
write the Gompertz function in the following convenient form

$x(t)=x_{sat}e^{-e^{-s(t-t_{0})}},$ (2)

where constant $t_{0}$ appears in the integration process of (1) and is
connected with the initial condition $x(0)=x_{0}=x_{sat}e^{-e^{st_{0}}}$, thus
$t_{0}=\frac{1}{s}\log\log(x_{sat}/x_{0})$. It is easy to check that $t_{0}$
is also the inflection point of $x(t)$ (2). Fig 1 shows an exemplary Gompertz
funcion.

Figure 1: Exemplary Gompertz function with parameters
$x_{sat}=100,s=0.15,t_{0}=20$

The Gompertz function (2) is an example of the so-called S-shaped curves and
was first described and applied in actuarial mathematics in 1825 by Benjamin
Gompertz [5]. Since then, the Gompertz function has found applications in
probability theory (Gumbel distribution), biology, medicine, economics,
engineering, physics and other fields. Many interesting applications of the
Gompertz curve are given by Waliszewski and Konarski [17].The first hundred
years of the use of this function are well described by Winsor [18]. The
interesting story of the next almost one hundred years can be found in the
article by Tjørve and Tjørve [16]. In recent years, many papers have appeared
in which the Gompertz function was used to describe the spread of COVID-19
pandemy (see Ohnishi et al [13], Dhahbi et al [2], Kundu et al [8], Estrada
and Bartesaghi [3]).

The logistic equation defining the logistic function $x=x(t)$ has the form

$x^{\prime}(t)=\frac{s}{x_{sat}}\>x(x_{sat}-x),\quad x(0)=x_{0}.$ (3)

where $t$ is time, and the parameters $s$-steepness or slope coefficient and
$x_{sat}$-saturation level are real constants. We assume here that the
saturation level can be a positive or negative number. The integral curve
$x(t)$ of equation (3) satisfying the condition that $x(t)$ lies between zero
and $x_{sat}$ is called the logistic function. The logistic function is used
to describe and model various phenomena in physics, economics, medicine,
biology, engineering, sociology and many other sciences. Logistic functions
now seem even more important from the point of view of their possible
applications, due to the theory of the Triple Helix (TH) developed in the
1990s by Etzkowitz and Leydesdorff [4] (see also Leydesdorff [9]). This theory
explains the phenomenon of creating and introducing innovations under the
influence of the interaction of three factors University-Industry-Government
and relations between them. According to the TH theory, the phenomenon of the
emergence of innovations can be described by means of logistic functions.
Ivanova [7], [6] has shown that the KdV equation naturally appears in TH
theory and has also applied it to other fields such as the COVID-19 pandemic
or financial markets.

After solving the differential equation (3) we obtain the logistic function in
the form

$x(t)=\frac{x_{sat}}{1+e^{-s(t-t_{0})}},$ (4)

where $t_{0}$ is the inflection point associated with the initial condition
$\displaystyle x(0)=x_{0}=\frac{x_{sat}}{1+e^{st_{0}}}$, then $\displaystyle
t_{0}=\frac{1}{s}\log\Big{(}\frac{x_{sat}-x_{0}}{x_{0}}\Big{)}$. At the point
$t_{0}$, $x(t_{0})=x_{sat}/2$. An exemplary logistic function is shown in the
Fig 2.

Figure 2: Exemplary logistic function with parameters
$x_{sat}=100,s=0.15,t_{0}=20$

Mahjan et al [10] show areas in which various S-shaped curves are used as
diffusion models. The authors find that the Bass function is useful for
modeling of consumer durable goods, retail services, agricultural, education,
and industrial innovations. The logistic curve serves as model in industrial,
high technology, administrative innovations, and the Gompertz function can be
used for modeling consumer durable goods and agriculture innovations.

The aim of this paper is to show that, for a given Gompertz function, it is
possible to find some logistic functions (waves) such that their sum
(multilogistic function) approximates the Gompertz function with high
accuracy.

The structure of the article is as follows. In Sec. 2 we describe the basic
properties of the second-order logistic wavelets. Sec. 3 is devoted to show
connections between the Gompertz function and the logistic function. We prove
that the Gompertz function can be approximated with high accuracy by a
multilogistic function. The paper is concluded in Sec. 4.

##  2 Logistic wavelets

We briefly outline the basic general properties of wavelets (cf. [1, 12, 11]),
which we will need later. A wavelet or mother wavelet (see Daubechies [1],
p.24 ) is a function $\psi\in L^{1}(\mathbb{R})$ such that the following
admissibility condition holds:

$C_{\psi}=2\pi\int_{-\infty}^{\infty}|\xi|^{-1}|\widehat{\psi}(\xi)|^{2}d\xi<\infty,$
(5)

where $\widehat{\psi}(\xi)$ is the Fourier transform $F(\psi)$ of $\psi$,
i.e.,

$F(\psi)(\xi)=\widehat{\psi}(\xi)=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty}\psi(x)e^{-i\xi
x}dx.$

Since for $\psi\in L^{1}(\mathbb{R})$, $\widehat{\psi}(\xi)$ is continuous
then condition (5) is only satisfied if $\widehat{\psi}(0)=0$, which is
equivalent to $\int_{-\infty}^{\infty}\psi(x)dx=0$. On the other hand,
Daubechies [1], p.24 points out that condition
$\int_{-\infty}^{\infty}\psi(x)dx=0$ together with a slightly stronger than
the integrability condition
$\int_{-\infty}^{\infty}|\psi(x)|(1+|x|)^{\alpha}dx<\infty$, for some
$\alpha>0$ are sufficient for (5). Usually, in practice much more is assumed
for the function $\psi$ hence, from a practical point of view, conditions
$\int_{-\infty}^{\infty}\psi(x)dx=0$ and (5) are equivalent. Suppose the
function $\psi$ is also square-integrable, $\psi\in L^{2}(\mathbb{R})$ with
the norm

$||\psi||=\left(\int_{-\infty}^{\infty}|\psi(x)|^{2}dx\right)^{1/2}.$

From a mother wavelet, one can generate a doubly-indexed family of wavelets
(called children wavelets), by dilating and translating,

$\psi^{a,b}(x)=\frac{1}{\sqrt{a}}\psi\Big{(}\frac{x-b}{a}\Big{)},$

where $a,b\in\mathbb{R},\;a>0$. The normalization has been chosen so that
$||\psi^{a,b}||=||\psi||$ for all $a,b$. In order to be able to compare
different wavelets, it is convenient to normalize the wavelets, i.e.,
$||\psi||=1$.

The continuous wavelet transform (CWT) of a function $f\in L^{2}(\mathbb{R})$
for this wavelet family is defined as

$(T^{wav}f)(a,b)=\langle
f,\psi^{a,b}\rangle=\int_{-\infty}^{\infty}f(x)\psi^{a,b}(x)dx.$ (6)

Rza̧dkowski and Figlia [15] introduced the logistics wavelets of any order and
then Rza̧dkowski [14] presented their standardized form. The formula for the
normalized second-order logistic mother wavelet $\psi_{2}(t)$ (see Fig. 3) is
as follows

$\psi_{2}(t)=\frac{\sqrt{30}}{1+e^{-t}}\Big{(}1-\frac{1}{1+e^{-t}}\Big{)}\Big{(}1-\frac{2}{1+e^{-t}}\Big{)}=\frac{\sqrt{30}(e^{-2t}-e^{-t})}{(1+e^{-t})^{3}}.$
(7)

Formula (7) is simply the second derivative of the basic logistic function
$\displaystyle x(t)=\frac{1}{1+e^{-t}}$ multiplied by $\sqrt{30}$.

Figure 3: Logistic mother wavelet $\psi_{2}$

As usually, we generate from the mother wavelet a doubly-indexed family of
wavelets from $\psi_{2}$ by dilating and translating

$\psi_{2}^{a,b}(t)=\frac{1}{\sqrt{a}}\psi_{2}\Big{(}\frac{t-b}{a}\Big{)},$

where $a,b\in\mathbb{R},\;a>0,\;n=2,3,\ldots$.

##  3 Applications

Let $(y_{n})$ be a time series. To calculate the CWT (Continuous Wavelet
Transform) coefficients for the central second differences

$\Delta^{2}y_{n}=y_{n+1}-2y_{n}+y_{n-1},$

we use MATLAB’s Wavelet Toolbox. Assume that the time series $(y_{n})$ locally
follows a logistic function $\displaystyle
y(t)=\frac{y_{sat}}{1+\exp(-\frac{t-b}{a})}$, i.e., $\displaystyle
y_{n}\approx y(n)=\frac{y_{sat}}{1+\exp(-\frac{n-b}{a})}$. By definition (7)
we have

$y^{\prime\prime}(t)=\frac{y_{sat}}{\sqrt{30}a^{3/2}}\psi_{2}^{a,b}(t).$

######  Lemma.

The continuous wavelet transform CWT (6) of the function
$y^{\prime\prime}(t)$, based on the logistic wavelets $\psi_{2}^{c,d}$

$(T^{wav}y^{\prime\prime})(c,d)=\langle
y^{\prime\prime},\psi_{2}^{c,d}\rangle=\int_{-\infty}^{\infty}y^{\prime\prime}(t)\psi_{2}^{c,d}(t)dt,$

takes the maximum (for $y_{sat}>0$) or minimum (for $y_{sat}<0$) value when
$c=a$ and $d=b$.

###### Proof.

Assume that $y_{sat}>0$. By the Cauchy-Schwartz inequality

$|(T^{wav}y^{\prime\prime})(c,d)|=|\langle
y^{\prime\prime},\psi_{2}^{c,d}\rangle|\leq||y^{\prime\prime}||\;||\psi_{2}^{c,d}||=\frac{y_{sat}}{\sqrt{30}a^{3/2}}||\psi_{2}^{a,b}||\;||\psi_{2}^{c,d}||=\frac{y_{sat}}{\sqrt{30}a^{3/2}}.$

However the maximum is reached for $c=a$, $d=b$, because:

$(T^{wav}y^{\prime\prime})(a,b)=\langle
y^{\prime\prime},\psi_{2}^{a,b}\rangle=\frac{y_{sat}}{\sqrt{30}a^{3/2}}\langle\psi_{2}^{a,b},\psi_{2}^{a,b}\rangle=\frac{y_{sat}}{\sqrt{30}a^{3/2}}.$

Similarly we consider the case $y_{sat}<0$.

∎

In view of Lemma, for the maximal value of Index we get successively

$\displaystyle\max(\textrm{Index})$
$\displaystyle=\sum\limits_{n}\Delta^{2}y_{n}\psi_{2}^{a,b}(n)\approx\sum\limits_{n}\Delta^{2}y(n)\psi_{2}^{a,b}(n)\approx\int_{-\infty}^{\infty}y^{\prime\prime}(t)\psi_{2}^{a,b}(t)dt$
$\displaystyle=\int_{-\infty}^{\infty}\frac{y_{sat}}{\sqrt{30}a^{3/2}}\psi_{2}^{a,b}(t)\psi_{2}^{a,b}(t)dt=\frac{y_{sat}}{\sqrt{30}a^{3/2}}\int_{-\infty}^{\infty}(\psi_{2}^{a,b}(t))^{2}dt=\frac{y_{sat}}{\sqrt{30}a^{3/2}}.$
(8)

Two parameters of a logistic wave, $b$ \- shift (translation) and $a$ \-
dilation, can be read from the CWT scalogram by finding a point where the sum
(8) (denoted in the scalogram by Index) is locally maximal (or locally
minimal). It remains to determine the third parameter of the wave, i.e., its
saturation level $y_{sat}$. Using (8) we can estimate the saturation level
$y_{sat}$ as follows

$y_{sat}\approx\sqrt{30}a^{3/2}\sum\limits_{n}\Delta^{2}y_{n}\psi_{2}^{a,b}(n)=\sqrt{30}a^{3/2}\max(\textrm{Index}).$
(9)

Similarly we get

$y_{sat}\approx\sqrt{30}a^{3/2}\sum\limits_{n}\Delta^{2}y_{n}\psi_{2}^{a,b}(n)=\sqrt{30}a^{3/2}\min(\textrm{Index}).$
(10)

Consider the Gompertz function

$x(t)=100,000\exp(-\exp(-\frac{t-50}{10})),$

with parameters $x_{sat}=100,000;\;t_{0}=50;\;s=0.1$ and a time series $y_{n}$
following the same exact Gompertz growth trend (Fig. 4(a))

$y_{n}=x(n)=100,000\exp(-\exp(-\frac{n-50}{10}))\quad\quad
n=0,1,2,\ldots,201.$ (11)

Then we calculate the central first differences (Fig. 4(b))

$\Delta^{1}y_{n}=(y_{n+1}-y_{n-1})/2,\;\;n=1,2,\ldots 200,$

and the central second differences (Fig. 4(c))

$\Delta^{2}y_{n}=y_{n+1}-2y_{n}+y_{n-1},\;\;n=1,2,\ldots,200.$

For the central second differences, we apply the MATLAB’s cwt function, which
uses second-order logistic wavelets (Fig. 4(d)). The marked point indicates
the maximum of the Index. Using (9) we can estimate the parameters of the
first logistic wave, which best fits the Gompertz trend $(y_{n})$:

$a=6.115,\;b=50,\;y_{sat}=\sqrt{30}\cdot 6.115^{3/2}\cdot 1062=87959.$

Thus we obtain logistic function

$h(t)=\frac{87959}{1+e^{-\frac{t-50}{6.115}}}.$

|
---|---

(a) Time series $y_{n}$ with the Gompertz growth (11)

(b) First differences $\Delta^{1}y_{n}$

|
---|---

(c) Second differences $\Delta^{2}y_{n}$

(d) CWT scalogram for the second differences (c)

Figure 4: Graphs and the CWT scalogram for the time series (11), with exact
Gompertz growth

Then we repeat this procedure for the time series $(y_{n}-h(n))$, Fig. 5(a).
We calculate its first differences (Fig. 5(b)) and the second differences
(Fig. 5(c)), for which we again apply the MATLAB’s cwt function. (Fig. 5(d))
shows two points (the minimum and the maximum of Index) giving the next two
logistic waves with the following parameters:

$a=4.028,\;b=34,\;y_{sat}=\sqrt{30}\cdot 4.028^{3/2}\cdot(-246.4)=-10910,$

and

$a=6.74,\;b=66,\;y_{sat}=\sqrt{30}\cdot 6.74^{3/2}\cdot 226.4=21698.$

For calculation of the saturations levels we used (10) and (9) respectively.

|
---|---

(a) Time series ($y_{n}-h(n)$)

(b) First differences of (a)

|
---|---

(c) Second differences of (a)

(d) CWT scalogram for second differences (c)

Figure 5: Graphs and the CWT scalogram showing the other two logistic waves

The three logistic waves described above, when summed to a multilogistic
function, approximate the time series $y_{n}$ with the maximum absolute error
equal to 1808, with the root mean square error (RMSE) equal to 969 and
$R^{2}=0.9998$. The RMSE error is mainly influenced by deviations, related to
the mismatch of saturation levels, for values of $n$ greater than 100. The
value of these deviations is approximately $100000-87959+10910-21698=1253.$
After optimizing the parameters with the objective function - minimization of
the maximum absolute error - we get the multilogistic approximating function
in the following form:

$f(t)=\frac{88057}{1+e^{-\frac{t-50}{6.17}}}-\frac{10919}{1+e^{-\frac{t-33.55}{5.12}}}+\frac{22846}{1+e^{-\frac{t-67.17}{8.77}}}.$
(12)

The multilogistic function (12), $f(t)$ approximates the Gompertz growth trend
$y_{n}$, (11) with errors (see Fig 6):

$\max\limits_{0\leq n\leq
201}|y_{n}-f(n)|=525,\quad\textrm{RMSE}=\sqrt{\frac{1}{202}\sum\limits_{n=0}^{201}(y_{n}-f(n))^{2}}=160,\quad
R^{2}=0.999985.$

Figure 6: Approximating function $f(t)$, (12)

##  4 Conclusions

In the present paper we showed that the Gompertz curve can be approximated,
with high accuracy, by a multilogistic function consisting of three logistic
functions. We have shown this using an example of one specific time series
having the exact Gompertz growth. Obviously, this applies to any Gompertz
curve by linearly replacing the variables.

The question arises whether it is possible to interpret somehow the subsequent
components of the multilogistic function (12). It seems that the first
component in (12) could be interpreted as a diffusion model in a favorable
business environment. The second function, with a negative saturation level,
could play the role of an inhibitory influence of business competitors, trying
to prevent a new, innovative product from entering the market. This last
function can be interpreted as a kind of strengthening function (boost
function) that appears after the role of the competitors is effectively
limited. All these interactions cause the appearance of the Gompertz curve.

Conflict of Interests

The author declares no conflict of interests related to the submitted
manuscript.

Funding statement

The research was partially funded by the ’IDUB against COVID-19’ project
granted by the Warsaw University of Technology (Warsaw, Poland) under the
program Excellence Initiative: Research University (IDUB), grant no
1820/54/201/2020.

## References

  * [1] I Daubechies.  Ten Lectures on Wavelets.  Society for Industrial and Applied Mathematics, 1992.
  * [2] A B Dhahbi, Y Chargui, S Boulaaras, S Rahali, and A Mhamdi.  The proliferation of COVID-19 in Saudi Arabia according to Gompertz model.  Fractals, 10, 2022.
  * [3] E Estrada and P Bartesaghi.  From networked SIS model to the Gompertz function.  Appl. Math. Comput., 419, 2022.
  * [4] H Etzkowitz and L Leydesdorff.  The Triple Helix – University-Industry-Government relations: A laboratory for knowledge based economic development.  EASST Review, 14(1):14–19, 1995.
  * [5] B Gompertz.  On the nature of the function expressive of the law of human mortality, and on a new method of determining the value of life contingencies.  Phil. Trans. Roy. Soc., 1:513–585, 1825.
  * [6] I Ivanova.  Evolutionary dynamics of investors expectations and market price movement.  ArXiv, preprint, arXiv:1912.11216:1–57, 2022.
  * [7] I Ivanova.  Information exchange, meaning and redundancy generation in anticipatory systems: selforganization of expectations – the case of COVID-19.  International Journal of General Systems, 51(7):675–690, 2022.
  * [8] R Kundu, H Basak, P K Singh, A Ahmadian, M Ferrara, and R Sarkar.  Fuzzy rank-based fusion of CNN models using Gompertz function for screening COVID-19 CT-scans.  Sci. Rep., 11:14133, 2021.
  * [9] L Leydesdorff.  The Evolutionary Dynamics of Discursive Knowledge: Communication-Theoretical Perspectives on an Empirical Philosophy of Science.  Springer Cham, 2021.  https://doi.org/10.1007/978-3-030-59951-5.
  * [10] V Mahjan, E Muller, and F M Bass.  New product diffusion models in marketing: A review and directions for research.  Journal of Marketing, 54:1–26, 1990.
  * [11] Y Meyer.  Wavelets, Vibrations and Scalings.  American Mathematical Society, 1997.
  * [12] Y Meyer and D Ryan.  Wavelets: Algorithms and Applications.  Society for Industrial and Applied Mathematics, 1996.
  * [13] A Ohnishi, Y Namekawa, and T Fukui.  Universality in COVID-19 spread in view of the Gompertz function.  Progress of Theoretical and Experimental Physics, 12:123J01, 2020\.
  * [14] G Rza̧dkowski.  Normalized logistic wavelets: Applications to COVID-19 data in Italy.  ArXiv, preprint, arXiv:2305.05620 [math.GM]:1–10, 2023.
  * [15] G Rza̧dkowski and G Figlia.  Logistic wavelets and their application to model the spread of COVID-19 pandemic.  Appl. Sci., 11(17):8147, 2021.
  * [16] K M C Tjørve and E Tjørve.  The use of Gompertz models in growth analyses, and new Gompertz-model approach: An addition to the Unified-Richards family.  PLoS ONE, 12(6):e0178691, 2017.
  * [17] P Waliszewski and J Konarski.  A mystery of the Gompertz function.  In Gabriele A. Losa, editor, Fractals in Biology and Medicine, pages 277–286. Birkhäuser Verlag, 2005.
  * [18] C P Winsor.  The Gompertz curve as a growth curve.  Proceedings of the National Academy of Sciences, 1:1–8, 1932.


# PHINC: A Parallel Hinglish Social Media Code-Mixed Corpus for Machine
Translation

Vivek Srivastava, Mayank Singh
Indian Institute of Technology Gandhinagar
Gujarat India 382355
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Code-mixing is the phenomenon of using more than one language in a sentence.
It is a very frequently observed pattern of communication on social media
platforms. Flexibility to use multiple languages in one text message might
help to communicate efficiently with the target audience. But, it adds to the
challenge of processing and understanding natural language to a much larger
extent. This paper presents a parallel corpus of the 13,738 code-mixed
English-Hindi sentences and their corresponding translation in English. The
translations of sentences are done manually by the annotators. We are
releasing the parallel corpus to facilitate future research opportunities in
code-mixed machine translation. The annotated corpus is available at
https://doi.org/10.5281/zenodo.3605597.

## Introduction

Code-mixing is the phenomenon of switching between two or more languages by
the speaker in a single sentence of a text or speech. Code mixing is one of
the most frequent styles of communication in multilingual communities, such as
India. High spelling variations in the Romanized Hindi words (e.g.,  mujhe,
mjhe, mujhee, etc., are some variants for the Hindi counterpart of the English
word me.) presents the challenge to effective machine translation task. We
frquently observe code-mixing on social media platforms such as Twitter,
Facebook, etc. in contrast to the formal literature sources such as books,
poems, and newspapers. We, therefore, use several social media platforms such
as Twitter, Facebook, etc. as the source of data for our purpose.

With the ever-increasing volume of user engagement on social media platforms,
there is an upraise in the interest to study and build systems that support
code-mixing of multiple resource-constraint Indian languages. Barman et al.
(2014) discuss the language identification task for the code-mixed data
involving Bengali-Hindi-English. Das and Gambäck (2014) present various
techniques to identify languages at the token-level for the Bengali-English
and Hindi-English code-mixed corpus. Singh et al. (2018) discuss various
techniques to identify the named-entities in the code-mixed Hindi-English
corpora consisting of 3,638 tweets. Vyas et al. (2014) discuss various
experiment to identify POS tags of the 1,062 code-mixed Hindi-English Facebook
posts. They collected data from three popular celebrity Facebook public pages
of Mr. Amitabh Bachchan, Mr. Shahrukh Khan, and Mr. Narendra Modi. Besides,
they leverage the BBC Hindi news articles. Sinha and Thakur (2005) present a
rule-based machine translation system to translate the code-mixed Hindi-
English sentence to monolingual Hindi and English forms. Dhar, Kumar, and
Shrivastava (2018) propose a machine translation augmentation pipeline to use
on top of the standard machine translation systems. They also create a
parallel corpus of 6,096 English-Hindi code-mixed sentences and their
corresponding translation in English.

In this paper, we propose a large-scale parallel corpus for code-mixed
English-Hindi social media text messages. In contrast to similar works (Dhar,
Kumar, and Shrivastava (2018)), the proposed dataset is significantly more
extensive and comprises of multiple social media platforms. Also, the dataset
spans diverse topics such as sports, entertainment, news, etc. We also
showcase the limitations of state-of-the-art machine translation systems on
code-mixed datasets and present a translation architecture that outperforms
state-of-the-art systems.

## Dataset

We use multiple code-mixing datasets (Singh et al. (2018), Swami et al.
(2018), Prabhu and Verma (2016), Barman et al. (2014), and Vrishank Shete and
Mittal (2016)). One major advantage of using these datasets is the
availability of high-quality code-mixed sentences without manual filtering.
Also, it offers diversity in terms of the source of the data collection as
most of the major social networking platforms (e.g., Twitter, Facebook, etc.)
are present. It also reduces the bias towards the topics in discussion.

### Description, Collection, and Pre-processing

We collect a total of 49,602 sentences from multiple sources. We then shuffle,
pre-process and share these sentences with the annotators to provide the
corresponding English translation. Each sentence in the corpus is written in
the Roman script. Pre-processing of the dataset involves the following steps:

  * •

We remove sentences with less than five or more than 40 tokens. We introduce
the upper limit on the sentence length to speed up the annotation process.

  * •

We remove sentences having a percentage of out of vocabulary (OOV) words less
than 50% or more than 90%. Lower limit (i.e., 50%) helps to filter out the
sentences with the majority of English words whereas the upper limit (i.e.,
90%) filter out the sentences with high Hindi words. We consider alphanumeric
tokens as part of the vocabulary. We are using the English dictionary of the
Natural Language Toolkit (NLTK) to identify the out of vocabulary words.

Post pre-processing, we obtain a total of 25,346 code-mixed sentences.

### Annotation

The objective of the annotation is to provide the English translation of the
code-mixed English-Hindi sentence. We employ 54 student annotators studying in
several disciplines (Computer Science and Engineering, Electrical Engineering,
etc.) at Indian Institute of Technology Gandhinagar in the annotation task.
Each annotator has expert level proficiency in writing, speaking, and
understanding English and Hindi languages. We assign randomly selected 400
samples to each annotator, and the annotator has to provide the corresponding
translation of each sentence in English. Each sentence in the final dataset is
annotated by a single annotator. We provide a set of guidelines for each
annotator for the annotation task. The annotation guidelines are listed below:

  * •

Special characters and emoticons: Use the best understanding to include or
skip these symbols and characters in the translated English sentences.

  * •

Links, mentions(e.g., @some_user) and hashtags: Keep the same links, mentions,
and hashtags in the translated sentence.

  * •

Incorrect spellings (u, hm, pls, coz, etc.): Translated sentence should have
the correct spelling for each word.

  * •

Lower case: Write the translated sentence in lowercase.

  * •

Proper English sentence: If the input sentence is already in English and also
grammatically correct with no spelling mistakes, then its translation will
only be “&” (without quotes). E.g., “I can translate the sentence quickly”, do
not require any modification.

  * •

Ambiguous sentence: Do not translate an ambiguous sentence. If the sentence is
unclear to translate in English, mark it as “#” (without quotes).

  * •

Abusive words: Do not translate sentences containing abusive/cuss words. Mark
it as “#” (without quotes).

We refrain from translating sentences containing abusive words. Also, the
annotators do not provide any translation for the ambiguous sentences. Post
annotation, we obtain 21,597 sentences. It also includes sentences that are
refrained from the translation (i.e., proper English sentence, ambiguous
sentence, and sentences containing abusive words). Finally, we obtain 13,738
code-mixed sentences111https://doi.org/10.5281/zenodo.3605597 with the
corresponding English translation.

Figure 1 shows three example of the code-mixed sentence and their
corresponding translation in the corpus. First two examples are the high-
quality translation by the annotators and do not require any changes, whereas
the translation in the last example is of poor quality as they do not satisfy
the semantic requirements and require modification for the correct
translation. We are not making any changes to the poor quality translation of
the code-mixed sentences.

Figure 2 shows three examples of the sentences that come under the refrain
category of sentences for translation. Example I is a sentence that is already
in English and requires no translation. The sentence in example II contains
the abusive word, whereas the sentence in example III is ambiguous to
translate.

Example I Code-mixed Sentence: is seat me girne ka koi chance nhi hai English
Translation: there is no chance of falling down from this seat Require changes
in the English translation?: No Example II Code-mixed Sentence: Thnks buds!
Kabhi kabhi aajate hai achhe photos English Translation: Thank you buddy,
sometime good photos are captured. Require changes in the English
translation?: No Example III Code-mixed Sentence: Australia ke saath abhi
jeete nahi hai, magar NZ ke saath final kaise jeetenge iss soch mein bhartiya
yuvak on twitter. English Translation: Indian youth on twitter thinking that -
We have not won against Australia yet, but how would we win final with NZ?
Require changes in the English translation?: Yes Figure 1: Example translation
of the code-mixed sentences in the corpus. The annotators provide translations
to the code-mixed sentences. A change in the translation is required if the
translation does not meet the semantic requirements. Example I Code-mixed
Sentence: RT: Today is the birth anniversary of Maharana Pratap, whose bravery
& indomitable spirit doesn’t fail to inspire even today. Label: & Reason for
no translation: Sentence already in English Example II Code-mixed Sentence:
sach bolu ? Aap Cuss hai Label: # Reason for no translation: Presence of
abusive/cuss word in sentence. Example III Code-mixed Sentence: yuhi kat
jaayega safar sath tweetne se , ki manzil aayegi nazar sath tweetne se . Hum
raahi Twitter ke Label: # Reason for no translation: Unclear/ambiguous
sentence. Figure 2: Example of the code-mixed sentences with no translation by
the annotators. We replace the cuss word in Example II with the word “Cuss”.

### Exploratory Analysis

  * •

Out of vocabulary (OOV) words: Figure 3 shows the distribution of the OOV
words in the code-mixed sentences. We are using the NLTK English dictionary
for this study. Apart from the Romanized Hindi words, hashtags and mentions
also fall into the category of OOV words. We consider alphanumeric tokens as
part of the vocabulary. We have sentences with the percentage of OOV words
greater than 50% and less than 90%. This distribution is also indicative of
the non-standard writing style of the users on social media sites such as
Twitter, Facebook, etc.

Figure 3: Distribution of out of vocabulary words in the code-mixed messages.

  * •

Code Mixing Index (CMI): CMI is the metric introduced by Das and Gambäck
(2014). It is the measure of the degree of code-mixing in a corpus. CMI is
calculated as follows

$CMI=\begin{cases}100*[1-\frac{max(w_{i})}{n-u}]&n>u\\\ 0&n=u\end{cases}$

Here, $w_{i}$ is the number of words of the language $i$, max{$w_{i}$}
represents the number of words of the most prominent language, $n$ is the
total number of tokens, $u$ represents the number of language-independent
tokens (such as named entities, abbreviations, mentions, hastags, etc.). CMI
value range from 0 to 100. A value close to 0 suggest multilingualism in the
corpus, whereas high CMI values indicate a high degree of code-mixing. To
calculate the value of CMI, we randomly sample 50 code-mixed sentences from
the corpus and annotate them at the token level with three language tags
English, Hindi, and Other. The CMI calculated for this set of sentences is 77.

  * •

Topics: Figure 4 shows the word cloud of the code-mixed and English translated
sentences. It is evident from the word cloud that words from multiple domains
such as politics, entertainment, sports, etc., are very frequently used. We
remove the hyperlinks, mentions, and hashtags from the translated sentences to
create the word cloud.

|
---|---
(a) | (b)
Figure 4: Word cloud of the (a) code-mixed and (b) translated sentences.

  * •

Message length: Figure 5 shows the distribution of the message length for the
code-mixed and the translated sentences. Distribution of message length for
code-mixed and the translated sentences follows a similar trend.

  * •

Quality of Translations (QT): To evaluate the quality of the translations by
the annotators, we have randomly sampled 50 sentences from the corpus. We
provide two labels to each of the translation correct translation and require
change. The quality of translation is calculated as follows

${QT}=\frac{Count\>of\>correct\>translations}{Sample\>size}$

42 samples out of 50 do not require any changes. Thus, the quality of
translation is 0.84.

## Evaluation of Machine Translation Systems

We experiment with various machine translation systems and evaluate their
performance on our proposed corpus. The majority of these systems perform well
for the monolingual translation tasks. However, these systems demonstrate
severe limitations in translating code-mixed text prevalent on social media
platforms. In the following experiments, we randomly sample 50 code-mixed
sentences from the corpus. We use human translated sentences as reference.

|
---|---
(a) | (b)
Figure 5:  Distribution of the message length for the (a) code-mixed and (b)
translated sentences.

  * •

Bing Translate (BT): We evaluate the performance of BT on the code-mixed
corpus. We set the language of the code-mixed input sentence as Hindi on the
BT platform. The BLEU-1 score for this set of sentences is 0.139. This value
is close to 0 and suggests the poor performance of the system on the code-
mixed data.

  * •

Google Translate (GT): Next, we evaluate the performance of the GT on the
code-mixed corpus. We set GT to auto-detect the language of the code-mixed
input sentence. The BLEU-1 score for this set of sentences is 0.14. Even
though we observe a slightly higher value of BLEU-1 for GT, the translations
can still be improved.

  * •

Proposed Pipeline + Google Translate (PPGT): We propose a pipeline to use on
top of GT. This pipeline helps to improve the quality of input being fed to
GT. Various steps of PPGT translation are:

    * –

We provide a label for each token of the code-mixed sentence based on the
language ( English, Hindi, and Other).

    * –

We create chunks of Type-I using Hindi tokens with at most two English/Other
token allowed to be part of any chunk. A chunk of Type-I starts with a Hindi
token.

    * –

We create chunks of Type-II using the tokens that are labeled as
English/Others and not part of any Type-I chunk.

    * –

We only translate the Type-I chunks using GT. We keep the chunks of Type-II as
it is.

Figure 6 shows the example translation of code-mixed sentences using all the
three techniques BT, GT, and PPGT. The Type-I and Type-II chunks in each
example are used for translating the sentence using the PPGT technique. In
PPGT, we maintain the original order of the chunks as that of the code-mixed
sentence while translating.
For instance, the order of the chunks for Example II in Figure 6 is [[par if
its], [possible and any other guest needs a room ,], [mera room de de kisi ko
bhi]].
The BLEU-1 score of this system is 0.153, which is an improvement over both
the other systems.

As most of the state-of-the-art machine translation systems do not perform
well on the code-mixed data, we can build pipelines on top of these systems
that can preprocess the input to these systems. Dhar, Kumar, and Shrivastava
(2018) also present one such augmentation pipeline to improve the performance
of these systems. These pipelines can address the challenges to code-mixed
machine translation, as outlined in the next section.

## Challenges to Code-Mixed Machine Translation

The nature of code-mixed text presents several challenges to the various
natural language processing techniques. Google translate, one of the most
sophisticated systems for translating texts in one language to another, also
fails at times to translate code-mixed texts efficiently. Figure 7 shows the
three different instances of translating text using Google translate. Example
I and II are relatively better translations as compared to example III. We use
the auto-detection of language by Google translate in all the translations.
Table 1 shows the comparison of the BLEU score of all the three examples in
Figure 7. The value of the BLEU score can vary between 0 and 1. Some prominent
reasons for the failure of the standard machine translations systems on the
code-mixed data are:

  * •

Ambiguity in language identification: Hindi words written in the Roman script
present some significant challenges to identify the language of the text at
the token level.
is, me, to are some examples of the words that are ambiguous to classify as
English and Hindi without proper knowledge of context.
Hashtags are often used on the social medium platforms, and code-mixed
hashtags make it challenging to identify the boundaries of code-switching.

  * •

Spelling variations: Romanized Hindi also presents a challenge with no
standard spelling of the words. Various spellings for the same word is used
based on the user’s pronunciation of the word, emotions, etc.
E.g., jaldi, jldi, jldiii,.. are some variations for the word hurry in
English.
At times, people use repeated instances of some particular character to
emphasize emotion, such as in jaldiii.

  * •

Informal style of writing: Writing style of the users on social media sites is
informal. At times, we do not follow the standard rules of sentence structure
on these platforms. This presents a challenge to translate the sentence in
monolingual style where the formal sentence structure is required for the
semantic purpose.
E.g., Sad kabhi dekha h usko.. me never.
The corresponding English translation is Have you ever seen him sad? I have
never seen him sad.

  * •

Misplaced/ skipped punctuation: In the informal writing style on social media
platforms, punctuations are usually skipped, misplaced, or repeatedly used to
express an opinion, and that makes it difficult for the machine translation
system to translate such sentences.
E.g., Aap kb se cricket khelne lage..never saw u bfr.
The sentence in the example misses a question mark(?) apart from other
necessary modifications to make the structure of the sentence correct.

  * •

Missing context: Lack of knowledge of the context makes the task of machine
translation difficult and challenging. Hidden sarcasm might get unnoticed
while translating the sentence with missing context.
E.g., Note kr lijiye.. Bandi chal rahi h ;) is a code-mixed sentence and
demonetisation (notebandi) is the hidden context.

## FAIR Principles

We are making the dataset FAIR by employing the various guiding
principles222https://www.force11.org/fairprinciples.

  * •

Findable: A globally unique and eternally persistent identifier for our data
is available for access333https://doi.org/10.5281/zenodo.3605597. We describe
our data with rich metadata, and the previous section explains all the
exploratory analysis for the metadata. The data is indexed in
OpenAIRE444http://bit.ly/2FH4hDU.

  * •

Accessible: The metadata and the data are retrievable and accessible by humans
and machines by the identifier using a standardized communications protocol
such as HTTP.

  * •

Interoperable: The metadata and the data use the English language for
knowledge representation, which is formal, accessible, shared, and broadly
applicable language.

  * •

Re-usable: The data is released with a clear and accessible data usage
license555https://creativecommons.org/licenses/by/4.0/legalcode. Metadata and
the data are sufficiently well-described and rich.

## Conclusion

In this paper, we present a parallel corpus for the English-Hindi code-mixed
machine translation task. We discuss various challenges that state of the art
machine translation system build for monolingual corpus face while dealing
with code-mixed corpora. We also demonstrate the performance of the various
systems on our parallel corpus. It is evident that we have a large scope to
build systems that can overcome the challenges associated with the code-mixed
text from various social media platforms.

Example I Code-mixed Sentence: @Prankoholic tumko matlab kya time hai din ka,
kuch samaj nahi aata na Type-I chunks: [tumko matlab kya time hai din ka, kuch
samaj nahi aata na] Type-II chunks: [@Prankoholic] English Translation using
BT: @prankoholic what time do you mean of the day, some society does not come.
English Translation using GT: @Prankoholic you mean what is the time of day,
don’t understand anything English Translation using PPGT: @Prankoholic Do you
mean what is the time of day, no sense Example II Code-mixed Sentence: par if
its possible and any other guest needs a room , mera room de de kisi ko bhi
Type-I chunks: [par], [mera room de de kisi ko bhi] Type-II chunks: [if its
possible and any other guest needs a room ,] English Translation using BT: On
if its possible egg any other guest needs coming room , my room day to anyone
English Translation using GT: par if its possible and any other guest needs a
room , mera room de de kisi ko bhi English Translation using PPGT: par if its
possible and any other guest needs a room , Give my room to anyone Example III
Code-mixed Sentence: @UPGovt @yadavakhilesh Great progress Sir. Iss baar bhi
aap kuch nahi karoge. Type-I chunks: [Iss baar bhi aap kuch nahi karoge.]
Type-II chunks: [@UPGovt @yadavakhilesh Great progress Sir.] English
Translation using BT: @upgovt @yadavakhilesh great progress sir. This time too
you karoge nothing. English Translation using GT: @UPGovt @yadavakhilesh Great
progress Sir. Iss baar bhi aap kuch nahi karoge. English Translation using
PPGT: @UPGovt @yadavakhilesh Great progress Sir. You will not do anything this
time. Figure 6: Example translation of code-mixed sentences using BT, GT, and
PPGT. We use Type-I and Type-II chunks while translating the code-mixed
sentence using PPGT. Example I Sentence: Bahot time baad suna ye! Achha laga
Google Translation: Heard this after a lot of time! It was nice Human
Translation: Heard this after a long time! It was nice Performance of Google
translate: Good Example II Sentence: Is shaher ko ye Hua kya hai.. Kahi rakh
hai to kahi dhua dhua.. Play interrupted due to bad weather Google
Translation: What has happened to this city .. If there is smoke somewhere,
then smoke somewhere .. Play interrupted payable then bad weather Human
Translation: What has happened to this city. there is ash and smoke
everywhere. play interrupted due to bad weather Performance of Google
translate: Average Example III Sentence: Bhai IIT wale hai pehle relationship
toh bane laundon ki, break up par nacha rahe ho. Google Translation: Brother-
in-law is the first relationship to be made of laundries, you are dancing on
the brake sub. Human Translation: Brother, you are an IITian. First get in to
a relation. Then you can worry about break up. Performance of Google
translate: Poor Figure 7: Comparison of translation of code-mixed sentences by
Google translate and human annotators.

| BLEU 1 | BLEU 2 | BLEU 3 | BLEU 4
---|---|---|---|---
Example I | 0.814 | 0.704 | 0.565 | 0.388
Example II | 0.560 | 0.362 | 0.191 | 0.152
Example III | 0.272 | 0.095 | 0.050 | 0

Table 1: BLEU score for various examples of Figure 7

## Future Work

We can use the parallel corpus to build efficient machine translation systems
for social media platforms. We can also explore various other code-mixed
languages, especially those that are low resource and endangered. We can also
extend the corpus presented here for various other code-mixing tasks such as
language identification, named-entity recognition, etc.

## References

  * Barman et al. (2014) Barman, U.; Das, A.; Wagner, J.; and Foster, J.  2014\.  Code mixing: A challenge for language identification in the language of social media.  In Proceedings of the First Workshop on Computational Approaches to Code Switching, 13–23.  Doha, Qatar: Association for Computational Linguistics.
  * Das and Gambäck (2014) Das, A., and Gambäck, B.  2014\.  Identifying languages at the word level in code-mixed indian social media text.  In Proceedings of the 11th International Conference on Natural Language Processing, 378–387.
  * Dhar, Kumar, and Shrivastava (2018) Dhar, M.; Kumar, V.; and Shrivastava, M.  2018\.  Enabling code-mixed translation: Parallel corpus creation and mt augmentation approach.  In Proceedings of the First Workshop on Linguistic Resources for Natural Language Processing, 131–140.
  * Prabhu and Verma (2016) Prabhu, Ameya, J. A. S. M., and Verma, V.  2016\.  Towards sub-word level compositions for sentiment analysis of hindi-english code mixed text.  arXiv preprint arXiv:1611.00472.
  * Singh et al. (2018) Singh, V.; Vijay, D.; Akhtar, S. S.; and Shrivastava, M.  2018\.  Named entity recognition for hindi-english code-mixed social media text.  In Proceedings of the Seventh Named Entities Workshop, 27–35.
  * Sinha and Thakur (2005) Sinha, R. M. K., and Thakur, A.  2005\.  Machine translation of bi-lingual hindi-english (hinglish) text.  10th Machine Translation summit (MT Summit X), Phuket, Thailand 149–156.
  * Swami et al. (2018) Swami, S.; Khandelwal, A.; Singh, V.; Akhtar, S. S.; and Shrivastava, M.  2018\.  A corpus of english-hindi code-mixed tweets for sarcasm detection.  arXiv preprint arXiv:1805.11869.
  * Vrishank Shete and Mittal (2016) Vrishank Shete, G. S. C., and Mittal, L.  2016\.  Sentiment analysis on hindi-english code mixed data using svm.  [Online; accessed 08-Jan-2020].
  * Vyas et al. (2014) Vyas, Y.; Gella, S.; Sharma, J.; Bali, K.; and Choudhury, M.  2014\.  Pos tagging of english-hindi code-mixed social media content.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 974–979.