# Type I-Type II Mixture Censoring Scheme for Lifetime Data Analysis

K. K. Anakhaa and V. M. Chackoa CONTACT K. K. Author. Email:
<EMAIL_ADDRESS>aDepartment of Statistics, St.Thomas College
(Autonomous), Thrissur, University of Calicut, Kerala, India

###### Abstract

The Type-I and Type-II censoring schemes are the most prominent and commonly
used censoring schemes in practice. In this work, a mixture of Type-I and
Type-II censoring schemes, named the Type I-Type II mixture censoring scheme,
has been introduced. Different censoring schemes have been discussed, along
with their benefits and drawbacks. For the proposed censoring scheme, we
analyze the data under the assumption that failure times of experimental units
follow the Weibull distribution. The computational formulas for the expected
number of failures and the expected failure time are provided. Maximum
likelihood estimation and Bayesian estimation are used to estimate the model
parameters. On the basis of thorough Monte Carlo simulations, the investigated
inferential approaches are evaluated. Finally, a numerical example is provided
to demonstrate the method of inference discussed here.

###### keywords:

Type-I and Type-II censoring; Maximum likelihood estimaton; Bayesian
estimation

##  1 Introduction

Effective censoring schemes are crucial in the field of statistical analysis,
as they play a vital role in accurately evaluating and deriving significant
insights from censored data. Censoring, a common phenomenon in many fields
such as survival analysis, reliability engineering, and clinical trials,
occurs when the complete information about the event of interest is not
available for some observations. Traditional censoring schemes (Type-I and
Type-II), while widely used, may not capture the full potential of the
available data or adequately handle complex scenarios.

In this article, we delve into a new censoring scheme that promises to advance
statistical inference techniques and unlock new avenues of knowledge. This
innovative approach challenges conventional methodologies, offering
researchers a fresh perspective on handling censored data. By introducing a
novel framework for censoring, we aim to bridge the gap between existing
methods and the increasing complexity of real-world datasets, enabling more
accurate and insightful analysis.

The conventional censoring schemes, Type-I and Type-II, have specific
features. Type-I censoring scheme involves running the experiment until a
fixed time point $T$ is reached, while Type-II censoring scheme allows the
experiment to continue until the occurrence of the $m$th failure. These
schemes lack complete event time information for censored observations. This
loss of information can limit the precision and accuracy of statistical
analysis, potentially leading to biased estimates or reduced statistical
power.

To address the drawback, Epstein[5] introduced, a mixture of Type-I and Type-
II censoring scheme, known as hybrid censoring scheme. Here the experiment is
terminated at the time $T_{1}^{*}=min\\{x_{m:n},T\\}$, where
$T\epsilon(0,\infty)$ and $1\leq m\leq n$ are prefixed and $x_{m:n}$
represents the $m^{th}$ failure time when $n$ units are on the experiment.
This scheme also referred to as Type-I hybrid censoring scheme. Childs et
al.[2] introduced a new censoring scheme known as, Type-II hybrid censoring
scheme, in which the stopping time is $T_{2}^{*}=max\\{x_{m:n},T\\}$. However,
in recent years, hybrid censoring scheme has grown increasingly prevalent in
reliability and life-testing investigations, these schemes lack the
flexibility to allow units to be removed from the experiment before it reaches
its termination.

Later, Cohen[3] introduced progressive censoring scheme. Progressive censoring
permits the removal of experimental objects at various stages to preserve the
total cost and time associated with the experiment. From the perspective of
experimental design, this approach could be useful in terms of identifying an
early appropriate censoring scheme. Nevertheless, while this notion is
frequently made in the literature, it’s not really met in real-world trials as
the investigator can alter the censoring numbers during the study regardless
of the circumstances. As a result, having a model that supports this type of
adaptation is critical. In exchange for the efficiency benefit, Burkschat[1]
points out that progressive censoring has a longer test time than the
conventional Type-II censoring strategy. In order to overcome the drawback of
progressive Type-II censoring scheme Kundu and Joarder introduced Type-II
progressively hybrid censoring scheme having fixed experimental time.

Thereafter Ng et al.[11] introduced a censoring scheme called adaptive Type-II
progressive censoring scheme. In which the effective sample size $m$ is fixed
in advance, and the progressive censoring scheme is provided. But the number
of items progressively removed from the experiment upon failure may change
during the experiment. If the experimental time exceeds a prefixed time $T$
but the number of observed failures does not reach $m$, the experiment can
terminate as soon as possible by adjusting the number of items progressively
removed from the experiment upon failure.

Most of the censoring schemes discussed above do not guarantee a minimum of
$m$ failures and a time constraint simultaneously. As a result, we’ve come up
with a new censoring scheme named, Type I-Type II (T1-T2) mixture censoring
scheme that ensures a minimum of $m$ and a maximum of $n$ failures with a
prefixed supplementary time $S$. In this work, an experiment is carried out
across $n$ units with the assumptions that the effective sample size $m$ and
supplementary time $S$ are fixed prior to the experiment. At the time of $m$th
failure the experiment is further continued for a prefixed supplementary time
of $S$. The test is terminated at a time $T^{*}=min\\{X_{n:n},X_{m:n}+S\\}$.
For $S=0$, the proposed method converge to the conventional T2 censoring
method.

The remaining sections of the paper are arranged as follows. In the second
section, a description of the introduced censoring scheme using Weibull
distribution is given. The third and fourth sections are the attempts to
derive expressions for the expected duration of the experiment and the Fisher
information matrix, respectively. The parameter estimation using maximum
likelihood method (MLE) and Bayes method are given in Sections 5 and 6,
respectively. A simulation study is performed in Section 7. In Section 8, real
data is used to demonstrate the findings, and finally the paper is concluded
in Section 9.

##  2 Model Description

Suppose n units are put on a life-testing experiment and let
$X_{1},X_{2},X_{3},…,X_{n}$ be their corresponding lifetimes. If $r\,(m\leq
r\leq n)$ denotes the number of failures occur before $T^{*}$, then the
available data will be of the form;

$\displaystyle\text{Case I
}:\\{X_{1:n},X_{2:n},…,X_{n:n}\\}\quad\text{if}\quad T^{*}=X_{n:n}$
$\displaystyle\text{Case
II}:\\{X_{1:n},X_{2:n},…,X_{r:n}\\}\quad\text{if}\quad T^{*}=X_{m:n}+S.$

The forms of likelihood function corresponding to both cases are respectively,

$L(x_{i:n};\theta)\propto\left\\{\begin{array}[]{lll}\prod_{i=1}^{n}f(x_{i:n};\theta)&if&T^{*}=X_{n:n}\\\
\prod_{i=1}^{r}f(x_{i:n};\theta)\left\\{1-F(X_{m:n}+S)\right\\}^{n-r}&if&T^{*}=X_{m:n}+S\end{array}\right.$

In the fields of life-testing and reliability, the weibull distribution is one
of the most often used lifetime model. Based on the above expression, the log-
likelihood function for the weibull distribution with density function

$f(x;\gamma,\delta)=\gamma\delta x^{\gamma-1}e^{-\delta
x^{\gamma}};\,x,\delta,\gamma>0,$ (1)

and cumulative distribution function

$F(x;\gamma,\delta)=1-e^{-\delta x^{\gamma}};\,x,\delta,\gamma>0,$ (2)

is obtained as,

$\log
L(\theta)\propto\Bigg{\\{}\begin{array}[]{lll}n\log\gamma+n\log\delta+(\gamma-1)\sum_{i=1}^{n}\log
x_{i}-\delta\sum_{i=1}^{n}x_{i}^{\gamma}&;&r=n\\\ \\\
r\log\gamma+r\log\delta+(\gamma-1)\sum_{i=1}^{r}\log
x_{i}-\delta\sum_{i=1}^{r}x_{i}^{\gamma}-\delta(n-r)U^{\gamma}&;&r=m,m+1,...,n-1,\end{array}$

where $U=X_{m:n}+S$.

##  3 Expected Duration of Experiment

Let $K$ be the number of failures observed and $T^{*}$ denotes the duration of
experiment under T1-T2 censoring scheme. Clearly $K$ and $T^{*}$ are random
variables.
The distribution of $K$ is given by,

$\displaystyle P(K=r)=$
$\displaystyle\binom{n}{r}F(T^{*};\theta)^{r}(1-F(T^{*};\theta))^{n-r};\quad
m\leq r\leq n.$

Therefore, the expected value of $K$ is,

$\displaystyle
E[K]=\sum_{r=m}^{n}r\binom{n}{r}F(T^{*};\theta)^{r}(1-F(T^{*};\theta))^{n-r}$

Now consider the expected value of the duration of experiment,

$\displaystyle E[T^{*}]=$ $\displaystyle E[\min\\{X_{n:n},X_{m:n}+S\\}]$
$\displaystyle=$ $\displaystyle E\left[X_{m:n}+S|X_{m:n}+S\leq
X_{n:n}\right]P\left[X_{m:n}+S\leq X_{n:n}\right]+$ $\displaystyle
E\left[X_{n:n}|X_{n:n}<X_{m:n}+S\right]P\left[X_{n:n}<X_{m:n}+S\right]$

One additional way of determining the expected duration of experiment is as
follows.

$\displaystyle E[T^{*}]=$ $\displaystyle E[min\\{X_{n:n},X_{m:n}+S\\}]$
$\displaystyle=$
$\displaystyle\int_{0}^{\infty}P(min\\{X_{n:n},X_{m:n}+S\\}\geq x)dx$
$\displaystyle=$ $\displaystyle\int_{0}^{\infty}P(X_{n:n}\geq
x)P(X_{m:n}+S\geq x)dx$ $\displaystyle=$
$\displaystyle\int_{0}^{\infty}(1-F_{n:n}(x;\theta))(1-F_{m:n}(x+S))dx$

######  Lemma 1.

$E[K]$ is invariant under scalar multiplication of $T$.

###### Proof.

Consider $T^{{}^{\prime}}=\alpha T$, where $\alpha$ is the multiplication
constant. Then $T^{*}$ changes to some $T^{**}$ and $P[T^{{}^{\prime}}\leq
T^{**}]=P[\alpha T\leq\alpha T^{*}]=P[T\leq T^{*}]=F(T^{*},\theta)$. ∎

######  Lemma 2.

Assume that $T^{+}$ represents the censoring time obtained by multiplying some
constant $\alpha$ to $T$. Then, $E[T^{+}]=\alpha E[T^{*}]$.

###### Proof.

$\displaystyle E[T^{+}]=$ $\displaystyle E[min\\{\alpha
X_{n:n},\alpha(X_{m:n}+S)\\}]$ $\displaystyle=$ $\displaystyle
E[\alpha(X_{m:n}+S)\;|\;\alpha(X_{m:n}+S)\leq\alpha
X_{n:n}]\;P[\alpha(X_{m:n}+S)\leq\alpha X_{n:n}]+$ $\displaystyle E[\alpha
X_{n:n}\;|\;\alpha X_{n:n}<\alpha(X_{m:n}+S)]\;P[\alpha
X_{n:n}<\alpha(X_{m:n}+S)]$ $\displaystyle=$ $\displaystyle\alpha
E[X_{m:n}+S\;|\;X_{m:n}+S\leq X_{n:n}]\;P[X_{m:n}+S\leq X_{n:n}]+$
$\displaystyle E[X_{n:n}\;|\;X_{n:n}<X_{m:n}+S]\;P[X_{n:n}<X_{m:n}+S]$
$\displaystyle=$ $\displaystyle\alpha E[T^{*}]$

∎

##  4 Fisher Information

Effron and Johnstone [4] shown that the Fisher information about its
parameters of a continuous random variable can be expressed in terms of hazard
function of the random variable. Park and Balakrishnan [12] derived Fisher
information in different hybrid censoring schemes. Based on this, we can
derive Fisher information in the T1T2 censoring scheme.

$I(\gamma,\delta)=\left[\begin{array}[]{cc}I_{11}&I_{12}\\\
I_{21}&I_{22}\end{array}\right]$

where

$\displaystyle I_{11}=$
$\displaystyle\int_{0}^{\infty}\biggl{\\{}\frac{\partial}{\partial\gamma}\log
h(t)\biggr{\\}}^{T}\biggl{\\{}\frac{\partial}{\partial\gamma}\log
h(t)\biggr{\\}}\sum_{i=1}^{r}f_{i:n}(t;\gamma,\delta)dt$ $\displaystyle=$
$\displaystyle\int_{0}^{\infty}\biggl{\\{}\frac{1}{\gamma}+\log
t\biggr{\\}}^{2}\sum_{i=1}^{r}f_{i:n}(t;\gamma,\delta)dt$ $\displaystyle
I_{12}=$
$\displaystyle\int_{0}^{\infty}\biggl{\\{}\frac{\partial}{\partial\gamma}\log
h(t)\biggr{\\}}^{T}\biggl{\\{}\frac{\partial}{\partial\delta}\log
h(t)\biggr{\\}}\sum_{i=1}^{r}f_{i:n}(t;\gamma,\delta)dt$ $\displaystyle=$
$\displaystyle\frac{1}{\delta}\int_{0}^{\infty}\biggl{\\{}\frac{1}{\gamma}+\log
t\biggr{\\}}\sum_{i=1}^{r}f_{i:n}(t;\gamma,\delta)dt$ $\displaystyle=$
$\displaystyle I_{21}$ $\displaystyle I_{22}=$
$\displaystyle\int_{0}^{\infty}\biggl{\\{}\frac{\partial}{\partial\delta}\log
h(t)\biggr{\\}}^{T}\biggl{\\{}\frac{\partial}{\partial\delta}\log
h(t)\biggr{\\}}\sum_{i=1}^{r}f_{i:n}(t;\gamma,\delta)dt$ $\displaystyle=$
$\displaystyle\int_{0}^{\infty}\left(\frac{1}{\delta}\right)^{2}\sum_{i=1}^{r}f_{i:n}(t;\gamma,\delta)dt$
$\displaystyle\text{and}\quad f_{i:n}(t;\gamma,\delta)=$ $\displaystyle
i\binom{n}{i}\gamma\delta t^{\gamma-1}e^{-\delta
t^{\gamma}(n-i+1)}\left(1-e^{-\delta t^{\gamma}}\right)^{i-1}.$

##  5 Maximum Likelihood Estimation

The derivatives with respect to $\gamma$ and $\delta$ are respectively given
by,

$\frac{\partial\log
L}{\partial\gamma}=\Bigg{\\{}\begin{array}[]{lll}\frac{n}{\gamma}+\sum_{i=1}^{n}\log
x_{i}-\delta\sum_{i=1}^{n}x_{i}^{\gamma}\log x_{i}&;&r=n\\\ \\\
\frac{r}{\gamma}+\sum_{i=1}^{r}\log
x_{i}-\delta\sum_{i=1}^{r}x_{i}^{\gamma}\log x_{i}-\delta(n-r)U^{\gamma}\log
U&;&r=m,m+1,...,n-1.\end{array}$

$\frac{\partial\log
L}{\partial\delta}=\Bigg{\\{}\begin{array}[]{lll}\frac{n}{\delta}-\sum_{i=1}^{n}x_{i}^{\gamma}&;&r=n\\\
\\\
\frac{r}{\delta}-\sum_{i=1}^{r}x_{i}^{\gamma}-(n-r)U^{\gamma}&;&r=m,m+1,...,n-1.\end{array}$

Now, combining Case I and Case II and equating to zero, the above equations
become,

$\displaystyle\frac{\partial\log L}{\partial\gamma}$
$\displaystyle=\frac{w}{\gamma}+\sum_{i=1}^{w}\log x_{i}-\delta P=0$ (3)
$\displaystyle\frac{\partial\log L}{\partial\delta}$
$\displaystyle=\frac{w}{\delta}-Q=0$ (4)

where $w=n$, $P=\sum_{i=1}^{n}x_{i}^{\gamma}\log x_{i}$ and
$Q=\sum_{i=1}^{n}x_{i}^{\gamma}$ for case I and $w=r$,
$P=\sum_{i=1}^{r}x_{i}^{\gamma}\log x_{i}+(n-r)U^{\gamma}\log U$ and
$Q=\sum_{i=1}^{r}x_{i}^{\gamma}+(n-r)U^{\gamma}$ for case II.

From (2), we get

$\hat{\delta}=\frac{w}{Q}=v(\gamma),\text{(say)}.$ (5)

So that

$\hat{\gamma}=\frac{w}{v(\gamma)P-\sum_{i=1}^{w}\log x_{i}}=u(\gamma)$ (6)

where $v(\gamma)$ and $u(\gamma)$ are represents the functions of $\gamma$.

The MLEs, $\hat{\gamma}$ and $\hat{\delta}$ can be obtained by an iterative
scheme proposed by Kundu [8].

##  6 Bayes Estimation

In this part, we investigate the Bayes estimates of the unknown parameters
$\gamma$ and $\delta$ while using squared error loss function under T1-T2
censoring scheme. Here we assume that the prior distributions of $\gamma$ and
$\delta$ are follows independent $Gamma(\alpha_{1},\beta_{1})$ and
$Gamma(\alpha_{2},\beta_{2})$, respectively, where
$\alpha_{1},\alpha_{2},\beta_{1},\&\beta_{2}$ are the hyper parameters. When
$\gamma$ and $\delta$ are both unknown, their conjugate prior won’t exist.
Therefore, gamma priors can be used in these circumstances because they are
quite flexible and also take into account noninformative priors. Thus the
independent prior distributions of $\gamma$ and $\delta$ are,

$\pi_{1}(\gamma)\propto\gamma^{\alpha_{1}-1}e^{-\beta_{1}\gamma};\,\gamma\geq
0,\,\alpha_{1},\beta_{1}>0,$
$\pi_{2}(\delta)\propto\delta^{\alpha_{2}-1}e^{-\beta_{2}\delta};\,\delta\geq
0,\alpha_{2},\beta_{2}>0.$

The joint prior distribution of the parameters $\gamma$ and $\delta$ is,

$\pi(\gamma,\delta)\propto\gamma^{\alpha_{1}-1}\delta^{\alpha_{2}-1}e^{-\beta_{1}\gamma-\beta_{2}\delta};\,\gamma,\delta\geq
0,\alpha_{1},\alpha_{2},\beta_{1},\beta_{2}>0$ (7)

Using Bayes Theorem, the joint posterior density function of $\gamma$ and
$\delta$ is given by

$\pi(\gamma,\delta|\underline{X})\propto
l(\gamma,\delta|\underline{X})\pi(\gamma,\delta)$
$\pi(\gamma,\delta|\underline{X})\propto\gamma^{\alpha_{1}+w-1}\delta^{\alpha_{2}+w-1}\prod_{i=1}^{w}x_{i}^{\gamma-1}e^{-\delta\left[\beta_{2}+\sum_{i=1}^{w}x_{i}^{\gamma}+(n-w)U^{\gamma}\right]-\beta_{1}\gamma}$
(8)

To determine the Bayes estimates, we take into account both symmetric and
asymmetric loss functions. For the symmetric loss function, the squared error
loss function $(L_{SE})$ is considered. The $L_{SE}$ can be defined as,

$L_{SE}\left(g(\theta),\hat{g}(\theta)\right)=\left(g(\theta)-\hat{g}(\theta)\right)^{2},$
(9)

where $\hat{g}(\theta)$ is the estimate of $g(\theta)$.
For the asymmetric loss function, the LINEX loss function $(L_{LL})$ is
considered. The $L_{LL}$ can be defined as,

$L_{LL}\big{(}g(\theta),\hat{g}(\theta)\big{)}=e^{d\left(g(\theta)-\hat{g}(\theta)\right)}-d\left(g(\theta)-\hat{g}(\theta)\right)-1,\quad
d\neq 0,$ (10)

where $d$ is the loss parameter. For the LINEX loss function, the Bayes
estimate $\hat{\theta}_{LL}$ can be calculated as,

$\hat{\theta}_{LL}=-\frac{1}{d}\log\left\\{E_{\theta}\left(e^{-d\theta}\right)\right\\},$
(11)

where the expectation exist and finite.

The conditional distribution of the unknown parameters,

$\pi(\gamma|\delta,\underline{X})\propto\gamma^{\alpha_{1}+w-1}\prod_{i=1}^{w}x_{i}^{\gamma-1}e^{-\delta\left[\sum_{i=1}^{w}x_{i}^{\gamma}+(n-w)U^{\gamma}\right]-\beta_{1}\gamma}$
(12)

$\pi(\delta|\gamma,\underline{X})\propto\text{Gamma}\left(\alpha_{2}+w,\beta_{2}+\sum_{i=1}^{w}x_{i}^{\gamma}+(n-w)U^{\gamma}\right)$
(13)

As the conditional distribution of $\gamma$ lacks an explicit form, we need to
utilize the Metropolis-Hastings (M-H) technique to calculate Bayes estimates
and to construct HPD credible intervals.

###  6.1 M-H Algorithm

The M-H algorithm proposed by Metropolis et al. [10] and Hasting [6] is used
to generate Bayes estimates of $\gamma$ and $\delta$. This method can be
extremely helpful for constructing posterior samples using random proposal
distributions when the posterior density is operationally infeasible. Based on
the constructed posterior samples, Bayesian inference can be drawn regarding
the unknown parameters $\gamma$ and $\delta$. The following steps can be used
to construct the posterior samples.

STEPS
Step 1: Choose an initial value $(\gamma^{0},\delta^{0})$, and let
$\sigma=\gamma^{i-1}$.
Step 2: Proposing a new point $\gamma^{i}$ from
$\pi_{1}(\gamma|\delta^{0},\underline{X})$ from the proposing distribution
$g(\gamma)\equiv N(\gamma^{0}1),\gamma>0$.
Step 3: Generate a random sample $u$ from Uniform(0,1) and a random sample
$\psi$ from the proposal distribution. If
$u\leq\frac{\pi(\psi)g(\sigma)}{\pi(\sigma)g(\psi)}$, then set $\gamma=\psi$.
Here, the acceptance and rejection probabilities of $\psi$ are
$min\left\\{1,\frac{\pi(\psi)g(\sigma)}{\pi(\sigma)g(\psi)}\right\\}$ and
$1-min\left\\{1,\frac{\pi(\psi)g(\sigma)}{\pi(\sigma)g(\psi)}\right\\}$,
respectively.
Step 4: Generate $\delta^{i}$’s from
Gamma$\left(\alpha_{2}+w,\beta_{2}+\sum_{i=1}^{w}x_{i}^{\gamma}+(n-w)U^{\gamma}\right)$,
as in step 3.
Step 5: Repeat steps 2 to 4, N times and obtained
$(\gamma_{1},\delta_{1}),(\gamma_{2},\delta_{2}),...,(\gamma_{N},\delta_{N})$.

To ensure convergence and eliminate the impact of initial value selection, the
first $M$ estimates are excluded. The remaining sample, consisting of
$\gamma^{i}$ and $\delta^{i}$ with $i$ ranging from $M+1$ to $N$, where $N$ is
sufficiently large, constitutes an estimated posterior sample that can be
utilized for Bayesian inference. The $100(1-\alpha)\%$ HPD credible interval
of the unknown parameter $\gamma$ can be constructed from the ordered samples
$(\gamma_{(M+1)},\gamma_{(M+2)},...,\gamma_{(N)})$by selecting the smallest
interval, and similarly for $\delta$.

##  7 Simulation Study

This section is dedicated to presenting experimental results, primarily
focused on examining the behavior of various methods under different sample
sizes and censoring schemes for various values of parameters of
$f(x;\gamma,\delta)$ under T1-T2 censoring scheme. The statistical software R
is used to perform the computations.

A MCMC approach is used to generate samples for different values of $(n,m,S)$
in order to assess the performance of MLE. Without loss of generality, we
choose two different values for the parametres, $(\gamma,\delta)$ = (1, 1) and
(1.5, 2). The censoring schemes $(n,m)$ = (100, 100), (100, 90), (100, 85),
(60, 60), (60, 55), (60, 50), (60, 45), (30, 30), (30, 25), (30, 20), (30,
15), (15, 15), (15, 12), (15, 10), and (15, 7) are chosen for the
supplymentary times $S=0.1$ and $S=0.2$ with 1000 replications. Bias and mean
squared error (MSE) are computed for the unknown parameters in order to
examine the effectiveness of MLE, and the results are given in the Table 1. As
can be seen from Table 1, the MLE of parameters performs pretty satisfactorily
with respect to bias and MSE. The MSE value is declining as sample size
increases, indicating that the MLEs of the unknown parameters are consistent.
Moreover, we can observe that the MSE value obtained for S equal to 0.2 is
smaller compared to the MSE value obtained for S equal to 0.1. As the
supplementary time increases, there is a greater chance of more failures
occurring, which, in turn, leads to more precise statistical results. The
average confidence length (CL) and coverage probability (CP) based on $95\%$
confidence interval are determined and the results are given in Table 4 and 5.
As the sample size grows, the value of CL gets shorter, and the CP remains at
its nominal level.

The Bayes estimates for squared error and the LINEX loss functions are derived
using the M-H algorithm. Under LINEX, the arbitrarily chosen values of the
loss parameter $d$ are -1 and 1. For the prior distribution, hyperparameters
are chosen in such a way that the corresponding prior means are somewhat close
to the actual parameter values. Hence, the selected values are
$(\alpha_{1}=1,\beta_{1}=1,\alpha_{2}=1,\beta_{2}=1)$ and
$(\alpha_{1}=2.25,\beta_{1}=1.5,\alpha_{2}=4,\beta_{2}=2)$ (see, Kundu [9]).
Tables 2-3 show the estimated bias and risk for various censoring schemes
under the aforementioned loss functions. When we compare the bias and risk of
each estimate, it is clear that the estimates obtained under the LINEX loss
function perform better. The values of bias and risk are decreasing with the
increase in the effective sample size. Table 6 displays the CL and CP based on
the $95\%$ HPD credible interval (HPDC). As the effective sample size rises,
the CL of HPDC becomes narrower. Meanwhile, the nominal level of the CP is
maintained. Comparing CL of HPDC with MLE, it is clear that CL of HPDC
performs better for small sample sizes while they both have values that are
nearly equivalent for higher sample sizes.

Table 1: Average values of the biases and MSEs for the maximum likelihood
estimates of ($\gamma$, $\delta$) = (1, 1) and (1.5, 2) under different
censoring schemes.

Censoring Scheme | $\gamma$=1, $\delta$=1 | $\gamma$=1.5, $\delta$=2
---|---|---
| Bias($\gamma$) | Bias($\delta$) | MSE($\gamma$) | MSE($\delta$) | Bias($\gamma$) | Bias($\delta$) | MSE($\gamma$) | MSE($\delta$)
n=100, m=100, S=0.1 | 0.0101 | 0.0077 | 0.0064 | 0.0115 | 0.0151 | 0.0291 | 0.0144 | 0.0466
n=100, m=90, S=0.1 | 0.0112 | 0.0096 | 0.0077 | 0.0122 | 0.0152 | 0.0329 | 0.0168 | 0.0550
n=100, m=85, S=0.1 | 0.0122 | 0.0109 | 0.0083 | 0.0126 | 0.0169 | 0.0370 | 0.0178 | 0.0582
n=100, m=80, S=0.1 | 0.0125 | 0.0118 | 0.0089 | 0.0131 | 0.0174 | 0.0401 | 0.0188 | 0.0640
n=60, m=60, S=0.1 | 0.0189 | 0.0080 | 0.0111 | 0.0183 | 0.0284 | 0.0415 | 0.0251 | 0.0751
n=60, m=55, S=0.1 | 0.0219 | 0.0109 | 0.0135 | 0.0192 | 0.0306 | 0.0489 | 0.0294 | 0.0865
n=60, m=50, S=0.1 | 0.0264 | 0.0153 | 0.0161 | 0.0205 | 0.0365 | 0.0641 | 0.0333 | 0.1009
n=60, m=45, S=0.1 | 0.0306 | 0.0210 | 0.0192 | 0.0237 | 0.0397 | 0.0760 | 0.0386 | 0.1240
n=30, m=30, S=0.1 | 0.0472 | 0.0284 | 0.0274 | 0.0454 | 0.0709 | 0.1234 | 0.0616 | 0.2104
n=30, m=25, S=0.1 | 0.0577 | 0.0436 | 0.0387 | 0.0576 | 0.0814 | 0.1659 | 0.0799 | 0.3046
n=30, m=20, S=0.1 | 0.0805 | 0.0811 | 0.0568 | 0.0988 | 0.1008 | 0.2485 | 0.1069 | 0.5430
n=30, m=15, S=0.1 | 0.1105 | 0.1468 | 0.0853 | 0.1876 | 0.1388 | 0.4397 | 0.1579 | 1.3586
n=15, m=15, S=0.1 | 0.0992 | 0.0558 | 0.0722 | 0.1133 | 0.1488 | 0.2633 | 0.1625 | 0.6763
n=15, m=12, S=0.1 | 0.1294 | 0.1062 | 0.1107 | 0.1910 | 0.1739 | 0.3879 | 0.2191 | 1.2546
n=15, m=10, S=0.1 | 0.1640 | 0.1719 | 0.1501 | 0.2932 | 0.2090 | 0.5749 | 0.2843 | 2.4203
n=15, m=7, S=0.1 | 0.2473 | 0.4116 | 0.2840 | 1.2957 | 0.2950 | 1.1548 | 0.4437 | 8.1042
n=100, m=100, S=0.2 | 0.0101 | 0.0077 | 0.0064 | 0.0115 | 0.0151 | 0.0291 | 0.0144 | 0.0466
n=100, m=90, S=0.2 | 0.0104 | 0.0090 | 0.0076 | 0.0120 | 0.0126 | 0.0281 | 0.0158 | 0.0524
n=100, m=85, S=0.2 | 0.0108 | 0.0097 | 0.0080 | 0.0124 | 0.0142 | 0.0313 | 0.0168 | 0.0548
n=100, m=80, S=0.2 | 0.0124 | 0.0114 | 0.0086 | 0.0128 | 0.0149 | 0.0333 | 0.0177 | 0.0585
n=60, m=60, S=0.2 | 0.0189 | 0.0080 | 0.0111 | 0.0183 | 0.0284 | 0.0415 | 0.0251 | 0.0751
n=60, m=55, S=0.2 | 0.0209 | 0.0101 | 0.0134 | 0.0191 | 0.0275 | 0.0426 | 0.0281 | 0.0817
n=60, m=50, S=0.2 | 0.0246 | 0.0139 | 0.0153 | 0.0202 | 0.0325 | 0.0537 | 0.0305 | 0.0904
n=60, m=45, S=0.2 | 0.0279 | 0.0178 | 0.0179 | 0.0222 | 0.0345 | 0.0620 | 0.0335 | 0.1050
n=30, m=30, S=0.2 | 0.0472 | 0.0284 | 0.0274 | 0.0454 | 0.0709 | 0.1234 | 0.0616 | 0.2104
n=30, m=25, S=0.2 | 0.0546 | 0.0399 | 0.0367 | 0.0542 | 0.0742 | 0.1435 | 0.0741 | 0.2639
n=30, m=20, S=0.2 | 0.0687 | 0.0619 | 0.0496 | 0.0748 | 0.0817 | 0.1811 | 0.0886 | 0.3714
n=30, m=15, S=0.2 | 0.0930 | 0.1071 | 0.0704 | 0.1271 | 0.1000 | 0.2584 | 0.1152 | 0.5804
n=15, m=15, S=0.2 | 0.0992 | 0.0558 | 0.0722 | 0.1133 | 0.1488 | 0.2633 | 0.1625 | 0.6763
n=15, m=12, S=0.2 | 0.1197 | 0.0896 | 0.1032 | 0.1608 | 0.1536 | 0.3106 | 0.1910 | 0.8809
n=15, m=10, S=0.2 | 0.1428 | 0.1323 | 0.1300 | 0.2247 | 0.1660 | 0.3807 | 0.2210 | 1.1855
n=15, m=7, S=0.2 | 0.1901 | 0.2315 | 0.1960 | 0.4099 | 0.2008 | 0.5661 | 0.2876 | 2.1479

Table 2: Average values of the biases and Risks for the Bayesian estimates of
parameters ($\gamma$, $\delta$) = (1, 1) under different censoring schemes.

$\gamma$=1, $\delta$=1 | SE | LL
---|---|---
Bias | Risk | Bias | Risk
v=-1 | v=1 | v=-1 | v=1
$\gamma$ | $\delta$ | $\gamma$ | $\delta$ | $\gamma$ | $\delta$ | $\gamma$ | $\delta$ | $\gamma$ | $\delta$ | $\gamma$ | $\delta$
n=100, m=100, S=0.1 | -0.1807 | 0.0245 | 0.0402 | 0.0107 | 0.0038 | 0.0051 | -0.0037 | -0.0049 | 0.0185 | 0.0054 | 0.0219 | 0.0053
n=100, m=90, S=0.1 | -0.2151 | 0.0367 | 0.0544 | 0.0157 | 0.0041 | 0.0075 | -0.0041 | -0.0069 | 0.0248 | 0.0078 | 0.0299 | 0.0080
n=100, m=85, S=0.1 | -0.1917 | 0.0409 | 0.0457 | 0.0145 | 0.0044 | 0.0066 | -0.0045 | -0.0062 | 0.0210 | 0.0073 | 0.0249 | 0.0072
n=100, m=80, S=0.1 | -0.1923 | 0.0264 | 0.0458 | 0.0128 | 0.0044 | 0.0062 | -0.0045 | -0.0059 | 0.0210 | 0.0065 | 0.0251 | 0.0064
n=60, m=60, S=0.1 | -0.1385 | -0.0012 | 0.0270 | 0.0154 | 0.0039 | 0.0077 | -0.0040 | -0.0076 | 0.0126 | 0.0076 | 0.0145 | 0.0078
n=60, m=55, S=0.1 | -0.1854 | -0.0077 | 0.0498 | 0.0223 | 0.0077 | 0.0112 | -0.0077 | -0.0111 | 0.0226 | 0.0111 | 0.0276 | 0.0113
n=60, m=50, S=0.1 | -0.1912 | -0.0158 | 0.0551 | 0.0243 | 0.0094 | 0.0122 | -0.0092 | -0.0118 | 0.0248 | 0.0118 | 0.0309 | 0.0126
n=60, m=45, S=0.1 | -0.1458 | 0.0050 | 0.0406 | 0.0263 | 0.0098 | 0.0135 | -0.0095 | -0.0127 | 0.0184 | 0.0129 | 0.0225 | 0.0136
n=30, m=30, S=0.1 | 0.0065 | 0.0269 | 0.0206 | 0.0409 | 0.0103 | 0.0203 | -0.0102 | -0.0200 | 0.0103 | 0.0211 | 0.0103 | 0.0203
n=30, m=25, S=0.1 | 0.0175 | 0.0774 | 0.0257 | 0.0404 | 0.0126 | 0.0175 | -0.0128 | -0.0170 | 0.0133 | 0.0216 | 0.0126 | 0.0192
n=30, m=20, S=0.1 | -0.1085 | -0.0158 | 0.0527 | 0.0525 | 0.0209 | 0.0273 | -0.0201 | -0.0251 | 0.0239 | 0.0252 | 0.0297 | 0.0282
n=30, m=15, S=0.1 | -0.2812 | -0.3702 | 0.1647 | 0.3088 | 0.0456 | 0.0999 | -0.0408 | -0.0768 | 0.0675 | 0.1160 | 0.1053 | 0.2299
n=15, m=15, S=0.1 | -0.1103 | 0.0076 | 0.0669 | 0.0664 | 0.0286 | 0.0355 | -0.0261 | -0.0313 | 0.0296 | 0.0321 | 0.0388 | 0.0359
n=15, m=12, S=0.1 | -0.1113 | -0.0013 | 0.0768 | 0.0744 | 0.0334 | 0.0392 | -0.0311 | -0.0355 | 0.0343 | 0.0361 | 0.0444 | 0.0400
n=15, m=10, S=0.1 | -0.1843 | -0.0774 | 0.1151 | 0.0930 | 0.0417 | 0.0446 | -0.0396 | -0.0425 | 0.0496 | 0.0431 | 0.0693 | 0.0524
n=15, m=7, S=0.1 | -0.2589 | -0.2617 | 0.1997 | 0.3329 | 0.0685 | 0.1587 | -0.0641 | -0.1144 | 0.0819 | 0.1247 | 0.1285 | 0.2608
n=100, m=100, S=0.2 | -0.1807 | 0.0245 | 0.0302 | 0.0107 | 0.0038 | 0.0051 | -0.0037 | -0.0049 | 0.0185 | 0.0054 | 0.0219 | 0.0053
n=100, m=90, S=0.2 | -0.2043 | 0.0181 | 0.0326 | 0.0136 | 0.0055 | 0.0069 | -0.0044 | -0.0064 | 0.0190 | 0.0067 | 0.0221 | 0.0069
n=100, m=85, S=0.2 | -0.1738 | 0.0333 | 0.0493 | 0.0147 | 0.0046 | 0.0070 | -0.0045 | -0.0066 | 0.0187 | 0.0075 | 0.0215 | 0.0073
n=100, m=80, S=0.2 | -0.1962 | 0.0307 | 0.0589 | 0.0147 | 0.0058 | 0.0071 | -0.0052 | -0.0067 | 0.0223 | 0.0074 | 0.0269 | 0.0074
n=60, m=60, S=0.2 | -0.1385 | -0.0012 | 0.0270 | 0.0154 | 0.0039 | 0.0077 | -0.0040 | -0.0076 | 0.0126 | 0.0076 | 0.0145 | 0.0078
n=60, m=55, S=0.2 | -0.1589 | -0.0064 | 0.0364 | 0.0188 | 0.0055 | 0.0095 | -0.0056 | -0.0093 | 0.0167 | 0.0093 | 0.0198 | 0.0096
n=60, m=50, S=0.2 | -0.1748 | -0.0191 | 0.0397 | 0.0244 | 0.0091 | 0.0117 | -0.0087 | -0.0113 | 0.0167 | 0.0116 | 0.0197 | 0.0130
n=60, m=45, S=0.2 | -0.1818 | 0.0203 | 0.0399 | 0.0259 | 0.0079 | 0.0129 | -0.0076 | -0.0125 | 0.0150 | 0.0130 | 0.0281 | 0.0130
n=30, m=30, S=0.2 | 0.0065 | 0.0269 | 0.0206 | 0.0409 | 0.0103 | 0.0103 | -0.0102 | -0.0110 | 0.0103 | 0.0211 | 0.0103 | 0.0203
n=30, m=25, S=0.2 | 0.0480 | 0.0891 | 0.0264 | 0.0450 | 0.0121 | 0.0186 | -0.0120 | -0.0185 | 0.0138 | 0.0244 | 0.0127 | 0.0210
n=30, m=20, S=0.2 | -0.1038 | -0.0157 | 0.0502 | 0.0429 | 0.0201 | 0.0221 | -0.0195 | -0.0206 | 0.0229 | 0.0206 | 0.0281 | 0.0228
n=30, m=15, S=0.2 | -0.1553 | -0.1185 | 0.0899 | 0.0976 | 0.0341 | 0.0426 | -0.0319 | -0.0407 | 0.0392 | 0.0437 | 0.0533 | 0.0563
n=15, m=15, S=0.2 | -0.1103 | 0.0076 | 0.0669 | 0.0664 | 0.0286 | 0.0355 | -0.0261 | -0.0313 | 0.0296 | 0.0321 | 0.0388 | 0.0359
n=15, m=12, S=0.2 | -0.1147 | 0.0149 | 0.0814 | 0.0690 | 0.0348 | 0.0365 | -0.0336 | -0.0328 | 0.0368 | 0.0339 | 0.0465 | 0.0368
n=15, m=10, S=0.2 | -0.1636 | -0.0467 | 0.1055 | 0.0775 | 0.0399 | 0.0388 | -0.0387 | -0.0366 | 0.0462 | 0.0366 | 0.0620 | 0.0426
n=15, m=7, S=0.2 | -0.1570 | -0.0917 | 0.1355 | 0.1877 | 0.0585 | 0.0997 | -0.0532 | -0.0819 | 0.0584 | 0.0819 | 0.0835 | 0.1193

Table 3: Average values of the biases and Risks for the Bayesian estimates of
parameters ($\gamma$, $\delta$) = (1.5, 2) under different censoring schemes.

$\gamma$=1.5, $\delta$=2 | SE | LL
---|---|---
Bias | Risk | Bias | Risk
v=-1 | v=1 | v=-1 | v=1
$\gamma$ | $\delta$ | $\gamma$ | $\delta$ | $\gamma$ | $\delta$ | $\gamma$ | $\delta$ | $\gamma$ | $\delta$ | $\gamma$ | $\delta$
n=100, m=100, S=0.1 | -0.3000 | -0.2028 | 0.1100 | 0.0931 | 0.0100 | 0.0265 | -0.0100 | -0.0255 | 0.0483 | 0.0404 | 0.0635 | 0.0549
n=100, m=90, S=0.1 | -0.2921 | -0.2416 | 0.1045 | 0.1167 | 0.0098 | 0.0297 | -0.0095 | -0.0287 | 0.0459 | 0.0499 | 0.0603 | 0.0700
n=100, m=85, S=0.1 | -0.2543 | -0.1387 | 0.0880 | 0.0796 | 0.0116 | 0.0303 | -0.0117 | -0.0300 | 0.0389 | 0.0357 | 0.0503 | 0.0455
n=100, m=80, S=0.1 | -0.2664 | -0.1790 | 0.0962 | 0.0828 | 0.0128 | 0.0265 | -0.0124 | -0.0245 | 0.0421 | 0.0358 | 0.0556 | 0.0491
n=60, m=60, S=0.1 | -0.2141 | -0.1919 | 0.0733 | 0.1100 | 0.0138 | 0.0381 | -0.0136 | -0.0353 | 0.0325 | 0.0470 | 0.0419 | 0.0667
n=60, m=55, S=0.1 | -0.2368 | -0.2132 | 0.0895 | 0.1295 | 0.0170 | 0.0446 | -0.0165 | -0.0399 | 0.0391 | 0.0541 | 0.0521 | 0.0809
n=60, m=50, S=0.1 | -0.2476 | -0.2585 | 0.1019 | 0.1762 | 0.0208 | 0.0573 | -0.0197 | -0.0526 | 0.0438 | 0.0724 | 0.0603 | 0.1128
n=60, m=45, S=0.1 | -0.2355 | -0.2172 | 0.0947 | 0.1581 | 0.0200 | 0.0594 | -0.0192 | -0.0524 | 0.0410 | 0.0653 | 0.0556 | 0.1015
n=30, m=30, S=0.1 | 0.0164 | 0.1118 | 0.0410 | 0.1105 | 0.0209 | 0.0507 | -0.0199 | -0.0475 | 0.0205 | 0.0608 | 0.0209 | 0.0526
n=30, m=25, S=0.1 | -0.0041 | 0.1723 | 0.0545 | 0.1293 | 0.0277 | 0.0510 | -0.0267 | -0.0485 | 0.0270 | 0.0748 | 0.0283 | 0.0581
n=30, m=20, S=0.1 | -0.0830 | -0.0682 | 0.0763 | 0.2182 | 0.0363 | 0.1183 | -0.0331 | -0.0975 | 0.0344 | 0.0980 | 0.0437 | 0.1368
n=30, m=15, S=0.1 | -0.1491 | -0.2686 | 0.0986 | 0.4535 | 0.0388 | 0.2268 | -0.0375 | -0.1657 | 0.0435 | 0.1709 | 0.0576 | 0.3726
n=15, m=15, S=0.1 | -0.1570 | -0.0601 | 0.1141 | 0.2312 | 0.0463 | 0.1195 | -0.0433 | -0.1079 | 0.0495 | 0.1091 | 0.0684 | 0.1367
n=15, m=12, S=0.1 | -0.0868 | 0.0384 | 0.1218 | 0.2507 | 0.0584 | 0.1324 | -0.0560 | -0.1168 | 0.0564 | 0.1294 | 0.0694 | 0.1369
n=15, m=10, S=0.1 | -0.1774 | -0.1020 | 0.1865 | 0.3356 | 0.0838 | 0.1788 | -0.0725 | -0.1489 | 0.0778 | 0.1500 | 0.1211 | 0.2222
n=15, m=7, S=0.1 | -0.2125 | -0.3120 | 0.2017 | 0.6883 | 0.0800 | 0.3317 | -0.0759 | -0.2579 | 0.0849 | 0.2593 | 0.1273 | 0.5915
n=100, m=100, S=0.2 | -0.3000 | -0.2028 | 0.1100 | 0.0931 | 0.0100 | 0.0265 | -0.0100 | -0.0255 | 0.0483 | 0.0404 | 0.0635 | 0.0549
n=100, m=90, S=0.2 | -0.3016 | -0.2259 | 0.1107 | 0.1039 | 0.0099 | 0.0268 | -0.0098 | -0.0262 | 0.0485 | 0.0449 | 0.0639 | 0.0616
n=100, m=85, S=0.2 | -0.2929 | -0.2453 | 0.1165 | 0.1166 | 0.0106 | 0.0280 | -0.0101 | -0.0283 | 0.0466 | 0.0502 | 0.0617 | 0.0690
n=100, m=80, S=0.2 | -0.3172 | -0.2655 | 0.1253 | 0.1282 | 0.0124 | 0.0295 | -0.0123 | -0.0282 | 0.0544 | 0.0543 | 0.0733 | 0.0776
n=60, m=60, S=0.2 | -0.2141 | -0.1919 | 0.0733 | 0.1100 | 0.0138 | 0.0381 | -0.0136 | -0.0353 | 0.0325 | 0.0470 | 0.0419 | 0.0667
n=60, m=55, S=0.2 | -0.2224 | -0.2001 | 0.0805 | 0.1198 | 0.0156 | 0.0416 | -0.0155 | -0.0384 | 0.0355 | 0.0508 | 0.0463 | 0.0733
n=60, m=50, S=0.2 | -0.2549 | -0.2446 | 0.1045 | 0.1472 | 0.0208 | 0.0462 | -0.0189 | -0.0416 | 0.0447 | 0.0609 | 0.0625 | 0.0929
n=60, m=45, S=0.2 | -0.2654 | -0.2651 | 0.1122 | 0.1742 | 0.0213 | 0.0535 | -0.0204 | -0.0507 | 0.0481 | 0.0721 | 0.0666 | 0.1101
n=30, m=30, S=0.2 | 0.0164 | 0.1118 | 0.0410 | 0.1105 | 0.0209 | 0.0507 | -0.0199 | -0.0475 | 0.0205 | 0.0608 | 0.0209 | 0.0526
n=30, m=25, S=0.2 | 0.0215 | 0.1535 | 0.0479 | 0.1240 | 0.0244 | 0.0517 | -0.0230 | -0.0486 | 0.0240 | 0.0705 | 0.0244 | 0.0567
n=30, m=20, S=0.2 | -0.0673 | 0.0020 | 0.0682 | 0.1549 | 0.0334 | 0.0807 | -0.0304 | -0.0741 | 0.0311 | 0.0771 | 0.0387 | 0.0838
n=30, m=15, S=0.2 | -0.0398 | 0.0052 | 0.0717 | 0.2535 | 0.0361 | 0.1389 | -0.0342 | -0.1153 | 0.0342 | 0.1229 | 0.0390 | 0.1482
n=15, m=15, S=0.2 | -0.1570 | -0.0601 | 0.1141 | 0.2312 | 0.0463 | 0.1195 | -0.0433 | -0.1079 | 0.0495 | 0.1091 | 0.0684 | 0.1367
n=15, m=12, S=0.2 | -0.0064 | 0.1491 | 0.0910 | 0.2526 | 0.0457 | 0.1221 | -0.0452 | -0.1086 | 0.0460 | 0.1449 | 0.0471 | 0.1225
n=15, m=10, S=0.2 | -0.1921 | -0.1076 | 0.1435 | 0.2756 | 0.0635 | 0.1424 | -0.0549 | -0.1224 | 0.0639 | 0.1225 | 0.0892 | 0.1764
n=15, m=7, S=0.2 | -0.0583 | 0.0916 | 0.1439 | 0.3993 | 0.0715 | 0.2239 | -0.0688 | -0.1702 | 0.0689 | 0.2077 | 0.0903 | 0.2330

Table 4: Confidence length and coverage probability for the parameters
($\gamma$,$\delta$) under different censoring schemes.

|  $\gamma$=1, $\delta$=1 |  $\gamma$=1.5, $\delta$=2
---|---|---
| Bayesian HPDC | MLE | Bayesian HPDC | MLE
| CL | CP | CL | CP | CL | CP | CL | CP
n=100, m=100, S=0.1 | (0.3180, | (0.951, | (0.3099, | (0.952, | (0.5288, | (0.952, | (0.4649, | (0.952,
0.3563) | 0.952) | 0.4159) | 0.96) | 0.8520) | 0.952) | 0.8187) | 0.957)
n=100, m=90, S=0.1 | (0.3688, | (0.952, | (0.3442, | (0.946, | (0.5515, | (0.951, | (0.5068, | (0.95,
0.4162) | 0.951) | 0.4242) | 0.954) | 0.9159) | 0.951) | 0.8822) | 0.947)
n=100, m=85, S=0.1 | (0.3686, | (0.953, | (0.3601, | (0.954, | (0.5738, | (0.952, | (0.5270, | (0.947,
0.4303) | 0.953) | 0.4315) | 0.956) | 0.9019) | 0.951) | 0.9228) | 0.957)
n=100, m=80, S=0.1 | (0.3377, | (0.952, | (0.3758, | (0.952, | (0.5815, | (0.951, | (0.5471, | (0.95,
0.4366) | 0.951) | 0.4410) | 0.955) | 0.8816) | 0.954) | 0.9688) | 0.956)
n=60, m=60, S=0.1 | (0.3242, | (0.957, | (0.4038, | (0.954, | (0.6451, | (0.951, | (0.6056, | (0.954,
0.4537) | 0.952) | 0.5375) | 0.948) | 1.0304) | 0.953) | 1.0671) | 0.959)
n=60, m=55, S=0.1 | (0.4955, | (0.953, | (0.4425, | (0.954, | (0.7269, | (0.952, | (0.6526, | (0.955,
0.5992) | 0.955) | 0.5468) | 0.953) | 1.0754) | 0.951) | 1.1389) | 0.956)
n=60, m=50, S=0.1 | (0.4951, | (0.954, | (0.4782, | (0.951, | (0.7693, | (0.951, | (0.6979, | (0.959,
0.6076) | 0.954) | 0.5642) | 0.952) | 1.3370) | 0.952) | 1.2349) | 0.962)
n=60, m=45, S=0.1 | (0.5466, | (0.954, | (0.5151, | (0.956, | (0.7519, | (0.951, | (0.7445, | (0.955,
0.6179) | 0.951) | 0.5921) | 0.961) | 1.2934) | 0.951) | 1.3555) | 0.966)
n=30, m=30, S=0.1 | (0.5174, | (0.953, | (0.5891, | (0.947, | (0.8079, | (0.951, | (0.8836, | (0.947,
0.7871) | 0.951) | 0.7750) | 0.943) | 1.1309) | 0.951) | 1.5953) | 0.966)
n=30, m=25, S=0.1 | (0.5806, | (0.951, | (0.6969, | (0.957, | (0.8977, | (0.951, | (1.0151, | (0.958,
0.6939) | 0.953) | 0.8267) | 0.952) | 1.1974) | 0.951) | 1.8826) | 0.963)
n=30, m=20, S=0.1 | (0.8191, | (0.952, | (0.8210, | (0.953, | (0.9861, | (0.951, | (1.1686, | (0.959,
0.8541) | 0.953) | 0.9678) | 0.951) | 1.7118) | 0.951) | 2.4091) | 0.968)
n=30, m=15, S=0.1 | (1.0720, | (0.951, | (0.9854, | (0.948, | (1.0137, | (0.951, | (1.3746, | (0.954,
1.5067) | 0.951) | 1.3004) | 0.965) | 2.3050) | 0.951) | 3.4663) | 0.958)
n=15, m=15, S=0.1 | (0.8566, | (0.954, | (0.8816, | (0.945, | (1.0880, | (0.953, | (1.3225, | (0.945,
0.9960) | 0.951) | 1.1293) | 0.934) | 1.8097) | 0.951) | 2.5000) | 0.965)
n=15, m=12, S=0.1 | (0.9598, | (0.951, | (1.0852, | (0.959, | (1.3420, | (0.951, | (1.5635, | (0.95,
1.0907) | 0.952) | 1.2963) | 0.94) | 1.7570) | 0.951) | 3.2631) | 0.962)
n=15, m=10, S=0.1 | (1.1078, | (0.951, | (1.2523, | (0.961, | (1.4652, | (0.951, | (1.7649, | (0.952,
1.1124) | 0.951) | 1.5622) | 0.947) | 2.1629) | 0.951) | 4.2863) | 0.959)
n=15, m=7, S=0.1 | (1.3373, | (0.951, | (1.6139, | (0.959, | (1.4974, | (0.951, | (2.1922, | (0.958,
1.8838) | 0.951) | 2.7710) | 0.963) | 2.8849) | 0.951) | 7.8173) | 0.969)

Table 5: Confidence length and coverage probability for the parameters
($\gamma$,$\delta$) under different censoring schemes.

|  $\gamma=1$, $\delta=1$ |  $\gamma=1.5$, $\delta=2$
---|---|---
| Bayesian HPDC | MLE | Bayesian HPDC | MLE
| CL | CP | CL | CP | CL | CP | CL | CP
n=100, m=100, S=0.2 | (0.3179, | (0.951, | (0.3099, | (0.952, | (0.5288, | (0.952, | (0.4648, | (0.952,
0.3562) | 0.952) | 0.4158) | 0.96) | 0.8519) | 0.952) | 0.8187) | 0.957)
n=100, m=90, S=0.2 | (0.3660, | (0.951, | (0.3411, | (0.948, | (0.5274, | (0.956, | (0.4953, | (0.947,
0.4885) | 0.952) | 0.4229) | 0.954) | 0.9306) | 0.952) | 0.8611) | 0.952)
n=100, m=85, S=0.2 | (0.3942, | (0.952, | (0.3553, | (0.953, | (0.5649, | (0.956, | (0.5108, | (0.951,
0.4954) | 0.954) | 0.4288) | 0.953) | 0.9012) | 0.951) | 0.8890) | 0.951)
n=100, m=80, S=0.2 | (0.4006, | (0.951, | (0.3695, | (0.951, | (0.6132, | (0.951, | (0.5261, | (0.947,
0.4243) | 0.952) | 0.4368) | 0.953) | 0.9005) | 0.952) | 0.9200) | 0.95)
n=60, m=60, S=0.2 | (0.3241, | (0.957, | (0.4037, | (0.954, | (0.6451, | (0.951, | (0.6056, | (0.954,
0.4537) | 0.952) | 0.5375) | 0.948) | 1.0303) | 0.953) | 1.0670) | 0.959)
n=60, m=55, S=0.2 | (0.3853, | (0.952, | (0.4389, | (0.953, | (0.6768, | (0.954, | (0.6394, | (0.956,
0.5344) | 0.954) | 0.5454) | 0.954) | 1.1031) | 0.954) | 1.1143) | 0.961)
n=60, m=50, S=0.2 | (0.5138, | (0.951, | (0.4709, | (0.96, | (0.7775, | (0.951, | (0.6738, | (0.955,
0.6211) | 0.953) | 0.5598) | 0.95) | 1.1386) | 0.953) | 1.1795) | 0.96)
n=60, m=45, S=0.2 | (0.4386, | (0.952, | (0.5034, | (0.952, | (0.7577, | (0.951, | (0.7089, | (0.957,
0.6178) | 0.953) | 0.5818) | 0.956) | 1.2459) | 0.951) | 1.2593) | 0.962)
n=30, m=30, S=0.2 | (0.5174, | (0.953, | (0.5890, | (0.947, | (0.8078, | (0.951, | (0.8836, | (0.947,
0.7870) | 0.951) | 0.7749) | 0.943) | 1.1308) | 0.951) | 1.5953) | 0.966)
n=30, m=25, S=0.2 | (0.5693, | (0.951, | (0.6851, | (0.959, | (0.7871, | (0.953, | (0.9777, | (0.953,
0.7666) | 0.951) | 0.8172) | 0.951) | 1.2227) | 0.951) | 1.7784) | 0.963)
n=30, m=20, S=0.2 | (0.8479, | (0.951, | (0.7903, | (0.954, | (0.8870, | (0.951, | (1.0878, | (0.961,
0.7257) | 0.951) | 0.9183) | 0.949) | 1.4659) | 0.953) | 2.0885) | 0.97)
n=30, m=15, S=0.2 | (0.9756, | (0.953, | (0.9230, | (0.959, | (1.0329, | (0.951, | (1.2294, | (0.953,
1.0953) | 0.951) | 1.1366) | 0.951) | 1.8774) | 0.951) | 2.6186) | 0.959)
n=15, m=15, S=0.2 | (0.8566, | (0.954, | (0.8816, | (0.945, | (1.0879, | (0.953, | (1.3224, | (0.945,
0.9959) | 0.951) | 1.1293) | 0.934) | 1.8096) | 0.951) | 2.5000) | 0.965)
n=15, m=12, S=0.2 | (0.9949, | (0.951, | (1.0575, | (0.958, | (1.1353, | (0.951, | (1.4842, | (0.947,
0.9821) | 0.951) | 1.2549) | 0.939) | 1.8291) | 0.951) | 2.9176) | 0.964)
n=15, m=10, S=0.2 | (1.0483, | (0.952, | (1.1943, | (0.955, | (1.3200, | (0.951, | (1.6201, | (0.959,
1.0680) | 0.952) | 1.4344) | 0.944) | 1.9196) | 0.952) | 3.4056) | 0.966)
n=15, m=7, S=0.2 | (1.2669, | (0.952, | (1.4606, | (0.964, | (1.4345, | (0.952, | (1.8905, | (0.955,
1.5338) | 0.951) | 1.9870) | 0.959) | 2.2919) | 0.951) | 4.6995) | 0.962)

##  8 Real Data Example

For the real-data application, we use the data from Hinkley [7], which
consists of 30 values of March precipitation readings (in inches) for
Minneapolis/St. Paul. The data set is provided in Table 6. First, it must be
verified that this data set can be used to analyze the Weibull distribution.
Also, for the sake of comparison, Lindley and Inverse Weibull (IW)
distributions are also taken into consideration. We looked at the Kolmogorov-
Simirnov (KS) test to determine the quality of fit. In order to confirm the
data’s suitability, other criteria are also taken into account, including the
Akaike information criterion (AIC), the corrected Akaike information criterion
(AICC), the Bayesian information criterion (BIC), the Hannan-Quinn criterion
(HQC), and negative log-likelihood (NLL). The distribution with the smallest
estimated values of the KS statistic, AIC, AICC, BIC, HQC, and NLL and a
comparatively high p-value is the better lifetime model. Table 7 indicates
that the Weibull model fits the data far better than the other models.
Additionally, we provide fitted density, empirical distribution function, PP-
plot and QQ-plot in Figure 1. Thus, Weibull distribution is used to draw
conclusions about the data under consideration.

Table 6: Data of 30 successive values of March precipitation 0.77 | 1.74 | 0.81 | 1.20 | 1.95 | 1.20 | 0.47 | 1.43 | 3.37 | 2.20
---|---|---|---|---|---|---|---|---|---
3.00 | 3.09 | 1.51 | 2.10 | 0.52 | 1.62 | 1.31 | 0.32 | 0.59 | 0.81
2.81 | 1.87 | 1.18 | 1.35 | 4.75 | 2.48 | 0.96 | 1.89 | 0.90 | 2.05
Table 7: The goodness-of-fit statistics for the data set

Distribution | Estimates | KS | p-value | AIC | AICC | BIC | HQC | NLL
---|---|---|---|---|---|---|---|---
Weibull | 1.8089 | 0.3155 | 0.0689 | 0.9988 | 81.2866 | 81.7310 | 84.0890 | 82.1831 | 77.2866
Lindley | 0.9096 |  | 0.1882 | 0.2383 | 88.2875 | 88.4303 | 89.6886 | 88.7357 | 86.2875
IW | 1.0162 | 1.5496 | 0.1524 | 0.4893 | 154.9602 | 155.4046 | 157.7626 | 155.8567 | 150.9602

Figure 1: Probability plots from the fitted Weibull distribution

In order to analyse the data set under the T1T2 censoring scheme, we created
artificial data sets from the real data for the schemes,
$(m,S)=(20,1),(20,2),(15,1)$ and $(15,2)$. For each sample, MLE, the
associated asymptotic confidence interval (ACI), the Bayes estimates under the
squared error loss function, and the HPDC are calculated and are provided in
the Table 8. Bayes estimates are calculated with the assumption that all
hyperparameters are zeros. For each censoring scheme, MLE and Bayes estimates
are nearly, but not quite, identical. In the case of confidence intervals, the
HPDC is shorter than the ACI. For a fixed value of $m$, interval length
decreases with supplymentary time S. The interval length of the scale
parameter is shorter than that of the shape parameter.

Table 8: Estimate values under different censoring schemes Scheme | MLE | Bayesian
---|---|---
(m, S) | Estimates | ACI | Estimates | HPDC
(20,1) |  $\hat{\gamma}$=1.8461 | (1.2618, 2.4304) |  $\hat{\gamma}$=1.9131 | (1.4346, 2.4902)
$\hat{\delta}$=0.3099 | (0.1307, 0.4891) |  $\hat{\delta}$=0.2989 | (0.1677, 0.5092)
(20,2) |  $\hat{\gamma}$=1.8534 | (1.3243, 2.3826) |  $\hat{\gamma}$=1.7649 | (1.3577, 2.2958)
$\hat{\delta}$=0.3105 | (0.1327, 0.4882) |  $\hat{\delta}$=0.3491 | (0.1813, 0.5321)
(15,1) |  $\hat{\gamma}$=1.9386 | (1.2786, 2.5985) |  $\hat{\gamma}$=1.887 | ( 1.2267, 2.4044)
$\hat{\delta}$=0.3042 | (0.1259, 0.4825) |  $\hat{\delta}$= 0.3269 | (0.1614, 0.4939)
(15,2) |  $\hat{\gamma}$=1.9174 | (1.3644, 2.4704) |  $\hat{\gamma}$=1.8477 | (1.3800, 2.3001)
$\hat{\delta}$=0.3031 | (0.1272, 0.4791) |  $\hat{\delta}$=0.3325 | (0.2195, 0.5441)

##  9 Conclusion

In this paper, we have introduced the T1-T2 mixture censoring scheme and
examined its statistical inference based on the Weibull distribution. The new
scheme guarantees a maximum possible number of failures within the prefixed
time limit. The expected value of the failures and the expected duration of
experiment are derived and it has been seen that both are invariant under
scalar multiplication. The Fisher information for the new censoring scheme is
presented. The performances of MLE and Bayesian estimates for squared error
and LINEX loss function, demonstrate their effectiveness in providing reliable
information. A real data is used to illustrate the functionality of the T1-T2
mixture censoring scheme.

## References

  * [1] M Burkschat.  On optimality of extremal schemes in progressive type ii censoring.  Journal of Statistical Planning and Inference, 138(6):1647–1659, 2008.
  * [2] A Childs, B Chandrasekar, N Balakrishnan, and D Kundu.  Exact likelihood inference based on type-i and type-ii hybrid censored samples from the exponential distribution.  Annals of the Institute of Statistical Mathematics, 55(2):319–330, 2003.
  * [3] A Clifford Cohen.  Progressively censored samples in life testing.  Technometrics, 5(3):327–339, 1963.
  * [4] Bradley Efron and Iain M Johnstone.  Fisher’s information in terms of the hazard rate.  The Annals of Statistics, pages 38–62, 1990.
  * [5] Benjamin Epstein.  Truncated life tests in the exponential case.  The Annals of Mathematical Statistics, pages 555–564, 1954.
  * [6] W Keith Hastings.  Monte carlo sampling methods using markov chains and their applications.  1970\.
  * [7] David Hinkley.  On quick choice of power transformation.  Journal of the Royal Statistical Society: Series C (Applied Statistics), 26(1):67–69, 1977.
  * [8] Debasis Kundu.  On hybrid censored weibull distribution.  Journal of Statistical Planning and Inference, 137(7):2127–2142, 2007.
  * [9] Debasis Kundu.  Bayesian inference and life testing plan for the weibull distribution in presence of progressive censoring.  Technometrics, 50(2):144–154, 2008.
  * [10] Nicholas Metropolis, Arianna W Rosenbluth, Marshall N Rosenbluth, Augusta H Teller, and Edward Teller.  Equation of state calculations by fast computing machines.  The journal of chemical physics, 21(6):1087–1092, 1953.
  * [11] Hon Keung Tony Ng, Debasis Kundu, and Ping Shing Chan.  Statistical analysis of exponential lifetimes under an adaptive type-ii progressive censoring scheme.  Naval Research Logistics (NRL), 56(8):687–698, 2009.
  * [12] Sangun Park and N Balakrishnan.  On simple calculation of the fisher information in hybrid censoring schemes.  Statistics & Probability Letters, 79(10):1311–1319, 2009.


# DiaLex:
A Benchmark for Evaluating Multidialectal Arabic Word Embeddings

Muhammad Abdul-Mageed1, Shady Elbassuoni2, Jad Doughman2, AbdelRahim
Elmadany1,
El Moatez Billah Nagoudi1, Yorgo Zoughby2, Ahmad Shaher1, Iskander Gaba2 ,
Ahmed Helal3, Mohammed El-Razzaz4
1 Natural Language Processing Lab, The University of British Columbia,
Vancouver, Canada
2 American University of Beirut, Beirut, Lebanon
3 Concordia University, Montreal, Canada
4 Arab Academy for Science and Technology, Cairo, Egypt
1<EMAIL_ADDRESS>
2 sd58,jad17,ytz00,amh90<EMAIL_ADDRESS>3<EMAIL_ADDRESS>4
<EMAIL_ADDRESS>
utf8

Abstract

Word embeddings are a core component of modern natural language processing
systems, making the ability to thoroughly evaluate them a vital task. We
describe DiaLex, a benchmark for intrinsic evaluation of dialectal Arabic word
embeddings. DiaLex covers five important Arabic dialects: Algerian, Egyptian,
Lebanese, Syrian, and Tunisian. Across these dialects, DiaLex provides a
testbank for six syntactic and semantic relations, namely male to female,
singular to dual, singular to plural, antonym, comparative, and genitive to
past tense. DiaLex thus consists of a collection of word pairs representing
each of the six relations in each of the five dialects. To demonstrate the
utility of DiaLex, we use it to evaluate a set of existing and new Arabic word
embeddings that we developed. Beyond evaluation of word embeddings, DiaLex
supports efforts to integrate dialects into the Arabic language curriculum. It
can be easily translated into Modern Standard Arabic and English, which can be
useful for evaluating word translation. Our benchmark, evaluation code, and
new word embedding models will be publicly available.
111https://github.com/UBC-NLP/dialex.

##  1 Introduction

Word embeddings are the backbone of modern natural language processing (NLP)
systems. They encode semantic and syntactic relations between words by
representing them in a low-dimensional space. Many techniques have been
proposed to learn such embeddings Pennington et al. (2014); Mikolov et al.
(2013a); Mnih and Kavukcuoglu (2013) from large text corpora. As of today, a
large number of such embeddings are available in many languages including
Arabic. Due to their importance, it is vital to be able to evaluate word
embeddings, and various methods have been proposed for evaluating them. These
methods can be broadly categorized into intrinsic evaluation methods and
extrinsic evaluation ones. For extrinsic evaluation, word embeddings are
assessed based on performance in downstream applications. For intrinsic
evaluation, they are assessed based on how well they capture syntactic and
semantic relations between words.

Figure 1: A map of the five Arab countries covered by DiaLex. The five
countries cover different regions in the Arab world: two in the western region
(Algeria and Tunisia), one in the middle (Egypt), and two in the eastern
region (Lebanon and Syria).

Although there exists a benchmark for evaluating modern standard Arabic (MSA)
word embeddings Elrazzaz et al. (2017), no such resource that we know of
exists for Arabic dialects. This makes it difficult to measure progress on
Arabic dialect processing. In this paper, our goal is to facilitate intrinsic
evaluation of dialectal Arabic word embeddings. To this end, we build a new
benchmark spanning five different Arabic dialects, from Eastern, Middle, and
Western Arab World. Namely, our benchmark covers Algerian (ALG), Egyptian
(EGY), Lebanese (LEB), Syrian (SYR), and Tunisian (TUN). Figure 1 shows a map
of the five Arab countries covered by DiaLex. For each one of these dialects,
DiaLex consists of a set of word pairs that are syntactically or semantically
related by one of six different relations: Male to Female, Singular to Plural,
Singular to Dual, Antonym, Comparative, and Genitive to Past Tense. Overall,
DiaLex consists of over $3,000$ word pairs in those five dialects, evenly
distributed. To the best of our knowledge, DiaLex is the first benchmark that
can be used to assess the quality of Arabic word embeddings in the five
dialects it covers.

To be able to use DiaLex to evaluate Arabic word embeddings, we generate a set
of word analogy questions from the word pairs in DiaLex. A word analogy
question is generated from two word pairs from a given relation. These
questions have recently become the standard in intrinsic evaluation of word
embeddings Mikolov et al. (2013a); Gao et al. (2014); Schnabel et al. (2015).
To demonstrate the usefulness of DiaLex in evaluating Arabic word embeddings,
we use it to evaluate a set of existing and new Arabic word embeddings. We
conclude that both available and newly-developed word embedding models have
moderate-to-serious coverage issues and are not sufficiently representative of
the respective dialects under study. In addition to the benchmark of word
pairs, our newly-developed dialectal Arabic word embeddings will also be
publicly available.222Our benchmark, evaluation code, and new word embedding
models will be available at: https://github.com/UBC-NLP/dialex.

Beyond evaluation of word embeddings, we envision DiaLex as a basis for
creating multidialectal Arabic resources that can facilitate study of the
semantics and syntax of Arabic dialects, for example for pedagogical
applications Mubarak et al. (2020). More broadly, we hope DiaLex will
contribute to efforts for integrating dialects in the Arabic language
curriculum Al-Batal (2017). DiaLex can also be used to complement a growing
interest in contextual word embeddings Peters et al. (2018) and self-
supervised language models Devlin et al. (2019), including in Arabic Antoun et
al. (2020); Abdul-Mageed et al. (2020a); Lan et al. (2020a). Extensions of
DiaLex can also be valuable for NLP, for example, DiaLex can be easily
translated into MSA, other Arabic dialects, English, or other languages. This
extension can enable evaluation of word-level translation systems, including
in cross-lingual settings Aldarmaki et al. (2018); Aldarmaki and Diab (2019).
Our resources can also be used in comparisons against contextual embeddings
Peters et al. (2018) and embeddings acquired from language models such as BERT
Devlin et al. (2019). For example, it can be used in evaluation settings with
Arabic language models such as AraBert Antoun et al. (2020), GigaBERT Lan et
al. (2020b), and the recently developed ARBERT and MARBERT Abdul-Mageed et al.
(2020a). More generally, our efforts are motivated by the fact that the study
of Arabic dialects and computational Arabic dialect processing is a nascent
area with several existing gaps to fill Bouamor et al. (2019); Abbes et al.
(2020); Abdul-Mageed et al. (2020b, 2021, c).

The rest of the paper is organized as follows. In Section 2, we describe how
DiaLex was constructed. Section 3 offers our methodical generation of a
testbank for evaluating word embeddings. In Section 4, we provide a case study
for evaluating various word embedding models, some of which are newly
developed by us. Section 5 is about related work. Section 6 is where we
conclude and present future directions.

##  2 Benchmark Construction

Dialect | antonym | compara | genitive-pt | male-fem | sing-pl | sing-dual | all
---|---|---|---|---|---|---|---
Algerian | 100 | 100 | 100 | 100 | 102 | 105 | 607
Egyptian | 98 | 98 | 998 | 99 | 97 | 98 | 588
Lebanese | 98 | 99 | 109 | 96 | 105 | 126 | 633
Syrian | 98 | 97 | 98 | 101 | 97 | 102 | 593
Tunisian | 100 | 100 | 100 | 100 | 102 | 147 | 649
Total | 494 | 494 | 505 | 496 | 503 | 578 | 3,070
Table 1: Statistics of DiaLex across different relations. (Genitive-pt=
genitive-past tense).

DiaLex consists of a set of word pairs in five different Arabic dialects and
for six different semantic and syntactic relations, namely Male to Female,
Singular to Plural, Singular to Dual, Antonym, Comparative, and Genitive to
Past Tense. We chose only this set of relations as they are standard in
previous literature. In addition, they are comprehensive enough to reflect
dialect specificity. A good word embeddings model should thus have close
representation of word pairs for each of these relations in the embeddings
space.

For each dialect, word pairs were manually generated by at least one native
speaker of the dialect. Each person independently came up with the word pairs
representing a given relation based on their knowledge of the dialect and
while trying to include words that are typically representative and unique in
that dialect. That is, to the best of our ability, the words were chosen so
that they are frequently-used words in the dialect and are not the same as in
MSA. One challenge we faced when generating the word pairs was orthographic
variation. For example, consider the antonym of the word ¡ورا¿ (“behind”) in
the Egyptian dialect. It can be written as ¡قدام¿ or ¡أودام¿ (“in front of”) .
We decided to include all variations of the same word in the benchmark. A
second challenge was the need to present the relationship using more than one
word. For instance, consider Algerian and the relationship dual, the word
¡زوج¿ (“pair”) is sometimes used to describe two items of something. So, for
instance, for the word ¡ساعه¿ (“hour”), the dual can be either ¡زوج سوايع¿
(“pair of hours”) or ¡ساعتين¿ (“two hours”). Again, we opted for including
both variants in the benchmark. Overall, for each dialect and each relation,
around 100 word pairs were generated. Table 1 shows the statistics of DiaLex’s
word pairs lists and Table 2 shows some example word pairs in DiaLex and their
English and MSA translations. Overall, DiaLex consists of a total of $3,070$
word pairs distributed evenly among the dialects and the relations.

Dialect | male-fem | sing-pl | sing-dual
---|---|---|---
Algerian | ¡دجاجة¿ - ¡سردوك¿ | ¡زوج كسكروطات¿ - ¡كسكروط¿ | ¡زوج بليغات¿ - ¡بليغة¿
MSA | ¡دجاجة¿ - ¡ديك¿ | ¡شطيرتان¿ - ¡شطيرة¿ | ¡نعل¿ - ¡نعلان¿
Eng. | rooster - chicken | sandwich - two sandwiches | shoe - pair of shoes
Egyptian | ¡عسولة¿ - ¡عسول¿ | ¡إوض¿ - ¡أوضة¿ | ¡ودنين¿ - ¡ودن¿
MSA | ¡لطيفة¿ - ¡لطيف¿ | ¡غرف¿ - ¡غرفة¿ | ¡أذنين¿ - ¡أذن¿
Eng. | nice (m) - nice (f) | room - rooms | ear - two ears
Lebanese | ¡مرا¿ - ¡رجال¿ | ¡اداديح¿ - ¡إداحة¿ | ¡ملئعتين¿ - ¡ملئعة¿
MSA | ¡إمرأة¿ - ¡رجل¿ | ¡ولاعات¿ - ¡ولاعة¿ | ¡ملعقتين¿ - ¡ملعقة¿
Eng. | man - woman | lighter - lighters | spoon - two spoons
Syrian | ¡فارة¿ - ¡جردون¿ | ¡كاتويات¿ - ¡كاتو¿ | ¡زنارين¿ - ¡زنار¿
MSA | ¡فأرة¿ - ¡جرذ¿ | ¡كعكات¿ - ¡كعكة¿ | ¡حزامين¿ - ¡حزام¿
Eng. | mouse (m) - mouse (f) | cake - cakes | belt - belts
Tunisian | ¡تحفونة¿ - ¡تحفون¿ | ¡علالش¿ - ¡علوش¿ | ¡زوز ريدووات¿ - ¡ريدو¿
MSA | ¡جميلة¿ - ¡وسيم¿ | ¡خرفان¿ - ¡خروف¿ | ¡ستاران¿ - ¡ستار¿
Eng. | handsome - beautiful | sheep (sing) - sheep (pl) | curtain - two curtains
Dialect | antonym | comparative | genitive-past tense
Algerian | ¡ضعيف¿ - ¡قاوي¿ | ¡قرعاج أكثر¿ - ¡قرعاج¿ | ¡تحواس¿ - ¡حوّس¿
MSA | ¡ضعيف¿ - ¡قوي¿ | ¡أكثر فضولا¿ - ¡فضولي¿ | ¡نزهة¿ - ¡تنزه¿
Eng. | strong - weak | nosy - nosier | promenade - promenaded
Egyptian | ¡أودام¿ - ¡ورا¿ | ¡أوحش¿ - ¡وحش¿ | ¡عياط¿ - ¡عيط¿
MSA | ¡أمام¿ - ¡خلف¿ | ¡أسوأ¿ - ¡سئ¿ | ¡بكاء¿ - ¡بكي¿
Eng. | back - front | bad - worse | cried - crying
Lebanese | ¡مزعوج¿ - ¡مرتاح¿ | ¡اجكل¿ - ¡جكل¿ | ¡فوتة¿ - ¡فات¿
MSA | ¡منزعج¿ - ¡مرتاح¿ | ¡أوسم¿ - ¡وسيم¿ | ¡دخول¿ - ¡دخل¿
Eng. | restful - restless | handsome - more handsome | entered - entering
Syrian | ¡خبط¿ - ¡دفش¿ | ¡افهم¿ - ¡فهمان¿ | ¡بحبشة¿ - ¡بحبش¿
MSA | ¡اصطدم¿ - ¡دفع¿ | ¡أذكي¿ - ¡ذكي¿ | ¡تفتيش¿ - ¡فتش¿
Eng. | push - bump into | smart - smarter | inspect - inspection
Tunisian | ¡مزهار¿ - ¡منحوس¿ | ¡ميزر أكثر¿ - ¡ميزر¿ | ¡تصرفيق¿ - ¡صرفق¿
MSA | ¡محظوظ¿ - ¡منكود¿ | ¡أفقر¿ - ¡فقير¿ | ¡صفع¿ - ¡صفع¿
Eng. | unlucky - lucky | poor - poorer | slap - slapping

Table 2: Example DiaLex word pairs in every dialect across the various
relations. For each pair, we also provide MSA and English translations.

##  3 Testbank for Evaluating Word Embeddings and Evaluation Metric

Given the word pair lists in DiaLex, we generate a testbank consisting of
260,827 tuples. Each tuple consists of two word pairs $(a,b)$ and $(c,d)$ from
the same relation and the same dialect. For each of our five dialects and for
each of our six relations, we generate a tuple by combining two different word
pairs from the same relation in the same dialect. Once tuples have been
generated, they can be used as word analogy questions to evaluate different
word embeddings as defined by Mikolov et al. Mikolov et al. (2013a). A word
analogy question for a tuple consisting of two word pairs $(a,b)$ and $(c,d)$
can be formulated as follows: ”$a$ to $b$ is like $c$ to $?$”. Each such
question will then be answered by calculating a target vector $t=b-a+c$. We
then calculate the cosine similarity between the target vector $t$ and the
vector representation of each word $w$ in a given word embeddings $V$.
Finally, we retrieve the most similar word $w$ to $t$, i.e., $argmax_{w\in
V\&w\notin\\{a,b,c\\}}\frac{w\cdot t}{||w||||t||}$. If $w=d$ (i.e., the same
word) then we assume that the word embeddings $V$ has answered the question
correctly.

Moreover, we extend the traditional word analogy task by taking into
consideration if the correct answer is among the top $K$, with
$K\in\\{5,10\\}$, closest words in the embedding space to the target vector
$t$, which allows us to more leniently evaluate the embeddings. This is
particularly important in the case of Arabic since many forms of the same word
exist, usually with additional prefixes or suffixes such as the equivalent of
the article “the” or possessive determiners such as “her”, “his”, or “their”.
For example, consider one question which asks ¡راجل¿ to ¡ست¿ is like ¡أمير¿ to
“ ?”, i.e., “man” to “woman” is like “prince” to “?”, with the answer being
“¡أميره¿” or “princess”. Now, if we rely only on the top-1 word and it happens
to be “¡للأميرة¿” which means “for the princess” in English, the question
would be considered to be answered wrongly. To relax this, and ensure that
different forms of the same word will not result in a mismatch, we use the
top-5 and top-10 words for evaluation rather than just the top-1.

Note that we consider a question to be answered wrongly if at least one of the
words in the question are not present in the word embeddings. That is, we take
into consideration the coverage of the embeddings as well Gao et al. (2014).

Finally, we report the number of questions that were answered correctly over
the total numbers of questions available. That is, assume the number of
questions for a given dialect and a given relation is $n$, and assume that a
given embeddings model $M$ correctly answered $m$ out of those $n$ questions
as explained above. Then, the accuracy of the model $M$ will be $\frac{m}{n}$.

##  4 Evaluation of Arabic Word Embeddings Using DiaLex

In this section, we demonstrate how DiaLex can be used to evaluate word
embeddings across the different dialects it covers. Particularly, we evaluate
two large word embeddings models based on Word2Vec Mikolov et al. (2013b)
released by Zahran et al. (2015). One model is based on skip grams (Zah_SG)
and the other is a continuous bag-of-words (Zah_CBOW). Both of these models
have a vocabulary size of $626,3435$ words. We also create four CBOW Word2Vec
models, all of which have 300 word vector embedding dimensions, as we describe
next.

###  4.1 Newly-Developed Word Embedding Models

The following are our four newly-developed dialectal Arabic word embedding
models:

Twitter-1B. Our first model was trained using a one billion in-house Arabic
tweet collection. All tweets were crawled by putting a bounding box crawler
around the whole Arab world. Since this collection is large, we only performed
light pre-processing on it. This involved removing hashtags, URLs, and
reducing consecutive repetitions of the same character into only 2. We then
trained a CBOW Word2Vec model using the Python library gensim. We set the
minimum word frequency at 100 and a window size of 5 words. This model has a
vocabulary size of $929,803$ words.

Dialect Model K=1 K=5 K=10 ALG Zah_SG 1.79 6.16 7.96 Zah_CBOW 3.40 8.51 10.78
2-5 Ours-1B 1.32 3.60 5.53 Ours-MC-50 3.02 8.25 10.90 Ours-Seeds 3.88 9.19
11.51 Ours-MC-100 3.87 11.33 15.03 EGY Zah_SG 1.90 5.93 8.17 Zah_CBOW 2.06
6.75 8.98 2-5 Ours-1B 2.09 5.86 8.65 Ours-MC-50 4.68 11.59 15.67 Ours-Seeds
3.21 8.19 11.63 Ours-MC-100 4.33 11.49 15.97 SYR Zah_SG 1.53 4.14 5.20
Zah_CBOW 1.86 4.76 6.15 2-5 Ours-1B 1.23 4.036 6.85 Ours-MC-50 2.49 6.50 8.66
Ours-Seeds 2.54 5.88 8.02 Ours-MC-100 3.14 9.39 12.97 LEB Zah_SG 5.14 10.97
13.69 Zah_CBOW 6.72 12.88 15.51 2-5 Ours-1B 2.29 5.71 7.70 Ours-MC-50 3.97
9.71 13.83 Ours-Seeds 3.96 8.77 11.66 Ours-MC-100 5.29 12.90 17.87 TUN
Zahran_SG 1.38 6.06 8.26 Zah_CBOW 2.63 8.19 10.77 2-5 Ours-1B 2.53 5.54 8.47
Ours-MC-50 3.81 8.07 12.20 Ours-Seeds 3.80 9.08 12.49 Ours-MC-100 2.49 7.68
12.14

Table 3: Evaluation of six word embedding models using our benchmark across
the five dialects. In most cases, our Twitter-250K-MC100 (Ours-MC-100)
achieves the best performance.

Twitter-250K-MC50. This model uses the same data as Twitter-1B, yet with
stricter pre-processing. Namely, we normalize usually orthographically
confused Arabic characters by converting Alif maksura to Ya, reducing all
hamzated Alif to plain Alif, and removing all non-Arabic characters. We then
only keep tweets with length $\geq 5$ words. This gives us a total of
$223,387,189$ tweets (and hence the name Twitter-250K). We then use the same
parameters as the 1B model, but we set the minimum count to 50 words (again,
hence the name Twitter-250K-MC50 where MC50 meaning minimum count of 50). We
acquire a model with a vocabulary size of $536,846$ words.

Twitter-250K-MC100. This model is identical with the Twitter-250K-MC50 model,
but uses a minimum count of 100 words when training Word2Vec. This model has a
vocabulary size of $202,690$ words.

Twitter-Seeds. We use all unigram entries in our benchmark to crawl Twitter,
using the search API. In order to avoid overfitting to our benchmark, we
randomly sample only $400$ words from it for this process. This gives us about
$\sim 3M$ tweets. We then crawl up to $3,200$ tweets from the timelines of all
$\sim 300K$ users who have posted the initially collected $3M$ tweets. The
resulting collection is at $214,161,138$ tweets after strict cleaning and
removal of all tweets of length $<5$ words. To train an embedding model on
this dataset, we use the same Word2Vec parameters as with Twitter-250K-MC50
(i.e., with a minimum count of $50$ words). This model has a vocabulary size
of $773,311$ words.

Dialect Relation K=1 K=5 K=10 ALG sing-dual None None None sing-pl 17.39 38.51
47.20 gen-pt 0.00 2.52 2.52 antonym 0.83 5.19 7.91 comp None None None male-
fem 10.00 35.00 35.00 2-5 total acc 3.87 11.33 15.03 EGY sing-dual 13.89 27.78
36.11 sing-pl 27.62 54.29 60.95 gen-pt 3.40 8.85 13.55 antonym 3.02 9.51 13.71
comp None None None male-fem 5.00 15.00 20.00 2-5 total acc 4.33 11.49 15.97
SYR sing-dual 0.00 0.00 0.00 sing-pl 7.00 30.00 43.00 gen-pt 0.46 3.72 4.80
antonym 0.40 2.81 4.81 comp 3.83 10.71 14.49 male-fem 6.89 16.70 23.17 2-5
total acc 3.14 9.39 12.97 LEB sing-dual 0.00 2.04 6.12 sing-pl 20.99 41.67
50.31 gen-pt 1.66 4.92 8.36 antonym 0.43 5.18 9.31 comp 13.14 26.72 33.62
male-fem 5.56 13.89 22.22 2-5 total acc 5.29 12.90 17.87 TUN sing-dual 0.00
0.00 0.00 sing-pl 11.44 23.20 30.39 gen-pt 0.00 4.94 11.11 antonym 0.41 4.00
7.67 comp None None None male-fem 0.00 0.00 0.00 2-5 total acc 2.48 7.68 12.14

Table 4: Evaluation across all relations for our Twitter-MC-100 model. Values
shown as “None” are for relationships where the model did not include any of
the word pairs in the question tuples in the model vocabulary. Zeros mean the
model includes the words in its vocabulary but no correct answers were
returned in the top-K.

###  4.2 Model Evaluation

We evaluate the two models from Zahran et al. (2015) and our four models
described above using DiaLex. Table 3 shows average evaluation results across
the different word relationships. For top-1 performance, one or another of our
models scores best. For top-5 results, our Twitter-250K-MC100 (Ours-MC-100)
acquires best performance for most dialects. Exceptions are EGY and TUN
dialects. Ours-MC-100 also performs best on all but TUN dialect. These results
show the necessity of developing dialectal resources for the various varieties
and that a model trained on large MSA data such as that of Zahran et al.
(2015) is quite sub-optimal.

Table 4 shows per-relation results evaluation of the Twitter-250K-MC100 model.
We show all-relations results only for this model for space limitations, and
we choose this model since it tends to perform well compared to other models.
As Table 4 shows, the model works best on the LEB dialect (17.87 accuracy for
top-10) and worst on TUN (12.14 accuracy for top-10). The table also shows
that the sing-plural relationship is the one most challenging for the model.
Clearly, the dual feature either involves (1) bigrams (in which case these
unigram models do not work and hence the “None” values) or (2) different ways
of expressing the same meaning of duality. For example, in the LEB pair
¡كوريدورين¿ - ¡كوريدور¿ “corridor-two corridors”, duality can be expressed
also by the phrase ¡2 كوريدور¿, using the digit “2” instead of the dual suffix
¡ين¿ in ¡كوريدورين¿. Overall, these results suggest that even our developed
models are neither sufficiently powerful nor large enough (even with large
vocabularies close to 1M words) to cover all the dialects. We also observe
that models need to see enough contexts (words $>$ 50 minimum count) to scale
well. This calls for the development of more robust models with wider coverage
and more frequent contexts. Our Twitter collection of 5M words can be used
towards that goal, but we opted for not exploiting it for building a word
embeddings model since this would be considered overfitting to our benchmark.

##  5 Related Work

###  5.1 Arabic Word Embeddings

The number of available Arabic word embeddings is increasing rapidly. Some of
these are strictly trained using textual corpora written in MSA, while others
were trained using dialectal data. We review the most popular embedding models
we are aware of here.

Zahran et al. (2015) built three models for Arabic word embeddings (CBOW,
SKIP-G, and GloVe). To train these models, they used a large collection of MSA
texts totaling $\sim 5.8$B words. The sources used include Arabic Wikipedia,
Arabic Gigaword Parker et al. (2009), Open Source Arabic Corpora (OSAC) Saad
and Ashour (2010), OPUS Tiedemann (2012), MultiUN Chen and Eisele (2012), and
a few others. Soliman et al. (2017) proposed AraVec a set of Arabic word
embedding models. It consist of six word embedding models built on top of
three different Arabic content domains; Wikipedia Arabic, World Wide Web
pages, and Tweets with more than $3.3$ billion word. Both CBOW and SKIP-Gram
architecture are investigated in this work.

Abdul-Mageed et al. (2018) build an SKIP-G model using $\sim 234$M tweets,
with vector dimensions at $300$. The authors, however, do not exploit their
model in downstream tasks. Abu Farha and Magdy (2019) built two word
embeddings models (CBOW and SKIP-Gram) exploiting $250$M tweets. The authors
used the models in the context of training the sentiment analysis system
Mazajak. The dimensions of each embedding vector in the Mazajak models are at
$300$.

More recently, El-Haj (2020) developed Habibi, a Multi Dialect Multi-National
Arabic Song Lyrics Corpus which comprises more than $30,000$ Arabic song
lyrics from $18$ Arab countries and six Arabic dialects for singers. Habibi
contains $500,000$ sentences (song verses) with more than $3.5$ million words.
Moreover, the authors provided a $300$ dimension CBOW word embeddings of the
corpus. Doughman et al. (2020) built a set of word embeddings learnt from
three large Lebanese news archives, which collectively consist of $609,386$
scanned newspaper images and spanning a total of 151 years, ranging from 1933
till 2011. To train the word embeddings, Optical Character Recognition (OCR)
was employed to transcribe the scanned news archives, and various archive-
level as well as decade-level word embeddings were trained. In addition,
models were also built using a mixture of Arabic and English data. For
example, Lachraf et al. (2019) presented AraEngVec an Arabic-English cross-
lingual word embedding models. To train their bilingual models, they used a
large dataset with more than $93$ million pairs of Arabic-English parallel
sentences (with more than $1.8$ billion words) mainly extracted from the Open
Parallel Corpus Project (OPUS) Tiedemann (2012). In order to train the models,
they have chosen CBOW and SKIP-Gram as an architecture. Indeed, they propose
three methods for pre-processing the opus dataset: parallel sentences, word-
level alignment and random shuffling. Both extrinsic and intrinsic evaluations
for the different AraEngVec model variants. The extrinsic evaluation assesses
the performance of models on the Arabic-English Cross-Language Semantic
Textual Similarity (CL-STS) task Nagoudi et al. (2018), while the intrinsic
evaluation is based on the Word Translation (WT) task.

Some works have also investigated the utility of using morphological knowledge
to enhance word embeddings. For example, Erdmann and Habash (2018)
demonstrated that out-of-context rule-based knowledge of morphological
structure can complement what word embeddings can learn about morphology from
words’ in-context behaviors. They quantified the value of leveraging subword
information when learning embeddings and the further value of noise reduction
techniques targeting the sparsity caused by complex morphology such as in the
Arabic language case.

El-Kishky et al. (2019) tackled the problem of root extraction from words in
the Semitic language family. They proposed a constrained sequence-to-sequence
root extraction method. Furthermore, they demonstrated how one can leverage
the root information alongside a simple slot-based morphological decomposition
to improve upon word embedding representations as evaluated through word
similarity, word analogy, and language modeling tasks. In this paper, we have
demonstrated the effectiveness of our benchmark DiaLex in the evaluation of a
chosen set of these available word embedding models and compared them to
newly-developed ones by us as we explain in the next section.

###  5.2 Word Embeddings Evaluation

There is a wealth of research on evaluating unsupervised word embeddings,
which can be broadly divided into intrinsic and extrinsic evaluations.
Intrinsic evaluations mostly rely on word analogy questions and measure the
similarity of words in the low-dimensional embedding space Mikolov et al.
(2013a); Gao et al. (2014); Schnabel et al. (2015). Extrinsic evaluations
assess the quality of the embeddings as features in models for other tasks,
such as semantic role labeling and part-of-speech tagging Collobert et al.
(2011), or noun-phrase chunking and sentiment analysis Schnabel et al. (2015).
However, all of these tasks and benchmarks are built for English and thus
cannot be used to assess the quality of Arabic word embeddings, which is the
main focus here.

To the best of our knowledge, only a handful of recent studies attempted
evaluating Arabic word embeddings. Zahran et al. (2015) translated the English
benchmark in Mikolov et al. (2013a) and used it to evaluate different
embedding techniques when applied on a large Arabic corpus. However, as the
authors themselves point out, translating an English benchmark is not the best
strategy to evaluate Arabic embeddings. Zahran et al. (2015) also consider
extrinsic evaluation on two NLP tasks, namely query expansion for Information
Retrieval and short answer grading.

Dahou et al. (2016) used the analogy questions from Zahran et al. (2015) after
correcting some Arabic spelling mistakes resulting from the translation and
after adding new analogy questions to make up for the inadequacy of the
English questions for the Arabic language. They also performed an extrinsic
evaluation using sentiment analysis. Finally, Al-Rfou et al. (2013) generated
word embeddings for 100 different languages, including Arabic, and evaluated
the embeddings using part-of-speech tagging, however the evaluation was done
only for a handful of European languages. Elrazzaz et al. (2017) built a
benchmark in MSA that can be utilized to perform intrinsic evaluation of
different word embeddings using word analogy questions. They then used the
constructed benchmark to evaluate various Arabic word embeddings. They also
performed extrinsic evaluation of these word embeddings using two NLP tasks,
namely Document Classification and Named Entity Recognition.

Salama et al. (2018) investigated enhancing Arabic word embeddings by
incorporating morphological annotations to the embeddings model. They tuned
the generated word vectors to their lemma forms using linear compositionality
to generate lemma-based embeddings. To assess the effectiveness of their
model, they used the benchmark built by Elrazzaz et al. (2017). Taylor and
Brychcín (2018b) demonstrated several ways to use morphological Arabic word
analogies to examine the representation of complex words in semantic vector
spaces. They presented a set of morphological relations, each of which can be
used to generate many word analogies.

El Bazi and Laachfoubi (2017) investigated the effect of stemming on Arabic
word representations. They applied various stemmers on different word
representations approaches, and conducted an extrinsic evaluation to assess
the quality of these word vectors by evaluating their impact on the Named
Entity Recognition task for Arabic. Taylor and Brychcín (2018a) provided a
corpus of Arabic analogies focused on the morphological constructs which can
participate in verb, noun and prepositional phrases. They conducted an
examination of ten different semantic spaces to see which of them is most
appropriate for this set of analogies, and they illustrated the use of the
corpus to examine phrase-building. Barhoumi et al. (2020) proposed intrinsic
and extrinsic methods to evaluate word embeddings for the specific task of
Arabic sentiment analysis. For intrinsic evaluation, they proposed a new
method that assesses what they define as the ”sentiment stability” in the
embedding space. For extrinsic evaluation, they relied on the performance of
the word embeddings to be evaluated for the task of sentiment analysis. They
also trained various word embeddings using different types of corpora (polar
and non-polar) and evaluated them using their proposed methods. To the best of
our knowledge, our proposed benchmark is the first benchmark developed in
various Arabic dialects and that can be used to perform intrinsic evaluation
of Arabic word embedding with respect to those dialects.

##  6 Conclusion

We described DiaLex, a benchmark for evaluating dialectal Arabic word
embeddings. DiaLex comes in five major Arabic dialects, namely Algerian,
Egyptian, Lebanese, Syrian, and Tunisian. Across these dialects, DiaLex offers
a testbank of word pairs for six syntactic and semantic relations, namely male
to female, singular to dual, singular to plural, antonym, comparative, and
genitive to past tense. To demonstrate the utility of DiaLex, we used it to
evaluate a set of available and newly-developed Arabic word embedding models.
Our evaluations are intended to showcase the utility of our new benchmark.
DiaLex as well as the newly-developed word embeddings will be publicly
available.

DiaLex can be used to support integration of dialects in the Arabic language
curriculum, and for the study of the syntax and semantics of Arabic dialects.
It can also complement evaluations of contextual word embeddings. In the
future, we plan to use DiaLex to for more extensive evaluation of all
publicly-available Arabic word embedding models. We will also train a
dialectal Arabic word embeddings model with a larger dataset and evaluate it
using DiaLex. Finally, we will translate DiaLex into MSA and English to
facilitate use of the resource for evaluating word translation.

## References

  * Abbes et al. (2020) Ines Abbes, Wajdi Zaghouani, Omaima El-Hardlo, and Faten Ashour. 2020.  Daict: A dialectal Arabic irony corpus extracted from twitter.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 6265–6271.
  * Abdul-Mageed et al. (2018) Muhammad Abdul-Mageed, Hassan Alhuzali, and Mohamed Elaraby. 2018.  You tweet what you speak: A city-level dataset of arabic dialects.  In _LREC_ , pages 3653–3659.
  * Abdul-Mageed et al. (2020a) Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020a.  ARBERT & MARBERT: Deep bidirectional transformers for Arabic.  _arXiv preprint arXiv:2101.01785_.
  * Abdul-Mageed et al. (2020b) Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020b.  Nadi 2020: The first nuanced arabic dialect identification shared task.  In _Proceedings of the Fifth Arabic Natural Language Processing Workshop_ , pages 97–110.
  * Abdul-Mageed et al. (2021) Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2021.  NADI 2021: The Second Nuanced Arabic Dialect Identification Shared Task.  In _Proceedings of the Sixth Arabic Natural Language Processing Workshop (WANLP 2021)_.
  * Abdul-Mageed et al. (2020c) Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, and Lyle Ungar. 2020c.  Micro-dialect identification in diaglossic and code-switched environments.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5855–5876.
  * Abu Farha and Magdy (2019) Ibrahim Abu Farha and Walid Magdy. 2019.  Mazajak: An online Arabic sentiment analyser.  In _Proceedings of the Fourth Arabic Natural Language Processing Workshop_ , pages 192–198, Florence, Italy. Association for Computational Linguistics.
  * Al-Batal (2017) Mahmoud Al-Batal. 2017.  _Arabic as one language: Integrating dialect in the Arabic language curriculum_.  Georgetown University Press.
  * Al-Rfou et al. (2013) Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013.  Polyglot: Distributed word representations for multilingual nlp.  _arXiv preprint arXiv:1307.1662_.
  * Aldarmaki and Diab (2019) Hanan Aldarmaki and Mona Diab. 2019.  Context-aware cross-lingual mapping.  _arXiv preprint arXiv:1903.03243_.
  * Aldarmaki et al. (2018) Hanan Aldarmaki, Mahesh Mohan, and Mona Diab. 2018.  Unsupervised word mapping using structural similarities in monolingual embeddings.  _Transactions of the Association for Computational Linguistics_ , 6:185–196.
  * Antoun et al. (2020) Wissam Antoun, Fady Baly, and Hazem Hajj. 2020.  Arabert: Transformer-based model for Arabic language understanding.  _arXiv preprint arXiv:2003.00104_.
  * Barhoumi et al. (2020) Amira Barhoumi, Nathalie Camelin, Chafik Aloulou, Yannick Estève, and Lamia Hadrich Belguith. 2020.  Toward qualitative evaluation of embeddings for arabic sentiment analysis.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 4955–4963.
  * Bouamor et al. (2019) Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019.  The madar shared task on arabic fine-grained dialect identification.  In _Proceedings of the Fourth Arabic Natural Language Processing Workshop (WANLP19), Florence, Italy_.
  * Chen and Eisele (2012) Yu Chen and Andreas Eisele. 2012.  Multiun v2: Un documents with multilingual alignments.  In _LREC_ , pages 2500–2504.
  * Collobert et al. (2011) Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011.  Natural language processing (almost) from scratch.  _The Journal of Machine Learning Research_ , 12:2493–2537.
  * Dahou et al. (2016) Abdelghani Dahou, Shengwu Xiong, Junwei Zhou, Mohamed Houcine Haddoud, and Pengfei Duan. 2016.  Word embeddings and convolutional neural network for arabic sentiment classification.  In _International Conference on Computational Linguistics_ , pages 2418–2427.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Doughman et al. (2020) Jad Doughman, Fatima Abu Salem, and Shady Elbassuoni. 2020.  Time-aware word embeddings for three lebanese news archives.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 4717–4725.
  * El Bazi and Laachfoubi (2017) Ismail El Bazi and Nabil Laachfoubi. 2017.  Is stemming beneficial for learning better arabic word representations?  In _First International Conference on Real Time Intelligent Systems_ , pages 508–517. Springer.
  * El-Haj (2020) Mahmoud El-Haj. 2020.  Habibi - a multi dialect multi national Arabic song lyrics corpus.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 1318–1326, Marseille, France. European Language Resources Association.
  * El-Kishky et al. (2019) Ahmed El-Kishky, Xingyu Fu, Aseel Addawood, Nahil Sobh, Clare Voss, and Jiawei Han. 2019.  Constrained sequence-to-sequence semitic root extraction for enriching word embeddings.  In _Proceedings of the Fourth Arabic Natural Language Processing Workshop_ , pages 88–96.
  * Elrazzaz et al. (2017) Mohammed Elrazzaz, Shady Elbassuoni, Khaled Shaban, and Chadi Helwe. 2017.  Methodical evaluation of arabic word embeddings.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 454–458.
  * Erdmann and Habash (2018) Alexander Erdmann and Nizar Habash. 2018.  Complementary strategies for low resourced morphological modeling.  In _Proceedings of the Fifteenth Workshop on Computational Research in Phonetics, Phonology, and Morphology_ , pages 54–65.
  * Gao et al. (2014) Bin Gao, Jiang Bian, and Tie-Yan Liu. 2014.  Wordrep: A benchmark for research on learning word representations.  _arXiv preprint arXiv:1407.1640_.
  * Lachraf et al. (2019) Raki Lachraf, El Moatez Billah Nagoudi, Youcef Ayachi, Ahmed Abdelali, and Didier Schwab. 2019.  ArbEngVec : Arabic-English cross-lingual word embedding model.  In _Proceedings of the Fourth Arabic Natural Language Processing Workshop_ , pages 40–48, Florence, Italy. Association for Computational Linguistics.
  * Lan et al. (2020a) Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020a.  An empirical study of pre-trained transformers for arabic information extraction.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4727–4734.
  * Lan et al. (2020b) Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020b.  Gigabert: Zero-shot transfer learning from english to arabic.  In _Proceedings of The 2020 Conference on Empirical Methods on Natural Language Processing (EMNLP)_.
  * Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_.
  * Mikolov et al. (2013b) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b.  Distributed representations of words and phrases and their compositionality.  In _Advances in neural information processing systems_ , pages 3111–3119.
  * Mnih and Kavukcuoglu (2013) Andriy Mnih and Koray Kavukcuoglu. 2013.  Learning word embeddings efficiently with noise-contrastive estimation.  In _Advances in Neural Information Processing Systems_ , pages 2265–2273.
  * Mubarak et al. (2020) Hamdy Mubarak, Shimaa Amer, Ahmed Abdelali, and Kareem Darwish. 2020.  Arabic curriculum analysis.  In _Proceedings of the 28th International Conference on Computational Linguistics: System Demonstrations_ , pages 80–86.
  * Nagoudi et al. (2018) El Moatez Billah Nagoudi, Jérémy Ferrero, Didier Schwab, and Hadda Cherroun. 2018.  Word embedding-based approaches for measuring semantic similarity of arabic-english sentences.  In _Arabic Language Processing: From Theory to Practice_ , pages 19–33, Cham. Springer International Publishing.
  * Parker et al. (2009) Robert Parker, David Graff, Ke Chen, Junbo Kong, and Kazuaki Maeda. 2009.  Arabic gigaword.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global Vectors for Word Representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , volume 1, pages 2227–2237.
  * Saad and Ashour (2010) Motaz K Saad and Wesam Ashour. 2010.  Osac: Open source arabic corpora.  In _6th ArchEng Int. Symposiums, EEECS_ , volume 10.
  * Salama et al. (2018) Rana Aref Salama, Abdou Youssef, and Aly Fahmy. 2018.  Morphological word embedding for arabic.  _Procedia computer science_ , 142:83–93.
  * Schnabel et al. (2015) Tobias Schnabel, Igor Labutov, David M Mimno, and Thorsten Joachims. 2015.  Evaluation methods for unsupervised word embeddings.  In _EMNLP_ , pages 298–307.
  * Soliman et al. (2017) Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017.  Aravec: A set of arabic word embedding models for use in arabic nlp.  _Procedia Computer Science_ , 117:256–265.
  * Taylor and Brychcín (2018a) Stephen Taylor and Tomáš Brychcín. 2018a.  Arabic word analogies and semantics of simple phrases.  In _2018 2nd International Conference on Natural Language and Speech Processing (ICNLSP)_ , pages 1–6. IEEE.
  * Taylor and Brychcín (2018b) Stephen Taylor and Tomáš Brychcín. 2018b.  The representation of some phrases in arabic word semantic vector spaces.  _Open Computer Science_ , 8(1):182–193.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in opus.  In _LREC_ , volume 2012, pages 2214–2218.
  * Zahran et al. (2015) Mohamed A Zahran, Ahmed Magooda, Ashraf Y Mahgoub, Hazem Raafat, Mohsen Rashwan, and Amir Atyia. 2015.  Word representations in vector space and their applications for arabic.  In _International Conference on Intelligent Text Processing and Computational Linguistics_ , pages 430–443. Springer.