Zero-Inflated Poisson Cluster-Weighted Models: Properties and Applications

Kehinde Olobatuyi

Department of Statistics and Mathematics Finance,

University of Milan-Bicocca, Milan, Italy.

<EMAIL_ADDRESS>
###### Abstract

In this paper, I propose a new class of Zero-Inflated Poisson models into the
family of Cluster Weighted Models (CWMs) called Zero-Inflated Poisson CWMs
(ZIPCWM). ZIPCWM extends Poisson cluster weighted models and other mixture
models. I propose an Expectation-Maximization (EM) algorithm via an
iteratively reweighted least squares for the model. I theoretically and
analytically investigate the identifiability of the proposed model through an
extensive simulation study. Parameter recovery, classification assessment, and
performance of different information criteria are investigated through broad
simulation design. ZIPCWM is applied to real data which accounts for excess
zeros of over $40\%$. We explore the classification performance of ZIPCWM,
Fixed Zero-inflated Poisson mixture model (FZIP), and Poisson cluster weighted
model (PCWM) on the data. Based on the confusion matrix, ZIPCWM achieves
$97.4\%$ classification power, PCWM achieves $67.30\%$, while FZIP has the
worst classification performance. In conclusion, ZIPCWM outperforms both PCWM
and FZIP models.

keywords: Cluster-weighted model, Class Imbalance, Zero-inflated Poisson
regression models, Poisson regression models, Fixed Zero-inflated Poisson
model

##  1 Introduction

Finite mixture of Poisson regression models with constant weights parameters
have been developed by [Weldel et al., 1993]; [Brannas and Rosenqvist, 1994];
[Wang et al., 1996]; and [Alfo and Trovato, 2004]. [Wang et al., 1998]
incorporated covariates in the weight parameters of finite mixture Poisson
regression models, treating the covariates as the concomitant variables. To
account for different covariates with count response variables, [Ingrassia et
al., 2015] proposed a generalized linear mixed cluster-weighted model where
the response variable is allowed to follow the Poison distribution. As an
alternative to Poisson regression model for handling overdispersion in data, a
Negative binomial (NB) regression model can be used. The count variable of
interest may contain more zeros than expected under a Poisson model, which is
observed in many applications. Inflated zeros can cause instability in a
predictive model. Class imbalance also can be a consequence of zero inflation
in a response variable we wish to predict therefore causing irregularities in
the prediction accuracy and result in overfitting. Zero-Inflated Poisson (ZIP)
regression model has been proposed with an application to defects in
manufacturing ([Lambert, 1992]). The ZIP distribution is a mixture of a
Poisson distribution and a degenerate at zero. This regression setting allows
for the covariates in both Poisson mean and weight parameter. Furthermore, in
a situation where overdispersion takes precedence, a zero-inflated negative
binomial (ZINB) regression model can a better fit. However, if a population
has excess zeros and several sub-populations in non-zero counts, a single
component of the ZINB regression model may fail to capture the excess or
sufficient to describe the non-zero counts.

In this paper, we wish to address the problem of class imbalance by proposing
a zero-inflated Poisson cluster weighted model that is capable of handling
zero-inflation in data. We also wish to investigate further the effect of
mixed covariates in the zero-inflated Poisson cluster weighted models. On this
proposed model, we study the Expectation Maximization algorithm via
iteratively re-weighted least squares (IRLS). At the E step, we follow the
optimization conventionally. While at the M step, we used IRLS for the Poisson
distribution. We compared the proposed model to the two existing models such
as Poisson CWM ([Ingrassia et al., 2015]) and Zero-inflated Poisson regression
model ([Hwa et al., 2014]). The paper is organized as follows. We describe
briefly mixed continuous and categorical variables in Section (2). Section (3)
gives the details of the proposed model. We discuss the identifiability
condition for the proposed model in Section (4). In Section (5). We derived
the EM algorithm via Iteratively rewieghted least squares for the proposed
model. An extensive simulation study is carried out in section (6). Section
(7) demonstrates real data application of the model. Finally we conclude with
a discussion in Section (8).

##  2 Mixed Continuous and Categorical Variables

We consider now the problem of fitting a family of cluster weighted model

$p(\vec{x},y)=\sum_{g=1}^{G}=\pi_{g}p(y|\vec{x},\mathcal{D}_{g})p(\vec{x}|\mathcal{D}_{g})$
(1)

to some pair of data $(\mathbf{X}^{\prime},Y)^{\prime}$ where, $\mathbf{X}$ is
a matrix of covariates, and $Y$ is a random variable defined on some space
$\mathcal{D}$ where some of the feature variables are categorical. Basically,
we can assume that the categorical variables are independent of each other and
of the continuous variables which can be taken to have any continuous
distribution such as the multivariate normal distribution. This idea of mixed
covariates has come into prominence more recently due to its appearance in the
graphical modeling of mixed variables known as the conditional Gaussian
distribution model ([Whittaker, 1990]; [Cox and Wermuth, 1992]). The location
model has been used for fitting the mixture models to mixed categorical and
continuous variables, ([Jorgensen and Hunt, 1996]; [Lawrence and Krzanowski,
1996]; and [Lawrence and Krzanowski, 1999]).

##  3 The Zero-Inflated Poisson CWM

Suppose that $\mathbf{X}$ can be decomposed as
$(\mathbf{Q}^{\prime},\mathbf{W})^{\prime}$, where $\mathbf{Q}$ is a
$q$-variate vector of continuous covariates and $\mathbf{W}$ is a $p$-variate
vector of categorical variable respectively, being $d=q+p$. In this case,
$\mathcal{X}=\mathbf{R}^{p\times\\{1,...,r_{1}\\},\times,...,\times\\{1,...,r_{p}\\}}$.
With the location model, the $\mathbf{W}$ categorical variables are replaced
by a single multinomial random variable $\mathbf{W}_{i}$ with $p$ cells. Let
$Y$ be a count response variable, then the probability of observing zeros is
$\pi_{1}$, and the probability of observing Poisson cluster weighted model is
$1-\pi_{1}$. The ZIP cluster weighted model is given as follows:

$p(\mathbf{x},y;\mathbf{\Theta})=\pi_{1}I_{(y=0)}+\sum_{g=2}^{G}p\big{(}y\big{|}\mathbf{x};\vec{\zeta}_{g}\big{)}p\big{(}\mathbf{x};\vec{\psi}_{g}\big{)}\pi_{g},$
(2)

where Equation (2) is decomposed into categorical and continuous covariates as
follows;

$=\pi_{1}I_{(y=0)}+\sum_{g=2}^{G}p\big{(}y\big{|}\mathbf{x};\vec{\zeta}_{g}\big{)}p\big{(}\mathbf{q};\vec{\psi}^{*}_{g}\big{)}p\big{(}\mathbf{w};\vec{\psi}^{**}_{g}\big{)}\pi_{g},$
(3)

since $y=0$, and its mean $\mu_{i1}(\mathbf{x};\vec{\beta}_{1})=0$, then
$p(y|\mathbf{x};\vec{\zeta}_{1})=1$. If we assume that the response variable
contains many zeroes, then $p(y|\mathbf{x};\zeta_{g})$ is modeled as a Zero-
Inflated Poisson regression, $p(\mathbf{w}|\vec{\psi}^{**}_{g})$ follows a
multinomial distribution and $p(\mathbf{q}|\vec{\psi}^{*}_{g})$ follows a
Gaussian distribution, where $G$ is the number of mixing components, $\pi_{g}$
is the mixing weight of component $g$ such that $0<\pi_{g}<1$, $g=1,...,G$ and
$\sum_{g=1}^{G}\pi_{g}=1$ by the decomposition
$\pi_{1}=1-\sum_{g=2}^{G}\pi_{g}$. The weight $\pi_{1}$ determines the
proportion of excess zeros compared with an ordinary Poisson mixture model
determined by $1-\pi_{1}$. Assuming that $Y$ takes values in $\mathcal{Y}$ and
that the conditional density $Y|\mathbf{x},\mathcal{D}_{g}$ is Poisson with
parameter $\mu_{ig}(\mathbf{x};\vec{\beta}_{g})$; that is,
$Y|\mathbf{x},\mathcal{D}_{g}\sim P[\mu_{ig}(\mathbf{x};\vec{\beta}_{g})]$. We
allow the mean of the response variable to depend on the covariates
$\mu_{ig}(\mathbf{x};\mathbf{\beta}_{g})$ using the following regression
models that

$\mu_{ig}(\mathbf{x};\vec{\beta}_{g})=\exp(\mathbf{x}_{i}^{\prime}\vec{\beta}_{g}),\hskip
7.22743pti=1,...,N,g=2,...,G$ (4)

In this case,

$p(y|\mathbf{x};\vec{\beta}_{g})=\exp[-\mu_{ig}(\mathbf{x};\vec{\beta}_{g})]\frac{[-\mu_{ig}(\mathbf{x};\vec{\beta}_{g})]^{y}}{y!}$
(5)

The mixing weights $\\{\pi_{g}\\}_{g=1}^{G}$ can be treated as the multinomial
logit of $\pi_{g}$ to be linear function of covariates which is commonly known
in the literature as a concomitant variable.

$\pi_{ig}(\mathbf{v}_{i},\gamma)=\frac{\exp(\mathbf{v}_{i}\gamma_{g})}{\sum_{g=1}^{G}\exp(\mathbf{v}_{i}\gamma_{g})}$
(6)

In this case, $\mathbf{x}_{i}=(x_{i1},...,x_{ip})$ and
$\mathbf{v}_{i}=(v_{i1},...,v_{ir})$ are $1\times p$ and $1\times q$ rows
vectors of covariates (including an intercept), respectively. They can be the
same or have nothing in common. The regression coefficient for the $g$th
component are $\beta_{g}$ and $\gamma_{g}$ which are vectors of $p\times 1$
and $q\times 1$ respectively. We note that the mixing probability of the first
component $\pi_{i1}(\mathbf{v}_{i},\gamma)$ is taken to be the baseline for
the multinomial logistic model. To handle response variable with excess zeros,
we assume that the density of $p(y|\mathbf{x};\vec{\eta}_{g})$ belong to
exponential family. It is a common practice that the exponential family is
strictly related to the generalized linear models with a monotone and
differential link function $f(.)$ that makes the expected value $\mu_{g}$, of
$Y|\mathcal{D}_{g}$ depend on the covariate $\mathbf{X}$ through the linear
combination $f(\mu_{g})=\mathbf{x}^{\prime}\vec{\beta}_{g}$. Note here that we
have not used $\vec{\beta}_{g}$ explicitly, so it contains the coefficient and
$\mathbf{X}$ will have extra column of ones. The distribution of
$Y|\mathcal{D}_{g}$ is denoted by $p(y|\mathbf{x};\vec{\beta}_{g})$. The term
$p(\mathbf{q};\vec{\psi}^{*}_{g})$ in Equation (3) is modeled as a $q$-variate
Gaussian density with mean $\vec{\mu}_{g}$ and covariance matrix
$\vec{\Sigma}_{g}$, i.e.,
$p(\mathbf{q}|\vec{\psi}^{*}_{g})=\phi(\mathbf{q};\vec{\mu}_{g},\vec{\Sigma}_{g})$.
With respect to the term $p(\mathbf{w};\vec{\psi}^{**}_{g})$ in Equation (3),
we assume that each categorical covariate can be taken to be a binary vector
$\mathbf{w}^{k}=(w^{k1},...,w^{kr_{k}})^{\prime}$, where $w^{ks}=1$ if $r_{k}$
is equal to the value $s$, with $s\in\\{1,...,r_{k}\\}$, and $w^{ks}=0$
otherwise. Furthermore, assume that $q$ categorical covariates are independent
of each other. Then,

$p(w;\vec{\alpha}_{g})=\prod_{k=1}^{p}\prod_{s=1}^{r_{k}}(\alpha_{gks})^{w^{ks}},$
(7)

where $g=1,...,G$,
$\vec{\alpha}_{g}=(\vec{\alpha}_{g1}^{\prime},...,\vec{\alpha}^{\prime}_{gp})^{\prime}$
and $\vec{\alpha}_{gk}=(\alpha_{gk1},...,\alpha_{gkr_{p}})^{\prime}$. We take
the density of $p(\mathbf{w};\vec{\alpha}_{g})$ in Equation (7) to be a
multinomial distribution of parameters $\vec{\alpha}_{gk}$, where $k=1,...,p$
and $\sum_{s=1}^{r_{k}}\alpha_{gks}=1$ with constraint $\alpha_{gks}>0$. The
ZIPCWM over all observation can be formulated as follows:

$p(\vec{x},y;\mathbf{\Theta})=\pi_{1}I_{(y=0)}+\sum_{g=2}^{G}Pois\big{(}y\big{|}\vec{x};\vec{\beta}_{g}\big{)}\phi\big{(}\vec{q};\vec{\mu}_{g},\vec{\Sigma}_{g}\big{)}p\big{(}\vec{w};\vec{\alpha}_{g}\big{)}\pi_{g}$
(8)

where $I_{(.)}$ is an indicator function that outputs $1$ when the specified
condition is satisfied and $0$ otherwise, and Pois(.) denotes the probability
mass function of $y_{i}$ and $x_{i}$ with a mean of
$\mu_{ig}(\mathbf{x}_{i};\vec{\beta}_{g})$.

###  3.1 Related Mixture Model

This section highlights the special cases of the proposed model. We show a
list of special cases of ZIPCWM below;

####  3.1.1 Poisson CWM [Ingrassia et al., 2015]

Let $p(\mathbf{x},y;\mathbf{\Theta})$ in Equation (8) be a ZIPCWM. Assuming
that $Y$ takes values $\mathrm{y}=\mathcal{N}$, $Y|\mathbf{x};\mathcal{D}_{g}$
is Poisson regression model with parameter
$\mu_{g}(\mathbf{x};\vec{\beta}_{g})$ and the $\mathbf{X}|\mathcal{D}_{g}$
with parameter $(\vec{\mu}_{g},\vec{\Sigma}_{g})$. If there is no account of
excess zeros, then $Y|\mathbf{x},\mathcal{D}_{g}$ is modeled with Poisson
regression and the $X|\mathcal{D}_{g}$ is taken to be the Gaussian
distribution, then ZIPCWM reduces to Poisson CWM as follows;

$p(\mathbf{x},y;\mathbf{\Theta})=\sum_{g=1}^{G}Pois\big{(}y\big{|}\mathbf{x};\vec{\beta}_{g}\big{)}\phi\big{(}\mathbf{q};\vec{\mu}_{g},\vec{\Sigma}_{g}\big{)}p\big{(}\mathbf{w};\vec{\alpha}_{g}\big{)}\pi_{g}$
(9)

####  3.1.2 ZIP Regression mixture model [Hwa et al., 2014]

If $p(\mathbf{x})=1$ in Equation (2), then ZIPCWM reduces to GZIP mixture
distribution if the mixing weight depends on a concomitant variables and FZIP
if the mixing weight is fixed. Generalized ZIP model:

$p(y;\mathbf{\Theta})=\pi_{i1}(\mathbf{v}_{i},\gamma)I_{(y=0)}+\sum_{g=2}^{G}Pois\big{(}y\big{|}\mathbf{x};\vec{\beta}_{g}\big{)}\pi_{ig}(\mathbf{v}_{i},\gamma)$
(10)

and the fixed ZIP model is as follows:

$p(y;\mathbf{\Theta})=\pi_{1}I_{(y=0)}+\sum_{g=2}^{G}Pois\big{(}y\big{|}\mathbf{x};\vec{\beta}_{g}\big{)}\pi_{g}$
(11)

####  3.1.3 ZIP distribution [Lambert, 1992]:

If $p(\mathbf{x})=1$ and $G=2$ in Equation (2), then ZIPCWM reduces to ZIP
distribution.

####  3.1.4 Standard Poisson mixture model:

If both $\pi_{g}$ and $\mu_{g}(\mathbf{x}_{i},\beta_{g})$ are constant
functions and $p(\mathbf{x})=1$, ZIPCWM reduces to the standard Poisson
mixture model, denoted by

$p(Y=y_{i})=\sum_{g=1}^{G}\pi_{g}Pois(y_{i}|\lambda_{g})$ (12)

where the constancy of $\mu_{ig}(\mathbf{x}_{i},\beta_{g})$ is taken to be
$\lambda_{g}$. It should be noted that we have used $\lambda_{g}$ to preserve
the original parameter notation of the standard Poisson mixture model. In the
following, we present the derivative of an estimation method based on the EM
algorithm for the ZIPCWM regression model described in Equation (3).

##  4 Identifiability

To reliably estimate the parameters of Equation (8), we require that the
ZIPCWM be identifiable, that is, two sets of parameters which do not agree
after permutation cannot produce the same mixture distribution. [Teicher,
1961] proves that the class of finite mixtures of Poisson distribution is
identifiable without covariates. Similarly, [Follmann and Lambert, 1991] give
sufficient conditions for identifiability, and [Wang et al., 1996] extend the
definition of identifiability of finite Poisson mixtures with covariates. We
provide the condition for identifiability of ZIPCWM as follows; Consider the
collection of probability models
$\\{p(\mathbf{x}_{1},y_{1},\vec{\Theta}),...,p(\mathbf{x}_{N},y_{N};\vec{\Theta})\\}$,
with a restriction that $\pi_{1}<...<\pi_{g}$, sample space
$\mathcal{Y}_{1},...,\mathcal{Y}_{N}$, parameter space $\vec{\Theta}$, and
fixed covariates vectors $x_{1},...,x_{N}$ that is decomposed to categorical
$w$ and continuous variables $q$, where $w_{i}\in\mathcal{R}^{p}$ and
$q_{i}\in\mathcal{R}^{d}$ for $i=1,...,N$. The collection of probability model
is identifiable if for $(\vec{\Omega},\vec{\pi},\mathbf{k},\mathbf{h})$,
$(\vec{\Omega}^{*},\vec{\pi}^{*},\mathbf{k}^{*},\mathbf{h}^{*})$
$\in\vec{\Theta}$,

$p(\mathbf{x},y,\vec{\Omega},\vec{\pi},\mathbf{k},\mathbf{h})=p(\mathbf{x},y,\vec{\Omega}^{*},\vec{\pi}^{*},\mathbf{k}^{*},\mathbf{h}^{*})$
(13)

for all $y_{i}\in\mathcal{Y}_{i},i=1,...,N$, implies that
$(\vec{\Omega},\vec{\pi},\mathbf{k},\mathbf{h})=(\vec{\Omega}^{*},\vec{\pi}^{*},\mathbf{k}^{*},\mathbf{h}^{*})$.

We note here that the restriction on the mixing probability is a sufficient
condition for label switching problems and it means that two models are
equivalent if they agree up to permutation of parameters.

We now provide a sufficient condition for identifiability. Suppose that
$(\vec{\Omega},\vec{\pi},\mathbf{k},\mathbf{h})$,
$(\vec{\Omega}^{*},\vec{\pi}^{*},\mathbf{k}^{*},\vec{h}^{*})$ satisfy Equation
13. It then implies that

$\pi_{1}I_{(y=0)}+\sum_{g=2}^{G}\text{Pois}\big{(}y\big{|}\mathbf{x};\vec{\beta}_{g}\big{)}\phi\big{(}\mathbf{q};\vec{\mu}_{g},\vec{\Sigma}_{g}\big{)}p\big{(}\mathbf{w};\vec{\alpha}_{g}\big{)}\pi_{g}=$
$\pi_{1}^{*}I_{(y=0)}+\sum_{c=2}^{C}\text{Pois}\big{(}y\big{|}\mathbf{x};\vec{\beta}_{c}^{*}\big{)}\phi\big{(}\mathbf{q};\vec{\mu}_{c}^{*},\vec{\Sigma}_{c}^{*}\big{)}p\big{(}\mathbf{w};\vec{\alpha}_{c}^{*}\big{)}\pi_{c}^{*}$
(14)

for each $i$ and $y_{i}\in\mathcal{Y}_{i},i=1,...,N$. Teicher’s and Hennig’s
results imply that

$G=C,\hskip 3.61371pt\pi_{g}=\pi^{*}_{c},\hskip
3.61371pt\vec{\beta}_{g}=\vec{\beta}_{c}^{*},\hskip
3.61371pt\vec{\mu}_{g}^{*}=\vec{\mu}^{*}_{c},$
$\vec{\Sigma}_{g}=\vec{\Sigma}^{*}_{c},\hskip 3.61371pt\text{and}\hskip
3.61371pt\vec{\alpha}_{g}=\vec{\alpha}_{c}^{*}$

for $i=1,...,N$ $g=1,...,G$, and $c=1,...,C$. By definition of ZIPCWM, we
obtain

$\exp(\mathbf{x}_{i}^{\prime}\vec{\beta}_{g})=\exp(\mathbf{x}_{i}^{\prime}\vec{\beta}^{*}_{c}),$
(15)

Equation 15 means

$(\vec{\beta}_{g}-\vec{\beta}^{*}_{c})^{\prime}\mathbf{x}_{i}=0,\hskip
3.61371pt\text{for}\hskip 3.61371pti=1,...,N.$ (16)

Then, we say

$\text{Pois}\big{(}y\big{|}\mathbf{x};\vec{\beta}_{g}\big{)}=\text{Pois}\big{(}y\big{|}\mathbf{x};\vec{\beta}_{c}^{*}\big{)},\phi\big{(}\mathbf{q};\vec{\mu}_{g},\vec{\Sigma}_{g}\big{)}=\phi\big{(}\mathbf{q};\vec{\mu}_{c}^{*},\vec{\Sigma}_{c}^{*}\big{)},$
$p\big{(}\mathbf{w};\vec{\alpha}_{g}\big{)}=p\big{(}\mathbf{w};\vec{\alpha}_{c}^{*}\big{)},\pi_{g}=\pi_{c}^{*}.$
(17)

Hence a sufficient condition for identifiability is that $\mathbf{X}$ is a
full rank matrix, where $\mathbf{X}={\mathbf{x}_{1},...,\mathbf{x}_{N}}$.

##  5 Model Estimation by EM Algorithm

Let
$(\mathbf{x}^{\prime}_{i},y_{i}),...,(\mathbf{x}^{\prime}_{n},y_{n})^{\prime}$
be a sample of $n$ independent observation pairs drawn from model Equation
(8). The corresponding likelihood, for a fixed number of component $G$ is
given by

$L(\mathbf{\Theta})=\prod_{i=1}^{n}p(\mathbf{x}_{i},y_{i};\mathbf{\Theta})=\prod_{i=1}^{n}\bigg{[}\pi_{1}I_{(y=0)}+\sum_{g=2}^{G}Pois\big{(}y\big{|}\mathbf{x};\vec{\beta}_{g}\big{)}\phi\big{(}\mathbf{q};\vec{\mu}_{g},\vec{\Sigma}_{g}\big{)}p\big{(}\mathbf{w};\vec{\alpha}_{g}\big{)}\pi_{g}\bigg{]}$
(18)

Here, we assume the number of $G$ is fixed and known a priori and
$\mathbf{z}_{i}=(z_{i1},...,z_{iG})^{\prime}$ be the latent vector of
component indicator variables, where $z_{ig}=1$ if ith subject
$(\mathbf{x}^{\prime}_{i},y_{i})^{\prime}$ comes from $\mathcal{D}_{g}$
belongs to the $gth$ latent group and $z_{ig}=0$ otherwise. By assumption,
each belongs to one of the $G$ unobservable components therefore, considered
missing or incomplete. Using a multinomial distribution for the unobserved
vector $\mathbf{z}_{i}$,where
$\\{(\mathbf{x}^{\prime}_{i},y_{i},\mathbf{z}^{\prime}_{i})^{\prime};i=1,...,n\\}$
is th complete data. The the complete-data log-likelihood can be written as

$L_{\mathbf{c}}(\mathbf{\Theta})=\prod_{i=1}^{n}\bigg{[}\pi_{1}I_{(y=0)}+\prod_{g=2}^{G}Pois\big{(}y\big{|}\mathbf{x};\vec{\beta}_{g}\big{)}\phi\big{(}\mathbf{q};\vec{\mu}_{g},\vec{\Sigma}_{g}\big{)}p\big{(}\mathbf{w};\vec{\alpha}_{g}\big{)}\pi_{g}\bigg{]}^{z_{ig}}$
(19)

The corresponding complete-data log-likelihood by taking the logarithm of
Equation (18) can be written as

$l_{\mathbf{c}}(\mathbf{\Theta}|y,\vec{\omega},\mathbf{x})=\sum_{i=1}^{N}z_{i1}\ln\pi_{1}I_{y_{i}=0}+\sum_{i=1}^{N}\sum_{g=2}^{G}z_{ig}\bigg{[}\ln\pi_{g}+\ln
Pois\big{(}y_{i}|\mu_{ig}(\mathbf{x}_{i},\vec{\beta}_{g})\big{)}$
$+\ln\phi\big{(}\mathbf{q}_{i}|\vec{\mu}_{g},\vec{\Sigma}_{g}\big{)}+\ln
p\big{(}\mathbf{w}_{i};\vec{\alpha}_{g}\big{)}\bigg{]}$ (20)

where $\pi_{1}=1-\sum_{g=2}^{G}\pi_{g}$ and
$\mathbf{\Theta}=(\vec{\mu}_{g},\vec{\Sigma_{g}},\vec{\beta}_{g})$ for
$g=2,...G$ is the set of model parameters to be estimated.

###  5.1 E-step

Using the current estimates $\mathbf{\Theta}^{(t)}$, we compute the
probability $z^{(t)}_{ig}$ that the subject $i$ comes from $gth$ component of
the mixture:

$E(z_{ig}|y_{i},\vec{\omega}_{i},\mathbf{x}_{i},\mathbf{\Theta}^{(t)})={z}^{(t)}_{ig}=\frac{\pi_{g}Pois\bigg{(}y_{i}|\mu_{ig}(\mathbf{x}_{i},\vec{\beta}_{g}^{(t)})\bigg{)}\phi\bigg{(}\mathbf{x}_{i}|\vec{\mu}_{g}^{(t)},\vec{\Sigma}_{g}^{(t)}\bigg{)}}{p\bigg{(}\mathbf{x}_{i},y_{i};\vec{\Theta}^{(t)}\bigg{)}}$
(21)

which correspond to the posterior probability that the unlabeled observation
$(\mathbf{x}^{\prime}_{i},y_{i})^{\prime}$ belong to the $g$th component of
the mixture, using the current fit $\mathbf{\Theta}^{(t)}$ for
$\mathbf{\Theta}$ and ${z}^{(t)}_{i1}=1-\sum_{g=2}^{G}{z}^{(t)}_{ig}.$

###  5.2 M-step

At this step, we obtain the $Q(.)$ with respect to $\mathbf{\Theta}^{(t+1)}$
where $t=0,1,\ldots$, The conditional expectation of $l_{c}(\mathbf{\Theta})$
given the observed data , say $Q(\mathbf{\Theta};\mathbf{\Theta}^{(t)})$ is
maximized with respect to $\mathbf{\Theta}$. The $z_{ig}$ are simply replaced
by the current expectations $z_{ig}^{(t)}$ yielding,

$Q\bigg{(}\mathbf{\Theta};\mathbf{\Theta}^{(t)}\bigg{)}=\sum_{i=1}^{N}z^{(t)}_{i1}\ln\pi_{1}I_{y_{i}=0}+\sum_{i=1}^{N}\sum_{g=2}^{G}z^{(t)}_{ig}\ln\pi_{g}+\sum_{i=1}^{N}\sum_{g=2}^{G}z^{(t)}_{ig}\ln
p\big{(}\mathbf{w}_{i};\vec{\alpha}_{g}\big{)}$
$+\sum_{i=1}^{N}\sum_{g=2}^{G}z^{(t)}_{ig}\ln
Pois\big{(}y_{i}|\mu_{ig}(\mathbf{x}_{i},\vec{\beta}_{g})\big{)}+\sum_{i=1}^{N}\sum_{g=2}^{G}z^{(t)}_{ig}\ln\bigg{(}\phi\big{(}\vec{q}_{i}|\vec{\mu}_{g},\vec{\Sigma}_{g}\big{)}\bigg{)}$
(22)

####  5.2.1 Parameter Estimation of $Y$

We maximize Equation (22) independently of the $G$ expression as the four
terms on the right of equation 22 have zero cross-derivatives, i.e.

$\frac{\partial}{\partial\vec{\beta}_{g}}\sum_{i=1}^{N}z^{(t)}_{ig}\ln\bigg{(}Pois\big{(}y_{i}|\mu_{ig}(\mathbf{x}_{i},\vec{\beta}_{g})\big{)}\bigg{)}$
(23)
$\frac{\partial}{\partial\vec{\beta}_{g}}\sum_{i=1}^{n}z^{(t)}_{ig}\log\frac{\bigg{(}\mu_{ig}(\mathbf{x}_{i};\vec{\beta}_{g})\bigg{)}^{y_{i}}\exp\bigg{(}-\mu_{ig}(\mathbf{x}_{i};\vec{\beta}_{g})\bigg{)}}{y_{i}!}$
(24)
$\frac{\partial}{\partial\vec{\beta}_{g}}\sum_{i=1}^{n}z_{ig}^{(t)}\bigg{[}y_{i}\log\mu_{ig}(\mathbf{x}_{i};\vec{\beta}_{g})-\mu_{ig}(\mathbf{x}_{i};\vec{\beta}_{g})-\log
y_{i}!\bigg{]}$ (25)
$\frac{\partial}{\partial\vec{\beta}_{g}}\mu_{ig}(\mathbf{x}_{i};\vec{\beta}_{g})=\mathbf{x}_{i}\exp(\mathbf{x}_{i}^{\prime}\vec{\beta}_{g})=\mathbf{x}_{i}\mu_{ig}(\mathbf{x}_{i};\vec{\beta}_{g})$
(26)
$\frac{\partial}{\partial\vec{\beta}_{g}}\sum_{i=1}^{n}z_{ig}^{(t)}\bigg{[}y_{i}\log\mu_{ig}(\mathbf{x}_{i};\vec{\beta}_{g})-\mu_{ig}(\mathbf{x}_{i};\vec{\beta}_{g})-\log
y_{i}!\bigg{]}$ (27)

From Equation (25), we have the following

$\sum_{i=1}^{n}z_{ig}^{(t)}\mathbf{x}_{i}\mu_{ig}\big{(}\mathbf{x}_{i};\vec{\beta}_{g}\big{)}\bigg{(}\frac{y_{i}}{\mu_{ig}(\mathbf{x}_{i};\vec{\beta}_{g})}-\frac{\mu_{ig}(\mathbf{x}_{i};\vec{\beta}_{g})}{\mu_{ig}(\mathbf{x}_{i};\vec{\beta}_{g})}\bigg{)}=0$
(28)
$S(\vec{\beta}^{(t)}_{g})=\sum_{i=1}^{n}z_{ig}^{(t)}\mathbf{x}_{i}\bigg{(}y_{i}-\mu_{ig}(\mathbf{x}_{i};\vec{\beta}_{g})\bigg{)}$
(29)

Using the expression
$\vec{\beta}_{g}^{(t+1)}=\vec{\beta}_{g}^{(t)}+\big{[}I(\vec{\beta}_{g}^{(t)})\big{]}^{-1}S(\vec{\beta}_{g}^{(t)})$,
where $I(\vec{\beta}_{g}^{(t)})$ is the Fisher Information matrix and
$S(\vec{\beta}_{g}^{(t)})$ is the score function obtained in Equation (29).
So,

$I(\vec{\beta}_{g}^{(t)})=-\frac{\partial}{\partial}S(\vec{\beta}_{g}^{(t)})=\sum_{i=1}^{n}z^{(t)}_{ig}\mu_{ig}(\mathbf{x}_{i};\vec{\beta}_{g})\mathbf{x}^{\prime}_{i}\mathbf{x}_{i}$
(30)
$\vec{\beta}_{g}^{(t+1)}=\vec{\beta}_{g}^{(t)}+\bigg{(}\sum_{i=1}^{n}z^{(t)}_{ig}\mu_{ig}\big{(}\mathbf{x}_{i};\vec{\beta}_{g}\big{)}\mathbf{x}^{\prime}_{i}\mathbf{x}_{i}\bigg{)}^{-1}\bigg{(}\sum_{i=1}^{n}z_{ig}^{(t)}\mathbf{x}_{i}\big{(}y_{i}-\mu_{ig}(\mathbf{x}_{i};\vec{\beta}_{g})\big{)}\bigg{)}$
(31)

finally Equation (31) becomes

$\vec{\beta}_{g}^{(t+1)}=\bigg{(}\sum_{i=1}^{n}\mathbf{x}^{\prime}_{i}s_{ig}\mathbf{x}_{i}\bigg{)}^{-1}\bigg{(}\sum_{i=1}^{n}\mathbf{x}_{i}^{\prime}s_{ig}\delta_{ig}^{(t)}\bigg{)}$
(32)

where $\delta^{(t)}=\mathbf{x}^{\prime}_{i}\vec{\beta}_{g}+\delta_{ig}^{*}$
with
$\delta_{ig}^{*}=\big{(}y_{i}-\mu_{ig}(\mathbf{x}_{i};\vec{\beta}_{g})\big{)}\big{/}\mu_{ig}(\mathbf{x}_{i};\vec{\beta}_{g})$
and $s_{ig}=z^{(t)}_{ig}\mu^{(t)}_{ig}(\mathbf{x}_{i};\vec{\beta}_{g})$

####  5.2.2 Parameter Estimation for Mixing Weights

The maximum of $Q(\mathbf{\Theta})$ with respect to $\vec{\pi}$, subject to
the constraints on these parameters. is achieved by maximizing the augmented
function

$=\sum_{i=1}^{N}z^{(t)}_{i1}\ln\pi_{1}-\gamma_{1}\biggl{(}\pi_{1}-1\biggr{)}I_{y_{i}=0}+\sum_{i=1}^{N}\sum_{g=2}^{G}z^{(t)}_{ig}\ln\pi_{g}+\gamma\biggl{(}\sum_{g=2}^{G}\pi_{g}-1\biggr{)}$
(33)

where $\gamma_{1}$ is a Lagrangian multiplier. Setting the derivative in
Equation (33) with respect to $\pi_{1}$ for a degenerate distribution and
$\pi_{g}$ for a Poisson cluster weighted models equal to zero and solving
yields

$\pi_{1}^{(t+1)}=\sum_{i=1}^{n}z^{(t+1)}_{i1}\bigg{/}n$
$\pi_{g}^{(t+1)}=\sum_{i=1}^{n}z^{(t+1)}_{ig}\bigg{/}n$ (34)

where $g=2,\ldots,G$.

####  5.2.3 Parameter Related to q

Maximizing Equation (22) with respect to $\vec{\mu}_{g}$ and
$\vec{\Sigma}_{g}$, $g=1,...,G$, is equivalent to maximizing independently
each of the $G$ expression

$\sum_{i=1}^{n}z^{(t)}_{ig}\ln\bigg{(}\phi\big{(}\mathbf{q}_{i}|\vec{\mu}_{g},\vec{\Sigma}_{g}\big{)}\bigg{)}$
(35)

we obtain

$\vec{\mu}_{g}^{(t+1)}=\sum_{i=1}^{n}z^{(t)}_{(ig)}\mathbf{q}_{i}\bigg{/}\sum_{i=1}^{n}z^{(t)}_{(ig)}$
(36)

and

$\vec{\Sigma}_{g}^{(t+1)}=\sum_{i=1}^{n}z_{ig}^{(t)}\big{(}\mathbf{q}_{i}-\vec{\mu}_{g}^{(t+1)}\big{)}\big{(}\mathbf{q}_{i}-\vec{\mu}_{g}^{(t+1)}\big{)}^{\prime}\Bigg{/}\sum_{i=1}^{n}z_{ig}^{(t)}$
(37)

####  5.2.4 Parameters Related to W

Maximizing $Q(\mathbf{\Theta};\mathbf{\Theta}^{(t)})$ with respect to
$\vec{\alpha}_{g}$, $g=1,\ldots,G$ is equivalent to maximizing each of the $G$
expression considering the constraints

$\sum_{i=1}^{n}z_{ig}^{(t)}\ln
p(\mathbf{w}_{i};\vec{\alpha}_{g})=\sum_{k=1}^{p}\sum_{i=1}^{n}z_{ig}^{(t)}\sum_{s=1}^{r_{k}}w^{ks}\ln\alpha_{gks}.$
(38)

Using the Lagrangian multiplier, Equation (38) can be expanded as follows;

$\sum_{i=1}^{n}z_{ig}^{(t)}\sum_{s=1}^{r_{k}}w^{ks}\ln\alpha_{gks}-\gamma_{2}\bigg{(}\sum_{s=1}^{r_{k}}\alpha_{gks}-1\bigg{)},$
(39)

the $\gamma_{2}$ is used here as the Lagrangian multiplier. Setting the
derivative of Equation (39) to zero

$\sum_{i=1}^{n}z_{ig}^{(t)}\frac{w^{ks}}{\alpha_{gks}}-\gamma_{2}=0,\implies\sum_{i=1}^{n}z_{ig}^{(t)}w^{ks}\bigg{/}\gamma_{2}=\alpha_{gks},$
(40)

we find $\gamma_{2}=\sum_{i=1}^{n}z_{ig}^{(t)}$, then finally we have

$\alpha_{gks}^{(t+1)}=\sum_{i=1}^{n}z_{ig}^{(t)}w^{ks}\bigg{/}\sum_{i=1}^{n}z_{ig}^{(t)},$
(41)

Figure 1: The Visualization of the simulated data with sample size $500$

Figure 2: The Visualization of the simulated data with sample size $1000$

##  6 Simulation Study

A simulation study was performed to evaluate the performance of the maximum
likelihood estimates of the model obtained via EM algorithm. We generated
samples of size $n$ from the following ZIPCWM with three components.

$p(\mathbf{x},y;\mathbf{\Theta})=\pi_{1}I_{(y=0)}+\sum_{g=2}^{3}Pois\big{(}y\big{|}\mathbf{x};\vec{\beta}_{g}\big{)}\phi\big{(}\mathbf{q};\vec{\mu}_{g},\vec{\Sigma}_{g}\big{)}p\big{(}\mathbf{w};\vec{\alpha}_{g}\big{)}\pi_{g}$
(42)

The log-link for the Poisson mean $\mu_{ik}$ is as follows:

$\log(\mu_{i2}(\mathbf{x}_{i},\beta_{2}))=\beta_{20}+\beta_{21}\mathbf{x}_{i}$
$\log(\mu_{i3}(\mathbf{x}_{i},\beta_{3}))=\beta_{30}+\beta_{31}\mathbf{x}_{i}$
(43)

where $\mathbf{x}_{i}$ in this case is the combination of
$\mathbf{q}_{i},\text{and}\hskip 4.33601pt\mathbf{w}_{i}$ random variables.
Figure (2) and Figure (2) present the visualization of the simulated data with
sample sizes $500$ and $1000$ but we intentionally not include visualization
for $n=200$. We considered the distribution
$\mathbf{q}|\mathcal{D}_{g}\in\mathcal{R}^{d}$ to follow a Gaussian
distribution. Table (1) shows the true values of the mean vectors, and the
covariance matrices are the spherical covariance matrices. For the finite
discrete variables $w_{1}$ and $w_{2}$, we generated two different variables
from binomial distribution where $p=2$ discrete covariates and $r_{1}=2$ and
$r_{2}=3$ levels with the probability of success $0.5$ and $1/3$ respectively.
The true values of the ZIPCWM regression coefficients are presented in Table
(2). We selected the $\pi_{1}\approx 0.5$, $\pi_{2}\approx 0.3$, and
$\pi_{3}\approx 0.2$ according to [Hwa et al., 2014]. We generated a random
number $U$ from Uniform distribution. To generate samples from the above model
for each subject $(i=1,...,n)$, we adopted the following conditions; if $U$ is
less than $\pi_{1}$, $Y_{i}$ takes the value $0$, where $\mu_{i1}=0$. If $U$
is between $\pi_{1}$ and $\pi_{1}+\pi_{2}$, then $Y_{i}$ is a draw from
Pois($\mu_{i2}$). Otherwise, $Y_{i}$ is generated from Pois$(\mu_{i3})$.

###  6.1 Result for Parameters Estimated

Here, the ZIPCWM with three component is fitted to the simulated data of sizes
$n=200,500$, and $1000$. We compute the misclassification based on the
simulated data. We ran each simulation $10$ times as suggested by[McLachlan
and Peel, 2000] to avoid local maxima. However, we ran the simulated $10$
times and recorded the average values to avoid the biased choice of selecting
the best result. Table (1) shows the recovered estimates of the
$\vec{\mu}_{g}$, the diagonal values for $\vec{\Sigma}_{g}$, and the mixing
proportions $\vec{\pi_{g}}$. For estimated parameters of $\mu_{1}$ in
component $1$ for $n=200,500$, and $n=1000$, the values are omitted.
Generally, in Table (1), we observe that the closeness of the parameter values
is not independent of the sample size $n$. Also, in Table (2), considering the
coefficients $\vec{\beta}_{g}$ where $g=2,\ldots,G$. The parameter estimates
for $n=1000$ are closer to the true parameter values compared to the parameter
estimates for $n=200$ and $n=500$. We investigate the ability of the proposed
model to identify the number of

Table 1: True and Recovered values of the mean, sigma, and mixing weight
for $n=200,500$ and $1000$ | $n$ | $g$ | $\mu_{1}$ | $\mu_{2}$ | $\mu_{3}$ | $\sigma_{11}$ | $\sigma_{22}$ | $\sigma_{33}$ | $\vec{\pi}$ |
---|---|---|---|---|---|---|---|---|---|---
True |  | $1$ | - | $-$ | $-$ | $-$ | - | - | $0.50$ |
|  | $2$ | $0.10$ | $2.00$ | $1.00$ | $1.00$ | $1.00$ | $1.00$ | $0.30$ |
|  | $3$ | $-2.00$ | $0.00$ | $3.00$ | $1.00$ | $1.00$ | $1.00$ | $0.20$ |
Recovered | $200$ | $1$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $0.54$ |
|  | $2$ | $-0.77$ | $1.50$ | $2.20$ | $1.99$ | $1.62$ | $1.48$ | $0.25$ |
|  | $3$ | $-1.07$ | $0.92$ | $2.40$ | $2.14$ | $1.57$ | $1.86$ | $0.21$ |
| $500$ | $1$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $0.53$ |
|  | $2$ | $-0.50$ | $1.52$ | $2.03$ | $1.72$ | $1.39$ | $1.25$ | $0.25$ |
|  | $3$ | $-1.44$ | $0.62$ | $2.79$ | $1.45$ | $1.30$ | $0.99$ | $0.22$ |
| $1000$ | $1$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $0.53$ |
|  | $2$ | $0.17$ | $2.08$ | $1.17$ | $0.70$ | $0.81$ | $1.11$ | $0.27$ |
|  | $3$ | $-1.93$ | $-0.07$ | $3.09$ | $1.00$ | $1.05$ | $1.21$ | $0.20$ |

Table 2: True and Recovered values of the regression coefficients for $n=200,500,1000$ Values | $n$ | $g$ | $\beta_{0}$ | $\beta_{1}$ | $\beta_{2}$ | $\beta_{3}$ | $\beta_{4}$ | $\beta_{5}$ |
---|---|---|---|---|---|---|---|---|---
True |  | $2$ | $0.00$ | $0.88$ | $0.28$ | $0.96$ | $0.09$ | $0.33$ |
|  | $3$ | $0.00$ | $0.77$ | $0.53$ | $0.98$ | $0.07$ | $0.37$ |
Recovered | $200$ | $2$ | 0.33 | 0.73 | 0.26 | 0.87 | 0.07 | 0.39 |
|  | $3$ | 0.30 | 0.71 | 0.29 | 0.84 | 0.18 | 0.41 |
| $500$ | $2$ | 0.25 | 0.82 | 0.29 | 0.92 | 0.09 | 0.32 |
|  | $3$ | 0.24 | 0.75 | 0.42 | 0.89 | 0.086 | 0.38 |
| $1000$ | $2$ | $0.02$ | $0.92$ | $0.27$ | $0.86$ | $0.09$ | $0.39$ |
|  | $3$ | $0.51$ | $0.81$ | $0.49$ | $0.86$ | $0.04$ | $0.41$ |

Table 3: The values of AIC, BIC, ICL of ZIPCWM for different $n=200,500,1000$ with true component $G=3$. For $n=200$, all the selection criteria perform poorly. However, the performance gets improved with increased sample size. n | G | AIC | BIC | ICL | AWE | AIC3 | AICc | AICu | Caic |
---|---|---|---|---|---|---|---|---|---|---
$200$ | $2$ | $2192.91$ | $2080.77$ | $2080.76$ | $1798.62$ | $2158.91$ | $2178.48$ | $2140.01$ | $2046.77$ |
| $3$ | $2055.82$ | $1831.49$ | $1822.53$ | $1267.24$ | $1987.82$ | $1984.18$ | $1899.56$ | $1763.53$ |
| $4$ | $1945.12$ | $1608.69$ | $1598.41$ | $762.26$ | $1843.12$ | $1728.50$ | $1583.78$ | $1506.69$ |
| $5$ | $\vec{1771.85}$ | $\vec{1323.28}$ | $\vec{1315.37}$ | $\vec{194.70}$ | $\vec{1635.85}$ | $\vec{1180.35}$ | $\vec{949.32}$ | $\vec{1187.28}$ |
$500$ | $2$ | $7557.49$ | $7414.20$ | $7413.55$ | $7100.90$ | $7523.49$ | $7552.37$ | $7516.09$ | $7380.20$ |
| $3$ | $\vec{5128.59}$ | $4841.99$ | $4826.93$ | $4215.40$ | $\vec{5060.59}$ | $\vec{5106.82}$ | $5032.57$ | $4773.99$ |
| $4$ | $5212.08$ | $4782.19$ | $4750.51$ | $3842.30$ | $5110.08$ | $5159.16$ | $5043.82$ | $4680.19$ |
| $5$ | $5270.85$ | $\vec{4697.66}$ | $\vec{4649.80}$ | $\vec{3444.47}$ | $5134.85$ | $5168.19$ | $\vec{5008.09}$ | $\vec{4561.66}$ |
$1000$ | $2$ | $12074.24$ | $11907.38$ | $11907.37$ | $11570.51$ | $12040.24$ | $12071.78$ | $12036.15$ | $11873.38$ |
| $3$ | $\vec{10234.37}$ | $\vec{9863.09}$ | $\vec{9900.64}$ | $9226.91$ | $\vec{10166.37}$ | $\vec{10224.29}$ | $\vec{10152.79}$ | $\vec{9832.64}$ |
| $4$ | $10860.90$ | $10360.31$ | $10268.41$ | $9349.72$ | $10758.90$ | $10837.48$ | $10728.78$ | $10258.31$ |
| $5$ | $10907.28$ | $10239.83$ | $10127.11$ | $\vec{8892.37}$ | $10771.28$ | $10864.10$ | $10716.76$ | $10103.83$ |

Figure 3: The Visualization of Confusion Matrix and Statistics of the ZIPCWM
with $n=200$, $G=3$.

Figure 4: The Visualization of Confusion Matrix and Statistics of the ZIPCWM
with $n=500$, $G=3$.

Figure 5: The Visualization of Confusion Matrix and Statistics of the ZIPCWM
with $n=1000$, $G=3$.

Figure 6: The Visualization of Confusion Matrix and Statistics of the FZIP
regression mixture model with $n=1000$, $G=3$.

components using eight different model selection criteria such as Akaike’s
information criterion (AIC) and Bayesian information criterion (BIC) model
selection criteria, AWE, AICc, AIC3, AICu, Caic. The values presented are the
average values of ten runs for each of the criterion. In Table (3), the effect
of sample size is significantly evident. It can be seen that all the selection
criteria select wrong number of components with small size. They all select
the model with too many components (clusters) when the sample size is $n=200$.
However, when the sample size increases to $500$, AIC, AIC3, and AIC3
correctly select the true cluster component. Moreover, when the sample size is
$100$, all the selection criteria except AWE pick the right choice of the
model component. AWE displays a poor performance of overestimating the number
of components in the data. All criteria except the AWE perform satisfactorily
with higher sample size. Obviously, the performance of the criteria roughly
gets improved with increasing sample size $n$. Table (4) presents the
misclassification rates by counting the number of observation classified into
a different components to the original components. The overall
misclassification rate is $5.5\%$ for $n=200$, $8.4\%$ for $n=500$, and
$6.7\%$ for $n=1000$, We also compare the result with the fixed zero-inflated
Poisson mixture model with the misclassification rate of $13.1\%$. The
classification power of fixed zero-inflated Poisson mixture model agrees with
the simulated study provided in [Hwa et al., 2014]. PCWM achieves a slight
higher classification accuracy of $89.8\%$ compared yo FZIP and
misclassification rate of $10.2\%$. To validate the classification power, we
report the ARI value for each model, which measures the agreement between the
true cluster and classification result of the proposed model. The ARI of the
ZIPCWM is $0.892$ when $n=1000$ and FZIP has ARI of $0.847$. This means the
predicted result of ZIPCWM agrees more with the true classification than the
FZIP. The AUC for both sample sizes are $0.923$, while the AUC of FZIP model
is $0.829$.

Table 4: Confusion Matrix in the three component Model for $n=200,500,1000$. For $1000$, ZIPCWM, PCWM, and FZIP are compared together. It was observed that ZIPCWM outperforms PCWM and FZIP. $G$ | Real Component |  |  | $1$ |  | $2$ |  | $3$ |  |  | Misclass. rate (%) |  |
---|---|---|---|---|---|---|---|---|---|---|---|---|---
$200$ | $1$ |  |  | $102$ |  | $0$ |  | $0$ |  |  | $0.00$ |  |
| $2$ |  |  | $6$ |  | $42$ |  | $0$ |  |  | $12.50$ |  |
| $3$ |  |  | $0$ |  | $5$ |  | $45$ |  |  | $10.00$ |  |
Misclassification |  |  | $5.50$ |  |  |  |  |  |  |  |  |  |
Accuracy |  |  | $94.50\%$ |  |  |  |  |  |  |  |  |  |
$500$ | $1$ |  |  | $241$ |  | $0$ |  | $0$ |  |  | $0.00$ |  |
| $2$ |  |  | $24$ |  | $121$ |  | $0$ |  |  | $16.55$ |  |
| $3$ |  |  | $0$ |  | $18$ |  | $96$ |  |  | $15.79$ |  |
rate |  |  | $8.40$ |  |  |  |  |  |  |  |  |  |
Accuracy |  |  | $91.60\%$ |  |  |  |  |  |  |  |  |  |
$1000$ | $1$ |  |  | $485$ |  | $0$ |  | $0$ |  |  | $0.00$ |  |
| $2$ |  |  | $45$ |  | $253$ |  | $0$ |  |  | $15.10$ |  |
| $3$ |  |  | $0$ |  | $22$ |  | $195$ |  |  | $10.14$ |  |
Misclassification |  |  | $6.70$ |  |  |  |  |  |  |  |  |  |
Accuracy |  |  | $93.30\%$ |  |  |  |  |  |  |  |  |  |
PCWM | $1$ |  |  | $485$ |  | $0$ |  | $0$ |  |  | $0.00$ |  |
| $2$ |  |  | $15$ |  | $196$ |  | $87$ |  |  | $34.23$ |  |
| $3$ |  |  | $0$ |  | $0$ |  | $217$ |  |  | $0.00$ |  |
Overall |  |  | $10.20$ |  |  |  |  |  |  |  |  |  |
ccuracy |  |  | $89.80\%$ |  |  |  |  |  |  |  |  |  |
FZIP | $1$ |  |  | $485$ |  | $0$ |  | $0$ |  |  | $0.00$ |  |
| $2$ |  |  | $45$ |  | $167$ |  | $86$ |  |  | $43.96$ |  |
| $3$ |  |  | $0$ |  | $0$ |  | $217$ |  |  | $0.00$ |  |
Overall |  |  | $13.10$ |  |  |  |  |  |  |  |  |  |
ccuracy |  |  | $86.90\%$ |  |  |  |  |  |  |  |  |  |

##  7 Modeling of ZIPCWM on Real Data

###  7.1 The Use of Contraceptive Among married women

This dataset is a subset of the $1987$ National Indonesia Contraceptive
Prevalence Survey. The samples are married women who were either not pregnant
or do not know if they were at the time of interview [Dua and Graff, 2017].
The problem is to cluster the current contraceptive method choices (no use,
long-term methods, or short-term methods) of a woman based on her demographic
and socio-economic characteristics. The demographic and socio-economic
attributes are as follows: Wife’s age (numerical), Wife’s education
(categorical) 1=low, 2, 3, 4=high, Husband’s education (categorical) 1=low, 2,
3, 4=high, Number of children ever born (numerical), Wife’s religion (binary)
0=Non-Islam, 1=Islam, Wife’s now working? (binary) 0=Yes, 1=No, Husband’s
occupation (categorical) 1, 2, 3, 4, Standard-of-living index (categorical)
1=low, 2, 3, 4=high, Media exposure (binary) 0=Good, 1=Not good, Contraceptive
method used (class attribute) 1=No-use, 2=Long-term, 3=Short-term. The values
of the method used range between $0$ and $3$. Approximately, $43\%$ of the
sample values are equal to zero, which correspond to no use of contraceptive
method.

First, we carried out a dispersion test to ascertain the absence of
overdispersion in the data. The dispersion parameter is $0.83983$ which
signifies the absence of overdispersion. Furthermore, the Poisson regression
model was carried out to check the significant power of the variables in the
data. The result shows that the wife’s age has a significant effect on the use
of contraception among married women. This is reasonable since the estimate
$\beta_{\text{age}}<0$ which is $-0.04375$. This means the younger married
women are $0.09572$ times more expected to use contraception than their older
married counterparts. The wife’s education also contributes significantly to
the use of contraception with a p-value of $<2e-16$. From the

Figure 7: The Visualization of Confusion Matrix and Statistics of the ZIPCWM
that achieves $97\%$ classification accuracy

Figure 8: The Visualization of Confusion Matrix and Statistics of the PCWM
that achieves about $59\%$ classification accuracy.

Table 5: Comparison of classification performance of ZIPCWM and PCWM with classification accuracy of $97\%$, and $58.5\%$ respectively. FZIP performs so poorly on the data as the model could not identify the reasonable component for the data. Model | Real Component |  |  | $1$ |  | $2$ |  | $3$ |  |  | Misclassification rate (%) |
---|---|---|---|---|---|---|---|---|---|---|---|---
ZIPCWM | $1$ |  |  | $629$ |  | $0$ |  | $0$ |  |  | $0.00$ |
| $2$ |  |  | $0$ |  | $294$ |  | $39$ |  |  | $11.71$ |
| $3$ |  |  | $0$ |  | $0$ |  | $511$ |  |  | $0.00$ |
Misclassification |  |  | $2.60$ |  |  |  |  |  |  |  |  |
Accuracy |  |  | $97.40\%$ |  |  |  |  |  |  |  |  |
PCWM | $1$ |  |  | $351$ |  | $95$ |  | $183$ |  |  | $44.20$ |
| $2$ |  |  | $0$ |  | $0$ |  | $333$ |  |  | $100.00$ |
| $3$ |  |  | $0$ |  | $0$ |  | $511$ |  |  | $0.00$ |
Misclassification |  |  | $41.50$ |  |  |  |  |  |  |  |  |
Accuracy |  |  | $58.50\%$ |  |  |  |  |  |  |  |  |

result, we deduce that well educated married women have $1.18$ more chance of
using conception. The rest of the significant variables considered in the
study are as follows; Number of children ever born, Standard-of-living index
and Media exposure. We performed a comparison study of three different models
such as Zero-Inflated Poisson Cluster-Weighted Models (ZIPCWM), Poisson
Cluster-weighted Models (PCWM, [Ingrassia et al., 2015]), and Fixed Zero-
Inflated Poisson Mixture models (FZIP, [Hwa et al., 2014]).

Here, the ZIPCWM, PCWM, and FZIP models with three components are fitted to
the method of contraceptive among the married women based on their
demographics and they are used to classify the data into three components. We
computed the misclassification rate by counting the number of the incorrectly
classified observation. Visualization results are presented in Figure (8) and
Figure (8). In Table (5), ZIPCWM has the overall misclassification rate is
$2.6\%$. Moreover, it can be seen that most of the misclassifications are in
component two which has $11.71\%$. In group one the number of married women
according to the prediction of the model is $629$. The component two according
to ZIPCWM has $294$ married women correctly classified as the married women
that have long-term use of contraceptive. $39$ observations of the women that
have long-term use were misclassified into cluster three (short-term use).
ZIPCWM correctly predicts that $511$ married women have a short-term use of
contraceptive. We observe that PCWM has a high overall misclassification rate
of $41.50\%$. Moreover, all the component two was totally misclassified as
component three. This is a problem of label switching. FZIP provides the worst
classification. The model could not identify the components at all but
classified all the women as coming from component three.

Table 6: The significant student related variables Attribute |  |  |  | Description |
---|---|---|---|---|---
school |  |  |  | student’s school (binary: Gabriel Pereira or Mousinho da Silveira) |
sex |  |  |  | student’s sex (binary: female or male) |
age |  |  |  | student’s age (numeric: from 15 to 22) |
Pstatus |  |  |  | parent’s cohabitation status (binary: urban or rural) |
Medu |  |  |  | mother’s education (numeric: 0 to 4) |
Fedu |  |  |  | father’s education (numeric: 0 to 4) |
Mjob |  |  |  | mother’s job (nominal) |
Fjob |  |  |  | father’s job (nominal) |
guardian |  |  |  | student’s guardian (nominal: ”mother”, ”father” or ”other”) |
studytime |  |  |  | weekly study time (numeric) |
failures |  |  |  | number of past class failures (numeric: $n$ if 1 $\leq n<3$, else 4) |
famsize |  |  |  | family size (binary: $\leq 3$ or $>3$) |
schoolsup |  |  |  | extra educational support (binary: yes or no) |
famsup |  |  |  | family educational support (binary: yes or no) |
paidclass |  |  |  | extra paid classes(binary: yes or no) |
higher |  |  |  | wants to take higher education? (binary: yes or no) |
romantic |  |  |  | with a romantic relationship (binary: yes or no) |
famrel |  |  |  | quality of family relationships (numeric: from 1 - very bad to 5 - excellent) |
freetime |  |  |  | free time after school (numeric: from 1 - very low to 5 - very high) |
Dalc |  |  |  | workday alcohol consumption (numeric: from 1 - very low to 5 - very high) |
Walc |  |  |  | weekend alcohol consumption (numeric: from 1 - very low to 5 - very high) |
health |  |  |  | current health status (numeric: from 1 - very bad to 5 - very good) |
absences |  |  |  | number of school absences (numeric: from 0 to 93) |

###  7.2 Modeling the Number of Absence of students

The data consists of the performance of students in Portugal. The secondary
education consists of $3$ years of schooling, preceding 9 years of basic
education followed by the higher education. Due to some factors, most of the
students join the public and free education system. The data was collected
primarily with questionnaires. The structure of the education system in
Portugal is a 20-point grading scale where 0 is the lowest and 20 is the
perfect grade. The data was collected during the 2005-2006 school year from
two public schools by [Cortez and Silva, 2008]. They designed the latter with
closed questions related to demographic, social/emotional ([Pritchard and
Wilson, 2003]) and school related questions. The goal of the study is to
classify the students according to the number of absence in the class. First
of, we identify the significant factors that contribute to the absence. We
modeled the data with a Poisson regression to investigate the significant
effect of the variables on the presence of the students in class. Table (6)
shows the significant variables that contributed to the absence/presence of
the students in class. It is interesting to know how some factors can strongly
contribute to or negatively affect the success of the students such as parent
cohabitation status. The response variable is taken to be a Poisson count. We
also note the presence of class imbalance in the data i.e. one class makes up
about $63\%$ of the response variable which makes it suitable for Zero-
inflated model. The response variable is a number of event occurring (absence)
in the class. The number of absence ranges from $0$ to $93$ where $0$
represents no absence and other numbers represent the number of absence. The
response variable can also be seen as a binary variable of presence and
absence. However, we performed a model selection test using the combination of
eight model selection criteria with ZIPCWM and the result is presented below.
Table (7) confirms our intuition about the response variable. We observe that
all the model selection criteria agree with the selection of the model with
two components. This shows us that there are two categories of students in the
class viz; those students that are regular in class and those that are
irregular whether short-term or long-term.

Figure 9: The Visualization of Confusion Matrix and Statistics of the ZIPCWM
that achieves $100\%$ classification accuracy.

Figure 10: The Visualization of Confusion Matrix and Statistics of the PCWM
that achieves about $90\%$ classification accuracy.

Table 7: The values of model selection criteria of ZIPCWM for different $G=2,3,4$ and $5$. Before proceeding with the clustering analysis, we carried out a model selection test. All the selection criteria used suggest the model with two components. G | AIC | BIC | ICL | AWE | AIC3 | AICc | AICu | Caic |
---|---|---|---|---|---|---|---|---|---
$2$ | $\vec{6505.81}$ | $\vec{6420.77}$ | $\vec{6419.84}$ | $\vec{6240.74}$ | $\vec{6486.81}$ | $\vec{6504.60}$ | $\vec{6484.28}$ | $\vec{6401.77}$ |
$3$ | $7784.37$ | $7614.30$ | $7481.70$ | $7254.24$ | $7746.37$ | $7779.51$ | $7739.29$ | $7576.30$ |
$4$ | $8059.74$ | $7804.64$ | $7633.21$ | $7264.54$ | $8002.74$ | $8048.55$ | $7987.79$ | $7747.64$ |
$5$ | $7515.60$ | $7175.47$ | $7062.70$ | $6455.34$ | $7439.60$ | $7495.14$ | $7413.18$ | $7099.49$ |

Table 8: Comparison of classification power of the ZIPCWM and PCWM Models. We intentionally omitted FZIP because we are interested in zero-inflated model against ordinary model. ZIPCWM provides higher classification accuracy relative to PCWM. Model | Real Component |  |  | $1$ |  |  |  | $2$ |  |  | Misclassification rate (%) |
---|---|---|---|---|---|---|---|---|---|---|---|---
ZIPCWM | $1$ |  |  | $244$ |  |  |  | $0$ |  |  | $0.00$ |
| $2$ |  |  | $0$ |  |  |  | $405$ |  |  | $0.00$ |
Misclassification |  |  | $0.00$ |  |  |  |  |  |  |  |  |
Accuracy |  |  | $100.00\%$ |  |  |  |  |  |  |  |  |
PCWM | $1$ |  |  | $242$ |  |  |  | $2$ |  |  | $0.82$ |
| $2$ |  |  | $66$ |  |  |  | $339$ |  |  | $16.30$ |
Misclassification |  |  | $10.48$ |  |  |  |  |  |  |  |  |
Accuracy |  |  | $89.52\%$ |  |  |  |  |  |  |  |  |

After confirming the component of the model, we compare the classification
power of ZIPCWM and PCWM models to distinguish between the zero-inflation
model and ordinary model. We observe that the ZIPCWM outperforms PCWM. Table
(8) shows the misclassification rate of both models. ZIPCWM has the zero
misclassification rate and achieves $100\%$ classification accuracy while the
PCWM has $89.52\%$ classification accuracy. Also, Figure (10) and Figure (10)
present the visualization of the confusion matrix with the classification
accuracy. Thus, we conclude that ZIPCWM has a high classification power
compared to PCWM. We measure the agreement between actual class and predicted
class using Adjusted Rand Index (ARI). ZIPCWM has ARI of $1$ while PCWM has
ARI of $0.81$.

##  8 Concluding Remarks

In this paper, we have introduced the Zero-Inflated Poisson cluster-weighted
model for class imbalance in data commonly known as censored information in
medical data. Zero-Inflated Poisson cluster weighted model is a generalized
form of the previously existing models such as Poisson cluster weighted model,
Fixed Zero-Inflated Poisson mixture model. ZIPCWM allows modeling the data
with mixed-type covariates which is the combination of the finite discrete and
continuous variables. This provides an advantage over the limitations of GZIP
and FZIP. Furthermore, we have extensively described an Expectation-
Maximization algorithm for parameter estimation. To investigate the parameter
recovery of the algorithm and the performance of various model selection
criteria to select the number of mixture components, explicit simulation
studies were carried out. Our simulation studies show that the proposed model
work satisfactorily and the estimation technique performs well. The results
show that the ZIPCWM with three components provides the best fit. These
results lend support to the use of cluster weighted model and establish that
ZIPCWM provides better fitting performance than the PCWM and FZIP when
explaining zero-inflated heterogeneous data. Additionally, the proposed model
was applied to the real data and a comparison study of the classification
performance with its special models was investigated. It was observed that the
proposed model outperformed its special models. In this work, we used eight
different model selection criteria to select the number of mixing components,
$G$.

## References

  * [Alfo and Trovato, 2004] Alfo, M. and Trovato, G. (2004).  Semiparametric mixture models for multivariate count data, with application.  Econmics Journal, 7(2):426–454.
  * [Brannas and Rosenqvist, 1994] Brannas, K. and Rosenqvist, G. (1994).  Semiparametric estimation of heterogeneous count data models.  European Journal of Operating Research, (76):247–258.
  * [Cortez and Silva, 2008] Cortez, P. and Silva, A. (2008).  Using data mining to predict secondary school student performance.  In A. Brito and J. Teixeira Eds., Proceedings of 5th FUture BUsiness TEChnology Conference (FUBUTEC 2008), (5):5–12.
  * [Cox and Wermuth, 1992] Cox, D. and Wermuth, N. (1992).  Response models for mixed binary and quintitative variables.  Biometrika, 79:441–461.
  * [Dua and Graff, 2017] Dua, Dheer, u. and Graff, C. (2017).  Uci, machine leaning repository.  University of California, Irvine, School of Information and Computer Sciences.
  * [Follmann and Lambert, 1991] Follmann, D. and Lambert, D. (1991).  Identifiability of finite mixtures of logistics regression models.  Journal of Statistical Planning and Inference, (27):375–381.
  * [Hwa et al., 2014] Hwa, K. L., Wai, K., and Philip, L. (2014).  Zero-inflated poisson regression mixture model.  Computational Statistics and Data Analysis., (71):151–158.
  * [Ingrassia et al., 2015] Ingrassia, S., Punzo, A., Vittadini, G., and Minotti, S. C. (2015).  The generalized linear mixed cluster-weighted model.  Journal of Classification, 32(1):85 – 113.
  * [Jorgensen and Hunt, 1996] Jorgensen, M. and Hunt, L. (1996).  Mixture modelling clustering of data sets with categorical and continuous variables.  In ISIS: Information, Statistics and induction in Science, D.L. Dowe, K.B. Korb, and J.J. Oliver (Eds.). Singapore: World Scientific Publishing,, pages 375–384.
  * [Lambert, 1992] Lambert, D. (1992).  Zero-inflated poisson regression, with an application to defects in manufacturing.  Technometrics, (34):1–14.
  * [Lawrence and Krzanowski, 1996] Lawrence, C. and Krzanowski, W. (1996).  Mixture separation for mixed-mode data.  Statistics and Computing, 6:85–92.
  * [Lawrence and Krzanowski, 1999] Lawrence, C. and Krzanowski, W. (1999).  Fitting a mixture model to three-mode three-way data with categorical and continuous variables.  Journal of Classification, 16:283–296.
  * [McLachlan and Peel, 2000] McLachlan, G. and Peel, D. (2000).  Finite mixture models.  Wiley Series In Probability and Statistics Applied Probability and Statistics Section,Wiley, New York., 1(1):1–438.
  * [Pritchard and Wilson, 2003] Pritchard, M. and Wilson, S. (2003).  Using emotional and social factors to predict student success.  College Student Development, 44, (1):18–28.
  * [Teicher, 1961] Teicher, H. (1961).  Identifiability of mixtures.  Annals of Mathematical Statistics., (34):244–248.
  * [Wang et al., 1998] Wang, P., Cockburn, I., and Putterman, M. (1998).  Analysis of patent data: a mixed poisson regression model approach.  journal Business Econometric Statistics, 16(1):27–41.
  * [Wang et al., 1996] Wang, P., Putterman, M., Cockburn, I., and Le, N. (1996).  Mixed poisson regression models with covariate dependent rates.  Biometrics, 52(2):381–400.
  * [Weldel et al., 1993] Weldel, M., DeSarbo, W., Bult, J., and Ramaswamy, V. (1993).  A latent class poisson regression model for heterogeneous count data.  Journal of Application Econometrics, (8):397–411.
  * [Whittaker, 1990] Whittaker, J. (1990).  Graphical models in applied multivariate analysis.  New York: Wiley., 55:1–17.


there are shared norms regarding the tone and style of discourse in these
communities. Both subreddits also demonstrate moderate preferences for
politeness and formality, indicating a mutual appreciation for respectful and
well-mannered discussions.

In contrast, r/libertarian stands out with a strong preference for supportive,
formal, and verbose comments, indicating a community that values thorough and
well-structured discourse. This unique preference set suggests that
r/libertarian places a higher emphasis on detailed and supportive interactions
compared to the other two subreddits. These findings highlight the nuanced
differences and similarities in community norms within political subreddits,
with r/democrats and r/republican sharing many conversational norms, while
r/libertarian adopts a distinctively detailed and supportive approach to ore
structured discourse.

Figure 24: RPM plots for politics subreddits on the supportiveness dimension.
Figure 25: RPM plots for politics subreddits on the formality dimension.
Figure 26: RPM plots for politics subreddits on the politeness dimension.
Figure 27: RPM plots for politics subreddits on the sarcasm dimension. Figure
28: RPM plots for politics subreddits on the humor dimension. Figure 29: RPM
plots for politics subreddits on the verbosity dimension.

###  N.4 Science Subreddits

We examined three subreddits related to science: r/askscience,
r/shittyaskscience, and r/asksciencediscussion. First, it is notable that
r/shittyaskscience shows very weak preference across all norm dimensions, and
relatively weaker disapproval than the other two subreddits.
r/shittyaskscience disproves of toxic, casual and rude comments to a lesser
extent than r/askscience and r/asksciencediscussion (Figures 30–32). This
could be attributed to the fact that it is a spin-off subreddit created to
mock r/askscience, which makes it more tolerant to toxic, casual or rude
comments. On the other hand, the three subreddits have similar overall
preference patterns despite different magnitudes, except for r/askscience in
the serious–humorous norm dimension. As shown in Figure 34, r/askscience
exhibits significantly stronger preference for humorous comments compared to
the other two subreddits. While seemingly counter-intuitive, since
r/askscience is a tightly moderateed subreddit, but its preference for
humorous data implies that comments that both adhere to the subreddit rules
_and_ humorous would typically be a high quality comment preferred by
community members.

Figure 30: RPM plots for science subreddits on the supportiveness dimension.
Figure 31: RPM plots for science subreddits on the formality dimension. Figure
32: RPM plots for science subreddits on the politeness dimension. Figure 33:
RPM plots for science subreddits on the sarcasm dimension. Figure 34: RPM
plots for science subreddits on the humor dimension. Figure 35: RPM plots for
science subreddits on the verbosity dimension. Figure 36: Human annotation UI
for the binary norm dimension classification task. For each question, two
options (1, 2) were provided without a tie option. Additionally, there were
two extra options to mark samples that could not be properly annotated with
the given context (hard-to-tell, media-needed). Figure 37: Annotation
guideline provided to human annotators. Figure 38: Annotation guideline
provided to human annotators. Figure 39: Annotation guideline provided to
human annotators. Figure 40: Human annotation UI for validating the quality of
the filtered synthetic data. Figure 41: The annotators are presented with two
versions of comments on a post: one synthetic and the other the original seed
comment. Then, the annotators evaluated these two comments for their
qualities, such as fluency and content preservation.