Solution of the Kac–Bernstein functional equation

on Abelian groups in the class of positive functions

G.M. Feldman

The general form of the solutions of the Kac–Bernstein functional equation
$f(x+y)g(x-y)=f(x)f(y)g(x)g(-y),\ x,y\in X,$ on an arbitrary Abelian group $X$
in the class of positive functions is obtained. We also study the solutions of
this equation in the class of complex-valued functions that do not vanish and
satisfy the Hermitian condition.

Subject Classification (2010): 39B52, 39A10

Keywords. Abelian group, Kac–Bernstein functional equation

##  1  Introduction

Let $X$ be an Abelian group. We remaind that the Kac–Bernstein functional
equation on the group $X$ is an equation of the form

$f(x+y)g(x-y)=f(x)f(y)g(x)g(-y),\ x,y\in X.$ (1)

The Kac–Bernstein functional equation arises in the proof of the following
characterization theorem of mathematical statistics on the real line.

######  The Kac–Bernstein theorem

Let $\xi_{1}$ and $\xi_{2}$ be independent random variables. If the sum
$\xi_{1}+\xi_{2}$ and the difference $\xi_{1}-\xi_{2}$ are also independent,
then $\xi_{1}$ and $\xi_{2}$ are Gaussian random variables.

This theorem was independently proved by M. Kac [12] and S.N. Bernstein [2].
The Kac–Bernstein theorem is one of the first, and apparently, the most famous
characterization theorem of mathematical statistics. Some generalizations of
this theorem were studied in the case when independent random variables take
values either in an arbitrary locally compact Abelian group or in specific
locally compact Abelian groups ([1, 4, 5, 8, 9, 11, 14, 15, 16, 17], see also
[6, Chapter 3]). In this case, either the densities of the distributions of
independent random variables or their characteristic functions satisfy the
equation (1). In the latter case the equation (1) is considered on the
character group of the original group. We also remark that the equation (1)
was studied in [10] on locally compact Abelian groups for complex-valued
continuous and measurable functions that satisfy the Hermitian condition.

In the article [13] Pl. Kannappan considered the equation (1) on the real line
assuming that the functions $f(x)$ and $g(x)$ are positive. Moreover, no
restrictions were imposed on the functions $f(x)$ and $g(x)$ such as
continuity, measurability, positive definiteness. Pl. Kannappan proved the
following theorem.

######  Theorem A

Let $f(x)$ and $g(x)$ be positive functions on the real line satisfying the
Kac–Bernstein functional equation $(\ref{1})$. Then the functions $f(x)$ and
$g(x)$ may be represented in the form

$f(x)=\exp\\{P(x)+l(x)+r\\},\quad g(x)=\exp\\{P(x)+m(x)-r\\},$

where the function $P(x)$ satisfies the equation

$P(x+y)+P(x-y)=2[P(x)+P(y)],$ (2)

each of the functions $l(x)$ and $m(x)$ satisfies the Cauchy equation

$l(x+y)=l(x)+l(y),$ (3)

$r$ is a constant.

The main result of this note is the proof of a theorem in which a complete
description of the solutions of the equation (1) in the class of positive
functions on an arbitrary Abelian group $X$ is given.

##  2 Solution of the Kac–Bernstein functional equation in the class of
positive functions

Let $X$ be an Abelian group, $n$ be a positive integer. Denote by $X^{(n)}$
the image of the group $X$ under the mapping $x\rightarrow nx$. Let

$X=\bigcup_{\alpha}\left(x_{\alpha}+X^{(2)}\right)$

be a decomposition of the group $X$ with respect to the subgroup $X^{(2)}$,
where an index $\alpha$ runs through the factor-group $X/X^{(2)}$. Moreover,
we assume that the element $x_{0}=0$ corresponds to the coset $X^{(2)}$.

Let $T(x)$ be a function on the group $X$, and $h$ be an arbitrary element of
the group $X$. Denote by $\Delta_{h}$ the finite difference operator

$\Delta_{h}T(x)=T(x+h)-T(x).$

A function $T(x)$ is called a polynomial on the group $X$ if for some
nonnegative integer $n$ the function $T(x)$ satisfies the equation

$\Delta_{h}^{n+1}T(x)=0,\quad x,h\in X.$ (4)

The minimal $n$ at which this equation holds is called the degree of the
polynomial $T(x)$. We remark that if a function $P(x)$ satisfies the equation
$(\ref{2})$, then either $P(x)\equiv 0$ for $x\in X$ or $P(x)$ is a polynomial
of degree 2. If a function $l(x)$ satisfies the equation $(\ref{3})$, then
either $l(x)\equiv 0$ for $x\in X$ or $l(x)$ is a polynomial of degree 1.

The main result of the article is the proof of the following statement.

######  Theorem 2.1

Let $X$ be an Abelian group. Let $f(x)$ and $g(x)$ be positive functions on
the group $X$ satisfying the Kac–Bernstein functional equation $(\ref{1})$.
Then the functions $f(x)$ and $g(x)$ can be represented in the form

$f(x)=\exp\\{P(x)+l(x)+r(x)\\},\quad g(x)=\exp\\{P(x)+m(x)-r(x)\\},\quad x\in
X,$ (5)

where the function $P(x)$ satisfies the equation $(\ref{2})$, each of the
functions $l(x)$ and $m(x)$ satisfies the equation $(\ref{3})$, the function
$r(x)$ is a constant on each of the cosets of the subgroup $X^{(2)}$ in $X$.

To prove Theorem 2.1 we need the following lemmas.

######  Lemma 2.1

([3]) Let $X$ be an Abelian group, $T(x)$ be a function on the group $X$
satisfying the equation $(\ref{4})$. Then there are symmetric $k$-additive
functions $g_{k}(x_{1},\dots,x_{k})$ on the group $X^{k}$, $k=1,2,\dots,n$,
such that

$T(x)=\sum_{k=0}^{n}g_{k}^{*}(x),\quad x\in X,$

where $g^{*}_{0}(x)=T(0)$, $g^{*}_{k}(x)=g_{k}(x,\dots,x)$, $k=1,2,\dots,n$.

######  Corollary 2.1

Let $X$ be an Abelian group, $T(x)$ be a polynomial on the group $X$ of degree
$\leq 2$. Then $T(x)$ may be represented in the form

$T(x)=P(x)+l(x)+r,\quad x\in X,$

where the function $P(x)$ satisfies the equation $(\ref{2})$, the function
$l(x)$ satisfies the equation $(\ref{3})$, $r$ is a constant.

######  Lemma 2.2

Let $X$ be an Abelian group, $T(x)$ be a function on the group $X$ satisfying
the equation

$\Delta_{2k}\Delta_{h}^{2}T(x)=0,\quad k,h,x\in X.$ (6)

Then the function $T(x)$ may be represented in the form

$T(x)=P(x)+l(x)+r(x),\quad x\in X,$ (7)

where the function $P(x)$ satisfies the equation $(\ref{2})$, the function
$l(x)$ satisfies the equation $(\ref{3})$, the function $r(x)$ is a constant
on each of the cosets of the subgroup $X^{(2)}$ in $X$.

Proof. Note that the equation (6) on specific groups $X$ for an even function
$T(x)$ arose in solving characterization problems of mathematical statistics
on locally compact Abelian groups. Lemma 2.2 on an arbitrary locally compact
abelian group $X$ for a continuous even function $T(x)$ was proved in [7,
Lemma 6]. We will follow the scheme of the proof of Lemma 6 in [7].

Obviously, without loss of generality, we can assume that $T(0)=0$. We fix an
element $x_{\alpha}\in X$. It follows from the equation (6) that the
restriction of the function $T(x_{\alpha}+x)$ to the subgroup $X^{(2)}$ is a
polynomial of degree $\leq 2$. Therefore, by Corollary 2.1 applied to the
subgroup $X^{(2)}$, there is a representation

$T(x_{\alpha}+x)=P_{\alpha}(x)+l_{\alpha}(x)+s_{\alpha},\quad x\in X^{(2)},$
(8)

where the function $P_{\alpha}(x)$ satisfies the equation $(\ref{2})$, the
function $l_{\alpha}(x)$ satisfies the equation $(\ref{3})$, $s_{\alpha}$ is a
constant. In particular, $s_{0}=0$. Moreover, by Lemma 2.1,
$P_{\alpha}(x)=A_{\alpha}(x,x)$, where $A_{\alpha}(x_{1},x_{2})$,
$x_{1},x_{2}\in X^{(2)}$, is a symmetric biadditive function. We extend the
function $A_{\alpha}(x_{1},x_{2})$ from the subgroup $(X^{(2)})^{2}$ as a
symmetric biadditive function $\tilde{A}_{\alpha}(x_{1},x_{2})$ to the group
$X^{2}$, setting $\tilde{A}_{\alpha}(x_{1},x_{2})={1\over
4}A_{\alpha}(2x_{1},2x_{2})$, $x_{1},x_{2}\in X$. Denote by
$\tilde{P}_{\alpha}(x)$ the corresponding extension of the function
$P_{\alpha}(x)$ from $X^{(2)}$ to $X$. We extend the function $l_{\alpha}(x)$
from the subgroup $X^{(2)}$ as an additive function $\tilde{l}_{\alpha}(x)$ to
the group $X$, setting $\tilde{l}_{\alpha}(x)={1\over 2}l_{\alpha}(2x)$, $x\in
X$. It follows from $(\ref{7})$ that

$T(-x_{\alpha}-x)=T(x_{\alpha}+(-2x_{\alpha}-x)=P_{\alpha}(-2x_{\alpha}-x)+l_{\alpha}(-2x_{\alpha}-x)+s_{\alpha}$
$=4\tilde{P}_{\alpha}(x_{\alpha})+4\tilde{A}_{\alpha}(x_{\alpha},x)+P_{\alpha}(x)-2\tilde{l}_{\alpha}(x_{\alpha})-l_{\alpha}(x)+s_{\alpha},\quad
x\in X^{(2)}.$ (9)

Substituting $k=h$ in $(\ref{5})$, we obtain

$T(x+4h)-2T(x+3h)+2T(x+h)-T(x)=0,\quad x,h\in X.$ (10)

Substituting $x=0$, $h=x_{\alpha}+2x$ in (10), we find

$T(4x_{\alpha}+8x)-2T(3x_{\alpha}+6x)+2T(x_{\alpha}+2x)=0,\quad x\in X.$ (11)

1\. Suppose that $T(-x)=T(x)$, i.e. $T(x)$ is an even function. Then (8) and
(2) imply that

$l_{\alpha}(x)=2\tilde{A}_{\alpha}(x_{\alpha},x),\quad x\in X^{(2)}.$ (12)

Since
$P_{\alpha}(x)=\tilde{P}_{\alpha}(x_{\alpha}+x)-2\tilde{A}_{\alpha}(x_{\alpha},x)-\tilde{P}_{\alpha}(x_{\alpha})$,
we find from (8) and (12) that

$T(x_{\alpha}+x)=\tilde{P}_{\alpha}(x_{\alpha}+x)+r_{\alpha},\quad x\in
X^{(2)},$ (13)

where $r_{\alpha}=-\tilde{P}_{\alpha}(x_{\alpha})+s_{\alpha}$. We note that if
$\alpha=0$, then $r_{0}=0$. Taking into account (13), we obtain from (11)

$\tilde{P}_{0}(4x_{\alpha}+8x)-2\tilde{P}_{\alpha}(3x_{\alpha}+6x)-2r_{\alpha}+2\tilde{P}_{\alpha}(x_{\alpha}+2x)+2r_{\alpha}=0,\quad
x\in X.$

This implies that

$4(\tilde{P}_{0}(x)-\tilde{P}_{\alpha}(x))+4(\tilde{A}_{0}(x_{\alpha},x)-\tilde{A}_{\alpha}(x_{\alpha},x))+\tilde{P}_{0}(x_{\alpha})-\tilde{P}_{\alpha}(x_{\alpha})=0,\quad
x\in X.$ (14)

It follows from (14) that $\tilde{P}_{0}(x)=\tilde{P}_{\alpha}(x)$ for all
$x\in X$ and any $\alpha$. Taking this into account, (13) implies for the
function $T(x)$ the representation

$T(x)=\tilde{P}_{0}(x)+r_{\alpha},\quad x\in x_{\alpha}+X^{(2)}.$ (15)

Put

$P(x)=\tilde{P}_{0}(x),\quad x\in X,\quad r(x)=r_{\alpha},\quad x\in
x_{\alpha}+X^{(2)}.$ (16)

The desired representation for the function $T(x)$ follows from (15) and (16).
Moreover, in the case when $T(x)$ is an even function, we proved that
$l(x)\equiv 0$ for $x\in X$ in the representation (7).

2\. Suppose that $T(-x)=-T(x)$, i.e. $T(x)$ is an odd function. Then (8) and
(2) imply that

$P_{\alpha}(x)=0,\quad x\in
X^{(2)},\quad\tilde{l}_{\alpha}(x_{\alpha})=s_{\alpha}.$ (17)

Taking into account (17), it follows from (8) that

$T(x_{\alpha}+x)=\tilde{l}_{\alpha}(x_{\alpha}+x),\quad x\in X^{(2)}.$ (18)

Hence, the equation (11) takes the form

$\tilde{l}_{0}(4x_{\alpha}+8x)-2\tilde{l}_{\alpha}(3x_{\alpha}+6x)+2\tilde{l}_{\alpha}(x_{\alpha}+2x)=0,\quad
x\in X.$

This implies that $\tilde{l}_{0}(x)=\tilde{l}_{\alpha}(x)$ for $x\in X$. In
view of this, (18) implies that

$T(x)=\tilde{l}_{0}(x),\quad x\in X.$ (19)

Put $l(x)=\tilde{l}_{0}(x)$, $x\in X.$ The desired representation for the
function $T(x)$ follows from (19). Moreover, in the case when $T(x)$ is an odd
function, we proved that $P(x)\equiv 0$, $r(x)\equiv 0$ for $x\in X$ in the
representation (7).

To complete the proof of the lemma we note that an arbitrary function $T(x)$
is the sum of an even and an odd function. $\blacksquare$

Proof of Theorem 2.1. Put $T(x)=\log f(x)$ and $S(x)=\log g(x)$. It follows
from (1) that the functions $T(x)$ and $S(x)$ satisfy the equation

$T(x+y)+S(x-y)=A(x)+B(y),\quad x,y\in X,$ (20)

where $A(x)=T(x)+S(x)$, $B(y)=T(y)+S(-y)$. We solve the equation (20) using
the finite difference method. We replace $x$ with $x+k$ and $y$ with $y+k$ in
(20), where $k$ is an arbitrary element of the group $X$. Subtract (20) from
the obtained equation. We have

$\Delta_{2k}T(x+y)=\Delta_{k}A(x)+\Delta_{k}B(y).$ (21)

We replace $y$ with $y+h$ in (21), where $h$ is an arbitrary element of the
group $X$. Subtract (21) from the obtained equation. We get

$\Delta_{h}\Delta_{2k}T(x+y)=\Delta_{h}\Delta_{k}B(y).$ (22)

We replace $x$ with $x+h$ in (22) and subtract (22) from the obtained
equation. We have

$\Delta_{h}^{2}\Delta_{2k}T(x+y)=0.$

Putting here $y=0$, we receive that the function $T(x)$ satisfies the equation
(6). By Lemma 2.2, the function $T(x)$ is represented in the form (7). This
implies that the function $f(x)$ can be represented in the form

$f(x)=\exp\\{P(x)+l(x)+r(x)\\},\quad x\in X,$

where the function $P(x)$ satisfies the equation $(\ref{2})$, the function
$l(x)$ satisfies the equation $(\ref{3})$, the function $r(x)$ is a constant
on each of the cosets of the subgroup $X^{(2)}$ in $X$.

Reasoning similarly, we obtain that the function $S(x)$ also satisfies the
equation (6) and by Lemma 2.2, the function $g(x)$ may be represented in the
form

$g(x)=\exp\\{Q(x)+m(x)+s(x)\\},\quad x\in X,$

where the function $Q(x)$ satisfies the equation $(\ref{2})$, the function
$m(x)$ satisfies the equation $(\ref{3})$, the function $s(x)$ is a constant
on each of the cosets of the subgroup $X^{(2)}$ in $X$. Substituting the
obtained expressions for the functions $f(x)$ and $g(x)$ into the equation
(1), we find that $P(x)=Q(x)$, $x\in X$. Moreover, assuming that in the
expression obtained after substitution $x,y\in x_{\alpha}+X^{(2)}$, we get
that $s(x)=-r(x)$, $x\in X$. $\blacksquare$

######  Corollary 2.2

If we assume in Theorem 2.1 that the group $X$ satisfies the condition
$X^{(2)}=X$, then the functions $f(x)$ and $g(x)$ may be represented in the
form

$f(x)=\exp\\{P(x)+l(x)+r\\},\quad g(x)=\exp\\{P(x)+m(x)-r\\},\quad x\in X,$

where the function $P(x)$ satisfies the equation $(\ref{2})$, each of the
functions $l(x)$ and $m(x)$ satisfies the equation $(\ref{3})$, $r$ is a
constant.

Obviously, for the group of real numbers $X=\mathbb{R}$ Theorem A follows from
Corollary 2.2.

######  Corollary 2.3

If we assume in Theorem 2.1 that $f(x)=g(x)$, i.e. instead of the equation
$(\ref{1})$ we consider the equation

$f(x+y)f(x-y)=f^{2}(x)f(y)f(-y),\ x,y\in X,$ (23)

then the function $f(x)$ may be represented in the form

$f(x)=\exp\\{P(x)+l(x)\\},\quad x\in X,$

where the function $P(x)$ satisfies the equation $(\ref{2})$ and the function
$l(x)$ satisfies the equation $(\ref{3})$.

######  Remark 2.1

The description of the solutions of the equation (1) given in Theorem 2.1 is
complete in the following sense. If functions $f(x)$ and $g(x)$ are
represented in the form (5), then they satisfy the equation $(\ref{1})$. To
verify this, it is enough to note that for any $x,y\in X$ the elements $x+y$
and $x-y$ are in the same coset of the subgroup $X^{(2)}$ in $X$. It is
obvious that the similar statement also holds for Corollaries 2.2 and 2.3.

##  3 Solution of the Kac–Bernstein functional equation in the class of
complex-valued functions satisfying the Hermitian condition

Let $X$ be an Abelian group. We say that a complex-valued function $f(x)$ on
the group $X$ satisfies the Hermitian condition if

$f(-x)=\overline{f(x)},\quad x\in X.$ (24)

The characteristic functions of probability distributions on a locally compact
Abelian group satisfy, in particular, the Hermitian condition. We note that in
the article [9] the solutions of the equation (1) in the class of
characteristic functions on such groups were studied. In this section we study
the solutions of the equation (1) in the class of functions satisfying the
Hermitian condition on an arbitrary group $X$. We complete Theorem 2.1 with
the following statement.

######  Theorem 3.1

Let $X$ be an Abelian group. Let $f(x)$ and $g(x)$ be complex-valued non-
vanishing functions on the group $X$, each of which satisfies the Hermitian
condition $(\ref{19})$. If the functions $f(x)$ and $g(x)$ satisfy the
Kac–Bernstein functional equation $(\ref{1})$, then the functions $f(x)$ and
$g(x)$ may be represented in the form

$f(x)=\alpha(x)a(x)\exp\\{P(x)+r(x)\\},\quad
g(x)=\beta(x)b(x)\exp\\{P(x)-r(x)\\},\quad x\in X,$ (25)

where each of the complex-valued functions $\alpha(x)$ and $\beta(x)$
satisfies the equation

$\alpha(x+y)=\alpha(x)\alpha(y),\quad x,y\in X,$ (26)

and the condition

$|\alpha(x)|=1,\quad x\in X,$ (27)

the functions $a(x)$ and $b(x)$ take the values $\pm 1$ and they are constant
on each of the cosets of the subgroup $X^{(4)}$ in $X$, the real-valued
function $P(x)$ satisfies the equation $(\ref{2})$, the real-valued function
$r(x)$ is a constant on each of the cosets of the subgroup $X^{(2)}$ in $X$.

Proof. It is clear that without loss of generality we can assume that
$f(0)=g(0)=1$. Since the functions $|f(x)|$ and $|g(x)|$, obviously also
satisfy the equation $(\ref{1})$ and they are even, by Theorem 2.1, there is a
representation

$|f(x)|=\exp\\{P(x)+r(x)\\},\quad|g(x)|=\exp\\{P(x)-r(x)\\},\quad x\in X,$
(28)

where the function $P(x)$ is real-valued and satisfies the equation
$(\ref{2})$, the function $r(x)$ is real-valued and is a constant on each of
the cosets of the subgroup $X^{(2)}$ in $X$. Put

$p(x)=f(x)/|f(x)|,\quad q(x)=g(x)/|g(x)|,\quad x\in X.$ (29)

Then $|p(x)|=|q(x)|=1$ for $x\in X$, and the functions $p(x)$ and $q(x)$
satisfy the equation $(\ref{1})$ which takes the form

$p(x+y)q(x-y)=p(x)p(y)q(x)q(-y),\quad x,y\in X.$ (30)

Putting $y=x$, then $y=-x$ in (30) we obtain

$p(2x)=p^{2}(x),\quad q(2x)=q^{2}(x),\quad x\in X.$ (31)

Interchange $x$ and $y$ in (30), and multiply the resulting equation and the
equation (30). Taking into account (24), we find

$p^{2}(x+y)=p^{2}(x)p^{2}(y),\quad x,y\in X.$ (32)

It follows from (31) and (32) that the restriction of the function $p(x)$ to
the subgroup $X^{(2)}$ satisfies the equation (26). Reasoning similarly we get
that the restriction of the function $q(x)$ to the subgroup $X^{(2)}$ also
satisfies the equation (26).

The group $X$ can be considered as a discrete Abelian group. It is well known
that then every function on an arbitrary subgroup of the group $X$ satisfying
the equation (26) and the condition (27) is a restriction to this subgroup a
function on the group $X$, satisfying on $X$ the equation (26) and the
condition (27). We apply this result to the subgroup $X^{(2)}$. Let
$\alpha(x)$ and $\beta(x)$ be the corresponding extensions from the subgroup
$X^{(2)}$ to the group $X$ of the functions $p(x)$ and $q(x)$. Note that the
functions $\alpha(x)$ and $\beta(x)$ satisfy the equation (1) on $X$. Put

$a(x)=p(x)/\alpha(x),\quad b(x)=q(x)/\beta(x),\quad x\in X.$ (33)

Since $a(x)=b(x)=1$ for $x\in X^{(2)}$, it follows from (31) that the
functions $a(x)$ and $b(x)$ are even and take the values $\pm 1$. It is
obvious that the functions $a(x)$ and $b(x)$ satisfy the equation (30) which
takes the form

$a(x+y)b(x-y)=a(x)a(y)b(x)b(y),\quad x,y\in X.$ (34)

If $x,y\in x_{\alpha}+X^{(2)}$, then $x\pm y\in X^{(2)}$, and the left-hand
side of the equation (34) is equal to 1. This implies that $a(x)b(y)=a(y)b(x)$
for $x,y\in x_{\alpha}+X^{(2)}$, and hence either $a(x)=b(x)$ or $a(x)=-b(x)$
for $x\in x_{\alpha}+X^{(2)}$. Given this, it follows from (34) that

$a(x+y)a(x-y)=1,\quad b(x+y)b(x-y)=1,\quad x,y\in
X^{(2)}{\cup}\big{(}x_{\alpha}+X^{(2)}\big{)},$ (35)

and (35) implies that the functions $a(x)$ and $b(x)$ are constant on each of
the cosets of the subgroup $X^{(4)}$ in $X$. The statement of the theorem
follows from (28), (29) and (33). $\blacksquare$

######  Remark 3.1

The functions $a(x)$ and $b(x)$ arising in the proof of Theorem 3.1 are
constant on each of the cosets of the subgroup $X^{(4)}$ in $X$. Surprisingly
is the fact that in this statement we cannot replace $X^{(4)}$ by $X^{(2)}$.
Below we construct an example that shows it (compare with Remark 2 in [9]).

Denote by $\mathbb{Z}(4)=\\{0,1,2,3\\}$ the group of residue classes modulo 4.
Let $X=(\mathbb{Z}(4))^{2}$. Denote by $x=(m,n)$, $m,n\in\mathbb{Z}(4)$,
elements of the group $X$. Then $X^{(2)}=\\{(0,0),(0,2),(2,0),(2,2)\\}$,
$X^{(4)}=\\{(0,0)\\}.$ Consider on the group $X$ the functions

$f(m,n)=\begin{cases}1,\ \ \
(m,n)\in\\{X^{(2)},(1,0),(3,0),(0,1),(0,3),(1,1),(3,3)\\},\\\ -1,\
(m,n)\in\\{(1,2),(3,2),(2,1),(2,3),(1,3),(3,1)\\}\\\ \end{cases}$

and

$g(m,n)=\begin{cases}1,\ \ \
(m,n)\in\\{X^{(2)},(1,0),(3,0),(0,1),(0,3),(1,3),(3,1)\\},\\\ -1,\
(m,n)\in\\{(1,2),(3,2),(2,1),(2,3),(1,1),(3,3)\\}.\end{cases}$

Verify that the functions $f(x)$ and $g(x)$ satisfy the equation (1). Let

$X=X^{(2)}\cup\big{(}(1,0)+X^{(2)}\big{)}\cup\big{(}(0,1)+X^{(2)}\big{)}\cup\big{(}(1,1)+X^{(2)}\big{)}$

be a decomposition of the group $X$ with respect to the subgroup $X^{(2)}$. It
follows from the expressions for the functions $f(x)$ and $g(x)$ that
$f(x)g(x)=1$ for $x\in
X^{(2)}\cup\big{(}(0,1)+X^{(2)}\big{)}\cup\big{(}(1,0)+X^{(2)}\big{)}$ and
$f(x)g(x)=-1$ for $x\in(1,1)+X^{(2)}$.

We consider successively the cases when $x$ and $y$ belong to the given cosets
s of the subgroup $X^{(2)}$ in $X$.

Let $x,y\in X^{(2)}$. Then $x\pm y\in X^{(2)}$, and the right-hand and left-
hand sides of the equation (1) are equal to $1$.

Let $x\in X^{(2)}$ and $y\in(1,0)+X^{(2)}$. Then the right-hand side of the
equation (1) is equal to $1$. We note that $x\pm y\in(1,0)+X^{(2)}$. If the
left-hand side of the equation (1) is equal to $-1$, then either
$x+y\in\\{(1,0),(3,0)\\}$, $x-y\in\\{(1,2),(3,2)\\}$ or
$x+y\in\\{(1,2),(3,2)\\}$, $x-y\in\\{(1,0),(3,0)\\}$. In each of these cases
$2x\neq(0,0)$, which contradicts the fact that $x\in X^{(2)}$. Hence, the
left-hand side of the equation (1) is also equal to $1$.

Let $x,y\in(1,0)+X^{(2)}$. Then the right-hand side of the equation (1) is
equal to $1$. We note that $x\pm y\in X^{(2)}$. Hence, the left-hand side of
the equation (1) is also equal to $1$.

Let $x\in(1,0)+X^{(2)}$ and $y\in(0,1)+X^{(2)}$. Then the right-hand side of
the equation (1) is equal to $1$. We note that $x\pm y\in(1,1)+X^{(2)}$. If
the left-hand side of the equation (1) is equal to $-1$, then either $x\pm
y\in\\{(1,1),(3,3)\\}$ or $x\pm y\in\\{(1,3),(3,1)\\}$. In each of these cases
$2x\in\\{(0,0),(2,2)\\}$, which contradicts the fact that $x\in(1,0)+X^{(2)}$.
Hence, the left-hand side of the equation (1) is also equal to $1$.

Let $x\in(1,0)+X^{(2)}$ and $y\in(1,1)+X^{(2)}$. Then the right-hand side of
the equation (1) is equal to $-1$. We note that $x\pm y\in(0,1)+X^{(2)}$. If
the left-hand side of the equation (1) is equal to $1$, then either $x\pm
y\in\\{(0,1),(0,3)\\}$ or $x\pm y\in\\{(2,1),(2,3)\\}$. In each of these cases
$2x\in\\{(0,0),(0,2)\\}$, which contradicts the fact that $x\in(1,0)+X^{(2)}$.
Hence, the left-hand side of the equation (1) is also equal to $-1$.

The remaining cases are similar to those considered above.

In the constructed example in the representation (25) $\alpha(x)\equiv 1$,
$\beta(x)\equiv 1$, $P(x)\equiv 0$ and $r(x)\equiv 0$ for $x\in X$, but the
functions $a(x)=f(x)$ and $b(x)=g(x)$ are not constant on each of the cosets
of the subgroup $X^{(2)}$ in $X$.

######  Remark 3.2

The example in Remark 3.1 also shows that both the function $a(x)$ and the
function $b(x)$ arising in the proof of Theorem 3.1 need not satisfy the
equation (26). At the same time, assuming that the group $X$ is such that
$X/X^{(2)}\cong\mathbb{Z}(2)$, where $\mathbb{Z}(2)$ is the group of residue
classes modulo 2, each of the functions $a(x)$ and $b(x)$ satisfies the
equation (26). Indeed, if $X/X^{(2)}\cong\mathbb{Z}(2)$, then a decomposition
of the group $X$ with respect to the subgroup $X^{(2)}$ is of the form
$X=X^{(2)}\cup(x_{1}+X^{(2)})$. Therefore, (35) implies that the functions
$a(x)$ and $b(x)$ are constant on each of the cosets of the subgroup $X^{(2)}$
in $X$. Since $a(x)=b(x)=1$ for $x\in X^{(2)}$ and both functions $a(x)$ and
$b(x)$ on the coset $x_{1}+X^{(2)}$ take the values $\pm 1$, then, obviously,
each of them satisfies the equation (26). The representation (25) then takes
the form

$f(x)=\pm\alpha(x)\exp\\{P(x)+r(x)\\},\quad
g(x)=\pm\beta(x)\exp\\{P(x)-r(x)\\},\quad x\in X.$

If we assume that the group $X$ satisfies the condition $X^{(2)}=X$, then
Theorem 3.1 implies directly that in this case the representation (25) takes
the form

$f(x)=\pm\alpha(x)\exp\\{P(x)+r\\},\quad
g(x)=\pm\beta(x)\exp\\{P(x)-r\\},\quad x\in X,$ (36)

where $r$ is a real constant.

######  Remark 3.3

Generally speaking, if functions $f(x)$ and $g(x)$ are represented in the form
(25), then they need not satisfy the equation (1). An example can be easily
constructed on the group $(\mathbb{Z}(4))^{2}$. Nevertheless, if we
additionally require that the functions $a(x)$ and $b(x)$ in the
representation (25) are constant on each of the cosets of the subgroup
$X^{(2)}$ in $X$ and $a(x)b(x)=1$ for $x\in X$, it is easy to verify that in
this case the functions $f(x)$ and $g(x)$ already satisfy the equation (1).

######  Corollary 3.1

If we assume in Theorem 3.1 that $f(x)=g(x)$, i.e. instead of the equation
$(\ref{1})$ we consider the equation $(\ref{n4})$, then the function $f(x)$
may be represented in the form

$f(x)=\alpha(x)a(x)\exp\\{P(x)\\},\quad x\in X,$ (37)

where the complex-valued function $\alpha(x)$ satisfies the equation
$(\ref{20})$ and the condition $(\ref{21})$, the function $a(x)$ takes the
values $\pm 1$ and is a constant on each of the cosets of the subgroup
$X^{(2)}$ in $X$ and the real-valued function $P(x)$ satisfies the equation
$(\ref{2})$.

Proof. Following the scheme of the proof of Theorem 3.1, we arrive at the
equation $(\ref{26})$. Since $f(x)=g(x)$, we have $a(x)=b(x)$ in (34) and we
get that the function $a(x)$ satisfies the equation

$a(x+y)a(x-y)=1,\quad x,y\in X.$ (38)

It follows from (38) that the function $a(x)$ is a constant on each of the
cosets of the subgroup $X^{(2)}$ in $X$. $\blacksquare$

######  Remark 3.4

Despite the fact that the function $a(x)$ in the representation (37) in
Corollary 3.1 is a constant on each of the cosets of the subgroup $X^{(2)}$ in
$X$, unlike the representation (25) in Theorem 3.1, where the function $a(x)$
is a constant on each of the cosets of the subgroup $X^{(4)}$ in $X$,
generally speaking the function $a(x)$ need not satisfy the equation
$(\ref{20})$. We construct an example. Denote by $\mathbb{Z}$ the group of
integers. Let $X=\mathbb{Z}^{2}$. Denote by $x=(m,n)$, $m,n\in\mathbb{Z}$,
elements of the group $X$. Put

$f(m,n)=\exp\\{i\pi mn\\},\quad(m,n)\in X.$

It is obvious that the function $f(x)$ satisfies the equation (23). It is easy
to see that in this case in the representation (37) $\alpha(x)\equiv 1$,
$P(x)\equiv 0$ for $x\in X$, but the function $a(x)=f(x)$ does not satisfy the
equation $(\ref{20})$.

######  Remark 3.5

In contrast to Theorem 3.1, the description of the solutions of the equation
(23) given in Corollary 3.1 is complete in the following sense. If a function
$f(x)$ is represented in the form (37), then it satisfies the equation (23).
It follows from the fact that if a function $a(x)$ takes the values $\pm 1$
and is a constant on each of the cosets of the subgroup $X^{(2)}$ in $X$, then
it satisfies the equation (38), and hence also satisfies the equation (23).

Assume that an Abelian group $X$ satisfies the condition $X^{(2)}=X$. Based on
the representation (36), we prove the following statement.

######  Theorem 3.2

Let $X$ be an Abelian group, such that $X^{(2)}=X$. Let $f(x)$ and $g(x)$ be
complex-valued functions that are not identically zero on the group $X$, each
of which satisfies the Hermitian condition $(\ref{19})$. If the functions
$f(x)$ and $g(x)$ satisfy the Kac–Bernstein functional equation $(\ref{1})$,
then they can be represented in the form

$f(x)=\begin{cases}\pm\alpha(x)\exp\\{P(x)+r\\},\ x\in G,\\\
0,\qquad\qquad\qquad\quad\qquad x\notin G,\end{cases}\quad
g(x)=\begin{cases}\pm\beta(x)\exp\\{P(x)-r\\},\ x\in G,\\\
0,\qquad\qquad\qquad\quad\qquad x\notin G,\end{cases}$ (39)

where $G$ is a subgroup of $X$, such that the factor-group $X/G$ contains no
elements of order $2$, each of the complex-valued functions $\alpha(x)$ and
$\beta(x)$ satisfies the equation $(\ref{20})$ and the condition $(\ref{21})$,
the real-valued function $P(x)$ satisfies the equation $(\ref{2})$, $r$ is a
real constant.

Proof. It follows from the equation (1) that either $f(0)g(0)=0$ or
$f(0)g(0)=1$. Let $f(0)g(0)=0$ and assume for definiteness that $f(0)=0$.
Then, putting $y=-x$ in the equation (1) we obtain that $|f(x)|^{2}g^{2}(x)=0$
for $x\in X$. Hence, $f(x+y)g(x-y)=0$ for $x,y\in X$. In view of $X^{(2)}=X$,
this implies that either $f(x)\equiv 0$ or $g(x)\equiv 0$ for $x\in X$,
contrary to the condition. Therefore $f(0)g(0)=1$. Without loss of generality,
we can assume that $f(0)=g(0)=1$.

Putting $y=x$, then $y=-x$ in (1) we obtain

$f(2x)=f^{2}(x)|g^{2}(x)|,\quad g(2x)=|f^{2}(x)|g^{2}(x),\quad x\in X.$ (40)

In view of $X^{(2)}=X$, it follows from (40) that

$|f(x)|=|g(x|,\quad x\in X.$ (41)

Taking into account (41), the equation (1) implies that the set

$G=\\{x\in X:f(x)\neq 0\\}=\\{x\in X:g(x)\neq 0\\}$

is a subgroup of $X$. It follows from (40) that the subgroup $G$ has the
property: if $2x\in G$, then $x\in G$. This implies that the factor-group
$X/G$ contains no elements of order $2$. It also follows from (40) that the
subgroup $G$ satisfies the condition $G^{(2)}=G$. We apply the representation
(36) to the group $G$. $\blacksquare$

######  Remark 3.6

The description of the solutions of the equation (1) given in Theorem 3.2 is
complete in the following sense. Let $X$ be an Abelian group, such that
$X^{(2)}=X$. If functions $f(x)$ and $g(x)$ are represented in the form (39),
then they satisfy the equation (1).

## References

  * [1] Baryshnikov, Y., Eisenberg, B., Stadje W.: Independent variables with independent sum and difference: S1-case. J. Multivariate Anal. 45 (2), 161–170 (1993).
  * [2] Bernstein, S. N.: On a property which characterizes a Gaussian distribution. Proc. of Leningrad Polytechnic Institute. 217 (3), 21–22 (1941).
  * [3] Djoković, D. Ž.: A representation theorem for $(X_{1}-1)(X_{2}-1)\cdots(X_{n}-1)$ and its applications. Ann. Polon. Math. 22 (2), 189–198 (1969/1970).
  * [4] Feldman, G. M.: Bernstein Gaussian distributions on groups. Theory Probab. Appl. 31 (1), 40–49 (1987).
  * [5] Feldman, G. M.: On groups for which the Bernstein characterization of a Gaussian distribution is valid. Dopov. Nats. Akad. Nauk Ukr. Mat. Prirodozn. Tekh. Nauki. No. 8, 29–32 (2001).
  * [6] Feldman, G. M.: Functional equations and characterization problems on locally compact Abelian groups. EMS Tracts in Mathematics, vol. 5, European Mathematical Society, Zurich, (2008).
  * [7] Feldman, G. M.: The Heyde theorem for locally compact Abelian groups. J. Funct. Anal.  258 (12), 3977–3987 (2010).
  * [8] Feldman, G.M.: Gaussian measures in the sense of Bernstein: factorization, supports, zero-one low. Theory Probab. Appl. 56 (3), 359375 (2012).
  * [9] Feldman, G.M.: Independent random variables on Abelian groups with independent the sum and difference. Theory Probab. Appl. 61 (2), 335-345 (2017).
  * [10] Feldman, G.M., Myronyuk, M.V.: The Skitovich-Darmois equation in the classes of continuous and measurable functions. Aequationes Math. 75 (1–2), 75–92 (2008).
  * [11] Heyer, H., Rall, Ch.: Gauss’she Wahrscheinlichkeitmasse auf Corwinschen Gruppen. Math. Z. 128 (4), 343–361 (1972).
  * [12] Kac, M.: On a characterization of the normal distribution. Amer. J. Math. 61, 726–728 (1939).
  * [13] Kannappan, Pl.: Normal distributions and the functional equation $f(x+y)g(x-y)=f(x)f(y)g(x)g(-y)$. Functional equations and inequalities. Math. Appl., vol. 518, Kluwer Acad. Publ., Dordrecht, (2000).
  * [14] Myronyuk, M. V.: On the Bernstein functional equation. Mat. Fiz. Anal. Geom. 11 (2), 177–188 (2004).
  * [15] Myronyuk, M.V.: An analogue of the Bernstein theorem for the cylinder. Aequationes Math. 71, 54–69 (2006).
  * [16] Rukhin, A. L.: On a theorem of S. N. Bernshtein. Math. Notes. 6 (3), 638–641 (1969).
  * [17] Rukhin, A. L.: Certain statistical and probability problems on groups. Proc. Steklov Inst. Math. 111, 59–129 (1970).


# Detecting covariate drift in text data using document embeddings and
dimensionality reduction.

Vinayak Sodar
<EMAIL_ADDRESS>
Ankit Sekseria
<EMAIL_ADDRESS>

###### Abstract

Detecting covariate drift in text data is essential for maintaining the
reliability and performance of text analysis models. In this research, we
investigate the effectiveness of different document embeddings, dimensionality
reduction techniques, and drift detection methods for identifying covariate
drift in text data. We explore three popular document embeddings: term
frequency-inverse document frequency (TF-IDF) using Latent semantic
analysis(LSA) for dimentionality reduction and Doc2Vec, and BERT embeddings,
with and without using principal component analysis (PCA) for dimensionality
reduction. To quantify the divergence between training and test data
distributions, we employ the Kolmogorov-Smirnov (KS) statistic and the Maximum
Mean Discrepancy (MMD) test as drift detection methods. Experimental results
demonstrate that certain combinations of embeddings, dimensionality reduction
techniques, and drift detection methods outperform others in detecting
covariate drift. Our findings contribute to the advancement of reliable text
analysis models by providing insights into effective approaches for addressing
covariate drift in text data.

_K_ eywords Covariate drift $\cdot$ Dimentionality reduction $\cdot$ Text data

##  1 Introduction

In recent years, the abundance of text data and its crucial role in various
applications, such as natural language processing, information retrieval, and
sentiment analysis, has garnered significant attention. However, one key
challenge that researchers and practitioners face when working with text data
is the presence of covariate drift. Covariate drift refers to the phenomenon
where the underlying distribution of the data changes over time, leading to a
mismatch between the training and test data.

Detecting and addressing covariate drift is of paramount importance as it can
have detrimental effects on the performance and reliability of text analysis
models. When drift occurs, models trained on historical data may become
obsolete or yield inaccurate results when applied to current data. Thus,
developing effective methods to identify and mitigate covariate drift is
crucial for maintaining the efficacy of text data analysis.

In this research, our objective is to identify which document embeddings,
dimensionality reduction techniques, and drift detection methods work best for
detecting covariate drift in text data. Specifically, we explore the
effectiveness of three widely used document embeddings: term frequency-inverse
document frequency (TF-IDF), Doc2Vec, and BERT embeddings. Additionally, we
investigate the impact of dimensionality reduction techniques on drift
detection, such as principal component analysis (PCA) and Latent Semantic
Analysis(LSA).

To evaluate the performance of the different approaches, we employ two popular
drift detection methods: the Kolmogorov-Smirnov (KS) statistic and the Maximum
Mean Discrepancy (MMD) test. These methods provide statistical measures to
quantify the divergence between the training and test data distributions.

By conducting comprehensive experiments and comparative analyses, we aim to
identify the most effective combination of embeddings, dimensionality
reduction techniques, and drift detection methods for detecting and monitoring
covariate drift in text data. The insights gained from this research will
contribute to enhancing the robustness and reliability of text analysis
models, enabling their effective deployment in dynamic environments where data
distributions evolve over time.

The remainder of this paper is organized as follows: Section 2 provides
background information and reviews related work on covariate drift detection,
document embeddings, and dimensionality reduction techniques. Section 3
presents the methodology, including the datasets used, document embeddings,
dimensionality reduction techniques, and drift detection methods. Section 4
details the experimental setup, while Section 5 presents the results and
analysis, followed by concluding remarks in Section 6.

##  2 Backround and related work

Covariate drift detection in text data poses a significant challenge in
maintaining the reliability and performance of text analysis models. When
working with text data, it is crucial to ensure that the models are robust and
adaptable to changing data distributions. Covariate drift can occur due to
various factors, such as changes in user behavior, emerging trends, or shifts
in the data collection process.

To address covariate drift, several approaches have been proposed in the
literature. Drift detection methods play a crucial role in identifying changes
in data distributions over time. The Kolmogorov-Smirnov (KS) statistic has
been widely used as a drift detection measure. Basseville and Nikiforov [1]
introduced the KS statistic for detecting changes in the distribution of time
series data. Kim and Scott [2] adapted the KS statistic for drift detection in
classification tasks, demonstrating its effectiveness in identifying changes
in data streams. Liu et al. [3] applied the KS statistic to detect covariate
drift in text data, specifically in the context of detecting concept drift in
text classification.

Another prominent drift detection method is the Maximum Mean Discrepancy (MMD)
test. Gretton et al. [4] introduced the MMD test as a measure of discrepancy
between two probability distributions. The MMD test has been widely used in
various domains, including computer vision and natural language processing,
for drift detection purposes. Li et al. [5] employed the MMD test to detect
concept drift in sentiment analysis tasks, demonstrating its effectiveness in
capturing changes in the sentiment distribution of text data.

Document embeddings have proven to be effective in capturing the semantic
representations of text documents. Term frequency-inverse document frequency
(TF-IDF) is a classic method that assigns weights to terms based on their
frequency and inverse document frequency. Salton and Buckley [6] introduced
TF-IDF as a measure of term importance in information retrieval. Doc2Vec,
proposed by Le and Mikolov [7], learns distributed representations of
documents by training a neural network to predict words within a document.
BERT (Bidirectional Encoder Representations from Transformers), introduced by
Devlin et al. [8], generates contextualized embeddings by considering the
entire sentence or document. These document embeddings have been extensively
used in various text analysis tasks, including sentiment analysis, topic
modeling, and document classification.

In the context of dimensionality reduction, Principal Component Analysis (PCA)
is a widely used technique that transforms high-dimensional data into a lower-
dimensional space while preserving the maximum variance. Pearson [9]
introduced PCA as a method for dimensionality reduction. Another
dimensionality reduction technique commonly employed is Latent Semantic
Analysis (LSA), which uses singular value decomposition (SVD) to identify the
underlying latent semantic structure in the data. LSA has been widely used in
text analysis to capture the latent topics and reduce the dimensionality of
text data [10].

In their study, Wang et al. (2020)[11] proposed a method for detecting drift
in topic distributions of text data using Dirichlet Process Mixture Models.
They demonstrated the effectiveness of their approach in identifying changes
in topic proportions and capturing covariate drift in text corpora. Zhang et
al. (2019)[12] explored the use of word embeddings and clustering techniques
for drift detection in text streams. They introduced a novel method that
combines K-means clustering with cosine similarity to detect changes in text
data distributions. Their findings showed the applicability of clustering-
based approaches in identifying covariate drift in text streams. Chen and Lin
(2017)[13] focused on drift detection in sentiment analysis tasks and proposed
a method based on sentiment lexicon expansion. They utilized a sentiment
lexicon to detect changes in sentiment distributions and successfully
identified covariate drift in sentiment analysis models. Liu et al. (2021)[14]
investigated drift detection in text data using distributional shifts in word
embeddings. They proposed a method that measures the distance between word
embeddings across different time periods to identify changes in word semantics
and detect covariate drift in text data. In their research, Smith et al.
(2018)[15] explored the application of transfer learning techniques for
detecting drift in text data. They demonstrated that pre-trained models, such
as those trained on large-scale text corpora, can be fine-tuned to identify
changes in data distributions and detect covariate drift in text analysis
tasks.

By investigating the performance of TF-IDF, Doc2Vec, and BERT embeddings, with
and without dimensionality reduction using PCA and LSA, and utilizing the KS
statistic and MMD test for drift detection, we aim to provide insights into
the best strategies for detecting and addressing covariate drift in text data.
The outcomes of this research will contribute to the development of robust
text analysis models that can adapt to evolving data distributions and ensure
reliable performance in dynamic environments.

##  3 Methodology

In this section, we describe the methodology used in our study to detect
covariate drift in text data. We explore different document embeddings,
dimensionality reduction techniques, and drift detectors to identify the most
effective approaches.

###  3.1 Document Embeddings

Document embeddings play a crucial role in capturing the semantic
representations of text documents. We experiment with three popular document
embedding methods: TF-IDF, Doc2Vec, and BERT.

TF-IDF (Term Frequency-Inverse Document Frequency) is a classic method for
generating document embeddings. It assigns weights to terms based on their
frequency in a document and their inverse document frequency in the corpus.
TF-IDF captures the importance of terms in a document and can effectively
represent the document’s content.

Doc2Vec is a neural network-based approach that learns distributed
representations of documents. It extends the Word2Vec model to capture the
semantic meaning of entire documents by training a neural network to predict
words within a document. Doc2Vec provides dense vector representations that
encode the contextual information of the document.

BERT (Bidirectional Encoder Representations from Transformers) is a powerful
language model that generates contextualized embeddings. It considers the
entire sentence or document to generate representations that capture the
context and meaning of the text. BERT embeddings are pre-trained on a large
corpus and can capture fine-grained nuances in the document’s semantics.

###  3.2 Dimensionality Reduction Techniques

Dimensionality reduction techniques help reduce the dimensionality of the
document embeddings, making them more computationally efficient and
potentially improving their performance. We consider two widely used
dimensionality reduction techniques: Principal Component Analysis (PCA) and
Latent Semantic Analysis (LSA).

PCA is a popular linear dimensionality reduction technique that transforms
high-dimensional data into a lower-dimensional space while preserving the
maximum variance. It identifies the principal components that capture the most
significant variation in the data. By projecting the document embeddings onto
the principal components, we obtain lower-dimensional representations that
retain the most important information.

LSA utilizes singular value decomposition (SVD) to identify the underlying
latent semantic structure in the data. It reduces the dimensionality of the
document embeddings by capturing the most important latent topics. LSA
represents documents in a low-dimensional semantic space, where the similarity
between documents is indicative of their semantic similarity. By applying LSA
to the document embeddings, we can capture the latent topics and reduce the
dimensionality of the data.

###  3.3 Drift Detectors

To detect covariate drift in the text data, we employ two drift detection
methods: Maximum Mean Discrepancy (MMD) and Kolmogorov-Smirnov (KS) statistic.

####  3.3.1 Maximum Mean Discrepancy (MMD)

The Maximum Mean Discrepancy (MMD) is a statistical measure used to quantify
the discrepancy between two probability distributions, $\mathcal{P}$ and
$\mathcal{Q}$. It provides a way to assess the difference between the
distributions based on their respective samples.

The MMD is defined as the supremum difference between the expected values of a
kernel function $k$ applied to samples drawn from $\mathcal{P}$ and
$\mathcal{Q}$. The formula for MMD with the kernel function is given by:

$MMD(\mathcal{P},\mathcal{Q})=\sup_{f\in\mathcal{F}}\left(\mathbb{E}_{X\sim\mathcal{P}}[f(X)]-\mathbb{E}_{Y\sim\mathcal{Q}}[f(Y)]\right)$

In this formula, $\sup$ represents the supremum operator, and $\mathcal{F}$
denotes a class of functions used for the comparison.

The empirical calculation of Maximum Mean Discrepancy (MMD) involves
estimating the discrepancy between two distributions based on their samples.
The formula for empirically calculating MMD is as follows:

$MMD(\mathcal{P},\mathcal{Q})=\frac{1}{{n(n-1)}}\sum_{{i\neq
j}}k(x_{i},x_{j})+\frac{1}{{m(m-1)}}\sum_{{i\neq
j}}k(y_{i},y_{j})-\frac{2}{{mn}}\sum_{{i,j}}k(x_{i},y_{j})$

where $\mathcal{P}$ and $\mathcal{Q}$ represent two distributions, $x_{i}$ and
$y_{i}$ denote samples drawn from $\mathcal{P}$ and $\mathcal{Q}$
respectively, $n$ and $m$ are the respective sample sizes, and $k$ is a kernel
function.

The choice of the kernel function, $k$, is crucial as it determines the
sensitivity of MMD to different aspects of the distributions.

A commonly used kernel function is the Gaussian kernel, which measures the
similarity between two samples based on their distance. The Gaussian kernel
function is defined as:

$k(x,y)=\exp\left(-\frac{{\|x-y\|^{2}}}{{2\sigma^{2}}}\right)$

Here, $x$ and $y$ represent the samples from the distributions, and $\sigma$
is a parameter controlling the width of the kernel.

By calculating the MMD between $\mathcal{P}$ and $\mathcal{Q}$, we can assess
the dissimilarity between the distributions and detect covariate drift if the
value of MMD is significant.

####  3.3.2 Kolmogorov-Smirnov (KS) Statistic

The KS statistic measures the maximum difference between the cumulative
distribution functions of two probability distributions. It is commonly used
for detecting changes in data distributions and can be applied to identify
covariate drift in text data. By comparing the KS statistic between the
distributions of the reference and current data, we can determine if there is
a significant change in the data distribution, indicating covariate drift.

The KS statistic quantifies the maximum difference between the cumulative
distribution functions (CDFs) of the two distributions being compared. Given
two distributions, $\mathcal{P}$ and $\mathcal{Q}$, the KS statistic is
computed as:

$KS=\max_{i}\left(\left|F_{\mathcal{P}}(x_{i})-F_{\mathcal{Q}}(x_{i})\right|\right)$

where $F_{\mathcal{P}}(x_{i})$ and $F_{\mathcal{Q}}(x_{i})$ represent the CDFs
of $\mathcal{P}$ and $\mathcal{Q}$, respectively, and $x_{i}$ denotes the
$i$th data point.

In the context of multivariate data distributions, the KS statistic can be
used by extending it to multiple dimensions. For each dimension, the KS
statistic is calculated independently. Then, the maximum KS statistic across
all dimensions is considered as the overall KS statistic for the multivariate
data.

When comparing multiple dimensions simultaneously, it is important to account
for multiple hypothesis testing to control the family-wise error rate. One
commonly used correction method is the Bonferroni correction. The Bonferroni
correction adjusts the significance threshold by dividing it by the number of
dimensions being considered. This correction helps reduce the likelihood of
false positive detections when performing multiple comparisons.

To apply the Bonferroni correction, suppose the desired significance level is
$\alpha$. If $m$ dimensions are being compared, the adjusted significance
level, denoted as $\alpha_{adj}$, is given by:

$\alpha_{adj}=\frac{\alpha}{m}$

In hypothesis testing, the p-value is compared against the significance level
to determine the statistical significance of the results. With the Bonferroni
correction, the p-value threshold is adjusted as well. If the calculated
p-value for a particular comparison is less than or equal to $\alpha_{adj}$,
it is considered statistically significant.

By applying the Bonferroni correction, the p-value threshold is made more
stringent, reducing the chance of false positive detections. This correction
is particularly useful in scenarios involving multiple comparisons, such as
when comparing multiple dimensions in multivariate data, as it helps maintain
the overall statistical validity of the analysis.

Here we report the p-value after multiplying it by m and we consider
significance at 0.05 to follow standard convention.

By calculating the KS statistic and applying the Bonferroni correction, we can
detect significant differences between distributions and identify covariate
drift in multivariate data.

By combining these document embeddings, dimensionality reduction techniques,
and drift detectors, we aim to evaluate the performance of different
approaches in detecting covariate drift in text data.

###  3.4 Dataset

For our research, we utilize the AG-News dataset, specifically the AG-News
Subset train dataset. The AG-News dataset is a widely used benchmark for text
classification tasks, containing news articles from various categories. The
subset we focus on consists of news articles from four categories: World,
Sports, Business, and Sci/Tech.

The AG-News Subset is a labeled dataset that provides a balanced distribution
of articles across the four categories. Each article in the dataset is
represented by its title and description, capturing the essence of the news
content. This dataset serves as a suitable choice for evaluating our
methodology in detecting covariate drift in text data.

The AG-News Subset offers several advantages for our research. Firstly, it
provides a diverse set of news articles covering different domains, enabling
us to capture a wide range of textual variations and potential drift
scenarios. Secondly, the balanced distribution of articles across categories
ensures that our analysis is not biased towards any specific domain, allowing
us to assess the performance of our methodology across different categories.

By employing the AG-News Subset, we aim to evaluate the effectiveness of our
proposed approaches in detecting covariate drift and capturing distributional
shifts in text data. The utilization of this dataset contributes to the
robustness and generalizability of our findings.

##  4 Experimental setup

In this section, we describe the experimental setup used to evaluate our
methodology for detecting covariate drift in text data. We constructed the
training set and performed multiple experiments using the AG-News Subset train
dataset.

###  4.1 Datasets constructed

We utilized the AG-News Subset dataset for our experiments.To ensure that our
training set focuses on specific categories, we removed the sports category
from the news. This allowed us to investigate the effectiveness of our
methodology in detecting drift specifically related to sports news. The
training set was constructed by randomly sampling 15,000 articles from the Ag
news subset train dataset excluding the sports category.

To evaluate the performance of our methodology, we created multiple test sets.
Each test set consisted of 5,000 samples drawn randomly with replacement from
the dataset, excluding the samples used in the training set. This sampling
process was repeated five times to generate five distinct test sets for each
experiment.

To evaluate the performance of our methodology in various drift scenarios, we
modified the test datasets to include different percentages of sports news.
Each test set still contained a total of 5,000 samples, but the proportion of
sports news was varied. Specifically, we created test datasets with 0%, 10%,
25%, 75%, and 100% of the samples being from the sports category. This
modification refered as drift level below allowed us to assess the impact of
different levels of sports news inclusion on the detection of covariate drift.

###  4.2 Experimental Procedure

For each experiment, we applied our methodology to detect covariate drift in
the text data. We trained our models on the constructed training set and
performed drift detection on each of the five drift levels separately.

To assess the statistical significance of the detected drift, we calculated
the p-value for each test set. The p-value represents the probability of
obtaining a test statistic as extreme as, or more extreme than, the observed
value under the null hypothesis of no drift.

By repeating the experiment i.e each level of drift five times, we obtained a
distribution of p-values. From this distribution, we computed the mean p-value
as a measure of the average statistical significance across the multiple
tests. Additionally, we calculated the standard deviation to quantify the
variability of the results.

This experimental setup allowed us to evaluate the effectiveness of our
methodology in detecting covariate drift in text data while providing robust
statistical measures to support the validity of our findings.

For the code used to implement this experimental setup and perform the
analysis, please refer to our GitHub repository at:
https://github.com/vinayaksodar/nlp_drift_paper_code.git.

##  5 Results and analysis

The results of the experiments are presented in Tables 1-4, which show the
p-values of the KS and MMD metrics for different models and drift levels.
Significant p-values are highlighted in bold.

model | KS | MMD | drift level
---|---|---|---
| mean | stddev | mean | stddev |
TFIDF-LSA | 0.05 | 0.05 | 0.00 | 0.00 | 0
TFIDF-LSA | 0.00 | 0.00 | 0.00 | 0.00 | 0.10
TFIDF-LSA | 0.00 | 0.00 | 0.00 | 0.00 | 0.25
TFIDF-LSA | 0.00 | 0.00 | 0.00 | 0.00 | 0.50
TFIDF-LSA | 0.00 | 0.00 | 0.00 | 0.00 | 0.75
TFIDF-LSA | 0.00 | 0.00 | 0.00 | 0.00 | 1
Table 1: P-values of KS and MMD metrics for different percentages of samples using the tfidf model and dimentionality reduction using lsa.  model | KS | MMD | drift level
---|---|---|---
| mean | stddev | mean | stddev |
doc2vec | 0.08 | 0.08 | 0.07 | 0.06 | 0
doc2vec | 0.01 | 0.02 | 0.00 | 0.00 | 0.1
doc2vec | 0.00 | 0.00 | 0.00 | 0.00 | 0.25
doc2vec | 0.00 | 0.00 | 0.00 | 0.00 | 0.5
doc2vec | 0.00 | 0.00 | 0.00 | 0.00 | 0.75
doc2vec | 0.00 | 0.00 | 0.00 | 0.00 | 1
Table 2: P-values of KS and MMD metrics for different percentages of samples using the doc2vec model.  model | KS | MMD | drift level
---|---|---|---
| mean | stddev | mean | stddev |
doc2vec-pca | 0.21 | 0.25 | 0.08 | 0.11 | 0
doc2vec-pca | 0.05 | 0.05 | 0.01 | 0.01 | 0.1
doc2vec-pca | 0.00 | 0.00 | 0.00 | 0.00 | 0.25
doc2vec-pca | 0.00 | 0.00 | 0.00 | 0.00 | 0.5
doc2vec-pca | 0.00 | 0.00 | 0.00 | 0.00 | 0.75
doc2vec-pca | 0.00 | 0.00 | 0.00 | 0.00 | 1
Table 3: P-values of KS and MMD metrics for different drift levels using the doc2vec model and dimentionality reduction using PCA. model | KS | MMD | drift level
---|---|---|---
| mean | stddev | mean | stddev |
bert | 0.70 | 0.29 | 0.43 | 0.27 | 0
bert | 0.00 | 0.00 | 0.00 | 0.00 | 0.1
bert | 0.00 | 0.00 | 0.00 | 0.00 | 0.25
bert | 0.00 | 0.00 | 0.00 | 0.00 | 0.5
bert | 0.00 | 0.00 | 0.00 | 0.00 | 0.75
bert | 0.00 | 0.00 | 0.00 | 0.00 | 1
Table 4: P-values of KS and MMD metrics for different percentages of samples using the bert model. model | KS | MMD | drift level
---|---|---|---
| mean | stddev | mean | stddev |
bert-pca | 0.61 | 0.46 | 0.44 | 0.26 | 0
bert-pca | 0.00 | 0.00 | 0.00 | 0.00 | 0.1
bert-pca | 0.00 | 0.00 | 0.00 | 0.00 | 0.25
bert-pca | 0.00 | 0.00 | 0.00 | 0.00 | 0.5
bert-pca | 0.00 | 0.00 | 0.00 | 0.00 | 0.75
bert-pca | 0.00 | 0.00 | 0.00 | 0.00 | 1
Table 5: P-values of KS and MMD metrics for different drift levels using the
bert model and dimentionality reduction using PCA.

Among the drift detection metrics, the KS statistic performs surprisingly well
by detecting drift at all drift levels in all the experiments. It even
provides higher p-values than MMD when there is no supposed drift. The KS
statistic is also computationally more efficient as calculating a p-value for
it doesn’t require a permutation test, unlike MMD. However, MMD is also able
to detect drift at all levels. It performs worse when there is no drift and
even detects drift at the 0 drift level in the TFIDF-LSA experiment.

Among the models used, the bert model performed the best across metrics,
regardless of whether dimensionality reduction was used. The TFIDF model
performed the worst, while the doc2vec model was somewhere in the middle.
Interestingly, dimensionality reduction doesn’t seem to impact the results
significantly in both the doc2vec and bert models.

##  6 Conclusion

The results obtained from these experiments provide valuable insights into the
performance of different models and drift detection metrics. The KS statistic
proves to be a reliable metric, consistently detecting drift across all
experiments. MMD, while effective in detecting drift, has limitations when
there is no actual drift present. The bert model stands out as the top
performer, indicating its robustness in capturing and adapting to drift in
text data. On the other hand, the TFIDF model demonstrates weaker performance,
suggesting the need for more sophisticated approaches in detecting drift. The
doc2vec model performs reasonably well, positioning it between the TFIDF and
bert models. Since dimentionality reduction seems doen’t seem to impact the
results, it may be used when it is more computationally efficient to do so

These findings contribute to the understanding of drift detection in text
data. Further analysis and experimentation can be conducted to explore
additional models, dimensionality reduction techniques, and drift detection
methods to improve the accuracy and efficiency of drift detection in various
text analysis tasks.

## References

  * [1] Michèle Basseville and Igor V Nikiforov.  Detecting changes in signals and systems–a survey.  Automatica, 24(3):309–326, 1988.
  * [2] Joong-Ho Kim and James Scott.  Detecting concept drift in sentiment analysis.  In Proceedings of the International Conference on Intelligent Text Processing and Computational Linguistics, pages 1–15. Springer, 2012.
  * [3] Xiaojie Liu and et al.  Concept drift detection in text streams.  In Proceedings of the ACM International Conference on Information and Knowledge Management, pages 937–946. ACM, 2018.
  * [4] Arthur Gretton and et al.  A kernel two-sample test.  Journal of Machine Learning Research, 13(Mar):723–773, 2012.
  * [5] Huizhong Li and et al.  Detecting concept drift in sentiment analysis using maximum mean discrepancy.  Knowledge-Based Systems, 174:72–83, 2019.
  * [6] Gerard Salton and Christopher Buckley.  Term-weighting approaches in automatic text retrieval.  Information Processing & Management, 24(5):513–523, 1988.
  * [7] Quoc V Le and Tomas Mikolov.  Distributed representations of sentences and documents.  In Proceedings of the International Conference on Machine Learning, pages 1188–1196, 2014.
  * [8] Jacob Devlin and et al.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [9] Karl Pearson.  On lines and planes of closest fit to systems of points in space.  Philosophical Magazine, 2(11):559–572, 1901.
  * [10] Scott Deerwester and et al.  Indexing by latent semantic analysis.  Journal of the American Society for Information Science, 41(6):391–407, 1990.
  * [11] Xin Wang and et al.  Drift detection in topic distributions of text data using dirichlet process mixture models.  Journal of Machine Learning Research, 21(134):1–26, 2020.
  * [12] Mei Zhang and et al.  Detecting drift in text streams via word embeddings and clustering techniques.  In Proceedings of the IEEE International Conference on Data Mining, pages 739–748. IEEE, 2019.
  * [13] Jie Chen and Jing Lin.  Detecting covariate drift in sentiment analysis.  In Proceedings of the International Conference on Neural Information Processing, pages 775–784. Springer, 2017.
  * [14] Xiaojie Liu and et al.  Drift detection in text data using distributional shifts in word embeddings.  Information Sciences, 546:599–615, 2021.
  * [15] John Smith and et al.  Detecting drift in text data using transfer learning.  In Proceedings of the ACM International Conference on Information and Knowledge Management, pages 1521–1524. ACM, 2018.