# Tridiagonal Toeplitz Matrices and Bipartite Quantum Correlations

Varsha S. Sambhaje1, Suprabhat Sinha2, Anju Chaurasia3, Kapil K. Sharma⋆4
(1,2,3,4)D Y Patil International University,
Akurdi, Pune, Maharashtra 411044, India
E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this article, we focus on tridiagonal Toeplitz Hermitian matrices, which
fulfill the requirement of a valid Hamiltonian often used in Quantum
Information. We investigate the behavior of such matrices to pursue the
dynamics of quantum correlations (entanglement and quantum discord) for
bipartite Werner state and maximally entangled mixed states. We have found
interesting results that the main diagonal terms in the Toeplitz matrices
never affect the quantum correlations in both quantum states. However, super-
diagonal and sub-diagonal terms play the important role in the dynamics. We
investigate the phenomenon of entanglement sudden death and also observe the
presence of quantum discord in the absence of entanglement. Most importantly
it is found that MEMS is more sensitive in comparison to the Werner state.

Toeplitz matrix, QITE, Werner state, MEMS, Lanczos algorithm

##  I Introduction

Quantum correlations are the backbone of quantum computation and information;
their dynamical study is central to this discipline [1]. To execute many
quantum applications like quantum teleportation, super-dense coding, quantum
image processing, quantum machine learning, and quantum algorithms [2, 3, 4,
5, 6, 7]; the quantum correlations are primary requirement. There are several
kinds of quantum correlations but here we are focusing on concurrence and
quantum discord [8, 9]. To pursue the dynamical study of quantum correlations,
we need mathematical tools for their detection and quantification [10, 11]. We
have enough mathematical tools to quantify and detect such quantum
correlations in bipartite Hilbert space but in higher dimensions, (i.e
$d\otimes d)$; the development of such tools is still a challenging
mathematical problem. It is always an essential requirement to maintain the
sustainability of quantum correlations in a physical system for a long time.
However, these correlations are very much sensitive and entanglement often
passes through sudden death as time evolves; this phenomenon is called
entanglement sudden death (ESD) [12]. It has been observed that in the absence
of entanglement, quantum discord has existence but there are limited studies
to use quantum discord for quantum applications in a practical scenario.
Calculating quantum discord is an NP-complete problem [13]. A bipartite system
is the primary one, which can be the subsystem of a larger system. In the
context of quantum application development, if ESD takes place then no quantum
application will produce the result. Therefore it is always interesting to
investigate ESD in a physical system under certain conditions. ESD also has
been observed in varieties of quantum states under Dzyaloshinsky-Moriya
interactions [14, 15, 16, 17, 18].

Toeplitz matrices have significant applications in classical as well as in
quantum domains [19]. Toeplitz operators in quantum mechanics are known for a
long time in literature and their special properties have been studied in all
kinds of Hilbert spaces such as Banach, Hardy, and Fock spaces, etc. [20, 21,
22]. The theory of the Toeplitz determinant is developed in 1967, which is
used in calculating the spin correlations in the two-dimensional Ising model
[23, 24]. In 2017 the properties of these matrices have been studied in Long-
Range Kitaev spin chains to investigate topological phase transition [25].
Such metrics are around in quantum mechanics at a larger scale although in
quantum chemistry [26]. Often if the Hamiltonian of a physical system is a
large dimensional sparse Hermitian matrix then such Hamiltonian is mapped to
tridiagonal Toeplitz matrices by using the Lanczos algorithm to simplify the
calculations [27, 28, 29]. The Lanczos algorithm is numerically stable if the
elements of a given Hamiltonian are non-floating numbers [30].

From a quantum algorithms perspective, recently these matrices have had a
prominent role in the development of the quantum imaginary time evolution
(QITE) model to do progress in quantum algorithms [31, 32, 33, 34]. In QITE we
deal with a larger dimensional k-local many-body Hamiltonian i.e.
$(H=\sum_{i}^{n}h_{i}\in\mathbb{B}((\mathbb{C}^{d})^{\otimes n})$, where each
$h_{i}$ is k-local. This Hamiltonian is used to find out the quantum imaginary
time evolution by assuming time as an imaginary component [35]. The time
evolution operator is given as $e^{iHt}$ which is non-unitary. Further, this
non-unitary operator is approximated by choosing the appropriate k-Local
Hamiltonian $H$ of the system and mapped it to a unitary operator by using
Trotter expansion with errors [31]. The error is reduced by considering the
nearest neighbour interactions by selecting the proper k-local Hamiltonian.
This scheme provides a better convergence time of the expectation value of
energy in comparison to the variational classical-quantum approach [36].
However, selecting the better k-Local Hamilton with less amount of error is an
art and there is no perfect method known for the same. In general, the
Hamiltonian matrix considered for QITE has a very large dimension, so in this
case, one need to map the given matrix onto tridiagonal Toeplitz form by using
recently proposed Quantum Lanczos algorithms [31], which maps a k-local
Hamiltonian to Toeplitz matrices.

Recently Farenick et al. developed the Toeplitz operator structure over the
kronecker product of bipartite Hilbert space by using the $C^{*}$ algebra;
further, they have studied entanglement in Toeplitz matrices [37, 38].
Following the literature on Toeplitz matrices, it is very important to
investigate the dynamical behavior of quantum correlations under such
matrices. Here we mention that the impact of tridiagonal Toeplitz matrices on
different quantum states is missing in quantum information. Hence we plan to
investigate the role of tridiagonal Toeplitz (special case) matrices to study
the dynamics of quantum correlations in bipartite Werner state and maximally
entangled mixed states (MEMS) [39, 40]. In the present study we consider the
specific form of such matrices with the elements as exponential functions of
the parameters i.e. $(a^{n},b^{n})$, the reason to choose such elements is
that by putting the parameter values $(a,b,n)$, different Hamiltonian can be
produced with numerical entries, hence it generalize the results.

The paper is organized as follows, in section II, we present the tridiagonal
Toeplitz Hamiltonian with its eigenvalues and eigenvectors. Also, we discussed
the unitary time evolution density matrix. Section III gives information on
initial quantum states. In section IV, we provide the quantum correlation
measures. Further, section V is dealing with the eigenvalue spectrum and
density matrices for quantum dynamics. In section VI, we discuss the results
based on the dynamical behavior of the Werner state and MEMS respectively. In
the last section VII, we provide the conclusion.

##  II Toeplitz matrices and unitary time evolution

In this section, we present the Hamiltonian of the physical system, which has
the structure of a tridiagonal Toeplitz matrix with ($H=H^{\dagger}$). This
matrix has three diagonals namely main diagonal, super-diagonal, and sub-
diagonal. The reader can go through the formation of the tridiagonal Toeplitz
operator with the references therein [38]. The structure of the matrix
considered for the present study is given below,

$H_{n}=\begin{pmatrix}a^{n}&b^{n}&0&0\\\ b^{n}&a^{n}&b^{n}&0\\\
0&b^{n}&a^{n}&b^{n}\\\ 0&0&b^{n}&a^{n}\end{pmatrix},$ (1)

where $(a,b,n)\in R$. The eigenvalues and eigenvectors of $H$ are given below,

$\lambda_{j}=b^{n}+2a^{n}\sqrt{\frac{b^{n}}{a^{n}}}\cos{\frac{j\pi}{m+1}},\quad\quad
with\quad(j,m)=(1,2,3,4)$ (2)

and

$x_{j}=\left(\frac{b^{n}}{a^{n}}\right)^{\frac{m}{2}}\sin\left(\frac{{mj\pi}}{m+1}\right),\quad\quad
with\quad(j,m)=(1,2,3,4)$ (3)

where $\lambda_{j}$ are the eigenvalues and $x_{j}$ are the corresponding
eigenvectors.

In general, the entries of a Hamiltonian give information about the
interaction strength in a physical system. We are intended to look at how fast
the dynamics of quantum correlations change as the interaction strength in
$H_{n}$ grows exponentially; hence we recall the reason that we choose the
entries of the $H_{n}$ as $(a^{n},b^{n})$.

According to the postulates of quantum mechanics [18], the time-dependent
Schrödinger equation shown below governs the unitary dynamics of the physical
system,

$i\hbar\frac{d}{dt}|\psi(t)\rangle=H|\psi(t)\rangle,$ (4)

where $H$ is the Hamiltonian of the physical system with ($H=H^{\dagger}$).
The solution of this equation is obtained as,

$|\psi(t)\rangle=e^{\frac{-iHt}{\hbar}}|\psi(0)\rangle.$ (5)

Using the above equation, we can obtain the time evolution density matrix as
follows,

$\rho(t)=U(t).\rho(0).U(t)^{\dagger},$ (6)

where $U(t)=e^{\frac{-iHt}{\hbar}}$ is the unitary matrix that has the
Hamiltonian $H$ in the exponential. Here we assume ($\hbar=1$) to maintain the
simplicity of the current study. The Eq.$(\ref{tm1})$ is used in current work
to develop the dynamics of quantum correlations in Werner state and MEMS.

We prepare the initial state of the bipartite system in Werner state and MEMS
respectively. The density matrices of these quantum states are formed by the
two qubits provided in the next section.

##  III Initial quantum states

In this section, we present the mathematical structure for the Werner state
[39] and MEMS [40] with their corresponding density matrices. Both states are
parametrized; hence their quantum correlations are the functions of parameters
involved in these states. The respective quantum states are described in
successive subsections.

###  III.1 Werner state

The Werner state is invariant under unitary transformation in $d-$dimensional
Hilbert space. This property reads as below,

$\rho=(U\otimes U)\rho(U^{\dagger}\otimes U^{\dagger}),$ (7)

where $\rho$ denotes the density matrix of the system. The Werner state adopt
the following form while dealing with two qubits,

$\rho^{WS}=\gamma|\psi^{-}\rangle\langle\psi^{-}|+(1-\gamma)\frac{I}{4}\quad
with\quad(0\leq\gamma\leq 1),$ (8)

where $|\psi^{-}\rangle$ indicates a singlet state and I is the ($4\times 4$)
dimensional identity matrix. The matrix form is expanded below,

$\rho^{WS}=\left(\begin{array}[]{cccc}\frac{1-\gamma}{4}&0&0&0\\\
0&\frac{\gamma+1}{4}&-\frac{\gamma}{2}&0\\\
0&-\frac{\gamma}{2}&\frac{\gamma+1}{4}&0\\\ 0&0&0&\frac{1-\gamma}{4}\\\
\end{array}\right)$ (9)

###  III.2 Maximally entangled mixed states (MEMS)

Another bipartite quantum state that is closely related to the bipartite
Werner state is MEMS. The two qubits MEMS, which is empirically proven as well
as being more entangled than the two qubits Werner state in terms of
concurrence measure [41, 42], the state is prepared by Munro et al. [40]. Two-
qubits bipartite MEMS’ density matrix can be expressed as,

$\rho^{MEMS}=\left[\begin{array}[]{cccc}g(\gamma)&0&0&\frac{\gamma}{2}\\\
0&1-2g(\gamma)&0&0\\\ 0&0&0&0\\\
\frac{\gamma}{2}&0&0&g(\gamma)\end{array}\right].$ (10)

The above state incorporates a function $g(\gamma)$ with following conditions,

$g(\gamma)=\delta=\left\\{\begin{array}[]{cc}\frac{1}{3},&\quad
0\leq\gamma<\frac{2}{3}\\\ \frac{\gamma}{2},&\quad\frac{2}{3}\leq\gamma\leq
1\end{array}\right..$ (11)

For the sake of convenience, we will refer to $g(\gamma)$ as $\delta$
throughout this study.

To proceed with the dynamical study in specified quantum states, we need
theoretical measures to quantify the entanglement and quantum discord. These
measures are given in the next section.

##  IV Measures of quantum correlations

The quantum correlation will provide information about the entanglement of the
system. In this section we explore the quantum correlation of the system with
the help of two important measures of quantum correlations viz., concurrence
and quantum discord [9].

###  IV.1 Concurrence

Concurrence is one of the quantum correlation measure that is widely used to
quantify the entanglement of bipartite quantum systems [41, 42]. Concurrence
of a bipartite quantum system governed by the corresponding density matrix
$\rho$ can be defined as,

$C(\rho)=max\,\\{0,\lambda_{1}-\lambda_{2}-\lambda_{3}-\lambda_{4}\\},$ (12)

where $\lambda_{i}$’s are the decreasing order square root of eigenvalues of
$\rho\tilde{\rho}$ and

$\tilde{\rho}=(\sigma_{y}\otimes\sigma_{y})\rho^{*}(\sigma_{y}\otimes\sigma_{y}),$
(13)

where $\tilde{\rho}$ is the result of the spin-flip operation applied on
density matrix $\rho$. Here $\sigma_{y}$ is the Pauli Y matrix and $\rho^{*}$
is the complex conjugate of the density matrix $\rho$.

###  IV.2 Quantum discord

Quantum discord is another measure of quantum correlations which characterized
mathematically in terms of quantum mutual information. In the classical domain
the mutual information between two random variables $[X,Y]$ can be expressed
in equations given below,

$\displaystyle I(X;Y)=H(X)+H(Y)-H(X,Y),$ (14) $\displaystyle
I(X;Y)=H(X)-H(X|Y).$ (15)

Here we mention that both the above equations are different in the quantum
domain as they depends on quantum measurements. Considering a bipartite
composite density matrix $\rho$, incorporating $\rho_{A}$ and $\rho_{B}$, we
can express the quantum version of the above equations as follows,

$\displaystyle I(\rho)=S(\rho_{A})+S(\rho_{B})-S(\rho)$ (16) $\displaystyle
I(\rho)=S(\rho_{A})-S(\rho_{A}|\rho_{B})$ (17)

The term $S(\rho_{A}|\rho_{B})$ is obtained by performing the quantum
measurement on either sub-system $A$ or $B$ of the composite system $AB$. It
is important to note that quantum discord is not symmetric with respect to the
measurement performed. Here we consider that the measurement $M$ is performed
on subsystem $A$, so in this direction the term $I(\rho|M)$ is obtained as,

$I(\rho|M)=S(\rho_{B})-S(\rho|M)).$ (18)

We have to maximize the term $I(\rho|M)$ over all possible measurements, and
the alternate form of the expression $I(\rho|M)$ can be given by,

$C(\rho)=S(\rho_{B})-\text{min}_{M}S(\rho|M).$ (19)

The difference between the expressions $I(\rho)$ and $C(\rho)$ is called
quantum discord, which is given below,

$Q(\rho)=I(\rho)-C(\rho)=S(\rho_{A})-S(\rho)+\text{min}_{M}S(\rho|M).$ (20)

##  V Eigenvalue spectrum and density matrix

We proceed to develop the dynamical form Eq.(6) by using the Hamiltonian given
in Eq.(1). This dynamical equation is explored with ($\rho(0)=\rho^{WS}$) and
($\rho(0)=\rho^{MEMS}$) respectively. For Werner state the eigenvalue spectrum
of $\rho^{W}(t)$ is obtained as,

$\left\\{\frac{1-\gamma}{4},\frac{1-\gamma}{4},\frac{1-\gamma}{4},\frac{1}{4}(3\gamma+1)\right\\}$
(21)

We observe that the above eigenvalue spectrum is free from all the Hamiltonian
parameters $(a,b,n,t)$, hence the concurrence and quantum discord are never
affected in Werner state from these parameters.

We have found that the eigenvalue spectrum $\rho(t)$ of MEMS is very
complicated to calculate analytically as it is a function of the parameters
$(a,b,n,\gamma,t)$. Further, we pursue the numerical study and presented the
density matrix $\rho^{MEMS}(t)$ as below.

$\left(\begin{array}[]{cccc}\frac{a_{1}+b_{1}}{5}&\frac{\phi_{6}P_{1}}{20}&e_{7}[n_{3}+n_{4}+n_{5}]&\frac{c+d}{10}\\\
\\\
e_{6}[m_{1}\theta_{1}+n_{1}\theta_{2}+m_{3}\phi_{4}]&\frac{-d_{1}[e_{8}P_{3}+P_{4}]}{10}&e_{8}[P_{5}+P_{6}+P_{7}]&e_{6}m_{5}\\\
\\\
e_{6}[n_{1}\theta_{1}-m_{1}\theta_{2}+n_{2}\phi_{4}]&e_{8}[P_{6}-P_{5}-P_{7}]&e_{8}[P_{5}-m_{4}+P_{8}]&e_{6}m_{5}\\\
\\\
\frac{c-d}{10}&\frac{\phi_{6}P_{2}}{5}&e_{7}[n_{3}-n_{4}+n_{6}]&\frac{a_{1}-b_{1}}{5}\\\
\\\ \end{array}\right)$ (22)

where, the terms of matrix are defined as follows,

$\displaystyle a_{1}=1+\delta+d_{2}\phi_{3},\quad b_{1}=d_{1}\phi_{1}(-1+\phi
3),\quad c=\gamma(3+2\phi_{3}),\quad d=2id_{1}\phi_{2}(-1+\phi_{3}),$
$\displaystyle
P_{1}=-(-1+\sqrt{5})e^{\frac{ie_{3}}{2}}(\gamma-2\delta)\phi_{5}+e^{\frac{-ie_{3}}{2}}[-(-1+\sqrt{5}+e_{1}+\sqrt{5}e_{1})\gamma\phi_{5}+2(1-\sqrt{5}+e_{2}+\sqrt{5}e_{2})\delta\phi_{5}]$
$\displaystyle-4\phi_{4}(\gamma\phi_{4}+2i(-1+\delta)\phi_{5})\phi_{6}+4\sqrt{5}\phi_{4}(2i(-1+3\delta)\phi_{4}+\gamma\phi_{5})\phi_{7}$
$\displaystyle P_{2}$
$\displaystyle=(-d_{1}+d_{1}\phi_{1})\phi_{6}+\sqrt{5}(i\gamma+d_{1}\phi_{2})\phi_{7},$
$\displaystyle P_{3}$ $\displaystyle=[h_{1}e_{3}+h_{1}-h_{2}e_{1}]^{2},$
$\displaystyle P_{4}$
$\displaystyle=\frac{\delta}{5}[2-e^{-i\sqrt{5}b^{n}t}+e^{i\sqrt{5}b^{n}t}],$
$\displaystyle P_{5}$
$\displaystyle=d_{1}h_{3}-d_{1}e^{2ie_{3}}h_{3}+d_{1}e^{2ib^{n}h_{4}},$
$\displaystyle P_{6}$
$\displaystyle=d_{1}e^{2i\sqrt{5}b^{n}t}h_{4}+8e^{ie_{3}}\gamma-4e^{ie_{4}}\gamma-4e^{ib^{n}t}\gamma,$
$\displaystyle P_{7}$
$\displaystyle=4d_{1}e^{i\sqrt{5}b^{n}t}-4d_{1}e^{ie_{5}},$ $\displaystyle
P_{8}$ $\displaystyle=4d_{1}e^{ie_{5}}+4d_{1}e^{i\sqrt{5}b^{n}t},$
$\displaystyle e_{1}$ $\displaystyle=e^{ib^{n}t}+e^{i\sqrt{5}b^{n}t},$
$\displaystyle e_{2}$ $\displaystyle=e^{ib^{n}t}-e^{\sqrt{5}b^{n}t},$
$\displaystyle e_{3}$ $\displaystyle=(1+\sqrt{5})b^{n}t,$ $\displaystyle
e_{4}$ $\displaystyle=(1+2\sqrt{5})b^{n}t,$ $\displaystyle e_{5}$
$\displaystyle=(2+\sqrt{5})b^{n}t,$ $\displaystyle e_{6}$
$\displaystyle=\frac{1}{20\sqrt{5}}e^{\frac{-ie_{3}}{2}},$ $\displaystyle
e_{7}$ $\displaystyle=\frac{1}{20\sqrt{5}}e^{\frac{-ie_{5}}{2}}\phi_{6},$
$\displaystyle e_{8}$ $\displaystyle=\frac{1}{40}e^{-ie_{3}},$ $\displaystyle
d_{1}$ $\displaystyle=-1+2\delta,$ $\displaystyle d_{2}$
$\displaystyle=-1+4\delta,$ $\displaystyle h_{1}$ $\displaystyle=5+\sqrt{5},$
$\displaystyle h_{2}$ $\displaystyle=-5+\sqrt{5},$ $\displaystyle h_{3}$
$\displaystyle=3+\sqrt{5},$ $\displaystyle h_{4}$ $\displaystyle=-3+\sqrt{5},$
$\displaystyle\theta_{1}$ $\displaystyle=2\delta cos(\frac{b^{n}t}{2})-i\gamma
sin(\frac{b^{n}t}{2}),$ $\displaystyle\theta_{2}$ $\displaystyle=\gamma
cos(\frac{b^{n}t}{2})-2\delta isin(\frac{b^{n}t}{2}),$
$\displaystyle\theta_{3}$ $\displaystyle=i\gamma cos(\frac{b^{n}t}{2})+2\delta
sin(\frac{b^{n}t}{2}),$ $\displaystyle\theta_{4}$ $\displaystyle=2i\delta
cos(\frac{b^{n}t}{2})+sin(\frac{b^{n}t}{2}),$ $\displaystyle\phi_{1}$
$\displaystyle=cos(b^{n}t),$ $\displaystyle\phi_{2}$
$\displaystyle=sin(b^{n}t),$ $\displaystyle\phi_{3}$
$\displaystyle=cos(\sqrt{5}b^{n}t),$ $\displaystyle\phi_{4}$
$\displaystyle=cos(\frac{b^{n}t}{2}),$ $\displaystyle\phi_{5}$
$\displaystyle=sin(\frac{b^{n}t}{2}),$ $\displaystyle\phi_{6}$
$\displaystyle=sin(\frac{1}{2}\sqrt{5}b^{n}t),$ $\displaystyle\phi_{7}$
$\displaystyle=cos(\frac{1}{2}\sqrt{5}b^{n}t),$ $\displaystyle m_{1}$
$\displaystyle=(-5+\sqrt{5})e^{i(1+\sqrt{5})b^{n}t}-(5+\sqrt{5})(e^{ib^{n}t}+e^{i\sqrt{5}b^{n}t})+h2),$
$\displaystyle m_{2}$
$\displaystyle=(-5+\sqrt{5})+e_{2}h_{1}-e^{ie_{3}}h_{2},$ $\displaystyle
m_{3}$ $\displaystyle=2[h_{1}e^{ie_{3}}-h_{2}e_{1}+h_{1}]d_{1},$
$\displaystyle m_{5}$
$\displaystyle=[m_{1}\theta_{3}-m_{2}\theta_{4}+m_{3}\phi_{5}],$
$\displaystyle m_{4}$
$\displaystyle=d_{1}e^{2i\sqrt{5}b^{n}t}h_{4}+e^{ie_{4}}(4-16\delta)+e^{ib^{n}t}(4-16\delta)+e^{ie_{4}}(12-8\delta),$
$\displaystyle n_{1}$ $\displaystyle=-(h_{2}+e_{2}h_{1}-e^{ie_{3}}h_{2}),$
$\displaystyle n_{2}$ $\displaystyle=2(-h_{1}e^{ie_{5}}+h_{2}e_{2}+h_{1}),$
$\displaystyle n_{3}$
$\displaystyle=-d_{1}h_{1}+d_{1}e^{i\sqrt{5}b^{n}t}h_{2},$ $\displaystyle
n_{4}$ $\displaystyle=d_{1}e^{ie_{5}}h_{1}-d_{1}e^{2ib^{n}t}h_{2},$
$\displaystyle n_{5}$
$\displaystyle=2e^{ie_{3}}[\sqrt{5}d_{2}+5\gamma]+2e^{ib^{n}t}[5\gamma-\sqrt{5}d_{2}],$
$\displaystyle n_{6}$
$\displaystyle=2e^{ie_{3}}[\sqrt{5}\gamma+5d_{2}]+2e^{ib^{n}t}[5d_{2}-\sqrt{5}\gamma]$

By looking at the elements of density matrix, we have found that the matrix is
free from the parameter $a$. Here the diagonal elements of tridiagonal matrix
does not play the role in the dynamics.

##  VI Quantum dynamics for Werner state and MEMS

Based on the results derived in the simulation of previous sections, here we
present the dynamical aspects of quantum correlations (Concurrence and quantum
discord). We recall that the eigenvalue spectrum of Werner state i.e.
$\rho^{W}(t)$ given in Eq.(21) is free from all the parameters $(a,b,n,t)$
hence the quantum correlations in Werner state are unaffected under the
tridiagonal Toeplitz matrix.

Further, we present the density matrix $\rho^{MEMS}(t)$ of MEMS for the value
$(t=0)$, the $\rho^{MEMS}(t)$ maps to $\rho^{MEMS}(0)$ and eigenvalue spectrum
is obtained as below for $(\delta=\frac{1}{3})$ and
$(\delta=\frac{\gamma}{2})$ respectively,

$\left\\{\frac{1}{3},0,\frac{1}{6}(2-3\gamma),\frac{1}{6}(3\gamma+2)\right\\},$
(23) $\\{0,0,1-\gamma,\gamma\\}$ (24)

Figure 1: Plot of quantum corrections for (b=2) and $(\gamma$=0.3)

Here, we observe that both the eigenvalue spectrums are free from all the
parameters $(n,a,b,t)$, and hence the initial quantum correlations are never
affected by these parameters; these remain the same as in $\rho^{MEMS}(0)$. It
is observed in this state that the quantum correlations are very sensitive
with the parameters $(b,n,\gamma)$, and independent of the parameter $a$.
Further we divide the study into two subsections with different parameter
ranges.

Figure 2: Plot of quantum corrections for (b=2) and $(\gamma$=0.6)

###  VI.1 Entanglement and quantum discord in MEMS with $(b=2)$ and varying
$(n,\gamma)$.

In this subsection, we present the results with the fixed value of $b$ and
vary the parameters $(n,\gamma)$. These results are presented in Figs. 1, 2
and 3. The concurrence is depicted by black dotted color and quantum discord
is sketched by a red solid color.

####  VI.1.1 Case 1: $(\gamma=0.3)$

In Fig. 1, we plot the results for the parameter $(b=2)$ with varying values
of $n$ (keeping $\gamma$ fixed). With the specified values of parameters $b$
and $n$ the super-diagonal and sub-diagonal of the tridiagonal Toeplitz
Hamiltonian becomes an exponential function $2^{n}$ which rises exponentially
w.r.t to $n$. We investigate ESD with the initial values of
$(n=1,\gamma=0.3)$. Further when the value of $n$ increases the periodic ESD
takes place but the width of the ESD zone decreases. In the absence of
entanglement quantum discord exists that can be used for quantum applications.
We also observe the increasing value of $n$ increases the frequency of quantum
correlations (entanglement, quantum discord). For ($n=7.2,\gamma=0.3$), the
frequency of oscillations is very high and quantum discord achieves a higher
amplitude than entanglement.

Figure 3: Plot of quantum corrections for (b=2) and $(\gamma$=0.9)

####  VI.1.2 Case 2: $(\gamma=0.6)$

In Fig. 2, we obtain the results for $(\gamma=0.6)$ and varying $n$. Following
the previous case, it is observed here in the current case that the increasing
values of $\gamma$ make disappear the sudden death of entanglement and rise
the amplitude of quantum correlations. The initial amplitude of quantum
discord at $(t=0)$ is approximately $0.5$ with $(n=1)$. Here we mention that
increasing values of $n$ freeze the initial amplitude of quantum discord but
this amplitude of entanglement rise with increasing values of $n$. For
$(n=7.2)$, the entanglement achieves the initial amplitude as $0.48$ which is
still less than the previously mentioned initial amplitude of quantum discord.
It is interesting to note that the increasing values of $n$ increase the
oscillations of both the quantum correlations.

Figure 4: Plot of quantum corrections for (b=e) and $(\gamma$=0.3)

####  VI.1.3 Case 3: $(\gamma=0.9)$

In Fig. 3, we plot the graphical results for $(\gamma=0.9)$. We revise that
the elements of MEMS are the functions of parameter $\gamma$, so the
increasing value of this parameter generates different quantum states (MEMS).
With the value $(n=1,\gamma=0.9)$, the initial amplitude of quantum discord is
approximately $0.8$ and for entanglement, it is $0.7$. We also observed that
the initial amplitude of quantum discord freeze with varying $\gamma$. Next,
we observed an interesting behavior that at $((t/T)=0.7)$, the amplitude of
entanglement overshoots the quantum discord as time advances, the crossing of
quantum correlations are phrased as a balancing point [18]. As the value of
$n$ increases the number of balancing points increases.

###  VI.2 Entanglement and quantum discord in MEMS with $(b=e)$ and varying
$(n,\gamma)$.

In this section, we present the results for $(b=e)$ and varying values of
$(n,\gamma)$. The graphical results are depicted in Figs. 4, 5 and 6. When we
consider $(b=e)$, the elements of tridiagonal Toeplitz Hamiltonian take the
form of the exponential function as $e^{n}$, which is the upper bound of an
exponential function $2^{n}$ and have the higher growth rate. This study is
taken over in subsequent cases with the varying value of $\gamma$.

####  VI.2.1 Case 1: $(\gamma=0.3)$

In this subsection, we discuss the results in Fig. 4. With the parameter
values $(n=1,\gamma=0.3)$ we obtain the sudden death results with the time
interval $(0.1\leq t\leq 0.78)$. Moreover, as the value of $n$ increases the
frequency of ESD and quantum discord increases. Here we mention that at
$(t=0)$ the amplitude of quantum discord is less as compared to case 1 of
$(b=2^{n})$ also the amplitude of entanglement is higher and goes on
decreasing with time. Further, we have found the two sudden death zones with
the parameters $(n=1.8,\gamma=0.3)$.

####  VI.2.2 Case 2: $(\gamma=0.6)$ and $(\gamma=0.9)$

In this subsection, we explore the dynamics with the parameters
$(\gamma=0.6,\gamma=0.9)$ in Figs. 5 and 6. We have found that as the
condition $(\gamma\geq 0.6)$ satisfied, the sudden death disappears in the
system. Further, we mention that the frequency and amplitude behavior remains
the same as discussed in previous subsections.

For the value $(\gamma=0.9)$, there is a trade-off between the initial
amplitude of entanglement and quantum discord. With the values
$(n=1,\gamma=0.9)$, the initial amplitude of entanglement is higher than
quantum discord, but with $(n\geq 2)$, the quantum discord at $(t=0)$,
achieves higher amplitude than entanglement and this behavior remain same with
the range $(2\leq n\leq 7)$. Further with $(n=9.4,\gamma=0.9)$, the initial
behavior of entanglement dominates the quantum discord. As time advances the
sinusoidal pattern of quantum correlations arises.

Figure 5: Plot of quantum corrections for (b=e) and $(\gamma$=0.6) Figure 6:
Plot of quantum corrections for (b=e) and $(\gamma$=0.9)

##  VII Conclusion

In this work, we present the analysis of the quantum dynamics in two different
kinds of quantum states i.e. Werner state and MEMS respectively. The study is
focused to investigate the role of tridiagonal Toeplitz Hamiltonian in quantum
information. We have considered the generalized elements of such Hamiltonian
as the exponential functions $(a^{n},b^{n})$ and found interesting results
under unitary dynamics. We analysed that the eigenvalue spectrum of the Werner
state is free from all the parameters $(a,b,n)$, hence the quantum
correlations in this state are unaffected by the Hamiltonian parameters.
Further for MEMS, the eigenvalue spectrum is independent of diagonal
parameters $(a^{n})$, it is depending only on $(b^{n})$ and the parameter
$\gamma$. We explored this study with the two different cases $(b=2)$ and
$(b=e)$. We noticed that the quantum dynamics in MEMS are more fragile under
such parameter changes and it passes through sudden death of entanglement. The
frequency of quantum correlations in this state increases as the value of the
parameter $n$ increases. Further we also investigated the balancing points of
quantum correlations. The present study can be explored in more detail by
expanding the range of the parameters $(a,b,n)$ in Toeplitz matrices, and hope
it will be beneficial for the quantum information community.

## Compliance with ethical standards

The present work does not use any kind of experimental or human data. Also the
work does not have any financial or competing interest.

##  Appendix A Lanczos algorithm

Algorithm 1 Lanczos Algorithm

1:Let $B\in F^{n\times n}$ be a Hermitian matrix. This algorithm computes the
Lanczos iterations, i.e., an orthonormal basis $V_{m}=[v_{1},...,v_{m}]$ for
$V_{m}(x)$ where m is the smallest index such that $V_{m}(x)=V_{m+1}(x)$ , and
the tridiagonal Toeplitz matrix $T_{m}$

2:$v_{1}\in C^{n}$

3:$q:=\frac{x}{k\times k};Q_{1}=[q]$;

4:$r:=Aq$;

5:for $j={2,3,...}$ do

6: $q:=\frac{w_{1}}{\beta_{j-1}}$;

7: $Q_{j}:=[Q_{j-1},q]$;

8: $r:=Aq-\beta-{j}-v_{1}$;

9: $\alpha_{j}:=q^{*}r$;

10: $r:=r-\alpha_{j}q$;

11: $\beta_{j}:=krk$;

12: if $(\beta_{j}=0)$ then

13: $(Q\in F^{n\times j};\alpha_{1},...,\alpha_{j};\beta_{1},...,\beta_{j-1})$

14: end if

15:end for

## References

  * [1] Nielsen, M.A., Chuang, I.L., Quantum computation and quantum information, Cambridge University Press, Cambridge (2000).
  * [2] Bell, J., Speakable and unspeakable in quantum mechanics Cambridge University Press, New York (1987).
  * [3] Sharma, K. K., Gerdt, V. P., Gerdt, P. V., Milestone developments in quantum information and No-Go theorems, In Distributed Computer and Communication Networks, Springer International Publishing, 510 (2020).
  * [4] Bennett, C. H., Brassard, G., Crépeau, C., Jozsa, R., Peres, A., Wootters, W. K., Teleporting an unknown quantum state via dual classical and Einstein-Podolsky-Rosen channels, Phys. Rev. Lett. 70, 1895 (1993).
  * [5] Yan, F., Iliyasu, A. M., Le, P. Q., Quantum image processing: a review of advances in its security technologies, International Journal of Quantum Information, 15(03), 1730001 (2017).
  * [6] Peter Wittek book , Quantum Machine Learning: What Quantum Computing Means to Data Mining, Academic Press, ISBN: 9780128009536 (2018).
  * [7] Biamonte, J., Wittek, P., Pancotti, N., Rebentrost, P., Wiebe, N., Lloyd, S., Quantum machine learning, Nature, 549(7671), 195 (2017).
  * [8] Horodecki, R., Horodecki, P., Horodecki, M., Horodecki, K.: Quantum entanglement. Rev. Mod. Phys. 81, 865 (2009)
  * [9] Ollivier, H., Zurek, W. H., Quantum siscord, A measure of the quantumness of correlations, Phys. Rev. Lett., 88, 017901 (2001).
  * [10] Horodecki, M., Entanglement measures, Quantum Inf. Comput., 1(1), 3 (2001).
  * [11] Adesso, G., Bromley, T. R., Cianciaruso, M., Measures and applications of quantum correlations, Journal of Physics A: Mathematical and Theoretical, 49(47), 473001 (2016).
  * [12] Yu, T., Eberly, J. H., Sudden death of entanglement: classical noise effects, Optics Communications, 264(2), 393(2006).
  * [13] Huang, Y., Computing quantum discord is NP-complete, New journal of phys., 16, 3,033027 (2014).
  * [14] Sharma, K. K., Awasthi, S. K., Pandey, S. N., Entanglement sudden death and birth in qubit–qutrit systems under Dzyaloshinskii–Moriya interaction, Quantum information processing, 12, 3437 (2013).
  * [15] Sharma, K. K., Pandey, S. N., Influence of Dzyaloshinshkii–Moriya interaction on quantum correlations in two-qubit Werner states and MEMS, Quantum Information Processing, 14, 1361 (2015).
  * [16] Sharma, K. K., Pandey, S. N., Entanglement dynamics in two-parameter qubit–qutrit states under Dzyaloshinskii–Moriya interaction, Quantum information processing, 13, 2017 (2014).
  * [17] Sharma, K. K., Sinha, S., Trade-off between squashed entanglement and concurrence in bipartite quantum states, International Journal of Theoretical Physics, 60(9), 3651 (2021).
  * [18] Sharma, K. K., Gerdt, V. P., Quantum information scrambling and entanglement in bipartite quantum states, Quantum Information Processing, 20(6), 195 (2021).
  * [19] Ye, K., Lim, L. H., Every matrix is a product of Toeplitz matrices, Foundations of Computational Mathematics, 16, 577 (2016).
  * [20] Berger, C. A., Coburn, L. A., Toeplitz operators and quantum mechanics, Journal of Functional Analysis, 68(3), 273 (1986).
  * [21] Al-Qabani, A., Hilberdink, T., Virtanen, J. A. Fredholm theory of Toeplitz operators on doubling Fock Hilbert spaces, Mathematica scandinavica, 126(3), 593 (2020).
  * [22] Petkova, V., Multipliers and Toeplitz operators on Banach spaces of sequences, Journal of Operator Theory, 283 (2010).
  * [23] Wu, T. T., Theory of Toeplitz determinants and the spin correlations of the two-dimensional Ising model, I, Physical Review, 149(1), 380 (1966).
  * [24] McCoy, B. M., Wu, T. T., Theory of Toeplitz determinants and the spin correlations of the two-dimensional Ising model, IV, Physical Review, 162(2), 436 (1967).
  * [25] Pradhan, Sunny, Toeplitz matrices for the long-range Kitaev model, Università di Bologna (Ph.D. thesis, 2019).
  * [26] Álvarez-Nodarse, R., Petronilho, J., Quintero, N. R., On some tridiagonal k-Toeplitz matrices: Algebraic and analytical aspects, Applications, Journal of computational and applied mathematics, 184(2), 518 (2005).
  * [27] Lanczos, C., An iteration method for the solution of the eigenvalue problem of linear differential and integral operators, (1950).
  * [28] Paige, C. C., Computational variants of the Lanczos method for the eigenproblem, IMA Journal of Applied Mathematics, 10(3), 373 (1972).
  * [29] Coakley, E. S., Rokhlin, V., A fast divide-and-conquer algorithm for computing the spectra of real symmetric tridiagonal matrices, Applied and Computational Harmonic Analysis, 34(3), 379 (2013).
  * [30] Cahill, E., Irving, A., Johnston, C., Sexton, J., Ukqcd Collaboration, Numerical stability of Lanczos methods, Nuclear Physics B-Proceedings Supplements, 83, 825 (2000).
  * [31] Motta, M., Sun, C., Tan, A. T., O’Rourke, M. J., Ye, E., Minnich, A. J., Chan, G. K. L., Determining eigenstates and thermal states on a quantum computer using quantum imaginary time evolution, Nature Physics, 16(2), 205 (2020).
  * [32] McArdle, S., Jones, T., Endo, S., Li, Y., Benjamin, S. C., Yuan, X., Variational ansatz-based quantum simulation of imaginary time evolution, npj Quantum Information, 5(1), 75 (2019).
  * [33] Gomes, N., Zhang, F., Berthusen, N. F., Wang, C. Z., Ho, K. M., Orth, P. P., Yao, Y., Efficient step-merged quantum imaginary time evolution algorithm for quantum chemistry, Journal of Chemical Theory and Computation, 16(10), 6256 (2020).
  * [34] Lin, S. H., Dilip, R., Green, A. G., Smith, A., Pollmann, F., Real-and imaginary-time evolution with compressed quantum circuits, PRX Quantum, 2(1), 010342 (2021).
  * [35] Haah, J., Hastings, M. B., Kothari, R., Low, G. H., Quantum algorithm for simulating real time evolution of lattice Hamiltonians, SIAM Journal on Computing, (0), FOCS,18 (2021).
  * [36] Cerezo, M., Arrasmith, A., Babbush, R., Benjamin, S. C., Endo, S., Fujii, K., Coles, P. J., Variational quantum algorithms, Nature Reviews Physics, 3(9), 625 (2021).
  * [37] Farenick, D., The operator system of Toeplitz matrices. Transactions of the American Mathematical Society, Series B, 8(32), 999 (2021).
  * [38] Farenick, D., McBurney, M., Toeplitz separability, entanglement, and complete positivity using operator system duality. arXiv preprint arXiv:2208.03236 (2022).
  * [39] Werner, R. F., Quantum states with Einstein-Podolsky-Rosen correlations admitting a hidden-variable model, Phys. Rev. A, 40, 4277 (1989).
  * [40] Munro, W. J., James, D. F. V., White, A. G., Kwiat, P. G., Maximizing the entanglement of two mixed qubits, Phys. Rev. A, 64, 030302 (2001).
  * [41] Hill, S. A., Wootters, W. K., Entanglement of a pair of quantum bits. Physical review letters, 78(26), 5022 (1997).
  * [42] Wootters, W. K., Entanglement of formation of an arbitrary state of two qubits. Physical Review Letters, 80(10), 2245 (1998).


Jeongkyun Park∗††thanks: * Equal contribution., Kwanghee Choi∗, Hyunjun Heo∗,
Hyung-Min Park

# Unsupervised Speech Representation Pooling Using Vector Quantization

###### Abstract

With the advent of general-purpose speech representations from large-scale
self-supervised models, applying a single model to multiple downstream tasks
is becoming a de-facto approach. However, the pooling problem remains; the
length of speech representations is inherently variable. The naive average
pooling is often used, even though it ignores the characteristics of speech,
such as differently lengthed phonemes. Hence, we design a novel pooling method
to squash acoustically similar representations via vector quantization, which
does not require additional training, unlike attention-based pooling. Further,
we evaluate various unsupervised pooling methods on various self-supervised
models. We gather diverse methods scattered around speech and text to evaluate
on various tasks: keyword spotting, speaker identification, intent
classification, and emotion recognition. Finally, we quantitatively and
qualitatively analyze our method, comparing it with supervised pooling
methods.

Index Terms: pooling, vector quantization, speech representations, self-
supervised models, wav2vec 2.0

##  1 Introduction

wav2vec 2.0 [1] has swept the speech processing community like a storm with
the surprising effectiveness of transformer-based self-supervised models [2].
Self-supervision leverages large quantities of unlabeled data, which is much
easier to obtain than labeled data. Many task designs exist, but one of the
most common is to mask the input and train the neural network to reconstruct
the original input based on the surrounding information [3, 1]. The task
definition enables the neural network to model the input distribution so that
it can yield a general representation useful for various downstream tasks.

After building a general self-supervised model, supervised training ensues to
fine-tune the model to each specific need. One often attaches the penultimate
fully connected layer to the frozen shared model, acting as the prediction
head [2]. However, input lengths often vary for both speech and text, a
fundamental characteristic of sequential data. Transformer's output
representation length linearly increases as the input size gets longer, while
the final prediction head requires the fixed-size input representation. It
raises the problem of representation pooling; given a variable number of
sequential representations, one has to summarize the representation to have a
fixed size.

The pooling task is often addressed as the unsupervised or paraphrastic
sentence embedding problem in the NLP literature [4, 5], especially after the
success of word embeddings on various tasks. It aims to yield a general-
purpose sentence embedding independent of each downstream task. By starting
from simply averaging the word representations [4], many unparameterized
methods were introduced, such as considering the word frequency while
averaging [5], whitening the representations [6, 7], or utilizing the singular
value transformation [8]. However, existing methods are often dependent on the
underlying tokenizer, making it nontrivial to apply to speech. Also, simply
averaging the speech representations poses a fundamental problem; each phone
has a different length, so vowels will be overrepresented compared to
consonants.

In the meantime, many pooling methods have been introduced for summarizing the
frame-wise speech representations, notably for speaker recognition. For
example, statistics pooling (SP) [9, 10] concatenates the first- and second-
order statistics of representations. Also, the attention layer is often used
to obtain which representation matters more for each downstream task [11, 12,
13]. However, there has been limited interest in developing an unsupervised
way of pooling. Modern methods, except for SP, are often heavily
parameterized, requiring labeled samples. In contrast, unsupervised pooling
removes the training procedure altogether, being more closer to the idea of
general speech embedding.

To avoid parameterization while being speech-friendly, we shed light on vector
quantization (VQ) to substitute the tokenizer. Decades of research have been
conducted on VQ [14], where it crept into the modern self-supervised models,
such as vq-wav2vec [15, 16]. VQ focuses on translating the real-valued vectors
into countable indices, segmenting the latent embedding space with well-chosen
centroids. Given that essential components comprising speech, such as
frequency, amplitude, and formants, are embedded inside the model
representation[17], we expect the clusters to successfully gather similar
phone representations to have the same cluster indices. We design various
intuitive approaches that utilize VQ to effectively summarize the speech
representations, and experimentally demonstrate its effectiveness in a wide
range of settings.

To further boost the research on unsupervised pooling methods, we devise a
benchmark that evaluates various pooling methods across multiple backbone
networks and a wide range of tasks, namely, keyword spotting [18], speaker
identification [19], intent classification [20], and emotion recognition [21].
To avoid supervision altogether, we closely follow the recent contrastive
learning literature [22, 23] to evaluate the effectiveness of the pooled
representation directly via the nearest-neighbor approach [24]. Also, we
modify the existing unsupervised sentence embedding methods to compare their
downstream performance. We further conduct various analyses that accurately
depict the effectiveness and behaviors of our method by comparing with the
parameterized methods via supervised learning, visualizing the weights
directly, and exploring different settings for benchmark evaluation.

##  2 Method

In this section, we briefly overview the vector quantization method used in
vq-wav2vec [16] and wav2vec 2.0 [1, 25] in Section 2.1. Then, we discuss
various ways to utilize the quantization approach. Section 2.2 covers the idea
of merging a similar representation, while Section 2.3 utilizes the quantized
results to estimate the prior probability of each representation.

###  2.1 Preliminaries: vector quantization

We briefly cover the internals of wav2vec 2.0 [1] and its vector quantizer
module before introducing our method. Let us denote the raw speech input as
$\mathbf{x}$, latent speech representations from the 1D convolutional features
as $\mathbf{Z}$, context representations from the transformer as $\mathbf{C}$,
and the quantized representation as $\mathbf{Q}$:

$\displaystyle\mathbf{Z}$
$\displaystyle=\texttt{CNN}(\mathbf{x})\in\mathbb{R}^{T\times F},$ (1)
$\displaystyle\mathbf{C}$
$\displaystyle=\texttt{transformer}(\mathbf{Z})\in\mathbb{R}^{T\times F}.$ (2)
$\displaystyle\mathbf{Q}$
$\displaystyle=\texttt{quantizer}(\mathbf{Z})=\\{q_{t}^{g}\\}\in[V]^{T\times
G},$ (3)

where $G$ is the number of codebook groups, where sharing the codebooks often
yield comparable performance with a much better efficiency [16]. We denote an
integer $i$ between $1\leq i\leq V$ as $[V]$. Each representation of the
$t$-th frame is represented as $G$ integers, where each integer is $1\leq
q_{t}^{g}\leq V$, where $V$ is the number of centroids. Hence, for each frame-
wise speech representation $\mathbf{Z}_{t}\in\mathbb{R}^{F}$ where $t$ is the
frame index, the quantizer maps $\mathbf{Z}_{t}$ to the $G$-dimensional
integers $(q_{t}^{1},q_{t}^{2},\cdots,q_{t}^{G})$. The quantized results are
designed to map the nearby representations to have the same indices, such that
$\texttt{quantizer}(\mathbf{Z})=\texttt{quantizer}(\mathbf{Z}+\epsilon)$ for a
negligible acoustic difference $\epsilon$.

###  2.2 Squashing via quantized results

For the ease of notation, we regard pooling as the weighted sum of frame
representations $\mathbf{C}_{t}$ with a scalar $w_{t}$:

$\bar{\mathbf{C}}=\textstyle\sum_{t=1}^{t=T}\zeta w_{t}\mathbf{C}_{t}.$ (4)

where the normalizing constant $\zeta$ is used to ensure that the sum of
weights equals $1$, i.e., $\textstyle\sum_{t=1}^{t=T}\zeta w_{t}=1$.

For instance, for average pooling (AP), $w_{t}=1$ and $\zeta=1/T$. We observe
that the AP regards every frame to be equally important. However, the length
of speech signals is not proportional to their importance. For instance, even
if there is a long silence, the AP will average them, and the resulting
representation will be highly biased towards silence representation. Also,
vowels and consonants have inherently different lengths; hence the
representation will be fitted towards vowels more.

Hence, we develop a novel pooling approach based on VQ, which clusters similar
frames into each partition, borrowing the idea from greedy decoding of
Connectionist Temporal Classification (CTC) acoustic models [26]. By averaging
with respect to partitions, the method effectively penalizes the
representations within the larger partition:

$\bar{\mathbf{C}}_{\text{Squash}}=\frac{1}{|\mathcal{S}(\mathbf{Q})|}\sum_{P\in\mathcal{S}(\mathbf{Q})}\frac{1}{|P|}\sum_{t\in
P}\mathbf{C}_{t}.$ (5)

$\mathcal{S}:[V]^{T\times G}\rightarrow 2^{[V]^{T\times G}}$ is the squashing
function, where we write the superset of a set $A$ as $2^{A}$. The squash
function returns a partition of the frame indices, where each partition is
constructed by squashing similar representations. The inner fraction exists
for averaging within a single partition that contains similar representations
($\nicefrac{{1}}{{|P|}}$), where the outer fraction
($\nicefrac{{1}}{{|\mathcal{S}(\mathbb{Q})|}}$) exists for averaging across
partitions. We explore two perspectives on squashing, summarized in Figure 1.

Firstly, the equality of two representations at different steps
$\mathbf{C}_{i}$ and $\mathbf{C}_{j}$ can be defined in two ways. We can check
all the indices of $\mathbf{C}_{i}$ and $\mathbf{C}_{j}$ (AND), or allow
partial matches (OR). Compared to AND, OR aims to merge more neighboring
samples by using each quantized group while tolerating some errors.

Further, one has to define which temporal frames should be merged. We develop
a strategy Squash, which penalizes the features that occur repeatedly in long
sequences, similar to CTC, where neighboring equal values are squashed. On the
other hand, we observe that AP works in an orderless manner. Hence one can
extend the definition of Squash to be averaging all the occurrences,
penalizing the features that occur more often than others. We call this
strategy AllSquash.

Figure 1:  Partition Methods for Quantized Representations.

###  2.3 Prior probability estimation via quantized results

We extend the idea of AllSquash to estimate the prior probability $P(q)$ that
each quantized representation occurs. For sentence embeddings, smooth inverse
frequency (SIF) [5] is often used to penalize more frequent words. Given a
word $W$ and a hyperparameter $a$, the weight $w_{i}=a/(a+p(W))$. [5] points
out that SIF has direct connections with TF-IDF [27] and the subsampling
probabilities of word2vec [28].

We extend SIF for speech representations by using VQ. For this, we have to
count the quantized indices of whole training samples and penalize the
representations by their frequency:

$w^{\text{SIF}}_{t}={a}/({a+N(q_{t})}),$ (6)

where we denote the count of $q$ as $N(q)$ and $a$ is the smoothing
hyperparameter, similar to the original SIF [5].

However, there can be other strategies for counting, i.e., the criterion for
the penalty $w_{t}$. We developed a counting strategy that searches the
indices separately in their groups, fully utilizing the group-wise similarity
while avoiding sparseness [16]:

$w^{\text{GP}}_{t}={1}/{\textstyle\sum^{G}_{g=1}{N_{g}(q_{t})}}$ (7)

where $N_{g}$ denotes the number of indices inside the group $g$. We also
dropped the smoothing hyperparameter $a$ for simplicity.

On the other hand, in Section 2.2, we already suggested that similar
representations may have less importance within the instance. While we
proposed the counting method that considers the global probability of VQ
indices (GP), we also devised another approach that counts indices only in the
instance representation. The only difference is the scope of counting, from
global to local. We call this counting method LP.

Finally, we can use both local and global probabilities by multiplying the
weights of LP and GP, denoting it BP:
$w^{\text{BP}}_{t}=w^{\text{LP}}_{t}\cdot w^{\text{GP}}_{t}$, where
$w^{\text{LP}}_{t}$ and $w^{\text{GP}}_{t}$ are weights of $\mathbf{C}_{t}$
from LP and GP, respectively. This pooling strategy aims to reduce the weights
of quantized indices that are redundant in each instance and frequently appear
across the entire training set.

##  3 Experimental results

We closely follow the SUPERB benchmark that measures the efficacy of self-
supervised speech models [2], but with a key difference in the evaluation for
unsupervised pooling methods (Section 3.2). For more details, consult our
repository.111https://github.com/IIP-Sogang/speech-pooling-benchmark

###  3.1 Tasks

We choose four speech tasks that can represent various aspects of speech
processing. Keyword spotting (KWS) aims to automatically detect words within
the predefined vocabulary. We use the Google Speech Commands dataset V2 [18],
which consists of 35 keywords including numbers and words. Speaker
identification (SID) finds the speaker with a given utterance. Our benchmark
focuses on text-independent closed-set SID [29]. We used the VoxCeleb1 dataset
[19] with the provided data splits. Intent classification (IC) extracts
semantic information from speech, such as the speaker's intent or emotions
[30]. We use the fluent speech command dataset[20] that provides labels for
three different information: action, object, and location. Emotion recognition
(ER) is a task that predicts an emotion class depending on verbal channels. We
adopt IEMOCAP [21] and select 4 emotion classes to follow [2]. We train our
models on 5-fold cross-validation using 5 sessions provided in the IEMOCAP
dataset.

###  3.2 Benchmark design

We design our benchmark considering the unsupervised setting: not allowing
additional trainable parameters to fairly compare the effectiveness of
unsupervised pooling methods. We evaluate the representations on various tasks
by finding the nearest neighbors of each test dataset representation by
measuring the distances with the utterance-level training dataset
representations. To measure the accuracy, the labels of test samples and their
nearest training sample are compared, which is a commonly used setting in
self-supervised representations based on contrastive losses [22, 23]. We use
the approximated nearest neighbor method [24] to calculate distances
efficiently.

###  3.3 Models

We adopted three different self-supervised speech models based on wav2vec2.0
which can generate quantized representations: base, large[1] and XLS-R[25].
base and large differ by the model size, while large and XLS-R differ by the
pretraining dataset, which is book-reading speech data and multilingual speech
corpus from various sources, respectively. We chose the widely used three
models as we can assess the impact of pooling techniques under various
conditions.

###  3.4 Baselines

We chose four unsupervised pooling baselines. Average Pooling (AP) is the most
naive and widely used baselines among various pooling methods [2]. Statistics
Pooling (SP) extends upon AP by utilizing not only the first-order but the
second-order statistics, namely, mean and standard deviation [9, 10].
Whitening applies the whitening transformation to the representations so that
their covariance becomes an identity matrix [7, 6]. SoftDecay is the current
state-of-the-art for sentence embedding, where a non-linear function is
applied to make the singular value distribution of the representations less
skewed [8].

Table 1:  Unsupervised benchmark for speech representation pooling. Test
accuracy (%) via the nearest neighbor method (Section 3.2)) is reported. Best
accuracies are written in bold.

| Method | KWS | SID | IC | ER | AVG
---|---|---|---|---|---|---
base | AP | 58.44 | 13.85 | 26.71 | 53.44 | 38.11
SP | 68.43 | 15.63 | 33.11 | 53.30 | 42.62
Whitening | 60.55 | 15.13 | 36.59 | 50.88 | 40.79
SoftDecay | 58.66 | 13.66 | 26.71 | 53.44 | 38.12
VQ-Squash-AND | 58.46 | 14.82 | 26.97 | 52.76 | 38.26
VQ-Squash-OR | 55.83 | 14.12 | 24.97 | 52.00 | 36.73
VQ-AllSquash-AND | 65.16 | 14.80 | 32.96 | 53.50 | 41.60
VQ-AllSquash-OR | 65.92 | 12.99 | 37.81 | 52.61 | 42.33
VQ-SIF | 59.62 | 13.33 | 36.54 | 54.28 | 40.94
VQ-LP | 74.48 | 15.53 | 48.11 | 55.48 | 48.40
VQ-GP | 67.98 | 12.91 | 41.55 | 54.11 | 44.14
| VQ-BP | 73.48 | 12.94 | 50.99 | 55.13 | 48.14
large | AP | 71.87 | 17.62 | 29.95 | 54.37 | 43.45
SP | 77.09 | 19.15 | 36.91 | 56.77 | 47.48
Whitening | 67.44 | 18.75 | 36.75 | 55.45 | 44.60
SoftDecay | 72.02 | 17.37 | 30.21 | 54.37 | 43.49
VQ-Squash-AND | 70.21 | 17.05 | 31.43 | 54.12 | 43.20
VQ-Squash-OR | 66.31 | 17.40 | 29.21 | 54.27 | 41.80
VQ-AllSquash-AND | 76.75 | 18.77 | 39.12 | 55.34 | 47.50
VQ-AllSquash-OR | 75.75 | 16.64 | 41.76 | 56.04 | 47.55
VQ-SIF | 73.09 | 17.60 | 32.64 | 54.59 | 44.48
VQ-LP | 83.07 | 19.42 | 55.00 | 57.29 | 53.69
VQ-GP | 76.57 | 15.89 | 51.86 | 55.28 | 49.90
| VQ-BP | 81.00 | 14.71 | 58.53 | 54.28 | 52.13
XLS-R | AP | 68.42 | 11.62 | 26.13 | 55.33 | 40.38
SP | 77.26 | 13.42 | 36.70 | 56.54 | 45.98
Whitening | 60.72 | 15.22 | 30.11 | 50.69 | 39.18
SoftDecay | 68.35 | 12.02 | 26.13 | 55.33 | 40.46
VQ-Squash-AND | 70.80 | 12.06 | 34.27 | 55.35 | 43.12
VQ-Squash-OR | 73.31 | 12.46 | 41.79 | 55.55 | 45.78
VQ-AllSquash-AND | 75.97 | 12.82 | 44.53 | 56.72 | 47.51
VQ-AllSquash-OR | 75.37 | 12.02 | 40.36 | 55.12 | 45.72
VQ-SIF | 80.45 | 11.89 | 46.08 | 57.46 | 48.97
VQ-LP | 82.38 | 12.71 | 49.14 | 58.84 | 50.77
VQ-GP | 82.87 | 12.00 | 52.49 | 57.74 | 51.27
| VQ-BP | 86.30 | 11.71 | 52.89 | 57.60 | 52.12

Figure 2:  t-SNE visualization of the pooled features from Google Speech
Commands V2 dataset. Each point represents a single data instance, and colors
indicate four different classes.

###  3.5 Results

Table 1 shows the unsupervised benchmark results. We found that squashing by
AND and OR improved overall performance with XLS-R by 2.74% and 5.4%,
respectively, compared to AP, while there was no notable improvement in other
models. Further, the AllSquash strategy showed better performance than Squash
while outperforming AP on all variants of wav2vec 2.0. Additionally, VQ-LP
achieved the best average accuracy in base and large, while VQ-BP performed
remarkably well on IC with every backbone, outscoring AP by a factor of two.
We also employed the t-distributed Stochastic Neighbor Embedding (t-SNE) [31]
to visualize the pooled representations, as shown in Figure 2. Our
observations indicate that the VQ-based approach helps differentiate between
the representations of keywords with similarly pronounced vowels. Overall, our
results demonstrate the effectiveness of VQ-based pooling approaches in
capturing the temporal importance of speech representations.

Table 2:  Performance comparison with supervised methods. Test accuracy(%) of
the best model kept by the validation set is reported. Best accuracies are
written in bold.

| Method | KWS | SID | IC | ER | AVG
---|---|---|---|---|---|---
base | AP | 94.89 | 65.36 | 91.88 | 62.59 | 78.68
SP | 95.15 | 64.72 | 97.39 | 55.66 | 79.18
VQ-LP | 95.49 | 64.82 | 95.70 | 62.73 | 79.69
SAP | 95.15 | 40.86 | 97.39 | 55.66 | 72.27
ASP | 95.70 | 36.52 | 97.47 | 59.34 | 72.26
VAP | 96.74 | 68.50 | 99.50 | 65.50 | 82.56
large | AP | 96.05 | 79.23 | 95.07 | 65.47 | 83.96
SP | 96.23 | 77.71 | 96.73 | 67.02 | 84.42
VQ-LP | 96.32 | 78.12 | 97.76 | 66.58 | 84.70
SAP | 96.26 | 58.48 | 96.78 | 57.78 | 77.33
ASP | 96.42 | 46.36 | 98.55 | 61.94 | 75.82
VAP | 97.14 | 77.08 | 99.58 | 67.42 | 85.30
XLS-R | AP | 97.32 | 67.85 | 91.30 | 71.77 | 82.06
SP | 97.51 | 60.89 | 94.12 | 72.07 | 81.15
VQ-LP | 97.59 | 68.13 | 94.38 | 70.67 | 82.69
SAP | 97.63 | 61.73 | 98.86 | 64.83 | 80.76
ASP | 96.69 | 13.48 | 99.02 | 61.08 | 67.57
VAP | 97.49 | 61.71 | 99.63 | 71.15 | 82.49

Figure 3:  Visualized weights induced by VQ-LP. Two distinct peaks indicate
the presence of ``r'' and ``t'' sounds, while values dropped in regions
corresponding to vowel sounds.

##  4 Analysis

###  4.1 Comparing with parameterized methods

In this section, we compared the effectiveness of our approach with
parameterized methods. We selected the parameterized pooling methods often
used for speaker recognition: Self Attentive Pooling (SAP)[32], Attentive
Statistics Pooling (ASP)[11], and Vector-based Attentive Pooling (VAP)[13].

Comparing all methods in supervised settings, we used the Adam optimizer[33]
with a default learning rate of 0.001 and 200 epochs, except for KWS using a
learning rate of 0.01 and SID using 50 epochs. Note that parameterized methods
have additional trainable parameters to fit towards downstream tasks, making
the comparison more favorable towards them.

While the methods using quantized representations brought a significant
improvement in unsupervised experiments, the same level of enhancement was not
observed in Table 2. However, VQ-LocalProb is comparable to the parameterized
methods, even getting higher average accuracy in XLS-R than the best
parameterized method, VAP. The results indicate the possibility of further
progress in unsupervised over supervised pooling methods, worthwhile for
future investigations.

###  4.2 Pooling weight analysis

We analyzed the behavior of our proposed pooling methods by comparing the
weights of different strategies with SAP. We use the KL divergence
$D_{\text{KL}}(P\|Q)$, putting the weights induced by SAP and the others into
$P$ and $Q$, respectively. Figure 4 show that our proposed methods generally
approximate SAP better than AP. We also found that accuracy and KL divergence
negatively correlate, confirming that better similarity to SAP results in
better performance. Additionally, we visualize a sample speech and its pooling
weights induced by our VQ-LP method in Figure 3 from Speech Commands V2, where
our method detects speech while suppressing areas with ambient noise.

Figure 4:  Comparing different pooling methods' weights. KL divergence
measures the similarity between the weights from SAP and others, regarding SAP
as the oracle. We observed that closer to SAP, generally better the downstream
performance.  Table 3:  Test accuracy (%) of average pooling on base with
varying number of nearest neighbors (K).

Task K | 1 | 3 | 4 | 5 | 6 | 7 | 8 | 9
---|---|---|---|---|---|---|---|---
KWS | 58.44 | 60.42 | 61.88 | 62.48 | 62.58 | 62.85 | 63.22 | 63.12
SID | 13.85 | 13.59 | 13.38 | 13.17 | 12.94 | 12.63 | 12.14 | 11.93
IC | 26.71 | 27.58 | 27.74 | 27.42 | 28.16 | 28.16 | 28.08 | 28.05
ER | 53.44 | 56.68 | 57.54 | 58.45 | 59.30 | 59.30 | 60.11 | 59.51

###  4.3 Exploring different evaluation schemes

As the benchmark design in Section 3.2 utilizes only the nearest neighbor's
information, we further explore the case considering a group of neighbors. We
experiment with whether increasing the number of nearest neighbors and
choosing the final label via voting can increase the downstream performance.
If a tie occurs, we follow the label of the sample that is the closest, which
is the reason why we do not consider the 2-nearest case. Table 3 shows the
performance differences with respect to the number of nearest neighbors.
Consistent improvements in performance are shown across all tasks except for
SID, where the task contains relatively a large number of classes. It
indicates that the current benchmark settings can be potentially improved for
more realistic use cases.

##  5 Conclusion

In this paper, we design an unsupervised yet effective way of pooling
variable-length speech features obtained from self-supervised speech models.
Downstream performance on various speech-processing tasks and self-supervised
representations validate the effectiveness of our VQ-based approach. Further,
we compare our unsupervised pooling method with supervised pooling methods
based on attention mechanisms to understand its inner workings. Further
research should be done to investigate the potential of VQ approaches on a
broader array of general speech models without an embedded VQ module, such as
by incorporating the idea of HuBERT [34].

## References

  * [1] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, ``wav2vec 2.0: A framework for self-supervised learning of speech representations,'' _Advances in neural information processing systems_ , vol. 33, pp. 12 449–12 460, 2020.
  * [2] S. Yang, P. Chi, Y. Chuang, C. J. Lai _et al._ , ``SUPERB: Speech processing universal performance benchmark,'' in _Proc. INTERSPEECH_ , 2021\.
  * [3] J. D. M.-W. C. Kenton and L. K. Toutanova, ``BERT: Pre-training of deep bidirectional transformers for language understanding,'' in _Proceedings of NAACL-HLT_ , 2019, pp. 4171–4186.
  * [4] J. Wieting, M. Bansal, K. Gimpel, and K. Livescu, ``Towards universal paraphrastic sentence embeddings,'' in _Proc. ICLR_ , 2016.
  * [5] S. Arora, Y. Liang, and T. Ma, ``A simple but tough-to-beat baseline for sentence embeddings,'' in _Proc. ICLR_ , 2017.
  * [6] J. Huang, D. Tang, W. Zhong, and S. e. a. Lu, ``WhiteningBERT: An easy unsupervised sentence embedding approach,'' in _Proc. EMNLP_ , 2021, pp. 238–244.
  * [7] J. Su, J. Cao, W. Liu, and Y. Ou, ``Whitening sentence representations for better semantics and faster retrieval,'' _arXiv preprint arXiv:2103.15316_ , 2021.
  * [8] H. Yan, L. Gui, W. Li, and Y. He, ``Addressing token uniformity in transformers via singular value transformation,'' in _Proc. Uncertainty in Artificial Intelligence_. PMLR, 2022, pp. 2181–2191.
  * [9] D. Snyder, P. Ghahremani, D. Povey, D. Garcia-Romero, Y. Carmiel, and S. Khudanpur, ``Deep neural network-based speaker embeddings for end-to-end speaker verification,'' in _Proc. Spoken Language Technology Workshop (SLT)_. IEEE, 2016, pp. 165–170.
  * [10] D. Snyder, D. Garcia-Romero, D. Povey, and S. Khudanpur, ``Deep neural network embeddings for text-independent speaker verification.'' in _Proc. INTERSPEECH_ , 2017, pp. 999–1003.
  * [11] K. Okabe, T. Koshinaka, and K. Shinoda, ``Attentive statistics pooling for deep speaker embedding,'' in _Proc. INTERSPEECH_ , 2018.
  * [12] P. Safari, M. India, and J. Hernando, ``Self-attention encoding and pooling for speaker recognition,'' in _Proc. INTERSPEECH_ , 2020, pp. 941–945.
  * [13] Y. Wu, C. Guo, H. Gao, X. Hou, and J. Xu, ``Vector-based attentive pooling for text-independent speaker verification.'' in _Proc. INTERSPEECH_ , 2020, pp. 936–940.
  * [14] F. K. Soong, A. E. Rosenberg, B.-H. Juang, and L. R. Rabiner, ``Report: A vector quantization approach to speaker recognition,'' _AT &T technical journal_, vol. 66, no. 2, pp. 14–26, 1987.
  * [15] H. Jegou, M. Douze, and C. Schmid, ``Product quantization for nearest neighbor search,'' _IEEE transactions on pattern analysis and machine intelligence_ , vol. 33, no. 1, pp. 117–128, 2010.
  * [16] A. Baevski, S. Schneider, and M. Auli, ``vq-wav2vec: Self-supervised learning of discrete speech representations,'' in _Proc. ICLR_ , 2020.
  * [17] K. Choi and E. J. Yeo, ``Opening the black box of wav2vec feature encoder,'' _arXiv preprint arXiv:2210.15386_ , 2022.
  * [18] P. Warden, ``Speech commands: A dataset for limited-vocabulary speech recognition,'' _arXiv preprint arXiv:1804.03209_ , 2018.
  * [19] A. Nagrani, J. S. Chung, and A. Zisserman, ``VoxCeleb: A large-scale speaker identification dataset,'' in _Proc. INTERSPEECH_. ISCA, 2017, pp. 2616–2620.
  * [20] L. Lugosch, M. Ravanelli, P. Ignoto, V. S. Tomar, and Y. Bengio, ``Speech model pre-training for end-to-end spoken language understanding,'' in _Proc. INTERSPEECH_ , 2019.
  * [21] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, ``IEMOCAP: Interactive emotional dyadic motion capture database,'' _Language resources and evaluation_ , vol. 42, pp. 335–359, 2008.
  * [22] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, ``A simple framework for contrastive learning of visual representations,'' in _Proc. International Conference on Machine Learning_. PMLR, 2020, pp. 1597–1607.
  * [23] K. Choi and S. Lee, ``Combating the instability of mutual information-based losses via regularization,'' in _Proc. Uncertainty in Artificial Intelligence_. PMLR, 2022, pp. 411–421.
  * [24] E. Bernhardsson, ``Annoy: approximate nearest neighbors in c++/python optimized for memory usage and loading/saving to disk,'' _GitHub https://github.com/spotify/annoy_ , 2017.
  * [25] A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli, ``XLS-R: Self-supervised cross-lingual speech representation learning at scale,'' in _Proc. INTERSPEECH_ , 2022, pp. 2278–2282.
  * [26] T. Zenkel, R. Sanabria, F. Metze, J. Niehues, M. Sperber, S. Stüker, and A. Waibel, ``Comparison of decoding strategies for ctc acoustic models,'' _arXiv preprint arXiv:1708.04469_ , 2017.
  * [27] K. Sparck Jones, ``A statistical interpretation of term specificity and its application in retrieval,'' _Journal of documentation_ , vol. 28, no. 1, pp. 11–21, 1972.
  * [28] T. Mikolov, W.-t. Yih, and G. Zweig, ``Linguistic regularities in continuous space word representations,'' in _Proceedings of the conference of the north american chapter of the association for computational linguistics: Human language technologies_ , 2013, pp. 746–751.
  * [29] M. M. Kabir, M. F. Mridha, J. Shin, I. Jahan, and A. Q. Ohi, ``A survey of speaker recognition: Fundamental theories, recognition methods and opportunities,'' _IEEE Access_ , vol. 9, pp. 79 236–79 263, 2021.
  * [30] S. Seo, D. Kwak, and B. Lee, ``Integration of pre-trained networks with continuous token interface for end-to-end spoken language understanding,'' in _Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 7152–7156.
  * [31] G. E. Hinton and S. Roweis, ``Stochastic neighbor embedding,'' in _NeurIPS_ , vol. 15, 2002.
  * [32] W. Cai, J. Chen, and M. Li, ``Exploring the encoding layer and loss function in end-to-end speaker and language recognition system,'' in _Proc. Odyssey The Speaker and Language Recognition Workshop_ , 2018, pp. 74–81.
  * [33] D. P. Kingma and J. Ba, ``Adam: A method for stochastic optimization,'' in _Proc. ICLR_ , Y. Bengio and Y. LeCun, Eds., 2015.
  * [34] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.


ResourceswhiteResources |  Financial or people resources are limited, especially time it takes to do counterspeech |  “[The biggest challenge is] time. We’re doing a max already but for security reasons we cannot be too big. This is not our job.”
Finding hate speechwhiteFinding hate speech |  Takes time to find hate speech |  “It’s so time consuming looking for articles.”
TrainingwhiteTraining |  People don’t have the training |  “People don’t always know what to say.”
ReachwhiteReach |  It’s hard to reach people, which is discouraging |  “When you feel unheard and it’s like I’m doing this for nothing - it’s not really getting the word out - it’s frustrating”
RiskwhiteRisk |  There are risks of online or offline attacks |  “And the risk – the risk is real. I don’t use anonymous posting. In our community, there is fear and so it’s risky when you put yourself out there.”
Mental healthwhiteMental health |  It is too hard on mental health (stress or boredom) |  “You are just overwhelmed with what you are seeing.”
Counterspeaker Motivations
[regular] RightwhiteRight |  It’s the right thing to do / civic responsibility |  “I think it’s because it’s the right thing to do – I feel that at least I tried. ”
ImpactwhiteImpact |  Seeing evidence of successful impact |  “When you can see the comment section change. When you can see other non-members speaking out. We take screen shots and save our successes.”
Scale - smallwhiteScale - small |  Quotes about the individual-level impact |  “Probably when we get in real time that we’ve helped someone, helped someone who had maybe been reading the comments and had been upset by them, they say something like, ‘oh thank goodness, I was in a pit of dispair before seeing your comment.”’
Table 5. Codes developed from analysis of interview responses discussing counterspeech barriers and motivations. Code |  |  Description |  Representative Quote
---|---|---|---
Activist Counterspeakers
TimewhiteTime |  |  References to time or efficiency |  “Anything that could help us be more efficient - help us produce more.”
Finding - AIwhiteFinding - AI |  |  AI would help by locating hateful speech |  “I’ve spent up to two hours looking for good actions, so that would be super helpful. ”
Scale - bigwhiteScale - big |  |  AI would help scale the work of counterspeakers |  “A tool that would amplify voice against hate speech. That would assist in amplifying counterspeech and helping it reach the target audience.”
Lay-users
EfficiencywhiteEfficiency |  |  The user wants to use the tool to save time. |  “It would be make responding to hatred so much easier if I could just click a few boxes and let an AI do the work. Even though it won’t change the hater’s heart, it would provide a counter to their hate speach.”
AI-betterwhiteAI-better |  |  The user thinks that AI would be better thant they are. |  “[An AI] could give me a constructive framework for a much more impactful response than I could otherwise generate on my own.”
Capability-dependentwhiteCapability-dependent |  |  The user wants to explore the capabilities of the tool before deciding. |  “I would be willing to see the suggestion that the AI offered and decide whether or not to use it.”
InformationwhiteInformation |  |  The user thinks a tool would be helpful to compile information to counter hate. |  “if it was fact based i sure would use it, since i feel we all can have our own oppinions”
GuidancewhiteGuidance |  |  The user would use the tool to get guidance on how to respond effectively: formulating response and creating more diverse response in a more collaborative way, or to help them understand the hate or detection of hate. |  “I would potentially use it because it could give me a constructive framework for a much more impactful response than I could otherwise generate on my own”
EmotionswhiteEmotions |  |  The user wants help with regulating their emotions to communicate clearly or with effectively communicating user’s emotions. |  “… It would help me to stay calm and collected. When I am faced with hateful speech, I can sometimes get emotional. This can make it difficult for me to respond effectively. The AI tool would help me to stay calm and collected, so that I could focus on responding to the hateful speech in a thoughtful and reasoned way. It would help me to feel more confident in my responses…”
EmpowermentwhiteEmpowerment |  |  The user feels empowered by being able to speak up in addressing hate speech to create a positive impact. |  “Because it would help me speak up more.”
Reduce-stresswhiteReduce-stress |  |  The user feels that having the tool could help reduce stress while responding to hate speech. |  “It might be less stressful to use than making a more personal comment.”
AI-proxywhiteAI-proxy |  |  The user would rather have the AI get involved, rather themselves (often under the guise of anonymity) either in responding or reporting. |  “To be honest it sort of makes me feel like I have some plausible deniability if an issue arises. In a worst case scenario I would be able to ‘blame’ it on the AI.”
Lay-users - AI Tools “”
Existing-technologywhiteExisting-technology |  |  The user refers to existing technology a specific AI tool that current exists in the market. |  “I use most of the time ChatGPT.”
ReportwhiteReport |  |  The user would like a system that can automatically or with minimal input report hate speech. |  “[An AI] that identifies the hate speech and remove or block the comment.”
Response-supportwhiteResponse-support |  |  The user wants an AI tool that suggests or automatically replies with a response to hate speech, which could also be refined by the user with minimal input that is well-written and thoughtful. The user usually wants efficient and time saving support with minimal engagement and is easy to use. |  “The one which suggest the reply in very decent manner.”
FactualwhiteFactual |  |  The user expresses that it would be beneficial to have an assistive technology that can gather factual information to formulate arguments or fact-check hate speech. The users also want help in creating responses that rational, intellectual, and logical arguments. |  “I would use it if it gave out information that was correct and if it was reliable.”
CollaborativewhiteCollaborative |  |  The user wishes to have more collaborative interaction to improve their responses. Some examples of interactions include correction to their written response such as grammar, emotional, or factual and checking their own biases. |  “Something I could be ”checked” on, making sure MY post wasn’t also toxic.”
Effective-communicationwhiteEffective-communication |  |  The user would like to use an AI tool that is sensitive to human emotions while addressing hate speech, and is capable of expressing nuanced emotions. The user wants support communicating clearly with the understanding of emotional, human factors focusing on meaning and impact. |  “An AI assistance that is nice and helps alleviate the situation.”
AlignedwhiteAligned |  |  The user would like an AI tool that is personally and/or culturally aligned and provide responses just like how they would or in an unbiased way. |  “One that is trained off my data and personality that I approve of.”
ProtectivewhiteProtective |  |  The user would like an AI tool that will protect them from retaliation often through anonimity. |  “I would like an AI tool that could prepare a message…[avoids] making myself a target.”
Table 6. Codes developed from analysis of responses showing openness to adopt AI tools in SQ14 and in the interview studie as well as responses to SQ15. The icons indicate the theme of barriers relevant to the code. The codes are separated into three sections: benefits identified by activist counterspeakers, benefits identified by lay-users, and AI tools discussed by lay-users. The colors of icons were chosen to match Figure 2. Code |  |  Description |  Representative Quote
---|---|---|---
Activist CounterSpeakers
Authenticity - strategywhiteAuthenticity - strategy |  |  Worries that inauthentic counterspeech would not be credible |  “The moment we deploy this online, a lot of people who share hateful content and know a lot about tech will recognize it.”
RealwhiteReal |  * |  Quotes comparing AI to what is ”real” |  “We do not need, neither for us nor for the haters, the possibility to create a fake sentiment and take away our voices. It will boil down to who has the money to pay it more.”
Long-termwhiteLong-term |  * |  Concerns that AI counterspeech has long-term negative consequences |  “Really using the bot at all is tricky. You aren’t inspiring real people to participate. If we are actually going to make change, we need those people to be engaged. We need people to get involved in their communities.”
CounterspeakerwhiteCounterspeaker |  [regular] |  Counterspeakers are aided by doing counterspeech |  “And there is something incredibly magic about turning something really hateful in the other direction. You feel you aren’t hopeless or helpless.”
Becoming the monsterwhiteBecoming the monster |  [regular] |  Troll farms are bad. Would we become just as bad by using a counterspeech bot? |  “I can see the appeal to that for sure, but I think that it takes out the human component. We’re kind of no better than what is being used.”
Emotional IntelligencewhiteEmotional Intelligence |  [regular] |  Emotional intelligence, empathy, you need a human, authenticity |  “The process itself I find very satisfying. Having a sense of not being alone… For me, what’s really touching is that someone out there is just there to support me. I’m not the only one who thinks this.”
Functionality - technicalwhiteFunctionality - technical |  [regular] |  Skepticism that AI counterspeech would work |  “I’m not sure about it getting the facts right. It’s not good for fact checking.”
Lay-users
AuthenticitywhiteAuthenticity |  |  The user expresses concern that what AI communicates is not their own words and would not represent what they are thinking especially their intentions (alignment) or would be considered inauthentic focusing on ”who” is behind counterspeech. |  “Using AI is too impersonal and it sounds very generic.”
Engaging-not-helpfulwhiteEngaging-not-helpful |  * |  The user believes engaging with people who espouse hate speech is not helpful in reducing that behavior or do not want hate speech getting more attention. These users sometimes express that they would engage if they knew that it would make an impact. |  “I don’t think it matters if I get help with what I want to say if it’s just falling on deaf ears.”
AvoidancewhiteAvoidance |  * |  The user rather wishes to avoid hate speech rather than engage and availability of an AI tool will not change this. |  “I don’t engage in any online hate/drama. I just scroll right through.”
AgencywhiteAgency |  [regular] |  The user does not want AI help especially because they are able to perform the task themselves. The user prefers humans to respond focusing on ”how” counterspeech is generated. |  “I believe in stating things that I feel not what an AI tells me to feel.”
Capability-doubtswhiteCapability-doubts |  [regular] |  The user expresses that they do not think that AI response will have an impact or it will contain other functionality problems. |  “AI tools are often wrong and I wouldn’t want it’s bias’ to affect what I am posting.”
Become-targetwhiteBecome-target |  [regular] |  The user does not want to become the target of hate. |  “Having an AI help me write a response would not keep people from sending me hateful replies. I cannot handle that.”
Table 7. Codes from analysis of responses resistant to adopting AI tools from
SQ14 and concerns raised by interview participants. The icons indicate
relevant themes of motivations that are negatively affected and new themes of
functionality doubts and long-term impact. The codes are presented in two
sections: responses from activist counterspeakers and from lay-users.