&\big{[}\frac{t-\Delta}{t+\Delta}\big{]}^{\frac{l}{2}}\sin\big{(}\frac{n\pi l}{2L+2}\big{)} = \big{[}\frac{t-\Delta}{t+\Delta}\big{]}^{\frac{l}{2}}\sin\big{(}\frac{n\pi l}{N+1}\big{)},\quad  \text{if} \quad l=\text{even},\\
& 0,\quad \text{if} \quad l=\text{odd}.
\end{cases}\\
\Psi_2(l) =&
\begin{cases}
& 0,\quad \text{if} \quad l=\text{even},\\
&\big{[}\frac{t-\Delta}{t+\Delta}\big{]}^{\frac{l+1}{2}}\sin\big{(}\frac{n\pi (l+1)}{2L+4}\big{)} = \big{[}\frac{t-\Delta}{t+\Delta}\big{]}^{\frac{l+1}{2}}\sin\big{(}\frac{n\pi (l+1)}{N+3}\big{)},\quad  \text{if} \quad l=\text{odd}.
\end{cases}
\end{align}

Again, we get the same expression as in Eqn. <ref> for $\Psi_1(l)$ with $\theta_1=\frac{n\pi}{N+1}$ and $\Psi_2(l)=\Psi_1(l+1)$ but with $\theta_1=\frac{n\pi}{N+3}$.

Let us now extrapolate to the case when $R_1=R_2$. This is possible if $\frac{t_1}{\Delta_1}=\pm\frac{t_2}{\Delta_2}$. Consider now $R_2e^{i\theta_2}=R_1e^{i\delta}e^{i\theta_1}$, which implies $\theta_2-\theta_1=\delta$. This is a generalization of the previous case, where we had $\delta=\pi$. The quantization condition in this case is,

\begin{equation}
\frac{1+(e^{i\delta})^{2(N+2)}-2(e^{i\delta})^{N+2}\cos(2(N+1)\theta_1)}{1+e^{2i\delta}-2e^{i\delta}\cos2\theta_1}=\frac{1+(e^{i\delta})^{2(N+2)}-2(e^{i\delta})^{N+2}}{1+e^{2i\delta}-2e^{i\delta_1}}.
\label{deltaquant}
\end{equation}

Let us consider the next simplest case i.e., the phase difference is the third root of unity, denoted as $\delta=e^{i\frac{2\pi}{3}}=\omega$. Following our previous analysis, we must consider three kinds of system sizes, $N=3L,3L+1$ and $3L+2$,

* Case 1:(N=3L) From the Eqn. <ref>, we get,

\begin{equation}
\begin{split}
&\frac{1+\omega-2\omega\cos(2(N+2)\theta_1)}{1+\omega^2-2\omega\cos 2\theta_1}=\omega,\implies 4\omega^2\sin((N+1)\theta_1)\sin((N+3)\theta_1)=0,\\ \implies &\theta_1=\frac{n\pi}{N+1},\frac{m\pi}{N+3}, \quad n\in\{1,...,N\},m\in\{1,...,N+2\}.
\end{split}
\end{equation}

* Case 2:(N=3L+1) From Eqn. <ref>, we get,

\begin{equation}
\begin{split}
&\frac{2-2\cos(2(N+2)\theta_1)}{1+\omega^2-2\omega\cos 2\theta_1}=0,\implies  4\sin^2((N+2)\theta_1)=0,\\
&\implies \theta_1=\frac{n\pi}{N+2}, \quad n\in\{1,...,N+1\}.
\end{split}
\end{equation}

* Case 3:(N=3L+2) From Eqn. <ref>, we have,

\begin{equation}
\begin{split}
&\frac{1+\omega^2-2\omega\cos((N+2)\theta_1)}{1+\omega^2-2\omega\cos 2\theta_1}=1, \implies 4\omega\sin((N+1)\theta_1)\sin((N+3)\theta_1)=0,\\
\implies & \theta_1 = \frac{n\pi}{N+1},\frac{m\pi}{N+3},\quad n\in\{1,...,N\}, m\in\{1,...,N+2\}.
\end{split}
\end{equation}

§.§ MKC perpendicular zero energy modes:

Here, we compute the zero energy modes for the MKC perpendicular system given by the Hamiltonian,

\begin{equation}
\begin{split}
H_{MKC,\perp}(k_x,k_y) =& [-(2t_1\cos k_x+\mu_1)\tau^z+2\Delta_1\sin k_x \tau^y]\otimes [(2t_2\cos k_y+\mu_2)\sigma^z+2\Delta_2\sin k_y\sigma^y],\\
=& \mathbf{d}_1(k_x)\cdot\boldsymbol{\tau}\otimes \mathbf{d}_2(k_y)\cdot\boldsymbol{\sigma}.
\end{split}
\end{equation}

Localizing the Hamiltonian in the $\hat{x}$-direction via $k_x\rightarrow iq_x$,

\begin{equation}
H_{MKC,\perp}(iq_x,k_y) = [-(2t_1\cosh q_x+\mu_1)\tau^z+2i\Delta_1\sinh q_x \tau^y]\otimes [(2t_2\cos k_y+\mu_2)\sigma^z+2\Delta_2\sin k_y\sigma^y],
\end{equation}

the null condition must be,

\begin{equation}
\begin{split}
&((2t_1\cosh q_x+\mu_1)^2-4\Delta_1^2\sinh^2 q_x)((2t_2\cos k_y+\mu_2)^2+4\Delta_2^2\sin^2k_y)=0,\\
\implies & 2t_1\cosh q_x+\mu_1=\pm 2\Delta_1\sinh q_x.
\end{split}
\label{MKCperp_OBCx}
\end{equation}

As we show for the KC case, the edge states must be of the form, $\Psi(j,k_y)\sim (e^{-q_{x,+}j}-e^{-q_{x,-}j})e^{ik_yy}\Phi$, where $\Phi$ is derived from the null vectors of $H_{MKC,\perp}(iq_x,k_y)$. Denoting $M_2=2t_2\cos k_y+\mu_2$ and $R_2=2\Delta_2\sin k_y$, the two signs in Eqn. (<ref>) imply,

\begin{equation}
\Phi_\pm = \frac{1}{2\sqrt{M_2(M_2\pm \sqrt{M_2^2+R_2^2})}}
\begin{pmatrix}
\pm 1
\end{pmatrix}
\otimes
\begin{pmatrix}
\end{pmatrix}.
\end{equation}

We obtain two solutions for $e^{-q_x}$ from each of the two signs in Eqn. (<ref>),

\begin{equation}
+: e^{-q_x} = \frac{-\mu_1\pm\sqrt{\mu_1^2-4(t_1^2-\Delta_1^2)}}{2(t_1+\Delta_1)},\quad -: e^{-q_x} = \frac{-\mu_1\pm\sqrt{\mu_1^2-4(t_1^2-\Delta_1^2)}}{2(t_1-\Delta_1)}.
\end{equation}

Similarly, localizing only along the y-direction, $k_y\rightarrow iq_y$,

\begin{equation}
H_{MKC,\perp}(k_x,iq_y) = [-(2t_1\cos k_x+\mu_1)\tau^z+2\Delta_1\sin k_x \tau^y]\otimes [(2t_2\cosh q_y+\mu_2)\sigma^z+2i\Delta_2\sinh q_y\sigma^y],
\end{equation}

the null condition stands as,

\begin{equation}
\begin{split}
&((2t_1\cos k_x+\mu_1)^2+4\Delta_1^2\sin^2 k_x)((2t_2\cosh q_y+\mu_2)^2-4\Delta_2^2\sinh^2q_y)=0,\\
\implies & 2t_2\cosh q_y+\mu_2 = \pm 2\Delta_2\sinh q_y.
\end{split}
\label{MKCperp_OBCy}
\end{equation}

Similar to the previous case, the edge states must be of the form, $\Psi(k_x,j)\sim e^{ik_xx}(e^{-q_{y,+}j}-e^{-q_{y,-}j})\Phi$. Denoting $M_1=-(2t_1\cos k_x+\mu_1)$ and $R_1=2\Delta_1\sin k_x$, for the two signs of Eqn. (<ref>) respectively, the null vector is given as,

\begin{equation}
\Phi_\pm = \frac{1}{2\sqrt{M_1(M_1\pm\sqrt{M_1^2+R_1^2})}}
\begin{pmatrix}
\end{pmatrix}
\otimes
\begin{pmatrix}
\mp 1
\end{pmatrix}
\end{equation}

We obtain two solutions for $e^{-q_y}$ from each of the two signs in Eqn. (<ref>),

\begin{equation}
+: e^{-q_y} = \frac{-\mu_2\pm\sqrt{\mu_2^2-4(t_2^2-\Delta_2^2)}}{2(t_2+\Delta_2)},\quad -: e^{-q_y} = \frac{-\mu_2\pm\sqrt{\mu_2^2-4(t_2^2-\Delta_2^2)}}{2(t_2-\Delta_2)}.
\end{equation}

Again, if we localize along both the $\hat{x}$- and $\hat{y}$-directions, $k_x\rightarrow iq_x$, $k_y\rightarrow iq_y$,

\begin{equation}
H_{MKC,\perp}(iq_x,iq_y) = [-(2t_1\cosh q_x+\mu_1)\tau^z+2i\Delta_1\sinh q_x \tau^y]\otimes [(2t_2\cosh q_y+\mu_2)\sigma^z+2i\Delta_2\sinh q_y\sigma^y],
\end{equation}

the null condition is realized as,

\begin{equation}
((2t_1\cosh q_x+\mu_1)^2-4\Delta_1^2\sinh^2 q_x)((2t_2\cosh q_y+\mu_2)^2-4\Delta_2^2\sinh^2q_y)=0.
\end{equation}

This relation does not provide the conditions which occur simultaneously, for which we need to consider the component Bloch Hamiltonians for the MKC perpendicular,

\begin{equation}
\begin{split}
\mc{H}_{\perp,1}(k) =& -[2\mu_2t_1\cos k_x+2\mu_1t_2\cos k_y+2(t_1t_2+\Delta_1\Delta_2)\cos(k_x+k_y)+2(t_1t_2-\Delta_1\Delta_2)\cos(k_x-k_y)+\mu_1\mu_2]\sigma^z\\
&+[2\mu_2\Delta_1\sin k_x+2\mu_1\Delta_2\sin k_y+2(t_2\Delta_1+t_1\Delta_2)\sin(k_x+k_y)+2(t_2\Delta_1-t_1\Delta_2)\sin(k_x-k_y)]\sigma^y\\
\end{split}
\label{Hperp1_Bloch}
\end{equation}

\begin{equation}
\begin{split}
\mc{H}_{\perp,2}(k) =& -[2\mu_2t_1\cos k_x+2\mu_1t_2\cos k_y+2(t_1t_2-\Delta_1\Delta_2)\cos(k_x+k_y)+2(t_1t_2+\Delta_1\Delta_2)\cos(k_x-k_y)+\mu_1\mu_2]\sigma^z\\
&+[2\mu_2\Delta_1\sin k_x-2\mu_1\Delta_2\sin k_y+2(t_2\Delta_1-t_1\Delta_2)\sin(k_x+k_y)+2(t_2\Delta_1+t_1\Delta_2)\sin(k_x-k_y)]\sigma^y\\
\end{split}
\label{Hperp2_Bloch}
\end{equation}

Performing localization $k_x\rightarrow iq_x$ and $k_y\rightarrow iq_y$ then provides the following conditions,

\begin{equation}
\begin{split}
&[2\mu_2t_1\cosh q_x+2\mu_1t_2\cosh q_y+2(t_1t_2+\Delta_1\Delta_2)\cosh(q_x+q_y)+2(t_1t_2-\Delta_1\Delta_2)\cosh(q_x-q_y)+\mu_1\mu_2]\\
&=\pm [2\mu_2\Delta_1\sinh q_x+2\mu_1\Delta_2\sinh q_y+2(t_2\Delta_1+t_1\Delta_2)\sinh(q_x+q_y)+2(t_2\Delta_1-t_1\Delta_2)\sinh(q_x-q_y)],\\
\implies &[(2t_1\cosh q_x+\mu_1)\pm 2\Delta_1\sinh q_x][(2t_2\cosh q_y+\mu_2)\pm 2\Delta_2\sinh q_y]=0,
\end{split}
\end{equation}

\begin{equation}
\begin{split}
&[2\mu_2t_1\cosh q_x+2\mu_1t_2\cosh q_y+2(t_1t_2-\Delta_1\Delta_2)\cosh(q_x+q_y)+2(t_1t_2+\Delta_1\Delta_2)\cosh(q_x-q_y)+\mu_1\mu_2]\\
&=\pm [2\mu_2\Delta_1\sinh q_x-2\mu_1\Delta_2\sinh q_y+2(t_2\Delta_1-t_1\Delta_2)\sinh(q_x+q_y)+2(t_2\Delta_1+t_1\Delta_2)\sinh(q_x-q_y)],\\
\implies &[(2t_1\cosh q_x+\mu_1)\pm 2\Delta_1\sinh q_x][(2t_2\cosh q_y+\mu_2)\mp 2\Delta_2\sinh q_y]=0,
\end{split}
\end{equation}

Each condition for each component Hamiltonian gives rise to two solutions for $e^{-q_x}$ and two for $e^{-q_y}$. The Bloch Hamiltonians after localization are given as,

\begin{equation}
\begin{split}
\mc{H}_{\perp,1}(iq_x,iq_y)=& -[(2t_1\cosh q_x+\mu_1)(2t_2\cosh q_y+\mu_2)+4\Delta_1\Delta_2\sinh q_x\sinh q_y]\sigma^z\\
&+[2\Delta_1\sinh q_x(2t_2\cosh q_y+\mu_2)+2\Delta_2\sinh q_y(2t_1\cosh q_x+\mu_1)]\sigma^y,
\end{split}
\end{equation}

\begin{equation}
\begin{split}
\mc{H}_{\perp,2}(iq_x,iq_y)=& -[(2t_1\cosh q_x+\mu_1)(2t_2\cosh q_y+\mu_2)-4\Delta_1\Delta_2\sinh q_x\sinh q_y]\sigma^z\\
&+[2\Delta_1\sinh q_x(2t_2\cosh q_y+\mu_2)-2\Delta_2\sinh q_y(2t_1\cosh q_x+\mu_1)]\sigma^y.
\end{split}
\end{equation}

Similar to the parallel system, even here we must work with the basis for the full system, $(\tilde{c}_{\boldsymbol{k},\uparrow},c\tilde{c}_{\boldsymbol{k},\downarrow},c^{\dagger}_{-\boldsymbol{k},\uparrow},c^\dagger_{-\boldsymbol{k},\downarrow})^T$. The eigenvectors for the MZMs then depend on whether the parents are topological or trivial, and which boundary conditions are open,  so that we have two cases,

* Case 1: If both the x and y boundary conditions are open, and both the parents are topological, let us have the following condition fulfilled,

\begin{equation}
[(2t_1\cosh q_x+\mu_1)-2\Delta_1\sinh q_x][(2t_2\cosh q_y+\mu_2)-2\Delta_2\sinh q_y]=0.
\end{equation}

Here both the factors are zero since both the parents are topological, so that $2t_1\cosh q_x+\mu_1=2\Delta_1\sinh q_x$ and $2t_2\cosh q_y+\mu_2=2\Delta_2\sinh q_y$. Substituting into Eqns. (S47a) and (S47b), we see that $\mc{H}_{\perp,2}(iq_x,iq_y)=0$ and then the MZM eigenvectors must be null vectors of the matrix,

\begin{equation}
\begin{pmatrix}
d_{1,z}(iq_x,iq_y) & 0 & 0 & d_{1,z}(iq_x,iq_y)\\
0 & 0 & 0 & 0\\
0 & 0 & 0 & 0\\
-d_{1,z}(iq_x,iq_y) & 0 & 0 & -d_{1,z}(iq_x,iq_y)
\end{pmatrix}
\end{equation}

The MZM eigenvectors are then given as $\{\frac{1}{\sqrt{2}}(\ket{00}-\ket{11}),\ket{01},\ket{10}\}$.

* Case 2: Lets suppose that only the x boundary condition is open or only parent 1 is topological. One can show that we must facilitate the following condition,

\begin{equation}
2t_1\cosh q_x+\mu_1=2\Delta_1\sinh q_x.
\end{equation}

The MZM eigenvectors are then null vectors for the following matrix,

\begin{equation}
\begin{pmatrix}
d_{1,z}(iq_x,iq_y/k_y) & 0 & 0 & d_{1,z}(iq_x,iq_y/k_y)\\
0 & d_{2,z}(iq_x,iq_y/k_y) & d_{2,z}(1q_x,iq_y/k_y) & 0\\
0 & -d_{2,z}(iq_x,iq_y/k_y) & -d_{2,z}(1q_x,iq_y/k_y) & 0\\
-d_{1,z}(iq_x,iq_y/k_y) & 0 & 0 & -d_{1,z}(iq_x,iq_y/k_y)
\end{pmatrix}
\end{equation}

which are given as, $\{\frac{1}{\sqrt{2}}(\ket{00}-\ket{11}),\frac{1}{\sqrt{2}}(\ket{01}-\ket{10})\}$.

§ WILSON LOOP

The Wilson loop <cit.> is a unitary operator defined over a closed path as:

\begin{equation}
\begin{aligned}
\mathcal{W} = \overline{\exp}{\Big [ i \int_{BZ} d \bm{k} \cdot \bm{A}(\bm{k}) \Big ]}.
\end{aligned}
\end{equation}

Here $\bm{A} = A_x \hat{k}_x + A_y \hat{k}_y + A_z \hat{k}_z$ is the non-Abelian Berry connection:

\begin{equation}
\begin{aligned}
\bm{A}_{mn}(\bm k) = i\bra{u_m(\bm k)} \nabla_{\bm k} \ket{u_n(\bm k)},
\end{aligned}
\end{equation}

a Hermitian operator as $A_{mn}=A^\ast_{nm}$ in this convention. In the definition above $\ket{u_n(\bm k)}$ are eigenvectors of a Bloch Hamiltonian satisfying $H(\bm k) \ket{u_{n}(\bm k)} = E_{n}(\bm k)  \ket{u_{n}(\bm k)}$ and $1 \leq m, n \leq M$, with $M$ being the number of occupied bands.

The eigenvalues of the Wilson operator defined over a closed path, a Wilson loop, are Gauge independent and unitary. Therefore they can be expressed as $e^{i2\pi \nu_i}$, with $\nu_i$ corresponding to the Wannier centers <cit.>.

To numerically compute the Wilson loop and avoid complications related to a lack of Gauge fixing, we compute the Wilson loop over a discrete and closed path in momentum space divided into $R+1$ segments <cit.>:

\begin{equation}
\begin{aligned}
\mathcal{W}_{mn} = \bra{u_{m}(\bm k_0)} \lim_{R \to \infty} \prod_{i=R}^1  P(\bm k_i) \ket{u_n(\bm k_0)},
\end{aligned}
\end{equation}

where $P(\bm k) = \sum_{n'=1}^M \ket{u_{n'}(\bm k)} \bra{u_{n'}(\bm k)}$ is the projector operator in the occupied subspace as $1 \leq n, m \leq M$, with $M$ the number of occupied bands.

§.§ Properties of the Wilson loop spectrum of a child Hamiltonian

We analytically derive here the numerical results presented in Fig. <ref>, where the Wannier centers of a child Hamiltonian correspond to the addition of the parents' Wannier centers of charge.

The child Hamiltonian for the parallel multiplicative chain is given by Eq. <ref>, repeated here for convenience,

\begin{equation}
\begin{aligned}
H^c_{MKC,||}(k) =& [-(2t_1\cos k+\mu_1)\tau^z+2\Delta_1\sin k\tau^y]\\
&\otimes [(2t_2\cos k+\mu_2)\sigma^z+2\Delta_2\sin k\sigma^y].
\end{aligned}
\end{equation}

Given the tensor product construction of the Hamiltonian, its eigenvectors can be represented using the parents eigendecomposition,

\begin{equation}
\begin{aligned}
\ket{u_1(k)} = \ket{v_{1+}(k)} \otimes \ket{v_{2-}(k)} ; \quad H^c_{MKC,||}(k)\ket{u_1(k)} = \epsilon^{(1)}_+ \epsilon^{(2)}_- \ket{u_1(k)}\\
\ket{u_2(k)} = \ket{v_{1-}(k)} \otimes \ket{v_{2+}(k)} ; \quad H^c_{MKC,||}(k)\ket{u_2(k)} = \epsilon^{(1)}_- \epsilon^{(2)}_+ \ket{u_2(k)} \\
\ket{u_3(k)} = \ket{v_{1-}(k)} \otimes \ket{v_{2-}(k)} ; \quad H^c_{MKC,||}(k)\ket{u_3(k)} = \epsilon^{(1)}_- \epsilon^{(2)}_- \ket{u_3(k)}\\
\ket{u_4(k)} = \ket{v_{1+}(k)} \otimes \ket{v_{2+}(k)} ; \quad H^c_{MKC,||}(k)\ket{u_4(k)} = \epsilon^{(1)}_+ \epsilon^{(2)}_+ \ket{u_4(k)}
\end{aligned}
\end{equation}

where $\ket{v_{1 \pm}(k)}$, are the first parent's eigenvectors with corresponding eigenvalues $\epsilon^{(1)}_{\pm}$, and $\ket{v_{2 \pm}(k)}$ are the eigenvectors of the second parent, with corresponding eigenvalues $\epsilon^{(2)}_{\pm}$, after mapping $t_2 \rightarrow -t_2$ and $\mu_2 \rightarrow -\mu_2$. As $\epsilon^{(1)}_{\pm}=\epsilon^{(2)}_{\pm}$, we obtain doubly degenerate bands and identify $\ket{u_1(k)}, \ket{u_2(k)}$ as the eigenvectors in the occupied subspace at half-filling. This results in a $2 \times 2$ matrix representation for the non-Abelian Berry connection,

\begin{equation}
\begin{aligned}
A(k) &= i
\begin{pmatrix}
\bra{v_{1-}(k)}\partial_k \ket{v_{1-}(k)} + \bra{v_{2+}(k)}\partial_k \ket{v_{2+}(k)}& 0\\
0 & \bra{v_{1+}(k)}\partial_k \ket{v_{1+}(k)} + \bra{v_{2-}(k)}\partial_k \ket{v_{2-}(k)}
\end{pmatrix} \\
\begin{pmatrix}
\bra{v_{1-}(k)}\partial_k \ket{v_{1-}(k)} & 0\\
0 & \bra{v_{1+}(k)}\partial_k \ket{v_{1+}(k)}
\end{pmatrix}
\begin{pmatrix}
\bra{v_{2+}(k)} \partial_k \ket{v_{2+}(k)}& 0\\
0 & \bra{v_{2-}(k)} \partial_k \ket{v_{2-}(k)}
\end{pmatrix}
\\
\implies A &= A_1 + A_2
\end{aligned}
\end{equation}

where $A_1$ and $A_2$ are not the Berry connections for each parent, as they are constructed from the full space and not just the occupied subspace. Importantly, $A_1$ and $A_2$ are Hermitian and diagonal.

\begin{equation}
\begin{aligned}
\mathcal{W} &=
\overline{\exp}{\Big [ i \int_{BZ} dk A(k) \Big ]} \\
&= \overline{\exp}{\Big [ i \int_{BZ} dk A_1(k) +i \int_{BZ} dk A_2(k)\Big ]} \\
&= \overline{\exp} \Big [ i \int_{BZ} dk A_1(k) \Big ] \quad \overline{\exp}{\Big [ i \int_{BZ} dk A_2(k)\Big ]} \\
\implies \mathcal{W}&= \mathcal{W}_1 \mathcal{W}_2
\end{aligned}
\label{eq:wilson_loop_equivalence}
\end{equation}

In order to separate the exponential, we have used the fact that as $A_1$ and $A_2$ are diagonal matrices, they satisfy $[A_1, A_2]=0$. It is worth noting that the hermiticity of $A_1$ and $A_2$ makes $\mathcal{W}_1$ and $\mathcal{W}_2$ unitary operators with unitary eigenvalues. Moreover, due to $A_1$ and $A_2$ being diagonal we conclude $\mathcal{W}_1$, $\mathcal{W}_2$, and consequently $\mathcal{W}$, are diagonal too.

As stated earlier, the Wilson loop $\mathcal{W}$ is a unitary operator, and its  $i$th eigenvalue can therefore be represented by $\exp(i2\pi \nu_i)$. Using Eq. <ref>, we establish a direct relation between the eigenvalues of $\mathcal{W}$ and the ones of $\mathcal{W}_1$ and $\mathcal{W}_2$. We first denote the $i$th eigenvalue of $\mathcal{W}_1$ and $\mathcal{W}_2$ as $\exp(i2\pi \nu^{(1)}_i)$ and $\exp(i2\pi \nu^{(2)}_i)$, respectively. Noting that $\nu_1^{(1)}=\nu_2^{(1)}$ and $\nu_1^{(2)}=\nu_2^{(2)}$, $\exp(i2\pi \nu_i)$ may then be expressed as:

\begin{equation}
\begin{aligned}
\implies
\exp(i2\pi \nu_i) &=  \exp(i2\pi \nu^{(1)}) \exp(i2\pi \nu^{(2)}) \\
\implies \nu_i &= \nu^{(1)} + \nu^{(2)} \mod 1
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
\nu_i &= \nu^{(1)} + \nu^{(2)} \mod 1
\end{aligned}
\end{equation}

This result demonstrates that a child obtained from two topological parents ($\nu^{(1)}= \nu^{(2)}=0.5$) is not distinguished from a child of trivial parents ($\nu^{(1)}= \nu^{(2)}=0$) by means of a Wilson loop spectrum, as $\nu_i=0$ for $i \in \{1, 2\}$ in each case.

We can also examine another formulation for the Wilson loop, in terms of projectors onto occupied states, which is widely-used for numerical calculations <cit.>, and arrive at the same conclusion.

\begin{equation}
\begin{aligned}
\mathcal{W}_{mn} = \bra{u_{m}(\bm k_0)} \lim_{R \to \infty} \prod_{i=R}^1  P(\bm k_i) \ket{u_n(\bm k_0)},
\end{aligned}
\label{WLProj}
\end{equation}

At half-filling the projector operator corresponds to

\begin{equation}
\begin{aligned}
&= \ket{u_{1}(k)} \bra{u_1(k)} + \ket{u_{2}(k)} \bra{u_2(k)} \\
&= \ket{v_{1-}(k)} \bra{v_{1-}(k)} \otimes \ket{v_{2+}(k)} \bra{v_{2+}(k)} + \ket{v_{1+}(k)} \bra{v_{1+}(k)} \otimes \ket{v_{2-}(k)} \bra{v_{2-}(k)}
\end{aligned}
\end{equation}

Consequently, after a discretization of the BZ into $R+1$ segments such that the wavefunctions vary smoothly enough,

\begin{equation}
\begin{aligned}
\lim_{R \to \infty} \prod_{i=R}^{1} P(k_i)
&= \lim_{R \to \infty} \prod_{i=R}^{1} \Big [ \ket{v_{1-}(k)} \bra{v_{1-}(k)} \otimes \ket{v_{2+}(k)} \bra{v_{2+}(k)} + \ket{v_{1+}(k)} \bra{v_{1+}(k)} \otimes \ket{v_{2-}(k)} \bra{v_{2-}(k)}  \Big ] \\
&= \lim_{R \to \infty} \prod_{i=R}^{1} \Big [ \ket{v_{1-}(k)} \bra{v_{1-}(k)} \otimes \ket{v_{2+}(k)} \bra{v_{2+}(k)} \Big ] + \prod_{i=R}^{1} \Big [ \ket{v_{1+}(k)} \bra{v_{1+}(k)} \otimes \ket{v_{2-}(k)} \bra{v_{2-}(k)} \Big ] \\
&= \lim_{R \to \infty} \prod_{i=R}^{1}  \ket{v_{1-}(k)} \bra{v_{1-}(k)} \otimes \prod_{i=R}^{1}  \ket{v_{2+}(k)} \bra{v_{2+}(k)}  + \prod_{i=R}^{1} \ket{v_{1+}(k)} \bra{v_{1+}(k)} \otimes \prod_{i=R}^{1}  \ket{v_{2-}(k)} \bra{v_{2-}(k)}  \\
&= \lim_{R \to \infty} \prod_{i=R}^{1}  P_{1-}(k_i) \otimes \prod_{i=R}^{1}  P_{2+}(k_i)  + \prod_{i=R}^{1}  P_{1+}(k_i) \otimes \prod_{i=R}^{1}  P_{2-}(k_i)
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
\implies \mathcal{W}_{mn} = \bra{u_{m}(k_0)} \lim_{R \to \infty} \prod_{i=R}^{1}  P_{1-}(k_i) \otimes \prod_{i=R}^{1}  P_{2+}(k_i)   \ket{u_n(k_0)} +
\bra{u_{m}(k_0)} \lim_{R \to \infty} \prod_{i=R}^{1}  P_{1+}(k_i) \otimes \prod_{i=R}^{1}  P_{2-}(k_i)  \ket{u_n(k_0)}
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
\mathcal{W}_{11} &=
\bra{v_{1+}(k_0)} \lim_{R \to \infty} \prod_{i=R}^{1}  P_{1+}(k_i) \ket{v_{1+}(k_0)} \bra{v_{2-}(k_0)} \prod_{i=R}^{1}  P_{2-}(k_i)  \ket{v_{2-}(k_0)} \\
\mathcal{W}_{11} &=
\bra{v_{1-}(k_0)} \lim_{R \to \infty} \prod_{i=R}^{1}  P_{1-}(k_i) \ket{v_{1-}( k_0)} \bra{v_{2+}(k_0)} \prod_{i=R}^{1}  P_{2+}(k_i)  \ket{v_{2+}(k_0)} \\
\mathcal{W}_{21} &= \mathcal{W}_{12} = 0 \\
\end{aligned}
\end{equation}

indicating that $\mathcal{W}_{mn}$ is represented by a diagonal matrix that can be written as $\mathcal{W}=\mathcal{W}_1 \mathcal{W}_2$, leading to the same conclusion.

One can similarly work out the Wannier spectra for the perpendicular MKC, given by the child Hamiltonian in Eqn. (<ref>),

\begin{equation}
H^c_{MKC,\perp}=[-(2t_1\cos k_x+\mu_1)\tau^z+2\Delta_1\sin k_x\tau^y]\otimes [(2t_2\cos k_y+\mu_2)\sigma^z+2\Delta_2\sin k_y\sigma^y].
\end{equation}

Assuming a similar convention to the MKC parallel case described previously while replacing $k$ by the vector $\boldsymbol{k}=(k_x,k_y)$, the definition of $\mathbf{A}(\boldsymbol{k})$ indicates that one should have the following non-Abelian Berry connection vector,

\begin{equation}
\begin{split}
\mathbf{A}(\boldsymbol{k}) =& i
\begin{pmatrix}
\bra{v_{1-}(k_x)}\partial_{k_x}\ket{v_{1-}(k_x)}\boldsymbol{\hat{e}_x}+\bra{v_{2+}(k_y)}\partial_{k_y}\ket{v_{2+}(k_y)}\boldsymbol{\hat{e}_y} & 0\\
0 & \bra{v_{1+}(k_x)}\partial_{k_x}\ket{v_{1+}(k_x)}\boldsymbol{\hat{e}_x}+\bra{v_{2-}(k_y)}\partial_{k_y}\ket{v_{2-}(k_y)}\boldsymbol{\hat{e}_y}
\end{pmatrix}
=& i
\begin{pmatrix}
\bra{v_{1-}(k_x)}\partial_{k_x}\ket{v_{1-}}\boldsymbol{\hat{e}_x} & 0\\
0 & \bra{v_{1+}(k_x)}\partial_{k_x}\ket{v_{1+}}\boldsymbol{\hat{e}_x}
\end{pmatrix}
\begin{pmatrix}
\bra{v_{2+}(k_y)}\partial_{k_y}\ket{v_{2+}(k_y)}\boldsymbol{\hat{e}_y} & 0\\
0 & \bra{v_{2-}(k_y)}\partial_{k_y}\ket{v_{2-}(k_y)}\boldsymbol{\hat{e}_y}
\end{pmatrix}
=& A_1\boldsymbol{\hat{e}_x}+A_2\boldsymbol{\hat{e}_y}.
\end{split}
\end{equation}

We first consider the formulation of the Wilson loop in terms of projectors onto occupied states as written in Eqn. (<ref>). At half-filling, the projector onto occupied states for the perpendicular MKC is given as,

\begin{equation}
\begin{split}
P(k_x,k_y) =& \ket{u_1(k_x,k_y)}\bra{u_1(k_x,k_y)}+\ket{u_2(k_x,k_y)}\bra{u_2(k_x,k_y)},\\
=& \ket{v_{1+}(k_x)}\bra{v_{1+}(k_x)}\otimes\ket{v_{2-}(k_y)}\bra{v_{2-}(k_y)}+\ket{v_{1-}(k_x)}\bra{v_{1-}(k_x)}\otimes\ket{v_{2+}(k_y)}\bra{v_{2+}(k_y)}
\end{split}
\end{equation}

One can compute the Wilson loop by integrating over either $k_x$ or $k_y$. Let us assume that the BZ along $k_x$ direction is discretized into $R+1$ segments for a given $k_y$ (assuming sufficient smoothness for the wavefunctions), the other case follows similarly,

\begin{equation}
\begin{split}
\lim_{R\rightarrow\infty}\prod_{i=R}^1P(k_{xi},k_y)=& \lim_{R\rightarrow\infty}\prod_{i=R}^1\bigg{[}\ket{v_{1+}(k_{xi})}\bra{v_{1+}(k_{x1})}\otimes\ket{v_{2-}(k_y)}\bra{v_{2-}(k_y)}+\ket{v_{1-}(k_{xi})}\bra{v_{1-}(k_{xi})}\otimes\ket{v_{2+}(k_y)}\bra{v_{2+}(k_y)}\bigg{]},\\
=& \lim_{R\rightarrow\infty}\bigg{[}\prod_{i=R}^1\ket{v_{1+}(k_{xi})}\bra{v_{1+}(k_{xi})}\bigg{]}\otimes\ket{v_{2-}(k_{y})}\bra{v_{2-}(k_{y})}\\
=& \lim_{R\rightarrow\infty}\bigg{[}\prod_{i=R}^1P_{1+}(k_{xi})\bigg{]}\otimes P_{2-}(k_y)+\lim_{R\rightarrow\infty}\bigg{[}\prod_{i=R}^1P_{1-}(k_x)\bigg{]}\otimes P_{2+}(k_y).
\end{split}
\end{equation}

We get the Wilson loop matrix from the definition as follows,

\begin{equation}
\begin{split}
\mathcal{W}_{mn}(k_y)=\bra{u_m(k_{x0},k_y)}\lim_{R\rightarrow\infty}\bigg{[}\prod_{i=R}^1P_{1+}(k_{xi})\bigg{]}\otimes P_{2-}(k_y)+\lim_{R\rightarrow\infty}\bigg{[}\prod_{i=R}^1P_{1-}(k_x)\bigg{]}\otimes P_{2+}(k_y)\ket{u_n(k_{x0},k_y)},
\end{split}
\label{WLMKCperp}
\end{equation}

so that $\mathcal{W}_{11}(k_y) = \mathcal{W}_{11}$ and $\mathcal{W}_{22}(k_y) = \mathcal{W}_{22}$ are expressed as,

\begin{equation}
\begin{split}
\mathcal{W}_{11} =& \bra{v_{1+}(k_{x0})}\lim_{R\rightarrow\infty}\bigg{[}\prod_{i=R}^1P_{1+}(k_{xi})\bigg{]}\ket{v_{1+}(k_{x0})},\\
\mathcal{W}_{22} =& \bra{v_{1-}(k_{x0})}\lim_{R\rightarrow\infty}\bigg{[}\prod_{i=R}^1P_{1-}(k_{xi})\bigg{]}\ket{v_{1-}(k_{x0})},\\
\mathcal{W}_{12} =& \mathcal{W}_{21}=0.
\end{split}
\end{equation}

The projector due to the eigenvectors of the second parent contract with the respective eigenvectors of the second parent in the tensor product eigenvectors for the occupied basis to produce all four matrix elements $\mathcal{W}_{mn}(k_y)=\mathcal{W}_{mn}$ as $k_y$ independent terms. This implies that the Wilson loop computed as an integral over a given momentum component is independent of the other momentum component. Previously we mentioned that the Wilson loop eigenvalues are of the form $e^{i2\pi\nu_i}$ due to the unitary nature of $\mathcal{W}$. Here $\nu_i$ is the $i$th Wannier charge center. Let us refer to the Wannier charge spectra due to the Wilson loop along $k_x$ as $\{\nu_{i}(k_y)\}_x$. The $i$th Wannier charge center for the child Hamiltonian computed as an integral over $k_x$ for each value of $k_y$, $\nu_{i}(k_y)$, is then equal to the Wannier charge center of the parent Hamiltonian that is calculated along a loop across $k_x$ (here parent 1). We similarly find the $i$th Wannier charge center for the child Hamiltonian computed by integrating over $k_y$ for a given $k_x$, $\nu_{i}(k_x)$, is equal to the Wannier charge center of the parent Hamiltonian that is calculated across a loop along $k_y$ (here parent 2), such that

\begin{equation}
\begin{split}
\nu_{i}(k_y)=&\nu^{(1)}\text{mod 1},\\
\nu_{i}(k_x)=&\nu^{(2)}\text{mod 1},
\end{split}
\end{equation}

where $\nu^{(j)}$ is the Wannier charge center due to the $j$th parent. Here the spectrum for both $\{\nu_i(k_y)\}$ and $\{\nu_i(k_x)\}$ are doubly degenerate (up to mod 1) and equal to the respective parent Wannier charge center values.

§ CALCULATION FOR WINDING NUMBER IN THE MKC PERPENDICULAR CASE:

Consider the Bloch Hamiltonians for the component Hamiltonians in the MKC perpendicular case,

\begin{equation}
\begin{split}
\mc{H}_{\perp,1}(\boldsymbol{k})=&-((\mu_1+2t_1\cos k_x)(\mu_2+2t_2\cos k_y)-4\Delta_1\Delta_2\sin k_x\sin k_y)\sigma^z\\
&+(2\Delta_1\sin k_x(\mu_2+2t_2\cos k_y)+2\Delta_2\sin k_y(\mu_1+2t_1\cos k_x))\sigma^y = (0,d_{1,y},d_{1,z})\cdot\boldsymbol{\sigma},
\end{split}
\end{equation}

\begin{equation}
\begin{split}
\mc{H}_{\perp,2}(\boldsymbol{k})=&-((\mu_1+2t_1\cos k_x)(\mu_2+2t_2\cos k_y)+4\Delta_1\Delta_2\sin k_x\sin k_y)\sigma^z\\
&+(2\Delta_1\sin k_x(\mu_2+2t_2\cos k_y)-2\Delta_2\sin k_y(\mu_1+2t_1\cos k_x))\sigma^y=(0,d_{2,y},d_{2,z})\cdot\boldsymbol{\sigma}.
\end{split}
\end{equation}

Since we want to plot the Bloch vectors $(d_{1,y},d_{1,z})$ and $(d_{2,y},d_{2,z})$ for varying $k_x$ at given $k_y$ and vice-versa in PBC on both directions, let us find the locus of the curve for parameter $k_x$ at given $k_y$. Denote, $M_2=\mu_2+2t_2\cos k_y$ and $R_2=2\Delta_2\sin k_y$. Further denote, $\cos\theta=\frac{M_2}{\sqrt{M_2^2+R_2^2}}$ and $\sin\theta=\frac{R_2}{\sqrt{M_2^2+R_2^2}}$. Then the locus of the parametric curve $(d_{1,y},d_{1,z})$ is shown below,

\begin{equation}
\frac{(\cos\theta d_{1,y}+\sin\theta d_{1,z})^2}{4\Delta_1^2(M_2^2+R_2^2)}+\frac{(\cos\theta d_{1,z}-\sin\theta d_{1,y}+\mu_1\sqrt{M_2^2+R_2^2})^2}{4t_1^2(M_2^2+R_2^2)} = 1.
\end{equation}

Since we plot for $t_1=\Delta_1$ in the main text, implementing this condition we further get,

\begin{equation}
(\cos\theta d_{1,y}+\sin\theta d_{1,z})^2+(\cos\theta d_{1,z}-\sin\theta d_{1,y}+\mu_1\sqrt{M_2^2+R_2^2}) = (2t_1\sqrt{M_2^2+R_2^2})^2.
\end{equation}

It is easy to notice that this is a circle whose coordinates have been rotated by and angle $\theta$ and the center, and radii have been modulated by the other parent by $\sqrt{M_2^2+R_2^2}$, which however does not change the range of parameters where the system is topological. For PBC with $L_y$ sites in the y-direction, we obtain $L_y$ number of such circles rotated at $L_y$ angles, which is essentially obtain in the main text.

§ DEPENDENCE OF ENERGY WITH PARAMETER $\MU$ NEAR CRITICAL POINTS:

We will consider two cases for the dispersion of the MKC parallel Hamiltonian, (i)$\mu_1=\mu_2=\mu$, $t_1=t_2=t$ and $\Delta_1=\Delta_2=\Delta$ and (ii)$\frac{\mu_1}{t_1}=-\frac{\mu_2}{t_2}=\frac{\mu}{t}$ and $\Delta_1=\Delta_2=\Delta$. For (i), the child dispersion is of the form,

\begin{equation}
E(k) = (2t\cos k+\mu)^2+4\Delta^2\sin^2k.
\end{equation}

Let $\mu = -2t+\delta\mu$ near the critical point at $k=0$, so that we must have,

\begin{equation}
E(k=0)\approx (2t-2t+\delta\mu)^2, \implies E(k=0)\sim \delta\mu^2.
\label{Evsmuexplplus}
\end{equation}

For (ii), the child dispersion is given as,

\begin{equation}
\begin{split}
E(k) =& \sqrt{((2t\cos k+\mu)^2+4\Delta^2\sin^2k)((2t\cos k-\mu)^2+4\Delta^2\sin^2k)},\\
=& \sqrt{(4t^2\cos^2k+4\Delta^2\sin^2k+\mu^2)^2-16t^2\mu^2\cos^2k}.
\end{split}
\end{equation}

Again, we let $\mu=-2t+\delta\mu$ near the critical point, $k=0$, and we then get,

\begin{equation}
\begin{split}
E(k=0)&\approx \sqrt{(4t^2+(-2t+\delta\mu)^2)^2-16t^2(-2t+\delta\mu)^2},\\
&\approx 4t^2-(-2t+\delta\mu)^2\approx 4t\delta\mu+\delta\mu^2\implies E(k=0)\sim \delta\mu.
\end{split}
\label{Evsmuexplminus}
\end{equation}

§ ROBUSTNESS OF MMZMS IN THE MKC PARALLEL SYSTEM:

Here, we show additional slab spectra for the MKC as a function of chemical potential $\mu_1$ for a variety of disorder terms, for $\mu_1 = -\mu_2$ in Fig. <ref>, $\mu_2 = 0$ in Fig. <ref>, and $\mu_2 = 3$ in Fig. <ref>. Stability of MMZMs against on-site disorder terms proportional to  $\tau^i \sigma^j$ correspond to presence of zero-energy modes over a finite interval in $\mu_1$.

figureFor the case $\mu_1=-\mu_2$ again the MMZMs for the MKC parallel system are robust for all $\tau^i\sigma^j$ disorders except $\tau^x\sigma^x$.

figureChecking for robustness for given $\mu_2=0$ and varying across $\mu_1$ for all disorder $\tau^i\sigma^j$ combinations. The MMZM in the range $(-2t_1, 2t_1)$ is always robust except $\tau^x\sigma^x$, while MZMs beyond that range succumb to disorder for $\tau^i\sigma^x$.

figureChecking for robustness for given $\mu_2=3$ and varying across $\mu_1$ for all disorder $\tau^i\sigma^j$ combinations. The MMZM in the range $(-2t_1, 2t_1)$ is always robust except $\tau^x\sigma^x$, while MZMs beyond that range succumb to disorder for $\tau^x\sigma^j$.

Bogoliubov basis and edge modes for the finite length MKC:

The two-band Kitaev chain is diagonalized in the Bogoliubov basis, as $H = \sum_k\Lambda_k(\eta^\dagger_k\eta_k+\frac{1}{2})$ so that the Bogoliubov operator in the Majorana representation may be written as,

\begin{equation}
\begin{aligned}
\eta_k = \frac{1}{2}\sum_j\phi_{kj}\gamma_{2j-1}+i\psi_{kj}\gamma_{2j}.
\end{aligned}
\end{equation}

The edge modes are given by the momenta(complex) for which one gets the energy $\Lambda_k^2=0$. It is intuitively clear that the coefficients in the Majorana representation signify the two edge modes, and comprised of fully odd or fully even Majorana components.

Therefore, taking advantage of the fact that the Multiplicative phase can be expressed as a tensor product of individual two band Hamiltonians, the Bogoliubov transformation is given by unitary operators which must be also in tensor product structure and must individually diagonalize each of the Hamiltonian in the tensor product. Let us denote the unitary transformations in terms of the Bogoliubov angles of the two Hamiltonians as $\theta_{k,1}$ and $\theta_{k,2}$ respectively. Then the similarity transformation is given as,

\begin{equation}
\begin{aligned}
U_1\otimes U_2 =
\begin{pmatrix}
\cos\theta_{k,1} & -i\sin\theta_{k,1}\\
i\sin\theta_{k,1} & \cos\theta_{k,1}
\end{pmatrix}
\otimes
\begin{pmatrix}
\cos\theta_{k,2} & -i\sin\theta_{k,2}\\
i\sin\theta_{k,2} & \cos\theta_{k,2}
\end{pmatrix}
\end{aligned}
\end{equation}

Operating on the basis $(c_{k,\uparrow}, c_{k,\downarrow}, c^\dagger_{-k,\uparrow}, c^\dagger_{-k,\downarrow})^T$, and then carrying out an inverse Fourier transform, one gets the Bogoliubov operators for the MKC(parallel),

\begin{equation}
\begin{aligned}
\eta_{k,\uparrow} = & \frac{1}{2}\sum_j\frac{1}{\sqrt{N}}e^{-ikj}\cos\theta_{k,2}e^{i\theta_{k,1}}\gamma_{2j-1,\uparrow}\\
\end{aligned}
\end{equation}

Consider now for the MKC. For the sake of convenience, we express the Bloch Hamiltonian as,

\begin{equation}
\begin{aligned}
H_{MKC} = (-M_1\tau^z+D_1\tau^y)\otimes (M_2\sigma^z+D_2\sigma^y),
\end{aligned}
\end{equation}

where $M_i$, $D_i$ for $i\in (1,2)$ is $k_x$ or $k_y$ dependent based on the orientation of the parents. As done in the KC, we subject the $H_{MKC}$ to the following similarity transformation, $S=\frac{1}{\sqrt{2}}(\tau^0-i\tau^y)\sigma^0$, and put off the Fourier transformation until later,

\begin{equation}
\begin{aligned}
\tilde{H}_{MKC} = SH_{MKC}S^\dagger = (-M_1\tau^x+D_1\tau^y)\otimes(M_2\sigma^z+D_2\sigma^y).
\end{aligned}
\end{equation}

Notice that the Hamiltonian is now block off-diagonal, so that it one can express it in a similar way as in KC,

\begin{equation}
\begin{aligned}
\tilde{H}_{MKC} =& \tau^+\tilde{H}^R_{MKC}+\tau^-\tilde{H}^L_{MKC},\\ =&\tau^+[(-M_1-iD_1)(M_2\sigma^z+D_2\sigma^y)]\\
\end{aligned}
\end{equation}

where $\tilde{H}^R_{MKC}=(-M_1-iD_1)(M_2\sigma^z+D_2\sigma^y)$ and $\tilde{H}^L_{MKC}=(-M_1+iD_1)(M_2\sigma^z+D_2\sigma^y)$. Expressing in the position basis, we now have,

\begin{equation}
\begin{aligned}
\tilde{\mathbb{H}}^L_{MKC} =& [-(2t_1t_2+\mu_1\mu_2)\sigma^z+2i\Delta_1\Delta_2\sigma^y]\mathbb{I}_x\\
\tilde{\mathbb{H}}^R_{MKC} =&
\end{aligned}
\end{equation}

where $\mathbb{M}_\pm = \delta_{x+1,y}\pm\delta_{x,y+1}$ and $\mathbb{N}_{\pm}=\delta_{x+2,y}\pm\delta_{x,y+2}$. From the Schrodinger equation, it is easy to see,

\begin{equation}
\begin{aligned}
\tilde{\mathbb{H}}^R_{MKC}\tilde{\mathbb{U}}_R=E\tilde{\mathbb{U}}_L;\quad \tilde{\mathbb{H}}^L_{MKC}\tilde{\mathbb{U}}_L=E\tilde{\mathbb{U}}_R,
\end{aligned}
\end{equation}

where $\tilde{\mathbb{U}}_{R/L}=(\tilde{\mathbb{U}}_{R/L,\uparrow},\tilde{\mathbb{U}}_{R/L,\downarrow})^T$. We intend to solve for $E=0$, and therefore assume solutions of the form $\tilde{\mathbb{U}}^{R/L}=(u_{R/L,\uparrow}, u_{R/L,\downarrow})^Ts^j$, for the above difference equations. We illustrate for $\tilde{\mathbb{H}^{L/R}_{MKC}}$, where we must have

\begin{equation}
\begin{aligned}
\text{det}
\begin{pmatrix}
A_{L/R} & B_{L/R}\\
-B_{L/R} & -A_{L/R}
\end{pmatrix}
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}

\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
\end{aligned}
\end{equation}

Now for the MKC parallel, one may write the two wavefunctions as,

\begin{equation}
\begin{aligned}
\Psi_1(j) =& as_{1+}^j+bs_{1-}^j+as_{2+}^j+bs_{2-}^j,\\
\Psi_2(j) =& as_{1+}^j+bs_{1-}^j+bs_{2+}^j+as_{2-}^j,
\end{aligned}
\end{equation}

which for similar boundary conditions must give us,

\begin{equation}
\begin{aligned}
&R_1^{N+1}\sin(N+1)\theta_1+R_2^{N+1}\sin(N+1)\theta_2 = 0,\\
&R_1^{N+1}\sin(N+1)\theta_1-R_2^{N+1}\sin(N+1)\theta_2 = 0,
\end{aligned}
\end{equation}

which for $R_1=R_2$, must give us,

\begin{equation}
\begin{aligned}
\sin\frac{1}{2}(N+1)(\theta_1+\theta_2)\cos\frac{1}{2}(N+1)(\theta_1-\theta_2) = 0,
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
\sin\frac{1}{2}(N+1)(\theta_1-\theta_2)\cos\frac{1}{2}(N+1)(\theta_1+\theta_2) = 0,
\end{aligned}
\end{equation}

which for the sine as zero must give us,

\begin{equation}
\begin{aligned}
&\theta_1-\theta_2 = \frac{2n\pi}{N+1}, \quad \theta_1+\theta_2 = \frac{2m\pi}{N+1},\\
&\mu_1 = \sqrt{t_1^2-\Delta_1^2}\cos\frac{2(n+m)\pi}{N+1},\\
&\mu_2 = \sqrt{t_2^2-\Delta_2}\cos\frac{2(n-m)\pi}{N+1}
\end{aligned}
\end{equation}

Let us look at the three types of solutions one may get from the above equation,

* The first solution, $2t_1\cosh q+\mu_1=\pm2\Delta_1\sinh q$ with the other term non-zero, provides,

\begin{pmatrix}
\end{pmatrix}
\begin{pmatrix}
\pm 1
\end{pmatrix}

while $(a_2,b_2)^T$ is function of both sets of parameters, $(\mu_i,t_i,\Delta_i)$ for $(i=1,2)$.

* The second solution, $2t_2\cos q+\mu_2=\pm2\Delta_2\sinh q$ with the first term non-zero, provides,

\begin{pmatrix}
\end{pmatrix}
\begin{pmatrix}
\pm 1
\end{pmatrix}

and vice versa.
The first two solutions are valid when either one has $|\mu_1|<2t_1$ or $|\mu_2|<2t_2$ respectively but not both of the conditions together. This implies that they are always valid when one of the parent Hamiltonians exist in the topological phase and account for the two $e^{-q}<1$ solutions which make up the x-dependent part of the wavefunction.

* The third case is where both the first and second solution are fulfilled and the two vector part is of the form,

\begin{pmatrix}
\end{pmatrix}
\otimes
\begin{pmatrix}
\end{pmatrix}
\begin{pmatrix}
\pm 1
\end{pmatrix}
\otimes
\begin{pmatrix}
\pm 1
\end{pmatrix}

But the x-dependent part now has four solutions for $e^{-q}<1$ for both parents in the topological phase which must account for two Majorana zero modes at each end of the chain. Using the following expressions,

\begin{equation}
\begin{aligned}
e^{-q_{1+/2+}} =& \frac{\mp\mu_{1/2}+\sqrt{\mu_{1/2}^2-4(t_{1/2}^2-\Delta_{1/2}^2)}}{2(\Delta_{1/2}\pm t_{1/2})} \\
e^{-q_{1-/2-}} =& \frac{\mp\mu_{1/2}-\sqrt{\mu_{1/2}^2-4(t_{1/2}^2-\Delta_{1/2}^2)}}{2(\Delta_{1/2}\pm t_{1/2})}
\end{aligned}
\end{equation}

we express the two edge modes satisfying the boundary conditions as,

\begin{aligned}
\Psi(x)\sim
\begin{pmatrix}
\pm 1
\end{pmatrix}
\otimes
\begin{pmatrix}
\pm 1
\end{pmatrix}
\begin{cases}
\end{cases}
\end{aligned}

These solutions account for the cases when any two $e^{-q}<1$ values arising from different parent Hamiltonians are equal, which are along the lines $\frac{\mu_1}{2t_1}=\pm\frac{\mu_2}{2t_2}$ and $\mu_1=0=\mu_2$.

Let us consider $\theta_+=\theta$ and $\theta_-=\delta$, which means the phase difference $\theta_1-\theta_2=2\delta$ and we have previously considered the case where $2\delta=\pi$. For the more general case,

\begin{equation}
\begin{split}
&\frac{1-\cos 2(N+2)\theta}{1-\cos\theta}=\frac{1-\cos 2(N+2)\delta}{1-\cos 2\delta},\\
&1-\frac{\sin^2(N+2)\delta}{\sin^2\delta}=\cos 2(N+2)\theta-\frac{\sin^2(N+2)\delta}{\sin^2\delta}\cos 2\theta.
\end{split}
\end{equation}

Here, a similar situation to the case for which $2\delta=\pi$ occurs if the following condition is satisfied,

\begin{equation}
\begin{split}
\implies \delta = \frac{n\pi}{N+2},\quad n\in\{1,...,N+2\},\\
\implies \delta = \frac{n\pi}{N+1},\frac{m\pi}{N+3},\quad n\in\{1,...,N+1\},m\in\{1,...,N+3\},
\end{split}
\end{equation}

which again leads to the values of $\theta$ equivalent to Eqn. (<ref>),

\begin{equation}
\begin{split}
&1-\cos 2(N+2)\theta =0,\implies, \theta = \frac{n\pi}{N+2},\quad n\in\{1,...,N+2\},\\
&\cos 2\theta-\cos 2(N+2)\theta =0,\implies \theta = \frac{n\pi}{N+1},\frac{m\pi}{N+3}, \quad n\in\{1,...,N+1\},m\in\{1,...,N+3\}.
\end{split}
\end{equation}

Considering $R_1=R_2$, let us assume, $A_1=-A_2$, which automatically implies, $B_1=-B_2$, so that the general wavefunction, (<ref>) is a superposition of two sine waves,

\begin{equation}
\Psi(j) = 2iR_1^j(A_1\sin(\theta_1j)+B_1\sin(\theta_2j)).
\end{equation}


# Towards Modelling Coherence in Spoken Discourse

Rajaswa Patil∗
IIIT Delhi, BITS Pilani
<EMAIL_ADDRESS>Yaman Kumar Singla∗
IIIT Delhi, Adobe, SUNY at Buffalo
<EMAIL_ADDRESS>Rajiv Ratn Shah
IIIT Delhi
<EMAIL_ADDRESS>Mika Hama
Second Language Testing Inc.
<EMAIL_ADDRESS>Roger Zimmermann
National University of Singapore
<EMAIL_ADDRESS>∗Authors (listed in alphabetical order) contributed
equally.

###### Abstract

While there has been significant progress towards modelling coherence in
written discourse, the work in modelling spoken discourse coherence has been
quite limited. Unlike the coherence in text, coherence in spoken discourse is
also dependent on the prosodic and acoustic patterns in speech. In this paper,
we model coherence in spoken discourse with audio-based coherence models. We
perform experiments with four coherence-related tasks with spoken discourses.
In our experiments, we evaluate machine-generated speech against the speech
delivered by expert human speakers. We also compare the spoken discourses
generated by human language learners of varying language proficiency levels.
Our results show that incorporating the audio modality along with the text
benefits the coherence models in performing downstream coherence related tasks
with spoken discourses.

##  1 Introduction

Discourse coherence is a property of the organization and semantic
relationships within a structured group of written or spoken utterances.
Coherence can be used as an auxiliary metric to evaluate the quality of a
given discourse. Previously, discourse coherence has been used to evaluate
written discourses for tasks like essay scoring and readability assessment
(Miltsakaki and Kukich, 2004; Burstein et al., 2010; Mesgar and Strube, 2018).
Coherence based metrics and objectives have also been used to evaluate and
improve text-based artificial natural language generation systems (Lapata and
Barzilay, 2005; Park and Kim, 2015; Kiddon et al., 2016) for tasks like text-
summarization (Barzilay and Lapata, 2008; Nenkova and McKeown, 2011), machine-
translation (Smith et al., 2016), language modelling (Iter et al., 2020; Lan
et al., 2020) and conversation thread reconstruction (Joty et al., 2018).
Therefore, modelling discourse coherence has become an essential task in
computational linguistics with a variety of downstream applications.

Most of the previous work dealing with coherence modelling has been limited to
text-based coherence Joty et al. (2019), and there has been limited work on
modelling spoken discourse coherence as a task (Wang et al., 2013). The few
studies which have tried to work on spoken discourse have done so by
transcribing speech and then applying text-coherence modelling methods on it
Wang et al. (2019). Modelling coherence of a spoken discourse with its text-
transcriptions is an inherently lossy and challenging task (Tappe and
Schilder, 1998). On the one hand, crucial cues of speech such as pauses, tonal
variations, speed changes, stress, rhythm and intensity are lost while
transcribing it, and on the other, the transcription is itself a cumbersome
process with involved logistics and errors from automatic speech recognition
(ASR) systems (Errattahi et al., 2018). Various studies in linguistics have
highlighted the importance of prosody in providing a structure to the spoken
discourse (Nakajima and Allen, 1993; Degand and Simon, 2009). By incorporating
the audio modality along with the text-transcription of the speech, we can
better model the coherence in spoken discourse. In this paper, we make the
following contributions:

  * •

We model coherence in spoken discourse with audio and text-based neural
coherence models, which to the best of our knowledge, has not been previously
explored.

  * •

We perform experiments with four coherence-related tasks to inspect the role
of audio modality in modelling spoken discourse coherence: _Speaker Change
Detection_ (SCD), _Artificial Speech Evaluation_ (ASE), _Topic Change
Detection_ (TCD) and _Speech Response Scoring_ (SRS).

  * •

We show that incorporating the audio modality from speech benefits a coherence
model in capturing discourse coherence and in performing various downstream
applications with human and machine-generated speech.

  * •

We also show that the audio modality becomes significantly important while
modelling coherence in the less structured and noisy spoken discourses.

##  2 Background

Early work in modelling discourse coherence focused on extracting features
based on the Centering Theory (Grosz et al., 1995) and entity transitions in
the text (Lapata and Barzilay, 2005; Elsner et al., 2007). Barzilay and Lapata
(2008) introduced the entity grid representation of discourse, which was based
on discourse entities and their grammatical role transitions. The entity grid
model was further improved for coherence-related tasks by Elsner and Charniak
(2011), Feng and Hirst (2012) and Louis and Nenkova (2012). Parallely, many
works (Pitler and Nenkova, 2008; Lin et al., 2011; Feng et al., 2014)
performed coherence-related tasks based on discourse relations in the text,
parsed with theories like the Rhetorical Structure Theory (RST) (Thompson and
Mann, 1987) and the Lexicalized Tree Adjoining Grammar for discourse (D-LTAG)
(Webber, 2004) with the Penn Discourse Treebank (PDTB) (Prasad et al., 2008)
styled annotations. Notably, the features based on the RST-encodings were
found to be useful for modelling coherence in spoken discourse (Wang et al.,
2019) and more efficient than the PDTB-encodings for modelling text coherence
as well (Feng et al., 2014). Further, Guinaudeau and Strube (2013) and Mesgar
and Strube (2015) proposed graph representation-based approaches to model
coherence in text.

Following the advances in deep neural network architectures and distributed
representations, there has been much progress towards developing neural models
of discourse coherence which provide significant performance gains over the
traditional feature-based models. The entity grid representation of discourse
got extended with neural architectures. Tien Nguyen and Joty (2017) proposed
the neural entity grid model, which performed convolutions over the entity
grid representations. Further, Joty et al. (2018) lexicalized the neural
entity grid model by attaching the entities to their respective grammatical
roles in the entity grid embeddings.

Neural coherence models can be broadly classified into two categories:
generative models and discriminative models. On the one hand, generative
coherence models deal with modelling the conditional probabilities of a
sentence being coherent with a given set of preceding sentences (Li and
Jurafsky, 2017; Logeswaran et al., 2018). On the other hand, discriminative
coherence models are trained to classify coherent and incoherent texts. It has
been previously shown that modelling local coherence with discriminative
models can be beneficial for capturing both the local and the global contexts
of coherence with good approximation (Moon et al., 2019; Xu et al., 2019).
Similarly, capturing relations and similarities between sentences at a local
level with neural models can be helpful with coherence-related tasks (Li and
Hovy, 2014; Mesgar and Strube, 2018). Recent work in coherence modelling has
focused on building models in open-domain (Li and Jurafsky, 2017) and cross-
domain (Xu et al., 2019) settings. More recently Lai and Tetreault (2018)
built coherence models and datasets for real-world texts. Coherence modelling
has also been beneficial for the recent breakthroughs in transformer-based
language modelling (Iter et al., 2020; Lan et al., 2020). Some recent work has
also focused on building benchmarks for applying coherence models in
evaluating text-based natural language generation systems (Mohiuddin et al.,
2020).

Coherence deals with the perception of the discourse rather than the discourse
content itself (Cohen et al., 2001; Wang and Guo, 2014; Li, 2017). While the
perception of a written discourse is only affected by the semantic
organization of its lexical content, the perception of a spoken discourse is
additionally dependent on its prosodic and acoustic features (Hirschberg and
Nakatani, 1996). Previous work in linguistics has highlighted the role of
prosody in defining the structure for spoken discourse. Nakajima and Allen
(1993) performed experiments with cooperative dialogues and demonstrated the
role of prosodic information in defining the topic structure of a given spoken
discourse. Further, Degand and Simon (2009) introduced prosodic segmentation
to define basic discourse units in speech.

Various previous studies have used prosodic attributes to perform coherence-
related tasks with spoken discourse. Nakajima and Allen (1993) analysed the
role of intonation and pause durations in modelling semantic relationships
between discourse utterances at topic boundaries. Further, Tür et al. (2001)
used duration and pitch based features to perform the task of topic
segmentation, which is closely related to both the local and global coherence
of spoken discourse. Stifelman (1995) used pitch patterns to perform emphasis
detection with automated discourse segmentation. This was further used to
summarize and skim through spoken discourses, a task which is highly relevant
to the comprehensibility and the perception of spoken discourse.

Apart from the above mentioned tasks, automated speech scoring is an another
important application of modelling discourse structure and coherence.
Explicitly annotated coherence based measures (Wang et al., 2013, 2017) and
features extracted from discourse structures in text-transcriptions of spoken
discourses (Wang et al., 2019) help in improving the performance of the
automated speech scoring systems significantly. Unlike the work done with
essay scoring as an auxiliary evaluation task for coherence modelling, work in
speech scoring has been limited to include discourse coherence related
features from the text-transcriptions of spoken discourse along with other
features relevant to speech scoring.

##  3 Methodology

###  3.1 Coherence Models for Spoken Discourse

A coherence model for spoken discourse should be able to capture both the
prosodic (pitch, intonation and stress) and the acoustic features (fundamental
frequency, intensity and duration) of an audio sample. It is relatively easier
to procure text datasets of varying levels of structure from sources like
Wikipedia, social networks, and other online medium. Given the lack of such
structured discourse-rich datasets for speech, the model should generalize
beyond closed-domain settings (Li and Jurafsky, 2017; Xu et al., 2019) and
perform well on more open and cross-domain settings with limited training data
(Iter et al., 2020; Mohiuddin et al., 2020). This becomes more important with
spoken discourse as it has been shown that the audio modality is more
vulnerable to change in data domain and background as compared to text (Yan et
al., 2020). Given the above mentioned challenges and pitfalls related to
modelling coherence in spoken discourse, an ideal audio-based coherence model
should:

  1. 1.

learn a discourse coherence signal with a limited number of training samples

  2. 2.

generalize across speech samples which vary in terms of the acoustic and
prosodic features of the speech audio (Ex: accent, gender, rhythm, age, speed
and intonation) and differences in the background of data like recording
quality, sampling frequency, background noise etc.

  3. 3.

and, similar to the text domain, generalize across speech samples which vary
in terms of the spoken discourse’s topic and content.

The local discrimination algorithm proposed by Xu et al. (2019) is designed to
maximize the local coherence scores of adjacent (positive) pairs of sentences
and minimize the local coherence scores for the non-adjacent (negative) pairs
of sentences in a given discourse. Unlike the older discrimination models
which suffer with class-imbalance between the coherent and incoherent
permutations of written discourses, this approach captures local coherence
with an effective negative sampling of the incoherent non-adjacent sentences.
The model takes in a pair of sentence representations as an input, which are
further passed through a multi-layered perceptron with a single hidden layer
(Figure 1) to obtain a local coherence score. Experiments done by Xu et al.
(2019) show that the global aspects of coherence can be approximated by using
the local coherence scores from their models with techniques like score-
averaging across the discourse. Further, the local discrimination model learns
to generalize in open-domain as well as cross-domain settings (as shown by
their sentence-ordering and paragraph-reconstruction experiments with domain-
separated Wikipedia articles), and is agnostic to the modality and background
of the input data. Hence, for all our experiments, we use the local
discrimination model proposed by Xu et al. (2019).

Figure 1: The local discrimination coherence model. Figure 2: Model
architecture across the three input settings: Coh-T, Coh-A and Coh-AT.

To incorporate the audio modality into the coherence model, we encode an audio
based sentence embedding, similar to the sentence embedding obtained from the
text modality. We use a pre-trained audio language model: wav2vec (Schneider
et al., 2019) to encode the audio segment of a sentence into its corresponding
audio embedding as shown in Figure 2. Wav2vec is pre-trained with an
unsupervised objective of next time-step prediction task for audio segments.
This objective function aligns significantly with that of the local
discrimination coherence model, providing rich audio representations for our
task. Similarly, we use pre-trained GloVe embeddings (Pennington et al., 2014)
to encode the text from the sentence into its corresponding text embedding.

In order to inspect the role of the audio modality in modelling coherence for
spoken discourses, we experiment with three different learning settings
(Coh-T, Coh-A and Coh-AT) for the local discrimination coherence model (Figure
2). Similar to all the previous work, with the first setting, we just use the
text modality as the input to the coherence model (Coh-T). In the second
setting, we use only the audio modality as the input to the coherence model,
to establish an audio-only control setting for our experiments (Coh-A).
Finally, in the third setting we obtain a multimodal input representation by
fusing the text and audio modalities together (Coh-AT). In order to get a
minimal trainable aggregated fusion of the two modalities, we pass the audio
and the text embeddings through a bi-linear layer as shown in Figure 2.
Following Xu et al. (2019)’s approach, we aggregate the coherence scores from
both the forward model _(sentence-1, sentence-2)_ and the backward model
_(sentence-2, sentence-1)_.

###  3.2 Datasets

We perform all our experiments with the debate speech samples from the IBM
Debater
dataset111https://www.research.ibm.com/haifa/dept/vst/debating_data.shtml
(Lavee et al., 2019). The dataset consists of recordings of debate speeches
delivered by nine expert debaters, with debate speeches on 200 distinct
Wikipedia topics (such as social media, nuclear weapons, gambling, etc.) as
the debate motions. Each motion topic is contested with two debate recordings
from distinct experts, resulting in a total number of 400 unique speech
samples. This makes the dataset rich with a variety of coherent speeches
spread across a variety of open-domain topics, providing a high quality
training signal to our coherence models. We generate new datasets222The
datasets are available here for various evaluation tasks (explained in Section
3.3) using the responses from the IBM Debater dataset. For this, we use the
text-transcriptions from the debate speech recordings to synthesize artificial
speech responses with a standard text-to-speech (TTS) system based on the
Microsoft Speech API (SAPI5) Microsoft (2009). We use two distinct TTS voices
across all our experiments: S1 (US-male voice) and S2 (US-female voice).

Task | Number of speech responses
---|---
Train | Validation | Test
SCD* | - | - | 398
ASE* | 197 | 78 | 120
TCD* | - | - | 786
SRS | 463 | 181 | 234
Table 1: Number of unique speech responses across the datasets for various
evaluation tasks (*each unique speech response is further sampled with two TTS
settings, resulting in twice as many evaluation samples).

Further, we also use another dataset of speech responses from non-native
English language learners for one of the evaluation tasks. We use the speech
samples delivered by human speakers to train the coherence models (task ASE
and SRS), whereas the speech samples synthesized with the TTS voices are used
for evaluating the models on various coherence-related tasks (task SCD, TCD,
and ASE). Further details about the training and testing of coherence models
on the evaluation tasks are given in Section 3.3. The statistics for all the
datasets are mentioned in Table 1.

###  3.3 Evaluation Tasks

Empirical results from previous work in coherence modelling for text has shown
that the traditional synthetic tasks like sentence ordering do not effectively
capture the models’ ability to perform downstream discourse coherence related
tasks with real-world data (Lai and Tetreault, 2018; Mohiuddin et al., 2020).
An ideal coherence model should perform well (assign appropriate coherence
scores) for spoken discourses delivered by humans as well as machine-generated
speech. Keeping this in mind, we design four tasks for evaluating coherence in
spoken discourse: _Speaker Change Detection_ (SCD), _Artificial Speech
Evaluation_ (ASE), _Discourse Topic Change Detection_ (TCD) and _Speech
Response Scoring_ (SRS). While the first three tasks focus on artificially
generated speech and speech delivered by expert human speakers, the fourth
task focuses on comparing the spoken discourses delivered by non-native
language learners of varying proficiency levels.

#### Speaker Change Detection (SCD):

Modelling coherence in a conversational setting is a very important task with
various downstream applications (Joty et al., 2018; Vakulenko et al., 2018).
To obtain a structured conversational text-transcript from a given speech
audio, we first need to perform speaker diarization. Hence, speaker
diarization is an important aspect of modelling conversational spoken
discourse (Moattar and Homayounpour, 2012). Acoustic cues play an important
role in speaker segmentation of conversational speech (Park and Georgiou,
2018). Since all the samples from the IBM Debater dataset (Lavee et al., 2019)
are monologue debate speeches, we construct a new dataset for this task using
these monologues. We sample a ten-sentences long segment from the middle of
every response. The first five sentences from the sampled segment are
synthesized with a TTS voice (S1) and the next five sentences are synthesized
with another TTS voice (S2). Consequently, the overall synthesized speech
response consists of a speaker change at the end of the fifth sentence, while
maintaining a continuation in the discourse topic. This speaker change in the
response can be detected with an audio-based coherence model, where the event
of speaker change can be depicted with the least inter-sentence local
coherence score. Further in a separate experiment, we reverse the order of
speakers, such that the first five sentences are synthesized with TTS voice S2
and the next five sentences are synthesized with TTS voice S1.

#### Artificial Speech Evaluation (ASE):

Following the work done in evaluating the quality of text-based natural
language generation systems with coherence-based measures (Lapata and
Barzilay, 2005; Park and Kim, 2015; Kiddon et al., 2016), we propose
evaluating the TTS systems with coherence models of spoken discourse. TTS
systems often face issues while naturalizing the synthesized speech, and
continuous efforts are being made to make TTS-speech more human-like and
intelligible across longer contexts (Taylor, 2009). Modelling discourse
relations and coherence can benefit a TTS system in delivering expressive and
intelligible speech (Delmonte and Tripodi, 2015). In this task, we evaluate
and compare a TTS speech sample which lacks a certain amount of prosodic
variation in terms of intonation, stress and speaking rate, to the speech from
a expert human speaker. The underlying hypothesis for this particular task is
that the lack of prosodic variation in the TTS sample makes it relatively
incoherent as compared to the human speech, even though the delivered lexical
content is same across both the samples. Given that we train the coherence
models with human speech only, a direct comparison between the coherence
scores of human and machine generated speech might include some biases
emerging from audio-data background. To handle this, we perform the comparison
using the relative difference between the mean coherence scores of the
positive and negative sentence pairs of a given speech response (explained
further in Section 5).

#### Topic Change Detection (TCD):

Following the work done by Tür et al. (2001) in automatic topic segmentation
with prosodic cues, we propose a coherence-based topic change detection task
for spoken discourse. Since, all the samples from the IBM Debater dataset are
restricted to a single topic (debate motion), we construct a new dataset for
this task using the responses from the IBM Debater dataset (Lavee et al.,
2019). These samples majorly evaluate the extent to which a model captures the
prosodic features at topic boundaries (Nakajima and Allen, 1993). For
constructing a sample, we select a five-sentences long segment from the middle
of every response, so that the sampled segment represents a developed topic
rather than introductory definitions or concluding statements. Subsequently,
we combine it with a similarly sampled segment from a different motion topic.
This results in a new ten-sentences long speech response which covers a
particular topic in its first five sentences and a different topic in the next
five sentences. We use a text-to-speech system to synthesize the speech audio
for this newly generated response. The underlying hypothesis for this
particular task is that the local inter-sentence coherence score should be the
lowest for the fifth and the sixth sentence, depicting a change in discourse
topic for the given speech response.

#### Speech Response Scoring (SRS):

Coherence scores follow a monotonic relationship with the holistic language
proficiency grades. Previous work in text-based coherence modelling has used
essay scoring as an auxiliary evaluation task Miltsakaki and Kukich (2004);
Mesgar and Strube (2018). Conversely, modelling coherence has also proved to
be beneficial in essay-scoring benchmarks (Tay et al., 2017; Li et al., 2018).
In a like manner, coherence-based features extracted from the text-
transcriptions of spoken discourses have proved to be useful in scoring speech
responses from language learners and non-native English speakers (Wang et al.,
2013, 2017). Following this, we test our coherence models on a dataset of
spoken discourses delivered by non-native English language learners from
Philippines (Grover et al., 2020). The dataset comprises of speech responses
recorded in a test environment where the candidates are asked to respond to
one of the $48$ distinct prompts. They are subsequently double scored by
expert annotators using a holistic language proficiency level on a $6$-point
CEFR scale (2001). Since there is no publicly available speech dataset with
explicit coherence-score annotations, these proficiency grades provide us with
an approximate human evaluation of coherence. Using this data, we construct
pairs of speech responses, such that every pair contains speech responses from
two different speakers, for the same prompt (to eliminate prompt-based biases
in the holistic proficiency grading). Given such a pair of speech responses,
we hypothesize that the response graded with higher holistic proficiency
level, should be assigned higher coherence scores by a coherence model.

Spoken discourse from non-native speakers is usually less structured as
compared to a discourse delivered by a native expert speaker (Yan et al.,
2020), making it more challenging to model discourse coherence for non-native
speakers. The text-transcripts of the speech responses in the dataset are not
structured with proper punctuation, which is needed to obtain sentence-level
segments of the speech response. So, we punctuate the text-transcripts from
the dataset with a punctuator model (Tilk and Alumäe, 2016). Given the
background and the pre-processing involved, this dataset is more noisy and
challenging as compared to the IBM Debater dataset. Moreover, while the
discourses from the IBM Debater dataset are more argumentative and informative
in nature, the responses in this dataset are more descriptive and narrative in
nature. Hence, both the datasets vastly differ in terms of discourse modes
(Song et al., 2017; Dhanwal et al., 2020).

##  4 Experimental Setup

#### Audio Processing:

Across all our experiments, we use a speaking-rate of $150$ words per minute
to synthesize speech responses with the TTS systems, a value which is
recognized as the average speaking rate for a native US-English speaking adult
NCVS (2020). Unlike structured texts, there are no explicit cues to perform a
sentence-level segmentation in speech. We use a pre-trained Montreal Forced
Aligner (MFA333https://montreal-forced-aligner.readthedocs.io/) model
(McAuliffe et al., 2017) and the punctuation from the structured text-
transcriptions to get the sentence-level alignments for the speech audio
files. Further, all the speech responses are resampled to a $16$kHz mono-
channel audio file as required by the pre-trained wav2vec model.

#### Coherence Modelling:

Following Xu et al. (2019)’s training protocol, we sample the incoherent pair
of sentences within the same speech response. This avoids the model pitfalls
related to the topic and speech based features with the local discrimination
setting . Further, to make the model generalize well across various domains
and sources of the data, we do not fine-tune the pre-trained audio and text
encoders on the training data. We train the model to optimize the local
coherence scoring based margin loss objective as shown in Equation 1, where
$f\textsuperscript{+}$ and $f\textsuperscript{-}$ are coherence scores for the
adjacent (coherent) and non-adjacent (incoherent) pairs of sentences,
respectively. We use $50\%$ of the topics in the dataset to train our model,
while the rest $20\%$ and $30\%$ of the topics are used for validation and
testing purposes, respectively. The model parameters are optimized with Adam
optimizer (Kingma and Ba, 2015) with a learning rate of $0.001$. We validate
the models with an early stop callback on the validation loss, with a patience
of two epochs. Given, the cross-domain adaptation abilities of the local
discrimination model, we borrow the hyperparameter settings from Xu et al.
(2019) and do not perform any extensive hyperparameter tuning during our
experiments.

$L(f\textsuperscript{+},f\textsuperscript{-})=\max(0,5-f\textsuperscript{+}+f\textsuperscript{-})$
(1)

$k_{change}=\textstyle\min_{k\in[1,N-1]}\\{f\textsuperscript{+}_{k}\\}$ (2)

$coherence\ score=\frac{1}{N-1}\sum_{k=1}^{N-1}f\textsuperscript{+}_{k}$ (3)

While the SCD task and TCD task are evaluated at a local level with inter-
sentence coherence scores (Equation 2), the SRS task is evaluated with
response-level coherence scores as shown in Equation 3, where $N$ is the
number of sentences in the speech response.

##  5 Results and Discussion

#### Task SCD:

For a ten-sentences long response, task SCD results in a nine-way
classification setting. The top-k444here a prediction is considered to be
correct if the ground-truth value lies in the first $k$ predictions given by
the model (k=1,2,3) accuracy scores for the SCD task are shown in Table 2. As
expected, the Coh-T model fails to capture the speaker change boundaries (with
the accuracy scores being almost equal to that of random guessing) due to the
lack of access to the acoustic information from the speech audio. The Coh-A
model shows impressive accuracy for this task, consistent across both the
orders of speaker-change. The model predicts almost all the speaker change
boundaries for k=3. The Coh-AT model does not match up in performance against
the Coh-A model, suggesting a difference in audio-based learning between the
two input settings.

Model | Change | k=1 | k=2 | k=3
---|---|---|---|---
Coh-T | - | 0.0944 | 0.2041 | 0.3138
Coh-A | S1 $\rightarrow$ S2 | 0.9770 | 0.9898 | 0.9949
S2 $\rightarrow$ S1 | 0.9796 | 0.9974 | 1.0000
Coh-AT | S1 $\rightarrow$ S2 | 0.7398 | 0.8954 | 0.9311
S2 $\rightarrow$ S1 | 0.5026 | 0.6913 | 0.7730
Table 2: Top-k accuracy scores for the SCD task.

Negative sampling within the same speech response restricts the model to look
at audio segments from different speakers under the audio-based settings.
Consequently, the model is only exposed to small prosodic and acoustic
variations from the same speaker during training. This shows that the local
discrimination model captures even large acoustic changes in a given
conversational spoken discourse, by modelling local coherence with cues from
small acoustic and prosodic variations in monologue speech.

Speaker | Coh-T | Coh-A | Coh-AT
---|---|---|---
Human Expert | $f\textsuperscript{+}$ | 0.75 | 0.97 | 0.84
$f\textsuperscript{-}$ | -1.14 | -1.76 | -2.52
% diff | -252% | -281.40% | -400%
TTS voice (S1) | $f\textsuperscript{+}$ | 0.75 | 1.76 | 1.08
$f\textsuperscript{-}$ | -1.14 | 1.62 | -0.92
% diff | -252% | -7.90% | -185.20%
TTS voice (S2) | $f\textsuperscript{+}$ | 0.75 | 2.73 | 0.77
$f\textsuperscript{-}$ | -1.14 | 2.55 | -1.39
% diff | -252% | -6.60% | -280.50%

Table 3: Mean coherence scores of positive (adjacent) and negative (non-
adjacent) pairs of sentences from the speech samples in the test set for task
ASE, along with the relative difference (% diff) between them. A higher % diff
value is indicative of better coherence models and more coherent spoken
discourses.

#### Task ASE:

Given the difference in the audio data backgrounds for expert human speakers
and TTS systems, we compare their coherence by monitoring the relative
difference between the mean coherence scores of the coherent (adjacent) and
incoherent (non-adjacent) pairs of sentences sampled from the speech responses
generated by them (Table 3). In accordance with the training objective
function, the incoherent sentences are scored lesser than the coherent
sentences (negative relative difference) across all the speaker and model
settings. A higher relative difference between the coherence scores of
coherent and incoherent pairs of sentences not only indicates the coherence
model’s ability to effectively model coherence (horizontal traversal across
Table 3), but it also indicates the speaker’s ability to produce more coherent
discourses (vertical traversal across Table 3). While the Coh-T model gives a
relative difference of $-252\%$ on the samples from human experts, the audio-
based Coh-A and Coh-AT models give much higher relative differences of
$-281.40\%$ and $-400\%$, respectively. This shows that incorporating the
audio modality highly benefits a coherence model to capture the difference
between coherent and incoherent samples. Further, while the Coh-T model is
independent of any changes in the speech audio (same mean coherence scores of
$0.75$ and $-1.14$ for all the speakers), comparing the relative differences
across the speakers, we observe that the TTS voices S1 and S2 show
significantly lower relative differences across both the audio-based settings
($-7.90\%$ and $-6.60\%$ for Coh-A and, $-185.20\%$ and $-280.50\%$ for Coh-
AT, respectively). Hence, under the ASE task, we find that the speech
synthesized by TTS systems is relatively incoherent and more difficult to
perceive as compared to human-generated speech.

Model | Speaker | k=1 | k=2 | k=3
---|---|---|---|---
Coh-T | - | 0.2513 | 0.3980 | 0.5434
Coh-A | S1 | 0.1148 | 0.2156 | 0.3444
S2 | 0.1135 | 0.2742 | 0.3801
Coh-AT | S1 | 0.2385 | 0.4031 | 0.5383
S2 | 0.2449 | 0.4056 | 0.5663
Table 4: Top-k accuracy scores for the TCD task.

#### Task TCD:

Similar to the SCD task, the TCD task comes up with a nine-way classification
setting. The top-k (k=1,2,3) accuracy scores for the TCD task are shown in
Table 4. The Coh-A model does not perform well on the TCD task individually,
with the accuracy scores being almost equal to that of random guessing. While
the Coh-T model slightly outperforms the Coh-AT model for k=1, the Coh-AT
model shows slight improvements over the Coh-T model for k=2 with both the TTS
voices S1 and S2. Further, for k=3, the Coh-AT model shows significant
improvement over the Coh-T model for TTS voice S2. Even though topic
segmentation is predominantly a text-based task, the slight improvements shown
by Coh-AT model over the text-only settings can be explained by the presence
of cues related to the prosodic patterns observed at topic boundaries
(Nakajima and Allen, 1993).

| Coh-T | Coh-A | Coh-AT
---|---|---|---
Accuracy | 0.4641 | 0.7046 | 0.5569
Table 5: Accuracy scores for the SRS task with non-native speech dataset.

#### Task SRS:

For this task, we monitor the accuracy scores for the binary classification
setting based on the holistic language proficiency grades, using response-
level coherence scores as the proficiency measure (Table 5). While the Coh-A
model performs significantly well with an accuracy score of $0.70$ on the test
set, the Coh-T and Coh-AT models fail to perform well. Given the lack of
structure in non-native speech and the noise in the text-transcriptions of the
speech-responses, text-based settings do not capture the complex holistic
grades efficiently. On the other hand, the audio-based setting seems to be
resistant to this lack of structure and noise in transcriptions and it
effectively captures the holistic language proficiency grades.

The near-perfect performance of the Coh-A model in predicting the speaker
changes suggests that modelling coherence with the audio modality can turn out
to be quite beneficial for a variety of discourse related tasks in
conversational speech such as speech act detection, conversation
disentanglement, etc. To further the efforts made in naturalizing the speech
synthesized with text-to-speech systems, one can come up with better
coherence-based objectives to train the TTS systems. Building up on the topic
change detection task, coherence models for spoken discourses can be also
evaluated on related downstream applications like topic-segmentation in spoken
lectures, podcasts, political spoken discourses, etc. Further, the
significantly higher performance of the Coh-A model on the SRS task shows that
modelling coherence with audio modality can highly compensate for the lack of
structure and errors in text-transcriptions of the speech. This can be quite
useful while modelling coherence with data from non-native speakers, language
learners or while using error-prone text-transcriptions from automatic speech
recognition (ASR) systems.

##  6 Conclusion

In this paper, we performed experiments with four coherence-related tasks for
spoken discourse. In our experiments, we compare the speech synthesized with
text-to-speech systems against the expert human speakers. We also evaluate
coherence in spoken discourses delivered by non-native language learners of
varying language proficiency levels. Our experiments show that incorporating
the audio-modality betters the coherence-modelling for spoken discourses
significantly.

## References

  * Barzilay and Lapata (2008) Regina Barzilay and Mirella Lapata. 2008.  Modeling local coherence: An entity-based approach.  _Computational Linguistics_ , 34(1):1–34.
  * Burstein et al. (2010) Jill Burstein, Joel Tetreault, and Slava Andreyev. 2010.  Using entity-based features to model coherence in student essays.  In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 681–684, Los Angeles, California. Association for Computational Linguistics.
  * Cohen et al. (2001) Henri Cohen, Josée Douaire, and Mayada Elsabbagh. 2001.  The role of prosody in discourse processing.  _Brain and Cognition_ , 46(1):73 – 82.
  * Degand and Simon (2009) Liesbeth Degand and Anne Catherine Simon. 2009.  On identifying basic discourse units in speech: theoretical and empirical issues.  _Discours. Revue de linguistique, psycholinguistique et informatique. A journal of linguistics, psycholinguistics and computational linguistics_ , (4).
  * Delmonte and Tripodi (2015) Rodolfo Delmonte and Rocco Tripodi. 2015.  Semantics and discourse processing for expressive TTS.  In _Proceedings of the First Workshop on Linking Computational Models of Lexical, Sentential and Discourse-level Semantics_ , pages 32–43, Lisbon, Portugal. Association for Computational Linguistics.
  * Dhanwal et al. (2020) Swapnil Dhanwal, Hritwik Dutta, Hitesh Nankani, Nilay Shrivastava, Yaman Kumar, Junyi Jessy Li, Debanjan Mahata, Rakesh Gosangi, Haimin Zhang, Rajiv Ratn Shah, and Amanda Stent. 2020.  An annotated dataset of discourse modes in Hindi stories.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 1191–1196, Marseille, France. European Language Resources Association.
  * of Europe. Council for Cultural Co-operation. Education Committee. Modern Languages Division (2001) Council of Europe. Council for Cultural Co-operation. Education Committee. Modern Languages Division. 2001.  _Common European Framework of Reference for Languages: learning, teaching, assessment_.  Cambridge University Press.
  * Elsner et al. (2007) Micha Elsner, Joseph Austerweil, and Eugene Charniak. 2007.  A unified local and global model for discourse coherence.  In _Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference_ , pages 436–443, Rochester, New York. Association for Computational Linguistics.
  * Elsner and Charniak (2011) Micha Elsner and Eugene Charniak. 2011.  Extending the entity grid with entity-specific features.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 125–129, Portland, Oregon, USA. Association for Computational Linguistics.
  * Errattahi et al. (2018) Rahhal Errattahi, Asmaa El Hannani, and Hassan Ouahmane. 2018.  Automatic speech recognition errors detection and correction: A review.  _Procedia Computer Science_ , 128:32 – 37.  1st International Conference on Natural Language and Speech Processing.
  * Feng and Hirst (2012) Vanessa Wei Feng and Graeme Hirst. 2012.  Extending the entity-based coherence model with multiple ranks.  In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 315–324, Avignon, France. Association for Computational Linguistics.
  * Feng et al. (2014) Vanessa Wei Feng, Ziheng Lin, and Graeme Hirst. 2014.  The impact of deep hierarchical discourse structures in the evaluation of text coherence.  In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_ , pages 940–949, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
  * Grosz et al. (1995) Barbara J. Grosz, Aravind K. Joshi, and Scott Weinstein. 1995.  Centering: A framework for modeling the local coherence of discourse.  _Computational Linguistics_ , 21(2):203–225.
  * Grover et al. (2020) Manraj Singh Grover, Yaman Kumar, Sumit Sarin, Payman Vafaee, Mika Hama, and Rajiv Ratn Shah. 2020.  Multi-modal automated speech scoring using attention fusion.
  * Guinaudeau and Strube (2013) Camille Guinaudeau and Michael Strube. 2013.  Graph-based local coherence modeling.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 93–103, Sofia, Bulgaria. Association for Computational Linguistics.
  * Hirschberg and Nakatani (1996) Julia Hirschberg and Christine H. Nakatani. 1996.  A prosodic analysis of discourse segments in direction-giving monologues.  In _34th Annual Meeting of the Association for Computational Linguistics_ , pages 286–293, Santa Cruz, California, USA. Association for Computational Linguistics.
  * Iter et al. (2020) Dan Iter, Kelvin Guu, Larry Lansing, and Dan Jurafsky. 2020.  Pretraining with contrastive sentence objectives improves discourse performance of language models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4859–4870, Online. Association for Computational Linguistics.
  * Joty et al. (2019) Shafiq Joty, Giuseppe Carenini, Raymond Ng, and Gabriel Murray. 2019.  Discourse analysis and its applications.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts_ , pages 12–17, Florence, Italy. Association for Computational Linguistics.
  * Joty et al. (2018) Shafiq Joty, Muhammad Tasnim Mohiuddin, and Dat Tien Nguyen. 2018.  Coherence modeling of asynchronous conversations: A neural entity grid approach.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 558–568, Melbourne, Australia. Association for Computational Linguistics.
  * Kiddon et al. (2016) Chloé Kiddon, Luke Zettlemoyer, and Yejin Choi. 2016.  Globally coherent text generation with neural checklist models.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 329–339, Austin, Texas. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Lai and Tetreault (2018) Alice Lai and Joel Tetreault. 2018.  Discourse coherence in the wild: A dataset, evaluation and methods.  In _Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 214–223, Melbourne, Australia. Association for Computational Linguistics.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  Albert: A lite bert for self-supervised learning of language representations.  In _International Conference on Learning Representations_.
  * Lapata and Barzilay (2005) Mirella Lapata and Regina Barzilay. 2005.  Automatic evaluation of text coherence: Models and representations.  In _Proceedings of the 19th International Joint Conference on Artificial Intelligence_ , IJCAI’05, page 1085–1090, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
  * Lavee et al. (2019) Tamar Lavee, Matan Orbach, Lili Kotlerman, Yoav Kantor, Shai Gretz, Lena Dankin, Michal Jacovi, Yonatan Bilu, Ranit Aharonov, and Noam Slonim. 2019.  Towards effective rebuttal: Listening comprehension using corpus-wide claim mining.  In _Proceedings of the 6th Workshop on Argument Mining_ , pages 58–66.
  * Li and Hovy (2014) Jiwei Li and Eduard Hovy. 2014.  A model of coherence based on distributed sentence representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2039–2048, Doha, Qatar. Association for Computational Linguistics.
  * Li and Jurafsky (2017) Jiwei Li and Dan Jurafsky. 2017.  Neural net models of open-domain discourse coherence.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 198–209, Copenhagen, Denmark. Association for Computational Linguistics.
  * Li (2017) Junyi Li. 2017.  From discourse structure to text specificity: Studies of coherence preferences.
  * Li et al. (2018) Xia Li, Minping Chen, Jianyun Nie, Zhenxing Liu, Ziheng Feng, and Yingdan Cai. 2018\.  Coherence-based automated essay scoring using self-attention.  In _Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data_ , pages 386–397. Springer.
  * Lin et al. (2011) Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2011.  Automatically evaluating text coherence using discourse relations.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 997–1006, Portland, Oregon, USA. Association for Computational Linguistics.
  * Logeswaran et al. (2018) Lajanugen Logeswaran, Honglak Lee, and Dragomir R Radev. 2018.  Sentence ordering and coherence modeling using recurrent neural networks.  In _AAAI_.
  * Louis and Nenkova (2012) Annie Louis and Ani Nenkova. 2012.  A coherence model based on syntactic patterns.  In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ , pages 1157–1168, Jeju Island, Korea. Association for Computational Linguistics.
  * McAuliffe et al. (2017) Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017.  Montreal forced aligner: Trainable text-speech alignment using kaldi.
  * Mesgar and Strube (2015) Mohsen Mesgar and Michael Strube. 2015.  Graph-based coherence modeling for assessing readability.  In _Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics_ , pages 309–318, Denver, Colorado. Association for Computational Linguistics.
  * Mesgar and Strube (2018) Mohsen Mesgar and Michael Strube. 2018.  A neural local coherence model for text quality assessment.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4328–4339, Brussels, Belgium. Association for Computational Linguistics.
  * Microsoft (2009) Microsoft. 2009.  Microsoft Speech API (SAPI) 5.4.
  * Miltsakaki and Kukich (2004) E. Miltsakaki and K. Kukich. 2004.  Evaluation of text coherence for electronic essay scoring systems.  _Nat. Lang. Eng._ , 10(1):25–55.
  * Moattar and Homayounpour (2012) Mohammad Hossein Moattar and Mohammad M Homayounpour. 2012.  A review on speaker diarization systems and approaches.  _Speech Communication_ , 54(10):1065–1103.
  * Mohiuddin et al. (2020) Tasnim Mohiuddin, Prathyusha Jwalapuram, Xiang Lin, and Shafiq Joty. 2020.  Coheval: Benchmarking coherence models.
  * Moon et al. (2019) Han Cheol Moon, Tasnim Mohiuddin, Shafiq Joty, and Chi Xu. 2019.  A unified neural coherence model.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2262–2272, Hong Kong, China. Association for Computational Linguistics.
  * Nakajima and Allen (1993) Shin Nakajima and James F. Allen. 1993.  A study on prosody and discourse structure in cooperative dialogues.  Technical report, USA.
  * NCVS (2020) NCVS. 2020.  National center for voice and speech (ncvs).
  * Nenkova and McKeown (2011) Ani Nenkova and Kathleen McKeown. 2011.  _Automatic summarization_.  Now Publishers Inc.
  * Park and Kim (2015) Cesc C Park and Gunhee Kim. 2015.  Expressing an image stream with a sequence of natural sentences.  In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, _Advances in Neural Information Processing Systems 28_ , pages 73–81. Curran Associates, Inc.
  * Park and Georgiou (2018) Tae Jin Park and Panayiotis Georgiou. 2018.  Multimodal speaker segmentation and diarization using lexical and acoustic cues via sequence to sequence neural networks.  _Interspeech 2018_.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global vectors for word representation.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543.
  * Pitler and Nenkova (2008) Emily Pitler and Ani Nenkova. 2008.  Revisiting readability: A unified framework for predicting text quality.  In _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing_ , pages 186–195, Honolulu, Hawaii. Association for Computational Linguistics.
  * Prasad et al. (2008) Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008.  The Penn discourse TreeBank 2.0. In _LREC 2008_.
  * Schneider et al. (2019) Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019.  wav2vec: Unsupervised pre-training for speech recognition.  _Interspeech 2019_.
  * Smith et al. (2016) Karin Sim Smith, Wilker Aziz, and Lucia Specia. 2016.  The trouble with machine translation coherence.  In _Proceedings of the 19th Annual Conference of the European Association for Machine Translation_ , pages 178–189.
  * Song et al. (2017) Wei Song, Dong Wang, Ruiji Fu, Lizhen Liu, Ting Liu, and Guoping Hu. 2017.  Discourse mode identification in essays.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 112–122, Vancouver, Canada. Association for Computational Linguistics.
  * Stifelman (1995) Lisa J Stifelman. 1995.  A discourse analysis approach to structured speech.
  * Tappe and Schilder (1998) Heike Tappe and Frank Schilder. 1998.  Coherence in spoken discourse.  In _COLING 1998 Volume 2: The 17th International Conference on Computational Linguistics_.
  * Tay et al. (2017) Yi Tay, Minh C. Phan, Luu Anh Tuan, and Siu Cheung Hui. 2017.  Skipflow: Incorporating neural coherence features for end-to-end automatic text scoring.
  * Taylor (2009) Paul Taylor. 2009.  _Text-to-speech synthesis_.  Cambridge university press.
  * Thompson and Mann (1987) Sandra A Thompson and William C Mann. 1987.  Rhetorical structure theory: A framework for the analysis of texts.  _IPRA Papers in Pragmatics_ , 1(1):79–105.
  * Tien Nguyen and Joty (2017) Dat Tien Nguyen and Shafiq Joty. 2017.  A neural local coherence model.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1320–1330, Vancouver, Canada. Association for Computational Linguistics.
  * Tilk and Alumäe (2016) Ottokar Tilk and Tanel Alumäe. 2016.  Bidirectional recurrent neural network with attention mechanism for punctuation restoration.  In _Interspeech 2016_.
  * Tür et al. (2001) Gökhan Tür, Dilek Hakkani-Tür, Andreas Stolcke, and Elizabeth Shriberg. 2001\.  Integrating prosodic and lexical cues for automatic topic segmentation.  _Computational Linguistics_ , 27(1):31–57.
  * Vakulenko et al. (2018) Svitlana Vakulenko, Maarten de Rijke, Michael Cochez, Vadim Savenkov, and Axel Polleres. 2018.  Measuring semantic coherence of a conversation.  In _The Semantic Web – ISWC 2018_ , pages 634–651, Cham. Springer International Publishing.
  * Wang et al. (2013) Xinhao Wang, Keelan Evanini, and Klaus Zechner. 2013.  Coherence modeling for the automated assessment of spontaneous spoken responses.  In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 814–819, Atlanta, Georgia. Association for Computational Linguistics.
  * Wang et al. (2017) Xinhao Wang, Keelan Evanini, Klaus Zechner, and Matthew Mulholland. 2017.  Modeling discourse coherence for the automated scoring of spontaneous spoken responses.  In _Proc. 7th ISCA Workshop on Speech and Language Technology in Education_ , pages 132–137.
  * Wang et al. (2019) Xinhao Wang, Binod Gyawali, James V. Bruno, Hillary R. Molloy, Keelan Evanini, and Klaus Zechner. 2019.  Using rhetorical structure theory to assess discourse coherence for non-native spontaneous speech.  In _Proceedings of the Workshop on Discourse Relation Parsing and Treebanking 2019_ , pages 153–162, Minneapolis, MN. Association for Computational Linguistics.
  * Wang and Guo (2014) Yuan Wang and Minghe Guo. 2014.  A short analysis of discourse coherence.  _Journal of Language Teaching and Research_ , 5(2):460.
  * Webber (2004) Bonnie Webber. 2004.  D-ltag: extending lexicalized tag to discourse.  _Cognitive Science_ , 28(5):751 – 779.  2003 Rumelhart Prize Special Issue Honoring Aravind K. Joshi.
  * Xu et al. (2019) Peng Xu, Hamidreza Saghir, Jin Sung Kang, Teng Long, Avishek Joey Bose, Yanshuai Cao, and Jackie Chi Kit Cheung. 2019.  A cross-domain transferable neural coherence model.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 678–687, Florence, Italy. Association for Computational Linguistics.
  * Yan et al. (2020) Duanli Yan, André A Rupp, and Peter W Foltz. 2020.  _Handbook of automated scoring: Theory into practice_.  CRC Press.