##

Tianhao Zhang
University of Minnesota, Twin Cities
<EMAIL_ADDRESS>
Zhexiao Lin11footnotemark: 1
University of California, Berkeley
<EMAIL_ADDRESS>
Zhecheng Sheng11footnotemark: 1
University of Minnesota, Twin Cities
<EMAIL_ADDRESS>
Chen Jiang11footnotemark: 1
University of Minnesota, Twin Cities
<EMAIL_ADDRESS>
Dongyeop Kang
University of Minnesota, Twin Cities
<EMAIL_ADDRESS>
Equal contribution.

## On the Sequence Evaluation based on Stochastic Processes

Tianhao Zhang
University of Minnesota, Twin Cities
<EMAIL_ADDRESS>
Zhexiao Lin11footnotemark: 1
University of California, Berkeley
<EMAIL_ADDRESS>
Zhecheng Sheng11footnotemark: 1
University of Minnesota, Twin Cities
<EMAIL_ADDRESS>
Chen Jiang11footnotemark: 1
University of Minnesota, Twin Cities
<EMAIL_ADDRESS>
Dongyeop Kang
University of Minnesota, Twin Cities
<EMAIL_ADDRESS>
Equal contribution.

###### Abstract

Modeling and analyzing long sequences of text is an essential task for Natural
Language Processing. Success in capturing long text dynamics using neural
language models will facilitate many downstream tasks such as coherence
evaluation, text generation, machine translation and so on. This paper
presents a novel approach to model sequences through a stochastic process. We
introduce a likelihood-based training objective for the text encoder and
design a more thorough measurement (score) for long text evaluation compared
to the previous approach. The proposed training objective effectively
preserves the sequence coherence, while the new score comprehensively captures
both temporal and spatial dependencies. Theoretical properties of our new
score show its advantages in sequence evaluation. Experimental results show
superior performance in various sequence evaluation tasks, including global
and local discrimination within and between documents of different lengths. We
also demonstrate the encoder achieves competitive results on discriminating
human and AI written text.

###  1 Introduction

Nowadays, generative models are becoming increasingly popular in Natural
Language Processing (NLP) [Zou et al., 2023, Yang et al., 2023, Yi et al.,
2024], and integrating stochastic processes with these models has gained
traction due to their proven effectiveness. However, we find that stochastic
representations of the latent space can encode not only the target domain
features but also a common dynamic structure across most articles within the
domain. Inspired by this, we observe that stochastic representations can
exhibit superior performance in multiple downstream tasks, such as evaluating
long text coherence. Furthermore, while similar methods may heavily rely on
the training domain Lai and Tetreault [2018], Jeon and Strube [2022], our
approach shows significant potential in out-of-domain tasks, likely benefiting
from the stochastic representation’s robust generative properties. In
practice, this capability can be realized by fine-tuning the evaluation
function on a small dataset without changing any parameters in the encoder.

Learning a stochastic representation typically needs to balance between
achieving higher generative ability and fitting the target stochastic
processes better. In application, we can either use the likelihood, or a
likelihood-free method, the contrastive learning van den Oord et al. [2018],
Durkan et al. [2020]. In this work, we utilize the contrastive learning method
from Wang et al. [2022] and design new likelihood-based methods. Comparing
between these two methods. We found that the contrastive learning may be
effective for evaluating coherence but may not adequately learn the target
stochastic process representation, which can limit its performance on other
tasks, e.g. out-of-domain tasks.

To assess the coherence of long texts, we utilize the idea from linguistic
theory and stochastic processes. Notice that there are always some common
properties among coherent articles, for example, (1) a main theme should be
maintained throughout the text; (2) sentences near the beginning and end of
the article should emphasize the main idea; and (3) sentences in the middle
can deviate slightly from the main idea but should remain controlled by the
theme. This provides an ideal scenario for the Brownian bridge (abbr. BB),
which is a special stochastic process with a “fixed” starting and ending
position Øksendal and Øksendal [2003], and has shown superior performance in
many areas: e.g. text generation Wang et al. [2022], human mobility analysis
Krumm [2021], and animal migration studies Horne et al. [2007]. Therefore,
Sheng et al. [2024] proposed a heuristic BB-based method, which showed better
performance on multiple tasks. However, it was limited to simple coherence
evaluation tasks and lacked a good interpretation of the latent space
dimensions.

In this paper, unlike the settings in Wang et al. [2022] and Sheng et al.
[2024], we fully utilize the temporal and spatial dependencies of the
stochastic process by incorporating $\Sigma$ and $\Sigma_{T}$, as described in
Section 3. We derive the log-likelihood of the unknown parameter $\Sigma$
based on the observed sequences. As a direct result, we design a BB encoder
that minimizes the negative log-likelihood. This encoder maps the sequence to
a general BB while preserving important information from the sequence, which
in application shows a better performance on out-of-domain test. By leveraging
the stochastic process structure of the sequence, we propose a new BBScore to
evaluate the sequence. This new BBScore is supported by theory and has
demonstrated strong performance across various downstream tasks and it improve
performance by 20% on global coherence discrimination tasks and 10% on local
coherence discrimination tasks. Additionally, it overcomes previous
limitations and enabling comparisons among different articles. Furthermore,
benefit from the fact that $\widehat{\Sigma}$ can encode domain-specific
property, we designed a training-free method to implement human-AI
discrimination task, and achieving 70% accuracy.

###  2 Related work

Stochastic processes have demonstrated their robust capability in modeling
complex tasks across various fields, including biology Horne et al. [2007],
and finance Øksendal and Øksendal [2003]. Recently, when integrated with
machine learning and deep learning techniques, methods inspired by stochastic
processes have proven exceptionally effective in handling noisy, high-
dimensional real-world data. Notably, they excel in generation tasks, such as
image generation Lim et al. [2023], video generation Zhang et al. [2023], and
long text generation Wang et al. [2022]. A critical aspect of these tasks is
learning the stochastic representation from a given dataset. Traditionally, if
the target stochastic processes are well-defined, one can directly construct a
likelihood function and train the system. Recently, a likelihood-free training
paradigm can be employed Durkan et al. [2020] —specifically, the contrastive
learning method —which has shown significant power and potential in handling
high-dimensional data van den Oord et al. [2018]. This approach provides a way
to learn about the predictive density indirectly, rather than through direct
reconstruction Mathieu et al. [2021]. Moreover, it has been demonstrated that
a properly constructed contrastive learning task can estimate the transition
kernel in some diffusion modeling problems Liu et al. [2021], offering further
theoretical support for its impressive empirical performance.

Coherence, as described by Reinhart [1980], defines the discourse structure of
a text, where high-quality text should present a logical flow with well-
connected ideas and topics. Previous studies have reported that neural
language models, such as transformers, often struggle to effectively capture
coherence structures [Deng et al., 2022]. To better guide language models in
learning the dynamics of long text, methods that construct latent spaces have
been developed [Bowman et al., 2016, Gao et al., 2021]. These methods aim to
model sentence embeddings by leveraging information from neighboring
utterances. However, most of these approaches yield static representations and
overlook the dynamics of the entire text. In a recent study, Wang et al.
[2022] proposed a method utilizing the BB as an objective to model long-range
text dependencies, achieving notable performance in generating long coherent
texts from the learned latent plans. Despite advancements in coherence
modeling, automatically assessing coherence for any given text remains a
challenge.Maimon and Tsarfaty [2023] Sheng et al. [2024] designed a
likelihood-based score metric for coherence assessment, built upon the
unsupervised learning approach from Wang et al. [2022]. This score
demonstrated strong performance on artificial shuffle tasks and downstream
tasks such as Human-AI discrimination. Shuffle tasks have long been used as a
standard approach for coherence evaluation [Barzilay and Lapata, 2005]. In
shuffle tasks, the model is asked to recognize a shuffled document from its
original copy. Shuffling documents with different sentence block sizes
provides insights into a model’s ability to discern global coherence. A more
challenging variant of the shuffle task is the local window shuffle [Moon et
al., 2019], where only a small set of consecutive sentences is shuffled within
the original document, and the model is asked to detect these shuffled
articles from the unshuffled ones.

###  3 Method

####  3.1 Setup

In this section, we introduce the setup of our method. We model the sequences
using BBs, which are among the most commonly used stochastic processes. First,
we introduce the standard BB $\\{B(t):t\in[0,T]\\}$ with $B(0)=0$ and
$B(T)=0$. For any $t\in[0,T]$, the process $B(t)$ follows a normal
distribution $B(t)\sim N(0,\frac{t(T-t)}{T})$. Additionally, for $s,t\in[0,T]$
with $s<t$, the covariance structure is given by ${\rm
Cov}(B(s),B(t))=\frac{s(T-t)}{T}$. The general BB can then be constructed
using $a+\frac{t}{T}(b-a)+\sigma B(t)$, where $a$ and $b$ are arbitrary fixed
start and end points, respectively, and $\sigma$ is the standard deviation of
the process.

For an input sequence $\bar{{\mathbf{s}}}=(s_{0},\ldots,s_{T})$ where
$s_{t}\in\mathbb{R}^{d}$ for $t\in\llbracket T\rrbracket$, we use standard BBs
to capture temporal dependence. To account for spatial dependence, we consider
$d$ independent standard BBs $B_{1}(t),\ldots,B_{d}(t)$ over the interval
$[0,T]$. For any $t\in\llbracket T\rrbracket$, the sequence is modeled as
$s_{t}=\mu_{t}+{\mathbf{W}}(B_{1}(t),\ldots,B_{d}(t))^{\top}$, where
${\mathbf{W}}\in{\mathbb{R}}^{d\times d}$ is the transformation matrix and
$\mu_{t}=s_{0}+\frac{t}{T}(s_{T}-s_{0})$ represents the mean at time $t$. Let
${\mathbf{s}}=(s_{1},\ldots,s_{T-1})$ be the sequence excluding the start and
end points, and $\boldsymbol{\mu}=(\mu_{1},\ldots,\mu_{T-1})$ be the
corresponding means.

The encoder and the BBScore are based on the likelihood function of the input
sequence $\bar{{\mathbf{s}}}$. The following theorem presents the likelihood
function.

######  Theorem 1.

Let $\Sigma_{T}\in{\mathbb{R}}^{(T-1)\times(T-1)}$ be the covariance matrix
with $[\Sigma_{T}]_{s,t}=\frac{s(T-t)}{T}$. Let
$\Sigma={\mathbf{W}}{\mathbf{W}}^{\top}$. Then the log-likelihood function is
given by

$\small\ell(\Sigma|\bar{{\mathbf{s}}})=-\frac{d(T-1)}{2}\log(2\pi)-\frac{d}{2}\log(\lvert\Sigma_{T}\rvert)-\frac{(T-1)}{2}\log(\lvert\Sigma\rvert)-\frac{1}{2}\mathrm{tr}(\Sigma^{-1}({\mathbf{s}}-\boldsymbol{\mu})\Sigma_{T}^{-1}({\mathbf{s}}-\boldsymbol{\mu})^{\top}).$

Generally, for $n$ independent input sequence
$\bar{\mathbf{s}}_{1},\ldots,\bar{\mathbf{s}}_{n}$ with lengths
$T_{1}+1,\ldots,T_{n}+1$, generated by the same ${\mathbf{W}}$ (or $\Sigma$),
the log-likelihood function is

$\displaystyle\small\ell(\Sigma|\\{\bar{{\mathbf{s}}}_{i}\\}_{i=1}^{n})=$
$\displaystyle\frac{d\sum_{i=1}^{n}(T_{i}-1)}{2}\log(2\pi)-\frac{d}{2}\sum_{i=1}^{n}\log(\lvert\Sigma_{T_{i}}\rvert)$
$\displaystyle-\frac{\sum_{i=1}^{n}(T_{i}-1)}{2}\log(\lvert\Sigma\rvert)-\frac{1}{2}\sum_{i=1}^{n}\mathrm{tr}(\Sigma^{-1}({\mathbf{s}}_{i}-\boldsymbol{\mu}_{i})\Sigma_{T_{i}}^{-1}({\mathbf{s}}_{i}-\boldsymbol{\mu}_{i})^{\top}).$

By Theorem 1, we obtain the log-likelihood function of the sequence, enabling
us to derive the maximum likelihood estimation (MLE) of $\Sigma$.

######  Theorem 2.

Under the setting of Theorem 1, the MLE for $\ell(\Sigma|\bar{{\mathbf{s}}})$
is

$\small\widehat{\Sigma}=(T-1)^{-1}({\mathbf{s}}-\boldsymbol{\mu})\Sigma_{T}^{-1}({\mathbf{s}}-\boldsymbol{\mu})^{\top},$

and the MLE for $\ell(\Sigma|\\{\bar{{\mathbf{s}}}_{i}\\}_{i=1}^{n})$ is

$\small\widehat{\Sigma}=\Big{(}\sum_{i=1}^{n}(T_{i}-1)\Big{)}^{-1}\Big{(}\sum_{i=1}^{n}({\mathbf{s}}_{i}-\boldsymbol{\mu}_{i})\Sigma_{T_{i}}^{-1}({\mathbf{s}}_{i}-\boldsymbol{\mu}_{i})^{\top}\Big{)}.$

The training of the encoder and the definition of BBScore are based on the MLE
of $\Sigma$.

####  3.2 Encoders

The encoder architecture consists of a frozen, pretrained GPT-2 model from
Huggingface and a trainable MLP network. The GPT-2 model’s last layer hidden
state corresponding to the end-of-sentence (EOS) token is extracted, and a
4-layer MLP is trained on top of this hidden state. The purpose of the encoder
is to learn a nonlinear mapping from the raw input space to the latent space,
denoted as $f_{\theta}:{\mathcal{X}}\to{\mathcal{S}}$. In practice, we use two
loss functions to train the encoder: the contrastive learning loss $L_{\rm
CL}$ and the negative log-likelihood loss $L_{\rm NLL}$. The $L_{\rm CL}$ loss
is primarily designed for the prediction task van den Oord et al. [2018],
promoting the encoder to distinguish between positive and negative samples
from the data. On the other hand, the $L_{\rm NLL}$ loss is designed directly
to enforce the output to follow a certain distribution given by the
likelihood, ensuring that the latent space representations are well-structured
and meaningful. While Wang et al. [2022] utilized contrastive learning for the
encoder, the negative log-likelihood loss exhibits certain advantages over
contrastive learning, both theoretically and empirically.

Figure 1: Training framework.

#####  3.2.1 Contrastive Learning Encoder (CL Encoder)

In the context of Wang et al. [2022], a structure assumption $\Sigma=I_{d}$
was imposed, where $I_{d}$ is the $d$-dimensional identity matrix. For an
arbitrary start point $s_{0}$ at time $t=0$ and end point $s_{T}$ at time
$t=T$, the marginal distribution of $s_{t}$ at time $t$ is given by $s_{t}\mid
s_{0},s_{T}\sim
N((1-\frac{t}{T})s_{0}+\frac{t}{T}s_{T},\frac{t(T-t)}{T}I_{d})$. Consider any
triplet observations $(x_{1},x_{2},x_{3})$ with
$x_{1},x_{2},x_{3}\in{\mathcal{X}}$. The goal was to ensure that
$f_{\theta}(x_{2})$ follows the above marginal distribution with the starting
point $f_{\theta}(x_{1})$ and the end point $f_{\theta}(x_{3})$. For a
sequence of observations $(x_{0},\ldots,x_{T})$, let
$B=\\{(x_{0},x_{t},x_{T})\\}$ be a set of batches consisting of randomly
sampled positive triplets $(x_{0},x_{t},x_{T})$ with $0<t<T$. Then, the
contrastive learning loss function is defined as:

$\displaystyle\footnotesize L_{\rm CL}$
$\displaystyle=E\left[-\log\frac{\exp(d(x_{0},x_{t},x_{T};f_{\theta}))}{\sum_{(x_{0},x_{t^{\prime}},x_{T})\in
B}\exp(d(x_{0},x_{t^{\prime}},x_{T};f_{\theta}))}\right]$
$\displaystyle\text{with}\quad d(x_{0},x_{t},x_{T};f_{\theta})$
$\displaystyle=-\frac{1}{2\sigma^{2}}\lVert
f_{\theta}(x_{t})-(1-\frac{t}{T})f_{\theta}(x_{0})-\frac{t}{T}f_{\theta}(x_{T})\rVert_{2}^{2},~{}~{}\sigma^{2}=t(T-t)/T.$

There are two theoretical drawbacks to this contrastive learning approach.
First, the assumption $\Sigma=I_{d}$ implies independence and homogeneity for
each dimension of the output from the encoder. Second, the use of the marginal
distribution ignores the covariance structure of the stochastic process.
Empirically, it has been observed that only a few dimensions of the encoder’s
output are effective, which is not desirable. To address these issues and
incorporate both spatial and temporal dependence, we propose our new Brownian-
Bridge Encoder, which minimizes the negative log-likelihood loss.

#####  3.2.2 Brownian-Bridge Encoder (BB Encoder)

Consider the multi-domain problem where we have $m$ domains
${\mathcal{D}}_{1},{\mathcal{D}}_{2},\ldots,{\mathcal{D}}_{m}$. For each
domain ${\mathcal{D}}_{j}$, we consider $n_{j}$ independent raw inputs
${\mathbf{x}}_{j1},\ldots,{\mathbf{x}}_{jn_{j}}$. Let
$\bar{\mathbf{s}}_{ji}^{\theta}=f_{\theta}({\mathbf{x}}_{ji})$ for
$j\in\llbracket m\rrbracket$ and $i\in\llbracket n_{j}\rrbracket$. When
$\theta$ reaches its optimal value $\hat{\theta}$, the sequences
$[\bar{{\mathbf{s}}}_{ji}^{\theta}]_{i=1}^{n_{j}}$ should be i.i.d. from the
BBs and share the same true parameters $\Sigma_{j}$ for each domain
${\mathcal{D}}_{j}$.

We use negative log-likelihood as the loss function. Note that for each
$\theta$, the negative log-likelihood in Theorem 1 for domain
${\mathcal{D}}_{j}$ depends on $\Sigma_{j}$ and
$[{\mathbf{x}}_{ji}]_{i=1}^{n_{j}}$ by

$\displaystyle\small\sum_{i=1}^{n_{j}}(T_{i}-1)\log(\lvert\Sigma_{j}\rvert)+\sum_{i=1}^{n_{j}}\mathrm{tr}(\Sigma_{j}^{-1}({\mathbf{s}}_{i}^{\theta}-\boldsymbol{\mu}_{i}^{\theta})\Sigma_{T_{i}}^{-1}({\mathbf{s}}_{i}^{\theta}-\boldsymbol{\mu}_{i}^{\theta})^{\top}).$

We consider the following training process. We divide the
$[{\mathbf{x}}_{ji}]_{i=1}^{n_{j}}$ into several batches. For each batch
${\mathcal{B}}$, we consider the loss within the batch as

$\displaystyle\small\sum_{i\in{\mathcal{B}}}\mathrm{tr}(\widehat{\Sigma}_{j}^{-1}({\mathbf{s}}_{i}^{\theta}-\boldsymbol{\mu}_{i}^{\theta})\Sigma_{T_{i}}^{-1}({\mathbf{s}}_{i}^{\theta}-\boldsymbol{\mu}_{i}^{\theta})^{\top}),$

where $\widehat{\Sigma}_{j}$ is the current estimation for $\Sigma_{j}$.

In cases where $T_{i}$ might be large, we consider randomly sampling a triplet
from ${\mathbf{x}}_{i}$. For a random sample $t=(t_{1},t_{2},t_{3})$ with
$1\leq t_{1}<t_{2}<t_{3}\leq T_{i}-1$, we let
$[{\mathbf{s}}_{i}^{\theta}]_{t}$ and $[\boldsymbol{\mu}_{i}^{\theta}]_{t}$ be
the $d\times 3$ sub-matrices corresponding to $t$ from
${\mathbf{s}}_{i}^{\theta}$ and $\boldsymbol{\mu}_{i}^{\theta}$, respectively.
Let $[\Sigma_{T_{i}}]t$ be the $3\times 3$ sub-matrix corresponding to $t$
from $\Sigma_{T_{i}}$. The corresponding loss for each $i$ will be

$\displaystyle\small\mathrm{tr}(\widehat{\Sigma}_{j}^{-1}([{\mathbf{s}}_{i}^{\theta}]_{t}-[\boldsymbol{\mu}_{i}^{\theta}]_{t})[\Sigma_{T_{i}}]_{t}^{-1}([{\mathbf{s}}_{i}^{\theta}]_{t}-[\boldsymbol{\mu}_{i}^{\theta}]_{t})^{\top}),$

and the within-batch loss will be the sum over all $i\in{\mathcal{B}}$.

After iterating over all the batches, we update the estimate for $\Sigma_{j}$
as

$\displaystyle\small\widehat{\Sigma}_{j}=\Big{(}\sum_{i=1}^{n_{j}}(T_{i}-1)\Big{)}^{-1}\Big{(}\sum_{i=1}^{n_{j}}({\mathbf{s}}_{i}^{\theta}-\boldsymbol{\mu}_{i}^{\theta})\Sigma_{T_{i}}^{-1}({\mathbf{s}}_{i}^{\theta}-\boldsymbol{\mu}_{i}^{\theta})^{\top}\Big{)},$

and then move on to the next domain. To stabilize the training process, we
shift $\widehat{\Sigma}_{j}$ towards the identity matrix. We compute
$\widehat{\sigma}^{2}_{j}$, and update $\widehat{\Sigma}_{j}$ as following,
given some small $\epsilon>0$,

$\displaystyle\footnotesize\widehat{\sigma}^{2}_{j}$
$\displaystyle=\Big{(}\sum_{i=1}^{n_{j}}(T_{i}-1)d\Big{)}^{-1}\Big{(}\sum_{i=1}^{n_{j}}\mathrm{tr}(({\mathbf{s}}_{i}^{\theta}-\boldsymbol{\mu}_{i}^{\theta})\Sigma_{T_{i}}^{-1}({\mathbf{s}}_{i}^{\theta}-\boldsymbol{\mu}_{i}^{\theta})^{\top})\Big{)},$
$\displaystyle\widehat{\Sigma}_{j}$
$\displaystyle=(1-\epsilon)\Big{(}\sum_{i=1}^{n_{j}}(T_{i}-1)\Big{)}^{-1}\Big{(}\sum_{i=1}^{n_{j}}({\mathbf{s}}_{i}^{\theta}-\boldsymbol{\mu}_{i}^{\theta})\Sigma_{T_{i}}^{-1}({\mathbf{s}}_{i}^{\theta}-\boldsymbol{\mu}_{i}^{\theta})^{\top}\Big{)}+\epsilon\widehat{\sigma}^{2}_{j}{\mathbf{I}}_{d}.$

After iterating over all the domains, the total empirical loss function is

$\displaystyle\footnotesize L_{\rm
NLL}=\sum_{j=1}^{m}\sum_{i=1}^{n_{j}}(T_{i}-1)\log(\lvert\Sigma_{j}\rvert)+\sum_{j=1}^{m}\sum_{i=1}^{n_{j}}\mathrm{tr}(\Sigma_{j}^{-1}({\mathbf{s}}_{i}^{\theta}-\boldsymbol{\mu}_{i}^{\theta})\Sigma_{T_{i}}^{-1}({\mathbf{s}}_{i}^{\theta}-\boldsymbol{\mu}_{i}^{\theta})^{\top}).$

####  3.3 BBScore

From Sheng et al. [2024], the BBScore therein imposed a structural assumption
$\Sigma=\sigma^{2}I_{d}$, where $I_{d}$ is the $d$-dimensional identity
matrix. This assumption implies that different coordinates are independent and
share the same variance. Additionally, Sheng et al. [2024] assumed the
observations $s_{t}$ are independent for different $t$. The final BBScore was
defined as the log-likelihood with a maximum likelihood estimate (MLE) for
$\sigma^{2}$.

We define a new BBScore with improvements in two key aspects. First, we
incorporate temporal dependence through $\Sigma_{T}$. Second, we incorporate
spatial dependence through $\Sigma$.

Consider the sequence $\bar{{\mathbf{s}}}=(s_{0},\ldots,s_{T})$, with
${\mathbf{s}}$ and $\boldsymbol{\mu}$ defined as before. We assume
$\bar{{\mathbf{s}}}$ follows a BB with an unknown parameter $\Sigma$. Let
$\widehat{\Sigma}$ be the estimate of $\Sigma$ from Theorem 2. A natural
generalization of Sheng et al. [2024] is to construct the BBScore based on the
log density in Theorem 1:

$\displaystyle\small\log
p(\bar{{\mathbf{s}}}|\widehat{\Sigma})=-\frac{d(T-1)}{2}\log(2\pi)-\frac{d}{2}\log(\lvert\Sigma_{T}\rvert)-\frac{(T-1)}{2}\log(\lvert\widehat{\Sigma}\rvert)-\frac{1}{2}\mathrm{tr}(\widehat{\Sigma}^{-1}({\mathbf{s}}-\boldsymbol{\mu})\Sigma_{T}^{-1}({\mathbf{s}}-\boldsymbol{\mu})^{\top}).$

However, the log density $\log p(\bar{{\mathbf{s}}}|\widehat{\Sigma})$ is not
scale-invariant with respect to $T$. This means that for two sequences of
different lengths generated by the same $\Sigma$, the log densities are not
comparable. Additionally, the log density can range over the entire real
space, while a positive score is desirable. To address these issues, we define
the new BBScore, motivated by hypothesis testing in statistics.

Note that $\mathrm{vec}({\mathbf{s}}-\boldsymbol{\mu})\sim
N(0,\Sigma_{T}\otimes\Sigma)$, where $\mathrm{vec}(\cdot)$ is the
vectorization operator and $\otimes$ is the Kronecker product. Then we have

$\displaystyle\small[\Sigma_{T}\otimes\Sigma]^{-1/2}$
$\displaystyle\mathrm{vec}({\mathbf{s}}-\boldsymbol{\mu})\sim
N(0,{\mathbf{I}}_{(T-1)\times d}),$
$\displaystyle\lVert[\Sigma_{T}\otimes\Sigma]^{-1/2}\mathrm{vec}({\mathbf{s}}-\boldsymbol{\mu})\rVert^{2}$
$\displaystyle=\mathrm{vec}({\mathbf{s}}-\boldsymbol{\mu})^{\top}[\Sigma_{T}\otimes\Sigma]^{-1}\mathrm{vec}({\mathbf{s}}-\boldsymbol{\mu})\sim\chi_{(T-1)\times
d}^{2},$

where $\chi_{(T-1)\times d}^{2}$ is the chi-square distribution with
$(T-1)\times d$ degrees of freedom. Notice that

$\displaystyle\small\mathrm{vec}({\mathbf{s}}-\boldsymbol{\mu})^{\top}[\Sigma_{T}\otimes\Sigma]^{-1}\mathrm{vec}({\mathbf{s}}-\boldsymbol{\mu})=\mathrm{tr}(\Sigma^{-1}({\mathbf{s}}-\boldsymbol{\mu})\Sigma_{T}^{-1}({\mathbf{s}}-\boldsymbol{\mu})^{\top}),$

we can thus define the new BBScore.

######  Theorem 3.

The BBScore is defined as

$\displaystyle
B(\bar{{\mathbf{s}}}|\widehat{\Sigma})=\mathrm{tr}(\widehat{\Sigma}^{-1}({\mathbf{s}}-\boldsymbol{\mu})\Sigma_{T}^{-1}({\mathbf{s}}-\boldsymbol{\mu})^{\top})/[(T-1)d].$

This BBScore is always positive, and a larger value implies that the sequence
is less likely to have been generated by the given covariance matrix.

###  4 Experiments

####  4.1 Problems

In the experimental section, we begin by comparing the heuristic BBScore,
defined in Sheng et al. [2024], on coherence evaluation tasks. We then
demonstrate that this new BBScore performs better on multiple tasks, including
out-of-domain tasks, highlighting its significant potential. Moreover, we
illustrate that our new score enables comparisons among different articles,
achieving excellent results. Following this, we present results from the BB
encoder, which show similar outcomes to the contrastive learning results.
Although it does not surpass them in coherence evaluation tasks, it shows a
better fitting to the target distribution (BB). Finally, we demonstrate that
our computation scheme can be used for human-AI differentiation tasks.

####  4.2 Datasets

###### WikiSection

We use dataset introduced in [Arnold et al., 2019] which contains selected
Wikipedia articles on the topic of global cities and have clear topic
structures. Each article in this collection follows a pattern certain sections
such as abstract, history, geographics and demographics. The training split
contains 2165 articles and the test split has 658 articles.

###### WikiText

WikiText language modeling dataset [Merity et al., 2016] is a much larger set
of verified good and featured articles extracted from Wikipedia compared to
WikiSection. We used WikiText-103-v1 collection in specific for experiments.
This dataset encompass over 100 million tokens from 29,061 full articles. The
dataset is assessible through Huggingface
111https://huggingface.co/datasets/EleutherAI/WikiText_document_level

###### GCDC

This benchmark dataset [Lai and Tetreault, 2018] is specifically for
evaluating text coherence and each text in the dataset is annotated by human
judges for coherence (low, middle, high). These annotations provide reliable
ground truth for evaluating the performance of a coherence model. However, the
average article length is much smaller comparing to wikipedia articles.

###### HC3

The Human ChatGPT Comparison Corpus (HC3) Guo et al. [2023] includes
comparative responses from human experts and ChatGPT, covering questions from
various fields such as open-domain, finance, medicine, law, and psychology.

####  4.3 Evaluation methods

###### Global and Local discrimination

Evaluation through discrimination tasks has been a widely adopted approach in
coherence evaluation. In global discrimination tasks, we shuffle the entire
article with varying shuffle sentence block sizes (1, 2, 5, and 10). For local
discrimination tasks, we followed the practice in Moon et al. [2019] and
randomly sampled a set of windows, denoted as $w$, from each article. Each
window consists of a fixed size of 3 sentences. We then shuffle sentences
within each window to create perturbed copies. For each article and each task,
we randomly create 20 shuffled copies and discard any duplicates.

###### Relative scores

To evaluate BBScore’s ability to compare the coherence between different
articles, we introduce the following relative accuracy score.

$\small\textbf{Accuracy}=\frac{\sum_{A_{i}^{1}\in\mathcal{A}_{1}}\sum_{A_{j}^{2}\in\mathcal{A}_{2}}|f(A_{i}^{1},A_{j}^{2})|}{|\mathcal{A}_{1}||\mathcal{A}_{2}|}$

where $\mathcal{A}_{1}$ and $\mathcal{A}_{2}$ are sets of articles. In our
test, $\mathcal{A}_{2}$ can also consist of shuffled version of articles from
$\mathcal{A}_{1}$. $f(\cdot,\cdot)$ is defined by:

$\small f(A_{1},A_{2})=\begin{cases}1&\text{If
}\left(\left(A_{1}>_{B}A_{2}\right)\land\left(A_{1}>_{c}A_{2}\right)\right)\lor\left(\left(A_{1}<_{B}A_{2}\right)\land\left(A_{1}<_{c}A_{2}\right)\right),\\\
0&\text{Otherwise.}\end{cases}$

where $A_{1}>_{c}A_{2}$ ($A_{1}<_{c}A_{2}$) means article $A_{1}$ is more
(less) coherent than article $A_{2}$, and $A_{1}>_{B}A_{2}$
($A_{1}<_{B}A_{2}$) means article $A_{1}$ is more (less) coherent than article
$A_{2}$ evaluated by the BBScore.

###  5 Results

####  5.1 Coherence Evaluation Tasks

In global and local discrimination tasks on WikiSection, as shown in Tables 1
and 2, our BBScore (p-value) significantly outperforms the heuristic BBScore
and matches state-of-the-art (SOTA) results. Unlike the SOTA method, which
rely on pairwise training and can’t compare different articles, our BBScore
stands out in this aspect. (See Appendix C for more details on methods we
compared to.)

Methods | Train | Test
---|---|---
$\mathcal{D}_{b=1}$ |  $\mathcal{D}_{b=2}$ |  $\mathcal{D}_{b=5}$ |  $\mathcal{D}_{b=10}$ |  $\mathcal{D}_{b=1}$ |  $\mathcal{D}_{b=2}$ |  $\mathcal{D}_{b=5}$ |  $\mathcal{D}_{b=10}$
Entity Grid Barzilay and Lapata [2005] |  79.17 |  86.20 |  74.51 |  60.70 |  85.73 |  82.79 |  75.81 |  64.65
Unified Coherence Moon et al. [2019] |  99.75 |  98.60 |  97.10 |  96.21 |  99.73 |  97.86 |  96.90 |  96.09
BBScore, heuristic Sheng et al. [2024] |  76.29 |  75.12 |  73.04 |  73.12 |  83.39 |  80.71 |  79.36 |  78.66
BBScore, p-value |  96.35 |  95.90 |  96.54 |  97.34 |  95.06 |  94.72 |  95.13 |  95.67
BBScore, p-value [Relative score] |  91.63 |  86.74 |  80.02 |  78.31 |  90.32 |  86.03 |  79.26 |  77.89

Table 1: Global Discrimination Task Results on WikiSection.

Methods | Train | Test
---|---|---
$\mathcal{D}_{w=1}$ |  $\mathcal{D}_{w=2}$ |  $\mathcal{D}_{w=3}$ |  $\mathcal{D}_{w=1}$ |  $\mathcal{D}_{w=2}$ |  $\mathcal{D}_{w=3}$
Entity Grid Barzilay and Lapata [2005] |  55.37 |  62.94 |  65.44 |  53.04 |  60.83 |  66.67
Unified Coherence Moon et al. [2019] |  84.02 |  83.64 |  89.59 |  77.47 |  82.98 |  87.87
BBScore, heuristic Sheng et al. [2024] |  47.17 |  55.84 |  63.10 |  50.29 |  60.15 |  64.06
BBScore, p-value |  55.30 |  68.73 |  74.16 |  56.35 |  68.83 |  74.24

Table 2: Local Discrimination Task Results on WikiSection.
$\mathcal{D}_{w=1,2,3}$ represents the joint set of all three other datasets.

We test our BBScore with global discrimination tasks on the GCDC dataset,
which includes several human-labeled samples. This allows direct comparison
with human labels. As BBScore being continuous and label-independent, we focus
on two sub-tasks: 1) Discretize the BBScore based on class labels by
approximating the threshold from the training set, then compare it with true
labels using Spearman rank correlation. 2) Compare relative scores between
articles, defining coherence based on their human labels. As shown in Table 3,
our score surpasses the baseline from Lai and Tetreault [2018], which
discretizes the Flesch-Kincaid grade level using training data thresholds. Our
method achieves 65% accuracy in coherence comparison against human labels,
compared to the 55-58% accuracy of current SOTA models designed to predict
true human labels Lai and Tetreault [2018], Jeon and Strube [2022]. Our
approach emphasizes relative relationships between articles, enhancing its
predictive capabilities.

Methods | Spearman | Relative acc (%)
---|---|---
Enron |  Clinton |  Yahoo |  Yelp |  Enron |  Clinton |  Yahoo |  Yelp
BBScore, p-value |  0.3523 |  0.4394 |  0.4335 |  0.5169 |  59.00 |  68.23 |  65.67 |  70.98
SOTA [Lai and Tetreault, 2018] |  0.454 |  0.505 |  0.519 |  0.329 |  – |  – |  – |  –
Baseline [Lai and Tetreault, 2018] |  0.244 |  0.323 |  0.089 |  0.200 |  – |  – |  – |  –

Table 3: GCDC 3-way classification results.

####  5.2 Coherence Evaluation Tasks (O.O.D)

We evaluate BBScore on an out-of-domain dataset by training the encoder with
contrastive learning and BB loss on WikiText, and then applying it to compute
the latent trajectory of WikiSection. We assess performance on the global
discrimination task (Table 4). The results show that BBScore achieve high
performance in out-of-domain tests, with the BB encoder outperforming the CL
encoder. It indicates that the BB encoder effectively learns both the
distribution and the dynamics of the entire article. Additionally, the
approximation of $\widehat{\Sigma}$ acts as “parameter-free fine-tuning” in
our opinion.

Methods | Train | Test
---|---|---
$\mathcal{D}_{b=1}$ |  $\mathcal{D}_{b=2}$ |  $\mathcal{D}_{b=5}$ |  $\mathcal{D}_{b=10}$ |  $\mathcal{D}_{b=1}$ |  $\mathcal{D}_{b=2}$ |  $\mathcal{D}_{b=5}$ |  $\mathcal{D}_{b=10}$
BBScore, p-value, (wiki text, CL) |  91.26 |  87.11 |  86.21 |  86.78 |  91.30 |  87.22 |  86.14 |  88.18
BBScore, p-value, (wiki text, BB) |  94.15 |  90.69 |  85.26 |  86.65 |  93.03 |  89.32 |  83.02 |  88.31

Table 4: Global Discrimination Task Results on WikiSection (out of domain
test)

####  5.3 Comparision between BB Encoder and CL Encoder

In this task, we conduct a comparison between the BB encoder and the CL
encoder. First, as shown in Fig 2, we plot the rough shape of the latent
trajectory (blue and red regions) and the target BB (green region), where we
observe that the BB encoder appears to learn the target BB more accurately.

Figure 2: Contrastive learning vs BBEncoder in the latent space

We then analyze the performance of two encoders: global discrimination and
article-wise comparison on WikiSection, and out-of-domain analysis with the
encoder trained on WikiText and tested on WikiSection. As shown in Figure 3,
the CL encoder excels in both global discrimination and article-wise
comparison, effectively distinguishing between shuffled and unshuffled
articles as well as different articles and their shuffled counterparts.
However, the BB encoder shows better performance in out-of-domain tasks. This
suggests that the BB encoder captures common structures or dynamics among
articles, helping it maintain performance in out-of-domain tasks.

Figure 3: Contrastive learning vs BBEncoder for different tasks on WikiSection
testset

####  5.4 Results of human-AI comparison

In this section, we operate under the assumption that comparing with AI
writing, human writing exhibits similar dynamics and structure to other human-
written articles. As shown in Table 5, we test our assumption using the
dataset from Guo et al. [2023]. We compute the BBScore with different
$\widehat{\Sigma}$ values, where the subscript indicates the dataset used for
approximation. We then compare the BBScore between human and AI-generated
articles, assuming human articles will have a smaller BBScore due to their
similarity to other human writings. Our results show that
$\widehat{\Sigma}_{wiki}$ performs best in both the Human-AI (no questions)
and Human-AI (Q&A) benchmarks without any additional training. This
demonstrates its potential strength, as it is training-free and based purely
on comparing article dynamics and writing structure.

| Human AI comparison | Human AI comparison with Q&A
---|---|---
Human ($\widehat{\Sigma}_{human}$) | Human ($\widehat{\Sigma}_{ai}$) | Human ($\widehat{\Sigma}_{wiki}$) | Human ($\widehat{\Sigma}_{human}$) | Human ($\widehat{\Sigma}_{ai}$) | Human ($\widehat{\Sigma}_{wiki}$)
AI ($\widehat{\Sigma}_{human}$) | 70.07 | 70.55 | - | 69.00 | 69.60 | -
AI ($\widehat{\Sigma}_{ai}$) | 59.98 | 61.52 | - | 58.19 | 59.74 | -
AI ($\widehat{\Sigma}_{wiki}$) | - | - | 70.67 | - | - | 69.71

Table 5: Combined accuracy of human AI comparison and human AI comparison with
Q&A

###  6 Discussion

In this paper, we propose a new BB encoder and compare it with the traditional
CL encoder, both theoretically and empirically. Our findings indicate that the
BB encoder performs better on out-of-domain tasks. One reason for this
improvement is that the BB encoder can more effectively learn the stochastic
representation, capturing both spatial and temporal dependencies within the
sequence. Building on the outputs of the BB encoder or CL encoder, we define a
new BBScore with theoretical motivation, which shows significant empirical
improvements over the heuristic BBScore in Sheng et al. [2024]. Using the
BBScore, we explore various downstream tasks, such as comparing the relative
coherence between coherent articles, beyond simple shuffling tasks, and
distinguishing between AI and human-written articles.

However, there are still some limitations in our current scheme: 1) The
hyperparameter search for BB encoder training is heuristic. 2) More
experiments are needed on multi-domain tasks to demonstrate the benefits of
our BB encoder and BBScore, aligning with the theoretical arguments in Section
3. 3) Additional empirical evidence is needed to show the advantages of our BB
encoder over the CL encoder since it learns the stochastic representation with
more information.

For future work, we aim to implement a more thorough analysis of how to use
BBScore for multi-domain tasks, such as domain identification. Additionally,
we plan to explore downstream tasks like text generation and applications with
large language models, which will be of independent interest.

### References

  * Zou et al. [2023] Hao Zou, Zae Myung Kim, and Dongyeop Kang.  A survey of diffusion models in natural language processing, 2023.
  * Yang et al. [2023] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang.  Diffusion models: A comprehensive survey of methods and applications.  _ACM Comput. Surv._ , 56(4), nov 2023.  ISSN 0360-0300.  doi: 10.1145/3626235.  URL https://doi.org/10.1145/3626235.
  * Yi et al. [2024] Qinghua Yi, Xiaoyu Chen, Chen Zhang, Zhen Zhou, Ling Zhu, and Xin Kong.  Diffusion models in text generation: a survey.  _PeerJ Computer Science_ , 10:e1905, 2024.  doi: 10.7717/peerj-cs.1905.  URL https://doi.org/10.7717/peerj-cs.1905.
  * Lai and Tetreault [2018] Alice Lai and Joel Tetreault.  Discourse coherence in the wild: A dataset, evaluation and methods.  In _Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 214–223, Melbourne, Australia, July 2018. Association for Computational Linguistics.  doi: 10.18653/v1/W18-5023.  URL https://aclanthology.org/W18-5023.
  * Jeon and Strube [2022] Sungho Jeon and Michael Strube.  Entity-based neural local coherence modeling.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7787–7805, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.537.  URL https://aclanthology.org/2022.acl-long.537.
  * van den Oord et al. [2018] Aäron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _CoRR_ , abs/1807.03748, 2018.  URL http://arxiv.org/abs/1807.03748.
  * Durkan et al. [2020] Conor Durkan, Iain Murray, and George Papamakarios.  On contrastive learning for likelihood-free inference.  In Hal Daumé III and Aarti Singh, editors, _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 2771–2781. PMLR, 13–18 Jul 2020.  URL https://proceedings.mlr.press/v119/durkan20a.html.
  * Wang et al. [2022] Rose E Wang, Esin Durmus, Noah Goodman, and Tatsunori Hashimoto.  Language modeling via stochastic processes.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=pMQwKL1yctf.
  * Øksendal and Øksendal [2003] Bernt Øksendal and Bernt Øksendal.  _Stochastic differential equations_.  Springer, 2003.
  * Krumm [2021] John Krumm.  Brownian bridge interpolation for human mobility?  In _Proceedings of the 29th International Conference on Advances in Geographic Information Systems_ , SIGSPATIAL ’21, pages 175–183, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450386647.  doi: 10.1145/3474717.3483942.  URL https://doi.org/10.1145/3474717.3483942.
  * Horne et al. [2007] Jon S Horne, Edward O Garton, Stephen M Krone, and Jesse S Lewis.  Analyzing animal movements using brownian bridges.  _Ecology_ , 88(9):2354–2363, Sep 2007.  ISSN 0012-9658 (Print); 0012-9658 (Linking).  doi: 10.1890/06-0957.1.
  * Sheng et al. [2024] Zhecheng Sheng, Tianhao Zhang, Chen Jiang, and Dongyeop Kang.  Bbscore: A brownian bridge based metric for assessing text coherence.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 38(13):14937–14945, 2024.
  * Lim et al. [2023] Sungbin Lim, Eunbi Yoon, Taehyun Byun, Taewon Kang, Seungwoo Kim, Kyungjae Lee, and Sungjoon Choi.  Score-based generative modeling through stochastic evolution equations in hilbert spaces.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=GrElRvXnEj.
  * Zhang et al. [2023] Heng Zhang, Daqing Liu, Qi Zheng, and Bing Su.  Modeling video as stochastic processes for fine-grained video representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2225–2234, June 2023.
  * Mathieu et al. [2021] Emile Mathieu, Adam Foster, and Yee Teh.  On contrastive representations of stochastic processes.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, _Advances in Neural Information Processing Systems_ , volume 34, pages 28823–28835. Curran Associates, Inc., 2021.  URL https://proceedings.neurips.cc/paper_files/paper/2021/file/f19c44d068fecac1d6d13a80df4f8e96-Paper.pdf.
  * Liu et al. [2021] Bingbin Liu, Pradeep Ravikumar, and Andrej Risteski.  Contrastive learning of strong-mixing continuous-time stochastic processes.  In Arindam Banerjee and Kenji Fukumizu, editors, _Proceedings of The 24th International Conference on Artificial Intelligence and Statistics_ , volume 130 of _Proceedings of Machine Learning Research_ , pages 3151–3159. PMLR, 13–15 Apr 2021.  URL https://proceedings.mlr.press/v130/liu21h.html.
  * Reinhart [1980] Tanya Reinhart.  Conditions for text coherence.  _Poetics Today_ , 1(4):161–180, 1980.
  * Deng et al. [2022] Yuntian Deng, Volodymyr Kuleshov, and Alexander Rush.  Model criticism for long-form text generation.  In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11887–11912, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.emnlp-main.815.  URL https://aclanthology.org/2022.emnlp-main.815.
  * Bowman et al. [2016] Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio.  Generating sentences from a continuous space.  In Stefan Riezler and Yoav Goldberg, editors, _Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning_ , pages 10–21, Berlin, Germany, August 2016. Association for Computational Linguistics.  doi: 10.18653/v1/K16-1002.  URL https://aclanthology.org/K16-1002.
  * Gao et al. [2021] Tianyu Gao, Xingcheng Yao, and Danqi Chen.  SimCSE: Simple contrastive learning of sentence embeddings.  In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.552.  URL https://aclanthology.org/2021.emnlp-main.552.
  * Maimon and Tsarfaty [2023] Aviya Maimon and Reut Tsarfaty.  A Novel Computational and Modeling Foundation for Automatic Coherence Assessment.  _arXiv e-prints_ , art. arXiv:2310.00598, October 2023.  doi: 10.48550/arXiv.2310.00598.
  * Barzilay and Lapata [2005] Regina Barzilay and Mirella Lapata.  Modeling local coherence: An entity-based approach.  In Kevin Knight, Hwee Tou Ng, and Kemal Oflazer, editors, _Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)_ , pages 141–148, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics.  doi: 10.3115/1219840.1219858.  URL https://aclanthology.org/P05-1018.
  * Moon et al. [2019] Han Cheol Moon, Tasnim Mohiuddin, Shafiq Joty, and Chi Xu.  A unified neural coherence model.  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2262–2272, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1231.  URL https://aclanthology.org/D19-1231.
  * Arnold et al. [2019] Sebastian Arnold, Benjamin Schrauwen, Verena Rieser, and Katja Filippova.  Sector: A neural model for coherent topic segmentation and classification.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 241–253, Hong Kong, China, 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1024.  URL https://aclanthology.org/D19-1024.
  * Merity et al. [2016] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.  Pointer sentinel mixture models, 2016.
  * Guo et al. [2023] Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu.  How close is chatgpt to human experts? comparison corpus, evaluation, and detection.  _arXiv preprint arXiv:2301.07597_ , 2023.

## Appendix

###  Appendix A Proofs

####  A.1 Proof of Theorem 1

###### Proof.

We fix the start and end points $s_{0}$ and $s_{T}$, and calculate the
likelihood function of the input sequence ${\mathbf{s}}$.

By $s_{t}-\mu_{t}={\mathbf{W}}(B_{1}(t),\ldots,B_{d}(t))^{\top}$, the
independence between $B_{1}(t),\ldots,B_{d}(t)$, and the properties of the
standard BB, we have for any $t,t^{\prime}\in\llbracket T-1\rrbracket$,
$\mathrm{E}[s_{t}-\mu_{t}]=0$,
$\mathrm{Var}[s_{t}]=[\Sigma_{T}]_{t,t}{\mathbf{W}}{\mathbf{W}}^{\top}$ and
${\rm
Cov}[s_{t},s_{t^{\prime}}]=[\Sigma_{T}]_{t,t^{\prime}}{\mathbf{W}}{\mathbf{W}}^{\top}$.
Then we have

$\displaystyle\mathrm{vec}({\mathbf{s}}-\boldsymbol{\mu})\sim
N(0,\Sigma_{T}\otimes({\mathbf{W}}{\mathbf{W}}^{\top})),$

where $\mathrm{vec}(\cdot)$ is the vectorization operator and $\otimes$ is the
Kronecker product.

By the likelihood function of the multivariate normal distribution, we have

$\displaystyle\small L({\mathbf{W}}|\bar{{\mathbf{s}}})=$
$\displaystyle(2\pi)^{-d(T-1)/2}\lvert\Sigma_{T}\otimes({\mathbf{W}}{\mathbf{W}}^{\top})\rvert^{-1/2}\exp[-\mathrm{vec}({\mathbf{s}}-\boldsymbol{\mu})^{\top}[\Sigma_{T}\otimes({\mathbf{W}}{\mathbf{W}}^{\top})]^{-1}\mathrm{vec}({\mathbf{s}}-\boldsymbol{\mu})/2].$

Since $\Sigma={\mathbf{W}}{\mathbf{W}}^{\top}$, the only unknown parameters
are $\Sigma$. Note that
$\lvert\Sigma_{T}\otimes\Sigma\rvert=\lvert\Sigma_{T}\rvert^{d}\lvert\Sigma\rvert^{T-1}$
and

$\displaystyle\mathrm{vec}({\mathbf{s}}-\boldsymbol{\mu})^{\top}[\Sigma_{T}\otimes\Sigma]^{-1}\mathrm{vec}({\mathbf{s}}-\boldsymbol{\mu})=\mathrm{vec}({\mathbf{s}}-\boldsymbol{\mu})^{\top}[\Sigma_{T}^{-1}\otimes\Sigma^{-1}]\mathrm{vec}({\mathbf{s}}-\boldsymbol{\mu})$
$\displaystyle=$
$\displaystyle\mathrm{vec}({\mathbf{s}}-\boldsymbol{\mu})^{\top}\mathrm{vec}(\Sigma^{-1}({\mathbf{s}}-\boldsymbol{\mu})\Sigma_{T}^{-1})=\mathrm{tr}(({\mathbf{s}}-\boldsymbol{\mu})^{\top}\Sigma^{-1}({\mathbf{s}}-\boldsymbol{\mu})\Sigma_{T}^{-1})$
$\displaystyle=$
$\displaystyle\mathrm{tr}(\Sigma^{-1}({\mathbf{s}}-\boldsymbol{\mu})\Sigma_{T}^{-1}({\mathbf{s}}-\boldsymbol{\mu})^{\top}).$

Then we have the likelihood function is

$\displaystyle L(\Sigma|\bar{{\mathbf{s}}})=$
$\displaystyle(2\pi)^{-d(T-1)/2}\lvert\Sigma_{T}\rvert^{-d/2}\lvert\Sigma\rvert^{-(T-1)/2}\exp[-\mathrm{tr}(\Sigma^{-1}({\mathbf{s}}-\boldsymbol{\mu})\Sigma_{T}^{-1}({\mathbf{s}}-\boldsymbol{\mu})^{\top})/2],$

and then the log-likelihood function is

$\displaystyle\ell(\Sigma|\bar{{\mathbf{s}}})=$
$\displaystyle-\frac{d(T-1)}{2}\log(2\pi)-\frac{d}{2}\log(\lvert\Sigma_{T}\rvert)-\frac{(T-1)}{2}\log(\lvert\Sigma\rvert)-\frac{1}{2}\mathrm{tr}(\Sigma^{-1}({\mathbf{s}}-\boldsymbol{\mu})\Sigma_{T}^{-1}({\mathbf{s}}-\boldsymbol{\mu})^{\top}).$

Then for $n$ independent input sequence
$\bar{\mathbf{s}}_{1},\ldots,\bar{\mathbf{s}}_{n}$ with lengths
$T_{1}+1,\ldots,T_{n}+1$, generated by the same ${\mathbf{W}}$ (or $\Sigma$),
we can define the corresponding ${\mathbf{s}}_{i}$ and $\boldsymbol{\mu}_{i}$,
and then we have

$\displaystyle
L(\Sigma|\\{{\mathbf{s}}_{i}\\}_{i=1}^{n})=\Pi_{i=1}^{n}L(\Sigma|{\mathbf{s}}_{i}),$

and

$\displaystyle\ell(\Sigma|\\{{\mathbf{s}}_{i}\\}_{i=1}^{n})=\sum_{i=1}^{n}\ell(\Sigma|{\mathbf{s}}_{i})$
$\displaystyle=$
$\displaystyle-\frac{d\sum_{i=1}^{n}(T_{i}-1)}{2}\log(2\pi)-\frac{d}{2}\sum_{i=1}^{n}\log(\lvert\Sigma_{T_{i}}\rvert)-\frac{\sum_{i=1}^{n}(T_{i}-1)}{2}\log(\lvert\Sigma\rvert)-\frac{1}{2}\sum_{i=1}^{n}\mathrm{tr}(\Sigma^{-1}({\mathbf{s}}_{i}-\boldsymbol{\mu}_{i})\Sigma_{T_{i}}^{-1}({\mathbf{s}}_{i}-\boldsymbol{\mu}_{i})^{\top}).$

∎

####  A.2 Proof of Theorem 2

###### Proof.

Maximizing $L(\Sigma|\\{{\mathbf{s}}_{i}\\}_{i=1}^{n})$ is equivalent to
minimize

$\displaystyle
g(\Sigma)=\sum_{i=1}^{n}(T_{i}-1)\log(\lvert\Sigma\rvert)+\sum_{i=1}^{n}\mathrm{tr}(\Sigma^{-1}({\mathbf{s}}_{i}-\boldsymbol{\mu}_{i})\Sigma_{T_{i}}^{-1}({\mathbf{s}}_{i}-\boldsymbol{\mu}_{i})^{\top}).$

Then since $\Sigma={\mathbf{W}}{\mathbf{W}}^{\top}$ is positive definite
matrix, then

$\displaystyle\frac{\mathrm{d}}{\mathrm{d}\Sigma}\mathrm{tr}(\Sigma^{-1}({\mathbf{s}}_{i}-\boldsymbol{\mu}_{i})\Sigma_{T_{i}}^{-1}({\mathbf{s}}_{i}-\boldsymbol{\mu}_{i})^{\top})$
$\displaystyle=$
$\displaystyle-(\Sigma^{-1}({\mathbf{s}}_{i}-\boldsymbol{\mu}_{i})\Sigma_{T_{i}}^{-1}({\mathbf{s}}_{i}-\boldsymbol{\mu}_{i})^{\top}\Sigma^{-1})^{\top}=-\Sigma^{-1}({\mathbf{s}}_{i}-\boldsymbol{\mu}_{i})\Sigma_{T_{i}}^{-1}({\mathbf{s}}_{i}-\boldsymbol{\mu}_{i})^{\top}\Sigma^{-1}.$

Note that
$(\mathrm{d}/\mathrm{d}\Sigma)\log(\lvert\Sigma\rvert)=(\Sigma^{\top})^{-1}=\Sigma^{-1}$.
Then

$\displaystyle\frac{\mathrm{d}}{\mathrm{d}\Sigma}g(\Sigma)=\Big{(}\sum_{i=1}^{n}(T_{i}-1)\Big{)}\Sigma^{-1}-\Sigma^{-1}\Big{(}\sum_{i=1}^{n}({\mathbf{s}}_{i}-\boldsymbol{\mu}_{i})\Sigma_{T_{i}}^{-1}({\mathbf{s}}_{i}-\boldsymbol{\mu}_{i})^{\top}\Big{)}\Sigma^{-1}$

By taking $(\mathrm{d}/\mathrm{d}\Sigma)g(\Sigma)=0$, we have

$\displaystyle\widehat{\Sigma}=\Big{(}\sum_{i=1}^{n}(T_{i}-1)\Big{)}^{-1}\Big{(}\sum_{i=1}^{n}({\mathbf{s}}_{i}-\boldsymbol{\mu}_{i})\Sigma_{T_{i}}^{-1}({\mathbf{s}}_{i}-\boldsymbol{\mu}_{i})^{\top}\Big{)}.$

∎

###  Appendix B Training Details

####  B.1 BB Encoder

The WikiSection BB encoder was trained on 1 A100 GPU for about 10 hours using
the training set of WikiSection for 100 epochs. We used SGD optimizer and set
the learning rate to be 1e-9. The $\epsilon$ in the loss function $L_{NLL}$ is
chosen as 1e-7. The WikiText BB encoder was trained on 4 A100 GPUs for roughly
20 hours for 4 epochs with WikiText dataset. For this dataset, we trained with
AdamW optimizer with learning rate 1e-9 and batch size 32. The $\epsilon$ in
the loss function $L_{NLL}$ is chosen as 1e-3. Other hyperparameters can be
accessed from the configuration file in the submitted code. Our empirical
results show incorporating $\hat{\sigma}_{j}$ into the $\widehat{\Sigma}_{j}$
makes no significant results in the downstream tasks, thus we disregard
$\hat{\sigma}_{j}$ during encoder training.

####  B.2 Hyper-parameter Tuning

While training the WikiSection BB encoder, we experimented with different
$\epsilon$ in $L_{NLL}$ to see its impact on the performance of the trained
encoder. Note that $\epsilon$ determines the perturbation added to the matrix
$\widehat{\Sigma}$. The eigenvalues of the initial $\widehat{\Sigma}$ range
from $10^{-6}$ to $10^{-1}$, with the majority of which lying in
$[10^{-3},10^{-5}]$. Thus we tested the following three different $\epsilon$:

  * •

Large $\epsilon=10^{-3}$ that is larger that most eigenvalues of
$\widehat{\Sigma}$.

  * •

Medium $\epsilon=10^{-5}$ that is about the same scale of most eigenvalues of
$\widehat{\Sigma}$.

  * •

Small $\epsilon=10^{-7}$ that is smaller than most eigenvalues of
$\widehat{\Sigma}$.

We choose the small $\epsilon$ based on the performance.

###  Appendix C Other scores used in this paper

###### Entity Grid

Barzilay and Lapata [2005] is the most recognized entity-based approach. It
creates a two-way contingency table for each input document to track the
appearance of entities in each sentence. We use Stanford’s CoreNLP to annotate
the documents and the implementation provided in the Coheoka
library222https://github.com/kigawas/coheoka to obtain the Entity Grid score.

###### Unified Coherence

Moon et al. [2019] presents a neural-based entity-grid method that integrates
sentence grammar, inter-sentence coherence relations, and global coherence
patterns, achieving state-of-the-art results in artificial tasks.

###### BBScore, Heuristic

Sheng et al. [2024] introduces a framework for the computation of BBScore. The
score is supposed to represent the deviation from the latent Brownian
trajectory from training data. (Also see 3.3)