# A Multivariate Point Process Model for
Simultaneously Recorded Neural Spike Trains

Reza Ramezan<EMAIL_ADDRESS>Meixi Chen  Martin Lysy  Paul Marriott
Department of Statistics and Actuarial Science, University of Waterloo, Canada

###### Abstract

The current state-of-the-art in neurophysiological data collection allows for
simultaneous recording of tens to hundreds of neurons, for which point
processes are an appropriate statistical modelling framework. However,
existing point process models lack multivariate generalizations which are both
flexible and computationally tractable. This paper introduces a multivariate
generalization of the Skellam process with resetting (SPR), a point process
tailored to model individual neural spike trains. The multivariate SPR (MSPR)
is biologically justified as it mimics the process of neural integration. Its
flexible dependence structure and a fast parameter estimation method make it
well-suited for the analysis of simultaneously recorded spike trains from
multiple neurons. The strengths and weaknesses of the MSPR are demonstrated
through simulation and analysis of experimental data.

###### keywords:

multivariate point processes; Skellam process with resetting; records; spike
trains; neurophysiology data

##  1 Introduction

The statistical modelling of neural spike trains falls under the general
framework of point processes. Among point processes, the (inhomogeneous)
Poisson process is a popular choice [1, 14], but its theoretical properties
fail to address biological phenomena such as refractoriness and burst spiking
activities [15]. Furthermore, the Poisson process has very limited
multivariate generalizations in terms of dependence structure [12, 9, 4, 11,
6, 7]. Some alternative models include Integrate-and-Fire models [3, 13], GLMs
[8, 2, 17, 18], renewal processes [10], and latent variable models [16, 19].

The Skellam process with resetting (SPR) is a point process recently developed
to model spike trains collected from an individual neuron [15]. A Skellam
process $\\{S(t),t\geq 0\\}$ is defined as the difference between two
independent Poisson processes, i.e. $S(t)=N_{1}(t)-N_{2}(t)$, where
$N_{i}(t)\sim Poisson(\lambda_{i}\,t)$, $i=1,2$. Let $T_{n}=\min\\{t:S(t)\geq
n\\}$. The SPR processs is then defined as
$\tilde{S}(t)=\\{S(t)-\sum_{j=1}^{\infty}\delta(t\geq T_{j}),~{}t\geq 0\\}$,
where $\delta(\cdot)$ is the indicator function. Thus, resetting refers to
bringing the path of the Skellam process back to state 0 after each spike,
i.e. after each record of the process, to mimic the refractoriness of neurons
[15]. In this paper we generalize the results of [15] and introduce the
multivariate Skellam process with resetting (MSPR) for the analysis of
simultaneously recorded spike trains from multiple neurons.

##  2 Multivariate Skellam Process with Resetting

We first introduce the multivariate Skellam random vector, following a similar
construction to that the multivariate Poisson random vector [6, 7].

Definition 1: Let $\mathbf{Z}$ be a $p$-variate random vector where

$\mathbf{Z}=\begin{pmatrix}Z_{1}\\\ Z_{2}\\\ \vdots\\\ Z_{i}\\\ \vdots\\\
Z_{p}\\\ \end{pmatrix}=\begin{pmatrix}Y_{1}&+&\sum_{j=2}^{p}a_{1j}Y_{1j}&&\\\
Y_{2}&+&a_{21}Y_{12}&+&\sum_{j=3}^{p}a_{2j}Y_{2j}\\\ \vdots\\\
Y_{i}&+&\sum_{j=1}^{i-1}a_{ij}Y_{ji}&+&\sum_{j=i+1}^{p}a_{ij}Y_{ij}\\\
\vdots\\\ Y_{p}&&&+&\sum_{j=1}^{p-1}a_{pj}Y_{jp}\\\ \end{pmatrix},$

where $a_{ij}\in\\{-1,0,1\\}$, $Y_{i}\sim Skellam(\lambda_{i1},\lambda_{i2})$,
and $Y_{ij}\sim Skellam(\gamma_{ij},\gamma_{ij})$. All Skellam random
variables $Y_{i}$ and $Y_{ij}$, $\forall i,j$ are independent of one another,
and also $a_{ij}=0\implies a_{ji}=0$. The random vector $\mathbf{Z}\sim
MSk(\boldsymbol{\lambda}_{1},\boldsymbol{\lambda}_{2},\boldsymbol{\gamma})$ is
called a multivariate Skellam random vector with parameters
$\boldsymbol{\lambda}_{1}=(\lambda_{11},...,\lambda_{p1})$,
$\boldsymbol{\lambda}_{2}=(\lambda_{12},...,\lambda_{p2})$, and
$\boldsymbol{\gamma}=(\gamma_{12},...,\gamma_{(p-1)p})$.

Definition 2: Let $\mathbf{S}(t)=\\{S^{(i)}(t),t\geq 0,i=1,\ldots,p\\}$ be a
$p$-dimensional Skellam processes, i.e., using Definition 1 with Skellam
processes instead of random variables. Then for
$T^{(i)}_{n}=\min\\{t:S^{(i)}(t)\geq n\\}$, the multivariate Skellam process
with resetting (MSPR) is defined as

$\tilde{\mathbf{S}}(t)=\Big{\\{}S_{0}^{(i)}(t)-\sum_{j=1}^{\infty}\delta(t\geq
T^{(i)}_{j}),~{}t\geq 0,~{}i=1,\ldots,p\Big{\\}}.$

The value of a $p$-variate Skellam process at time $t=t_{0}$ is a multivariate
Skellam random vector, $\mathbf{S}(t_{0})\sim
MSk(\boldsymbol{\lambda}_{1}t_{0},\boldsymbol{\lambda}_{2}t_{0},\boldsymbol{\gamma}\,t_{0})$.
Within this process, the spike times of neuron $i$ are modelled as the record
times of the $i$th SPR within the multivariate Skellam process. These
individual processes are potentially dependent to allow modelling the
relationships between cells in a neuronal ensemble.

To the best of our knowledge, the MSPR is the first multivariate point process
model for neural spike trains which allows for the full range of correlation
coefficient. Indeed, neurons $i$ and $j$ are positively correlated if
$sign(a_{ij})=sign(a_{ji})$, negatively correlated if
$sign(a_{ij})=-sign(a_{ji})$, and uncorrelated if $a_{ij}=a_{ji}=0$.

###  2.1 Parameter Estimation

We assume that the spike times from multiple independent trials are available
for each of the simultaneously recorded neurons. The duration of each trial is
assumed to be “long” relative to the average inter-spike interval so that
asymptotic results apply.

Following Definitions 1 and 2, and the parameterization of [15], the two
parameters of the marginal SPR for neuron $i$ are
$\lambda_{1}^{(i)}=\lambda_{i1}+\sum_{j}\gamma_{ij}$,
$\lambda_{2}^{(i)}=\lambda_{i2}+\sum_{j}\gamma_{ij}$, for $i=1,..,p$ . We use
the method of moments to estimate $\gamma_{ij}$ using spike counts in each
trial, i.e. $\widehat{\gamma}_{ij}=|0.5\widehat{\sigma}_{ij}|$ in which
$\widehat{\sigma}_{ij}$ is the estimate of the covariance between spike counts
of the two neurons $i$ and $j$. We then plug in the moment estimate
$\widehat{\gamma}_{ij}$ in the likelihood function of the marginal SPR
processes and compute maximum profile likelihood estimates of parameters
$\lambda_{1}^{(i)}$ and $\lambda_{2}^{(i)}$ for $i=1,...,p$ using constrained
optimization. The standard error of estimates are obtained by boostrapping
over the independent trials.

##  3 Data Analysis

We assessed the MSPR model using a 3-neuron simulation study as well as a
5-neuron experimental dataset from the prefrontal cortex of a mouse (3 in OFC
and 2 in PL) in a classical conditioning study where the observation window is
10 seconds after a reward delivery [5].

Table 1: Parameter estimation for 50 repeated trials from an ensemble of three simulated neurons | $\lambda_{11}$ | $\lambda_{12}$ | $\lambda_{21}$ | $\lambda_{22}$ | $\lambda_{31}$ | $\lambda_{32}$ | $\gamma_{12}$ | $\gamma_{13}$ | $\gamma_{23}$
---|---|---|---|---|---|---|---|---|---
True | 15 | 10 | 20 | 15 | 10 | 7 | 5 | 15 | 10
Est. | 17.8 | 11.6 | 19.7 | 12.9 | 15.2 | 10.9 | 5.0 | 11.5 | 8.3
SE | 7.4 | 7.2 | 6.9 | 7.0 | 7.8 | 7.6 | 3.2 | 5.2 | 5.7
Table 2: Moment estimates of the ISIs from 13 repeated trials from an ensemble of five prefrontal cortex neurons |  | Mean | Variance
---|---|---|---
| Observed ISI (SE) | 0.075 (0.035) | 0.007 (0.008)
Neuron #1 (OFC) | Model ISI (SE) | 0.121 (0.069) | 0.015 (0.025)
| Observed ISI (SE) | 0.059 (0.011) | 0.003 (0.002)
Neuron #2 (OFC) | Model ISI (SE) | 0.096 (0.021) | 0.009 (0.004)
| Observed ISI (SE) | 0.054 (0.011) | 0.002 (0.001)
Neuron #3 (OFC) | Model ISI (SE) | 0.083 (0.021) | 0.007 (0.004)
| Observed ISI (SE) | 0.076 (0.025) | 0.009 (0.008)
Neuron #4 (PL) | Model ISI (SE) | 0.118 (0.047) | 0.014 (0.014)
| Observed ISI (SE) | 0.057 (0.021) | 0.007 (0.006)
Neuron #5 (PL) | Model ISI (SE) | 0.080 (0.037) | 0.006 (0.008)

Table 1 shows the results for 50 spike trains from a 3-neuron simulation
study. Table 2 and Figure 1 present the results for the 5-neuron experiment.
To put the parameter estimates into context, Table 2 displays the means and
variances of the inter-spike intervals (ISI) as well as those estimated by the
MSPR model. Comparing the mean and variance estimates from the model to those
of the empirical ISI data, it seems that a slight systematic upward bias
exists in the model fit. While PP-plots for each neuron (not shown here) did
not reveal a noticeable lack of fit, bias correction for the parameter
estimation method could potentially be explored.

Figure 1:  Spike count correlation (left) and statistically significant
correlations only (right).

Figure 1 suggests that all OFC neurons are correlated, whereas PL neurons are
not. The correlations between OFC and PL neurons were not statistically
significant.

##  4 Discussion

This paper introduces the multivariate Skellam process with resetting for the
analysis of simultaneously recorded neural spike trains. Unlike other
multivariate point processes, the MSPR has a flexible dependency structure and
can be fit with relative ease. Future work includes the additional modelling
of the crossing times $T_{n}^{(i)}=\min\\{t:S^{(i)}\geq nk_{i}\\}$ for
$k_{i}$, $i=1,\ldots,p$ to be estimated from the data, as was done in the
univariate SPR model of [15]. The challenge is to perform parameter estimation
under this more complicated model without sacrificing computational
efficiency.

##  5 Acknowledgments

This work was supported by the Natural Sciences and Engineering Research
Council of Canada. We thank David Moorman of UMass Amherst for the classical
conditioning data.

## References

  * [1] E. N. Brown, L. M. Frank, D. Tang, M. C. Quirk, and M. A. Wilson.  A statistical paradigm for neural spike train decoding applied to position prediction from ensemble firing patterns of rat hippocampal place cells.  Journal of Neuroscience, 18(18):7411–7425, 1998.
  * [2] I. Dimatteo, C. R. Genovese, and R. E. Kass.  Bayesian curve-fitiing with free-knot splines.  Biometrika, 88(4):1055–1071, 2001.
  * [3] G. L. Gerstein and B. Mandelbrot.  Random walk models for the spike activity of a single neuron.  Biophysical Journal, 4(1):41–68, 1964.
  * [4] N. L. Johnson, S. Kotz, and N. Balakrishnan.  Discrete Multivariate Distributions.  John Wiley & Sons, New York, NY, 1997.
  * [5] Beata Kaminska.  Value and Action Encoding for Natural Outcomes and Ethanol in the Medial Prefrontal and Orbitofrontal Cortex.  PhD thesis, University of Massachusetts Amherst, 2020.
  * [6] D. Karlis.  Analysis of sports data by using bivariate Poisson models.  Journal of the Royal Statistical Society. Series D (The Statistician), 52(3):381–393, 2003.
  * [7] D. Karlis and L. Meligkotsidou.  Multivariate Poisson regression with covariance structure.  Statistics and Computing, 15:255–265, 2005.
  * [8] R. E. Kass and V. Ventura.  A spike-train probability model.  Neural Computation, 13(8):1713–1720, 2001.
  * [9] S Kocherlakota and K. Kocherlakota.  Bivariate Discrete Distributions.  Marcel Dekker Inc., New York, NY, 1992.
  * [10] S. Koyama.  On the spike train variability characterized by variance-to-mean power relationship.  Neural Computation, 27(7):1530–1548, July 2014.
  * [11] J. Lakshminarayana, S. N. N. Pandit, and K. S. Rao.  On a bivariate Poisson distribution.  Communications in Statistics - Theory and Methods, 28(2):267–276, 1999.
  * [12] A. W. Marshall and I. Olkin.  A family of bivariate distributions generated by the bivariate Bernoulli distribution.  Journal of American Statistical Association, 80:332–338, 1985.
  * [13] L. Paninski, E. N. Brown, S. Iyengar, and R. E. Kass.  Statistical models of spike trains.  In C. Liang and G. Lord, editors, Stochastic Methods in Neuroscience. Oxford University Press, 2008.
  * [14] R. Ramezan, P. Marriott, and S. Chenouri.  Multiscale analysis of neural spike trains.  Statistics in Medicine, 33(2):238–256, 2014.
  * [15] R. Ramezan, P. Marriott, and S. Chenouri.  Skellam process with resetting: A neural spike trains model.  Statistics in Medicine, 35(30):5717–5729, 2016.
  * [16] B. Shahbaba, B. Zhou, S. Lan, H. Ombao, D. Moorman, and S. Behseta.  A semiparametric bayesian model for detecting synchrony among multiple neurons.  Neural Computation, 26(9), 2014.
  * [17] W. Truccolo, U. T. Eden, M. R. Fellows, J. P. Donoghue, and E. N. Brown.  A point process framework for relating neural spiking activity to spiking history, neural ensemble, and extrinsic covariate effects.  Journal of Neurophysiology, 93:1074–1089, 2005.
  * [18] D. K. Wójcik, G. Mochol, W. Jakuczun, M. Wypych, and W.J. Waleszczyk.  Direct estimation of inhomogeneous markov interval models of spike trains.  Neural Computation, 21(8):2105–2113, August 2009.
  * [19] B. Zhou, D. Moorman, S. Behseta, H. Ombao, and B. Shahbaba.  A dynamic bayesian model for characterizing cross-neuronal interactions during decision-making.  Journal of the American Statistical Association, 111(514), 2016\.


# CHEAT: A Large-scale Dataset for Detecting ChatGPT-writtEn AbsTracts

Peipeng Yu
Jinan University
<EMAIL_ADDRESS>
Jiahan Chen
Jinan University
<EMAIL_ADDRESS>
Xuan Feng
Jinan University
<EMAIL_ADDRESS>
Zhihua Xia
Jinan University
<EMAIL_ADDRESS>

###### Abstract

The powerful ability of ChatGPT has caused widespread concern in the academic
community. Malicious users could synthesize dummy academic content through
ChatGPT, which is extremely harmful to academic rigor and originality. The
need to develop ChatGPT-written content detection algorithms call for large-
scale datasets. In this paper, we initially investigate the possible negative
impact of ChatGPT on academia, and present a large-scale CHatGPT-writtEn
AbsTract dataset (CHEAT) to support the development of detection algorithms.
In particular, the ChatGPT-written abstract dataset contains 35,304 synthetic
abstracts, with $Generation$, $Polish$, and $Mix$ as prominent
representatives. Based on these data, we perform a thorough analysis of the
existing text synthesis detection algorithms. We show that ChatGPT-written
abstracts are detectable, while the detection difficulty increases with human
involvement. Our dataset is available in https://github.com/botianzhe/CHEAT.

##  1 Introduction

ChatGPT, a natural language processing tool based on artificial intelligence
technology, has attracted widespread attention in recent times. Based on user
needs, ChatGPT could complete tasks with high quality, such as coding,
translation, thesis writingMacdonald et al. (2023), and so on. While ChatGPT
brings convenience to human life, the potential harm of its synthetic content
has gradually emergedUfuk (2023). As an important part of the academic thesis,
the abstract has been proven to be brilliantly synthesized by ChatGPTGao et
al. (2022); Mok (2023). Malicious researchers could synthesize plausible
academic content without practical research, which seriously undermines
academic originality.Thorp (2023). Meanwhile, although the synthetic content
exhibit high levels of authenticity, the academic rigor and the conclusion
correctness cannot be guaranteedElse (2023). How to detect ChatGPT-written
abstracts and ensure the academic originality have been pressing issues that
need to be addressed.

Figure 1: The different distributions of human-written and ChatGPT-written
abstracts. The visualization results are obtained by GLTR Gehrmann et al.
(2019).

As concerns over ChatGPT escalate, there has been a surge of interest in
detecting ChatGPT content. The availability of large-scale datasets is an
enabling factor for the development of ChatGPT content detection methods. So
far, researchers have constructed multiple detection datasets for question
answering and querying scenariosGuo et al. (2023); Mitrović et al. (2023).
Some studies also evaluated the quality of ChatGPT-written text with a small
amount of dataHuang et al. (2023); Cao et al. (2023); Gao et al. (2022).
However, due to the limitations of OpenAI, there is currently no large-scale
dataset to support the development of ChatGPT-written abstract detection
algorithms.111https://www.nature.com/articles/d41586-023-00553-9.

In this paper, we focus on the practical need to generate large-scale and
diverse ChatGPT-written abstracts. We explore different abstract synthesis
methods using ChatGPT, including $Generation$, $Polish$, and $Mix$. As shown
in Figure 1, ChatGPT-written abstracts are almost indistinguishable from the
human-written ones, but there are some differences in semantic distribution.
We aim to construct a large-scale dataset to analyze the abnormal distribution
patterns in ChatGPT-written abstracts. The contributions are summarized as
follows:

  1. (1)

We present a large-scale ChatGPT-written abstract dataset, CHEAT, which is
currently the largest dataset available for ChatGPT-written abstract
detection.

  2. (2)

We analyze the distribution differences between human-written and ChatGPT-
written abstracts. Compared with human-written abstracts, ChatGPT-written
abstracts are more informative, but lack of logic and definition precision.

  3. (3)

We conduct an evaluation of current detection algorithms on the CHEAT dataset.
The experimental results show that existing schemes lack effectiveness in
detecting ChatGPT-written abstracts, and the detection difficulty increases
with human involvement.

##  2 Related Work

In this section, we will introduce the related works of Text synthesis and
Text synthesis detection.

###  2.1 Text Synthesis

ChatGPT, an artificial intelligence chatbot program, is developed based on the
Generative Pre-trained Transformer (GPT) models. The earliest GPT-1Radford et
al. (2018) was proposed in 2018, which could perform various natural language
processing tasks such as semantic inference, question answering, and
classification. Although GPT-1 could effectively handle some unknown tasks,
its generalization ability was still not sufficient compared to fine-tuned
supervised models. After that, Radford _et al._ Radford et al. (2019) proposed
GPT-2 in 2019, which used more network parameters and larger datasets to learn
more universal knowledge and achieved the best performance in multiple
specific language modeling tasks. Later, Brown _et al._ Brown et al. (2020)
proposed GPT-3. They employed self-supervised mechanisms to learn general
knowledge and performed well in various tasks. However, due to directly
learning from large-scale text corpora, GPT-3 was prone to generating
erroneous or offensive content, which limited its application. To solve this
problem, Ouyang _et al._ Ouyang et al. (2022) designed the InstructGPT
language model to improve the quality of synthetic content. They combined
supervised learning and Reinforcement Learning from Human Feedback(RLHF) to
promote the synthetic content better following user intent. The recent ChatGPT
furtherly learned to rank the quality of output results, and improved the
model’s understanding ability successfully. The Instruct Learning and
Reinforcement Learning strategies had driven the success of ChatGPT, enabling
it to synthesize academic contents that can deceive most detectors.

###  2.2 Text Synthesis Detection

Existing text synthesis detection algorithms could be roughly divided into
traditional detection algorithms and deep learning-based detection algorithms.
Traditional detection algorithms analyze abnormal patterns of synthetic text
by extracting hand-crafted features, while deep learning-based algorithms
train detection models on large-scale datasets to achieve accurate detection
of synthetic text. The following provides a detailed introduction to the two
types of approaches.

Traditional text synthesis detection Early text synthesis algorithms left
obvious synthesis traces and abnormal distribution patterns in statistical
features. Sebastian _et al._ Gehrmann et al. (2019) developed a visualization
tool called GLTR. They integrated multiple detection methods based on
statistical features(word order, predicted distribution entropy, and so on.)
to identify anomalies in synthetic text. Fröhling _et al._ Fröhling and
Zubiaga (2021) found that most synthetic texts were repetitive and lacked
purpose, so they extracted statistical features of text style and successfully
identified various synthetic text using Random Forests and Support Vector
Machines. Lundberg _et al._ Levin et al. (2023) generated 50 abstracts using
ChatGPT and then analyzed the differences between human-written abstracts and
ChatGPT-written abstracts using Grammarly. They found that ChatGPT was able to
generate more unique words and had fewer grammatical errors. Recently, Guo _et
al._ Guo et al. (2023) discovered that ChatGPT-written text tends to use more
connecting words, providing new ideas for detecting synthetic text.

Table 1: The searching keywords used for collecting human-written abstracts.

Natural language processing | Feature extraction | Artificial Intelligence | Knowledge Representation and Reasoning | Internet of Things
---|---|---|---|---
Computational modeling | Labeling | Machine Learning | Expert Systems | Cloud Computing
Training | Neural networks | Deep Learning | Fuzzy Logic | Cybersecurity
Supervised learning | Nonlinear systems | Computer Vision | Genetic Algorithms | Data Mining
Brightness | Convergence | Robotics | Swarm Intelligence | Predictive Analytics
Estimation | networks | Reinforcement Learning | Big Data Analytics | Decision Support Systems

Deep learning-based text synthesis detection The rich information stored in
large language models could provide guidance for text synthesis detection.
Solaiman _et al._ Solaiman et al. (2019) fine-tuned the detection model based
on RoBERTa and achieved the best performance in the generated-web detection
task. Tay _et al._ Tay et al. (2020) found that different language models
would leave different defects in the synthesized text. Thus, they trained
corresponding detectors for specific language models to achieve high-accuracy
detection, but performed poorly in cross-model detection. After that, Ippolito
_et al._ Ippolito et al. (2020) constructed a novel text synthesis dataset and
fine-tuned the BERT classification model, significantly improving the
detection accuracy on synthetic texts. Although deep learning algorithms could
detect synthetic text well, their interpretability greatly limits their
application. The research focus of current deep learning based solutions is to
explain the underlying principles behind the detection results.

##  3 The ChatGPT-written Abstract Dataset

Although some datasets for ChatGPT-written content detection are available,
there is no large-scale ChatGPT-written abstract datasets until now. To
provide more relevant data for developing detection methods, we constructed
the ChatGPT-written abstract (CHEAT) dataset.

###  3.1 Basic information

The CHEAT dataset is consisted of 15,395 human-written abstracts and 35,304
ChatGPT-written abstracts. The average length of all abstracts is 163.9 and
the total vocabulary size is 130,272. The human-written abstracts are searched
from IEEE Xplore, a mega repository of scholarly literature. As shown in Table
1, we select 30 keywords to search for matching article abstracts. The
collected abstracts all originate from the field of computer science, and
cover areas such as natural language processing, computer vision, and machine
learning. Among them, abstracts with less than 100 words accounted for 11.6%,
abstracts with 100-200 words accounted for 67.7%, and abstracts with more than
200 words accounted for 20.7%. ChatGPT-written abstracts are then synthesized
through the interface provided by OpenAI.

###  3.2 Abstract Synthesis Methods

The synthetic abstracts in the CHEAT dataset are consisted of $Generation$
abstracts, $Polish$ abstracts, and $Mix$ abstracts. We input the human-written
abstracts into ChatGPT (gpt-3.5-turbo) through the OpenAI interface to obtain
the corresponding output. Specifically, we obtain synthesized abstracts in
three ways:

Generation: ChatGPT is capable of generating plausible abstracts from
keywords. Malicious users could use ChatGPT to generate abstracts directly for
publication, thus obtaining scientific results without any cost. To detect
such synthesis, we create the ChatGPT-Generation dataset to develop detection
algorithms. Specifically, we use ChatGPT to output the generated abstract by
entering the following command: "_Generate a 200-word abstract of the paper in
English based on the title and keywords; your answer only needs to include the
generated paragraph._ ", followed by the title and keywords in the human-
written data.

Polish: Unlike traditional text polishing algorithms, ChatGPT is able to
utilize its rich linguistic knowledge to optimize the original text and
enhance its readability. Malicious users are likely to evade paper checking by
text polish, causing damage to academic originalityKhalil and Er (2023). In
this paper, we create the ChatGPT-Polish dataset to support the relevant
detection algorithm. Specifically, we employ ChatGPT to output a polished
abstract by entering the following command: "_Polish the following paragraphs
in English, your answer just needs to include the polished text._ ", followed
by the human-written abstract.

Mix: Malicious users are likely to mix human-written abstracts with polished
abstracts to evade detection algorithms. To address this problem, we create a
more challenging dataset, ChatGPT-Mix, based on the polished abstracts.
Specifically, we first decompose the polished abstracts and human-written
abstracts according to their semantics, then construct a random mask to
determine which sentences need to be replaced, and finally replace the
polished abstracts with text from the human-written abstracts to obtain the
final mixed abstract. By controlling the number of 1 in the mask, we are able
to effectively control the text replacement rate and synthesize mixed
abstracts with different detection difficulties.

##  4 Evaluation and Analysis

ChatGPT’s ability to synthesize abstracts is potentially harmful to academic
originality and correctness. In this section, we first analyze the linguistic
differences between human-written and ChatGPT-written abstracts, and then
evaluate the detection performance of existing algorithms on the CHEAT
dataset. After that, we explore the judgment basis of the deep learning based
detection methods.

###  4.1 Linguistic Analysis

In this subsection, we evaluate the synthesis quality of ChatGPT-written
abstracts from a linguistic perspective, including lexical analysis and
dependency analysis.

Lexical analysis In the field of natural language processing, each word could
be classified as one of lexical categories. The part-of-speech (POS) tagging
task aims to determine the grammatical class of each word in a given sentence.
In this part, we use the POS module in NLTK Bird (2006) to calculate the
lexical distributions of abstract texts in the CHEAT dataset, and sort them by
lexical percentage. As shown in Figure 2, we show the statistics of the top
ten lexicalities. It can be seen that noun ($NOUN$) occupies the largest
proportion of all lexicalities, while punctuation ($PUNCT$), verb ($VERB$),
adposition ($ADP$), adjective ($ADJ$), and determiner ($DET$) occupy most of
the others. Comparing human-written and ChatGPT-written abstracts, the
following findings could be made:

(1) The proportions of $NOUN$, $VERB$ in ChatGPT-written abstracts are higher
than those of human-written abstracts. It can be argued that the rich
knowledge contained in ChatGPT can provide a more diverse vocabulary for
abstract synthesis and make it more informative.

(2) The proportions of adposition($ADP$), proper noun($PROPN$), and
auxiliary($AUX$) in human-written abstracts are larger than those of ChatGPT-
written abstracts. This indicates that humans tend to pay more attention to
abstract structure, consistency, and logic, while ChatGPT is weaker in these
aspects.

Figure 2: The lexical distribution for human-written, ChatGPT-polished, and ChatGPT-generated abstracts. Figure 3: The dependency distribution forhuman-written, ChatGPT-polished, and ChatGPT-generated abstracts. Table 2: The detection performance of existing schemes. The accuracy and AUC(Area Under the ROC Curve) are used as metrics. Methods | Datasets
---|---
Generation | Polish | Mix
ACC | AUC | ACC | AUC | ACC | AUC
Grover Zellers et al. (2019) | 54.24 | 56.34 | 53.33 | 55.45 | 50.89 | 51.71
ZerogptZeroGPT (2023) | 67.32 | 78.8 | 52.71 | 57.35 | 50.61 | 52.59
OpenAI-detector Solaiman et al. (2019) | 75.97 | 84.41 | 54.07 | 56.17 | 52.18 | 55.23
ChatGPT-detector-roberta Guo et al. (2023) | 75.54 | 81.91 | 53.65 | 47.28 | 51.92 | 63.71
Chatgpt-qa-detector-roberta Guo et al. (2023) | 85.56 | 97.6 | 53.53 | 64.39 | 51.67 | 65.28

Dependency analysis Dependency grammar is an important tool for natural
language understanding. It is able to point out the syntactic collocation
relations between words. In this paper, we calculate the dependency properties
between individual words and attempt to analyze the differences between human-
written and ChatGPT-written abstracts. We compute statistical histograms of
dependencies in each type of abstracts, and then rank them according to the
percentage. As shown in Figure 3, we present the statistics for the top ten
dependencies. There is some similarity in the distribution of the human-
written abstracts and ChatGPT-written abstracts, but there are also some
differences. Specifically, we have the following findings:

(1) The Adjectival modifier ($amod$), Prepositional modifier ($prep$), and
Compound modifier ($compound$) are more used in human-written abstracts.
Compared with ChatGPT-written abstracts, human-written abstracts tend to apply
more modifiers to define the words precisely.

(2) The proportions of the Direct Object ($dobj$), Nominal subject ($nsubj$),
and Root words($ROOT$) in the human-written abstracts are smaller than those
of ChatGPT-written abstracts. This is similar with the distributions in the
lexical analysis. ChatGPT could provide more the rich vocabulary for
synthesizing abstracts.

Figure 4: ROC curves of existing detection schemes on three datasets(ChatGPT-Generation, ChatGPT-Polish, and ChatGPT-Mix). Table 3: The detection performance of different models. The models are trained on the CHEAT training dataset and then evaluated on the corresponding test dataset. Models |  Datasets
---|---
Polish |  Generation |  Mix
|  Distilbert Sanh et al. (2019)
---
99.43 |  100 |  85.07
|  BERT Kenton et al. (2019)
---
99.48 |  100 |  86.62
|  Roberta Liu et al. (2019)
---
99.72 |  100 |  52.93
|  BERT-multilingual Pires et al. (2019)
---
99.49 |  100 |  60.16
|  PubMedBERT Gu et al. (2021)
---
99.56 |  100 |  87.83

###  4.2 Text Synthesis Detection Evaluation

The distribution differences between ChatGPT-written abstracts and human-
written abstracts provide the feasibility of their detection. This subsection
evaluate the detection performance of existing algorithms on ChatGPT-written
abstracts.

####  4.2.1 Compared Text Synthesis Detection Methods

We consider five text synthesis detection methods in our experiment. Due to
the need to evaluate each method on the CHEAT dataset, we only capture those
schemes that have code or the model parameters publicly available.

  * •

Grover Detector Grover is a controlled text generation model proposed by Rowan
Zellers et al.Zellers et al. (2019). The Grover detector is trained based on
the Grover generation model to determine whether the text is generated by the
neural network model.

  * •

ZeroGPT ZeroGPT detectorZeroGPT (2023) is an online detector, which is trained
based on 10 million articles and texts. It is capable of performing machine-
generated text detection in multiple languages with high accuracy.

  * •

OpenAI-detector The detectorSolaiman et al. (2019) is officially provided by
OpenAI. It employs the original text and the GPT-2 generated text to fine-tune
the RoBERTa model to determine whether the text is machine generated or not.

  * •

ChatGPT-detector-roberta Guo _et al._ Guo et al. (2023) constructed the HC3
(Human ChatGPT Comparison Corpus) dataset consisting of nearly 40K questions
and their corresponding human/ChatGPT answers, and then used these data to
fine-tune the RoBERTa model to obtain an accurate synthetic text detector.

  * •

ChatGPT-qa-detector-roberta The correlation between the question and answer is
also used to detect the synthetic text. Guo _et al._ Guo et al. (2023) trained
a detection model using Q&A statements in HC3 dataset, and achieved high
accuracy in synthetic text detection.

####  4.2.2 Detection Performance

In this part, we evaluate the performance of existing detection algorithms on
our CHEAT dataset. The detection accuracy (ACC) and area under the ROC curve
(AUC) are applied to evaluate the effectiveness of the detection algorithms.
We first evaluate the detection performance when the algorithm is not trained
with our CHEAT dataset. As shown in Table 2, due to the differences of
training corpus, existing algorithms have difficulty in detecting ChatGPT-
written abstracts. The ROC curve shown in Figure 4 also demonstrates the
weakness of existing detection schemes. The Chatgpt-qa-detector-roberta could
obtain high detection accuracy on the ChatGPT-Generation dataset, but fails on
the ChatGPT-Polish and ChatGPT-Mix dataset. Existing schemes are still
inefficient in detecting ChatGPT-written abstracts.

After that, we trained multiple models using ChatGPT-Generation, ChatGPT-
Polish, and ChatGPT-Mix, respectively. The detection performance of trained
models is then evaluated on the corresponding test datasets. The AUC scores
are presented in Table 3. Due to the specialized training on the CHEAT
dataset, these models obtain better detection performance on the ChatGPT-
written abstracts. In particular, the PubMedBERT, associated with abstract
content detection, obtains the best detection performance on our CHEAT
dataset. From the results, we believe that the fully generated abstracts are
highly feasible to detect. However, similar with the evaluation of existing
schemes, the detection difficulty increases with the human involvement. These
models could obtain auc scores of 100 on the ChatGPT-Generation dataset while
get lower auc score on the ChatGPT-Polish dataset, which are influenced by
human writing. Considering the most difficult case, these models obtain the
lowest auc score when the human-written abstracts are mixed with ChatGPT-
polished abstracts. Detecting ChatGPT-written content with human involvement
is still a challenge for existing detection algorithms.

Figure 5: Visualization of SHAP value statistics. The top 10 words ranked by
contribution are listed for human-written and ChatGPT-written abstracts.
Figure 6: Visualization of SHAP value for single abstract. The sentence
semantics and the top 10 words ranked by contribution are calculated for the
selected abstract.

####  4.2.3 Explainability Analysis

The well-trained detection model could obtain high accuracy, but have
difficulty interpreting the output. It is essential for us to understand
whether detection models are perceiving particular word patterns to make
judgments. In this subsection, we use SHAP (SHapley Additive exPlanations)
Lundberg and Lee (2017) to explain the judgments made by detection models.
Specifically, SHAP uses the classical Shapley values from game theory to link
optimal credit allocation with local explanations. It is able to assign
feature importance values to each input word of the detection model. To obtain
the model interpretion, we first train a high-accuracy detection model based
on the PubMedBERT. Then, we compute the SHAP values between the detection
results and the input abstracts. We count the judgment bases for each kind of
abstracts, and present the top ten terms of judgment bases in Figure 5. When
we attempt to discriminate human-written and ChatGPT-generated abstracts, it
can be found that ChatGPT tend to use more special-patterns, such as ’zi’,
’iv’, ’tunv’, and ’gran’. When the human involvement increases, precise words
gradually play a more important role in detection tasks. We also visualized
the SHAP values for two abstracts in Figure 6. The positive SHAP value
indicates a positive impact on classifying the text as ChatGPT-written
abstract, and vice versa. It can be seen that detectors could correlate
certain words, such as "_it holds great potential_ ", "_mapping of mobile
robots_ ", and "_and security areas._ ", to detect abnormal distributions of
ChatGPT-written abstracts. When human involvement increases, words biased
towards human-written abstracts would appear, increasing the difficulty of
text synthesis detection.

##  5 Discussion & Conclusion

Although the current ChatGPT demonstrates stunning synthesis ability, we
demonstrate that the ChatGPT-written abstracts could be detected by a well-
trained detector. To train the detection model using domain-specific
knowledge, we construct a large-scale CHatGPT-writtEn AbsTract (CHEAT)
dataset, which exceeds the size of all existing publicly available synthetic
abstract datasets.

In this paper, we evaluate the performance of existing detection algorithms on
the CHEAT dataset. The experimental results in this paper show that _the
abstracts generated entirely by ChatGPT are detectable, while the synthetic
abstracts with human guidance possess detection difficulty, especially when
mixed with human-written text._ We need not be too alarmed by the abstracts
fully generated by ChatGPT. Rather, we must direct our attention towards
ChatGPT-written abstracts with human guidance. We aspire for this dataset to
serve as a stepping stone for ChatGPT-written content detection research,
particularly in the realm of paper abstracts.

## References

  * Bird (2006) Steven Bird. 2006.  Nltk: the natural language toolkit.  In _Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions_ , pages 69–72.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Cao et al. (2023) Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023.  Assessing cross-cultural alignment between chatgpt and human societies: An empirical study.  _arXiv preprint arXiv:2303.17466_.
  * Else (2023) Holly Else. 2023.  Abstracts written by chatgpt fool scientists.  _Nature_ , 613(7944):423–423.
  * Fröhling and Zubiaga (2021) Leon Fröhling and Arkaitz Zubiaga. 2021.  Feature-based detection of automated language models: tackling gpt-2, gpt-3 and grover.  _PeerJ Computer Science_ , 7:e443.
  * Gao et al. (2022) Catherine A Gao, Frederick M Howard, Nikolay S Markov, Emma C Dyer, Siddhi Ramesh, Yuan Luo, and Alexander T Pearson. 2022.  Comparing scientific abstracts generated by chatgpt to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers.  _bioRxiv_ , pages 2022–12.
  * Gehrmann et al. (2019) Sebastian Gehrmann, SEAS Harvard, Hendrik Strobelt, and Alexander M Rush. 2019.  Gltr: Statistical detection and visualization of generated text.  _ACL 2019_ , page 111.
  * Gu et al. (2021) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021.  Domain-specific language model pretraining for biomedical natural language processing.  _ACM Transactions on Computing for Healthcare (HEALTH)_ , 3(1):1–23.
  * Guo et al. (2023) Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023.  How close is chatgpt to human experts? comparison corpus, evaluation, and detection.  _arXiv preprint arXiv:2301.07597_.
  * Huang et al. (2023) Fan Huang, Haewoon Kwak, and Jisun An. 2023.  Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech.  _arXiv preprint arXiv:2302.07736_.
  * Ippolito et al. (2020) Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2020.  Automatic detection of generated text is easiest when humans are fooled.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1808–1822.
  * Kenton et al. (2019) Jacob Devlin Kenton, Chang Ming-Wei, and Lee Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL-HLT_ , pages 4171–4186.
  * Khalil and Er (2023) Mohammad Khalil and Erkan Er. 2023.  Will chatgpt get you caught? rethinking of plagiarism detection.  _arXiv preprint arXiv:2302.04335_.
  * Levin et al. (2023) Gabriel Levin, Raanan Meyer, Eva Kadoch, and Yoav Brezinov. 2023.  Identifying chatgpt-written obgyn abstracts using a simple tool.  _American Journal of Obstetrics & Gynecology MFM_, 5(6).
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Lundberg and Lee (2017) Scott M Lundberg and Su-In Lee. 2017.  A unified approach to interpreting model predictions.  _Advances in neural information processing systems_ , 30.
  * Macdonald et al. (2023) Calum Macdonald, Davies Adeloye, Aziz Sheikh, and Igor Rudan. 2023.  Can chatgpt draft a research article? an example of population-level vaccine effectiveness analysis.  _Journal of Global Health_ , 13:01003.
  * Mitrović et al. (2023) Sandra Mitrović, Davide Andreoletti, and Omran Ayoub. 2023.  Chatgpt or human? detect and explain. explaining decisions of machine learning model for detecting short chatgpt-generated text.  _arXiv preprint arXiv:2301.13852_.
  * Mok (2023) Kimberley Mok. 2023.  Chatgpt writes scientific abstracts that can fool experts.  https://thenewstack.io/chatgpt-writes-scientific-abstracts-well-enough-to-fool-experts/.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019.  How multilingual is multilingual bert?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4996–5001.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018.  Improving language understanding by generative pre-training.  _OpenAI Tech Report_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Solaiman et al. (2019) Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. 2019\.  Release strategies and the social impacts of language models.  _OpenAI Tech Report_.
  * Tay et al. (2020) Yi Tay, Dara Bahri, Che Zheng, Clifford Brunk, Donald Metzler, and Andrew Tomkins. 2020.  Reverse engineering configurations of neural text generation models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 275–279.
  * Thorp (2023) H. Holden Thorp. 2023.  Chatgpt is fun, but not an author.  _Science_ , 379(6630):313–313.
  * Ufuk (2023) Furkan Ufuk. 2023.  The role and limitations of large language models such as chatgpt in clinical settings and medical journalism.  _Radiology_ , page 230276.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019.  Defending against neural fake news.  In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ , pages 9054–9065.
  * ZeroGPT (2023) ZeroGPT. 2023.  Ai text detector.  https://www.zerogpt.com.