# Contrasting Linguistic Patterns in Human and LLM-Generated Text

Alberto Muñoz-Ortiz, Carlos Gómez-Rodríguez and David Vilares
Universidade da Coruña, CITIC
Departamento de Ciencias de la Computación y Tecnologías de la Información
Campus de Elviña s/n, 15071
A Coruña, Spain
{alberto.munoz.ortiz, carlos.gomez<EMAIL_ADDRESS>

###### Abstract

We conduct a quantitative analysis contrasting human-written English news text
with comparable large language model (LLM) output from 4 LLMs from the LLaMa
family. Our analysis spans several measurable linguistic dimensions, including
morphological, syntactic, psychometric and sociolinguistic aspects. The
results reveal various measurable differences between human and AI-generated
texts. Among others, human texts exhibit more scattered sentence length
distributions, a distinct use of dependency and constituent types, shorter
constituents, and more aggressive emotions (fear, disgust) than LLM-generated
texts. LLM outputs use more numbers, symbols and auxiliaries (suggesting
objective language) than human texts, as well as more pronouns. The sexist
bias prevalent in human text is also expressed by LLMs.

##  1 Introduction

Natural language generation (NLG) models Radford et al. ; Scao et al. (2022);
Touvron et al. (2023) and instruction-tuned variants OpenAI (2023); Taori et
al. (2023) output fluent, human-like text in many languages, English being the
best represented. The extent to which these models truly understand semantics
Landgrebe and Smith (2021); Søgaard (2022), encode representations of the
world Li et al. (2022), generate fake statements Kumar et al. (2023), or
propagate specific moral and ethical values Santurkar et al. (2023) is
currently under active debate. Regardless, a crucial factor contributing to
the persuasiveness of these models lies, in the very first place, in their
exceptional linguistic fluency.

Figure 1: We gather contemporary articles from the New York Times API and use
their headlines as a prompt to produce LLaMa-generated news. We then compare
both types of texts, measuring differences across aspects such as vocabulary,
morphosyntactic structures, and semantic attributes.

A question that arises regards whether their storytelling strategies align
with the linguistic patterns observed in human-generated texts. Do these
models tend to use more flowery or redundant language? Do they exhibit
preferences for specific voices or syntactic structures in sentence
generation? Are they prone to certain psychometric dimensions? However,
contrasting such linguistic patterns is not trivial. Firstly, the creators of
these models often insufficiently document the training data used. Even with
available information, determining the extent of the training set’s influence
on a sentence or whether it’s similar to an input sample remains challenging.
Second, different domains exhibit distinct linguistic complexities, e.g.,
children textbooks differ from technical articles, and war articles may convey
more anger compared to hotel reviews. Third, language is subject to societal
influences, cultural norms, social factors, and geographic variations, which
shape linguistic preferences and conventions. Thus, to contrast linguistic
patterns between humans and machines, it is advisable to rely on a controlled
environment. Yet, few efforts have been dedicated to measure differences, if
any, in syntax, grammar, and other linguistic particularities between both
types of texts.

##### Contribution

We study text generation models from the LLaMa Touvron et al. (2023) family,
and contrast several linguistic patterns against human text using English news
text. To do so, we recover human-generated news and ask LLaMa models to
generate news based on the headline and first words of the news. We query the
New York Times API, retrieving news as they are published, to guarantee
sterilization from the training set. We analyze various linguistic patterns
such as: differences in the distribution of the vocabulary, sentence length,
part-of-speech (PoS) tags, differences in the distribution of syntactic
structures, psychometric features such as the tone of the news articles and
emotions detectable in the text, and sociolinguistic aspects like gender bias.
We depict the overview in Figure 1. We also explore if these disparities
change across models of different sizes.

##  2 Data preparation

Next, we will delve into our data collection process for both human-generated
and machine-generated content.

###  2.1 Data

We generate the evaluation dataset relying on news published strictly after
the release date of the models that we will use in this work. This strategy
ensures that they do not have exposure to the news headlines and their content
during pre-training. It is also in line with strategies proposed by other
authors - such as Liu et al. (2023) \- who take an equivalent angle to
evaluate LLMs in the context of generative search engines. The reference
human-generated texts will be the news (lead paragraph) themselves.

##### Crawling

We use New York Times news, which are accessible through its
API111https://developer.nytimes.com/apis that we access through the pynytimes
Python library Den Heijer (2022).. We retrieve the data based on their release
dates, processing them in batches of 300 to avoid exceeding the API limit. The
articles are retrieved in JSON format, and include relevant metadata such as
the URL, section name, type of material, keywords, or publication date. Table
1 shows some information about the articles retrieved. Our main interest
focuses on two specific fields: the headline and the lead paragraph. The lead
paragraph serves as a concise summary of the primary information presented in
the article. We discarded those articles that had an empty lead paragraph.
Particularly, we collected 11 133 articles from April 20 to July 13, 2023. The
collected articles primarily consist of news pieces, although around 30% also
include other types of texts, such as movie or book reviews.

Section | Category (%)
---|---
Section name | U.S. (15.81), World (11.85),
Opinion (8.04), Arts (7.64),
Business Day (6.92), New York (4.91),
Sports (4.87), Books (3.71), Style (3.27),
Movies (3.02), Crosswords & Games (2.58),
Briefing (2.58), Food (1.96), Real State (1.84),
Podcasts (1.61), Theater (1.58)
Type of material | News (70.62), Op-Ed (6.60), Review (5.18),
Other (4.06), Interactive Feature (3.23),
Briefing (2.86), Video (2.31), Obituary (2.30),
Recipe (1.55), Letter (1.28)
Document type | Article (90.31), Multimedia (6.95),
Recipe (1.55), Other (1.19)
Table 1: Information about the articles used for this work. Each category
shows the most frequent topics.

###  2.2 Generation

Let $\mathcal{H}$ = $[h_{1},h_{2},...,h_{N}]$ be a set of human-generated
texts, such that $h_{i}$ is a tuple of the form $(t_{i},s_{i})$ where $t_{i}$
is a headline and $s_{i}$ is a paragraph of text with a summary of the
corresponding news. Similarly, we will define $\mathcal{M}$ =
$[m_{1},m_{2},...,m_{N}]$ as the set machine-generated news articles produced
by a LLM such that $m_{i}$ is also a tuple of the from
($t^{\prime}_{i},s^{\prime}_{i})$ where $t^{\prime}_{i}=t_{i}$ and
$s^{\prime}_{i}=[w^{\prime}_{1},w^{\prime}_{2},...,w^{\prime}_{|s_{i}|}]$ is a
piece of synthetic text. For the generation of high-quality text, language
models aim to maximize the probability of the next word based on the previous
content. To ensure that the models keep on track with the domain and topic, we
initialize the previous content with the headline (the one chosen by the
journalist that released the news) and the first three words of the human-
generated lead paragraph to help the model start and follow the topic.222In
preliminary experiments, certain LLM outputs encountered difficulties in
adhering to a minimal coherent structure when a minimum number of the body’s
words were absent from the prompt. Formally, we first condition the model on
$c=t^{\prime}_{i}\cdot s_{i[0:2]}$ and every next word ($i\geq 3$) will be
predicted from a conditional distribution $P(w^{\prime}_{i}|c_{i}\cdot
s^{\prime}_{i[3:t-1]})$.

##### LLaMa models Touvron et al. (2023)

To approximate such distribution, we use pre-trained generative language
models. We rely on all models from the LLaMA family: 7B, 13B, 30B and 65B. The
range of versions enables us to study the impact of model size in our
framework. Also, we focus on this family because it has been widely adopted by
the community, the pre-training data is described, and the models are released
under an open-source license. To generate a piece of synthetic text
$s^{\prime}$, we condition LLaMa with a prompt that includes the headline and
first words, as described above, and we will keep generating the text news
until the model decides to stop.333In preliminary experiments, we explored
hyperparameter values that generated the more fluent and coherent texts:
temperature of 0.7, 0.9 top p tokens, and a repetition penalty of 1.1. We
enable the model to output text without any forced criteria, except for not
exceeding 200 tokens. The length limit serves two main purposes: (i) to manage
computational resources efficiently444We ran the models on 2xA100 GPUs for 10
days to generate all texts. The inference speed varies from 5.80 tokens per
second for the 7B model to 2.15 tokens per second for the 65B model. To
address memory costs, we use 8-bit precision., and (ii) to ensure that the
generated content resembles the typical length of human-written lead
paragraphs, making it comparable to human-produced content. We arrived at this
limit after comparing the average and standard deviation of number of tokens
between humans and models in early experiments.

##  3 Analysis of linguistic patterns

Next, we compare human- and machine-generated texts. We first analyze the
texts under a morphosyntactic viewpoint and then focus on psychometric and
sociolinguistic aspects, in particular emotions, text similarities, and gender
biases.

###  3.1 Morphosyntactic Analysis

To compute the linguistic representations, we rely on Stanza Qi et al. (2020)
to do segmentation, tokenization, part-of-speech (PoS) tagging, and dependency
and constituent parsing. For these tasks, and in particular for the case of
English and news text, the performance is high-enough to be used for
applications Manning (2011); Berzak et al. (2016).

####  3.1.1 Sentence length

Figure 2 illustrates the length distribution for the LLaMA models in
comparison to human-generated news articles. We excluded a few outliers from
the plot by ignoring sentences with lengths over 80 tokens. The four LLMs
exhibit a similar distribution across different sentence lengths, presenting
less variation when compared to human-generated sentences, which display a
wider range of lengths and greater diversity. Specifically, the LLaMa models
exhibit a higher frequency of sentence generation within the 10 to 30 token
range compared to humans, whereas the latter tend to produce longer sentences
with greater frequency.

Figure 2: Sentence length distribution for the human- and LLaMa-generated
texts.

####  3.1.2 Richness of vocabulary and lexical variation

We analyze the diversity of vocabulary used by the LLMs, categorized it down
based on their sizes, and compared them against human texts. To assess
vocabulary diversity, we considered the total number of tokens, the count of
unique tokens, and the Type-Token Ratio (TTR). The TTR is a measure of lexical
variation and is calculated by dividing the number of unique tokens (types) by
the total number of tokens (types + tokens). Table 2 presents the results.
Regarding the count of unique tokens, we found that human texts exhibited a
slight increase in richness compared to the 65B LLaMa model. However, a more
substantial difference - approximately 2 000 extra used tokens - was evident
when comparing human texts with the 7B, 13B, and 30B versions of LLaMa.
Regarding TTR, humans exhibited a larger value than the two largest LLaMa
models. LLaMa smallest versions showed the poorest lexical variation across
the board. While the discrepancy was not drastic, it suggests that the larger
models possess a certain ability to achieve human-like level in this respect.
Yet, human journalists still display a slightly richer vocabulary.

Model | Tokens | Unique | Type-token ratio
---|---|---|---
Human | 577 756 | 39 947 | 0.067
7B | 671 956 | 36 429 | 0.054
13B | 617 426 | 35 900 | 0.058
30B | 598 762 | 36 665 | 0.061
65B | 655 205 | 39 927 | 0.059
Table 2: Statistics related to the vocabulary of the articles generated by
humans and each of the LLaMa models.

####  3.1.3 Part-of-speech tag distributions

Table 3 presents the frequency of universal part-of-speech (UPOS) tags Petrov
et al. (2012) for both human and LLaMa texts. Figure 3 shows relative
differences observed across humans and each LLaMa model, for a better
understanding of the overuse or underuse of certain grammatical categories.
Humans exhibit a systematic preference for using certain kinds of content
words, such as nouns and adjectives. Interestingly, humans also make more
frequent use of words tagged as punctuation symbols. This observation may be
connected to our analysis on sentence length, as human users tend to rely on
longer sentences, thereby requiring increased usage of punctuation.
Alternatively, LLaMa models exhibit a pronounced inclination towards relying
on categories such as symbols or numbers, possibly indicating an excessive
attempt by language models to furnish specific data in order to sound
convincing. Moreover, they exhibited a higher frequency of pronoun usage;
however, a comprehensive analysis of this aspect will be undertaken later from
a sociolinguistic perspective, specifically analyzing gender biases.

UPOS | H | 7B | 13B | 30B | 65B | UPOS | H | 7B | 13B | 30B | 65B
---|---|---|---|---|---|---|---|---|---|---|---
NOUN | 19.20 | 17.96 | 17.82 | 17.98 | 18.07 | ADV | 3.45 | 3.66 | 3.70 | 3.44 | 3.56
PUNCT | 12.03 | 11.29 | 11.30 | 11.94 | 11.71 | CCONJ | 2.60 | 2.69 | 2.68 | 2.58 | 2.60
ADP | 10.83 | 10.38 | 10.26 | 10.44 | 10.45 | PART | 2.38 | 2.46 | 2.44 | 2.30 | 2.35
VERB | 9.76 | 9.69 | 9.64 | 9.46 | 9.70 | NUM | 2.31 | 3.21 | 3.20 | 3.28 | 3.09
PROPN | 9.40 | 8.88 | 9.00 | 9.51 | 9.39 | SCONJ | 1.44 | 1.68 | 1.68 | 1.48 | 1.60
DET | 8.81 | 8.47 | 8.48 | 8.59 | 8.46 | SYM | 0.20 | 0.33 | 0.31 | 0.29 | 0.29
ADJ | 7.37 | 6.80 | 6.76 | 6.72 | 6.69 | X | 0.09 | 0.07 | 0.07 | 0.10 | 0.07
PRON | 5.74 | 6.89 | 7.05 | 6.58 | 6.67 | INTJ | 0.08 | 0.07 | 0.08 | 0.08 | 0.08
AUX | 4.31 | 5.48 | 5.53 | 5.23 | 5.23 |  |  |  |  |  |
Table 3: UPOS frequencies (%) in human- and LLaMa-generated texts. Figure 3:
Percentage differences, following Table 3, in the use of each UPOS category
between each LLaMa model in comparison to humans.

####  3.1.4 Dependency arc lengths

Table 4 shows various information with respect to the syntactic dependency
arcs that arise in human and machine-generated texts. The results indicate
that their length and distribution are nearly identical for the four LLMs and
the human texts. This finding holds true despite the variation in sentence
length distribution between human-generated and LLM-generated text.

$l$ | Model | %L | %R | $\bar{l}$ | $\bar{l}_{L}$ | $\bar{l}_{R}$ | $\sigma_{l}$ | $\sigma_{l_{L}}$ | $\sigma_{l_{R}}$ | # Sent
---|---|---|---|---|---|---|---|---|---|---
1-10 | Human | 48.42 | 51.58 | 2.36 | 2.84 | 1.84 | 1.67 | 1.90 | 1.19 | 4 396
7B | 49.66 | 50.34 | 2.37 | 2.91 | 1.82 | 1.67 | 1.90 | 1.19 | 4 885
13B | 49.26 | 50.74 | 2.37 | 2.90 | 1.83 | 1.70 | 1.92 | 1.21 | 5 123
30B | 49.40 | 50.60 | 2.37 | 2.90 | 1.83 | 1.68 | 1.89 | 1.20 | 5 302
65B | 49.51 | 50.49 | 2.37 | 2.90 | 1.82 | 1.67 | 1.89 | 1.19 | 5 205
11-20 | Human | 59.47 | 40.53 | 3.17 | 4.67 | 2.16 | 3.08 | 3.83 | 1.84 | 6 736
7B | 60.05 | 39.95 | 3.14 | 4.66 | 2.13 | 3.05 | 3.82 | 1.78 | 9 763
13B | 60.19 | 39.81 | 3.15 | 4.67 | 2.15 | 3.05 | 3.82 | 1.82 | 9 772
30B | 59.90 | 40.10 | 3.14 | 4.65 | 2.13 | 3.04 | 3.81 | 1.80 | 9 624
65B | 59.81 | 40.19 | 3.14 | 4.64 | 2.13 | 3.05 | 3.81 | 1.80 | 9 666
21-30 | Human | 60.84 | 39.16 | 3.60 | 5.53 | 2.36 | 4.32 | 5.63 | 2.52 | 6 297
7B | 61.28 | 38.72 | 3.53 | 5.50 | 2.29 | 4.26 | 5.64 | 2.36 | 8 722
13B | 61.59 | 38.41 | 3.56 | 5.54 | 2.32 | 4.28 | 5.66 | 2.42 | 8 191
30B | 61.18 | 38.82 | 3.56 | 5.50 | 2.33 | 4.27 | 5.58 | 2.47 | 7 913
65B | 61.08 | 38.92 | 3.54 | 5.49 | 2.30 | 4.27 | 5.63 | 2.40 | 8 436
31-40 | Human | 61.19 | 38.81 | 3.87 | 6.05 | 2.50 | 5.41 | 7.20 | 3.16 | 3 955
7B | 61.42 | 38.58 | 3.81 | 5.99 | 2.44 | 5.34 | 7.17 | 3.05 | 4 496
13B | 61.46 | 38.54 | 3.83 | 6.03 | 2.44 | 5.31 | 7.12 | 3.04 | 3 925
30B | 61.24 | 38.76 | 3.83 | 5.95 | 2.49 | 5.30 | 7.06 | 3.12 | 3 879
65B | 60.92 | 39.08 | 3.82 | 5.96 | 2.45 | 5.31 | 7.06 | 3.07 | 4 411
+41 | Human | 60.66 | 39.34 | 4.03 | 6.33 | 2.54 | 6.24 | 8.39 | 3.58 | 1 987
7B | 60.63 | 39.37 | 3.99 | 6.24 | 2.52 | 6.26 | 8.41 | 3.62 | 1 602
13B | 60.72 | 39.28 | 4.01 | 6.25 | 2.56 | 6.25 | 8.36 | 3.72 | 1 361
30B | 60.56 | 39.44 | 4.03 | 6.29 | 2.56 | 6.20 | 8.30 | 3.62 | 1 276
65B | 60.73 | 39.27 | 4.02 | 6.34 | 2.52 | 6.21 | 8.37 | 3.52 | 1 585
Table 4: Statistics for dependency arcs in sentences of different lengths for
the texts generated by human writers and the LLaMa models. The meaning of the
columns is as follows: (%L) percentage of left arcs, (%R) percentage of right
arcs, ($\bar{l}$) average arc length, ($\bar{l}_{L}$) average left arc length,
($\bar{l}_{R}$) average right arc length, ($\sigma_{l}$) standard deviation of
arc length, ($\sigma_{l_{L}}$) standard deviation of left arc length,
($\sigma_{l_{R}}$) standard deviation of right arc length, and number of
sentences.

####  3.1.5 Dependency types

Table 5 lists the frequencies for the main syntactic dependency types in human
and machine-generated texts. Figure 4 depicts relative differences in their
use (humans versus each LLaMa model). For space reasons, we only comment a few
relevant cases. For instance, nummod dependencies are much more common in
LLaMa-generated texts compared to human texts. This is coherent with the
higher use of the NUM tag in the part-of-speech tag distribution analysis.
Additionally, we observed higher ratios for other dependency types, such as
aux (for which the use of auxiliary verbs was also significantly higher
according to the UPOS analysis) and copula. Furthermore, syntactic structures
from LLaMa models exhibit significantly fewer subtrees involving adjective
modifiers (amod dependency type).

deprel | H | 7B | 13B | 30B | 65B | deprel | H | 7B | 13B | 30B | 65B
---|---|---|---|---|---|---|---|---|---|---|---
punct | 11.31 | 11.94 | 11.71 | 11.30 | 12.03 | mark | 2.97 | 2.69 | 2.86 | 3.00 | 2.63
case | 10.48 | 10.69 | 10.68 | 10.61 | 11.12 | cc | 2.71 | 2.61 | 2.63 | 2.73 | 2.61
det | 8.29 | 8.41 | 8.27 | 8.28 | 8.64 | nmod:poss | 2.17 | 2.10 | 2.11 | 2.13 | 2.22
amod | 5.65 | 5.74 | 5.71 | 5.73 | 6.61 | aux | 2.50 | 2.35 | 2.38 | 2.50 | 2.04
nsubj | 6.99 | 6.76 | 6.85 | 6.85 | 6.28 | flat | 1.84 | 2.01 | 1.95 | 1.80 | 2.00
obl | 5.15 | 5.19 | 5.21 | 5.17 | 5.29 | advcl | 1.65 | 1.52 | 1.64 | 1.61 | 1.72
compound | 4.55 | 4.74 | 4.72 | 4.59 | 4.99 | cop | 2.03 | 1.90 | 1.85 | 1.95 | 1.49
nmod | 4.28 | 4.43 | 4.42 | 4.38 | 4.63 | nummod | 1.96 | 2.03 | 1.90 | 1.92 | 1.46
root | 4.67 | 4.74 | 4.56 | 4.48 | 4.21 | appos | 1.02 | 1.15 | 1.12 | 1.00 | 1.22
obj | 4.01 | 3.92 | 4.01 | 3.98 | 4.17 | acl:relcl | 1.24 | 1.22 | 1.27 | 1.31 | 1.20
advmod | 3.94 | 3.65 | 3.78 | 3.88 | 3.67 | xcomp | 1.30 | 1.18 | 1.23 | 1.30 | 1.13
conj | 2.82 | 2.72 | 2.76 | 2.91 | 3.00 |  |  |  |  |  |
Table 5: Percentage of words generated by humans and LLaMa models that are
labeled with a specific dependency type (deprel). For space reasons, we only
include relations with a frequency surpassing 1% within the human texts.
Figure 4:  Percentage differences, following Table 5, in the use of dependency
relations between each LLaMa model in comparison to humans.

####  3.1.6 Optimality of dependencies

We compare the degree of optimality of syntactic dependencies between human
texts and LLMs. It has been observed in human language that dependencies tend
to be much shorter than expected by chance, a phenomenon known as dependency
length minimization (Ferrer-i-Cancho, 2004; Futrell et al., 2015). This can be
quantified in a robust way (with respect to sentence length, tree topology and
other factors) by the $\Omega$ optimality score introduced in Ferrer-i Cancho
et al. (2022). This score measures where observed dependency lengths sit with
respect to random word orders and optimal word orders, and is defined as:
$\Omega=\frac{D_{rla}-D}{D_{rla}-D_{min}}$, where $D$ is the sum of dependency
lengths in the sentence, $D_{rla}$ is the expected sum of lengths, and
$D_{min}$ is the optimal sum of lengths for the sentence’s tree structure.
Thus, for optimally-arranged trees $D=D_{min}$ and $\Omega$ takes a value of
1, whereas for a random arrangement it has an expected value of 0. Negative
values are possible (albeit uncommon in attested language) if dependency
lengths are larger than expected by chance.

Figure 5 displays the distribution of $\Omega$ values across sentences for
both human and LLM-generated texts. The results indicate that $\Omega$ values
remain remarkably close between human sentences and those generated by the
LLMs, exhibiting almost no discernible difference.

Figure 5: $\Omega$ value distribution for the human- and LLM-generated texts.

####  3.1.7 Constituent lengths

Table 6 shows the comparison between the distribution of constituent lengths
across both types of texts. While human-generated sentences, on average,
surpass the length of those generated by LLMs, the average length of a
sentence constituent for LLMs is observed to be greater than that for humans.
The standard deviation of constituent lengths exhibits similar values across
all models for each sentence length range.

| Model | 1-10 | 11-20 | 21-30 | 31-40 | +41
---|---|---|---|---|---|---
$\bar{l}$ | Human | 4.30 | 6.38 | 7.98 | 9.37 | 10.51
7B | 4.36 | 6.47 | 8.23 | 9.72 | 11.05
13B | 4.36 | 6.42 | 8.13 | 9.59 | 10.77
30B | 4.37 | 6.42 | 8.08 | 9.51 | 10.67
65B | 4.33 | 6.42 | 8.16 | 9.50 | 10.78
$\sigma_{l}$ | Human | 2.33 | 4.65 | 6.92 | 9.08 | 11.10
7B | 2.34 | 4.68 | 6.95 | 9.12 | 11.25
13B | 2.36 | 4.65 | 6.90 | 9.00 | 11.01
30B | 2.34 | 4.65 | 6.88 | 8.99 | 11.00
65B | 2.35 | 4.66 | 6.94 | 9.03 | 11.10
# Sent | Human | 4 397 | 6 736 | 6 297 | 3 955 | 1 987
7B | 4 885 | 9 763 | 8 722 | 4 496 | 1 602
13B | 5 123 | 9 772 | 8 191 | 3 925 | 1 361
30B | 5 302 | 9 624 | 7 913 | 3 879 | 1 276
65B | 5 205 | 9 666 | 8 436 | 4 411 | 1 585
Table 6: Statistics for constituents that arise in sentences of different
lengths for the text generated by human writers and the LLaMa models. The
meaning of the rows are as follows: ($\bar{l}$) average constituent length,
($\sigma_{l}$) standard deviation of constituent length, and number of
sentences.

####  3.1.8 Constituent types

Table 7 and Figure 6 examine the disparities in constituent types between
human- and LLaMa-generated texts. Our focus was on constituent types
constituting over 1% of the total occurrences. Some outcomes are in the same
line of earlier findings: human-generated content displays heightened use of
noun, adjective, and prepositional phrases (NP, ADJP, and PP, respectively).
On the contrary, there is minimal divergence in the frequency of adverb
phrases (ADVP). Interestingly, despite the parallel occurrence of VERB UPOS
tag in human and LLM-generated texts, the latter exhibits a more pronounced
propensity for verb phrases (VP). Finally, we see that language models use a
considerable larger amount of subordinate clauses (SBAR).

Type | H | 7B | 13B | 30B | 65B | Symbol | H | 7B | 13B | 30B | 65B
---|---|---|---|---|---|---|---|---|---|---|---
NP | 42.39 | 40.80 | 40.76 | 41.67 | 41.29 | ADVP | 2.54 | 2.60 | 2.59 | 2.47 | 2.54
VP | 18.42 | 19.62 | 19.61 | 19.12 | 19.31 | ADJP | 2.08 | 1.92 | 2.02 | 1.93 | 1.90
PP | 13.52 | 12.75 | 12.61 | 12.98 | 12.88 | NML | 1.87 | 1.81 | 1.81 | 1.89 | 1.83
S | 12.14 | 12.83 | 13.00 | 12.66 | 12.75 | WHNP | 1.41 | 1.53 | 1.44 | 1.46 | 1.47
SBAR | 3.69 | 4.13 | 4.10 | 3.81 | 3.99 |  |  |  |  |  |
Table 7: Percentage of spans generated by humans and LLaMa models labeled with
a specific constituent type. Only constituent types that conform more than 1%
of the human’s texts spans are shown. Figure 6:  Percentage differences,
following Table 7, in the use of constituent labels between each LLaMa model
in comparison to humans.

###  3.2 Semantic Analysis

To perform the psychometric and sociolinguistic analysis we also relied on
pre-existing NLP models. Below, we detail the results of analyzing emotions,
text similarities, and gender biases.

####  3.2.1 Emotions

To study differences in the emotions conveyed by human- and LLaMa-generated
outputs, we relied on the Hartmann (2022) emotion model. Table 8 provides the
percentage of articles labeled with distinct emotional categories, including
anger, disgust, fear, joy, sadness, surprise, and a special tag neutral to
denote that no emotion is present in the text. For a visual representation of
the variations in emotional content, Figure 7 depicts the percentage of
articles associated with each emotion for each LLaMa version, as compared to
human-written texts. As anticipated in journalistic texts, a substantial
majority of the lead paragraphs are classified as neutral. This category
accounts for over 50% of the texts across all models and human-generated
samples, with the LLM-generated text demonstrating a slightly higher
inclination towards neutrality.

Concerning the remainder of the samples, human texts demonstrate a greater
inclination towards negative and aggressive emotions, encompassing anger,
disgust, and fear. In contrast, LLMs tend to generate more texts imbued with
positive emotions, such as joy and surprise.555LLaMa also produces a many sad
texts, a passive but negative emotion, yet less toxic than emotions such as
anger or fear. Across various LLaMa models, fear and anger increase as the
number of parameters grows, making them more akin to human texts. Since LLaMa
(version 1 models) were not fine-tuned with reinforcement learning with human
feedback, we hypothesize the main source contributing to this issue might be
some pre-processing steps used for the LLaMa models, such as removing toxic
content from its data. Yet, it is noteworthy that the LLaMa technical report
Touvron et al. (2023) mentioned an increase in model toxicity as they scaled
up in size despite using the same pre-processing in all cases, which is
coherent with our findings.

Model | Emotion
---|---
anger | disgust | fear | joy | neutral | sadness | surprise
Human | 8.04 | 9.35 | 10.77 | 8.30 | 52.16 | 8.51 | 2.87
7B | 7.13 | 7.19 | 8.68 | 8.97 | 55.57 | 9.43 | 3.01
13B | 7.72 | 7.41 | 8.69 | 9.00 | 53.95 | 9.72 | 3.51
30B | 7.39 | 7.45 | 8.61 | 9.54 | 54.23 | 9.59 | 3.19
65B | 7.45 | 8.26 | 9.25 | 9.10 | 53.65 | 8.80 | 3.49
Table 8: Percentage of articles generated by humans and each LLaMa model that
are labeled with different emotions. Figure 7:  Relative difference of emotion
labels of articles generated by LLaMa models in comparison to human text.

####  3.2.2 Text similarity

We conducted an analysis of the cosine semantic similarity between lead
paragraphs generated by various LLMs and their human-authored counterparts.
Our objective was to investigate the impact of model sizes on the semantic
similarity between these texts. To achieve so, we used a a state-of-the-art
sentence similarity model called all-mpnet-
base-v2666https://huggingface.co/sentence-transformers/all-mpnet-base-v2
Reimers and Gurevych (2019). Figure 8 illustrates the distribution of the
similarity scores obtained from our analysis. The results show that smaller-
sized LLMs do not necessarily result in a decrease in sentence similarity
compared to the human-authored texts.

Figure 8:  Distribution of similarity scores between the sentences generated
by the LLaMa models and human text.

####  3.2.3 Gender bias

Although related as well with our study with part-of-speech tags distribution,
we here separately analyze the proportion between masculine and feminine
pronouns employed in both human- and LLaMa-generated text. Based on the
morphological output by Stanza, we find the words that are pronouns have the
features Gender=Masc and Gender=Fem, respectively. Results in Table 9 indicate
that the models not only replicate the gender bias present in human texts but
actually exacerbate it. On average, the models use male pronouns twice as
often as female pronouns, whereas the already biased human texts use male
pronouns only 1.85 times more frequently than female pronouns.

Model | Male-Female ratio | Increment
---|---|---
Human | 1.85 | -
7B | 2.06 | 11.19%
13B | 1.97 | 6.32 %
30B | 2.06 | 11.24 %
65B | 1.97 | 6.23 %
Table 9: Male-to-female ratio of pronouns used by the text generated by humans
and each of the LLaMa models.

##  4 Related work

Next, we survey relevant work to the subject of this paper: (i) analyzing
inherent linguistic properties of machine-generated text, (ii) distinguishing
between machine- and human-generated texts, and (iii) applications of machine-
generated text to economize human labor and time.

##### Analysis of linguistic properties of AI-generated text

Cognitive scientists Cai et al. (2023) have exposed models such as ChatGPT to
experiments initially designed for humans. They verified that it was able to
replicate human patterns such as associating unfamiliar words to meanings,
denoising corrupted sentences, or reusing recent syntactic structures, among
other abilities. Yet, they also showed that ChatGPT tends to refrain from
using shorter words to compress meaning, as well as from using context to
resolve syntactic ambiguities. Similarly, Zhou et al. (2023) conducted a
thorough comparison between AI-created and human-created misinformation. They
first curated a dataset of human-created misinformation pertaining to the
COVID-19 pandemic. Then, they used these representative documents as prompts
for GPT-3 to generate synthetic misinformation. By analyzing and contrasting
the outputs from both sources, the study revealed notable differences.
Specifically, the AI-generated misinformation exhibited heightened emotional
content, used more salient expressions, and displayed a higher tendency to
express uncertainties and draw unwarranted conclusions. Very recently, Xu et
al. (2023) have shed light on the lexical conceptual representations of
GPT-3.5 and GPT-4. Particularly, their study demonstrated that these AI
language models exhibited strong correlations with human conceptual
representations in specific dimensions, such as emotions and salience.
However, they encountered challenges when dealing with concepts linked to
perceptual and motor aspects, such as visual, gustatory, hand/arm, or
mouth/throat aspects, among others. With the goal of measuring differences
across both types of texts, Pillutla et al. (2021) introduced MAUVE, a new
metric designed to compare the learned distribution of a language generation
model with the distributions observed in human-generated texts. Given the
inherent challenge in open-ended text generation tasks, where no single
correct output exists, the authors address the issue of gauging proximity
between distributions by leveraging the concept of a divergence curve.

##### Identification of synthetically-generated text

This challenge has been addressed from different angles including statistical,
feature-based methods Nguyen-Son et al. (2017); Fröhling and Zubiaga (2021)
and neural approaches Rodriguez et al. (2022); Zhan et al. (2023). Yet,
Crothers et al. (2022) recently concluded that except from neural methods, the
other approaches have little applicability when it comes to detecting
synthetic text coming from modern large language models. Ippolito et al.
(2020) observed two interesting behaviors related to this classification task:
(i) that fancier sampling methods for generation (e.g., nucleus or untruncated
random sampling) are helpful to better fool humans, but make the detection
easier for machines, and (ii) that showing longer excerpts help both machines
and humans to better detect synthetically-generated strings. Munir et al.
(2021) showed that it was possible to attribute a given synthetically-
generated text to the specific LLM model that produced it, using a standard
machine learning classification architecture that used XLNet Yang et al.
(2019) as its backbone. In a different line, Dugan et al. (2020) studied
whether humans could identify the fencepost where an initially human-generated
text transitions to a machine-generated one. There are also methods that have
been specifically designed to generate or detect machine-generated texts for
highly sensible domains, warning about the dangers of language technologies.
The SCIgen software Stribling et al. (2005) was able to create semantically
non-sense but grammatically correct research papers, whose content was
accepted at some conferences with poor peer-review processes. More recently,
Liao et al. (2023) showed that medical texts generated by ChatGPT were easy to
attribute to it: although the syntax is correct, the texts were usually more
vague and did not provide effective information but only general terminology
or knowledge.

##### LLMs as a replacement for human NLP tasks

The quality of current synthetically-generated text has encouraged researchers
to explore their potential for complementing labor-intensive tasks, such as
annotation and evaluation. For instance, He et al. (2022) generated synthetic
unlabeled text tailored for a specific NLP task. Then, they used an existing
supervised classifier to silver-annotate those sentences, aiming to establish
a fully synthetic process for generating, annotating, and learning instances
relevant to the target problem. Related, Chiang and Lee (2023) investigated
whether LLMs can serve as a viable replacement for human evaluators in
downstream tasks. Particularly, they conducted experiments where LLMs are
prompted with the same instructions and samples as provided to humans,
revealing a correlation between the ratings assigned by both types of
evaluators.

##  5 Conclusion

This paper presented a comprehensive study on linguistic patterns in texts
produced by both humans and machines, comparing them under controlled
conditions. To keep up with current trends, we used modern generative models,
specifically the LLaMA models. To ensure the novelty of texts and address
memorization concerns, we fed the LLaMA models headlines from news articles
published after the release data of the models. The study revealed that
despite generating highly fluent text, these models still exhibited noticeable
differences when compared to human-generated texts. More precisely, at the
lexical level, LLaMA models relied on a more restricted vocabulary.
Additionally, at the morphosyntactic level, discernible distinctions existed
between human and machine-generated texts, the latter having a preference for
parts of speech displaying (a sense of) objectivity - such as symbols or
numbers - while using substantially fewer adjectives. We also observed
variations in terms of syntactic structures, both for dependency and
constituent representations, specifically in the use of dependency and
constituent types, as well as the length of spans across both types of texts.
In terms of semantics, while exhibiting a great text similarity with respect
to the human texts, LLaMA models manifested less propensity than humans for
displaying aggressive negative emotions, such as fear or anger. However, we
noted a rise in the volume of negative emotions with the models’ size. This
aligns with prior findings that associate larger sizes with heightened
toxicity Touvron et al. (2023). Finally, we detected an inclination towards
the use of male pronouns, surpassing the frequency in comparison to their
human counterparts.

## Acknowledgements

This paper has received funding from grant SCANNER-UDC (PID2020-113230RB-C21)
funded by MCIN/AEI/10.13039/501100011033, grant FPI 2021
(PID2020-113230RB-C21) funded by MCIN/AEI/10.13039/501100011033, the European
Research Council (ERC), which has supported this research under the European
Union’s Horizon Europe research and innovation programme (SALSA, grant
agreement No 101100615), Xunta de Galicia (ED431C 2020/11), and Centro de
Investigación de Galicia “CITIC”, funded by Xunta de Galicia and the European
Union (ERDF - Galicia 2014-2020 Program), by grant ED431G 2019/01.

## References

  * Berzak et al. (2016) Yevgeni Berzak, Yan Huang, Andrei Barbu, Anna Korhonen, and Boris Katz. 2016.  Anchoring and agreement in syntactic annotations.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2215–2224, Austin, Texas. Association for Computational Linguistics.
  * Cai et al. (2023) Zhenguang G Cai, David A Haslett, Xufeng Duan, Shuqi Wang, and Martin J Pickering. 2023.  Does chatgpt resemble humans in language use?  _arXiv preprint arXiv:2303.08014_.
  * Chiang and Lee (2023) Cheng-Han Chiang and Hung-yi Lee. 2023.  Can large language models be an alternative to human evaluations? In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15607–15631, Toronto, Canada. Association for Computational Linguistics.
  * Crothers et al. (2022) Evan Crothers, Nathalie Japkowicz, Herna Viktor, and Paula Branco. 2022.  Adversarial robustness of neural-statistical features in detection of generative transformers.  In _2022 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–8. IEEE.
  * Den Heijer (2022) Micha Den Heijer. 2022.  pynytimes.
  * Dugan et al. (2020) Liam Dugan, Daphne Ippolito, Arun Kirubarajan, and Chris Callison-Burch. 2020.  RoFT: A tool for evaluating human detection of machine-generated text.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 189–196, Online. Association for Computational Linguistics.
  * Ferrer-i-Cancho (2004) Ramon Ferrer-i-Cancho. 2004.  Euclidean distance between syntactically linked words.  _Physical Review E_ , 70:056135.
  * Ferrer-i Cancho et al. (2022) Ramon Ferrer-i Cancho, Carlos Gómez-Rodríguez, Juan Luis Esteban, and Lluís Alemany-Puig. 2022.  Optimality of syntactic dependency distances.  _Physical Review E_ , 105(1):014308.
  * Fröhling and Zubiaga (2021) Leon Fröhling and Arkaitz Zubiaga. 2021.  Feature-based detection of automated language models: tackling gpt-2, gpt-3 and grover.  _PeerJ Computer Science_ , 7:e443.
  * Futrell et al. (2015) Richard Futrell, Kyle Mahowald, and Edward Gibson. 2015.  Large-scale evidence of dependency length minimization in 37 languages.  _Proceedings of the National Academy of Sciences_ , 112(33):10336–10341.
  * Hartmann (2022) Jochen Hartmann. 2022.  Emotion english distilroberta-base.  https://huggingface.co/j-hartmann/emotion-english-distilroberta-base/.
  * He et al. (2022) Xuanli He, Islam Nassar, Jamie Kiros, Gholamreza Haffari, and Mohammad Norouzi. 2022\.  Generate, annotate, and learn: NLP with synthetic text.  _Transactions of the Association for Computational Linguistics_ , 10:826–842.
  * Ippolito et al. (2020) Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2020.  Automatic detection of generated text is easiest when humans are fooled.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1808–1822, Online. Association for Computational Linguistics.
  * Kumar et al. (2023) Sachin Kumar, Vidhisha Balachandran, Lucille Njoo, Antonios Anastasopoulos, and Yulia Tsvetkov. 2023.  Language generation models can cause harm: So what can we do about it? an actionable survey.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 3299–3321, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Landgrebe and Smith (2021) Jobst Landgrebe and Barry Smith. 2021.  Making ai meaningful again.  _Synthese_ , 198:2061–2081.
  * Li et al. (2022) Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2022.  Emergent world representations: Exploring a sequence model trained on a synthetic task.  _arXiv preprint arXiv:2210.13382_.
  * Liao et al. (2023) Wenxiong Liao, Zhengliang Liu, Haixing Dai, Shaochen Xu, Zihao Wu, Yiyang Zhang, Xiaoke Huang, Dajiang Zhu, Hongmin Cai, Tianming Liu, et al. 2023.  Differentiate chatgpt-generated and human-written medical texts.  _arXiv preprint arXiv:2304.11567_.
  * Liu et al. (2023) Nelson F Liu, Tianyi Zhang, and Percy Liang. 2023.  Evaluating verifiability in generative search engines.  _arXiv preprint arXiv:2304.09848_.
  * Manning (2011) Christopher D Manning. 2011.  Part-of-speech tagging from 97% to 100%: is it time for some linguistics?  In _International conference on intelligent text processing and computational linguistics_ , pages 171–189. Springer.
  * Munir et al. (2021) Shaoor Munir, Brishna Batool, Zubair Shafiq, Padmini Srinivasan, and Fareed Zaffar. 2021.  Through the looking glass: Learning to attribute synthetic text generated by language models.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1811–1822, Online. Association for Computational Linguistics.
  * Nguyen-Son et al. (2017) Hoang-Quoc Nguyen-Son, Ngoc-Dung T Tieu, Huy H Nguyen, Junichi Yamagishi, and Isao Echi Zen. 2017.  Identifying computer-generated text using statistical analysis.  In _2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_ , pages 1504–1511. IEEE.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Petrov et al. (2012) Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012.  A universal part-of-speech tagset.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 2089–2096, Istanbul, Turkey. European Language Resources Association (ELRA).
  * Pillutla et al. (2021) Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021.  Mauve: Measuring the gap between neural text and human text using divergence frontiers.  _Advances in Neural Information Processing Systems_ , 34:4816–4828.
  * Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020\.  Stanza: A python natural language processing toolkit for many human languages.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 101–108, Online. Association for Computational Linguistics.
  * (26) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Rodriguez et al. (2022) Juan Rodriguez, Todd Hay, David Gros, Zain Shamsi, and Ravi Srinivasan. 2022.  Cross-domain detection of GPT-2-generated technical text.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1213–1233, Seattle, United States. Association for Computational Linguistics.
  * Santurkar et al. (2023) Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023.  Whose opinions do language models reflect?  _arXiv preprint arXiv:2303.17548_.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Søgaard (2022) Anders Søgaard. 2022.  Understanding models understanding language.  _Synthese_ , 200(6):443.
  * Stribling et al. (2005) Jeremy Stribling, Max Krohn, and Dan Aguayo. 2005.  Scigen-an automatic cs paper generator.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023.  Alpaca: A strong, replicable instruction-following model.  _Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html_ , 3(6):7.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Xu et al. (2023) Qihui Xu, Yingying Peng, Minghua Wu, Feng Xiao, Martin Chodorow, and Ping Li. 2023\.  Does conceptual representation require embodiment? insights from large language models.  _arXiv preprint arXiv:2305.19103_.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  _Advances in neural information processing systems_ , 32.
  * Zhan et al. (2023) Haolan Zhan, Xuanli He, Qiongkai Xu, Yuxiang Wu, and Pontus Stenetorp. 2023.  G3detector: General gpt-generated text detector.  _arXiv preprint arXiv:2305.12680_.
  * Zhou et al. (2023) Jiawei Zhou, Yixuan Zhang, Qianni Luo, Andrea G Parker, and Munmun De Choudhury. 2023.  Synthetic lies: Understanding ai-generated misinformation and evaluating algorithmic and human solutions.  In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ , pages 1–20.