# Direct and indirect evidence of compression of word lengths.
Zipf’s law of abbreviation revisited.

###### Abstract

Zipf’s law of abbreviation, the tendency of more frequent words to be shorter,
is one of the most solid candidates for a linguistic universal, in the sense
that it has the potential for being exceptionless or with a number of
exceptions that is vanishingly small compared to the number of languages on
Earth. Since Zipf’s pioneering research, this law has been viewed as a
manifestation of a universal principle of communication, i.e. the minimization
of word lengths, to reduce the effort of communication. Here we revisit the
concordance of written language with the law of abbreviation. Crucially, we
provide wider evidence that the law holds also in speech (when word length is
measured in time), in particular in 46 languages from 14 linguistic families.
Agreement with the law of abbreviation provides indirect evidence of
compression of languages via the theoretical argument that the law of
abbreviation is a prediction of optimal coding. Motivated by the need of
direct evidence of compression, we derive a simple formula for a random
baseline indicating that word lengths are systematically below chance, across
linguistic families and writing systems, and independently of the unit of
measurement (length in characters or duration in time). Our work paves the way
to measure and compare the degree of optimality of word lengths in languages.

Keywords: word length, compression, law of abbreviation

##  1 Introduction

It has been argued that linguistic universals are a myth [8], but this
neglects the statistical regularities that the quantitative linguistic
community has been investigating for many decades. A salient case is Zipf’s
law of abbreviation, the tendency of more frequent words to be shorter [28].
It holds across language families [19, 2, 15, 16, 14], writing systems [26,
21] and modalities [3, 24], and also when word length in characters is
replaced by word duration in time [12]. Furthermore, the number of species
where a parallel of this law has been confirmed is growing over time [22]
111The interested reader can check the latest discoveries on this law in
Bibliography on laws of language outside human language.. In language
sciences, research on the law of abbreviation in languages measures word
length in discrete units (e.g., characters) whereas, in biology, research on
the law in other species typically uses duration in time. Here, we aim to
reduce the gulf that separates these two traditions by promoting research on
the law of abbreviation on word durations.

G. K. Zipf believed that the law of abbreviation constituted indirect evidence
of the minimization of the cost of using words [28]. At present, Zipf’s view
is supported by standard information theory and its extensions: the main
argument is that the minimization of $L$, the mean word length, that is indeed
a simplification of Zipf’s cost function222He referred to the cost function as
“minimum equation” [28], leads to the law of abbreviation [10, 9]. In the
language of information theory, the minimization of mean word length is known
as compression. In the language of quantitative linguistics, $L$ is the
average length of tokens from a repertoire of $n$ types, that is defined as

(1) $L=\sum_{i=1}^{n}p_{i}l_{i},$

where $p_{i}$ and $l_{i}$ are, respectively, the probability and the length of
the $i$-th type. In practical applications, $L$ is calculated replacing
$p_{i}$ by the relative frequency of a type, that is

$p_{i}=f_{i}/T,$

where $f_{i}$ is the absolute frequency of a type and $T$ is the total number
of tokens, i.e.

$T=\sum_{i=1}^{n}f_{i}.$

This leads to a definition of $L$ that is

$L=\frac{1}{T}\sum_{i=1}^{n}f_{i}l_{i}.$

At present, the mathematical link between the law of abbreviation and
compression has been established under the assumption that words are coded
optimaly so as to minimize $L$. If words are coded optimaly, the correlation
between the frequency of a word and its duration cannot be positive [9]. Thus,
a lack of correlation between the frequency of a word and its duration does
not imply absence of compression. Furthermore, it is not a warranted
assumption that languages code words optimaly. Therefore, an approach to find
direct evidence of compression getting rid of the assumption of optimal coding
is required.

As a first approach, one could compare the value of $L$ of a language against
$L_{max}$, the maximum value that $L$ could achieve in this language. The
larger the gap between $L$ and $L_{max}$, the higher the level of compression
in the language. However, the problem is that $L_{max}$ can be infinite a
priori. To fix that problem, one could restrict $L_{max}$ to be finite but
then this raises the question of what should be the finite value of $L_{max}$
and why. For these reasons, here we resort to the notion of random baseline,
an imaginary line that we use for comparison against the actual value of $L$,
that is defined assuming some random mapping of word types into strings. In
previous research, the random baseline was defined by the average word length
resulting from a shuffling of the current length/duration of types [10, 11] so
as to check if $L$ was smaller than expected by chance in that random mapping
[10, 11]. Critically, an exact method to compute the random baseline, namely
the expected word length in these shufflings, is missing.

The remainder of the article is organized as follows. In section 2, we
introduce the definition of $L_{r}$, the random baseline, that we will use to
explore direct evidence of compression. In particular, we derive a simple
formula for $L_{r}$ that will simplify future research on compression in
natural communication systems. In section 3 and section 4, we present,
respectively, the materials and methods that will be used to provide further
evidence of compression and the law of abbreviation in real languages with
emphasis on word durations. In section 4, we present a new unsupervised method
to exclude words with foreign characters in line with good practices for
research on linguistic laws and communicative efficiency [16]. In section 5,
we show that the law of abbreviation holds without exceptions in a wide sample
of languages, independently of the unit of measurement of word length, namely
characters or duration in time, providing further indirect evidence of
compression in languages. In addition, the random baseline indicates that word
lengths are systematically below chance, across linguistic families and
writing systems, independently of the unit of measurement (length in
characters or duration in time), providing direct evidence of compression.
Finally, in section 6, we discuss the findings in relation to the potential
universality of the law of abbreviation and the universality of compression in
languages. We also make proposals for future research.

##  2 A random baseline revisited

In our statistical setting, the null hypothesis states that compression
(minimization of word lengths) has no effect on word lengths. The alternative
hypothesis states that compression has an effect on word lengths as Zipf
hypothesized. If the null hypothesis is rejected then word lengths are shorter
than expected by chance.

Table 1:  Matrix indicating the frequency and length of three types. The mean type length is $L=\frac{235}{125}=1.88$.  $i$ | $f_{i}$ | $l_{i}$
---|---|---
1 | 100 | 2
2 | 20 | 1
3 | 5 | 3

Consider a matrix with two columns, $f_{i}$ and $l_{i}$, that are used to
compute the average word length $L$. The matrix in Table 1 gives
$L=\frac{235}{125}=1.88$. We consider the null hypothesis of a random mapping
of probabilities into lengths, namely that the ordering of the $f_{i}$’s or
the $l_{i}$’s in Table 1 is arbitrary and results from a random shuffling of
one of these variables or both. We use $f_{i}^{\prime}$, $l_{i}^{\prime}$ and
$p_{i}^{\prime}$ for the new values of $f_{i}$, $l_{i}$ and $p_{i}$ that
result from one of these shufflings.

This null hypothesis was introduced in research on compression in human
language and animal communication to test if $L$ is significantly small using
a permutation test [10, 11]. Later, it was used to estimate the degree of
optimality of word lengths [17, 20]. Our new contribution here is a precise
mathematical characterization of the null hypothesis and the derivation of a
simple formula the expected word length.

In the context of computing average word length, the matrix in Table 1 is
equivalent to a matrix where the column $f_{i}$ is replaced by a column with
$p_{i}$ thanks to

$p_{i}^{\prime}=\frac{f_{i}^{\prime}}{T}.$

Indeed, the null hypothesis has three variants

  1. 1.

Single column shuffling. Only the column of $f_{i}$ or $p_{i}$ is shuffled.

  2. 2.

Single column shuffling. Only the column of $l_{i}$ is shuffled.

  3. 3.

Dual column shuffling. The column of $f_{i}$ or $p_{i}$ and the column of
$l_{i}$ are both shuffled.

In each of the variants, all random shufflings of a specific column are
equally likely. In case of dual shuffling, the shuffling of one column is
independent of the shuffling of the other column. The outcome of a dual
shuffling on Table 1 is shown in Table 2.

Table 2:  Matrix indicating the frequency and length of three types. The mean type length is $L=\frac{345}{125}=2.76$.  $i$ | $f_{i}^{\prime}$ | $l_{i}^{\prime}$
---|---|---
1 | 20 | 2
2 | 100 | 3
3 | 5 | 1

The random baseline, $L_{r}$, is the expected value of $L$ under the null
hypothesis 333Notice that $L$ is indeed the expected value of the length of a
token but under a distinct setting (a distinct null hypothesis), where one
picks a token uniformly at random over all tokens of a text and looks at its
length. . $L_{r}$ can be defined in more detail in two main equivalent ways:

  1. 1.

The value of $L$ that is expected if $L$ is recomputed after pairing the
$f_{i}$’s and the $l_{i}$’s at random and recomputing $L$. The new value of
$L$ depends on the variant of the null hypothesis. When shuffling the column
for $f_{i}$ in the matrix (Table 1), the new $L$ is

$L^{\prime}=\frac{1}{T}\sum_{i=1}^{n}f_{i}^{\prime}l_{i}.$

When shuffling the column for $l_{i}$ and recomputing $L$, the new $L$ is

$L^{\prime}=\frac{1}{T}\sum_{i=1}^{n}f_{i}l_{i}^{\prime}.$

When shuffling both columns, the new $L$ is

$L^{\prime}=\frac{1}{T}\sum_{i=1}^{n}f_{i}^{\prime}l_{i}^{\prime}.$

  2. 2.

The average value of $L$ that is expected over all possible shufflings in one
of the variants of the null hypothesis. In the example in Table 3, on
shuffling only the $l_{i}$ column,

$L_{r}=\frac{\frac{155}{125}+\frac{170}{125}+\frac{235}{125}+\frac{265}{125}+\frac{330}{125}+\frac{345}{125}}{6}=\frac{155+170+235+265+330+345}{125\cdot
6}=2.$

We use $\operatorname{\mathbb{E}}[X]$ to refer to the expected value of a
random variable $X$ under some variant of the null hypothesis above. Then

$L_{r}=\operatorname{\mathbb{E}}[L^{\prime}],$

where $L^{\prime}$ is the value of $L$ resulting from some shuffling.

In quantitative linguistics, the mean length of tokens ($L$) is also known as
dynamic word length [4] and corresponds to the mean length of the words in a
text. The mean length of types ($M$), defined as

$M=\frac{1}{n}\sum_{i=1}^{n}l_{i},$

is also known as the static word length and corresponds to average length of
the headwords in a dictionary [4]. Interestingly, the following property
states that $L_{r}$ turns out to be $M$ independently of the variant of the
null hypothesis under consideration.

######  Property 2.1.

The expected value of $L^{\prime}$ under any variant of the null hypothesis is
$L_{r}=M$.

###### Proof.

We analyze $\operatorname{\mathbb{E}}[L^{\prime}]$ under each of the variants
of the null hypothesis.

Dual shuffling. Applying the linearity of expectation and independence between
the shuffling of the $p_{i}$ column of the that of the $l_{i}$ column, we
obtain

$\displaystyle\operatorname{\mathbb{E}}[L^{\prime}_{1}]$ $\displaystyle=$
$\displaystyle\operatorname{\mathbb{E}}\left[\sum_{i=1}^{n}p_{i}^{\prime}l_{i}^{\prime}\right]$
$\displaystyle=$
$\displaystyle\sum_{i=1}^{n}\operatorname{\mathbb{E}}[p_{i}^{\prime}l_{i}^{\prime}]$
$\displaystyle=$
$\displaystyle\sum_{i=1}^{n}\operatorname{\mathbb{E}}[p_{i}^{\prime}]\operatorname{\mathbb{E}}[l_{i}^{\prime}].$

Noting that

$\displaystyle\operatorname{\mathbb{E}}[p_{i}^{\prime}]$ $\displaystyle=$
$\displaystyle\frac{1}{n}\sum_{i=1}^{n}p_{i}=\frac{1}{n}$
$\displaystyle\operatorname{\mathbb{E}}[l_{i}^{\prime}]$ $\displaystyle=$
$\displaystyle\frac{1}{n}\sum_{i=1}^{m}l_{i}=M,$

we finally obtain

(2) $\operatorname{\mathbb{E}}[L^{\prime}]=\sum_{i=1}^{n}\frac{M}{n}=M.$

Single shuffling of the $l_{i}$ column. Applying the linearity of expectation
and the fact that the column of $p_{i}$ remains constant, we obtain

$\displaystyle\operatorname{\mathbb{E}}[L^{\prime}_{1}]$ $\displaystyle=$
$\displaystyle\operatorname{\mathbb{E}}\left[\sum_{i=1}^{n}p_{i}l_{i}^{\prime}\right]$
$\displaystyle=$
$\displaystyle\sum_{i=1}^{n}p_{i}\operatorname{\mathbb{E}}[l_{i}^{\prime}].$

Recalling $\operatorname{\mathbb{E}}[l_{i}^{\prime}]=M$, we finally obtain

(3) $\operatorname{\mathbb{E}}[L^{\prime}]=M\sum_{i=1}^{n}p_{i}=M.$

Single shuffling of the $p_{i}$ column. Applying the linearity of expectation
and the fact that the column of $l_{i}$ remains constant, we obtain

$\displaystyle\operatorname{\mathbb{E}}[L^{\prime}_{1}]$ $\displaystyle=$
$\displaystyle\operatorname{\mathbb{E}}\left[\sum_{i=1}^{n}p_{i}^{\prime}l_{i}\right]$
$\displaystyle=$
$\displaystyle\sum_{i=1}^{n}\operatorname{\mathbb{E}}[p_{i}^{\prime}]l_{i}.$

Recalling $\operatorname{\mathbb{E}}[p_{i}^{\prime}]=\frac{1}{n}$, we finally
obtain

(4) $\operatorname{\mathbb{E}}[L^{\prime}]=\frac{1}{n}\sum_{i=1}^{n}l_{i}=M.$

∎

The previous finding indicates that the random baseline for $L$ is equivalent
to assuming that all word types are equally likely, namely, replacing each
$p_{i}$ by $1/n$.

Table 3:  All the $3!=6$ permutations of the column $l_{i}$ in Table 1 that can be produced. Each permutation is indicated with letters from A to F. $L^{\prime}$, the mean length of types in a shuffling, is shown at the bottom for each permutation.  |  | A | B | C | D | E | F
---|---|---|---|---|---|---|---
$i$ | $f_{i}$ | $l_{i}^{\prime}$ | $l_{i}^{\prime}$ | $l_{i}^{\prime}$ | $l_{i}^{\prime}$ | $l_{i}^{\prime}$ | $l_{i}^{\prime}$
1 | 100 | 1 | 1 | 2 | 2 | 3 | 3
2 | 20 | 2 | 3 | 1 | 3 | 1 | 2
3 | 5 | 3 | 2 | 3 | 1 | 2 | 1
| $L^{\prime}$ | $\frac{155}{125}=1.24$ | $\frac{170}{125}=1.36$ | $\frac{235}{125}=1.88$ | $\frac{265}{125}=2.12$ | $\frac{330}{125}=2.64$ | $\frac{345}{125}=2.76$

##  3 Material

###  3.1 General information about corpora and languages

We investigate the relationship between the frequency of a word and its length
in languages from two collections: Common Voice Forced Alignments
(subsubsection 3.2.1), hereafter CV, and Parallel Universal Dependencies
(subsubsection 3.2.2), hereafter PUD.

All the preprocessed files used to produce the results from the original
collections are available in the repository of the article444In the data
folder of https://github.com/IQL-course/IQL-Research-Project-21-22..

PUD comprises 20 distinct languages from 7 linguistic families and 8 scripts
(subsection 3.1). CV comprises 46 languages from 14 linguistic families (we
include ’Conlang’, i.e. ’constructed languages’, as a family for Esperanto and
Interlingua) and 10 scripts (subsection 3.1). Both PUD and CV are biased
towards the Indo-European family and the Latin script. The typological
information (language family) is obtained from Glottolog
4.6555https://glottolog.org/. The writing systems are determined according to
ISO-15924 codes666https://unicode.org/iso15924/iso15924-codes.html. In
subsection 3.1 and subsection 3.1, we show the scripts using their standard
English names. For example, most languages from the Indo-European family are
written in Latin scripts. We also categorize Chinese Pinyin and Japanese
Romaji as Latin scripts.

Table 4:  Summary of the main characteristics of the languages in the PUD
collection. For each language, we show the linguistic family, the writing
system (namely script name according to ISO-15924) and various numeric
parameters: $A$, the observed alphabet size (number of distinct characters),
$n$, the number of word types, and $T$, the number of word tokens.

Language | Family | Script | $A$ | $n$ | $T$
---|---|---|---|---|---
Arabic | Afro-Asiatic | Arabic | 20 | 3309 | 11667
Indonesian | Austronesian | Latin | 23 | 4501 | 16702
Russian | Indo-European | Cyrillic | 23 | 4666 | 11749
Hindi | Indo-European | Devanagari | 44 | 4343 | 20071
Czech | Indo-European | Latin | 33 | 7073 | 15331
English | Indo-European | Latin | 25 | 5001 | 18028
French | Indo-European | Latin | 26 | 5214 | 20407
German | Indo-European | Latin | 28 | 6116 | 18331
Icelandic | Indo-European | Latin | 32 | 6035 | 16209
Italian | Indo-European | Latin | 24 | 5606 | 21266
Polish | Indo-European | Latin | 31 | 7188 | 15191
Portuguese | Indo-European | Latin | 38 | 5661 | 21855
Spanish | Indo-European | Latin | 32 | 5750 | 21067
Swedish | Indo-European | Latin | 25 | 5624 | 16378
Japanese | Japonic | Japanese | 1549 | 4852 | 24737
Japanese-strokes | Japonic | Japanese | 1549 | 4852 | 24737
Japanese-romaji | Japonic | Latin | 24 | 4849 | 24734
Korean | Koreanic | Hangul | 379 | 6218 | 12307
Thai | Kra-Dai | Thai | 50 | 3573 | 20860
Chinese | Sino-Tibetan | Han (Traditional variant) | 2038 | 4970 | 17845
Chinese-strokes | Sino-Tibetan | Han (Traditional variant) | 2038 | 4970 | 17845
Chinese-pinyin | Sino-Tibetan | Latin | 50 | 4970 | 17845
Turkish | Turkic | Latin | 28 | 6587 | 13799
Finnish | Uralic | Latin | 24 | 6938 | 12701

Table 5:  Summary of the main characteristics of the languages in the CV
collection. For every language we show its linguistic family, the writing
system (namely script name according to ISO-15924) and various numeric
parameters: $A$, the observed alphabet size (number of distinct characters),
$n$, the number of word types, and, $T$, the number of word tokens. ’Conlang’
stands for ’constructed language’, that is an artificially created language.
This is not a family in the proper sense as Conlang languages are not related
in the common linguistic family sense.

###  3.2 The datasets

We measure word length in two main ways: duration in time and length in
characters. Concerning Chinese and Japanese, we additionally consider the
number of strokes and the number of characters of their romanization (i.e.
Pinyin for Chinese and Romaji for Japanese).

Given these datasets, word durations are obtained only from CV. Word lengths
in characters are obtained from both CV as well as from PUD. Word lengths in
strokes, and word lengths in characters after romanization, are obtained only
from PUD.

####  3.2.1 Common Voice Forced Alignments

The Common Voice Corpus777https://commonvoice.mozilla.org/en/datasets is an
open source dataset of recorded voices uttering sentences in many different
languages. The amount of data, as well as the source and topic of each
sentence, depends considerably on the language and the corpus version.
Specifically, the Common Voice Corpus 5.1 contains information on 54 languages
and dialects.

Common Voice Forced Alignments (CVFA)888https://github.com/JRMeyer/common-
voice-forced-alignments were created by Josh Meyer using the Montreal Forced
Aligner999https://github.com/MontrealCorpusTools/Montreal-Forced-Aligner on
top of the Common Voice Corpus 5.1. Kabyle, Upper Sorbian and Votic were left
out of the alignments for an undocumented reason. Therefore, CVFA contains
information on 51 languages.

In our analyses, Japanese and the three Chinese dialects were excluded as the
forced aligner failed to correctly extract words from sentences. In addition,
both Romansh dialects were fused into a single Romansh language. Indeed, given
the nature of this corpus, all languages are likely to be represented by more
than one dialect.

Notice that Abkhazian, Panjabi, and Vietnamese have a critically low number of
tokens ($T<1000$ in subsection 3.1). However, we decided to include them in
the analyses so as to understand their limitations related to corpus size.

####  3.2.2 Parallel Universal Dependencies

The Universal Dependencies (UD)101010https://universaldependencies.org/
collection is an open source dataset of annotated sentences, in which the
amount of data depends on each language. The Parallel Universal Dependencies
(PUD) collection is a parallel subset of 20 languages from the UD collection,
consisting of 1000 sentences. It allows for a cross-language comparison,
controlling for content and annotation style.

In subsection 3.1, we show the characteristics of the languages in PUD. For
traditional Chinese and Japanese, we also include word lengths in
romanizations (Pinyin and Romaji respectively), as well as word lengths
measured in strokes, resulting in a total of 24 language files. Notice that
three Japanese words that are hapax legomena could not be romanized and thus
the number of tokens and types varies slightly with respect to the original
Japanese characters (subsection 3.1).

##  4 Methodology

All the code used to produce the results is available in the repository of the
article 111111 In the code folder of https://github.com/IQL-course/IQL-
Research-Project-21-22..

###  4.1 The units of length

####  4.1.1 Duration

The duration of a word for a given language is estimated by computing the
median duration in seconds across all its occurrences in utterances in the CV
corpus. All words with equal orthographic form are assumed to be the same
type. The median is preferred over the mean as it is less sensitive to
outliers (that may be produced by forced alignment errors) and better suited
to deal with heavy-tailed distributions [12]. Given the oral nature of the
data, we do expect to observe some variation in the duration of words, due to
differences between individuals, and variation within a single individual.
This is more generally in line with speakers acting as complex dynamical
systems [13]. For these reasons, median duration is preferred for research on
the law of abbreviation in acoustic units [24, 27].

####  4.1.2 Length in characters

Word length in characters is measured by counting every Unicode UTF-8
character present in a word. Special characters such as “=” were removed.
Characters with stress accents are considered as different from their non-
stressed counterpart (e.g. “a” and “à” are considered separate characters).
Following best practices from [16], characters were always kept in UTF-8.

####  4.1.3 Length in strokes

Japanese Kanji and Chinese Hanzi were turned into strokes using the cihai
Python library121212https://github.com/cihai/cihai. In Japanese characters
other than Kanji, namely Japanese Kana, the number of strokes in printed
versus hand-written modality can differ (Chinese Hanzi and Japanese Kanji have
the same number of strokes in printed version or hand-written version). Here
we counted the number of strokes in printed form. Japanese Kana were converted
into printed strokes by using a hand-crafted correspondence table, since Kana
is not part of the CJK unified character system. This table was created by us
and checked by a native linguist (S. Komori from Chubu University, Japan). It
is available in the repository of the article 131313In the data/other folder
of https://github.com/IQL-course/IQL-Research-Project-21-22..

In case of discrepancies on the number of strokes for a given character, the
most typical printed version was chosen.

####  4.1.4 Length in Pinyin and Romaji

Chinese Pinyin was obtained using the cihai package as above, while the
Japanese Romaji was obtained with the cutlet Python
library141414https://github.com/polm/cutlet. The latter uses Kunrei-shiki
romanization (since it is the one used officially by the government of Japan)
and the spelling of foreign words is obtained in its native reading (e.g.
“カレー” is romanized as “karee” instead of “curry”). There are some
particularities with the romanization of Kanji characters by cutlet. For
example, in the case of the word “year” (年), it chose the reading of “Nen”
instead of “Tosi”, which would be the expected one.

A more systematic issue with Japanese romanization is that it does not provide
means to indicate pitch accents, which are implicitly present in Kanji. For
example, “日本” “Ni$\uparrow$hon” (“Japan”) is romanized as simply “Nihon”.
Therefore, the alphabet size of romanized Japanese is smaller than it should
be, compared to other languages where, as stated before, stress accents are
counted as distinctive features of characters.

###  4.2 Tokenization

Tokenization is already given in each dataset and we borrow it for our
analyses. Thus tokenization methods are not uniform for CV and PUD and are not
guaranteed to be uniform among languages even within each of these datasets.

###  4.3 Filtering of tokens

Examining our datasets, we noticed that in some text files there was a
considerable number of unusual character strings, as well as foreign words
(written in different scripts). These need to be filtered out in order to
obtain a “clean” set of word types. To this end we filter out tokens following
a two step procedure:

  1. 1.

Mandatory elementary filtering. This filter consists of:

     * •

Common filtering. In essence, it consists of the original tokenization and the
removal of tokens containing digits. In each collection, the original
tokenizer yields tokens that may contain certain punctuation marks. Due to the
nature of the CV dataset, the bulk of punctuation was already removed via the
Montreal Forced Aligner with some exceptions. For instance, single quotation
(in particular “”’) is a punctuation sign that is kept within a word token in
CV, as it is necessary for the formation of clitics in multiple languages,
such as in English or French. In PUD, as a part of UD, contractions are split
into two word types. “can’t” is split into “ca” “n’t” (in CV “can’t” would
remain as just one token). In both collections, words containing ASCII digits
are removed because they do not reflect phonemic length and can be seen as
another writing system.

     * •

Specific filtering. In case of the PUD collection, we excluded all tokens with
Part-of-Speech (POS) tag ‘PUNCT’. In case of the CV collection, we removed
tokens tagged as ¡unk¿ or null tokens, namely tokens that either could not be
read or that represent pauses.

     * •

Lowercasing. Every character is lowercased. In the case of CV, this is already
given by the Montreal Forced Aligner, while in the case of PUD, tokens are
lowercased by means of the spaCy python package151515https://spacy.io/.

  2. 2.

Optional filtering. This is a new method that is applied after the previous
filter and described in subsection 4.4.

###  4.4 A new method to filter out unusual characters

It has been pointed out that “chunk” words and loanwords can distort the
results of quantitative analyses of word lengths [16]. Indeed, especially the
files of the Common Voice Corpus feature a considerable number of word tokens
which do not consist of characters belonging to the primary alphabet of the
respective writing system. [16] proposed to use dictionaries to exclude such
anomalous words. However, this is not feasible for our multilingual datasets,
as loanword dictionaries are not available for this large number of diverse
languages (subsection 3.1 and subsection 3.1). The Intercontinental Dictionary
Series161616https://ids.clld.org/, for example, contains only around half of
the languages in our analysis, so it is not applicable to many of them. Hence,
this approach would lead to a non-uniform treatment of different languages and
texts. Selecting a matched set of semantic concepts across languages using a
lexical database is also infeasible due to similar reasons.

Against this backdrop, we decided to develop an unsupervised method to filter
out words which contain highly unusual characters. For a given language, the
method starts by assuming that the strings (after the mandatory filtering
illustrated above) contain characters of two types: characters of the
working/primary alphabet as well as other characters. We hypothesize that the
latter are much less frequent than the former. Following this rationale, we
apply the $k$-means algorithm of the Ckmeans R
package171717https://cran.r-project.org/web/packages/Ckmeans.1d.dp/index.html
to split the set of characters into the two groups based on the logarithm of
the frequency of the characters181818The motivation for taking logarithms of
frequencies is three-fold: First, this brings observations closer together.
Note that the $k$-means algorithm prefers high-density areas. Second, this
transforms the frequencies into a measure of surprisal, following standard
information theory [23]. Third, manual inspection suggests that the
logarithmic transformation is required to produce an accurate split.. To
maximize the power of the clustering method, we use the exact method with
$k=2$ for one dimension instead of the customary approximate method. We then
keep the high frequency cluster as the real working alphabet and filter out
the word tokens that contain characters not belonging to this high frequency
cluster.

We illustrate the power of the method by showing working alphabets that are
obtained on CV, that is the noisiest one of the collections.

In English, the working alphabet is defined by the 26 English letters and
quotation marks (“”’, “”’). These quotation marks are used often in clitics,
and as such are correctly identified as part of the encoding, since, for
example, “can’t” and “cant” are different words in meaning, with “can’t”
meaning “can not”, while “cant” is a statement on a religious or moral subject
that is not believed by the person making the statement, with the
differentiating feature being the “”’. Therefore, the working alphabet becomes
5 vowels (“a”, “e”, “i”, “o”, “u”), 21 consonants (“b”, “c”, “d”, “f”, “g”,
“h”, “j”, “k”, “l”, “m”, “n”, “p”, “q”, “r”, “s”, “t”, “v”, “w”, “x”, “y”,
“z”) and 2 kinds of quotation marks (“”’, “”’).

In Russian, the working alphabet comprises 9 vowels ( “ а”, “о”, “у”, “ы”,
“э”, “я”, “ю”, “и”, “е”), a semivowel / consonant “й”, 20 consonants ( “б”,
“в”, “г”, “д”, “ж”, “з”, “к”, “л”, “м”, “н”, “п”, “р”, “с”, “т”, “ф”, “х”,
“ц”, “ч”, “ш”, “щ”) and 2 modifier letters (“ъ”, “ь”).

In Italian, it comprises 5 vowels (“a”, “e”, “i”, “o”, “u”), 21 consonants
(“b”, “c”, “d”, “f”, “g”, “h”, “j”, “k”, “l”, “m”, “n”, “p”, “q”, “r”, “s”,
“t”, “v”, “w”, “x” , “y”, “z”) and 6 instances of the 5 vowels containing a
diacritic mark (“à”, “è”, “é”, “ì”, “ò”, “ù”).

The unsupervised filter method filter is not applied to Chinese, Japanese and
Korean as, given their nature, this would exclude letters that actually belong
to the real alphabet. In subsection B.1 we analyze the impact of the optional
filter and provide arguments for not applying the unsupervised filter to these
languages. As a compensation, strings that contain non-CJK characters are
filtered out in Chinese and Japanese as a part of the optional filter. In
Korean, only a few characters are not proper Hangul and thus such a
complementary filtering is not necessary.

###  4.5 Immediate constituents in writing systems

When measuring word length in written languages, we are using immediate
constituents of written words. In Romance languages, the immediate
constituents are letters of the alphabet, which are a proxy for phonemes. For
syllabic writing systems (as Chinese in our dataset), these are characters
that correspond to syllables. In addition, for Chinese and Japanese, we are
considering two other possible units for word length, which are not immediate
constituents, but alternative ways of measuring word lengths which could
provide useful insights: strokes and letters in Latin script romanizations.
That means that for each of these languages words are unfolded into three
systems, one for each unit of encoding (original characters, strokes,
romanized letters/characters). In the hierarchy from words to other units,
only the original characters are immediate constituents.

###  4.6 Statistical testing

####  4.6.1 Correlation

When measuring the association between two variables, we use both Pearson
correlation and Kendall correlation [5]. Note that the traditional view of
Pearson being a measure of linear association and Kendall correlation being a
measure of monotonic non-linear association has been challenged [25].

####  4.6.2 How to test for the law of abbreviation

We used a left-sided correlation test to verify the presence of the law of
abbreviation. In a purely exploratory or atheoretic exploration, one should
use a two-sided test. In an exploration guided by theory, namely regarding the
law of abbreviation as a manifestation of compression, the test should be
left-sided as theory predicts that $\tau(p,l)$ cannot be positive in case of
optimal coding [9].

####  4.6.3 How to test for compression

In the context of the null hypothesis of a random mapping of type
probabilities into type lengths, testing that compression (minimization of
$L$) has some effect on actual word lengths is easy because $L$ is a linear
function of $r$, the Pearson correlation between word length and word
probability (Appendix A). In particular, L = a r + L_r, where
$a=(n-1)s_{p}s_{l}$, being $n$ the number of types and $s_{p}$ and $s_{l}$,
respectively the standard deviation of type probabilities and type lengths. In
such random mappings, $L_{r}$, $s_{p}$ and $s_{l}$ remain constant and then
testing if $r$ is significantly small is equivalent to testing if $L$ is
significantly small (notice $a\geq 0$).

####  4.6.4 Controlling for multiple testing

When performing multiple correlation tests at the same time, it becomes easier
to reject the null hypothesis simply by chance. To address this problem we
used a Holm-Bonferroni correction to
p-values191919https://stat.ethz.ch/R-manual/R-devel/library/stats/html/p.adjust.html.
We applied the correction when checking the law of abbreviation in the
languages of a collection, so as to exclude the possibility that the law of
abbreviation is found many times simply because we are testing it in many
languages.

##  5 Results

In section 1, we highlighted the importance of distinguishing between direct
and indirect evidence of compression. Against this theoretical backdrop, here
we first investigate the presence of Zipf’s law of abbreviation in languages.
Then we investigate direct evidence of compression with the help of the new
random baseline.

###  5.1 The law of abbreviation revisited

We investigate the presence of the law of abbreviation by means of left-sided
correlation tests for the association between frequency and length. We use
both Kendall correlation, as suggested by theory on the origins of the law
[9], and Pearson’s. For each language, we show the significance level of the
relationship, color-coded by the value of the correlation coefficient. Figure
1 (a,b) indicates that the law holds in all languages – regardless of the
definition of word length – when Kendall $\tau$ correlation is used. In both
collections, we find Kendall $\tau$ correlation coefficients significant at
the 99% confidence level, except for Dhivehi in the CV collection when length
is measured in characters, and Abkhazian, Dhivehi, Panjabi and Vietnamese when
length is measured in duration. However, note that these are all still
significant at the 95% confidence level. When Pearson correlation is used
instead, Figure 1 (c) shows that the picture remains the same in PUD. The main
findings are the same also in CV (Figure 1 (d)), but when length is measured
in duration Panjabi ceases to be significant at the 95% confidence level.
Overall, we only fail to find the law of abbreviation in Panjabi given word
durations, and using Pearson correlation. This is certainly related to
undersampling, as this particular language only features 98 tokens (subsection
3.1).

(a)

(b)

(c)

(d)

Figure 1:  The correlation between frequency and length across languages.
’***’ indicates a Holm-Bonferroni corrected $p$-value lower than or equal to
0.01, ’**’ indicates lower than or equal to 0.05 but smaller than 0.1 and ’*’
indicates lower than or equal to 0.1. Here ’*’ symbols are not used to
indicate significance but p-value ranges. (a) Kendall $\tau$ correlation in
PUD (word length in characters). (b) Kendall $\tau$ correlation in CV (left:
word length in characters; right: word length in duration). (c) Same as (a)
with Pearson $r$ correlation. (d) Same as (b) with Pearson $r$ correlation.

###  5.2 Real word lengths versus the random baseline

We investigate the relationship between the actual mean word length ($L$) and
the random baseline ($L_{r}$). We find that $L<L_{r}$ for all languages in
every collection (Figure 2 and Tables B.2, B.2, B.2). Interestingly, there is
a large gap between $L$ and $L_{r}$ in the majority of languages, which is
more compelling in CV with word durations (Figure 2). Exceptions to the large
gap – as in the case of Panjabi and Abkhazian when length is measured in
duration – mainly concern languages with reduced sample sizes. The result
holds even when alternative units of measurement are considered for Chinese
and Japanese.

Figure 2 is reminiscent of Figure 4 of [20] but our setting is much simpler
(it only involves $L$ and $L_{r}$).

(a)

(b)

(c)

Figure 2:  Mean word length ($L$) as a function of the random baseline
($L_{r}$) in languages. Every point stands for a language. The diagonal (long
dashed line) indicates the line $L=L_{r}$. Languages with $L<L_{r}$ are
located below the diagonal. (a) Languages in PUD with word length measured in
characters (or strokes for Chinese and Japanese). (b) Languages in CV with
word length measured in characters. (c) Languages in CV with word length
measured in duration (seconds).

###  5.3 Impact of disabling the filter of words that contain “foreign”
characters

All results presented in this section have been obtained after applying the
new method to filter out highly unusual characters and words described in
subsection 4.4. If the filter is disabled, we obtain some slight changes in
the values, but the qualitative results summarized above remain the same.

##  6 Discussion

###  6.1 The universality of Zipf’s law

The first step of our analysis consisted in checking the universality of the
law of abbreviation in the languages of our samples through a Kendall $\tau$
correlation test. Here, we introduced two methodological improvements with
respect to previous research: using the Bonferroni-Holm correction for
$p$-values, as well as word length in time given spoken utterances, rather
than just characters in written form [2]. We also computed Pearson
correlations for two reasons: (a) to verify the robustness of the conclusions
and (b) to check the significance of the gap between $L$ and $L_{r}$ ((b) is
addressed in the next subsection). We find that the law of abbreviation holds
in nearly all languages in our sample at a 95% confidence level, independently
from how word length is measured, and even after controlling for multiple
testing. The only exception is Panjabi in CV, but only when length is measured
in duration and Pearson $r$ correlation is used. Panjabi is also the language
suffering most from under-sampling (only 98 tokens). Therefore, Panjabi cannot
be considered a true exception to the law of abbreviation.

Given the rather scarce evidence of the law of abbreviation in word durations
in human language [24], we have taken step forward by providing evidence of it
in 46 languages from 14 linguistic families. The massive agreement of the law
of abbreviation even when orthographic word lengths are replaced by word
durations in human languages provides stronger support for the law of
abbreviation as a potentially universal pattern of human languages with
respect to previous research relying on word length in characters [2] and
often on a small number of linguistic families [19, 15, 16, 14].

###  6.2 Direct evidence of compression

We have found that word lengths are shorter than expected by chance
($L<L_{r}$) in all languages in every collection (Figure 2). Such a systematic
finding is unlikely to be accidental and strongly indicates that compression
is acting in all languages in our sample. Crucially, the finding holds
independently of how word length is measured. The ample evidence of
compression even when orthographic word lengths are replaced by word durations
in human languages provides stronger support for compression as a universal
principle of the organization of languages with respect to previous research
relying on word length in characters [10].

It could be argued that these findings constitute evidence of compression in
ensembles of language but not in individual languages. The reason is that
$L<L_{r}$ does not imply that the difference between the actual word length
and the random baseline is statistically significant for a single language.
However, we have shown that the Pearson correlation is indeed a linear
function of $L$ and $L_{r}$ (Appendix A) and thus $L$ is significantly small
in every language where the law of abbreviation has been confirmed using a
Pearson correlation test.

Finally, the direct correspondence we have established between the average
length of types ($M$) and the random baseline sheds new light on previous
research. For instance, it has been shown that $M<L$ in Chinese characters in
six time periods spanning two millennia [4, Fig. 4], which now can be
reinterpreted as a sign of compression of word lengths in Chinese in light of
our theoretical findings.

### Future research

In this article, we have introduced a new random baseline and unveiled a
systematic gap between that random baseline and real mean word lengths that we
have interpreted as direct evidence of compression. Figure 2 suggests that the
gap is wider when word lengths are measured in duration rather than in
characters. However, we have not quantified the magnitude of that gap and we
have neither taken into consideration the gap between actual mean words
lengths and the minimum baseline, that would be defined as the minimum word
length that could be achieved under certain constraints [6, 9, 20]. Future
research should quantify the first gap in relation to the minimum baseline. As
the random baseline is crucial to asses the degree of optimality of word
lengths, we have paved the way for exploring the degree of optimality of word
lengths in characters or duration in languages.

## Authors’ contributions

SP: Conceptualization, Formal Analysis, Investigation, Software, Supervision,
Validation, Visualization, Writing-original draft, Writing-review & editing;
ACM: Data curation, Resources, Software; JCM: Writing-review & editing; MW:
Writing-review & editing, Software, Visualization; CB: Conceptualization,
Writing-review & editing; RFC: Conceptualization, Formal analysis, Funding
acquisition, Methods, Project Administration, Supervision, Writing-original
draft, Writing-review & editing.

## Acknowledgments

This article is one of the products of the research project of the 1st edition
of the master degree course “Introduction to Quantitative Linguistics” at
Universitat Politècnica de Catalunya. We are specially grateful to two
students of that course: L. Alemany-Puig for helpful discussions and
computational support, and M. Michaux for comments on early versions of the
manuscript. We thank M. Farrús and A. Hernández-Fernández for advice on voice
datasets. We also thank S. Komori for advice on Japanese, Y. M. Oh for advice
on Korean and S. Semple for helping us to improve English. Last but not least,
we are very grateful to an anonymous reviewer for very helpful comments.

SP is funded by the grant ’Thesis abroad 2021/2022’ from the University of
Milan. CB was partly funded by the Deutsche Forschungsgemeinschaft (FOR 2237:
Words, Bones, Genes, Tools - Tracking Linguistic, Cultural and Biological
Trajectories of the Human Past), and the Schweizerischer Nationalfonds zur
Förderung der Wissenschaftlichen Forschung (Non-randomness in Morphological
Diversity: A Computational Approach Based on Multilingual Corpora, 176305).

## References

  * [1] V.. Balasubrahmanyan and S. Naranan  “Quantitative Linguistics and Complex system Studies”  In _Journal of Quantitative Linguistics_ 3.3, 1996, pp. 177–228
  * [2] Christian Bentz and Ramon Ferrer-i-Cancho  “Zipf’s law of abbreviation as a language universal”  In _Proceedings of the Leiden Workshop on Capturing Phylogenetic Algorithms for Linguistics_ University of Tübingen, 2016
  * [3] C. Börstell, T. Hörberg and R. Östling  “Distribution and duration of signs and parts of speech in Swedish Sign Language”  In _Sign Language & Linguistics_ 19, 2016, pp. 143–196  DOI: 10.1075/sll.19.2.01bor
  * [4] Heng Chen, Junying Liang and Haitao Liu  “How Does Word Length Evolve in Written Chinese?”  In _PLOS ONE_ 10.9 Public Library of Science, 2015, pp. 1–12  DOI: 10.1371/journal.pone.0138567
  * [5] W.. Conover  “Practical nonparametric statistics” 3rd edition  New York: Wiley, 1999
  * [6] T.. Cover and J.. Thomas  “Elements of information theory” 2nd edition  New York: Wiley, 2006
  * [7] Weibing Deng, Armen E. Allahverdyan, Bo Li and Qiuping A. Wang  “Rank–frequency relation for Chinese characters”  In _The European Physical Journal B_ 87.2, 2014, pp. 47  DOI: 10.1140/epjb/e2014-40805-2
  * [8] N. Evans and S.. Levinson  “The myth of language universals: language diversity and its importance for cognitive science”  In _Behavioral and Brain Sciences_ 32, 2009, pp. 429–492  DOI: 10.1017/s0140525x0999094x
  * [9] R. Ferrer-i-Cancho, C. Bentz and C. Seguin  “Optimal coding and the origins of Zipfian laws”  In _Journal of Quantitative Linguistics_ 29.2, 2019, pp. 165–194  DOI: 10.1080/09296174.2020.1778387
  * [10] R. Ferrer-i-Cancho, A. Hernández-Fernández, D. Lusseau, G. Agoramoorthy, M.. Hsu and S. Semple  “Compression as a universal principle of animal behavior”  In _Cognitive Science_ 37.8, 2013, pp. 1565–1578  DOI: 10.1111/cogs.12061
  * [11] R. Heesen, C. Hobaiter, R. Ferrer-i-Cancho and S. Semple  “Linguistic laws in chimpanzee gestural communication”  In _Proceedings of the Royal Society B: Biological Sciences_ 286, 2019, pp. 20182900  DOI: 10.12775/3991-1.039
  * [12] Antoni Hernández-Fernández, Iván G., Juan-María Garrido and Lucas Lacasa  “Linguistic Laws in Speech: The Case of Catalan and Spanish”  In _Entropy_ 21.12, 2019  DOI: 10.3390/e21121153
  * [13] C.. Kello, G… Brown, R. Ferrer-i-Cancho, J.. Holden, K. Linkenkaer-Hansen, T. Rhodes and G.. Orden  “Scaling laws in cognitive sciences”  In _Trends in Cognitive Sciences_ 14.5, 2010, pp. 223–232  DOI: 10.1016/j.tics.2010.02.005
  * [14] Alexander Koplenig, Marc Kupietz and Sascha Wolfer  “Testing the Relationship between Word Length, Frequency, and Predictability Based on the German Reference Corpus”  In _Cognitive Science_ 46.6, 2022, pp. e13090  DOI: 10.1111/cogs.13090
  * [15] Natalia Levshina  “Frequency, Informativity and Word Length: Insights from Typologically Diverse Corpora”  In _Entropy_ 24.2, 2022  DOI: 10.3390/e24020280
  * [16] Stephan C. Meylan and Thomas L. Griffiths  “The Challenges of Large-Scale, Web-Based Language Datasets: Word Length and Predictability Revisited”  In _Cognitive Science_ 45.6, 2021, pp. e12983  DOI: 10.1111/cogs.12983
  * [17] Javier Moreno Fernández  “The optimality of word lengths”, 2021
  * [18] S. Naranan and V.. Balasubrahmanyan  “Information theoretic model for frequency distribution of words and speech sounds (phonemes) in language”  In _Journal of Scientific and Industrial Research_ 52, 1993, pp. 728–738
  * [19] S.. Piantadosi, H. Tily and E. Gibson  “Word lengths are optimized for efficient communication”  In _Proceedings of the National Academy of Sciences_ 108.9, 2011, pp. 3526–3529  DOI: 10.1073/pnas.1012551108
  * [20] Tiago Pimentel, Irene Nikkarinen, Kyle Mahowald, Ryan Cotterell and Damián Blasi  “How (Non-)Optimal is the Lexicon?”  In _North American Chapter of the Association for Computational Linguistics_ , 2021  DOI: 10.18653/v1/2021.naacl-main.350
  * [21] H. Sanada  “Investigations in Japanese historical lexicology”  Göttingen: Peust & Gutschmidt Verlag, 2008
  * [22] S. Semple, R. Ferrer-i-Cancho and M. Gustison  “Linguistic laws in biology”  In _Trends in Ecology and Evolution_ 37.1, 2022, pp. 53–66  DOI: 10.1016/j.tree.2021.08.012
  * [23] C.. Shannon  “A mathematical theory of communication”  In _Bell Systems Technical Journal_ 27, 1948, pp. 379–423 bibrangessep 623–656
  * [24] Iván G. Torre, Bartolo Luque, Lucas Lacasa, Christopher T. Kello and Antoni Hernández-Fernández  “On the physical origin of linguistic laws and lognormality in speech”  In _Royal Society Open Science_ 6.8, 2019, pp. 191023  DOI: 10.1098/rsos.191023
  * [25] Edwin Heuvel and Zhuozhao Zhan  “Myths About Linear and Monotonic Associations: Pearson’s $r$, Spearman’s $\rho$, and Kendall’s $\tau$”  In _The American Statistician_ 76.1 Taylor & Francis, 2022, pp. 44–52  DOI: 10.1080/00031305.2021.2004922
  * [26] Y. Wang and X. Chen  “Structural Complexity of Simplified Chinese Characters”  In _Recent Contributions to Quantitative Linguistics_ De Gruyter, 2015, pp. 229–239
  * [27] Stuart K. Watson, Raphaela Heesen, Daniela Hedwig, Martha M. Robbins and Simon W. Townsend  “An exploration of Menzerath’s law in wild mountain gorilla vocal sequences”  In _Biology Letters_ , 2020, pp. 162020038020200380  DOI: 10.1098/rsbl.2020.0380
  * [28] G.. Zipf  “Human behaviour and the principle of least effort”  Cambridge (MA), USA: Addison-Wesley, 1949  DOI: 10.2307/2226729

##  Appendix A Theory

Here we review the relationship between $L$, $L_{r}$ and Pearson correlation

Given two random variables $x$ and $y$ and a sample of $n$ points,
$\left\\{(x_{1},y_{1}),...,(x_{i},y_{i}),...,(x_{n},y_{n})\right\\}$, the
sample covariance is defined as s_xy = 1n-1(∑_i=1^n x_i y_i - n ¯x¯y), where
$\bar{x}$ is the sample mean of $x$ and $\bar{y}$ is the sample mean for $y$,
i.e.

$\displaystyle\bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_{i}$
$\displaystyle\bar{y}=\frac{1}{n}\sum_{i=1}^{n}y_{i}.$

Now consider than the random variables are $p$ (the probability of a type) and
$l$ (the length/duration of a type) instead of $x$ and $y$. Then our sample of
$n$ points is
$\left\\{(p_{1},l_{1}),...,(p_{i},l_{i}),...,(p_{n},l_{n})\right\\}$, one
point per type. Accordingly, the covariance between $p$ and $l$ in a sample of
points is s_pl = 1n-1(∑_i=1^n p_i l_i - n ¯p¯l). Recalling the definition of
$L$ (Equation 1) and noting that $\bar{p}=\frac{1}{n}$ and $\bar{l}=M=L_{r}$
(recall Property 2.1), we finally obtain s_pl = 1n-1(L - L_r).

The sample Pearson correlation is r = sxysxsy, where $s_{x}$ and $s_{y}$ are
the sample standard deviation of $x$ and $y$, i.e.

$\displaystyle s_{x}=\sqrt{\frac{1}{n-1}\sum_{i=1}^{n}(x_{i}-\bar{x})^{2}}$
$\displaystyle s_{y}=\sqrt{\frac{1}{n-1}\sum_{i=1}^{n}(y_{i}-\bar{y})^{2}}.$

Proceeding as we did for the covariance, we find that the Pearson correlation
between $p$ and $l$ is r = L - Lr(n-1)spsl. Then it is easy to see that $L$ is
a linear function of the Pearson correlation $r$ or $s_{pl}$. For instance, L
= ar + b, where

$\displaystyle a=(n-1)s_{p}s_{l}$ $\displaystyle b=L_{r}.$

Other linear relationships are left as an exercise.

##  Appendix B Analysis

We here present complementary analyses, tables and plots.

###  B.1 The impact of the unsupervised filter

subsection B.1 and subsection B.1 show the impact of the unsupervised filter
in the optional filter. PUD is a controlled setting for the impact of the
filter because it is a collection where tokens are of high quality compared to
CV. Thus we expect that the impact of the optional filter is low in PUD.
Unexpectedly, the number of tokens reduces substantially (a reduction of the
order of thousands) in Chinese, Japanese and Korean. An additional drastic
reduction in the observed alphabet size in these languages strongly suggests
that the optional filter is not adequate for them. For these reasons, we
believe we should not apply the unsupervised filter to these languages because
their writing system is essentially a syllabary. We suspect that the actual
need for the exclusion could be a combination of sampling problems relating to
a large alphabet size (compared to the Latin script) and a heavy- tailed rank
distribution that breaks the optional filter. It is well-known that the rank
distribution of Chinese characters is long-tailed, spanning two orders of
magnitude [7], while that of phonemes (the counterpart of letters in many
languages using the Latin script) is exponential-like [18, 1]. However, that
issue should be the subject of future research.

In CV, we find that the optional filter has a similar impact in languages
concerning the reduction in the number of tokens but higher impacts concerning
the reduction of the alphabet sizes, suggesting that presence of strings with
strange characters. The three languages with the most marked reduction in
alphabet size are French, Spanish, German and Italian, with an alphabet size
greater then 100.

Table B1:  The impact of the unsupervised filter in the PUD collection. For every language, we show its linguistic family, the writing system (namely script name according to ISO-15924) and various numeric parameters after applying the mandatory filter but before applying the unsupervised filter, that are $A$, the observed alphabet size (number of distinct characters), $n$, the number of types, and, $T$, the number of tokens. $A^{\prime}$, $n^{\prime}$ and $T^{\prime}$ are the respective values of $A$, $n$ and $T$ after applying the unsupervised filter.  Language | Script | Family | $A$ | $A^{\prime}$ | $n$ | $n^{\prime}$ | $T$ | $T^{\prime}$
---|---|---|---|---|---|---|---|---
Arabic | Arabic | Afro-Asiatic | 47 | 39 | 6600 | 6596 | 18214 | 18201
Indonesian | Latin | Austronesian | 39 | 23 | 4596 | 4501 | 16819 | 16702
Russian | Cyrillic | Indo-European | 61 | 31 | 7358 | 7113 | 15870 | 15588
Hindi | Devanagari | Indo-European | 84 | 50 | 4920 | 4716 | 21184 | 20796
Czech | Latin | Indo-European | 49 | 33 | 7360 | 7073 | 15700 | 15331
English | Latin | Indo-European | 39 | 25 | 5082 | 5001 | 18135 | 18028
French | Latin | Indo-European | 48 | 26 | 5593 | 5214 | 21084 | 20407
German | Latin | Indo-European | 39 | 28 | 6215 | 6116 | 18446 | 18331
Icelandic | Latin | Indo-European | 43 | 32 | 6175 | 6035 | 16385 | 16209
Italian | Latin | Indo-European | 42 | 24 | 5944 | 5606 | 21815 | 21266
Polish | Latin | Indo-European | 47 | 31 | 7329 | 7188 | 15386 | 15191
Portuguese | Latin | Indo-European | 47 | 38 | 5678 | 5661 | 21873 | 21855
Spanish | Latin | Indo-European | 39 | 32 | 5765 | 5750 | 21083 | 21067
Swedish | Latin | Indo-European | 39 | 25 | 5842 | 5624 | 16653 | 16378
Japanese | Japanese | Japonic | 1549 | 609 | 4990 | 3345 | 24899 | 22538
Japanese-strokes | Japanese | Japonic | 1549 | 609 | 4852 | 3345 | 24737 | 22538
Japanese-romaji | Latin | Japonic | 23 | 19 | 4984 | 4860 | 24892 | 24743
Korean | Hangul | Koreanic | 1002 | 401 | 8031 | 6424 | 14475 | 12540
Thai | Thai | Kra-Dai | 89 | 52 | 3818 | 3599 | 21642 | 21121
Chinese | Han (Traditional variant) | Sino-Tibetan | 2038 | 814 | 5224 | 3154 | 18129 | 15436
Chinese-strokes | Han (Traditional variant) | Sino-Tibetan | 2038 | 814 | 4970 | 3154 | 17845 | 15436
Chinese-pinyin | Latin | Sino-Tibetan | 49 | 44 | 5224 | 5038 | 18129 | 17885
Turkish | Latin | Turkic | 42 | 28 | 6793 | 6587 | 14092 | 13799
Finnish | Latin | Uralic | 39 | 24 | 7076 | 6938 | 12853 | 12701
Table B2:  The impact of the unsupervised filter in the CV collection. The
content is the same as in subsection B.1. ’Conlang’ stands for ’constructed
language’, that is an artificially created language. This is not a family in
the proper sense as Conlang languages are not related in the common linguistic
family sense.

###  B.2 Mean word length and the law of abbreviation

In subsection B.2, subsection B.2 and subsection B.2, we show the mean word
length ($L$) and the random baseline ($L_{r}$) as well as the outcome of the
correlation test between length and frequency for PUD and for CV when length
is measured in characters and also in duration, respectively.

Table B3:  Mean word length and the correlation between frequency and length
in PUD. Word length is measured in number of characters. Mean word length
($L$) is followed by the random baseline ($L_{r}$). Each correlation statistic
(Kendall $\tau$ or Pearson $r$) is followed by p-values after applying Holm-
Bonferroni correction (rather than being the direct output of the correlation
test).  Table B4:  Mean word length and the correlation between frequency and
length in CV. Word length is measured in number of characters. Content is the
same as in B.2. ’Conlang’ stands for ’constructed language’, that is an
artificially created language. This is not a family in the proper sense, and
Conlang languages are not related in the common family sense.  Table B5:  Mean
word length and the correlation between frequency and length in CV. Word
length is measured in duration. Content is the same as in B.2.