# Simplicity bias, algorithmic probability, and the random logistic map

Boumediene Hamzi1,2, Kamaludin Dingle3 1Department of Computing and
Mathematical Sciences, Caltech, CA, USA.  2The Alan Turing Institute, London,
UK<EMAIL_ADDRESS>3Centre for Applied Mathematics and
Bioinformatics, Department of Mathematics and Natural Sciences, Gulf
University for Science and Technology, Kuwait<EMAIL_ADDRESS>
###### Abstract.

_Simplicity bias_ is an intriguing phenomenon prevalent in various input-
output maps, characterized by a preference for simpler, more regular, or
symmetric outputs. Notably, these maps typically feature high-probability
outputs with simple patterns, whereas complex patterns are exponentially less
probable. This bias has been extensively examined and attributed to principles
derived from algorithmic information theory and algorithmic probability. In a
significant advancement, it has been demonstrated that the renowned logistic
map $x_{k+1}=\mu x_{k}(1-x_{k})$, a staple in dynamical systems theory, and
other one-dimensional maps exhibit simplicity bias when conceptualized as
input-output systems. Building upon this foundational work, our research
delves into the manifestations of simplicity bias within the random logistic
map, specifically focusing on scenarios involving additive noise. This
investigation is driven by the overarching goal of formulating a comprehensive
theory for the prediction and analysis of time series.

Our primary contributions are multifaceted. We discover that simplicity bias
is observable in the random logistic map for specific ranges of $\mu$ and
noise magnitudes. Additionally, we find that this bias persists even with the
introduction of small measurement noise, though it diminishes as noise levels
increase. Our studies also revisit the phenomenon of noise-induced chaos,
particularly when $\mu=3.83$, revealing its characteristics through
complexity-probability plots. Intriguingly, we employ the logistic map to
underscore a paradoxical aspect of data analysis: more data adhering to a
consistent trend can occasionally lead to reduced confidence in extrapolation
predictions, challenging conventional wisdom.

We propose that adopting a probability-complexity perspective in analyzing
dynamical systems could significantly enrich statistical learning theories
related to series prediction and analysis. This approach not only facilitates
a deeper understanding of simplicity bias and its implications but also paves
the way for novel methodologies in forecasting complex systems behavior,
especially in scenarios dominated by uncertainty and stochasticity.

_Keywords:_ Random dynamical systems; algorithmic probability; simplicity
bias; time series; machine learning

##  1\. Introduction

Dynamical systems appear in many fields including financial, engineering,
meteorological, medical, and other areas. Because of this, developing methods
to predict or understand the behavior and trajectories of dynamical systems is
an important area of applied mathematics research. Historically, such
predictions have been based on modeling the relevant system (e.g., with ODEs).
More recently, however, machine learning advances have given tools for the
prediction and analysis of dynamical systems. The machine learning approach
instead relies on directly extracting patterns from data rather than detailed
modeling of the underlying mechanics [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
13, 14, 15, 16, 17, 18, 19].

It follows that an interesting question is the extent to which machine
learning methods apply to dynamical systems analysis and prediction. Because
information theory and machine learning are very closely connected [20], this
motivates exploring the interface between information theory and dynamical
systems research. With this in mind, here we expand on the interface between
_algorithmic information theory_ [21, 22, 23] (AIT), and specifically
_algorithmic probability_ [21, 24, 25, 26], and dynamical systems. Our current
study builds on the work of Ref. [27], in which various 1D maps from chaos
theory were numerically examined from these perspectives.

Random dynamical systems have been investigated for some decades [28].
Initially, the field of dynamical systems began with works on deterministic
systems, but following this randomly perturbed dynamics became a focus area,
and indeed random dynamical systems are a natural way to model physical and
other systems (e.g., [29]). Random dynamical systems also provide a rich area
of mathematics for study, and they can yield behaviors that are qualitatively
different from deterministic dynamical systems, e.g. via noise-induced chaos
[30].

Numerous studies of _simplicity bias_ have been made in input-output maps, in
which a general inverse relationship between the complexity of outputs and
their respective probabilities has been observed [31, 32]. This work is based
on information theory and computation results and was derived from arguments
inspired by AIT, and can be viewed as an attempt to apply algorithmic
probability in real-world settings [31]. The simplicity bias upper bound [31]
gives a way to bound probability $P(x)$ of observing output pattern $x$ based
essentially only on estimating the Kolmogorov complexity of the pattern in
$x$. Simplicity bias analysis has been applied in many contexts, including
machine learning and deep neural networks [33, 34, 35, 36, 37, 38, 39].
Simplicity bias and algorithmic probability are closely related to Occam’s
razor [40], a fundamental basis of scientific reasoning and model selection,
that simpler theories/models should be preferred over more complex
theories/models provided they explain the data equally well.

The goal of this paper is to bridge the gap between machine learning and
algorithmic information theory using dynamical systems, particularly focusing
on the concept of simplicity bias within random logistic maps. We aim to
extend the understanding of simplicity bias, traditionally studied in
deterministic settings, to encompass the inherently stochastic nature of real-
world systems as modeled by random dynamical systems. Our approach is twofold:
first, to elucidate the manifestation of simplicity bias in the random
logistic map under various conditions and noise levels, thereby providing
insights into the behavior of these systems; second, to leverage these
insights to improve the theoretical foundations and practical applications of
machine learning in analyzing and predicting dynamical systems.

We posit that the intersection of dynamical systems, algorithmic information
theory, and machine learning represents a fertile ground for advancing our
understanding of complex systems. By studying the random logistic map—a
prototypical example in dynamical systems theory—we aim to uncover new
principles and methods that can inform both the development of machine
learning algorithms and the theoretical underpinnings of algorithmic
information theory. In doing so, we hope to provide a more robust framework
for series prediction and analysis, particularly in contexts where data
exhibit complex, chaotic, or noisy behavior.

Ultimately, this paper seeks to not only advance the theoretical discourse in
these domains but also to provide a robust set of tools and insights that can
be employed in practical applications. Whether it be in financial modeling,
climate prediction, or understanding biological systems, the implications of
this work aim to be far-reaching, contributing to better predictive models and
a deeper understanding of the simplicity and complexity inherent in natural
and artificial systems.

In this work, we extend the applications of simplicity bias to random
dynamical systems, and in particular the (digitized) trajectories of the
logistic map with additive noise [41]. Our work is part of a broader research
project into the relevance of simplicity bias and algorithmic probability to
dynamical systems and time series forecasting [42, 27, 43], and indeed to
mathematics and science more generally. Furthermore, this study should be
viewed in the same context as that presented in Ref. [44], where Martin-Löf
randomness was employed to examine Brownian motions, thereby demonstrating the
utility of Algorithmic Information Theory (AIT) tools in analyzing random
dynamical systems.

Most results in AIT, and the original algorithmic probability formulation, are
given in terms of binary strings. While AIT is not restricted only to
applications involving binary strings, applications involving binary strings
are more natural and arguably easier to implement. Hence, in this study, we
will digitize trajectories so as to study binary string trajectories (Cf. [45,
27, 43]). We begin by studying simplicity bias in the random logistic map and
then investigate the effects of adding measurement noise also. The following
section looks at complexity-probability graphs and simplicity bias in the
noise-induced regime of the logistic map. Finally, we look at the logistic map
in the context of the counter-intuitive claim of induction based on
algorithmic probability, that sometimes has more data, all of which follows
the same trend, can result in _less_ confidence in our ability to extrapolate
the trend.

##  2\. Background and problem set up

###  2.1. Background theory and pertinent results

####  2.1.1. AIT and Kolmogorov complexity

At the intersection of computer science and information theory lies
_algorithmic information theory_ [21, 22, 23] (AIT). AIT concerns the
information content of individual objects, such as binary strings and discrete
geometries [40]. The central quantity of AIT is _Kolmogorov complexity_ ,
$K(x)$, which quantifies the shortest length of a program to produce some
object $x$. Alternatively $K(x)$ can be thought of as the size in bits of the
compressed version of $x$ (assuming a perfect compressor). An object $x$ which
has a complex or random pattern is hard to compress, and so has a high
Kolmogorov complexity, while an object $x$ containing simple repeating
patterns like $x=01010\dots 10101$ is highly compressible and so has a low
Kolmogorov complexity value.

Because we will not be using or invoking detailed results from AIT, we will
not go into details here giving formal definitions and proofs. There are many
standard texts that the interested reader can refer to if needed, e.g., Refs.
[40, 46, 47, 48].

####  2.1.2. The coding theorem and algorithmic probability

An important result in AIT is the coding theorem [25], which establishes a
connection between $K(x)$ and probability prediction estimates. It states that

$P(x)=2^{-K(x)+O(1)}$ (1)

where $P(x)$ is the probability that an output $x$ is generated by a generic
computing device (technically a prefix optimal universal Turing machine) when
fed with a random binary program. Thus, high complexity outputs have
exponentially low probability, and simple outputs must have high probability.
$P(x)$ is also known as the _algorithmic probability_ of $x$ (see Ref. [26]
for an overview). AIT results are often difficult to apply directly in real-
world contexts, due to a number of issues including the fact that $K(x)$ is
formally uncomputable.

####  2.1.3. The simplicity bias bound

While the original theory regarding algorithmic probability is beautiful and
powerful, as mentioned, it is not easy to directly apply the theory to
practical real-world problems. However, several studies have tried to apply
numerical approximations of the theory to concrete problems [49, 42, 31]. Such
applications have led to the observation of a phenomenon called _simplicity
bias_ [31]. Simplicity bias has three levels of meaning, or we could say,
precision:

  * (i)

A loose general inverse relation between probability and complexity

  * (ii)

A scatter plot of complexity vs. probability showing a roughly linear upper
bound decay in log probability with increasing complexity

  * (iii)

The roughly linear upper bound in (ii) follows the simplicity bias bound (see
below) with $a\approx 1$

If any of these are observed in a complexity-probability scatter plot, then it
is considered as simplicity bias. The simplicity bias bound was presented in
Ref. [31], and has the form

$P(x)\leq 2^{-a\tilde{K}(x)-b}$ (2)

where $P(x)$ is the probability of observing output $x$ on random choice of
inputs, and $\tilde{K}(x)$ is the estimated Kolmogorov complexity of the
output $x$. The bound says that complex outputs from input-output maps have
lower probabilities, while high-probability outputs must be simpler. The
constants $a>0$ and $b$ can be fit with sampling and often even predicted
without recourse to sampling [31]. Typically we assume that $a\approx 1$,
which is the theoretical prediction. However, sometimes in practice, it can be
hard to correctly estimate $a$ due to insufficient sampling. We will also
assume throughout this work that $b=0$ in Eq. (2), which is a default
assumption as argued and discussed in Ref. [31].

There is also a conditional version of the simplicity bias equation [50].
Because Eq. (2) is an upper bound, it allows for output patterns to fall below
the bound. See Refs. [32, 51] for more on these types of patterns.

Simplicity bias and the related bound have been found to apply in many systems
including RNA shapes [31, 38], protein shapes [52, 50], models of plant growth
[31], ordinary differential equation solution profiles [31], finite state
transducers [32], natural time series [43], and others. In these systems, the
probability of different output shapes, upon a random sampling of inputs, was
non-trivially bounded and thereby predicted by Eq. (2).

The main motivations for studying simplicity bias are (a) uncovering a
‘universal’ property of input-output maps; (b) identifying a basic and general
mechanism for the appearance of simplicity and symmetry in nature; and (c)
exploring a fundamentally different way to predict the probability of output
shapes as compared to standard methods like frequency sampling. Simplicity
bias means that some predictions about output probabilities can be made
directly by estimating the complexity of outputs, independently of the
underlying system mechanisms and without having to make frequency counts.

###  2.2. Digitised map trajectories

AIT results, including algorithmic probability, are typically given in terms
of random inputs or ‘programs’ which are computed to produce output strings or
patterns. Further, these results of AIT are normally given in terms of binary
strings. While dynamical systems may not appear to fit this input-output
framework, they can be viewed as input-output functions if the initial value
and parameters are considered input ‘programs’ which are then computed to
generate ‘output’ dynamical system trajectories. As above, because output
pattern complexities and probabilities are more easily calculated if outputs
are binary strings, we will digitize the real-valued trajectories into binary
strings.

In Ref. [27] we explored simplicity bias in the standard deterministic
logistic map with equation

$x_{k+1}=\mu x_{k}(1-x_{k})$ (3)

where the inputs are the pair of values $x_{0}\in$(0.0, 1.0) and $\mu\in$
(0.0, 4.0], and $k=1,2,3,\dots,n$. Here, we will extend the analysis to the
logistic map with added noise, as studied by Sato et al. [53]. This random
logistic map has the form

$x_{k+1}=\mu x_{k}(1-x_{k})+\omega_{k}$ (4)

where $\omega_{k}$ is an i.i.d. random number sampled uniformly from
$[-\epsilon,\epsilon]$, so that $\epsilon$ determines the magnitude of the
noise.

The random logistic map can be viewed as an input-output map because the
values of $\mu$ and $x_{0}$ are like ‘inputs’ which are computed (in addition
to the noise) to produce a binary sequence trajectory ‘output’. In more
detail, the map is first iterated to obtain a sequence of $n$ real values
$x_{1},x_{2},x_{3},\dots,x_{n}$ in [0,1]. Similar to the field of symbolic
dynamics [54], the real-valued trajectory is digitized to become a binary
string output sequence $x$ by applying a threshold, writing 1 if $x_{k}\geq
0.5$ and 0 if $x_{k}<0.5$ [45, 55, 27]. Hence a binary sequence output $x$ of
$n$ bits is generated for each input pair $\mu$ and $x_{0}$, and random
realization of $\omega_{k}$ for $k=1,2,3,\dots,n$.

For example, some choice of $\mu$, $x_{0}$, and $\epsilon$ after iterating Eq.
(4) with $k=1,2,...,25$ might yield the real-valued trajectory

$x_{1},x_{2},\dots,x_{25}=0.12,0.47,0.66,\dots,0.21,0.05,0.78,0.97$

which after digitization becomes the binary string

$x=001\dots 0011$ (5)

Following Ref. [27], we will use $n=25$, but see the same reference for the
impact of larger are smaller $n$. In brief, $n$ should be large enough to
yield many different patterns and complexity values, but small enough so that
decent frequency and probability estimates can be made for up to $2^{n}$
different strings without the need for excessive sampling.

Simulations will proceed by fixing some choice of $\mu$ and $\epsilon$, and
then randomly sampling $x_{0}\in(0.0,1.0)$, and iterating for $n=25$ steps.
Each sample will yield a binary string $x\in\\{0,1\\}^{n}$, i.e., a binary
string of length $n$. By counting the relative frequency of strings $x$, we
can find $P(x)$, the probability of each string $x$. The estimated complexity
$\tilde{K}(x)$ of each string can be calculated using a compression-based
metric, based on the Lempel-Ziv 1976 complexity measure [56] (see Appendix for
more details on the complexity measure). Using the values of $P(x)$ and
$\tilde{K}(x)$ for each binary string, we make a scatter plot of these
quantities and examine the connection (if any) between complexity and
probability.

##  3\. Results

###  3.1. Simplicity bias in the random logistic map

In this study, we are primarily interested in values of $\mu$-values for which
the trajectories of the deterministic map converge quite quickly to fairly
trivial fixed points or periods. If instead $\mu\approx 4.0$ then it is known
that there is no simplicity bias in trajectories [27], so adding noise will
only serve to increase the complexity and variability of the trajectories,
making any kind of bias and therefore simplicity bias, not appear. Note that
we use the word ‘bias’ to describe a strongly non-uniform (low entropy)
probability distribution, such that certain binary string outputs are much
more likely the others.

It is known that if $\mu\in[0.0,1.0]$, then the trajectories of the
deterministic logistic map tend to 0 [57]. Because we are adding noise to the
trajectories, the trajectories will be prevented from becoming arbitrarily
close to zero. If the noise is small enough, then $x_{k}\approx 0<0.5$ so that
the resulting digitized binary string (after truncating at 0.5) will be
$x=...0000$. If the noise is larger, then it may be that $x_{k}>0.5$ for some
$k$, so that more varied patterns including 1s can emerge for $x$. For
$\mu\in(1.0,3.0]$ the trajectories of the deterministic logistic map tend to a
fixed point with value $1-(1/\mu)$ [58]. If $\epsilon=0.0$, then for any
initial value of $x_{0}$ the trajectories will tend to be either $x=\dots 000$
or $x=\dots 111$ depending on $\mu$, which are trivial patterns. Not only are
they trivial, they are not varied enough to have different complexity and
probability values, and so the corresponding complexity-probability plot is
trivial. Therefore, simplicity bias cannot emerge in this scenario, at least
if we ignore the initial transient dynamics (but see also Ref. [27] for more
on this). However, if we add noise then different and non-trivial patterns can
emerge.

Figure 1. Simplicity bias in the digitized random logistic map. Each blue data
point corresponds to a different binary digitized trajectory $x$ of length 25
bits. The black line is the upper bound of Eq. (2). (a) $\mu=3.0$ and
$\epsilon=0.125$ including transient dynamics; (b) $\mu=1.0$ and
$\epsilon=0.375$ including transient dynamics; and (c) $\mu=1.0$ and
$\epsilon=0.375$ excluding transient dynamics.

The parameter values used in the numerical experiments are $\mu=$ 0.0, 1.0,
2.0, and 3.0 in combination with $\epsilon=$ 0.125, 0.25, 0.375, and 0.5. For
many of these parameter combinations, we can expect no simplicity bias in the
binary trajectories, due to the following: If the noise $\epsilon$ is large,
then the trajectory is essentially a random erratic pattern across the
interval [0,1] with little correlation between adjacent bits, yielding a
roughly uniform distribution. Therefore, there is unlikely to be bias for some
specific patterns, and consequently no bias for simple patterns. On the other
hand, if $\epsilon$ is very small, then the ‘kicks’ imposed by the noise will
not be large enough to prevent the trajectories from converging. As examples,
if $\mu=1.5$ and $\epsilon\approx 0$, then the trajectory values $x_{k}$ will
fluctuate around the value $1-(1/\mu)=0.33<0.5$, yielding a binary trajectory
$x=...0000$. If $\mu=2.0$, then the fixed point would be $1-(1/\mu)=0.5$, so
even small fluctuations around this value would yield almost random binary
strings $x$. From this reasoning, we can conclude that to observe simplicity
bias in the random logistic map requires having a balance of noise that is not
too large as to overly randomize the trajectories, and not too small as to
allow the trajectories to converge to a fixed point with only modest
fluctuations around $1-(1/\mu)$.

Turning to numerical experiments, we make complexity-probability plots for
different values of $\mu$ and noise magnitude values $\epsilon$. We separately
analyze the cases of including initial transient dynamics before the
trajectories settle, and ignoring the transient dynamics by excluding the
first 50 iterations of the map. Ignoring the initial iterations is common
practice in the mathematical physics community [58]. However, from the machine
learning perspective, the initial transient dynamics where a larger fraction
of the space is sampled is more important for statistical learning. So we will
consider both regimes.

Figure 1 shows some examples of simplicity bias. The different binary string
outputs that appeared in sampling are shown as blue dots. The black line bound
of Eq. (2) is also depicted. Figure 1(a) shows $\mu=3.0$ and $\epsilon=0.125$,
and the initial dynamics are included. There is simplicity bias because we see
that some low complexity patterns have high probability. Further, we see a
roughly linear upper bound decay in the probabilities, as complexity
$\tilde{K}(x)$ increases. However, the gradient appears to be less steep than
the $a=1.0$ default prediction suggests. The reason that the slope is not
accurate may be due to undersampling because estimating the slope $a$ assumes
that nearly all of the possible strings have appeared in the sample, which may
be overly coarse. Another reason may be due to the introduction of random
noise (see next section for more on this). Despite these considerations, it is
still clear that simplicity bias is observed. Without the additive noise, the
trajectory would have converged to $1-1/\mu=0.67$ and shown no simplicity
bias.

In Figure 1(b) we see data for $\mu=1.0$ and $\epsilon=0.375$, and the initial
dynamics are included. Again there is a clear simplicity bias. Without the
additive noise, the trajectory would have converged to $1-1/\mu=0.0$ and shown
no simplicity bias. In addition to the inverse relation between probability
and complexity, even the gradient of the decay of the upper bound is not
strongly divergent from the black line default prediction. In Figure 1(c) we
see data again for $\mu=1.0$ and $\epsilon=0.375$, but this time after
ignoring the first 50 iterations (to exclude the transient dynamics). Here we
see simplicity bias in the form of an inverse relation between probability and
complexity. Having said that, the simplicity bias in panel (c) is not as
pronounced as in panel (b).

For each complexity value, we see outputs with a range of probabilities. That
is, there are some patterns that are simple, yet have very low probability
values. These low complexity, low probability outputs have been explored in
detail earlier [31, 51]. However, it is known that these low complexity, low
probability outputs collectively do not account for much of the probability
mass [32].

In conclusion, we see that for some parameter values, simplicity bias is
observed in the random logistic map trajectories, even while the deterministic
counterparts with the same $\mu$ values would have converged to fixed points
and not shown simplicity bias.

Figure 2. Simplicity bias with measurement noise. In (a), a very small amount
of measurement noise ($\delta=0.01$) is added, and simplicity bias is clear.
In (b) a larger amount of noise is added ($\delta=0.17$), and simplicity bias
remains. In (c) a larger value of noise is added ($\delta=0.45$), and the
slope becomes much less steep, thus weakening the probability-complexity
connection.

###  3.2. Adding noise to the strings

There are different ways in which noise can be included in the modeling of the
random dynamical system. Above, we included additive noise in the dynamics
themselves, to model the scenario in which the dynamics themselves are not
deterministic but perturbed stochastically. In this section, we now consider
another type of noise, measurement noise, which models the scenario that when
measuring the trajectory, we do so inaccurately.

It is important to study if simplicity bias is maintained under different
types of perturbation, because if it does not, then it may not be a physically
relevant body of theory. This is related to arguments supporting the study of
random dynamical systems, over deterministic systems, because some results
from chaos theory disappear in the presence of even small noise, suggesting
that they are not relevant to the study of physical dynamical systems (which
are inherently stochastic, typically).

We introduce measurement noise as follows: If $x_{1}x_{2}x_{3}\dots x_{n}$ is
a trajectory of real-values, then we replace each $x_{k}$ with
$x_{k}+\delta_{k}$, where $\delta_{k}$ is some small uniform noise in
$[-\delta,\delta]$. After adding the noise, we use the 0.5 threshold as above
and digitize the noisy trajectory. We can then, as above, make a complexity-
probability plot and look for simplicity bias.

In Figure 2 we see the effects of adding measurement noise. For illustration
purposes, we use the same values of $\mu$ and $\epsilon$ as in Figure 1, and
increasing values of $\delta$. In Figure 2(a), a very small noise level of
$\delta=0.01$ is used, and hence the simplicity bias is still clearly
observed. In (b), a larger noise level of $\delta=0.17$ is used, and we see
that the simplicity bias still remains, but the slope of decay in log
probability with increasing complexity is slightly less steep as compared to
(a). Finally, in (c), a larger noise level of $\delta=0.45$ is used, and the
bias and simplicity bias has almost completely disappeared with only a modest
slope decay in log probability. In sum, the simplicity bias gradually
disappears with increasing $\delta$ via the slope gradually becoming more
horizontal.

It is noteworthy that in some earlier studies of simplicity bias, the slope of
the decay in log probability was observed to be considerably less steep than
the upper bound of Eq. (2) but it was not stated why this was the case [43,
32]. The results here suggest a simple explanation for this observation, i.e.,
noise (in some form) corrupts the simplicity bias, adjusting the slope.

###  3.3. Noise-induced chaos around $\mu=3.83$

For a large fraction of values of $\mu\geq 3.56994567...\approx 3.57$
(https://oeis.org/A098587), the trajectories of the deterministic logistic map
are chaotic [57]. However, there are also islands of stability for some
$\mu\in(3.56994567...,4.0]$, in which the dynamics are simple and periodic.
One example is when $\mu=3.83$, which has an attracting period-3 trajectory in
the deterministic case. It has been known for many years, however, that with a
small amount of noise added to the random logistic map, noise-induced chaos
can appear [30, 59]. That is, with $\mu=3.83$ it is known that
$\epsilon\approx 0.00146$ is the critical noise amplitude at which the chaotic
attractor appears. At this value, the Lyapunov exponent becomes positive.

We will study this noise-induced chaos in terms of simplicity bias. Above, we
saw that simple fixed point dynamics combined with additive noise can lead to
simplicity bias in the trajectories. This suggests that the simple period-3
trajectory when $\mu=3.83$ may show simplicity bias when combined with
additive noise of $\epsilon=0.00146$. In contrast, for the chaotic regime, the
logistic map does not exhibit bias, nor simplicity bias [27], because no
particular patterns are much more likely (and therefore no simple patterns,
either). Therefore, we will also examine $\epsilon=0.00146$ combined with
$\mu=3.82$ and $\mu=3.84$ which potentially will not show simplicity bias,
assuming that the dynamics are chaotic for these values.

We perform numerical experiments as above, digitizing the trajectories. Figure
3(a) shows the complexity-probability plot for $\mu=3.82$, and there is no
evidence of simplicity bias, as expected. There is some bias in the
distribution, as inferred from the fact that some binary strings have
probability $\approx 10^{-3}$, while some others have probability $\approx
10^{-5}$. However, the connection to complexity is very weak. In (b) however,
with $\mu=3.83$ there is clear evidence of simplicity bias, with a linear
upper-bound decay in log probability with increasing complexity. The slope of
the decay is not the same as the black line bound, but somewhat less steep. In
Figure 3(c), interestingly with $\mu=3.84$ there is still simplicity bias, but
not quite as clear as in (b) as evidenced by the less clearly linear upper
bound.

While there are already other ways to detect and measure noise-induced chaos
[30], this perspective of simplicity bias also provides another perspective.

Figure 3. Complexity-probability plots for $\mu\approx 3.83$. (a) With
$\mu=3.82$ there is no simplicity bias; (b) with $\mu=3.83$ there is clear
simplicity bias with a roughly linear upper bound decay; and (c) with
$\mu=3.84$ there is still simplicity bias, but the linear upper bound decay is
more noisy than with $\mu=3.83$.

###  3.4. More data does not always give more confidence in predictions

In the preceding sections, we looked at how simplicity bias, which is a form
of algorithmic probability, can be used to predict or bound the probability of
patterns in time series from the logistic map. Another application of
algorithmic probability, and in fact one of the original motivations for its
development, is for inference or induction of series. Solomonoff [21, 24] was
interested in questions like, given a (finite) binary series $x$ representing
a historical series, what is the best prediction for the next bit in the
series? More quantitatively, what is the probability that the next bit is a
zero, $P(0|x)$, and what is the probability the next bit is a 1, $P(1|x)$? If
the underlying process generating the series is known, e.g. it is a Bernoulli
process, then standard statistical theory can give methods to estimate the
probabilities of $P(0|x)$ and $P(1|x)$. In many practical situations, however,
the underlying process is not known, and a central goal of the study of time
series is forecasting using just the series itself. So, is there a general way
to predict time series, which does not require (strong) assumptions or
knowledge about the process that made the series? Solomonoff induction [40] is
the method proposed by AIT, which is essentially prediction based on
algorithmic probability.

Hutter [60] has also argued that algorithmic probability should be central to
artificial intelligence and the theory of machine learning. The main problem
with algorithmic probability, and hence Solomonoff induction, is that it is
formally uncomputable [40], so cannot be applied directly to in practice (or
at least not straightforwardly). Indeed, some have argued that this makes the
theory almost useless [61]. However, developing the theory is useful because
some aspects can shed light on real-world problems, and some of the theory can
be approximated and thereby applied.

Here we will consider one rather surprising result from Solomonoff induction,
which is that sometimes with _more_ data — even if the data points all conform
to one trend — we should be _less_ confident in our ability to predict the
series (see Li and Vitanyi [40] for a mention of this result, and more in
Hutter [62]). We will see how this result can apply to the logistic map.

Suppose we observe some binary series, which happens to be just 1 repeated $n$
times, i.e., $1^{n}=$ 111…111. What should be our prediction for the next bit?
And how confident should we be of our prediction? Laplace famously argued that
if there are $k$ 1s out of $n$ binary observations, then we should predict
that the next bit is 1 with probability

$P(1|111...111)=\frac{k+1}{n+2}=\frac{n+1}{n+2}=1-\frac{1}{n+2}\approx 1-1/n$
(6)

where we have set $k=n$ because all observations are ‘1’. According to Laplace
then, if we observe more data (larger $n$) all of which conforms to the same
trend (all 1s), then we should have greater confidence that the next bit will
also be 1 (because $1-1/n$ is closer to one). This reasoning also accords with
common-sense statistics. However, using a general form of inference based on
Solomonoff induction and algorithmic probability, we do not always arrive at
the same conclusion.

Solomonoff induction (or prediction) is based on imagining that a computer has
been fed a random input program, and it produces an infinite binary string
output. We observe the first $n$ bits of the output as a binary string $x$,
and we try to predict the probability that the next bit is a 0 or a 1.
Technically this computer should be a monotone universal Turing machine, but
we will not explore the details of this because they are not relevant to our
example, see [26, 62, 40] for more details. We will write $M(x)$ for the
probability that the infinite string begins with $x$, and write the
conditional probability as

$M(x_{n+1}|x_{1}x_{2}x_{3}\dots x_{n})=\frac{M(x_{1}x_{2}x_{3}\dots
x_{n}x_{n+1})}{M(x_{1}x_{2}x_{3}\dots x_{n})}$ (7)

Essentially, $M(x_{n+1}|x_{1}x_{2}x_{3}\dots x_{n})$ gives the conditional
probability prediction based on compressing the sequence $x_{1}x_{2}x_{3}\dots
x_{n}x_{n+1}$, and assigns the predicted probabilities of
$M(0|x_{1}x_{2}x_{3}\dots x_{n})$ and $M(1|x_{1}x_{2}x_{3}\dots x_{n})$ based
on how well the following bit (0 or 1) fits the preceding sequence, as
measured by how well the following bit (0 or 1) compresses the sequence.

Using algorithmic probability now as a method for induction or prediction, the
probability that the next bit in the series after observing $x=111\dots 111$
is 1 is given by [62]

$M(1|111...111)=1-2^{-K(n)+O(1)}$ (8)

For most $n$, or typical $n$, it is known that the complexity of an integer
$n$ is given by $K(n)\approx\log_{2}(n)$ [40], so that

$M(1|111...111)=1-2^{-K(n)+O(1)}\sim 1-2^{-\log_{2}(n)}=1-1/n$ (9)

roughly agreeing with Laplace.

A difference in predictions between the Laplacian and algorithmic probability
approaches can arise when certain values of $n$ are encountered, which are
‘special’, and therefore (algorithmically) ‘simple’. Such values of $n$ are
those that can be described succinctly and unambiguously. For example,
$n=2^{2^{2^{2^{2}}}}\approx 2\times 10^{19728}$, meaning that if the integer
$n$ is written out in full it has nearly 20,000 digits, while at the same
time, we can describe it very succinctly in just a few characters. Therefore
we say that $n$ is simple, and the Kolmogorov complexity of this value $n$ is
low. As another example of ‘special’ or ‘simple’ integers, the factorial
$n=(8!)!\approx 3\times 10^{168186}$, which has nearly 170,000 digits if
written out explicitly in full, yet can be described exactly in just a few
characters. There are many ‘special’ and algorithmically simple numbers, but
nonetheless, they are rare in the sense that a vanishingly small fraction of
integers are special or simple [40].

For such ‘special’ or ‘simple’ numbers $n$, the Kolmogorov complexity is
small, or we can say $O(1)$ bits, to represent just a few bits. In other
words, for a simple $n$ which has $K(n)\ll\log_{2}(n)$, the predicted
probability for the next bit is

$M(1|111...111)\approx 1-2^{-K(n)+O(1)}<1-1/n$ (10)

which can be substantially different from Laplace’s estimate. This is not to
say that algorithmic probability would predict a 0 in place of a 1 as the next
bit, but instead, it means that the prediction that the next bit is 1 — for
this $n$ — is less confident (further from 1) than it would be for a typical
$n$. This is a kind of quantification of the intuition that typically we do
not expect there to be a change in a series after a certain pattern has been
followed for a long time (e.g., 1s to change to 0s), but on ‘special
occasions’ when $n$ is somehow special, then it is more likely that something
will ‘happen’ (i.e., a change).

Rather surprisingly, what this means also is that for some few special values
of $n$, more data does not necessarily mean higher confidence in our
predictions: consider $n$ is some large random integer such that
$K(n)\approx\log_{2}(n)$, while $n^{\prime}$ is some other simple integer such
that $n^{\prime}>n$ but at the same time

$K(n^{\prime})\ll\log_{2}(n)<\log_{2}(n^{\prime})\Rightarrow
M(1|1^{n})>M(1|1^{n^{\prime}})$ (11)

Thus, even though we have observed $n^{\prime}-n$ more 1s repeated, we are
_less_ confident in our prediction that the next bit is 1, because
$M(1|1^{n})$ is closer to 1 than $M(1|1^{n^{\prime}})$.

We will now consider how this somewhat counterintuitive result from
algorithmic probability theory applies to the logistic map. We will construct
an example of when the trend of just 0s changes after a long time and becomes
a string of 1s.

If $\mu=2.5$, $\epsilon=0.0$, $(x_{0})^{-1}=2^{2^{2^{2^{2}}}}$ so that
$x_{0}\approx 10^{-19728}$, then this series begins with $x_{0}<0.5$. Because
$\mu=2.5$, the fixed point which the trajectory approaches is
$1-1/\mu=1-1/2.5=0.6>0.5$. Therefore, eventually, the series will change from
0s to 1s. Call $n^{*}$ the integer at which the series transitions from 0s to
1s, i.e. the series is $0000...011111$.

Because $x_{0}\approx 0$ and $\epsilon=0.0$, we have $(1-x_{k})\approx 1$ so
that the logistic equation Eq. (3) simplifies to $x_{k+1}\approx 2.5x_{k}$, or
$x_{k+1}\approx 2.5^{k}x_{0}$, meaning that when $x_{k}\approx 0$ the terms
grow exponentially fast initially, but slows as $x_{k}$ gets larger. Even with
exponential growth, $n^{*}$ must be a large number because $x_{0}$ is so
small, indeed because $10^{19728}\approx 2.5^{49575}$, the number of
iterations $n^{*}$ required to reach the 0.5 threshold is at least of the
order of 50,000.

So, having observed a series of $n^{*}$ 0s, $x=0^{n^{*}}$, according to
Laplace’s induction method, the probability that the next bit is also 0 will
be around

$P(0|000...000)=\frac{k+1}{n+2}=\frac{n^{*}+1}{n^{*}+2}\approx
1-1/n^{*}>1-1/50000=0.99998$ (12)

so that $P(1|000...000)<0.00002$. Therefore, the sudden change in the logistic
map from 0s to 1s at $n^{*}$ would come as rather a ‘surprise’ for Laplace.

In contrast, using algorithmic probability we have the following reasoning:
Because $x_{0}$ is an algorithmically simple (small) number, and $\mu$ is a
simple number also, and the logistic equation Eq. (3) can be written in only a
few characters also, and because the value of $n^{*}$ can be derived via a
simple function of these values (e.g. by iterating the logistic map to find
the value of $n^{*}$), then this implies that $n^{*}$ is a special number and
$K(n^{*})$ is small. Therefore we have

$M(0|000...000)=1-2^{-K(n^{*})+O(1)}<1-1/n^{*}$ (13)

so that $M(1|000...000)>1/n^{*}$, so that the change from 0s to 1s is less of
a ‘surprise’ if using the algorithmic probability approach.

Note that the algorithmic probability estimate does not give an exact figure
for the probability of changing to 1s after the 0s (which would depend on the
specific computer or language used), but it does reveal that the probability
prediction can be significantly different when $n$ is simple compared to
typical values of $n$. See the Appendix for an example that does not depend on
the specific computer or language used.

##  4\. Discussion

Arguments inspired by algorithmic information theory (AIT) have been used to
predict the occurrence of _simplicity bias_ in many real-world input-output
maps, where complex patterns have exponentially low probabilities, and high
probability patterns are simple [31]. This phenomenon has been observed in a
very wide range of systems, including RNA shapes [31, 38], protein shapes [52,
50], models of plant growth [31], ordinary differential equation solution
profiles [31], finite state transducer [32], natural time series [43], deep
neural networks [33], 1D dynamical systems [27], and others. In this work, we
have numerically investigated the presence of simplicity bias in digitized
trajectories of the random logistic map. This current work sits in the context
of a wider research program of investigating the ‘universal’ presence of
simplicity bias in input-output maps, which can aid in an understanding of the
origins of symmetry and simplicity in nature, and also serve the practical aim
of aiding _a priori_ probability predictions, based on directly measuring the
complexity of patterns (instead of, e.g. sampling).

Our main conclusions are that (i) we observe simplicity bias in the random
logistic map for some restricted set of parameter values; (ii) adding
measurement noise to trajectories gradually erases simplicity bias by
flattening the slope in complexity-probability plots; (iii) the regime of
noise-induced chaos when $\mu=3.83$ shows-up in complexity-probability plots,
and (iv) algorithmic probability-based induction shows how sometimes in the
logistic map trajectories, counterintuitively more data does not improve our
confidence in extrapolating the trend.

The presence of a bias towards simplicity in this example of the logistic map
with additive noise could be explained in a mechanistic fashion, and indeed
many of the other examples of simplicity bias may admit mechanistic
explanations based on the details of the specific way inputs are assigned to
outputs. Despite this, we propose that it is beneficial to explain the
observed simplicity bias from another perspective entirely, that of
information theory and the simplicity bias bound Eq. (2). This latter
perspective forms a unifying theory to unite the disparate observations in
different maps. These different types of explanations are complementary.

It is noteworthy that digitized logistic map trajectories have been studied in
terms of Lempel-Ziv complexity by Kaspar and Schuster [55], and in this sense
is related to our work. Despite this, their work is fundamentally different
from ours because it is not concerned with simplicity bias or in estimating
the probability of different outputs.

The theory of algorithmic probability was developed several decades ago [21,
24, 63, 25], and is important and potentially widely applicable in science and
mathematics [40, 60, 64, 26, 65]. Applications of the field have been held
back, partly at least, by the difficulty of applying the theory due to the
incomputability of important quantities. However, it has been argued that
algorithmic probability predictions should “serve as ‘gold standards’ that
practitioners should aim at … [but which] have to be (crudely) approximated in
practice” [62]. The simplicity bias bound, including the results here, can be
seen in this light. Further, many other researchers have built on and applied
the compression-learning-prediction link [66]. Indeed, very recently it has
been argued that even the currently highly influential “large language models
[e.g., ChatGPT] are powerful general-purpose predictors and that the
compression viewpoint provides novel insights” [67].

In future work, we suggest continuing the application of (approximations to)
algorithmic probability to develop novel approaches to probability predictions
and advance statistical and machine learning methods.

##  5\. Conclusion

In this study, we’ve demonstrated the presence of simplicity bias within the
random logistic map, affirming its role across a range of dynamic systems. We
observed that simplicity bias persists within specific parameter regimes but
is gradually diminished by measurement noise, particularly within noise-
induced chaos regimes. These findings not only reinforce the intricacies of
dynamical systems but also emphasize the counterintuitive nature of
algorithmic probability-based predictions.

Our research underscores the value of interpreting simplicity bias through an
information-theoretic lens, offering a unifying framework that transcends
specific system mechanics. This perspective aligns with broader efforts to
employ algorithmic probability as a theoretical standard in understanding
complex systems, despite its practical challenges.

As we look forward, the application of algorithmic probability and related
concepts remains a promising frontier for advancing prediction and analysis
methods in various fields. By continuing to bridge theory with practical
applications, we aim to deepen our understanding of the principles that govern
simplicity and complexity in the natural world.

Acknowledgments: BH thanks Prof. Jeroen Lamb (Imperial College London) for
useful discussions about random dynamical systems that inspired the work in
this paper. KD thanks Muhammad Alaskandarani for useful discussions on the
early parts of this work. This work has been partially supported by the Gulf
University for Science and Technology, including by project code: ISG Case 9.

## References

  * [1] R. González-García, R. Rico-Martínez, and I.G. Kevrekidis.  Identification of distributed parameter systems: A neural net based approach.  Computers & Chemical Engineering, 22:S965–S968, 1998.  European Symposium on Computer Aided Process Engineering-8.
  * [2] Ashesh Chattopadhyay, Pedram Hassanzadeh, Krishna V. Palem, and Devika Subramanian.  Data-driven prediction of a multi-scale lorenz 96 chaotic system using a hierarchy of deep learning methods: Reservoir computing, ann, and RNN-LSTM.  CoRR, abs/1906.08829, 2019.
  * [3] Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz.  Discovering governing equations from data by sparse identification of nonlinear dynamical systems.  Proceedings of the National Academy of Sciences, 113(15):3932–3937, 2016.
  * [4] Jaideep Pathak, Zhixin Lu, Brian R. Hunt, Michelle Girvan, and Edward Ott.  Using machine learning to replicate chaotic attractors and calculate lyapunov exponents from data.  Chaos: An Interdisciplinary Journal of Nonlinear Science, 27(12):121102, 2017.
  * [5] A. Nielsen.  Practical Time Series Analysis: Prediction with Statistics and Machine Learning.  O’Reilly Media, 2019.
  * [6] Alan A. Kaptanoglu, Kyle D. Morgan, Chris J. Hansen, and Steven L. Brunton.  Physics-constrained, low-dimensional models for magnetohydrodynamics: First-principles and data-driven approaches.  Physical Review E, 104(1):015206, July 2021.
  * [7] J. Nathan Kutz and Steven L. Brunton.  Parsimony as the ultimate regularizer for physics-informed machine learning.  Nonlinear Dynamics, January 2022.
  * [8] B.Haasdonk ,B.Hamzi , G.Santin , D.Wittwar.  Kernel methods for center manifold approximation and a weak data-based version of the center manifold theorems.  Physica D, 2021.
  * [9] P. Giesl, B. Hamzi, M. Rasmussen, and K. Webster.  Approximation of Lyapunov functions from noisy data.  Journal of Computational Dynamics, 2019.  https://arxiv.org/abs/1601.01568.
  * [10] Boumediene Hamzi and Houman Owhadi.  Learning dynamical systems from data: A simple cross-validation perspective, part i: Parametric kernel flows.  Physica D: Nonlinear Phenomena, 421:132817, 2021.
  * [11] Boumediene Hamzi and Fritz Colonius.  Kernel methods for the approximation of discrete-time linear autonomous and control systems.  SN Applied Sciences, 1(7):674, July 2019.
  * [12] Stefan Klus, Feliks Nuske, and Boumediene Hamzi.  Kernel-based approximation of the koopman generator and schrödinger operator.  Entropy, 22, 2020.  https://www.mdpi.com/1099-4300/22/7/722.
  * [13] Stefan Klus, Feliks Nüske, Sebastian Peitz, Jan-Hendrik Niemann, Cecilia Clementi, and Christof Schütte.  Data-driven approximation of the koopman generator: Model reduction, system identification, and control.  Physica D: Nonlinear Phenomena, 406:132416, 2020.
  * [14] Romeo Alexander and Dimitrios Giannakis.  Operator-theoretic framework for forecasting nonlinear time series with kernel analog techniques.  Physica D: Nonlinear Phenomena, 409:132520, 2020.
  * [15] Andreas Bittracher, Stefan Klus, Boumediene Hamzi, Peter Koltai, and Christof Schutte.  Dimensionality reduction of complex metastable systems via kernel embeddings of transition manifold, 2019.  https://arxiv.org/abs/1904.08622.
  * [16] Jake Bouvrie and Boumediene Hamzi.  Empirical estimators for stochastically forced nonlinear systems: Observability, controllability and the invariant measure.  Proc. of the 2012 American Control Conference, pages 294–301, 2012\.  https://arxiv.org/abs/1204.0563v1.
  * [17] Jake Bouvrie and Boumediene Hamzi.  Kernel methods for the approximation of nonlinear systems.  SIAM J. Control and Optimization, 2017.  https://arxiv.org/abs/1108.2903.
  * [18] Jake Bouvrie and Boumediene Hamzi.  Kernel methods for the approximation of some key quantities of nonlinear systems.  Journal of Computational Dynamics, 1, 2017.  http://arxiv.org/abs/1204.0563.
  * [19] Boumediene Hamzi, Christian Kuehn, and Sameh Mohamed.  A note on kernel methods for multiscale systems with critical transitions.  Mathematical Methods in the Applied Sciences, 42(3):907–917, 2019\.
  * [20] David J. C. MacKay.  Information Theory, Inference, and Learning Algorithms.  Copyright Cambridge University Press, 2003.
  * [21] R. J. Solomonoff.  A preliminary report on a general theory of inductive inference (revision of report v-131).  Contract AF, 49(639):376, 1960.
  * [22] A.N. Kolmogorov.  Three approaches to the quantitative definition of information.  Problems of information transmission, 1(1):1–7, 1965.
  * [23] Gregory J Chaitin.  A theory of program size formally identical to information theory.  Journal of the ACM (JACM), 22(3):329–340, 1975.
  * [24] Ray J Solomonoff.  A formal theory of inductive inference. part i.  Information and control, 7(1):1–22, 1964.
  * [25] L.A. Levin.  Laws of information conservation (nongrowth) and aspects of the foundation of probability theory.  Problemy Peredachi Informatsii, 10(3):30–35, 1974.
  * [26] Marcus Hutter, Shane Legg, and Paul MB Vitanyi.  Algorithmic probability.  Scholarpedia, 2(8):2572, 2007.
  * [27] Kamaludin Dingle, Mohammad Alaskandarani, Boumediene Hamzi, and Ard Louis.  Exploring simplicity bias in 1d dynamical systems.  2023\.
  * [28] Ludwig Arnold, Christopher KRT Jones, Konstantin Mischaikow, Geneviève Raugel, and Ludwig Arnold.  Random dynamical systems.  Springer, 1995.
  * [29] Kamaludin Dingle, Jeroen SW Lamb, and Joan-Andreu Lázaro-Camí.  Knudsen’s law and random billiards in irrational triangles.  Nonlinearity, 26(2):369, 2012.
  * [30] G Mayer-Kress and H Haken.  The influence of noise on the logistic model.  Journal of Statistical Physics, 26:149–171, 1981.
  * [31] Kamaludin Dingle, Chico Q Camargo, and Ard A Louis.  Input–output maps are strongly biased towards simple outputs.  Nature communications, 9(1):761, 2018.
  * [32] Kamaludin Dingle, Guillermo Valle Pérez, and Ard A Louis.  Generic predictions of output probability based on complexities of inputs and outputs.  Scientific reports, 10(1):1–9, 2020.
  * [33] Guillermo Valle-Perez, Chico Q Camargo, and Ard A Louis.  Deep learning generalizes because the parameter-function map is biased towards simple functions.  arXiv preprint arXiv:1805.08522, 2018.
  * [34] Chris Mingard, Joar Skalse, Guillermo Valle-Pérez, David Martínez-Rubio, Vladimir Mikulik, and Ard A Louis.  Neural networks are a priori biased towards boolean functions with low entropy.  arXiv preprint arXiv:1909.11522, 2019.
  * [35] Satwik Bhattamishra, Arkil Patel, Varun Kanade, and Phil Blunsom.  Simplicity bias in transformers and their ability to learn sparse boolean functions.  arXiv preprint arXiv:2211.12316, 2022.
  * [36] Greg Yang and Hadi Salman.  A fine-grained spectral perspective on neural networks.  arXiv preprint arXiv:1907.10599, 2019.
  * [37] Santiago Hernández-Orozco, Hector Zenil, Jürgen Riedel, Adam Uccello, Narsis A Kiani, and Jesper Tegnér.  Algorithmic probability-guided machine learning on non-differentiable spaces.  Frontiers in artificial intelligence, 3:567356, 2021.
  * [38] Kamaludin Dingle, Pau Batlle, and Houman Owhadi.  Multiclass classification utilising an estimated algorithmic probability prior.  Physica D: Nonlinear Phenomena, 448:133713, 2023.
  * [39] Chris Mingard, Henry Rees, Guillermo Valle-Pérez, and Ard A Louis.  Do deep neural networks have an inbuilt occam’s razor?  arXiv preprint arXiv:2304.06670, 2023.
  * [40] M. Li and P.M.B. Vitanyi.  An introduction to Kolmogorov complexity and its applications.  Springer-Verlag New York Inc, 2008.
  * [41] Thai Son Doan, Maximilian Engel, Jeroen SW Lamb, and Martin Rasmussen.  Hopf bifurcation with additive noise.  Nonlinearity, 31(10):4567, 2018.
  * [42] H. Zenil and J.P. Delahaye.  An algorithmic information theoretic approach to the behaviour of financial markets.  Journal of Economic Surveys, 25(3):431–463, 2011.
  * [43] Kamaludin Dingle, Rafiq Kamal, and Boumediene Hamzi.  A note on a priori forecasting and simplicity bias in time series.  Physica A: Statistical Mechanics and Its Applications, 609:128339, 2023.
  * [44] Kelty Ann Allen.  Martin-Löf Randomness and Brownian Motion.  PhD thesis, University of California, Berkeley, Berkeley, May 2014.  Available at https://escholarship.org/uc/item/20072582.
  * [45] Ali Kanso and Nejib Smaoui.  Logistic chaotic maps for binary numbers generations.  Chaos, Solitons & Fractals, 40(5):2557–2568, 2009.
  * [46] C.S. Calude.  Information and randomness: An algorithmic perspective.  Springer, 2002.
  * [47] P. Gács.  Lecture notes on descriptional complexity and randomness.  Boston University, Graduate School of Arts and Sciences, Computer Science Department, 1988.
  * [48] Alexander Shen, Vladimir A Uspensky, and Nikolay Vereshchagin.  Kolmogorov complexity and algorithmic randomness, volume 220.  American Mathematical Society, 2022.
  * [49] Hector Zenil, Fernando Soler-Toscano, Kamaludin Dingle, and Ard A Louis.  Correlation of automorphism group size and topological properties with program-size complexity evaluations of graphs and complex networks.  Physica A: Statistical Mechanics and its Applications, 404:341–358, 2014.
  * [50] Kamaludin Dingle, Javor K Novev, Sebastian E Ahnert, and Ard A Louis.  Predicting phenotype transition probabilities via conditional algorithmic probability approximations.  Journal of the Royal Society Interface, 19(197):20220694, 2022.
  * [51] Mohammad Alaskandarani and Kamaludin Dingle.  Low complexity, low probability patterns and consequences for algorithmic probability applications.  Complexity, 2023, 2023.
  * [52] Iain G Johnston, Kamaludin Dingle, Sam F Greenbury, Chico Q Camargo, Jonathan PK Doye, Sebastian E Ahnert, and Ard A Louis.  Symmetry and simplicity spontaneously emerge from the algorithmic nature of evolution.  Proceedings of the National Academy of Sciences, 119(11):e2113883119, 2022.
  * [53] Yuzuru Sato, Thai Son Doan, Jeroen SW Lamb, and Martin Rasmussen.  Dynamical characterization of stochastic bifurcations in a random logistic map.  arXiv preprint arXiv:1811.03994, 2018.
  * [54] Douglas Lind and Brian Marcus.  An introduction to symbolic dynamics and coding.  Cambridge university press, 1995.
  * [55] F. Kaspar and HG Schuster.  Easily calculable measure for the complexity of spatiotemporal patterns.  Physical Review A, 36(2):842, 1987.
  * [56] A. Lempel and J. Ziv.  On the complexity of finite sequences.  Information Theory, IEEE Transactions on, 22(1):75–81, 1976.
  * [57] Boris Hasselblatt and Anatole Katok.  A first course in dynamics: with a panorama of recent developments.  Cambridge University Press, 2003.
  * [58] Arno Berger.  Chaos and Chance: An Introduction to Stochastic Apects of Dynamics.  Walter de Gruyter, 2001.
  * [59] James Patrick Crutchfield, J Doyne Farmer, and Bernardo A Huberman.  Fluctuations and simple chaotic dynamics.  Physics Reports, 92(2):45–82, 1982.
  * [60] Marcus Hutter.  Universal artificial intelligence: Sequential decisions based on algorithmic probability.  Springer Science & Business Media, 2004.
  * [61] Sven Neth.  A dilemma for solomonoff prediction.  Philosophy of Science, 90(2):288–306, 2023.
  * [62] Marcus Hutter.  On universal prediction and bayesian confirmation.  Theoretical Computer Science, 384(1):33–48, 2007.
  * [63] Ray Solomonoff.  Complexity-based induction systems: comparisons and convergence theorems.  IEEE transactions on Information Theory, 24(4):422–432, 1978.
  * [64] Paul MB Vitányi.  Similarity and denoising.  Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 371(1984):20120091, 2013.
  * [65] Hector Zenil, Liliana Badillo, Santiago Hernández-Orozco, and Francisco Hernández-Quiroz.  Coding-theorem like behaviour and emergence of the universal distribution from resource-bounded algorithmic probability.  International Journal of Parallel, Emergent and Distributed Systems, 34(2):161–180, 2019.
  * [66] Pieter Adriaans.  Learning as data compression.  In Computation and Logic in the Real World: Third Conference on Computability in Europe, CiE 2007, Siena, Italy, June 18-23, 2007. Proceedings 3, pages 11–24. Springer, 2007.
  * [67] Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al.  Language modeling is compression.  arXiv preprint arXiv:2309.10668, 2023.

##  Appendix A Estimating pattern complexity

To estimate complexity, we follow Ref. [31] and use

$C_{LZ}(x)=\begin{cases}\log_{2}(n),&\text{$x=0^{n}$ or $1^{n}$}\\\
\log_{2}(n)[N_{w}(x_{1}...x_{n})+N_{w}(x_{n}...x_{1})]/2,&\text{otherwise}\end{cases}$
(14)

where $N_{w}(x)$ comes from the 1976 Lempel and Ziv complexity measure [56],
and where the simplest strings $0^{n}$ and $1^{n}$ are separated out because
$N_{w}(x)$ assigns complexity $K=1$ to the string 0 or 1, but complexity 2 to
$0^{n}$ or $1^{n}$ for $n\geq 2$, whereas the true Kolmogorov complexity of
such a trivial string actually scales as $\log_{2}(n)$ for typical $n$ because
one only needs to encode $n$. Having said that, the minimum possible value is
$K(x)\approx 0$ for a simple set, and so e.g. for binary strings of length
$n$, we can expect $0\leq K(x)\leq n$ bits. Because for a random string of
length $n$ the value $C_{LZ}(x)$ is often much larger than $n$, especially for
short strings, we scale the complexity so that $a$ in Eq. (2) is set to $a=1$
via

$\tilde{K}(x)=\log_{2}(M)\cdot\frac{C_{LZ}(x)-\min_{x}(C_{LZ})}{\max_{x}(C_{LZ})-\min_{x}(C_{LZ})}$
(15)

where $M$ is the maximum possible number of output patterns in the system, and
the min and max complexities are over all strings $x$ which the map can
generate. $\tilde{K}(x)$ is the approximation to Kolmogorov complexity that we
use throughout. This scaling results in $0\leq\tilde{K}(x)\leq n$ which is the
desirable range of values.

##  Appendix B Comparison of Laplace and algorithmic probability for induction

Here we give another example in which Laplace’s induction method gives
different conclusions as compared to induction based on algorithmic
probability.

In the logistic map, assume that $\mu=2.5$ and $\epsilon=0.0$. As in the main
text, if we assume that $x_{0}\approx 0$, then the digitised trajectory will
begin as zeros, and then change to 1s as the trajectory approaches $1-1/\mu=$
0.6$>$0.5. Define $x_{0}$ to be the value such that the series of 0s changes
to 1s after $m^{m}$ zeros are observed, where $m$ is some positive integer.
Because the logistic map can be described in a few bits, and $\mu=2.5$ and
$\epsilon=0.0$ can also be described in a few bits, then the complexity of the
series comes from the value of $m$. We have

$K(m^{m})=K(m)+O(1)\approx\log_{2}(m)$ (16)

for typical $m$. Therefore the algorithmic probability prediction that the
next bit after the zeroes will be 1 is $2^{-\log_{2}(m)}\sim 1/m$. In
contrast, after observing $m^{m}$ zeros, the Laplacian prediction for the
probability that the next bit is 1 is $1/m^{m}$ which is substantially
smaller. Therefore the observation of a 1 is more ‘surprising’ from the
perspective of the Laplacian method.


#  _Trialstreamer_ : Mapping and Browsing Medical Evidence in Real-Time

Benjamin E. Nye
Northeastern University
<EMAIL_ADDRESS>Ani Nenkova
UPenn
<EMAIL_ADDRESS>Iain J. Marshall
King’s College London
<EMAIL_ADDRESS>
Byron C. Wallace
Northeastern University
<EMAIL_ADDRESS>
###### Abstract

We introduce _Trialstreamer_ , a living database of clinical trial reports.
Here we mainly describe the _evidence extraction_ component; this extracts
from biomedical abstracts key pieces of information that clinicians need when
appraising the literature, and also the relations between these. Specifically,
the system extracts descriptions of trial participants, the treatments
compared in each arm (the _interventions_), and which outcomes were measured.
The system then attempts to infer which interventions were reported to work
best by determining their relationship with identified trial outcome measures.
In addition to summarizing individual trials, these extracted data elements
allow automatic synthesis of results across many trials on the same topic. We
apply the system at scale to all reports of randomized controlled trials
indexed in MEDLINE, powering the automatic generation of _evidence maps_ ,
which provide a global view of the efficacy of different interventions
combining data from all relevant clinical trials on a topic. We make all code
and models freely available111https://github.com/bepnye/evidence_extraction/
alongside a demonstration of the web interface.222http://bit.ly/trialstreamer

##  1 Introduction and Motivation

The highest-quality evidence to inform healthcare practice comes from
randomized controlled trials (RCTs). The results of the vast majority of these
trials are communicated in the form of unstructured text in journal articles.
Such results accumulate quickly, with over 100 articles describing RCTs
published daily, on average. It is difficult for healthcare providers and
patients to make sense of and keep up with this torrent of unstructured
literature.

Consider a patient who has been newly diagnosed with diabetes. She would like
to consult (in collaboration with her healthcare provider) the available
evidence regarding her treatment options. But she may not even be aware of
what her treatment options are. Further, she may only care about particular
outcomes (for instance, managing her blood pressure). Currently, it is not
straightforward to retrieve and browse the evidence pertaining to a given
condition, and in particular to ascertain which treatments are best supported
for a specific outcome of interest.

_Trialstreamer_ is a first attempt to solve this problem, making evidence more
browseable via NLP technologies. Figure 1 shows one of the key features of the
system: an automatically generated _evidence map_ that displays treatments
(vertical axis) and outcomes (horizontal) identified for a condition specified
by the user (here, migraines). We elaborate on this particular example to
illustrate the use of the system in Section 3.

Figure 1: A portion of an example evidence mapping Interventions and their
inferred efficacy for Outcomes, given the condition (or Population) of _Type
II Diabetes_. These maps are generated automatically using the NLP system we
describe in this work.

Trialstreamer aims to facilitate efficient evidence mapping with a user
friendly method of presenting a search across a broad field (here, being a
clinical condition) Miake-Lye et al. (2016). We use NLP technologies to
provide browseable, interactive overviews of large volumes of literature, on-
demand. These may then inform subsequent, formal syntheses, or they may simply
guide exploration of the primary literature. In this work we describe an open-
source prototype that enables evidence mapping, using NLP to generate
interactive overviews and visualizations of all RCT reports indexed by MEDLINE
(and accessible via PubMed).

When mapping the evidence one is generally interested in the following basic
questions:

  * •

What interventions and outcomes have been studied for a given condition
(population)?

  * •

How much evidence exists, both in terms of the number of trials and the number
of participants within these?

  * •

Does the evidence seem to support use of a particular intervention for a given
condition?

In the remainder of this paper we describe a prototype system that facilitates
interactive exploration and mapping of the evidence base, with an emphasis on
answering the above questions. The Trialstreamer mapping interface allows
structured search over study populations, interventions/comparators, and
outcomes — collectively referred to as PICO elements Huang et al. (2006). It
then displays key clinical attributes automatically extracted from the set of
retrieved trials. This is made possible via NLP modules trained on recently
released corpora Nye et al. (2018); Lehman et al. (2019), described below.

##  2 System Overview

Figure 2: Overview of the evidence extraction pipeline, applied to all RCT
article abstracts automatically identified. Text spans are first extracted
from these abstracts, then assembled into relations that reflect the structure
of the trials, and finally used to infer the effect interventions were
reported to have on measured outcomes, as compared to the control treatment.

The evidence extraction pipeline is composed of four primary phases. First,
text snippets that convey information about the trial’s treatments (or
_interventions_), outcome measures, and results are extracted from abstracts.
Relations between these snippets are then inferred to identify which
treatments were compared against each other, and which outcomes were measured
for these comparisons. The extracted relations and evidence statements are
then used to infer an overall conclusion about the comparative efficacy of the
trial’s interventions. Finally, the clinical concepts expressed in the
extracted spans are normalized to a structured vocabulary in order to ground
them in an existing knowledge base and allow for aggregations across trials.

A typical RCT report would pertain to a single clinical condition (the
_population_), but might report multiple numerical results, each concerning a
particular intervention, comparator, and outcome measure (which we describe as
an ICO triplet).

Because the end-to-end task combines NLP subtasks that are supported by
different datasets, we collected new development and test sets — 160 abstracts
in all, exhaustively annotated — in order to evaluate the overall performance
of our system. Two medical doctors333Hired via Upwork (http://www.upwork.com).
annotated these documents with the all of the expressed entities, their
mentions in the text, the relations between them, the conclusions reported for
each ICO triplet and the sentence that contains the supporting evidence for
this Lehman et al. (2019).

We were unable to obtain normalized concept labels for the ICO triplets due to
the excessive difficulty of the task for the annotators.

Modeling decisions were informed by the 60 document development set, and we
present evaluations of the first four information extraction modules with
regard to the 100 documents in the unseen test set.

###  2.1 Preprocessing

Enabling search over RCT reports requires first compiling and indexing all
such studies. This is, perhaps surprisingly, non-trivial. One may rely on
“Publication Type” (PT) tags that codify study designs of articles, but these
are manually applied by staff at the National Library of Medicine.
Consequently, there is a lag between when a new study is published and when a
PT tag is applied. Relying on these tags may thus hinder access to the most
up-to-date evidence available. Therefore, we instead use an automated tagging
system that uses machine learning to classify articles as RCT reports (or
not). This model has been validated extensively in prior work Marshall et al.
(2018), and we do not describe it further here.

Next, we replace all abbreviations with their long forms using the Ab3P
algorithm Sohn et al. (2008). Using long forms has the complementary
advantages of improving PICO labeling accuracy while also reducing the amount
of context needed for prediction by downstream model components.

###  2.2 Study Descriptor Recognition

#### PICO Elements

In order to identify the spans of text corresponding to the PICO elements of
the trial, we use the _EBM-NLP_ corpus Nye et al. (2018). This is a dataset
comprising $\sim$5,000 abstracts of RCT reports that have been annotated to
demarcate textual spans that describe the respective PICO elements. In
addition to these spans, it contains more granular annotations on information
within spans (e.g., specific Population attributes like age and sex).

| F1 | Precision | Recall
---|---|---|---
Tokens | 0.63 | 0.56 | 0.72
Clinical Entities | 0.67 | 0.55 | 0.87
Table 1: Macro-averaged scores for ICO span prediction at both the token and
clinical entity level.

We follow our prior work Nye et al. (2018) in training a BiLSTM-CRF model that
learns to jointly predict each PICO element using EBM-NLP. Recent work has
shown the efficacy of BERT Devlin et al. (2018) representations in this space,
e.g., Beltagy _et al._ achieved state-of-the-art performance on EBM-NLP using
this approach Beltagy et al. (2019). Therefore, for all text encoding we use
BioBERT Lee et al. (2019), which was pre-trained on PubMed documents.444For
PICO tagging on EBM-NLP we found that BioBERT performed comparably to SciBERT
Beltagy et al. (2019).

Results for Interventions/Comparators and Outcomes on our test set are
reported in Table 1. Since these spans will serve as inputs to downstream
models in the pipeline, high recall at the expense of precision is preferable;
we will allow subsequent classifiers to discard spurious spans. We achieve
0.87 recall at the clinical concept level.

#### Evidence Statements

In addition to PICO elements, we extract all sentences in the abstract that
are predicted to contain evidence concerning the relative efficacy of an
Intervention. Our training data for this model is sourced from the _Evidence-
Inference_ corpus Lehman et al. (2019), which comprises $\sim$10,000 annotated
‘prompts’ across $\sim$2,400 unique full-text articles. Each prompt specifies
an Intervention, a Comparator, and an Outcome. Doctors have annotated the
prompts for each article, supplying an extracted snippet that presents the
conclusion for these ICO elements, as well as an inference concerning whether
the Outcome increased, decreased, or remained the same in the intervention
group (as compared to the comparator group).

| F1 | Precision | Recall
---|---|---|---
Evidence | 0.69 | 0.53 | 0.97
Table 2: Performance for identifying evidence-bearing sentences.

We frame evidence identification as a sentence classication task, and train a
linear classification layer on top of BioBERT outputs. Our positive training
examples are the sentences containing evidence snippets in Evidence-Inference,
and we draw an equal number of length-matched negatives randomly from the rest
of the document. As shown in Table 2, we achieve extremely high recall on the
test set, but only middling precision. On inspection, many of these false
positives are sentences from the conclusion that provide a high-level summary
of the evidence, but aren’t the best evidence statement — as provided by the
annotator — for any given ICO prompt.

###  2.3 Relation Extraction

To transform the extracted spans into a semantic representation of the trial
that can be used to construct an evidence map, we must identify all instances
of an outcome being reported, and infer which two treatments were being
directly compared as the intervention and comparator with respect to said
outcome. Finally, given each assembled ICO prompt, we can then predict the
trial’s findings regarding whether the outcome increased, decreased, or was
not statistically different under the intervention versus the comparator. In
effect, we are aiming to jointly extract ICO prompts _and_ infer the
directionality of the results reported concerning these, whereas prior work
Nye et al. (2018); Lehman et al. (2019) has considered these problems only in
isolation.

Our strategy for assembling ICO prompts is informed by the style in which
results are commonly described in abstracts.When results are described in an
article the outcome is typically referenced explicitly, while the intervention
and especially the comparator are often referenced either indirectly (“Mean
headache duration was similar between groups”), or not at all (“No significant
difference was observed for recovery time”). In the fully annotated dev set
collected for this work, 87% of outcomes were described explicitly in an
evidence span, while only 28% of treatments were explicit.

Motivated by this observation, we use the (explicit) outcomes extracted from
an evidence snippet as a starting point; for each of these outcomes, the
associated intervention and comparator are then inferred. This has the
significant advantage of explicitly linking each outcome to the evidence that
will be used to infer the directionality of the reported finding. This also
provides the end-user with an interpretable rationale for the inference
concerning treatment efficacy.

To link candidate extracted treatments to specific outcome mentions, we train
a model that takes in a candidate treatment, an evidence statement containing
the outcome, and the surrounding context from the document, and predicts if
the treatment is the participating intervention, the participating comparator,
or if it is not involved in this particular evaluation. We use the evidence-
inference corpus to provide training examples for the first two classes, and
manually generate negative samples for the final class. The negatives are
constructed to mimic common errors that the treatment extraction module made
on the dev set, including: mislabeling an outcome as a treatment; extracting
compound phrases containing multiple individual treatments; and, finally,
extracting spurious spans that don’t represent a study descriptor.

The model is a linear classifier on top of BioBERT. Inputs are constructed as:
[cls] treatment [sep] evidence. context. [sep]. We experimented with different
slices of the document as the context, and achieved the highest dev
performance using the first four sentences of the article. The class
probabilities from this model are used to rank the possible interventions and
comparators for each outcome, and when sufficiently probable candidates are
identified we generate a complete ICO prompt.

| F1 | Precision | Recall
---|---|---|---
No Difference | 0.91 | 0.94 | 0.89
Increased | 0.73 | 0.69 | 0.77
Decreased | 0.76 | 0.75 | 0.78
Table 3: Per-class prediction scores for each outcome in the test set.

After assembling all ICO prompts in a document, we feed them to a final
classifier to predict the directionality of findings for each outcome, with
respect to the given intervention and comparator. This model is trained over
the evidence-inference corpus using the provided I, C, and O spans coupled
with the sentences that contain the corresponding evidence statement.
Empirically, we found that signal for the classifier is dominated by the
outcome text and evidence span, with almost no contribution from the
intervention and comparator. This is unsurprising given the regularity of the
language used to describe conclusions. The reported directionality of the
result is almost exclusively framed with respect to the intervention, and only
4.0% of all outcomes ever have different results for another I+C linking
within the same document. The best performing model input was simply [cls]
outcome [sep] evidence [sep], and the results on the test set are reported in
Table 3.

###  2.4 Normalizing PICO Terms

In order to standardize the language used to categorize the articles with
respect to their PICO elements, we turn to the structured vocabulary provided
by the National Libaray of Medicine (NLM) in the form of Medical Subject
Heading (MeSH) terms. This resource codifies a comprehensive set of medical
concepts into an ontology that includes their descriptions, properties, and
the structured relationships between them. Each article in the MEDLINE
database maintained by the NLM is annotated with the relevant MeSH terms by
expert library scientists (subject to the same lag that necessitates an RCT
classifier instead of relying on annotated Publication Types).

To induce relevant MeSH terms for an extracted text span, we reproduced the
method described in the Metamap Lite paper Demner-Fushman et al. (2017) to
extract MeSH terms describing the PICO elements. In short, we generated a
large dictionary of synonyms for medical terms algorithmically using data from
the UMLS Metathesaurus, with synonyms being matched to unique identifiers
pertaining to concepts in the MeSH vocabulary. We used this dictionary to map
matching strings in our extracted PICO text to MeSH terms, yielding a set of
normalized concepts describing each of the population, intervention, and
outcome spans in the documents.

To evaluate the accuracy of this approach, we compare the differences between
the MeSH terms produced by our system against those provided by the NLM for
the 191 articles that comprise the test set for EBM-NLP.

Strict | Precision | Recall
---|---|---
Extracted spans | 0.26 | 0.24
Expert spans | 0.23 | 0.26
Relaxed | Precision | Recall
Extracted spans | 0.32 | 0.34
Expert spans | 0.31 | 0.34
Table 4: Performance for predicting an article’s exact MeSH terms using the
rule-base system, run on both the automatically extracted spans and the
expert-provided test spans.

The test articles are provided with an average of $14.8$ MeSH terms per
article, while our system induces $14.0$ terms on average. The strictest
evaluation for this module is to require exact matches between the predicted
MeSH terms and the official MEDLINE terms – a daunting task given the $30,000$
possible labels we have to chose from. However, because the concepts in the
ontology exist in varying levels of specificity (for example _Migraine with
Aura_ is a subset of _Migraine Disorders_), it is often the case that the
predicted MeSH term is sufficiently close to the provided MeSH term for
practical purposes, but differs in the level of specificity.

To better characterize the performance of our approach, we therefore also
consider relaxing the equivalence criteria to include matching immediate
parents or children in the MeSH hierarchy. This modification results in a
$42\%$ relative increase in recall and a $23\%$ increase in precision, as
shown in Table 4.

False Neg | False Pos
---|---
Name / Count | Name / Count
Humans |  185 |  Patients |  115
Middle Aged |  93 |  Aging |  42
Adult |  84 |  Therapeutics |  42
Aged |  62 |  Weights/Measures |  33
Double-Blind Method |  50 |  Placebos |  33
Treatment Outcome |  42 |  Time |  21
Adolescent |  39 |  Serum |  17
Prospective Studies |  27 |  Safety |  17
Time Factors |  20 |  Pain |  16
Child, Preschool |  20 |  Women |  14
Table 5: Ten most common over- and under-predicted MeSH terms for the test set
of 191 articles.

We observe that while the absolute accuracy is not high, this technique
generally captures the key terms for the PICO elements. The most common
mistakes, shown in Table 5, mostly involve missing age or publication type
terms, and systematic differences between the general MeSH terms commonly
applied to articles (for example, we might apply _Patients_ rather than
_Humans_).

A more sophisticated aligment between the way MeSH terms are applied by
experts and the terms produced by our system has the potential to improve the
overall effectiveness of the tool; we intend to pursue this in future work.

##  3 Illustrative Example

To illustrate the envisioned use of our automatic mapping system, we return to
the example we began with at the outset of this paper: seeking evidence
concerning treatment of Type II Diabetes. To begin, the user specifies a
condition (Population) of interest. We rely on Medical Subject Headings (MeSH)
terms,555https://www.ncbi.nlm.nih.gov/mesh which as discussed above is a
structured vocabularly maintained by the NLM. We allow users to enter a search
string and provide auto-complete options from the MeSH vocabulary. Users can
additionally provide interventions or outcomes of interest to further narrow
the search. We show an example of a constructed set of filters in Figure 3.

Figure 3: View of a collected set of concepts used to specify trials of
interest. The search interface allows concepts to be combined using and/or
operators.

Once a set of search terms is specified, relevant RCTs are retrieved from the
comprehensive and up-to-date database.666We update this database nightly by
scanning MEDLINE for new RCT reports using our RCT classifier Marshall et al.
(2018). The interface then displays counts of unique interventions and
outcomes covered by the retrieved trials. Each bar in these plots can be
clicked to explicitly include that concept in the search terms, allowing for a
data-driven approach to building up the search parameters via iterative
refinement.

Figure 4: Detailed view of selected abstracts that contribute to the evidence
map. These are automatically annotated with all extracted information.

At this point, the evidence map shown in Figure 1 is also displayed, providing
a summary of the evidence available for the effectiveness of the selected
interventions with respect to their co-occurring outcomes. The user can mouse-
over plot elements to view tooltips that include snippets of contributing
evidence from the underlying abstracts, or click through to browse these texts
annotated with all of the extracted information, as shown in Figure 4.

##  4 User Study

To evaluate the system’s utility for a real-world task, we provided the tool
to a team of researchers at Cures Within Reach for Cancer
(CWR4C).777https://www.cwr4c.org/ Domain experts reviewed the extracted ICO
conclusions and automatically generated plots for a randomly selected subset
of documents pertaining to cancer trials, a domain that is particularly
challenging given the prevalence of complex compound interventions that often
share individual components between trial arms.

The reviewers were asked to evaluate the types of mistakes made by the system
as well as the overall precision and recall of the extracted conclusions for
each document. Across 21 documents average precision was 54% and average
recall was 75%, and the team expressed excitement about the efficacy of the
system for their purposes. CWR4C has continued to work with this tool as a
source of information about cancer-related clinical trials.

##  5 Conclusions

We have presented the evidence extraction component of Trialstreamer, an open-
source prototype that performs end-to-end identification of published RCT
reports, extracts key elements from the texts (intervention and outcomes
descriptions), and performs relation extraction between these, i.e., attempts
to determine which intervention was reported to work for which outcomes.

We use this pipeline to provide fast, on-demand overviews of all published
evidence pertaining to a condition of interest. Moving forward, we hope to
refine the linking of extracted snippets to structured vocabularies to run a
more comprehensive user-study to evaluate the use of the system in practice by
different types of users. We also hope to develop a joint extraction and
inference model, rather than relying on the current pipelined approach.

## Acknowledgements

This work was funded in part by the National Institutes of Health (NIH) under
the National Library of Medicine (NLM) grant 2R01LM012086, and by the National
Science Foundation (NSF) CAREER award 1750978.

## References

  * Beltagy et al. (2019) Iz Beltagy, Arman Cohan, and Kyle Lo. 2019.  Scibert: Pretrained contextualized embeddings for scientific text.  _arXiv preprint arXiv:1903.10676_.
  * Demner-Fushman et al. (2017) Dina Demner-Fushman, Willie J Rogers, and Alan R Aronson. 2017.  Metamap lite: an evaluation of a new java implementation of metamap.  _Journal of the American Medical Informatics Association_ , 24(4):841–844.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Huang et al. (2006) Xiaoli Huang, Jimmy Lin, and Dina Demner-Fushman. 2006.  Evaluation of pico as a knowledge representation for clinical questions.  In _AMIA annual symposium proceedings_ , volume 2006, page 359. American Medical Informatics Association.
  * Lee et al. (2019) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _CoRR_ , abs/1901.08746.
  * Lehman et al. (2019) Eric Lehman, Jay DeYoung, Regina Barzilay, and Byron C. Wallace. 2019.  Inferring Which Medical Treatments Work from Reports of Clinical Trials.  In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_ , pages 3705–3717.
  * Marshall et al. (2018) Iain J Marshall, Anna Noel-Storr, Joël Kuiper, James Thomas, and Byron C Wallace. 2018.  Machine learning for identifying randomized controlled trials: an evaluation and practitioner’s guide.  _Research synthesis methods_ , 9(4):602–614.
  * Miake-Lye et al. (2016) Isomi M Miake-Lye, Susanne Hempel, Roberta Shanman, and Paul G Shekelle. 2016.  What is an evidence map? a systematic review of published evidence maps and their definitions, methods, and products.  _Syst. Rev._ , 5:28.
  * Nye et al. (2018) Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain J Marshall, Ani Nenkova, and Byron C Wallace. 2018.  A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature.  In _Proceedings of the conference. Association for Computational Linguistics. Meeting_ , volume 2018, page 197. NIH Public Access.
  * Sohn et al. (2008) Sunghwan Sohn, Donald C Comeau, Won Kim, and W John Wilbur. 2008.  Abbreviation definition identification based on automatic precision estimates.  _BMC bioinformatics_ , 9(1):402.