#  Abductive Inference and C. S. Peirce: 150 Years Later

Subhadeep (DEEP) Mukhopadhyay Email<EMAIL_ADDRESS>
###### Abstract

Two pillars of the paper. This paper is about two things: (i) Charles Sanders
Peirce (1837–1914)—an iconoclastic philosopher and polymath who is among the
greatest of American minds. (ii) Abductive inference—a term coined by C. S.
Peirce, which he defined as “the process of forming explanatory hypotheses. It
is the only logical operation which introduces any new idea.”

Abductive inference and quantitative economics. Abductive inference plays a
fundamental role in empirical scientific research as a tool for discovery and
data analysis. Heckman and Singer (2017) strongly advocated “Economists should
abduct.” Arnold Zellner (2007) stressed that “much greater emphasis on
reductive [abductive] inference in teaching econometrics, statistics, and
economics would be desirable.” But currently, there are no established theory
or practical tools that can allow an empirical analyst to abduct. This paper
attempts to fill this gap by introducing new principles and concrete
procedures to the Economics and Statistics community. I termed the proposed
approach as Abductive Inference Machine (AIM). The historical Peirce’s
experiment. In 1872, Peirce conducted a series of experiments to determine the
distribution of response times to an auditory stimulus, which is widely
regarded as one of the most significant statistical investigations in the
history of nineteenth-century American mathematical research (Stigler, 1978).
On the 150th anniversary of this historical experiment, we look back at the
Peircean-style abductive inference through a modern statistical lens. Using
Peirce’s data, it is shown how empirical analysts can abduct in a systematic
and automated manner using AIM.

Keywords: Abductive inference machine; Artificial intelligence; Density
sharpening; Informative component analysis; Problem of surprise; Laws of
discovery; Self-corrective models.

##  1 Introduction

Charles Sanders Peirce (1839–1914), America’s greatest philosopher of science,
was also a brilliant statistician and experimental scientist. For 32 years,
from 1859 until 1891, he worked for the United States Coast and Geodetic
Survey111U. S. Coast and Geodetic Survey was established on February 10, 1807,
by President Thomas Jefferson. It was the nation’s first civilian scientific
agency.. During this time, he developed an unfailing passion for experimental
research. He was deeply involved in developing theoretical and practical
methods for acquiring high-precision scientific measurements, which ultimately
earned him an international reputation as an expert in ‘measurement error’ in
physics. Robert Crease (2009), a philosopher and historian of science, noted:
‘His [Peirce’s] work helped remove American metrology from under the British
shadow and usher in an American tradition.’

Figure 1: First look at Peirce’s auditory response data. The x-axis denotes 24
different days of experiments and the y-axis displays the data of individual
experiment as boxplot.

1872 Experimental Data. In 1872, he conducted a series of famous experiments
to determine the distribution of response times to an auditory stimulus. He
measured the time that elapsed between the making of a sharp sound and the
record of reception of the sound by an observer, employing a Hipp chronoscope
(some kind of sophisticated clock). Fig. 1 shows the dataset, which consists
of roughly $500$ measurements (recorded in nearest milliseconds) each day for
$k=24$ different days222For further details on the experimental setup and the
full dataset, consult the online Peirce Edition Project: vol 3, p. 133–160 of
the chronological edition (Peirce, 2009). It’s also available in the R-package
quantreg. . Note that the first-day observations are systematically different
from others (also called systematic bias), and the inconsistency was due to
the lack of experience of the observer, which was corrected on the next day.
The next 23 days show much more consistent (comparable) measurements.

##  2 Gauss’ Law of Error

> What observation has to teach us is [density] function, not a mere number.
>
> — C. S. Peirce (1873)

Deciphering the Law of Errors. Peirce’s actual motivation for doing the
experiment was to study the probabilistic laws of fluctuations (also called
errors) in the measurements and to investigate how response time distributions
differ from the standard Gauss’ law.

Nineteenth-century statistical learning. Peirce (1873) presented a detailed
empirical investigation of the reaction-time densities for each day. He was
driven by two goals: to understand the shape of the reaction time densities
and to compare them with the expected Gaussian distribution. His approach had
a remarkably modern conceptual basis: first, he developed smooth kernel
density-type probability density estimates to understand the shape of error
distributions333Peirce made a pioneering contribution to American statistics
by developing the concepts that underpin nonparametric density estimation.;
second, he performed a goodness-of-fit (GOF) type assessment through visual
comparison between the shape of Gaussian distribution and the
nonparametrically estimated densities444 However, at that time no theory of
GOF was available. It took 30 more years for an English mathematician, Karl
Pearson, to make the breakthrough contribution in developing the formal
language of the GOF., and concluded that the reaction-time distributions
‘differed very little from’ the expected normal probability law555The term
“normal distribution” was coined by Peirce..

Twentieth-century statistical learning. Almost sixty years later, the same
dataset was reanalyzed by Wilson and Hilferty (1929), and they came to a very
different conclusion. Wilson and Hilferty performed a battery of tests to
verify the appropriateness of the normal distribution. For each series of
measurements, they computed $23$ statistics (e.g., mean, standard deviation,
skewness, kurtosis, interquartile range, etc.) to justify substantial
departures from Gaussianity. Interestingly, without any formal statistical
test, simply by carefully looking at the boxplots in Fig. 1, we can see the
presence of significant skewness (the median cuts the boxes into two unequal
pieces), heavy-tailedness (long whiskers relative to the box length), and
ample outlying observations—which is good enough to suspect the adequacy of
Gaussian distribution as a model for the data.

######  Remark 1.

The non-Gaussian nature of the error distribution of scientific measurements
is hardly surprising666Even Wilson and Hilferty (1929) noted the same:
‘according to our previous experience such long series of observations
generally reveal marked departures from the normal law.’ — in fact, it is the
norm, not the exception (Bailey, 2017), which arises primarily because it is
hard to control all the factors of a complex measurement process. But what is
startling is that even Peirce’s experiment, a simple investigation of
recording response times with the same instrument by the same person under
more or less similar conditions, can produce so much heterogeneity.

> ‘Does statistics help in the search for an alternative hypothesis? There is
> no codified statistical methodology for this purpose. Text books on
> statistics do not discuss either in general terms or through examples how to
> elicit clues from data to formulate an alternative hypothesis or theory when
> a given hypothesis is rejected.’
>
> — C. R. Rao (2001)

Revised Goal: From Testing to Discovery. Confirmatory analysis through
hypothesis testing provides investigators absolutely no clues on what might
actually be going on.777An average statistician uses data to confirm or reject
a particular theory/model. A competent statistician uses data to sharpen their
theory/model. Simply rejecting a hypothesis—saying that it is non-
Gaussian—does not add any new insight into the underlying laws of error. Thus,
our focus will be on developing a data analysis technique that can identify
the most questionable aspects of the existing model and can also provide
concrete recommendations on how to rectify those deficiencies in order to
build a better and more realistic model for the measurement uncertainties.

##  3 The Problem of Surprise

> It is not enough to, look for what we anticipate. The greatest gains from
> data come from surprises. — John Tukey (1972)

Modeling the Surprise. All empirical laws are approximations of
reality—sometimes good, sometimes bad. We will be fooling ourselves if we
think there is a single best model that fits Peirce’s experimental data. Any
statistical model, irrespective of how sophisticated it is, should be ready to
be surprised by data. The goal of empirical modeling is to develop a general
strategy for describing how a model should react and adapt itself to reduce
the surprise.

Without surprise, there is no discovery. The ‘process’ of discovering new
knowledge from data starts by answering the following questions: Is there
anything surprising in the data? If so, what makes it surprising? How should
the current model react to the surprise? How can it modify itself to
rationalize the empirical surprise? To develop a model and principle for
statistical discovery, we need to first address these fundamental data
modeling questions. In subsequent sections, we develop one such general
theory.

Basic notations used throughout the paper: $X$ is a continuous random variable
with cdf $F(x)$, pdf $f(x)$. The quantile function is given by
$Q(u)=F^{-1}(u)$. Expectation with respect to the initial working model
$F_{0}(x)$ is abbreviated as
$\mathbb{E}_{0}(\psi(X)):=\int\psi\;\mathrm{d}F_{0}$, and expectation with
respect to the empirical $\widetilde{F}$ is simply written as
${\widetilde{\mathbb{E}}}(\psi(X)):=\int\psi\;\mathrm{d}\widetilde{F}$. The
inner product of two functions $\psi_{1}$ and $\psi_{2}$ in
$\mathcal{L}^{2}(dF_{0})$ will be denoted by
$\langle\psi_{1},\psi_{2}\rangle_{F_{0}}:=\int\psi_{1}\psi_{2}\;\mathrm{d}F_{0}$.

##  4 A Model for Empirical Discovery

> There is no established practice for dealing with surprise, even though
> surprise is an everyday occurrence. Is there a best way to respond to
> empirical surprises?
>
> — Heckman and Singer (2017)

###  4.1 A Dyadic Meta-Model

Science is a “self-corrective” enterprise that seeks new knowledge by refining
the known.888According to Peirce, every branch of scientific inquiry exhibits
“the vital power of self-correction” that permits us to make progress and grow
our knowledge; see, Burch and Parker (2022). The same is true for statistical
modeling: it explores and discovers unknown patterns by smartly utilizing the
known (expected) model. In the following, we formalize this general principle.

######  Definition 1 (A Dyadic Meta-Model).

$X$ be a continuous random variable with true unknown density $f(x)$. Let
$f_{0}(x)$ represents a possibly misspecified predesignated working model for
$X$, whose support includes the support of $f(x)$. Then the following density
decomposition formula holds:

$f(x)\,=\,f_{0}(x)\,d\big{(}F_{0}(x);F_{0},F\big{)},$ (1)

where the $d(u;F_{0},F)$ is defined as

$d(u;F_{0},F)=\dfrac{f(F_{0}^{-1}(u))}{f_{0}(F_{0}^{-1}(u))},\leavevmode\nobreak\
\,0<u<1,$ (2)

which is called ‘comparison density’ because it compares the initial model-0
$f_{0}(x)$ with the true $f(x)$ and it integrates to one:

$\int_{0}^{1}d(u;F_{0},F)\;\mathrm{d}u\,=\,\int_{x}d(F_{0}(x);F_{0},F)\;\mathrm{d}F_{0}(x)\,=\,\int_{x}\big{(}f(x)/f_{0}(x)\big{)}\;\mathrm{d}F_{0}(x)\,=\,1.\leavevmode\nobreak\
\leavevmode\nobreak\ $

To simplify the notation, $d(F_{0}(x);F_{0},F)$ will be abbreviated as
$d_{0}(x)$. One can view (1) as a “meta-model”—a model comprising two sub-
models that blends existing imprecise knowledge $f_{0}(x)$ with new empirical
knowledge $d_{0}(x)$ to provide us complete picture of the uncertainty.

The above density representation formula can be interpreted from many
different angles: 1\. Model-Editing Tool. The dyadic model provides a general
statistical mechanism for designing a “better” model by editing or sharpening
the existing version. For that reason, we call $d$ the density-sharpening
function (DSF). Next section presents how to learn DSF from data. The
$d$-modulated repaired $f_{0}$-density in Eq. (1) will be referred to
$d$-sharp $f_{0}$.

2\. Surprisal Function. The process of data-driven discovery starts with a
surprise—a deviation between the data and the expected model. The DSF
$d(u;F_{0},F)$ gets activated only when model-0 encounters surprise, and its
shape encodes the nature of surprise. When there is no surprise,
$d(u;F_{0},F)$ takes the shape of a ‘flat’ uniform density.

Surprise to information gain: It is not enough to simply detect an empirical
surprise. For statistical learning, it is critical to know: What information
can we gain from the observed surprise? And how can we use that information to
revise our initial model of reality? The density-sharpening function
$d(u;F_{0},F)$ provides a pathway from surprise to information gain that
bridges the gap between the initial belief and knowledge.

3\. Simon’s Means-Ends Analysis. The model (1) interacts with the outer data
environment through two information channels:

  * •

Afferent (or ‘inward’) information: it captures and represents the
‘difference’ between the desired and present model using $d_{0}(x)$999 It
would be pointless to waste computational resources on the redundant part of
the data..

  * •

Efferent (or ‘outward’) information: it intelligently searches and provides
the best course of ‘actions’ that changes the present model through (1) to
reduce the difference101010$d_{0}(x)$ “fires” actions when the difference in
information content between $F_{0}$ and $\widetilde{F}$ reaches a threshold..

Herbert Simon (1988) noted that any general-purpose computational learning
system must have these two information processing components. Models equipped
with this special structure are known as the ‘Means-Ends analysis model’ in
the artificial intelligence community.

4\. Detective’s Microscope111111The name was inspired from John Tukey (1977,
p. 52). Information in the data can be broken down into two parts:

Data Information = Anticipated part + Unexpected surprising part. (3)

Model-0 explains the first part, whereas $d_{0}(x)$ captures everything that
is unexplainable by the initial $f_{0}(x)$. Accordingly, $d_{0}(x)$ performs
dual tasks: it reveals the incompleteness of our starting assumptions and
provides strategies on how to revise it to account for the observed puzzling
facts. In short, $d(u;F_{0},F)$ plays the role of a detective’s microscope,
permitting data investigators to assemble clues to initiate a systematic
search for new explanatory hypotheses that fit the evidence and solve the
puzzle.

5\. System-1 and System-2 Architecture121212This ‘Two Systems’ analogy was
inspired by Daniel Kahneman’s work on ‘Thinking, Fast and Slow.’. In our
dyadic model, System-1 is denoted by $f_{0}(x)$ that captures the background
knowledge component. Model-0 interacts with the environment through System 2
$d$-function. The DSF $d$ allows model-0 to self-examine its limitations and
also offers strategies for self-correcting to adapt to new situations. The DSF
plays the role of a ‘supervisor’ whose goal is model management. It helps the
subordinate $f_{0}$ to figure out what’s missing and how to fix this. Our
dyadic model combines both system-1 and system-2 into one integrated modeling
system.

6\. A Change Agent: The great philosopher Heraclitus taught us that change is
the only constant thing in this world. If we believe in this doctrine then we
should focus on modeling the change, not the model itself.131313Isaac Newton
confronted a similar problem in the mid-1600s: He wanted to describe a falling
object, which changes its speed every second. The challenge was: How to
describe a “moving” object? His revolutionary idea was to focus on modeling
the “change,” which led to the development of Calculus and Laws of Motion.
Here we are concerned with a similar question: How to change probability
distribution when confronted with new data? In our dyadic model (1), the
sharpening function $d$ provides the necessary “push to change.” The dyadic
model operationalizes this philosophy by providing a universal law of model
evolution: how to produce a useful model by changing an imperfect model-0 in a
data-adaptive manner. The rectified $f_{0}(x)$ inherits new characteristics
through $d_{0}(x)$ that give them a better chance of survival in the new data
environment.

###  4.2 A Robust Nonparametric Estimation Method

To operationalize the density-sharpening law, we need to estimate from data
the function $d_{0}(x)$, which is the cause of change in the state of a
probability distribution. We describe a theory of robust nonparametric
estimation whose core concepts and methodological tools are introduced in a
‘programmatic’ style—making it easy to translate the theory into a concrete
algorithm.

######  Remark 2 (What are we approximating?).

Before describing the approximation theory, it is vital to emphasize that,
unlike traditional nonparametric (or machine learning) methods where the goal
is to produce a density estimate $\widehat{f}(x)$, in our case, the focus is
on estimating the sharpening kernel $\widehat{d}_{0}(x)$—the ‘gap’ between
theory and measurements. This will provide rational explanations for the
surprising facts and, because of Eq. (1), concurrently rectify the initial
model $f_{0}$. Also, see sec. 8.

Step 0. Data and Setup. We observe a random sample
$X_{1},\ldots,X_{n}\mathrel{\dot{\sim}}F_{0}$. By “$\mathrel{\dot{\sim}}$” we
mean that $F_{0}$ is a tentative (approximate theoretical) model for $X$ that
is given to us. And, $f(x)$ denotes the unknown true model from which the data
were generated.

Step 1. LP-orthogonal System. Note that DSF $d_{0}(x)$ is a function of
rank-$F_{0}$ transform (i.e., probability integral transform with respect to
the base measure) $F_{0}(X)$. Hence, one can efficiently approximate $d\circ
F_{0}(x)\in\mathcal{L}^{2}({dF_{0}})$ by expanding it in a Fourier series of
polynomials that are function of $F_{0}(x)$ and orthonormal with respect to
the user-selected base-model $f_{0}(x)$. One such orthonormal system is the
LP-family of rank-polynomials (Mukhopadhyay and Fletcher, 2018, Mukhopadhyay,
2017), whose construction is given below.

LP-basis construction for an arbitrary continuous $F_{0}$: Define the first-
order LP-basis function as standardized rank-$F_{0}$ transform:

$T_{1}(x;F_{0})\,=\,\sqrt{12}\big{\\{}F_{0}(x)-1/2\big{\\}}.$ (4)

Note that $\mathbb{E}_{0}(T_{1}(X;F_{0}))=0$ and
$\operatorname{Var}_{0}(T_{1}(X;F_{0}))=1$. Next, apply Gram-Schmidt procedure
on $\\{T_{1}^{2},T_{1}^{3},\ldots\\}$ to construct a higher-order LP
orthogonal system $T_{j}(x;F_{0})$:

$\displaystyle T_{2}(x;F_{0})$ $\displaystyle=$
$\displaystyle\sqrt{5}\big{\\{}6F^{2}_{0}(x)-6F_{0}(x)+1\big{\\}}$ (5)
$\displaystyle T_{3}(x;F_{0})$ $\displaystyle=$
$\displaystyle\sqrt{7}\big{\\{}20F^{3}_{0}(x)-30F^{2}_{0}(x)+12F_{0}(x)-1\big{\\}}$
(6) $\displaystyle T_{4}(x;F_{0})$ $\displaystyle=$
$\displaystyle\sqrt{9}\big{\\{}70F^{4}_{0}(x)-140F^{3}_{0}(x)+90F^{2}_{0}(x)-20F_{0}(x)+1\big{\\}},$
(7)

and so on. For data analysis, we compute them by executing the Gram-Schmidt
process numerically. Hence, there is no need for bookkeeping the explicit
formulae of these polynomials. By construction, the LP-sequence of polynomials
satisfy the following conditions:

$\int\nolimits_{x}T_{j}(x;F_{0})\;\mathrm{d}F_{0}\,=\,0;\leavevmode\nobreak\
\leavevmode\nobreak\
\,\int\nolimits_{x}T_{j}(x;F_{0})T_{k}(x;F_{0})\;\mathrm{d}F_{0}=\delta_{jk},$
(8)

where $\delta_{jk}$ is the Kronecker’s delta function. The notation for LP-
polynomials $\\{T_{j}(x;F_{0})\\}$ is meant to emphasize: (i) they are
polynomials of $F_{0}(x)$ (not raw $x$) and hence are inherently robust. (ii)
they are orthonormal with respect to the distribution $F_{0}$, since they
satisfy (8). We also define the Unit LP-bases via quantile transform:
$S_{j}(u;F_{0})=T_{j}(Q_{0}(u);F_{0})$, $0\leq u\leq 1$.

Step 2. LP-Fourier Approximation. LP-orthogonal series representation of the
density-sharpening function $d_{0}(x)$ is given by

$d_{0}(x):=d(F_{0}(x);F_{0},F)\,=\,1+\sum_{j}\operatorname{LP}[j;F_{0},F]\,T_{j}(x;F_{0}),$
(9)

where the expansion coefficients $\operatorname{LP}[j;F_{0},F]$ satisfy

$\operatorname{LP}[j;F_{0},F]=\big{\langle}d\circ
F_{0},T_{j}\big{\rangle}_{F_{0}},\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ (j=1,2,\ldots,).$ (10)

Step 3. Nonparametric Estimation. To estimate the LP-Fourier coefficients from
data, rewrite Eq. (10) in the following form:

$\operatorname{LP}[j;F_{0},F]=\int_{-\infty}^{\infty}d_{0}(x)T_{j}(x;F_{0})f_{0}(x)\;\mathrm{d}x=\int_{-\infty}^{\infty}T_{j}(x;F_{0})f(x)\;\mathrm{d}x=\mathbb{E}_{F}[T_{j}(X;F_{0})]$
(11)

which expresses $\operatorname{LP}[j;F_{0},F]$ as the expected value of
$T_{j}(X;F_{0})$. Accordingly, estimate the LP-parameter as

$\widetilde{\operatorname{LP}}_{j}:=\operatorname{LP}[j;F_{0},\widetilde{F}]\,=\,{\widetilde{\mathbb{E}}}[T_{j}(X;F_{0})]\,=\,\frac{1}{n}\sum_{i=1}^{n}T_{j}(x_{i};F_{0}).$
(12)

These expansion coefficients act as the coordinates of true $f(x)$ relative to
assumed $f_{0}(x)$:

$\big{[}F\big{]}_{F_{0}}\leavevmode\nobreak\ :=\leavevmode\nobreak\
\Big{(}\operatorname{LP}[1;F_{0},\widetilde{F}],\ldots,\operatorname{LP}[m;F_{0},\widetilde{F}]\Big{)},\leavevmode\nobreak\
\leavevmode\nobreak\ \,1\leq m<r\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ $ (13)

where $r$ is the number of unique values observed in the data. Step 4.
Surprisal Index. Can we quantify the surprise of $f_{0}$ when it comes in
contact with the new data? We define the surprise index of the hypothesized
model as follows:

${\rm SI}(F_{0},F)\,=\,\sum_{j}\Big{|}\operatorname{LP}[j;F_{0},F]\Big{|}^{2}$
(14)

which can be computed by substituting (12) into (14). The motivation behind
this definition comes from recognizing that ${\rm
SI}(F_{0},F)=\int_{0}^{1}d^{2}-1$, i.e., the divergence-measure (14) captures
the departure of $d(u;F_{0},F)$ from uniformity. Note that when $d$ takes the
form of ${\rm Uniform}(0,1)$, then no correction is required in (1)—i.e., the
assumed model $f_{0}(x)$ is capable of fully explaining the data without being
surprised at anything. An additional desirable property of the measure (14) is
that it is invariant to monotone transformations of the data.

######  Remark 3 (Information is an observer-dependent concept).

Our definition (14) is different from the classical Shannon-style measure of
surprise or information. We view surprise as a “fundamentally relativistic,”
not an absolute quantity. The same data can have different surprising
information content for different background-knowledge-based initial
models/agents. In more philosophical terms: ${\rm SI}(F_{0},F)$ captures
observer-specific useful information of a dataset.

Step 5. Key Elements of Surprise. A ‘large’ value of ${\rm SI}(F_{0},F)$
indicates that the model $f_{0}(x)$ got shocked by the data. But what caused
this? This is the same as asking: what are the main ‘broken components’ of the
initial believable model $f_{0}(x)$ that need repair? As George Box (1976)
said:

> Since all models are wrong the scientist must be alert to what is
> importantly wrong. It is inappropriate to be concerned about mice when there
> are tigers abroad.

Note that the value of $\widetilde{\operatorname{LP}}_{j}$ is expected to be
“small” when underlying distribution is close to the assumed $F_{0}$; verify
this from (12). We discuss two pruning strategies that effectively remove the
noisy LP-components that can cause the density estimate $\widehat{f}(x)$ to be
unnecessary wiggly. Identify the ‘significant’ non-zero LP-
coefficients141414Under the null model, sample LP-statistic follows
asymptotically $\mathcal{N}(0,n^{-1/2})$. with
$|\widetilde{\operatorname{LP}}_{j}|>2/\sqrt{n}$. One can further refine the
denoising method as follows: sort them in descending order based on their
magnitude (absolute value) and compute the penalized ordered sum of squares.
This Ordered PENalization scheme will be referred as OPEN:

$\texttt{OPEN}(m)\leavevmode\nobreak\ =\leavevmode\nobreak\ \text{Sum of
squares of top $m$ LP coefficients}\leavevmode\nobreak\ -\leavevmode\nobreak\
\dfrac{\gamma_{n}}{n}m.\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ $ (15)

For AIC penalty choose $\gamma_{n}=2$, for BIC choose $\gamma_{n}=\log n$,
etc. For more details see Mukhopadhyay and Parzen (2020) and Mukhopadhyay
(2017). Find the $m$ that maximizes the ${\rm OPEN}(m)$. Store the selected
indices $j$ in the set $\mathcal{J}$; the set of functions
$\\{T_{j}(x;F_{0})\\}_{j\in\mathcal{J}}$ then denote the key ‘surprising
directions’ that need to be incorporated into the current model to make it
data-consistent. The OPEN-smoothed LP-coefficients will be denoted by
$\widehat{\operatorname{LP}}_{j}$.

Step 6. MaxEnt Lazy Update. We build an improved exponential density estimate
for $d(u;F_{0},F)$, which, unlike the previous $\mathscr{L}^{2}$-Fourier
series model (9), is guaranteed to be non-negative estimate and integrates to
1. The basic idea is to choose a model for $d$ to sharpen $f_{0}$ in order to
provide a better explanation of the data by minimizing surprises as much as
possible. We can formalize this idea using the notion of relative-entropy (or
Kullback-Leibler divergence) between $f_{0}$ and the $d$-sharp $f_{0}$:

$\displaystyle{\rm KL}(f_{0}\|f)$ $\displaystyle=$ $\displaystyle\int
f(x)\log\Big{\\{}\frac{f(x)}{f_{0}(x)}\Big{\\}}\;\mathrm{d}x\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ $ $\displaystyle=$
$\displaystyle\int\frac{f(x)}{f_{0}(x)}\log\Big{\\{}\frac{f(x)}{f_{0}(x)}\Big{\\}}\,f_{0}(x)\;\mathrm{d}x\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ $ $\displaystyle=$
$\displaystyle\int
d_{0}(x)\log\\{d_{0}(x)\\}\,f_{0}(x)\;\mathrm{d}x\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ $

Since $d_{0}(x)=d(F_{0}(x);F_{0},F)$, substituting $F_{0}(x)=u$ in the above
equation, we get the following important result in terms of entropy of $d$:
$H(d)=-\int d\log d$

${\rm KL}(f_{0}\|f)=\int_{0}^{1}d\log d=-{\rm Entropy}(d),$ (16)

which can also be viewed as a measure of surprise. Thus the goal of searching
for $d$ by minimizing the KL-divergence between the old and new model reduces
to the problem of finding a $d$ by maximizing its entropy. This is known as
the principle of maximum entropy (MaxEnt), first expounded by E.T. Jaynes
(1957).151515See, for example, the work of Amos Golan (2018) and Esfandiar
Maasoumi (1993) for an excellent review of the usefulness of ‘maxent
information-theoretic thinking’ for econometrics and decision sciences.
Additional recent works on the application of maximum-entropy techniques in
empirical economics can be found in Buansing et al. (2020), Mao et al. (2020),
Lee et al. (2021). However, an maximization of $H(d)=-\int_{0}^{1}d\log d$
under the normalization constraint $\int_{0}^{1}d=1$, among all continuous
distributions supported over unit interval, will lead to the trivial solution:

$d(u;F_{0},F)\,=\,1,\leavevmode\nobreak\ \leavevmode\nobreak\
0<u<1.\vspace{-.2em}$

######  Remark 4.

Despite its elegance, the classical Jaynesian inference is an incomplete data
modeling principle since it only tells us how to assign probabilities, not how
to design and select appropriate constraints. Discovery, by definition, can’t
happen by imposing preconceived constraints. The core ‘intelligence’ part of
any empirical modeling involves appropriately designing and searching for
relevant ‘directions’ (constraints) that neatly capture the surprising
information. More discussion on this is given in Mukhopadhyay (2022a).

Law of Lazy Update. The key question is: how to determine the informative
constraints? Jaynes’ maximum entropy principle remains completely silent on
this issue and assumes we know the relevant constraints ab initio (i.e.,
sufficient statistic functions)—which, in turn, puts restrictions on the
possible ‘shape’ of the probability distribution. We avoid this assumption as
follows, using what we call ‘Law of Lazy Update’: (i) Identify a small set of
most important LP-sufficient statistics functions161616These sets of
specially-designed functions provide the simplest and most likely explanation
of how the model $f_{0}$ differs from reality. using Step 5, which filters out
the ‘directions’ where close attention should be focused. (ii) Find a sparse
(smoother) probability distribution by maximizing the entropy $H(d)$ under the
normalization constraint $\int d=1$ and the following LP-moment constraints:

$\mathbb{E}[S_{j}(U;F_{0})]\,=\,\operatorname{LP}[j;F_{0},\widetilde{F}],\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \text{for $j\in\mathcal{J}$}.$ (17)

The solution of the above maxent-constrained optimization problem can be shown
to take the following exponential (18) form

$d_{\bm{\theta}}(u;F_{0},F)\,=\,\exp\Big{\\{}\sum_{j\in\mathcal{J}}\theta_{j}S_{j}(u;F_{0})\,-\,\Psi(\bm{\theta})\Big{\\}},\leavevmode\nobreak\
\leavevmode\nobreak\ 0<u<1$ (18)

where
$\Psi(\bm{\theta})=\log\int_{0}^{1}\exp\\{\sum_{j}\theta_{j}S_{j}(u;F_{0})\\}\;\mathrm{d}u.$

######  Remark 5 (Economical and Explanatory Construction).

The principle of ‘maxent lazy update’ provides a model for $d(u;F_{0},F)$,
which acts as a policymaker for $f_{0}$—one who formulates the preferred
course of action on how to amend the existing model $f_{0}$ (incorporating eqs
17) in a cost-effective way to achieve the most “economical description” of
the current reality.

######  Remark 6.

Incidentally, Peirce also had a strong interest in building ‘economical’
models171717Also see Peirce’s 1979 article on “Economy of Research,” which is
widely regarded as the first real attempt to establish the fundamental
principles of marginal utility theory. Stephen Stigler brought this to my
attention. and was influenced by the English philosopher William of Ockham.
During the 1903 Harvard Lectures on Pragmatism, Peirce noted: “There never was
a sounder logical maxim of scientific procedure than Ockham’s razor: Entia non
sunt multiplicanda praeter necessitatem.”

######  Remark 7 (Rational agent interpretation).

$d_{0}(x)$ acts as a rational agent for $f_{0}(x)$, which designs and selects
best possible actions (alternatives) by minimizing the surprise (16), subject
to the constraints (17). This kind of rationalistic empirical models were
called machina economicus by Parkes and Wellman (2015).

######  Remark 8.

Our style of learning of $d$ function from data performs two critical
operations: The formation of new hypotheses (design of LP-sufficient statistic
functions of $d$) and selection or adoption of some of the most prominent ones
(through OPEN model selection).

###  4.3 Repair-Friendly $\operatorname{DS}(F_{0},m)$ Models

######  Definition 2.

$\operatorname{DS}(F_{0},m)$ stands for Density-Sharpening of $f_{0}(x)$ using
$m$-term LP-series approximated $d_{0}(x)$. Two categories of
$\operatorname{DS}(F_{0},m)$ class of distributions are given below:

$\text{Orthogonal series $\operatorname{DS}(F_{0},m)$}:\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\
f(x)\,=\,f_{0}(x)\Big{[}1\,+\,\sum_{j=1}^{m}\operatorname{LP}[j;F_{0},F]\,T_{j}(x;F_{0})\Big{]},\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ $ (19) $\text{Maximum Entropy
$\operatorname{DS}(F_{0},m)$}:\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
f(x)\,=\,f_{0}(x)\exp\Big{\\{}\sum_{j=1}^{m}\theta_{j}T_{j}(x;F_{0})\,-\,\Psi(\bm{\theta})\Big{\\}}.\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \,$ (20)

They are obtained by replacing (9) and (18), respectively, into the dyadic
model (1). The truncation point $m$ indicates the search-radius around the
expected $f_{0}(x)$ to create permissible models. $\operatorname{DS}(F_{0},m)$
models with higher $m$ entertains alternative models of higher complexity.
However, to exclude absurdly rough densities, it is advisable to focus on the
vicinity of $f_{0}$ by choosing an $m$ that is not too large. In our
experience, $m=6$ (or at most $8$) is often sufficient for real data
applications—since $f_{0}(x)$ is a knowledge-based sensible starting model.

######  Remark 9.

The goal of empirical science is to progressively sharpen the existing
knowledge by discovering new patterns in the data, thereby leading to a new
revised theory. The ‘density-sharpening’ mechanism facilitates and automates
this process.

######  Remark 10 (Blending the old with the new).

The above density-editing schemes modify the initial probability law
$f_{0}(x)$ with a small set of new additional ‘shape functions’ (i.e., LP-
sufficient statistics $\\{T_{j}(x;F_{0})\\}_{j\in\mathcal{J}}$) that serve as
explanations for the surprising phenomenon. This will be more clear in the
next section where we carry out Peirce data analysis using the density-
sharpening principle, governed by the simple general law described in
Definition 2.

######  Remark 11.

$\operatorname{DS}(F_{0},m)$ models can be viewed as ‘descent with
modification,’ which partially inherits characteristics of model-0 and adds
some new shapes to it. This shows how new models are born out of a pre-
existing inexact model with some modification dictated by the density-
sharpening kernel $d_{0}(x)$ —thereby helping $f_{0}(x)$ to broaden its
initial knowledge repertoire.

######  Remark 12 (Model Economy).

Recall, in Section 3, we raised the question: how should a model adapt and
generalize in the face of surprise? $\operatorname{DS}(F_{0},m)$ is a class of
nonparametrically-modified parametric models that precisely answer this
question. In particular, density models (19) and (20) allow modelers to ‘fix’
their broken models (in a fully automated manner) rather than completely
replacing them with a brand new model built from scratch. Two practical
advantages of constructing auto-adaptive models: Firstly, it reduces the waste
of computational resources, and secondly, it extends the life span of of the
initial, imprecise knowledge-model $f_{0}(x)$ by making it reusable and
sustainable—we call this “model economy.”

##  5 Laplace’s Two Laws of Error

Our search for the laws of errors begins with the question: what is the most
natural choice of the error distribution that we anticipate to hold at least
approximately. Two candidates are:

  * •

Laplace distribution. In 1774, i.e., almost 100 years before Peirce’s
experiment, Laplace postulated that the frequency of an error could be
expressed as an exponential function of the numerical magnitude of the error,
disregarding sign (Laplace, 1774). This is known as Laplace’s first law of
error.

  * •

Gaussian distribution. Laplace proposed Gaussian distribution as the second
candidate for the error curve in 1778.

These two distributions provide a simple yet believable model-0 to start our
search for a realistic error distribution for the Peirce data. Our strategy
will be as follows: first, would like to know which of Laplace’s laws provides
a more reasonable choice as an initial candidate model. In other words, which
distribution is relatively less surprised by the Peirce data. Second, we like
to understand the nature of misspecifications of these two models over the set
of $24$ experimental datasets. This will ultimately help us repair $f_{0}(x)$
by informing us which components are damaged.181818Also, some
misspecifications may be harmless as far as the final decision-making is
concerned. Knowing the nature of deficiency can help us avoid over-
complicating the model-0. As John Tukey (1969) said: ‘Amount, as well as
direction, is vital.’

###  5.1 Informative Component Analysis

Generating new hypotheses in response to the rejection of the initial
candidate model is one of the central objectives of Informative Component
Analysis (ICA).

Gaussian Error Distribution. We devise a graphical explanatory method, called
Informative Component Analysis (ICA), to perform ‘informative’ data-model
comparison in a way that is easily interpretable for large number of parallel
experiments like Peirce data. The general process goes as follows:

Algorithm: Informative Component Analysis (ICA)

Step 0. Data and notation. For the $t$-th day experiment: we observe ${\bf
x}_{t}=(x_{t1},x_{t1},\ldots,x_{tn_{t}})$ with empirical distribution
$\widetilde{F}_{t}$.

Step 1. For each day, we estimate the best-fitted Gaussian distribution
$\varphi_{t}=\mathcal{N}(\tilde{\mu}_{t},\tilde{\sigma}_{t})$, where the
parameters are robustly estimated: $\tilde{\mu}_{t}$ is estimated by the
median and $\tilde{\sigma}_{t}$ is estimated by dividing the interquartile
range (IQR) by $1.349$. The presence of large outlying observations makes the
IQR-based robust-scale estimate more appropriate than the usual standard
deviation based estimate of $\sigma_{t}$; see Fig. 2.

Step 2. For each experiment, compute the LP-coefficients between the assumed
$\varPhi_{t}$ and the empirical distribution $\widetilde{F}_{t}$.

$\operatorname{LP}[j;\Phi_{t},\widetilde{F}_{t}]\leavevmode\nobreak\
=\leavevmode\nobreak\
\mathbb{E}\big{[}T_{j}(X_{t};\Phi_{t});\widetilde{F}_{t}\big{]}\leavevmode\nobreak\
=\leavevmode\nobreak\
\frac{1}{n_{t}}\sum_{i=1}^{n_{t}}T_{j}(x_{ti};\Phi_{t}).\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ $ (21)

for $t=1,2,\ldots,24$ and $j=1,\ldots,4$. The smoothed LP-coefficients
(applying OPEN model selection method based on AIC-penalty; see equation 15)
are stored in
$\operatorname{LP}[t,j]\,:=\,\widehat{\operatorname{LP}}[j;\Phi_{t},\widetilde{F}_{t}]$.
Step 3. LP-Map: Display the $24\times 4$ LP-matrix as an image for easy
visualization and interpretation. This is shown in Fig. 3(a).

Figure 2: Two normal distributions are compared with different scale
estimates: The red curve is based on the robust IQR-based scale estimate and
the blue one is the usual standard deviation-based curve. Clearly, the blue
curve underestimates the peak and overestimates the width of the density (due
to the presence of few large values in the tails).

Figure 3: Informative component analysis: LP-Map for Gauss and Laplace models.
The rows denote different time points and columns are the order of LP
coefficients $\operatorname{LP}[t,j]$. This graphical explanatory method
uncovers what are some of the most prominent ways a large number of related
distributions differ from the anticipated $f_{0}(x)$.

Interpretation. What can we learn from the LP-map? It tells us the nature of
non-Gaussianity of the error distributions for all 24 of the experiments in a
compact way. The ICA-diagram detects three major directions of departure (from
assumed Gaussian law) that more or less consistently appeared across different
days of the experiment: (i) excess variability: This is indicated by the large
positive values of the second-order LP-coefficients (2nd column of the LP-
matrix) $\\{\operatorname{LP}_{t2}\\}_{1\leq t\leq 24}$. (ii) Asymmetry: It is
interesting to note that the values of $\\{\operatorname{LP}_{t3}\\}_{1\leq
t\leq 24}$ change from positive to negative somewhere around the 15th
day—which implies that the skewness of the distributions switches from left-
skewed to right-skewed around the middle of the experiment. (iii) Long-
taildness: Large positive values of $\\{\operatorname{LP}_{t4}\\}_{1\leq t\leq
24}$ strongly indicate that the measurement densities are heavily leptokurtic,
i.e., they possess larger tails than normal. These fatter tails generate large
(or small) discrepant errors more frequently than expected—as we have
witnessed in Fig. 1.

Laplace Error Distribution. Here we choose $f_{0}(x)$ as the
$\texttt{Laplace}(\mu,s)$ distribution:

$f_{0}(x)={\frac{1}{2s}}\exp\left(-{\frac{|x-\mu|}{s}}\right)\,\\!,\leavevmode\nobreak\
\leavevmode\nobreak\ x\in\mathcal{R}$

where $\mu\in\mathcal{R}$ and $s>0$. The unknown parameters are estimated
using the maximum likelihood (MLE) method that automatically yields robust
estimates: sample median for the location parameter $\mu$ and mean absolute
deviation from the median for the scale parameter $s$.

The LP-map after applying the ICA algorithm is displayed in Fig. 3(b), which
shows that a moderate degree of skewness and a major tail-repairing are needed
to make Laplace consistent with the data. It is important to be aware of them
to build a more realistic model of errors—which is pursued in the next
section.

Laplace or Gauss? Following (14), compute

${\rm SI}(F_{0t},\widetilde{F}_{t})\leavevmode\nobreak\ =\leavevmode\nobreak\
\sum_{j=1}^{4}\Big{|}\,\widehat{\operatorname{LP}}[t,j]\,\Big{|}^{2},$ (22)

by taking the sum of squares of each row of the LP-matrix. Fig. 4 compares the
surprisal-index for the normal and Laplace distribution over $24$ experiments.
From the plot, it is evident that Peirce’s data were better represented by
Laplace than by Gaussian.

Figure 4: Shows the surprisal-indices (22) for the Gauss and Laplace models,
over the sequence of $24$ experiments. A ‘small’ value of ${\rm
SI}(F_{0},\widetilde{F})$ indicates that it is comparatively easier to repair
$f_{0}(x)$ to fit the data. The plot provides a clear rational basis for
choosing Laplace as a preferred model-0 since the blue curve consistently
exceeds the orange curve for most experiments.

###  5.2 Examples

The primary goal here is to show how density-sharpening provides a statistical
method for repairing a scientifically meaningful model based on observed data.
To that end, we apply the theory of density-sharpening to two specific day
studies with $f_{0}(x)$ as the Laplace distribution.

Figure 5: Mechanism of density-sharpening for 11th (1st column) and 13th (2nd
column) day experiments. Top row: Display the best-fitted Laplace
distributions $f_{0}$. Middle row: Displays the estimate
$\widehat{d}(u;F_{0},\widetilde{F})$, which reveals the nature of statistical
uncertainties of the Laplace models. Put it simply, the shape of
$\widehat{d}(u;F_{0},\widetilde{F})$ answers the central question of
discovery: What have we learned from the data that we did not already know?
This helps to invent new hypotheses that are worthy of pursuit. Last row:
Estimated SharpLaplace models $f_{0}(x)\times\widehat{d}_{0}(x)$ are shown
(red curves). Here $\widehat{d}_{0}(x)$ rectifies the shortcomings of Laplace
model by “bending” it in a data-adaptive manner.

Study #11 (July 19, 1872). The best fitted Laplace distribution with mean
$213$ and scale parameter $25.7$ is shown in the top left of Fig. 5. The
estimated sharpening kernel with smoothed LP-coefficients (see Sec. 4.2) is
given below:

$\widehat{d}_{0}(x)\,:=\,\widehat{d}(F_{0}(x);F_{0},F)\,=\,1+0.095T_{3}(x;F_{0})-0.148T_{4}(x;F_{0}),$
(23)

and is shown in the middle panel.

The graphical display of $d_{0}(x)$ provides actionable insights into how to
modify the Laplace distribution to reduce the empirical surprise. The non-zero
$\widehat{\operatorname{LP}}_{3}$ and $\widehat{\operatorname{LP}}_{4}$
indicates that the Laplace distribution needs to be corrected for skewness and
kurtosis, which is accomplished via LP-orthogonal series
$\operatorname{DS}(F_{0},m)$ model (19):

$\widehat{f}(x)\leavevmode\nobreak\ =\leavevmode\nobreak\
{\frac{1}{2s_{0}}}\exp\left(-{\frac{|x-\mu_{0}|}{s_{0}}}\right)\,\Big{[}1\,+\,0.095T_{3}(x;F_{0})\,-\,0.148T_{4}(x;F_{0})\Big{]},$
(24)

where $(\mu_{0},s_{0})=(213,25.7)$. The model (24) sharpens the assumed
Laplace law to render it more closer to the observed fact. The bottom-left
panel of Fig. 5 displays the asymmetric Laplace distribution with a shorter
left-tail. The maxent $\operatorname{DS}(F_{0},m)$ estimate

$\widehat{\widehat{f\,}}(x)\leavevmode\nobreak\ =\leavevmode\nobreak\
{\frac{1}{2s_{0}}}\exp\left(-{\frac{|x-\mu_{0}|}{s_{0}}}\right)\,\exp\Big{\\{}\,0.098T_{3}(x;F_{0})\,-\,0.153T_{4}(x;F_{0})\,-\,0.0152\Big{\\}},$
(25)

whose shape is almost indistinguishable from that of (24), and thus not
displayed.

Study #13 (July 22, 1872). We apply the same steps to derive the error
distributions of the day-13 experiment. We choose $f_{0}(x)$ as
Laplace$(244,20)$ and estimate the density-sharpening function:

$\widehat{d}_{0}(x)\,:=\,\widehat{d}(F_{0}(x);F_{0},F)\,=\,1-0.256T_{4}(x;F_{0}).$
(26)

The shape of $\widehat{d}_{0}(x)$ clearly indicates that the peak and the
tails of the initial Laplace distribution need repairing.

Ampliative character of our modeling: The DSF $\widehat{d}_{0}(x)$ allows the
Laplace to identify its own limitations and drives it to evolve into a new,
more complete one:

$\widehat{f}(x)\leavevmode\nobreak\ =\leavevmode\nobreak\
{\frac{1}{2s_{0}}}\exp\left(-{\frac{|x-\mu_{0}|}{s_{0}}}\right)\,\Big{[}1\,-0.256T_{4}(x;F_{0})\Big{]},$
(27)

with $(\mu_{0},s_{0})=(244,20)$; shown in the bottom-right panel of Fig. 5.
Compared with the Laplace distribution (the blue curve), the $d$-modified
Laplace (the red curve) is much wider with a rounded peak and clipped tails.

##  6 A Generalized Law of Errors

We have seen that for Peirce data, Laplace distribution was surprised in
different manners for different experiments (refer Fig. 3b and Fig. 5): e.g.,
on day 11, the Laplace model got puzzled by the discrepancy in skewness, and
tail of the measurement distribution, whereas on day 13, the surprise was
mainly due to tail differences. The question naturally arises: how should a
Laplace model respond to unexpected changes in data? A proposal for
generalized law of errors is given that allows Laplace to automatically adapt
to new data environments.

######  Definition 3 (Self-improving Laplace model).

We call $X\sim$ SharpLaplace($m$), when the density of $X$ obeys the following
parameterizable form:

$f(x)\leavevmode\nobreak\ =\leavevmode\nobreak\
{\frac{1}{2s}}\exp\left(-{\frac{|x-\mu|}{s}}\right)\,\exp\Big{\\{}\sum_{j=1}^{m}\theta_{j}T_{j}(x;F_{0})\,-\,\Psi(\bm{\theta})\Big{\\}},\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ x\in\mathcal{R}.\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ $ (28)

The insights gained from the analysis done in Sec. 5.2 suggest that $m=4$ or
$6$ could be perfectly reasonable for most practical purposes. The power of
this model lies in its capacity to self-modify its structure in a data-driven
manner.

######  Remark 13.

SharpLaplace class of error models has inbuilt “rules” (principles and
mechanisms) that tell a Laplace how to adapt with the data in a completely
autonomous fashion without being pre-programmed into them. This auto-adaptive
nature makes this model realistic enough to be useful for a wide range of
scientific applications.

##  7 Peirce’s Law of Discovery

> Not the smallest advance can be made in knowledge beyond the stage of vacant
> staring, without making an abduction at every step. — C. S. Peirce (1901)

All empirical scientific inquiry goes through three fundamental inferential
phases:

  * •

Discovery: developing new testworthy hypotheses;

  * •

Hypothesis testing: confirming the plausibility of a hypothesis;

  * •

Prediction: predicting by extrapolating the acceptable model.

Over the last century or so, statistical inference has been dominated by
hypothesis testing and prediction191919Discovery is much harder than
prediction because one can go away with good prediction without understanding.
But for discovery, understanding ‘how and why’ is a must. problems, virtually
neglecting the key question of where the reasonable hypothesis came from,
leaving it to the scientists’ imagination and speculation. In the article
‘Statistics for Discovery,’ George E. P. Box argued that

> [S]tatistics has been overly influenced by mathematical methods rather than
> the scientific method and consequently the subject has been greatly skewed
> towards testing rather than discovery. — George Box (2001)

Our focal interest is in the problem of discovery, not confirmation or
prediction. We showed how density-sharpening based modeling can provide the
basis for developing statistical laws of discovery.

Charles Sanders Peirce introduced the idea of abductive inference (as opposed
to inductive inference) to describe the process of generating hypotheses in
order to explain surprising facts. He developed abductive reasoning over $50$
years (between 1865 and 1914), and it is considered as Peirce’s most
significant contribution to the logic of science. According to Peirce,
abduction ‘is the only logical operation which introduces any new idea.’ The
importance of abduction for scientific discovery was further stressed by
Heckman and Singer (2017):

> Abduction is different from falsification or corroboration. It moves
> descriptions of the world forward rather than just confirming or falsifying
> hypotheses. It is part of a process of discovery where model reformulation,
> revision of hypotheses and addition of new information are part of the
> process.

Abductive Inference Machine (AIM). This paper lays out a proposal for
algorithmic operationalization of Peircean style abductive inference and
discovery. In particular, we described how the density-sharpening principle
can help us design an Abductive Inference Machine (AIM202020It tells empirical
analysts where to AIM as they search for possible new discoveries.) that (i)
allows us to properly handle model uncertainty and misspecifications; (ii)
produces abductive instinct—by guiding us to make better decisions (than
depending on pure luck alone) in formulating and adopting new promising
hypotheses that have a better chance of being true; and (iii) generates a
preferred course of actions for extracting statistical models from
experimental data by revising an initially misspecified scientific model.

##  8 AIM: Science of Model Development and Revision

AIM is a theory of model-revision, not parameter estimation (MLE/Bayes/robust
methods) or curve-fitting (machine learning methods). There are some unique
objectives and challenges, which set it apart from traditional data modeling
cultures. In the following, we will highlight a few major ones (D1-D7), taking
help from the Peirce data analysis done in Section 5.

Four stages of abductive model building: 1\. Initial state. AIM starts with an
approximate model $f_{0}$ (based on, say, some economic theory) and
measurements.212121In our context, the theory of Laplace’s law of error was
confronted with Peirce’s experimental data. The top left density in Fig. 5
shows the best-fitted theoretical model—the Laplace distribution for
experiment #11, where the unknown parameters were estimated using MLE.

D1

Non-standard inferential questions: Justification $\rightarrow$ discovery.
However, for modern econometricians and policymakers, parameter estimation or
significance testing routines (classical inference; see Haavelmo (1944)) are
not the most interesting issues. Modern quantitative economists are more
concerned with questions like: “How far is our speculated model from reality?
What are the most important gaps in our understanding? In which directions can
I improve my theory-based model?” Developing a general approach to answering
these modeling questions is the central imperative of AIM .

2\. Encountering surprise. ‘Surprise’ jumpstarts the abductive learning
process. Surprise essentially means that there is something new in the data
relative to the assumed model $f_{0}$, which we estimate by ${\rm
SI}(F_{0},F)$ following Eq. (14). Simply put, it quantifies how much new
information is left on the data to be explained; see the orange curve in Fig.
4. Intelligent learners (agents) utilize surprise as a source of additional
information to learn something new about the phenomena.

D2

Model-disequilibrium theory. A ‘large’ value of ${\rm SI}(F_{0},F)$ indicates
the model is ‘out of equilibrium’ with the current environment, and to restore
equilibrium, a careful revision of the current theoretical model (beyond
parameter tuning) is necessary. But how do we sharpen the existing model? Can
we develop an automatic procedure to generate the sharpening rules? These
questions are beyond the reach of classical statistical learning methods. AIM
approaches these questions by first characterizing the “knowledge-gap” between
the postulated theory and the observed measurements.

3\. Discovering the knowledge-gap. Fig. 5 (left of middle panel) displays the
estimated sharpening kernel $\widehat{d}(u;F_{0},F)$ for experiment #11, which
acts as a ‘channel’ through which information flows from the data to the
model—obeying the density-sharpening principle—to bring the system (model-
data) back to equilibrium. Accordingly, $\widehat{d}(u;F_{0},F)$ acts as a
“bridge” between a theorized model and actual measurements. The following
remark by Trygve Haavelmo (1944) highlights how crucial this accomplishment
is:

> The method of econometric research aims, essentially, at a conjunction of
> economic theory and actual measurements, using the theory and technique of
> statistical inference as a bridge pier.

D3

Hypotheses generation. Abduction is the logic of discovery. Why do we need a
logic for discovery? Charles Peirce, Herbert Simon, and many other prolific
researchers believed that a trial-and-error search for “invention” is seldom a
worthwhile strategy, especially for complex systems (like economics, biology,
etc.). Norwood Hanson said it beautifully in his book:

> “If establishing an hypothesis through its predictions has a logic, so has
> the conceiving of an hypothesis.” —Patterns of Discovery (1958)

AIM helps scientists to make educated guesses—on what’s the next best
hypothesis to try from a vast pool of conceivable collections—by autonomously
learning new realities from the data through $d_{0}(x)$. Classical inference,
on the other hand, mainly deals with the confirmatory or predictive side of
data analysis, not hypotheses generation and discovery.

D4

Modeling surprise, not the full data. AIM searches for patterns in the
“unexplained rest”—the parts of the data that were left unexplained by the
existing theory. Notice that we are not blindly searching for patterns in the
full data; we are only focusing on the novel parts of the data that contain
new information. It is important to distinguish between these two aspects. Our
goal is to accelerate discovery by synthesizing a simple explanatory model
$\widehat{d}(u;F_{0},F)$ for the surprising phenomena.

D5

Information-filtering unit. The density-sharpening kernel acts as a filter
that discards redundant information and compactly summarizes the new
information (knowledge-gap) as a probability density function.222222As Herbert
Simon said: “Anything that gives us new knowledge gives us an opportunity to
be more rational.” From that perspective, AIM could be a powerful tool to
guide economic agents in making rational decisions under uncertainty. More
details can be found in Mukhopadhyay (2022b). The non-zero LP-coefficients of
$\widehat{d}(u;F_{0},F)$ identify the missing elements of reality in the
current theoretical model. For example, Eq. 23 implies that the Laplace law is
misspecified in terms of symmetry (3rd order) and long-tailedness (4th order);
also check the 11th row of the LP-map shown in Fig. 3. Standard statistical
learning methods don’t have any such capabilities.

4\. Model-editing. AIM fills the knowledge-gaps by revising initial
probability model based on the principle of density-sharpening, which can be
described by a simple logical formula:

$\framebox{\text{Hypothesized-Model}}\leavevmode\nobreak\ \leavevmode\nobreak\
+\leavevmode\nobreak\ \leavevmode\nobreak\ \framebox{\text{New
data}}\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\
\xRightarrow[\texttt{Sharpening}]{d(u;F_{0},F)}\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\framebox{\text{Improved-Model}}\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ $ (29)

Every successive iteration of the above procedure generates a more realistic
model than its predecessors. Here the density-sharpening kernel $d_{0}(x)$
represents the progress in our understanding, which makes the hypothesized
simplified model elastic enough to be adaptable for real-data.232323In other
words, we don’t believe in the ‘one-fits-all’ model. Our goal is to provide
economists with a systemic principle for iteratively revising their
preliminary models by confronting them with real-world data.

By executing (29), AIM designs a class of most pursuitworthy alternative
models for the data and selects the best one using the ‘law of lazy
update.’242424It is fundamentally different from model selection or multiple
hypothesis testing, which deals with a pre-determined set of alternative
models. Model discovery and model selection are two very different things. Eq.
25 shows the sharp-Laplace model for study #11, also displayed in the bottom
of Fig. 5. The parsimonious $d_{0}(x)$ keeps the final model ‘sophisticatedly
simple’252525Arnold Zellner (2007): ‘a much heavier emphasis on sophisticated
simplicity in econometrics is needed.’ by smoothly extending the hypothesized
Laplace model to explain the data.

D6

AIM $\neq$ Curve-fitting. One of the non-standard aspects of AIM is that it’s
not about building fancy empirical models starting from a clean slate—it’s
about building a statistical structure on top of the already existing
scientific knowledge base to advance the current theory.262626In more simple
terms, AIM = Learning from data by standing on the foundation of existing
knowledge. In doing so, it provides efficient ways of handling an idealized
simple model for discovering new knowledge from complex real-world data.

D7

Conservative-liberal coalition. Note that the derived sharp-probabilistic law
(e.g., eq. 25) combines the generality (generic features) of Laplace’s laws of
error with the specificity (stylized features) of Peirce’s data. This
coalition of conservative (sticking to current dogma) and liberal (openness to
course correction when necessary) ideologies makes our data modeling
philosophy stands out from traditional statistical and machine learning data-
fitting methods.

##  9 AI = Abductive Intelligence

When can we say a model is behaving intelligently? This is no easy question.
However, at least part of the answer, I believe, lies in inspecting how the
model reacts to surprise and adapts to changes. An intelligent model should
not be ‘brittle,’ which collapses all of a sudden upon encountering surprises
from the data. The model should have an ‘internal’ mechanism that help it rise
to the occasion by providing recommendations for how to put together
incomplete pre-existing knowledge (encoded in model-0) with the data. We refer
to this core intellectual component of any learning problem as “abductive
intelligence.”

Building models that are capable of improving themselves has been a dream of
computer scientists since the inception of the artificial intelligence field
in 1956. It was at the top of the agenda in the proposal written for the
Dartmouth Summer Research Project on artificial intelligence:

> “Probably a truly intelligent machine will carry out activities which may
> best be described as self-improvement.” —John McCarthy et al. (1955)

More than developing new ways of building data models, we need new principles
for sharpening an existing model’s infrastructure. Such a model, equipped with
self-improving capability, then gradually acquires more knowledge about the
environment by building increasingly refined models of reality.

> “Once we have devised programs with a genuine capacity for self-improvement
> a rapid evolutionary process will begin. As the machine improves both itself
> and its model of itself, we shall begin to see all the phenomena associated
> with the terms ‘consciousness,’ ‘intuition’ and ‘intelligence’ itself.”
> —Marvin Minsky (1966)

######  Remark 14 (Intelligence of a Model).

A model’s ‘intelligence’ is its capacity to change and remodel itself when
confronted with new data. Any model which is not capable of ‘self-improving’
is a disposable, dead model.

######  Remark 15 (Designing an Intelligent Model).

Model is a work in progress; there is no such thing as the ‘final model.’ The
important part is knowing how to expand the knowledge base by sharpening
yesterday’s version. With that being said, the focus of present-day machine
learning has primarily been on developing a good subordinate-model $f_{0}(x)$
but not so much on designing the supervisory-model $d_{0}(x)$. To achieve the
goal of designing intelligent machines, it seems inevitable that we have to
shift our attention to the ‘supervisory’ part of the model—that which knows
how to react to surprise in order to evolve to its next avatar. In the long
run, the durability of a model depends more on its skill to adapt than its
built-in skill. And to design such self-improving models, we will need to
instill ‘abductive intelligence’ in the existing machine learning
systems—which is the goal of AIM.

##  10 Peirce: An Explorer Upon Untrodden Ground

I was an explorer upon untrodden ground — C. S. Peirce (1902)

Bertrand Russell (1959) described Charles Sanders Peirce as “one of the most
original minds of the later nineteenth century, and certainly the greatest
American thinker ever.” He was a craftsman of the highest order, who made some
eminent contributions to the development of nineteenth-century American
Statistics. Peirce analyzed his 1872 experimental data in the paper “On the
theory of errors of observation”—which is a gold mine of ideas. His techniques
and philosophy of data analysis reported in the paper were a testament to his
brilliance as a master applied statistician. On the landmark occasion of 150th
anniversary of his famous 1872 study,272727A miraculous year: In 1872, exactly
the same year of Peirce’s experiment, Ludwig Boltzmann established the
probabilistic (statistical) foundation of the entropy function—the birth of
modern information theory. As we have seen in this article, Peircean abduction
and information theory are intimately connected through the concept of
density-sharpening. As a matter of fact, AIM stands on three pillars:
Abductive inference + information theory + density-sharpening law. we have
looked back at his views on empirical modeling. The current article offers a
framework that embraces and operationalizes the Peircean view of discovery and
statistical modeling. We called this framework AIM–Abductive Inference
Machine—which is grounded in the principle of density sharpening and a new
class of models called “dyadic models.” We have illustrated the key
algorithmic steps and philosophical aspects of our modeling approach using
Peirce’s 1872 experimental data to reveal new insights on the probabilistic
nature of measurement uncertainties.

## Statements and Declarations

The Author declares that there is no conflict of interest/competing interests.

## Acknowledgement

The author was benefited from conversations and/or correspondence with James
Heckman, Burton Singer, and Stephen Stigler. We also thank the editorial board
for their thoughtful and philosophical comments on the manuscript.

2022 marks the 150th anniversary of C. S. Peirce’s historic experiment—which
according to Stigler (1978) is one of the most significant statistical
investigations in the history of nineteenth-century American mathematical
research. This paper celebrates this landmark occasion by introducing a modern
approach to Peircean abduction, which is readily applicable in practice.

## REFERENCES

  * Bailey (2017) Bailey, D. C. (2017).  Not normal: the uncertainties of scientific measurements.  Royal Society open science 4(1), 160600.
  * Box (2001) Box, G. (2001).  Statistics for discovery.  Journal of Applied Statistics 28(3-4), 285–299.
  * Box (1976) Box, G. E. (1976).  Science and Statistics.  Journal of the American Statistical Association 71(356), 791–799.
  * Buansing et al. (2020) Buansing, T. T., A. Golan, and A. Ullah (2020).  An information-theoretic approach for forecasting interval-valued sp500 daily returns.  International Journal of Forecasting 36(3), 800–813.
  * Burch and Parker (2022) Burch, R. and K. A. Parker (2022).  Charles Sanders Peirce.  In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy (Summer 2022 ed.). Stanford University.
  * Crease (2009) Crease, R. P. (2009).  Charles Sanders Peirce and the first absolute measurement standard.  Physics Today 62(12), 39.
  * Golan (2018) Golan, A. (2018).  Foundations of info-metrics: Modeling, inference, and imperfect information.  Oxford University Press.
  * Haavelmo (1944) Haavelmo, T. (1944).  The probability approach in econometrics.  Econometrica: Journal of the Econometric Society, iii–115.
  * Hanson (1965) Hanson, N. R. (1965).  Patterns of discovery: An inquiry into the conceptual foundations of science.  CUP Archive.
  * Heckman and Singer (2017) Heckman, J. J. and B. Singer (2017).  Abducting economics.  American Economic Review 107(5), 298–302.
  * Jaynes (1957) Jaynes, E. T. (1957).  Information theory and statistical mechanics.  Physical review 106(4), 620.
  * Laplace (1774) Laplace, P. S. (1774).  Mémoire sur la probabilité de causes par les évenements.  Mémoire de l’académie royale des sciences.
  * Lee et al. (2021) Lee, T.-H., M. Y. Mao, and A. Ullah (2021).  Maximum entropy analysis of consumption-based capital asset pricing model and volatility.  Journal of Econometric Methods 10(1), 1–19.
  * Maasoumi (1993) Maasoumi, E. (1993).  A compendium to information theory in economics and econometrics.  Econometric reviews 12(2), 137–181.
  * Mao et al. (2020) Mao, Y., A. Ullah, et al. (2020).  Information theoretic estimation of econometric functions.  Forthcoming in Advances in Info-Metrics: Information and Information Processing across Disciplines.
  * McCarthy et al. (1955) McCarthy, J., M. L. Minsky, N. Rochester, and C. E. Shannon (1955).  A proposal for the Dartmouth Summer Research Project on Artificial Intelligence. Online: http://www-formal.stanford.edu/jmc/history/dartmouth.pdf.
  * Minsky (1966) Minsky, M. (1966).  Artificial intelligence.  Scientific American, 215(3), 257.
  * Mukhopadhyay (2017) Mukhopadhyay, S. (2017).  Large-scale mode identification and data-driven sciences.  Electronic Journal of Statistics 11(1), 215–240.
  * Mukhopadhyay (2022a) Mukhopadhyay, S. (2022a).  A maximum entropy copula model for mixed data: Representation, estimation, and applications.  in press, arXiv:2203.03040, 1–35.
  * Mukhopadhyay (2022b) Mukhopadhyay, S. (2022b).  Modelplasticity and abductive decision making.  Technical Report, arXiv:2203.03040, 1–27.
  * Mukhopadhyay and Fletcher (2018) Mukhopadhyay, S. and D. Fletcher (2018).  Generalized empirical Bayes modeling via frequentist goodness-of-fit.  Nature: Scientific Reports 8(9983), 1–15.
  * Mukhopadhyay and Parzen (2020) Mukhopadhyay, S. and E. Parzen (2020).  Nonparametric universal copula modeling.  Applied Stochastic Models in Business and Industry, special issue on “Data Science” 36(1), 77–94.
  * Parkes and Wellman (2015) Parkes, D. C. and M. P. Wellman (2015).  Economic reasoning and artificial intelligence.  Science 349(6245), 267–272.
  * Peirce (1902) Peirce, C. (1902).  Minute logic, Chapter I. Intended characters of this treatise.  Digital Companion to CS Peirce.
  * Peirce (1873) Peirce, C. S. (1873).  On the theory of errors of observation.  Report of the Superintendent of the U.S. Coast Survey, 200–224.
  * Peirce (1901) Peirce, C. S. (1901).  The proper treatment of hypotheses: A preliminary chapter, toward an examination of hume’s argument against miracles, in its logic and in its history.  MS 692, 890–904.
  * Peirce ( 58) Peirce, C. S. (1931–58).  The collected papers of Charles Sanders Peirce, eds.  C Hartshorne, P Weiss (vols. 1-6) and A Burks (vols. 7-8), Cambridge, MA: Harvard University Press, 1931-58 and 1960-66.
  * Peirce (2009) Peirce, C. S. (2009).  Writings of Charles S. Peirce: a chronological edition, volume 3: 1872–1878, Volume 3.  Indiana University Press.
  * Rao (2001) Rao, C. R. (2001).  Statistics: Reflections on the past and visions for the future.  Communications in Statistics-Theory and Methods 30(11), 2235–2257.
  * Russell (1959) Russell, B. (1959).  Wisdom of the West, a Historical Survey of Western Philosophy in Its Social and Political Setting.  Garden City, NY: Doubleday.
  * Simon (1988) Simon, H. A. (1988).  The science of design: Creating the artificial.  Design Issues, 67–82.
  * Stigler (1978) Stigler, S. M. (1978).  Mathematical statistics in the early states.  The Annals of Statistics, 239–265.
  * Tukey (1969) Tukey, J. W. (1969).  Analyzing data: Sanctification or detective work?  American Psychologist 24(2), 83.
  * Tukey (1972) Tukey, J. W. (1972).  Exploratory data analysis: As part of a larger whole.  In Proceedings of the 18th conference on design of experiments in Army research and development I. Washington, DC, Volume 1010.
  * Tukey (1977) Tukey, J. W. (1977).  Exploratory data analysis, Volume 2.  Reading, Mass.
  * Wilson and Hilferty (1929) Wilson, E. B. and M. M. Hilferty (1929).  Note on C. S. Peirce’s experimental discussion of the law of errors.  Proceedings of the National Academy of Sciences of the United States of America 15(2), 120.
  * Zellner (2007) Zellner, A. (2007).  Philosophy and objectives of econometrics.  Journal of Econometrics 136(2), 331–339.