#  The end of spacetime

Mir Faizal<EMAIL_ADDRESS>Irving K. Barber School of Arts and
Sciences, University of British Columbia - Okanagan, Kelowna, British Columbia
V1V 1V7, Canada Canadian Quantum Research Center 204-3002 32 Ave Vernon, BC
V1T 2L7 Canada

###### Abstract

We will highlight that despite there being various approaches to quantum
gravity, there are universal approach-independent features of quantum gravity.
The geometry of spacetime becomes an emergent structure, which emerges from
some purely quantum gravitational degrees of freedom. We argue that these
quantum gravitational degrees of freedom can be best understood using quantum
information theory. Various approaches to quantum gravity seem to suggest that
quantum gravity could be a third quantized theory, and such a theory would not
be defined in spacetime, but rather in an abstract configuration space of
fields. This supports the view that spacetime geometry is not fundamental,
thus effectively ending the spacetime description of nature.

In Newtonian mechanics, space and time occur as distinct physical concepts,
and events occur at a place in space and at a point in time. This concept does
not change with respect to who is observing those events. In this sense,
Newtonian mechanics makes space and time an absolute objective observer-
independent concept n1 . Special relativity challenges this notion of space
and time and merges them into a single physical quantity, which is usually
called spacetime rela . This concept of spacetime still remains an absolute
objective observer-independent concept, but both space and time cease to exist
as independent concepts in special relativity. If taken individually, they
explicitly depend on the relative motion of the person making the measurement.
Despite space and time losing independent meaning, the geometrical structure
of spacetime is flat and fixed in special relativity. General relativity makes
spacetime dynamic, and according to general relativity, spacetime curves in
presence of matter gene . In fact, it is this very curvature of spacetime that
causes gravity to act on objects. Among other things, proof of this concept
occurs by correctly predicting the bending of light by stars light . According
to Newtonian gravity, only massive particles can be affected by gravity. As
light classically is a massless wave (neglecting the quantum theory of light),
it should not have been affected by the gravitational force. Now according to
special relativity, light has to travel in a straight line, so it should have
continued to do so even in the neighborhood of stars. However, its bending
near stars, demonstrates that gravity is actually the curvature of spacetime,
and due to the curvature of spacetime, a straight line is actually a curve.

In both special and general relativity, even though space and time are
subjective notions, spacetime is an objective observer-independent concept.
However, the very concept of objective observer independence has to be totally
discarded in quantum mechanics quan . Quantum mechanics is not a mere
probabilistic theory, like statistical mechanics. In quantum mechanics, the
very act of measuring any event changes the system. As an observer performs
these measurements, it becomes impossible to define reality in an objective
observer-independent way. However, apart from the measurement problem meas ,
which only becomes important in purely quantum phenomena like entanglement
enta  (a quantum property by which two quantum particles remain connected even
when separated by vast distances, and causes purely quantum effects described
by the Bell’s theorem), quantum mechanics for all practical purposes can be
reduced to a statistical theory. This is because, even though we cannot
predict the outcomes of individual experiments, we can assign probabilities to
them, and these probabilities match with the observations for a larger number
of such measurements. This deep relation between quantum averages and
statistical averages is the reason quantum mechanical tools like path
integrals can be used as calculation devices in statistical mechanics path .
But despite the mathematical similarity between classical statistical
mechanics and quantum mechanics, these theories are conceptually very
different from each other.

It has been possible to reconcile special relativity with quantum mechanics,
and the theory which does this is called quantum field theory qft .
Conceptually, it is known that if we naively try to write a relativistic
generalization of the Schrödinger equation, we get inconsistencies. These
inconsistencies known to occur in the Klein-Gordon equation could be resolved
in the Dirac equation. However, if we treat even the Dirac equation as a
quantum mechanical equation, we cannot account for the dynamical creation and
annihilation of particles, which occurs in quantum field theory. The deep
relation between the quantum systems and statistical mechanics is used to
first express the relativistic equation like the Klein-Gordon equation as a
’classical’ equation of a classical field, and then it is quantized again seco
. The quantum mechanical observer dependence is absorbed in this second
quantization of the field, and the original Klein-Gordon equation is treated
as a classical deterministic equation. The advantage of doing this is that now
it becomes a quantum theory of a field, where its perturbative modes act as
particles, and hence the particles can be created and annihilated by any non-
linear terms involving those fields. Thus, elementary particles (such as
electrons, photons, etc) of nature stop being fundamental (as irreducible
building blocks) as they can be produced due to the collision of other
particles. It may be noted that this is only possible in second quantized
theories, with non-linear terms. In first quantized theories, these particles
are truly fundamental, and cannot transform into different particles.
Furthermore, it is possible to even reconcile quantum mechanics, with general
relativity, if spacetime is not quantized. This is done using quantum field
theory in curved spacetime qftc . In this theory, a stationary observer does
not observe any particles, whereas an accelerating observer observes particles
at a certain temperature un ; un12 . This effect called the Unruh effect has
even been experimentally observed un14 . Thus, the very notion of particles
also becomes an observer-dependent concept and depends on the acceleration of
an observer.

However, it has not been possible to fully unify quantum mechanics with
general relativity, in a complete theory, where even spacetime will be
quantized. Even though we do not have a fully consistent quantum theory of
gravity, we have several approaches to quantum gravity. The interesting
observation is that these approaches seem to make certain universal
predictions about the nature of quantum gravity. To investigate these
universal predictions, we observe that we need more energy to probe spacetime
at smaller lengths scales. We need Planck energy to probe a Planckian region.
However, if we put Planck energy in the Planckian region, we will form a mini
black hole, which will in turn prevent such measurement from taking place bl .
Now any theory of quantum gravity has to be consistent with classical black
hole physics. Thus, it seems that in any quantum theory of gravity, spacetime
should have a natural minimal length le  and a minimal time ti . This would
also solve another problem in quantum field theories. In quantum field
theories, we get divergences or infinite answers. To get finite answers, we
need to introduce a cutoff by hand, which is a scale beyond which we do not
probe our system. However, if spacetime does have a minimal length and a
minimal time, it will naturally act as a cutoff for any quantum field theory
described on such a spacetime cut .

Perturbative string theory does have such a minimal length, as it is not
possible to probe spacetime below string length scale pert . This is because
in perturbative string theory, string length is the smallest probe available,
and we cannot probe spacetime below the length of the smallest probe. Then
taking the foundations of quantum mechanics seriously, any physical object
that cannot be probed does not physically exist, so string theory suggests
that spacetime does not exist below the string length scale. Even though we
have point-like objects known as D0-branes in non-perturbative string theory,
this conclusion still holds because of dualities in string theory. Due to a
duality, called T-duality, between string theory at larger and small scales,
it has been demonstrated that the spectrum of string theory above string
length scale is similar to the spectrum of string theory below string length
scale tdual . Hence, new information is not obtained by probing string theory
below the string length scale, and this length scale acts as a minimal length.
Similarly, a minimal length occurs in other discrete models of spacetime, like
the causal sets c1 , quantum graphity c2 , and causal dynamical triangulation
c4 , where it is not possible to probe the geometry below the length scale at
which discreteness occurs. Thus, the occurrence of a minimal length, or more
generally a minimal geometrical quantity as a natural cutoff of spacetime
seems to be an approach to independent observation in quantum gravity. Such a
minimal length also occurs in noncommutative nc10  and nonlocal nl10  quantum
field theories. In loop quantum cosmology a background independent quantized
called the polymer quantization has to be employed c5 ; c51 . In this
quantization, there is a natural minimal length called the polymer length.
Apart from that, a minimal area and a minimal volume occur naturally in loop
quantum gravity c6  and spin foam c10 . Thus, we still have a minimal value
for some geometric quantity, which acts as a natural cutoff for geometry. Such
a cutoff seems to be a generic feature of any theory of quantum gravity, even
though the details of its origin are approach dependent.

Such a cutoff for geometry resolves some intrinsic problems in general
relativity. General relativity predicts its own breakdown due to the
occurrence of singularities, where spacetime description of reality does not
hold. Furthermore, the Penrose-Hawking singularity theorems demonstrate that
these singularities are an intrinsic property of general relativity ph1 ; ph2
. However, quantum gravitational effects should modify these theories by
including a cutoff on geometry and prevent the occurrence of these
singularities li1 ; li2 . In string theory, a minimal length occurs due to
T-duality tdual , and T-duality also prevents the formation of singularities
tdual1 . Singularities are also removed in loop quantum cosmology due to the
cutoff on the geometry lqgs . As the geometry in any theory of quantum gravity
has a natural cutoff, it can be argued that the absence of singularity will be
a general feature of all approaches to quantum gravity. This is because the
Penrose-Hawking singularity theorems have been directly related to the
Bekenstein-Hawking entropy hpbe , so a bound on such an entropy from the
cutoff would naturally prevent the occurrence of singularities. As the
Bekenstein-Hawking entropy is directly related to geometry in the Jacobson
formalism jc , a modification to the Bekenstein-Hawking entropy will directly
modify the geometry of spacetime. In fact, it has been explicitly demonstrated
that the bound on the Bekenstein-Hawking entropy from the minimal length in
quantum gravity will prevent the formation of singularities in spacetime gup ;
gup2 . The singularities are prevented as a minimum value for a geometric
quantity would imply that spacetime geometry is an emergent structure, which
emerges from quantum gravitational degrees of freedom, and breaks down below
such a minimal value. Singularities occur in general relativity when we apply
it to describe scenarios, where the spacetime description of nature does not
hold. It is also interesting to note that this bound on the geometry follows
most directly from a bound on the quantum information gup ; gup2 , which seems
to indicate that spacetime geometry emerges from quantum information.

The emergence of spacetime geometry from purely non-geometric quantum states
can be most explicitly seen using the holographic principle. The holographic
principle relates a theory in a volume of a region to the theory on its
boundary holog . The holographic principle has been thoroughly applied to
study various aspects of string theory. Here, the holographic principle
relates a gravitational theory to the quantum field theory on its flat
boundary ads . Thus, it is studied as a duality between string theory, or more
commonly its supergravity approximation on anti-de Sitter spacetime (spacetime
with a constant negative scalar curvature) and a boundary conformal field
theory (a theory where angles are preserved but not lengths). The emergence of
spacetime geometry from non-geometric quantum degrees of freedom follows
directly from this form of the holographic principle ep1 ; bits ; b12 ; b14 .
This has been done by relating the geometric structures to the abstract notion
of information, which has been made concrete in quantum information theory. As
discussed before, even though most quantum mechanical phenomena can be mapped
onto statistical mechanical systems, there are purely quantum effects like
entanglement for which this cannot be done. Interestingly, it is the
entanglement of the quantum states in the boundary theory which gives rise to
the geometric structure. Removing entanglement in the boundary theory amounts
to destroying the geometric structures for dual theory. However, entanglement
has been demonstrated to be an observer-dependent phenomenon obser ; obser1 ;
obser2 ; obser4 . This also makes the geometric structures dual to such
entanglement also observer-dependent.

In loop quantum gravity c6  and spin foam c10 , geometry at macroscopic scales
emerges from loops woven into a network at the microscopic scale. Here, again
the geometry of spacetime emerges only as an approximation at a larger scale
compared to the Planck scale, much like the geometry of a table emerges from
atomic physics at a larger scale compared to the atomic scale. This also
occurs in other models of quantum gravity, with discrete degrees of freedom,
such as causal sets c1 , quantum graphity c2 , and causal dynamical
triangulation c4 . Thus, the idea that geometry emerges from non-geometric
states seems to be a universal prediction of various approaches to quantum
gravity. Furthermore, even though these other approaches to quantum gravity
are fundamentally different from string theory, even in those approaches,
geometry seems to emerge from quantum information theory Makela:2019vgf ;
bombelli ; konopka ; Skenderis:2006ah .

The idea of emergence can be made dynamical in third quantized theories t1 .
Just like in the second quantized theory, it is possible to dynamically create
and annihilate particles, it is possible to define a theory, where it is
dynamically possible to create and annihilate fields, and even spacetime t2 .
This theory naturally is not defined in spacetime, but in an abstract
configuration space of fields, including the metric, which accounts for the
geometry. Then the wave function of quantum field theory, which is defined for
fields, is again treated like a classical field, and a second quantized theory
is defined as a classical theory of an ensemble of fields. This theory is
quantized again, and the measurement problem is absorbed in the third
quantization of fields. Now it is possible to dynamically create and
annihilate those fields, including metric, and hence it is possible to
dynamically create and annihilate geometries. Various, approaches to quantum
gravity, if taken to their logical conclusion tend to end in some sort of a
third quantized theory. In loop quantum gravity along with some other models
of quantum gravity, third quantization has appeared in the form of group field
theory, where the group theoretical structure of certain groups is the
fundamental variable from which geometry emerges lt . In string theory, third
quantization has appeared as string field theory st02 , even though for
historical reasons string theory is called a first quantized theory (despite
being a second quantized conformal field theory), and string field theory is
called a second quantized theory (despite being a third quantized theory, with
dynamical creation and annihilation of fields).

Apart from quantum gravity, the calculations of particle scattering amplitudes
also suggest that spacetime geometry is an emergent structure, and physical
processes such as scattering amplitudes can be better understood without
reference to spacetime ar . It seems natural even from a purely cosmological
perspective that spacetime should be emergent. As the universe, which is
represented by geometry, was formed at the big bang, and so geometry could not
be fundamental but emergent cos1 ; cos2 . So, in all approaches to quantum
gravity, spacetime geometry emerges from the purely quantum gravitational
degrees of freedom, and it is the information contained in those degrees of
freedom that causes the geometrical structures to emerge. This view that
spacetime emerges from information goes back to John Wheeler and is usually
stated as ’it from bit not bit from it’ itfrombit . Here, ’it’ represents
geometry and structures such as matter defined on that geometry, and ’bit’
represents the abstract notion of information, from which that geometry
emerges. The quantum gravitational variant of this initial proposal can be
stated as ’it from qubit and not qubit from it’ with the information being
replaced by quantum information representing quantum gravitational degrees of
freedom. This quantum informational need not be defined in spacetime, and in
third quantized theories, it is not defined in spacetime. Furthermore, various
theories of quantum gravity, such as string theory, and loop quantum gravity,
along with some other approaches to quantum gravity are better understood
using a third quantized formalism. Here, we would like to also remark that the
spacetime in string theory is continuous despite there being a minimal length,
as the minimal length just sets a limit to the ability to probe the system.
However, this is equivalent to a discrete spacetime from an information
theoretical perspective, as it has been argued that like the information in
information theory, spacetime can be simultaneously described as a discrete
and continuous structure cutoff2 . Thus, from an information theoretical view,
these different approaches to quantum gravity might be different
representations of the same or some very similar theory. Thus, the universal
feature of fundamentally different approaches to quantum gravity is the
emergence of spacetime. The journey, which started with Newtonian mechanics,
with the definition of space and time as an absolute objective observer-
independent concept, seems to have reached a point, where the very concept of
spacetime has ended.

## References

  * (1) R. Rynasiewicz, Newton’s views on space, time, and motion. Stanford Encyclopedia of Philosophy (2008)
  * (2) U. E. Schroder, World Sci. Lect. Notes Phys. 33, 1 (1990)
  * (3) D. W. Sciama, Rev. Mod. Phys. 36, 463-469 (1964)
  * (4) P. Meszaros and H. Riffert, Astrophys. J. 327, 712-722 (1988)
  * (5) T. J. Hollowood, J. Phys. A 47, 185301 (2014)
  * (6) M. Schlosshauer, Rev. Mod. Phys. 76, 1267-1305 (2004)
  * (7) D. Paneru, E. Cohen, R. Fickler, R. W. Boyd and E. Karimi, Rept. Prog. Phys. 83, no.6, 064001 (2020)
  * (8) B. M. McCoy, Contribution to: 7th Physics Summer School on Statistical Mechanics and Field Theory, 26-128 (1994)
  * (9) S. Weinberg, Cambridge University Press (2005)
  * (10) M. Mcguigan, Physica A 158, 546-554 (1989)
  * (11) S. Hollands and R. M. Wald, Commun. Math. Phys. 293, 85-125 (2010)
  * (12) J. Hu, L. Feng, Z. Zhang and C. Chin, Nature Phys. 15, 785–789 (2019)
  * (13) E. Martín-Martínez, I. Fuentes and R. B. Mann, Phys. Rev. Lett. 107, 131301 (2011)
  * (14) M. H. Lynch, E. Cohen, Y. Hadad and I. Kaminer, Phys. Rev. D 104, 025015 (2021)
  * (15) M. Maggiore, Phys. Lett. B 304, 65-69 (1993)
  * (16) I. Pikovski, M. R. Vanner, M. Aspelmeyer, M. Kim and C. Brukner, Nature Phys. 8, 393 (2012)
  * (17) M. Faizal, M. M. Khalil and S. Das, Eur. Phys. J. C 76, no.1, 30 (2016)
  * (18) A. Kempf, Phys. Rev. Lett. 103, 231301 (2009)
  * (19) D. Amati, M. Ciafaloni and G. Veneziano, Phys. Lett. B 216, 41-47 (1989)
  * (20) M. Fontanini, E. Spallucci and T. Padmanabhan, Phys. Lett. B 633, 627-630 (2006)
  * (21) A. Belenchia, D. M. T. Benincasa and F. Dowker, Class. Quant. Grav. 33, no.24, 245018 (2016)
  * (22) S. A. Wilkinson and A. D. Greentree, Phys. Rev. D 90, no.12, 124003 (2014)
  * (23) J. Ambjorn, D. Coumbe, J. Gizbert-Studnicki and J. Jurkiewicz, Phys. Rev. D 93, no.10, 104032 (2016)
  * (24) M. Rinaldi, Class. Quant. Grav. 28, 105022 (2011)
  * (25) R. Brout, C. Gabriel, M. Lubo and P. Spindel, Phys. Rev. D 59, 044005 (1999)
  * (26) A. Ashtekar and P. Singh, Class. Quant. Grav. 28, 213001 (2011)
  * (27) M. Bojowald, Nature Phys. 3, 523–525 (2007)
  * (28) A. Ashtekar and E. Bianchi, Rept. Prog. Phys. 84, no.4, 042001 (2021)
  * (29) A. Perez, Class. Quant. Grav. 20, R43 (2003)
  * (30) S. W. Hawking and R. Penrose, Proc. Roy. Soc. Lond. A 314, 529 (1970)
  * (31) R. Penrose, Phys. Rev. Lett. 14, 57 (1965)
  * (32) S. Alsaleh, L. Alasfar, M. Faizal and A. F. Ali, Int. J. Mod. Phys. A 33, no.10, 1850052 (2018)
  * (33) E. C. Vagenas, L. Alasfar, S. M. Alsaleh and A. F. Ali, Nucl. Phys. B 931, 72-78 (2018)
  * (34) H. Wu and H. Yang, JCAP 07, 024 (2014)
  * (35) G. Date and G. M. Hossain, Phys. Rev. Lett. 94, 011302 (2005)
  * (36) M. Salah, F. Hammad, M. Faizal and A. F. Ali, JCAP 02, 035 (2017)
  * (37) A. Awad and A. F. Ali, JHEP 1406, 093 (2014)
  * (38) T. Jacobson, Phys. Rev. Lett. 75, 1260 (1995)
  * (39) R. Bousso and A. Shahbazi-Moghaddam, Phys. Rev. Lett. 128, no.23, 231301 (2022)
  * (40) A. Kempf, New J. Phys. 12, 115001 (2010)
  * (41) R. Bousso, Rev. Mod. Phys. 74, 825-874 (2002)
  * (42) J. L. Petersen, Int. J. Mod. Phys. A 14, 3597-3672 (1999)
  * (43) D. Jafferis, A. Zlokapa, J. D. Lykken, D. K. Kolchmeyer, S. I. Davis, N. Lauk, H. Neven and M. Spiropulu, Nature 612, 7938, 51-55 (2022)
  * (44) M. Van Raamsdonk, Science 370, no.6513, 198-202 (2020)
  * (45) L. Susskind, JHAP 1, no.1, 1-22 (2021)
  * (46) L. Susskind, JHAP 3, no.1, 1-30 (2023)
  * (47) A. Peres, P. F. Scudo, and D. R. Terno, Phys. Rev. Lett. 88, 230402 (2002)
  * (48) P. M. Alsing, I. Fuentes-Schuller, R. B. Mann and T. E. Tessier, Phys. Rev. A 74, 032326 (2006)
  * (49) J. Fan and X. Li, Phys. Rev. D 97, 016011 (2018)
  * (50) J. León and E. Martín-Martínez, Phys. Rev. A 80, 012314 (2009)
  * (51) J. Mäkelä, Int. J. Mod. Phys. D 28, no.10, 1950129 (2019)
  * (52) L. Bombelli, J. H. Lee, D. Meyer and R. Sorkin, Phys. Rev. Lett. 59, 521 (1987)
  * (53) T. Konopka, F. Markopoulou and S. Severini, Phys. Rev. D 77, 104029 (2008)
  * (54) K. Skenderis and M. Taylor, Phys. Rev. Lett. 98, 071601 (2007)
  * (55) J. A. Wheeler, in the Proceedings of the Third International Symposium on the Foundations of Quantum Mechanics, (Tokyo 1989)
  * (56) S. B. Giddings and A. Strominger, Nucl. Phys. B 321, 481-508 (1989)
  * (57) Y. Ohkuwa and Y. Ezawa, Class. Quant. Grav. 29, 215004 (2012)
  * (58) H. M. Haggard, New J. Phys. 20, no.7, 071001 (2018)
  * (59) T. Erler, Phys. Rept. 851, 1-36 (2020)
  * (60) N. Arkani-Hamed and J. Trnka, JHEP 10, 030 (2014)
  * (61) E. R. Harrison, Nature 215, 151–152 (1967)
  * (62) E. Hand, Nature 458, 820–824 (2009)


*arg min MIDLMedical Imaging with Deep Learning 2019 MIDL 2019 – Extended Abstract Track Max-Heinrich Laves<EMAIL_ADDRESS>
Sontje Ihler<EMAIL_ADDRESS>
Tobias Ortmaier<EMAIL_ADDRESS>
Institute of Mechatronic Systems, Leibniz Universität Hannover, Germany

# Uncertainty Quantification in Computer-Aided Diagnosis: Make Your Model say
“I don’t know” for Ambiguous Cases

###### Abstract

We evaluate two different methods for the integration of prediction
uncertainty into diagnostic image classifiers to increase patient safety in
deep learning111Code is available at https://github.com/mlaves/uncertainty-
midl. In the first method, Monte Carlo sampling is applied with dropout at
test time to get a posterior distribution of the class labels (Bayesian
ResNet). The second method extends ResNet to a probabilistic approach by
predicting the parameters of the posterior distribution and sampling the final
result from it (Variational ResNet). The variance of the posterior is used as
metric for uncertainty. Both methods are trained on a data set of optical
coherence tomography scans showing four different retinal conditions. Our
results shown that cases in which the classifier predicts incorrectly
correlate with a higher uncertainty. Mean uncertainty of incorrectly diagnosed
cases was between 4.6 and 8.1 times higher than mean uncertainty of correctly
diagnosed cases. Modeling of the prediction uncertainty in computer-aided
diagnosis with deep learning yields more reliable results and is anticipated
to increase patient safety.

###### keywords:

bayesian approximation, variational inference, optical coherence tomography

##  1 Introduction

In recent years, deep learning methods have received significant attention for
computer-aided diagnosis in a variety of fields of medical imaging. They
outperform former methods in capability and accuracy. However, deep models
trained for diagnosis of specific cases currently lack the ability to say “I
don’t know” for ambiguous or unknown cases. Standard models do not provide
prediction uncertainty, which is indispensable for the acceptance of deep
learning in medical practice.

Existing approaches for uncertainty estimation in deep learning try to
approximate a Bayesian neural network (BNN), where distributions are placed
over the weights [Gal and Ghahramani(2016)]. BNNs provide the mathematical
tool to model uncertainty, but are usually associated with limiting
computational cost. Gal et al. have shown, that the use of Monte Carlo
sampling with dropout at training and test time acts as approximation to a
Bayesian model without aggravating the complexity or the performance of the
model [Gal and Ghahramani(2016), Kendall et al.(2015)Kendall, Badrinarayanan,
and Cipolla]. This idea has attracted attention in medical segmentation [Roy
et al.(2019)Roy, Conjeti, Navab, and Wachinger, Laves et al.(2019)Laves,
Bicker, Kahrs, and Ortmaier]. Another method for uncertainty estimation is a
variational inference approach, where a deep model is trained to learn the
parameters of a probability distribution from which the final prediction is
sampled [Kingma and Welling(2013)]. Variational inference has been used
recently for uncertainty estimation in deformable registration of brain MRI
[Dalca et al.(2019)Dalca, Balakrishnan, Guttag, and Sabuncu].

In this work, the aforementioned approaches for uncertainty estimation are
integrated into diagnostic classifiers and compared in order to increase
patient safety and acceptance for deep models in medical imaging.

##  2 Methods

In the following, we will briefly revise the Bayesian and variational
inference approach for uncertainty estimation. Given a set of training images
$\bm{X}$ with corresponding labels $\bm{Y}$ from medical experts, we try to
find a probabilistic function $f_{\theta}:x\rightarrow y$ yielding the most
likely label prediction $\hat{y}$ of a test image $x$ with probability

$p(\hat{y}|x,\bm{X},\bm{Y})=\int
p(\hat{y}|x,\theta)p(\theta|\bm{X},\bm{Y})\mathrm{d}\theta~{},$ (1)

with parameters $\theta$ of the deep model $f$. The posterior distribution in
(1) is generally intractable and therefore, the integral can be approximated
by summing Monte Carlo samples obtained from $f_{\theta}$ with dropout at test
time [Gal and Ghahramani(2016)]. The mean of these samples is used as label
prediction $\hat{y}$ and the variance is interpreted as the uncertainty of the
prediction. We train the ResNet-18 image classifier on a dataset of 84,484
optical coherence tomographies showing four different retinal conditions [He
et al.(2016)He, Zhang, Ren, and Sun, Kermany et al.(2018)Kermany, Goldbaum,
Cai, Valentim, et al.]. Dropout with $p=0.5$ is added before the last fully
connected layer (referred to as _bayesian1_) and before every building block
of ResNet-18 (referred to as _bayesian2_), creating a bayesian classifier. In
Monte Carlo experiments, 100 forward passes are performed to get an
approximation of the posterior distribution of the class labels.

In the variational inference approach, we assume a normal distribution for the
posterior and replace the last fully-connected layer of ResNet with two fully-
connected layers predicting the parameters $\bm{\mu}$ and $\bm{\sigma^{2}}$ of
the posterior distribution (referred to as _variational_). The final
prediction is sampled from this distribution using the reparameterization
trick $\hat{\bm{y}}=\bm{\mu}+\bm{\sigma}\bm{\varepsilon}$ with
$\bm{\varepsilon}\sim\mathcal{N}(\bm{0},\bm{\mathrm{I}})$. As proposed in
[Kingma and Welling(2013)], we additionally try to bring the estimated
posterior closer to a standard normal distribution by adding the Kullback-
Leibler divergence (KLD) to the overall loss function. In this case, the KLD
can be solved analytically as
$\mathcal{L}_{\mathrm{KLD}}(\vec{\mu},\vec{\sigma}^{2})=-\frac{1}{2}\sum_{j=1}^{N}\left(1+\log(\sigma_{j}^{2})-\mu_{j}^{2}-\sigma_{j}^{2}\right)$.

After training the three approaches and a _baseline_ ResNet-18 for comparison,
we investigate the uncertainties for all true and false predictions for images
from the test set.

##  3 Results

Figure 1: Results from different approaches for uncertainty estimation of correctly (true) and incorrectly (false) diagnosed OCT scans. All methods yield a higher uncertainty for incorrect predictions. | baseline | bayesian1 | bayesian2 | variational
---|---|---|---|---
precision | 0.96 | 0.96 | 0.93 | 0.94
recall | 0.95 | 0.96 | 0.93 | 0.94
F1 score | 0.95 | 0.96 | 0.93 | 0.94
Table 1: _Bayesian1_ does not affect test set accuracy. Many dropout layers in
_bayesian2_ and the noise introduced by the reparameterization trick in
_variational_ seem to have a negative effect.

Fig. 1 show boxplots (top row) and relative frequencies (bottom row) for
uncertainties of correctly (true) and incorrectly (false) predicted cases from
the test set. The results shown that cases in which the network predicts
incorrectly correlate with a higher uncertainty. Mean uncertainty of
incorrectly diagnosed cases was 4.6 (variational), 6.0 (bayesian2) and 8.7
(bayesian1) times higher than mean uncertainty of correctly diagnosed cases.
Test set accuracies compared to a baseline ResNet-18 are listed in Tab. 1.

##  4 Conclusion

Modeling of the prediction uncertainty in computer-aided diagnosis with deep
learning yields more reliable results and is therefore anticipated to increase
patient safety. This can help to transfer such systems into clinical routine
and to increase the acceptance of physicians and patients for machine learning
in diagnosis. In future work, the uncertainties can be used to further
increase classification accuracy.

This research has received funding from the European Union as being part of
the EFRE OPhonLas project.

## References

  * [Dalca et al.(2019)Dalca, Balakrishnan, Guttag, and Sabuncu] Adrian V Dalca, Guha Balakrishnan, John Guttag, and Mert R Sabuncu.  Unsupervised Learning of Probabilistic Diffeomorphic Registration for Images and Surfaces.  _arXiv preprint arXiv:1903.03545_ , 2019.
  * [Gal and Ghahramani(2016)] Yarin Gal and Zoubin Ghahramani.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning.  In _International Conference on Machine Learning_ , pages 1050–1059, 2016.
  * [He et al.(2016)He, Zhang, Ren, and Sun] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep Residual Learning for Image Recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 770–778, 2016.
  * [Kendall et al.(2015)Kendall, Badrinarayanan, and Cipolla] Alex Kendall, Vijay Badrinarayanan, and Roberto Cipolla.  Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding.  _arXiv preprint arXiv:1511.02680_ , 2015.
  * [Kermany et al.(2018)Kermany, Goldbaum, Cai, Valentim, et al.] Daniel S. Kermany, Michael Goldbaum, Wenjia Cai, Carolina C.S. Valentim, et al.  Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning.  _Cell_ , 172(5):1122–1131, 2018.  10.1016/j.cell.2018.02.010.
  * [Kingma and Welling(2013)] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * [Laves et al.(2019)Laves, Bicker, Kahrs, and Ortmaier] Max-Heinrich Laves, Jens Bicker, Lüder A. Kahrs, and Tobias Ortmaier.  A dataset of laryngeal endoscopic images with comparative study on convolution neural network-based semantic segmentation.  _International Journal of Computer Assisted Radiology and Surgery_ , 14(3):483–492, 2019.  10.1007/s11548-018-01910-0.
  * [Roy et al.(2019)Roy, Conjeti, Navab, and Wachinger] Abhijit Guha Roy, Sailesh Conjeti, Nassir Navab, and Christian Wachinger.  QuickNAT: A fully convolutional network for quick and accurate segmentation of neuroanatomy.  _NeuroImage_ , 186:713–727, 2019.


11institutetext: University of Catania, Italy 22institutetext: ISTC - National
Research Council, Rome and Catania, Italy 33institutetext: University of
Bologna, Italy

# Do Language Models Understand Morality? Towards a Robust Detection of Moral
Content

Luana Bulla 1122 0000-0003-1165-853X Aldo Gangemi 2233 0000-0001-5568-2684
Misael Mongiovì 1122 0000-0003-0528-5490

###### Abstract

The task of detecting moral values in text has significant implications in
various fields, including natural language processing, social sciences, and
ethical decision-making. Previously proposed supervised models often suffer
from overfitting, leading to hyper-specialized moral classifiers that struggle
to perform well on data from different domains. To address this issue, we
introduce novel systems that leverage abstract concepts and common-sense
knowledge acquired from Large Language Models (LLMs) and Natural Language
Inference (NLI) models during previous stages of training on multiple data
sources. By doing so, we aim to develop versatile and robust methods for
detecting moral values in real-world scenarios. Our approach uses the GPT-
based Davinci model as a zero-shot ready-made unsupervised multi-label
classifier for moral values detection, eliminating the need for explicit
training on labeled data. To assess the performance and versatility of this
method, we compare it with a smaller NLI-based zero-shot model. The results
show that the NLI approach achieves competitive results compared to the
Davinci model. Furthermore, we conduct an in-depth investigation of the
performance of supervised systems in the context of cross-domain multi-label
moral value detection. This involves training supervised models on different
domains to explore their effectiveness in handling data from different sources
and comparing their performance with the unsupervised methods. Our
contributions encompass a thorough analysis of both supervised and
unsupervised methodologies for cross-domain value detection. We introduce the
Davinci model as a state-of-the-art zero-shot unsupervised moral values
classifier, pushing the boundaries of moral value detection without the need
for explicit training on labeled data. Additionally, we perform a comparative
evaluation of our approach with the supervised models, shedding light on their
respective strengths and weaknesses.

###### Keywords:

Large Language Models Value Detection Natural Language Inference Natural
Language Processing

##  1 Introduction

The detection of moral values is a critical area of research with wide-ranging
implications in fields such as Natural Language Processing (NLP), social
sciences, and ethical decision-making. Building robust automatic systems
capable of predicting the nuanced moral aspects of the text remains an ongoing
challenge, especially due to the limitations of current supervised systems,
which often suffer from overfitting the data distribution presented in the
training dataset [4]. This phenomenon results in the development of hyper-
specialized moral classifiers that excel at handling the specific distribution
of textual documents and domains to which they have been exposed during the
training phase. However, when confronted with data from sources with uncertain
distribution or originating from different domains, these classifiers have a
considerable drop in performance [23, 17].

To address this issue, we aim to leverage the unintentional acquisition of
abstract conceptions and concepts connected to the field of social value by
models during previous stages of training on multiple commonsense data [1].
This can avoid the phenomena of overfitting the training data distribution and
develop versatile and robust methods applicable in real-world scenarios for
reliable moral value detection. Additionally, we conduct an in-depth
investigation of the performance of state-of-the-art unsupervised models by
comparing them to supervised systems in a cross-domain framework. This
evaluation allows us to assess the robustness and versatility of systems in
effectively detecting moral values in text.111All materials and code are
accessible at https://github.com/LuanaBulla/Detection-of-Morality-in-
Text/tree/main

Our novel approach introduces the use of the GPT-3-based Davinci model [3] as
a zero-shot ready-made unsupervised multi-label classifier for moral values
detection. This allows us to detect moral values without the need for explicit
training on labeled data. To evaluate the performance and versatility of our
approach, we compare it with a smaller, more flexible zero-shot method based
on Natural Language Inference (NLI). This comparative analysis demonstrates
that the NLI approach achieves competitive results compared to the Davinci
model. Furthermore, we investigate the benefits and limitations of supervised
systems in cross-domain value frameworks. By training supervised models based
on RoBERTa architecture [18] on different domains, we examine the results
achieved in a cross-domain setting and compare their performance with the
unsupervised systems. Through a detailed analysis of the results, we shed
light on the strengths and weaknesses of both supervised and unsupervised
methodologies, offering valuable tools for researchers and practitioners. We
consider for our study the Moral Foundation Reddit Corpus (MFRC) [23], which
contains $16,123$ Reddit comments split into three different sub-corpora
belonging to different domains and tagged with Graham and Haidt’s Moral
Foundation Theory (MFT) [7]. Based on the work of Trager et al. [23], we
calculate the agreement between annotators to estimate the moral values
associated with each comment in the sub-corpora.

The main contribution of this paper can be summarized as follows:

  * •

We provide a comprehensive analysis of supervised (cross-domain) and
unsupervised methodologies for moral value detection.

  * •

We present a novel method that leverages the GPT-3-based Davinci model as a
zero-shot ready-made unsupervised moral values classifier.

  * •

We compare the approach based on GPT-3 with a smaller and more versatile NLI
zero-shot-based method, that achieves competitive results compared with the
Davinci model.

  * •

We train different supervised RoBERTa models, one for each sub-corpus of the
MFRC. Our findings demonstrate the remarkable effectiveness of these models in
a cross-domain context, establishing a new state-of-the-art for MFRC.

The paper is organized as follows. Section 2 provides a summary of the current
state-of-the-art results in this field. In Section 3, we briefly describe the
theoretical grounding of MFT. Section 4 focuses on the unsupervised and
supervised methodologies we employed. In Section 5, we present an overview of
our experimental settings (Sect. 5.1) and results (Sect. 5.2). Section 6
discusses the aforementioned results, comparing the unsupervised methods both
internally and against the supervised systems in cross-domain settings.
Finally, Section 7 concludes the paper and discusses potential future
developments of our approach.

##  2 Related Works

In the field of identifying moral values within text, previous research has
predominantly focused on two main approaches: word-count-based methods [5] and
feature-based methods utilizing word embeddings and sequences [6, 13]. These
methods can be broadly categorized as supervised and unsupervised approaches.
While the former relies on systems supported by external framing annotations,
the latter does not require any specific external framing annotations.
Unsupervised methods have explored architectures like the Frame Axis technique
[15], demonstrated in works by Mokhberian et al. [21] and Priniski et al.
[22]. These approaches project words onto micro-frame dimensions,
characterized by two opposing sets of words. Additionally, some unsupervised
approaches leverage the extended version of the Moral Foundation Dictionary
(MFD) [10], which comprises words related to the virtues, vices, and general
morality aspects of the five dyads of MFT. Kobbe et al. [14] contribute by
linking MFD entries to WordNet, thus extending and disambiguating the lexicon
within a dictionary-based framework. Hulpus et al. [12] propose an
unsupervised approach exploring the capture of moral values through Knowledge
Graphs (KG), which integrate data using a graph-structured data model. Their
study evaluates the relevance of entities within WordNet 3.1, ConceptNet, and
DBpedia in relation to MFT. In recent advancements, Asprino et al. [1]
introduce two distinct unsupervised approaches for detecting moral content in
natural language. The first method uses KG to identify moral values within the
text. By employing this approach, researchers aim to capture explicit
references to moral concepts and values in the content. The second approach
adopts a zero-shot machine learning model based on an NLI system as a zero-
shot classifier. Authors leverage the commonsense knowledge of the NLI system,
fine-tuned on a commonsense dataset for language inference (MNLI) [24]. The
indirect acquisition of commonsense knowledge by the NLI model enables the
detection of main values in the input text solely based on the semantic
interpretation conveyed from the MFT label taxonomy. Additionally, the authors
demonstrate that incorporating the emotional tone expressed in the input
sentence can significantly improve the classifier’s performance. Both methods
are evaluated on the Moral Foundation Twitter Corpus (MFTC) [9], a dataset
comprising $35$k items divided into seven domains spanning various socio-
political areas, annotated with the value taxonomy of the MFT. While these
unsupervised methods offer valuable insights into the detection of moral
values based on the MFT taxonomy, they do not provide a comprehensive cross-
domain analysis. Furthermore, they lack a comparison with current state-of-
the-art unsupervised methods represented by GPT-3.

From the cross-domain perspective, an interesting work is the Tomea system
[16], which is an explainable method for comparing a supervised text
classifier’s representation of moral rhetoric across different domains. This
approach aims to understand whether text classifiers learn domain-specific
expressions of moral language, shedding light on differences and similarities
in moral concepts across social domains. Tomea utilizes the SHapley Additive
exPlanations (SHAP) method [20], which uses the Shapley values to quantify the
extent to which an input component (a word) contributes toward predicting a
label (a moral element), to compile domain-specific moral lexicons,
facilitating direct comparisons of linguistic cues for predicting morality
across diverse domains. Additionally, other studies focus on supervised
methods for cross-domain analysis, such as [17], [19], and [11]. The first
evaluates the performance of supervised systems for cross-domain value
detection using the MFTC. The second focuses on the performance of a
supervised model trained on non-extremist Twitter data and tested on data from
extremist forums, representing a specific case of supervised cross-domain
moral classification. The results demonstrate that cross-domain classification
is feasible, albeit with some reduction in performance. The study compares the
efficacy of Word2Vec and BERT embeddings, with BERT exhibiting slightly better
generalization capabilities. The third describes Learning to Adapt Framework
(L2AF), a framework for addressing domain variations in the morality
classification task when the training and testing data come from different
domains. L2AF consists of four main modules: neural feature extractor
(utilizing models like RNN and BERT), prediction network (predicting moral
values), weighting network (adapting to domain shifts), and joint
optimization. The neural feature extractor encodes input documents into
feature representations. The prediction network predicts moral values using a
fully connected network and softmax function. The weighting network adapts to
domain shifts by assigning higher weights to out-domain instances with similar
language usage to in-domain data. Joint optimization involves two tasks: moral
value and domain predictions, with separate optimizers for the prediction
network and weighting network. The framework dynamically adjusts and balances
multi-domain data, ensuring effective adaptation to domain variations in
morality classification. The authors test their framework on the MFTC,
demonstrating superior performance compared to the state of the art in a
cross-domain context on the same dataset. While the method shares similarities
with our task and displays a high level of innovation, it primarily approaches
the task from a supervised perspective and requires a substantial amount of
training data, making it unfeasible when such data is not available. In this
context, the work of Guo et al. [8] addresses the recognition of moral
sentiment in textual content, enabling researchers to gain insights into the
role of morality in human life. Various ground truth datasets annotated with
moral values have been released, each differing in data collection methods,
domains, topics, and annotation instructions. Merging these heterogeneous
datasets during training can result in models that struggle to generalize
effectively. To overcome this limitation, the paper introduces a data fusion
framework that employs adversarial domain training to align datasets into a
shared feature space, thereby enhancing model generalizability. Additionally,
the proposed approach uses a weighted loss function to account for differences
in label distribution across datasets. As a result, the study achieves
cutting-edge performance in morality inference across different datasets
compared to previous methods. While these works provide valuable insights, our
focus remains on exploring effective unsupervised methods that are naturally
portable across domains, unaffected by specific training data that might bias
their performance. Furthermore, our study offers a comprehensive comparison of
current unsupervised methods, including GPT-3 and NLI, with the performance of
existing supervised approaches across domains, to verify their adaptability
and robustness.

##  3 Theoretical Grounding

Our work focuses on Haidt’s MFT [7], which represents the theoretical
grounding for our research. The MFT is grounded on the idea that, while
morality may vary across different geographical, temporal, and cultural
contexts, there are recurring patterns in its core principles, forming a
psychological system of “intuitive ethics” [7]. MFT adopts a nativist,
cultural-developmental, intuitionist, and pluralist approach to studying
morality. It acknowledges neurophysiological bases for moral responses
(nativist), considers environmental influences on moral beliefs (cultural-
developmental), suggests that moral judgments result from various patterns
(intuitionist), and allows for multiple narratives to explain moral reasoning
(pluralist). Central to MFT are six dyads of values and violations
representing distinct moral dimensions. At the core of MFT lie six dyads of
values and violations, each representing distinct MFT dimensions:

  * •

Care/Harm: This dyad is concerned with caring and harming behaviors,
encompassing virtues such as gentleness, kindness, and nurturance.

  * •

Fairness/Cheating: Grounded in social cooperation and reciprocal altruism,
this foundation underlies ideas of justice, rights, and autonomy.

  * •

Loyalty/Betrayal: Based on the benefits of cohesive coalitions and the
rejection of traitors, this dyad emphasizes loyalty and betrayal in social
interactions.

  * •

Authority/Subversion: Focusing on societal hierarchies, this foundation
relates to concepts of leadership, deference to authority, and respect for
tradition.

  * •

Purity/Degradation: Derived from the psychology of disgust, this dyad is
associated with notions of elevated spiritual life, often expressed through
metaphors like “the body as a temple” and incorporating spiritual aspects of
religious beliefs.

These five MFT dimensions form the basis of MFT, providing a comprehensive
framework for understanding the diverse aspects of human morality. In a
subsequent version, the liberty/oppression dyad was introduced. This dyad
represents the desire for freedom and the feeling of oppression when freedom
is denied. However, we do not use it, as it is not present in the MFRC
dataset.

##  4 Methodology

Our study encompasses the implementation of both unsupervised and supervised
methodologies. In the unsupervised approach, we delve into the exploration of
the LLMs GPT-3 generative model, with a specific focus on the Davinci model
[3]. This system plays a central role in our research as a zero-shot ready-
made unsupervised multi-label moral values classifier, enabling us to detect
moral values without the need for explicit training on labeled data. Built
upon the GPT-3 architecture, the Davinci model represents a cutting-edge
language model renowned for its exceptional performance across a wide range of
NLP tasks. Through extensive training on a vast corpus of text data, this
model has acquired the ability to generate coherent and contextually relevant
responses. In our methodology, we leverage the Davinci model as a zero-shot
ready-made unsupervised classifier for moral value detection. By utilizing the
model’s intrinsic knowledge and contextual understanding, we prompt it to
predict the moral values conveyed by an input text. Specifically, we employ
the prompt: “Does the sentence [Input text] convey a moral content or not?
(answer with one word: moral or not moral). If yes, based on the Moral
Foundation Theory, what moral values does the text reflect? (categorize text
with Care/Harm, Fairness/Cheating, Loyalty/Betrayal, Authority/Subversion,
Purity/Degradation).” This prompts the model to first assess whether an item
conveys moral content and, if affirmative, to label it with one or more MFT
dimensions.

As part of our exploration into unsupervised approaches for moral values
detection, we delve into the capabilities of NLI systems as zero-shot
classifiers. NLI models are fine-tuned to classify the relationship between
two given input text: a premise and a hypothesis. Specifically, the model can
accurately assess the degree of entailment, neutrality, and contradiction of
the hypothesis concerning the premise. This capability allows NLI models to
effectively understand the contextual relationship between textual pairs and
make informed judgments about their logical connections. Starting from the
methodology proposed in a previous work [1], we construct a hypothesis for
each potential label, utilizing the input text as the NLI premise. To form
these hypotheses, we employ the prompt “This text conveys the moral values of
<label>.” Our approach considers all ten moral values and violations expressed
in the MFT as possible labels. To identify the prevalent values in the input
text from an NLI perspective, we perform a multi-label classification task.
Specifically, we select values with a scoring entailment output of $0.50$ or
higher, indicating a strong association. Additionally, we account for
neutrality, representing the items that do not convey any moral content in the
dataset, by applying a $0.50$ cut-off on the normalized entailment score (for
a visual representation of the methodology see Figure 1).

Figure 1: Overview of the NLI-based approach: The NLI system processes the
input sentence as a premise and evaluates its relationship with all the labels
in the MFT taxonomies, considering them as hypotheses. We select the labels
with an entailment score equal to or greater than 0.50 as the classification
results. If no label meets this threshold, the sentence is categorized as non-
moral.

In contrast to the approach defined in Asprino et al. [1], where the
entailment score for each moral value is obtained by normalizing the model’s
entailment and contradiction outcomes related to it, we shift our focus to the
entailment and neutrality scores. We make this choice considering that the
contradiction of each positive or negative label is implicitly encompassed
within the range of labels we assign, taking into account the polarity of each
moral foundation separately. As a result, normalizing the entailment outcome
with the degree of neutrality within the same context becomes more appropriate
and relevant for our analysis. By adopting this method, we can effectively
identify non-moral content through exclusion. If the degree of neutrality in a
sentence exceeds the entailment for all values associated with the same item,
then that item is considered to lack moral content. Consequently, the highest
moral values derived from the normalization stage will fall below the 0.50
threshold. Through this approach, we enhance the precision and relevance of
our moral values detection, enabling us to efficiently distinguish between
moral and non-moral elements in the input text. For our experiments, we employ
the checkpoints of MNLI-RoBERTa-large222https://huggingface.co/roberta-large-
mnli, which were trained on the MNLI dataset [24].

Finally, we aim to explore and compare the performance of unsupervised methods
based on GPT-3 and NLI with the supervised models’ outcomes in a cross-domain
setting. To achieve this, we implement a moral value classifier based on the
RoBERTa-large model. This classifier utilize the pre-trained RoBERTa-large
checkpoint333https://huggingface.co/roberta-large and incorporate a dropout
layer, a linear layer, and a softmax function applied to the pooled output
embedding of the CLS token. During training, we set the learning rate to
$1e-5$, employ a batch size of $64$, and use a dropout rate of $0.1$, with
AdamW as the optimizer. To evaluate the proximity of the model’s predictions
to the actual values, we employ the MultiLabel Soft Margin
Loss444https://pytorch.org/docs/stable/generated/torch.nn.MultiLabelSoftMargin
Loss.html, a commonly used loss function for multi-label classification tasks.
This evaluation allows us to assess the performance of the supervised models
when faced with data from different domains, providing valuable insights into
their robustness and versatility.

##  5 Results and Evaluation

We conducted an extensive experimental analysis to assess the effectiveness of
our approaches in classifying moral values, achieving a new state-of-the-art
for both supervised and unsupervised methods in a cross-domain setting. In
Sect. 5.1 we detail the description of the MFRC while in Sect. 5.2 we present
the performance achieved by the unsupervised and supervised methods in the
multi-label classification task for moral values.

###  5.1 The Moral Foundation Reddit Corpus

For our experiments, we use the MFRC dataset [23], which enables the
examination of moral language dynamics and its relationship to online and
offline behaviors. Recognizing that different online platforms have distinct
linguistic and social environments, the MFRC aims to address these variations
by providing a diverse dataset sourced from Reddit. The corpus comprises
$16,123$ Reddit comments from $12$ subreddits, which have been meticulously
annotated for $8$ categories of MFT dimensions based on the updated MFT [2].
In our study, we employ an earlier version of the Moral Foundation taxonomy,
in which we merge the labels “Equality” and “Proportionality” into the moral
category of “Fairness”. While the Atari et al. [2] version of the MFT
separates the Fairness dimension to address distinct moral concerns related to
procedural fairness and equality of outcomes, previous evidence suggests that
the concept of “Fairness” effectively encompasses both notions. Moreover,
unsupervised models demonstrate a high level of understanding and successful
detection of this unified “Fairness” category [1]. Furthermore, we decided not
to include the label “Thin Morality” introduced by Trager et al., as it
represents a moral judgment or concern that does not clearly align with any of
the five moral dimensions, resulting in ambiguity. Finally, we identified
items that were consistently labeled as having no moral value by the majority
of annotators. Consequently, we associated these items exclusively with the
“non-moral” label. The corpus comprises comments from subreddits associated
with US Politics, French Politics, and Everyday Moral Life. The Everyday Moral
Life sub-corpus (i.e. Corpus A) encompasses topics related to various aspects
of daily life, collected for their non-political moral judgment and expression
of moral emotions, and includes comments from four subreddits. The US Politics
sub-corpus (i.e. Corpus B) consists of comments from three subreddits that
capture political moral language in general, encompassing both the moral
rhetoric of the right and the left. The French Politics sub-corpus (i.e.
Corpus C) contains comments from subreddits related to relevant keywords
associated with the presidential race (e.g., “Macron,” “Le Pen,” “France,”
“French,” and “Hollande’). The MFRC provides information relating to the
degree of confidence of the annotator for each individual item. In our study,
we exclude items labeled by annotators with uncertain confidence levels and
items labeled by a single annotator due to their higher subjectivity and
potential noise. Consequently, the examined sub-corpora comprises a total of
$3,472$ items labeled by two to six different annotators for the Everyday
Moral Life sub-corpus; $3,949$ items labeled by two to five different
annotators for the US Politics sub-corpus, and $5,443$ items labeled by two to
five different annotators for the French Politics sub-corpus. Moreover, to
establish performance baselines, Reddit annotations were processed by
calculating the majority vote for each moral value, considering the majority
to be 50%.

###  5.2 Classification of the MFT Dimensions

We assess the effectiveness of both GPT-3-based and NLI-based unsupervised
systems on sub-corpora A, B, and C of the MFRC dataset. The Davinci
GPT-3-based model provides a classification output that includes five distinct
MFT dimensions, alongside the identification of non-moral content. Conversely,
for the NLI-RoBERTa system, we present the results by converting the
predictions of individual moral values or violations into their respective MFT
dimensions. To validate the performance of the supervised RoBERTa-base models
in a cross-domain setting, we train them on one sub-corpus and evaluate them
on the remaining sub-corpora. For this purpose, we develop three distinct
supervised classifiers based on the RoBERTa-large architecture, named
A-RoBERTa, B-RoBERTa, and C-RoBERTa, each trained on its respective sub-
corpus: Everyday Moral Life (A), US Politics (B), and French Politics (C).
Both the unsupervised and supervised systems are tested on the complete data
available in each sub-corpus. For an overview of the label distribution in
each sub-corpora, please refer to Table 1.

Table 1: Label distribution in each sub-corpora of the MFRC. Corpus A refers to the Everyday Moral Life sub-corpus, Corpus B indicated the US Politics sub-corpus and Corpus C represents the French sub-corpus Moral Dimension | Corpus A | Corpus B | Corpus C
---|---|---|---
Non-moral | 2278 | 2684 | 4330
Fairness | 510 | 731 | 638
Care | 708 | 473 | 424
Purity | 102 | 90 | 75
Loyalty | 105 | 122 | 167
Authority | 74 | 211 | 350
All | 3777 | 4311 | 5802

Table 2 presents overall results for all the methods, reporting weighted
precision, recall, and F1-score for sub-corpus A, B, and C of MFRC. Among
unsupervised models, the Davinci GPT-3 system achieves the highest F1 scores
of 0.66, 0.59, and 0.69 for Corpus A, B, and C, respectively, while the NLI
system exhibits comparable performance with F1 scores of 0.62, 0.58, and 0.69
for the corresponding sub-corpora. Considering the supervised models,
A-RoBERTa performs relatively poorer in cross-domain scenarios compared to
models trained on sub-corpora B and C, with a difference of approximately ten
percentage points. Notably, the supervised model trained on the USA Politics
sub-corpus (B-RoBERTa) performs well in a cross-domain context, achieving F1
scores of 0.71 and 0.75 on the Everyday Life Morality (A) and French Politics
(C) sub-corpora, respectively.

Tables 3 to 7 report results for all MFT dimensions in terms of weighted
F1-score. Davinci GPT-3 model slightly outperforms NLI system on four out of
five moral dimensions, with notable differences in performance. GPT-3 exhibits
a significantly better trend in the “Fairness” dimensions and, to some extent,
in the “Authority” dimension, while NLI excels in “Purity”, “Care”, and
“Loyalty” dimensions, consistently exceeding GPT-3 by a small margin.
Supervised models consistently outperform unsupervised models across all the
MFT dimensions, aligning with overall trends observed in Table 2.
Particularly, the A-RoBERTa model demonstrates better results in capturing
nuances of moral values.

The behavior of supervised models is most accurately reflected in the
prediction of non-moral content (Table 8), which significantly influences the
overall weighted F1 score reported in Table 2, where the high support of non-
moral elements within the MFRC dataset plays a crucial role. In predicting
elements that do not convey any moral content, the NLI system outperforms the
GPT-3-based model by 2, 6, and 5 percentage points. Supervised models
demonstrate their effectiveness in classifying nonmoral elements,
outperforming unsupervised models by a narrow margin of 6, 2, and 3 percentage
points in corpus B, C, and A, respectively.

Table 2: Unsupervised (i.e. GPT-3 and NLI-RoBERTa) and supervised model (i.e. A-RoBERTa, B-RoBERTa, C-RoBERTa) overall performance for the task of multi-label moral values detection in terms of precision, recall and weighted F1 score. Hyphens indicate when the evaluation sub-corpus is used in training. Underlined values show the best performance by supervised models, and bold values highlight the best among unsupervised results. Models | Metrics | Corpus A | Corpus B | Corpus C
---|---|---|---|---
GPT-3 |  | Precision
---
Recall
F1
| 0.64
---
0.72
0.66
| 0.58
---
0.64
0.59
| 0.69
---
0.71
0.69
NLI-RoBERTa |  | Precision
---
Recall
F1
| 0.60
---
0.68
0.62
| 0.58
---
0.63
0.58
| 0.68
---
0.74
0.69
A-RoBERTa |  | Precision
---
Recall
F1
- |  | 0.70
---
0.63
0.65
| 0.78
---
0.68
0.65
B-RoBERTa |  | Precision
---
Recall
F1
| 0.76
---
0.73
0.71
- |  | 0.77
---
0.78
0.75
C-RoBERTa |  | Precision
---
Recall
F1
| 0.77
---
0.74
0.75
| 0.70
---
0.67
0.68
-
Table 3: Unsupervised (i.e. GPT-3 and NLI-RoBERTa) and supervised model (i.e. A-RoBERTa, B-RoBERTa, C-RoBERTa) performance for predicting the MFT “Care” dimension in terms of F1 score. Hyphens indicate when the evaluation sub-corpus is used in training. Underlined values show the best performance by supervised models, and bold values highlight the best among unsupervised results. Models | Corpus A | Corpus B | Corpus C
---|---|---|---
GPT-3 | 0.54 | 0.32 | 0.20
NLI-RoBERTa | 0.47 | 0.32 | 0.20
A-RoBERTa | - | 0.48 | 0.27
B-RoBERTa | 0.53 | - | 0.23
C-RoBERTa | 0.67 | 0.46 | -
Table 4: Unsupervised (i.e. GPT-3 and NLI-RoBERTa) and supervised model (i.e. A-RoBERTa, B-RoBERTa, C-RoBERTa) performance for predicting the MFT “Fairness” dimension in terms of F1 score. Hyphens indicate when the evaluation sub-corpus is used in training. Underlined values show the best performance by supervised models, and bold values highlight the best among unsupervised results. Models | Corpus A | Corpus B | Corpus C
---|---|---|---
GPT-3 | 0.42 | 0.41 | 0.33
NLI-RoBERTa | 0.18 | 0.17 | 0.05
A-RoBERTa | - | 0.54 | 0.50
B-RoBERTa | 0.52 | - | 0.45
C-RoBERTa | 0.50 | 0.52 | -
Table 5: Unsupervised (i.e. GPT-3 and NLI-RoBERTa) and supervised model (i.e. A-RoBERTa, B-RoBERTa, C-RoBERTa) performance for predicting the MFT “Purity” dimension in terms of F1 score. Hyphens indicate when the evaluation sub-corpus is used in training. Underlined values show the best performance by supervised models, and bold values highlight the best among unsupervised results. Models | Corpus A | Corpus B | Corpus C
---|---|---|---
GPT-3 | 0.10 | 0.12 | 0.07
NLI-RoBERTa | 0.14 | 0.14 | 0.14
A-RoBERTa | - | 0.20 | 0.20
B-RoBERTa | 0.26 | - | 0.05
C-RoBERTa | 0.29 | 0.17 | -
Table 6: Unsupervised (i.e. GPT-3 and NLI-RoBERTa) and supervised model (i.e. A-RoBERTa, B-RoBERTa, C-RoBERTa) performance for predicting the MFT “Loyalty” dimension in terms of F1 score. Hyphens indicate when the evaluation sub-corpus is used in training. Underlined values show the best performance by supervised models, and bold values highlight the best among unsupervised results. Models | Corpus A | Corpus B | Corpus C
---|---|---|---
GPT-3 | 0.18 | 0.21 | 0.18
NLI-RoBERTa | 0.17 | 0.20 | 0.20
A-RoBERTa | - | 0.19 | 0.24
B-RoBERTa | 0.22 | - | 0.28
C-RoBERTa | 0.15 | 0.33 | -
Table 7: Unsupervised (i.e. GPT-3 and NLI-RoBERTa) and supervised model (i.e. A-RoBERTa, B-RoBERTa, C-RoBERTa) performance for predicting the MFT “Authority” dimension in terms of F1 score. Hyphens indicate when the evaluation sub-corpus is used in training. Underlined values show the best performance by supervised models, and bold values highlight the best among unsupervised results. Models | Corpus A | Corpus B | Corpus C
---|---|---|---
GPT-3 | 0.16 | 0.18 | 0.21
NLI-RoBERTa | 0.09 | 0.15 | 0.13
A-RoBERTa | - | 0.26 | 0.22
B-RoBERTa | 0.16 | - | 0.06
C-RoBERTa | 0.20 | 0.29 | -
Table 8: Unsupervised (i.e. GPT-3 and NLI-RoBERTa) and supervised model (i.e. A-RoBERTa, B-RoBERTa, C-RoBERTa) performance for predicting the Moral Sentiment in terms of F1 score. Hyphens indicate when the evaluation sub-corpus is used in training. Underlined values show the best performance by supervised models, and bold values highlight the best among unsupervised results. Models | Corpus A | Corpus B | Corpus C
---|---|---|---
GPT-3 | 0.81 | 0.75 | 0.83
NLI-RoBERTa | 0.83 | 0.81 | 0.88
A-RoBERTa | - | 0.78 | 0.84
B-RoBERTa | 0.87 | - | 0.91
C-RoBERTa | 0.89 | 0.83 | -

##  6 Discussion

The findings of our study present valuable insights into the performance of
unsupervised and supervised models for classifying moral values in a cross-
domain framework. Davinci GPT-3 model exhibits a slight advantage over the NLI
model in the unsupervised setting, showcasing its competitiveness. This
suggests that smaller and more versatile models, such as NLI, can also be
effective in this domain, achieving comparable performance to LLMs.
Furthermore, GPT-3 proves to be more proficient in detecting semantic moral
dimensions, particularly in opaque concepts like “Fairness” and “Authority”.
This advantage is attributed to GPT-3’s extensive commonsense knowledge,
enabling better comprehension and association with these complex moral
concepts. However, certain MFT dimensions, such as “Authority”, “Purity”, and
“Loyalty”, exhibit poor performance in terms of F1 score across all sub-
corpora. These findings indicate the difficulty in predicting these labels,
possibly due to their opaque and ambiguous meanings within the dataset’s
context. On the other hand, concepts like “Care” and “Fairness” are more
transparent and easily identifiable by GPT-3, and to some extent by NLI.

The training process facilitates value learning by models in a cross-domain
context as models learn to discriminate moral dimensions based on the data
they see during the fine-tuning phase. Nonetheless, the improvement in
performance between supervised and unsupervised models is not substantial,
especially for difficult-to-interpret labels. The impact of the dataset’s
imbalanced nature is evident, as the varying number of items assigned to each
moral dimension affects the supervised models’ ability to predict specific
moral values, such as the moral dimensions of “Purity”, “Authority”, and
“Loyalty”. Supervised models exhibit a slight advantage over unsupervised
models in discriminating elements that do not convey any moral content.
Indeed, while supervised models can leverage the data distribution in the
training set, which includes a greater number of items labeled as non-moral,
their performance does not significantly surpass that of unsupervised models
in this context. This indicates that unsupervised models are more proficient
at recognizing non-moral content.

The indirect prior knowledge acquired by unsupervised models proves useful for
moral detection. Particularly in predicting moral dimensions that are easier
to associate at a semantic level and in discriminating items containing moral
content, unsupervised models perform comparably to supervised models. This
highlights the benefit of unsupervised models’ general understanding of moral
concepts in diverse contexts. However, more challenging-to-interpret dyads
remain an open research challenge for both supervised and unsupervised models.
These ambiguous moral values present complexities in their discrimination
within textual content, leading to lower accuracy in predicting them.

In conclusion, both supervised and unsupervised models exhibit strengths and
limitations in classifying moral values in a cross-domain context. While GPT-3
showcases competitiveness and efficiency in detecting certain moral
dimensions, challenges persist in accurately predicting difficult-to-interpret
MFT dimensions. Training improves value learning, but supervised model
enhancements are not consistently significant. Unsupervised models benefit
from indirect prior knowledge, particularly in predicting easily traceable
moral content. However, accurately discriminating opaque moral values remains
a research challenge.

##  7 Conclusion

In this paper, we addressed the challenge of detecting moral values in text
using both supervised and unsupervised methodologies, focusing on cross-domain
scenarios. It introduces a novel approach utilizing the GPT-3-based Davinci
model as an unsupervised multi-label classifier for moral value detection and
compares it with a smaller, NLI-based zero-shot method. The study also
investigates the performance of three supervised RoBERTa models trained on
different sub-corpora of the MFRC dataset, establishing their effectiveness in
a cross-domain context. Through a comprehensive analysis, we compared the
performance of unsupervised and supervised methodologies, shedding light on
their strengths and weaknesses in cross-domain scenarios. The GPT-3-based
Davinci model exhibited advantages in detecting semantic moral dimensions,
especially in opaque concepts like “Fairness” and “Care”. However, some moral
dimensions, such as “Authority”, “Purity”, and “Loyalty”, proved difficult to
predict for both unsupervised and supervised models. The dataset’s imbalanced
nature influenced the models’ ability to predict specific moral values,
particularly the challenging ones. While supervised models showed a slight
advantage in discriminating non-moral elements, unsupervised models
demonstrated proficiency in recognizing non-moral content. Overall, both
supervised and unsupervised models presented strengths and limitations in
cross-domain moral value detection. Unsupervised models benefitted from
indirect prior knowledge, while supervised models excelled in certain moral
dimensions in the cross-domain setting. The challenges of accurately
predicting difficult-to-interpret moral values persist for both methodologies,
urging further research in this area. As part of our future work, we intend to
investigate the potential use of LLMs for the generation of additional data.
This approach aims to enrich the input data, to effectively address and
mitigate both the labeling imbalance and subjectivity present in the dataset.

In conclusion, our study offers significant insights into the effectiveness of
unsupervised and supervised models in detecting moral values across different
domains. By exploring the strengths and weaknesses of both approaches, we
provide researchers and practitioners with valuable tools and knowledge to
enhance the development of reliable moral value detection methods. As a
result, our research lays the foundation for future advancements in
understanding moral content in diverse contexts, facilitating ethical
decision-making, and driving progress in the fields of NLP and social
sciences.

####  7.0.1 Acknowledgements

We acknowledge financial support from the H2020 projects TAILOR: Foundations
of Trustworthy AI – Integrating Reasoning, Learning and Optimization – EC
Grant Agreement number 952215 – and the Italian PNRR MUR project
PE0000013–FAIR: Future Artificial Intelligence Research.

## References

  * [1] Asprino, L., Bulla, L., De Giorgis, S., Gangemi, A., Marinucci, L., Mongiovì, M.: Uncovering values: detecting latent moral content from natural language with explainable and non-trained methods. In: Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures. pp. 33–41 (2022)
  * [2] Atari, M., Haidt, J., Graham, J., Koleva, S., Stevens, S.T., Dehghani, M.: Morality beyond the weird: How the nomological network of morality varies across cultures. Journal of Personality and Social Psychology (2022)
  * [3] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020)
  * [4] Bulla, L., Gangemi, A., et al.: Towards distribution-shift robust text classification of emotional content. In: Findings of the Association for Computational Linguistics: ACL 2023. pp. 8256–8268 (2023)
  * [5] Fulgoni, D., Carpenter, J., Ungar, L., Preoţiuc-Pietro, D.: An empirical exploration of moral foundations theory in partisan news sources. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16). pp. 3730–3736 (2016)
  * [6] Garten, J., Boghrati, R., Hoover, J., Johnson, K.M., Dehghani, M.: Morality between the lines: Detecting moral sentiment in text. In: Proceedings of IJCAI 2016 workshop on Computational Modeling of Attitudes (2016)
  * [7] Graham, J., Haidt, J., Koleva, S., Motyl, M., Iyer, R., Wojcik, S.P., Ditto, P.H.: Moral foundations theory: The pragmatic validity of moral pluralism. In: Advances in experimental social psychology, vol. 47, pp. 55–130. Elsevier (2013)
  * [8] Guo, S., Mokhberian, N., Lerman, K.: A data fusion framework for multi-domain morality learning. In: Proceedings of the International AAAI Conference on Web and Social Media. vol. 17, pp. 281–291 (2023)
  * [9] Hoover, J., Portillo-Wightman, G., Yeh, L., Havaldar, S., Davani, A.M., Lin, Y., Kennedy, B., Atari, M., Kamel, Z., Mendlen, M., et al.: Moral foundations twitter corpus: A collection of 35k tweets annotated for moral sentiment. Social Psychological and Personality Science 11(8), 1057–1071 (2020)
  * [10] Hopp, F.R., Fisher, J.T., Cornell, D., Huskey, R., Weber, R.: The extended moral foundations dictionary (emfd): Development and applications of a crowd-sourced approach to extracting moral intuitions from text. Behavior research methods 53(1), 232–246 (2021)
  * [11] Huang, X., Wormley, A., Cohen, A.: Learning to adapt domain shifts of moral values via instance weighting. In: Proceedings of the 33rd ACM Conference on Hypertext and Social Media. pp. 121–131 (2022)
  * [12] Hulpus, I., Kobbe, J., Stuckenschmidt, H., Hirst, G.: Knowledge graphs meet moral values. In: Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics. pp. 71–80 (2020)
  * [13] Kennedy, B., Atari, M., Davani, A.M., Hoover, J., Omrani, A., Graham, J., Dehghani, M.: Moral concerns are differentially observable in language. Cognition 212, 104696 (2021)
  * [14] Kobbe, J., Rehbein, I., Hulpus, I., Stuckenschmidt, H.: Exploring morality in argumentation. In: Proceedings of the 7th Workshop on Argument Mining. Association for Computational Linguistics, ACL (2020)
  * [15] Kwak, H., An, J., Jing, E., Ahn, Y.Y.: Frameaxis: characterizing microframe bias and intensity with word embedding. PeerJ Computer Science 7, e644 (2021)
  * [16] Liscio, E., Araque, O., Gatti, L., Constantinescu, I., Jonker, C., Kalimeri, K., Murukannaiah, P.K.: What does a text classifier learn about morality? an explainable method for cross-domain comparison of moral rhetoric. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 14113–14132 (2023)
  * [17] Liscio, E., Dondera, A., Geadau, A., Jonker, C., Murukannaiah, P.: Cross-domain classification of moral values. In: Findings of the Association for Computational Linguistics: NAACL 2022. pp. 2727–2745 (2022)
  * [18] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
  * [19] van Luenen, A.F.: Recognising moral foundations in online extremist discourse: A cross-domain classification study (2020)
  * [20] Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. Advances in neural information processing systems 30 (2017)
  * [21] Mokhberian, N., Abeliuk, A., Cummings, P., Lerman, K.: Moral framing and ideological bias of news. In: International Conference on Social Informatics. pp. 206–219. Springer (2020)
  * [22] Priniski, J.H., Mokhberian, N., Harandizadeh, B., Morstatter, F., Lerman, K., Lu, H., Brantingham, P.J.: Mapping moral valence of tweets following the killing of george floyd. arXiv preprint arXiv:2104.09578 (2021)
  * [23] Trager, J., Ziabari, A.S., Davani, A.M., Golazazian, P., Karimi-Malekabadi, F., Omrani, A., Li, Z., Kennedy, B., Reimer, N.K., Reyes, M., et al.: The moral foundations reddit corpus. arXiv preprint arXiv:2208.05545 (2022)
  * [24] Williams, A., Nangia, N., Bowman, S.R.: A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426 (2017)


# Applications of the Theory of Aggregated Markov Processes in Stochastic
Learning Theory

Fangyuan Lin
University of California, Berkeley

(Year: 2023)

###### Abstract

A stochastic process that arises by composing a function with a Markov process
is called an aggregated Markov process (AMP). The purpose of composing a
Markov process with a function can be a reduction of dimensions, e.g., a
projection onto certain coordinates. The theory around AMP has been
extensively studied e.g. by Dynkin, Cameron, Rogers and Pitman, and Kelly, all
of whom provided sufficient conditions for an AMP to remain Markov. In another
direction, Larget provided a canonical representation for AMP, which can be
used to verify the equivalence of two AMPs.

The purpose of this paper is to describe how the theory of AMP can be applied
to stochastic learning theory as they learn a particular task.

##  1 Learning Theory Background

Stochastic learning theory serves to provide a stochastic model of the process
of behavior modification in animals or people. Learning has a stochastic
nature (at least it appears to), e.g., a person asked to memorize a list of
words sometimes recall more words in an early trial than in a later trial.

> Even in simple repetitive experiments, the sequences of choices made by the
> subject are typically erratic, suggesting a probabilistic nature of
> learning. (Sternberg 1963)

The models of stochastic learning theory serve to describe the trial-to-trial
changes in probabilities to make certain responses and are usually either
Markov or a function of a Markov process.

  * •

AMPs are generally not Markov, so the aggregated model does not necessarily
enjoy Markovanity.

  * •

The aggregating functions are assumed to be onto. Also, we are only interested
in many-to-one aggregating functions since a process derived by a one-to-one
aggregating function always contains the same information as the underlying
process that is relevant to the next state of the derived process.

###  1.1 Stochastic Learning Theory Framework

The following is a brief description of a framework (Norman 1968) for the
construction of a learning model.

  * •

Consider the process $X_{n}$ with state space $(X,\mathscr{B})$. Here $X_{n}$
represents the subject’s response probability on trial $n$.

  * •

Consider also the event space $(E,\mathscr{G})$, which represents the set of
possible outcomes of a trial that can happen to the subject.

  * •

The process $E_{n}$ represents the outcome on trial $n$ and usually contains
which response the subject makes as one of its coordinates.

  * •

Let $\mu:X\times E\to X$ be a measurable function with respect to
$\mathscr{B}\times\mathscr{G}$ and $\mathscr{B}$. $\mu$ has the role of
transforming $X_{n}$ based on $E_{n}$, i.e., $X_{n+1}=\mu(X_{n},E_{n})$. In
other words, the trial-to-trial change of response-probability is governed by
the deterministic operator $\mu$, which transforms the response-probability
depending on the event that occurs in the previous trial. (Most models are
assumed to be time-homogeneous, meaning $\mu$ does not depend on the trial
$n$.)

  * •

Let $P$ be a stochastic kernel on $X\times\mathscr{G}$ and $P$ has the role
that $P(X_{n},A)=P(E_{n}\in A|X_{n},E_{n-1,...}$ for $A\in\mathscr{G}$.

The system $((X,\mathscr{B}),(E,\mathscr{G}),P,\mu)$ is called a random system
with complete connection, and it is a formalization of learning model.

##  2 Applications of the Theory of Aggregated Markov Processes

######  Setup.

Let $Z_{n}$ be a Markov chain on the countable state space $S_{Z}$ with
transition probability function $P$. Let $Y_{n}=f(Z_{n})$, for $f:S_{Z}\to
S_{Y}$ be the aggregated process.

######  Theorem 2.1 (Burke and Rosenblatt (1958) Simplest condition to
preserve Markovianity).

Suppose that

$\displaystyle
P\\{Y_{n+1}|Z_{n}=z_{n}\\}=P\\{Y_{n+1}|Y_{n}=f(z_{n})\\}=P\\{Y_{n+1}|Z_{n}\in
f^{-1}(f(z_{n}))\\},$ (1)

then $Y$ is Markov.

######  Remark.

The condition of the theorem says that $Y_{n}$ contains the same information
as $Z_{n}$ does for predicting $Y_{n+1}$.

######  Remark.

The Markov process $Y$ is not necessarily homogeneous.

Here is a straight-forward application of the above theorem.

######  Application (Theorem 2.1).

A model based on the continuous pattern model of Suppes (1959). A subject is
asked to predict the location on the rim of a disk that will be illuminated.
Let $Y_{n}$ be the subject’s prediction and let $Z_{n}$ be the actual spot of
light that will be revealed after the subject makes their choice. Assume the
set of locations $Y$ is discrete so that the theorem applies (we can always
consider a higher granularity for better accuracy).

Let $S$ be a set of $N$ stimulus patterns. The state of learning $x:S\to Y$ is
a function mapping a stimulus pattern to a point on the rim of the disk, and
$Y_{n}$ has distribution $\lambda_{s}(x_{s},dy)$.

On each trial, the subject samples a stimulus pattern $s$ uniformly randomly
from $S$ (i.e. with probability $\frac{1}{N}$) and makes a prediction
according to $\lambda_{s}(x_{s},dy)$.

Given a prediction $y$, the outcome $z$ has distribution $\pi(y,dz)$. In this
model, the event process is $(S_{n},Y_{n},Z_{n}).$

After observing the spot illuminated $z$, $x_{s}$ is updated to $z$. The
transformation $u$ plays the role of updating $x_{s}$:

$\displaystyle u(x_{s},e)$ $\displaystyle=z$ $\displaystyle u(x_{t},e)$
$\displaystyle=x_{t}$ for $t\neq s$

Note that the original model also considers the probability of successful
conditioning and I neglect this aspect for simplicity.

One observation is that the transformation of $x$, the state of learning, does
not depend on the prediction $y$ that the subject just made, so we have the
motivation to reduce the model from $(x,s,y,z)$ to $(x,s,z)$, i.e. we do not
keep track of the subject’s response.

To see the reduced model still has the Markov property, we can apply Theorem
2.1. The distribution of $S_{n}$ is always the uniform distribution on $S$.
The next state of learning $X_{n}$ depends only on $S_{n}$ and $Z_{n}$ and is
independent of $Y_{n}$, as noted previously. The distribution of $Z_{n+1}$
does depend on $Y_{n+1}$; however, the distribution of $Y_{n+1}$ follows
$\lambda(X_{n+1},dy)$ and $X_{n+1}$ does not depend on $Y_{n}$.

Therefore, the reduced model is Markov by Theorem 2.1.

Next, we introduce a theorem applicable to process with uncountable state
space.

###  2.1 Theorem 2.2

######  Theorem 2.2 (Cameron (1973)).

If for $z,z^{\prime}\in S_{Z}$, if $f(z)=f(z^{\prime})$, then for any $n$ and
event $B_{n}\subseteq S_{Y}$, the $n$-step transition probability
$P_{n}(z,f^{-1}(B_{n}))=P_{n}(z^{\prime},f^{-1}(B_{n}))$,
then $Y_{n}$ is Markov.

######  Remark.

The condition says that what matters is where the starting point is mapped to,
not the starting point itself.

######  Application (Theorem 2.2).

Reduction of dimensionality in the Zeaman-House-Lovejoy (ZHL) Model. Suppose
we have a T-maze where on each trial, one of the left and right arms is chosen
to be bright (white) uniformly at random.

A rat then chooses which arm to go to. On each trial, we assume that the rat
can only pay attention to either brightness or the position of the arm. In
this model, the food (reward) is always provided on the black arm. Therefore,
the position of the arms is actually completely irrelevant, and the rat needs
to learn to pay attention to brightness instead of the position of the arms.

We can formulate the stochastic process as follows:

Let $\mathrm{br}$ be the event that the rat pays attention to brightness, and
$\mathrm{pos}$ be the event that the rat pays attention to the position of the
arm. Let $W$ be the event that the rat chooses the white (bright) arm, and let
$B$ be the event that the rat chooses the black (dark) arm.
We are interested in:

$\displaystyle V_{n}=Pr(\mathrm{br})$ on trial $n$ $\displaystyle
Y_{n}=Pr(B|\mathrm{br})$ on trial $n$ $\displaystyle Z_{n}=Pr(L|\mathrm{pos})$
on trial $n$

We now describe the trial-to-trial change of probability: The probability $V$
of attending to brightness is assumed to increase linearly with some parameter
if the rat attends to brightness and is fed or does not attend to brightness
and is not fed. Otherwise, it decreases. For example,
$V_{n+1}=u(V_{n},e)=\begin{cases}V_{n}+\phi_{1}(1-V_{n})&e=(B,W)\mathrm{br}B\\\
V_{n}-\phi_{2}V_{n}&e=(B,W)\mathrm{br}W\\\ \text{and so on}\end{cases}$
where $\phi_{1},\phi_{2}\in(0,1)$ are parameters representing how fast the rat
learns from the presence of reward.

The conditional response probabilities $Y$ and $Z$ are also changed according
to the rat’s attention: For example, if the rat pays attention to brightness,
the $Y$ should always be increased (linearly in this model) since we assume
that the rat learns from the presence or lack of reward! In this case, $Z_{n}$
does not change at all, which is intuitive. If the rat pays attention to
position, $Z_{n}$ is increased (linearly in this model) only if the left arm
is, luckily, the black arm. In this case, $Y_{n}$ does not change at all,
which is intuitive.

The stochastic process that we have an interest in is
$X_{n}=(V_{n},Y_{n},Z_{n})$, which represents the response probabilities and
it has a continuous state space, so Theorem 2.1 does not apply. The event
process is $E_{n}=(S_{n},A_{n},R_{n})$ where $S_{n}$ represents the location
of the black arm, $A_{n}$ is either $\mathrm{br}$ or $\mathrm{pos}$,
representing the rat’s attention, and $R_{n}$ is $B$ or $W$, representing
which arm the rat chooses on trial $n$.

######  Claim.

As the assumption of the model says, the location of the arms is irrelevant.
We can ask whether we can reduce the dimensionality of the model to reduce its
complexity while preserving the Markovianity. The answer is yes. Theorem 2.2
applies in this example.

###### Proof.

Consider the coordinate projection $\Phi(v,y,z)=(v,y)$. We apply $\Phi$ to
$X_{n}$ so that from now on, we stop caring about the probability of the rat
choosing the left arm when it pays attention to the position of the arms,
motivated by the fact we just want to know how well the rat learns to pay
attention to brightness and go to the black arm.

Note that we can in fact also apply the projection $\Psi(s,a,r)=(a,r)$ to
$E_{n}$ so that we stop keeping track of where the black arm is. After all, it
is the brightness, not the location, that matters.

To apply Theorem 2.2, we need to show that for any $n$ and
$A_{n}\subset[0,1]\times[0,1]$ measurable,
$P_{n}((v,y,z),f^{-1}(A_{n}))=P_{n}((v,y,z^{\prime}),f^{-1}(A_{n}))$, for any
$z$ and $z^{\prime}$, since $\Phi(v,y,z)=\Phi(v,y,z^{\prime})=(v,y)$ for any
$z$ and $z^{\prime}$.

Recall that $X_{n+1}$ is determined by a transformation that takes $E_{n}$ and
$X_{n}$ (i.e., $X_{n+1}=u(X_{n},E_{n})$).

  * •

If $\mathrm{b}$, $V_{n}$ is changed according to the occurrence of $B$: it
increases linearly if the rat receives the reward by choosing to go to the
black arm (which happens with probability $Y_{n}$) and decreases linearly with
probability $1-Y_{n}$.

  * •

If $\mathrm{pos}$, $V_{n}$ is increased linearly if the rat does not receive
the reward, which happens with probability:

$\displaystyle Pr(W_{n}|X_{n},pos)$
$\displaystyle=Pr(L|X_{n},\mathrm{pos},(W,B))Pr((W,B))+Pr(R|X_{n},\mathrm{pos},(B,W))Pr((B,W))$
by conditioning on the position of the white and black arms (where $(W,B)$
stands for the case where the white arm is on the left)
$\displaystyle=Z_{n}\cdot\frac{1}{2}+(1-Z_{n})\cdot\frac{1}{2}$
$\displaystyle=1/2$

Similarly, $V_{n}$ is decreased linearly if the rat does receive the reward
while paying attention to the position of the arms, which happens with
probability $Pr(B_{n}|X_{n},\mathrm{pos})=\frac{1}{2}$.

In both cases, $\mathrm{br}$ and $\mathrm{pos}$, the transformation of $V_{n}$
does not depend on $Z_{n}$. Also, the occurrence of $\mathrm{br}$ and
$\mathrm{pos}$ is dictated by $V_{n}$ only.

  * •

If $\mathrm{br}$, the conditional probability $Y_{n}$ is always increased
linearly.

  * •

If $\mathrm{pos},$ $Y_{n}$ remains the same.

The transformation of $Y_{n}$ depends solely on the occurrence of
$\mathrm{br}$, which, again, does not depend on $Z_{n}$.
Therefore,
$P_{n}((v,y,z),f^{-1}(A_{n}))=P_{n}((v,y,z^{\prime}),f^{-1}(A_{n}))$ since as
long as we start with the same $v,y$, the distribution of the future of
$V_{n},Y_{n}$ is independent of $z,$. We do not need to consider the
distribution of the future of $Z_{n}$, since $f^{-1}(A_{n})=A_{n}\times[0,1]$,
i.e., there is no restriction on what value the future of $Z_{n}$ takes. The
condition of Theorem 2.2 is thus met, and the proof is complete. ∎

Acknowledgment: The author thanks Steven N. Evans for helpful discussions
during the preparation of this paper.

## Reference

  1. 1.

Bush, R. R., & Mosteller, F. (1955). Stochastic models for learning. John
Wiley & Sons Inc. https://doi.org/10.1037/14496-000

  2. 2.

Cameron, M. A. (1973). A Note on Functions of Markov Processes with an
Application to a Sequence of $\chi^{2}$ Statistics. Journal of Applied
Probability, 10(4), 895–900. https://doi.org/10.2307/3212394

  3. 3.

Kelly, F. P. (1982). Markovian Functions of a Markov Chain. Sankhyā: The
Indian Journal of Statistics, Series A (1961-2002), 44(3), 372–379.
http://www.jstor.org/stable/25050326

  4. 4.

Norman, M. F. (1970). [Review of Random Processes and Learning., by M.
Iosifescu & R. Theodorescu]. The Annals of Mathematical Statistics, 41(4),
1381–1383. http://www.jstor.org/stable/2240182

  5. 5.

Rogers, L. C. G., & Pitman, J. W. (1981). Markov Functions. The Annals of
Probability, 9(4), 573–582. http://www.jstor.org/stable/2243410

  6. 6.

Rosenblatt, M. (1971). Markov processes structure and asymptotic behavior.
Springer.

  7. 7.

Sternberg, S. (1963). Stochastic learning theory. John Wiley & Sons Inc.

  8. 8.

Suppes, P. (1959). A linear model for a continuum of responses. Stanford
University Press.