# Einstein Completeness as Categoricity

Iulian D. Toader

(Institute Vienna Circle, University of Vienna
<EMAIL_ADDRESS>
###### Abstract

This paper provides an algebraic reconstruction of Einstein’s own argument for
the incompleteness of quantum mechanics – the one that he thought did not make
it into the EPR paper – in order to clarify the assumptions that underlie an
understanding of Einstein completeness as categoricity, the sense in which it
is a type of descriptive completeness, and some of the various ways in which
it has been more often misconstrued.

##  1 Introduction

Einstein’s argument for the incompleteness of quantum mechanics, which did not
make it into the EPR paper (Einstein, Podolski, and Rosen 1935) in the way
Einstein thought it would, was clearly formulated in letters to Schrödinger
and Popper, as well as in several publications (e.g. Einstein 1936). After
Arthur Fine brought it to philosophical attention (Fine 1981), Don Howard
suggested that the argument might be understood as deploying a notion of
completeness known as categoricity, i.e., model uniqueness up to isomorphism
(Howard 1990). This suggestion was motivated by Einstein’s claim that quantum
mechanics fails to assign a unique wavefunction to the real state of one
subsystem of an EPR system, since the assignment depends on the measurement
that could be performed on the other subsystem. If multiple wavefunctions can
be assigned to the same subsystem, and if one is justified in considering them
as non-isomorphic models, then this would be enough to show quantum mechanics
non-categorical. If Howard’s suggestion is taken seriously, then Einstein
completeness turns out to be a rather different type of completeness than the
one articulated in the EPR paper.

In the present paper, I provide an algebraic reconstruction of Einstein’s
argument, which I think can clarify the assumptions underlying an
understanding of Einstein completeness as categoricity. The key idea to be
rigorously articulated is the following: “if one understands a theoretical
state as, in effect, a model for a set of equations plus boundary conditions
…, then Einstein’s conception of a completeness requirement should really be
understood as a categoricity requirement.” (Howard 1992, 208) To stay as close
as possible to Einstein’s own argument, I focus on the original EPR state,
with observables having a continuous spectrum, suitably defined within an
algebraic framework (Arens and Varadarajan 2000, Werner 1999), and I explain
under what conditions one would be justified to read Einstein completeness as
categoricity. On my reconstruction, the argument assumes that representations
on (tensor products of) Hilbert spaces are the models of quantum mechanics of
(composite) systems. It assumes as well that the unitary equivalence of such
representations is a necessary (though not a sufficient) condition for
categoricity. The argument then points out that there are representations of
the (tensor product of) algebras describing a subsystem of an EPR system that
are not unitarily equivalent. As I reconstruct it, following Howard’s
suggestion, Einstein’s argument concludes that categoricity is logically
inconsistent with separability and locality.

It is, of course, difficult to say that the reconstruction I propose is
actually entirely faithful to Einstein’s own thought. One worry one might
raise is that while my reconstruction works for the original EPR state, it
does not work for entangled spins, since on finite-dimensional Hilbert spaces
there are no unitarily inequivalent representations. But Einstein, as is well
known, never cared much about Bohm’s version of the EPR argument.111The only
place where Einstein formulated a spin version of his argument appears to be
in a late manuscript from around 1955: for discussion, see Sauer 2007.
Focusing on the infinite-dimensional case is thus historically reasonable.
Moreover, for Einstein’s objection to stand, it is of course sufficient that
his argument goes through in one case; it is not required that it should do so
in all cases. In any event, I will argue that my reconstruction is preferable
to others, according to which Einstein’s argument should be taken to establish
“overcompleteness” (Lehner 2014) or unsoundness (Gömöri and Hofer-Szabó 2021),
rather than non-categoricity. Furthermore, I suggest that my reconstruction
sheds some new light on the Bohr-Einstein controversy. Bohr’s doctrine of
complementarity has been interpreted in terms of the unitary inequivalence of
non-regular Hilbert space representations (Halvorson 2004), an interpretation
that vindicates a common view that Bohr’s notion of completeness was
significantly distinct from the descriptive completeness articulated in the
EPR paper.222For an expression of this view, see, e.g., Norsen 2017, 148. But
on my reconstruction, Einstein completeness fails precisely due to this
unitary inequivalence. Thus, from an algebraic point of view, it appears that
the sense in which Bohr thought quantum mechanics was complete is exactly the
sense in which Einstein argued it wasn’t. Considered from a model-theoretical
perspective, their views would turn out to be precisely antithetical. Of
course, in order to properly justify this claim, a more formal undertaking
would be needed than what this paper can offer. In particular, one would need
a fully articulated formal semantics of standard quantum mechanics that
deploys a suitable logic for a formalization of the algebraic structures
described below. Deferring this undertaking to further work seems, however, in
order for now.

##  2 Einstein completeness

Recall the completeness condition that the EPR paper purported to argue it is
not satisfied by quantum mechanics: “Whatever the meaning assigned to the term
complete, the following requirement for a complete theory seems to be a
necessary one: every element of the physical reality must have a counterpart
in the physical theory.” (1935, 777) This condition has typically been
understood to formulate a type of descriptive completeness, since it applies
to a physical theory just in case that theory is able to describe all of the
physical reality that it aims to describe. Related ideals of completeness were
endorsed by other foundational thinkers, before and after the 1930s, within
and without quantum physics, and particularly with respect to systems of
axioms. For instance, Russell and Whitehead stated that a system of axioms for
pure mathematics (like that presented in their Principia Mathematica) is
complete just in case it is able to capture “as much as may seem necessary” of
the domain that it aims to describe, i.e., the entire class of theorems of
ordinary mathematics. This was a notion of completeness that Gödel took to
prove that, on some reasonable assumptions, it cannot be attributed to the
system in the Principia.333See (Detlefsen 2014) for a discussion of
completeness as understood by Russell and Whitehead, and an argument that
Gödel’s own understanding of completeness, as negation-completeness, is
relevantly different than the descriptive variety. Similarly, von Neumann
considered a system of axioms for quantum mechanics (like that presented in
his Mathematical Foundations of Quantum Mechanics), to be complete just in
case it is able to derive all statistical formulas of quantum mechanics, and
rejected the view that their statistical nature was due to “an ambiguity
(i.e., incompleteness) in our description of nature”.444See (Acuña 2021) for a
recent re-evaluation of (the debate on) von Neumann’s proof of completeness.

The EPR paper argued that quantum mechanics does not satisfy the completeness
condition because in the case of a system in an EPR state there are elements
of physical reality, i.e., properties of a subsystem of that system, that the
theory aims to describe, but fails to do so. As is well known, crucial to the
EPR argument is the following criterion of reality: “If, without in any way
disturbing a system, we can predict with certainty (i.e., with probability
equal to unity) the value of a physical quantity, then there exists an element
of physical reality corresponding to this physical quantity.” (1935, 777;
italics removed) The role and character of this criterion, as well as the
formal structure of the argument, have been often addressed.555Perhaps the
most detailed formal reconstruction of the EPR argument has been given in
(McGrath 1978) where the following is also noted: “Regrettably EPR equate two
notions of completeness: ‘complete representation by a wave function’ and
‘complete theory’ are used interchangeably.” (560) See also (Gömöri and Hofer-
Szabó 2021) for a nice discussion of the EPR criterion of reality, its
indispensable role within the EPR argument for incompleteness, and its absence
from Einstein’s own argument. What has been often ignored is the fact that EPR
completeness and Einstein completeness are not the same type of descriptive
completeness.666The fact that EPR completeness and Einstein completeness are
not the same type of completeness has been, to my knowledge, first explicitly
noted by Arthur Fine: “Einstein does not give [the latter] a catchy name, but
… [we can] call this more technical conception bijective completeness.” (Fine
1981, 72) In one widely quoted passage from his letter to Schrödinger,
Einstein writes: “In the quantum theory, one describes a real state of a
system through a normalized function, $\psi$, of the coordinates (of the
configuration-space). … Now one would like to say the following: $\psi$ is
correlated one-to-one with the real state of the real system. … If this works,
then I speak of a complete description of reality by the theory. But if such
an interpretation is not feasible, I call the theoretical description
‘incomplete’.” (Letter to Schrödinger, 19 June 1935; translated in Howard
1985, 179) Einstein completeness can thus be attributed to quantum mechanics
if and only if there exists a one-to-one correlation between a $\psi$-function
and the real state of a system the theory aims to describe. Einstein’s point
that a one-to-one correlation does not exist in the case of an EPR system is
meant to be supported by his separability and locality assumptions, so his
argument can be formulated in the following way:

1\. Spacelike separated physical systems have real states, which cannot
causally influence one another.

2\. Consider a system with two subsystems, A and B, in an EPR state.

3\. Thus, each subsystem has a real state, no matter what measurements can be
carried out on its other subsystem.

4\. Quantum mechanics assigns different $\psi$-functions to A, depending on
which observable one can choose to measure on B.

5\. But if quantum mechanics is complete, these $\psi$-functions should be
identical.

6\. Thus, quantum mechanics is incomplete.

Most of my discussion in this paper will be focused on premises 4 and 5. It
should be clear that the notion of completeness in premise 5 is directly
implied by that defined in the letter to Schrödinger. In a letter to Pauli,
Schrödinger put it in the following metaphorical terms: “He [i.e., Einstein]
has a model of that which is real consisting of a map with little flags. To
every real thing there must correspond on the map a little flag, and vice
versa.” (Schrödinger to Pauli, July 1935, translated in Howard 1990, 106) It
immediately follows that a map with little flags is an Einstein complete
description of physical reality only if the map is essentially unique, in the
sense that there is a bijection between any two maps describing that reality.
In the case of quantum mechanics, by analogy, Einstein completeness requires
that the $\psi$-function representing the real state of a system be
essentially unique.

Howard’s suggestion as to how to properly understand Einstein completeness,
first offered in a footnote to his presentation of Einstein’s argument, is
this: “A complete theory assigns one and only one theoretical state to each
real state of a physical system. [Footnote: This is a curious conception of
completeness, more akin to what is called in formal semantics “categoricity.”]
But in EPR-type experiments involving spatio-temporally separated, but
previously interacting systems, A and B, quantum mechanics assigns different
theoretical states, different ‘psi-functions,’ to one and the same real state
of A, say, depending upon the kind of measurement we choose to carry out on B.
Hence quantum mechanics is incomplete.” (Howard 1990, 64) The same suggestion
is then uplifted from footnote to the main text of a later paper: “Let me
conclude with one really outrageous suggestion. … the operative criterion of
completeness in Einstein’s thinking was this: a theory is complete if and only
if it assigns a unique theoretical state, such as a psi-function, to every
unique real physical state. But if one understands a theoretical state as, in
effect, a model for a set of equations plus boundary conditions …, then
Einstein’s conception of a completeness requirement should really be
understood as a categoricity requirement. In other words, Einstein is saying
that a ‘complete’ (read ‘categorical’) theory is one that determines a unique
(eindeutige) model for the reality it aims to represent.” (Howard 1992, 208;
my emphasis) The conception of completeness that Howard thought should be
credited to Einstein is that according to which a theory is Einstein complete
if and only if all its models belong to one and the same isomorphism class.

In the background to Howard’s suggestion lies, of course, a complex model-
theoretical machinery. Models are defined as set-theoretical structures of
elements in a domain that can be assigned to a formal version of the language
of quantum mechanics (including symbols for variables, constants, functions,
and relations) by an interpretation map. But in order to reach a good
understanding of Howard’s suggestion, starting from my reconstruction of
Einstein’s argument, we will not actually need this formal machinery. It will
be enough to motivate informally an algebraic requirement for categoricity,
and then argue that Einstein can be understood to have doubted that quantum
mechanics satisfies that requirement. It will then follow that what Einstein
argued for is that quantum mechanics is descriptively incomplete in the sense
that, roughly, even though it may be able to describe all of the physical
reality that it aims to describe, it ends up describing a lot more besides
that as well.

The reconstruction of Einstein’s argument, in an algebraic framework, will be
given further below. That will require a preliminary consideration of premise
2, in particular, a properly algebraic definition of the original EPR state,
for continuous observables. It will also require a discussion of premise 4,
that is an algebraic account of the difference between the $\psi$-functions
assigned to susbsystem A. The outcome of all that will, I hope, be three-fold:

(i) a good understanding of the assumptions behind Howard’s suggestion to read
Einstein completeness as categoricity,

(ii) an explanation of what makes Einstein completeness different from EPR
completeness as a type of descriptive completeness, and finally,

(iii) an unequivocal sense of where some of the misreadings of Einstein
completeness have gone wrong.

Although my real emphasis will be on (i), let me briefly address (iii) right
away, in order to candidly raise the reader’s interest in what is to come
later, in section 4, which takes care of (i). Then (ii) will be presented in
section 5, where I also briefly indicate how (i) bears on the Bohr-Einstein
debate.

##  3 Misconstruals

Einstein’s argument for incompleteness is sometimes considered too confused to
take seriously. Klaas Landsman, for example, maintained that it is a “muddled”
argument (Landsman 2006, 234) for the following reason: “Unfortunately,
Einstein (and EPR) insisted on a further elaboration of this disjunction
[i.e., completeness or separability], namely the idea that there exists some
version of quantum mechanics that is separable … at the cost of assigning more
than one state to a system (two in the simplest case). It is this unholy
version of quantum mechanics that Einstein (and EPR) called ‘incomplete’. Now,
within the formalism of quantum mechanics such a multiple assignment of states
(except in the trivial sense of wave functions differing by a phase factor)
makes no sense at all, for the entanglement property lying at the root of the
non-separability of quantum mechanics is so deeply entrenched in its formalism
that it simply cannot be separated from it.” (op. cit., 227) However, if it
implies that Einstein’s argument requires that entanglement be relinquished,
this is simply missing the point. One would have thought it obvious that,
quite the contrary, the argument is essentially based on entanglement.
Clearly, Landsman’s criticism fails insofar as it neglects the fact that
Einstein separability (expressed in premise 1 above) and entanglement
(implicit in premise 2) are compatible.777See (Murgueitio Ramírez 2020) for
the point that Einstein’s argument would be trivially unsound, if separability
and entanglement were not compatible. Landsman argues that, as a consequence
of Raggio’s theorem, Einstein separability is mathematically equivalent to
Bohr’s doctrine of the necessity of classical concepts. But his argument takes
Einstein separability as state decomposability. The alleged reason why
multiple assignment of states is nonsensical doesn’t stand: there is no
“unholy version” of quantum mechanics, i.e., a version of the theory without
entanglement, and Einstein did not think otherwise. Furthermore, and this is a
more significant problem, Landsman seems to misconstrue Einstein’s view on the
difference between the $\psi$-functions assigned to subsystem A as a trivial
phase difference. As we will see further below, the multiple assignment of
$\psi$-functions that Einstein actually considered can be aptly interpreted
algebraically in terms of the unitary inequivalence of non-regular Hilbert
space representations. Were this nonsensical, Bohr’s complementarity doctrine
(at least as interpreted in Halvorson 2004, see below section 5) would appear
to be nonsensical as well, but this is a doctrine that Landsman professes to
defend.

In The Cambridge Companion to Einstein, Christoph Lehner characterized
Einstein completeness, and Einstein’s argument that this cannot be attributed
to quantum mechanics, in these terms: “Einstein … concludes that the quantum
mechanical description is not biunique because it is incomplete. … This
conclusion is not warranted logically.” (Lehner 2014, 334) A description that
is not biunique is one that assigns to subsystem A a $\psi$-function which is
not correlated one-to-one with its real state. The bottom half of Einstein’s
argument would, on this reconstruction, be as follows:

4L. Quantum mechanics assigns different $\psi$-functions to A, depending on
which observable one can choose to measure on B.

5L. Thus, quantum mechanics is incomplete.

6L. Thus, quantum mechanics is not biunique.

Lehner maintained that the inference from 5L to 6L is not justified. But this
clearly puts Einstein’s cart before his horse: the one-to-one correlation does
not fail because of incompleteness; rather, its failure is the actual reason
for incompleteness, just as the reason why the one-to-one correlation fails is
provided by the possible assignment of different $\psi$-functions to A
(expressed by 4L). Moreover, since a $\psi$-function is just a theoretical
description and such “a description that is not invariant is not necessarily
incomplete … it is ‘overcomplete’ or nonabsolute” (loc. cit.), i.e., it
assigns to A a theoretical state that is correlated many-to-one with its real
state, what Einstein should have allegedly argued is the following:

4L. Quantum mechanics assigns different theoretical states to A, depending on
which observable one can choose to measure on B.

5L. Thus, quantum mechanics is overcomplete.

6L. Thus, quantum mechanics is not biunique.

The inference from 5L to 6L is immediate, but trivial. However, Lehner appears
to reduce Einstein’s incompleteness to a quite uninteresting case of empirical
underdetermination. Jos Uffink has recently pointed out that reading Einstein
incompleteness as overcompleteness is “surprising”, since “cases of
overcompleteness are ubiquitous in physics… Indeed one might wonder whether
overcompleteness is a worrisome issue at all in theoretical physics.” (Uffink
2020, 557) In quantum mechanics, as Uffink emphasizes, such cases of
overcompleteness as illustrated by phase differences between $\psi$-functions
are not worrisome at all. But, of course, Einstein did not think otherwise. As
already noted, we will see below that the essential differences between
$\psi$-functions that Einstein did think worrisome can be aptly interpreted
algebraically in terms of the unitary inequivalence of non-regular Hilbert
space representations. If this is enough to conclude that quantum mechanics is
not categorical, then Lehner’s reconstruction of Einstein’s argument turns out
to conflate (the problem of) categoricity and (the problem of) empirical
underdetermination.

Along different lines, Márton Gömöri and Gábor Hofer-Szabó argue for a similar
conclusion, that Einstein’s argument should not really be taken to establish
incompleteness: “According to Einstein’s later [than the EPR] argument, the
Copenhagen interpretation is committed to the existence of elements of reality
that cannot be out there in the world –- under the assumptions of locality and
no-conspiracy. Hence, given these assumptions, the Copenhagen interpretation
is unsound –- as opposed to being incomplete.” (Gömöri and Hofer-Szabó 2021,
13453) Supposing it right that Einstein intended his argument against the
Copenhagen interpretation of quantum mechanics, the bottom half of Einstein’s
argument would change as follows:

4G. Quantum mechanics assigns different $\psi$-functions to A, depending on
which observable one can choose to measure on B.

5G. Thus, at least some $\psi$-functions represent states of A that cannot
exist.

6G. Thus, quantum mechanics is unsound.

Gömöri and Hofer-Szabó readily acknowledge that their reconstruction of
Einstein’s argument is in conflict with his own understanding of completeness.
Pursuing “logical reconstruction” in spite of what “Einstein actually thought
to argue”, they find his own notion “not quite apt”, in part because it is not
identical to the EPR notion of completeness (loc. cit.). But one would have
thought that the difference between EPR completeness and Einstein completeness
cannot be sufficient to deny the latter as an apt notion, since there is
nothing to indicate that EPR completeness is the only possible type of
descriptive completeness attributable to physical theories. Still, their main
reason for having Einstein’s argument conclude that quantum mechanics is
unsound, rather than incomplete, is the incompatibility of the different
$\psi$-functions assigned to subsystem A of an EPR system. This is correct as
an understanding of the essential difference between $\psi$-functions,
expressed by premise 4G. In the next section, this incompatibility will be
precisely interpreted algebraically as the unitary inequivalence of non-
regular Hilbert space representations. If my interpretation is adequate, I do
not see why that incompatibility would make quantum mechanics unsound, rather
than sound but non-categorical. To argue that my interpretation is adequate, I
turn now to reconstructing Einstein’s argument in the framework of
C∗-algebras.

##  4 An algebraic reconstruction of Einstein’s argument

Taking Howard’s suggestion seriously requires, as I pointed out in the
introduction, that one consider the theoretical states assigned to a system by
quantum mechanics as models of the physical theory. But it’s not immediately
clear how a $\psi$-function, and in particular one that represents the EPR
state of a system, can be considered in this way. For one thing, such a
$\psi$-function cannot be construed as a unit vector in a standard infinite-
dimensional Hilbert space $\mathcal{H}$. This is for multiple related reasons,
such as that the $\psi$-function has infinite norm, and the observables (like
position and momentum) of the subsystems of an EPR system have continuous
spectra, so their probability distribution will have a probability density
that cannot be concentrated on a single point in $\mathcal{H}$. This is why
the definition of the EPR state for a system composed of subsystems A and B as
a unit vector in $\mathcal{H}_{A}\otimes\mathcal{H}_{B}$ for observables in
subalgebras $\textbf{B}(\mathcal{H}_{A})\otimes\mathcal{I}$ and
$\mathcal{I}\otimes\textbf{B}(\mathcal{H}_{B})$ is not entirely adequate
(Arens and Varadarajan 2000, 638).

But that definition has been generalized, as we will presently see, and the
EPR state has been identified with a normalized positive linear functional on
$\mathfrak{A}_{A}\otimes\mathfrak{A}_{B}$ for observables in the von Neumann
algebras $\pi(\mathfrak{A}_{A}\otimes\mathcal{I})^{\prime\prime}$ and
$\pi(\mathcal{I}\otimes\mathfrak{A}_{B})^{\prime\prime}$ (Werner 1999,
Halvorson 2000). The generalized definition will be used below to reformulate
premises 4 and 5 in Einstein’s argument. The essential difference between the
theoretical states assigned to subsystem A of the EPR system will be
interpreted in terms of the unitary inequivalence of non-regular
representations of $\mathfrak{A}_{A}\otimes\mathcal{I}$ on
$\mathcal{H}_{A}\otimes\mathcal{I}$. This can support a model-theoretical
approach to Einstein’s argument, suggested by Howard, provided one takes
representations as models of quantum mechanics. But this would not be at all
unusual. Consider, for example, R.I.G. Hughes’ view: “Quantum mechanics, we
may say, uses the models supplied by Hilbert spaces.” (Hughes 1989,
79)888Similarly, but more recently: “The models of NRQM are Hilbert spaces,
along with a suitable subalgebra of the bounded operators on that Hilbert
space.” (Weatherall 2019, 7) These models are assumed to be definable as
formal structures in a suitable metatheory of quantum mechanics, but as
already noted at the outset, for my purposes in this paper no such definitions
are needed.999This assumption would be problematic, if one dispensed with
Hilbert space representations in the formalism of quantum mechanics. For the
∗-automorphisms of the Weyl algebra are unique up to unitary equivalence, so
at the abstract algebraic level there are no non-regular states. But I think
this problem for my reconstruction can be ignored, since Einstein, interested
as he was in local wavefunctions, was surely not an Algebraic Imperialist of
the kind introduced and described in Arageorgis 1995 (see also Ruetsche 2011).

Let’s start by considering observables $O_{A}\otimes
I\in\textbf{B}(\mathcal{H}_{A})\otimes\mathcal{I}$ and $I\otimes
O_{B}\in\mathcal{I}\otimes\textbf{B}(\mathcal{H}_{B})$. As defined by Richard
Arens and Veeravalli S. Varadarajan, $\omega$ is an EPR state if and only if
the joint distribution of $O_{A}\otimes I$ and $I\otimes O_{B}$ is a measure
on $\mathbb{R}^{2}$ concentrated on the diagonal, $\mu_{\omega}^{O_{A}\otimes
I,I\otimes O_{B}}(\left\\{(x,x)|x\in\mathbb{R}\right\\})=1$. Such pairs of
observables are typically called EPR-doubles: the outcome of measuring one
predicts with certainty the outcome of measuring the other. These EPR-doubles
form type I factors (isomorphic to von Neumann algebras
$\textbf{B}(\mathcal{H}_{A})\otimes\mathcal{I}$ and
$\mathcal{I}\otimes\textbf{B}(\mathcal{H}_{B})$ respectively), so they have a
discrete spectrum (their distributions relative to $\omega$ take discrete
values only) (Arens and Varadarajan 2000, 647). Thus, this definition is not
enough to characterize the original EPR state – the state that Einstein was
concerned with. In order to overcome this limitation, Reinhard F. Werner
considered $\mathfrak{A}_{A}\otimes\mathcal{I}$ and
$\mathcal{I}\otimes\mathfrak{A}_{B}$ as the mutually commuting subalgebras of
$\mathfrak{A}_{A}\otimes\mathfrak{A}_{B}$, and took
$\pi(\mathfrak{A}_{A}\otimes\mathcal{I})^{\prime\prime}$ as a self-adjoint
unital subalgebra of $\textbf{B}(\mathcal{H}_{A})\otimes\mathcal{I}$, and
$\pi(\mathcal{I}\otimes\mathfrak{A}_{B})^{\prime\prime}$ as a self-adjoint
unital subalgebra of $\mathcal{I}\otimes\textbf{B}(\mathcal{H}_{B})$, both
closed in the weak operator topology. Then $\pi(O_{A}\otimes
I)\in\pi(\mathfrak{A}_{A}\otimes\mathcal{I})^{\prime\prime}$ and $\pi(I\otimes
O_{B})\in\pi(\mathcal{I}\otimes\mathfrak{A}_{B})^{\prime\prime}$ are EPR-
doubles and have continuous spectra, since
$\pi(\mathfrak{A}_{A}\otimes\mathcal{I})^{\prime\prime}$ and
$\pi(\mathcal{I}\otimes\mathfrak{A}_{B})^{\prime\prime}$ are type II1 factors
(Werner 1999).

Now, consider the representations $(\pi,\mathcal{H}_{A}\otimes I)$ and
$(\pi,I\otimes\mathcal{H}_{B})$ of $\mathfrak{A}_{A}\otimes\mathfrak{A}_{B}$.
If $\omega$ is an original EPR state and $\pi$ is faithful, then there is a
state $\tau$ in each of these representations such that for any element
$O_{A}\otimes O_{B}\in\mathfrak{A}_{A}\otimes\mathfrak{A}_{B}$, we will have
$\omega(O_{A}\otimes O_{B})=\tau(\pi(O_{A}\otimes O_{B}))$ (Halvorson 2000,
327). This entails, on the same conditions, that for any two representations
$(\pi_{1},\mathcal{H}_{A}\otimes I)$ and $(\pi_{2},\mathcal{H}_{A}\otimes I)$
of $\mathfrak{A}_{A}\otimes\mathcal{I}$, there are different states $\tau_{1}$
and $\tau_{2}$ in these representations, respectively, such that for two
different elements $O^{1}_{A}\otimes I,O^{2}_{A}\otimes
I\in\mathfrak{A}_{A}\otimes\mathcal{I}$, we have the corresponding
restrictions of $\omega$, that is $\omega(O^{1}_{A}\otimes
I)=\tau_{1}(\pi_{1}(O^{1}_{A}\otimes I))$ and $\omega(O^{2}_{A}\otimes
I)=\tau_{2}(\pi_{2}(O^{2}_{A}\otimes I))$. All this suggests the following
reformulation of the bottom half of Einstein’s argument:

4T. Quantum mechanics assigns different states, $\tau_{1}$ or $\tau_{2}$, to
subsystem A, depending on which EPR-double, $\pi_{1}(I\otimes O^{1}_{B})$ or
$\pi_{2}(I\otimes O^{2}_{B})$, one can choose to measure on B.

5T. But if quantum mechanics is complete, $\tau_{1}$ and $\tau_{2}$ should be
identical.

6T. Thus, quantum mechanics is incomplete.

How should we understand premise 4T? Within this framework, what makes
$\tau_{1}$ and $\tau_{2}$ different states? As we have seen above, some
commentators took Einstein’s argument to be confused on this very point, for
the reason that the only differences between such states allowed by standard
quantum mechanics are “trivial” phase differences. But this is not what
Einstein had in mind, as Howard already pointed out: “Might there not be
situations in which the differences between two $\psi$-functions (phase
differences, for example) are inessential from the point of view of the system
whose real state they aim to describe? Einstein’s completeness condition
would, indeed, be too strong if it required that literally every difference
between $\psi$-functions mirror a difference in the real state of the system
in question; but such was not Einstein’s intention.” (Howard 1985, 181) What
are then the non-trivial differences between theoretical states that Einstein
did have in mind? Here is Howard, again: “The kind of difference with which
Einstein was concerned is clear from his argument: [$\tau_{1}$] and
[$\tau_{2}$] differ in the predictions they yield for the results of certain
objective, local measurements on A. … (For example, if [$\tau_{1}$] attributed
a definite position to [A], but not a definite momentum, it would be
incomplete in its description of [A]’s momentum; but, of course, Einstein’s
argument does not require any such reference to specific parameters or
‘elements of reality’.)” (loc. cit., modified for uniform notation) The
essential differences between $\tau_{1}$ and $\tau_{2}$ concern their
predictions of the measurement outcomes for A’s observables. For instance,
$\tau_{1}$ and $\tau_{2}$ are essentially different if one of them “lives” in
$(\pi_{1},\mathcal{H}_{A}\otimes I)$, say a position representation, and the
other in $(\pi_{2},\mathcal{H}_{A}\otimes I)$, say a momentum representation.
More generally then, $\tau_{1}$ and $\tau_{2}$ are essentially different if
$(\pi_{1},\mathcal{H}_{A}\otimes I)$ and $(\pi_{2},\mathcal{H}_{A}\otimes I)$
are unitarily inequivalent representations of
$\mathfrak{A}_{A}\otimes\mathcal{I}$. Taking this into account, one obtains
the following reconstruction of the bottom half of Einstein’s argument:

4T. Quantum mechanics allows unitarily inequivalent representations of
$\mathfrak{A}_{A}\otimes\mathcal{I}$, depending on which EPR-double,
$\pi_{1}(I\otimes O^{1}_{B})$ or $\pi_{2}(I\otimes O^{2}_{B})$, one can choose
to measure on subsystem B.

5T. But if quantum mechanics is complete, the representations
$(\pi_{1},\mathcal{H}_{A}\otimes I)$ and $(\pi_{2},\mathcal{H}_{A}\otimes I)$
should be unitarily equivalent.

6T. Thus, quantum mechanics is incomplete.

I believe that there is a good chance that this captures exactly what Einstein
thought on the matter. But one immediate criticism against this reconstruction
concerns the apparent conflict between premise 4T and the Stone-von Neumann
theorem, conjectured by Marshall Stone (Stone 1930) and then proved by John
von Neumann (von Neumann 1931), of which Einstein was no doubt fully aware by
1935. The theorem states that any irreducible faithful Hilbert space
representation of the Weyl algebra generated by the canonical commutation
relations describing a quantum mechanical system (or any system with a finite
number of degrees of freedom) is uniquely determined up to a unitary
transformation, and in fact unitarily equivalent to the Schrödinger
representation.101010This entails that all bounded operators are intertwined
by an isometric isomorphism. For details, see Summers 2001, Rosenberg 2004,
Ruetsche 2011. The existence of this isomorphism has been considered
sufficient to justify an interpretation of the Stone-von Neumann theorem as a
categoricity result for quantum mechanics. For a critical discussion of such
an interpretation, see Toader 2021.

The Stone-von Neumann theorem applies, of course, to EPR systems. But the
conflict with my reconstruction is merely apparent, since the representations
mentioned in premise 4T may be taken as non-regular and, therefore, not in the
range of the Stone-von Neumann theorem.111111There are other ways in which the
conflict can be avoided, for example, by turning to phase spaces with
nontrivial topologies, on which representations can be regular but unitarily
inequivalent (Landsman 1990). This reply, however, just seems to have opened
the door to further criticism. For if Einstein’s argument required that
regularity be dropped, then one might seem justified to consider it muddled
after all, for it is not at all clear what physical meaning can be given to
non-regular representations. If, besides the Schrödinger representation, both
position and momentum representations are taken to be physically significant
despite their non-regularity, then unitary dynamics must be given up.121212For
details, see Feintzeig et al. 2019, 126sq. This observation is essentially
based on an unpublished result by David Malament, according to which if a free
dynamics is assumed in the position representation, then exact localizabilty
is violated. But I think that this captures precisely the sense of
incompatibility that Gömöri and Hofer-Szabó, as we have seen above, associate
with the multiple assignment of $\psi$-functions to subsystem A. The dynamical
incompatibility between (regular and) non-regular representations not only
does not muddle Einstein’s argument, but rather helps clarify his
justification for rejecting completeness.

Now, if my algebraic reconstruction is correct, a formalization in a suitable
metatheory would be required to fully support Howard’s suggestion.131313Given
the metric completeness of the mathematical structures of quantum mechanics,
this formalization would be impossible in classical first-order logic. For a
survey of recent work in the formal semantics of C∗-algebras, see Lupini 2019.
But an informal version of the reconstructed argument should be enough here:

4H. Quantum mechanics allows non-isomorphic models of its description of A,
depending on which EPR-double one can choose to measure on subsystem B.

5H. But if quantum mechanics is categorical, these models should be
isomorphic.

6H. Thus, quantum mechanics is not categorical.

As noted already, this version of the argument assumes that unitarily
inequivalent representations (in premise 4T) can be construed as non-
isomorphic models of the quantum mechanical description of subsystem A (in
premise 4H). In other words, it assumes that unitary equivalence is a
necessary, though not a sufficient, condition for categoricity. On this
assumption, which I consider unproblematic, if the validity of the argument is
to be preserved, premise 5T must be reformulated as premise 5H. Einstein
completeness should indeed be understood as categoricity. I think that this
clarifies and supports, at least in part, Howard’s “outrageous” suggestion.
Einstein’s no-go result is really that a local, separable, and categorical
quantum mechanics cannot exist.

##  5 Consequences

Reconstructing Einstein’s argument as above points to a series of general
consequences often discussed, in other theoretical contexts, throughout the
rich history of philosophical concerns with the categoricity of logic and
mathematics. For instance, one often takes it to be the case that “A non-
categorical set of sentences … does not give the impression of a closed and
organic unity and does not seem to determine precisely the meaning of the
concepts contained in it.” (Tarski 1934, 311) The models of a categorical
theory agree on all theorems or predictions that can be expressed in its
language. But only a categorical theory determines the meaning of its concepts
precisely. For instance, the meaning of arithmetical terms is precisely
determined only if Peano arithmetic is categorical with respect to the
isomorphism class of an omega sequence. The meaning of classical logical
connectives is precisely determined only if classical logic is categorical
with respect to the isomorphism class of a two-element Boolean algebra. This
suggests that, quite similarly, the meaning of concepts like quantum state,
observable, etc. is precisely determined only if quantum mechanics is
categorical with respect to the relevant isomorphism class of models. Thus,
Einstein’s argument, if reconstructed as I proposed above, attributes to
quantum mechanics a kind of semantic indeterminacy that may help explain the
difference between EPR completeness and Einstein completeness as types of
descriptive completeness. The question is what kind of descriptive failure is
the incompleteness that Einstein attributed to quantum mechanics? As a non-
categorical theory, quantum mechanics is descriptively incomplete, just not in
the sense that it fails to describe all of the physical reality that it aims
to describe, but rather in the sense that it describes more than just the
physical reality that it aims to describe. This does not necessarily imply
that the theory describes states that cannot exist; rather, it describes
dynamically incompatible states, construed as (elements of) non-isomorphic
models.

Why is the non-categoricity of quantum mechanics not considered a problem
today? Why is it’s having fallen out of sight not itself considered as a
serious problem? Why is categoricity not a primary goal of contemporary
quantum physicists? These are important questions that I cannot address here.
What I want to briefly emphasize before I conclude this paper are some
implications I take my reconstruction of Einstein’s argument to have for our
understanding of his debate with Bohr.

Let us recall one important point in that debate, that is, that Bohr’s reply
to the EPR paper, as was well understood by Einstein, denies separability
(Howard 2007). Thus, it also applies to Einstein’s own argument (by rejecting
the first half of premise 1). Einstein famously commented on this point as
follows: “By this way of looking at the matter it becomes evident that the
paradox forces us to relinquish one of the following two assertions: (1) the
description by means of the $\psi$-function is complete. (2) the real states
of spatially separated objects are independent of each other.” (Einstein 1949,
682) Einstein thought that relinquishing assertion (2), i.e., denying that
spatially separated objects have real states, would make physics impossible.
But a second important point made by Bohr is that EPR-doubles are
complementary (Bohr 1935). What does this point amount to, when considered in
the algebraic framework described above? Bohr’s complementarity has been
rigorously interpreted in terms of the unitary inequivalence of non-regular
Hilbert space representations (Halvorson 2004). Since Einstein’s argument, on
my reconstruction, points out that there are representations describing
subsystem A of an EPR system that are unitarily inequivalent (premise 4T),
Bohr’s reply can be understood as a rejection of premise 5T: a complete
quantum mechanics does not require unitary equivalence of representations.
Furthermore, if it is correct to reformulate premise 5T as premise 5H, then
one can see Einstein and Bohr holding opposite positions in a spectrum of
model-theoretical views regarding quantum mechanics: while Einstein would
deplore its non-categoricity, Bohr would embrace it as a theoretical asset.

##  6 Conclusion

The reconstruction of Einstein’s argument offered in this paper clarifies the
assumptions behind Howard’s suggestion to read Einstein completeness as
categoricity. It also reinforces criticism of some of the misconstruals of
Einstein completeness in the literature. Finally, it explains the sense in
which Einstein completeness is different than EPR completeness as a type of
descriptive completeness, and it sheds some light on the Bohr-Einstein
controversy. A more thorough understanding of these issues would certainly
benefit from a fully articulated formal semantics of quantum mechanics.

##  7 Statement

There are no conflicts of interest in connection to this paper.

##  8 Acknowledgements

I am grateful to Don Howard for making the suggestion discussed in this paper
and for his now decades-long mentorship; to Guido Bacciagaluppi for reminding
me of the suggestion, in Utrecht, in 2017; to Jos Uffink for drawing my
attention to the interpretation of Einstein completeness as overcompleteness,
in Salzburg, in 2019; and more recently, to Marton Gömöri, Gábor Hofer-Szabó,
Sebastian Horvat, and Magdalena Zych for comments and discussion that helped
improve the paper. Any remaining errors are my sole responsibility.

##  9 References

Acuña, P. (2021) von Neumann’s Theorem Revisited, Foundations of Physics, 51,
73

Arageorgis, A. (1995) Fields, particles, and curvature: Foundations and
philosophical aspects of quantum field theory in curved spacetime, Ph.D.
thesis, University of Pittsburgh

Arens, R. and V. S. Varadarajan (2000) On the concept of
Einstein–Podolsky–Rosen states and their structure, Journal of Mathematical
Physics, 41, 638–651

Bohr, N. (1935) Can Quantum-Mechanical Description of Physical Reality Be
Considered Complete? Physical Review, 48, 696–702

Detlefsen, M. (2014) Completeness and the ends of axiomatization, Interpreting
Gödel, 59–77

Einstein A., B. Podolski, and N. Rosen (1935) Can Quantum-Mechanical
Description of Physical Reality Be Considered Complete? Physical Review, 47,
777–780

Einstein A. (1936) Physics and Reality, Ideas and opinions, 1954, 290–323

Einstein A. (1949) Remarks Concerning the Essays Brought together in this Co-
operative Volume, Albert Einstein: Philosopher-Scientist, 665–688

Feintzeig, B., J.B. Le Manchak, S. Rosenstock, & J.O. Weatherall (2019) Why Be
regular?, part I, Studies in History and Philosophy of Modern Physics, 65,
122–132

Fine, A. (1981) Einstein’s Critique of Quantum Theory: The Roots and
Significance of EPR, The Shaky Game, 1986, 26–39

Gömöri, M. and G. Hofer-Szabó (2021) On the meaning of EPR’s Reality
Criterion, Synthese 199, 13441–13469.

Halvorson, H. (2000) The Einstein-Podolsky-Rosen State Maximally Violates
Bell’s Inequalities, Letters in Mathematical Physics 53, 321–329

Halvorson, H. (2004) Complementarity of representations in quantum mechanics,
Studies in History and Philosophy of Modern Physics, 35, 45–56

Howard, D. (1985) Einstein on Locality and Separability, Studies in History
and Philosophy of Science, 16, 171–201

Howard, D. (1990) ‘Nicht Sein Kann Was Nicht Sein Darf’, Or the Prehistory of
the EPR, 1909-1935: Einstein’s Early Worries About the Quantum Mechanics of
Composite Systems, Sixty-Two Years of Uncertainty: Historical, Philosophical,
and Physical Inquiries into the Foundations of Quantum Mechanics, ed. by A. I.
Miller, Plenum Press, 61–111

Howard, D. (1992) Einstein and Eindeutigkeit: A Neglected Theme in the
Philosophical Background to General Relativity, Studies in the History of
General Relativity, ed. by J. Eisenstaedt and A. J. Kox, Birkhäuser, 154–243

Howard, D. (2007) Revisiting the Einstein-Bohr Dialogue, Iyyun: The Jerusalem
Philosophical Quarterly, 56, 57–90

Hughes, R. I. G. (1989) The Structure and Interpretation of Quantum Mechanics,
Harvard University Press

Landsman, N. P. (1990) C∗-algebraic quantization and the origin of topological
quantum effects, Letters in Mathematical Physics, 20, 11–18

Landsman, N. P. (2006) When champions meet: Rethinking the Bohr–Einstein
debate, Studies in History and Philosophy of Modern Physics, 37, 212–242

Lehner, C. (2014) Einstein’s Realism and His Critique of Quantum Mechanics,
The Cambridge Companion to Einstein, CUP, 306–353

Lupini, M. (2019) An Invitation to Model Theory and C∗-Algebras, The Bulletin
of Symbolic Logic, 25, 34–100

McGrath, J. H. (1978) A Formal Statement of the Einstein-Podolsky-Rosen
Argument, International Journal of Theoretical Physics, 17, 557–571

Murgueitio Ramírez, S. (2020) Separating Einstein’s separability, Studies in
History and Philosophy of Modern Physics, 72, 138–149

Norsen, T. (2017) Foundations of Quantum Mechanics, Springer

Rosenberg, J. (2004) A selective history of the Stone–von Neumann Theorem,
Operator Algebras, Quantization, and Noncommutative Geometry, Providence, RI,
American Mathematical Society, 331–354

Ruetsche, L. (2011) Interpreting Quantum Theories. The Art of the Possible,
Oxford University Press.

Sauer, T. (2007) An Einstein manuscript on the EPR paradox for spin
observables, Studies in History and Philosophy of Modern Physics 38, 879–-887

Stone, M. H. (1930) Linear Transformations in Hilbert Space, Proceedings of
the National Academy of Sciences of the USA, 16, 172–175

Summers, S. J. (2001) On the Stone-von Neumann Uniqueness Theorem and Its
Ramifications, John von Neumann and the Foundations of Quantum Physics, ed. by
M. Rédei and M. Stöltzner, Springer, 135–152

Tarski, A. (1934) Some methodological investigations on the definability of
concepts, Logic, Semantics, Metamathematics, Hackett, 298–319

Toader, I. D. (2021) On the Categoricity of Quantum Mechanics, European
Journal for Philosophy of Science, 11, 17

Uffink, J. (2020) Schrödinger’s reaction to the EPR paper. Quantum,
Probability, Logic, ed. by M. Hemmo and O. Shenker, Springer, 545–566

von Neumann, J. (1931) Die Eindeutigkeit der Schrödingerschen Operatoren,
Mathematische Annalen, 104, 570–578

Werner, R. F. (1999) EPR states for von Neumann algebras, arXiv:quant-
ph/9910077

Weatherall, J. O. (2019) Part 1: Theoretical equivalence in physics,
Philosophy Compass, 10.1111/phc3.12592


# Jointly Explicit and Implicit Cross-Modal Interaction Network for Anterior
Chamber Inflammation Diagnosis

Qian Shao1, Ye Dai2, Haochao Ying3, Kan Xu2, Jinhong Wang1, Wei Chi2, Jian Wu4
Corresponding authors.Corresponding authors.

###### Abstract

Uveitis demands the precise diagnosis of anterior chamber inflammation (ACI)
for optimal treatment. However, current diagnostic methods only rely on a
limited single-modal disease perspective, which leads to poor performance. In
this paper, we investigate a promising yet challenging way to fuse multimodal
data for ACI diagnosis. Notably, existing fusion paradigms focus on empowering
implicit modality interactions (_i.e.,_ self-attention and its variants), but
neglect to inject explicit modality interactions, especially from clinical
knowledge and imaging property. To this end, we propose a jointly Explicit and
implicit Cross-Modal Interaction Network (EiCI-Net) for Anterior Chamber
Inflammation Diagnosis that uses anterior segment optical coherence tomography
(AS-OCT) images, slit-lamp images, and clinical data jointly. Specifically, we
first develop CNN-Based Encoders and Tabular Processing Module (TPM) to
extract efficient feature representations in different modalities. Then, we
devise an Explicit Cross-Modal Interaction Module (ECIM) to generate attention
maps as a kind of explicit clinical knowledge based on the tabular feature
maps, then integrated them into the slit-lamp feature maps, allowing the CNN-
Based Encoder to focus on more effective informativeness of the slit-lamp
images. After that, the Implicit Cross-Modal Interaction Module (ICIM), a
transformer-based network, further implicitly enhances modality interactions.
Finally, we construct a considerable real-world dataset from our collaborative
hospital and conduct sufficient experiments to demonstrate the superior
performance of our proposed EiCI-Net compared with the state-of-the-art
classification methods in various metrics.

## Introduction

Uveitis, the inflammation inside the eye, accounts for approximately 10$\%$ of
blindness worldwide, which occurs in every age group, including even children
(Jabs 2008; Williams et al. 2007; Gutteridge and Hall 2007). Definitely,
uveitis can be anatomically and easily classified into anterior, intermediate,
posterior, and panuveitis by the primary site of inflammation. Importantly,
the accurate diagnosis of inflammation type can effectively assist in guiding
therapy. Prior work demonstrated that community ophthalmologists
overwhelmingly see anterior uveitis (91$\%$) in clinical (McCANNEL et al.
1996). Thus, the anterior chamber inflammation (ACI) diagnosis and
classification of disease severity are crucial for choosing appropriate
therapeutic measures.

Recently, the Standardization of Uveitis Nomenclature (SUN) scoring system has
been widely used to grade ACI by slit-lamp images in clinical (D.A., R.B., and
J.T. 2005). Slit-lamp images can reflect not only the overall state outside
the eyeball, but also various important indicators inside the anterior chamber
(AC) for diagnosing ACI (shown in Figure 1 a$\sim$d), including the number of
AC cells, iris pigmentation, keratic precipitates (KPs), and so on. However,
the SUN system is a subjective scoring and grading system that can easily lead
to marked differences by interobservers depending on different experiences
(Kempen et al. 2008). Additionally, when the severity level of ACI is high,
the distribution of AC cells becomes relatively dense which makes it both
time-consuming and labor-intensive for doctors to diagnose ACI by counting
these cells using the naked eye as shown in Figure 1c. As a result,
automatically diagnosing and classifying ACI needs to be developed.

Figure 1: Comparison of slit-lamp images (a$\sim$d) and AS-OCT images (e): a.
ciliary hemorrhage at the limbus; b. Iris pigmentation; c. AC cells; d. KPs;
e. AC cells. Slit-lamp images, despite containing rich information indicative
of ACI, challenge automated algorithms to detect AC cells due to their complex
backgrounds and the small area of AC cells. Conversely, AC cells are more
detectable in AS-OCT images, though without portraying the comprehensive state
outside the eyeball state and iris pigmentation.

However, it is very challenging for current automated algorithms to accurately
detect and count these cells in slit-lamp images due to the complexity of the
background in slit-lamp images and the small area occupied by AC cells within
the images as shown in Figure 1c. Consequently, an increasing number of
researchers resort to anterior segment optical coherence tomography (AS-OCT)
images analysis (Invernizzi et al. 2017; Sharma et al. 2015; Baghdasaryan et
al. 2019). The background information in AS-OCT images is simple, allowing for
accurate and easy segmentation of the AC. After applying threshold processing,
AC cells can be detected as demonstrated in Figure 1e. Nevertheless, AS-OCT
imaging serves as a cross-sectional imaging technique for the AC, generating
hundreds of AS-OCT images per eye (_i.e.,_ 128 images in our experiments),
while some indicators (_e.g.,_ KPs) typically appear on a limited section of
these cross-sectional AS-OCT images. In addition, AS-OCT images solely depict
the internal features of the AC, without the ability to reflect the overall
state outside the eyeball and iris pigmentation which also helps the
classification of ACI.

In summary, slit-lamp images offer more comprehensive information to identify
the presence of ACI, but quantifying detailed features using automated
algorithms is challenging such as the number of AC cells. On the other hand,
AS-OCT images provide limited information, yet it is easier to employ
automated algorithms to quantify the number of AC cells. Therefore,
efficiently exploiting and fusing each modal strength will significantly
improve prediction performance. But, existing fusion paradigms focus on
empowering implicit modality interactions (_i.e.,_ self-attention and its
variants), but neglect to inject explicit modality interactions, especially
from expert knowledge and imaging property. For example, Zheng et al. (Zheng
et al. 2022) and Saeed et al. (Saeed et al. 2022) adopt transformer to fuse
multimodal data implicitly for the respective tasks. Thus, exploring the
explicit and implicit modality interactions for ACI diagnosis needs to be
further studied.

To this end, we propose a jointly explicit and implicit cross-modal
interaction network called EiCI-Net, which mainly consists of Feature Encoders
(CNN-Based Encoders), a Tabular Processing Module (TPM), an Explicit Cross-
Modal Interaction Module (ECIM) and an Implicit Cross-Modal Interaction Module
(ICIM). Specifically, two CNN-Based Encoders are used to learn rich
informativeness from two types of slit-lamp images (_i.e.,_ the overall state
images outside eyeball and the observation images of AC) respectively.
Meanwhile, the TPM first quantifies the features of AS-OCT images which are
difficult to obtain from slit-lamp images, and then encodes these features
together with other clinical data that may be closely related to ACI to
establish tabular feature as the explicit cross-modal interaction features.
Then, the ECIM generates attention maps by calculating the correlation between
slit-lamp image feature maps and tabular feature maps, guiding the slit-lamp
feature maps to focus more on effective task-related informativeness. Finally,
the ICIM adopts a transformer-based network for implicit modality
interactions.

The main contributions can be summarized as follows:

  * •

We propose EiCI-Net to diagnose ACI using multimodal data. To our best
knowledge, we are the first to diagnose ACI under the fusion of multimodal
data.

  * •

We focus on the implicit and explicit modality interactions jointly for the
diagnosis of ACI. The experiment demonstrates that explicit modality
interactions improve the performance significantly, which current methods
always neglect to inject.

  * •

We construct the first real-world multimodal ACI diagnosis dataset for
research and conduct extensive experiments to evaluate the effectiveness of
EiCI-Net. The results validate the superior performance over current methods
in various metrics.

## Related Work

### Anterior Chamber Inflammation Diagnosis

The current standard for measuring ACI was established by the SUN
International Workshop which assigns the grade of inflammation a score between
0 (no inflammation) to 4+ (severe inflammation) based on the number of visible
cells in a 1 × 1 mm slit-lamp beam in the AC through the central cornea
(Maring et al. 2022). Although manual estimation of cell count on the slit-
lamp exam has become the gold standard for grading ACI, several limitations
exist (Sharma et al. 2015). The SUN system is a subjective scoring and grading
system that leads to marked differences by interobservers depending on
different experiences. Therefore, there have been efforts to quantitatively
grade the ACI by the advances in optical imaging technology, such as Anterior
segment optical coherence tomography (AS-OCT). For example, some researchers
(Akbarali et al. 2021; Invernizzi et al. 2017; Lu et al. 2020) calculate the
number of hyperreflective dots representing AC cells and optical density ratio
for flare qualification from AS-OCT images to diagnose ACI manually. However,
manual methods are time-consuming and labor-intensive, so automated algorithms
(Sharma et al. 2015; Baghdasaryan et al. 2019; Li et al. 2013) are developed
to measure the number of cells seen in the AS-OCT images for ACI diagnosis.
For example, Baghdasaryan et al. (Baghdasaryan et al. 2019) use the Image J
Particle Analysis algorithm to analyze the degree of ACI automatically. And
deep learning algorithm is also adopted to count and measure the size of AC
cells (Wu et al. 2018). In summary, each modality data provides different
advantages for the diagnosis of ACI in clinical, and none of the above methods
pay attention to the integration of them.

Figure 2: The overall architecture of our proposed EiCI-Net. \small{C}⃝
denotes the concatenate operation, while $\cdot$⃝ denotes multiplying by
element.

### Fusion of Multimodal Data

In the field of medical artificial intelligence, the most common method of
multimodal data fusion is to map the different modal data to a high-
dimensional feature space, then fuse these features using different
algorithms, and finally to achieve specific tasks, such as classification,
segmentation or survival prediction (Zheng et al. 2022; Fang et al. 2022;
Saeed et al. 2022; Mallya and Hamarneh 2022). The current methods usually
adopt implicit modality interactions (i.e., self-attention and its variants).
For example, Zheng et al. (Zheng et al. 2022) use a multimodal transformer for
feature fusion and a decision-level fusion method to predict survival of
nasopharyngeal carcinoma patients; Saeed et al. (Saeed et al. 2022) adopt a
transformer for fusion of different modality data to achieve the tasks of
segmentation and survival prediction. But existing work neglects to inject
explicit modality interactions, especially from expert knowledge and imaging
property. As an illustration, ophthalmologists usually diagnose ACI by
combining the advantages of slit-lamp and AS-OCT examination. Slit-lamp images
reflect the overall condition of the eyeball and KPs intuitively that are not
easily observed on AS-OCT images. However, the quantification of the number of
AC cells and the degree of flare through slit-lamps may be largely influenced
by the experience of ophthalmologists, while these indicators can be
objectively and accurately quantified in AS-OCT images. Therefore, on the
basis of implicit cross-modal interaction, we also perform cross-modal
interaction based on these explicit features for the diagnosis of ACI.

## Methodology

### Overview

The overall architecture of our EiCI-Net seeks to efficiently exploit the
advantages of each modality to enhance ACI diagnosis performance through
explicitly and implicitly multimodal features fusion. As shown in Figure 2,
EiCI-Net consists of two Convolutional Neural Network based Encoders (CNN-
Based Encoders), a Tabular Processing Module (TPM), an Explicit Cross-Modal
Interaction Module (ECIM) and an Implicit Cross-Modal Interaction Module
(ICIM). Specifically, in order to extract rich informativeness from slit-lamp
images, we use two CNN-Based Encoders to encode two types of slit-lamp images
(_i.e.,_ the overall state images outside eyeball and the observation images
of AC) respectively. At the same time, the TPM first quantifies the features
of AS-OCT images which are difficult to obtain from slit-lamp images such as
the number of AC cells, and then encodes these features together with other
clinical data that may be closely related to ACI, such as best corrected
vision acuity (BCVA), intraocular pressure and KPs to establish tabular
feature as the explicit cross-modal interaction features. Next, the ECIM
generates attention maps by calculating the correlation between slit-lamp
image feature maps and tabular feature maps, then integrating them into the
slit-lamp feature maps, allowing the CNN-Based Encoder to focus on more
effective informativeness of the slit-lamp images such as AC cells. Afterward,
the ICIM employs a transformer-based network for implicit modality
interactions. Next, we depict the details of each module in the following
subsections.

### Convolutional Neural Network Based Encoder

In order to obtain comprehensive information for the diagnosis of ACI,
ophthalmologists usually use diffuse light to observe the overall state
outside the eyeball, and use slit light to observe the condition of different
parts of the AC. Therefore, based on this expert experience, we collect two
types of slit-lamp images for each case, including two overall images that
reflect the overall state outside the eyeball and two observation images that
reflect various important indicators inside the AC. Note that we bring two
images in each type to further magnify feature representation because
ophthalmologists often observe AC from different angles. To extract image
features from each type, technically, we can use any image encoder
architecture. Here, we use a Resnet18 (He et al. 2016) as the backbone because
it is a kind of classic and efficient network for feature extraction.
Additionally, the complexity of Resnet18 matches our dataset which can be
verified in the later experiments. Formally, in the process of overall image
feature extraction, we take each image $I_{i}^{Overall}\in\mathbb{R}^{H\times
W\times C},\,i=\\{1,2\\}$ as input, where $H$, $W$ and $C$ are the height,
width, and channel respectively. The output image features from the encoder is
represented as:

$f_{i}^{Overall}=CNNEncoder(I_{i}^{Overall}),\,i=\\{1,2\\}.$ (1)

Then we take the concatenated results of two feature maps along the channel
dimension $f^{Overall}\in\mathbb{R}^{H^{Overall}\times W^{Overall}\times
C^{Overall}}$ as the final features. Note that the process of the observation
images is the same as above, and the final feature of AC observation images
can be represented as $f^{AC}$.

### Tabular Processing Module

The Tabular Processing Module (TPM) first quantifies the features of AS-OCT
images which are difficult to obtain from slit-lamp images including area,
number of AC cells and aqueous-to-air relative intensity (ARI). In order to
establish tabular feature maps, TPM then encodes these 3 features together
with other 6 clinical features that may be closely related to ACI including
gender, age, BCVA, intraocular pressure, grading of keratic precipitates (KPs)
and grading of aqueous flare. All these 9 features are considered tabular data
which can be divided into two types: numerical data and categorical data.
Among them, gender, grading of KPs and grading of aqueous flare are
categorical, while age, BCVA, intraocular pressure, area and number of AC cell
and aqueous-to-air relative intensity (ARI) are numerical.

In conclusion, the overall process is divided into two steps: quantification
of AS-OCT image features and establishment of tabular feature maps.

#### Quantification of AS-OCT Image Features

First, we use U-net (Ronneberger, Fischer, and Brox 2015) to segment the AC
area, and obtain the boundary between the AC and the cornea. We take the two
endpoints of the boundary as the coordinate points of the AC angle. Next, a
2mm by 4mm (true intraocular distance) rectangular area $Area_{1}$ is
specified in the center of the AC, the long side of which is close to the
boundary and parallel to the line connecting these two endpoints. Then, a 1mm
by 2mm rectangular area $Area_{2}$ is specified at the top left of the image,
which is used for the calculation of air intensity. In order to calculate the
number of AC cells, we perform threshold processing on the image. The points
whose pixel value is higher than the threshold in $Area_{1}$ are candidate
points for AC cells. The adjacent candidate points can be considered one AC
cell. Finally, we calculate ARI according to the following formulas:

$\begin{split}ARI&=\frac{\frac{1}{n_{Area_{1}}-n_{CP}}\sum_{i\in{Area_{1}}-CP}PV_{i}}{\frac{1}{n_{Area_{2}}}\sum_{j\in{Area_{2}}}PV_{j}}\\\
&=\frac{n_{Area_{2}}\sum_{i\in{Area_{1}}-CP}PV_{i}}{(n_{Area_{1}-n_{CP}})\sum_{j\in{Area_{2}}}PV_{j}},\end{split}$
(2)

among which $n_{Area_{1}}$, $n_{Area_{2}}$ are the number of pixels in
$Area_{1}$ and $Area_{2}$ respectively, $CP$ denotes all the candidate points
in $Area_{1}$, while $n_{CP}$ is the number of candidate points. And $PV_{i}$
and $PV_{j}$ denote the pixel value of the point $i$ and $j$ separately.

#### Establishment of Tabular Feature Maps

We expand the dimensions of tabular data to make them consistent with that of
image features for explicit and implicit cross-modal interactions in the later
experiments. The specific method is as follows: we copy its value and
construct a matrix matching the width and height of image features. Then we
obtain the tabular feature denoted as:

$f^{Tabular}=TabularEncoder(Tabular\,data),$ (3)

where the height and width of the tabular feature are the same as
$H^{Overall}$ and $W^{Overall}$, while the channel size is half of
$C^{Overall}$. By ensuring uniformity of all values within each channel of
$f^{Tabular}$, this operation enables it to serve as a channel attention
mechanism that introduces feature maps from other modalities, thereby
adaptively recalibrating channel-wise feature responses.

### Explicit Cross-Modal Interaction Module

The ECIM introduces explicit modeling of the interdependencies among the
feature map channels of the slit-lamp images by incorporating external expert
knowledge (_i.e.,_ $f^{Tabular}$), which consequently allows the CNN-Based
Encoder to concentrate on the more effective informativeness of the slit-lamp
images, such as AC cells and KPs.

First, we multiply each element of $f^{Tabular}$ and
$f_{i}^{AC},\,i=\\{1,2\\}$, then perform Softmax operations on the channel
dimension to obtain the attention map. Then, we multiply each element of the
attention map and $f^{AC}$ to obtain the attention weighted feature map
$f^{AC\\_attention}$, allowing the CNN Based Encoder to focus on more detailed
information of the slit-lamp images. The above process can be expressed as:

$Attention\,Map_{i}=\sigma(Mul(f^{Tabular},\,f_{i}^{AC}),\,i=\\{1,2\\},$ (4)
$f_{i}^{AC\\_attention}=Mul(Attention\,Map_{i},\,f_{i}^{AC}),\,i=\\{1,2\\},$
(5)

where $\sigma$ and $Mul$ represent the element-wise Soft-max function and
multiplying by element operation. Note that, aligning with the unified
attention mechanism with key, query, and value, $f^{Tabular}$ means query and
$f_{i}^{AC}$ is equal to both key and value.

### Implicit Cross-Modal Interaction Module

Self-attention based transformer essentially aligns data from different
modalities by calculating the similarity between inputs, making it very
suitable for the fusion of multimodal data. Therefore, we propose ICIM which
is a transformer-based network for implicit modality interactions.

First, we concat $f^{Overall}$, $f^{AC\\_attention}$ and $f^{Tabular}$ along
the channel dimension to obtain the fusion feature $f^{Fusion}$. In order to
reduce the computational effort, we transform the channel size of $f^{Fusion}$
to $C^{Fusion}$ (_i.e.,_ from 2560 to 512 in our experiment). Since the
Transformer encoder expects sequence-style as input, we treat each voxel of
the feature along the channel dimension as a sequence of fusion feature
(Vaswani et al. 2017), therefore the Transformer receives $(H^{Overall}\times
W^{Overall})$ sequences as input and outputs a vector denoted as
$V\in\mathbb{R}^{(H^{Overall}\times W^{Overall})\times C^{Fusion}}$ to form
$C^{Fusion}$ predicted values as the final feature vector. Finally, we perform
a fully-connected (FC) layer and Softmax on the final feature vector to
predict the probability of ACI. The prediction loss function can be expressed
as:

$Loss=-\frac{1}{B}\sum_{j=1}^{B}\sum_{i=1}^{n}y_{ji}log(\hat{y_{ji}}),$ (6)

where $B$ is the batch size, $n$ is the number of categories, $y_{ji}$ is the
real probability that the $j$-th sample belongs to $i$-th category, and
$\hat{y_{ji}}$ is the predicted probability that the $j$-th sample belongs to
$i$-th category.

## Experiments

### Subjects

This study includes 104 eyes with uveitis and 35 healthy eyes of 82
participants from our collaborating hospital between May 23rd, 2018, and
October 10th, 2022. All patients are diagnosed with unilateral or bilateral
uveitis involving the anterior segment. During the same period, 35 healthy
eyes without a history of ocular inflammation, injury, surgery, or other
ocular diseases are recruited as a normal control group. Eyes with corneal
opacity or shallow AC are excluded. Informed consent is obtained from each
participant at enrollment. This study is approved by the Ethics Committee of
State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-
sen University, Guangdong Provincial Key Laboratory of Ophthalmology and
Visual Science, Guangdong Provincial Clinical Research Center for Ocular
Diseases. Finally, we evenly divide the dataset into 5 parts and used 5-fold
cross-validation to evaluate model performance.

### Clinical Examination

All participants underwent a comprehensive ocular examination by a uveitis
specialist, including best corrected vision acuity (BCVA), intraocular
pressure measurements, slit-lamp observation of the AC, prefacing lens
observation of the posterior segment and scoring of ACI according to SUN
criteria (D.A., R.B., and J.T. 2005). Clinical data collected included gender,
age, BCVA, intraocular pressure, grakeratic precipitates (KPs), aqueous flare
and uveitis diagnosis. Eyes with uveitis were divided into two groups based on
clinical evaluation: inactive eyes when both AC cells and aqueous flare
clinically were graded 0, while active eyes with any other grading score. In
addition, all participants were examined by CASIA SS-1000 (Tomey, CASIA1), a
Swept Source OCT (SS-OCT) which scans at a rate of 30,000 A scans per second
in the anterior segment scan (16mm*6mm) mode. Each eye contains 3 AS-OCT
images close to 0°-180° cross section.

### Experimental Settings and Metrics

All models are implemented in PyTorch and trained on an RTX 3090 with 24GB
memory. Specifically, we use stochastic gradient descent (SGD) with an initial
learning rate $1e^{-3}$ to optimize the network. For a fair comparison, the
epoch and the batch size are set to $100$ and $8$. The code is available in
Supplementary Material. We will release publicly the code and dataset.

We calculate the true positive (TP), true negative (TN), false positive (FP),
and false negative (FN) values from the confusion matrix to obtain four
widely-used evaluation metrics, including $accuracy$, $F_{1}\,score$,
$precision$, and $recall$. Note that we report the average indicators of the
5-fold cross-validation to evaluate the model performance.

Method | $Accuracy$ | $F_{1}\,score$ | $Precision$ | $Recall$
---|---|---|---|---
Overall images only | Resnet18 | 0.878 | 0.923 | 0.889 | 0.962
ViT | 0.835 | 0.897 | 0.854 | 0.952
Observation images only | Resnet18 | 0.886 | 0.924 | 0.908 | 0.943
ViT | 0.834 | 0.897 | 0.836 | 0.971
AS-OCT images only | Resnet18 | 0.921 | 0.950 | 0.921 | 0.981
ViT | 0.784 | 0.873 | 0.791 | 0.981
Clinical data only | MLP | 0.843 | 0.908 | 0.837 | 1.000
SVM | 0.906 | 0.941 | 0.914 | 0.971
Multimodal data | EiCI-Net | 0.935 | 0.958 | 0.950 | 0.971
Table 1: Comparison With Other Methods Using a Single Modality Data.

### Comparison With Other Methods Using a Single Modality Data

In order to prove the superiority of our proposed EiCI-Net, we compared the
performance with four classification methods by using a single modality data,
including:

(1) Resnet18 (He et al. 2016): a classical CNN using overall state images,
slit-lamp observation images or AS-OCT images as input only;

(2) ViT (Dosovitskiy et al. 2021): a network of self-attention-based
architecture using overall state images, observation images, or AS-OCT images
only;

(3) Multilayer Perceptron (MLP): a network of three hidden layers for
classification using clinical data only;

(4) Support Vector Machine (SVM) (Pedregosa et al. 2011): a traditional
machine learning method for classificaiton using clinical data only.

Method | $Accuracy$ | $F_{1}\,score$ | $Precision$ | $Recall$
---|---|---|---|---
Multi-transSP | 0.921 | 0.950 | 0.912 | 0.990
TMSS | 0.907 | 0.944 | 0.897 | 1.000
EiCI-Net | 0.935 | 0.958 | 0.950 | 0.971
Table 2: Comparison With Other Methods Using Multimodal Data.

The experimental results are shown in Table 1 on which we have several
observations: (1) Our method has obtained the consistent best performance
under the metrics of $accuracy$, $F_{1}\,score$, and $precision$, and
competitive performance under $recall$, which is superior to other methods
based on single modality data. This indicates that fusing multimodal data to
diagnose ACI is effective. (2) The performance of the Resnet18 using slit-lamp
images only is almost superior to that of ViT. This may be due to the high
complexity of ViT and the small size of our dataset, which results in over-
fitting, whereas the complexity of Resnet18 matches our dataset to achieve
better performance. (3) Resnet18 trained by AS-OCT images performed second
only to EiCI-Net, outperforming slit-lamp image-based methods. This may be
because the background of AS-OCT images is relatively simple, which
facilitates the automatic extraction of features, while there are several
parts in the slit-lamp images that interfere with the diagnosis such as
eyelashes, slit-lamp bands, etc. (4) The performance of SVM based on clinical
data is superior to the methods based on slit-lamp images. From the point of
view of the modality itself, the clinical data is relatively streamlined and
contains indicators that are strongly correlated with ACI, such as aqueous
flare, KPs, etc. While slit-lamp image modality is relatively redundant, and
the information it provides for diagnosing task largely depends on the feature
extraction method. In a word, traditional feature extraction methods cannot
extract the features of slit-lamp images relatively effectively.

### Comparison With Other Methods Using Multimodal Data

In order to prove the superiority of our proposed EiCI-Net, we compared the
performance with other two fusion methods (Zheng et al. 2022; Saeed et al.
2022) in the field of medical artificial intelligence. Multimodal data
includes: two types of slit-lamp images, AS-OCT images and clinical data. Two
fusion methods are as follows:

(1) Multi-transSP (Zheng et al. 2022): implicit feature-level fusion and
decision-level fusion methods to predict survival of nasopharyngeal carcinoma
patients;

(2) TMSS (Saeed et al. 2022): transformer for implicit fusion of different
modality data to achieve the tasks of segmentation and survival prediction.

Because our task is different from the above methods, only the feature fusion
part of their methods was used in the comparison experiment.

The experimental results are shown in Table 2 on which we have several
observations: (1) The performance of our method is better than the two
methods. Specifically, the $accuracy$ of our model is 0.014 and 0.028 higher
than that of Multi-transSP (Zheng et al. 2022) and TMSS (Saeed et al. 2022),
respectively. While the $F_{1}\,score$ is 0.008 and 0.014 higher than that of
Multi-transSP (Zheng et al. 2022) and TMSS (Saeed et al. 2022), respectively.
The results demonstrate that the method using jointly implicit and explicit
cross-modal interactions is superior to the method using only implicit cross-
modal interactions; (2) Several methods using single modality data (_i.e.,_
Resnet18 using AS-OCT images only and SVM using clinical data only) achieve
performance comparable to the above methods, which may be because multimodal
data contains a large amount of redundant information, making it difficult to
interact and fuse effective information using implicit cross-modal interaction
method only, thereby limiting the performance of the model. While our model
explicitly applies this effective information directly to the interaction
between modalities, thus achieving higher performance.

Key Component | $Accuracy$ | $F_{1}\,score$ | $Precision$ | $Recall$
---|---|---|---|---
$Encoder^{Overall}$ | $Encoder^{AC}$ | TPM | ECIM | ICIM
✓ | ✓ | ✓ |  | ✓ | 0.893 | 0.932 | 0.888 | 0.981
✓ | ✓ | ✓ | ✓ |  | 0.893 | 0.936 | 0.893 | 0.990
| ✓ | ✓ | ✓ | ✓ | 0.928 | 0.954 | 0.937 | 0.971
✓ |  | ✓ |  | ✓ | 0.863 | 0.913 | 0.888 | 0.942
✓ | ✓ |  |  | ✓ | 0.886 | 0.929 | 0.885 | 0.981
✓ | ✓ | ✓ | ✓ | ✓ | 0.935 | 0.958 | 0.950 | 0.971
Table 3: Ablation Study of the Contribution of Key Components in EiCI-Net.

### Ablation Study

Finally, we conducted an ablation study to investigate the contribution of key
components in our EiCI-Net. The experimental setup can be summarized as
follows: (1) EiCI-Net without ECIM; (2) EiCI-Net without ICIM but a
convolutional layer for feature fusion. (3) EiCI-Net without CNN-Based Encoder
for feature extraction of the overall state images; (4) EiCI-Net without CNN-
Based Encoder for feature extraction of the observation images; (5) EiCI-Net
without TPM. We should note that the ECIM generates cross-modal attention maps
based on $f^{AC}$ and $f^{Tabular}$. Therefore, when the CNN-Based Encoder for
feature extraction of the observation images or TPM is removed in the ablation
experiments, the ECIM cannot work.

Figure 3: Visualization of EiCI-Net w or w/o ECIM.

The quantitative results are shown in Table 3 on which we have several
observations: (1) When we remove any module, the performance of the model
decreases to varying degrees, indicating that each module is valid. (2) When
$Encoder^{AC}$ or TPM is removed, the most significant drop in model
performance occurs ($accuracy$ decreased by 0.072 and 0.049 respectively,
while $F_{1}\,score$ decreased by 0.045 and 0.029 respectively), indicating
that observation images and tabular data are more important modalities. This
is consistent with the results shown in Table 1, where observation images or
clinical data based methods perform better than other modality-based methods.
(3) When $Encoder^{Overall}$ is removed, the model performance degradation is
least noticeable, indicating that overall image modality is secondary. This is
consistent with the results shown in Table 1, where overall image based
methods perform less well than other modality based methods. (4) When ECIM or
ICIM is removed, the degree of decrease in $accuracy$ is the same at 0.042,
which demonstrates that the explicit cross-modal interaction and the implicit
cross-modal interaction are almost equally important.

In addition, we employ Grad-CAM (Selvaraju et al. 2017) to visually observe
the area of interest of EiCI-Net, proving that ECIM allows the CNN-Based
Encoder to focus on more effective informativeness of the slit-lamp images, as
shown in Figure 3. The first column images in Figure 3 are the original
observation images, the second are heat maps generated based on
$f^{AC\\_attention}$ with ECIM and the third are heat maps generated based on
$f^{AC}$ without ECIM. The specific process of heat map generation is that we
use the gradients of logits for ACI, flowing into the final convolutional
layer of CNN Based Encoder which is used for feature extracion of observation
images to produce the heat map highlighting the important regions in the
image. In the heat maps, the red area is what the model focuses on.

In Figure 3, we can see that the model with ECIM focuses more on indicators
that ophthalmologists will pay attention to in clinical examinations, such as:
(1) the pupil in Figure 3a is not round which reflects uveitis in the past;
(2) AC cells in Figure 3b and Figure 3c which is one of the most important
indicators of ACI; (3) the vitreous body and lens in Figure 3d are turbid
which suggests the possibility of ACI. In contrast, the model without ECIM
pays less attention to these information, which proves that the ECIM plays an
important role in the cross-modal interaction.

## Conclusion

In this paper, we proposed a cross-modal interaction network called EiCI-Net
to jointly learn the explicit and implicit feature correlations among all
modalities to diagnose ACI. The experimental results on our in-house dataset
demonstrated that it is superior to the state-of-the-art classification
methods. However, our method also has certain limitations which can only
predict whether there is ACI but cannot predict the grade of inflammation.
Clinically, it is also difficult to accurately grade the ACI, especially for
the grade larger than 4, because the AC cells of patients with high-grade
inflammation are usually very dense under slit-lamp observation, and may
appear cloudy on AS-OCT images, which is difficult to evaluate quantitatively
by naked eyes or automated algorithms. Therefore, our subsequent research will
focus on the grade classification of ACI.

## References

  * Akbarali et al. (2021) Akbarali, S.; Rahi, J. S.; Dick, A. D.; Parkash, K.; Etherton, K.; Edelsten, C.; Liu, X.; and Solebo, A. L. 2021.  Imaging-Based Uveitis Surveillance in Juvenile Idiopathic Arthritis: Feasibility, Acceptability, and Diagnostic Performance.  _Arthritis Rheumatol._ , 73(2): 330–335.
  * Baghdasaryan et al. (2019) Baghdasaryan, E.; Tepelus, T. C.; Marion, K. M.; Huang, J.; Huang, P.; Sadda, S. R.; and Lee, O. L. 2019.  Analysis of ocular inflammation in anterior chamber—involving uveitis using swept-source anterior segment OCT.  _Int. Ophthalmol._ , 39(8): 1793–1801.
  * D.A., R.B., and J.T. (2005) D.A., J.; R.B., N.; and J.T., R. 2005.  Standardization of Uveitis Nomenclature for Reporting Clinical Data. Results of the First International Workshop.  _Amer. J. Ophthalmol._ , 140(3): 509–516.
  * Dosovitskiy et al. (2021) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In _International Conference on Learning Representations_.
  * Fang et al. (2022) Fang, J.; Li, A.; OuYang, P.-Y.; Li, J.; Wang, J.; Liu, H.; Xie, F.-Y.; and Liu, J. 2022.  Weighted Concordance Index Loss-Based Multimodal Survival Modeling for Radiation Encephalopathy Assessment in Nasopharyngeal Carcinoma Radiotherapy.  In Wang, L.; Dou, Q.; Fletcher, P. T.; Speidel, S.; and Li, S., eds., _Medical Image Computing and Computer Assisted Intervention – MICCAI 2022_ , 191–201. Cham: Springer Nature Switzerland.
  * Gutteridge and Hall (2007) Gutteridge, I. F.; and Hall, A. J. 2007.  Acute anterior uveitis in primary care.  _Clin. Exp. Optom._ , 90(2): 70–82.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep Residual Learning for Image Recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 770–778.
  * Invernizzi et al. (2017) Invernizzi, A.; Marchi, S.; Aldigeri, R.; Mastrofilippo, V.; Viscogliosi, F.; Soldani, A.; Adani, C.; Garoli, E.; Viola, F.; Fontana, L.; McCluskey, P.; and Cimino, L. 2017.  Objective Quantification of Anterior Chamber Inflammation: Measuring Cells and Flare by Anterior Segment Optical Coherence Tomography.  _Ophthalmol._ , 124(11): 1670–1677.
  * Jabs (2008) Jabs, D. A. 2008.  Epidemiology of Uveitis.  _Ophthal. Epidemiol._ , 15(5): 283–284.
  * Kempen et al. (2008) Kempen, J. H.; Ganesh, S. K.; Sangwan, V. S.; and Rathinam, S. R. 2008.  Interobserver Agreement in Grading Activity and Site of Inflammation in Eyes of Patients with Uveitis.  _Amer. J. Ophthalmol._ , 146(6): 813–818.e1.
  * Li et al. (2013) Li, Y.; Lowder, C.; Zhang, X.; and Huang, D. 2013.  Anterior Chamber Cell Grading by Optical Coherence Tomography.  _Invest. Ophth. Vis. Sci._ , 54(1): 258–265.
  * Lu et al. (2020) Lu, M.; Wang, X.; Lei, L.; Deng, Y.; Yang, T.; Dai, Y.; Li, Y.; Gan, X.; Hu, Y.; Chen, H.; Li, M.; Su, L.; Yuan, J.; and Chi, W. 2020.  Quantitative Analysis of Anterior Chamber Inflammation Using the Novel CASIA2 Optical Coherence Tomography.  _Amer. J. Ophthalmol._ , 216: 59–68.
  * Mallya and Hamarneh (2022) Mallya, M.; and Hamarneh, G. 2022.  Deep Multimodal Guidance for Medical Image Classification.  In Wang, L.; Dou, Q.; Fletcher, P. T.; Speidel, S.; and Li, S., eds., _Medical Image Computing and Computer Assisted Intervention – MICCAI 2022_ , 298–308. Cham: Springer Nature Switzerland.
  * Maring et al. (2022) Maring, M.; Saraf, S. S.; Blazes, M.; Sharma, S.; Srivastava, S.; Pepple, K. L.; and Lee, C. S. 2022.  Grading Anterior Chamber Inflammation with Anterior Segment Optical Coherence Tomography: An Overview.  _Ocul. Immunol. Inflamm._ , 30(2): 357–363.
  * McCANNEL et al. (1996) McCANNEL, C. A.; HOLLAND, G. N.; HELM, C. J.; CORNELL, P. J.; WINSTON, J. V.; RIMMER, T. G.; and THE UCLA COMMUNITY-BASED UVEITIS STUDY GROUP. 1996.  Causes of Uveitis in the General Practice of Ophthalmology.  _Amer. J. Ophthalmol._ , 121(1): 35–46.
  * Pedregosa et al. (2011) Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; Vanderplas, J.; Passos, A.; Cournapeau, D.; Brucher, M.; Perrot, M.; and Duchesnay, E. 2011.  Scikit-learn: Machine Learning in Python.  _J. Mach. Learn. Res._ , 12: 2825–2830.
  * Ronneberger, Fischer, and Brox (2015) Ronneberger, O.; Fischer, P.; and Brox, T. 2015.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  In Navab, N.; Hornegger, J.; Wells, W. M.; and Frangi, A. F., eds., _Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015_ , volume 9351, 234–241. Cham: Springer International Publishing.  Series Title: Lecture Notes in Computer Science.
  * Saeed et al. (2022) Saeed, N.; Sobirov, I.; Al Majzoub, R.; and Yaqub, M. 2022.  TMSS: An End-to-End Transformer-Based Multimodal Network for Segmentation and Survival Prediction.  In Wang, L.; Dou, Q.; Fletcher, P. T.; Speidel, S.; and Li, S., eds., _Medical Image Computing and Computer Assisted Intervention – MICCAI 2022_ , 319–329. Cham: Springer Nature Switzerland.
  * Selvaraju et al. (2017) Selvaraju, R. R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; and Batra, D. 2017.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , 618–626.
  * Sharma et al. (2015) Sharma, S.; Lowder, C. Y.; Vasanji, A.; Baynes, K.; Kaiser, P. K.; and Srivastava, S. K. 2015.  Automated Analysis of Anterior Chamber Inflammation by Spectral-Domain Optical Coherence Tomography.  _Ophthalmol._ , 122(7): 1464–1470.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017.  Attention is All You Need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, 6000–6010. Red Hook, NY, USA: Curran Associates Inc.
  * Williams et al. (2007) Williams, G. J.; Brannan, S.; Forrester, J. V.; Gavin, M. P.; Paterson-Brown, S. P.; Purdie, A. T.; Virdi, M.; and Olson, J. A. 2007.  The prevalence of sight-threatening uveitis in Scotland.  _Brit. J. Ophthalmol._ , 91(1): 33–36.
  * Wu et al. (2018) Wu, Y.; Xiao, S.; Rokem, A.; Lee, C. S.; Wilson, L.; Pepple, K.; Sabesan, R.; and Lee, A. Y. 2018.  Fully automated quantification of retinal cones and anterior chamber cells using deep learning.  _Invest. Ophth. Vis. Sci._ , 59(9): 1222.
  * Zheng et al. (2022) Zheng, H.; Lin, Z.; Zhou, Q.; Peng, X.; Xiao, J.; Zu, C.; Jiao, Z.; and Wang, Y. 2022.  Multi-transSP: Multimodal Transformer for Survival Prediction of Nasopharyngeal Carcinoma Patients.  In Wang, L.; Dou, Q.; Fletcher, P. T.; Speidel, S.; and Li, S., eds., _Medical Image Computing and Computer Assisted Intervention – MICCAI 2022_ , 234–243. Cham: Springer Nature Switzerland.