# Background Independence and Quantum Causal Structure

L. Parker<EMAIL_ADDRESS>Centre for Engineered Quantum Systems, School of
Mathematics and Physics, University of Queensland, QLD 4072 Australia  F.
Costa<EMAIL_ADDRESS>Centre for Engineered Quantum Systems, School of
Mathematics and Physics, University of Queensland, QLD 4072 Australia

###### Abstract

One of the key ways in which quantum mechanics differs from general relativity
is that it requires a fixed background reference frame for spacetime. In fact,
this appears to be one of the main conceptual obstacles to uniting the two
theories. Additionally, a combination of the two theories is expected to yield
‘indefinite’ causal structures. In this paper, we present a background-
independent formulation of the process matrix formalism—a form of quantum
mechanics that allows for indefinite causal structure—while retaining
operationally well-defined measurement statistics. We do this by imposing that
the probabilities arising in the formalism—which we ascribe to measurement
outcomes across the points of a discrete spacetime—be invariant under
permutations of spacetime points. We find (a) that one still obtains
nontrivial, indefinite causal structures with background independence, (b)
that we lose the idea of local operations in distinct laboratories, but can
recover it by encoding a reference frame into the physical states of our
system, and (c) that permutation invariance imposes surprising symmetry
constraints that, although formally similar to a superselection rule, cannot
be interpreted as such.

##  1 Introduction

In a quantum theory of gravity it is expected that spacetime itself will be
quantised, giving rise to indefinite, or ‘quantum’, causal structures [1, 2].
The process matrix formalism was developed to describe these causal structures
[3]. In fact, it describes the most general causal relations between a finite
set of regions, or ‘parties’, compatible with the local validity of quantum
mechanics in each region. However, the framework relies on an _a priori_
labelling of the parties, which tacitly presupposes the existence of a
background reference frame. This is in conflict with the background
independence of general relativity, which associates no absolute meaning to
individual spacetime points or regions [4, 5]. Incorporating background
independence into the quantum formalism is in fact one of the main challenges
in the development of a theory of quantum gravity [6, 7].

In order to represent a viable approach to quantum gravity, the process matrix
formalism should be able to describe indefinite causal structures without
reference to a fixed background. Here, we show how this can be done within a
toy model, where we treat a process matrix as a particular configuration of a
discretised spacetime, with laboratories that correspond to the discrete units
of that spacetime. A process matrix will be background independent if it is
invariant under any arbitrary permutation of ‘laboratories’ or points of
spacetime.

In this paper, we introduce background independent processes and describe some
of their properties. First, we note that indefinite causal structures still
arise in permutation-invariant processes. We show that imposing permutation
invariance results in the loss of a distinction between spacetime points. As
in general relativity, one recovers a distinction between spacetime points by
using a material reference frame (a reference frame made up of physically
observable systems, the ‘rods and clocks’ picture).

Finally, we discuss the symmetry properties of permutation-invariant
processes. We expect, in analogy to symmetries in ordinary quantum mechanics,
that permutation invariance should give rise to a superselection rule for some
‘charge’—a property that can only take definite values, with coherence between
different charge values forbidden (but classical probabilistic mixtures
allowed). However, we find that, while indeed coherence between charge sectors
is suppressed, processes cannot in general be understood as mixtures of
different charge values, as the definite-charge components of the mixture are
not valid processes. We show explicitly why this occurs in the case of a
bipartite qubit process (where a ‘qubit’ process is just one with two-
dimensional local Hilbert spaces). We also present a partial proof
generalising that result to any process matrix dwelling in the symmetric or
antisymmetric subspaces of the symmetric group. Our results suggest that no
invariant processes with a definite charge may exist, although more work will
be needed to substantiate this conjecture. The breakdown in the association
between symmetries and superselection rules may indicate that background
independence in quantum mechanics cannot be interpreted analogously to other
known symmetries of nature.

##  2 Quantum processes in spacetime

We will first review the process matrix formalism in its most common
presentation—using a language borrowed from quantum information. Afterwards,
we will discuss how this formalism can be used as a model of discretised
spacetime.

###  2.1 The process matrix formalism

The process matrix formalism is a framework for quantum mechanics that does
not assume any global background causal structure, just that quantum mechanics
is obeyed locally. Conceptually, it extends quantum mechanics in a similar way
in which, relaxing global Lorentz invariance, one can extend special
relativity to general relativity. Relaxing the assumption of causal structure
allows one to obtain new, ‘indefinite’ causal relations that are incompatible
with a fixed ordering of events. Relationships of this type have been observed
in the laboratory [8, 9, 10, 11, 12, 13, 14, 15, 16], where the lack of causal
order arises from temporally delocalised events, rather than from a quantum
spacetime [17, 18]. Much of the experimental interest derives from the
applications of indefinite causal relations to computation and communication
[19, 20, 21, 22, 23, 24, 25, 26, 27]. Here, we briefly describe the aspects of
the process formalism that are relevant to this work. For more details, see
references [3, 28, 29].

The simplest way to think of process matrices is as follows. Consider a system
of $N$ laboratories. Each laboratory is occupied by an experimenter capable of
performing all of the preparations, operations, and measurements compatible
with the standard measurement formalism of quantum mechanics. Formally, this
means that each experimenter has the ability to perform a quantum instrument—a
set $\mathcal{I}^{x}=\\{\mathcal{M}_{i}^{X}\\}_{i=1}^{n}$ of completely
positive (CP) maps that sum to a completely positive and trace preserving
(CPTP) map. The superscript $X$ denotes that the maps
$\mathcal{M}^{X}:\mathcal{L}\left(\mathcal{H}^{X_{I}}\right)\to\mathcal{L}\left(\mathcal{H}^{X_{O}}\right)$
act on laboratory $X$. The Hilbert spaces $\mathcal{H}^{X_{I}}$,
$\mathcal{H}^{X_{O}}$, respectively represent the incoming and outgoing state-
spaces of laboratory $X$, with $\mathcal{L}\left(\mathcal{H}\right)$ denoting
the linear operators on $\mathcal{H}$.

Consider the case where we have two parties, Alice and Bob, who respectively
have access to instruments $\mathcal{I}^{A}=\\{\mathcal{M}^{A}_{i}\\}$ and
$\mathcal{I}^{B}=\\{\mathcal{N}^{B}_{j}\\}$. The probability that Alice and
Bob realise a particular combination of operations
$\mathcal{M}^{A}_{i},\mathcal{N}^{B}_{j}$ is given by some probability
distribution $P(\mathcal{M}^{A}_{i},\mathcal{N}^{B}_{j})$. To be consistent
with quantum mechanics, $P$ must be a multilinear map [28]. The Choi-
Jamiołkowski isomorphism [30, 31] allows us to represent these operations by
sending CP maps $\mathcal{M}^{X}$ to positive semidefinite linear operators
$M^{X_{I}X_{O}}_{i}:=[\mathcal{I}\otimes\mathcal{M}^{X}_{i}(\outerproduct{\phi^{+}}{\phi^{+}})]^{T}\in\mathcal{L}(\mathcal{H}^{X_{I}}\otimes\mathcal{H}^{X_{O}})$,
where $\ket{\phi^{+}}=\sum_{i}\ket{i}^{X_{I}}\otimes\ket{i}^{X_{I}}$ is a non-
normalised maximally entangled state and T denotes transposition in the
computational basis. These operators act over an input Hilbert space $X_{I}$
and an output Hilbert space $X_{O}$. In this representation, the trace
preserving condition reads $\Tr_{X_{O}}M^{X_{I}X_{O}}=\mathbbm{1}^{X_{I}}$;
this means that, for a set of maps that form an instrument, we must have
$\Tr_{X_{O}}[\sum_{i}M^{X_{I}X_{O}}_{i}]=\mathbbm{1}^{X_{I}}$.

Figure 1: A graphical depiction of a process matrix $W=W^{AB}$ with two
laboratories, $A$ and $B$. Operations $M_{i}^{A}$ and $N_{j}^{B}$ are
performed on the systems entering laboratories $A$ and $B$, respectively. The
probability of $M_{i}^{A}$ and $N_{j}^{B}$ being realised out of their
respective instruments is given by
$P(M_{i}^{A},N_{j}^{B})=\Tr[W^{A}\cdot(M^{A}_{i}\otimes N^{B}_{j})]$ (Eq. (1).
Note that we have shortened the laboratory labellings from those used in Eq.
(1), so that, in the superscripts, $A=A_{I}A_{O}$ and $B=B_{I}B_{O}$.

Our complete list of probabilities $P$ now becomes a multilinear map over
linear operators. This map is equivalent to [32, prop. 2.38]

$\displaystyle P(M^{A_{I}A_{O}}_{i}\otimes N^{B_{I}B_{O}}_{j})$
$\displaystyle=\Tr[W^{A_{I}A_{O}B_{I}B_{O}}\cdot(M^{A_{I}A_{O}}_{i}\otimes
N^{B_{I}B_{O}}_{j})],$ (1)

for some linear operator
$W^{A_{I}A_{O}B_{I}B_{O}}\in\mathcal{L}(\mathcal{H}^{A_{I}A_{O}B_{I}B_{O}})$.

$W^{A_{I}A_{O}B_{I}B_{O}}$ is called a process matrix, and is the
generalisation of a joint quantum state (from the point of view of a
probability measure) to correlations that can be spacelike, timelike, or
neither—those with indefinite causal structure. Process matrices must satisfy
the constraints

$\displaystyle W^{A_{I}A_{O}B_{I}B_{O}}\geq 0,$ (2)
$\displaystyle\Tr[W^{A_{I}A_{O}B_{I}B_{O}}\cdot(M^{A_{I}A_{O}}N^{B_{I}B_{O}})]=1,$
(3) $\displaystyle\forall\,M,N\geq 0,$
$\displaystyle\Tr_{A_{O}}[M^{A_{I}A_{O}}]=\mathbbm{1}^{A_{I}},\Tr_{B_{O}}[N^{B_{I}B_{O}}]=\mathbbm{1}^{B_{I}},$

which ensure that probabilities are nonnegative and sum to one. We have left
out tensor product symbols for convenience, and will continue to do so where
it is clear.

In a Hilbert-Schmidt basis, i.e., a basis $\\{\sigma^{X}_{i}\\}$ of
$\mathcal{L}\left(\mathcal{H}\right)$ satisfying
$\sigma_{0}^{X}=\mathbbm{1}^{X}$,
$\Tr[\sigma^{X}_{i}\sigma^{X}_{j}]=d_{X}\delta_{ij}$,
($d_{X}:=\dim(\mathcal{H}^{X})$) and $\Tr[\sigma^{X}_{i}]=0$ for $i>0$, a
process matrix can be represented as

$W^{A_{I}A_{O}B_{I}B_{O}}=\sum_{ijkl}w_{ijkl}\sigma_{i}^{A_{I}}\sigma_{j}^{A_{O}}\sigma_{k}^{B_{I}}\sigma_{l}^{B_{O}},$
(4)

where $w_{ijkl}\in\mathbb{R}$ since $W^{A_{I}A_{O}B_{I}B_{O}}$ is hermitian.

The probability normalisation requirement forbids certain Hilbert-Schmidt
terms from appearing in the decomposition of an allowed process. We call terms
with identity on all outputs except $X$ type $X$ process terms, all outputs
except $X,Y$ terms of type $XY$ etc. Forbidden bipartite process terms are
terms of the form $A_{O}$, $B_{O}$, $A_{O}B_{O}$, $A_{I}A_{O}B_{O}$,
$A_{O}B_{I}B_{O}$, and $A_{I}A_{O}B_{I}B_{O}$. The process constraint, Eq.
(3), requires that $\Tr[W\sigma]=0$ for any such terms $\sigma$. Thus, we
effectively have linear constraints on the matrix elements of allowed
processes.

One consequence of not making _a priori_ assumptions about the causal
structure is the appearance of novel types of causal order that cannot be
expressed in the standard formalism of quantum mechanics. Process matrices can
be causally ordered, which corresponds to the familiar situation where A comes
before B comes before C, or they can be causally separable, convex
combinations of processes that have different causal orders such as ‘A before
B’ and ‘B before A’, representing classical ignorance of causal order. One
novel aspect of the process matrix formalism is that one can also have
indefinite causal order, where it does not make sense to say that ‘A is before
B’ or vice versa: there are signalling correlations from A to B and also from
B to A, which cannot be interpreted as classical ignorance.

Throughout this section we have only discussed bipartite processes, for
simplicity. Everything we have discussed generalises straightforwardly to an
arbitrary number of parties. We refer the reader to references [3, 28, 29, 33,
34, 35] for a more complete discussion.

###  2.2 Process matrices as toy spacetime configurations

The crux of this paper is in the fact that we can use process matrices as toy
models of spacetime. It is worth going into some detail about what exactly
this means.

Most of the literature on quantum causal structures, in particular that
concerned with experiments or practical applications, regards a process matrix
as a representation of a finite number of finite-size ‘regions’, which can be
thought of as physical laboratories in which measurements are made on some
quantum state. This is the interpretation that has been used in our discussion
of the process matrix formalism so far.

It is now time to move beyond this interpretation. For the rest of this paper,
we will treat a given process matrix as a toy model of a discretised
spacetime. This is consistent with several approaches to quantum gravity,
which stipulate that spacetime is discrete at a fundamental level [36, 37, 38,
39, 40]. However, here we do not consider any explicit model, but simply a
generic framework that allows us to avoid the complications associated with
continuous spacetime.

Within this framework, each of what we have been referring to as
‘laboratories’ now corresponds to a fundamental, indivisible unit of our
discrete spacetime —in other words, to a single spacetime point. Each
indivisible point can be understood as the theoretical maximal spatio-temporal
resolution of a measurement apparatus. Occasionally, we will still refer to
the elementary spacetime points as ‘laboratories’ or ‘parties’, in keeping
with the terminology in the literature, but it must be remembered that this
does not entail the existence of a physical laboratory at each point.

We can make the connection with the more ordinary conception of spacetime more
explicit. This is usually defined as a differentiable manifold, consisting of
an uncountable set $\mathfrak{M}$ that is locally diffeomorphic to Minkowski
space. Any classical physical property, including the metric, is represented
in the form of fields, namely as functions $\phi(x)$ defined at each spacetime
point $x\in\mathfrak{M}$. In a quantum theory, the fields at each point are
not uniquely defined: a measurement at $x$ can yield a (possibly uncountable)
set of outcomes $\left\\{\phi_{i}(x)\right\\}_{i}$, where $i$ labels the
outcomes. A measurement at the point $x$ is clearly an idealisation, with any
physical measurement extending to a finite region, but the basic operational
meaning of a spacetime point is still the (limit of) the smallest possible
region where a measurement can be made.

Figure 2: A visual representation of a diffeomorphisms $\phi$ (smooth
relabelling of points in a manifold) and a permutation $\sigma$ (relabelling
of elements in a finite set). A permutation is the discrete analogue of a
diffeomorphism.

Our model simply replaces the uncountable set $\mathfrak{M}$ with a finite set
of spacetime points, which we label $x=1,\dots,n$. (In section 2.1 above, we
used $x=A,B,\dots$ Other labellings will be used below according to
convenience). As before, physical quantities can be defined at each point.
Importantly, pairs of points in spacetime can be timelike separated (or even
have undefined causal relations), meaning that a measurement at one point can
influence measurements at different points. An instrument is the most general
representation of a measurement (or operation), including how it affects the
measured system. An individual CP map $\mathcal{M}^{1}_{i_{1}}$ within an
instrument represents the measurement of a physical quantity at point $1$,
yielding outcome $i_{1}$, with a transformation of the system described by
$\mathcal{M}^{1}_{i_{1}}$. The Born rule for processes—Eq. (1) extended to $n$
points—governs the probability distribution $P(i_{1},\dots,i_{n})$ to observe
outcomes $i_{1},\dots,i_{n}$ at spacetime points $1,\dots,n$. Note that it is
not necessary to imagine an observer acting at each point in spacetime: We can
have points where no operation is performed, corresponding to an instrument
with a single element—the identity map.

The process matrix contains all the information relevant for evaluating the
probabilities for arbitrary operations at any spacetime point. In a
traditional model, such information would be encoded in an initial state, in
the dynamical laws describing the propagation of systems across spacetime, and
in the metric encoding the causal connections between points. In a successful
formulation of quantum gravity, the process matrix should unify such
information in a way that does not depend on a background causal structure.

##  3 Why permutation invariance?

By thinking of a process matrix as representing a particular configuration of
a discrete spacetime, we can make an analogy between background independence
in the process matrix formalism and background independence in general
relativity.

In general relativity, background independence is a consequence of the fact
that observable quantities must be invariant under any arbitrary coordinate
transformation111Note that this is a much stronger requirement than simply
stating that we can use arbitrary coordinate systems. Indeed, most theories
include observables that are not invariant under coordinate transformations.
For example, in relativistic quantum field theory, we can measure the value of
a field at a point $x$ or at a different point $y$, and these are different
observables that can yield different outcomes. Such observables are not
allowed in general relativity, as coordinate transformations act both on the
fields and on the background spacetime.. Formally, these transformations are
smooth, invertible mappings from a manifold to itself, and are called
diffeomorphisms. In a discrete spacetime model, the smoothness requirement
does not apply. Hence, a discrete coordinate transformation is any invertible
map from the set of spacetime points onto itself. As our spacetime toy model
contains only a finite set of points, its coordinate relabellings are defined
as arbitrary permutations. Therefore, our task is to characterise invariance
of observable quantities under permutations in the process matrix formalism.

In the process matrix formalism, the statistical properties of observables are
given by e.g. Eq. (1), the Born-rule generalisation for processes. In general,
Eq. (1) generates a multipartite probability distribution $P(i_{1},...,i_{n})$
given a particular process $W$, where $P(i_{1},...,i_{n})$ denotes the
probability that measurement at point $1$ obtains outcome $i_{1}$, measurement
at point $2$ obtains outcome $i_{2}$, etc. Although $P$ does not assume any
causal structure, it does in general assume that it is possible to distinguish
between and label the different points. Operationally, this implies the
existence of some background reference frame, which allows one to determine
that outcome $i_{1}$ corresponds to party 1, outcome $i_{2}$ to party 2, and
so on. In other words, the labels of the parties form a (discrete) system of
coordinates. In a background-independent theory, no background reference frame
is available, so it is not possible to identify a point in spacetime with a
coordinate. Therefore, the theory should be invariant under relabelling.

As discussed above, in a discrete spacetime model relabellings are simply
permutations. As a consequence, in order for observable quantities to be
invariant under relabellings, the probability distribution $P$ must be
invariant under permutations of the parties, i.e.
$P(i_{1},i_{2},...,i_{n})=P(i_{g(1)},i_{g(2)},...,i_{g(n)}),$ for all
permutations $g\in S_{n}$. Permutation invariance as a discrete analogue of
diffeomorphism invariance is also discussed, for example, in Ref. [41].

Invariance under permutations is a particular case of invariance under an
arbitrary symmetry group. A general framework for dealing with this has been
developed in Ref. [42]. Although this framework deals with Lie groups rather
than finite groups (such as permutations), the main results, which we will use
below, also hold for finite groups.

First, we must introduce a mathematical representation for permutations, which
we will use throughout the paper. Just as the group of diffeomorphisms are
represented by a (continuous) diffeomorphism group, the (finite) group of
permutations of a set of $n$ elements is known as the symmetric group and is
denoted $S_{n}$.

We define the representation of the symmetric group $S_{n}$ on the space of
$n$-party process matrices as the map from elements $g\in S_{n}$ to operators
$U_{g}$ such that $U_{g}WU^{\dagger}_{g}$ performs a permutation on the
laboratories. For example, the action of the ‘swap’ permutation $U_{AB}$ on a
bipartite process in the Hilbert-Schmidt basis is

$U_{AB}\left(\sum_{ijkl}\alpha_{ijkl}\sigma^{A_{I}}_{i}\sigma^{A_{O}}_{j}\sigma^{B_{I}}_{k}\sigma^{B_{O}}_{l}\right)U^{\dagger}_{AB}\\\
=\sum_{ijkl}\alpha_{ijkl}\sigma^{A_{I}}_{k}\sigma^{A_{O}}_{l}\sigma^{B_{I}}_{i}\sigma^{B_{O}}_{j}.$
(5)

Note that input and output spaces are always swapped together. In order to
make permutations well-defined, we assume that the input spaces of all
laboratories have equal dimensions, and similarly all output spaces. It is not
difficult to generalise, but we will not do so here [43].

We say that a linear operator $A$ (which can be a process matrix or a
measurement, or more generally even a quantum state or POVM element) is
permutation invariant if it is unchanged by the action of any permutation,
i.e. $U_{g}A{U_{g}}^{\dagger}=A,\ \forall g\in S_{n}$. Equivalently, $A$ is
permutation invariant if $\mathcal{G}[A]=A$, where $\mathcal{G}$ is the twirl
operation,

$\mathcal{G}[W]:=\frac{1}{n!}\sum_{g\in S_{n}}U_{g}W{U_{g}}^{\dagger}.$ (6)

##  4 Process matrices without spacetime events.

Now we can formalise the ideas we introduced in the previous section. Our
overarching goal is to develop a framework for processes in which measurement
statistics are permutation-invariant, so that the processes are background-
independent. We find that there are different ways to achieve this.

One way to ensure permutation invariant statistics is to restrict measurements
to be permutation invariant. As we will discuss below, invariant measurements
are not necessarily of the product form. Therefore, we will denote a general
measurement operator across all labs with a single symbol $M_{\bm{i}}$, where
$M_{\bm{i}}\equiv M_{i_{1}}^{A}M_{i_{2}}^{B}...$ for the particular case of
product measurements and $\bm{i}\equiv(i_{1},i_{2},\dots)$ denotes the
collection of outcomes (which are still associated with the distinct
labs/spacetime points). If $M_{\bm{i}}$ is invariant, we have
$P(i_{g(1)},i_{g(2)},...,i_{g(n)})=\Tr[WU_{g}M_{\bm{i}}U_{g}^{\dagger}]=\Tr[WM_{\bm{i}}]=P(i_{1},i_{2},...,i_{n})$,
as required. However,
$\Tr[WU_{g}M_{\bm{i}}U_{g}^{\dagger}]=\Tr[U_{g}^{\dagger}WU_{g}M_{\bm{i}}]$,
so $W=\mathcal{G}[W]$ also implies that measurement statistics are invariant,
even if $M_{\bm{i}}\neq\mathcal{G}[M_{\bm{i}}]$. Finally, the measurement
statistics will be permutation-invariant if both $W$ and $M_{\bm{i}}$ are
permutation-invariant. Although they all represent the same physics, each
choice is suggestive of a different way to describe permutation invariance.

If measurement operators are permutation-invariant, but not the processes
themselves, then we might think of processes as being described relative to
some fixed background that we cannot access, so that we are restricted to
using permutation-invariant measurements.

If process matrices are permutation-invariant, but not the measurement
operators, then we might instead say that we are making measurements relative
to some background reference frame, but that what we observe is permutation-
invariant—choosing a different reference frame will give us the same
statistics.

If both process and measurements are permutation-invariant, then we have
totally abandoned the concept of a fixed, absolute reference frame. As in
General Relativity, any reference frame must be defined using the available
degrees of freedom of the system, the ‘rods and clocks’ picture. The ordering
with respect to which we describe our system is not a physically relevant
aspect of the theory, just like the description of spacetime relative to a
particular choice of coordinates in General Relativity. In other words, the
original ordering, implied by the tensor product of local Hilbert spaces,
merely represents a fictitious, rather than physical, reference frame, and
permutations only represent transformations relative to such a fictitious
reference frame.

The three pictures we just described are operationally equivalent, and it is
sometimes helpful to work in one picture over another. For example, by
requiring that process matrices $W$ are permutation-invariant,
$W=\mathcal{G}[W]$, we can see that even with as strict a constraint as
permutation-invariance one still obtains nontrivial behaviour of processes.
Consider the process matrix

$W=\frac{1}{4}\big{[}\mathbbm{1}^{\otimes
4}+a^{\prime}_{0}\sigma_{z}\mathbbm{1}\sigma_{z}\mathbbm{1}-a^{\prime}_{1}(\sigma_{z}\mathbbm{111}+\mathbbm{11}\sigma_{z}\mathbbm{1})\\\
-a^{\prime}_{2}(\sigma_{z}\mathbbm{11}\sigma_{z}+\mathbbm{1}\sigma_{z}\sigma_{z}\mathbbm{1})+a^{\prime}_{3}(\sigma_{z}\mathbbm{1}\sigma_{z}\sigma_{z}+\sigma_{z}\sigma_{z}\sigma_{z}\mathbbm{1})\\\
+a^{\prime}_{4}(\sigma_{z}\mathbbm{1}\sigma_{x}\sigma_{x}-\sigma_{z}\mathbbm{1}\sigma_{y}\sigma_{y}+\sigma_{x}\sigma_{x}\sigma_{z}\mathbbm{1}-\sigma_{y}\sigma_{y}\sigma_{z}\mathbbm{1})\big{]},$
(7)

which was presented in Ref. [44], with coefficients
$\mathbf{a^{\prime}}=(0.0390,0.3355,0.2451,0.4291,0.2097)$. In Eq. (7) we have
omitted labels, so that $ABCD=A^{A_{I}}B^{A_{O}}C^{B_{I}}D^{B_{O}}$. As shown
in [44], this process can violate a causal inequality—a device-independent
test for indefinite causal order similar to a Bell inequality. Therefore, it
represents a minimal example of permutation-invariant process with no definite
causal order (in the sense that it involves only two parties, each with a
single-qubit system).

Another important consequence that arises is that permutation-invariant
process matrices cannot be causally ordered. We will demonstrate this with an
example. Consider, for simplicity, the framework in which we impose
permutation invariance on processes but not on measurements. Take a process
representing a state $\varrho$ prepared in laboratory $A$ and sent to
laboratory $B$ through a channel $T$,
$W=\varrho^{A_{I}}T^{A_{O}B_{I}}\mathbbm{1}^{B_{O}}$. This is not permutation-
invariant: $A$ can signal to $B$, but $B$ cannot signal to $A$. We can make
the process invariant by taking the mixture
$W^{\mathrm{inv}}=(\varrho^{A_{I}}T^{A_{O}B_{I}}\mathbbm{1}^{B_{O}}+\varrho^{B_{I}}T^{B_{O}A_{I}}\mathbbm{1}^{A_{O}})/2$,
noting the change in superscripts in the second term. $W^{\mathrm{inv}}$ may
be permutation-invariant, but we have lost the ability to determine whether
the state $\varrho$ was prepared in laboratory $A$ and then sent to laboratory
$B$, or the reverse: we cannot perform a measurement that will tell us where
the state preparation occurred. We have lost a way to label laboratories or,
equivalently, a definition of spacetime points—we no longer have a reference
frame for spacetime. This appears to be a general feature of background-
independence, as it is also found in general relativity.

A related phenomenon is that one cannot have an instrument where all
operations are (a) products of local operations, and (b) permutation-invariant
(aside from the degenerate case where each of the measurement operators
$M_{i}=N_{i}N_{i}...N_{i}$ act identically on every laboratory). This arises
because any permutation invariant product of local measurement operators
$M_{i_{1}}M_{i_{2}}...M_{i_{n}}$ must satisfy $M=U_{g}MU_{g}^{\dagger}$ for
all $U_{g}$ and therefore $M_{i}=M_{j}$ for all $i,j$. This might appear
alarming: one of the fundamental tenets of the process matrix formalism is
that measurements can be performed locally. However, in the next section we
will discuss how a definition of locality can be recovered.

##  5 Invariance with a reference frame.

Although in the previous section we found that permutation invariance removes
the distinction between points in spacetime, it is possible to recover a
definition of spacetime points using a material reference frame—a ‘rods and
clocks’ reference frame constructed out of physical systems. This corresponds
to techniques used in other frameworks for relational quantum theory, such as
[45, 46, 47, 48, 49, 50, 51]. In this way, we can encode all non-invariant
processes, such as those typically studied in the literature, into invariant
ones.

The idea of a material reference frame is to take a non-invariant process
matrix, which identifies the Hilbert spaces (e.g. $A_{I}A_{O}$, $B_{I}B_{O}$
etc.) with local laboratories labelled by some external reference frame, and
add to each laboratory a physical reference system whose quantum state encodes
a label uniquely specifying that laboratory. Then, a local observer can
measure this reference system to obtain information about which point of
spacetime they occupy. In this way, we have encoded the information from the
old abstract reference frame into a physical, observable reference frame.

This being done, we remove the external reference by making the joint process
invariant under permutations. The invariant process is simply the sum of all
possible permutations acting on the extended process (consisting of the system
and reference frame), which gives

$W^{\text{inv}}=\frac{1}{n!}\mathcal{R}(W),$ (8)

where

$\mathcal{R}(A):=\sum_{g\in
S_{n}}U^{SR}_{g}A^{S}[01...(n-1)]^{R_{I}}\mathbbm{1}^{R_{O}}{U^{{\dagger}}_{g}}^{SR}$
(9)

is a superoperator that applies to arbitrary operators (not necessarily
process matrices). In Eq. (9), the superscript on $A^{S}$ denotes that it is a
part of the system space $S=S^{1}_{I}S^{1}_{O}...S^{n}_{I}S^{n}_{O}$, while
the superscripts $R=R_{I}R_{O}=R^{1}_{I}R^{1}_{O}...R^{n}_{I}R^{n}_{O}$ denote
the reference frame space, which contains inputs and outputs. We have used the
notation $[\psi]=\outerproduct{\psi}{\psi}$, so that
$[01...(n-1)]^{R_{I}}=\outerproduct{0}{0}^{R^{1}_{I}}...\outerproduct{n-1}{n-1}^{R^{n}_{I}}$.
Finally, $U^{SR}_{g}=U^{S}_{g}U^{R}_{g}$, which acts separately on the system
and reference frame spaces. $U_{g}$ is a representation of the permutation
$g$. This means that a given permutation will act on the input and output
spaces of both the system and the reference frame together, so that each
reference system remains associated with its corresponding laboratory, and
each output space remains associated with its corresponding input space.

Essentially, we have moved from a particular process $W$ to an equivalence
class of processes (the terms in Eq. (9), related by permutations) described
by $\mathcal{R}(W)$, which we treat as the fundamentally meaningful physical
object, just as we consider equivalence classes of diffeomorphism-invariant
spacetimes as the meaningful physical system in relativity.

Using Eq. (9), we can construct permutation-invariant processes that reproduce
the statistics of arbitrary, non-invariant processes. However, as a
consequence of adding a locally observable reference system to each
laboratory, instruments now need to be extended so that the probability of
some measurement occurring is one. This completion turns out to be somewhat
arbitrary, suggesting that there exist physically distinct instruments that
are indistinguishable when using any reference frame.

To obtain permutation-invariant processes and measurements, we use the
following maps:

$\displaystyle W\to W^{\text{inv}}$
$\displaystyle\equiv\frac{1}{n!}\mathcal{R}(W),$ (10) $\displaystyle
M_{i_{1}...i_{n}}^{S}$ $\displaystyle\equiv
M^{S^{1}}_{i_{1}}...M^{S^{n}}_{i_{n}}$ $\displaystyle\to
M^{\text{inv}}_{i_{1}...i_{n}}$
$\displaystyle\equiv\mathcal{R}(M^{S^{1}}_{i_{1}}...M^{S^{n}}_{i_{n}})$
$\displaystyle+\frac{1}{Nd_{O}}\left(\mathbbm{1}^{SR}-\mathcal{R}\left(\mathbbm{1}^{S}\right)\right),$
(11)

where the instrument is composed of $N$ measurements. The additional term
included in the invariant measurement operators is an arbitrary completion
required in order for the instrument to be CPTP. $W^{\text{inv}}$ and
$M^{\text{inv}}_{i_{1}...i_{n}}$ are now invariant under the action of
$S_{n}$.

In the appendix, we prove that $W^{\text{inv}}$ are valid processes, that the
$M^{\mathrm{inv}}_{i_{1},...,i_{n}}$ are each valid elements of instruments,
and that $\sum_{\\{i\\}}M^{\mathrm{inv}}_{i_{1},...,i_{n}}$ is a CPTP map. In
addition, we can show that the Born rule is maintained by the permutation
invariance. Using Lemma 2 from the appendix,

$\displaystyle\begin{split}&\Tr[W^{\text{inv}}M_{i}^{\text{inv}}]\\\
&=\Tr[\frac{1}{n!}\mathcal{R}(W)(\mathcal{R}(M_{i})+\frac{1}{N}(\mathbbm{1}^{SR}-\mathcal{R}(\mathbbm{1}^{S}))]\\\
&=\Tr[\frac{1}{n!}\mathcal{R}(WM_{i})]+\Tr[\frac{1}{n!}(\mathcal{R}(W)-\mathcal{R}(W))]\\\
&=\Tr[\frac{1}{n!}\mathcal{R}(WM_{i})]=\Tr[WM_{i}].\end{split}$ (12)

In the previous section, we mentioned that it is impossible to have an
instrument in which all elements are both permutation-invariant and decompose
into a product of local measurements. Since one of the central ideas of the
process matrix formalism is locality, this was surprising. Here, we see that,
conditionally on measuring in a particular reference frame, the elements of a
permutation-invariant instrument once more decompose into local measurements.
This is because conditioning on a reference frame configuration
$[R_{0}R_{1}...R_{N}]$ is equivalent to projecting onto
$\mathbbm{1}\otimes[R_{0}R_{1}...R_{N}]$, which removes all terms that have a
reference frame state that is different to the one being projected onto. Thus,
we recover our definition of locality and see that it is only meaningful
relative to a physical reference frame.

##  6 Symmetry and superselection.

Usually, symmetry constraints in quantum mechanics give rise to superselection
rules on allowed states. That is, states have some ‘definite property’ and
coherences between different ‘types’ of that property cannot exist.

The archetypal example of a superselection rule is a $U(1)$ gauge symmetry.
For example, electromagnetism obeys a global $U(1)$ symmetry. This symmetry is
associated with a superselection rule for electric charge: states can have any
integer value of charge, but they cannot be in a superposition of two
different charge values. However, it is possible to have a classical
statistical mixture of positive and negative charge, such as for example when
there is some classical uncertainty as to the nature of the particle being
prepared such as an electron or positron.

The reason superselection rules arise can be seen by decomposing the Hilbert
space of states in terms of copies of irreducible representations of the
symmetry group (in our case the group of permutations $S_{n}$).

A Hilbert space with a representation $U_{g}$ of a group $G$ can be decomposed
into a direct sum of charge sectors $\mathcal{H}_{q}$, each containing an
inequivalent representation of $G$. Each charge sector can in turn be
decomposed into a tensor product of a gauge space $\mathcal{M}_{q}$, carrying
an irreducible representation of $G$, and a multiplicity space
$\mathcal{N}_{q}$, carrying an identity representation of $G$. The entire
space decomposes as

$\mathcal{H}=\bigoplus_{q}\mathcal{M}_{q}\otimes\mathcal{N}_{q},$ (13)

so that each charge sector contains a number of copies of a particular
irreducible representation. Each inequivalent representation corresponds to a
different ‘type’ of charge (in the $U(1)$ example, the number of elementary
electric charges). In this decomposition, the twirl \- which, we recall, is
the ‘average over all transformations’ superoperator - can be expressed as
[42]

$\mathcal{G}=\sum_{q}(\mathcal{D}_{\mathcal{M}_{q}}\otimes\mathcal{I}_{\mathcal{N}_{q}})\circ\mathcal{P}_{q},$
(14)

where $\mathcal{D}$ is the completely depolarising map that sends each state
to the maximally mixed state, $\mathcal{I}$ is the identity map, and
$\mathcal{P}_{q}$ is the projector onto $\mathcal{H}_{q}$. Eq. (14) tells us
that linear operators that are $G$-invariant (and therefore twirl-invariant)
must decompose as

$A=\sum_{q}\frac{1}{d_{\mathcal{M}_{q}}}\mathbbm{1}_{\mathcal{M}_{q}}\otimes
A_{\mathcal{N}_{q}},$ (15)

where $d_{\mathcal{M}_{q}}=\text{dim}(\mathcal{M}_{q})$. This restriction is a
superselection rule: requiring that allowed operators are block-diagonal in
the different inequivalent representations is the same as saying that there
can be no coherences between different charges. Additionally, Eq. (15) gives
us information about the physical degrees of freedom associated with each
‘type’ of charge: for a charge $q$, the physical state space reduces to the
invariant subspace $\mathcal{N}_{q}$.

So goes the typical interpretation of a superselection rule: physical objects
have some well-defined charge that can be subject to classical uncertainty but
not quantum indeterminacy. It turns out that, for processes, this standard
interpretation fails222Strictly speaking, the concept of superselection rule
does not apply directly to process matrices, because the normalisation
constraint, Eq. (3), implies that not all vectors in the underlying Hilbert
space represent valid processes. However, we retain a decomposition of the
space of processes into charge sectors, with coherence possible between
sectors. Symmetries should still imply loss of coherence; one could expect
that this corresponds to a superselection rule.. The reason it fails is
because some inequivalent representations do not contain any valid processes
(whether there are no representations that contain valid processes is an open
question). Here, we will show that for any $n$-partite process with two-
dimensional (qubit) local Hilbert spaces, the symmetric and antisymmetric
representations never contain valid processes. First, we will consider the
base case of a bipartite process, and then prove by induction that this will
hold for a process with any number of laboratories, as long as each laboratory
carries a single qubit.

A bipartite process with two-dimensional input and output spaces gives rise to
a 16-dimensional Hilbert space spanned by
$\ket{i}^{A_{I}}\ket{j}^{A_{O}}\ket{k}^{B_{I}}\ket{l}^{B_{O}}\equiv\ket{ijkl},\
i,j,k,l\in\\{0,1\\}$. Permutations of the two laboratories are obtained by the
action of $S_{2}$, which has two elements: the identity element and the swap
element $U_{AB}\ket{ijkl}=\ket{klij}$. There are two ineqivalent
representations, the symmetric and antisymmetric representations (denoted
$\mathcal{W}^{+}$ and $\mathcal{W}^{-}$), which are respectively spanned by

$\displaystyle\ket{\psi_{S}^{(1)}}$ $\displaystyle=\ket{ijij},\
i,j\in\\{0,1\\},$ (16) $\displaystyle\ket{\psi_{S}^{(2)}}$
$\displaystyle=\frac{1}{\sqrt{2}}(\ket{ijkl}+\ket{klij}),\ i,j\neq k,l,$ (17)

for the ‘symmetric representation’, and

$\displaystyle\ket{\psi_{A}}=\frac{1}{\sqrt{2}}(\ket{ijkl}-\ket{klij}),\
i,j\neq k,l,$ (18)

for the ‘anti-symmetric’. In all, the symmetric representation is
10-dimensional, and the antisymmetric is 6-dimensional. The superselection
rule tells us that any physically realisable (permutation invariant) process
must have the form

$W^{A_{I}A_{O}B_{I}B_{O}}=W^{+}+W^{-},$ (19)

where $W^{+}=\sum_{i,j}\alpha_{ij}\outerproduct{w^{+}_{i}}{w^{+}_{j}}$ and
$W^{-}=\sum_{ij}\beta_{ij}\outerproduct{w^{-}_{i}}{w^{-}_{j}}$, where
$w^{\pm}_{i}$ are respectively basis elements of the symmetric $(+)$ or
antisymmetric $(-)$ representations given in eqs. (16)-(18).

Matrices of the form of Eq. (19) will not in general be valid processes.
Process matrices must also satisfy Eq. (3). Solving algebraically for a
closed-form constraint on the diagonals (which can be done with a computer
algebra program such as Mathematica, or by hand) reveals that the trace of any
$W^{+}$ or $W^{-}$ containing no forbidden causal terms and belonging solely
to one of the two inequivalent representations must be zero. This violates the
normalisation constraint of Eq. (3). However, there are matrices
$W=W^{+}+W^{-}$ where the $W^{\pm}$ individually contain some causally
forbidden terms, but $W^{+}+W^{-}$ does not, allowing $W$ to be a valid
process satisfying the normalisation constraint.

We can use the result for bipartite qubit processes as the base case to show
that for any number of qubit laboratories, there will be no valid processes
living in the symmetric or antisymmetric representations. There are two
essential parts to this argument. The first is that, given a process matrix
$W$, tracing out any number of laboratories must result in a valid process. In
particular, for a process $W^{S^{1}...S^{n}}$ with $n$ laboratories,
$\frac{1}{d_{S^{1}...S^{n}\setminus S^{i}S^{j}}}\Tr_{S^{1}...S^{n}\setminus
S^{i}S^{j}}[W^{S^{1}...S^{n}}]=\overset{\sim}{W}{}^{S^{i}S^{j}}$, where
$\Tr_{S^{1}...S^{n}\setminus S^{i}S^{j}}$ denotes the trace over all
laboratories except $S^{i}$ and $S^{j}$ and $d_{S^{1}...S^{n}\setminus
S^{i}S^{j}}$ is the dimension of all the spaces except $S^{i},S^{j}$ , must be
a valid process. $\overset{\sim}{W}{}^{S^{i}S^{j}}$ is known as a reduced
process.

The second part of the argument is that for any state living in the symmetric
(antisymmetric) representation of $S_{n}$, any $(n-1)$-dimensional subsystem
of that state will be in the symmetric (antisymmetric) representation of
$S_{n-1}$. To see this, observe that we can write any $n$-dimensional state
$\ket{\psi}$ as

$\ket{\psi}=\sum_{j}c_{j}\ket{\psi_{j}}^{S^{1}...S^{n-1}}\ket{j}^{S^{n}},$
(20)

where $\ket{\psi_{j}}$ is an $(n-1)$-dimensional state, $c_{j}$ are some
coefficients, and $\ket{j}$, $j=1,...,n$ is a basis state of $S^{n}$. Then, we
have

$\displaystyle\bra{k}^{S^{n}}\ket{\psi}$
$\displaystyle=\bra{k}^{S^{n}}\sum_{j}c_{j}\ket{\psi_{j}}^{S^{1}...S^{n-1}}\ket{j}^{S^{n}}$
$\displaystyle=c_{k}\ket{\psi_{k}}^{S^{1}...S^{n-1}}.$ (21)

If $\ket{\psi}$ lives in the symmetric representation, then
$U_{g}\ket{\psi}=\ket{\psi}\forall g\in G$. In particular, this holds for all
$g$ that leave the state $\ket{j}^{S^{n}}$ in system $S^{n}$ unchanged. From
this, we can see that, for $g\in S_{n-1}$ and $U_{g}$ acting on the first
$n-1$ subsystems,

$\displaystyle\begin{split}&\qquad U_{g}\ket{\psi}=\ket{\psi}\\\
&\Rightarrow\sum_{j}c_{j}U_{g}\ket{\psi_{j}}^{S^{1}...S^{n-1}}\ket{j}^{S^{n}}\\\
&\qquad=\sum_{j}c_{j}\ket{\psi_{j}}^{S^{1}...S^{n-1}}\ket{j}^{S^{n}}\\\
&\Rightarrow\bra{k}^{S^{n}}\sum_{j}c_{j}U_{g}\ket{\psi_{j}}^{S^{1}...S^{n-1}}\ket{j}^{S^{n}}\\\
&\qquad=\bra{k}\sum_{j}c_{j}\ket{\psi_{j}}^{S^{1}...S^{n-1}}\ket{j}^{S^{n}}\\\
&\Rightarrow
c_{k}U_{g}\ket{\psi_{k}}^{S^{1}...S^{n-1}}=c_{k}\ket{\psi_{k}}^{S^{1}...S^{n-1}}\\\
&\Rightarrow
U_{g}\ket{\psi_{k}}^{S^{1}...S^{n-1}}=\ket{\psi_{k}}^{S^{1}...S^{n-1}}.\end{split}$
(22)

Therefore, the $\ket{\psi_{j}}^{S^{1}...S^{n-1}}$ will all be in the symmetric
subspace, and $\Tr_{S^{n}}[\outerproduct{\psi}{\psi}]/d_{S^{n}}$ will be a
linear combination of operators on the symmetric subspace. This holds
analogously for the antisymmetric subspace, where
$U_{g}\ket{\psi}=\text{sgn}(g)\ket{\psi}$. The same result holds if we
‘project out’ any number of subspaces. Taking the partial trace of a matrix in
the (anti)symmetric subspace will therefore result in a matrix that is still
in the (anti)symmetric subspace, where we define the (anti)symmetric subspace
for matrices as the space of matrices that act on the (anti)symmetric subspace
for states. We will equivalently say that these matrices belong to the
(anti)symmetric representation.

Combining these two arguments, we see that for an $n$-partite process $W$
living in the symmetric (antisymmetric) representation of $S_{n}$,
$\overset{\sim}{W}{}^{S^{i}S^{j}}$ must be a valid bipartite process and live
in the symmetric (antisymmetric) representation for all $i,j=1,..,n,i\neq j$.
But, we saw that there are no valid symmetric or antisymmetric bipartite
processes, so this is a contradiction. This tells us that there are no
symmetric or antisymmetric $n$-partite qubit processes. This proof generalises
to any local Hilbert space dimension once one has proved the base case.

##  7 Conclusion

In this paper, we have used the process matrix formalism to show that it is
possible to describe quantum causal order with background independence built
in, under the assumption of a discretised spacetime. We have also seen that
some properties of background independent processes have counterparts in
general-relativistic background independence, e.g. the ‘washing out’ of
spacetime and the need to construct a material reference frame to recover a
definition of spacetime points.

Our results show that background independence is consistent with the
principles of the process matrix formalism, including, with some
reinterpretation, locality—which must be defined relative to a reference
frame. This follows from our discussion on local vs. background independent
measurements.

We also investigated the general symmetry constraints imposed on processes by
permutation invariance, and discovered that the constraint is different to the
typical superselection rule: the standard interpretation is simply that
physical systems must have a well-defined ‘charge’, but for permutation-
invariance not all charges correspond to physically realisable processes.
Instead, valid processes can be block-diagonal combinations of matrices that
are not themselves valid processes. This implies that background independence
in quantum mechanics cannot be interpreted analogously to other known
symmetries of nature, and that a new interpretation may be necessary. Whether
or not this ‘charge’ can be taken seriously as a physical quantity is, for the
moment, an open question.

Finally, our attention has focused on permutations—namely relabellings of
laboratories. These can be understood as ‘classical’ coordinate
transformations, which do not change, for example, whether a particle is
localised at a point or in a superposition. It has been proposed that
combining quantum mechanics and general relativity requires considering more
general, ‘quantum’ coordinate transformations [52, 53]. It is an interesting
open question whether it is possible to extend our treatment to include such
‘quantum relabellings’.

This work was partially supported through an Australian Research Council (ARC)
Discovery Early Career Researcher Award (DE170100712) and by the ARC Centre of
Excellence for Engineered Quantum Systems (CE17010000). The University of
Queensland (UQ) acknowledges the Traditional Owners and their custodianship of
the lands on which UQ operates.

## References

  * [1] Jeremy Butterfield and Christopher J. Isham.  “Space-time and the philosophical challenge of quantum gravity”.  Pages 33–89.  Cambridge University Press. (1999).  arXiv:gr-qc/9903072.
  * [2] Lucien Hardy.  “Probability theories with dynamic causal structure: A New framework for quantum gravity” (2005).  arXiv:gr-qc/0509120.
  * [3] Ognyan Oreshkov, Fabio Costa, and Časlav Brukner.  “Quantum correlations with no causal order”.  Nat. Commun.3 (2012).
  * [4] C. Rovelli.  “What is observable in classical and quantum gravity?”.  Class. Quantum Grav.8 (1991).
  * [5] John D. Norton.  “The Hole Argument”.  In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy.  Metaphysics Research Lab, Stanford University (2019).  Summer 2019 edition.
  * [6] Abhay Ashtekar and Jerzy Lewandowski.  “Background independent quantum gravity: a status report”.  Classical and Quantum Gravity 21, R53–R152 (2004).
  * [7] Lee Smolin.  “The Case for Background Independence”.  In The Structural Foundations of Quantum Gravity.  Oxford University Press (2006).  arXiv:hep-th/0507235.
  * [8] L. Procopio, A. Moqnaki, M. Araujo, F. Costa, I. Calafell, E. Dowd, D. Hamel, L. Rozema, C. Brukner, and P. Walther.  “Experimental superposition of orders of quantum gates”.  Nat. Comms.6 (2015).
  * [9] Giulia Rubino, Lee A Rozema, Adrien Feix, Mateus Araújo, Jonas M Zeuner, Lorenzo M Procopio, Časlav Brukner, and Philip Walther.  “Experimental verification of an indefinite causal order”.  Science Advances 3, e1602589 (2017).
  * [10] Giulia Rubino, Lee A. Rozema, Francesco Massa, Mateus Araújo, Magdalena Zych, Časlav Brukner, and Philip Walther.  “Experimental entanglement of temporal order”.  Quantum 6, 621 (2022).
  * [11] K. Goswami, C. Giarmatzi, M. Kewming, F. Costa, C. Branciard, J. Romero, and A. G. White.  “Indefinite Causal Order in a Quantum Switch”.  Phys. Rev. Lett.121 (2018).
  * [12] K. Goswami, Y. Cao, G. A. Paz-Silva, J. Romero, and A. G. White.  “Increasing communication capacity via superposition of order”.  Phys. Rev. Research 2, 033292 (2020).
  * [13] Kejin Wei, Nora Tischler, Si-Ran Zhao, Yu-Huai Li, Juan Miguel Arrazola, Yang Liu, Weijun Zhang, Hao Li, Lixing You, Zhen Wang, et al.  “Experimental quantum switching for exponentially superior quantum communication complexity”.  Phys. Rev. Lett. 122, 120504 (2019).
  * [14] Márcio M. Taddei, Jaime Cariñe, Daniel Martínez, Tania García, Nayda Guerrero, Alastair A. Abbott, Mateus Araújo, Cyril Branciard, Esteban S. Gómez, Stephen P. Walborn, Leandro Aolita, and Gustavo Lima.  “Computational advantage from the quantum superposition of multiple temporal orders of photonic gates”.  PRX Quantum 2, 010320 (2021).
  * [15] Yu Guo, Xiao-Min Hu, Zhi-Bo Hou, Huan Cao, Jin-Ming Cui, Bi-Heng Liu, Yun-Feng Huang, Chuan-Feng Li, Guang-Can Guo, and Giulio Chiribella.  “Experimental transmission of quantum information using a superposition of causal orders”.  Phys. Rev. Lett. 124, 030502 (2020).
  * [16] Giulia Rubino, Lee A. Rozema, Daniel Ebler, Hlér Kristjánsson, Sina Salek, Philippe Allard Guérin, Alastair A. Abbott, Cyril Branciard, Časlav Brukner, Giulio Chiribella, and Philip Walther.  “Experimental quantum communication enhancement by superposing trajectories”.  Phys. Rev. Research 3, 013093 (2021).
  * [17] Philippe Allard Guérin and Časlav Brukner.  “Observer-dependent locality of quantum events”.  New Journal of Physics 20, 103031 (2018).
  * [18] Ognyan Oreshkov.  “Time-delocalized quantum subsystems and operations: on the existence of processes with indefinite causal structure in quantum mechanics”.  Quantum 3, 206 (2019).
  * [19] G. Chiribella, G. M. D’Ariano, P. Perinotti, and B. Valiron.  “Quantum computations without definite causal structure”.  Phys. Rev. A 88, 022318 (2013).
  * [20] G. Chiribella.  “Perfect discrimination of no-signalling channels via quantum superposition of causal structures”.  Phys. Rev. A 86, 040301(R) (2012).
  * [21] T. Colnaghi, G. D’Ariano, S. Facchini, and P. Perinotti.  “Quantum computation with programmable connections between gates”.  Phys. Lett. A376 (2012).
  * [22] M. Araújo, F. Costa, and Č. Brukner.  “Computational Advantage from Quantum-Controlled Ordering of Gates”.  Phys. Rev. Lett. 113, 250402 (2014).
  * [23] Adrien Feix, Mateus Araújo, and Časlav Brukner.  “Quantum superposition of the order of parties as a communication resource”.  Phys. Rev. A 92, 052326 (2015).
  * [24] Philippe Allard Guérin, Adrien Feix, Mateus Araújo, and Časlav Brukner.  “Exponential communication complexity advantage from quantum superposition of the direction of communication”.  Phys. Rev. Lett. 117, 100502 (2016).
  * [25] Daniel Ebler, Sina Salek, and Giulio Chiribella.  “Enhanced communication with the assistance of indefinite causal order”.  Phys. Rev. Lett. 120, 120502 (2018).
  * [26] Sina Salek, Daniel Ebler, and Giulio Chiribella.  “Quantum communication in a superposition of causal orders” (2018).  arXiv:1809.06655v2.
  * [27] Manish K. Gupta and Ujjwal Sen.  “Transmitting quantum information by superposing causal order of mutually unbiased measurements” (2019).  arXiv:1909.13125v1.
  * [28] S. Shrapnel, F. Costa, and G. Milburn.  “Updating the Born Rule”.  New J. Phys.20 (2018).
  * [29] M. Araujo, C. Branciard, F. Costa, A. Feix, C. Giarmatzi, and C. Brukner.  “Witnessing causal noseparability”.  New. J. Phys.17 (2015).
  * [30] A. Jamiołkowski.  “Linear transformations which preserve trace and positive semidefiniteness of operators”.  Rep. Math. Phys 3, 275–278 (1972).
  * [31] Man-Duen Choi.  “Completely positive linear maps on complex matrices”.  Linear Algebra Appl. 10, 285–290 (1975).
  * [32] T. Heinosaari and M. Ziman.  “The mathematical language of quanum theory”.  Cambridge University Press. (2012).
  * [33] A. Abbott, G. Giarmatzi, F. Costa, and C. Branciard.  “Multipartite causal correlations: polytopes and inequalities”.  Phys. Rev. A94 (2016).
  * [34] Alastair Abbott, Julian Wechs, Fabio Costa, and Cyril Branciard.  “Genuinely multipartite noncausality”.  Quantum 1, 39 (2017).
  * [35] J. Wechs, A. Abbott, and C. Branciard.  “On the definition and characterisation of multipartite causal (non)separability”.  New J. Phys.21 (2019).
  * [36] Jan Myrheim.  “Statistical geometry”.  Technical Report CERN-TH-2538.  CERN (1978).  url: http://cds.cern.ch/record/293594.
  * [37] Luca Bombelli, Joohan Lee, David Meyer, and Rafael D. Sorkin.  “Space-time as a causal set”.  Phys. Rev. Lett. 59, 521–524 (1987).
  * [38] G 't Hooft.  “Quantization of point particles in (2 + 1)-dimensional gravity and spacetime discreteness”.  Class. and Quantum Grav. 13, 1023–1039 (1996).
  * [39] Renate Loll.  “Discrete approaches to quantum gravity in four dimensions”.  Living Reviews in Relativity 1, 13 (1998).
  * [40] Fay Dowker.  “Causal sets as discrete spacetime”.  Contemp. Phys. 47, 1–9 (2006).
  * [41] Pablo Arrighi, Marios Christodoulou, and Amélia Durbec.  “Quantum superpositions of graphs” (2020).  arXiv:2010.13579.
  * [42] S. Bartlett, T. Rudolph, and R. Spekkens.  “Reference frames, superselection rules, and quantum information”.  Rev. Mod. Phys.79 (2007).
  * [43] Lachlan Parker.  “Permutation invariance in quantum processes.”.  Honours thesis.  The University of Queensland.  (2020).
  * [44] C. Branciard, M. Araujo, A. Feix, F. Costa, and C. Brukner.  “The simplest causal inequalities and their violation”.  New J. Phys.27 (2015).
  * [45] Don N. Page and William K. Wootters.  “Evolution without evolution: Dynamics described by stationary observables”.  Phys. Rev. D 27, 2885–2892 (1983).
  * [46] Carlo Rovelli.  “Relational quantum mechanics”.  International Journal of Theoretical Physics 35, 1637–1678 (1996).
  * [47] David Poulin.  “Toy model for a relational formulation of quantum theory”.  International Journal of Theoretical Physics 45, 1189–1215 (2006).
  * [48] Vittorio Giovannetti, Seth Lloyd, and Lorenzo Maccone.  “Quantum time”.  Phys. Rev. D 92, 045033 (2015).
  * [49] Takayuki Miyadera, Leon Loveridge, and Paul Busch.  “Approximating relational observables by absolute quantities: a quantum accuracy-size trade-off”.  Journal of Physics A: Mathematical and Theoretical 49, 185301 (2016).
  * [50] Flaminia Giacomini, Esteban Castro-Ruiz, and Časlav Brukner.  “Quantum mechanics and the covariance of physical laws in quantum reference frames”.  Nature Communications 10, 494 (2019).
  * [51] Alexander R. H. Smith and Mehdi Ahmadi.  “Quantizing time: Interacting clocks and systems”.  Quantum 3, 160 (2019).
  * [52] Lucien Hardy.  “Operational general relativity: Possibilistic, probabilistic, and quantum” (2016).  arXiv:1608.06940.
  * [53] Magdalena Zych, Fabio Costa, and Timothy C. Ralph.  “Relativity of quantum superpositions” (2018).  arXiv:1809.04999.

##  8 Appendix

Here, we prove some results stated in the main text.

###  8.1 Some facts about the $\mathcal{R}$ map.

It’s useful to first prove two properties of the $\mathcal{R}$ map, namely
that $\mathcal{R}(A)+\mathcal{R}(B)=\mathcal{R}(A+B)$ and
$\mathcal{R}(A)\mathcal{R}(B)=\mathcal{R}(AB)$. For the first, we see that

######  Lemma 1.

$\mathcal{R}(A)+\mathcal{R}(B)=\mathcal{R}(A+B)$, for $A$ and $B$ linear
operators.

###### Proof.

$\displaystyle\begin{split}\mathcal{R}(A)+\mathcal{R}(B)&=\sum_{g\in
S_{n}}U^{SR}_{g}A^{S}[01...(n-1)]^{R}{U^{\dagger}_{g}}^{SR}+\sum_{g\in
S_{n}}U^{SR}_{g}B^{S}[01...(n-1)]^{R}{U^{\dagger}_{g}}^{SR}\\\ &=\sum_{g\in
S_{n}}U^{SR}_{g}\left(A^{S}[01...(n-1)]^{R}+B^{S}[01...(n-1)]^{R}\right){U^{\dagger}_{g}}^{SR}\\\
&=\sum_{g\in
S_{n}}U^{SR}_{g}\left(A^{S}+B^{S}\right)[01...(n-1)]^{R_{i}}{U^{\dagger}_{g}}^{SR}=\mathcal{R}(A+B).\end{split}$
(23)

∎

The second takes slightly more work, but we obtain

######  Lemma 2.

$\mathcal{R}(A)\mathcal{R}(B)=\mathcal{R}(AB)$, for $A$ and $B$ linear
operators.

###### Proof.

$\displaystyle\begin{split}\mathcal{R}(A)\mathcal{R}(B)&=\sum_{g\in
S_{n}}\sum_{h\in
S_{n}}(U^{SR}_{g}A^{S}[01..(n-1)]^{R}{U^{\dagger}_{g}}^{SR})(U^{SR}_{h}B^{S}[01...(n-1)]^{R}{U^{\dagger}_{h}}^{SR})\\\
&=\sum_{g\in S_{n}}\sum_{h\in
S_{n}}(U_{g}^{S}A^{S}{U^{\dagger}_{g}}^{S}U_{h}^{S}B^{S}{U^{\dagger}_{h}}^{S})(U_{g}^{R}[01..(n-1)]^{R}{U^{\dagger}_{g}}^{R}U_{h}^{R}[01...(n-1)]^{R}{U^{\dagger}_{h}}^{R})\\\
&=\sum_{g\in S_{n}}\sum_{h\in
S_{n}}(U^{S}_{g}A{U^{\dagger}_{g}}^{S}U^{S}_{h}B{U^{\dagger}_{h}}^{S})(\delta(g^{-1}h)U^{R}_{g}[01..(n-1)]{U^{\dagger}_{g}}^{R})\\\
&=\sum_{g\in
S_{n}}U_{g}^{SR}(AB)^{S}[01...(n-1)]^{R}{U_{g}^{\dagger}}^{SR}=\mathcal{R}(AB),\end{split}$
(24)

using
$U_{g}[01...(n-1)]U^{\dagger}_{g}U_{h}[01...(n-1)]U^{\dagger}_{h}=[g(0)g(1)...g(n-1)][h(0)h(1)...h(n-1)]=\delta_{g(0)h(0)}...\delta_{g(n-1)h(n-1)}[g(0)...g(n-1)]=\delta(g^{-1}h)U_{g}[01...(n-1)]U^{\dagger}_{g}$,
with $\delta_{ij}$ being the kronecker delta and $\delta(g^{-1}h)$ being $1$
if $g^{-1}h$ is identity and $0$ otherwise (the delta function on $S_{n}$). ∎

###  8.2 Permutation-invariant processes.

For $W^{\text{inv}}$ to be a valid process, it must be a positive semi-
definite matrix that has trace 1 when multiplied with any tensor product of
CPTP maps. The positive semidefinite requirement is satisfied by Eq. (10)
because both the tensor product and the sum of positive semidefinite matrices
is positive semidefinite. We can also demonstrate that $W^{\text{inv}}$ gives
normalised probabilities.

First, note that a permutation of a valid process is again a valid process.
This is because $\Tr[UWU^{\dagger}M]=\Tr[WU^{\dagger}MU]$ (where $M$ denotes
the tensor product of local CP maps). Therefore, evaluating the Born rule for
a permuted process is the same as evaluating it for the original process and
permuted maps. If $\Tr[WM]=1$ for arbitrary CPTP maps, then the same is true
for permuted CPTP maps, meaning that the normalisation condition is preserved
under permutations. Furthermore, convex combinations preserve process
normalisation, so all we need is to prove that adding the reference frame
systems to a valid process we get a valid process.

Let us then consider the operator
$W^{S}[01...(n-1)]^{R_{I}}\mathbbm{1}^{R_{O}}$, where $W^{S}$ is a valid
process. For arbitrary CPTP maps acting on local systems and reference frames,
$M^{S^{1}R^{1}}_{1}\cdots M^{S^{n}R^{n}}_{n}$, we have

$\Tr\left[W^{S}[01...(n-1)]^{R_{I}}\mathbbm{1}^{R_{O}}\cdot\left(M^{S^{1}R^{1}}_{1}\cdots
M^{S^{n}R^{n}}_{n}\right)\right]=\Tr_{S}\left[W^{S}\cdot\left(\bar{M}_{1}^{S^{1}}\cdots\bar{M}_{n}^{S^{n}}\right)\right],$
(25)

where

$\bar{M}^{S^{j}}_{j}:=\Tr_{R^{j}}\left[\left([j-1]^{R^{j}_{I}}\mathbbm{1}^{R^{j}_{O}}\right)\cdot
M^{S^{j}R^{j}}_{j}\right],\qquad j=1,\dots n.$ (26)

We see that each $\bar{M}^{S^{j}}_{j}$ is CPTP, which, in the Choi
representation, means that
$\Tr_{S^{j}_{O}}\bar{M}^{S^{j}}_{j}=\mathbbm{1}^{S_{I}^{j}}$. Indeed,

$\Tr_{S^{j}_{O}}\bar{M}^{S^{j}}_{j}=\Tr_{R^{j}_{I}}\left[[j-1]^{R^{j}_{I}}\cdot\left(\Tr_{S^{j}_{O}R^{j}_{O}}M^{S^{j}R^{j}}_{j}\right)\right]=\Tr_{R^{j}_{I}}\left[[j-1]^{R^{j}_{I}}\mathbbm{1}^{S^{j}_{I}}\right]=\mathbbm{1}^{S^{j}_{I}},$
(27)

where we have used the CPTP property of the extended maps,
$\Tr_{S^{j}_{O}R^{j}_{O}}M^{S^{j}R^{j}}_{j}=\mathbbm{1}^{S^{j}_{I}R^{j}_{I}}$,
and the normalisation of the reference frame state, $\Tr\,[j-1]=1$. As the
$\bar{M}^{S^{j}}_{j}$ are CPTP, and $W^{S}$ is a valid process, the rhs of Eq.
(25) is equal to 1, which means the lhs is 1 too. This proves that
$W^{S}[01...(n-1)]^{R_{I}}\mathbbm{1}^{R_{O}}$ satisfies the process
normalisation condition.

###  8.3 Permutation-invariant instruments.

Here we show that the $M^{\text{inv}}_{i_{1}...i_{n}}$, as defined in Eq.
(11), is a valid instrument. To do this, we show that each element is a CP
map, and that the trace over the combined system-reference frame output is
$\Tr_{S_{O}R_{O}}[M^{\text{inv}}]=\mathbbm{1}^{S_{I}R_{I}}$, so that the maps
sum to a CPTP map:

$\displaystyle\begin{split}\Tr_{S_{O}R_{O}}[\sum_{i_{1}...i_{n}}M^{\text{inv}}_{i_{1}...i_{n}}]&=\Tr_{S_{O}R_{O}}[\sum_{i_{1}...i_{n}}\mathcal{R}(M_{i_{1}...i_{n}}^{S})+\frac{1}{Nd_{O}}\left(\mathbbm{1}^{SR}-\mathcal{R}(\mathbbm{1}^{S}\right)]\\\
&=\Tr_{S_{O}R_{O}}[\mathcal{R}(\sum_{i_{1}...i_{n}}M_{i_{1}...i_{n}}^{S})]+\frac{1}{d_{O}}\Tr_{S_{O}R_{O}}[\mathbbm{1}^{SR}]-\frac{1}{d_{O}}\Tr_{S_{O}R_{O}}[\mathcal{R}(\mathbbm{1}^{S})]\\\
&=\Tr_{S_{O}R_{O}}[\mathcal{R}(M^{S})]+\mathbbm{1}^{S_{I}R_{I}}-\Tr_{S_{O}R_{O}}[\sum_{g}U_{g}^{SR}\mathbbm{1}^{S}[01...(n-1)]^{R_{I}}\mathbbm{1}^{R_{O}}{U_{g}^{\dagger}}^{SR}]\\\
&=\Tr_{S_{O}R_{O}}[\sum_{g}U^{SR}_{g}M^{S}[01...(n-1)]^{R_{I}}\mathbbm{1}^{R_{O}}{U^{\dagger}_{g}}^{SR}]+\mathbbm{1}^{S_{I}R_{I}}\\\
&\qquad\qquad-\Tr_{S_{O}R_{O}}[\sum_{g}U^{SR}_{g}\mathbbm{1}^{S}[01...(n-1)]^{R_{I}}\mathbbm{1}^{R_{O}}{U^{\dagger}_{g}}^{SR}]\\\
&=\Tr_{S_{O}R_{O}}[\sum_{g}{U^{\dagger}_{g}}^{SR}U^{SR}_{g}M^{S}\otimes[01...(n-1)]^{R_{I}}\mathbbm{1}^{R_{O}}]+\mathbbm{1}^{S_{I}R_{I}}\\\
&\qquad\qquad-\Tr_{S_{O}R_{O}}[\sum_{g}{U^{\dagger}_{g}}^{SR}U^{SR}_{g}\mathbbm{1}^{S}\otimes[01...(n-1)]^{R_{I}}\mathbbm{1}^{R_{O}}]\\\
&=\sum_{g}\Tr_{S_{O}}[M^{S}]\Tr_{R_{O}}[[01...(n-1)]^{R_{I}}\mathbbm{1}^{R_{O}}]+\mathbbm{1}^{S_{I}R_{I}}\\\
&\qquad\qquad-\sum_{g}\Tr_{S_{O}}[\mathbbm{1}^{S}]\Tr_{R_{O}}[[01...(n-1)]^{R_{I}}\mathbbm{1}^{R_{O}}]\\\
&=(n!d_{S_{O}}\mathbbm{1}^{S_{I}}\Tr_{R_{O}}[[01...(n-1)]^{R_{I}}\mathbbm{1}^{R_{O}}]-n!d_{S_{O}}\mathbbm{1}^{S_{I}}\Tr_{R_{O}}[[R]^{R_{I}}\mathbbm{1}^{R_{O}}])+\mathbbm{1}^{S_{I}R_{I}}\\\
&=\mathbbm{1}^{S_{I}R_{I}}.\end{split}$ (28)

We now show that the $M_{i_{1}...i_{n}}^{\mathrm{inv}}$ are all positive
semidefinite. Observe that

$M_{i_{1}...i_{n}}^{\mathrm{inv}}=\mathcal{R}(M^{S^{1}}_{i_{1}}...M^{S^{n}}_{i_{n}})+\frac{1}{Nd_{O}}\left(\mathbbm{1}^{SR}-\mathcal{R}\left(\mathbbm{1}^{S}\right)\right)$
(29)

is a sum of two terms. It suffices to show that both of the two terms are
positive semidefinite. For the first, we see that

$\displaystyle\mathcal{R}(M^{S^{1}}_{i_{1}}...M^{S^{n}}_{i_{n}})=\sum
U_{g}(M^{S^{1}}_{i_{1}}...M^{S^{n}}_{i_{n}})U_{g}^{\dagger}[R],$ (30)

which, as $U_{g}(M^{S^{1}}_{i_{1}}...M^{S^{n}}_{i_{n}})U_{g}^{\dagger}$ and
$[R]$ are both positive semidefinite, is just a sum of positive semidefinite
operators. Therefore, it is positive semidefinite. For the second term, note
that $\mathbbm{1}^{SR}$ is simply the sum of all projectors, while
$\mathcal{R}(\mathbbm{1}^{S})=\sum\mathbbm{1}^{R_{I}}[R]^{R_{O}}$ is a sum
containing only projectors, so that

$\displaystyle\frac{1}{Nd_{O}}\left(\mathbbm{1}^{SR}-\mathcal{R}\left(\mathbbm{1}^{S}\right)\right)\geq
0$ (31)

because $\mathbbm{1}^{SR}$ contains all terms that appear in
$\mathcal{R}(\mathbbm{1}^{S})$, but the reverse is not true. Since both terms
are diagonal, we can see that this results in all eigenvalues being positive
or zero.


# Multi-contrast MRI Super-Resolution via Multi-stage Integration Network

###### Abstract

Super-Resolution (SR) plays a crucial role in improving image quality of
magnetic resonance imaging (MRI). By the nature of MRI, multi-contrast images
are available to provide a clear display of soft tissues. However, current
super-resolution methods only employ a single contrast, or take a simple
multi-contrast fusion mechanism for SR, ignoring the rich relations among
different contrast that are valuable for improving SR. In this work, we
propose a multi-stage integration network (i.e., MINet) for multi-contrast MRI
SR, which explicitly models the dependencies of multi-contrast images at
different stages to guide image SR. In particular, our MINet first learns a
hierarchical feature representation from multiple convolutional stages for
each contrast image. Subsequently, we introduce a multi-stage integration
module to mine the comprehensive relations between the representations of
multi-contrast images. For each representation, the module matches it with all
other features and summarizes these features according to the similarities to
obtain an enriched representation. Extensive experiments on fastMRI and real-
world clinical datasets demonstrate that 1) our MINet outperforms state-of-
the-art multi-contrast SR methods in terms of various metrics and 2) our
multi-stage integration module is able to excavate complex interactions among
multi-contrast features at different stages, leading to the improvement of
target image quality.

###### Keywords:

Magnetic resonance imaging Super-resolution Multi-contrast

##  1 Introduction

Magnetic resonance imaging (MRI) is a popular technique in medical imaging.
Compared with other modalities such as computed tomography (CT) and nuclear
imaging, MRI can provide clear information about tissue structure and function
without inducing ionizing radiation. However, due to the complex data
acquisition process, it is difficult to obtain High-Resolution (HR) MRI
images.

Since the Super-Resolution (SR) as a post- processing technology can improve
the image quality without changing the MRI hardware, it has been successfully
used in MRI. Bicubic and bi-spline interpolation are two basic SR
interpolation methods, however, they inevitably lead to blurred edges and
blocking artifacts. Relying on the inherent redundancy in the transform domain
iterative deblurring algorithms [hardie2007fast, tourbier2015efficient], low
rank [shi2015lrtv], dictionary learning method [bhatia2014super], have made
significant progress in MRI SR. Recently, deep learning offers higher
resolution and is widely used in MRI SR [oktay2016multi, pham2017brain,
mcdonagh2017context, chen2018brain, chaudhari2018super]. For example, Akshay
et al. applied the 3D residual network to generate thin-slice knee MR images
[chaudhari2018super]. Chen et al. used a densely connected SR Network to
restore HR details from one single Low-Resolution (LR) input image
[chen2018brain]. Lyu et al. used the Generative Adversarial Network (GAN)
framework for CT denoising and transfer them for MRI SR [lyu2018super].
However, the above methods focus on mono-contrast acquisition to restore HR
images, leading to the waste of multi-contrast complementary information.

MRI not only provides clear tissue structure and functional information, but
also generates multiple contrast images in different settings, e.g., T1 and T2
weighted images (T1WI and T2WI), Proton Density and Fat-Suppressed Proton
Density weight images (PDWI and FS-PDWI). However, different contrast have the
same anatomical structure and can provide complementary information. For
example, T1WI can be used to describe morphological and structural
information, while T2WI can be used to describe edema and inflammation. In
clinical routines, T1WI have shorter repetition time (TR) and echo time (TE)
than T2WI, and are easier to acquire than others. Therefore, we can add more
information to a single LR image to obtain better reconstruction results, such
as using relevant information in HR of T1WI to assist the generation of SR
images of T2WI [xiang2018deep]. Similarity, PDWI is of great significance for
the observation of structures such as articular cartilage, and has a high SNR
for tissues with little difference in PD, while FS-PDWI can inhibit fat signal
and highlight the contrast of tissue structures such as cartilage ligament
[chen2015accuracy]. However, FS-PDWI usually takes longer time than PD in
scanning process. Recently, several works on multi-contrast SR has been
proposed. For example, Zheng et al. used the local weight similarity and
relation model of gradient value between different contrast images to restore
a SR image from its counterpart LR image [zheng2017multi, zheng2018multi].
Zeng et al. proposed a deep convolutional neural network to simultaneously
obtaining single- and multi-contrast SR images [zeng2018simultaneous]. Lyu et
al. proposed a progressive network which integrate composite loss function for
multi-contrast SR [lyu2020multi]. Though significant progress have been made,
existing methods do not explore the interaction between different modalities
in various stages, which are especially critical for T2WI restoration. If we
treat the feature mapping of different contrast at each stage as a response to
a specific class, the responses from the different stages should be
interrelated.

To explore the correlation among hierarchical stages between different
contrast, we further explore the interactive relationship of multi-contrast
features at different stages. Specially, we propose a Multi-stage Integration
Network, named MINet, to interact and weight the multi-contrast features of
each stage. In our MINet, the T1WI features of each stage are fused with the
T2WI features of each stage, to obtain a comprehensive feature and guide the
learning of T2WI features.

Our contributions are as follows: 1) we design a novel multi-contrast SR
network to guide the SR restoration of the target contrast through the HR
auxiliary contrast features; 2) we explore the response of multi-contrast
fusion at different stages, obtain the dependency relationship between them,
and improve the representation ability of fusion features; and 3) we perform
extensive experiments on fastMRI and clinical datasets with our method which
obtain superiority results compared with state-of-the-art methods.

##  2 Methodology

Given a LR image $\mathbf{y}\in\mathbb{R}^{N\times N}$, we aim to learn a
neural network to obtain a HR image $\mathbf{x}\in\mathbb{R}^{M\times M}$.
While mainstream efforts address the task by directly restoring $\mathbf{x}$
from $\mathbf{y}$, we propose to solve this from a novel perspective by
introducing an extra contrast HR image with the same structural information to
get more effective results. For example, we employ HR T1WI as auxiliaries to
improve the resolution of LR T2WI, and the network $\mathcal{G}_{\theta}$ can
be optimized by minimizing the following loss function:

$L=\frac{1}{N}\sum_{n=1}^{N}\mathcal{L}\left(\mathcal{G}_{\theta}\left(\mathbf{y}_{T_{2}}^{n},\mathbf{x}_{T_{1}}^{n}\right),\mathbf{x}_{T_{2}}^{n}\right).$
(1)

Here, each training sample is a triplet of
$\\{\mathbf{y}_{T_{2}}^{n},\mathbf{x}_{T_{1}}^{n},\mathbf{x}_{T_{2}}^{n}\\}$,
and $N$ is the number of all samples.

Figure 1: Network architecture of the proposed two branch framework.

As shown in Fig. 1, our proposed MINet, consists of two branches: auxiliary
branch and target SR branch. We build a fusion between each stage of the two
branches so that the anatomical structure features of the auxiliary branch can
be fully propagated to the target SR branch. More importantly, we learn the
dependence of multi-contrast features in different stages, and allocate proper
attention weights to them, so as to automatically improve the expression
ability of the extracted features.

####  2.0.1 Hierarchical Feature Extraction.

Firstly, to obtain the anatomical structure of T1WI, we feed the HR T1WI
$\mathbf{x}_{T_{1}}$ to a convolutional layer

$\mathbf{F}_{T_{1}}^{0}=\texttt{Conv}\left({\mathbf{x}}_{T_{1}}\right),$ (2)

where $\mathbf{F}_{T_{1}}^{0}$ is the shallow structural feature of HR T1WI.
Accordingly, we extract the shallow feature $\mathbf{F}_{T_{2}}^{\prime}$ by
inputting the LR T2WI ${\mathbf{y}}_{T_{2}}$ to a convolutional layer. Then,
we use the sub-pixel convolution $\text{U}^{\uparrow}$ [shi2016real] as the
upsampling module to convert the scale sampling of the given magnification

$\mathbf{F}_{T_{2}}^{{}^{\prime}}=\texttt{Conv}\left({\mathbf{y}}_{T_{2}}\right),~{}~{}~{}\mathbf{F}_{T_{2}}^{0}=\text{U}^{\uparrow}(\mathbf{F}_{T_{2}}^{\prime}),$
(3)

where $\mathbf{F}_{T_{2}}^{0}$ is the shallow feature of SR T2WI. Then, we use
several residual groups to extract the intermediate features of T1WI

$\mathbf{F}_{T_{1}}^{i}=\mathcal{F}_{\text{RG}}^{i}\left(\mathbf{F}_{T_{1}}^{i-1}\right),~{}~{}i=1,2,..,Z,$
(4)

where $\mathcal{F}_{\text{RG}}^{i}$ denotes the $i^{th}$ residual group, and
$Z$ is the number of the residual groups. More importantly, to propagate the
structural information of ${\mathbf{x}}_{T_{1}}$, we concatenate the $i^{th}$
intermediate features of both T1WI and T2WI
$\mathbf{F}_{T_{1}}^{i-1},\mathbf{F}_{T_{2}}^{i-1}$ as the input of the
$\\!i\\!+\\!1\\!^{th}$ residual group

$\mathbf{F}_{T_{2}}^{i}=\mathcal{F}_{\text{RG}}^{i}\left(\texttt{Concat}\left(\mathbf{F}_{T_{1}}^{i-1},\mathbf{F}_{T_{2}}^{i-1}\right)\right),~{}~{}i=1,2,..,Z.$
(5)

Accordingly, $\mathbf{F}_{T_{1}}^{Z}$ and $\mathbf{F}_{T_{2}}^{Z}$ are the
final intermediate output features of both T1WI and T2WI.

####  2.0.2 Multi-contrast Feature Enhancement.

We further enhance and fuse the intermediate features, especially the
anatomical structure information in ${\mathbf{x}}_{T_{1}}$, which helps to
guide the restoration of ${\mathbf{y}}_{T_{2}}$. Firstly, we design a multi-
stage integration module $\mathcal{F}_{\text{M}}$ (see §2.1) to aggregate all
previous stage outputs of both T1WI and T2WI. It can be represented as

$\mathbf{F}_{M}=\mathcal{F}_{\text{M}}\left(\texttt{Concat}\left(\left\\{\mathbf{F}_{T_{1}}^{1},\mathbf{F}_{T_{2}}^{1}\right\\},\left\\{\mathbf{F}_{T_{1}}^{2},\mathbf{F}_{T_{2}}^{2}\right\\},...\left\\{\mathbf{F}_{T_{1}}^{Z},\mathbf{F}_{T_{2}}^{Z}\right\\}\right)\right)$
(6)

where $\mathbf{F}_{M}$ represents the aggregate output multi-contrast feature.
As a results, this module can enhance the correlation between each stage and
obtain the dependency relationship between them, to improve the expression
ability of aggregate features. Then, two channel-spatial attention modules to
adaptively adjust the features of the two contrasts, which can be expressed as

$\mathbf{F}_{CS,T_{1}}=\mathcal{F}_{\text{CS}}\left(\mathbf{F}_{T_{1}}^{Z}\right),\mathbf{F}_{CS,T_{2}}=\mathcal{F}_{\text{CS}}\left(\mathbf{F}_{T_{2}}^{Z}\right),$
(7)

where $\mathcal{F}_{\text{CS}}$ represents the channel-spatial attention
module (see §2.2), $\mathbf{F}_{CS,T_{1}}$ and $\mathbf{F}_{CS,T_{2}}$ denotes
the filtered multi-contrast features after channel-spatial attention.

####  2.0.3 Image Reconstruction.

Since our framework is a pre-upsampling SR, we use two convolutional layers
$\mathcal{F}_{Rec}$ to obtain the final restored image
$\hat{\bm{\boldsymbol{x}}}$, which can be written as

$\displaystyle\hat{\bm{\mathbf{x}}}_{T_{2}}$
$\displaystyle=\mathcal{F}_{\text{Rec}}\left(\mathbf{F}_{T_{1}}^{0}\oplus\mathbf{F}_{CS,T_{2}}\oplus\mathbf{F}_{M}\right),$
(8) $\displaystyle\hat{\bm{\mathbf{x}}}_{T_{1}}$
$\displaystyle=\mathcal{F}_{\text{Rec}}\left(\mathbf{F}_{CS,T_{1}}\right).$

####  2.0.4 Loss Function.

In our framework, we simply use $L_{1}$ to minimize the two terms of T1WI and
T2WI reconstruction, which can be represented as

$L=\frac{1}{N}\sum_{n=1}^{N}\alpha\left\|\hat{\bm{\mathbf{x}}}_{T_{2}}^{n}-{\bm{\mathbf{x}}}_{T_{2}}^{n}\right\|_{1}+\beta\left\|\hat{\bm{\mathbf{x}}}_{T_{1}}^{n}-{\bm{\mathbf{x}}}_{T_{1}}^{n}\right\|_{1},$
(9)

where $\alpha$ and $\beta$ is the weight controlling the trade-off between
T1WI and T2WI reconstruction. Note that we do not use other sophisticated loss
functions, because our framework can already achieve competitive results (see
§3).

###  2.1 Multi-stage Integration Module

Figure 2: Architecture of the proposed multi-modal cross stage attention
module.

Although we have aggregate the T1WI and T2WI features in the intermediate
stages for T2WI SR, the correlation among different stages has been missed,
especially for MRI in different contrasts. In multi-contrast SR, different
contrast can be considered as different classes, and features from different
stages can also be considered as responses to specific classes. Therefore, we
designed a multi-stage integration module. By obtaining the dependencies
between features of T1WI in each stage with other stages as well as T2WI in
each stage, the network can allocate different attention weights to the
features of different contrast and stages. Thus, while the network pays
attention to the more informative stages in T2WI, it emphasizes the stages in
T1WI that can guide the SR of T2WI.

As illustrated in Fig. 2, we concatenate the intermediate features
${\mathbf{F}_{T_{1}}}_{s}$ and ${\mathbf{F}_{T_{2}}}_{s}$ to get the input
feature
$\mathbf{H}_{s}\in\mathbb{R}^{2Z\\!\times\\!H\\!\times\\!W\\!\times\\!C}$,
which extracted from $Z$ residual groups of T1WI and T2WI, into the designed
module respectively. Then, $\mathbf{H}_{s}$ are reshaped to a 2D matrix with
the dimension of $2Z\\!\times\\!HWC$. To obtain the correlation matrix
$\boldsymbol{A}=\boldsymbol{w}_{i,j=1}^{Z}$, we apply matrix multiplication to
the 2D matrix and its corresponding transpose matrix, which can be expressed
as

$\mathbf{w}_{j,i}=\texttt{softmax}(\mathbf{H}_{i}\cdot\mathbf{H}_{j}^{\top}),\quad
i,j=1,2,\ldots,Z,$ (10)

where $\left(\mathbf{H}\right)^{i}$ represents the intermediate T1WI and T2WI
output of $i$-th residual group, $\delta(\cdot)\text{ and }\varphi(\cdot)$
represents the softmax and reshape operations. Then, the reshaped feature
groups $\mathbf{H}_{s}$ is multiplied by the prediction correlation matrix
with scale factor $\gamma$, and then summed with the input feature
$\mathbf{H}_{s}$ to obtain the enhanced multi-stage integrated feature.

$\boldsymbol{F}_{M}^{j}=\gamma\sum_{i=1}^{Z}\boldsymbol{w}_{i,j}\left(\mathbf{H})\right)^{i}+\left(\mathbf{H})\right)^{j},$
(11)

where the initial value of $\gamma$ is 0, but it is automatically updated in
the following epochs based on the importance of the features. Accordingly, the
weighted sum of features will force the network to focus on the stages that
can promote T2WI SR in the T1WI.

###  2.2 Channel-Spatial Attention Module

Figure 3: Architecture of our channel-spatial attention module.

Inspired by the spatial [woo2018cbam] and the channel attention mechanism
[zhang2018image], we design the channel-spatial attention module, which
reveals responses from all dimensions of the feature maps. As shown in Fig.
3, the features output by the residual groups $\mathbf{F}_{T_{1}}^{Z}$ or
$\mathbf{F}_{T_{1}}^{Z}$, are sent into a 3D convolution to generate attention
map $A_{CS}$ by capturing joint channels and spatial features, which can be
written as

$\mathbf{F}_{CS}=\lambda\sigma\left(\boldsymbol{A}_{CS}\right)\odot\mathbf{F}^{Z}+\mathbf{F}^{Z},$
(12)

where $\sigma(\cdot)$ is the sigmoid operation, $\lambda$ is initialized to 0
and updated in the following iterations, $\mathbf{F}^{Z}$ represents the
output feature of $Z^{th}$ residual groups of T1WI and T2WI. As a results,
$\mathbf{F}_{CS}$ aggregates all channel and spatial position features and
original features.

##  3 Experiments

####  3.0.1 Datasets and Baselines.

We evaluate our approach on three datasets: 1) SMS is constructed by a 3T
Siemens Magnetom Skyra system from 155 patients. Each MRI scan includes T1 and
T2 weight imaging with full $k$-space sampled (TRT1=2001ms, TET1=10.72ms,
slice thickness=5 mm, matrix=320$\times$320$\times$20, field of view
(FOV)=230$\times$200 mm2, TRT2=4511 ms, TET2=112.86ms,). 2) uMR is constructed
by a 3T whole body scanner (uMR 790, United Imaging Healthcare, Shanghai,
China) from 50 patients. Each MRI scan includes T1 and T2 weight imaging with
full $k$-space sampled (TRT1=2001 ms, TET1=10.72 ms, slice thickness=4 mm,
matrix=320$\times$320$\times$19, FOV=220$\times$250 mm2, TRT1=4511 ms,
TET1=112.86 ms). 3) fastMRI is the largest open-accessible raw k-space dataset
[zbontar2018fastmri]. Following [xuan2020learning], we filter out 227 and 24
pairs PD and PDFS volumes for training and validation. All the above dataset
are aligned before our experiments, and the collection of the clinical
datasets are approved by the Institutional Review Board. We split each
clinical dataset patient-wise into training/validation/test sets with a ratio
of 7 : 1 : 2. For comparison, we compare our method with tow single-contrast
SR methods: (EDSR [lim2017enhanced], SMORE [zhao2018deep]) and two multi-
contrast SR methods: (Zeng et al. [lyu2020multi], Lyu et al.
[zeng2018simultaneous]).

####  3.0.2 Experimental setup.

We train our model in PyTorch with two NVIDIA Tesla V100 GPUs and 32GB of
memory per card. Our model is trained using Adam optimizer with learning rate
5e-5 for 50 epochs. The $\alpha_{1}$ and $\beta$ are empirically set to 0.3,
0.7, respectively. All the compared methods are retrained using their default
parameter settings.

##  4 Results

####  4.0.1 Quantitative Results

Table 4 reports the average NMSE, SSIM and PSNR scores with respect to
different datasets on 2$\times$ and 4$\times$ enlargement. From this table,
our approach yield the best results in the two datasets. It demonstrates that
our model can effectively fuse the two modality, which that this is beneficial
to the restoration of the target modality. Specifically, the single-contrast
SR e.g.,DSR [lim2017enhanced] SMORE [zhao2018deep] is far less effective than
the other multi-contrast SR models. More importantly, the multi-contrast SR
models i.e., Zeng et al. [lyu2020multi] and Lyu et al. [zeng2018simultaneous]
are less effective in comparison with our model, since they missed the mining
of the fusion of two contrast features and the interaction of two modes at
each stage. In particular, under the scales of 4$\times$ enlargement, we
improve the PSNR from 32.2312 to 34.2194, and SSIM from 0.9290 to 0.9562, as
compared to the current best approach, i.e., Lyu et al.
[zeng2018simultaneous]. Although 4$\times$ enlargement is more difficult to
restore images than 2$\times$, our method still outperforms previous methods
in extremely challenging settings, which can be attributed to the strong
capability of our multi-stage fusion model in multi-contrast image
restoration.

####  4.0.2 Image quality comparison

Figure 4 illustrate the 2$\times$ and 4$\times$ enlargement T2 weight images
and the corresponding error maps. The more obvious the texture in the error
map, the worse the restoration effect is. However, our restoration exhibit
less chessboard effect and structural losses as compared to other methods,
which is owed to the fact that our method can effectively learn the aggregate
features in multi-stage. More importantly, the error maps in different
enlargement scales show the robustness of our method especially at various
datasets.

####  4.0.3 Ablation study

In this section, we evaluate the effectiveness of the key components of our
model through ablation study. We construct two models: w/o $\mathcal{F}_{M}$
which is our model with out $\mathcal{F}_{M}$, and the w/o $\mathcal{F}_{CS}$
which is our model without $\mathcal{F}_{CS}$. We summarized the 2$\times$
enlargement results in Table 2. From this table, we observe that w/o
$\mathcal{F}_{M}$ performs the worst results since the $\mathcal{F}_{M}$
reflects the interaction of two contrasts at different stages. However, w/o
$\mathcal{F}_{CS}$ outperforms w/o $\mathcal{F}_{M}$, because
$\mathcal{F}_{CS}$ only enhances the features of the single contrast, can not
learn the interaction between the two contrasts. Thus, the enhancement of
single contrast feature from $\mathcal{F}_{CS}$ and the fusion feature from
$\mathcal{F}_{M}$, showing the powerful capability of our model in the mining
of crucial information to guide T2 weight image restoration.

Table 1: Ablative results under 2$\times$ enlargement. The best and second-
best results are marked in red and blue, respectively

Variant |  NMSE$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$
---|---|---|---
w/o $\mathcal{F}_{M}$ |  0.0025$\pm$0.002 |  39.2001$\pm$0.108 |  0.9776$\pm$0.025
w/o $\mathcal{F}_{CS}$ |  0.0022$\pm$0.236 |  39.2654$\pm$0.001 |  0.9791$\pm$0.037
MCSAN |  0.0018$\pm$0.004 |  40.5491$\pm$0.014 |  0.9830$\pm$0.009

##  5 Conclusion

We proposed a Multi-modal Cross Stage Attention Network for Multi-contrast MRI
Super-resolution, with the aim to effectively restore the target contrast
image with guidance of a complementary contrast. Specifically, our model
explores modality-specific properties within each modality, as well as the
interaction between multi-contrast images. The $\mathcal{H}_{CS}$ and
$\mathcal{H}_{M}$ are collaboratively applied to multi-level and multi-
contrast features, which help to capture more informative features for the
target contrast image restore. This work provides a potential guide for
further research and extension of the interaction between multi- contrast of
MRI super-resolution.

Table 2: Quantitative results on the two dataset under different enlargement
scales, in terms of NMSE, SSIM, and PSNR. The best and second-best results are
marked in red and blue, respectively.

Dataset | IXI dataset | clinical dataset
---|---|---
Method | 2x | 4x | 2x | 4x
| PSNR | SSIM | NMSE | PSNR | SSIM | NMSE | PSNR | SSIM | NMSE | PSNR | SSIM | NMSE
Bicubic | 0.257 | 21.548 | 0.780 | 0.359 | 19.508 | 0.706 | 0.290 | 21.107 | 0.730 | 0.305 | 19.072 | 0.720
EDSR [lim2017enhanced] | 28.439 | 0.847 | 0.033 | 21.323 | 0.687 | 0.170 | 28.750 | 0.816 | 0.044 | 27.539 | 0.803 | 0.058
SMORE [zhao2018deep] | 27.535 | 0.802 | 0.041 | 21.696 | 0.731 | 0.156 | 28.781 | 0.765 | 0.064 | 26.197 | 0.751 | 0.079
Zeng et al. [lyu2020multi] | 28.426 | 0.847 | 0.033 | 21.895 | 0.710 | 0.149 | 28.839 | 0.817 | 0.043 | 27.700 | 0.815 | 0.056
Lyu et al. [zeng2018simultaneous] | 28.426 | 0.847 | 0.033 | 21.895 | 0.710 | 0.149 | 28.839 | 0.817 | 0.043 | 27.700 | 0.815 | 0.056
MINet | 29.397 | 0.872 | 0.027 | 28.659 | 0.850 | 0.032 | 30.400 | 0.841 | 0.030 | 30.252 | 0.840 | 0.031

Table 3: Quantitative results on three datasets, in terms of NMSE, SSIM, and
PSNR. The best and second-best results are marked in red and blue,
respectively.

| 2$\times$ | 4$\times$
---|---|---
Method |  NMSE$\downarrow$ |  PSNR$\uparrow$ | SSIM$\uparrow$ |  NMSE$\downarrow$ |  PSNR$\uparrow$ | SSIM$\uparrow$
Bicubic | 0.257 | 21.548 | 0.780 | 0.359 | 19.508 | 0.706
EDSR [lim2017enhanced] | 0.005 | 36.415 | 0.962 | 0.014 | 31.484 | 0.886
SMORE [zhao2018deep] | 0.003 | 38.106 | 0.972 | 0.012 | 32.091 | 0.901
Zeng et al. [lyu2020multi] | 0.003 | 38.164 | 0.973 | 0.011 | 32.484 | 0.912
Lyu et al. [zeng2018simultaneous] | 0.002 | 39.194 | 0.978 | 0.009 | 33.667 | 0.931
MCSAN | 0.002 | 40.549 | 0.983 | 0.006 | 35.032 | 0.948

| 2$\times$ | 4$\times$
---|---|---
Method |  NMSE$\downarrow$ |  PSNR$\uparrow$ | SSIM$\uparrow$ |  NMSE$\downarrow$ |  PSNR$\uparrow$ | SSIM$\uparrow$
Bicubic | 0.290 | 21.107 | 0.730 | 0.305 | 19.072 | 0.720
EDSR [lim2017enhanced] | 0.008 | 35.394 | 0.965 | 0.020 | 31.165 | 0.907
SMORE [zhao2018deep] | 0.006 | 36.547 | 0.972 | 0.017 | 31.791 | 0.918
Zeng et al. [lyu2020multi] | 0.006 | 36.435 | 0.971 | 0.017 | 31.859 | 0.921
Lyu et al. [zeng2018simultaneous] | 0.005 | 37.139 | 0.977 | 0.016 | 32.231 | 0.929
MCSAN | 0.004 | 37.997 | 0.980 | 0.010 | 34.219 | 0.956

| 2$\times$ | 4$\times$
---|---|---
Method |  NMSE$\downarrow$ |  PSNR$\uparrow$ | SSIM$\uparrow$ |  NMSE$\downarrow$ |  PSNR$\uparrow$ | SSIM$\uparrow$
EDSR [lim2017enhanced] | 0.103 | 26.669 | 0.512 | 0.747 | 18.363 | 0.208
SMORE [zhao2018deep] | 0.070 | 28.278 | 0.667 | 0.329 | 21.813 | 0.476
Zeng et al. [lyu2020multi] | 0.060 | 28.870 | 0.670 | 0.230 | 23.255 | 0.507
Lyu et al. [zeng2018simultaneous] | 0.032 | 31.769 | 0.709 | 0.059 | 29.219 | 0.604
MCSAN | 0.030 | 32.069 | 0.718 | 0.044 | 30.425 | 0.621

Figure 4: Architecture of our channel-spatial attention module.