††thanks: These authors contribute equally.††thanks: These authors contribute
equally.

#  Exact fixed-point tensor network construction for rational conformal field
theory

Gong Cheng  Department of Physics, Virginia Tech, Blacksburg, VA 24060, USA
Maryland Center for Fundamental Physics, University of Maryland, College Park,
MD 20740, USA  Lin Chen  School of Physics and Optoelectronics, South China
University of Technology, Guangzhou 510641, China  Zheng-Cheng Gu
corresponding to<EMAIL_ADDRESS>Department of Physics, The Chinese
University of Hong Kong, Shatin, New Territories, Hong Kong, China  Ling-Yan
Hung  corresponding to<EMAIL_ADDRESS>Yau Mathematical Sciences
Center, Tsinghua University, Haidian, Beijing 100084, China

###### Abstract

The novel concept of entanglement renormalization and its corresponding tensor
network renormalization technique have been highly successful in developing a
controlled real space renormalization group (RG) scheme for classical 2D
systems or $(1+1)$D quantum systems. Numerically approximate fixed-point (FP)
tensors are widely used to extract the conformal data of the underlying
conformal field theory (CFT) describing critical phenomena. In this paper, we
present an explicit analytical construction of the FP tensor for 2D rational
CFT (RCFT). We define it as a correlation function between the ”boundary-
changing operators” on triangles. Our construction fully captures all the
real-space RG conditions. We also provide a concrete example using the Ising
model to compute the scaling dimensions explicitly based on the corresponding
FP tensor. Interestingly, our construction of FP tensors is closely related to
a strange correlator, where the holographic picture naturally emerges.

###### pacs:

11.15.-q, 71.10.-w, 05.30.Pr, 71.10.Hf, 02.10.Kn, 02.20.Uw

Introduction — In the past two decades, the novel concept of entanglement
renormalization[1, 2, 3, 4, 5] has been developed to study critical systems.
In particular, computationally efficient algorithms has been proposed based on
tensor network techniques, such as various schemes of tensor network
renormalization (TNR) [6, 2, 5, 7, 8, 9, 10]. Although different in detailed
implementation, all these algorithms essentially combine tensors inside a
small patch recursively until an (approximately) fixed-point(FP) tensor is
reached. It is found that even with a moderate size of bond dimensions kept in
the coarse graining procedure, there are lots of important information such as
central charge, scaling dimensions and operator product expnasion(OPE)
coefficient of conformal field theory(CFT) can be read off from the FP
tensors[5, 10]. Despite the huge successes in numerically extracting conformal
data through tensor network simulations, constructing FP tensors analytically
for critical systems remains a significant challenge. While progress has been
made in understanding the components of FP tensors associated with primary
fields[11, 12], generalizing these constructions for descendant fields remains
unclear.

In this paper, we demonstrate that the collection of open string correlation
functions conformally related to an open pair of pants in every rational CFT
(RCFT) yields an exact infinite-dimensional FP tensor. By tiling these
correlators over a given manifold and summing over all intermediate states,
including primaries and descendants, we obtain the RCFT path integral.
However, this tiling process leaves behind holes, which must be reconciled for
the correlators to match with an FP tensor. Previous research [13] introduced
shrinkable boundary conditions that address this problem and was further
studied in [14]. By combining these boundary conditions with the open
correlators, we achieve a field theoretical construction of tensors that
satisfy the expected properties of a FP tensor. To validate our approach, we
provide explicit numerical examples, focusing on the Ising model. Our results
demonstrate convincingly that our proposed shrinkable boundary condition
successfully shrinks, and the FP tensors we propose can accurately recover the
closed spectrum of the exact CFT when tiling a cylinder.

Finally, we stress that our construction of FP tensors coincides with
constructing an eigenstate $\langle\Omega|$ of the topological RG operator
associated to a fusion category $\mathcal{C}$ [15, 16], and expressing the CFT
partition function as a strange correlator, namely
$Z_{CFT}=\langle\Omega|\Psi\rangle$, where $|\Psi\rangle$ is the ground state
wave-function of the Levin-Wen model [17], or Turaev-Viro topological quantum
field theory (TQFT) [18], associated also to category $\mathcal{C}$.

(a)

(b)

$\phi_{(i,I)}^{ab}$$\phi_{(j,J)}^{bc}$$\phi_{(k,K)}^{ca}$

(c)

Figure 1: (a) denotes the rank-3 tensor, corresponds to a path integral over
the shaded region (b). (c) denotes correlation function of three local
operators on a disk with conformal boundary condition on the red edge.

The structure of FP Tensor — The FP tensor we propose, denoted as
$\mathcal{T}^{abc}_{(i,I)(j,J)(k,K)}$, comprises nine indices. The labels $a$,
$b$, $c$ correspond to the conformal boundary conditions of the RCFT, while
$i$, $j$, $k$ represent the labels of the RCFT primaries, and the indices $I$,
$J$, $K$ pertain to the descendants of their respective primaries. In the
RCFT, $a$, $b$, $c$ and $i$, $j$, $k$ take values from a finite set, while
$I$, $J$, $K$ live in an infinite-dimensional space. Consequently, the exact
FP tensors possess an infinite bond dimension, as expected. The FP tensor,
$\mathcal{T}^{abc}_{(i,I)(j,J)(k,K)}$, can be interpreted as the path integral
of the CFT within an open triangle. To regulate the path integral, we slightly
modify the corners of the triangle and impose conformal boundary conditions
labeled as $a$, $b$, and $c$ at each respective corner. The edges of the
triangle correspond to states that represent boundary-changing operators that
connect the two conformal boundaries associated with the given edge.

To show that they correspond to FP tensors, we need to demonstrate two
properties: (a) the FP tensors should satisfy crossing relations, and FP
tensors covering a large patch upon contraction reproduce exactly the same FP
tensors covering a smaller patch. (b) Tiling the FP tensors on a surface and
assigning appropriate contraction of the indices recover the CFT path-integral
on the surface. These conditions are illustrated in Fig. 3 and Fig. 4. As we
will see, these requirements ensure that the FP tensors reconstruct the CFT
partition function exactly.

Figure 2: Diagramatical representation of the tensor. The base triangle
denotes the structure coefficient $C^{abc}_{ijk}$, and the rank-3 tensor on
top of it denotes the conformal block $\alpha^{ijk}_{IJK}$ which carries
descendants information.

In general, the FP tensor can be decomposed as:

$\mathcal{T}^{abc}_{(i,I)(j,J)(k,K)}=\alpha_{IJK}^{ijk}C^{abc}_{ijk}$ (1)

This is because a three-point correlation function of three boundary operators
carries two parts, represented diagramatically in Fig. 2, namely the structure
coefficients $C^{abc}_{ijk}$ and the conformal blocks carrying the dependence
of the correlation function on the precise descendent in the primary families,
the location of insertions, and the precise shape of the manifold in which
operators are inserted. To set our notations, the three point correlation
functions of three primary boundary changing operators on the upper-half plane
is given by:

$\displaystyle\langle\phi^{ab}_{(i_{1},0)}(x_{1})\phi^{bc}_{(i_{2},0)}(x_{2})\phi^{bc}_{(i_{3},0)}(x_{3})\rangle=C^{abc}_{i_{1}i_{2}i_{3}}\beta^{ijk}_{000}(x_{1},x_{2},x_{3}),$
(2) $\displaystyle\beta^{ijk}_{000}(x_{1},x_{2},x_{3})=$ (3)
$\displaystyle\frac{1}{|x_{1}-x_{2}|^{\Delta_{1}+\Delta_{2}-\Delta_{3}}|x_{1}-x_{3}|^{\Delta_{1}+\Delta_{3}-\Delta_{2}}|x_{3}-x_{2}|^{\Delta_{3}+\Delta_{2}-\Delta_{1}}},$

where $I=J=K=0$ denotes the fact that the inserted operators are all
primaries. Conformal blocks involving other descendents where $I,J,K\neq 0$
can be generated by repeated use of the Virasoro or generally Kac-Moody
operators in the primaries.

In our proposed FP tensor, $\alpha_{IJK}^{ijk}=\chi\circ\beta_{IJK}^{ijk}$,
where $x_{1,2,3}$ are fixed and suppressed in the following, and $\chi$
denotes a conformal map from the upper half plane to a triangle. This map is
detailed in the supplementary Material. These $\alpha^{ijk}_{IJK}$ satisfies:

$\sum_{M}\alpha^{ijm}_{IJM}\gamma^{mkl}_{MKL}=\sum_{n,N}[F^{ijk}_{l}]^{\textrm{blocks}}_{mn}\alpha^{inl}_{INL}\alpha^{jkn}_{JKN},$
(4)

where $[F^{ijk}_{l}]^{\textrm{blocks}}$ are the crossing coefficients
characterising this RCFT. The proposed FP tensor satisfies the crossing
relation in condition (a).

$\displaystyle\sum_{k,K}\mathcal{T}^{abc}_{(i,I)(j,J)(k,K)}\mathcal{T}^{acd}_{(k,K)(l,L)(p,P)}$
$\displaystyle=\sum_{h,H}\mathcal{T}^{bda}_{(h,H)(p,P)(i,I)}\mathcal{T}^{bcd}_{(j,J)(i,L)(h,H)}.$
(5)

Diagrammatically, this is illustrated in Fig. 3, which follows from the
crossing symmetry of the RCFT.

Figure 3: crossing symmetry

The FP tensor also satisfies the coarse graining condition (b), which is
illustrated in Fig. 4.

$\sum_{i}\omega_{i}$$i$$=$

Figure 4: coarse graining

Here, we note that the vertex degree of freedom at the center is summed over
with a weight $w_{i}$. For a diagonal RCFT,

$w_{i}=S_{00}^{1/2}S_{i0},$ (6)

Physically, the coarse-graining condition implies that when we sew the four
triangles by contracting the shared descendant labels between neighboring
triangles, a small hole is left in the middle. This hole disappears when we
sum over the conformal boundary conditions with weights given by (6). The idea
of this weighted sum of conformal boundary conditions was initially explored
in [13] within the context of entanglement brane boundary conditions. It
suggests that the boundaries arising in the computation of the entanglement
entropy are artificial and should be ”contractible.” These considerations
motivated the use of this particular weighted sum.

The open boundary can be transformed through a modular transformation into a
closed conformal boundary Cardy state $|i\rangle_{c}$. It can be shown that
the weighted sum of the boundaries yields:

$\sum_{i}\omega_{i}|i\rangle_{c}=|0\rangle\rangle,$ (7)

where the right-hand side corresponds to the identity of the Ishibashi state.
When the hole is small, the dominant contribution arises from the leading
term, which is the vacuum state. The leading corrections then come from the
leading descendant of the vacuum state, which can be viewed as an irrelevant
perturbation in the thermodynamic limit of the tiling, as explained in [14].

The partition function of the CFT on a manifold $M$ can be obtained using the
following procedure. We begin by triangulating the manifold $\mathcal{M}$ into
a collection of triangles ${\triangle}$. Each edge $e$ on a triangle is
labeled with a pair of primary and descendant labels $(i,I)$, and each vertex
$v$ is labeled with a conformal boundary condition $a$. On each triangle, we
assign a tensor $\mathcal{T}^{abc}_{(i,I)(j,J)(k,K)}$ based on the labeling of
the edges and vertices. The proposed partition function is then given by:

$\displaystyle
Z_{M}=\sum_{\\{(i,I)\\},\\{a\\}}\prod_{v}\omega_{a}\prod_{\triangle}\mathcal{T}^{abc}_{(i,I)(j,J)(k,K)}.$
(8)

A simple example: the Ising CFT — In the Ising example, we can put in explicit
expressions to the above construction. The closed Ising CFT has three
primaries $\mathcal{C}_{Is}=\\{I,\sigma,\psi\\}$. The theory has three
conformal boundary conditions. They are labeled as $\\{+,-,f\\}$,
corresponding to the respective primaries. The Hilbert space for an interval
with left and right boundary given by $a$ and $b$ respectively, where
$a,b\in\mathcal{C}_{Is}$ is given by
$\mathcal{H}_{ab}=\oplus_{c}N_{ab}^{c}V_{c}$, where $V_{c}$ is the space
corresponding to the primary representation labeled $a\in\mathcal{C}_{Is}$,
and $N_{ab}^{c}\in\mathbb{Z}_{\geq 0}$ are the fusion coefficient among the
objects $\mathcal{C}_{Is}$, with:

$N_{Ib}^{c}=\delta_{bc},\,\,\,N_{\sigma\sigma}^{c}=1-\delta_{c\sigma},\,\,\,N_{\sigma
a\neq\sigma}^{b}=\delta_{b\sigma},\,\,\,N_{\psi\psi}^{b}=\delta_{bI}.$ (9)

The structure coefficients $C^{abc}_{ijk}$ are given by

$\displaystyle C^{\pm\pm\pm}_{III}=C^{\pm\mp\pm}_{\psi\psi
I}=C^{fff}_{\sigma\sigma I}=C^{\pm f\pm}_{\sigma\sigma I}=2^{\frac{1}{4}},$
(10) $\displaystyle C^{fff}_{\psi\psi I}=1,C^{\pm
f\mp}_{\sigma\sigma\psi}=\frac{1}{2^{\frac{1}{4}}},C^{f+f}_{\sigma\sigma\psi}=\frac{1}{\sqrt{2}},C^{f-f}_{\sigma\sigma\psi}=-\frac{1}{\sqrt{2}}$

We have to compute the three point functions involving descendants, and then
transform them into the needed geometry using the conformal map that is
relegated to the Supplementary Materials. Explicitly, one has to look for the
orthogonal basis of the descendants. For example, in level one, the normalized
first descendant $O^{(-1)}$ is defined as $\frac{1}{\sqrt{2\Delta}}L_{-1}O$.
It’s transformation under the conformal map $\chi(z)$ is

$\chi_{*}[O^{(-1)}]=|\chi^{\prime}(0)|^{\Delta}\left(\chi^{\prime}(0)O^{(-1)}+\sqrt{\frac{\Delta}{2}}\frac{\chi^{\prime\prime}(0)}{\chi^{\prime}(0)}O\right).$
(11)

In the second level, we find three normalized operators.

$\displaystyle\mathbb{1}^{(-2)}=2L_{-2}\mathbb{1},$ (12)
$\displaystyle\psi^{(-2)}=\frac{6}{25}L_{-2}\psi+\frac{9}{25}L_{-1}^{2}\psi,$
(13)
$\displaystyle\sigma^{(-2)}=\frac{16\sqrt{2}}{25}L_{-2}\sigma+\frac{12\sqrt{2}}{25}L_{-1}^{2}\sigma.$
(14)

and the corresponding transformation rules given by:

$\displaystyle\chi_{*}[L_{-2}O]=(\chi^{\prime})^{2}L_{-2}O+\frac{3}{2}\chi^{\prime\prime}L_{-1}O+$
$\displaystyle\left(\frac{c\chi^{\prime\prime\prime}}{12\chi^{\prime}}-\frac{c(\chi^{\prime\prime})^{2}}{8(\chi^{\prime})^{2}}+\frac{2\chi^{\prime\prime\prime}\Delta}{3\chi^{\prime}}-\frac{(\chi^{\prime\prime})^{2}\Delta}{4(\chi^{\prime})^{2}}\right)O,$
(15)
$\displaystyle\chi_{*}[L_{-1}^{2}O]=(\chi^{\prime})^{2}L_{-1}^{2}O+(2\Delta+1)\chi^{\prime\prime}L_{-1}O+$
(16)
$\displaystyle\left(\frac{\chi^{\prime\prime\prime}\Delta}{\chi^{\prime}}+\frac{(\chi^{\prime\prime})^{2}\Delta(\Delta-1)}{(\chi^{\prime})^{2}}\right)O.$
(17)

For higher level descendants, we derive iteration equations to solve all the
transformation rules. Additionally, the three-point correlation functions for
descendant fields are also calculable by iteration methods. The details are
also illustrated in the Supplementary Materials. Together, these data allow us
to calculate the tensor values for arbitrary high descendants.

We checked crossing relations 1) and 2), keeping only three descendents in
each conformal family. Despite the very small bond dimension, we find that
they are satisfied to an accuracy of $2\times 10^{-3}$.

Consider the example of fixing the four external legs as
$(\mathbb{1},\mathbb{1},\mathbb{1},\mathbb{1})$ and the four boundary
conditions to be $(+,+,+,+)$. Then we compute the following contraction:

$T_{L.H.S.}:=[(T^{+++}_{\mathbb{1}\mathbb{1}\mathbb{1}})^{4}+(T^{+-+}_{\psi\psi\mathbb{1}})^{4}+\sqrt{2}(T^{+f+}_{\sigma\sigma\mathbb{1}})^{4}]/2\sqrt{2}.$
(18)

where we didn’t write the descendant field indices and they are understood as
being contracted implicitly. Similarly contracting two tensors we get

$T_{R.H.S.}=(T^{+++}_{\mathbb{1}\mathbb{1}\mathbb{1}})^{2}.$ (19)

A very important utility of the FP tensor is to recover the closed spectrum of
the CFT when tiling them on a cylinder. We demonstrate that our proposed FP
tensor constructed from open correlation functions can indeed recover the
closed spectrum with surprisingly high accuracy despite keeping only very few
descendents in each family. The cylinder is constructed using 4 squares formed
out of 8 triangles, as shown in Supplementary Materials.

The labels of the conformal boundaries at the top and the bottom edge of the
cylinders are treated alongside the primaries and descendent labels of the FP
tensors as input and output indices of the cylinder. One can solve for the
spectrum of the cylinder, which is listed in the Table 1 below.

States | Numerical dim | Accurate dim
---|---|---
$\mathbb{1}$ | 0.000 | 0.000
$\sigma$ | 0.127 | 0.125
$\epsilon$ | 1.035 | 1.000
$\partial\sigma,\bar{\partial}\sigma$ | 1.129 | 1.125
$\partial\epsilon,\bar{\partial}\epsilon$ | 2.092 | 2.000
$\mathbb{1}^{(-2)}$, $\bar{\mathbb{1}}^{(-2)}$ | 2.000 | 2.000
$\partial\bar{\partial}\sigma$ | 2.259 | 2.125
$\sigma^{(-2)}$, $\bar{\sigma}^{(-2)}$ | 2.211 | 2.125
$\partial\bar{\partial}\epsilon$ | 3.253 | 3.000
$\partial^{2}\epsilon$, $\bar{\partial}^{2}\epsilon$ | 3.142 | 3.000
$\mathbb{1}^{(-3)},\bar{\mathbb{1}}^{(-3)}$ | 3.230 | 3.000
$\bar{\partial}\sigma^{(-2)},\partial\bar{\sigma}^{(-2)}$ | 3.184 | 3.125
$\partial\sigma^{(-2)}$, $\bar{\partial}\bar{\sigma}^{(-2)}$ | 3.330 | 3.125
$\mathbb{1}^{(-2,-\bar{2})}$ | 3.973 | 4.000
Table 1: Conformal dimensions obtained by diagonalizing the transfer matrix,
compared with the accurate data.

FP tensors as eigenstates of topological RG operators — While the FP tensor
can be understood directly as a CFT correlation function without explicit
reference to an associated 3d TQFT, it is an important observation that these
FP tensors follows from an exact eigenstate of the topological RG operator in
[19, 15], and the CFT partition function can be written explicitly also as a
strange correlator.

To appreciate this connection, recall that the label set of primaries in an
RCFT are objects in a modular fusion cateogry $\mathcal{C}$. To avoid clutter,
we focus on diagonal RCFT so that the conformal boundary conditions are also
labeled by objects in $\mathcal{C}$. It is convenient to re-scale the three
point conformal block
$\alpha^{ijk}_{IJK}=\mathcal{N}_{ijk}\,\gamma^{ijk}_{IJK}$, where [20] ,

$\mathcal{N}_{ijk}=\sqrt{\theta(i,j,k)/\sqrt{d_{i}d_{j}d_{k}}},$ (20)

where $\theta(i,j,k)=d_{i}/\left[F^{jkkj}\right]^{\textrm{blocks}}_{1i}$, and
$d_{i}$ is the quantum dimension of object $i$, which is related to the
modular matrix by $d_{i}=S_{0i}/S_{00}$ for a diagonal RCFT. The value of the
FP tensor (1) does not change, except that it is decomposed instead as
$\mathcal{T}^{abc}_{(i,I)(j,J)(k,K)}=\gamma^{ijk}_{IJK}\hat{C}^{abc}_{ijk}$.
On this basis, the structure coefficients $\hat{C}^{abc}_{ijk}$ of a diagonal
RCFT (including the Ising CFT described above) can be written simply as [21],

$\hat{C}^{abc}_{ijk}=(d_{i}d_{j}d_{k})^{1/4}\left[\begin{tabular}[]{ccc}$i$&$j$&$k$\\\
$c$&$a$&$b$\end{tabular}\right],$ (21)

where the square bracket denotes the quantum 6j-symbols of the modular tensor
category $\mathcal{C}$ associated to the RCFT in a frame with tetrahedral
symmetry and chosen normalization. Several components in this gauge involving
the identity label are fixed to the values reviewed in the Supplementary
Material. All two point correlations are also normalised. These
$\gamma^{ijk}_{IJK}$ inherit the crossing relation of (4), with the crossing
kernel re-scaled as:

$[F^{ijk}_{l}]_{mn}=[F^{ijk}_{l}]^{\textrm{blocks}}_{mn}\frac{\mathcal{N}_{jkn}\mathcal{N}_{inl}}{\mathcal{N}_{ijm}\mathcal{N}_{mkl}}.$
(22)

These re-scaled crossing kernals $[F^{ijk}_{l}]_{mn}$ is related to the
quantum 6j-symbol above by:

$[F^{ijk}_{l}]_{mn}=\sqrt{d_{m}d_{n}}\left[\begin{tabular}[]{ccc}$i$&$j$&$m$\\\
$k$&$l$&$n$\end{tabular}\right].$ (23)

The explicit values of $[F^{ijk}_{l}]_{mn}$ and
$[F^{ijk}_{l}]^{\textrm{block}}_{mn}$ for the Ising CFT are given explicitly
in the Supplementary Materials.

Now it should be obvious that (8) can be rewritten as a strange correlator

$Z_{M}=\langle\Omega|\Psi\rangle,$ (24)

where $|\Psi\rangle$ is the ground state of the Levin-Wen model corresponding
to the fusion category $\mathcal{C}$. It is well known that the wave-function
on a two dimension surface (say topologically an $S^{2}$) can be constructed
using the Turaev Viro formulation of TQFT path-integral over a triangulated
three ball [18]. For a surface triangulation that matches the tiling as
specified in (8), the Levin-Wen ground state wavefunction can be written as
[18]

$\displaystyle|\Psi\rangle=\sum_{\\{a_{v}\\}}\sum_{\\{i\\}}\prod_{e}d_{i}^{1/2}\prod_{v}\omega_{a}\prod_{\triangle}\begin{bmatrix}i&j&k\\\
c&a&b\end{bmatrix}|\\{i\\}\rangle,$ (25)

The ket $|\\{i\\}\rangle$ are basis states living on the edges which carries a
label $i\in\mathcal{C}$, and

$\langle\Omega|=\sum_{\\{(i,I)\\}}\langle\\{i\\}|\prod_{e}(d_{i})^{-1/4}\prod_{\triangle}\gamma^{IJK}_{ijk}\dots.$
(26)

The crossing relation (4), together with (7) guarantees that $\langle\Omega|$
is an eigenstate of the RG operator proposed in [19].

We note that the entanglement brane boundary condition (7) follows simply from
the prescription of the Turaev-Viro formation of the path-integral. The
weights assigned to each internal edge that is summed agrees with the weighted
sum of the Cardy states in (7). In other words, the associated 3d TQFT
constructed from $\mathcal{C}$ knows about how to close holes in the RCFT.

This formulation provides a straightforward solution for determining
entanglement brane boundary conditions in non-diagonal CFTs. When constructing
non-diagonal RCFTs, one can utilize an alternative representation of the same
Levin-Wen ground state using the Projected Entangled Pair States (PEPS)
approach [22]. In this context, the boundary conditions of the CFT correspond
to corner variables placed on triangles, which are generally labeled by
objects from a ”module category” $\mathcal{M}_{\mathcal{C}}$ associated with
the fusion category $\mathcal{C}$. According to the TQFT framework[18], the
corner variable should be summed with the weights given by the quantum
dimension of the label as an object in the module category. This summation
procedure yields the appropriate entanglement brane boundary conditions for
general RCFTs. The strange correlator representation of the exact two-
dimensional CFT partition function serves as an explicit, practical, and
easily computable realization of the holographic relationship between a
quantum field theory with categorical symmetry and a TQFT in one higher
dimension, as advocated in Ref. [23, 24].

Conclusion and discussion — In conclusion, we present a concrete construction
of FP tensors for RCFTs based on the holographic principle. Specifically, the
FP tensor can be viewed as a correlation function of RCFT involving ”boundary-
changing operators” defined on triangles. Our proposed construction of the FP
tensor naturally fulfills all the requirements of real space RG conditions.
This approach provides a novel avenue for exploring the FP tensor of conformal
field theory in higher dimensions, offering exciting possibilities for further
investigation.

Despite satisfying all the real-space RG conditions, constructing the
conformal map $\chi$ for the FP tensor still poses a challenge due to gauge
freedom. We address this issue in the Supplementary Materials, where we
discuss two distinct conformal maps derived using different methodologies. One
of these constructions involves a continuous parameter $\theta$, which, when
adjusted, has the potential to generate a continuous spectrum of valid FP
tensors. It is worth noting that the gauge freedom also plays a significant
role in generating FP tensors through numerical TNR algorithms, as highlighted
in recent studies[11, 12]. While the gauge freedom complicates direct
comparisons between our constructed tensor components and those obtained
numerically, the successful reproduction of the bulk states spectrum, while
satisfying all RG conditions, serves as a robust validation of our approach.
Finally, we stress that our constructions can be naturally generalzied into
higher dimensions, which might allow us to reformulate all CFTs in terms of
tensor networks.

Acknowledgments – We acknowledge useful discussions with Yikun Jiang, Bingxin
Lao, Nicolai Reshetikhin, Gabriel Wong and Xiangdong Zeng. This work is
supported by funding from Hong Kong’s Research Grants Council (GRF
no.14301219) and Direct Grant no. 4053578 from The Chinese University of Hong
Kong. LYH acknowledges the support of NSFC (Grant No. 11922502, 11875111). LC
acknowledges the support of NSFC (Grant No. 12305080) and the start up funding
of South China University of Technology. GC acknowledges the support from
Commonwealth Cyber Initiative at Virginia Tech, U.S. Department of Energy,
Office of Science, Office of Advanced Scientific Computing Research.

## References

  * Vidal [2007] G. Vidal, Entanglement renormalization, Phys. Rev. Lett. 99, 220405 (2007).
  * Gu  _et al._ [2008] Z.-C. Gu, M. Levin, and X.-G. Wen, Tensor-entanglement renormalization group approach as a unified method for symmetry breaking and topological phase transitions, Phys. Rev. B 78, 205116 (2008).
  * Evenbly and Vidal [2009] G. Evenbly and G. Vidal, Algorithms for entanglement renormalization, Phys. Rev. B 79, 144108 (2009).
  * Pfeifer  _et al._ [2009] R. N. C. Pfeifer, G. Evenbly, and G. Vidal, Entanglement renormalization, scale invariance, and quantum criticality, Phys. Rev. A 79, 040301 (2009).
  * Gu and Wen [2009] Z.-C. Gu and X.-G. Wen, Tensor-entanglement-filtering renormalization approach and symmetry-protected topological order, Phys. Rev. B 80, 155131 (2009).
  * Levin and Nave [2007] M. Levin and C. Nave, Tensor renormalization group approach to two-dimensional classical lattice models, Phys. rev. lett. 99, 120601 (2007).
  * Xie  _et al._ [2009] Z.-Y. Xie, H.-C. Jiang, Q.-J. N. Chen, Z.-Y. Weng, and T. Xiang, Second renormalization of tensor-network states, Phys. rev. lett. 103, 160601 (2009).
  * Xie  _et al._ [2012] Z. Y. Xie, J. Chen, M. P. Qin, J. W. Zhu, L. P. Yang, and T. Xiang, Coarse-graining renormalization by higher-order singular value decomposition, Phys. Rev. B 86, 045139 (2012).
  * Evenbly and Vidal [2015] G. Evenbly and G. Vidal, Tensor network renormalization, Phys. rev. lett. 115, 180405 (2015).
  * Yang  _et al._ [2017] S. Yang, Z.-C. Gu, and X.-G. Wen, Loop optimization for tensor network renormalization, Phys. rev. lett. 118, 110504 (2017).
  * Li  _et al._ [2022] G. Li, K. H. Pai, and Z.-C. Gu, Tensor-network renormalization approach to the $q$-state clock model, Phys. Rev. Res. 4, 023159 (2022).
  * Ueda and Yamazaki [2023] A. Ueda and M. Yamazaki, Fixed-point tensor is a four-point function, (2023), arXiv:2307.02523 [cond-mat.stat-mech] .
  * Hung and Wong [2021] L. Y. Hung and G. Wong, Entanglement branes and factorization in conformal field theory, Phys. Rev. D 104, 026012 (2021), arXiv:1912.11201 [hep-th] .
  * Brehm and Runkel [2022] E. M. Brehm and I. Runkel, Lattice models from CFT on surfaces with holes: I. Torus partition function via two lattice cells, J. Phys. A 55, 235001 (2022), arXiv:2112.01563 [cond-mat.stat-mech] .
  * Vanhove  _et al._ [2018] R. Vanhove, M. Bal, D. J. Williamson, N. Bultinck, J. Haegeman, and F. Verstraete, Mapping topological to conformal field theories through strange correlators, Phys. Rev. Lett. 121, 177203 (2018).
  * Aasen  _et al._ [2020] D. Aasen, P. Fendley, and R. S. K. Mong, Topological Defects on the Lattice: Dualities and Degeneracies, (2020), arXiv:2008.08598 [cond-mat.stat-mech] .
  * Levin and Wen [2005] M. A. Levin and X.-G. Wen, String net condensation: A Physical mechanism for topological phases, Phys. Rev. B 71, 045110 (2005), arXiv:cond-mat/0404617 .
  * Turaev and Viro [1992] V. G. Turaev and O. Y. Viro, State sum invariants of 3 manifolds and quantum 6j symbols, Topology 31, 865 (1992).
  * Chen  _et al._ [2022] L. Chen, H. Zhang, H.-C. Zhang, K.-X. Ji, C. Shen, R.-s. Wang, X.-d. Zeng, and L.-Y. Hung, Exact Holographic Tensor Networks – Constructing CFTD from TQFTD+1, (2022), arXiv:2210.12127 [hep-th] .
  * Kojita  _et al._ [2018] T. Kojita, C. Maccaferri, T. Masuda, and M. Schnabl, Topological defects in open string field theory, JHEP 04, 057, arXiv:1612.01997 [hep-th] .
  * Fuchs  _et al._ [2005] J. Fuchs, I. Runkel, and C. Schweigert, TFT construction of RCFT correlators IV: Structure constants and correlation functions, Nucl. Phys. B 715, 539 (2005), arXiv:hep-th/0412290 .
  * Lootens  _et al._ [2021] L. Lootens, J. Fuchs, J. Haegeman, C. Schweigert, and F. Verstraete, Matrix product operator symmetries and intertwiners in string-nets with domain walls, SciPost Phys. 10, 053 (2021), arXiv:2008.11187 [quant-ph] .
  * Gaiotto and Kulp [2021] D. Gaiotto and J. Kulp, Orbifold groupoids, JHEP 02, 132, arXiv:2008.05960 [hep-th] .
  * Lin  _et al._ [2023] Y.-H. Lin, M. Okada, S. Seifnashri, and Y. Tachikawa, Asymptotic density of states in 2d CFTs with non-invertible symmetries, JHEP 03, 094, arXiv:2208.05495 [hep-th] .

##  Appendix A Convention for 6j-symbols and F symbols

The crossing kernals $[F^{ijk}_{l}]_{mn}$ after being rescaled in the main
text, which are often also referred to as the Racah coefficients in the
literature, are related to quantum 6j-symbols as follows:

$F_{mn}\left[\footnotesize\begin{tabular}[]{ccc}$j$&$k$\\\
$i$&$l$\end{tabular}\right]=\sqrt{d_{m}d_{n}}\left[\begin{tabular}[]{ccc}$i$&$j$&$m$\\\
$k$&$l$&$n$\end{tabular}\right].$ (27)

The quantum 6j symbols denoted by object in square brackets, enjoy full
tetrahedral symmetry. In this gauge it fixes a number of components to:

$\left[\begin{tabular}[]{ccc}$a$&$a$&$0$\\\
$b$&$b$&$c$\end{tabular}\right]=\left[\begin{tabular}[]{ccc}$a$&$b$&$c$\\\
$b$&$a$&$0$\end{tabular}\right]=\frac{N^{c}_{ab}}{\sqrt{d_{a}d_{b}}}.$ (28)

Correspondingly,

$[F^{aab}_{b}]_{0c}=\sqrt{\frac{d_{c}}{d_{a}d_{b}}}.$ (29)

As described in the main text, the Racah coefficients are related to the
crossing kernels describing crossing relations between canonically normalised
conformal blocks by a re-scaling.

For the Ising CFT, the $F^{\text{blocks}}$ are given by the expressions below
with the parameter $\lambda=1/2$:

$\displaystyle F_{11}\left[\footnotesize\begin{tabular}[]{ccc}$\psi$&$\psi$\\\
$\psi$&$\psi$\end{tabular}\right]$ $\displaystyle=$ $\displaystyle 1,$ (32)
$\displaystyle
F_{11}\left[\footnotesize\begin{tabular}[]{ccc}$\sigma$&$\sigma$\\\
$\sigma$&$\sigma$\end{tabular}\right]$ $\displaystyle=$ $\displaystyle-
F_{\psi\psi}\left[\footnotesize\begin{tabular}[]{ccc}$\sigma$&$\sigma$\\\
$\sigma$&$\sigma$\end{tabular}\right]=\frac{1}{\sqrt{2}},$ (37) $\displaystyle
F_{1\psi}\left[\footnotesize\begin{tabular}[]{ccc}$\sigma$&$\sigma$\\\
$\sigma$&$\sigma$\end{tabular}\right]$ $\displaystyle=$
$\displaystyle\frac{\lambda}{\sqrt{2}},\;\;\;F_{\psi
1}\left[\footnotesize\begin{tabular}[]{ccc}$\sigma$&$\sigma$\\\
$\sigma$&$\sigma$\end{tabular}\right]=\frac{1}{\sqrt{2}\lambda},$ (42)
$\displaystyle
F_{1\sigma}\left[\footnotesize\begin{tabular}[]{ccc}$\psi$&$\sigma$\\\
$\psi$&$\sigma$\end{tabular}\right]$ $\displaystyle=$ $\displaystyle
F_{1\sigma}\left[\footnotesize\begin{tabular}[]{ccc}$\sigma$&$\psi$\\\
$\sigma$&$\psi$\end{tabular}\right]=\lambda,$ (47) $\displaystyle F_{\sigma
1}\left[\footnotesize\begin{tabular}[]{ccc}$\psi$&$\psi$\\\
$\sigma$&$\sigma$\end{tabular}\right]$ $\displaystyle=$ $\displaystyle
F_{\sigma 1}\left[\footnotesize\begin{tabular}[]{ccc}$\sigma$&$\sigma$\\\
$\psi$&$\psi$\end{tabular}\right]=\frac{1}{\lambda},$ (52) $\displaystyle
F_{\sigma\sigma}\left[\footnotesize\begin{tabular}[]{ccc}$\psi$&$\sigma$\\\
$\sigma$&$\psi$\end{tabular}\right]$ $\displaystyle=$ $\displaystyle
F_{\sigma\sigma}\left[\footnotesize\begin{tabular}[]{ccc}$\sigma$&$\psi$\\\
$\psi$&$\sigma$\end{tabular}\right]=-1.$ (57)

The Racah coefficients of the Ising model are given by the same expressions
above with $\lambda=1$. The corresponding 6j symbols are given by

$\displaystyle\left[\begin{tabular}[]{ccc}$\sigma$&$\sigma$&$1$\\\
$\sigma$&$\sigma$&$1$\end{tabular}\right]$ $\displaystyle=$
$\displaystyle\left[\begin{tabular}[]{ccc}$\sigma$&$\sigma$&$1$\\\
$\sigma$&$\sigma$&$\psi$\end{tabular}\right]=\frac{1}{\sqrt{2}},\left[\begin{tabular}[]{ccc}$\sigma$&$\sigma$&$\psi$\\\
$\sigma$&$\sigma$&$\psi$\end{tabular}\right]=\frac{-1}{\sqrt{2}},$ (64)
$\displaystyle\left[\begin{tabular}[]{ccc}$1$&$1$&$1$\\\
$\sigma$&$\sigma$&$\sigma$\end{tabular}\right]$ $\displaystyle=$
$\displaystyle\left[\begin{tabular}[]{ccc}$1$&$\psi$&$\psi$\\\
$\sigma$&$\sigma$&$\sigma$\end{tabular}\right]=2^{-\frac{1}{4}}.$ (69)

One can readily check that the they are indeed related to
$F^{\textrm{blocks}}$ by a rescaling of the form

$[F^{ijk}_{l}]_{mn}=[F^{ijk}_{l}]^{\textrm{blocks}}_{mn}\frac{\mathcal{N}_{jkn}\mathcal{N}_{inl}}{\mathcal{N}_{ijm}\mathcal{N}_{mkl}}.$
(70)

##  Appendix B Conformal maps defining the fixed point tensor

In this section we explicitly evaluate the tensor defined as path integral on
Riemann surface with boundaries, as shown in Fig. 5(a). To achieve this we
construct a conformal transformation $\chi$ to map this region to upper-half-
plane, and shrink the three boundary states to local operators inserted on the
real axis (Fig. 5(b)).

(a)

$\Re$$\Im$$g(\xi)$$\phi_{i}^{ab}$$b$$\phi_{j}^{bc}$$\phi_{k}^{ca}$

(b)

Figure 5: Conformal transformation that maps triangular region to upper-half-
plane with three local operators inserted on the real-axis.

In the following, we present two distinct approaches for constructing the map.
The first approach leverages the state-operator correspondence principle. The
second approach takes a different route by treating the triangular region as
an amputated pants-diagram.

Method I: state-operator correspondence.— The state-operator correspondence is
implemented through a map $f$ from the Upper-Half-Plane (UHP) to a designated
segment region, as shown in Fig. 6(a). Through this map we prepare the state
$|O_{i}^{ab}\rangle$ by inserting operator $O_{i}^{ab}$ at origin.

$\Re$$\Im$$O_{i}^{ab}$$|O_{i}^{ab}\rangle$$f$$f_{*}O_{i}^{ab}$$\theta$

(a)

$\ \ \ f_{*}O_{i}^{ac}$$f_{*}O_{j}^{cb}$$f_{*}O_{k}^{ba}$

(b)

Figure 6: (a) A single segment region is mapped to upper-half-plane through a
function $f$. (b) The triangular region attaches three segment regions along
the open boundaries.

Then we apply these prepared states to the three open boundaries of the
triangular region, Fig. 5(a). Diagrammatically, this process is represented by
attaching the segments along the three open boundaries as shown in Fig. 6(b).

We further require that the prepared states form an orthonormal basis in the
Hilbert space of boundary CFT. The inner product of these states is determined
by the two-point function, which is evaluated over the double-segment region
as shown in Fig. 7. The condition of orthonormality is expressed through the
following relationship:

$\langle f_{*}[O_{j}(0)^{\dagger}]f_{*}[O_{i}(0)]\rangle_{\text{double-
segment}}=\delta_{ij}.$ (71)

$\langle\phi_{i}^{ab}|\phi_{j}^{ba}\rangle=$$f_{*}O_{i}^{ab}$$f_{*}O_{j}^{ba}{}^{\dagger}$

Figure 7: The double segment region obtained by gluing two segments regions.

The conformal map $f$ satisfying this condition is given by the following
complicate composite function:

$\begin{split}&f(z)=\xi\circ\eta\circ\omega(z)\\\ &\omega(z)=\frac{1+z}{1-z},\
\ \eta(\omega)=e^{-i\theta}\omega^{\frac{2\theta}{\pi}},\ \
\xi(\eta)=i\frac{\eta-1}{\eta+1}.\end{split}$ (72)

Here we have a free parameter $\theta$ which is the angle of the corner in the
segment region (see Fig. 6(a)). This angle serves as a gauge freedom of our
tensor construction. For computation simplicity we choose to set
$\theta=\frac{\pi}{4}$.

Following this, we proceed by shrinking the corners of the triangular region
(refer to Fig. 5(a)) to zero length. This leads to the derivation of the
tensor, now represented as a three-point function on the area shown in Fig.
8(a).

$f_{1*}O_{i}^{ac}$$f_{2*}O_{j}^{cb}$$f_{3*}O_{k}^{ba}$$a$$b$$c$$\xi$

(a)

$\Re$$\Im$$g(\xi)$$\chi_{1*}O_{i}^{ac}$$b$$\chi_{2*}O_{j}^{cb}$$\chi_{3*}O_{k}^{ba}$

(b)

Figure 8: Conformal transformation that maps triangular shaped disk to upper-
half-plane

In the final step we find a function $g(\xi)$ that maps the disk (Fig. 8(a))
to the UHP (Fig. 8(b)). Given the gauge choice $\theta=\frac{\pi}{4}$, we
write down the function $g(\xi)$ explicitly as,

$g(\xi)=(-i\frac{\xi+(1+i)}{\xi-(1+i)})^{\frac{4}{3}}.$ (73)

After applying the conformal transformation $g(\xi)$, The operators on upper-
half-plane Fig. 8(b) becomes $\chi_{1*}O_{i}^{ac}(0)$,
$\chi_{2*}O_{j}^{cb}(0)$ and $\chi_{3*}O_{k}^{ba}(0)$. So the tensor
components are equal to the following three point function on upper-half-
plane,

$\alpha^{ijk}_{IJK}=\langle\chi_{1*}O_{(I,i)}(0)\chi_{2*}O_{(J,j)}(0)\chi_{3*}O_{(K,k)}(0)\rangle_{UHP},$
(74)

where the $\chi$-functions are defined as,

$\begin{split}&\chi_{1}(z)=g(f(z)+1),\ \ \chi_{2}(z)=g(-if(z)-i),\\\
&\chi_{3}(z)=g(\sqrt{2}e^{i\frac{3\pi}{4}}f(z)).\end{split}$ (75)

As a simple example, consider the three operators all primary fields with
conformal dimension being $\Delta_{1}$, $\Delta_{2}$ and $\Delta_{3}$. Then
the tensor component are calculated as,

$\begin{split}\alpha^{\Delta_{1}\Delta_{2}\Delta_{3}}_{000}\approx
0.515^{\Delta_{1}+\Delta_{2}}0.839^{\Delta_{3}}.\end{split}$ (76)

Method II: Pants-digram— The second strategy we present here is to treat the
triangular region as an amputated pants-diagram, as shown in Fig. 9. The
function $\chi$ is a map from this pants-diagram to UHP.

$z$

Figure 9: pants-diagram

First we adopt the Schwarz–Christoffel transformation to find the map from
upper-half-plane to the pants-diagram,

$h(\xi)=\int^{\xi}dx\frac{\sqrt{2}(5x^{2}-1)^{\frac{1}{4}}}{x(x^{2}-1)}.$ (77)

This function maps the three points $-1$, $0$ and $1$ of the UHP to three
infinities along the legs of pants, where we attach free open string states.
Near these infinities, we have the following expansions:

$\begin{split}&h|_{\xi\rightarrow 1}\sim\ln|\xi-1|,\ \ h|_{\xi\rightarrow
0}\sim-(1+i)\ln|\xi|+i\pi,\\\ &h|_{\xi\rightarrow-1}\sim
i\ln|\xi+1|+\pi.\end{split}$ (78)

These are precisely the functions we can utilize to prepare open string states
at infinities. According to these relations, we define the conformal
functions,

$\begin{split}&\chi_{1}(z):=h^{-1}\left(\ln z\right),\
\chi_{2}(z):=h^{-1}\left(i\ln z+\pi\right),\\\
&\chi_{3}(z):=h^{-1}\left(-(1+i)\ln z+i\pi\right).\end{split}$ (79)

It’s hard to find a concise expression for the inverse function of the map
$h(\xi)$. Instead, we expand this function around the singularties as,

$\begin{split}&\chi_{1}(z)=1+1.628z+2.320z^{2}+3.795z^{3}+O(z^{4})\\\
&\chi_{2}(z)=-1+1.628z-2.320z^{2}+3.795z^{3}+O(z^{4})\\\
&\chi_{3}(z)=0.917z+0.096z^{3}+O(z^{5})\end{split}$ (80)

This allows us to evaluate the tensor numerically. Again, we present the
tensor component for primary fields,

$\alpha^{\Delta_{1}\Delta_{2}\Delta_{3}}_{000}\approx
0.902^{\Delta_{1}+\Delta_{2}}1.354^{\Delta_{3}}.$ (81)

Proof of RG equations— The construction in both approaches guarantee the
equivalence between contracting the tensors and gluing the Riemann surfaces.
The crossing symmetry condition follows direcly from this property.

$=$

(a)

$=$

(b)

Figure 10: (a) Two ways of contracting rank-3 tensors are equivalent. (b) Two
ways of gluing triangular-shaped Riemann surfaces are equivalent.

We can also prove the other properties diagrammatically. Fig. 11 shows proof
of the fixed point tensor property. The rank-4 tensor in Fig. 11(a) is
obtained by contracting four rank-3 tensors. This corresponds to the square-
shaped region in Fig. 11(b). The hole in the center of this region is shrinked
to zero dimension and projected to ground state due to long Eucilidean time
evolution and the closing condition (7).

$:=$$T^{\prime}$

(a)

$=$

(b)

Figure 11: (a) Contraction of four tensors in a loop. (b) Gluing four
triangles to a single Riemann surface.

Finally, Fig. 12 illustrates the process of self-contraction to obtain the
partition function on torus. The contraction of opposite legs corresponds to
fusion of opposing edges of the square-shaped region, as depicted in Fig.
12(b). The outcome of this procedure is a path integral on a torus defected by
a central hole. Again, the boundary state residing on this small hole can be
projected to the ground state as the dimension of the hole is reduced to zero.

(a)

(b)

Figure 12: Contraction the opposite legs of fixed point tensor produces
partition function on torus.

To obtain the closed string spectrum, we tiling multiple tensors into a
cylinder, as shown in Fig. 13. This can be viewed as a transfer matrix
$M_{IJ}$, where $I,J$ are collections of both the descendant indices and
boundary condition indices, namely
$I=\\{i_{1},i_{2},\cdots,i_{n},a_{1},a_{2},\cdots,a_{n}\\}$ and
$J=\\{j_{1},j_{2},\cdots,j_{n},b_{1},b_{2},\cdots,b_{n}\\}$. Then
diagonalizing this matrix produces the desired spectrum. In actual calculation
we used $n=4$.

$M_{IJ}$$:=$$i_{1}$$a_{1}$$b_{1}$$j_{1}$$i_{2}$$a_{2}$$b_{2}$$j_{2}$$a_{3}$$b_{3}$$\cdots$$\cdots$$a_{n}$$b_{n}$$i_{n}$$j_{n}$$a_{1}$$b_{1}$

Figure 13: The transfer matrix

##  Appendix C Iteration relations

Transformation rules — In this section, we present details in mapping
descendant fields under a conformal transformation $\chi(z)$. For more general
descendant fields, we can not give a simple expression for the transformation
coefficients, but deriving an iteration relation is possible.

Suppose that we already know the transformation rule for the operator
$O^{(-k_{l},\cdots,-k_{2},-k_{1})}:=L_{-k_{l}}\cdots L_{-k_{2}}L_{-k_{1}}O$.
The transformation under holomorphic function $\chi(z)$ is written as

$\chi_{*}O^{(-k_{l},\cdots,-k_{2},-k_{1})}(z)=\sum_{\\{k^{\prime}\\}\leq\\{k\\}}H^{k_{l},\cdots,k_{2},k_{1}}_{k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}}(z)O^{(-k^{\prime}_{l},\cdots,-k^{\prime}_{2},-k^{\prime}_{1})}(\eta).$
(82)

where $\eta=\chi(z)$, and the symbol $\\{k\\}$ is a shorthand notation of
$\\{k_{l},\dots,k_{2},k_{1}\\}$. $\\{k^{\prime}\\}\leq\\{k\\}$ means that
$\forall k_{i}\in\\{k\\}$, $k^{\prime}_{i}\leq k_{i}$. Moreover, suppose that
we also know the OPE between $T(z)$ and
$O^{(-k_{l},\cdots,-k_{2},-k_{1})}(z^{\prime})$:

$\begin{split}T(z)O^{(-k_{l},\cdots,-k_{2},-k_{1})}(z^{\prime})=&\sum_{k^{\prime}_{l+1}}(z-z^{\prime})^{k^{\prime}_{l+1}-2}O^{(-k^{\prime}_{l+1},-k_{l},\cdots,-k_{2},-k_{1})}(z^{\prime})\\\
&+\sum_{\\{k^{\prime}\\}\leq\\{k\\}}\frac{C^{k_{l},\cdots,k_{2},k_{1}}_{k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}}}{(z-z^{\prime})^{\sum_{p=1}^{l}k_{p}-\sum_{p=1}^{l}k^{\prime}_{p}+2}}O^{(-k^{\prime}_{l},\cdots,-k^{\prime}_{2},-k^{\prime}_{1})}(z^{\prime}).\end{split}$
(83)

Then we can derive the transformation rule for higher level descendant field
$O^{(-k_{l+1},-k_{l},\cdots,-k_{2},-k_{1})}$ as

$\begin{split}&\chi_{*}O^{(-k_{l+1},-k_{l},\cdots,-k_{2},-k_{1})}(z^{\prime})\\\\[5.0pt]
=&\oint_{z^{\prime}}\frac{dz}{2\pi
i}(z-z^{\prime})^{1-k_{l+1}}\chi_{*}T(z)\chi_{*}O^{(-k_{l},\cdots,-k_{2},-k_{1})}(z^{\prime})\\\\[5.0pt]
=&\oint_{\eta^{\prime}}\frac{d\eta}{2\pi
i}\frac{(z-z^{\prime})^{1-k_{l+1}}}{\chi^{\prime}(z)}[\chi^{\prime}(z)^{2}T(\eta)+\frac{c}{12}\\{\chi(z),z\\}]\sum_{\\{k^{\prime}\\}\leq\\{k\\}}H^{k_{l},\cdots,k_{2},k_{1}}_{k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}}(z^{\prime})O^{(-k^{\prime}_{l},\cdots,-k^{\prime}_{2},-k^{\prime}_{1})}(\eta^{\prime})\\\\[20.0pt]
=&\sum_{\\{k^{\prime}\\}\leq\\{k\\}}\sum_{k^{\prime}_{l+1}=0}^{k_{l+1}}H^{k_{l},\cdots,k_{2},k_{1}}_{k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}}(z^{\prime})a^{k_{l+1}}_{k_{l+1}-k^{\prime}_{l+1}}(z^{\prime})O^{(-k^{\prime}_{l+1},-k^{\prime}_{l},\cdots,-k^{\prime}_{2},-k^{\prime}_{1})}(\eta^{\prime})\\\
+&\sum_{\\{k^{\prime\prime}\\}\leq\\{k\\}}[\sum_{\\{k^{\prime\prime}\\}\leq\\{k^{\prime}\\}\leq\\{k\\}}H^{k_{l},\cdots,k_{2},k_{1}}_{k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}}(z^{\prime})a^{k_{l+1}}_{k_{l+1}+\sum
k^{\prime}-\sum
k^{\prime\prime}}(z^{\prime})C^{k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}}_{k^{\prime\prime}_{l},\cdots,k^{\prime\prime}_{2},k^{\prime\prime}_{1}}]O^{(-k^{\prime\prime}_{l},\cdots,-k^{\prime\prime}_{2},-k^{\prime\prime}_{1})}(\eta^{\prime})\\\
&+\frac{c}{12(k_{l+1}-2)!}(\frac{d}{dz})^{k_{l+1}-2}[\frac{\\{\chi(z),z\\}}{\chi^{\prime}(z)}]\sum_{\\{k^{\prime}\\}\leq\\{k\\}}H^{k_{l},\cdots,k_{2},k_{1}}_{k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}}(z^{\prime})O^{(-k^{\prime}_{l},\cdots,-k^{\prime}_{2},-k^{\prime}_{1})}(\eta^{\prime}),\end{split}$
(84)

where the coefficients $a^{n}_{m}$’s are defined by

$\chi^{\prime}(z)(z-z^{\prime})^{1-n}=(\eta-\eta^{\prime})^{1-n}\sum_{m=0}^{\infty}a_{m}^{n}(z^{\prime})(\eta-\eta^{\prime})^{m}.$
(85)

Comparing with the definition of these transformation coefficients we conclude
that,

$\begin{split}H^{k_{l+1},k_{l},\cdots,k_{2},k_{1}}_{k^{\prime}_{l+1},k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}}(z^{\prime})=&H^{k_{l},\cdots,k_{2},k_{1}}_{k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}}(z^{\prime})a^{k_{l+1}}_{k_{l+1}-k^{\prime}_{l+1}}(z^{\prime})\\\
H^{k_{l+1},k_{l},\cdots,k_{2},k_{1}}_{k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}}(z^{\prime})=&\sum_{\\{k^{\prime}\\}\leq\\{k^{\prime\prime}\\}\leq\\{k\\}}a^{k_{l+1}}_{k_{l+1}+\sum
k^{\prime\prime}-\sum
k^{\prime}}(z^{\prime})H^{k_{l},\cdots,k_{2},k_{1}}_{k^{\prime\prime}_{l},\cdots,k^{\prime\prime}_{2},k^{\prime\prime}_{1}}(z^{\prime})C^{k^{\prime\prime}_{l},\cdots,k^{\prime\prime}_{2},k^{\prime\prime}_{1}}_{k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}}\\\
+&\frac{c}{12(k_{l+1}-2)!}(\frac{d}{dz})^{k_{l+1}-2}[\frac{\\{\chi(z),z\\}}{\chi^{\prime}(z)}]|_{z=z^{\prime}}H^{k_{l},\cdots,k_{2},k_{1}}_{k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}}(z^{\prime}).\end{split}$
(86)

OPE coefficients — Now we derive an iteration relation of the OPE coefficient
between energy momentum tensor $T(z)$ and a general descendant field
$O^{(-k_{l},\cdots,-k_{2},-k_{1})}(z^{\prime})$. The OPE coefficients are
denoted by the symbol
$C^{k_{l},\cdots,k_{2},k_{1}}_{k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}}$
defined as,

$\begin{split}T(z)O^{(-k_{l},\cdots,-k_{2},-k_{1})}(z^{\prime})=&\sum_{k^{\prime}_{l+1}=1}^{\infty}(z-z^{\prime})^{k^{\prime}_{l+1}-2}O^{(-k^{\prime}_{l+1},-k_{l},\cdots,-k_{2},-k_{1})}(z^{\prime})\\\
&+\sum_{\\{k^{\prime}\\}\leq\\{k\\}}\frac{C^{k_{l},\cdots,k_{2},k_{1}}_{k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}}}{(z-z^{\prime})^{\sum_{p=1}^{l}k_{p}-\sum_{p=1}^{l}k^{\prime}_{p}+2}}O^{(-k^{\prime}_{l},\cdots,-k^{\prime}_{2},-k^{\prime}_{1})}(z^{\prime}).\end{split}$
(87)

Again, we use the symbol $\\{k\\}$ to denote the set
$\\{k_{l},\cdots,k_{2},k_{1}\\}$. $\\{k^{\prime}\\}\leq\\{k\\}$ means that
$k^{\prime}_{p}\leq k_{p}$ for any $1\leq p\leq l$.

For the higher level descendants $O^{(-k_{l+1},-k_{l},\cdots,-k_{2},-k_{1})}$,
its operator product with $T(z)$ is,

$\begin{split}&T(z)O^{(-k_{l+1},-k_{l},\cdots,-k_{2},-k_{1})}(z^{\prime})\\\
=&\oint_{z^{\prime}}\frac{dw}{2\pi
i}(w-z^{\prime})^{1-k_{l+1}}T(z)T(w)O^{(-k_{l},\cdots,-k_{2},-k_{1})}(z^{\prime})\\\
=&-[\oint_{z}\frac{dw}{2\pi
i}(w-z^{\prime})^{1-k_{l+1}}T(w)T(z)]O^{(-k_{l},\cdots,-k_{2},-k_{1})}(z^{\prime})\\\
&+\oint_{z^{\prime}}\frac{dw}{2\pi
i}(w-z^{\prime})^{1-k_{l+1}}T(w)[T(z)O^{(-k_{l},\cdots,-k_{2},-k_{1})}(z^{\prime})]\\\
=&\sum_{k^{\prime}_{l+1}=1}^{k_{l+1}}(2k_{l+1}-k^{\prime}_{l+1})\frac{O^{(-k^{\prime}_{l+1},-k_{l},\cdots,-k_{2},-k_{1})}}{(z-z^{\prime})^{k_{l+1}-k^{\prime}_{l+1}+2}}\\\
&+\sum_{\\{k^{\prime}\\}\leq\\{k\\}}C^{k_{l},\cdots,k_{2},k_{1}}_{k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}}\frac{O^{(-k_{l+1},-k^{\prime}_{l},\cdots,-k^{\prime}_{2},-k^{\prime}_{1})}(z^{\prime})}{(z-z^{\prime})^{\sum_{p=1}^{l}k_{p}-\sum_{p=1}^{l}k^{\prime}_{p}+2}}\\\
&+\sum_{\\{k^{\prime}\\}\leq\\{k\\}}(2k_{l+1}+\sum_{p=1}^{l}k_{p}-\sum_{p=1}^{l}k_{p}^{\prime})C^{k_{l},\cdots,k_{2},k_{1}}_{k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}}\frac{O^{(-k^{\prime}_{l},\cdots,-k^{\prime}_{2},-k^{\prime}_{1})}(z^{\prime})}{(z-z^{\prime})^{k_{l+1}+\sum_{p=1}^{l}k_{p}-\sum_{p=1}^{l}k_{p}^{\prime}+2}}\\\
&+\frac{c}{12}k_{l+1}(k_{l+1}^{2}-1)\frac{O^{(-k_{l},\cdots,-k_{2},-k_{1})}}{(z-z^{\prime})^{k_{l+1}+2}}+\cdots\end{split}$
(88)

where the ellipsis denotes any combination of descendant operators at level
higher than $\sum_{p=1}^{l+1}k_{p}$. We neglected them simply because we
already know their coefficients.

Comparing with the definition of
$C^{k_{l+1},k_{l},\cdots,k_{2},k_{1}}_{k^{\prime}_{l+1},k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}}$,
we conclude that,

$\begin{split}&C^{k_{l+1},k_{l},\cdots,k_{2},k_{1}}_{k^{\prime}_{l+1},k_{l},\cdots,k_{2},k_{1}}=2k_{l+1}-k^{\prime}_{l+1},\qquad
for\ 1\geq k^{\prime}_{l+1}<k_{l+1}\\\
&C^{k_{l+1},k_{l},\cdots,k_{2},k_{1}}_{k_{l+1},k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}}=C^{k_{l},\cdots,k_{2},k_{1}}_{k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}},\qquad
for\ \\{k^{\prime}\\}<\\{k\\}\\\
&C^{k_{l+1},k_{l},\cdots,k_{2},k_{1}}_{k_{l+1},k_{l},\cdots,k_{2},k_{1}}=k_{l+1}+C^{k_{l},\cdots,k_{2},k_{1}}_{k_{l},\cdots,k_{2},k_{1}}\\\
&C^{k_{l+1},k_{l},\cdots,k_{2},k_{1}}_{k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}}=(2k_{l+1}+\sum_{p=1}^{l}k_{p}-\sum_{p=1}^{l}k^{\prime}_{p})C^{k_{l},\cdots,k_{2},k_{1}}_{k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}}+\frac{c}{12}k_{l+1}(k_{l+1}^{2}-1)\delta^{\\{k\\}}_{\\{k^{\prime}\\}}\qquad\\\
\end{split}$ (89)

Correlation function — We can also derive an iteration equation of
correlators. Suppose that we already know all the correlators of lower level
descendants, we can derive the higher level ones using this equation.

To simplify the notation, we use $O^{(-\vec{k})}$ to denote
$O^{(-k_{l},\cdots,-k_{2},-k_{1})}$, and the OPE coefficient
$C^{\vec{k}}_{\vec{k^{\prime}}}$ to denote
$C^{k_{l},\cdots,k_{2},k_{1}}_{k^{\prime}_{l},\cdots,k^{\prime}_{2},k^{\prime}_{1}}$.
Then we will show that the correlator $\langle
L_{-m}O_{1}^{(-\vec{k})}(x)O_{2}^{(-\vec{p})}(y)O_{3}^{(-\vec{q})}(z)\rangle$
can be written as a linear combination of simpler correlators of the form
$\langle
O_{1}^{(-\vec{k^{\prime}})}(x)O_{2}^{(-\vec{p^{\prime}})}(y)O_{3}^{(-\vec{q^{\prime}})}(z)\rangle$,
with $\vec{k^{\prime}}\leq\vec{k}$, $\vec{p^{\prime}}\leq\vec{p}$ and
$\vec{q^{\prime}}\leq\vec{q}$ (in the sense of $\\{k^{\prime}\\}\leq\\{k\\}$
defined in the previous sections). We start from the equation,

$\begin{split}&\langle
L_{-m}O_{1}^{(-\vec{k})}(x)O_{2}^{(-\vec{p})}(y)O_{3}^{(-\vec{q})}(z)\rangle\\\
=&\oint_{x}\frac{dw}{2\pi
i}(w-x)^{1-m}\langle[T(w)O_{1}^{(-\vec{k})}(x)]O_{2}^{(-\vec{p})}(y)O_{3}^{(-\vec{q})}(z)\rangle\\\
=&-\oint_{y}\frac{dw}{2\pi i}(w-x)^{1-m}\langle
O_{1}^{(-\vec{k})}(x)[T(w)O_{2}^{(-\vec{p})}(y)]O_{3}^{(-\vec{q})}(z)\rangle\\\
&-\oint_{z}\frac{dw}{2\pi i}(w-x)^{1-m}\langle
O_{1}^{(-\vec{k})}(x)O_{2}^{(-\vec{p})}(y)[T(w)O_{3}^{(-\vec{q})}(z)]\rangle\\\
\end{split}$ (90)

Using the OPE,

$T(w)O^{(-\vec{p})}(y)=\sum_{\vec{p^{\prime}}\leq\vec{p}}C^{\vec{p}}_{\vec{p^{\prime}}}\frac{O^{(-\vec{p^{\prime}})}(y)}{(w-y)^{|\vec{p}|-|\vec{p^{\prime}}|+2}}+\frac{\partial
O^{(-\vec{p})}(y)}{w-y}+reg.,$ (91)

where $reg.$ means the regular terms in the limit $w\rightarrow y$, we can
expand the expression,

$\begin{split}&\oint_{y}\frac{dw}{2\pi i}(w-x)^{1-m}\langle
O_{1}^{(-\vec{k})}(x)[T(w)O_{2}^{(-\vec{p})}(y)]O_{3}^{(-\vec{q})}(z)\rangle\\\
=&\frac{\partial_{y}}{(y-x)^{m-1}}\langle
O_{1}^{(-\vec{k})}(x)O_{2}^{(-\vec{p})}(y)O_{3}^{(-\vec{q})}(z)\rangle\\\
&+(-1)^{|\vec{p}|-|\vec{p^{\prime}}|-1}\sum_{\vec{p^{\prime}}\leq\vec{p}}\frac{C^{\vec{p}}_{\vec{p^{\prime}}}}{(y-x)^{m+|\vec{p}|-|\vec{p^{\prime}}|}}\frac{(|\vec{p}|-|\vec{p^{\prime}}|+m-1)!}{(|\vec{p}|-|\vec{p^{\prime}}|+1)!(m-2)!}\langle
O_{1}^{(-\vec{k})}(x)O_{2}^{(-\vec{p^{\prime}})}(y)O_{3}^{(-\vec{q})}(z)\rangle.\end{split}$
(92)

The other term is calculated similarly. So the correlator is reduced to
combinations of simpler ones:

$\begin{split}&\langle
L_{-m}O_{1}^{(-\vec{k})}(x)O_{2}^{(-\vec{p})}(y)O_{3}^{(-\vec{q})}(z)\rangle\\\
=&-\frac{\partial_{y}}{(y-x)^{m-1}}\langle
O_{1}^{(-\vec{k})}(x)O_{2}^{(-\vec{p})}(y)O_{3}^{(-\vec{q})}(z)\rangle\\\
&+(-1)^{|\vec{p}|-|\vec{p^{\prime}}|}\sum_{\vec{p^{\prime}}\leq\vec{p}}\frac{C^{\vec{p}}_{\vec{p^{\prime}}}}{(y-x)^{m+|\vec{p}|-|\vec{p^{\prime}}|}}\frac{(|\vec{p}|-|\vec{p^{\prime}}|+m-1)!}{(|\vec{p}|-|\vec{p^{\prime}}|+1)!(m-2)!}\langle
O_{1}^{(-\vec{k})}(x)O_{2}^{(-\vec{p^{\prime}})}(y)O_{3}^{(-\vec{q})}(z)\rangle\\\
&+(y\rightarrow z,\ \vec{p}\rightarrow\vec{q}).\end{split}$ (93)


# Evaluation of autonomous systems under data distribution shifts

Daniel Sikar
City, University of London
Northampton Square, London EC1V 0HB, UK
<EMAIL_ADDRESS>Artur Garcez
City, University of London
Northampton Square, London EC1V 0HB, UK
<EMAIL_ADDRESS>
###### Abstract

We posit that data can only be safe to use up to a certain threshold of the
data distribution shift, after which control must be relinquished by the
autonomous system and operation halted or handed to a human operator. With the
use of a computer vision toy example we demonstrate that network predictive
accuracy is impacted by data distribution shifts and propose distance metrics
between training and testing data to define safe operation limits within said
shifts. We conclude that beyond an empirically obtained threshold of the data
distribution shift, it is unreasonable to expect network predictive accuracy
not to degrade.

##  1 Introduction

The development of autonomous systems such as self-driving cars is motivated
by a number of goals, of a practical, safety, public interest and economic
nature. From a practical perspective, the goal is ”to transport people from
one place to another without any help from a driver” [23]. From a public
health perspective, to transform the current approach to automotive safety
from reducing injuries after collisions to complete collision prevention [11].
From a public interest and economic perspective, AV fleets allow for new
shared autonomous mobility business models [31] though shared autonomous
electric vehicle (SAEVs) fleets [24]. Shared Autonomous Vehicles (SAVs) have
gained significant public interest as a possible less expensive, safer and
more efficient version of today’s transportation networking companies (TNCs)
and taxis.

This perceived superiority to human drivers is attributed to high-performance
computing that allows AVs to process, learn from and adjust their guidance
systems according to changes in external conditions at much faster rates than
the typical human driver [38].

The presence in public spaces of autonomous systems driven by AI is a concern.
Although models such as convolutional neural networks have been successfully
used to solve problems applied to Computer Vision, the ability to generalize
and robustness of such architectures has been increasingly scrutinised.

Zhang _et al_. [39] debated the need to rethink generalization, by
demonstrating how traditional benchmarking approaches fail to explain why
large neural networks generalize well in practice. By randomizing target
labels, the experiments show that state-of-the-art convolutional neural
networks for image classification trained with SGD (stochastic gradient
descent) are large enough to fit a random labelling of the training data. This
is achieved with a simple two-layer neural network, which presents a ”perfect
finite sample expressivity” once the number of parameters is greater than the
number of data points as often is the case with CNNs. This poses a challenge,
in the context of overfitting, for autonomous systems such as self-driving
cars, specially if relying only on images for perception of the surrounding
environment. Since testing real life scenarios is not practical due to the
associated cost and risk, we examine the use of game engines [8], which are
capable of generating labeled datasets and realistic environments where the
autonomous vehicle may be tested.

In this study, data distribution shifts are examined in the context of testing
data presented to a neural network trained on a known data distribution, that
is, with the weights and biases adjusted while minimizing an error function.

When the shift is of a sufficient quantity, relative to the training data, a
number of terms exist in the literature to express and deal with the shift,
such as anomaly detection [7, 6, 26, 32], out-of-distribution data [12, 33,
17, 16, 14], novelty detection [36, 27, 18, 9, 28], outlier detection [3, 15,
1] and covariate shift [35, 34, 30, 4]. In section 3 we propose metrics to
quantify the distance between training and testing data. We use a game engine
to generate our training data and train a model based on SOTA self-driving
architectures. Our aim is to determine safe i.e. acceptable shift quantities,
where predictions made by the model, given the shifted data, are safe to use,
as exemplified in Figure 1.

Figure 1: Simulated accident in the CARLA Simulator Town 10, where excessive
brightness i.e. high RGB values cause predictive accuracy of self-driving
models to degrade

##  2 Related Work

We examined literature related to out-of-distribution (OOD) data, and methods
for identifying such data and/or making neural networks robust to its adverse
and unwanted effect, such that network predictive accuracy is not affected by
data in the OOD regime.

Fort _et al_. [12] are motivated by out-of-distribution detection (OOD) and
point out that is high-stake applications such as healthcare and self-driving.
Techniques for detecting OOD inputs using neural networks include MSP (maximum
over softmax probabilities) proposed by Hendrycks _et al_. [14] i.e.
$score_{msp}(x)=max_{c=1,...,K}p(y=c|x)$, showing that the probability
prediction for correct examples tends to be higher than of incorrect and out-
of-distribution examples. Capturing prediction probability statistics about
in-sample examples is in most cases sufficient for detecting abnormal
examples.

Lee _et al_. [21] argue that Mahalanobis distance is more effective than
Euclidean distance in various OOD and adversarial data detection tasks, given
a defined distance-based confidence score $M(x)$ that is the Mahalanobis
distance between a test sample x and the closest class-conditional Gaussian
distribution:

$M(x)=\max\limits_{c}-(f(x)-\hat{\mu}_{c})^{T}\hat{\sum}^{-1}(f(x)-\hat{\mu}_{c}).$

Figure 2: LeNet predictive accuracy change under rotation (left) and
translation shift (right)

Noting that the metric corresponds to measuring the log of the probability
densities of the test sample. The value $\hat{\mu}_{c}$ represents the
empirical class mean while $\hat{\sum}$ the class covariance of the training
samples. The algorithm uses the weights and biases of the softmax classifier,
that is the penultimate (fully connected, dense) layer and assumes that the
class-conditional distribution follows a multivariate Gaussian distribution.

Shalev _et al_. [33] propose the use of a number of distinct word embeddings
as supervisors in ensemble models, thus generating shared representations. A
semantic structure is utilised for word embeddings to produce semantic
predictions, while the L2-norm of the output vectors is employed to detect OOD
inputs, testing the approach to detect adversarial and incorrectly classified
examples. Cosine distance is used in the space $Z$ to measure the distance
between two embeddings u,v $\in Z$:

$d_{cos}\mathbf{(u,v)=\frac{1}{2}\bigg{(}1-\frac{u\cdot v}{\lVert
u\lVert\lVert v\lVert}\bigg{)}}$

When the cosine distance is close to 0, the labels are considered similar,
when it is close to 1, the labels are considered to be semantically far apart.

Huang _et al_. [17] propose a scoring function based on group softmax for
different categories with the Minimum Others Score (MOS) function, allowing
differentiation between in and out-of-distribution data. The key observation
being that a pre-defined category others carries useful information for the
likelihood of a given imaged being OOD w.r.t. each group. An OOD input is
mapped to others in all groups and the lowest others score in all groups
determines OOD images. The MOS OOD scoring function being:

$S_{MOS}(\mathbf{x})=-\min\limits_{1\leq k\leq K}p_{others}^{k}(\mathbf{x})$

The sign is negated such that $S_{MOS}(\mathbf{x})$ is higher for in-
distribution and lower of out-of-distribution data.

Liang _et al_. [22] propose the Out-of-distribution detector (ODIN) to
distinguish in and out of distribution images on a pre-trained neural network
using a temperature scaling parameter $T$. The problem statement is defined by
$P_{\mathbf{x}}$ and $Q_{\mathbf{x}}$ denoting two distinct data distributions
defined on the image space $\chi$, where $P_{\mathbf{x}}$ are the in-
distribution and $Q_{\mathbf{x}}$ are the out-of-distribution images. During
training images are drawn from $P_{\mathbf{x}}$. If testing images are drawn
from a mixture distribution, can a distinction be made between in an out-of-
distribution images? For each input the network generates a label prediction
$\hat{y}(x)=\arg\max_{i}S_{i}(x;T)$ by computing the softmax output for each
class, using the temperature scaling term $T$ set to 1 during training:

$S_{i}(x;T)=\frac{\exp(f_{i}(x)/T)}{\sum_{j=1}^{N}\exp(f_{j}(x)/T)}$

Temperature scaling perhaps resonates with von Neumann [37] whose conviction
was that error should be treated and subject to themodynamical methods and
theory. In addition to temperature scaling, a small perturbation $\epsilon$,
inspired by the idea of adversarial examples [13], is added to the inputs. The
inputs are classified as in-distribution if the softmax score is greater than
threshold $\delta$, out-of-distribution otherwise. The out-of-distribution can
therefore be defined as a function of the input $x$, the perturbation
$\epsilon$, the temperature scaling term $T$ and the threshold $\delta$:

$g(x;\delta,T,\epsilon)=\begin{cases}1&\text{if
$\max_{i}p(\bar{x};T)\leq\delta$ },\\\ 0&\text{if
$\max_{i}p(\bar{x};T)>\delta$ }.\end{cases}$

Hsu _et al_. [16] use a combination of ODIN and Mahalanobis distance for OOD
data detection. Pointing out that ODIN requires OOD data to tune
hyperparameters for temperature scaling and input pre-processing, their
Generalized ODIN method addresses the problem of learning without OOD data,
adding a variable to declaring if the data is in or out of distribution,
closely related to the temperature scaling term in ODIN, except that the
variable depends on input instead of tuned hyperparameters.

Ovadia _et al_. [25] argue that high stake applications such as medical
diagnoses and self-driving require in addition to predictive accuracy,
quantification of predictive uncertainty, that is, class predictions and
confidence values. The study concluded that accuracy and confidence degrade
with dataset shifts (rotated or horizontally translated images), and that
ensembles help with robustness to dataset shifts. Figure 2 shows accuracy
shift as a function of rotated and translated MNIST datasets, as predicted
with the LeNet architecture [19] with standard training, validation, testing
and hyperparameter tuning protocols, and seven variants including vanilla,
ensemble and dropout. The work presented by Ovadia _et al_. [25] with respect
to accuracy and dataset shifts is the one we find our study most aligned with.

##  3 Methods

Figure 3: Two sets of images and pixel intensity value histograms, where the
set on the left is a negative shift (pixel intensity values decrease) and the
set on the right is a positive shift (pixel intensity values increase)

###  3.1 Applying data distribution shifts to RGB images

We interpret an RGB image as a data distribution of individual pixel intensity
values. This can be represented as a histogram where the sum of bucket counts
(distinct pixel values aggregated counts) is given in Equation 6, where the
number of buckets is equal to the number of values the pixel may take, i.e.
the pixel datatype range of values. Distribution shifts therefore will cause
counts to increase and decrease in each bucket, according to the direction
(left, negative, right, positive) of distribution shift. The distribution is
shifted by uniformly incrementing or decrementing each pixel intensity value
by the same quantity. We define the RGB pixel intensity shift function as:

$RGB_{is}(I,S)=\begin{cases}I_{jk}+S,&\text{if $I_{imin}\leq I_{jk}+S\leq
I_{imax}$ }.\\\ I_{imin},&\text{if $I_{imin}>I_{jk}+S$}.\\\ I_{imax},&\text{if
$I_{max}<I_{jk}+S$}.\end{cases}$ (1)

where $I$ is the RGB image pixel matrix, $S$ is the shift quantity to be added
to each pixel, $j$ is the number of dimensions in $I$ and $k$ is the number of
elements in $j$. $S$ takes both negative and positive values. Negative values
cause a left shift to the distribution and decrease the pixel intensity value.
Output values can be in the range of, and including, $I_{imin},I_{imax}$
defined respectively as the minimum and maximum values the pixel data type can
represent. A 24-bit RGB image, represented by 3 bytes, one for every red,
green and blue channel, can store values between 0 and 255, in this case, if
the value of $I_{jk}$ is 100 and a shift of -120 is applied, $I_{jk}$ will
take value 0. If $I_{jk}$ is 170 and a shift of 100 is applied $I_{jk}$ will
take value 255. We say in these cases that the pixel is saturated resulting
and darker or brighter images. The result and effect can be observed in Figure
3.

Figure 4: Left to right, the SDSandbox self-driving neural network training
application, the Generated Track circuit, a steering angle histogram showing
the distribution of steering angles when going around the track clockwise

###  3.2 Image pre-processing

One of the key pre-processing steps used in SOTA models such as [20] is moving
images from RGB to YUV space. The authors do not discuss the motivation behind
this procedure. [YUV article] points out that YUV is more related to human
vision. It can be said that YUV reduces the information content of the image.
We observed that when no datashift exists, the YUV mean is lower than the RGB
mean, and we decrease the RGB mean. we observed that after that when with a
mean shift of minus 5 the RGB and YUV means are equal. Increased negative
shift leads to an increase in the distance between RGB and YUV means.

We start with an image. We take the mean value of all combined RGB channel
values, we then move the image to YUV space and back to RGB space, then again
compute the mean. When there is no shift in pixel intensity value, the YUV
image has a lower mean than the corresponding RGB mean. Depending on the
shift, and the image becoming darker, or brighter, the YUV mean may be higher
(for darker images, or lower (for brighter images). The main takeaway being
the mean delta is lower in YUV space, creating lesser variability in the data
and validating the use of YUV images to train self-driving CNNs and
potentially other applications and architectures. This finding is a side-
effect, and not the main focus of this study.

One important aspect of training data for AVs is data pre-processing e.g.
cropping such that most relevant section of the image to the self-driving task
is kept. In the example shown, 25 rows of pixels were removed from the bottom
and 60 rows of pixels were removed from the top, resulting in a 75h x 320w
pixel image. The image is then resized to 66h x 200w, the geometry to be
presented to network in the example shown. Finally the image is moved from RGB
to YUV space. In the RGB scheme, each pixel is represented by three channel
intensities of red, green and blue. In YUV, also referred to as YCbCr space,
each pixel is represented by Y (luma), U (Cb - luminance value subtracted from
blue channel) and V (Cr - luminance value subtracted from red channel). It is
a ”lossy” process which degrades the data, and originally developed for colour
to black-and-white television backward-compatibility. Moving the image from
RGB to YUV space has been demonstrated to give ”better subjective image
quality than the RGB color space”, being better for computer vision
”implementations than RGB due to the perceptual similarities to the human
vision” [29]. This scheme was used in previous AV data pre-processing
pipelines e.g. Dave [20] and PilotNet [5].

###  3.3 Game engines

Game engines are ubiquitous and the concept of using virtual environment in
lieu of real ones, to train autonomous systems or humans, with the benefits of
cost savings and security, is long established. We used a Unity game engine
based self driving training and evaluation system called SDSandbox, short for
self-driving sandbox. Figure 4 shows left to right the SDSandbox user
interface, with options including to drive, self-drive with a PID algorithm
and self-drive with a neural network. In the middle is the Generated Track and
on the left a histogram of steering angles for a dataset generated by the
SDSandbox driving around the generated track, where the autonomous vehicle
drives clockwise around the track, and the start is on the top left of the
circuit.

The Unity SDSandbox can generate a number of circuits. We obtained a labelled
dataset of 20,061 frames for the Generated Road circuit and a labelled dataset
of 1,394 frames for the Generated Track circuit, each frame being a jpg image
of size 160 wide by 120 high, with a corresponding json file containing the
steering angle and throttle applied at the time the image was saved. We
trained networks to self-drive in the SDSandbox environment, a video of two
Donkey Cars [10] being driven by distinct trained networks, on a similar
random Generated Track, can be seen at [2]. Note the video is different in
that the frames are generated by the video game in real time, and the
simulated world adjusted accordingly, depending on the steering and position
of simulated vehicle on the road. With the labelled datasets, the world is not
adjusted, as the frames have been generated and saved previously, and whatever
steering prediction the model generates, it will not actually change the next
frame, as it would in realtime.

Figure 5: Plots of ground truth (SDSandbox PID steering output) and nvidia2
network predictions for images with pixel value intensity shifts of 40, 80 and
120, where the steering error (st. err.) for RGB shift is the MAE

###  3.4 Metrics

We examine 4 metrics for defining network prediction error, and 3 metrics for
defining the distance between training and testing data distributions.

####  3.4.1 Error functions

$MAE=\frac{1}{N}\sum\limits_{i=1}^{N}|y_{i}-\hat{y}_{i}|$ (2)

$MAPE=\frac{1}{N}\sum\limits_{i=1}^{N}\big{|}\frac{y_{i}-\hat{y}_{i}}{y_{i}}\big{|}$
(3)

$RMSE=\sqrt{\frac{1}{N}\sum\limits_{i=1}^{N}(y_{i}-\hat{y}_{i})^{2}}$ (4)

$MSE=\frac{1}{N}\sum\limits_{i=1}^{N}(y_{i}-\hat{y}_{i})^{2}$ (5)

Where MAE is the Mean Absolute Error, MAPE is Mean Absolute Percentage Error,
MSE is Mean Squared Error and RMSE is Root Mean Square Error. Where MSE is
quadratic, i.e., penalising larger errors while the other three are linear.
$y$ is the ground truth while $\hat{y}$ is the network prediction value, and
$N$ is the total number of images presented to the network, over which the
error is computed

####  3.4.2 Distance functions

We adapt three known distance functions to quantify distance between two
images, where the first is assumed to have been used during training, that is
of a known accuracy and not have expected to generate a steering value that
would have caused the simulated vehicle to drive off the road.

We consider an RGB image as distributions, as represented in a histogram,
corresponding to each red, green and blue channels where the ground truth and
shifted RGB histogram bin counts are $P,Q$ respectively.

We use the product $t_{p}$ obtained in equation 6 to normalise our bin counts
$P,Q$ where $N$ is the number of dimensions in the image array $I$ and $I_{n}$
is the number of elements in each image dimension $n$.

$t_{p}=\prod\limits_{n=1}^{N}I_{n}$ (6)

Bhattacharyya RGB distance we define as the negative natural logarithm of the
Batthacharyya RGB coefficient:

$D_{B}rgb(P,Q)=-ln(BCrgb(P,Q)$ (7)

where the Batthacharyya coefficient is defined as:

$BC(P,Q)=\sum\limits_{x\in\mathcal{X}}\sqrt{P(x)Q(x)}$ (8)

for every $x$ on the same domain $\mathcal{X}$. Since we are dealing with
multiple dimensions (channels), in our Batthacharyya RGB coefficient we
explicitly divide our histogram bin counts by $t_{p}$, and the expression
becomes:

$\begin{split}BCrgb(P,Q)&=\sum\limits_{x\in\mathcal{X}}\sqrt{\frac{P(x)}{t_{p}}\frac{Q(x)}{t_{p}}}\\\
&=\sum\limits_{x\in\mathcal{X}}\sqrt{\frac{P(x)Q(x)}{t_{p}^{2}}}\\\
&=\sum\limits_{x\in\mathcal{X}}\frac{\sqrt{P(x)Q(x)}}{t_{p}}\\\
&=\sum\limits_{x\in\mathcal{X}}\sqrt{P(x)Q(x)}\;t_{p}^{-1}\end{split}$ (9)

Relative entropy, also known as Kullback-Leibler divergence, for our specific
use with RGB images we express as:

$\begin{split}D_{\text{KLrgb}}(P\parallel
Q)&=\sum\limits_{x\in\mathcal{X}}P(x)t_{p}^{-1}\log\left(\frac{(P(x)+\epsilon)t_{p}^{-1}}{(Q(x)+\epsilon)t_{p}^{-1}}\right)\\\
&=\sum\limits_{x\in\mathcal{X}}P(x)t_{p}^{-1}\log\left(\frac{P(x)+\epsilon}{Q(x)+\epsilon}\right)\end{split}$
(10)

We add a small positive term $\epsilon$ to $P(x),Q(x)$ to avoid division by
zero, and also to avoid taking the logarithm base 10 of zero which is
undefined.

We note that the KL Divergence in addition to, for our purposes, being
fragile, is also asymmetric. that is the divergence from P to Q is not the
same as the divergence from Q to P.

Note we use discrete rather than the continuous values. That is, we use the
original byte value for each channel and the discrete relative entropy
equation. If we convert the distribution to a PDF, that is, with all
aggregated counts summing to one, we would use the continuous equation. The
PDF seems like the better way to go as it is a normalised value, though if we
are dealing with a known network, the input is always of the same the same
dimensions and can be considered normative, that is the RGB mean is
constrained by, and bound to, the input size. Histogram RGB intersection
between two distributions we define as:

$H_{irgb}(P,Q)=t_{p}^{-1}\sum\limits_{x\in\mathcal{X}}argmin(P(x),Q(x))$ (11)

If $P,Q$ are the same for every $x$, then the summation will be equal to the
product $t_{p}$, resulting in 1, meaning there is a complete overlap between
the two distributions. If $argmin(P(x),Q(x))$ is equal to zero for every $x$,
then the summation will be equal to zero meaning there is no overlap between
the histograms.

Since the histogram RGB intersection does not take square roots or logs, it is
the most computationally efficient of the three distance metrics we
investigate.

##  4 Experiments

A total of 10,360,826 predictions were made for our datashift and accuracy
analysis. Using both datasets, performing a total of 241 laps on each of the
Generated Track and Generated Road circuits, to obtain MAE, MAPE and MSE for
all steering predictions, similar to shown in Figure 5, where a shift of
positive 120 (red plot) clearly is not steering, and positive 80 is on the
limit of following the ground truth PID steering angle. The plot shows that
the limit of safety in this case is between positive 40 and positive 80, with
approximately the same for negative shifts, which are not shown here due to
space constraints.

###  4.1 Distribution shift metrics

We started our analysis by choosing one random image (1105_cam-
image_array_.jpg) from the Generated Track dataset. Using the equations
described in section 3.4.2, we computed the distance from the image to itself,
which is one for the Histogram Intersection (complete overlap), and zero
(distance) for Relative Entropy and Bhatthacharrya Distance. We then apply
positive and negative shifts, for a total of 241 shifts, from negative 120 to
positive 120.

Figure 6: Histogram Intersection, Relative Entropy and Bhattacharyya Distance
safe limits for RGB image pixel intensity value shifts Table 1: Histogram
Intersection in RGB space between random Generated Road frames

| ID1 | ID2 | -120 | -80 | -40 | 0 | 40 | 80 | 120
---|---|---|---|---|---|---|---|---|---
1 | 592 | 503 | 0.10 | 0.20 | 0.45 | 0.83 | 0.63 | 0.28 | 0.11
2 | 863 | 825 | 0.08 | 0.21 | 0.55 | 0.80 | 0.37 | 0.16 | 0.07
3 | 912 | 879 | 0.09 | 0.18 | 0.43 | 0.88 | 0.35 | 0.16 | 0.08
4 | 1096 | 410 | 0.14 | 0.33 | 0.69 | 0.62 | 0.29 | 0.16 | 0.08
5 | 519 | 772 | 0.11 | 0.29 | 0.62 | 0.81 | 0.44 | 0.18 | 0.09
6 | 365 | 1091 | 0.10 | 0.19 | 0.32 | 0.65 | 0.68 | 0.35 | 0.14
7 | 1082 | 1094 | 0.11 | 0.20 | 0.41 | 0.92 | 0.42 | 0.22 | 0.12
8 | 811 | 1300 | 0.07 | 0.15 | 0.26 | 0.63 | 0.72 | 0.32 | 0.17
9 | 146 | 1004 | 0.10 | 0.19 | 0.40 | 0.88 | 0.44 | 0.21 | 0.12
10 | 1199 | 593 | 0.15 | 0.34 | 0.70 | 0.65 | 0.33 | 0.19 | 0.10
11 | 797 | 1157 | 0.07 | 0.15 | 0.27 | 0.61 | 0.71 | 0.33 | 0.14
12 | 350 | 173 | 0.08 | 0.16 | 0.29 | 0.65 | 0.72 | 0.36 | 0.15
13 | 407 | 1260 | 0.07 | 0.15 | 0.27 | 0.61 | 0.72 | 0.38 | 0.17
14 | 157 | 1036 | 0.10 | 0.20 | 0.42 | 0.89 | 0.36 | 0.16 | 0.07
15 | 331 | 1139 | 0.09 | 0.18 | 0.31 | 0.65 | 0.69 | 0.36 | 0.16
16 | 623 | 1259 | 0.09 | 0.17 | 0.31 | 0.63 | 0.72 | 0.38 | 0.17
17 | 235 | 566 | 0.10 | 0.25 | 0.58 | 0.86 | 0.51 | 0.24 | 0.10
18 | 950 | 458 | 0.11 | 0.26 | 0.66 | 0.65 | 0.29 | 0.15 | 0.07
19 | 1237 | 158 | 0.11 | 0.20 | 0.41 | 0.90 | 0.41 | 0.20 | 0.11
20 | 850 | 1011 | 0.07 | 0.15 | 0.31 | 0.75 | 0.57 | 0.25 | 0.11

We observed that in RGB space the change for all three metrics is
approximately linear, which provides some advantage over using the same
analysis in YUV space where the change for all three metrics is approximately
exponential as shown in Figure 8, Appendix A. Two cyan vertical lines where
plotted at negative and positive 40, to represent a ”safe” shift range where
the autonomous system is expected to produce reliable predictions, as shown in
Figures 9 through 10, Appendix A.

We then performed an experiment with partial results shown in Tables 1 through
4. We chose fifty random image pairs from the Generated Track dataset (20 are
displayed) where column ID1 is the frame number of the first random image, and
ID2 is the frame number of the second random image.

To the second random image ID2 with apply six shifts, ranging from negative to
positive 120, and include the ID2 image with no shift, measuring the Histogram
Intersection between both ID1 and ID2 for all shifts and making a record of
the quantity, where values closer to 1, or most intersection, are the nearest,
and closest to zero, or least intersection, represent the furthest distance
between the pair of images.

Table 2: Relative Entropy in RGB space between random Generated Road frames

| ID1 | ID2 | -120 | -80 | -40 | 0 | 40 | 80 | 120
---|---|---|---|---|---|---|---|---|---
1 | 592 | 503 | 1.80 | 1.22 | 0.51 | 0.04 | 0.32 | 1.01 | 1.53
2 | 863 | 825 | 1.82 | 0.98 | 0.34 | 0.07 | 0.83 | 1.58 | 1.97
3 | 912 | 879 | 1.69 | 0.94 | 0.45 | 0.03 | 0.83 | 1.58 | 1.97
4 | 1096 | 410 | 1.49 | 0.70 | 0.22 | 0.24 | 0.98 | 1.54 | 1.90
5 | 519 | 772 | 1.71 | 0.97 | 0.32 | 0.07 | 0.67 | 1.38 | 1.81
6 | 365 | 1091 | 1.82 | 1.23 | 0.64 | 0.16 | 0.25 | 0.78 | 1.43
7 | 1082 | 1094 | 1.57 | 0.90 | 0.48 | 0.01 | 0.62 | 1.11 | 1.83
8 | 811 | 1300 | 1.99 | 1.30 | 0.75 | 0.20 | 0.17 | 0.71 | 1.18
9 | 146 | 1004 | 1.63 | 0.92 | 0.50 | 0.03 | 0.54 | 1.29 | 1.80
10 | 1199 | 593 | 1.47 | 0.68 | 0.22 | 0.18 | 0.82 | 1.43 | 1.86
11 | 797 | 1157 | 1.95 | 1.33 | 0.74 | 0.20 | 0.19 | 0.84 | 1.46
12 | 350 | 173 | 1.88 | 1.31 | 0.71 | 0.17 | 0.17 | 0.72 | 1.37
13 | 407 | 1260 | 1.93 | 1.33 | 0.74 | 0.20 | 0.16 | 0.68 | 1.32
14 | 157 | 1036 | 1.64 | 0.90 | 0.46 | 0.04 | 0.81 | 1.63 | 2.00
15 | 331 | 1139 | 1.83 | 1.25 | 0.68 | 0.16 | 0.25 | 0.80 | 1.41
16 | 623 | 1259 | 1.85 | 1.28 | 0.69 | 0.17 | 0.17 | 0.65 | 1.26
17 | 235 | 566 | 1.70 | 1.09 | 0.35 | 0.03 | 0.46 | 1.12 | 1.65
18 | 950 | 458 | 1.63 | 0.82 | 0.22 | 0.18 | 1.03 | 1.60 | 1.97
19 | 1237 | 158 | 1.63 | 0.91 | 0.48 | 0.02 | 0.67 | 1.15 | 1.81
20 | 850 | 1011 | 1.90 | 1.16 | 0.64 | 0.08 | 0.32 | 1.13 | 1.72

Figure 7: Histogram Intersection, Relative Entropy and Bhattacharyya Distance
in RGB space between frame 100 and others up to frame 1300, with a safe limit
determined empirically, that is the safe distance threshold the Histogram
Intersection should not fall below, or Relative Entropy and Bhattacharya
distance should not go above

Rows 11 and 13 are highlighted in Table 1 because the they represent the
largest distance between ID1 and ID2 in the range of negative to positive 40,
which is considered a ”safe” shift. We highlight values the largest values in
the negative and positive 80 columns, which represent the largest Histogram
Intersection we would expect between any two pair of random images taken from
e.g. a training dataset and an image acquired by the autonomous system in a
real life situation. We could then claim that for any such pairing, we would
accept the prediction of the network controlling the autonomous system as long
as the Histogram overlap is greater than 0.40, ensuring that we are close to
our safe limits, otherwise the autonomous system defers control to a human
operator. The value 0.40 is approximately the y axis value for the x value of
negative and positive 40, where the Histogram Intersection plot and the cyan
”safe” vertical line intersect.

Table 3: Bhattacharyya Distance in RGB space between random Generated Road
frames

| ID1 | ID2 | -120 | -80 | -40 | 0 | 40 | 80 | 120
---|---|---|---|---|---|---|---|---|---
1 | 592 | 503 | 1.59 | 0.85 | 0.32 | 0.03 | 0.15 | 0.54 | 1.06
2 | 863 | 825 | 1.74 | 0.81 | 0.23 | 0.04 | 0.39 | 0.88 | 1.51
3 | 912 | 879 | 1.66 | 0.85 | 0.32 | 0.01 | 0.40 | 0.83 | 1.43
4 | 1096 | 410 | 1.19 | 0.50 | 0.13 | 0.12 | 0.49 | 0.91 | 1.48
5 | 519 | 772 | 1.37 | 0.63 | 0.18 | 0.04 | 0.32 | 0.83 | 1.41
6 | 365 | 1091 | 1.62 | 0.87 | 0.45 | 0.10 | 0.11 | 0.43 | 0.96
7 | 1082 | 1094 | 1.51 | 0.75 | 0.35 | 0.01 | 0.30 | 0.64 | 1.23
8 | 811 | 1300 | 2.09 | 1.12 | 0.59 | 0.14 | 0.09 | 0.41 | 0.80
9 | 146 | 1004 | 1.55 | 0.75 | 0.35 | 0.02 | 0.29 | 0.69 | 1.21
10 | 1199 | 593 | 1.13 | 0.48 | 0.13 | 0.10 | 0.42 | 0.82 | 1.36
11 | 797 | 1157 | 2.17 | 1.20 | 0.59 | 0.14 | 0.09 | 0.45 | 0.95
12 | 350 | 173 | 1.78 | 0.98 | 0.49 | 0.10 | 0.08 | 0.39 | 0.91
13 | 407 | 1260 | 1.91 | 1.08 | 0.56 | 0.13 | 0.08 | 0.37 | 0.85
14 | 157 | 1036 | 1.52 | 0.73 | 0.33 | 0.02 | 0.39 | 0.89 | 1.52
15 | 331 | 1139 | 1.74 | 0.95 | 0.47 | 0.10 | 0.11 | 0.42 | 0.92
16 | 623 | 1259 | 1.76 | 0.96 | 0.48 | 0.11 | 0.08 | 0.36 | 0.81
17 | 235 | 566 | 1.43 | 0.72 | 0.22 | 0.02 | 0.24 | 0.65 | 1.22
18 | 950 | 458 | 1.43 | 0.65 | 0.15 | 0.10 | 0.49 | 0.90 | 1.49
19 | 1237 | 158 | 1.48 | 0.72 | 0.33 | 0.01 | 0.33 | 0.67 | 1.24
20 | 850 | 1011 | 2.00 | 1.04 | 0.50 | 0.05 | 0.15 | 0.58 | 1.13

We repeated the experiment for Relative Entropy, highlighting rows 4, 11 and
13, which represent the largest Relative Entropy values between ID1 and ID2,
where larger is further (greater distribution shift) and smaller is nearer
(smaller distribution shift). We highlight the smallest values in columns
negative and positive 80 shift, and claim that as long as the Relative Entropy
between a random image from the training dataset and one acquired from the
autonomous system camera, is less than 0.60, we will accept the prediction of
the network in control of the autonomous system, else the prediction is
considered to be unsafe and control is delegated to a human operator.

We repeated the experiment in RGB space and computed the Bhattacharyya
Distance, with results shown in table 3. We use the same images for all
distance metrics, i.e. row 1 will always be a comparison between frames 592
and 503 for all six, three RGB and three YUV tables, for the Histogram
Intersection, Relative Entropy and Bhattacharyya Distance metrics, that is the
50 image pairs were chosen randomly only once, and the same list of pairs is
used thereafter. Again we highlight the rows with the largest distances which
are coincident for all three tables described so far, rows 11 and 13. Again we
look at the smallest values in columns negative and positive 80 that a
Bhattacharyya Distance of 0.30 or less, between a randomly chosen image from
the training dataset, and an image acquired by the autonomous system’s camera,
is safe to use, and the network prediction controlling the autonomous system
will be accepted, otherwise the prediction will be rejected, and control will
be returned to a human operator.

We concluded that in the RGB space, any of the three distance metrics would be
adequate, and the least computationally intensive would be preferred, which is
the Histogram Intersection.

##  5 Conclusion

A formulation could be given by defining a range of maximum and minimum
distribution shift using pixels intensity value averages as a proxy, over the
entire training dataset, and the allowed data distribution shift. The output
being a boolean value where true means the image and resulting prediction are
safe to use, false otherwise.

$P_{safe}=\begin{cases}true,&\text{if
$|D_{B}rgb(P,Q)|<|D_{B}rgb(P,Q_{sr})|$}.\\\ true,&\text{if
$|D_{\text{KLrgb}}(P\parallel Q)|<|D_{\text{KLrgb}}(P\parallel Q_{sr})|$}.\\\
true,&\text{if $|H_{irgb}(P,Q)|>|H_{irgb}(P,Q_{sr})|$}.\\\
false,&\text{otherwise}.\end{cases}$ (12)

Where $P_{safe}$ is a boolean value determining if it is safe to make a
prediction from input $Q$ relative to $Q_{sr}$, and $Q_{sr}$ is the safe range
between negative and positive $sr$, and the comparisons are performed in RGB
space. Given the computational advantages, and strong linearity $H_{irgb}$ is
the preferred distance metric.

## References

  * [1] Charu C Aggarwal and Philip S Yu.  Outlier detection for high dimensional data.  In Proceedings of the 2001 ACM SIGMOD international conference on Management of data, pages 37–46, 2001.
  * [2] Anonymous.  https://youtu.be/v0Otdmtdhnk, 2020.
  * [3] Sabyasachi Basu and Martin Meckesheimer.  Automatic outlier detection for time series: an application to sensor data.  Knowledge and Information Systems, 11(2):137–154, 2007.
  * [4] Steffen Bickel, Michael Brückner, and Tobias Scheffer.  Discriminative learning under covariate shift.  Journal of Machine Learning Research, 10(9), 2009.
  * [5] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba.  End to end learning for self-driving cars, 2016.
  * [6] Saikiran Bulusu, Bhavya Kailkhura, Bo Li, Pramod K Varshney, and Dawn Song.  Anomalous example detection in deep learning: A survey.  IEEE Access, 8:132330–132347, 2020.
  * [7] Raghavendra Chalapathy and Sanjay Chawla.  Deep learning for anomaly detection: A survey.  arXiv preprint arXiv:1901.03407, 2019.
  * [8] Brent Cowan and Bill Kapralos.  A survey of frameworks and game engines for serious game development.  In 2014 IEEE 14th International Conference on Advanced Learning Technologies, pages 662–664. IEEE, 2014.
  * [9] Christopher P Diehl and John B Hampshire.  Real-time object classification and novelty detection for collaborative video surveillance.  In Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN’02 (Cat. No. 02CH37290), volume 3, pages 2620–2625. IEEE, 2002.
  * [10] Donkey Car.  An opensource diy self driving platform for small scale cars, 2022.
  * [11] Janet Fleetwood.  Public health, ethics, and autonomous vehicles.  Am J Public Health, 107(4)(9):532–537, 4 2017.
  * [12] Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan.  Exploring the limits of out-of-distribution detection.  Advances in Neural Information Processing Systems, 34:7068–7081, 2021.
  * [13] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  arXiv preprint arXiv:1412.6572, 2014.
  * [14] Dan Hendrycks and Kevin Gimpel.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  arXiv preprint arXiv:1610.02136, 2016.
  * [15] Victoria Hodge and Jim Austin.  A survey of outlier detection methodologies.  Artificial intelligence review, 22(2):85–126, 2004.
  * [16] Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira.  Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10951–10960, 2020.
  * [17] Rui Huang and Yixuan Li.  Mos: Towards scaling out-of-distribution detection for large semantic space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8710–8719, 2021.
  * [18] Hannah R Kerner, Danika F Wellington, Kiri L Wagstaff, James F Bell, Chiman Kwan, and Heni Ben Amor.  Novelty detection for multispectral images with application to planetary exploration.  In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 9484–9491, 2019.
  * [19] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [20] Y LeCun, E Cosatto, J Ben, U Muller, and B Flepp.  Dave: Autonomous off-road vehicle control using end-to-end learning.  Courant Institute/CBLL, http://www. cs. nyu. edu/yann/research/dave/index. html, Tech. Rep. DARPA-IPTO Final Report, 2004.
  * [21] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin.  A simple unified framework for detecting out-of-distribution samples and adversarial attacks.  Advances in neural information processing systems, 31, 2018.
  * [22] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant.  Enhancing the reliability of out-of-distribution image detection in neural networks.  arXiv preprint arXiv:1706.02690, 2017.
  * [23] Ming Lin, Jaewoo Yoon, and Byeongwoo Kim.  Self-driving car location estimation based on a particle-aided unscented kalman filter.  Sensors, 20(9), 2020.
  * [24] Benjamin Loeb and Kara M Kockelman.  Fleet performance and cost evaluation of a shared autonomous electric vehicle (saev) fleet: A case study for austin, texas.  Transportation Research Part A: Policy and Practice, 121:374–385, 2019.
  * [25] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek.  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.  Advances in neural information processing systems, 32, 2019.
  * [26] Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel.  Deep learning for anomaly detection: A review.  ACM Computing Surveys (CSUR), 54(2):1–38, 2021.
  * [27] Stanislav Pidhorskyi, Ranya Almohsen, and Gianfranco Doretto.  Generative probabilistic novelty detection with adversarial autoencoders.  Advances in neural information processing systems, 31, 2018.
  * [28] Marco AF Pimentel, David A Clifton, Lei Clifton, and Lionel Tarassenko.  A review of novelty detection.  Signal processing, 99:215–249, 2014.
  * [29] Michal Podpora, Grzegorz Pawel Korbas, and Aleksandra Kawala-Janik.  Yuv vs rgb-choosing a color space for human-machine interaction.  In FedCSIS (Position Papers), pages 29–34, 2014.
  * [30] Joaquin Quinonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence.  Dataset shift in machine learning.  Mit Press, 2008.
  * [31] William Riggs and Sven A Beiker.  Business models for shared and autonomous mobility.  In Automated Vehicles Symposium, pages 33–48. Springer, 2019.
  * [32] Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Müller.  A unifying review of deep and shallow anomaly detection.  Proceedings of the IEEE, 109(5):756–795, 2021.
  * [33] Gabi Shalev, Yossi Adi, and Joseph Keshet.  Out-of-distribution detection using multiple semantic label representations.  Advances in Neural Information Processing Systems, 31, 2018.
  * [34] Masashi Sugiyama and Motoaki Kawanabe.  Machine learning in non-stationary environments: Introduction to covariate shift adaptation.  MIT press, 2012.
  * [35] Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul Buenau, and Motoaki Kawanabe.  Direct importance estimation with model selection and its application to covariate shift adaptation.  Advances in neural information processing systems, 20, 2007.
  * [36] Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin.  Csi: Novelty detection via contrastive learning on distributionally shifted instances.  Advances in neural information processing systems, 33:11839–11852, 2020.
  * [37] John von Neumann.  Probabilistic logics and the synthesis of reliable organisms from unreliable components.  Automata studies, 34:43–98, 1956.
  * [38] Darrell M West.  Moving forward: self-driving vehicles in china, europe, japan, korea, and the united states.  Center for Technology Innovation at Brookings: Washington, DC, USA, 2016.
  * [39] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning requires rethinking generalization, 2017.

##  Appendix A Supporting Materials

Figure 8: Histogram Intersection, Relative Entropy and Bhattacharyya Distance safe limits for YUV image pixel intensity value shifts Table 4: Bhattacharyya Distance in YUV space between random Generated Road frames | ID1 | ID2 | -120 | -80 | -40 | 0 | 40 | 80 | 120
---|---|---|---|---|---|---|---|---|---
1 | 592 | 503 | 3.51 | 2.14 | 1.62 | 0.07 | 1.41 | 1.97 | 3.33
2 | 863 | 825 | 3.66 | 2.74 | 1.37 | 0.10 | 2.00 | 2.76 | 4.39
3 | 912 | 879 | 3.02 | 2.35 | 1.45 | 0.08 | 2.39 | 2.72 | 3.83
4 | 1096 | 410 | 3.20 | 2.41 | 1.31 | 0.20 | 2.23 | 2.95 | 4.46
5 | 519 | 772 | 2.90 | 1.84 | 1.35 | 0.13 | 1.63 | 2.44 | 3.59
6 | 365 | 1091 | 3.56 | 2.74 | 1.73 | 0.16 | 1.37 | 2.34 | 3.29
7 | 1082 | 1094 | 3.44 | 2.85 | 2.32 | 0.03 | 2.33 | 2.80 | 3.45
8 | 811 | 1300 | 4.95 | 2.91 | 2.35 | 0.24 | 1.23 | 2.11 | 3.27
9 | 146 | 1004 | 3.24 | 2.68 | 1.85 | 0.05 | 1.82 | 2.57 | 3.07
10 | 1199 | 593 | 2.87 | 2.07 | 1.25 | 0.18 | 1.67 | 2.50 | 3.43
11 | 797 | 1157 | 4.64 | 2.90 | 2.22 | 0.24 | 1.20 | 2.35 | 3.67
12 | 350 | 173 | 4.12 | 3.16 | 1.85 | 0.20 | 1.38 | 2.29 | 3.32
13 | 407 | 1260 | 4.71 | 3.01 | 2.37 | 0.18 | 1.20 | 2.01 | 2.87
14 | 157 | 1036 | 3.44 | 2.70 | 2.38 | 0.06 | 2.29 | 2.94 | 3.89
15 | 331 | 1139 | 3.64 | 2.66 | 1.81 | 0.15 | 1.32 | 2.25 | 3.21
16 | 623 | 1259 | 3.73 | 2.78 | 1.72 | 0.18 | 1.28 | 2.16 | 2.98
17 | 235 | 566 | 3.54 | 2.88 | 1.55 | 0.11 | 1.44 | 2.36 | 3.87
18 | 950 | 458 | 2.88 | 2.06 | 1.20 | 0.17 | 2.24 | 2.81 | 4.35
19 | 1237 | 158 | 4.72 | 2.95 | 1.93 | 0.06 | 2.28 | 2.75 | 3.47
20 | 850 | 1011 | 4.45 | 2.78 | 2.30 | 0.13 | 1.50 | 2.02 | 2.71
Figure 9: Plots of ground truth (SDSandbox PID steering output) and nvidia2
network predictions for images with pixel value intensity shifts of 0, -40,
-80 and -120 Figure 10: Plots of ground truth (SDSandbox PID steering output)
and nvidia1 network predictions for images with pixel value intensity shifts
of 40, 80 and 120