# On quantum Hall effect: Covariant derivatives, Wilson lines, gauge
potentials, lattice Weyl transforms, and Chern numbers

Felix A. Buot
CTCMP, Cebu Normal University, Cebu City 6000
Philippines,
C&LB Research Institute, Carmen, Cebu 6005
Philippines,
LCFMNN, University of San Carlos, Cebu City 6000,
Philippines

###### Abstract

We show that the gauge symmetry of the nonequilibrium quantum transport of
Chern insulator in a uniform electric field is governed by the Wilson line of
parallel transport operator coupled with the dynamical translation operator.
This is dictated by the minimal coupling of derivatives with gauge fields in
$U\left(1\right)$ gauge theory. This parallel transport symmetry consideration
leads to the integer quantum Hall effect in electrical conductivity obtained
to first-order gradient expansion of the nonequilibrium quantum transport
equations.

##  1 Introduction

In previous papers [1, 2, 3], we make use of the gapped energy-band structure
of solids under external electric field to derive the integer quantum Hall
effect (IQHE) of Chern insulator. We employ the real-time superfield and
lattice Weyl transform nonequilibrium Green’s function (SFLWT-NEGF) [4]
quantum transport formalism [5, 6] to the first-order gradient expansion to
derive the topological Chern number of the IQHE for two-dimensional systems,
as an integral multiple of quantum conductance, also known as the minimal
contact conductance in mesoscopic physics [4].

We find that the quantization of Hall effect occurs strictly not to first
order in the electric field per se but rather to first-order gradient
expansion in the nonequilibrium quantum transport equation. The Berry
connection and Berry curvature is the fundamental physics [7] behind the exact
quantization of Hall conductance in units of $\frac{e^{2}}{h}$, which also
happens to coincide with the source and drain contact conductance per spin in
a closed circuit of mesoscopic quantum transport [4].

In Ref.[1], we have shown that the $\left(p.q;E.t\right)$ phase-space is
renormalized to that of $\left(\mathcal{\vec{K}},\mathcal{E}\right)$ phase-
space, where

$\mathcal{\vec{K}}=\vec{p}+e\vec{F}t,$ (1)

and

$\mathcal{E}=E_{0}+e\vec{F}\cdot\vec{q}$ (2)

Here, the uniform electric field, $\vec{F}$, is in the $x$-direction, and the
Hall current is in the $y$-direction.

We have identified the topological invariant in
$\left(\mathcal{\vec{K}},\mathcal{E}\right)$ phase-space nonequilibrium
quantum transport equation leading to IQHE. Moreover, the formula is also
applicable to gapped Landau-level structure of a free electron gas in intense
magnetic field [8] since the variable $\mathcal{\vec{K}}$ can incorporates the
external vector potential, and its corresponding parallel transport, if
present. Note that the change of variables from $\mathcal{\vec{K}}$ to
$\vec{p}$ in the integration over the whole Brillouin zone has a Jacobian
unity. It was shown in previous paper [1] that the correct expression for the
IQHE conductivity given by

$\
\sigma_{yx}=\frac{e^{2}}{h}\sum\limits_{\alpha}\frac{\Delta\phi_{total}}{2\pi}=\sum\limits_{\alpha}\frac{e^{2}}{h}n_{\alpha}$

In the present paper, we show that Eqs, (1) and (2) directly arise from the
local gauge symmetry under a uniform electric field of localized Wannier
functions centered on lattice sites. These follow from covariant derivatives
leading to the dynamical spatio-temporal finite translation operators that is
coupled with the Wilson line parallel-transport operators.

##  2 Translation operators in uniform electric fields

In gauge theory, the minimally-coupled gauge covariant derivative is defined
as

$D_{\mu}=\partial_{\mu}+i\alpha^{\mu}A_{\mu}$

where $A_{\mu}$ is the electromagnetic four ”vector” potential, and
$\alpha^{\mu}$ is the serve as a coupling ”charge”. In electrodynamics,
$\alpha^{x,y,z}=\frac{e}{\hbar c}$, whereas $\alpha^{0}=\frac{e}{\hbar}$.
Therefore a finite translation operator $T\left(q_{\mu}\right)$ by the four
coordinates $q_{\mu}$ can be written as

$T\left(q_{\mu}\right)=\exp\left[q_{\mu}\left(D_{\mu}\right)\right]=\exp\left[q_{\mu}\left(\partial_{\mu}+i\alpha^{\mu}A_{\mu}\right)\right]$

where we have used the Einstein summation convention.

###  2.1 Wilson lines

The part of $T\left(q_{\mu}\right)$ given by
$\exp\left[q_{\mu}\left(i\alpha^{\mu}A_{\mu}\right)\right]$ represent the
Wilson lines. In general, for non-Abelain gauge theories, the Wilson line is
often written as,

$W_{C}=\mathcal{P}\exp\left[i\mathop{\displaystyle\int}\limits_{q=0}^{q_{\mu}}A_{\mu}dx^{\mu}\right]$

where $\alpha$’s are absorbed in $A_{\mu}$, $\mathcal{P}$ is the path ordering
operator since the $A_{\mu}$’s are generally non-Abelian and do not commute,
e.g., like the Pauli matrices.

The Wilson loop is defined as the average of the trace of Wilson lines around
a closed loop, where the average is taken using the Chern-Simons Lagrangian
which we will not go into. Moreover, we will not be dealing with non-Abelian
$A_{\mu}$’s and our Wilson lines are explicitly gauge invariant, being
expressed in terms of the gauge-field strength.

The importance of the Wilson line as a parallel transport operator in the
gauge independent formulation of gauge theories has been emphasized by
Mandelstam [9] and further developed by Wu and Yang [10]. The term given by
$\mathop{\displaystyle\int}\limits_{L}A_{\mu}dx^{\mu}$ is the electromagnetic
Berry phase. Precisely, the definition of the Berry phase $\phi$ reads

$\displaystyle\exp i\phi\ \left|\psi\left(x+q_{x},t+T\right)\right\rangle$
$\displaystyle=$ $\displaystyle
T\left(q_{\mu}\right)\left|\psi\left(x,t\right)\right\rangle=\exp\left[q_{\mu}\left(\partial_{\mu}+i\alpha^{\mu}A_{\mu}\right)\right]\left|\psi\left(x,t\right)\right\rangle$
(3) $\displaystyle=$
$\displaystyle\exp\left\\{\left[q_{x}\cdot\partial_{x}+T\partial_{t}\right]+i\phi\right\\}\left|\psi\left(x,t\right)\right\rangle$
$\displaystyle=$
$\displaystyle\exp\left\\{\frac{i}{\hbar}\left[\mathop{\displaystyle\int}\limits_{0}^{q_{x}}\hat{P}\cdot
dx-i\mathcal{H}\left(t^{\prime}\right)dt^{\prime}\right]+i\phi\right\\}\left|\psi\left(x,t\right)\right\rangle$

where $\hat{P}$ is the momentum operator, $\mathcal{H}$ is the energy
operator, $A_{\mu}$ is the electromagnetic Berry connection. The last line
serves as a generalization in terms of quantum mechanical operators, $\hat{P}$
and $\hat{H}$. The extra phase acquired in translation given by

$\phi=\mathop{\displaystyle\int}\alpha^{\mu}A_{\mu}dx^{\mu}$ (4)

is the geometrical phase. In this paper, it is identified with the generalized
Peierls phase factor in energy band dynamics of solid state physics. In
quantum physics, the adiabatic contour integral of Eq. (4) is now generally
known as the Berry phase and $A_{\mu}$ is now generally known mathematically
as the connection.

####  2.1.1 Peierls phase factor

The factor $\exp i\phi$ on the left side of Eq. (3) is the Peierls-phase
factor well-known in energy band dynamics of solids [11].

###  2.2 Gauge potential in uniform electric fields

In this section, we determined the gauge potential
$A_{\mu}=\left(A_{0},A_{x},A_{y},A_{z}\right)=\left(A_{0},\vec{A}\right)$,
where $A_{0}$ is the time component. We have

$\vec{A}=\vec{F}ct$

where $\vec{F}$ is the uniform electric field. Clearly,
$\alpha^{\mu}A_{\mu}=\frac{e}{\hbar c}\vec{A}$ has the same units as
$\vec{\partial}$ and $\frac{e}{c}\vec{A}$ has the same dimensional units as
the momentum, i.e.,

$\hat{P}+\frac{e}{c}\vec{A}$

represents the minimal coupling in quantum electrodynamics.

For $A_{0}$, we have

$\alpha^{0}A_{0}=-\frac{1}{\hbar}e\vec{F}\cdot\vec{q}$ (5)

has the same units as $\partial_{t}$ and hence from Eq. (3), we have

$\frac{1}{\hbar}\left(E_{o}+e\vec{F}\cdot\vec{q}\right)$

represents the coupling of the zero-field energy, $E_{o}$, to the electric
field at arbitrary lattice site $\vec{q}$. Note that the negative sign of Eq.
(5) is dictated by the equation for the force as the negative gradient of the
potential, namely,

$e\vec{F}=-\vec{\nabla}\left(-e\vec{F}\cdot\vec{q}\right)$

Therefore under uniform electric fields, we have the Wilson lines for our
Abelian gauge potential given by

$\displaystyle W_{C}$ $\displaystyle=$
$\displaystyle\exp\left[i\mathop{\displaystyle\int}\alpha^{\mu}A_{\mu}dx^{\mu}\right]$
$\displaystyle=$
$\displaystyle\exp\left[i\left(\alpha^{0}A_{0}t+\alpha\vec{A}\cdot
q\right)\right]$ $\displaystyle=$
$\displaystyle\exp\left[i\left(\left\\{-\frac{1}{\hbar}e\vec{F}\cdot\vec{q}\right\\}\
t+\left\\{\frac{e}{\hbar c}\vec{F}ct\right\\}\cdot q\right)\right]$

Thus, the Wilson lines serves as the generalization of Peierls phase factor in
solid state physics.

##  3 Alternative derivation of gauge potentials and Peierls phase factor

In contrast to the derivation given above, which directly use the covariant
derivative in $U\left(1\right)$ gauge theory, here we will use a self-
consistent Heisenberg equation for the spatio-temporal displacement operators
to determine the gauge potentials and hence the generalized Peierls phase
factor. Remarkably, the self-consistent Heisenberg equation of motion
automatically fixes the negative sign of the scalar gauge of Eq. (5).

Note that finite displacement operators are characteristically exponential
operators amenable to Fourier series expansion. Remarkably, phase factors are
generally acquired due to displacement or motion in parameter space in the
presence of electromagnetic fields. This is ubiquitous in solid-state physics
(e.g., Peierls phase factor in magnetic fields) before the Berry connection
become mainstream and fashionable. To begin, we write our bare Hamiltonian of
electrons as

$H=H_{o}-e\vec{F}\cdot\vec{r},$

where $H_{o}$ is the periodic Hamiltonian in the absence of the electric
field, $\vec{F}$.

In this section, we want to show that the displacement operator,

$\hat{T}\left(q\right)=\exp\left[\frac{i}{\hbar}\left(\left(\vec{q}\cdot-i\hbar\nabla_{\vec{r}}\right)\right)\right],$

acquires a phase factor in the presence of electric field, and is given by

$\displaystyle\tilde{T}\left(\vec{q}\right)$ $\displaystyle=$
$\displaystyle\exp\left[\frac{i}{\hbar}\left(\vec{q}\cdot\left(\frac{e}{c}\vec{F}ct\right)+\left(\vec{q}\cdot-i\hbar\nabla_{\vec{r}}\right)\right)\right],$
(6) $\displaystyle=$
$\displaystyle\exp\left[\frac{i}{\hbar}\left(\vec{q}\cdot\left(\frac{e}{c}\vec{F}ct\right)+\left(\vec{q}\cdot\hat{P}\right)\right)\right],$

where we indicate by $\tilde{T}$ the translation operator $\hat{T}$ with a
phase factor.

Similarly, for the time displacement operator, we will show that

$\hat{T}\left(t\right)=\exp\left[-\frac{i}{\hbar}\left(\left(t\cdot\
i\hbar\frac{\partial}{\partial t}\right)\right)\right],$ (7)

also acquires a phase factor and is given by

$\tilde{T}\left(t\right)=\exp\left[-\frac{i}{\hbar}\left(t\left(e\vec{F}\cdot\vec{q}\right)+\left(t\
\mathcal{H}\right)\right)\right],$ (8)

where $\tilde{T}\left(t\right)$ differs from $\hat{T}\left(t\right)$ by a
phase factor, and $i\hbar\frac{\partial}{\partial t}:\mathcal{=H}$ is the
energy operator of the system. In other words, displacement in space by
lattice vector $q$ acquires phase factors equal to
$\exp\left(\frac{i}{\hbar}e\vec{F}t\cdot\vec{q}\right)$. Smilarly,
displacement in time by $t$ acquires phase factor equal to
$\exp\left(-\frac{i}{\hbar}\left(e\vec{F}\cdot\vec{q}\right)t\right)$. In the
absence of the electric field these phase factors give unity, e.g.,
$\tilde{T}\left(t\right)\Longrightarrow_{\vec{F}\rightarrow
0}\hat{T}\left(t\right)$, reduces to ordinary translation operator.

The physics behind these phase factors is dictated by a selfconsistent
translation of local functions in space and time under a uniform electric
field, $\vec{F}$. These follow from the self-consistent Heisenberg equation of
motion for quantum operators. For efficient bookeeping and for ease i taking
lattice Weyl transform, it is usually more convenient to attach these phase
factors to displaced local functions themselves, Eq. (3) . This yields a
generalization of Peierls phase factor, well-known for solid-state problems in
magnetic fields. The derivation goes as follows.

####  3.0.1 Nonlocality in coordinates

The nature of the derivatives of exponential displacement operator is
determined, e.g., by the following operation,

$\displaystyle i\hbar\frac{\partial}{\partial
t}\tilde{T}\left(q\right)W_{\lambda}\left(r-0\right)$ $\displaystyle=$
$\displaystyle i\hbar\frac{\partial\phi}{\partial
t}\tilde{T}\left(q\right)W_{\lambda}\left(r-0\right),$
$\displaystyle\frac{\partial}{\partial t}\tilde{T}\left(q\right)$
$\displaystyle=$ $\displaystyle\frac{\partial\phi}{\partial
t}\tilde{T}\left(q\right),$
$\displaystyle\frac{d\tilde{T}\left(q\right)}{\tilde{T}\left(q\right)}$
$\displaystyle=$ $\displaystyle d\phi,$

where specifically $\hat{T}\left(-q\right)W_{\lambda}\left(r-0\right)$ is a
translation of the center coordinate of a localized Wannier function centered
in the origin to another lattice point $q$, yielding
$W_{\lambda}\left(r-q\right).$ Therefore $\hat{T}\left(-q\right)$ resembles
the physical process of transfering a localized function centered at the
origin to a localized function centered at another lattice point $q$. Here we
define total $\phi$ as undetermined for the moment as

$\phi=-\frac{i}{\hbar}\left(f\left(q,t\right)+\left(\vec{q}\cdot\hat{P}\right)\right),$

where $f\left(q,t\right)$ is to be determined. Note that the presence of
$f\left(q,t\right)$ is needed for selfconsistency in the presence of electric
field. Now consider the second term in the exponent, namely,

$\displaystyle\phi_{2}$ $\displaystyle=$
$\displaystyle-\frac{i}{\hbar}\vec{q}\cdot\hat{P}=-\frac{i}{\hbar}\vec{q}\cdot\left(-i\hbar\frac{\partial}{\partial\vec{r}}\right),$
$\displaystyle=$
$\displaystyle\left(-q\right)\cdot\frac{\partial}{\partial\vec{r}}$

leading to Fourier series expansion of the translation,
$\hat{T}\left(-q\right)$ . Then, we obtain

$-i\hbar\frac{\partial\phi_{2}}{\partial
t}=\left[H,\phi_{2}\right]=\frac{i}{\hbar}\left[H,\left(\left(-\vec{q}\right)\cdot\hat{P}\right)\right]=e\vec{F}\cdot\left(-\vec{q}\right),$
(9)

since $-\frac{i}{\hbar}f\left(q,t\right)$ commutes with the Hamiltonian.
Therefore, we have

$\frac{d\tilde{T}\left(-q\right)}{\tilde{T}\left(-q\right)}=d\phi=\frac{i}{\hbar}e\vec{F}\cdot\left(-\vec{q}\right)\
dt.$ (10)

We may thus write

$-i\hbar\frac{\partial}{\partial
t}\tilde{T}\left(\vec{q},t\right)=\left[H,\tilde{T}\left(\vec{q},t\right)\right]=e\vec{F}\cdot\left(\vec{q}\right)\
\tilde{T}\left(\vec{q},t\right).$ (11)

Therefore

$\displaystyle\tilde{T}\left(q\right)$ $\displaystyle=$
$\displaystyle\exp\frac{i}{\hbar}\left[\left(e\vec{F}\Delta
t\right)\cdot\vec{q}\ +\hat{P}\cdot\vec{q}\right],$ (12) $\displaystyle=$
$\displaystyle\exp\frac{i}{\hbar}\left[\hat{P}+e\vec{F}t\right]\cdot\vec{q}$

This means that a displacement by $\vec{q}$ of localized function acquires a
phase factor given by

$Peierls\ phase\ factor=\exp\left[\frac{i}{\hbar}\left(e\vec{F}t\cdot\vec{q}\
\right)\right].$ (13)

We can also deduce from Eq. (9) the relation for the momentum operator,

$\displaystyle\frac{i}{\hbar}\left[H,\left[\left(\vec{q}\right)\cdot-i\hbar\nabla_{\vec{r}}\right]\right]$
$\displaystyle=$ $\displaystyle
e\vec{F}\cdot\left(\vec{q}\right)=\frac{\partial\hat{P}}{\partial
t}\cdot\left(q\right),\Longrightarrow\frac{\partial\hat{P}}{\partial
t}=e\vec{F},$ (14) $\displaystyle\Longrightarrow$
$\displaystyle\hat{P}=\hat{P}_{o}+e\vec{F}t.$

which indicates a covariant derivative in spatial coordinates,

$\vec{D}_{\mu}=\vec{\partial}_{\mu}+i\frac{e}{\hbar c}\vec{A}$

with

$\vec{A}_{\mu}=\vec{F}ct$

as obtained before.

####  3.0.2 Simultaneous eigenvalues for $\hat{H}$ and
$\tilde{T}\left(\vec{q},t\right)$

One very important conclusion is implied in Eq. (11), which we rewrite here
for convenience

$\left[\hat{H},\tilde{T}\left(\vec{q},t\right)\right]=e\vec{F}\cdot\vec{q}\
\tilde{T}\left(\vec{q},t\right).$ (15)

What the above relation means is that if $\tilde{T}\left(\vec{q},t\right)$ is
diagonal then $\left[H,\tilde{T}\left(\vec{q},t\right)\right]$ is also
diagonal. But if $\tilde{T}\left(\vec{q},t\right)$ is diagonal, then $\hat{H}$
is also diagonal with the same eigenvalues. The eigenfunction of
$\tilde{T}\left(\vec{q},t\right)$ is labeled by the quantum label
$\mathcal{\vec{K}}=\vec{p}_{o}+e\vec{F}t$. This implies that $\hat{H}$ is also
diagonal in $\mathcal{\vec{K}}$. The electric Bloch function labeled by
$B\left(k_{o}+\frac{e}{\hbar}Ft,..\right)$ is the eigenfunction of
$\tilde{T}\left(\vec{q},t\right)$ as well as that of the renormalized

$H_{renormalized}\left(\hat{K},\hat{Q}\right)\Longleftrightarrow
W_{n}\left(\mathcal{\vec{K}},\mathcal{E}\right),$ (16)

where the double pointed arrow denotes lattice Weyl correspondence, see Eq.
(27) below, using the localized electric Wannier function.

We now show the the energy variable $\mathcal{E}$ in Eq. (16) does incorporate
the coordinates through the gauge potential $A_{0}$.

####  3.0.3 Nonlocality in time

Because nonlocal arguments in time acquires phase factor also, the energy
variable of the theory now varies with $e\vec{F}\cdot\vec{q},$ as discussed
next. From Eq. (7) for a displacement in time,

$\hat{T}\left(t\right)=\exp\frac{i}{\hbar}\left(-\mathcal{H}t\right)$

or

$\tilde{T}(t)\equiv\mathcal{F}\exp\left(\frac{-i}{\hbar}\left(i\hbar
t\frac{\partial}{\partial
t}\right)\right)=\mathcal{F}\exp\left(t\frac{\partial}{\partial t}\right)$

where $\mathcal{F}$ incorporates the scalar gauge potential, $A_{0}$. Let

$\displaystyle\mathcal{F}$ $\displaystyle=$
$\displaystyle\exp\left[i\alpha^{0}A_{0}t\right]$ (17) $\displaystyle=$
$\displaystyle\exp\left[-\frac{i}{\hbar}\left(f_{t}\left(q,t\right)\right)\right],$
(18)

is a phase factor to be determined. The total phase is

$\phi_{t}\left(q\right)=-\frac{i}{\hbar}\left(f_{t}\left(q,t\right)+t\
\left(i\hbar\frac{\partial}{\partial t^{\prime}}\right)\right).$

From the equation of motion,

$\displaystyle-i\hbar\frac{\partial\tilde{T}(t)}{\partial\vec{q}}$
$\displaystyle=$
$\displaystyle-i\hbar\frac{\partial\phi_{t}\left(q\right)}{\partial\vec{q}}\tilde{T}(t),$
$\displaystyle-i\hbar\frac{\partial\phi_{t}\left(q\right)}{\partial\vec{q}}$
$\displaystyle=$
$\displaystyle-\frac{i}{\hbar}\left[\mathcal{\vec{K}}\left(t^{\prime}\right),t\left(i\hbar\frac{\partial}{\partial
t^{\prime}}\right)\right]=\left[\vec{p}+\frac{e}{c}\vec{A},t\nabla_{t^{\prime}}\right],$
(19) $\displaystyle=$
$\displaystyle\left[\vec{p}+e\vec{F}t^{\prime},t\nabla_{t^{\prime}}\right]=-e\vec{F}t.$

Therefore

$\displaystyle-i\hbar\frac{\partial\phi_{t}\left(q\right)}{\partial\vec{q}}$
$\displaystyle=$ $\displaystyle-e\vec{F}t,$
$\displaystyle\frac{\partial\phi_{t}\left(q\right)}{\partial\vec{q}}$
$\displaystyle=$ $\displaystyle-\frac{i}{\hbar}e\vec{F}t.$

Thus, we obtained,

$\frac{\partial\tilde{T}(t)}{\partial\vec{q}}=\left(-\frac{i}{\hbar}e\vec{F}t\right)\tilde{T}(t),$

and hence,

$\displaystyle d\ln\tilde{T}(t)$ $\displaystyle=$
$\displaystyle\left(-\frac{i}{\hbar}e\vec{F}t\right)\cdot d\vec{q},$
$\displaystyle\ln\tilde{T}(t)$ $\displaystyle=$
$\displaystyle\left(-\frac{i}{\hbar}e\vec{F}t\right)\cdot\Delta\vec{q}.$

Hence a displacement in time carries a phase factor given by
$\exp\left(-\frac{i}{\hbar}\left(e\vec{F}t\right)\cdot\Delta\vec{q}\right)$
and

$\displaystyle\tilde{T}(t)$ $\displaystyle=$
$\displaystyle\exp\left\\{-\frac{i}{\hbar}\left[t\left(e\vec{F}\cdot\Delta\vec{q}\right)+t\left(i\hbar\frac{\partial}{\partial
t^{\prime}}\right)\right]\right\\}$ (20) $\displaystyle=$
$\displaystyle\exp\left\\{-\frac{i}{\hbar}\left[t\left(e\vec{F}\cdot\vec{q}\right)+t\left(i\hbar\frac{\partial}{\partial
t^{\prime}}\right)\right]\right\\}$

Once more this gives support to the covariant derivative,

$D_{0}=\partial_{t}+i\alpha^{0}A_{0}$

where

$i\alpha^{0}A_{0}=-i\frac{e\vec{F}\cdot\vec{q}}{\hbar}$

Therefore,

$-i\hbar\frac{\partial}{\partial\vec{q}}\tilde{T}\left(\vec{q},t\right)=\left[\mathcal{\vec{K}}\left(t\right),\tilde{T}\left(\vec{q},t\right)\right]=\left(-e\vec{F}t\right)\
\tilde{T}\left(\vec{q},t\right)$ (21)

Now from second line of Eq. (19), we have

$\displaystyle-\frac{i}{\hbar}\left[\mathcal{\vec{K}}\left(t^{\prime}\right),\left(t\right)i\hbar\frac{\partial}{\partial
t^{\prime}}\right]$ $\displaystyle=$
$\displaystyle\left[\vec{p}+e\vec{F}t^{\prime},\left(t\right)\frac{\partial}{\partial
t^{\prime}}\right]=-e\vec{F}t,$
$\displaystyle\left[\mathcal{\vec{K}}\left(t^{\prime}\right),\mathcal{H}\right]$
$\displaystyle=$ $\displaystyle-i\hbar
e\vec{F}=-i\hbar\frac{\partial\mathcal{E}}{\partial\vec{q}},$

which leads to to the expression for in Eq. (17) for $f_{t}$ in the phase
factor as

$\frac{\partial\mathcal{E}}{\partial\vec{q}}=e\vec{F}\Longrightarrow
f_{t}=e\vec{F}\cdot\vec{q}.$

All these results, Eq. (12) and Eq. (20), lead to the time-dependent wave
vector,

$\hbar\vec{k}=\hbar\vec{k}_{o}+\hbar e\vec{F}t$

and to the position-dependent energy,

$\mathcal{E}=E_{o}+e\vec{F}\cdot\vec{q},$ (22)

respectively. Again, for for taking lattice Weyl transform, it is more
convenient to attach these phase factor to the displaced local functions
themselves. This allows us to generalize the Peierls phase factor to space and
time displacements, originally well-known for solid-state problems for
magnetic fields.

As an example, for nonequilibrium translational symmetric and steady-state
condition.

$\left\langle
q_{1},t_{1}\right|\mathcal{O}\left|q_{2},t_{2}\right\rangle\Longrightarrow
e^{i\frac{e}{\hbar}\vec{F}t\cdot\left(\vec{q}_{2}-\vec{q}_{1}\right)}e^{-i\frac{e}{\hbar}\vec{F}\cdot\vec{q}\left(t_{2}-t_{1}\right)}O\left(\vec{q}_{2}-\vec{q}_{1},t_{2}-t_{1}\right)$
(23)

where

$\vec{q}=\frac{1}{2}\left(\vec{q}_{1}+\vec{q}_{2}\right)$
$t=\frac{1}{2}\left(t_{1}+t_{2}\right)$

##  4 Lattice Weyl transforms

Using the four dimensional notation: $p=\left(\vec{p},E\right)$ and
$q=\left(q,t\right)$, the lattice Weyl transform (LWT), $A\left(p,q\right)$ of
any operator $\hat{A}$ is defined by

$\displaystyle A_{\lambda\lambda^{\prime}}\left(p,q\right)$ $\displaystyle=$
$\displaystyle\mathop{\displaystyle\sum}\limits_{v}e^{\left(\frac{2i}{\hbar}\right)p.v}\left\langle
q-v,\lambda\right|\hat{A}\left|q+v,\lambda^{\prime}\right\rangle$ (24)
$\displaystyle=$
$\displaystyle\mathop{\displaystyle\sum}\limits_{u}e^{\left(\frac{2i}{\hbar}\right)q.u}\left\langle
p+u,\lambda\right|\hat{A}\left|p-u,\lambda^{\prime}\right\rangle$

Writing Eq. (24) explicitly, we have

$A_{\lambda\lambda^{\prime}}\left(\vec{p}.\vec{q};E,t\right)=\sum\limits_{\vec{v};\tau}e^{\left(\frac{2i}{\hbar}\right)\vec{p}\cdot\vec{v}}e^{\left(-\frac{i}{\hbar}\right)E\tau}\left\langle\vec{q}-\vec{v};t-\frac{\tau}{2},\lambda\right|\mathbf{\hat{A}}\left|\vec{q}+\vec{v};t+\frac{\tau}{2},\lambda^{\prime}\right\rangle\text{.}$
(25)

Using the form of matrix elements in Eq. (23), we have

$\displaystyle\left\langle\vec{q}-\vec{v};t-\frac{\tau}{2},\lambda\right|\mathbf{\hat{A}}\left|\vec{q}+\vec{v};t+\frac{\tau}{2},\lambda^{\prime}\right\rangle$
(26) $\displaystyle=$ $\displaystyle
e^{i\frac{e}{\hbar}\vec{F}t\cdot\left(\vec{q}_{1}-\vec{q}_{2}\right)}e^{-i\frac{e}{\hbar}\vec{F}\cdot\vec{q}\left(t_{1}-t_{2}\right)}A\left(\vec{q}_{1}-\vec{q}_{2},t_{1}-t_{2}\right)$
$\displaystyle=$ $\displaystyle
e^{i\frac{e}{\hbar}\vec{F}t\cdot\left(2\vec{v}\right)}e^{-i\frac{e}{\hbar}\vec{F}\cdot\vec{q}\tau}A_{\lambda\lambda^{\prime}}\left(\vec{q}_{1}-\vec{q}_{2},t_{1}-t_{2}\right)\text{.}$

Thus

$\displaystyle A_{\lambda\lambda^{\prime}}\left(\vec{p}.\vec{q};E,t\right)$
$\displaystyle=$
$\displaystyle\sum\limits_{\vec{v};\tau}e^{\left(\frac{i}{\hbar}\right)\vec{p}\cdot
2\vec{v}}e^{i\frac{e}{\hbar}\vec{F}t\cdot\left(2\vec{v}\right)}e^{\left(-\frac{i}{\hbar}\right)E\tau}e^{-i\frac{e}{\hbar}\vec{F}\cdot\vec{q}\tau}A_{\lambda\lambda^{\prime}}\left(\vec{q}_{1}-\vec{q}_{2},t_{1}-t_{2}\right),$
(27) $\displaystyle=$
$\displaystyle\sum\limits_{\vec{v};\tau}e^{\left(\frac{2i}{\hbar}\right)\left(\vec{p}+e\vec{F}t\right)\cdot\vec{v}}e^{\left(-\frac{i}{\hbar}\right)\left(E+e\vec{F}\cdot\vec{q}\right)\tau}A_{\lambda\lambda^{\prime}}\left(2v,\tau\right),$
$\displaystyle=$ $\displaystyle
A_{\lambda\lambda^{\prime}}\left[\left(\vec{p}+e\vec{F}t\right),\left(E+e\vec{F}\cdot\vec{q}\right)\right],$
$\displaystyle=$ $\displaystyle
A_{\lambda\lambda^{\prime}}\left(\mathcal{\vec{K}};\mathcal{E}\right)\text{.}$

Hence the expected dynamical variables in the phase space including the time
variable occurs in particular combinations of $\mathcal{\vec{K}}$ and
$\mathcal{E}$. Therefore, besides the crystal momentum varying in time as

$\mathcal{\vec{K}}=\vec{p}_{o}+e\vec{F}t\text{,}$ (28)

the energy variable vary with $\vec{q}$ as

$\mathcal{E}=E_{o}+e\vec{F}\cdot\vec{q}.$ (29)

This is the result of covariant derivatives for energy band dynamics in the
presence of uniform electric fields. Thus, differentiation with respect to
coordinate and time variables are now relegated to differentiation with
respect to energy $\mathcal{E}$ and $\mathcal{\vec{K}}$, respectively,

$\displaystyle\text{ \ \ }\frac{\partial}{\partial t}$ $\displaystyle=$
$\displaystyle\frac{\partial\mathcal{\vec{K}}}{\partial
t}\cdot\frac{\partial}{\partial\mathcal{\vec{K}}}=e\vec{F}\cdot\frac{\partial}{\partial\mathcal{\vec{K}}}\text{,}$
(30) $\displaystyle\text{\ }\frac{\partial}{\partial\vec{q}}$ $\displaystyle=$
$\displaystyle\frac{\partial\mathcal{E}}{\partial\vec{q}}\frac{\partial}{\partial\mathcal{E}}=e\vec{F}\frac{\partial}{\partial\mathcal{E}}\text{,}$
(31) $\displaystyle\frac{\partial}{\partial\mathcal{\vec{K}}}$
$\displaystyle=$
$\displaystyle\frac{\partial\mathcal{E}}{\partial\mathcal{\vec{K}}}\frac{\partial}{\partial\mathcal{E}}=\vec{v}_{g}\frac{\partial}{\partial\mathcal{E}}\text{,}$
(32)

where $v_{g}$ is the group velocity. The LWT of the effective or renormalized
lattice Hamiltonian $\mathcal{H}_{eff}\leftrightarrows
H\left(\vec{p},\vec{q};E_{o},t\right)$ can therefore be analyzed on
$\left(\mathcal{\vec{K}},E\right)$-space as

$H\left(\vec{p},\vec{q};E_{o},t\right)\Longrightarrow
H\left(\mathcal{\vec{K}},\mathcal{E}\right).$ (33)

The last line is by virtue of Eqs. (28)- (29). Of course in the absence of the
electric field, the dependence in phase space becomes the familiar
$H\left(\vec{p},\mathcal{\omega}\right)$ for translationally symmetric and
steady-state system. But with $\vec{F}\neq 0$ all gauge invariant quantities
are functions of $\left(\mathcal{\vec{K}},\mathcal{E}\right)$ such as the
electric Bloch function [11, 6] or Houston wavefunction [12]] and electric
Wannier function, i.e., the electric-field dependent generalization of Wannier
function. In particular, the Weyl transform of a commutator,

$\mathcal{W}\left[H,G^{<}\right]=\sin\Lambda\
\left\\{H\left(\mathcal{\vec{K}},\mathcal{E}\right)G^{<}\left(\mathcal{\vec{K}},\mathcal{E}\right)\right\\}\text{,}$
(34)

where $\Lambda$ is the Poisson bracket operator. We can therefore write the
Poisson bracket operator $\Lambda$, as

$\displaystyle\Lambda$ $\displaystyle=$
$\displaystyle\frac{\hbar}{2}\left[\frac{\partial^{\left(a\right)}}{\partial
t}\frac{\partial^{\left(b\right)}}{\partial\mathcal{E}}-\frac{\partial^{\left(a\right)}}{\partial\mathcal{E}}\frac{\partial^{\left(b\right)}}{\partial
t}\right],\text{ }$ (35) $\displaystyle=$
$\displaystyle\frac{\hbar}{2}\frac{\partial\mathcal{\vec{K}}}{\partial
t}\cdot\left[\frac{\partial^{\left(a\right)}}{\partial\mathcal{\vec{K}}}\frac{\partial^{\left(b\right)}}{\partial\mathcal{E}}-\frac{\partial^{\left(a\right)}}{\partial\mathcal{E}}\frac{\partial^{\left(b\right)}}{\partial\mathcal{\vec{K}}}\right],$
$\displaystyle=$
$\displaystyle\frac{\hbar}{2}e\vec{F}\cdot\left[\frac{\partial^{\left(a\right)}}{\partial\mathcal{\vec{K}}}\frac{\partial^{\left(b\right)}}{\partial\mathcal{E}}-\frac{\partial^{\left(a\right)}}{\partial\mathcal{E}}\frac{\partial^{\left(b\right)}}{\partial\mathcal{\vec{K}}}\right]\text{,
}$

on $\left(\mathcal{\vec{K}},\mathcal{E}\right)$-phase space.

##  5 Application to nonequilibrium quantum transport equation

The ballistic phase-space quantum transport equation [1] generally reads,

$\frac{\partial}{\partial
t}G^{<}\left(\vec{p},\vec{q};E,t\right)=\frac{2}{\hbar}\sin\hat{\Lambda}\left\\{H\left(p,q\right)G^{<}\left(p,q\right)\right\\}$
(36)

where in the right side of Eq. (36) the $4$-dimensional notation of phase
space is employed. Under a uniform electric field, this simplifies to

$\frac{\partial}{\partial
t}G^{<}\left(\mathcal{\vec{K}},\mathcal{E}\right)=\sin\Lambda\
\left\\{H\left(\mathcal{\vec{K}},\mathcal{E}\right)G^{<}\left(\mathcal{\vec{K}},\mathcal{E}\right)\right\\}$

If we expand Eq. (36) to first-order in the gradient, i.e.,
$\sin\Lambda\simeq\Lambda,$the phase-space transport equation [6] can be
written in a compact form as

$\frac{\partial}{\partial
t}G^{<}\left(\mathcal{\vec{K}},\mathcal{E}\right)=\frac{2}{\hbar}\frac{\hbar}{2}e\vec{F}\cdot\left[\frac{\partial^{\left(a\right)}}{\partial\mathcal{\vec{K}}}\frac{\partial^{\left(b\right)}}{\partial\mathcal{E}}-\frac{\partial^{\left(a\right)}}{\partial\mathcal{E}}\frac{\partial^{\left(b\right)}}{\partial\mathcal{\vec{K}}}\right]H^{\left(a\right)}\left(\mathcal{\vec{K}},\mathcal{E}\right)G^{<\left(b\right)}\left(\mathcal{\vec{K}},\mathcal{E}\right)\text{.}$
(37)

With the electric field in the $x$-direction, then we have

$G^{<}\left(\mathcal{\vec{K}},\mathcal{E}\right)=e\left|\vec{F}\right|\int dt\
\left[\frac{\partial^{\left(a\right)}}{\partial\mathcal{\vec{K}}_{x}}\frac{\partial^{\left(b\right)}}{\partial\mathcal{E}}-\frac{\partial^{\left(a\right)}}{\partial\mathcal{E}}\frac{\partial^{\left(b\right)}}{\partial\mathcal{\vec{K}}_{x}}\right]H^{\left(a\right)}\left(\mathcal{\vec{K}},\mathcal{E}\right)G^{<\left(b\right)}\left(\mathcal{\vec{K}},\mathcal{E}\right)\text{.}$
(38)

The Hall current in the $y$-direction is given by the following equation,

$J_{y}=\frac{a^{2}}{\left(2\pi\hbar\right)^{2}}\int\int
d\mathcal{\vec{K}}_{x}d\mathcal{\vec{K}}_{y}\left(\frac{e}{a^{2}}\frac{\partial\mathcal{E}}{\partial\mathcal{\vec{K}}_{y}}\right)\left(-iG^{<}\left(\mathcal{\vec{K}},\mathcal{E}\right)\right)$

which leads to $J_{y}=\sigma_{yx}\left|\vec{F}\right|,$

$\displaystyle\sigma_{yx}$ $\displaystyle=$ $\displaystyle
e^{2}\frac{1}{\left(2\pi\hbar\right)^{2}}\int\int\int
d\mathcal{\vec{K}}_{x}d\mathcal{\vec{K}}_{y}dt$ (39) $\displaystyle\times\
\left[\frac{\partial^{\left(a\right)}}{\partial\mathcal{\vec{K}}_{x}}\frac{\partial^{\left(b\right)}}{\partial\mathcal{\vec{K}}_{y}}-\frac{\partial^{\left(a\right)}}{\partial\mathcal{\vec{K}}_{y}}\frac{\partial^{\left(b\right)}}{\partial\mathcal{\vec{K}}_{x}}\right]H^{\left(a\right)}\left(\mathcal{\vec{K}},\mathcal{E}\right)\left(-iG^{<\left(b\right)}\left(\mathcal{\vec{K}},\mathcal{E}\right)\right)\text{.}$

By reverting to its equivalent matrix element expression and integrating with
respect to time, we obtain

$\sigma_{yx}=\frac{e^{2}}{h}\sum\limits_{\alpha}f\left(E_{\alpha}\right)\frac{i}{\left(2\pi\right)}\int\int
dk_{x}dk_{y}\left[\begin{array}[]{c}\left\langle\alpha,\frac{\partial}{\partial
k_{x}}\mathcal{\vec{K}},\mathcal{E}\right|\left|\alpha,\frac{\partial}{\partial
k_{y}}\mathcal{\vec{K}},\mathcal{E}\right\rangle\\\
-\left\langle\alpha,\frac{\partial}{\partial
k_{y}}\mathcal{\vec{K}},\mathcal{E}\right|\left|\alpha,\frac{\partial}{\partial
k_{x}}\mathcal{\vec{K}},\mathcal{E}\right\rangle\end{array}\right]\text{.}$
(40)

At low temperature, we can just write Eq. (40) as,

$\displaystyle\ \sigma_{yx}$ $\displaystyle=$
$\displaystyle\frac{ie^{2}}{2\pi\hbar}\frac{1}{\left(2\pi\right)}\sum\limits_{\alpha}\int\int_{occupiedBZ}dk_{x}dk_{y}\
\left[\nabla_{\vec{k}}\times\left\langle\alpha,\vec{k}\right|\frac{\partial}{\partial\vec{k}}\left|\alpha,\vec{k}\right\rangle\right]_{plane},$
(41) $\displaystyle=$
$\displaystyle\frac{e^{2}}{2\pi\hbar}\frac{i}{\left(2\pi\right)}\sum\limits_{\alpha}\mathop{\displaystyle\oint}dk_{c}\
\left[\left\langle\alpha,\vec{k}\right|\frac{\partial}{\partial
k_{c}}\left|\alpha,\vec{k}\right\rangle\right]_{contour}\text{.}$

where
$\left\langle\alpha,\vec{k}\right|\frac{\partial}{\partial\vec{k}}\left|\alpha,\vec{k}\right\rangle$
is the Berry connection in a band and $\mathop{\displaystyle\oint}dk_{c}\
\left[\left\langle\alpha,\vec{k}\right|i\frac{\partial}{\partial
k_{c}}\left|\alpha,\vec{k}\right\rangle\right]$ is the Berry phase. Thus, Eq.
(41) yields

$\sigma_{yx}=\sum\limits_{\alpha}\frac{e^{2}}{h}n_{\alpha}$

##  6 Concluding remarks

In this paper, we have shown that the direct route to generalized Peierls
phase factor or Wilson lines for crystalline solid under uniform electric
fields is embodied in the use of covariant derivatives (representing minimal
coupling in $U\left(1\right)$ gauge theory) in the finite translation
operators. The present calculation thus readily leads to the phase-space
topological invariant in nonequilibrium quantum transport equation which
yields the IQHE of electrical conductivity upon reverting to equivalent matrix
elements expression [1].

###### Acknowledgement 1

The author is grateful for the ’Balik Scientist’ Visiting Professor grant of
the PCIEERD-DOST, Philippines, at Cebu Normal University, Cebu City,
Philippines.

## References

  * [1] Felix A. Buot, Nonequilibrium superfield and lattice Weyl transform approach to quantum Hall effect, arXiv:2001.06993
  * [2] Felix A. Buot, Comments on the Weyl-Wigner calculus for lattice models,http://arxiv.org/abs/2103.10351
  * [3] Felix A. Buot, On the quantization of Hall effect in electrical conductivity: A nonequilibrium quantum superfield and lattice Weyl transform transport approach, AIP Conference Proceedings 2286, 030007 (2020).
  * [4] Felix A. Buot, ”Nonequilbrium Quantum Transport Physics in Nanosystems” (World Scientific, 2009) and references therein.
  * [5] F. A. Buot, Method for Calculating $TrH^{n}$ in Solid-State Theory, Phys. Rev. B10, 3700 (1974).
  * [6] F. A. Buot and K. L. Jensen, Lattice Weyl-Wigner Formulation of Exact Many-Body Quantum Transport Theory and Applications to Novel Quantum-Based Devices, Phys. Rev. B42, 9429-9456 (1990).
  * [7] F.A. Buot, A.R. Elnar, G. Maglasang, and R.E.S. Otadoy, On quantum Hall effect, Kosterlitz-Thouless phase transition, Dirac magnetic monopole, and Bohr-Sommerfeld quantization, J. Phys. Commun. 5, 025007 (2021).
  * [8] K. von Klitzing, G. Dorda, and M. Pepper, A New Method for High-Accuracy Determination of the Fine–Structure Constant Based on Quantized Hall Resistance, Phys. Rev. Lett, 45, 494 (1980).
  * [9] S. Mandelstam, Gauge Independent Formulation of Electrodynamics, Ann. Phys. (N.Y.) 19, 1 (1962).
  * [10] T.T. Wu and C.N. Yang, Concept Of Non-integrable Phase Factors And Global Formulation Of Gauge Fields, Phys. Rev. D12, 3845 (1975).
  * [11] G. H. Wannier, Dynamics of Band Electrons in Electric and Magnetic Fields, Rev. Mod. Phys. 34, 645 (1962).
  * [12] F.A. Buot, ”Zener Effect”, in Encyclopedia of Electrical and Electronics Engineering, Ed. John Webster, Vol. 23, pp. 669-688 (John Wiley, NY 1999). Wiley Online Library 2000 John Wiley & Sons, Inc.


# Novel View Synthesis with Neural Radiance Fields
for Industrial Robot Applications

Robert Langendörfer1 Max Heiken2 Max Mehltretter2 Andreas Schenk1 Martin
Weinmann1 Stefan Hinz1 Christian Heipke2 Markus Ulrich1 1Institute of
Photogrammetry and Remote Sensing, Karlsruhe Institute of Technology, Germany
- (markus.hillemann<EMAIL_ADDRESS>andreas.schenk@,
martin.weinmann@, stefan.hinz@, markus.ulrich@)kit.edu 2Institute of
Photogrammetry and GeoInformation, Leibniz Universität Hannover, Germany -
(heiken, mehltretter<EMAIL_ADDRESS>
###### Abstract

Neural Radiance Fields (NeRFs) have become a rapidly growing research field
with the potential to revolutionize typical photogrammetric workflows, such as
those used for 3D scene reconstruction. As input, NeRFs require multi-view
images with corresponding camera poses as well as the interior orientation. In
the typical NeRF workflow, the camera poses and the interior orientation are
estimated in advance with Structure from Motion (SfM). But the quality of the
resulting novel views, which depends on different parameters such as the
number and distribution of available images, the accuracy of the related
camera poses and interior orientation, but also the reflection characteristics
of the depicted scene, is difficult to predict. In addition, SfM is a time-
consuming pre-processing step, and its robustness and quality strongly depend
on the image content. Furthermore, the undefined scaling factor of SfM hinders
subsequent steps in which metric information is required. In this paper, we
evaluate the potential of NeRFs for industrial robot applications. To start
with, we propose an alternative to SfM pre-processing: we capture the input
images with a calibrated camera that is attached to the end effector of an
industrial robot and determine accurate camera poses with metric scale based
on the robot kinematics. We then investigate the quality of the novel views by
comparing them to ground truth, and by computing an internal quality measure
based on ensemble methods. For evaluation purposes, we acquire multiple
datasets that pose challenges for reconstruction typical of industrial
applications, like reflective objects, poor texture, and fine structures. We
show that the robot-based pose determination reaches similar accuracy as SfM
in non-demanding cases, while having clear advantages in more challenging
scenarios. We also report results of the novel view quality for different NeRF
approaches, showing that an additional online pose refinement may be
disadvantageous. Finally, we present first results of applying the ensemble
method to estimate the quality of the synthetic novel view in the absence of a
ground truth.

###### keywords:

neural radiance fields, novel view synthesis, uncertainty estimation,
industrial applications, robotics

##  1 Introduction

Neural Radiance Fields (NeRFs) (Mildenhall et al.,, 2020) have become a
rapidly growing research field. Given a set of oriented images, NeRFs allow
the generation of novel views of a learned 3D scene with impressive fidelity
to reality. In industrial applications, such novel views could be used, for
instance, to efficiently generate photorealistic training images for deep-
learning-based approaches like 6D object pose estimation in bin-picking
applications or image-based detection of anomalies. In general, the quality of
novel views depends on the accuracy of the input data (Lin et al.,, 2021;
Jäger et al.,, 2023). In the typical workflow, the camera poses and the
interior orientation of the camera are estimated in advance with established
Structure from Motion (SfM) approaches, such as COLMAP (Schönberger and
Frahm,, 2016). However, SfM is sensitive to the quality of local registration,
easily converges to suboptimal solutions (Lin et al.,, 2021), and sufficient
and well-distributed texture in the images is a prerequisite for accurate
results.

In this paper, we evaluate the potential of novel view synthesis with NeRFs
for industrial robot applications: An industrial robot is used to acquire the
input images with a camera that is mounted at its end effector. For each
acquired image that is used as input for NeRF training, the robot pose is
queried from the robot controller. In order to calculate the respective camera
poses, the pose between the tool coordinate system and the camera coordinate
system (hand–eye pose) must be known. It is determined by performing a
hand–eye calibration. The interior orientation of the camera is calibrated
simultaneously with the hand–eye calibration. Consequently, due to the very
high accuracy of modern industrial robots and target-based calibration
methods, the accuracy of our input data for NeRFs is high.

In addition, we investigate the quality of the novel views by comparing them
to ground truth using the established metrics Peak Signal to Noise Ratio
(PSNR)111https://lightning.ai/docs/torchmetrics/stable/image/peak_signal_noise_ratio.html
(last access 29/04/2024) and Structural SIMilarity (SSIM) (Wang et al.,, 2004)
incl. their respective standard deviations across the evaluation images for
different industrial scenes and different backgrounds. Because many industrial
applications benefit from knowing the uncertainty of the results, we further
compute an internal quality measure of the novel views based on ensemble
methods in order to be able to predict the quality of novel views without a
ground truth image.

##  2 Related work

The original concept of NeRF (Mildenhall et al.,, 2020) is to represent a
scene as continuous radiance field, where density and color are a function of
position and viewing angle. This function is commonly parameterized in terms
of a neural network. Images of a scene can be estimated from the radiance
field via volumetric rendering: Each pixel of an oriented image is regarded as
a ray intersecting the scene, where the color of a pixel is estimated by
integrating over the density and color along the ray. During training, this
concept is applied to render images for which there is a real counterpart in
the training data. The photometric error, i.e., the per pixel color error
between the rendered and the real image, is used to optimize the network
parameters. Barron et al., (2021) improve upon this method by regarding pixels
as cones instead of rays, which has an anti-aliasing effect and increases
rendering quality. This improvement is also adopted by Nerfacto (Tancik et
al.,, 2023), which additionally incorporates further extensions from various
other NeRF methods, such as the refinement of the camera poses from Bundle-
Adjusting Neural Radiance Fields (BARFs) (Lin et al.,, 2021), with the overall
goal of creating a compromise between quality and training time.

Inspired by NeRFs, Kerbl et al., (2023) developed 3D Gaussian Splatting
(3DGS), which is currently considered state-of-the-art in novel view synthesis
in terms of computation time and quality. 3DGS also uses volumetric rendering
to generate synthetic counterparts to real training images and uses the
photometric error as loss to optimize the scene representation. However,
instead of representing the scene as a continuous radiance field encoded in
the weights of a neural network, 3DGS uses a set of continuous 3D Gaussians.
The continuous representation with 3D Gaussians allows for optimization due to
its differentiability. At the same time, the scene is stored efficiently by
representing the 3D Gaussians as discrete ellipsoids with opacity and view-
dependent color information. Typically, as in other NeRF approaches, SfM is
employed to determine the required camera poses and interior orientation. The
sparse SfM point cloud is then used to initialize the set of 3D Gaussians,
which is subsequently densified and improved upon in an iterative
optimization. Like NeRFs, 3DGS can therefore be considered as an alternative
to Multi-View Stereo (MVS) approaches for dense 3D scene reconstruction in the
typical photogrammetric pipeline of SfM followed by MVS, see e.g. (Hackstein
et al.,, 2024) for a comparison of the resulting 3D surface and dense matching
results. However, unlike MVS, the aim of 3DGS is not to generate a 3D scene
reconstruction that is as dense and accurate as possible, but to find a
compact representation of the 3D scene in the form of 3D Gaussians that allows
novel views to be rendered as realistically as possible.

Lin et al., (2021) show that the accuracy of the camera poses limits the
quality of a radiance field and thus of rendered images. As the camera poses
obtained via SfM are often not accurate enough, especially in the presence of
poor or repetitive texture, some NeRF-based approaches (Lin et al.,, 2021;
Jeong et al.,, 2021; Wang et al.,, 2021; Bian et al.,, 2023; Tancik et al.,,
2023) jointly optimize the radiance field and the camera poses. However, these
approaches still require initial values for the camera poses, or employ a
coarse-to-fine schedule to try to avoid getting stuck in local minima, while
Bian et al., (2023) depend on matching point clouds of monocular depth maps.
Either strategy may fail in the industrial context. Furthermore, the
simultaneous estimation of camera poses and the radiance field introduces an
additional layer of ambiguity, leading to a slower convergence and a less
stable optimization (Bian et al.,, 2023). In the present work, we address this
problem by computing the camera poses directly from the known robot poses and
the calibrated hand–eye pose.

The DTU dataset (Aanæs et al.,, 2016) provides images that are often utilized
to characterize the performance of NeRFs. The images were captured with an
industrial robot arm and accurate camera poses are provided. A set of camera
poses was determined in advance with a high accuracy using a calibration
object. For image acquisition, the robot was moved to the same set of poses
again. As a result, the accuracy of the camera poses depends on the accuracy
of the determined poses of the calibration object and the pose repeatability
of the robot, which typically is very high. While this approach provides a
high-quality dataset, its use in practical robot applications is limited,
because it is impossible to flexibly choose the camera poses. Consequently,
distribution and number of camera poses cannot be automatically chosen, for
example based on the object’s characteristics, nor adapted, for example based
on previously occluded object parts. In contrast, our approach does not rely
on pre-calibrated camera poses but allows to use poses that are optimal for a
specific application. Since the absolute pose accuracy of industrial robots is
typically lower than its pose repeatability (Płaczek and Piszczek,, 2018), our
approach may lead to less accurate poses overall compared to the DTU dataset.
However, we consider the proposed data acquisition scheme to be more realistic
and flexible for industrial applications.

Jäger et al., (2023) study the impact of pose accuracy on the quality of
rendered images by simulating noisy poses. Šlapak et al., (2023) also study
the applicability of NeRFs in the industrial domain, but rely exclusively on
synthetic data. However, simulations like these are simplifications of reality
and the transferability to real scenarios remains open.

Lastly, most NeRF-based methods model a deterministic function that does not
allow estimating the uncertainty associated with the generated synthetic
images. Shen et al., (2022) address this limitation and approximate the
posterior distribution of the radiance field based on the training images.
Propagating this information through the volumetric rendering, the uncertainty
of the rendered color can be estimated per pixel in a synthetically generated
image. Sünderhauf et al., (2023) instead employ ensemble-based uncertainty
estimation, which is more straight-forward to implement and can be applied to
various NeRF architectures. Knowledge of the uncertainty associated with an
estimate is of particular importance in industry. However, to the best of the
author’s knowledge, the potential of uncertainty-aware NeRF-based methods has
not yet been investigated in the context of industrial applications. Thus, we
particularly focus on this aspect in our NeRF uncertainty investigations.

##  3 Setup of the industrial robot application

Our industrial robot setup consists of an IDS U3-31J0CP-C-HQ RGB camera with a
8 mm Tamron M23Fm08 lens rigidly attached to the end effector of a Universal
Robots UR3e robot arm, as shown in Figure 1(a). The UR3e has a maximum reach
of 500 mm, is calibrated by the manufacturer, and has a pose repeatability per
ISO 9283 of $\pm 0.03\,$mm. The RGB camera has a resolution of $2855\times
2848\,$px and the lens has a field of view of $58.5$°$\times 45.1$° as well as
a minimum object distance of $0.1\,$m.

(a)

(b)

Figure 1: (a) Hardware setup and data capturing. (b) Relevant 3D coordinate
systems, observed poses (green), and poses calculated during hand–eye
calibration (red); adapted from Ulrich and Hillemann, (2024).

We use the HALCON-implementation222www.mvtec.com/products/halcon (last access
29/04/2024) of the uncertainty-aware hand–eye calibration approach of Ulrich
and Hillemann, (2024) to simultaneously calculate the interior orientation of
the camera and the hand–eye pose. The approach minimizes the reprojection
error of points on a calibration object that was acquired by the camera in
multiple different robot poses. The uncertainty of the robot, which considers
pose repeatability and absolute pose accuracy, is explicitly taken into
account in this approach in order to increase the accuracy of the estimated
hand–eye pose. We capture $N=32$ images in total for calibration.

The result of hand–eye calibration is reported in terms of the root mean
square translation (RMST), root mean square rotation (RMSR), and reprojection
root mean square (RRMS) differences. Let
${{}^{\mathrm{b}}{\textnormal{{{H}}}}_{\mathrm{a}}}$ denote the 4$\times$4
homogeneous transformation matrix that represents a rigid 3D transformation of
points from coordinate system a to b. The basis for calculating RMST, RMSR,
and RRMS is the typical chain of transformations for hand–eye calibration
(Strobl and Hirzinger,, 2006)

${{}^{\mathrm{b}}{\textnormal{{{H}}}}_{\mathrm{w}}}={{}^{\mathrm{b}}{\textnormal{{{H}}}}_{\mathrm{t}}}_{,j}{{}^{\mathrm{t}}{\textnormal{{{H}}}}_{\mathrm{c}}}{{}^{\mathrm{c}}{\textnormal{{{H}}}}_{\mathrm{w}}}_{,j}\mathrm{,}$
(1)

with the coordinate systems world ($\mathrm{w}$), camera ($\mathrm{c}$), tool
($\mathrm{t}$), and base ($\mathrm{b}$), the $j$-th robot pose
${{}^{\mathrm{b}}{\textnormal{{{H}}}}_{\mathrm{t}}}_{,j}$, and camera pose
${{}^{\mathrm{c}}{\textnormal{{{H}}}}_{\mathrm{w}}}_{,j}$, as well as the
calibrated hand–eye pose ${{}^{\mathrm{t}}{\textnormal{{{H}}}}_{\mathrm{c}}}$
and world–base pose ${{}^{\mathrm{b}}{\textnormal{{{H}}}}_{\mathrm{w}}}$ as
shown in Figure 1(b). With the discrepancy matrix

$\Delta{\textnormal{{{H}}}}_{j}=({{}^{\mathrm{c}}{\textnormal{{{H}}}}_{\mathrm{t}}}{{}^{\mathrm{t}}{\textnormal{{{H}}}}_{\mathrm{b}}}_{,j}{{}^{\mathrm{b}}{\textnormal{{{H}}}}_{\mathrm{w}}})^{-1}{{}^{\mathrm{c}}{\textnormal{{{H}}}}_{\mathrm{w}}}_{,j}$
(2)

in this transformation chain, RMST and RMSR are calculated as follows:

$\mathrm{RMST}=\sqrt{\frac{1}{N}\sum_{j=1}^{N}\mathrm{trans}(\Delta{\textnormal{{{H}}}}_{j})^{2}}\enspace,$
(3)
$\mathrm{RMSR}=\sqrt{\frac{1}{N}\sum_{j=1}^{N}\mathrm{rot}(\Delta{\textnormal{{{H}}}}_{j})^{2}}\enspace.$
(4)

Here, trans($\cdot$) calculates the length of the translation component and
rot($\cdot$) the angle of the Rodrigues rotation of a matrix. The RRMS is
calculated by the root mean square of the differences of the observed image
points ${\textbf{{p}}}_{j,k}$ and the 3D points ${\textbf{{p}}}_{k}$
($k=1,\ldots,M$) of the calibration object projected into the image plane:

$\mathrm{RRMS}=\sqrt{\frac{1}{N}\sum_{j=1}^{N}\frac{1}{M}\sum_{k=1}^{M}\left({\textbf{{p}}}_{j,k}-\pi({{}^{\mathrm{c}}{\textnormal{{{H}}}}_{\mathrm{w}}}_{,j}\cdot{\textbf{{p}}}_{k},{\textbf{{i}}})\right)^{2}}\enspace,$
(5)

where $\pi({\textbf{{p}}}_{\mathrm{c}},{\textbf{{i}}})$ is the projection of
the point ${\textbf{{p}}}_{\mathrm{c}}$, which is given in the camera
coordinate system, into the image, i is the interior orientation, and

${{}^{\mathrm{c}}{\textnormal{{{H}}}}_{\mathrm{w}}}_{,j}={{}^{\mathrm{c}}{\textnormal{{{H}}}}_{\mathrm{t}}}{{}^{\mathrm{t}}{\textnormal{{{H}}}}_{\mathrm{b}}}_{,j}{{}^{\mathrm{b}}{\textnormal{{{H}}}}_{\mathrm{w}}}$
(6)

is the camera pose, which is calculated based on the transformation chain.

The values after hand–eye calibration are $\mathrm{RMST}=0.11$ mm,
$\mathrm{RMSR}=0.028$°, and $\mathrm{RRMS}=1.00\,$px. For comparison, we also
report the RRMS of a standard camera calibration with the same data, which is
$\mathrm{RRMS}=0.33\,$px. Here, the camera poses
${{}^{\mathrm{c}}{\textnormal{{{H}}}}_{\mathrm{w}}}_{,j}$ are estimated
directly and not based on the transformation chain via the robot. Moreover, we
report the mean standard deviations of the robot poses used for hand–eye
calibration. They are $\sigma_{t}=0.07$ mm for translation and
$\sigma_{R}=0.017$°for rotation, which is in the same order of magnitude.
Consequently, the accuracy of the robot poses corresponds to approximately two
to three times the pose repeatability of the robot. This is in line with
previous results (Ulrich and Hillemann,, 2024).

##  4 Experimental results

In this section, details of the data capturing are given, and the accuracy of
the camera poses calculated with our procedure as well as with the
conventional procedure using SfM is assessed. Subsequently, details of the
evaluated NeRFs are described before their results are presented
quantitatively and qualitatively on the basis of a meaningful selection of
examples. Finally, exemplary results of the uncertainty quantification are
presented.

###  4.1 Datasets

We capture images of three different industry-related objects (see Figure 2).
In principle, our approach allows us to select the optimal poses for each
object individually without additional computational effort. For the
evaluations in this paper, however, we select the same systematically arranged
poses for each dataset. The camera poses are sampled on a hemisphere with
radius $R=0.2$ m, with the camera always pointing approximately to the center
of the object. The camera is upside down, as this allows a larger radius. For
the sake of clarity, all shown images of the datasets are rotated by 180°. The
sampling interval is $d_{\mathrm{lat}}=5^{\circ}$ in latitude and
$d_{\mathrm{lon}}=5^{\circ}$ in longitude with an elevation between 55 and
85°, which results in 504 images per dataset. All objects have poor texture
and a (partially) reflective surface. Additionally, the light bulb (Figures
2(b) and (e)) is partly transparent and the metal brush (Figures 2(c) and (f))
contains fine structures. These characteristics pose challenges for image
processing in general and novel view synthesis with NeRFs in particular, and
allow us to explore the limits of NeRFs. At the same time, the objects are
relevant in practice, since they are typical for industrial applications.

(a) Dataset 1

(b) Dataset 2

(c) Dataset 3

(d) Dataset 4

(e) Dataset 5

(f) Dataset 6

Figure 2: Example image of each dataset.

In total, we capture six different datasets, whereby each object is captured
with two different backgrounds: An aluminum profile and a homogeneous green
background. The aluminum profile provides a good fine-grained texture and thus
allows for accurate SfM results. As already mentioned, such a good texture is
often not available in industrial applications. Therefore, we also collect
data with a homogeneous background on which no feature points can be
extracted. In such cases, the estimation of the camera poses with SfM relies
on the texture of the objects.

###  4.2 Pose accuracy

To analyze the accuracy of the camera poses determined with COLMAP and the
camera poses approached with the robot, we compare them to a ground truth
(GT). For this GT, the poses are determined in a third procedure with higher
accuracy by using a calibration object and spatial resection. To demonstrate
the higher accuracy of the GT, the RRMS is shown in Table 1 for each of the
three procedures. The calculation of the RRMS is based on Equation (5). Note
that ${\textbf{{p}}}_{k}$ are points of the calibration target for the GT and
the robot, and points of the sparse point cloud for COLMAP. In addition, for
the GT and COLMAP, the camera poses
${{}^{\mathrm{c}}{\textnormal{{{H}}}}_{\mathrm{w}}}_{,j}$ are calculated
directly, whereas for the robot it is calculated via the transformation chain
with Equation (6). The RRMS for the GT is considerably smaller compared to the
COLMAP and the robot variant. Further, Table 1 shows the Mean Translation
Error (MTE) and the Mean Rotation Error (MRE) of COLMAP and the robot compared
to GT. MTE and MRE333cf. translation_part and rotation_angle_deg of APE at
https://github.com/MichaelGrupp/evo/blob/master/notebooks/metrics.py_API_Documentation.ipynb
(last access 29/04/2024) are computed with the tool _evo_
444https://github.com/MichaelGrupp/evo (last access 29/04/2024), where the
poses are aligned in a pre-processing step. The alignment includes rotation
and translation for the robot poses and additionally scale for COLMAP poses.
Note, that these metrics differ from RMST and RMSR. As MTE and MRE are
influenced by the resection as well as by COLMAP / the robot, they should not
be interpreted as uncertainties of the COLMAP or of the robot results only.
Also, the metrics ignore the (potentially high) correlation between the
elements of the poses. We have chosen to still use them in order to simplify
the computations; see Förstner and Wrobel, (2016), p. 120 ff. for details
about taking into account the mentioned correlations by comparing the
respective covariance matrices.

Table 1: Reprojection Root Mean Squared difference (RRMS), Mean Translation Error (MTE), Mean Rotation Error (MRE), and processing time for the pose calculation with COLMAP and our approach with the robot. Dataset |  RRMS $\downarrow$ |  MTE $\downarrow$ |  MRE $\downarrow$ |  Time $\downarrow$
---|---|---|---|---
| in px | in mm | in ° | in sec
GT | 0.35 | - | - | -
COLMAP:
Dataset 1 | 1.13 | 0.08 | 0.051 | 10994
Dataset 2 | 1.10 | 0.14 | 0.091 | 9730
Dataset 3 | 1.23 | 0.08 | 0.054 | 10544
Dataset 4 | 1.36 | 0.13 | 0.123 | 1897
Dataset 5 | 1.93 | 3.77 | 179.037 | 1601
Dataset 6 | 1.59 | 0.11 | 0.074 | 5207
Robot | 0.96 | 0.10 | 0.079 | 0.2

The poses determined with the robot consistently have a high accuracy, and as
opposed to those computed with COLMAP, they are obviously independent of the
image content. The accuracy of the poses determined with COLMAP, on the other
hand, varies. For all datasets except dataset 5, the poses are similarly
accurate. For dataset 5, the accuracy of the poses computed with COLMAP is
significantly lower than that of the robot. This is probably due to strong
reflections on the light bulb in this dataset, which result in mirrored
feature points from the background being used to determine the poses. Lastly,
Table 1 contains the processing times for COLMAP and our approach with the
robot. COLMAP was processed with enabled GPU-acceleration on an NVIDIA RTX
2060 and takes between 26 min and 3 h for each of our datasets. Since no
feature points are found in the homogeneous background of datasets 4 to 6, the
related processing times are lower than for datasets 1 to 3. However, this
processing time is almost completely eliminated in our approach with the robot
although a non-optimized CPU implementation was used. The time measurement for
our approach includes the following three steps that need to be performed for
each image: i) Read joint angles, ii) apply forward kinematics to calculate
the robot poses, and iii) apply hand–eye pose to calculate the camera poses.

Figure 3 visualizes the result of COLMAP for dataset 4 in the form of the
poses in red and the reconstructed, sparse point cloud. The result is
plausible, however, no points can be reconstructed for the homogeneous
background. The poses of the images are therefore calculated based on a less
favorable distribution of the image points, which explains the lower pose
accuracy.

Figure 3: Result of COLMAP for Dataset 4, i.e., the reconstructed sparse point
cloud and the estimated poses.

###  4.3 Evaluation of the NeRF approaches

In this subsection, we report experiments investigating the quality of the
generated novel views by using different NeRF approaches from the literature
and different sets of camera poses. The criteria used for the evaluation are
the established metrics PSNR and SSIM, and their respective standard
deviations across the evaluation images.

From the large number of NeRF approaches that nowadays exist, we select
Nerfacto (Tancik et al.,, 2023) and 3DGS (Kerbl et al.,, 2023) for our
experiments. Nerfacto is the standard method of Nerfstudio (Tancik et al.,,
2023) and is a representative of the methods that can be trained in a short
time. We evaluate four different versions of Nerfacto: Nerfacto with and
without pose refinement as explained in Lin et al., (2021) as well as
Nerfacto-big with and without pose refinement. There are four main differences
between Nerfacto and Nerfacto-big: (a) The hashmap of Nerfacto-big has double
the resolution (4096 vs. 2048) and four times the capacity (2M vs 500k)
compared to Nerfacto; (b) Nerfacto-big trains 8192 rays per batch, in contrast
to 4096 for Nerfacto; (c) Nerfacto-big uses a larger network, i.e. Nerfacto-
big uses twice as many neurons per hidden layer as Nerfacto (128 vs. 64); (d)
Nerfacto-big computes 100,000 iterations, whereas Nerfacto only iterates
30,000 times.

3DGS (Kerbl et al.,, 2023) is currently considered the state-of-the-art method
in terms of training and inference time as well as of visual quality of the
rendered images. We use the 3DGS implementation Splatfacto from Nerfstudio
(Tancik et al.,, 2023) to ensure a uniform evaluation.

The various NeRF approaches are evaluated twice with different camera poses:

  1. 1.

COLMAP version: COLMAP is used to calculate the interior orientation and the
camera poses in a pre-processing step that needs to be performed for each
dataset separately. The COLMAP version corresponds to the conventional
procedure in the NeRF community.

  2. 2.

Robot version: Like mentioned in Section 3, the camera and the hand–eye
calibration are performed once in advance. The robot poses are queried from
the robot’s kinematics and together with the result of the hand–eye
calibration, the camera poses are computed based on Equation (1). The robot
version does not require a separate pre-processing step for each dataset.

For 3DGS, an initial point cloud needs to be provided. For the COLMAP version,
the calculation of this point cloud is straight-forward. For the robot
version, we use COLMAP to calculate the sparse point cloud but fix the
interior orientation and the camera poses of the robot version during
optimization.

Unless otherwise specified, we use the default parameter settings of
Nerfstudio for our experiments. In this way, we can evaluate the methods from
the point of view of how easily they can be transferred to our industrial
datasets. However, this means that higher visual quality may be achievable by
tuning the parameters.

###  4.4 Quantitative results of the NeRF methods

Table 2 shows the results of Nerfacto and 3DGS based on PSNR and SSIM and the
respective standard deviations across the evaluation images, which are not
part of the set of training images. The top part of the table shows the
results of the COLMAP version and the bottom part the results of the robot
version. The results in Table 2 are described in more detail below under
different aspects.

Table 2: Results and standard deviations of different NeRF methods for the
COLMAP version (top) and the robot version (bottom). ’pr’ means pose
refinement.

Dataset 1 Dataset 2 Dataset 3 Dataset 4 Dataset 5 Dataset 6 Method PSNR
$\uparrow$ SSIM $\uparrow$ PSNR $\uparrow$ SSIM $\uparrow$ PSNR $\uparrow$
SSIM $\uparrow$ PSNR $\uparrow$ SSIM $\uparrow$ PSNR $\uparrow$ SSIM
$\uparrow$ PSNR $\uparrow$ SSIM $\uparrow$ COLMAP version: Nerfacto w/o pr
31.3$\pm$1.4 0.892$\pm$0.006 31.0$\pm$1.5 0.884$\pm$0.005 32.1$\pm$1.8
0.894$\pm$0.007 29.0$\pm$2.1 0.941$\pm$0.006 26.1$\pm$2.5 0.909$\pm$0.032
37.8$\pm$1.5 0.963$\pm$0.004 Nerfacto w/ pr 25.0$\pm$2.5 0.785$\pm$0.035
23.8$\pm$2.5 0.784$\pm$0.032 23.1$\pm$3.1 0.734$\pm$0.036 23.4$\pm$0.7
0.911$\pm$0.005 27.3$\pm$2.2 0.917$\pm$0.024 31.1$\pm$0.5 0.908$\pm$0.007
Nerfacto-big w/o pr 30.1$\pm$1.8 0.909$\pm$0.005 29.1$\pm$2.4 0.895$\pm$0.010
32.2$\pm$2.2 0.918$\pm$0.006 25.6$\pm$2.1 0.937$\pm$0.012 21.3$\pm$1.2
0.817$\pm$0.032 21.6$\pm$1.0 0.846$\pm$0.016 Nerfacto-big w/ pr 23.9$\pm$0.9
0.776$\pm$0.030 25.8$\pm$1.3 0.799$\pm$0.017 20.3$\pm$0.9 0.695$\pm$0.032
20.2$\pm$2.2 0.901$\pm$0.012 17.6$\pm$1.0 0.716$\pm$0.036 24.2$\pm$0.9
0.849$\pm$0.015 3DGS 32.1$\pm$1.5 0.930$\pm$0.004 31.8$\pm$1.5 0.925$\pm$0.003
35.1$\pm$1.2 0.936$\pm$0.004 30.4$\pm$2.2 0.950$\pm$0.005 27.0$\pm$1.4
0.925$\pm$0.007 41.0$\pm$1.1 0.978$\pm$0.002 Robot version: Nerfacto w/o pr
31.3$\pm$1.4 0.883$\pm$0.009 30.1$\pm$1.4 0.870$\pm$0.012 31.3$\pm$1.8
0.876$\pm$0.012 26.6$\pm$2.1 0.933$\pm$0.007 33.3$\pm$1.5 0.951$\pm$0.006
37.1$\pm$1.0 0.957$\pm$0.004 Nerfacto w/ pr 25.1$\pm$2.5 0.783$\pm$0.042
25.2$\pm$1.5 0.791$\pm$0.028 23.2$\pm$3.3 0.734$\pm$0.050 22.8$\pm$1.1
0.904$\pm$0.005 28.1$\pm$2.6 0.938$\pm$0.011 31.1$\pm$1.5 0.909$\pm$0.013
Nerfacto-big w/o pr 29.9$\pm$1.5 0.894$\pm$0.008 28.9$\pm$2.2 0.881$\pm$0.016
31.1$\pm$2.1 0.893$\pm$0.016 24.9$\pm$2.2 0.927$\pm$0.012 24.8$\pm$1.9
0.886$\pm$0.033 21.2$\pm$0.7 0.823$\pm$0.014 Nerfacto-big w/ pr 22.5$\pm$2.8
0.756$\pm$0.039 26.5$\pm$2.3 0.812$\pm$0.022 23.5$\pm$2.0 0.731$\pm$0.033
23.3$\pm$2.4 0.914$\pm$0.014 23.4$\pm$1.6 0.866$\pm$0.024 25.2$\pm$1.2
0.882$\pm$0.022 3DGS 32.0$\pm$1.3 0.923$\pm$0.006 31.5$\pm$1.6 0.915$\pm$0.015
34.3$\pm$1.3 0.923$\pm$0.013 30.1$\pm$2.0 0.958$\pm$0.005 33.3$\pm$1.8
0.965$\pm$0.004 40.3$\pm$1.0 0.975$\pm$0.003

COLMAP vs. Robot. The results of the COLMAP and the robot version do not
differ significantly with one exception: For dataset 5, PSNR and SSIM are
considerably higher for the robot versions for all methods. The reason is the
higher accuracy of camera poses of the robot version compared to COLMAP (see
Table 1). This highlights the disadvantage of using SfM to determine the
camera poses in cases where the image content is challenging for SfM. Using
the robot for pose estimation is more robust and provides consistent accuracy,
whereas with COLMAP the accuracy strongly depends on the image content.

Best performing methods. On all datasets, 3DGS achieves higher values for PSNR
and SSIM than all Nerfacto methods while having faster training and inference
times. Nerfacto w/o pose refinement achieves the second-best results.
Interestingly, Nerfacto-big achieves worse results than Nerfacto, even though
it uses a larger network and trains for more iterations. Apparently, the size
of Nerfacto’s network is already sufficient to represent the given complexity
of the industrial scenes. With the exception of dataset 5, the choice of the
method has a larger impact on the evaluation metrics than the way in which the
poses are determined.

Pose refinement. The comparison of the Nerfacto methods with and without pose
refinement clearly shows that the pose refinement leads to a considerable
deterioration for our industrial datasets. The poses, which can already be
determined very accurately with both COLMAP and the robot, become less
accurate due to the simultaneous optimization during NeRF training, which
leads to lower PSNR and SSIM values. When looking at the rendered images,
difference between the results with and without pose refinement are small.
Nevertheless, it can be seen that the variant with pose refinement renders
images that are rotated and shifted in relation to the reference image. We
therefore argue that the pose refinement is responsible for the degradation.
We attribute this finding to the higher ambiguity during training of the
methods with simultaneous pose refinement and the already high accuracy of the
input poses.

###  4.5 Qualitative results of the NeRF methods

In general, the visual quality of the rendered images reflects the results in
Table 2 very well. For this reason, a small number of images are selected here
for the purpose of illustration.

Figure 4 visualizes the reference view and the rendered image of the robot
version of 3DGS for an evaluation image of dataset 3. With 3DGS,
extraordinarily realistic images can be rendered for this dataset. There is no
recognizable difference to the reference view. Consequently, the NeRF methods
are capable of densifying an industrial dataset with realistically rendered
images.

(a) Reference view

(b) 3DGS

Figure 4: Reference view and rendered result of the robot version of 3DGS for
an image of dataset 3.

(a) Reference view

(b) Nerfacto-big w/o pose refinement

Figure 5: Reference view and rendered result of the COLMAP version of
Nerfacto-big without pose refinement for an image of dataset 4.

Figure 5 shows the reference view and the rendered image of the COLMAP version
of Nerfacto-big without pose refinement for an image of dataset 4. In this
case, the image rendered by Nerfacto-big contains artifacts. In addition, the
reflections, which are generally a challenge, are not reproduced completely
correctly.

(a) Reference view

(b) COLMAP version of 3DGS

(c) Robot version of 3DGS

Figure 6: Reference view and rendered results of the COLMAP and the robot
version of 3DGS for an image of dataset 5.

Figure 6 visualizes the reference view as well as the results of the COLMAP
and the robot version for an image of dataset 5. Exemplary results from 3DGS
are shown, but the same applies to the other methods. The COLMAP version of
3DGS and also of the other methods struggle to model the base of the bulb.
Interestingly, at the same time, the reflections of the background are
reconstructed in fine detail. We assume that COLMAP has found a plausible
solution for calculating poses for the reflected image areas. These poses then
lead to a good reconstruction of the reflected background in the NeRFs. For
the area on the object that is not reflected, however, the poses do not
represent a plausible solution. In contrast, for the robot version of 3DGS,
the base of the bulb can be reconstructed and the rendered image looks very
similar to the reference view, which matches the higher PSNR and SSIM values
in Table 2. However, the reflections of the background look slightly blurrier.
Although the results for this difficult object are impressive, this example
shows that strong reflections are still a major challenge for novel view
synthesis, even for the state-of-the-art method.

###  4.6 Uncertainty quantification with a NeRF ensemble

In order to investigate the uncertainty of novel view generation using NeRF
approaches, in the following experiments, the simplified NeRF ensemble
approach as described in (Sünderhauf et al.,, 2023) is applied exemplarily to
dataset 1. Each of the $M$ ensemble members is a Nerfacto (Tancik et al.,,
2023) model, with pose optimization disabled and otherwise default
hyperparameters. Randomness is achieved through different initialization of
the learnable parameters.

Then, the ensemble prediction of the color $\hat{\mathbf{c}}_{E}(\mathbf{r})$,
$\mathbf{c}=(r,g,b)$ of a ray $\mathbf{r}$ is the arithmetic mean across all
ensemble members’ color predictions $\hat{\mathbf{c}}_{m}(\mathbf{r})$ for
this ray.

$\hat{\mathbf{c}}_{E}(\mathbf{r})=\frac{1}{M}\sum^{M}_{m=1}\hat{\mathbf{c}}_{m}(\mathbf{r})$
(7)

Similarly, the uncertainty is expressed as the standard deviation of the
individual predictions, aggregating the three color channels:

$\sigma_{\mathbf{c}_{E}}(\mathbf{r})=\sqrt{\frac{1}{3M}\sum_{m=1}^{M}\Delta\mathbf{c}_{m,E}^{2}(\mathbf{r})}$
(8)

with the residuals between ensemble prediction and reference view

$\Delta\mathbf{c}_{m,E}^{2}(\mathbf{r})=\sum_{i=1}^{3}\left(\hat{\mathbf{c}}_{m,i}(\mathbf{r})-\hat{\mathbf{c}}_{E}(\mathbf{r})\right)^{2}.$
(9)

####  4.6.1 In-distribution experiment.

For evaluation of the uncertainty quantification, each ensemble member
($M=30$) is trained using a subset of dataset 1 with 100 regularly distributed
images, but otherwise independently of each other. The evaluation images are
located in-between training views, i.e. in-distribution.

Figure 7 shows an example ensemble prediction of an evaluation image, allowing
for a qualitative evaluation. Although the shape appears to be captured well,
the color prediction (b) exhibits fewer details than the reference view (a),
resulting in a smoothed appearance. This is reflected in the residuals
$\Delta\mathbf{c}_{E,ref}$ (c) appearing grainy. The cause can be found in the
individual members’ predictions, which also appear smoothed, indicating that
the default Nerfacto model lacks capacity. Comparing (c) and (d), the
residuals visually correlate with the predicted standard deviation to some
degree. For example, the oblong specular reflection on the edge of the
cylinder visible in (a) is less pronounced in the prediction, leading to large
residuals, but also a large standard deviation where members are in
disagreement. A second area of high standard deviation can be found to the
left of the object. This is caused by the presence of a floater in a single
member’s prediction, which can be considered an outlier. This violates the
assumption of normally distributed colors.

It is apparent that the predicted standard deviation is too low in many areas
of large residuals. Prominently, the edge of the base of the object and the
grooves of the aluminum profile in the background exhibit very large
residuals, but only moderate standard deviations. The ensemble is over-
confident. Sünderhauf et al., (2023) propose to extend the uncertainty
equation by the squared inverse accumulated density (visualized in Figure
7(e), different color scale) to account for unobserved structure. In our case,
the comparatively lower density of the base plate does not coincide well with
the large residuals.

(a) Reference view

(b) Ensemble prediction

(c) Magnitude of residuals

(d) Ensemble standard deviation

(e) Inverse accumulated density

Figure 7: NeRF ensemble results for an image of dataset 1.

####  4.6.2 Out-of-distribution experiment.

In this experiment, only training views located in one half of the dome are
used in order to evaluate the ensemble’s performance with out-of-distribution
samples. With a total of 60 views for ensemble training, $M=10$ ensemble
members are trained with slightly different subsets of 50 training views each.

Figure 8 shows the ensemble prediction for an example evaluation image that is
located opposite the training views, i.e. on the back side. Remarkably, the
prediction (b) nevertheless shows that the general shape and color of the
scene have been captured to some degree. However, parts of the scene that were
unobserved during training exhibit artifacts, and the surface lacks
significantly in detail. View-dependent effects like specular reflection can
be observed to cause large residuals. These effects are not captured in the
standard deviation: The strong view-dependent component of the radiance field
at the reflective surface was underfit to a similar degree in each ensemble
member. This is a limitation of modeling the distribution in image space. On
the other hand, the patch of large predicted standard deviation stems from
inconsistent 3D structure, a typical artifact in unobserved regions of a NeRF
and thus a desired outcome.

(a) Reference view

(b) Ensemble prediction

(c) Magnitude of residuals

(d) Ensemble standard deviation

Figure 8: Ensemble prediction on out-of-distribution evaluation image

##  5 Conclusion

In this paper, we evaluated the potential of novel view synthesis with NeRFs
for industrial robot applications. We captured the input images with a camera
attached to the end effector of an industrial robot and determined accurate
camera poses based on the robot kinematics. Our datasets contain typical
challenges for industrial applications, like reflecting objects, poor texture,
and fine structures, and allow us to explore the limits of the conventional
approach of using SfM to determine the camera poses. Our experiments show that
with careful calibration and an accurate robot, this approach achieves
comparable quality to the conventional method. However, our approach is more
robust and leads to consistently high quality of images rendered by the NeRFs,
whereas the conventional approach requires image content favorable for image-
based determination of the camera poses. Furthermore, our approach determines
the camera poses with metric scale, which favors potential subsequent steps in
which metric information is required. Lastly, it is faster when flexible
adjustments of the poses are required, which is a critical factor across
various applications.

Since many industrial applications require a high degree of reliability, we
further quantify uncertainties of the rendered images. In principle, a
simplified ensemble approach provides the correct tendencies for the
uncertainties for in-distribution views, as shown by the correlation with the
residuals. However, the ensemble’s predicted standard deviation is often too
low in areas of large residuals, indicating overconfidence. Furthermore, the
experiment on out-of-distribution poses shows that NeRFs are, in principle,
able to generate novel views from perspectives unseen during training.
However, these views exhibit fewer details in texture and structure, and view-
dependent effects like specular reflection cause large residuals. These
effects are not captured in the ensembles’ standard deviation, which
demonstrates the limitations of the ensemble approach.

In the future, we will extend our analysis from novel view synthesis to the
task of 3D scene or object reconstruction, respectively, in the industrial
context. Furthermore, we will also look into the thermal domain for industrial
applications. Since determining the camera poses and interior orientation with
SfM is expected to be more challenging for thermal images, we consider the
potential of using a robot for pose estimation to be even higher when applied
to e.g. multi-modal NeRF approaches (Poggi et al.,, 2022) in order to enable
cross-spectral scene representation and analysis.

## References

  * Aanæs et al., (2016) Aanæs, H., Jensen, R. R., Vogiatzis, G., Tola, E., Dahl, A. B., 2016. Large-scale data for multiple-view stereopsis. International Journal of Computer Vision, 120, 153–168.
  * Barron et al., (2021) Barron, J. T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., Srinivasan, P. P., 2021. Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields. _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 5855–5864.
  * Bian et al., (2023) Bian, W., Wang, Z., Li, K., Bian, J.-W., Prisacariu, V. A., 2023. NoPe-NeRF: Optimising neural radiance field with no pose prior. _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 4160–4169.
  * Förstner and Wrobel, (2016) Förstner, W., Wrobel, B., 2016. Photogrammetric Computer Vision. Springer, Cham.
  * Hackstein et al., (2024) Hackstein, V., Fauth-Mayer, P., Rothermel, M., Haala, N., 2024. Depth Supervised Neural Surface Reconstruction from Airborne Imagery. arXiv preprint arXiv:2404.16429.
  * Jäger et al., (2023) Jäger, M., Landgraf, S., Jutzi, B., 2023. Density Uncertainty Quantification with NeRF-Ensembles: Impact of Data and Scene Constraints. arXiv preprint arXiv:2312.14664.
  * Jeong et al., (2021) Jeong, Y., Ahn, S., Choy, C., Anandkumar, A., Cho, M., Park, J., 2021. Self-calibrating neural radiance fields. _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 5846–5854.
  * Kerbl et al., (2023) Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G., 2023. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics, 42(4), 1–14.
  * Lin et al., (2021) Lin, C.-H., Ma, W.-C., Torralba, A., Lucey, S., 2021. BARF: Bundle-adjusting neural radiance fields. _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 5741–5751.
  * Mildenhall et al., (2020) Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., Ng, R., 2020. NeRF: Representing scenes as neural radiance fields for view synthesis. _Proceedings of the European Conference on Computer Vision_ , Springer, 405–421.
  * Płaczek and Piszczek, (2018) Płaczek, M., Piszczek, Ł., 2018. Testing of an industrial robot’s accuracy and repeatability in off and online environments. Eksploatacja i Niezawodnosc — Maintenance and Reliability, 20(3), 455–464.
  * Poggi et al., (2022) Poggi, M., Zama Ramirez, P., Tosi, F., Salti, S., Di Stefano, L., Mattoccia, S., 2022. Cross-spectral neural radiance fields. _Proceedings of the International Conference on 3D Vision_. 3DV.
  * Schönberger and Frahm, (2016) Schönberger, J. L., Frahm, J.-M., 2016. Structure-from-motion revisited. _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 4104–4113.
  * Shen et al., (2022) Shen, J., Agudo, A., Moreno-Noguer, F., Ruiz, A., 2022. Conditional-flow nerf: Accurate 3d modelling with reliable uncertainty quantification. _Proceedings of the European Conference on Computer Vision_ , Springer, 540–557.
  * Šlapak et al., (2023) Šlapak, E., Pardo, E., Dopiriak, M., Maksymyuk, T., Gazda, J., 2023. Neural radiance fields in the industrial and robotics domain: applications, research opportunities and use cases. arXiv preprint arXiv:2308.07118.
  * Strobl and Hirzinger, (2006) Strobl, K. H., Hirzinger, G., 2006. Optimal hand-eye calibration. _Proceedings of the 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 4647–4653.
  * Sünderhauf et al., (2023) Sünderhauf, N., Abou-Chakra, J., Miller, D., 2023. Density-aware NeRF Ensembles: Quantifying Predictive Uncertainty in Neural Radiance Fields. _Proceedings of the 2023 IEEE International Conference on Robotics and Automation_ , 9370–9376.
  * Tancik et al., (2023) Tancik, M., Weber, E., Ng, E., Li, R., Yi, B., Wang, T., Kristoffersen, A., Austin, J., Salahi, K., Ahuja, A. et al., 2023. Nerfstudio: A modular framework for neural radiance field development. _ACM SIGGRAPH 2023 Conference Proceedings_ , 1–12.
  * Ulrich and Hillemann, (2024) Ulrich, M., Hillemann, M., 2024. Uncertainty-Aware Hand–Eye Calibration. IEEE Transactions on Robotics, 40, 573–591.
  * Wang et al., (2004) Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E., 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600-612.
  * Wang et al., (2021) Wang, Z., Wu, S., Xie, W., Chen, M., Prisacariu, V. A., 2021. NeRF - -: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064.


CR7d_LRCR_c_all | 15 |  0.027 (0.022 - 0.03) |  0.104 (0.091 - 0.124) |  0.732 (0.716 - 0.746) |  1.176 (1.004 - 1.272) |  0.02 (0.014 - 0.036) |  0.963 (0.876 - 1.051)
CR7d_LRCR_c_all | 16 |  0.026 (0.022 - 0.03) |  0.101 (0.089 - 0.117) |  0.74 (0.717 - 0.757) |  1.192 (1.025 - 1.351) |  0.019 (0.012 - 0.033) |  1.038 (0.942 - 1.099)
CR7d_LRCR_c_all | 17 |  0.022 (0.017 - 0.026) |  0.094 (0.079 - 0.114) |  0.717 (0.697 - 0.734) |  1.05 (0.888 - 1.226) |  0.02 (0.012 - 0.029) |  0.888 (0.819 - 0.968)
CR7d_LRCR_c_all | 18 |  0.019 (0.013 - 0.024) |  0.085 (0.074 - 0.101) |  0.705 (0.679 - 0.738) |  0.996 (0.84 - 1.242) |  0.018 (0.011 - 0.029) |  0.983 (0.896 - 1.089)
CR7d_LRCR_c_all | 19 |  0.014 (0.009 - 0.02) |  0.067 (0.058 - 0.08) |  0.707 (0.676 - 0.726) |  1.006 (0.824 - 1.163) |  0.019 (0.012 - 0.025) |  1.037 (0.92 - 1.191)
CR7d_LRCR_c_all | 20 |  0.013 (0.007 - 0.017) |  0.066 (0.054 - 0.078) |  0.701 (0.672 - 0.728) |  0.975 (0.844 - 1.238) |  0.019 (0.012 - 0.031) |  1.051 (0.899 - 1.234)
CR7d_LRCR_c_all | 21 |  0.009 (0.003 - 0.016) |  0.058 (0.048 - 0.072) |  0.686 (0.656 - 0.719) |  0.938 (0.775 - 1.202) |  0.021 (0.012 - 0.033) |  1.15 (0.934 - 1.317)
CR7d_LRCR_c_all | 22 |  0.006 (-0.001 - 0.013) |  0.048 (0.04 - 0.069) |  0.687 (0.658 - 0.716) |  0.989 (0.803 - 1.161) |  0.028 (0.019 - 0.039) |  1.196 (0.974 - 1.429)
CR7d_LRCR_c_all | 23 |  0.006 (-0.001 - 0.011) |  0.048 (0.037 - 0.067) |  0.674 (0.648 - 0.706) |  0.933 (0.775 - 1.165) |  0.033 (0.026 - 0.047) |  1.106 (0.929 - 1.332)
CR7d_LRCR_c_all | 24 |  0.008 (-0.004 - 0.015) |  0.049 (0.036 - 0.069) |  0.697 (0.669 - 0.729) |  1.13 (0.893 - 1.413) |  0.032 (0.024 - 0.045) |  1.218 (0.991 - 1.48)
CR7d_LRCR_c_all | 25 |  0.007 (-0.006 - 0.014) |  0.048 (0.036 - 0.074) |  0.688 (0.657 - 0.721) |  0.987 (0.818 - 1.288) |  0.033 (0.022 - 0.049) |  1.178 (1.017 - 1.485)
CR7d_LRCR_c_all | 26 |  0.008 (-0.003 - 0.013) |  0.052 (0.038 - 0.085) |  0.681 (0.643 - 0.714) |  0.996 (0.758 - 1.208) |  0.041 (0.026 - 0.059) |  1.16 (0.954 - 1.473)
CR7d_LRCR_c_all | 27 |  0 (-0.007 - 0.007) |  0.04 (0.031 - 0.05) |  0.646 (0.61 - 0.686) |  0.837 (0.608 - 1.05) |  0.039 (0.023 - 0.058) |  1.142 (0.928 - 1.421)
CR7d_LRCR_c_all | 28 |  0.004 (-0.004 - 0.009) |  0.044 (0.034 - 0.056) |  0.658 (0.632 - 0.687) |  0.866 (0.682 - 1.103) |  0.043 (0.032 - 0.058) |  1.001 (0.861 - 1.2)
CR7d_LRCR_c_all | 29 |  0.009 (0.001 - 0.015) |  0.051 (0.04 - 0.065) |  0.698 (0.661 - 0.727) |  1.037 (0.814 - 1.403) |  0.038 (0.022 - 0.055) |  1.032 (0.848 - 1.208)
CR7d_LRCR_c_all | 30 |  0.011 (0.003 - 0.02) |  0.053 (0.04 - 0.071) |  0.714 (0.675 - 0.757) |  1.216 (0.93 - 1.59) |  0.04 (0.029 - 0.053) |  1.041 (0.858 - 1.237)
Table 12: Time-dependent metrics table (continued)

####  8.8.3 Pooled model evaluation

In contrast to the time-dependent evaluation, the “pooled” evaluation
considers all landmarks in the model pooled together (each landmark represents
an independent observation). The pooled evaluation metric are presented in
Figure 24 The findings are similar to the time-dependent evaluation, with the
survival model with competing events censoring at their event time showing
poorer performance especially in terms of calibration.

Figure 24: Prediction performance for dynamic models - pooled metrics

####  8.8.4 ROC curves

ROC curves are calculated using the “pooled” predictions (each landmark
represents an independent observation) and presented in Figure 25 for each
test set and each model.

Figure 25: ROC curves for dynamic models

####  8.8.5 Precision-recall curves

Precision-recall curves are calculated using the “pooled” predictions (each
landmark represents an independent observation) and presented in Figure 26 for
each test set and each model.

Figure 26: Precision-recall curves for dynamic models

####  8.8.6 Calibration curves - deciles

Calibration curves are calculated as for the baseline model, using the
“pooled” predictions (each landmark represents an independent observation).
The survival model with competing risks censored at the time of event shows
overestimated predictions. The deciles calibration curves for dynamic models
are presented in 27.

Figure 27: Calibration curves (deciles) for dynamic models

####  8.8.7 Calibration curves - splines

Calibration curves are calculated as for the baseline model, using the
“pooled” predictions (each landmark represents an independent observation).
The survival model with competing risks censored at the time of event shows
overestimated predictions. The calibration curves based on cubic splines are
presented in 28.

Figure 28: Calibration curves (splines) for dynamic models

####  8.8.8 Decision curves

Decision curves are calculated as for the baseline model, using the “pooled”
predictions (each landmark represents an independent observation). The
decision curves for dynamic models are presented in Figure 18.

Figure 29: Decision curves for dynamic models

####  8.8.9 Predictions density curves

Prediction density curves for dynamic models using the “pooled” predictions
(each landmark represents an independent observation) are presented in 30. As
opposed to the baseline models, there is no notable difference between models.

Figure 30: Decision curves for dynamic models

####  8.8.10 Tuned hyperparameters

The tuned hyperparameters for the dynamic models (mtry, nodesize and
sample.fraction) are shown in Figures 31, 32 and 33. As for baseline models,
models using all events in the outcome definition (multinomial and CR models
weighting all causes), have tuned nodesizes lower than the other models. The
sample fraction hyperparameter does not show any specific pattern.

Figure 31: Tune hyperparameters (nodesize in function of mtry) Figure 32: Tune
hyperparameters (sample.fraction in function of mtry) Figure 33: Tune
hyperparameters (nodesize in function of sample.fraction)

####  8.8.11 Variable importance

The minimal depth of the maximal subtree is used as a variable importance
metric (Ishwaran et al. 2021), which is the depth in a tree on which the first
split is made on a variable $v$, averaged over all trees in the forest. The
lowest possible value is 0 (root node split). The minimal depth of the maximal
subtree for dynamic models is presented in Figure 34. A guiding line has been
added to the plot on value 2, an aleatory choice to guide the focus on most
important variables.

Models using multiple levels of the outcome put more weight (splits closer to
the root) on variables: ICU, antibacterials, antineoplastic agents
(chemotherapy) and CRP, “other infection than BSI” and tunneled catheters. The
differences in the split depth for variables TPN and port-catheter are less
notable than for baseline models.

Figure 34: ROC curves for dynamic models

###  8.9 Supplementary material 9 - Timings table

The runtimes for baseline and dynamic models are presented in Table 13.

Table 13: Runtimes for all models

Model | Baseline / Dynamic | Tuning | Build final model | Predict
---|---|---|---|---
bin | Baseline | 209.404 (197.433 - 218.468) | 11.641 (10.403 - 12.321) | 0.346 (0.327 - 0.37)
multinom | Baseline | 226.332 (208.266 - 236.412) | 7.073 (6.541 - 7.567) | 0.377 (0.345 - 0.4)
surv7d | Baseline | 370.972 (355.951 - 385.155) | 9.201 (8.297 - 9.667) | 0.296 (0.288 - 0.306)
surv7d_cens7 | Baseline | 262.443 (249.593 - 274.153) | 7.129 (6.851 - 7.452) | 0.289 (0.278 - 0.307)
surv30d | Baseline | 459.341 (446.094 - 476.892) | 10.512 (9.328 - 11.825) | 0.319 (0.309 - 0.328)
surv30d_cens7 | Baseline | 378.045 (363.976 - 393.637) | 7.893 (7.504 - 8.279) | 0.323 (0.308 - 0.34)
CR7d_LR_c_1 | Baseline | 447.95 (431.07 - 472.655) | 8.851 (8.475 - 9.445) | 0.488 (0.465 - 0.509)
CR7d_LRCR_c_1 | Baseline | 468.8 (445.506 - 486.273) | 9.578 (8.996 - 11.015) | 0.466 (0.445 - 0.491)
CR7d_LR_c_all | Baseline | 401.085 (384.14 - 426.019) | 9.288 (8.962 - 10.001) | 0.481 (0.474 - 0.49)
CR7d_LRCR_c_all | Baseline | 426.47 (408.683 - 460.104) | 9.728 (9.195 - 10.307) | 0.514 (0.497 - 0.527)
CR30d_LR_c_1 | Baseline | 567.148 (545.541 - 590.394) | 10.082 (9.295 - 11.126) | 0.518 (0.493 - 0.532)
CR30d_LRCR_c_1 | Baseline | 543.608 (526.061 - 578.87) | 10.339 (9.373 - 11.483) | 0.486 (0.467 - 0.513)
CR30d_LR_c_all | Baseline | 539.688 (513.044 - 573.47) | 11.179 (10.126 - 13.222) | 0.526 (0.512 - 0.546)
CR30d_LRCR_c_all | Baseline | 552.918 (525.582 - 582.852) | 11.528 (10.614 - 13.562) | 0.528 (0.513 - 0.548)
bin | Dynamic | 760.216 (736.662 - 786.84) | 209.655 (205.827 - 213.852) | 1.884 (1.778 - 2.019)
multinom | Dynamic | 843.682 (815.876 - 866.967) | 214.542 (209.687 - 219.267) | 2.326 (2.178 - 2.511)
surv7d | Dynamic | 935.971 (896.896 - 970.382) | 207.88 (203.285 - 213.398) | 2.25 (2.025 - 2.441)
surv7d_cens7 | Dynamic | 776.992 (747.751 - 804.212) | 205.993 (201.512 - 209.779) | 2.114 (1.768 - 2.278)
CR7d_LR_c_1 | Dynamic | 1040.23 (998.582 - 1091.098) | 201.921 (197.449 - 208.597) | 1.969 (1.896 - 2.09)
CR7d_LRCR_c_1 | Dynamic | 1045.289 (1004.061 - 1095.314) | 202.667 (197.413 - 206.297) | 2.009 (1.939 - 2.1)
CR7d_LR_c_all | Dynamic | 1039.008 (996.17 - 1091.213) | 207.562 (202.457 - 211.716) | 2.359 (2.161 - 2.767)
CR7d_LRCR_c_all | Dynamic | 1057.966 (1004.95 - 1097.65) | 205.315 (201.36 - 211.757) | 2.446 (2.198 - 2.723)

###  8.10 Supplementary material 10 - Baseline models on test/train split 001

Additional results on the baseline test set 001 (corresponding to the first
data split in the 100 train/test splits) using the “pooled” predictions (each
landmark represents an independent observation) are presented in Figures 35,
36 and 37. To facilitate better visualization, survival and competing risk
models with administrative censoring at day 30 (surv30d, surv30d_cens7,
CR30d_LRCR_c_1, CR30d_LR_c_1, CR30d_LRCR_c_all, CR30d_LR_c_all) and competing
risk models with administrative censoring at day 7 and logrank split statistic
(CR7d_LR_c_1, CR7d_LR_c_all) have been excluded.

Figure 35: All curves. The x and y axis are: Recall and Precision for AUPRC; 1
- Specificity and Sensitivity for AUROC; Predicted probabilities and Observed
probabilities for Calibration (deciles) and Calibration (splines); cutoff and
Net benefit for DCA Figure 36: Prediction histogram Figure 37: Comparison of
predictions by model. The lower plots under the diagonal show the predictions
of two models plotted against each other. The diagonal contains the the
prediction density curves by class (CLABSI vs. no CLABSI). The upper plots
above the diagonal show the correlation (Pearson correlation coefficient) for
the predictions of the two models, as well as the correlation within in each
class.

###  8.11 Supplementary material 11 - Dynamic models on test/train split 001

Additional results on the dynamic test set 001 (corresponding to the first
data split in the 100 train/test splits) using the “pooled” predictions (each
landmark represents an independent observation) are presented in Figures 35,
36 and 37. To facilitate better visualization, competing risk models using
logrank split statistic (CR7d_LR_c_1, CR7d_LR_c_all) have been excluded.

Figure 38: All curves. The x and y axis are: Recall and Precision for AUPRC; 1
- Specificity and Sensitivity for AUROC; Predicted probabilities and Observed
probabilities for Calibration (deciles) and Calibration (splines); cutoff and
Net benefit for DCA Figure 39: Prediction histogram Figure 40: Comparison of
predictions by model. The lower plots under the diagonal show the predictions
of two models plotted against each other. The diagonal contains the the
prediction density curves by class (CLABSI vs. no CLABSI). The upper plots
above the diagonal show the correlation (Pearson correlation coefficient) for
the predictions of the two models, as well as the correlation within in each
class.