# Artificial Bee Colony-based Adaptive Position Control of Electrohydraulic
Servo Systems with Parameter Uncertainty

Babajide O. Ayinde
Electrical and Computer Engineering
University of Louisville
Louisville, KY 40218
<EMAIL_ADDRESS>
&Sami El-Ferik
Department of Systems Engineering
King Fahd University of Petroleum and Minerals
Dhahran, 31261, Saudi Arabia

###### Abstract

In this paper, a robust adaptive backstepping-based controller is developed
for positioning the spool valve of Electro-Hydraulic Servo System (EHSS) with
parameter fluctuations. Artificial Bee Colony (ABC) algorithm is utilized to
drive the parameters of the proposed controller to a good neighborhood in the
solution space. The optimization problem is formulated such that both the
tracking error and control signal are minimized concurrently. The results show
that the proposed controller guarantees the uniform ultimate boundedness of
the tracking error and control signal. Moreover, illustrative simulations
validate the potentials and robustness of the proposed schemes in the presence
of uncertainties. The proposed controller is also compared with sliding mode
control.

Keywords: Electro-hydraulic systems, backstepping control, nonlinear systems,
adaptive control, artificial bee colony

##  1 Introduction

The need for fast and powerful responses in many industrial applications has
made the Electro-Hydraulic Servo Systems (EHSS) very popular. Although EHSS
are very useful in many industrial applications, ranging from aerospace flight
control to manufacturing. In seismic applications, EHSS is one of the major
components in Vibroseis [1]. However, all the aforementioned applications
demand high precision control of the system’s output.
Many factors such as fluid inflow-outflow in the servo valve and friction on
actuator-valve moving parts contribute to the nonlinearity in the system
dynamics. Also, phenomena such as entrapment of air inside the hydraulic
valve, parameter variations (due to temperature changes), unknown model errors
and perturbations elevate the complexity of controller design. Many
methodologies have been developed to address this challenging task. In spite
of all the nonlinear behaviors of the EHSS, linear control theories have been
used for developing reasonably working controllers. One drawback of using this
approach is that linear control analysis breaks down with changing operating
conditions and uncertainties. The performance of linear controller was
enhanced using a feedback-feedforward iterative learning controller [2], and
the problem of parameters variation was addressed using adaptive schemes [3,
4].
Even though adaptivity can greatly handle the problem of parameter variation,
traditional adaptive schemes suffer a serious setback especially when system
to be controlled is not linear in parameters and/or if the dynamics has
uncertain or unmodeled part. Unfortunately, EHSS falls under the category of
such systems with parameter variations and uncertain dynamics. For instance,
EHSS parameters such as supply pressure and mass of the load/piston varies
with surrounding conditions. Lyapunov approach was used in [1] to design an
enhanced feedback linearization-based controller for EHSS with supply pressure
uncertainties, however, the effect of unknown disturbance at level of the
velocity was not taken into account. This ultimately narrows the scope of the
design.
Backstepping is one of the design strategies that employs a progressive
approach to formulate the control law [5] and it is generally used for
designing stabilizing controls for some class of dynamical nonlinear systems.
This method is developed by inserting new variables that depends on the state
variables, controlling parameters and the stabilizing functions. The essence
of this stabilizing function is to redress any nonlinearity that can impede
the stability of the system. Due to the stepwise nature of the control design,
formulation of the controller generally starts with a well known stable
system. Subsequently, virtual controllers are employed to stabilize the outer
subsystems progressively. The process of using the virtual controllers in the
stabilization of the subsystem continues until the external control is
accessible. In fact, it has been shown that backstepping technique can be used
to force nonlinear systems to behave like a linear system transformed into a
new set of coordinates.
One of the numerous advantages of using backstepping technique to design a
controller is its ability to avoid useful nonlinearity cancelation [6, 7]. The
objective of backstepping gravitates towards stabilization and tracking in
contrast with its corresponding feedback linearization method. This
facilitates controller designs for perturbed nonlinear system even if the
perturbation is nowhere around the equation containing the input. Backstepping
method is generally used for tracking and regulation problems [5]. In [8],
backstepping-based neural adaptive technique was used for velocity control of
EHSS with internal friction, flow nonlinearity, and noise. However, the effect
of external disturbance was not considered. More specifically, EHSS are prone
to parameter variations due to temperature change. For instance bulk modulus
and viscous friction coefficients are prone to variation due to fluctuations
in temperature. Owing to this fact, the need to design a controller that
adapts to these changes is of paramount importance.
One important consideration that has to be carefully addressed in many
mechanical systems is friction [9]. A LuGre model-based adaptive control
scheme was proposed to model and estimate the frictional effect [2]. This
approach does not only account for frictional effect, but also offers good
disturbance rejection and robustness to uncertainties. Variable structure
controllers have also been used to model both friction and load as external
disturbances [10]. Feedback controller with auto-disturbance rejection has
also been shown to control the position of EHSS with both internal and
external disturbances [11]. System model identification technique has also
been used to estimate the model of EHSS and adaptive Fuzzy PID controller was
developed for position control [12]. The problem of external load variation
and coulomb friction have been mitigated using a nonlinear adaptive feedback
linearization position control scheme with load disturbance rejection and
friction compensation [13]. Dynamic particle swarm optimization-based
algorithm has been proposed to optimize the control parameters and improve the
tracking performance of the closed loop system [14].
Evolutionary techniques are gaining popularity among the researchers in the
last few decades due to their ability to localize global optima, or in some
cases, good local optima [15, 16]. In this work, Artificial Bee Colony (ABC)
[17, 18] is proposed to tune the parameters of the controller for optimality.
Significance of ABC have been shown in path planning of multi robot and other
control applications [19].
This paper proposes a backstepping-based approach to design a robust adaptive
controller for a highly nonlinear EHSS - a single input single output (SISO)
system. The EHSS consists of a four-way spool valve supplying a double effect
linear cylinder with a double rodded piston. The piston drives a load modeled
by mass, spring and a sliding viscous friction. Under the proposed adaptive
controller, a practical stability of the closed loop system is ensured and the
uniform ultimate boundedness of the tracking error is guaranteed. ABC is used
to tune the controller parameters by minimizing both the tracking error and
control signal. The rest of this paper is organized as follows: Section II
presents the formulation of the problem and section III presents the adaptive
controller design. Section IV discusses the ABC optimization and its
implementation for the proposed adaptive backstepping controller. Section V
discusses the simulation results and finally, section VI concludes the paper.

##  2 System Model and Problem Formulation

The dynamics of the EHSS are highly nonlinear due to many factors such as
friction on the actuators, fluid inflow and outflow in the valve. Moreover,
entrapment of air and several other phenomena such as parameter variations,
unknown model errors aggravates the complexity of the control design. The EHSS
model in Fig. 1 with dynamics given below in 1 is considered.

Figure 1: Electro-hydraulic System

$\begin{split}{\dot{x}}_{1}&=\tfrac{4B}{V_{t}}(ku\sqrt{P_{d}-\text{sign}(u)x_{1}}-\tfrac{\alpha
x_{1}}{1+\gamma\lvert u\rvert}-Sx_{2}),\\\
{\dot{x}}_{2}&=\tfrac{1}{m_{t}}(Sx_{1}-bx_{2}-\beta x_{3}),\\\
{\dot{x}}_{3}&=x_{2}+d(t).\end{split}$ (1)

where $\beta=(k_{l}+\Delta\,k_{l})$. $x_{1}$ is the differential pressure
between the two chambers, $x_{2}$ and $x_{3}$ are the velocity and position of
the rod respectively. $k_{l}+\Delta\ k_{l}$ denotes the uncertain spring
stiffness and $b$ is the viscous damping coefficient, $V_{t}$ is the total
volume of the forward and return chambers, $P_{d}$ is the supply and return
pressure difference while $m_{t}$ is the total mass of the load and piston.
$B$ and $S$ are the bulk modulus and net cross-sectional area of one side of
the piston respectively. $k$, $\gamma$ , $\alpha$ are intrinsic constants of
the servo valve; $\gamma$ and $\alpha$ are used to model the leakage in the
servo valve.
In the sequel, we consider the following assumptions hold:

###### Assumption 1

  1. 1.

d(t) is an unknown but bounded disturbance with $|d(t)|<d_{max}$.

  2. 2.

$\Delta k_{l}$ is unknown and bounded with $|\Delta k_{l}|<\Delta
k_{l}^{max}$.

  3. 3.

The dynamics of the spool-valve is assumed fast enough so it can be ignored in
the dynamic model.

  4. 4.

The states of the system are available for controller design.

  5. 5.

Reference input ($r(t)$) is a known continuously differentiable bounded
trajectory.

The nonlinearities with respect to the input $u$ in the dynamics of the system
makes it challenging to control the output of the system. We are able to
circumvent this challenge by designing a backstepping based controller that
will drive the position of the rod to a desired reference $r(t)$. In order to
use the backstepping method to design the controller, a re-indexing of the
states variables is needed to transform the system into its standard strict
feedback form. Let

$\begin{split}\xi_{1}&=x_{3},\;\;\xi_{2}\;=x_{2},\;\;\xi_{3}=x_{1}.\end{split}$
(2)

The dynamics of the transformed system is then given in Eq. (3)

$\begin{split}{\dot{\xi}}_{1}&=\xi_{2}+d(t),\\\
{\dot{\xi}}_{2}&=\tfrac{1}{m_{t}}(S\xi_{3}-b\xi_{2}-\beta\xi_{1}),\\\
{\dot{\xi}}_{3}&=\tfrac{4B}{V_{t}}(ku\sqrt{P_{d}-\text{sign}(u)\xi_{3}}-\tfrac{\alpha\xi_{3}}{1+\gamma\lvert
u\rvert}-S\xi_{2}).\end{split}$ (3)

Let

$\begin{split}e_{1}&=\xi_{1}-r,\\\ e_{2}&=\xi_{2}-{\dot{r}},\\\
e_{3}&=f(\xi)-{\ddot{r}},\end{split}$ (4)

where $f(\xi)={\dot{\xi}_{2}}$, the error dynamics satisfy

$\begin{split}{\dot{e}_{1}}&=e_{2}+d,\\\ {\dot{e}_{2}}&=e_{3},\\\
{\dot{e}_{3}}&=(\tfrac{\partial
f(\xi)}{\partial\xi}){\dot{\xi}}-{\dddot{r}}.\end{split}$ (5)

##  3 Adaptive Control Law Design

More often than not, EHSS parameters are subject to variations due to
temperature rise. Owing to the fact that backstepping controller design relies
on actual system’s parameters, the need arises to design a controller that
adapts to these changes. To overcome the problem of variation in parameters,
adaptive control schemes are generally employed. In this section, we propose a
backstepping-based adaptive technique that is robust to uncertainties in the
system’s parameters and external disturbance. We assume the parameters of the
load (that is $\beta$) and b are unknown nonlinear functions whose parameters
will be estimated by the adaptive scheme. The schematic of the proposed
backstepping-based adaptive strategy is shown in Fig 2 below.

Figure 2: Adaptive Control Scheme

We also considered a scenario whereby a more complicated vibrator-ground model
can result due to the non-ideal contact stiffness that may exist at the
boundary interaction between the vibrator’s baseplate and ground as depicted
in Fig. 3. In order to achieve this, we replaced $\beta$ and $b$ as given in
(6).

Figure 3: A more detailed vibrator-ground model prototype [20]

$\begin{split}\beta&=\gamma_{3}\xi_{1}^{2}+\gamma_{4}\xi_{2}^{2}+\gamma_{5}\xi_{3}^{2}=\theta^{T}\phi(\xi)\\\
b&=b_{0}+\Delta f(\xi,b_{0})\end{split}$ (6)

where $\Delta f(\xi,b_{0})$ is unknown but bounded nonlinear function that
satisfies (7)

$\begin{split}\sup_{t\geq 0}|\Delta f(\xi,b_{0})|\leq F_{max}\end{split}$ (7)

Now, the error dynamics are given as:

$\begin{split}{\dot{e}_{1}}&=e_{2}+d,\\\ {\dot{e}_{2}}&=e_{3},\\\
{\dot{e}_{3}}&=\tfrac{b_{0}}{m_{t}^{2}}\theta^{T}\phi(\xi)\xi_{1}+(-\tfrac{1}{m_{t}}\theta^{T}\phi(\xi)+\tfrac{b_{0}^{2}}{m_{t}^{2}}-\tfrac{4BS^{2}}{m_{t}V_{t}})\xi_{2}\\\
&+(-\tfrac{b_{0}S}{m_{t}^{2}}-\tfrac{4BS\alpha}{m_{t}V_{t}(1+\gamma\lvert
u\rvert)})\xi_{3}+\Delta F_{1}\theta^{T}\phi(\xi)\xi_{1}\\\ &+\Delta
F_{2}\xi_{2}-\Delta
F_{3}\xi_{3}-\theta_{d}^{T}\phi(\xi)d(t)-{\dddot{r}}+Am(t)u.\end{split}$ (8)

where, $\theta_{d}^{T}\phi(\xi)=\tfrac{\beta}{m_{t}}$ and $\xi$ is a state
vector comprising of $\xi_{1}$, $\xi_{2}$ and $\xi_{3}$. Now, the goal is to
design an adaptive feedback such that:

$\begin{split}\lim_{t\rightarrow\infty}|\xi_{1}-r(t)|\leq\delta\end{split}$
(9)

where, $\delta$ is a sufficiently small positive number. The design task is to
make $\delta$ as small as possible and at the same time ensuring smooth
control law. The following gives the control design schemes.

$\begin{split}\Delta F_{1}&=\tfrac{\Delta f(\xi,b_{0})}{m_{t}^{2}}\leq
F_{max}\\\ \Delta F_{2}&=\tfrac{1}{m_{t}^{2}}(2b_{0}\Delta f(\xi,b_{0})+\Delta
f^{2}(\xi,b_{0}))\leq F^{3}_{max}\\\ \Delta F_{3}&=\tfrac{S}{m_{t}^{2}}\Delta
f(\xi,b_{0})\leq F_{max}\end{split}$ (10)

###### Theorem 1

Given that

$\displaystyle\alpha_{4}=\tfrac{3}{2}+\lambda,$
$\displaystyle\alpha_{5}=1+\tfrac{1}{\lambda^{3}}+\tfrac{1}{2\lambda},$ (11)
$\displaystyle\alpha_{6}=\tfrac{1}{\lambda^{4}}.$

and

$\begin{split}h(e)&=\alpha_{4}e_{1}+\alpha_{5}e_{2}+\alpha_{6}e_{3}\\\
g(e,\xi)&=e_{2}+\lambda e_{1}+\alpha_{4}e_{2}+\alpha_{5}e_{3}\\\
&+\alpha_{6}(\tfrac{b_{0}^{2}}{m_{t}^{2}}-\tfrac{4BS^{2}}{m_{t}V_{t}})\xi_{2}-\tfrac{b_{0}S}{m_{t}^{2}}\xi_{3}\\\
A_{1}&=\tfrac{\alpha_{6}b_{0}}{m_{t}^{2}},\ \ \
A_{2}=\tfrac{\alpha_{6}}{m_{t}}\ \ \
A_{3}=\tfrac{4BS\alpha\alpha_{6}}{m_{t}V_{t}}\\\
\tilde{\theta}&=\theta-\hat{\theta}\end{split}$ (12)

and let the adaptation law be given as

$\begin{split}{\dot{\hat{\theta}}}&=\gamma_{6}(A_{1}h(e)\phi(\xi)\xi_{1}-A_{2}h(e)\phi(\xi)\xi_{2}\\\
&+\alpha_{6}F_{max}|\xi_{1}|\phi(\xi))\\\
{\dot{\hat{\theta}}_{d}}&=-\gamma_{7}\alpha_{6}\phi(\xi)d_{max}\end{split}$
(13)

and let the adaptive feedback be given as

$\begin{split}u=\bigg{(}\tfrac{m_{t}V_{t}}{4\alpha_{3}SBk\text{min}(\sqrt{P_{d}-\xi_{3}},\sqrt{P_{d}+\xi_{3}}}\bigg{)}v\end{split}$
(14)

where,

$\begin{split}v=&-(|g(e,\xi)|+|\alpha_{4}-\alpha_{6}{\hat{\theta}}^{T}_{d}\phi(\xi)|d_{max}+\Phi(\xi,\hat{\theta})\\\
&+\alpha_{6}|\xi_{3}|F_{max}+\alpha_{6}|\xi_{2}|F^{3}_{max})-k_{o}h(e)\end{split}$
(15)

Then, system (3) under the adaptive feedback control law given in (14) is
practically stable and the solution of the error dynamic (8) is globally
uniformly ultimately bounded with ultimate bound satisfying the following
condition

$\begin{split}||e||^{2}\leq\frac{d_{max}^{2}}{\lambda\,\sigma_{min}(\phi\phi^{T})}\leq\frac{d_{max}^{2}}{2\left(\lambda+\lambda^{5}\right)\,\sigma_{min}(\phi\phi^{T})}\end{split}$
(16)

with

$\begin{split}\phi=\left[\begin{array}[]{lll}1&\lambda&\frac{\alpha_{1}}{\lambda^{2}}\\\
0&1&\frac{\alpha_{2}}{\lambda^{2}}\\\
0&0&\frac{\alpha_{3}}{\lambda^{2}}\end{array}\right]\end{split}$ (17)

###### Proof 1

To prove the boundedness of the error dynamics, we again choose the Lyapunov
functions as follows:

$\begin{split}V_{1}&=\tfrac{1}{2}e_{1}^{2},\ \ \
V_{2}=\tfrac{1}{2\lambda^{4}}(e_{2}+\lambda e_{1})^{2}\\\
V_{3}&=\tfrac{1}{2}(\alpha_{4}e_{1}+\alpha_{5}e_{2}+\alpha_{6}e_{3})^{2}\\\
V_{4}&=\tfrac{1}{2\gamma_{6}}{\tilde{\theta}}^{T}{\tilde{\theta}}^{T}+\tfrac{1}{2\gamma_{7}}{\tilde{\theta_{d}}}^{2},\end{split}$
(18)

and again using the Young’s inequality with $\lambda>0$ then,

$\begin{split}{\dot{V}}={\dot{V}_{1}}+{\dot{V}_{2}}+{\dot{V}_{3}}+{\dot{V}_{4}}\end{split}$
(19)

Therefore, the derivative of the Lyapunov function is thus given:

$\begin{split}{\dot{V}}&\leq-\tfrac{\lambda}{2}e_{1}^{2}+(\tfrac{1}{2\lambda}+\tfrac{1}{2\lambda^{5}})d^{2}-(e_{2}+\lambda
e_{1})^{2}\\\
&+h(e)[g(e,\xi)+A_{1}\tilde{\theta}^{T}\phi(\xi)\xi_{1}+A_{1}{\hat{\theta}}^{T}\phi(\xi)\xi_{1}\\\
&-A_{2}\tilde{\theta}^{T}\phi(\xi)\xi_{2}-A_{2}{\hat{\theta}}^{T}\phi(\xi)\xi_{2}-\tfrac{A_{3}}{1+\gamma\lvert
u\rvert})\xi_{3}\\\ &+\alpha_{4}d+\alpha_{6}\Delta
F_{1}\tilde{\theta}^{T}\phi(\xi)\xi_{1}+\alpha_{6}\Delta
F_{1}{\hat{\theta}}^{T}\phi(\xi)\xi_{1}\\\ &+\alpha_{6}\Delta
F_{2}\xi_{2}-\alpha_{6}\Delta
F_{3}\xi_{3}-\alpha_{6}\tilde{\theta}_{d}^{T}\phi(\xi)d(t)\\\
&-\alpha_{6}{\hat{\theta}}_{d}^{T}\phi(\xi)d(t)-\alpha_{6}{\dddot{r}}+\alpha_{6}Am(t)u]\\\
&-\tfrac{1}{\gamma_{6}}{\tilde{\theta}}^{T}{\dot{\hat{\theta}}}-\tfrac{1}{\gamma_{7}}\tilde{\theta}_{d}{\dot{\hat{\theta}}_{d}}\end{split}$
(20)

In order to annihilate the parametric error, the update laws are chosen as in
(13) Given that $F_{max}=\sup_{t\geq 0}|\Delta F_{1}|$ and
$d_{max}=\sup_{t\geq 0}|d(t)|$, and to ensure a uniformly ultimately bounded
tracking error, adaptive feedback $``u"$ is chosen as given in (14) and $v$ is
given as follows:

$\begin{split}v&=-(|g(e,\xi)|+|\alpha_{4}-\alpha_{6}{\hat{\theta}}^{T}_{d}\phi(\xi)|d_{max}+\Phi(\xi,\hat{\theta})\\\
&+\alpha_{6}|\xi_{3}|F_{max}+\alpha_{6}|\xi_{2}|F^{3}_{max})-k_{o}h(e)\end{split}$
(21)

Ultimately,

$\begin{split}{\dot{V}}&\leq-\tfrac{\lambda}{2}e_{1}^{2}+(\tfrac{1}{2\lambda}+\tfrac{1}{2\lambda^{5}})d^{2}-(e_{2}+\lambda
e_{1})^{2}\\\
&-(\alpha_{1}e_{1}+\alpha_{2}e_{2}+\alpha_{3}e_{3})^{2}\end{split}$ (22)

Therefore,for ${\dot{V}}\leq 0$, it is sufficient to verify that
$-V+\tfrac{1}{\lambda}d_{max}^{2}\leq 0$. Which means that ${\dot{V}}\geq 0$
if $e$ is such that $V\leq\tfrac{1}{\lambda}d_{max}^{2}$. This will lead to
increasing $e$ until $V\geq\tfrac{1}{\lambda}d_{max}^{2}$.
Let $z=\phi^{T}e$, with

$\begin{split}\phi=\left[\begin{array}[]{lll}\frac{1}{\sqrt{2}}&\lambda&\alpha_{1}\\\
0&1&\alpha_{2}\\\ 0&0&\alpha_{3}\end{array}\right]\end{split}$ (23)

Using (23), $V$ can be rewritten as $V=z^{T}\,z=||z||^{2}$. On the other hand,

$\begin{split}\sigma_{min}(\phi\phi^{T})||e||^{2}\leq
V=||z||^{2}\leq\sigma_{max}(\phi\phi^{T})||e||^{2}\end{split}$ (24)

where $\sigma_{min}(\phi\phi^{T})$ and $\sigma_{max}(\phi\phi^{T})$ represent
the max and min singular values of $\phi\phi^{T}$ respectively. Since the
Lyapunov function satisfies (22) for all $t\in\Re_{\geq 0}$, this implies that
the whole time during which the adaptation take place is finite. During the
finite time, the variables $\hat{\theta}$ and ${\hat{\theta}}_{d}$ cannot
escape to infinity since the adaptation laws in (13) are well defined. For any
bounded conditions $e(0)$, ${\hat{\theta}}(0)$ and ${\hat{\theta}}_{d}(0)$ and
by making use of (22), we infer that $e$, ${\hat{\theta}}$ and
${\hat{\theta}}_{d}$ are bounded for all $t\in\Re_{\geq 0}$. The proof ends
here.

###### Remark 1

The parameter $\lambda$ is a design parameter introduced in Young’s
inequality. Such parameter should be selected as big as possible to decrease
the ultimate bound in the tracking error dynamic. Nevertheless, a trade-off
must be made, since the larger the value of $\lambda$ the more oscillatory is
the transient and higher is the control input. However, $\epsilon$ has to be
chosen sufficiently small and its choice is neither dependent on system’s
parameters nor the bound of the disturbance.

The proposed controller has parameters that should be carefully selected for
optimal performance of the closed loop system. In this work, the ABC technique
is employed for optimal setting of the controller parameters $\lambda$ and
$\gamma_{1}$ based on a preassigned objective function minimization. The
proper selection of these parameters will be reflected on minimizing the
objective function. The proposed objective function is defined by the error
and control signal.

$Obj=\sum\limits_{t=0}^{t_{sim}}\big{(}\Gamma_{1}e_{1}^{2}(t)+\Gamma_{2}u^{2}(t)\big{)}$
(25)

where the tracking error $e_{1}(t)=\xi_{1}-r(t)$ and $u(t)$ is the control
signal, $\Gamma_{1}$ and $\Gamma_{2}$ are weighting parameters. The
controller’s parameters are selected within the bounds:

$\displaystyle\lambda^{min}\leq$ $\displaystyle\lambda\leq\lambda^{max}$
$\displaystyle\gamma_{1}^{min}\leq$
$\displaystyle\gamma_{1}\leq\gamma_{1}^{max}$

##  4 ABC Algorithm

Artificial Bees Colony is a meta-heuristic approach that gained its
inspiration from the work proposed in [17]. The algorithm is motivated from
the way life is structured in the colony of natural bees. The bees in the
colony are usually divided into three groups: employed, onlooker and scout
bees. The primary assignment of the employed bees is to randomly search for
food and the best solution of the food is identified as the optimal solution.
The employed bees dance in a systematic manner for the purpose of relaying
information about the food source and the amount of nectar to other bees in
the colony. Onlooker bees differentiate between the good and the bad food
sources based on the dance length, dance type and speed of shaking of the
employed bees. Onlooker bees also use these information to establish the
quality of food. The scout bees are chosen from the onlooker bees prior a new
search for food. The onlooker and scout bees may decide to switch roles with
the employed bees depending on the quality of food [18].
As detailed in [18], the employed and onlooker bees are responsible for
searching the solution space for the optimal parameters while the scout bees
control the search process. The summary of the ABC algorithm is shown in the
flowchart of Fig. 4 and the solution of the optimization problem is the
position of the food source while the amount of nectar with respect to the
quality is termed as the objective function of the optimization procedure. The
position of the food source in the search space can be described as follows:

$\begin{split}x^{new}_{ij}=x^{old}_{ij}+u(x_{ij}^{old}-x_{kj})\end{split}$
(26)

Figure 4: Artificial Bee Colony Algorithm

The probability of onlooker bees for choosing a food source:

$\begin{split}P_{i}=\tfrac{fitness_{1}}{\sum^{E_{b}}_{i=1}fitness_{i}}\end{split}$
(27)

where, $x$ is a candidate solution, $P_{i}$ is the probability of onlooker
solution, $i=1,2,\cdots,E_{b}$ is the half of the colony size,
$j=1,2,\cdots,D$ and and $j$ is the number of positions with D dimension where
$D$ refers to number of parameters to be optimized, $fitness_{i}$ is the
fitness function, $k$ is a random number where $k\in(1,2,\cdots,E_{b})$, $u$
is random number between 0 and 1.

##  5 Results and Discussions

The problem is divided into two stages. In the first stage, the proposed
backstepping controller (BSC) was implemented without ABC algorithm and in the
second stage, ABC algorithm is incorporated with the proposed controller for
optimal parameter tuning. The tracking capability of both stages is evaluated
using constant, sum of sinusoidal, and sinusoidal reference inputs. The
objective function of the optimization procedure is formulated as:

$\begin{split}Obj=\sum^{20}_{t=0.01}\Gamma_{1}e_{1}^{2}(t)+\Gamma_{2}u^{2}(t)\end{split}$
(28) $\displaystyle 9\leq$ $\displaystyle\lambda\leq 16$ $\displaystyle
10^{-7}\leq$ $\displaystyle\gamma_{1}\leq 10^{-10}$

In the optimization algorithm, the number of parameters to be optimized are
$\lambda$ and $\gamma_{1}$, the population size is selected as 50 and the
number of generations is 100, and the search space was constrained as given in
(28). The weighted values of $\Gamma_{1}$ and $\Gamma_{1}$ were chosen to be
1. For experimental purpose, $F_{max}$ is chosen to be $10$. Adaptive
backstepping controller with ABC-based optimizer is deployed on the system
model in (3). It must be remarked that this is a minimization task. The
parameters of the system’s model are given in table 1

Table 1: Numerical values for simulations Parameters | Value | Units
---|---|---
$B$ | $2.2e9$ | Pa
$P_{r}$ | $1e5$ | Pa
$V_{t}$ | $1e{-3}$ | $m^{3}$
$S$ | $1.5e{-3}$ | $m^{2}$
$\gamma$ | 8571 | $s^{-1}$
$b$ | 590 | $kg~{}s^{-1}$
$\Delta$$k_{l}$ | 2500 | $Nm^{-1}$
$k_{l}$ | 12500 | $Nm^{-1}$
$P_{s}$ | $300e5$ | Pa
$m_{t}$ | 70 | kg
$k$ | $5.12e{-5}$ | $m^{3}s^{-1}A^{-1}Pa^{1/2}$
$\alpha$ | $4.1816e-12$ | $m^{3}s^{-1}Pa^{-1}$
Table 2: Minimum objective function with optimal parameters. Experiment No | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
---|---|---|---|---|---|---|---|---
$Objective$ | 2.1368 | 2.1368 | 2.1368 | 2.1368 | 2.1369 | 2.1368 | 2.1368 | 2.1369
$\gamma_{1}$ | $10^{-10}$ | $10^{-10}$ | $10^{-10}$ | $10^{-10}$ | $10^{-10}$ | $10^{-10}$ | $10^{-10}$ | $10^{-10}$
$\lambda$ | 13.5585 | 13.5580 | 13.5583 | 13.5585 | 13.5580 | 13.5585 | 13.5581 | 13.5589

The simulation is implemented for 20 seconds and the values of error and
control signal are captured every 0.01 second. In the first set of
experiments, the reference input is a step function with $r(t)=0.2$. The
simulation was carried out 8 times with different initial populations in order
to ensure the robustness of the proposed solution. As shown in Table 2, the
objective functions were close in all experiments and Fig. 5 shows the output
performance of the proposed BSC with optimized parameters $\gamma_{1}\leq
10^{-10}$ and $\lambda\leq 13.5585$. Although steady state error is not equal
to zero due to the disturbance ($d(t)=0.1$), the error and control signal are
bounded and the output performance is very close to the reference. The
performance of the proposed controller was also compared with sliding mode
controller (SMC) and it can be seen that even though the SMC achieved smaller
tracking error, there is output chattering even at steady state. However in
the case of the proposed controller tuned with ABC, the control signal is
smooth and the transient oscillations have been eliminated. ABC was able to
find the best compromise solution by simultaneously minimizing the tracking
error and output oscillations in both transient and steady state. Also, as
shown in Fig. 6 the control signal is minimized and smooth for ABC-based BSC
compared to both SMC and BSC without ABC tuning.

Figure 5: Output performance of adaptive backstepping after ABC optimization
with $r=0.2$m Figure 6: Control input of adaptive backstepping after ABC
optimization with $r=0.2$m Figure 7: Output performance of adaptive
backstepping after ABC optimization with $r=0.3$m Figure 8: Control input of
adaptive backstepping after ABC optimization with $r=0.3$m Figure 9: Output
performance of adaptive backstepping after ABC optimization with
$r=0.05(sin(t)+sin(2t)+sin(3t))$m Figure 10: Output performance of adaptive
backstepping after ABC optimization with $r=0.05sin(t)$m

It must be noted that the proposed adaptive design does not involve the
differentiation of $m(t)$, which indicates the proposed scheme can handle the
effects of various types of slowly time-varying $m(t)$ and $d(t)$. The problem
we address is a robust adaptive control issue for EHSS. Using a constant
trajectory as seen in Fig. 5, the adaptive BSC input accomplishes a bounded
error tracking even in the presence of input nonlinearity, parameter
uncertainties and unknown but bounded disturbance. Fig. 7 shows the output
performance when the reference was changed to $r(t)=0.3$ and it can be seen
that BSC tuned with ABC was able to achieve better tracking accuracy, smooth
response, and less transient oscillations than both SMC and BSC. It can again
be observed in Fig. 8 that the SMC control signal is non-smooth and those of
BSC are smooth.
Fig. 9 demonstrates the excellent tracking of the proposed ABC-based BSC for
sum of sinusoids reference input compared to BSC without ABC. It is worth
mentioning that for sinusoidal references, the SMC went out of stability and
for this reason, we could not compare its response with BSC for experiments
involving sinusoidal references. Also in Fig. 10, excellent tracking accuracy
was noticed for ABC-based BSC compared to BSC when the reference was changed
to sinusoid. In a nutshell, the proposed solution is robust against parameter
uncertainties and disturbance if the ultimate bound satisfies the condition
$||e||^{2}\leq\frac{d_{max}^{2}}{2\left(\lambda+\lambda^{5}\right)\,\sigma_{min}(\phi\phi^{T})}$.
It must also be remarked that the results presented in this paper are for the
case when non-ideal contact stiffness exists at the boundary interaction
between the vibrator’s baseplate and ground. The $\beta$ and $b$ which
correspond to load and the frictional parameters have been replaced with
nonlinear functions to incorporate more realistic vibrator-ground model. As
shown, the tracking error still converges to the neighborhood of the origin at
the steady states. This shows the robustness of the backstepping-based
adaptive controller.

##  6 CONCLUSION

In this work, a robust backstepping based adaptive controller is proposed for
EHSS with uncertain and partially known parameters. ABC algorithm is
incorporated in the closed loop system to optimize the proposed controller’s
parameter and the adaptation gain while minimizing the tracking error, control
signal and its smoothness. Several experiments have been used to validate the
tracking capability of the proposed controller and the robustness of the
proposed approach. It is concluded that the proposed control approach ensures
uniform ultimate boundedness of the error and control signal.

## References

  * [1] H. A. Mintsa, R. Venugopal, J.-P. Kenne, and C. Belleau, “Feedback linearization-based position control of an electrohydraulic servo system with supply pressure uncertainty,” _Control Systems Technology, IEEE Transactions on_ , vol. 20, no. 4, pp. 1092–1099, 2012.
  * [2] X. Wang and S. Wang, “High performance adaptive control of mechanical servo system with LuGre friction model: identification and compensation,” _Journal of Dynamic Systems, Measurement, and Control_ , vol. 134, no. 1, p. 011021, 2012.
  * [3] H. Yanada and K. Furuta, “Adaptive control of an electrohydraulic servo system utilizing online estimate of its natural frequency,” _Mechatronics_ , vol. 17, no. 6, pp. 337–343, 2007.
  * [4] Z. Zhang and W. Chen, “Adaptive output feedback control of nonlinear systems with actuator failures,” _Information Sciences_ , vol. 179, no. 24, pp. 4249–4260, 2009.
  * [5] S. M. Rozali, M. N. Kamarudin, M. F. Rahmat, and A. R. Husain, “Asymptotic tracking position control for nonlinear systems using backsteppingtechnique,” _Procedia Engineering_ , vol. 53, pp. 255–263, 2013\.
  * [6] B. O. Ayinde, S. El Ferik, S. Ibrir, M. Feki, and B. A. Siddiqui, “Backstepping control of an electro-hydraulic servo system subject to disturbance and parameter uncertainty,” in _GCC Conference and Exhibition (GCCCE), 2015 IEEE 8th_. IEEE, 2015, pp. 1–6.
  * [7] S. El Ferik, B. O. Ayinde, S. Ibrir, and M. Feki, “Backstepping-based output feedback control of an electro-hydraulic servo system,” in _Systems, Signals & Devices (SSD), 2015 12th International Multi-Conference on_. IEEE, 2015, pp. 1–6.
  * [8] Z. A. S. Dashti, M. Gholami, M. Jafari, and M. A. Shoorehdeli, “Neural adaptive control based on backstepping and feedback linearization for electro hydraulic servo system,” in _Intelligent Systems (ICIS), 2014 Iranian Conference on_. IEEE, 2014, pp. 1–6.
  * [9] S. Hutamarn, P. Pratumsuwan, and W. Po-ngaen, “Neuro-fuzzy based on support vector machine friction compensator in servo hydraulic system,” in _Industrial Electronics and Applications (ICIEA), 2012 7th IEEE Conference on_. IEEE, 2012, pp. 2118–2122.
  * [10] A. Bonchis, P. I. Corke, D. C. Rye, and Q. P. Ha, “Variable structure methods in hydraulic servo systems control,” _Automatica_ , vol. 37, no. 4, pp. 589–595, 2001.
  * [11] R. Xiang and Z. Yinyin, “Electro-hydraulic position synchronization control system based on auto-disturbances rejection and feedback,” in _World Automation Congress (WAC), 2012_. IEEE, 2012, pp. 1–4.
  * [12] J. Shao, Z. Wang, J. Lin, and G. Han, “Model identification and control of electro-hydraulic position servo system,” in _Intelligent Human-Machine Systems and Cybernetics, 2009. IHMSC’09. International Conference on_ , vol. 1. IEEE, 2009, pp. 210–213.
  * [13] J. K. H. Angue-Mintsa, R. Venugopal and C. Belleau, “Adaptive position control of an electrohydraulic servo system with load disturbance rejection and friction compensation,” _Journal of Dynamic Systems, Measurement, and Control_ , vol. 133, no. 064506, pp. 1–7, 2011.
  * [14] L. Lai, “A synchronization position control method based on dynamic particle swarm optimization algorithm in electro-hydraulic servo system,” 2013.
  * [15] H. A. Hashim and M. A. Abido, “Fuzzy controller design using evolutionary techniques for twin rotor mimo system: a comparative study,” _Computational intelligence and neuroscience_ , vol. 2015, p. 49, 2015.
  * [16] H. A. Hashim, S. El-Ferik, and M. A. Abido, “A fuzzy logic feedback filter design tuned with pso for l1 adaptive controller,” _Expert Systems with Applications_ , vol. 42, no. 23, pp. 9077–9085, 2015.
  * [17] D. Karaboga, “An idea based on honey bee swarm for numerical optimization,” Erciyes university, computer engineering department, engineering faculty, Tech. Rep., 2005.
  * [18] D. Karaboga and B. Basturk, “A powerful and efficient algorithm for numerical function optimization: artificial bee colony (abc) algorithm,” _Journal of global optimization_ , vol. 39, no. 1, pp. 459–471, 2007.
  * [19] P. Bhattacharjee, P. Rakshit, I. Goswami, A. Konar, and A. K. Nagar, “Multi-robot path-planning using artificial bee colony optimization algorithm,” in _Nature and Biologically Inspired Computing (NaBIC), 2011 Third World Congress on_. IEEE, 2011, pp. 219–224.
  * [20] Z. Wei, “Modelling and modal analysis of seismic vibrator baseplate,” _Geophysical Prospecting_ , vol. 58, no. 1, pp. 19–32, 2010.


Previous point-based radiance field rendering methods provide great results in
many cases, but renderings can be aliased and incomplete (ADOP [RFS22] (left),
missing parts of the bike’s tire), or overblurred (3D Gaussian Splatting
[KKLD23] (middle), missing fine grass details). Our approach combines the
advantages of both to render crisp, complete, and alias-free images.

# TRIPS: Trilinear Point Splatting for Real-Time Radiance Field Rendering

Linus Franke, Darius Rückert, Laura Fink and Marc Stamminger
Visual Computing Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg,
Germany<EMAIL_ADDRESS>
###### Abstract

Point-based radiance field rendering has demonstrated impressive results for
novel view synthesis, offering a compelling blend of rendering quality and
computational efficiency. However, also latest approaches in this domain are
not without their shortcomings. 3D Gaussian Splatting [KKLD23] struggles when
tasked with rendering highly detailed scenes, due to blurring and cloudy
artifacts. On the other hand, ADOP [RFS22] can accommodate crisper images, but
the neural reconstruction network decreases performance, it grapples with
temporal instability and it is unable to effectively address large gaps in the
point cloud.

In this paper, we present TRIPS (Trilinear Point Splatting), an approach that
combines ideas from both Gaussian Splatting and ADOP. The fundamental concept
behind our novel technique involves rasterizing points into a screen-space
image pyramid, with the selection of the pyramid layer determined by the
projected point size. This approach allows rendering arbitrarily large points
using a single trilinear write. A lightweight neural network is then used to
reconstruct a hole-free image including detail beyond splat resolution.
Importantly, our render pipeline is entirely differentiable, allowing for
automatic optimization of both point sizes and positions.

Our evaluation demonstrate that TRIPS surpasses existing state-of-the-art
methods in terms of rendering quality while maintaining a real-time frame rate
of 60 frames per second on readily available hardware. This performance
extends to challenging scenarios, such as scenes featuring intricate geometry,
expansive landscapes, and auto-exposed footage.

The project page is located at: https://lfranke.github.io/trips

<ccs2012> <concept>
<concept_id>10010147.10010371.10010382.10010385</concept_id>
<concept_desc>Computing methodologies Image-based rendering</concept_desc>
<concept_significance>500</concept_significance> </concept> <concept>
<concept_id>10010147.10010371.10010372</concept_id> <concept_desc>Computing
methodologies Rendering</concept_desc>
<concept_significance>500</concept_significance> </concept> <concept>
<concept_id>10010147.10010178.10010224.10010245.10010254</concept_id>
<concept_desc>Computing methodologies Reconstruction</concept_desc>
<concept_significance>100</concept_significance> </concept> </ccs2012>
[500]Computing methodologies Rendering [500]Computing methodologies Image-
based rendering [100]Computing methodologies Reconstruction

##  1 Introduction

Novel view synthesis methods have been a significant driver for computer
graphics and vision, as they have revolutionized the way we perceive and
interact with 3D scenes. Many of these methods rely on explicit
representations, such as meshes or points. Typically, the explicit models are
derived from 3D reconstruction processes and can be efficiently rendered
through rasterization, which aligns well with contemporary GPU capabilities.
Nevertheless, these reconstructed models often fall short of perfection and
necessitate additional steps to mitigate artifacts.

A common strategy to handle these artifacts is to use scene-specific
optimization methods, known as inverse rendering. This allows for the
adjustment of the scene’s texture, geometry, and camera parameters to align
the rendering with the photograph. Prominent techniques in this domain
incorporate per-point descriptors [RFS22, ASK∗20, KPLD21], explicit
optimization of point sizes via Gaussians [KKLD23] and learned neural
refinement networks [TZN19, RFS22, KLR∗22]. While this generally extends
render times, it significantly enhances visual quality.

In the realm of point-based inverse and neural rendering techniques, two
successful recent approaches are 3D Gaussian Splatting[KKLD23] and
ADOP[RFS22]. The former method employs a unique strategy where each point is
rendered as a 3D Gaussian distribution, allowing for direct optimization of
the points’ shape and size. This process effectively fills gaps in point
clouds within the global coordinate space through the utilization of large
splats. Remarkably, this approach yields high-quality images without
necessitating the integration of a neural network for reconstruction. However,
a drawback is the potential loss of sharpness, as Gaussians tend to introduce
blurriness and cloudy artifacts, particularly when there are limited
observations available.

In contrast, ADOP rasterizes radiance fields as one-pixel points with depth
testing at multiple resolutions. Subsequently, it employs a neural network to
address gaps and enhance texture details in screen space. This approach
possesses the capability to reconstruct texture details that surpass the
resolution of the original point cloud, although the neural network adds an
additional computational overhead and shows weaknesses in filling large holes.

In this paper, we introduce TRIPS, a novel approach that seeks to harness the
strengths of both ADOP and 3D Gaussians without loosing real-time rendering
capabilities. Similar to 3D Gaussian Splatting, TRIPS rasterizes splats of
varying size, however, like ADOP, it also applies a reconstruction network to
generate hole-free and crisp images. More precisely, we first rasterize the
point cloud as $2\times 2\times 2$ trilinear splats into an image pyramid and
blend them using front-to-back alpha blending. Subsequently, we feed the image
pyramid through a compact and efficient neural reconstruction network, which
harmonizes the various layers, addresses remaining gaps, and conceals
rendering artifacts. To ensure the preservation of high levels of detail,
particularly in challenging input scenarios, we incorporate spherical
harmonics and a tone mapping module into our pipeline.

In our evaluations, we demonstrate that our approach can yield crisper images
compared to 3D Gaussians, with almost the same perfomance. Furthermore, it
surpasses ADOP in the task of filling sizable gaps and maintaining temporal
consistency throughout the rendering process. In summary, our contributions
are:

  * •

The introduction of TRIPS, a novel trilinear point splatting technique for
radiance field rendering.

  * •

A differentiable pipeline for optimization of all input parameters, including
point positions and sizes, creating a robust scene representations.

  * •

An implementation of the method resulting in high-quality real-time renderings
under varying capturing conditions at https://github.com/lfranke/TRIPS

Figure 1:  Our pipeline: TRIPS renders a point cloud trilinearly as
2$\times$2$\times$2 splats into multi-layered feature maps with the results
being passed though our small neural network, containing only a single gated
convolution per layer. This pipeline is completely differentiable, so that
point descriptors (colors) and positions, as well as camera parameters are
optimized via gradient descent.

##  2 Related Work

In this section, we provide an overview of the field of novel view synthesis
and choices for scene representations in this problem domain.

###  2.1 Novel View Synthesis and Traditional Approaches

Traditionally, real-world novel view synthesis relies on image-based rendering
techniques. Commonly, Structure-from-Motion (SfM) techniques [SSS06, SF16]
allow camera parameter estimations from a set of photographs which are then
used for directly warping source image colors to a target view [DYB98,
CDSHD13]. This relies on accurate proxy geometry (usually point clouds or
meshed), commonly enhanced via Multi-View Stereo (MVS) [SZPF16, GSC∗07]. In
real world datasets however, these techniques can present camera
miscalibrations and erroneous geometry [SK00]. For image-based rendering, this
can lead to warping artifacts, especially near object boundaries, or can cause
blurring of details. Recently, pipelines enhanced by neural rendering [TTM∗22]
provided powerful tools to lessen these artifacts.

###  2.2 Neural Rendering and Scene Representations

In the last years, multiple variants of deep learning for novel view synthesis
were introduced. Within proxy-based pipelines, several works have replaced the
blending operation by a deep neural networks [RK21, HPP∗18, RK20] or learned
textures [TZN19] during the warping stage. Other approaches use multi-plane
images [MSOC∗19, STB∗19, TS20, ZTF∗18] or estimate a warping fields [FNPS16,
GKSL16, ZTS∗16] to avoid the need of an scene specific proxy geometry.

This led the way for volumetric scene representations [PZ17] enhanced with
deep learning [SMB∗20, STH∗19] and rendered via ray-marching. Neural Radiance
Fields (NeRFs) [MST∗21] furthermore showed that compressing a full 3D scene
into a Multilayer Perceptron (MLP) achieve great results in this regard. This
representation however is challenging in its own right, which follow-up works
improve upon: long training times [CXZ∗21, CBLPM21, MESK22, TMW∗21, TRS22],
many well distributed input views [CBLPM21, YYTK21, KD23] and rendering times
[MESK22, BMT∗21, NSP∗21]. Improvements in quality [BMV∗23, MBRS∗21] allow
NeRFs to surpass visual quality of many proxy-based approaches, however render
times are still challenging, e.g. MipNeRF-360 [BMV∗22] ranging in the order of
seconds per image and training needing dozens of hours.

Lately, discretizing parts of the scene space [YLT∗21, HSM∗21] or even
replacing parts of it via voxel grids [FKYT∗22], octrees [RWL∗22] or tensor
factorization [CXG∗22] shrink computational costs as MLPs can be smaller or
even removed. In this area, InstantNGP [MESK22] made waves as it uses hash-
grids and a highly optimized MLP implementation for faster rendering and
trainings speeds while retaining many qualitative advantages of NeRFs.

For the scope of real-time radiance field rendering however, Kerbl and Kopanas
et al. [KKLD23] argue that ray-marching as a rendering concept is challenging
on current GPU hardware.

###  2.3 Real-Time Rendering for Radiance Fields via Points

In the domain of real-time radiance field rendering, point clouds as an
explicit proxy representation remain a great option. Point clouds are easily
captured via LiDAR-based mapping [LXG22], RGB-D cameras with fusion techniques
[DNZ∗17, WSMG∗16, KLL∗13] and SfM/MVS techniques [SZPF16]. They represent a
unstructured set of samples in space, with varying distances to neighbors, but
true to the originally captured data. Rendering these can be very fast [SKW21,
SKW22, SKW19], and augmenting points with neural descriptors [RFS22, ASK∗20,
RALB22] or optimized attributes [KPLD21, KLR∗22] provide high quality
renderings using differentiable point renderers [WGSJ20, YSW∗19]. However,
discrete rasterization of points can cause aliasing [SKW22] or overdraw
[RFS22] if many points are rendered to the same pixel.

Another problem shared by point rendering techniques is how to fill holes in
the unstructured data. Two main approaches have evolved over the years [KB04]:
splatting (in world-space) and screen-space hole filling.

In world-space hole-filling, points are represented as oriented discs, often
termed "splats" or "surfels", with disc radii precomputed based on point cloud
density. To reduce artifacts between neighboring splats, these discs can be
rendered using Gaussian alpha-masks and combined with a normalizing blend
function [AGP∗04, PZVBG00, ZPVBG01]. Recent techniques optimize splat sizes
[KKLD23, ZBRH22] or improve quality with neural networks [YCA∗20]. For
performance, overdraw poses a major issue as splats tend to overlap a lot.
Thus, special care has to be taken regarding the amount of splats drawn. 3D
Gaussian Splatting [KKLD23] can be considered state of the art in this domain.
They combine anisotropic Gaussians with a very fast tiled renderer and
optimize splat sizes via gradient descent. However, limiting Gaussian numbers
is necessary to avoid performance hits, which in turn can lead to over-blurred
small detailed elements.

The second direction involves screen-space hole-filling, where points, often
rendered as tiny splats, are post-processed either through traditional methods
[PGA11, MKC07, GD98] or using convolutional neural networks (CNNs) [ASK∗20,
MGK∗19, SCCL20]. While these techniques bridge large point distances, their
need for a large receptive field can result in artifacts or performance
issues. A multi-resolution pyramid rendering approach mitigates this by
assigning different network layers to varied resolutions[ASK∗20, RALB22],
albeit reintroducing overdraw issues at lower layers [RFS22]. Notably, ADOP
[RFS22] excels in screen-space hole-filling, enabling the rendering of
hundreds of millions of points for sharp object visualization [SKW22], but
encounters challenges with temporal aliasing and substantial hole-filling.

Our approach aims to take the best of both worlds. Using TRIPS, we can render
large splats by optimizing their size, but avoid high rasterization costs.
This allows rendering enormous point clouds and detailed textures, while still
being real-time capable without aliasing or temporal instability.

##  3 Method

Fig. 1 provides an overview of our rendering pipeline. The input data consists
of images with camera parameters and a dense point cloud, which can be
obtained through methods like multi-view stereo [SZPF16] or LiDAR sensing. To
render a specific view, we project the neural color descriptors of each point
into an image pyramid using the TRIPS technique (as detailed in Sec. 3.1) and
blends them (Sec. 3.2). Subsequently, a compact neural reconstruction network
(described in Sec. 3.3) integrates the layered representation, followed by the
application of a spherical harmonics module (discussed in Sec. 3.4) and a tone
mapper that transforms the resulting features into RGB colors.

Core to our method is the trilinear point renderer, which splats points
bilinearly onto the screen space position as well as linearly to two
resolution layers, determined by the projected point size. Our renderer uses
similar nomenclature and is inspired by previous point-rasterizing approaches
[SKW22, KPLD21, RFS22]. The neural image $I$ is the output of the render
function $\Phi$

$\text{I}=\Phi(C,R,t,x,E,\upsilon,\tau,\gamma),$ (1)

where $C$ are the camera intrinsics, $(R,t)$ the extrinsic pose of the target
view, $x$ the position of the points, $E$ the optional environment map,
$\upsilon$ the world space size of the points, $\tau$ the neural point
descriptors and $\gamma$ the transparency for each point.

In contrast to other approaches, we do not use multiple render passes with
progressively smaller resolutions, as this causes severe overdraw in the lower
resolution layers. Instead, we compute the two layers which best match the
point’s projected size and render it only into these layers as $2\times 2$
splat. By doing so, we mimic varying splat sizes, although effectively
rendering only $2\times 2$-splats. The layers are then later merged in a small
neural reconstruction network (Sec. 3.3) to the final image, resembling the
decoder part of a U-Net.

###  3.1 Differentiable Trilinear Point Splatting

Figure 2: Trilinear Point Splatting: (left) all points and their respective
size are projected into the target image. Based on this screen space size,
each point is written to the correct layer of the image pyramid using a
trilinear write (right). Large points are written to layers of lower
resolution and therefore cover more space in the final image.

Using $C$ and $(R,t)$, we project each point position $(x_{w},y_{w},z_{w})$ to
continuous (non-rounded) screen space coordinates $(x,y,z)$ and each world-
space point size $s_{w}$ to screen space size $s$ with the camera’s focal
length $f$:

$s=\frac{f\cdot s_{w}}{z}.$ (2)

Next, we render these points as a $2\times 2\times 2$ splats bilinearly and
handle point size by splatting into two neighboring resolution layers $L$ with
$L_{lower}=\lfloor\log(s)\rfloor$ and $L_{upper}=\lceil\log(s)\rceil$, as
shown in Fig. 2. The final opacity value $\gamma$ that is written to the image
pyramid for pixel $(x_{i},y_{i},s_{i})$ with $s_{i}=\lfloor 2^{L}\rfloor$ is

$\displaystyle\gamma$ $\displaystyle=\beta\cdot\iota\cdot\alpha,$ (3)
$\displaystyle\beta$ $\displaystyle=(1-|x-x_{i}|)\cdot(1-|y-y_{i}|)$ (4)
$\displaystyle\iota$ $\displaystyle=\begin{cases}1-|s-s_{i}|&s\geq 1\\\
\epsilon+(1-\epsilon)s&s_{i}=0\land s<1\end{cases}$ (5)

where $\beta$ is the bilinear weight inside the image layer, $\iota$ is the
linear layer weight, and $\alpha$ the opacity value of the point. The layer
weight $\iota$ is a standard linear interpolation if the point size $s$ is
inside the image pyramid. The second case of Equ. (5) handles far away points
that have a pixel size smaller than one. In order not to miss these, we always
add them to the finest level $0$. To avoid that their weight disappears, we
ensure that their contribution is at least $\epsilon=0.25$.

###  3.2 Multi Resolution Alpha Blending

Since each point is written to multiple pixels and multiple points can fall
into the same pixel, we collect all fragments in per pixel lists
$\Lambda_{l_{i},x_{i},y_{i}}$. These lists are sorted by depth and clamped to
a maximum size of 16 elements. Eventually, the color $C_{\Lambda}$ is computed
using front-to-back alpha blending (Fig. 3):

$C_{\Lambda}=\sum_{m=1}^{|\Lambda|}T_{m}\cdot\alpha_{m}\cdot c_{m}$ (6)
$T_{m}=\prod_{i=1}^{m-1}(1-\alpha_{i}),$ (7)

Figure 3: In each pixel of the image pyramid, a depth-sorted list of colors
and alpha values is stored. The final color of each pixel is computed using
front-to-back alpha blending on the sorted list. Figure 4: Our design of one
gated convolution block that processes the features of the image pyramid.

###  3.3 Neural Network

The result produced by our renderer consists of a feature image pyramid
comprising $n$ layers. These individual layers are finally consolidated into a
single full-resolution image by a compact neural network, as depicted in Fig.
1. Our network architecture incorporates a single gated convolution [YLY∗19]
in each layer, featuring a self-bypass connection and a filter size of 32.
Additionally, we include a bilinear upsampling operation for all layers except
the final one, merging the output with the subsequent level. This
configuration is shown in Fig. 4 and resembles an efficient decoder network,
due to its restrained number of features, pixels, and convolutional
operations.

Unlike well-established hole-filling neural networks [ASK∗20, RFS22, RALB22],
our approach demands a significantly smaller and more efficient network. This
reduced network size stems from the fact that our renderer is adept at filling
gaps autonomously and generates smooth output. Consequently, the network’s
primary task is to learn minimal hole-filling and outlier removal, allowing it
to concentrate its efforts on high-quality texture reconstruction.

###  3.4 Spherical Harmonics Module and Tone Mapping

To model view dependent effects and camera-specific capturing parameters (like
exposure time), we optionally interpret the network output as spherical
harmonics (SH) coefficients, convert them to RGB colors, and finally pass the
result to a physically-based tone mapper. This allows the system to make use
of explicit view directions. The SH-module makes use of 3 bands, which
corresponds to 27 input coefficients. The tone mapper follows the work of
Rückert et al. [RFS22], which models exposure time, white balance, sensor
response, and vignetting.

###  3.5 Optimization Strategy

Before novel views can be synthesized, the rendering pipeline is optimized to
reproduce the input photographs. This optimization includes point position,
size, and features, as well as the camera model and poses, neural network
weights, and tone mapper parameters. We train for 600 epochs, which, depending
on scene size, requires 2-4 hours to converge.

As training criterion, we use the VGG-loss [JAF16] which has been shown to
provide high-quality results [RFS22]. The VGG network, however, tends to be
slow to evaluate, thus increasing training times significantly compared to MSE
loss. Therefore, we use a combination of MSE and SSIM [KKLD23] in the first 50
epochs when the advantages of VGG are still negligible. This speeds up
training time by about $5\%$ percent.

Similar to Kerbl and Kopanas [KKLD23], we use a "warm-up" period of 20 epochs,
during which we train with half image resolutions. Afterwards we randomly zoom
in and out each epoch, so that all convolutions (whose weights are not shared)
are trained to contribute to the final result.

###  3.6 Implementation Details

Our implementation uses torch as auto-differentiable backend, however the
trilinear renderer is implemented in custom CUDA kernels, as they commonly
provide better performance [KKLD23, RFS22]. Fast spherical harmonics encodings
are provided by tiny-cuda-nn [Mül21].

The renderer is implemented in three stages: collecting, splatting and
accumulation, albeit diverging from other state-of-the-art multi-layer
blending strategies, this turned out to work best in our scenario [FHSS18,
LZ21, VVP20]. We first project each point $p_{i}$ to the desired view and
collect each point’s $(x,y,z)$ as well as $s$ in a buffer, and also count how
many elements are mapped to each pixel. This counting is then used for an
offset scan to allocate continuous arrays for all layers combined. The
following splatting pass duplicates each point and stores a pair of $(z,i)$ in
each pixels’ list.

Following, a combined sorting and accumulation pass is done. Regarding
performance, this part is critical, as such we opt to only use the front most
16 elements from each (sorted) list, a common practice when blending points
[LZ21]. We could not identify any loss of quality caused by this
approximation, as the blending contribution of later points is very low. This
limitation allows us to use GPU-friendly sorting, as we repeat warp-local (32
threads) and shuffle-based bitonic sorts, always replacing the latter 16
elements with new unsorted ones, until the lists are empty. For the backwards
pass, the sorted per-pixel lists are stored, allowing fast backpropagation.
The front-to-back alpha blending (see Sec. 3.2) is done in the same pass as
the sorting pass, because all relevant elements are already in registers.

For scenes with a large deviation in point density, we found that occlusion
may not be correctly evaluated by the neural network in edge cases. Therefore,
we include points from coarser layers during blending (in the usual way), of
which the additional cost is very small ($<0.5$ms).

Point sizes are initialized with the average distance to the four nearest
neighbor, which is then efficiently optimized during training (see Fig. 5).

Table 1: Results on the Tanks&Temples and MipNeRF-360 datasets, as well as Boat and Office. See also Fig. 7 for visual comparisons. | Tanks&Temples | MipNeRF-360 | Boat | Office
---|---|---|---|---
Method |  LPIPS$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$
InstantNGP | 0.309 | 21.74 | 0.692 | 0.289 | 25.94 | 0.697 | 0.539 | 15.34 | 0.455 | 0.316 | 13.45 | 0.803
Mip-NeRF 360 | 0.203 | 24.61 | 0.789 | 0.174 | 28.23 | 0.796 | 0.676 | 12.20 | 0.357 | 0.322 | 15.21 | 0.833
Gaussian Spl. | 0.197 | 24.63 | 0.818 | 0.166 | 26.94 | 0.792 | 0.467 | 15.30 | 0.470 | 0.184 | 18.77 | 0.878
ADOP | 0.121 | 23.78 | 0.802 | 0.163 | 23.26 | 0.707 | 0.179 | 20.49 | 0.650 | 0.117 | 21.47 | 0.899
Ours | 0.099 | 24.64 | 0.808 | 0.122 | 25.94 | 0.772 | 0.179 | 20.38 | 0.633 | 0.107 | 21.36 | 0.887

##  4 Evaluation

Next, we compare our approach with prior arts as well as showcase the
effectiveness of our design decisions in ablation studies.

###  4.1 Setup and Datasets

We have evaluated our approach on several scenes from the Tanks&Temples
[KPZK17] and the MipNeRF-360 [BMV∗22] datasets. Additionally, we use the Boat
and Office scene from Rückert et al. [RFS22] to evaluate robustness towards
difficult input conditions. The former contains outdoor auto-exposed images
while the later is a office floor with multiple distinct room and a large
LiDAR point cloud, but sparsely placed cameras.

From Tanks&Temples, we use the intermediate set containing eight scenes:
Train, Playground, M60, Lighthouse, Family, Francis, Horse and Panther. These
scenes are outdoor scenes captured under varying lighting conditions but with
good spatial coverage and can be seen as a good baseline for robustness. The
MipNeRF-360 dataset [BMV∗22] consists of 5 outdoor and 4 indoor scenes. This
dataset was captured with controlled setups and has capture positions well
suited for volumetric rendering with a hemispherical setup [KD23].

For the quantitative evaluation we use the LPIPS [ZIE∗18], PSNR, and SSIM
metrics. We note however, that neither of these metrics always reflect visual
impression. Some approaches are trained with MSE-loss or SSIM and therefore
naturally perform better in PSNR and SSIM. Our approach, on the other hand, is
trained with VGG-loss and thus usually shows better scores on LPIPS. For a
fair comparison, we recommend to look at all metrics and closely inspect the
provided image and video comparison.

###  4.2 Quality Comparison

In Tab. 1 and Fig. 7, we compare our approach to InstantNGP [MESK22],
MipNeRF-360 [BMV∗22], 3D Gaussian Splatting [KKLD23] and ADOP [RFS22]. The
latter two are the closest-related point-based radiance field rendering
approaches.

On the Tanks&Temples dataset, our approach achieves in average the best LPIPS
score with an improvement of 20% over the second best. In PSNR and SSIM the
score is on par with state-of-the-art. On the MipNeRF-360 dataset, we obtain
again the best LPIPS score, however the volumetric methods and Gaussian
Splatting show an improved PSNR and SSIM. The difference can be inspected in
Fig. 7. For example, in row 3, the TRIPS rendering provides better sharpness
with more details, but the MipNeRF and Gaussian output is overall cleaner with
less noise. On the difficult boat and office scenes, we can show that our
rendering pipeline, is robust to extreme input conditions.

###  4.3 Ablation Studies

In this section, first we show the effect of our design choices.

####  4.3.1 Point-Size Optimization

Figure 5:  The initial COLMAP reconstruction lacks point on the pedestal of
the statue (top left). Our approach distibutes the few present points and
increases their sizes (bottom left) thus rendering them also in lower layers
(middle). Thus our pipeline can avoid distracting holes (right).

With our trilinear splatting technique, point sizes can be optimized to fill
large holes in the scene. We show this capability in Fig. 5, where the initial
point cloud exhibits a large hole in the pedestal of the horse producing
artifacts in rendering (top row). To combat this, our pipeline efficiently
moves and enlarges the points to fill the hole (bottom row), thus providing
great render quality.

####  4.3.2 Point Position Optimization

Figure 6: We added noise to the converged point clouds of ADOP and ours, then
restarted optimization for positions only. Ours is able to converge back to
the correct result, ADOP fails at that.

To test the efficiency of our trilinear point position optimization compared
to the (cheaper) approximate gradients from ADOP, we added random noise (of
$0.01$) to the positions of all points after training, then optimize only
point positions for 100 epochs. The result can be seen in Fig. 6. Our pipeline
is able to reconstruct the correct rendering from that, while ADOP’s rendering
barely improves.

Table 2:  Number of resolution layers used (horse scene).  #Layers |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  Time $\downarrow$
---|---|---|---|---
3 | 0.121 | 21.24 | 0.818 | 7.10ms
4 | 0.108 | 22.40 | 0.826 | 7.35ms
5 | 0.106 | 22.76 | 0.826 | 7.42ms
6 | 0.105 | 23.06 | 0.829 | 7.50ms
7 | 0.105 | 23.10 | 0.826 | 7.54ms
8 | 0.104 | 23.34 | 0.828 | 7.61ms

####  4.3.3 Number of Render Layers

Due to our trilinear point rendering algorithm, increasing the number of
pyramid layers has almost no negative impact on render time. As seen in Tab.
2, having 8 layers improves quality, especially with PSNR. For reference,
other approaches make use of 4 [RFS22] or 5 [ASK∗20] layers and describe
significant performance impacts when increasing the number of layers [RFS22].

Table 3:  View dependency on different scenes. On scenes with strong view dependency (Garden), adding view dependant configurations, either via our SH network module (SH-net) or optimized per point (SH-point) increases quality, however the per-point point setup severely impacts performance. Our module gives a balanced trade off, which also avoids over-fitting on less view-dependent scenes (Playground).  |  Playground (12.5M Points) |  Horse (1.8M Points) |  Garden (8.2M Points)
---|---|---|---
View-dep |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  Time $\downarrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  Time $\downarrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  Time $\downarrow$
none | 0.112 | 24.85 | 0.720 | 11.1ms | 0.103 | 22.73 | 0.822 | 7.7ms | 0.130 | 24.82 | 0.756 | 17.9ms
SH-net | 0.119 | 24.88 | 0.724 | 13.3ms | 0.109 | 22.81 | 0.825 | 8.9ms | 0.112 | 24.46 | 0.752 | 18.5ms
SH-point | 0.127 | 24.32 | 0.702 | 27.4ms | 0.104 | 22.89 | 0.829 | 10.2ms | 0.122 | 25.15 | 0.756 | 27.3ms
Figure 7: Visual comparisons.

####  4.3.4 View Dependency

After the neural network, optionally we use a spherical harmonics module to
model view depended artifacts of the scene. This improves the rendering
quality for some scenes (Garden), while for others it makes little to no
difference (see Tab. 3). Applying the spherical harmonics before the network
achieves roughly the same quality, but also reduces efficiency due to
additional memory overhead. On scenes without reflective materials, skipping
the spherical harmonics module is thus possible.

####  4.3.5 Feature Vector Dimensions

Our pipeline uses by default four feature descriptors per point. More features
only marginally increase the quality, while requiring significantly more
memory and slightly increasing rendering time, as shown in Tab. 4.

Table 4:  Features per point on the playground scene.  # Features |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  Time $\downarrow$
---|---|---|---|---
4 | 0.112 | 24.85 | 0.720 | 11.1ms
6 | 0.113 | 24.61 | 0.701 | 11.7ms
8 | 0.112 | 25.04 | 0.727 | 12.2ms

####  4.3.6 Networks

In our pipeline, we use a simple decoder network made out of gated
convolutions as presented in Sec. 3.3. ADOP [RFS22] on the other hand uses a
four layer U-net with double convolutions for encoder and decoder (thus around
6 times more parameter). As seen in Tab. 5, in our pipeline our networks
provide similar quality to ADOP’s full network, while being much faster in
inference. With spherical harmonics, inference times slightly increase, but
the system is now able to model view dependency. Adding the SH-module to the
second finest layer (ours+SH${}_{\text{L2}}$) instead of the finest (ours+SH)
of the network improves efficiency but shows weaker results.

Table 5: Network configuration compared (Playground scene). Network |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  Time $\downarrow$
---|---|---|---|---
ADOP-net | 0.124 | 24.74 | 0.713 | 10.7ms
ours | 0.112 | 24.85 | 0.720 | 4.5ms
ours+SH | 0.119 | 24.88 | 0.724 | 5.6ms
ours+SH${}_{\text{L2}}$ | 0.143 | 24.34 | 0.684 | 2.2ms

####  4.3.7 Time Scaling on Number of Points

As seen in Tab. 6, TRIPS is very efficient in rendering large amounts of
points. Even for our largest scene with more than 70M points, the pipeline
remains real-time capable with only 15ms required for rasterization.

Table 6:  Efficiency of our approach regarding point cloud sizes.  Scene | Horse | Garden | Playgr. | Boat | Office
---|---|---|---|---|---
#Points | 1.8M | 7.8M | 12.5M | 53.0M | 72.5M
Time | 2.5ms | 5.9ms | 6.4ms | 13.1ms | 15.0ms

###  4.4 Rendering Efficiency

In Tab. 7, we evaluate training and rendering time for all examined methods.
Our method train for around 2-4h per scene on an Nvidia A100 and renders a
novel view in around 11ms on an RTX4090.

Table 7: Training and render times on the Garden (images resolution: 2594$\times$1681) and Playground scene (1920$\times$1080).  Method | Train |  Render(Garden) |  Render(Playgr.)
---|---|---|---
InstantNGP | 0.25h | 131ms | 172ms
Mip-NeRF360 | 36h | 38000ms | 18000ms
ADOP | 8h | 30.3ms | 14.5ms
Gaussian Spl. | 0.75h | 11.5ms | 8.6ms
Ours | 4h | 16.4ms | 11.1ms

###  4.5 Outlier Robustness

Figure 8:  Comparison of outlier robustness on the family scene. Only our
methods is able to remove floating artifacts while still retaining full color
precision on the sidewalk.

As seen in Fig. 8, our approach is robust to outlier measurements, for
example, people walking through the scene. Especially volumetric approach like
MipNeRF-360, suffer from severe artifacts in this case, due to strong view-
dependant over-fitting capability.

###  4.6 Comparison to Prior Work with Number of Points

We have seen in previous experiments that Gaussian Splatting [KKLD23] has
blurrier results compared to TRIPS, which can be confirmed by their weak LPIPS
scores. However, they start with fewer point primitives (the SfM
reconstruction) and thus are limited in the amount of detail to display. To
this end, we conducted an experiment, where the Gaussian Splatting pipeline is
provided with the dense point cloud (providing the same input as for our
pipeline). Gaussian splatting has a pruning mechanism to remove unwanted
Gaussian, thus after their full training, from the initial 12.5M points only
around 8M survived.

The results of this experiment are presented in Tab. 8. It can be seen that
LPIPS improves with more Gaussians (however PSNR declines) as fine details can
be reconstructed better. The qualitative comparison paints the same picture
(see Fig. 9), where the quality of the grass improves drastically, however
finer details such as the chains still can only be reconstructed by us.
Overall the technique cannot reach the quality and scores of TRIPS, as we can
keep more points to render efficiently as well as use neural descriptors to
encode more detailed information.

Table 8:  Performance of the methods on the Playground scene. Gaussian (dense) starts with COLMAP’s dense reconstruction of 12M points and prunes them to 8M, Gaussian (sparse) is the original sparse setup and has about 2M points. Also see Fig. 9.  Method |  LPIPS$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  Time $\downarrow$
---|---|---|---|---
Ours | 0.123 | 25.12 | 0.746 | 11.1ms
ADOP | 0.138 | 22.86 | 0.753 | 14.5ms
Gaussian (dense) | 0.225 | 24.06 | 0.773 | 11.4ms
Gaussian (sparse) | 0.264 | 24.61 | 0.776 | 8.6ms
Figure 9:  Visual results of Gaussian splatting with COLMAP’s dense point
cloud as input compared its normal setup as well as ours, which provides the
sharpest results (Playground scene).

##  5 Limitations

In the preceding section, we have demonstrated TRIPS’ effectiveness on
commonly encountered real-world datasets. Nonetheless, we have also identified
potential limitations. One such limitation arises from the prerequisite to
have an initial dense reconstruction (in contrast to Gaussian Splatting),
which may not be practical in certain scenarios.

Additionally, our lack of an anisotropic splat formulation can create
problems: When our method is tasked with strong holefilling of elongated,
slender object (such as poles), noisy artifacts surrounding their silhouettes
can be observed. An example of this is depicted in Fig. 10. In such instances,
the slightly blurred edge characteristic of Gaussian Splatting is often
preferred.

Furthermore, even though the temporal consistency compared to previous point
rendering approaches [ASK∗20, RFS22] has been drastically improved, slight
flickering can still occur in areas with too many or too little points.

##  6 Conclusion

In this paper, we presented TRIPS, a robust real-time point-based radiance
field rendering pipeline. TRIPS employs an efficient strategy of rasterizing
points into a screen-space image pyramid, allowing the efficient rendering of
large points and is completely differentiable, thus allowing automatic
optimization of point sizes and positions. This technique enables the
rendering of highly detailed scenes and the filling of large gaps, all while
maintaining a real-time frame rate on commonly available hardware.

We highlight that TRIPS achieves high rendering quality, even in challenging
scenarios like scenes with intricate geometry, large-scale environments, and
auto-exposed footage. Moreover, due to the smooth point rendering approach, a
comparably simple neural reconstruction network is sufficient, resulting in
real-time rendering performance.

An open source implementation is available under:

https://github.com/lfranke/TRIPS

Figure 10:  Limitation: Holefilling close to the camera exhibits fuzzy edges
and shine-through.

## Acknowledgements

Linus Franke was supported by the Bayerische Forschungsstiftung (Bavarian
Research Foundation) AZ-1422-20. The authors gratefully acknowledge the
scientific support and HPC resources provided by the Erlangen National High
Performance Computing Center (NHR@FAU) of the Friedrich-Alexander-Universität
Erlangen-Nürnberg (FAU) under the NHR project b162dc. NHR funding is provided
by federal and Bavarian state authorities. NHR@FAU hardware is partially
funded by the German Research Foundation (DFG) – 440719683.

## References

  * [AGP∗04] Alexa M., Gross M., Pauly M., Pfister H., Stamminger M., Zwicker M.:  Point-based computer graphics.  In _ACM SIGGRAPH 2004 Course Notes_. 2004, pp. 7–es.
  * [ASK∗20] Aliev K.-A., Sevastopolsky A., Kolos M., Ulyanov D., Lempitsky V.:  Neural point-based graphics.  In _European Conference on Computer Vision_ (2020), Springer, pp. 696–712.
  * [BMT∗21] Barron J. T., Mildenhall B., Tancik M., Hedman P., Martin-Brualla R., Srinivasan P. P.:  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ (October 2021), pp. 5855–5864.
  * [BMV∗22] Barron J. T., Mildenhall B., Verbin D., Srinivasan P. P., Hedman P.:  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2022), pp. 5470–5479.
  * [BMV∗23] Barron J. T., Mildenhall B., Verbin D., Srinivasan P. P., Hedman P.:  Zip-nerf: Anti-aliased grid-based neural radiance fields.  _arXiv preprint arXiv:2304.06706_ (2023).
  * [CBLPM21] Chibane J., Bansal A., Lazova V., Pons-Moll G.:  Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2021), pp. 7911–7920.
  * [CDSHD13] Chaurasia G., Duchene S., Sorkine-Hornung O., Drettakis G.:  Depth synthesis and local warps for plausible image-based navigation.  _ACM Transactions on Graphics (TOG) 32_ , 3 (2013), 1–12.
  * [CXG∗22] Chen A., Xu Z., Geiger A., Yu J., Su H.:  Tensorf: Tensorial radiance fields.  In _Computer Vision – ECCV 2022_ (Cham, 2022), Avidan S., Brostow G., Cissé M., Farinella G. M., Hassner T., (Eds.), Springer Nature Switzerland, pp. 333–350.
  * [CXZ∗21] Chen A., Xu Z., Zhao F., Zhang X., Xiang F., Yu J., Su H.:  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ (2021), pp. 14124–14133.
  * [DNZ∗17] Dai A., Nießner M., Zollhöfer M., Izadi S., Theobalt C.:  Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration.  _ACM Transactions on Graphics (ToG) 36_ , 4 (2017), 1.
  * [DYB98] Debevec P., Yu Y., Boshokov G.:  Efficient view-dependent ibr with projective texture-mapping.  In _EG Rendering Workshop_ (1998), vol. 4.
  * [FHSS18] Franke L., Hofmann N., Stamminger M., Selgrad K.:  Multi-layer depth of field rendering with tiled splatting.  _Proceedings of the ACM on Computer Graphics and Interactive Techniques 1_ , 1 (2018), 1–17.
  * [FKYT∗22] Fridovich-Keil S., Yu A., Tancik M., Chen Q., Recht B., Kanazawa A.:  Plenoxels: Radiance fields without neural networks.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ (2022), pp. 5491–5500.  doi:10.1109/CVPR52688.2022.00542.
  * [FNPS16] Flynn J., Neulander I., Philbin J., Snavely N.:  Deepstereo: Learning to predict new views from the world’s imagery.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ (2016), pp. 5515–5524.
  * [GD98] Grossman J. P., Dally W. J.:  Point sample rendering.  In _Eurographics Workshop on Rendering Techniques_ (1998), Springer, pp. 181–192.
  * [GKSL16] Ganin Y., Kononenko D., Sungatullina D., Lempitsky V.:  Deepwarp: Photorealistic image resynthesis for gaze manipulation.  In _European conference on computer vision_ (2016), Springer, pp. 311–326.
  * [GSC∗07] Goesele M., Snavely N., Curless B., Hoppe H., Seitz S. M.:  Multi-view stereo for community photo collections.  In _2007 IEEE 11th International Conference on Computer Vision_ (2007), IEEE, pp. 1–8.
  * [HPP∗18] Hedman P., Philip J., Price T., Frahm J.-M., Drettakis G., Brostow G.:  Deep blending for free-viewpoint image-based rendering.  _ACM Transactions on Graphics (TOG) 37_ , 6 (2018), 1–15.
  * [HSM∗21] Hedman P., Srinivasan P. P., Mildenhall B., Barron J. T., Debevec P.:  Baking neural radiance fields for real-time view synthesis.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ (2021), pp. 5875–5884.
  * [JAF16] Johnson J., Alahi A., Fei-Fei L.:  Perceptual losses for real-time style transfer and super-resolution.  _CoRR abs/1603.08155_ (2016).  URL: http://arxiv.org/abs/1603.08155.
  * [KB04] Kobbelt L., Botsch M.:  A survey of point-based techniques in computer graphics.  _Computers & Graphics 28_, 6 (2004), 801–814.
  * [KD23] Kopanas G., Drettakis G.:  Improving NeRF Quality by Progressive Camera Placement for Free-Viewpoint Navigation.  In _Vision, Modeling, and Visualization_ (2023), Guthe M., Grosch T., (Eds.), The Eurographics Association.  doi:10.2312/vmv.20231222.
  * [KKLD23] Kerbl B., Kopanas G., Leimkühler T., Drettakis G.:  3d gaussian splatting for real-time radiance field rendering.  _ACM Transactions on Graphics 42_ , 4 (2023).
  * [KLL∗13] Keller M., Lefloch D., Lambers M., Izadi S., Weyrich T., Kolb A.:  Real-time 3D reconstruction in dynamic scenes using point-based fusion.  In _Proc. of Joint 3DIM/3DPVT Conference (3DV)_ (June 2013), pp. 1–8.  Selected for oral presentation.
  * [KLR∗22] Kopanas G., Leimkühler T., Rainer G., Jambon C., Drettakis G.:  Neural point catacaustics for novel-view synthesis of reflections.  _ACM Transactions on Graphics (TOG) 41_ , 6 (2022), 1–15.
  * [KPLD21] Kopanas G., Philip J., Leimkühler T., Drettakis G.:  Point-based neural rendering with per-view optimization.  In _Computer Graphics Forum_ (2021), vol. 40, Wiley Online Library, pp. 29–43.
  * [KPZK17] Knapitsch A., Park J., Zhou Q.-Y., Koltun V.:  Tanks and temples: Benchmarking large-scale scene reconstruction.  _ACM Transactions on Graphics 36_ , 4 (2017).
  * [LXG22] Liao Y., Xie J., Geiger A.:  Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d.  _IEEE Transactions on Pattern Analysis and Machine Intelligence 45_ , 3 (2022), 3292–3310.
  * [LZ21] Lassner C., Zollhöfer M.:  Pulsar: Efficient sphere-based neural rendering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ (June 2021), pp. 1440–1449.
  * [MBRS∗21] Martin-Brualla R., Radwan N., Sajjadi M. S., Barron J. T., Dosovitskiy A., Duckworth D.:  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2021), pp. 7210–7219.
  * [MESK22] Müller T., Evans A., Schied C., Keller A.:  Instant neural graphics primitives with a multiresolution hash encoding.  _arXiv preprint arXiv:2201.05989_ (2022).
  * [MGK∗19] Meshry M., Goldman D. B., Khamis S., Hoppe H., Pandey R., Snavely N., Martin-Brualla R.:  Neural rerendering in the wild.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2019), pp. 6878–6887.
  * [MKC07] Marroquim R., Kraus M., Cavalcanti P. R.:  Efficient point-based rendering using image reconstruction.  In _PBG@ Eurographics_ (2007), pp. 101–108.
  * [MSOC∗19] Mildenhall B., Srinivasan P. P., Ortiz-Cayon R., Kalantari N. K., Ramamoorthi R., Ng R., Kar A.:  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  _ACM Transactions on Graphics (TOG) 38_ , 4 (2019), 1–14.
  * [MST∗21] Mildenhall B., Srinivasan P. P., Tancik M., Barron J. T., Ramamoorthi R., Ng R.:  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Communications of the ACM 65_ , 1 (2021), 99–106.
  * [Mül21] Müller T.:  tiny-cuda-nn, 4 2021.  URL: https://github.com/NVlabs/tiny-cuda-nn.
  * [NSP∗21] Neff T., Stadlbauer P., Parger M., Kurz A., Mueller J. H., Chaitanya C. R. A., Kaplanyan A., Steinberger M.:  Donerf: Towards real-time rendering of compact neural radiance fields using depth oracle networks.  In _Computer Graphics Forum_ (2021), vol. 40, Wiley Online Library, pp. 45–59.
  * [PGA11] Pintus R., Gobbetti E., Agus M.:  Real-time rendering of massive unstructured raw point clouds using screen-space operators.  In _Proceedings of the 12th International conference on Virtual Reality, Archaeology and Cultural Heritage_ (2011), pp. 105–112.
  * [PZ17] Penner E., Zhang L.:  Soft 3d reconstruction for view synthesis.  _ACM Transactions on Graphics (TOG) 36_ , 6 (2017), 1–11.
  * [PZVBG00] Pfister H., Zwicker M., Van Baar J., Gross M.:  Surfels: Surface elements as rendering primitives.  In _Proceedings of the 27th annual conference on Computer graphics and interactive techniques_ (2000), pp. 335–342.
  * [RALB22] Rakhimov R., Ardelean A.-T., Lempitsky V., Burnaev E.:  NPBG++: accelerating neural point-based graphics.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (June 2022), pp. 15969–15979.
  * [RFS22] Rückert D., Franke L., Stamminger M.:  Adop: Approximate differentiable one-pixel point rendering.  _ACM Transactions on Graphics (TOG) 41_ , 4 (2022), 1–14.
  * [RK20] Riegler G., Koltun V.:  Free view synthesis.  In _European Conference on Computer Vision_ (2020), Springer, pp. 623–640.
  * [RK21] Riegler G., Koltun V.:  Stable view synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2021), pp. 12216–12225.
  * [RWL∗22] Rückert D., Wang Y., Li R., Idoughi R., Heidrich W.:  Neat: Neural adaptive tomography.  _ACM Trans. Graph. 41_ , 4 (jul 2022).  URL: https://doi.org/10.1145/3528223.3530121, doi:10.1145/3528223.3530121.
  * [SCCL20] Song Z., Chen W., Campbell D., Li H.:  Deep novel view synthesis from colored 3d point clouds.  In _European Conference on Computer Vision_ (2020), Springer, pp. 1–17.
  * [SF16] Schonberger J. L., Frahm J.-M.:  Structure-from-motion revisited.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ (2016), pp. 4104–4113.
  * [SK00] Shum H., Kang S. B.:  Review of image-based rendering techniques.  In _Visual Communications and Image Processing 2000_ (2000), vol. 4067, SPIE, pp. 2–13.
  * [SKW19] Schütz M., Krösl K., Wimmer M.:  Real-time continuous level of detail rendering of point clouds.  In _2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)_ (2019), IEEE, pp. 103–110.
  * [SKW21] Schütz M., Kerbl B., Wimmer M.:  Rendering point clouds with compute shaders and vertex order optimization.  In _Computer Graphics Forum_ (2021), vol. 40, Wiley Online Library, pp. 115–126.
  * [SKW22] Schütz M., Kerbl B., Wimmer M.:  Software rasterization of 2 billion points in real time.  _arXiv preprint arXiv:2204.01287_ (2022).
  * [SMB∗20] Sitzmann V., Martel J., Bergman A., Lindell D., Wetzstein G.:  Implicit neural representations with periodic activation functions.  _Advances in Neural Information Processing Systems 33_ (2020).
  * [SSS06] Snavely N., Seitz S. M., Szeliski R.:  Photo tourism: exploring photo collections in 3d.  In _ACM siggraph 2006 papers_. 2006, pp. 835–846.
  * [STB∗19] Srinivasan P. P., Tucker R., Barron J. T., Ramamoorthi R., Ng R., Snavely N.:  Pushing the boundaries of view extrapolation with multiplane images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2019), pp. 175–184.
  * [STH∗19] Sitzmann V., Thies J., Heide F., Nießner M., Wetzstein G., Zollhofer M.:  Deepvoxels: Learning persistent 3d feature embeddings.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2019), pp. 2437–2446.
  * [SZPF16] Schönberger J. L., Zheng E., Pollefeys M., Frahm J.-M.:  Pixelwise view selection for unstructured multi-view stereo.  In _European Conference on Computer Vision (ECCV)_ (2016).
  * [TMW∗21] Tancik M., Mildenhall B., Wang T., Schmidt D., Srinivasan P. P., Barron J. T., Ng R.:  Learned initializations for optimizing coordinate-based neural representations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2021), pp. 2846–2855.
  * [TRS22] Turki H., Ramanan D., Satyanarayanan M.:  Mega-nerf: Scalable construction of large-scale nerfs for virtual fly-throughs.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2022), pp. 12922–12931.
  * [TS20] Tucker R., Snavely N.:  Single-view view synthesis with multiplane images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2020), pp. 551–560.
  * [TTM∗22] Tewari A., Thies J., Mildenhall B., Srinivasan P., Tretschk E., Yifan W., Lassner C., Sitzmann V., Martin-Brualla R., Lombardi S., et al.:  Advances in neural rendering.  In _Computer Graphics Forum_ (2022), vol. 41, Wiley Online Library, pp. 703–735.
  * [TZN19] Thies J., Zollhöfer M., Nießner M.:  Deferred neural rendering: Image synthesis using neural textures.  _ACM Transactions on Graphics (TOG) 38_ , 4 (2019), 1–12.
  * [VVP20] Vasilakis A.-A., Vardis K., Papaioannou G.:  A survey of multifragment rendering.  In _Computer Graphics Forum_ (2020), vol. 39, Wiley Online Library, pp. 623–642.
  * [WGSJ20] Wiles O., Gkioxari G., Szeliski R., Johnson J.:  Synsin: End-to-end view synthesis from a single image.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2020), pp. 7467–7477.
  * [WSMG∗16] Whelan T., Salas-Moreno R. F., Glocker B., Davison A. J., Leutenegger S.:  Elasticfusion: Real-time dense slam and light source estimation.  _The International Journal of Robotics Research 35_ , 14 (2016), 1697–1716.
  * [YCA∗20] Yang Z., Chai Y., Anguelov D., Zhou Y., Sun P., Erhan D., Rafferty S., Kretzschmar H.:  Surfelgan: Synthesizing realistic sensor data for autonomous driving.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ (June 2020).
  * [YLT∗21] Yu A., Li R., Tancik M., Li H., Ng R., Kanazawa A.:  Plenoctrees for real-time rendering of neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ (2021), pp. 5752–5761.
  * [YLY∗19] Yu J., Lin Z., Yang J., Shen X., Lu X., Huang T. S.:  Free-form image inpainting with gated convolution.  In _Proceedings of the IEEE/CVF international conference on computer vision_ (2019), pp. 4471–4480.
  * [YSW∗19] Yifan W., Serena F., Wu S., Öztireli C., Sorkine-Hornung O.:  Differentiable surface splatting for point-based geometry processing.  _ACM Transactions on Graphics (TOG) 38_ , 6 (2019), 1–14.
  * [YYTK21] Yu A., Ye V., Tancik M., Kanazawa A.:  pixelnerf: Neural radiance fields from one or few images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2021), pp. 4578–4587.
  * [ZBRH22] Zhang Q., Baek S.-H., Rusinkiewicz S., Heide F.:  Differentiable point-based radiance fields for efficient view synthesis.  _arXiv preprint arXiv:2205.14330_ (2022).
  * [ZIE∗18] Zhang R., Isola P., Efros A. A., Shechtman E., Wang O.:  The unreasonable effectiveness of deep features as a perceptual metric.  In _CVPR_ (2018).
  * [ZPVBG01] Zwicker M., Pfister H., Van Baar J., Gross M.:  Surface splatting.  In _Proceedings of the 28th annual conference on Computer graphics and interactive techniques_ (2001), pp. 371–378.
  * [ZTF∗18] Zhou T., Tucker R., Flynn J., Fyffe G., Snavely N.:  Stereo magnification: Learning view synthesis using multiplane images.  _ACM Transactions on Graphics (TOG) 37_ , 4 (2018), 1–12.
  * [ZTS∗16] Zhou T., Tulsiani S., Sun W., Malik J., Efros A. A.:  View synthesis by appearance flow.  In _European conference on computer vision_ (2016), Springer, pp. 286–301.