# Comparison of Bayesian and particle swarm algorithms for hyperparameter
optimisation in machine learning applications in high energy physics

Veelken, Christian1<EMAIL_ADDRESS>
(1 _National Institute Of Chemical Physics And Biophysics (NICPB), Akadeemia
tee 23, 12618 Tallinn, Estonia_ )

###### Abstract

When using machine learning (ML) techniques, users typically need to choose a
plethora of algorithm-specific parameters, referred to as hyperparameters. In
this paper, we compare the performance of two algorithms, particle swarm
optimisation (PSO) and Bayesian optimisation (BO), for the autonomous
determination of these hyperparameters in applications to different ML tasks
typical for the field of high energy physics (HEP). Our evaluation of the
performance includes a comparison of the capability of the PSO and BO
algorithms to make efficient use of the highly parallel computing resources
that are characteristic of contemporary HEP experiments.

##  1 Introduction

Machine learning (ML) methods often aid the analysis of the vast amounts of
data that are produced by contemporary high energy physics (HEP) experiments.
The ML algorithms often feature tunable parameters, referred to as
hyperparameters, which need to be chosen by the user and often have a
significant effect on the algorithm’s performance. In a previous publication
[1] we presented two different algorithms, particle swarm optimisation (PSO)
and the genetic algorithm, for the autonomous determination of these
hyperparameters. In the present paper we compare the performance of the PSO
algorithm, the more promising of the two algorithms studied in our previous
publication, to the performance of the Bayesian optimisation (BO) [2, 3, 4, 5,
6] algorithm. The latter is widely used for the task of finding optimal
hyperparameter values in the context of ML applications since the pioneering
work of Refs. [7, 8, 9]. The “asynchronous successive halving algorithm”
(ASHA) [10] is an alternative algorithm for optimising the values of
hyperparameters, which is popular in the ML community outside the field of
HEP.

As in our previous publication, we formulate the task of determining the set
of optimal hyperparameter values as a function maximisation problem. More
specifically, given an ML algorithm $\mathcal{A}$, we seek to find a point $h$
in the space $\mathcal{H}$ of hyperparameters, such that the performance of
the ML algorithm $\mathcal{A}$ attains its maximum at this point:

$\hat{h}=\mathop{\mathrm{argmax}}_{h\,\in\,\mathcal{H}}\,s(h)\,,$

where the objective (or “score”) function (OF) $s(h)$ quantifies the
performance of the ML algorithm $\mathcal{A}$, and the point $h$ at which
$s(h)$ attains its maximum is denoted by the symbol $\hat{h}$. We compare the
performance of the PSO and BO algorithms on two benchmark tasks, the task of
finding the minimum of the Rosenbrock function [11], and on a typical data
analysis task in the field of HEP, the “ATLAS Higgs boson machine learning
challenge” [12].

An important aspect in applications of ML algorithms in the field of HEP is an
algorithm’s capability to make efficient use of massively parallel computing
facilities. A single training of an ML algorithm on a single machine may take
several hours, days, or, in extreme cases, even weeks. In the context of the
hyperparameter optimisation task, such a single training corresponds to a
single evaluation of the OF $s(h)$. In order for the hyperparameter
optimisation task to finish within an acceptable time, different evaluations
of the OF, i.e. different ML trainings, need to be executed in parallel. The
computing facilities of contemporary HEP experiments typically allow users
concurrent access to hundreds, sometimes even thousands, of machines. It is
therefore of high practical relevance whether the PSO and BO algorithms can
organise the hyperparameter optimisation task such that hundreds of ML
trainings can be executed in parallel. We find that both algorithms fulfill
this requirement, but do exhibit some differences in performance compared to
the case that all ML trainings are executed sequentially on a single machine.

The manuscript is structured as follows: In Section 2, we present the main
concepts of the BO algorithm. The PSO algorithm is described in Ref. [1]. In
Section 3, we compare the performance of the PSO and BO algorithms on the two
benchmark tasks. The study of the parallelisation capability of the two
algorithms is presented in Section 4. We conclude the paper with a summary in
Section 5.

##  2 Bayesian Optimisation

The BO algorithm is designed to facilitate the numerical maximisation of
objective functions $s(h)$ which are time-consuming to evaluate and for which
the analytical form is in general not known. The BO algorithm further allows
to solve the maximisation task without using derivative information on $s(h)$.

This is achieved by performing the maximisation not on $s(h)$ directly, but on
an approximation of $s(h)$, which is referred to as the “surrogate” function
(SF). The SF is chosen such that it is fast to evaluate and its analytic form,
including its derivative, is known. We use a Gaussian process [13] with the
Matérn kernel [14] for the SF in this paper. For each point
$h\,\in\,\mathcal{H}$, the SF provides two values: an estimate for the value
$s(h)$ of the OF and a confidence interval. The latter represents an estimate
of the accuracy of the approximation of the OF by the SF at this point.

The numerical maximisation of the OF is performed by an iterative procedure.
Each iteration consists of two steps: The first step consists of finding the
next point $h$ for which to perform the time-consuming evaluation of the OF.
The task of finding this point is performed by an “acquisition function” (AF).
The inputs to the AF consist of the estimate for the values $s(h)$ provided by
the SF and the estimated accuracy of the approximation of the OF by the SF at
this point. In the second step, the SF is updated with the information of the
value of the OF at the point $h$, in order to improve the accuracy of the
approximation. Each evaluation of the OF thus serves two purposes: first, to
find points $h$ where the OF attains a higher value $s(h)$ compared to
previously found points and second, to improve the accuracy of the
approximation of the OF by the SF. We use the expected improvement (EI) [15]
for the AF in this paper. The time-consuming evaluation of the OF is performed
at the point where the AF reaches its maximum. The search for the maximum of
the AF is performed numerically, using the Broyden–Fletcher–Goldfarb–Shanno
(BFGS) [16, 17, 18, 19] algorithm. The numeric search for the maximum takes
advantage of the fact that both the SF and the AF are fast to evaluate, which
allows multiple evaluations of the SF and AF to be made in order to find the
best point for for which to make the time-consuming next evaluation of the OF.
The EI acquisition function has a parameter, denoted by the symbol $\xi$, that
allows to regulate how much importance is given to finding points $h$ where
the OF attains a high value (“exploitation”) versus finding points which
improve the accuracy of the approximation (“exploration”). The former are
typically located in the neighbourhood of previously found points, while the
latter are typically located in previously unexplored regions of the space
$\mathcal{H}$. The update of the SF after each evaluation of the OF is the
feature to which the Bayesian optimisation algorithm owes its name. The SF is
referred to as “prior function” before evaluating the OF and as “posterior
function” afterwards.

The BO algorithm is started by evaluating the OF at an initial set of points
and fitting the SF to the values $s(h)$ of the OF at these points. This
initial set of points is chosen with the objective of populating the space
$\mathcal{H}$ uniformly. We use the Latin hypercube [20] algorithm to obtain
this initial set of points. After fitting the SF to the values $s(h)$ of the
OF at these points, the BO algorithm enters the iterative phase: Given the SF,
the BFGS algorithm is used to find the location where the AF attains its
maximum. The OF is then evaluated at this point and the SF is updated. These
steps are repeated until either the algorithm has converged, indicated by
changes in the value of the OF that fall below a given threshold, or the
computing resources are exhausted, i.e. a maximum number of iterations or a
computing-time limit is reached.

The operation of the BO algorithm is illustrated by means of an example in
Figs. 1 and 2, which visualise the seventh and eighth iteration of the BO
algorithm, respectively. The OF is represented by the solid blue line and the
SF by the dashed black line in the upper part of each figure. The red diamond-
shaped markers indicate the points where the OF has been evaluated previously
(including $2$ initial points, which were chosen randomly in this example).
The shaded blue area represents the confidence interval that quantifies the
accuracy of the approximation of the OF by the SF. The AF is represented by
the solid red line in the lower part of the figures. The yellow circle along
the red line indicates the maximum of the AF, i.e. the point $h$ where the
next evaluation of the OF is performed.

Figure 1: Example for the operation of the BO algorithm: iteration 7. Figure
2: Example for the operation of the BO algorithm: iteration 8.

The BO algorithm has been implemented as described above by the authors. We
have validated our implementation by comparing its performance to the
implementation of the BO algorithm included in the scikit-optimize [21]
package.

The BO algorithm that we described above allows to evaluate the OF at one
point $h$ per iteration of the algorithm. This is well suited for executing
the BO algorithm sequentially on a single machine. For the purpose of
optimising the hyperparameters of ML algorithms the training of which is
performed on massively parallel computing facilities, the iterative phase of
the BO algorithm needs to be extended to provide multiple points $h$ per
iteration of the BO algorithm. This extension of the BO algorithm is non-
trivial and still a field of active research [22]. In this paper, we follow
the approach described in Ref. [23], referred to as “multi-points expected
improvement” ($q$-EI), and use the implementation provided by Ref. [24] when
running the hyperparameter parameter optimisation on multiple machines in
parallel.

##  3 Performance

In this section, the performance of the PSO and BO algorithms are compared on
the two benchmark tasks: the finding of the minimum of the Rosenbrock function
and the ATLAS Higgs boson machine learning challenge.

###  3.1 Rosenbrock function

The Rosenbrock function [11] is a widely used trial function for evaluating
the performance of function minimisation algorithms. It is defined as:

$R(x,y)=(a-x)^{2}+b(y-x^{2})^{2}\,.$ (1)

It’s domain is the x–y plane and it depends on two parameters $a$ and $b$. The
global minimum of the Rosenbrock function is located at $(x,y)=(a,a^{2})$. We
choose the two parameters to be $a=1$ and $b=10$, resulting in the minimum to
be located at $(x,y)=(1,1)$ and the value of the Rosenbrock function at the
minimum to be $R(1,1)=0$. See Fig. 3 in Ref. [1] for a visualisation of the
Rosenbrock function around the minimum.

We treat the task of finding the minimum of the Rosenbrock function as a two
dimensional hyperparameter optimisation problem. The optimal set of
hyperparameters is scanned in the range of $[-500,+500]\times[-500,+500]$.

Both the BO and the PSO algorithms were executed for $30$ iterations. During
each iteration, $100$ different hyperparameter sets were evaluated in
parallel. The parameter settings for both algorithms are given in Tables 1 and
2. The minimisation of the Rosenbrock function was repeated for $1000$ trials,
using a different random number seed for each trial.

Parameter | Value
---|---
$N^{points}_{init}$ | $100$
$\xi$ | $0.01$
Table 1:  Parameter settings for the BO algorithm. The parameter $N^{points}_{init}$ refers to the number of points, obtained from the Latin hypercube algorithm, that are used to initialise the BO algorithm. The parameter $\xi$ regulates the relative importance of exploitation versus exploration of the EI acquisition function.  Parameter | Value
---|---
_$N_{info}$_ | $10$
_$c_{1}$_ | $1.62$
_$c_{2}$_ | $1.62$
_$w_{min}$_ | $0.4$
_$w_{max}$_ | $0.8$
Table 2:  Parameter settings for the PSO algorithm. The parameters are
described in Ref. [1].

##### Performance

We denote the location of the minimum found in each trial $i$ by the symbol
$\hat{\vphantom{\rule{1.0pt}{5.71527pt}}\smash{\hat{h}}}_{i}$ and the value of
the Rosenbrock function at these points by the symbol
$\hat{\vphantom{\rule{1.0pt}{5.71527pt}}\smash{\hat{R}}}_{i}=R(\hat{\vphantom{\rule{1.0pt}{5.71527pt}}\smash{\hat{h}}}_{i})$.
The average of these values over the $1000$ trials,
$\langle\hat{\vphantom{\rule{1.0pt}{5.71527pt}}\smash{\hat{R}}}\rangle=\frac{1}{1000}\cdot\sum_{i=1}^{1000}\hat{\vphantom{\rule{1.0pt}{5.71527pt}}\smash{\hat{R}}}_{i}$,
is shown in Fig. 3. For the first $10$ iterations, the BO algorithm converges
faster to the minimum than the PSO algorithm, but improves less rapidly than
the PSO algorithm for more than $10$ iterations. This difference in the rate
of convergence is expected, since the BO algorithm has been developed for
applications where the number of evaluations of the OF is in the order of a
few hundred to a thousand [22] (here it is $3000$, given by the product of
$30$ iterations times $100$ different hyperparameter sets that are evaluated
in parallel per iteration). Neither the BO nor the PSO algorithm converges to
the true minimum $R=0$ within $30$ iterations. We have shown in Ref. [1] that
the PSO algorithm converges to the true minimum after $10^{4}$ iterations. The
shaded band in Fig. 3 represents the standard deviation of the
$\hat{\vphantom{\rule{1.0pt}{5.71527pt}}\smash{\hat{R}}}_{i}$ over the $1000$
trials.

Figure 3:  Minimum of the Rosenbrock function found by the BO and PSO
algorithms, as function of the number of iterations. The lines represent the
average performance over $1000$ trials and the shaded bands the variation
(standard deviation) of the performance.

###  3.2 The ATLAS Higgs Boson machine learning challenge

The _ATLAS Higgs Boson machine learning challenge (HBC)_ [12] constitutes the
second benchmark task for evaluating the performance of the BO and PSO
algorithms. As in our previous publication [1], we choose a boosted decision
tree (BDT), implemented in the XGBoost [25] package, for the ML algorithm. The
hyperparameter optimisation is performed with respect to the $7$
hyperparameters given in Table 3 of Ref. [1]. During the optimisation, each of
the $7$ hyperparameters is restricted to be within the range given in Table 3.

Hyperparameter | min | max
---|---|---
_num-boost-round_ | $1$ | $500$
_learning-rate_ | $10^{-5}$ | $1$
_max-depth_ | $1$ | $6$
_gamma_ | $0$ | $5$
_min-child-weight_ | $0$ | $500$
_subsample_ | $0.8$ | $1$
_colsample-bytree_ | $0.3$ | $1$
Table 3:  Minimum and maximum values of the hyperparameters for the HBC. The
hyperparameters are detailed in Ref. [12].

The optimisation of the hyperparameters by the BO and PSO algorithms was run
for $30$ iterations. $70$ BDTs, initialised with different random number
seeds, were trained in parallel per iteration. The parameter settings used for
the BO and PSO algorithm are the same as for the task of finding the minimum
of the Rosenbrock function and are given in Tables 1 and 2, except for the
parameter $N_{info}$ of the PSO algorithm, which was set to $7$ for the HBC
task. The $550000$ signal and background events provided by the organisers of
the HBC are split into training and test sets as described in Ref. [1]. The
“modified approximate mean significance” (d-AMS) defined by Eq. (8) of Ref.
[1] was used as objective function for the BDT training. The d-AMS scores were
computed setting the coefficient $\kappa$ that controls the penalty term
against overtraining to $0.3$. The hyperparameter optimisation was repeated
for $100$ trials.

####  3.2.1 Results

The average hyperparameter values found by the BO and PSO algorithms over the
$100$ trials and the standard deviation of these values are reported in Table
4. The values found by both algorithms are similar. The _gamma_ hyperparameter
has little effect on the d-AMS score and is thus not well constrained by the
optimisation.

Hyperparameter | BO | PSO
---|---|---
$\mu$ | $\sigma$ | $\mu$ | $\sigma$
_num-boost-round_ | $364.3$ | $85.2$ | $413.7$ | $85.4$
_learning-rate_ | $0.126$ | $0.041$ | $0.089$ | $0.024$
_max-depth_ | $3.9$ | $0.7$ | $4.5$ | $0.7$
_gamma_ | $1.78$ | $2.00$ | $2.81$ | $1.47$
_min-child-weight_ | $352.7$ | $130.1$ | $440.4$ | $72.4$
_subsample_ | $0.861$ | $0.081$ | $0.871$ | $0.066$
_colsample-bytree_ | $0.852$ | $0.268$ | $0.859$ | $0.145$
Table 4:  Average values ($\mu$) of the hyperparameters found by the BO and
PSO algorithms and their standard deviation ($\sigma$) for the HBC task.

The performance of BDTs is evaluated on two separate samples of events,
referred to as the _public_ and _private_ leaderboard samples [1, 12].
Following Ref. [1], the performance is quantified by the “approximate mean
significance” (AMS) score. The latter is averaged over the $100$ trials. The
motivation for using the d-AMS score as objective function for the training,
while the (final) performance is quantified using the AMS score is detailed in
Ref. [1]. The evolution of the average performance as function of the number
of iterations is shown in Fig. 4. The bottom part of the figure shows the
evolution of the d-AMS score, the objective function used for the BDT
training. While the d-AMS score keeps increasing monotonously with more
iterations, the AMS scores on the public and private leaderboard samples reach
a plateau already after $5$–$10$ iterations. In subsequent iterations, the AMS
scores start to fluctuate around the plateau. The magnitude of the
fluctuations is of $\mathcal{O}(10^{-2})$. The differences in performance
between the public and private leaderboard samples is compatible with a
statistical fluctuation of the signal and background events contained in these
samples [12]. The BO and PSO algorithms achieve a similar performance on the
HBC task.

Figure 4:  Evolution of the AMS score as function of the number of iterations
for the HBC task. The lines represent the average performance over $100$
trials.

##  4 Parallelisation capability

The capability of the BO and PSO algorithms to make efficient use of parallel
computing resources is studied on the task of finding the minimum of the
Rosenbrock function. We denote by $N_{parallel}$ the number of points $h$ in
hyperparameter space that are evaluated in parallel per iteration of the BO
and PSO algorithms. The value of $N_{parallel}$ represents the number of ML
trainings that are executed in parallel on different machines. Ideally, the
duration (“wall time”) of the hyperparameter optimisation task should decrease
inversely proportional to $N_{parallel}$, except for some (small) overhead
imposed by the BO and PSO algorithms.

##### Bayesian optimisation

Based on the literature [22], one expects the BO algorithm to perform best
when all ML trainings are performed sequentially on a single machine, with one
point $h$ in hyperparameter space evaluated per iteration. We have compared
the performance, quantified by the values of
$\langle\hat{\vphantom{\rule{1.0pt}{5.71527pt}}\smash{\hat{R}}}\rangle$,
obtained when executing our implementation of the BO algorithm (described in
Section 2) sequentially on a single machine ($N_{parallel}=1$) with the
performance obtained by the $q$-EI algorithm from Ref. [24], with the latter
running either $N_{parallel}=2$, $5$, $10$, $25$, $50$, $100$, $250$, $500$ or
$1000$ ML trainings in parallel. We find that the values of
$\langle\hat{\vphantom{\rule{1.0pt}{5.71527pt}}\smash{\hat{R}}}\rangle$ are
very similar in all cases and mainly depend on the total number of evaluations
of the Rosenbrock function, given by the product of the number of iterations
times $N_{parallel}$. For $3000$ evaluations, the value of
$\langle\hat{\vphantom{\rule{1.0pt}{5.71527pt}}\smash{\hat{R}}}\rangle$
amounts to about $10^{2}$ for all values of $N_{parallel}$ that we have tried.

The computing (CPU) time overhead imposed by the BO algorithm for updating the
SF and finding the maximum of the AF amounts to about $50$ CPU hours on a
$2.30$ GHz Intel® Xeon® E$5$-$2695$ v$3$ processor. The overhead does not vary
much as function of $N_{parallel}$. The task of updating the SF and finding
the maximum of the AF needs to run (as a “supervisor” task) on a single
machine. The overhead is sizeable compared to the computing time spent on
performing $3000$ evaluations of the Rosenbrock function, which only takes
$0.06$ CPU seconds, but becomes less important for “real” ML training
applications: The total computing time spend on the HPC task (with $30$
iterations and $70$ ML trainings running in parallel) amounts to about $500$
CPU hours.

##### Particle swarm optimisation

In contrast to the BO algorithm, the capability to make efficient use of
parallel computing resources is an intrinsic feature of the PSO algorithm. We
find that the performance of the PSO algorithm depends on the value of
$N_{parallel}$, which corresponds to the number of particles in the swarm, in
a non-trivial manner. Both too small and too large values of $N_{parallel}$
degrade the performance of the PSO algorithm compared to the optimal value. We
find that the optimal value of $N_{parallel}$ amounts to about $2\%$ times the
total number ($N_{tot}$) of evaluations of the Rosenbrock function, i.e. the
best performance of the PSO algorithm is achieved when using
$N_{parallel}=0.02\cdot N_{tot}$ particles in the swarm and a fixed number of
$50$ iterations (up to $10000$ evaluations of the Rosenbrock function that we
have tried). We recommend to set the parameter $N_{info}$ of the PSO algorithm
to $10\%$ times $N_{parallel}$.

We find that the CPU time overhead imposed by the PSO algorithm (spend on
updating the positions and momenta of the particles in the swarm) is
negligible for both benchmark tasks.

#####

The CPU time overhead limits the “speedup” (reduction in wall time) that one
can achieve by increasing the number of points in hyperparameter space that
are evaluated in parallel. The effect is referred to as “Amdahl’s law” [26] in
the literature and visualized in Fig. 5. An ideal algorithm with negligible
overhead would achieve a speedup factor equal to $N_{parallel}$ in this
figure. The speedup achieved by the PSO algorithm on both benchmark tasks is
very close to the ideal case. The CPU time overhead limits the speedup factor
achievable by the BO algorithm to about $3$ for the task of finding of the
minimum of the Rosenbrock function and to about $12$ for the HBC task.

Figure 5: Amdahl’s law: Parallelisation properties of the BO and PSO
algorithms on the two benchmark tasks. The two curves for the PSO algorithm
are very close.

##  5 Summary

We have compared the performance of two autonomous algorithms for the
optimisation of hyperparameters, Bayesian optimisation (BO) and particle swarm
optimisation (PSO), on two benchmark tasks typical for ML applications in the
field of high energy physics: the task of finding the minimum of the
Rosenbrock function and the ATLAS Higgs boson machine learning challenge.

We find that the BO algorithm performs better than the PSO algorithm when the
total number of evaluations of the Rosenbrock function (equivalent to the
number of ML trainings) is in the order of a few hundred to a few thousand. In
case the number of evaluations (ML trainings) is large, the PSO algorithm
outperforms the BO algorithm.

The capability of both algorithms to make efficient use of parallel computing
resources is good. In particular, we find that the “multi-points expected
improvement” of the BO algorithm provides similar performance when running on
parallel computing resources compared to executing the BO algorithm
sequentially on a single machine. In case the PSO algorithm, we found that the
best performance is achieved when setting the number of particles in the swarm
to $2\%$ times the total number of function evaluations (ML trainings) and
using a fixed number of $50$ iterations.

We found that the BO algorithm may add a significant computational overhead to
the task of finding the optimal hyperparameter values, while for the PSO
algorithm the overhead is insignificant.

### Acknowledgements

We would like to thank our colleague Joosep Pata for helpful discussions. This
work has been supported by the Estonian Research Council grant PRG445.

## References

  * [1] L. Tani, D. Rand, C. Veelken, and M. Kadastik, “Evolutionary algorithms for hyperparameter optimization in machine learning for application in high energy physics,” The European Physical Journal C, vol. 81, no. 2, p. 1, 2021.
  * [2] H. J. Kushner, “A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise,” Journal of Basic Engineering, vol. 86, p. 97, 03 1964.
  * [3] A. G. Zhilinskas, “Single-step Bayesian search method for an extremum of functions of a single variable,” Cybernetics, vol. 11, no. 1, p. 160, 1975\.
  * [4] J. Močkus, V. Tiesis, and A. Zilinskas, “The application of Bayesian methods for seeking the extremum,” Towards Global Optimization, vol. 2, p. 117, 09 2014.
  * [5] J. Močkus, “On Bayesian methods for seeking the extremum,” in Optimization Techniques IFIP Technical Conference Novosibirsk, July 1–7, 1974 (G. I. Marchuk, ed.), (Berlin, Heidelberg), p. 400, Springer, 1975.
  * [6] J. Močkus, Bayesian approach to global optimization.  Springer, 1989.
  * [7] J. Snoek, H. Larochelle, and R. P. Adams, “Practical Bayesian optimization of machine learning algorithms,” in Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 2, NIPS’12, (Red Hook, NY, USA), p. 2951, Curran Associates Inc., 2012.
  * [8] J. Bergstra, D. Yamins, and D. D. Cox, “Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms,” in Proceedings of the 12th Python in Science Conference (Stéfan van der Walt, Jarrod Millman, and Katy Huff, eds.), p. 13, 2013.
  * [9] C. Thornton, F. Hutter, H. H. Hoos, and K. Leyton-Brown, “Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms,” in Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’13, (New York, NY, USA), p. 847, Association for Computing Machinery, 2013.
  * [10] L. Li, K. Jamieson, A. Rostamizadeh, E. Gonina, M. Hardt, B. Recht, and A. Talwalkar, “A system for massively parallel hyperparameter tuning,” arXiv:1810.05934, 2020.
  * [11] Y.-W. Shang and Y.-H. Qiu, “A note on the extended Rosenbrock function,” Evolutionary Computation, vol. 14, no. 1, p. 119, 2006.
  * [12] ATLAS collaboration, “Dataset from the ATLAS Higgs boson machine learning challenge,” in CERN Open Data Portal, 2014.
  * [13] C. E. Rasmussen and C. K. I. Williams, Gaussian processes for machine learning.  The MIT Press, 2005.
  * [14] M. G. Genton, “Classes of kernels for machine learning: a statistics perspective,” J. Mach. Learn. Res., vol. 2, p. 299, Mar. 2002.
  * [15] D. R. Jones, M. Schonlau, and W. J. Welch, “Efficient global optimization of expensive black-box functions,” Journal of Global Optimization, vol. 13, no. 4, p. 455, 1998.
  * [16] C. G. Broyden, “The convergence of a class of double-rank minimization algorithms 1. general considerations,” IMA Journal of Applied Mathematics, vol. 6, p. 76, 03 1970.
  * [17] R. Fletcher, “A new approach to variable metric algorithms,” The Computer Journal, vol. 13, p. 317, 01 1970.
  * [18] D. Goldfarb, “A family of variable-metric methods derived by variational means,” Mathematics of Computation, vol. 24, no. 109, p. 23, 1970.
  * [19] D. F. Shanno, “Conditioning of quasi-Newton methods for function minimization,” Mathematics of Computation, vol. 24, no. 111, p. 647, 1970\.
  * [20] M. D. McKay, R. J. Beckman, and W. J. Conover, “A comparison of three methods for selecting values of input variables in the analysis of output from a computer code,” Technometrics, vol. 21, no. 2, p. 239, 1979.
  * [21] F. Pedregosa et al., “Scikit-learn: machine learning in Python,” JMLR, vol. 12, p. 2825, 2011.
  * [22] P. I. Frazier, “A tutorial on Bayesian optimization,” arXiv:1807.02811, 2018.
  * [23] D. Ginsbourger, R. L. Riche, and L. Carraro, “A multi-points criterion for deterministic parallel global optimization based on Gaussian processes,” tech. rep., HAL open science, hal-00260579, 2008.
  * [24] J. Wang, S. C. Clark, E. Liu, and P. I. Frazier, “Parallel Bayesian global optimization of expensive functions,” Oper. Res., vol. 68, p. 1850, 2020\.
  * [25] T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” in Proceedings, 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, p. 785, 2016.
  * [26] G. M. Amdahl, “Validity of the single processor approach to achieving large scale computing capabilities,” in Proceedings of the April 18-20, 1967, Spring Joint Computer Conference, AFIPS ’67 (Spring), (New York, NY, USA), p. 483, Association for Computing Machinery, 1967.


# TeleSHift: Telexisting TUI for Physical Collaboration & Interaction

Andrew Chen  National Taiwan University of Science and TechnologyTaipeiTaiwan
,  Tzu-Ling Yang  National Taiwan UniversityTaipeiTaiwan ,  Shu-Yan Cheng
National Taiwan University of Science and TechnologyTaipeiTaiwan ,  Po-Sheng
Cheng  National Taiwan UniversityTaipeiTaiwan ,  Tzu-Han Lin  National Taiwan
University of Science and TechnologyTaipeiTaiwan and  Kaiyuan Lin  National
Taiwan Normal UniversityTaipeiTaiwan

(2022)

###### Abstract.

Current workflow on co-editing and simultaneous presentation of 3-D shapes is
confined to on-screen manipulation, which causes loss of perceived information
when presenting perceptual concepts or complex shapes between members. Thus,
we create TeleSHift, a 3-D tangible user interface (TUI) with a telexisting
communication framework for group-based collaboration and demonstration. In
this work, we present a larger-scaled proof-of-concept prototype providing
hands-on operation for shape-based interactions including multi-sided
collaboration and one-to-many presentation. In contrast to previous works, we
further extend the use of TUIs to support cooperative tasks with telexistence
while enabling the linkage of manipulable bits to provide a better user
experience and interactivity.

Telexistence, Tangible User Interfaces, Collaboration, Co-editing

††journalyear: 2022††copyright: rightsretained††conference: Proceedings of the
2022 ACM International Joint Conference on Pervasive and Ubiquitous Computing;
September 11–15, 2022; Cambridge, United Kingdom††booktitle: Proceedings of
the 2022 ACM International Joint Conference on Pervasive and Ubiquitous
Computing (UbiComp/ISWC ’22 Adjunct), September 11–15, 2022, Cambridge, United
Kingdom††doi: 10.1145/3544793.3560323††isbn: 978-1-4503-9423-9/22/09††ccs:
Human-centered computing Interaction devices††ccs: Human-centered computing
Collaborative and social computing devices

##  1\. Introduction

Tangible User Interfaces (TUIs) enable humans to perceive spatial information
more intuitively. Ever since its introduction, its applications have extended
from simple static model-presenting tasks to providing touch-based feedback
with a degree of interactivity. Even so, we consider that its applications
have not been thoroughly explored and see the potential of TUIs in supporting
the collaboration of physical prototyping and dimensional presentation.

When it comes to presenting 3-D shapes and structures with 2-D interfaces,
especially during simultaneous editing, perceptual loss of accessed
information would inevitably occur. The concept of telexistence is to make two
separated objects in different physical spaces to sense co-presence via
interaction. On top of that, we proposed a 3-D TUI to achieve more realistic
feedback and thorough perception, while enhancing interaction experience on
both dimensional information knowing and object-initiated interpersonal
exchanges.

This work contributes by exploring the use of TUIs in physically manipulating
and presenting 3-D shapes while also creating a functional, larger-scaled
proof-of-concept prototype with telexisting abilities to provide parallel
display and simultaneous altering of shape-based information for multi-sided
physical collaboration.

##  2\. Related Works

Our work focuses on demonstrating cooperation and mutual knowing of
dimensional modeling in different physical spaces with a programmable, shape-
shifting TUI. Aside from exploring digital telexisting(Tachi, 2009) frameworks
for physical-to-physical interactions(Philippe et al., 2014; Nishida and
Suzuki, 2017), we focus on providing a better experience in physical modeling.

Bit-Scaled Operations in Tangible Interfaces:  The concept of Tangible
Bits(Ishii and Ullmer, 1997) was incorporated to create more immersive
interfaces with controllable small scaled bits to create a larger experience.
Approaches to this concept include Materiable(Nakagaki et al., 2016),
rendering material properties with bit-sized manipulation via physical
simulation with a large tabletop interface. Works in this scope focus solely
on the interactive interpretation of intended information, and this work aims
to utilize such techniques into supporting collaborative physical modeling
tasks.

Interactivity in Physical Modelling:  We see a gap within the scope of
previous works regarding hands-on control of dimensions as manipulable area
and methods of physical design is being limited, while only a handful of
designated usages could be performed with its confined interaction methods,
such as NURBSforms(Tahouni et al., 2020) enables prototyping of curved
surfaces with a magnetically actuated interface with the ability to record and
load previously saved dimensions. Thus, in this work, we focus on exploring
hands-on prototyping or presenting hardware for any 3-D shape on any scale.

This work aims to initiate an approach to extending such concepts into
cooperation tasks, creating controllable self-deforming objects with the
ability to connect with each other for a larger scale of interactivity while
implementing a telexistence framework for collaboration in separate physical
spaces.

##  3\. Implementation

Figure 1. The storyboard illustrating the concept of TeleSHift along with our
proof-of-concept prototype presented on the top left.

We present TeleSHift, which is a structure created by Telexisting shape-
shifting substructures and used as a TUI. Our implementation of the
substructure is constructed to have six extension arms able to perform
1-dimensional extend-interact movements. The arms, which consist of motorized
slide potentiometers fixed onto the aluminum ridges, are each separated by 90
degrees to present deformation on the positive and negative of the x- y- and
z-axes. Furthermore, the extending arms are also used to connect with other
substructures by magnets for swarming to shape-shift on a larger scale,
creating TeleSHifts. The substructures’ control and communication are
propelled with ESP32 and can exchange data with Google Realtime Firebase to
update its current status including dimensions and jointing states, or fetch
data to enable shape recovery and achieve synchronization with other connected
TeleSHifts in different physical spaces.

The TeleSHift storyboard along with its larger-scaled proof-of-concept
hardware interpretation is presented in Fig. 1.

##  4\. Proposing Scenarios

###  4.1. Into Prototype Collaboration

TeleSHift enables physical collaboration on editing the same 3-D structure in
separate spaces effectively, breaking space restrictions and creating real-
time interaction for consensus reaching on physical dimensions: When a group
of designers physically in different spaces are working together on the
creation of an ergonomic mouse, a common workflow would rely on 2-D interfaces
of computer-aided design tools. However, a loss of perceived information,
especially when collaborating simultaneously, would occur when presenting
perceptual concepts or complex shapes between members. With TeleSHift, the
designers can now work on the mouse’s prototype at the same time: Not only
could they present the deformation of the created shape, but parallel knowing
and simultaneous shape-shifting can also be achieved when one manually
modifies their TeleSHift by presenting the same deformation status on the
others simultaneously, and vice versa.

###  4.2. Into Dimensional Teaching

TeleSHift also provides specific functionalities such as single-sided
presentation, one-to-many demonstration, or saving shape-shift dimensions. An
instance needing such requirements would be when a teacher wants to teach
about the basic structure of a chair to several students and asks them to
create their own design with some high-cost material.

The teacher can manually demonstrate the shaping process with TeleSHift, while
students can watch and learn about the structural change with their own
TeleSHifts at the same time. Moreover, students’ manipulation of their devices
would not affect both the other students and the teacher, which allows
students to come up with their creations. On the other hand, students can
store the deformation status during their design at any time. When an undo
process is required, they can return to the specific shape and redesign the
chair. With TeleSHift as a substitute for expensive or raw materials, students
can constantly reshape their objects, thus reducing cost and mitigating the
waste of materials.

##  5\. Future Work

The concept of TeleSHift is to enable two-staged shape-shifting for the
presentation of any physical shapes at the users’ discretion. This is achieved
by substructure-to-substructure connection positioning for larger scale
changes and self-deformation of the substructures to create smaller scale
alterations.

In the future, we would like to work on scaling down the prototype and mass
producing the substructures to take TeleSHift to its full potential. Also, as
our current design only incorporates 6 extending arms to represent the axes in
3-D space, to add on more such structures would be worked on to provide a
higher resolution of shape presentation.

###### Acknowledgements.

The authors would like to thank Yu-Ju Yang for valuable suggestions and
assistance on scenario presenting as well as video making.

## References

  * (1)
  * Ishii and Ullmer (1997) Hiroshi Ishii and Brygg Ullmer. 1997.  Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms. In _Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems_ (Atlanta, Georgia, USA) _(CHI ’97)_. Association for Computing Machinery, New York, NY, USA, 234–241.  https://doi.org/10.1145/258549.258715
  * Nakagaki et al. (2016) Ken Nakagaki, Luke Vink, Jared Counts, Daniel Windham, Daniel Leithinger, Sean Follmer, and Hiroshi Ishii. 2016.  Materiable: Rendering Dynamic Material Properties in Response to Direct Physical Touch with Shape Changing Interfaces. In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_ (San Jose, California, USA) _(CHI ’16)_. Association for Computing Machinery, New York, NY, USA, 2764–2772.  https://doi.org/10.1145/2858036.2858104
  * Nishida and Suzuki (2017) Jun Nishida and Kenji Suzuki. 2017.  BioSync: A Paired Wearable Device for Blending Kinesthetic Experience. In _Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems_ (Denver, Colorado, USA) _(CHI ’17)_. Association for Computing Machinery, New York, NY, USA, 3316–3327.  https://doi.org/10.1145/3025453.3025829
  * Philippe et al. (2014) Bertrand Philippe, Daniel Gonzalez-Franco, Arthur Pointeau, and Christian Cherene. 2014.  The Machine to be Another-Embodied Telepresence using human performers.  _Prix Ars Electronica_ 96 (2014).
  * Tachi (2009) Susumu Tachi. 2009\.  _Telexistence_.  World Scientific Publishing Co., Inc., USA.
  * Tahouni et al. (2020) Yasaman Tahouni, Isabel P. S. Qamar, and Stefanie Mueller. 2020\.  NURBSforms: A Modular Shape-Changing Interface for Prototyping Curved Surfaces. In _Proceedings of the Fourteenth International Conference on Tangible, Embedded, and Embodied Interaction_ (Sydney NSW, Australia) _(TEI ’20)_. Association for Computing Machinery, New York, NY, USA, 403–409.  https://doi.org/10.1145/3374920.3374927


capbtabboxtable[][]

11institutetext: Microsoft 22institutetext: University of Oxford
22email<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Photo-realistic 360∘ Head Avatars in the Wild

Stanislaw Szymanowicz Work done while at Microsoft.1122 Virginia Estellers 11
Tadas Baltrušaitis 11
Matthew Johnson 11

###### Abstract

Delivering immersive, 3D experiences for human communication requires a method
to obtain 360∘ photo-realistic avatars of humans. To make these experiences
accessible to all, only commodity hardware, like mobile phone cameras, should
be necessary to capture the data needed for avatar creation. For avatars to be
rendered realistically from any viewpoint, we require training images and
camera poses from all angles. However, we cannot rely on there being trackable
features in the foreground or background of all images for use in estimating
poses, especially from the side or back of the head. To overcome this, we
propose a novel landmark detector trained on synthetic data to estimate camera
poses from 360∘ mobile phone videos of a human head for use in a multi-stage
optimization process which creates a photo-realistic avatar. We perform
validation experiments with synthetic data and showcase our method on 360∘
avatars trained from mobile phone videos.

##  1 Introduction

Immersive interaction scenarios on Mixed Reality devices require rendering
human avatars from all angles. To avoid the uncanny valley effect, these
avatars must have faces that are photo-realistic. It is likely that in the
future virtual spaces will become a ubiquitous part of every-day life,
impacting everything from a friendly gathering to obtaining a bank loan. For
this reason we believe high-quality, 360∘ avatars should be affordable and
accessible to all: created from images captured by commodity hardware, e.g.,
from a handheld mobile phone, without restrictions on the surrounding
environment.

Figure 1: Our system creates photo-realistic 360∘ avatars from captures from a
mobile phone capture and without constraints on the environment. Cameras are
registered from full 360∘ pose variation, and our multi-stage optimization
pipeline allows for high quality avatars.

Obtaining data to train a 360∘ photo-realistic avatar ‘in the wild’ is
challenging due to the potential difficulty of camera registration:
traditional Structure-from-Motion pipelines rely on reliable feature matches
of static objects across different images. Prior work limits the captures to a
120∘ frontal angle which allows the use of textured planar objects that are
amenable to traditional feature detectors and descriptors (e.g., a book,
markers, detailed wall decoration). However, in many 360∘ captures from a
mobile phone in unconstrained environments neither the background nor the
foreground can be depended upon to provide a source of such matches.

There are several properties of 360∘ captures in the wild which pose serious
challenges to camera registration and avatar model learning. First, the space
being captured is likely to either have plain backgrounds (e.g., white walls)
and/or portions of the capture in which the background is an open space,
leading to defocus blur and the inclusion of extraneous, potentially mobile
objects (e.g., pets, cars, other people). Second, in order to obtain the
needed details on the face and hair the foreground subject will likely occupy
much of the frame. While the face can provide some useful features for camera
registration, its non-planar nature combined with changes in appearance due to
lighting effects make it less than ideal. Further, while the back of the head
can produce many features for tracking the matching can become highly
ambiguous due to issues with hair, i.e., specular effects and repeated
texture.

To address the challenges of 360∘ captures we propose a multi-stage pipeline
to create 3D photo-realistic avatars from a mobile phone camera video. We
propose using head landmarks to estimate the camera pose. However, as most
facial landmark detectors are not reliable at oblique or backward-facing
angles, we propose using synthetic data to train landmark detectors capable of
working in full 360∘ range. We use the predicted landmarks to provide
initialization of the 6DoF camera poses, for a system which jointly optimizes
a simplified Neural Radiance Field with the camera poses. Finally, we use the
optimized camera poses to train a high quality, photo-realistic NeRF of the
subject.

The contributions of our work are three-fold: (1) a reliable system for camera
registration which only requires the presence of a human head in each photo,
(2) a demonstration of how to leverage synthetic data in a novel manner to
obtain a DNN capable of predicting landmark locations from all angles, and (3)
a multi-stage optimization pipeline which builds 360∘ photo-realistic avatars
with high-frequency visual details using images obtained from a handheld
mobile phone ‘in the wild’.

##  2 Related work

Photo-realistic avatars in the wild. Recent works on surface[1, 5, 16] and
volumetric avatars of human heads[3, 9] enroll the avatar from a dynamic video
of the subject, and they constrain the capture to only take frontal views,
either through limitations of the face tracker[1, 3, 5, 16] or by requiring a
highly textured, static background[9] to acquire camera poses. Instead, our
subject is static and captured by another person. In contrast to all of the
approaches above, this scenario enables photo-realistic novel-view renders
from the full 360∘ range, whereas previous works have only shown results from
within a frontal 180∘.

Detecting occluded landmarks. Landmark detection [15] is a challenging task
even under $\pm 60^{\circ}$ yaw angle variation [17]. Existing datasets and
approaches are limited to $\pm 90^{\circ}$[2, 17], partially due to the
difficulty of reliably annotating dense landmarks in such data. Fortunately,
recent work on using synthetic data in face understanding[14] has shown
evidence of being able to detect landmarks even under severe occlusions. We
push this capability to the limit: we aim to detect all landmarks even if
completely occluded, i.e., observing the back of the head. Unlike prior work,
we use a synthetic dataset with full 360∘ camera angle variations. The
synthetic nature of the dataset allows us to calculate ground truth landmark
annotations which could not be reliably labelled on real data.

Camera poses. We estimate initial camera poses from detected face landmarks,
similarly to object pose detection from keypoints[7]. We are not constrained
by the requirement of highly textured planar scene elements which Structure-
from-Motion (SfM) methods impose. We are therefore able to capture our
training data in the wild: we perform robust camera registration when a head
is observed in the image, while SfM approaches can fail to register all
cameras when insufficient matches are available, such as when observing dense
dark hair.

Optimizing camera poses jointly with NeRF has been the topic of BARF[6] and
NeRF–[13]. We use the optimization method from BARF, but it only uses one,
coarse, NeRF network. Therefore, for maximum visual quality we run a second
optimization, starting from the optimized camera poses but initializing the
coarse and fine networks from scratch and optimizing them jointly with camera
poses. We hence achieve the high-frequency visual detail that the two-stage
sampling provides, while optimizing the camera poses to avoid artefacts.

Figure 2: We build a 360∘ Neural Radiance Field of a person from a video
captured from a mobile phone. In each selected frame we detect dense
landmarks, and we use them to register the cameras from all angles. In the
joint optimization process, we refine camera poses jointly with NeRF,
obtaining a highly detailed volumetric avatar.

##  3 Method

We estimate camera pose by solving a Perspective-n-Point (PnP) problem that
matches 3D points in the world and their corresponding 2D projections in the
images. The 2D image points are dense face landmarks while the 3D world points
are the same landmarks annotated on a template face model[14]. We use standard
convolutional neural networks (CNN) to estimate landmarks and standard RANSAC
PnP solvers to solve for camera pose. Our results are possible because our
landmark detector, trained with synthetic data, provides good landmarks for
360∘ views of a face within $\pm 60^{\circ}$ of elevation.

Landmarks for Pose Estimation. The main difference between our landmark
detector and competing approaches is the number and quality of landmarks,
specially for non-frontal views. While a human can consistently label frontal
face images with tens of landmarks, it is impossible to annotate an image of
the back of the head with hundred of landmarks in a manner consistent in 3D.
Using synthetic data allows us to generate images with perfectly consistent
labels. We use the framework introduced by Wood et al. [14] to render a
synthetic training dataset of 100k images that contain extreme head and camera
pose variations, covering the 360∘ range. We use it to train an off-the-shelf
ResNet-101 landmark detector model to directly regress 703 landmark positions
from an image[15].

The camera poses estimated from landmarks might contain errors because the
occluded landmarks are a rough estimate based on the priors learned by the
CNN, but they are good enough for our avatar generation system.

Avatar Generation. We choose to represent our 3D avatar with a Neural Radiance
Field[8] (NeRF): a volumetric representation of the scene because it can
generate novel views with a high level of photo-realism and can handle human
hair and accessories seamlessly[12]. This is important for in-the-wild
captures where no assumptions of subjects appearance are possible.

We use the original representation, architecture and implementation details of
NeRF[8]. We optimize the camera poses jointly with the radiance field in a
coarse-to-fine manner, as in BARF[6]. While BARF[6] uses only the coarse MLP,
we improve on its proposed optimization scheme with a second optimization step
to achieve higher fidelity. The steps are as follows: (1) Initialize camera
poses from landmarks and optimize them jointly with the coarse MLP following
BARF[6]. Discard the coarse MLP. (2) Initialize camera poses from step 1 and
optimize them jointly with both coarse and fine MLPs initialized from scratch.

##  4 Data

We use two types of data: (1) synthetic data, allowing us to perform
experiments in a controllable manner and (2) 360∘ mobile phone captures of
real people.

We generate three synthetic faces. For each, we render 90 400x400 px training
and 10 testing images from camera poses varying across the entire 360∘ range
and elevation angle between $-25$ and $45$ degrees, simulating a 360∘ capture
that could be done by someone walking around a person sitting still in a
chair.

We collect two real datasets: in each, one person sits still in a chair, and
another person takes a video of them from a mobile phone. The capture covers
360∘ angles in forms of 3 orbits at different elevation levels. The data is
preprocessed for training the avatar by selecting 100 images without motion
blur, undistorting them, cropping to a square, segmenting out the background
and resizing to 400x400 resolution.

##  5 Experiments

Dense landmark detection and pose registration. Fig. 3 shows dense landmark
detection on real and synthetic images of faces. The results on real images of
the back of the head show that a network trained with synthetic data learns
correct priors to estimate plausible landmarks from weak cues like ears or the
head orientation. With these landmarks, we can register cameras from images of
360∘ angles around the head (blue frustums in Fig. 3) where traditional SfM
pipelines like COLMAP[10, 11] (red frustums in Fig. 3) partially fail because
images of the back of the head do not have enough reliable matches.

Figure 3: Detected landmarks are plausible on both synthetic and real faces from all viewpoints and allow to register all cameras (blue frustums). In our sequences SfM pipelines like COLMAP only register front-facing cameras (red frustums). Landmark + PnP | Optimized |
---|---|---
|  |  | Cameras |  PSNR |  R(deg) |  t(cm)
---|---|---|---
GT | 35.18 | - | -
PnP | 30.34 | 2.60 | 7.66
Optim. | 35.19 | 0.334 | 1.66
Figure 4: Novel viewpoints rendered from a NeRF trained with camera poses from
PnP exhibit artefacts (left). Optimizing (Optim.) poses jointly with NeRF
removes the artefacts (middle), recovers the visual quality of NeRF trained
with Ground Truth cameras (GT) and the registration error (right). Best viewed
zoomed in.

Avatars of synthetic faces. We compare avatar quality as trained from 1)
ground truth poses, 2) poses as estimated from dense landmarks, 3) poses
optimized jointly with NeRF when initialized from dense landmarks. Fig. 4
qualitatively compares novel view renders from different approaches and its
associated table provides quantitative metrics w.r.t. ground truth camera
poses.

Training a NeRF avatar with noisy poses estimated from landmarks leads to
artefacts in the renders, but using them as initialization and optimizing
jointly with NeRF removes the artefacts and successfully recovers camera
poses. The error in camera poses, leads to a significant drop in PSNR (Fig. 4)
of novel-view renders. However, optimizing camera poses jointly with NeRF
leads to novel-view quality on par with that obtained using the ground truth
cameras.

Figure 5: Novel view renders of 360∘ avatars trained from images captured from
a phone.

Avatars of real faces. We test our full pipeline on two real captures. Fig. 5
shows novel-view renders of avatars trained on 360∘ capture. The detail in
renders is visible even from the back, where the initial camera poses were
likely with the most error. We hypothesize that white regions on the torso
stem from being cropped by segmentation masks in training data.

Fig. 6 shows the comparison of our method to the baselines on real data: (1)
using camera poses as estimated from the dense landmarks, (2) using cameras
from manually joined models output by off-the-shelf Structure-from-Motion
software COLMAP, (3) optimizing a coarse NeRF jointly with camera poses
initialized with the estimate from dense landmarks, (4) two-stage optimization
process where second stage optimizes both fine and coarse NeRF from scratch
with poses initialized at the output of (3).

Error in camera poses result in avatars with artefacts (Fig. 6, left). NeRF
optimized jointly with camera poses in the same manner as in BARF does not
exhibit the artefacts but lacks in visual detail due to only training the
coarse network (Fig. 6, top right). Bottom right row of Fig. 6 illustrates
that adding the second optimization stage recovers the fine detail while
avoiding artefacts, because the second stage starts from a good initialization
of camera poses, and optimizes them jointly with both the coarse and fine
network.

Figure 6: Training a NeRF with cameras with a registration error results in
artefacts in renders from novel viewpoints (left). Using BARF[6] removes the
artefacts, but lacks the high-frequency details (top right). Adding our second
optimization stage leads to an avatar with high-frequency detail but without
artefacts (bottom right).

##  6 Conclusions

Limitations and future work. First, while our current formulation only renders
a static avatar, recent work[4] could be used to animate it using geometry-
driven deformations. Secondly, we train NeRF on images with background
segmented out, and segmentation errors, e.g., on the hair, can lead to
incorrectly learnt colour which is then revealed from different viewpoints
resulting in artefacts. Future work should aim to refine masks in the
optimization process by using the projection of the density field. Finally, we
observed that the camera pose estimates for the subject with hair covering the
ears resulted in larger camera pose error. While in our experiments we
observed that the pose optimization converged to a good solution, future work
should (1) investigate the basin of convergence of camera poses and (2) test
the method on more subjects.

Conclusions In this paper we demonstrate that we can obtain a 360∘ avatar from
a mobile phone video of a subject gathered ‘in the wild’. We train a facial
landmark DNN on synthetic data which is sampled from the full 360∘ range of
view angles. We then show that the landmarks predicted by this DNN can be used
to initialize the camera poses for the mobile phone image sequence. This is
the first stage of an optimization pipeline which we demonstrate can
progressively fine-tune the camera poses while training a NeRF that can be
used to create photo-realistic, highly detailed images of the subject from
novel views, including from the back and side. This can be used as the
foundation for a 360∘ avatar of the kind shown in [4].

## References

  * [1] Cao, C., Simon, T., Kim, J.K., Schwartz, G., Zollhoefer, M., Saito, S.S., Lombardi, S., Wei, S.E., Belko, D., Yu, S.I., Sheikh, Y., Saragih, J.: Authentic volumetric avatars from a phone scan. ACM Trans. Graph. 41(4) (2022). https://doi.org/10.1145/3528223.3530143, https://doi.org/10.1145/3528223.3530143
  * [2] Deng, J., Roussos, A., Chrysos, G., Ververas, E., Kotsia, I., Shen, J., Zafeiriou, S.: The menpo benchmark for multi-pose 2d and 3d facial landmark localisation and tracking. International Journal of Computer Vision 127 (2019)
  * [3] Gafni, G., Thies, J., Zollhöfer, M., Nießner, M.: Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. CVPR (2021)
  * [4] Garbin, S.J., Kowalski, M., Estellers, V., Szymanowicz, S., Rezaeifar, S., Shen, J., Johnson, M., Valentin, J.: Voltemorph: Realtime, controllable and generalisable animation of volumetric representations (2022), https://arxiv.org/abs/2208.00949
  * [5] Grassal, P.W., Prinzler, M., Leistner, T., Rother, C., Nießner, M., Thies, J.: Neural head avatars from monocular rgb videos. CVPR (2022)
  * [6] Lin, C.H., Ma, W.C., Torralba, A., Lucey, S.: Barf: Bundle-adjusting neural radiance fields. ICCV (2021)
  * [7] Lowe, D.: Object recognition from local scale-invariant features (1999)
  * [8] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. ECCV (2020)
  * [9] Park, K., Sinha, U., Barron, J.T., Bouaziz, S., Goldman, D.B., Seitz, S.M., Martin-Brualla, R.: Nerfies: Deformable neural radiance fields. ICCV (2021)
  * [10] Schönberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: CVPR (2016)
  * [11] Schönberger, J.L., Zheng, E., Pollefeys, M., Frahm, J.M.: Pixelwise view selection for unstructured multi-view stereo. In: ECCV (2016)
  * [12] Vicini, D., Jakob, W., Kaplanyan, A.: A non-exponential transmittance model for volumetric scene representations. SIGGRAPH 40(4), 136:1–136:16 (Aug 2021). https://doi.org/10.1145/3450626.3459815
  * [13] Wang, Z., Wu, S., Xie, W., Chen, M., Prisacariu, V.A.: NeRF$--$: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064 (2021)
  * [14] Wood, E., Baltrušaitis, T., Hewitt, C., Dziadzio, S., Johnson, M., Estellers, V., Cashman, T.J., Shotton, J.: Fake it till you make it: Face analysis in the wild using synthetic data alone (2021)
  * [15] Wood, E., Baltrušaitis, T., Hewitt, C., Johnson, M., Shen, J., Milosavljevic, N., Wilde, D., Garbin, S., Raman, C., Shotton, J., Sharp, T., Stojiljkovic, I., Cashman, T., Valentin, J.: 3d face reconstruction with dense landmarks (2022). https://doi.org/10.48550/ARXIV.2204.02776
  * [16] Zheng, Y., Abrevaya, V.F., Bühler, M.C., Chen, X., Black, M.J., Hilliges, O.: I M Avatar: Implicit morphable head avatars from videos. CVPR (2022)
  * [17] Zhu, X., Lei, Z., Liu, X., Shi, H., Li, S.Z.: Face alignment across large poses: A 3d solution. CVPR (2016)