# Following in Tini’s Giant Footsteps††thanks: Contribution to the special
volume of Acta Physica Polonica B commemorating
Martinus Veltman

John Ellis
King’s College London, Strand, London, WC2R 2LS, United Kingdom;
Theoretical Physics Department, CERN, Geneva, Switzerland;
National Institute of Chemical Physics & Biophysics, Rävala 10, 10143 Tallinn,
Estonia

###### Abstract

This paper describes my personal appreciation of some of Tini Veltman’s great
research achievements and how my own research career has followed the pathways
he opened. Among the topics where he has been the most influential have been
the pursuit and study of the Higgs boson and the calculation of radiative
corrections that enabled the masses of the top quark and the Higgs boson to be
predicted ahead of their discoveries. The search for physics beyond the
Standard Model may require a complementary approach, such as the search for
non-renormalizable interactions via the Standard Model Effective Field Theory.

KCL-PH-TH/2021-27, CERN-TH-2021-067

##  1 Introduction

It is an honour to have been offered the opportunity to contribute to this
memorial volume celebrating Martinus (Tini, as he was generally known)
Veltman, particularly because he was one of my scientific heroes. In addition
to his personal scientific research, he played a central role in putting Dutch
theoretical particle physics on the international map (think Gerard ’t Hooft,
Bernard De Wit, Peter van Nieuwenhuizen and many others), pioneered the
development of computer algebra with his Schoonschip programme, and was a key
early supporter of the LEP accelerator. Moreover, beneath his occasionally
idiosyncratic and outspoken exterior lurked a fiercely independent thinker
with empathy for young theorists whose work he appreciated, as well as for the
experimentalists whose work he encouraged.

##  2 Renormalization

Tini’s main scientific achievement was of course the demonstration (together
with ’t Hooft) how spontaneously-broken non-Abelian gauge theories could be
renormalized [1, 2, 3, 4], and my main scientific contacts with Tini resulted
from my own research work pursuing corollaries of that work. It is worth
remembering that Tini’s most consequential work was done somewhat in the
wilderness, struggling tirelessly with a problem that most theorists
disregarded: “sweeping an odd corner of weak interactions” as he reported
being told by Sidney Coleman [5]. For one thing, quantum field theory in
general was unfashionable for much of the 1960s, as causality, analyticity and
the S-matrix held sway. For another, very few theorists could see the interest
in non-Abelian gauge theories. Significant progress had been made by Feynman,
DeWitt, Faddeev and Popov in formulating massless non-Abelian theories at the
quantum level, but nobody understood how to renormalize them if the gauge
bosons were massive, and obtain sensible, finite results.

Many suggested that massive gauge vector bosons were the likely mediators of
the weak interactions, once the $V-A$ structure of their effective four-
fermion interaction was established experimentally. Among the proposals were
Glashow’s proposal in 1961 [6] of the SU(2)$\times$U(1) structure of the
electroweak sector of the Standard Model, in which he simply postulated masses
for such bosons without concerning himself with their origins, nor the
renormalizability of his model. Some years later, in 1967, Salam [7] and
Weinberg [8] revived his model with the suggestion that their masses might
arise from spontaneous symmetry breaking, using the mechanism that had been
proposed back in 1964 by Higgs [9], Englert and Brout [10], and Kibble [11].
They did not attempt to prove that the Standard Model was renormalizable,
though apparently Weinberg suspected that it might be, and set a student to
trying (unsuccessfully) to prove it. The numbers of citations of these papers
were minuscule until the papers of Tini and ’t Hooft showed how to renormalize
and obtain finite results from spontaneously-broken gauge theories.

##  3 The Higgs Boson

These papers triggered a tsunami of theoretical interest, joined by
experimental interest following the discovery of neutral currents in 1973 [12]
and charmonium in 1974 [13]. Mainstream efforts started targeting the
discovery of the massive electroweak gauge bosons, but that was not the
priority for Mary Gaillard, Dimitri Nanopoulos and myself. We reasoned that
the key element in the whole theoretical edifice of the Standard Model was the
Higgs-Englert-Brout mechanism for spontaneous gauge symmetry breaking, and the
existence of the physical scalar boson predicted by Higgs [9] (who had also
considered several of its physical properties [14]). This was why we set out
to write our paper on the phenomenological profile of the Higgs boson [15].
Because we were aware that we were out on what was considered by most senior
theorists in those days to be quite a hypothetical limb, we ended our paper by
saying (tongues somewhat in cheek) that “we [did] not want to encourage big
experimental searches for the Higgs boson”.

However, Tini was supportive of our efforts. I remember giving a talk on Higgs
phenomenology at a gauge theory workshop at the École Normale in Paris in
early 1976 about our paper. The reception was generally tepid, but Tini was
very positive about it. There had been only a handful of papers before ours on
signatures of the Higgs boson, and nobody had a clue what its mass might be.
Tini had already worked on this problem [16], arguing that a very small Higgs
mass would be incompatible with cosmology, and this problem continued to be
one of his principal theoretical interests in the following years [5]. In
particular, he was one of the first to point out that a massive Higgs boson
would necessarily be strongly-interacting, and to argue that this imposed a
qualitative upper limit on its mass of a few hundred GeV [17], at which stage
it would become strongly-interacting and some low-mass bound states might
emerge. 111The upper limit on the Higgs mass was also discussed in [18, 19].

The renormalizability of the Standard Model is a joint effort of all its
particles: remove any of them and uncontrollable infinities appear. This is
reflected in the growths of their quantum loop contributions to physical
quantities as their masses increase, which may be either quadratic or
logarithmic. Tini was a pioneer in exploring these renormalization effects,
and how they would affect physical observables. He considered these questions
in [17], showing that quadratic dependences on heavy-particle masses are the
general rule, but the one-loop effects of the Higgs boson in the Standard
Model increase only logarithmically with its mass, because of the screening
effects of a custodial symmetry in the Higgs sector (see [20] for a later
discussion). Tini went on to discuss the quadratic mass dependences of one-
loop corrections to vector boson masses in [21] imposed an upper limit of
several hundred GeV on mass differences within a fermion multiplet.

These pioneering papers were followed by a stream of calculations of one-loop
corrections to various specific electroweak processes, paying particular
attention to their sensitivities to the Higgs mass. These included the
violation of $\mu-e$ universality in lepton-hadron interactions [22] (very
topical at the moment! [23, 24]); the processes $e^{+}e^{-}\to\mu^{+}\mu^{-}$
[25] (with Giampiero Passarino) and $e^{+}e^{-}\to W^{+}W^{-}$ [26] (with M.
Lemoine); vector-boson masses [27]; and low-energy processes [28] (with Martin
Green). Somewhat later, Tini made a heroic calculation of two-loop corrections
to the ratio of $W$ and $Z$ masses (the $\rho$ parameter) [29] (with Jochum
van de Bij), showing that they are quadratically sensitive to the Higgs mass.
On this basis, Tini and Jochum argued that perturbation theory would breakdown
for a Higgs mass $>3$ TeV. These pioneering calculations played key roles in
the subsequent predictions of the top and Higgs masses on the basis of high-
precision LEP data, as discussed below.

Tini was also one of first to worry about the quadratic divergences in the
quantum corrections to the Higgs mass, considering the possibility of
cancellations between fermion and boson loops [30]. In particular, he
considered the possibility of cancelling the top quark contribution with those
of the massive vector and Higgs bosons in the Standard Model, and also
mentioned the possibility of supersymmetry. However, he was never strongly
enamoured of it, although it is the most systematic realization of his idea.
(It remains to be seen whether Nature likes it!)

In parallel to his theoretical work during this period, Tini was a member of
the CERN Scientific Policy Committee, where he was an enthusiastic advocate of
the construction of LEP. This project grew out of a paper written by Burt
Richter while he has on sabbatical at CERN in the academic year 1975/6 [31],
in which he analyzed the possible scaling up of circular $e^{+}e^{-}$
colliders from the few GeV of those operating at the time to a machine with
beams of energy $\sim 100$ GeV each. Mary Gaillard and I were tasked with
writing theoretical section of the first LEP physics study, which was
published in 1976 [32]. In addition to precise experiments at the $Z$ peak and
measurements of $W^{+}W^{-}$ production, we highlighted the importance of
searching for the Higgs boson, e.g., via its production in association with a
$Z$ boson. Tini supported strongly this physics programme, and pushed for LEP
to have the largest size compatible with CERN’s cramped geographical
surroundings, so as to reach the highest centre-of-mass energy possible. This
foresight also maximized the real estate available for the later construction
of the LHC, where the Higgs boson was finally discovered.

##  4 Radiative Corrections

During the 1980s, there was a sustained theoretical campaign to make and
refine calculations of quantum corrections to many physical quantities that
were to be measured at LEP. These made manifest the top and Higgs mass
dependences that had been foreshadowed by Tini in his pioneering calculations.
These theoretical contributions were gathered and reviewed in a series of CERN
reports [33, 34, 35, 36], where they were presented in an experimentalist-
friendly way.

Meanwhile, many low-energy experiments were providing a growing array of
constraints on the electroweak sector of the Standard Model, particularly on
the neutral-current interactions. Towards the end of the 1980s, they were
sufficiently precise to be sensitive to quantum loop corrections, and in 1987
it became possible [37, 38] to constrain the top quark mass through the
quadratic quantum effects that Tini had pointed out. We found an upper limit
on its mass similar to the value that was subsequently measured experimentally
[39]. We also pointed out that it would also be possible to establish a lower
bound on $m_{t}$ once the $Z$ mass was measured accurately [40], which was
done by the CDF experiment at Fermilab, the SLC and LEP in 1989. However,
these data were not yet accurate enough to provide any useful information
about the mass of the Higgs boson [41], whose effects were suppressed by
Tini’s screening theorem.

During the following few years, LEP (and SLC) produced many more high-
precision electroweak measurements, and the net around the top quark mass drew
tighter, enabling it to be estimated with $\sim 10\%$ accuracy [42]. The
plethora of LEP measurements also provided the first indications on the
possible mass of the Higgs boson, indicating that it probably weighed $<300$
GeV [43]. The first direct evidence for the top quark quark came in 1994 [44],
and in 1995 it became strong enough to claim discovery [45]. The measured mass
was quite consistent with the indirect estimate based on Tini’s loop
corrections. Moreover, the combination of the direct measurement with the
high-precision electroweak data made it possible to refine the estimate of the
Higgs mass [46].

The Nobel Physics Prize was awarded to Tini and Gerard ’t Hooft in 1999. To
quote the Nobel citation: “They showed that the non-Abelian quantum field
theories could make sense and provided a method for computing quantum
corrections in these theories … the mass of the top quark could be predicted,
using high precision data from the accelerator LEP … several years before it
was discovered”.

The story did not end there. The direct search for the Higgs boson had evolved
from a minority interest [15], supported enthusiastically by Tini, into a
central theme of the LEP experimental programme [32, 35, 36]. However,
searches at LEP were ultimately unsuccessful, constraining its mass to be
$>114$ GeV [47]. The torch was then passed to Fermilab, where unsuccessful
searches excluded a range of masses around 160 GeV [48]. A global analysis of
the combined direct and indirect information of the mass of the Higgs boson in
2011 quoted the 68% mass range $m_{H}=120^{+12}_{-5}$ GeV [49]. Finally, in
2012 the Higgs boson was discovered, with a mass of 125 GeV and an uncertainty
$<200$ MeV [50]. This provided the final experimental vindication of Tini’s
proof of the renormalizability of spontaneously-broken gauge theory, his
obsession with the Higgs boson, and his calculations of quantum loop
corrections.

##  5 What Next?

And the story continues. On the one hand, high-energy measurements at the LHC
continue obstinately to agree with the Standard Model, in particular those of
the production mechanisms and decays of the Higgs boson. On the other hand,
every term in its effective Lagrangian:

${\cal L}\;\ni\;yH{\bar{\psi}}\psi+\mu^{2}|H|^{2}-\lambda|H|^{4}-V_{0}+\dots$
(1)

poses a theoretical mystery. The pattern of Yukawa couplings $y$ is the
flavour problem of the Standard Model, to which recent LHCb measurements [23]
have added. While the Higgs can be responsible for the masses and mixings of
fundamental matter fermions, it does not explain their magnitudes. The
magnitude of the mass term $\mu$ raises the notorious naturalness/hierarchy
problem: why is it not of the same order of magnitude as the quantum loop
corrections that Tini strove to cancel? The magnitude of the quartic coupling
$\lambda$ corresponding to the measured values of the Higgs vacuum expectation
value (vev) and $m_{h}$ is so small that it is probably driven negative at
high renormalization scales by radiative corrections due to the top quark
[51], in which case the present electroweak vacuum would not be stable, and
how the Higgs evolved to reach its observed vev would pose a cosmological
puzzle [52]. The constant term $V_{0}$ corresponds to the comsological
constant (a.k.a. dark energy). It is measured to be ${\cal O}(\rm meV)^{3}$,
which is far smaller, e.g., than the difference in the vacuum energy between
the local maximum of the Higgs potential at the origin and its present-day
value after sinking into its vev [53], or the shift induced by non-
perturbative QCD effects. 222I remember having inclonclusive discussions about
the magnitude of the cosmological constant with Tini in early 1976. To my mind
the problem is not that it exists, but that it is so small. Finally, there
could be additional terms represented by the $\dots$, of higher order in the
Higgs and other Standard Model fields. These would not be renormalizable in
the same sense as the Standard Model, as proved by Tini and ’t Hooft, but they
could appear as terms in a low-energy effective field theory when massive
particles in some renormalizable extension of the Standard Model are
integrated out.

The possibilities for such terms are taken into account systematically by the
Standard Model Effective Field Theory (SMEFT) [54], which includes all
operators of dimensions $d\geq 5$ that are invariant under the Standard Model
gauge group and composed of Standard Model fields with their conventional
quantum numbers:

${\cal L}_{\rm SMEFT}\;=\;\sum_{i:d\geq 5}\frac{C_{i}}{\Lambda_{i}^{d-4}}{\cal
O}_{i}\,,$ (2)

where the $\Lambda_{i}$ are mass scales characteristic of the new physics
generating the operators ${\cal O}_{i}$ with coefficients $C_{i}$. The
operators of dimension 5 in the SMEFT can generate neutrino masses, but are
not of interest to us here. There are in general 2499 operators of dimension
6, many of which could in principle contribute to cross sections measured at
the LHC and are constrained by LHC measurements, as well as measurements of
Higgs decays and high-precision electroweak measurements at LEP and elsewhere.
If we could discover SMEFT interactions with dimension $\geq 6$ and
disentangle their structure, we would have clues to physics beyond the
Standard Model, just as the $V-A$ structure of the weak interactions led us
towards Tini’s giant footsteps.

We have recently performed a global analysis of the constraints on dimension-6
operators provided by all the available data, making simplifying assumptions
about their flavour structure, specifically that operators including fermions
have flavour-universal coefficients, apart possibly from those involving the
top quark [55]. Switching on individually the 34 dimension-6 operators in a
top-specific scenario with SU(2)${}^{2}\times$SU(3)3 symmetry, we see in Fig.
1 that the operator scales range between a few hundred GeV and $\sim 20$ TeV
if their coefficients $C_{i}=1$.

Figure 1:  Results from a global fit [55] to the electroweak, diboson, Higgs
and top data in a top-specific scenario with SU(2)${}^{2}\times$SU(3)3
symmetry [55]. The two panels show results for fits to 34 individual
dimension-6 operators, showing the 95% CL ranges for the operator coefficients
$C_{i}$ normalising all the new physics scales $\Lambda_{i}$ to 1 TeV, and the
ranges for the scales $\Lambda_{i}$ for different universal values of the
$C_{i}$.

Searching for possible indications of new physics beyond the Standard Model,
we have considered all the single-field extensions of the Standard Model
catalogued in [56], and the corresponding mass limits (in TeV) at the 95% CL
and upper limits on couplings assuming masses of 1 TeV are shown in Fig. 2. As
shown also, there are no significant pulls in the fit that might indicate the
presence of some new physics.

Figure 2: The yellow bars show the mass limits (in TeV) at the 95% CL for the
single-field extensions of the Standard Model catalogued in [55], setting the
corresponding couplings to unity. The grey boxes contain the coupling limits
obtained when setting the mass to 1 TeV, and the numbers in light blue are the
pulls in the fit that exceed 1-$\sigma$.

These results indicate that there is a non-trivial hierarchy of mass scales
between the particles of the Standard Model and whatever might appear at
higher energies. As mentioned earlier, Tini was concerned about the quadratic
divergence in the mass of the Higgs boson that appears in the Standard, and
speculated how to cancel it. One possibility that has attracted much attention
over the years is supersymmetry (though Tini did not pursue it). Since the
Standard Model is renormalizable by itself, and supersymmetry is an add-on,
its contributions to low-energy measurements via loop diagrams do not grow
with the supersymmetry mass scale, and are difficult to constrain via
precision measurements. Indeed, global analyses of LEP and LHC data using
either exact calculations of supersymmetric radiative corrections [57] or the
SMEFT approach are quite consistent with the lightest supersymmetric
particles, such as the lighter stop squark, weighing just a few hundred GeV
[55], as seen in Fig. 3.

Figure 3: Limits found in the global fit [55] on the mass of the lighter stop
squark, $m_{\tilde{t}}$, and the stop mixing parameter,
$\frac{X_{t}}{m_{\tilde{t}}}$, for one value of the ratio of supersymmetric
Higgs vacuum expectation values, $\tan\beta=20$. Figure 4:  The distributions
of pulls found [55] in global fits to 2 (upper left), 3 (upper right), 4
(lower left) and 5 (lower right) parameter subsets of operators, which have
been split into three categories: those including only operators that affect
$t\bar{t}$ production (blue), those without any operators that affect
$t\bar{t}$ production (orange), and the rest (green). The dashed vertical
lines mark the ranges for the pull distributions expected at the 95%
confidence level.

In general, supersymmetric particles may contribute to several SMEFT
coefficients, and the same holds for many other possible extensions of the
Standard Model. Accordingly, we performed [55] a general survey of all
possible extensions that contribute to 2, 3, 4 or 5 SMEFT coefficients, as
seen in Fig. 4, which shows the distributions of pulls found in three
categories of SMEFT operators: those that include only operators that affect
$t\bar{t}$ production (blue), those that do not include operators that affect
$t\bar{t}$ production (orange), and the rest (green). So far, we see no
significant evidence for possible physics beyond the Standard Model.

##  6 Reflections

What would Tini make of the current situation in particle physics? On the one
hand, he would be justifiably proud of the robust successes of the Standard
Model that did so much to place on a firm footing. On the other hand, some of
the problems that concerned him remain unresolved, notably the quadratic
instability in the mass of the Higgs boson and the magnitude of the dark
energy (cosmological constant). What would he think of the experimental
anomalies involving muons that have been strengthened recently [23, 24]. My
suspicion is that he would not yet be convinced, if only because they do not
have any obvious bearing on his theoretical preoccupations. My hunch is that
Tini would continue to focus his energies on the manifold problems associated
with the Higgs boson that he did so much to promote to theoretical
respectability and experimental reality.

## Acknowledgements

This work was supported in part by STFC grant ST/T000759/1 and by Estonian
Research Council grant MOBTT5.

## References

  * [1] M. J. G. Veltman, Nucl. Phys. B 7 (1968), 637-650 doi:10.1016/0550-3213(68) 90197-1.
  * [2] M. J. G. Veltman, Nucl. Phys. B 21 (1970), 288-302 doi:10.1016/0550-3213(70) 90521-3.
  * [3] G. ’t Hooft, Nucl. Phys. B 33 (1971), 173-199 doi:10.1016/0550-3213(71) 90395-6; Nucl. Phys. B 35 (1971), 167-188 doi:10.1016/0550-3213(71)90139-8.
  * [4] G. ’t Hooft and M. J. G. Veltman, Nucl. Phys. B 44 (1972), 189-213 doi:10.1016/0550-3213(72)90279-9.
  * [5] M. Veltman, Scientific American (1986),
https://www.scientificamerican.com/article/the-higgs-boson/.

  * [6] S. L. Glashow, Nucl. Phys. 22 (1961), 579-588 doi:10.1016/0029-5582(61) 90469-2.
  * [7] A. Salam, Conf. Proc. C 680519 (1968), 367-377 doi:10.1142/ 9789812795915_0034.
  * [8] S. Weinberg, Phys. Rev. Lett. 19 (1967), 1264-1266 doi:10.1103/ PhysRevLett.19.1264
  * [9] P. W. Higgs, Phys. Lett. 12 (1964), 132-133 doi:10.1016/0031-9163(64)91136-9; Phys. Rev. Lett. 13 (1964), 508-509 doi:10.1103/PhysRevLett.13.508.
  * [10] F. Englert and R. Brout, Phys. Rev. Lett. 13 (1964), 321-323 doi:10.1103/PhysRevLett.13.321.
  * [11] T. W. B. Kibble, Phys. Rev. 155 (1967), 1554-1561 doi:10.1103/ PhysRev.155.1554.
  * [12] F. J. Hasert et al. [Gargamelle Neutrino Collaboration], Phys. Lett. B 46 (1973), 121-124 doi:10.1016/0370-2693(73)90494-2; Phys. Lett. B 46 (1973), 138-140 doi:10.1016/0370-2693(73)90499-1.
  * [13] J. E. Augustin et al., Phys. Rev. Lett. 33 (1974), 1406-1408 doi:10.1103/PhysRevLett.33.1406; J. J. Aubert et al., Phys. Rev. Lett. 33 (1974), 1404-1406 doi:10.1103/PhysRevLett.33.1404.
  * [14] P. W. Higgs, Phys. Rev. 145 (1966), 1156-1163 doi:10.1103/PhysRev.145.1156.
  * [15] J. R. Ellis, M. K. Gaillard and D. V. Nanopoulos, Nucl. Phys. B 106 (1976), 292 doi:10.1016/0550-3213(76)90382-5
  * [16] M. J. G. Veltman, Phys. Rev. Lett. 34 (1975), 777 doi:10.1103/ PhysRevLett.34.777.
  * [17] M. J. G. Veltman, Acta Phys. Polon. B 8 (1977), 475.
  * [18] C. E. Vayonakis, Print-77-0432 (ATHENS).
  * [19] B. W. Lee, C. Quigg and H. B. Thacker, Phys. Rev. Lett. 38 (1977), 883-885 doi:10.1103/PhysRevLett.38.883.
  * [20] M. J. G. Veltman, Acta Phys. Polon. B 25 (1994), 1627-1636.
  * [21] M. J. G. Veltman, Nucl. Phys. B 123 (1977), 89-99 doi:10.1016/0550-3213(77)90342-X.
  * [22] M. J. G. Veltman, Phys. Lett. B 70 (1977), 253-254 doi:10.1016/0370-2693(77)90532-9.
  * [23] R. Aaij et al. [LHCb Collaboration], arXiv:2103.11769 [hep-ex].
  * [24] B. Abi et al. [Muon g-2 Collaboration], Phys. Rev. Lett. 126 (2021) no.14, 141801 doi:10.1103/PhysRevLett.126.141801 [arXiv:2104.03281 [hep-ex]].
  * [25] G. Passarino and M. J. G. Veltman, Nucl. Phys. B 160 (1979), 151-207 doi:10.1016/0550-3213(79)90234-7.
  * [26] M. Lemoine and M. J. G. Veltman, Nucl. Phys. B 164 (1980), 445-483 doi:10.1016/0550-3213(80)90521-0.
  * [27] M. J. G. Veltman, Phys. Lett. B 91 (1980), 95-98 doi:10.1016/0370-2693(80)90669-3.
  * [28] M. Green and M. J. G. Veltman, Nucl. Phys. B 169 (1980), 137-164 [erratum: Nucl. Phys. B 175 (1980), 547] doi:10.1016/0550-3213(80)90257-6.
  * [29] J. van der Bij and M. J. G. Veltman, Nucl. Phys. B 231 (1984), 205-234 doi:10.1016/0550-3213(84)90284-0.
  * [30] M. J. G. Veltman, Acta Phys. Polon. B 12 (1981), 437.
  * [31] B. Richter, Nucl. Instrum. Meth. 136 (1976), 47-60 doi:10.1016/0029-554X(76)90396-7.
  * [32] L. Camilleri et al., CERN Report 76-018, doi:10.5170/CERN-1976-018.
  * [33] G. Altarelli et al., in Physics at LEP, CERN Report 86-02 (1986), eds. J. Ellis and R. D. Peccei, Vol. 1,
http://cdsweb.cern.ch/record/166310/files/CERN-86-02-V-1.pdf, pp 203-296.

  * [34] F. Dydak, in ECFA Workshop on LEP200, CERN Report 87-08, eds. A. Böhm and W. Hoogland,
http://cdsweb.cern.ch/record/180188/files/CERN-87-08-V-1.pdf, pp 157-201.

  * [35] Z Physics at LEP, CERN Report 89-08, eds. G. Altarelli, R. Kleiss and C. Verzegnassi,
http://cdsweb.cern.ch/record/116932/files/CERN-89-08-V-1.pdf.

  * [36] Physics at LEP2, CERN Report 96-01, G. Altarelli, T. Sjöstrand and F. Zwirner,
http://cdsweb.cern.ch/record/300671/files/CERN-96-01-V1.pdf.

  * [37] U. Amaldi, A. Bohm, L. S. Durkin, P. Langacker, A. K. Mann, W. J. Marciano, A. Sirlin and H. H. Williams, Phys. Rev. D 36 (1987), 1385 doi:10.1103/PhysRevD.36.1385.
  * [38] G. Costa, J. R. Ellis, G. L. Fogli, D. V. Nanopoulos and F. Zwirner, Nucl. Phys. B 297 (1988), 244-286 doi:10.1016/0550-3213(88)90020-X.
  * [39] J. R. Ellis and G. L. Fogli, Phys. Lett. B 213 (1988), 526-530 doi:10.1016/0370-2693(88)91304-4.
  * [40] J. R. Ellis and G. L. Fogli, Phys. Lett. B 231 (1989), 189-194 doi:10.1016/0370-2693(89)90138-X.
  * [41] J. R. Ellis and G. L. Fogli, Phys. Lett. B 249 (1990), 543-550 doi:10.1016/0370-2693(90)91033-8.
  * [42] LEP Electroweak Working Group, http://lepewwg.web.cern.ch/LEPEWWG/; S. Schael et al. [ALEPH, DELPHI, L3, OPAL and SLD Collaborations, LEP Electroweak Working Group, SLD Electroweak Group and SLD Heavy Flavour Group], Phys. Rept. 427 (2006), 257-454 doi:10.1016/j.physrep.2005.12.006 [arXiv:hep-ex/0509008 [hep-ex]].
  * [43] J. R. Ellis, G. L. Fogli and E. Lisi, Phys. Lett. B 318 (1993), 148-154 doi:10.1016/0370-2693(93)91798-R.
  * [44] F. Abe et al. [CDF Collaboration], Phys. Rev. D 50 (1994), 2966-3026 doi:10.1103/PhysRevD.50.2966.
  * [45] F. Abe et al. [CDF Collaboration], Phys. Rev. Lett. 74 (1995), 2626-2631 doi:10.1103/PhysRevLett.74.2626 [arXiv:hep-ex/9503002 [hep-ex]]; S. Abachi et al. [D0 Collaboration], Phys. Rev. Lett. 74 (1995), 2632-2637 doi:10.1103/PhysRevLett.74.2632 [arXiv:hep-ex/9503003 [hep-ex]].
  * [46] J. R. Ellis, G. L. Fogli and E. Lisi, Phys. Lett. B 389 (1996), 321-326 doi:10.1016/S0370-2693(96)01283-X [arXiv:hep-ph/9608329 [hep-ph]].
  * [47] R. Barate et al. [LEP Working Group for Higgs boson searches, ALEPH, DELPHI, L3 and OPAL Collaborations], Phys. Lett. B 565 (2003), 61-75 doi:10.1016/S0370-2693(03)00614-2 [arXiv:hep-ex/0306033 [hep-ex]].
  * [48] CDF and D0 Collaborations, arXiv:1207.0449 [hep-ex].
  * [49] M. Baak, M. Goebel, J. Haller, A. Hoecker, D. Ludwig, K. Moenig, M. Schott and J. Stelzer, Eur. Phys. J. C 72 (2012), 2003 doi:10.1140/epjc/s10052-012-2003-4 [arXiv:1107.0975 [hep-ph]].
  * [50] G. Aad et al. [ATLAS Collaboration], Phys. Lett. B 716 (2012), 1-29 doi:10.1016/j.physletb.2012.08.020 [arXiv:1207.7214 [hep-ex]]; S. Chatrchyan et al. [CMS Collaboration], Phys. Lett. B 716 (2012), 30-61 doi:10.1016/j.physletb.2012.08.021 [arXiv:1207.7235 [hep-ex]].
  * [51] D. Buttazzo, G. Degrassi, P. P. Giardino, G. F. Giudice, F. Sala, A. Salvio and A. Strumia, JHEP 12 (2013), 089 doi:10.1007/JHEP12(2013)089 [arXiv:1307.3536 [hep-ph]].
  * [52] See, e.g., M. Fairbairn and R. Hogan, Phys. Rev. Lett. 112 (2014), 201801 doi:10.1103/PhysRevLett.112.201801 [arXiv:1403.6786 [hep-ph]].
  * [53] A. D. Linde, JETP Lett. 19 (1974), 183.
  * [54] W. Buchmuller and D. Wyler, Nucl. Phys. B 268 (1986), 621-653 doi:10.1016/0550-3213(86)90262-2.
  * [55] J. Ellis, M. Madigan, K. Mimasu, V. Sanz and T. You, arXiv:2012.02779 [hep-ph].
  * [56] J. de Blas, J. C. Criado, M. Perez-Victoria and J. Santiago, JHEP 03 (2018), 109 doi:10.1007/JHEP03(2018)109 [arXiv:1711.10391 [hep-ph]].
  * [57] E. Bagnaschi, et al. Eur. Phys. J. C 78 (2018) no.3, 256 doi:10.1140/ epjc/s10052-018-5697-0 [arXiv:1710.11091 [hep-ph]].


# Neural Relighting with Subsurface Scattering by Learning the Radiance
Transfer Gradient

Shizhan Zhu  University of California, BerkeleyUSA ,  Shunsuke Saito  Meta
Reality Labs ResearchUSA ,  Aljaž Božič  Meta Reality Labs ResearchUSA ,
Carlos Aliaga  Meta Reality Labs ResearchUSA ,  Trevor Darrell  University of
California, BerkeleyUSA and  Christoph Lassner  Meta Reality Labs ResearchUSA

(2021)

###### Abstract.

Reconstructing and relighting objects and scenes under varying lighting
conditions is challenging: existing neural rendering methods often cannot
handle the complex interactions between materials and light. Incorporating
pre-computed radiance transfer techniques enables global illumination, but
still struggles with materials with subsurface scattering effects. We propose
a novel framework for learning the radiance transfer field via volume
rendering and utilizing various appearance cues to refine geometry end-to-end.
This framework extends relighting and reconstruction capabilities to handle a
wider range of materials in a data-driven fashion. The resulting models
produce plausible rendering results in existing and novel conditions. We will
release our code and a novel light stage dataset of objects with subsurface
scattering effects publicly available.

††copyright: rightsretained††journalyear: 2020††doi: 0††journal:
TOG††journalvolume: 0††journalnumber: 0††article: 0††publicationmonth: 0
Figure 1. Our approach reconstructs objects with significant subsurface
scattering effects with high fidelity and inserts models into arbitrary
environments for relighting. It is fully data-driven and does not assume a
particular material representation (such as BRDF or BSSRDF), and can
faithfully render high quality appearance under varying lighting conditions
and view points. Please see our supplementary video for comprehensive
visualizations and comparisons.

##  1\. Introduction

The ability to relight objects and scenes under varying lighting conditions is
crucial in many areas, such as virtual reality, gaming, visual effects, and
architecture. It enables artists, designers, and engineers to experiment with
many lighting setups without having to physically recreate a scene. It also
allows for the creation of more realistic and immersive experiences by
accurately simulating the lighting conditions in a virtual environment.

However, relighting remains a challenging task due to the complex interaction
between the light and the materials in a scene. Traditional approaches have
sought to decompose rendering into geometry, material, and lighting to
simplify the problem. For example, opaque materials are represented in
Physically Based Rendering (PBR) by the Bidirectional Reflectance Distribution
Function (BRDF), which describes how light interacts with a material’s surface
[Zhang et al., 2021a; Munkberg et al., 2022; Zhang et al., 2021b; Boss et al.,
2021b, a]. Similarly, many relighting methods such as [Srinivasan et al.,
2021; Chen et al., 2020] rely on decomposing light into its components, such
as direct lighting and indirect lighting, to allow for more fine-grained
control.

While these approaches have been successful in many cases, they are limited in
their ability to handle objects with translucency or subsurface scattering
(SSS). This is because these materials are not well approximated by a simple
BRDF function and require more complex models, such as the Bidirectional
Surface Scattering Reflectance Distribution Function (BSSRDF). However,
modeling BSSRDF are computationally expensive and slow to evaluate, neglecting
textures beneath the surface (Fig. 3), limiting their practicality for inverse
rendering with complex geometry.

Recent works on neural radiance transfer fields [Lyu et al., 2022] have
incorporated the idea of pre-computed radiance transfer (PRT) into the neural
radiance fields (NeRF) literature, providing promising results for relighting
with global illumination effects. However, these approaches rely on a pre-
estimated surface, which is nontrivial to reconstruct for objects with SSS or
with translucency. Additionally, the separated geometry and appearance
optimization is suboptimal, leading to artifacts and unrealistic results.

In this work, we propose a novel framework for relighting that incorporates
the optimization of shape and radiance transfer using a volume rendering
approach (Fig. 1). Our framework extends the relighting capability to a wider
range of materials, including translucent objects with strong SSS effects and
textures beneath the surface. Specifically, we use a volume rendering approach
to estimate the transfer field and utilize appearance cues to refine the
geometry in an end-to-end fashion.

To evaluate our approach, we have recorded real-world objects featuring
subsurface scattering effects in a light stage and show that our method
produces high quality visual results in recorded and novel lighting
conditions. Quantitatively, our approach compares favorably with the current
state of the art with a 5 points higher PSNR on average across three datasets.

In summary, we propose a novel framework for neural radiance transfer fields
using volume rendering, optimizing appearance and geometry in an end-to-end
fashion, which to the best of our knowledge has not been achieved before for
optically-thick translucent materials. Additionally, we collected and will
release a dataset of objects that exhibit prominent subsurface scattering
effects for training and evaluation purposes. These objects have been recorded
with high fidelity featuring rich, high frequency spatially-varying details,
resulting in 15TiB of data, which is 3000 times larger and notably more
detailed than the current highest quality data for research in this area [Deng
et al., 2022].

##  2\. Related Work

Figure 2. Despite presented with significant subsurface scattering and
translucency in the scene, our approach provides the highest geometric
reconstruction quality compared to other approaches (NVMC [Hasselgren et al.,
2022]; NRTF [Lyu et al., 2022] via Neus [Wang et al., 2021]). For our
approach, we show the extracted mesh using marching cubes from the density in
the $512^{3}$ resolution. The high quality geometry is one of the key
advantages of our method.

Relighting and Surface Representations. The problem of relighting an object or
a scene under novel lighting conditions has been extensively studied. Usually,
the problem is tackled via decomposing the appearance into the lighting and
the surface material properties. Early works estimate material given known
illumination such as a single light source [Yu et al., 1999; Debevec et al.,
2000] or spherical gradient illumination [Fyffe, 2009; Guo et al., 2019] with
known geometry. [Zhang et al., 2021b] directly model light transports with
known illuminations and know geometry. More recently, neural scene
representations [Xie et al., 2022] and differentiable rendering [Nimier-David
et al., 2019] allow us to jointly optimize BRDF and geometry. Some methods
apply inverse rendering using implicit surface to obtain materials [Luan et
al., 2021; Zhang et al., 2022; Munkberg et al., 2022]. Other approaches
utilize volumetric representations with opacity fields [Bi et al., 2020b, a;
Zhang et al., 2021a, b; Boss et al., 2021a]. The required illumination setup
can be reduced to a co-located light [Bi et al., 2020b, a], and unknown
illuminations [Luan et al., 2021; Zhang et al., 2022, 2021a, 2021b; Boss et
al., 2021a]. To reduce the ambiguity in BRDF, the aforementioned methods use
parametric BRDFs such as a microfacet model [Walter et al., 2007; Burley and
Studios, 2012]. However, these parametric models do not support subsurface
scattering as they only consider reflectance. In contrast, our approach deals
with global light transport effects including subsurface scattering.

Figure 3. For relighting objects with subsurface scattering effects (e.g., the
translucent soap shown in this figure), the BRDF-based approach [Zhang et al.,
2022] renders the object with full opacity when the light comes from the
opposite directions, while the BSSRDF-based approach [Deng et al., 2022]
cannot capture the texture details and structures beneath the surface
(highlighted in the orange squares). In contrast, our approach can faithfully
render the right opacity of the object and retain appearance even given the
subsurface structure of the drill inside the candle (highlighted in the blue
squares).

Subsurface Scattering. Subsurface scattering refers to light transport inside
of a solid substance. It happens with some particular types of materials (such
as wax, jade, tiny furs or various fruits), and is quite common in the real
world. Since the light might leave the object surface at a different point
from where it enters, surface representations (e.g. various BRDFs) cannot
represent this type of light transmission. While subsurface scattering can be
accurately modeled by volumetric path tracing algorithms [Novák et al., 2018],
their run time is typically prohibitive in certain applications, despite
efforts to accelerate brute-force computation, e.g. through a shape adaptive
learned SSS model [Vicini et al., 2019] that relies on a conditional
variational auto encoder that learns to sample from a distribution of exit
points on the object surface. Some other works have focused on estimating the
scattering parameters from images of translucent objects. Inverse Transport
Networks [Che et al., 2020] infer the optical properties that control
subsurface scattering inside translucent objects of any shape under any
illumination. They rely on an encoder decoder where the latter is replaced by
a physically-based differentiable path tracer, trained with synthetic images.
Prior to that, another approach based on stochastic gradient descent, combined
with Monte Carlo rendering and a material dictionary was capable of estimating
the scattering materials, inverting the radiative transfer parameters
[Gkioulekas et al., 2013]. Nevertheless, since volumetric path tracing can be
costly, applying a BSSRDF can be a faster alternative [Deng et al., 2022].
Compared to BRDF-based representations, a higher dimension of inputs (usually
6D for homogeneous materials) is fed to query the outgoing radiance. A
relighting algorithm can thus seek to optimize the BSSRDF function with the
inverse rendering process so that the resulting material can be relit in
conventional rendering engines. Our work follows a different path - we learn
our relighting model in a fully data driven fashion, and learn the cached
outgoing radiance for each point using a deep neural network, where we bypass
the expensive BSSRDF computation in our optimization iteration.

Figure 4. Volume rendering leads to cleaner surface reconstructions and higher
rendering quality compared to NRTF [Lyu et al., 2022].

Neural Radiance Fields and Precomputed Radiance Transfer. Neural Radiance
Fields (NeRF) [Mildenhall et al., 2020; Barron et al., 2021, 2022] optimize a
parameterized volume rendering model from multiple views of the scene so that
at test time, novel views can be synthesized from the learned model. Despite
its superior rendering quality, NeRF bakes all the lighting and reflective
surface information into the RGBs without modeling the interaction of the
light and the material. Recent studies [Lyu et al., 2022] have shown promising
results for relightable models via incorporating the idea of “precomputed
radiance transfer” (PRT) [Sloan et al., 2002] from the real time rendering
community. Instead of precomputing and caching the intermediate representation
per location, they seek to optimize a cached intermediate representation in
the reconstruction process. Notably, [Lyu et al., 2022] relies on a fairly
accurate pre-computed surface [Wang et al., 2021], and keeps the lighting
appearance optimization separate from the geometry acquisition process.
Focused on synthetic images with varying but known illumination, a NeRF
extension [Zheng et al., 2021] was presented to reconstruct participating
media with full global illumination effects, achieving good results on
synthetic data. In contrast, our novel volume rendering framework not only
enables optimizing the geometry details with appearance cues, but also works
on scenes with partially opaque mass (e.g. thin rope or furs) and demonstrates
high quality results on synthetic and real data. It is worth mentioning that a
recent concurrent work [Yu et al., 2023] also addressed relighting with
translucent objects using scattering functions. In addition to distant point
lights, our approach efficiently relights the captured scenes with environment
maps with the help of the Median-cut algorithm. Further, we will release high-
resolution and large scale light stage dataset with rich lighting effects,
such as translucency coupled with specular highlights and translucent
shadowing, facilitating future research.

##  3\. Methodology

###  3.1. Notation

Our goal is to optimize a relightable neural model from a collection of photos
of the object, captured from different camera view points and under varying
lighting conditions, that is able to accurately represent strong subsurface
scattering effects. Our input includes the set of the input images
$\mathcal{I}=\\{\mathbf{I}_{c,l}\\}$, where
$\mathbf{I}_{c,l}\in(\mathbb{R}^{+})^{M\times N\times 3}$ are high dynamic
range (HDR) images, and $c$ and $l$ represent the camera viewpoint and
lighting condition, respectively. We assume the camera poses are known (e.g.,
computed using photogrammetry software such as Agisoft Metashape), and denote
them as $\mathcal{C}=\\{\mathbf{K}_{c},\mathbf{R}_{c},\mathbf{t}_{c}\\}$
(camera intrinsic parameters, rotations and translations, respectively). We
capture one-light-at-a-time (OLAT) images for training, and denote an OLAT
lighting condition as $\mathcal{L}=\\{\omega_{l}\\}$, where
$\omega_{l}\in\mathbb{R}^{3}$ is the $\ell_{2}$ normalized vector representing
the incident point light direction relative to the scene center. Since our
data capture system uses white light, we parameterize the light using a single
channel throughout this paper. During testing, we apply an environment map
(_envmap_) $\mathbf{E}_{l}\in\mathbb{R}^{M_{E}\times N_{E}}$, where each pixel
of the envmap can be considered as one light source.

We want our relightable model to render the scene under varying _unseen_
viewpoints
($\\{\mathbf{K}_{\text{query}},\mathbf{R}_{\text{query}},\mathbf{t}_{\text{query}}\\}$)
and lighting conditions ($\omega_{\text{query}}$ or
$\mathbf{E}_{\text{query}}$). Our framework optimizes the geometry as well as
the lighting- and viewpoint-varying appearance of the scene in an end-to-end
fashion. More precisely, we use the function $f_{\Theta}(\cdot)$ to denote our
model (parameterized by $\Theta$), and denote our model prediction as
$\hat{\mathbf{I}}(u,v;\omega\text{~{}or~{}}\mathbf{E})=f_{\Theta}(\mathbf{r};\omega\text{~{}or~{}}\mathbf{E})\in(\mathbb{R}^{+})^{3}$,
where $\mathbf{r}$ represents a pixel ray in the space, and $(u,v)$ represents
its related pixel coordinates on the image plane under the given camera pose
$\\{\mathbf{K},\mathbf{R},\mathbf{t}\\}$. We provide an overview of our
approach in Sec. 3.2, and provide details of our volume rendering scheme as
well as model details in Sec. 3.3 and Sec. 3.4.

###  3.2. Method Overview

Figure 5. Illustration of the proposed relighting framework. We devise two
MLPs to predict the gradient of the transfer vector for accumulating the HDR
value of each ray. See Sec. 3 for details.

We devise a volume rendering based neural relightable model that is optimized
directly from the image collections of varying camera views and lighting
conditions (Fig. 5). The core of our learning framework consists of a volume
renderer enabling an end-to-end optimization (Sec. 3.3) and the density-based
neural transfer field networks (Sec. 3.4). There are several key differences
compared to the existing (neural) relighting approaches. On one hand, unlike
[Lyu et al., 2022], our model can be trained from scratch, with no dependency
on known estimated surface or other explicit geometry cues whose geometric
details are difficult to obtain especially for materials with strong
subsurface scattering effects (Fig. 2). Furthermore, training images captured
under varying lighting conditions contain rich geometric cues via local micro
shadowing or micro reflections, where a direct geometric optimization via an
appearance loss is deemed necessary. On the other hand, thanks to our fully
data-driven learning scheme, our model does not make any explicit assumptions
about material (such as specifying a varying BRDF or BSSRDF) [Zhang et al.,
2022; Deng et al., 2022], making it applicable to a wide range of materials,
enabling global illuminations and subsurface scattering effects.

###  3.3. Volume Integration of the Transfer Gradient

The color of each pixel ray is computed using volume rendering. We denote the
points sampled along the ray $\mathbf{r}$ as
$\\{\mathbf{x}_{1},\mathbf{x}_{2},...,\mathbf{x}_{N}\\}$, where $N$ is the
total number of points. The model predicts
$\hat{\sigma}(\mathbf{x}_{i})\in\mathbb{R}^{+}$ and
$\hat{\mathbf{h}}(\mathbf{x}_{i};\mathbf{r};\omega)\in(\mathbb{R}^{+})^{3}$
for every sample point, representing the density and the gradient of the pre-
computed transfer vector, respectively. It is worth pointing out that instead
of predicting the transfer vector directly as in [Lyu et al., 2022], we
predict the transfer vector gradient prediction, which represents the HDR
contribution of a particular segment along a particular light transmission
direction. It is clear that no HDR delta would incur at a density-free
location, and among the non-zero density locations, the HDR contribution at a
segment can only be non-negative if a location is visible, i.e. when its
volume accumulation weight ($\hat{w}(\mathbf{x})$ in Eq. 1) is positive. This
intuition aligns well with our volume accumulation and rendering scheme. We
follow the volume integration from [Mildenhall et al., 2020] and obtain the
accumulated transfer vector prediction as:

(1) $\displaystyle\hat{\mathbf{I}}(u,v;\omega)$
$\displaystyle=\sum_{i=1}^{N}\hat{w}(\mathbf{x}_{i})\hat{\mathbf{h}}(\mathbf{x}_{i};\mathbf{r};\omega)$
$\displaystyle\text{where~{}~{}~{}~{}~{}~{}~{}~{}}\hat{w}(\mathbf{x}_{i})$
$\displaystyle=\hat{T}_{i}(1-\exp(-\hat{\sigma}(\mathbf{x}_{i})\delta_{i}))$
$\displaystyle\hat{T}_{i}$
$\displaystyle=\exp(-\sum_{j=1}^{i-1}\hat{\sigma}(\mathbf{x}_{j})\delta_{j})$
$\displaystyle\delta_{i}$ $\displaystyle=t_{i+1}-t_{i}.$

Our volume rendering scheme demonstrates several key benefits over a surface
representation [Lyu et al., 2022] (Fig. 4). First and foremost, obtaining a
fairly accurate pre-estimated surface for materials featuring subsurface
scattering with detailed geometry is non-trivial. Our model bypasses the
difficulties of pre-estimating the surface geometry by applying volume
rendering and optimizing the surface density together with appearance. In this
case, all local shadowing and reflection effects captured under different
lighting conditions are taken into account for geometry estimation, providing
stronger cues compared to surface estimation under a single lighting
condition. Second, volume rendering enables accurate appearance modeling of
semi-opaque materials (e.g. fur) with their subsurface scattering effects,
which cannot be trivially achieved using a surface-based rendering framework.
Third, similar to other volume rendering-based models, our model is able to
optimize the geometry as well as the relightable appearance end-to-end under
varying lighting conditions. We do not require model design changes to back
propagate the loss gradient back to the geometry prediction [Munkberg et al.,
2022]. Our results show that this rendering strongly result in higher fidelity
compared to previous surface-based rendering [Lyu et al., 2022].

Figure 6. Illustration of our light stage capture system. A full capture
consists of 9 capture groups, with each group labelled as “000”, “040”, “080”,
…, “320”, with their number denoting the 40 degree-stepped yaw rotation (see
“$\phi_{\text{ls}}$” in (b)). Lights are visualized as dots and cameras with
camera icons. All lights are of the same white color—the visualized dot colors
merely refer to the light bulb instance, highlighting that the lights are
locked with the cameras when rotating between groups. (a) Frontal view of the
system (group “000”). The radius of the light stage is 1.5 meters, with its
center at 1.1m height—the layout is a bottom-truncated sphere. The light stage
illustration in Fig. 1(a) is the elevated view of group “040”. (b) Rotating
from group “000” (b-left) to group “240” (b-right) according to the
“$\phi_{\text{ls}}$” rotation. (b-left) and (b-right) are visualized at an
elevated angle. (a) is viewed from the dashed line direction in (b-left).

###  3.4. End-to-end Learning of Neural Relighting

Architectures. We follow [Mildenhall et al., 2020] and use an MLP to predict
the density as well as the transfer vector gradient for each sample interval.
The MLP consists of 8 fully-connected layers (with a width of 256, and a skip
connection in the fourth layer) each for the density as well as the transfer
vector gradient prediction respectively. We devise two MLPs tackling the
coarse level and fine level of accumulation respectively. To ensure the
predicted transfer vector gradient to be non-negative, we use the exponential
function as the activation function, following [Mildenhall et al., 2022]. It
is worth pointing out that MLPs are just one option for modeling the
predictions of each point, we expect that more efficient models [Yu et al.,
2021; Fridovich-Keil et al., 2022; Müller et al., 2022; Chen et al., 2022] can
be used as well.

Loss functions. We utilize the weighted L2 tonemapped loss [Mildenhall et al.,
2022] to supervise the predicted HDR values of each pixel. We also impose an
auxiliary mask loss, where we pose L2 regularizers on the density of all
points that are sampled on a background ray. Our main focus is the modeling of
the foreground objects, and due to the inconsistency of the background
appearance (rotation nature of the camera groups, see Sec. 4), we set all the
ground truth HDR values in the background to be 0. We sample the rays (from
both the foreground as well as the background) with importance sampling, where
in each training batch, 1/2 of all rays are from the foreground, 3/8 of all
rays are from the near-silhouette area, and 1/8 of total rays are from
arbitrary locations in the background. In our real light stage data, since the
aspect ratio of the captured images is pretty large, we extend the background
area that is outside of the pixel map. We pad them with 0 as their ground
truth HDR values. We found this to be useful for clean free-space density
predictions.

Using environment map conditioning. We follow [Lyu et al., 2022] to obtain the
envmap relighting prediction via accumulating the OLAT HDR prediction. More
precisely, we treat each pixel on the envmap as an OLAT point light.
Practically, we apply the median cut algorithm [Debevec, 2008] to accelerate
inference. During accumulation, we reweight the predicted HDR value from each
OLAT location by the cosine value of the latitude angle on the envmap to
account for the area of lights on the envmap sphere. The aggregated predicted
rendering serves as our final prediction of the relighting given the query
envmap.

##  4\. Light Stage Data Acquisition

To facilitate studies on the light-dependent appearance modeling of objects
and scenes under significant subsurface scattering effects, it is critical to
acquire real-world objects featuring such effects. While existing datasets
(e.g., [Deng et al., 2022]) includes captures of two translucent objects, they
are often limited by resolution and fidelity of the acquired images, causing
local micro geometry details to not be fully captured. To reconstruct a
relightable model in a data-driven fashion, we aim to have real-world captures
with densely sampled camera viewpoints, complete incident light direction
coverage and high-resolution images retaining as much detail as possible.
Consequently, we propose a new dataset, consisting of 8 scenes with
significant subsurface scattering effects. Our captured data demonstrates high
fidelity, preserving rich appearance details, and represents a total of 15TB
(3000 times larger than the currently highest quality dataset with similar
goals to our knowledge, [Deng et al., 2022]).

As shown in Fig. 6, we place the cameras and the light sources on the
spherical light stage cage, while the objects to be captured are placed on a
holding table in the center with a height of roughly 1.1 meter. In particular,
when capturing the data, our cameras and the light bulbs are fixed on the
sphere, while a turntable in the middle can be freely rotated. Ignoring
background pixels, this is equivalent to keeping the object scene static to
satisfy the consistency of the scene among views, while rotating the cameras
and the light bulbs altogether. Throughout the text, we assume that the light
stage is configured in the latter case for notational convenience. Our
camera/light-bulb sphere radius is roughly 1.5 meter from the surface of the
holding table in the middle).***Our light bulbs only span roughly between [0,
$\frac{3}{2}\pi$) for $\theta_{ls}$, hence no light bulb has a negative
altitude even if the sphere radius is larger than the height of the center—the
holding table. The rotations of the sphere put the whole captured frames into
9 groups, with each group corresponding to one particular rotated setting of
the camera-light sphere. On the sphere, we have a total of 20 cameras as well
as 331 lighting bulbs (serving as 331 OLAT point lights).†††Notably, since the
point light locations are locked with the camera during rotation, the OLAT
location in different groups are different from each other. In other words, in
our whole dataset, there are only up to 20 images that have been recorded with
the same lighting. Consequently, in each group we captured a $20\times
331=6620$ frames, and for the total 9 groups, we captured a total of 59580
frames for one scene. Our camera captures high dynamic range value for the
RGBs, with the cutoff threshold at $4.4019$. The original captured frames come
with a resolution of $8192\times 5464$. We found a 4 times down-scaling
retains most of the texture details and hence we conduct all our experiments
on the down-scaled version ($2048\times 1366$). Notably, all the captures at
the resolution of $2048\times 1366$ still span 15TB of storage. During the
capture, the cameras always face toward the objects on the holding table, and
we tune the focal length of the camera to best suit the size of the particular
objects. We obtain the extrinsic camera poses via an off-the-shelf software
with manual corrections. Since the light bulbs shining in the opposite
direction of the camera incur significant noise to the reconstruction process
(especially considering that the rotation between the group would make the
background inconsistent), we introduced several heuristics, including RGB
variations and saturation to segment out the background. All the camera poses,
light locations as well as the masking information are used by all the
approaches in our evaluation sections (Sec. 5), and we shall make all the
details about the data publicly available to facilitate future research.

##  5\. Experiments

We use both the synthetic data (8 scenes) and the real data we captured (8
scenes) for evaluation and comparisons. All the details on data, training and
benchmarking protocols will be released.

Synthetic Data. We use the 8 scenes from the NeRF Blender dataset [Mildenhall
et al., 2020] and evaluate them with both their original materials (Synthetic-
Original) as well as their modified materials with the subsurface scattering
shader in Blender [Community, 2018] (Synthetic-SSS). During training, we use
the same 100 camera views given in the training set for each scene as provided
by the originally released data [Mildenhall et al., 2020]. To simulate OLAT
lighting, we evenly sample 112 incident lighting directions on the upper
hemisphere. More precisely, we sample evenly with 7 latitudes in the upper
hemisphere, evenly sampled 32 longitudes for each latitude, and left out every
other light (to be used during evaluation). The 7 $\times$ 32 OLAT directions
exactly correspond to Row 2 through Row 8 of the 16 $\times$ 32 envmap as used
in NRTF [Lyu et al., 2022]. We exclude the lower hemisphere for OLAT sampling,
mainly due to the fact that most of the scenes in the NeRF blender dataset are
rendered as top views, and the OLAT lighting from the bottom produces overall
dark renderings. This training setting gives us a total of 11200 training
images per scene. To mimic the light stage setting used for real-world data
capture, we use only white lights, and use the point light instead of the
envmap for rendering the ground truth. More precisely, the point lights are
placed roughly 100 units away from the scene center (with about 4 units being
the approximate size of each scene). During testing, we use unseen lighting
directions as well as unseen camera poses for each test sample. For
quantitative evaluation, we stick to the OLAT protocol where there is only one
light at a time. For saving evaluation time, we only test 10 out of the unseen
112 lights. We also provide qualitative samples by rendering results with
several envmaps downloaded from PolyHaven (e.g., Fig. 10). Since our point
lights are single-colored (white), we do the inference with the independent-
RGB-channel assumption when relighting under a colored envmap. Following [Lyu
et al., 2022] we cast them into a 32 $\times$ 16 envmap to serve as the input.
For test time camera poses, we apply the camera views from the test views
given in the NeRF blender dataset. For saving evaluation time, we only test 10
out of the unseen 200 test views. This test setup gives us 100 test cases in
total for each scene.

Light Stage Data. As introduced in Sec. 4, the proposed light stage data
contains 9 groups and 20 cameras per scene (a total of 180 views), with each
view consisting of 331 OLAT renderings, thus leading to a total of 59580 HDR
images per scene. During training, we use the first 18 cameras in each group,
and use 75 out of the 331 OLATs for training, leading to a total of 12150
training images per scene. Testing on real data also only includes samples
with both, unseen lighting directions and unseen views. For quantitative
evaluation, we use the remaining 2 cameras from each group (a total of 18
views) and 10 unseen OLATs to form our test set (180 images per scene). For
qualitative evaluation, we use the same input lighting envmaps as used in the
synthetic data benchmark. Since most of our real captures exhibit subsurface
scattering, we denote this data with Real-SSS.

Real-SSS | C-Red | J-Dragon | J-Fish | C-Cake | S-Lvd. | C-Blue | C-Head | C-S | Average
---|---|---|---|---|---|---|---|---|---
PSNR($\uparrow$) | IRON [Zhang et al., 2022] | 21.6 | 17.5 | 20.7 | 22.2 | 22.4 | 19.1 | 21.4 | 23.3 | 21.0
InverseTranslucent [Deng et al., 2022] | 23.3 | 21.6 | 23.6 | 22.9 | 25.1 | 21.8 | 25.0 | 26.8 | 23.8
NRTF [Lyu et al., 2022] | 27.5 | 28.5 | 28.4 | 29.7 | 30.7 | 29.0 | 30.7 | 32.0 | 29.6
Ours | 30.9 | 29.0 | 30.3 | 32.3 | 33.2 | 31.2 | 34.1 | 36.3 | 32.2
SSIM($\uparrow$) | IRON [Zhang et al., 2022] | 85.5 | 85.7 | 82.8 | 88.6 | 89.2 | 82.7 | 90.0 | 90.8 | 86.9
InverseTranslucent [Deng et al., 2022] | 86.2 | 89.6 | 84.6 | 89.7 | 90.7 | 86.3 | 92.3 | 93.3 | 89.1
NRTF [Lyu et al., 2022] | 92.3 | 94.0 | 92.5 | 94.1 | 94.7 | 92.8 | 95.8 | 96.5 | 94.1
Ours | 93.4 | 94.7 | 93.3 | 94.8 | 95.7 | 93.8 | 96.9 | 97.6 | 95.0
LPIPS($\downarrow$) | IRON [Zhang et al., 2022] | 0.131 | 0.143 | 0.173 | 0.108 | 0.109 | 0.179 | 0.109 | 0.106 | 0.132
InverseTranslucent [Deng et al., 2022] | 0.139 | 0.132 | 0.165 | 0.119 | 0.110 | 0.186 | 0.104 | 0.104 | 0.132
NRTF [Lyu et al., 2022] | 0.110 | 0.095 | 0.125 | 0.088 | 0.088 | 0.139 | 0.082 | 0.080 | 0.101
Ours | 0.099 | 0.089 | 0.123 | 0.078 | 0.077 | 0.132 | 0.071 | 0.067 | 0.092
|  |  |  |  |  |  |  |  |  |

Table 1. Comparison with several state-of-the-art methods on the “Real-SSS”
data (8 scenes). Despite optimized on the same data, our results consistently
outperform the existing approaches on all scenes and all evaluation metrics.
Material abbreviations: “C-” stands for “Candle”, “J-” stands for “Jade”, and
“S-” stands for “Soap”.

PSNR ($\uparrow$) | IRON | Inv. Translucent | NRTF | Ours
---|---|---|---|---
Synthetic-Original | 24.4 | 23.8 | 29.0 | 33.3
Synthetic-SSS | 23.1 | 26.9 | 31.1 | 39.3
|  |  |  |

Table 2. Comparison on Synthetic-Original and Synthetic-SSS. Please refer to
our supplementary materials for further details.

Evaluation Metrics. We evaluate the predicted pixel map following the standard
metric protocol [Lyu et al., 2022; Mildenhall et al., 2020], including PSNR,
SSIM and LPIPS [Zhang et al., 2018]. While our main focus is to evaluate the
objects of the scene, we follow existing protocols [Mildenhall et al., 2020]
to include all the pixels for evaluation. Following most of the recent
evaluation conventions (e.g. [Mildenhall et al., 2020]), we evaluate every
pixel on the predicted pixel maps (including the background regions). This
also includes the areas where the stand holds the captured objects.

Baseline approaches. We compare with several most representative state-of-the-
art approaches to highlight the strengths of our neural relightable model. All
models are trained with exactly the same data.

$\bullet$ IRON [Zhang et al., 2022] is a recent representative BRDF-based
relighting approach and achieves state-of-the-art performance with the
collocated GGX shader. We underwent major efforts to generalize it to the
general setting where the incident light direction, viewing direction and bi-
sector direction are no longer identical. Notably, while the GGX shader cannot
handle subsurface scattering, optimization in scenarios where lighting is
coming from the opposite side of the camera is essential, especially when
translucency is present. We used Mitsuba 3 [Jakob et al., 2022] to render the
trained textured models and fit the best HDR scaling with the ground truth
before computing PSNR.

$\bullet$ InverseTranslucent [Deng et al., 2022] is a recent representative
state-of-the-art BSSRDF relighting approach. We train the models using
spatially varying albedo, sigma (controlling light transmission underneath the
surface) and roughness all in the resolution of 256 $\times$ 256\. We found
[Deng et al., 2022] is sensitive to the geometry initialization, and thus we
provide the baseline with the optimized Neus reconstruction using their
original implementation [Wang et al., 2021].

$\bullet$ NRTF [Lyu et al., 2022] is a recent state-of-the-art fully data-
driven approach that is designed to handle global illumination and potentially
subsurface scattering.

For [Zhang et al., 2022; Lyu et al., 2022], we use the provided Neus
implementation rather than the original version to obtain object surfaces.

It is worth pointing out that [Zhang et al., 2022] was originally proposed to
handle only the PBR based materials with the assumptions that all the objects
are fully opaque, and hence it was not proposed to handle our evaluation data
of Synthetic-SSS and Real-SSS (our proposed light stage data). Meanwhile,
[Deng et al., 2022] was originally proposed to handle specifically objects
with translucency, but not necessarily opaque objects as present in our
evaluation data Synthetic-Original. We still include all results in the
experiments for reference purposes since our approach is able to handle all
the types of the materials, further showcasing the wide applicability of the
method.

Results. As shown in Tab. 1-2 and Fig. 8-10, our results demonstrate clear
advantages compared to all aforementioned methods. Notably, we achieve 5
points overall average PSNR gain (averaging over all the synthetic and real
data) over the best-performing existing method thanks to our end-to-end
learning framework. We conclude that our relighting approach can not only
handle a wider range of material types (in particular objects with subsurface
scattering effects) with significantly improved fidelity, but also stays
flexible representing vivid and rich geometric structures, such as the thin
ropes that are generally not easy to represent using meshes. In contrast to
other approaches [Zhang et al., 2022; Deng et al., 2022] that were designed to
handle a relatively narrow range of material types, our approach is able to
handle the full variety of materials present in the datasets. This underscores
the general applicability of our approach regarding material representations.
Please refer to our supplementary materials for additional results.

Figure 7. Failure cases on specular highlights (left) and translucent
shadowing (right). The proposed method does not explicitly model specularities
and shadowing.

Limitations. Our approach exhibits two main types of failure modes. First, the
proposed method may return blurry results for specular highlights (c.f., Fig.
7—left) since the model does not take specularities into account in a
dedicated way. Similarly, our approach does not contain a dedicated model for
shadows. In particular, when shadows “penatrate” a thin layer of translucent
material (e.g., Fig. 7-right) our model creates blurry boundaries on otherwise
hard shadow borders.

Another avenue for future improvement is rendering speed: the proposed model
does not yet meet the demand of real-time applications. Further, our
relighting algorithm is relying on a light stage capture system and is not yet
suited for in-the-wild use.

##  6\. Conclusion

We presented a novel volume-rendering based neural relighting approach adept
at handling subsurface scattering effects. Thanks to the end-to-end
optimization of the radiance transfer gradient on images recorded under
various lighting conditions in a light stage, the optimized geometry and
appearance reach high quality—even on real data with major subsurface
scattering effects. We extensively evaluated the proposed method and
established comparisons with several related optimization and modeling
approaches and found it to consistently and notably outperform existing work.

Figure 8. Detailed comparison for Soap-Lavender (left) and Candle-Head (right)
between our results (Row 4) with other state-of-the-art approaches (IRON
[Zhang et al., 2022] in Row 1, InverseTranslucent [Deng et al., 2022] in Row
2, and NRTF [Lyu et al., 2022] in Row 3). Recordings can be found in the last
row; all images are held out positions for lights and cameras. Our results
show a clear advantage in terms of visual fidelity and geometric accuracy.
Figure 9. Envmap relighting results on our real-SSS dataset (light stage
captures). The results in each row are from the same scene, while the results
in each column are relit using the same environment map. Figure 10. Relighting
results for various environment maps for the original as well as the
translucent version of the synthetic scenes from the Nerf-Blender synthetic
datasets (Synthetic-Original and Synthetic-SSS).

## References

  * [1]
  * Barron et al. [2021] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. 2021.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 5855–5864.
  * Barron et al. [2022] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. 2022\.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5470–5479.
  * Bi et al. [2020a] Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. 2020a.  Neural reflectance fields for appearance acquisition.  _arXiv preprint arXiv:2008.03824_ (2020).
  * Bi et al. [2020b] Sai Bi, Zexiang Xu, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. 2020b.  Deep reflectance volumes: Relightable reconstructions from multi-view photometric images. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16_. Springer, 294–311.
  * Boss et al. [2021a] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch. 2021a.  Nerd: Neural reflectance decomposition from image collections. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 12684–12694.
  * Boss et al. [2021b] Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan Barron, and Hendrik Lensch. 2021b.  Neural-pil: Neural pre-integrated lighting for reflectance decomposition.  _Advances in Neural Information Processing Systems_ 34 (2021), 10691–10704.
  * Burley and Studios [2012] Brent Burley and Walt Disney Animation Studios. 2012.  Physically-based shading at disney. In _Acm Siggraph_ , Vol. 2012. vol. 2012, 1–7.
  * Che et al. [2020] Chengqian Che, Fujun Luan, Shuang Zhao, Kavita Bala, and Ioannis Gkioulekas. 2020.  Towards learning-based inverse subsurface scattering. In _2020 IEEE International Conference on Computational Photography (ICCP)_. IEEE, 1–12.
  * Chen et al. [2022] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. 2022.  Tensorf: Tensorial radiance fields. In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII_. Springer, 333–350.
  * Chen et al. [2020] Zhang Chen, Anpei Chen, Guli Zhang, Chengyuan Wang, Yu Ji, Kiriakos N Kutulakos, and Jingyi Yu. 2020.  A neural rendering framework for free-viewpoint relighting. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5599–5610.
  * Community [2018] Blender Online Community. 2018\.  _Blender - a 3D modelling and rendering package_.  Blender Foundation, Stichting Blender Foundation, Amsterdam.  http://www.blender.org
  * Debevec [2008] Paul Debevec. 2008\.  A median cut algorithm for light probe sampling.  In _ACM SIGGRAPH 2008 classes_. 1–3.
  * Debevec et al. [2000] Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. 2000.  Acquiring the reflectance field of a human face. In _Proceedings of the 27th annual conference on Computer graphics and interactive techniques_. 145–156.
  * Deng et al. [2022] Xi Deng, Fujun Luan, Bruce Walter, Kavita Bala, and Steve Marschner. 2022.  Reconstructing Translucent Objects using Differentiable Rendering. In _ACM SIGGRAPH 2022 Conference Proceedings_. 1–10.
  * Fridovich-Keil et al. [2022] Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. 2022\.  Plenoxels: Radiance fields without neural networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5501–5510.
  * Fyffe [2009] Graham Fyffe. 2009\.  Cosine lobe based relighting from gradient illumination photographs.  In _SIGGRAPH’09: Posters_. 1–1.
  * Gkioulekas et al. [2013] Ioannis Gkioulekas, Shuang Zhao, Kavita Bala, Todd Zickler, and Anat Levin. 2013.  Inverse volume rendering with material dictionaries.  _ACM Transactions on Graphics (TOG)_ 32, 6 (2013), 1–13.
  * Guo et al. [2019] Kaiwen Guo, Peter Lincoln, Philip Davidson, Jay Busch, Xueming Yu, Matt Whalen, Geoff Harvey, Sergio Orts-Escolano, Rohit Pandey, Jason Dourgarian, et al. 2019\.  The relightables: Volumetric performance capture of humans with realistic relighting.  _ACM Transactions on Graphics (ToG)_ 38, 6 (2019), 1–19.
  * Hasselgren et al. [2022] Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg. 2022\.  Shape, light & material decomposition from images using monte carlo rendering and denoising.  _arXiv preprint arXiv:2206.03380_ (2022).
  * Jakob et al. [2022] Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, and Delio Vicini. 2022.  DR. JIT: a just-in-time compiler for differentiable rendering.  _ACM Transactions on Graphics (TOG)_ 41, 4 (2022), 1–19.
  * Luan et al. [2021] Fujun Luan, Shuang Zhao, Kavita Bala, and Zhao Dong. 2021\.  Unified shape and svbrdf recovery using differentiable monte carlo rendering. In _Computer Graphics Forum_ , Vol. 40. Wiley Online Library, 101–113.
  * Lyu et al. [2022] Linjie Lyu, Ayush Tewari, Thomas Leimkühler, Marc Habermann, and Christian Theobalt. 2022.  Neural Radiance Transfer Fields for Relightable Novel-view Synthesis with Global Illumination. In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVII_. Springer, 153–169.
  * Mildenhall et al. [2022] Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul P Srinivasan, and Jonathan T Barron. 2022\.  Nerf in the dark: High dynamic range view synthesis from noisy raw images. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 16190–16199.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020\.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.  arXiv:2003.08934 [cs.CV]
  * Müller et al. [2022] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Transactions on Graphics (ToG)_ 41, 4 (2022), 1–15.
  * Munkberg et al. [2022] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. 2022\.  Extracting triangular 3d models, materials, and lighting from images. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 8280–8290.
  * Nimier-David et al. [2019] Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019.  Mitsuba 2: A retargetable forward and inverse renderer.  _ACM Transactions on Graphics (TOG)_ 38, 6 (2019), 1–17.
  * Novák et al. [2018] Jan Novák, Iliyan Georgiev, Johannes Hanika, Jaroslav Krivánek, and Wojciech Jarosz. 2018\.  Monte Carlo methods for physically based volume rendering.. In _SIGGRAPH Courses_. 14–1.
  * Sloan et al. [2002] Peter-Pike Sloan, Jan Kautz, and John Snyder. 2002\.  Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. In _Proceedings of the 29th annual conference on Computer graphics and interactive techniques_. 527–536.
  * Srinivasan et al. [2021] Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron. 2021.  Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7495–7504.
  * Vicini et al. [2019] Delio Vicini, Vladlen Koltun, and Wenzel Jakob. 2019\.  A learned shape-adaptive subsurface scattering model.  _ACM Transactions on Graphics (TOG)_ 38, 4 (2019), 1–15.
  * Walter et al. [2007] Bruce Walter, Stephen R Marschner, Hongsong Li, and Kenneth E Torrance. 2007.  Microfacet models for refraction through rough surfaces. In _Proceedings of the 18th Eurographics conference on Rendering Techniques_. 195–206.
  * Wang et al. [2021] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. 2021\.  NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction.  _arXiv preprint arXiv:2106.10689_ (2021).
  * Xie et al. [2022] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. 2022\.  Neural fields in visual computing and beyond. In _Computer Graphics Forum_ , Vol. 41. Wiley Online Library, 641–676.
  * Yu et al. [2021] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. 2021\.  Plenoctrees for real-time rendering of neural radiance fields.  _arXiv preprint arXiv:2103.14024_ (2021).
  * Yu et al. [2023] Hong-Xing Yu, Michelle Guo, Alireza Fathi, Yen-Yu Chang, Eric Ryan Chan, Ruohan Gao, Thomas Funkhouser, and Jiajun Wu. 2023\.  Learning object-centric neural scattering functions for free-viewpoint relighting and scene composition.  _arXiv preprint arXiv:2303.06138_ (2023).
  * Yu et al. [1999] Yizhou Yu, Paul Debevec, Jitendra Malik, and Tim Hawkins. 1999\.  Inverse global illumination: Recovering reflectance models of real scenes from photographs. In _Proceedings of the 26th annual conference on Computer graphics and interactive techniques_. 215–224.
  * Zhang et al. [2022] Kai Zhang, Fujun Luan, Zhengqi Li, and Noah Snavely. 2022\.  Iron: Inverse rendering by optimizing neural sdfs and materials from photometric images. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5565–5574.
  * Zhang et al. [2021a] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. 2021a.  Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5453–5462.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018.  The unreasonable effectiveness of deep features as a perceptual metric. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 586–595.
  * Zhang et al. [2021b] Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron. 2021b.  Nerfactor: Neural factorization of shape and reflectance under an unknown illumination.  _ACM Transactions on Graphics (TOG)_ 40, 6 (2021), 1–18.
  * Zheng et al. [2021] Quan Zheng, Gurprit Singh, and Hans-Peter Seidel. 2021\.  Neural relightable participating media rendering.  _Advances in Neural Information Processing Systems_ 34 (2021), 15203–15215.