M0 | deg | 0.00002 | 0.00000 | 0.00010 | 2.07536 $\pm$ 1.83563 | 4.41051 $\pm$ 3.97091 | 4.07306 $\pm$ 3.66832
i | deg | 89.18430 | 89.42708 | 69.37943 | 89.34568 $\pm$ 0.77056 | 89.42539 $\pm$ 0.50304 | 71.01877 $\pm$ 3.23178
$\Omega$ | deg | 0.00004 | 0.00025 | 0.00000 | 1.68393 $\pm$ 1.27003 | 30.23809 $\pm$ 29.61307 | 14.80834 $\pm$ 13.91545
$\chi^{2}$ | … | 49.69869 | 5.23280 | … | 50.29577 | 2.51644 | …
$\chi^{2}_{\rm red}$ | … | 2.28881 | 13.73287 | … | 2.20051 | 13.20305 | …
$\ln\mathcal{L}$ | … | 217.53231 | 230.09827
$BIC$ | … | -353.33638 | -378.46832
$AIC_{c}$ | … | -358.84239 | -383.97433

Note. — The K’s listed here are unconstrained, but see $\S$6 for discussion
regarding the planets’ low K’s generated by Exo-Striker.

Table 25: Exo-Striker-generated best fit and MCMC modeling parameters for Pd =
14.06 days.

Parameter | Unit | Best Fit | MCMC
---|---|---|---
AU Mic b | AU Mic c | AU Mic d | AU Mic b | AU Mic c | AU Mic d
K | m s-1 | 0.35093 | 4.77345 | 0.11884 | 0.16407 $\pm$ 0.13072 | 6.18469 $\pm$ 2.98158 | 0.10151 $\pm$ 0.00548
Porb | day | 8.46312 | 18.85944 | 14.05900 | 8.46294 $\pm$ 0.00006 | 18.85896 $\pm$ 0.00002 | 14.06837 $\pm$ 0.00668
e | … | 0.00054 | 0.00000 | 0.00002 | 0.02493 $\pm$ 0.01168 | 0.00114 $\pm$ 0.00059 | 0.01868 $\pm$ 0.01544
$\omega$ | deg | 89.99978 | 224.00139 | 317.45994 | 88.70857 $\pm$ 1.62904 | 220.36756 $\pm$ 3.73928 | 296.28362 $\pm$ 7.31581
M0 | deg | 0.00005 | 0.00000 | 0.00010 | 1.35180 $\pm$ 1.11110 | 3.64407 $\pm$ 3.33291 | 19.59544 $\pm$ 19.32409
i | deg | 88.80255 | 89.44517 | 67.10440 | 88.99096 $\pm$ 0.39592 | 89.54381 $\pm$ 0.61789 | 61.44101 $\pm$ 0.69863
$\Omega$ | deg | 0.00010 | 0.00034 | 0.00000 | 25.46116 $\pm$ 24.71971 | 1.36176 $\pm$ 0.89242 | 14.51459 $\pm$ 14.12428
$\chi^{2}$ | … | 65.16546 | 10.92404 | … | 41.65098 | 3.66603 | …
$\chi^{2}_{\rm red}$ | … | 3.17040 | 19.02238 | … | 1.88821 | 11.32925 | …
$\ln\mathcal{L}$ | … | 203.49800 | 234.88087
$BIC$ | … | -325.26778 | -388.03351
$AIC_{c}$ | … | -330.77378 | -393.53951

Note. — The K’s listed here are unconstrained, but see $\S$6 for discussion
regarding the planets’ low K’s generated by Exo-Striker.

Figure 19: O–C diagram of AU Mic b (left) and AU Mic c (right), with
comparison between TTVs (green) and Exo-Striker-generated mcmc models (black)
for Pd = 5.07 days (top row) and 5.39 days (bottom row).

Figure 20: O–C diagram of AU Mic b (left) and AU Mic c (right), with
comparison between TTVs (green) and Exo-Striker-generated mcmc models (black)
for 11.84 days (top row), 13.29 days (middle row), and 14.06 days (bottom
row). Figure 21: Corner plot of AU Mic b and c’s orbital parameters from Exo-
Striker mcmc analysis for Pd = 5.07 days. Figure 22: Corner plot of AU Mic b
and c’s orbital parameters from Exo-Striker mcmc analysis for Pd = 5.39 days.
Figure 23: Corner plot of AU Mic b and c’s orbital parameters from Exo-Striker
mcmc analysis for Pd = 11.84 days. Figure 24: Corner plot of AU Mic b and c’s
orbital parameters from Exo-Striker mcmc analysis for Pd = 13.29 days. Figure
25: Corner plot of AU Mic b and c’s orbital parameters from Exo-Striker mcmc
analysis for Pd = 14.06 days.

Figure 26: rebound model of the stability of AU Mic system for Pd = 5.07 days
(top left), 5.39 days (top right), 11.84 days (middle left), 13.29 days
(middle right), and 14.06 days (bottom).

Figure 27: Raw TESS photometry of AU Mic overlaid with observed transits of AU
Mic b (red) and c (blue) and range of predicted transit midpoint times for AU
Mic d (green), with Pd = 5.07 days (first row), 5.39 days (second row), 11.84
days (third row), 13.29 days (fourth row), and 14.06 days (fifth row). Left
column is from cycle 1, and right column is from cycle 3. Owing to AU Mic d’s
estimated 150 ppm depth and AU Mic’s intense stellar activity, the potential
transit signatures from AU Mic d is not obvious in these plots.

## References

  * Abe et al. (2013) Abe, L., Gonçalves, I., Agabi, A., et al. 2013, A&A, 553, A49, doi: 10.1051/0004-6361/201220351
  * Addison et al. (2021) Addison, B. C., Horner, J., Wittenmyer, R. A., et al. 2021, AJ, 162, 137, doi: 10.3847/1538-3881/ac1685
  * Akeson et al. (2013) Akeson, R. L., Chen, X., Ciardi, D., et al. 2013, PASP, 125, 989, doi: 10.1086/672273
  * Arnold et al. (2022) Arnold, J. A., Weinberger, A. J., Videen, G., & Zubko, E. S. 2022, ApJ, 930, 123, doi: 10.3847/1538-4357/ac63a9
  * Astropy Collaboration et al. (2013) Astropy Collaboration, Robitaille, T. P., Tollerud, E. J., et al. 2013, A&A, 558, A33, doi: 10.1051/0004-6361/201322068
  * Astropy Collaboration et al. (2018) Astropy Collaboration, Price-Whelan, A. M., Sipőcz, B. M., et al. 2018, AJ, 156, 123, doi: 10.3847/1538-3881/aabc4f
  * Bailer-Jones et al. (2018) Bailer-Jones, C. A. L., Rybizki, J., Fouesneau, M., Mantelet, G., & Andrae, R. 2018, AJ, 156, 58, doi: 10.3847/1538-3881/aacb21
  * Benz et al. (2021) Benz, W., Broeg, C., Fortier, A., et al. 2021, Experimental Astronomy, 51, 109, doi: 10.1007/s10686-020-09679-4
  * Bertini et al. (2023) Bertini, L., Roccatagliata, V., & Kim, M. 2023, A&A, submitted
  * Borucki & Summers (1984) Borucki, W. J., & Summers, A. L. 1984, Icarus, 58, 121, doi: 10.1016/0019-1035(84)90102-7
  * Brown et al. (2013) Brown, T. M., Baliber, N., Bianco, F. B., et al. 2013, PASP, 125, 1031, doi: 10.1086/673168
  * Buchhave et al. (2016) Buchhave, L. A., Dressing, C. D., Dumusque, X., et al. 2016, AJ, 152, 160, doi: 10.3847/0004-6256/152/6/160
  * Butler et al. (1981) Butler, C. J., Byrne, P. B., Andrews, A. D., & Doyle, J. G. 1981, MNRAS, 197, 815, doi: 10.1093/mnras/197.3.815
  * Cale et al. (2021) Cale, B. L., Reefe, M., Plavchan, P., et al. 2021, AJ, 162, 295, doi: 10.3847/1538-3881/ac2c80
  * Cavanaugh (1997) Cavanaugh, J. E. 1997, Stat Probab Lett, 33, 201, doi: https://doi.org/10.1016/S0167-7152(96)00128-9
  * Chen & Kipping (2017) Chen, J., & Kipping, D. 2017, ApJ, 834, 17, doi: 10.3847/1538-4357/834/1/17
  * Choi et al. (2016) Choi, J., Dotter, A., Conroy, C., et al. 2016, ApJ, 823, 102, doi: 10.3847/0004-637X/823/2/102
  * Christiansen et al. (2017) Christiansen, J. L., Vanderburg, A., Burt, J., et al. 2017, AJ, 154, 122, doi: 10.3847/1538-3881/aa832d
  * Cincotta et al. (2003) Cincotta, P. M., Giordano, C. M., & Simó, C. 2003, Physica D Nonlinear Phenomena, 182, 151, doi: 10.1016/S0167-2789(03)00103-9
  * Cincotta & Simó (2000) Cincotta, P. M., & Simó, C. 2000, A&AS, 147, 205, doi: 10.1051/aas:2000108
  * Claret & Bloemen (2011) Claret, A., & Bloemen, S. 2011, A&A, 529, A75, doi: 10.1051/0004-6361/201116451
  * Collins et al. (2017) Collins, K. A., Kielkopf, J. F., Stassun, K. G., & Hessman, F. V. 2017, AJ, 153, 77, doi: 10.3847/1538-3881/153/2/77
  * Cully et al. (1993) Cully, S. L., Siegmund, O. H. W., Vedder, P. W., & Vallerga, J. V. 1993, ApJ, 414, L49, doi: 10.1086/186993
  * Cumming et al. (1999) Cumming, A., Marcy, G. W., & Butler, R. P. 1999, ApJ, 526, 890, doi: 10.1086/308020
  * Cutri et al. (2003) Cutri, R. M., Skrutskie, M. F., van Dyk, S., et al. 2003, 2MASS All Sky Catalog of point sources. (NASA/IPAC Infrared Science Archive)
  * David et al. (2019a) David, T. J., Petigura, E. A., Luger, R., et al. 2019a, ApJ, 885, L12, doi: 10.3847/2041-8213/ab4c99
  * David et al. (2019b) David, T. J., Cody, A. M., Hedges, C. L., et al. 2019b, AJ, 158, 79, doi: 10.3847/1538-3881/ab290f
  * Dotter (2016) Dotter, A. 2016, ApJS, 222, 8, doi: 10.3847/0067-0049/222/1/8
  * Eastman et al. (2013) Eastman, J., Gaudi, B. S., & Agol, E. 2013, PASP, 125, 83, doi: 10.1086/669497
  * Eastman et al. (2010) Eastman, J., Siverd, R., & Gaudi, B. S. 2010, PASP, 122, 935, doi: 10.1086/655938
  * Eastman et al. (2019) Eastman, J. D., Rodriguez, J. E., Agol, E., et al. 2019, arXiv e-prints, arXiv:1907.09480.  https://arxiv.org/abs/1907.09480
  * Fabrycky et al. (2014) Fabrycky, D. C., Lissauer, J. J., Ragozzine, D., et al. 2014, ApJ, 790, 146, doi: 10.1088/0004-637X/790/2/146
  * Fajardo-Acosta et al. (2000) Fajardo-Acosta, S. B., Beichman, C. A., & Cutri, R. M. 2000, ApJ, 538, L155, doi: 10.1086/312815
  * Feinstein et al. (2022) Feinstein, A. D., France, K., Youngblood, A., et al. 2022, AJ, 164, 110, doi: 10.3847/1538-3881/ac8107
  * Foreman-Mackey (2018) Foreman-Mackey, D. 2018, RNAAS, 2, 31, doi: 10.3847/2515-5172/aaaf6c
  * Foreman-Mackey et al. (2017) Foreman-Mackey, D., Agol, E., Ambikasaran, S., & Angus, R. 2017, AJ, 154, 220, doi: 10.3847/1538-3881/aa9332
  * Foreman-Mackey et al. (2013) Foreman-Mackey, D., Hogg, D. W., Lang, D., & Goodman, J. 2013, PASP, 125, 306, doi: 10.1086/670067
  * Foreman-Mackey et al. (2021) Foreman-Mackey, D., Luger, R., Agol, E., et al. 2021, The Journal of Open Source Software, 6, 3285, doi: 10.21105/joss.03285
  * Gaia Collaboration et al. (2018) Gaia Collaboration, Brown, A. G. A., Vallenari, A., et al. 2018, A&A, 616, A1, doi: 10.1051/0004-6361/201833051
  * Gallenne et al. (2022) Gallenne, A., Desgrange, C., Milli, J., et al. 2022, A&A, 665, A41, doi: 10.1051/0004-6361/202244226
  * Gelman & Rubin (1992) Gelman, A., & Rubin, D. B. 1992, Statistical Science, 7, 457, doi: 10.1214/ss/1177011136
  * Gilbert et al. (2022) Gilbert, E. A., Barclay, T., Quintana, E. V., et al. 2022, AJ, 163, 147, doi: 10.3847/1538-3881/ac23ca
  * Goodman & Weare (2010) Goodman, J., & Weare, J. 2010, Communications in Applied Mathematics and Computational Science, 5, 65, doi: 10.2140/camcos.2010.5.65
  * Grady et al. (2020) Grady, C. A., Wisniewski, J. P., Schneider, G., et al. 2020, ApJ, 889, L21, doi: 10.3847/2041-8213/ab65bb
  * Guillot et al. (2015) Guillot, T., Abe, L., Agabi, A., et al. 2015, Astronomische Nachrichten, 336, 638, doi: 10.1002/asna.201512174
  * Harris et al. (2020) Harris, C. R., Millman, K. J., van der Walt, S. J., et al. 2020, Nature, 585, 357, doi: 10.1038/s41586-020-2649-2
  * Huber et al. (2022) Huber, D., White, T. R., Metcalfe, T. S., et al. 2022, AJ, 163, 79, doi: 10.3847/1538-3881/ac3000
  * Hunter (2007) Hunter, J. D. 2007, Computing in Science and Engineering, 9, 90, doi: 10.1109/MCSE.2007.55
  * Hurvich & Tsai (1989) Hurvich, C. M., & Tsai, C.-L. 1989, Biometrika, 76, 297, doi: 10.1093/biomet/76.2.297
  * Ilin & Poppenhaeger (2022) Ilin, E., & Poppenhaeger, K. 2022, MNRAS, 513, 4579, doi: 10.1093/mnras/stac1232
  * Jensen (2013) Jensen, E. 2013, Tapir: A web interface for transit/eclipse observability, Astrophysics Source Code Library, record ascl:1306.007.  http://ascl.net/1306.007
  * Kalas et al. (2004) Kalas, P., Liu, M. C., & Matthews, B. C. 2004, Science, 303, 1990, doi: 10.1126/science.1093420
  * Kane et al. (2022) Kane, S. R., Foley, B. J., Hill, M. L., et al. 2022, AJ, 163, 20, doi: 10.3847/1538-3881/ac366b
  * Kreidberg (2015) Kreidberg, L. 2015, PASP, 127, 1161, doi: 10.1086/683602
  * Kundu et al. (1987) Kundu, M. R., Jackson, P. D., White, S. M., & Melozzi, M. 1987, ApJ, 312, 822, doi: 10.1086/164928
  * Laskar (1997) Laskar, J. 1997, A&A, 317, L75
  * Laskar (2000) —. 2000, Phys. Rev. Lett., 84, 3240, doi: 10.1103/PhysRevLett.84.3240
  * Laskar & Petit (2017) Laskar, J., & Petit, A. C. 2017, A&A, 605, A72, doi: 10.1051/0004-6361/201630022
  * Lightkurve Collaboration et al. (2018) Lightkurve Collaboration, Cardoso, J. V. d. M., Hedges, C., et al. 2018, Lightkurve: Kepler and TESS time series analysis in Python, Astrophysics Source Code Library, record ascl:1812.013.  http://ascl.net/1812.013
  * Lithwick et al. (2012) Lithwick, Y., Xie, J., & Wu, Y. 2012, ApJ, 761, 122, doi: 10.1088/0004-637X/761/2/122
  * Liu et al. (2004) Liu, M. C., Matthews, B. C., Williams, J. P., & Kalas, P. G. 2004, ApJ, 608, 526, doi: 10.1086/392531
  * MacGregor et al. (2013) MacGregor, M. A., Wilner, D. J., Rosenfeld, K. A., et al. 2013, ApJ, 762, L21, doi: 10.1088/2041-8205/762/2/L21
  * Mamajek & Bell (2014) Mamajek, E. E., & Bell, C. P. M. 2014, MNRAS, 445, 2169, doi: 10.1093/mnras/stu1894
  * Martioli et al. (2021) Martioli, E., Hébrard, G., Correia, A. C. M., Laskar, J., & Lecavelier des Etangs, A. 2021, A&A, 649, A177, doi: 10.1051/0004-6361/202040235
  * Martioli et al. (2020) Martioli, E., Hébrard, G., Moutou, C., et al. 2020, A&A, 641, L1, doi: 10.1051/0004-6361/202038695
  * Mékarnia et al. (2016) Mékarnia, D., Guillot, T., Rivet, J. P., et al. 2016, MNRAS, 463, 45, doi: 10.1093/mnras/stw1934
  * Olofsson et al. (2022) Olofsson, J., Thébault, P., Kral, Q., et al. 2022, MNRAS, 513, 713, doi: 10.1093/mnras/stac455
  * Osborn et al. (2021) Osborn, A., Armstrong, D. J., Cale, B., et al. 2021, MNRAS, 507, 2782, doi: 10.1093/mnras/stab2313
  * Palle et al. (2020) Palle, E., Oshagh, M., Casasayas-Barris, N., et al. 2020, A&A, 643, A25, doi: 10.1051/0004-6361/202038583
  * Perez & Granger (2007) Perez, F., & Granger, B. E. 2007, Computing in Science and Engineering, 9, 21, doi: 10.1109/MCSE.2007.53
  * Plavchan et al. (2005) Plavchan, P., Jura, M., & Lipscy, S. J. 2005, ApJ, 631, 1161, doi: 10.1086/432568
  * Plavchan et al. (2009) Plavchan, P., Werner, M. W., Chen, C. H., et al. 2009, ApJ, 698, 1068, doi: 10.1088/0004-637X/698/2/1068
  * Plavchan et al. (2020) Plavchan, P., Barclay, T., Gagné, J., et al. 2020, Nature, 582, 497, doi: 10.1038/s41586-020-2400-z
  * Rando et al. (2020) Rando, N., Asquier, J., Corral Van Damme, C., et al. 2020, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 11443, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, 1144314, doi: 10.1117/12.2567296
  * Rein & Liu (2012) Rein, H., & Liu, S. F. 2012, A&A, 537, A128, doi: 10.1051/0004-6361/201118085
  * Rein & Spiegel (2015) Rein, H., & Spiegel, D. S. 2015, MNRAS, 446, 1424, doi: 10.1093/mnras/stu2164
  * Sackett (1999) Sackett, P. D. 1999, in NATO Advanced Study Institute (ASI) Series C, Vol. 532, Planets Outside the Solar System: Theory and Observations, ed. J. M. Mariotti & D. Alloin, 189.  https://arxiv.org/abs/astro-ph/9811269
  * Schwarz (1978) Schwarz, G. 1978, Annals of Statistics, 6, 461
  * Song et al. (2002) Song, I., Weinberger, A. J., Becklin, E. E., Zuckerman, B., & Chen, C. 2002, AJ, 124, 514, doi: 10.1086/341164
  * Stauffer et al. (2010) Stauffer, J., Tanner, A. M., Bryden, G., et al. 2010, PASP, 122, 885, doi: 10.1086/655773
  * Strubbe & Chiang (2006) Strubbe, L. E., & Chiang, E. I. 2006, ApJ, 648, 652, doi: 10.1086/505736
  * Sugiura (1978) Sugiura, N. 1978, Communications in Statistics - Theory and Methods, 7, 13, doi: 10.1080/03610927808827599
  * Sun et al. (2019) Sun, L., Ioannidis, P., Gu, S., et al. 2019, A&A, 624, A15, doi: 10.1051/0004-6361/201834275
  * Szabó et al. (2021) Szabó, G. M., Gandolfi, D., Brandeker, A., et al. 2021, A&A, 654, A159, doi: 10.1051/0004-6361/202140345
  * Szabó et al. (2022) Szabó, G. M., Garai, Z., Brandeker, A., et al. 2022, A&A, 659, L7, doi: 10.1051/0004-6361/202243076
  * Tamayo et al. (2020) Tamayo, D., Cranmer, M., Hadden, S., et al. 2020, Proceedings of the National Academy of Science, 117, 18194, doi: 10.1073/pnas.2001258117
  * Tejada Arevalo et al. (2022) Tejada Arevalo, R., Tamayo, D., & Cranmer, M. 2022, ApJ, 932, L12, doi: 10.3847/2041-8213/ac70e0
  * Ter Braak (2006) Ter Braak, C. J. F. 2006, Statistics and Computing, 16, 239, doi: 10.1007/s11222-006-8769-1
  * Trifonov (2019) Trifonov, T. 2019, The Exo-Striker: Transit and radial velocity interactive fitting tool for orbital analysis and N-body simulations, Astrophysics Source Code Library, record ascl:1906.004.  http://ascl.net/1906.004
  * Tsikoudi & Kellett (2000) Tsikoudi, V., & Kellett, B. J. 2000, MNRAS, 319, 1147, doi: 10.1046/j.1365-8711.2000.03905.x
  * Virtanen et al. (2020) Virtanen, P., Gommers, R., Oliphant, T. E., et al. 2020, Nature Methods, 17, 261, doi: 10.1038/s41592-019-0686-2
  * Vizgan et al. (2022) Vizgan, D., Hughes, A. M., Carter, E. S., et al. 2022, ApJ, 935, 131, doi: 10.3847/1538-4357/ac80b8
  * White et al. (2019) White, R., Schaefer, G., Boyajian, T., et al. 2019, in American Astronomical Society Meeting Abstracts, Vol. 233, American Astronomical Society Meeting Abstracts #233, 259.41
  * Wit et al. (2012) Wit, E., Heuvel, E. v. d., & Romeijn, J.-W. 2012, Statistica Neerlandica, 66, 217, doi: https://doi.org/10.1111/j.1467-9574.2012.00530.x
  * Wittrock et al. (2022) Wittrock, J. M., Dreizler, S., Reefe, M. A., et al. 2022, AJ, 164, 27, doi: 10.3847/1538-3881/ac68e5
  * Zacharias et al. (2013) Zacharias, N., Finch, C. T., Girard, T. M., et al. 2013, AJ, 145, 44, doi: 10.1088/0004-6256/145/2/44
  * Zechmeister et al. (2018) Zechmeister, M., Reiners, A., Amado, P. J., et al. 2018, A&A, 609, A12, doi: 10.1051/0004-6361/201731483
  * Zicher et al. (2022) Zicher, N., Barragán, O., Klein, B., et al. 2022, MNRAS, 512, 3060, doi: 10.1093/mnras/stac614
  * Zuckerman (2001) Zuckerman, B. 2001, ARA&A, 39, 549, doi: 10.1146/annurev.astro.39.1.549


# Free Fermions Violate the Area Law For Entanglement Entropy

Robert C. Helling 11 Arnold Sommerfeld Center, Ludwig-Maximilians-Universität
München
Wolfgang Spitzer 22  Institut für Theoretische Physik, Universität Erlangen-
Nürnberg

###### Abstract

We show that the entanglement entropy associated to a region grows faster than
the area of its boundary surface. This is done by proving a special case of a
conjecture due to Widom that yields a surprisingly simple expression for the
leading behaviour of the entanglement entropy.

###### keywords:

Entanglement entropy, Area Law, LMU-ASC-08-10

##  1 Background

An interesting model for the area law of Bekenstein’s black-hole entropy is a
local quantum field theory for which one restricts all observations to the
complement of a compact spatial region $\Omega$. One should think of $\Omega$
as the interior of the horizon although one does not require $\Omega$ to have
any gravitational relevance. In fact, we will restrict our attention to a
region in the $n$-dimensional flat Euclidean space ${\mathbb{R}}^{n}$. Even if
the quantum field is globally in a pure state (e.g., the vacuum), the state
restricted to the complement of $\Omega$ will in general be mixed with finite
von Neumann entropy, called the entanglement entropy, $S(\Omega)$ [1, 2, 3].
For many types of quantum field theories, it was observed that in the semi-
classical limit of large $\Omega$, the entanglement entropy $S(\Omega)$ scales
like the area of $\partial\Omega$. In other words, if we rescale
$\Omega\subset{\mathbb{R}}^{n}$ by a factor $R$, the entanglement entropy,
$S(R\Omega)$, should asymptotically scale like $R^{n-1}$ for large $R$.

Recently, it was observed by Gioev and Klich [4, 5], that for free fermions at
zero temperature, however, the entanglement entropy of the ground state scales
like $R^{n-1}\log(R)$ for $R\to\infty$ based on a conjecture by Widom [6, 7].
They provide a lower bound on the entanglement entropy in terms of the trace
of a quadratic function of the restricted state. For the latter, we have
proved in [8] the leading asymptotic behaviour. This establishes indeed that
the entanglement entropy scales at least like $R^{n-1}\log(R)$, which violates
of the beforementioned area law scaling.

For notational clarity we will often write equalities that only hold
asymptotically for large $R$, that is, we will drop subleading terms that are
not central to our argument and only mention it in the text.

General arguments imply that for a pure state, the entanglement entropy of
this state with respect to $\Omega$ is the same as for the complement
${\mathbb{R}}^{n}\setminus\Omega$. For simplicity, we will restrict the state
to a compact region $\Omega$. The ground state, $\rho_{\Gamma}$, of a system
of non-interacting fermions in ${\mathbb{R}}^{n}$ is given in terms of the
Fermi surface at Fermi energy $\epsilon_{F}$: All one-particle states with
momentum
$\mathbf{p}\in\Gamma=\\{\mathbf{p}\in{\mathbb{R}}^{n}|E(\mathbf{p})\leq\epsilon_{F}\\}$
and energy111We will not discuss here assumptions on the dispersion relation
$E(\mathbf{p})$ but one may, of course, think of the example
$E(\mathbf{p})=\mathbf{p}^{2}$. We require that $\Gamma$ and $\Omega$ are
compact sets in ${\mathbb{R}}^{n}$ with sufficiently smooth
boundaries.$E(\mathbf{p})$ are occupied. The ground state $\rho_{\Gamma}$ is
then characterised by the one-particle Fermi projector, $P_{\Gamma}$, defined
in momentum space by the kernel
$P_{\Gamma}(\mathbf{p},\mathbf{p}^{\prime})=\chi_{\Gamma}(\mathbf{p})\delta(\mathbf{p}-\mathbf{p}^{\prime})$.
Here, and in the following, $\chi_{A}$ denotes the indicator function of a set
$A$. In position space, the kernel
$P_{\Gamma}(\mathbf{x},\mathbf{x}^{\prime})=\widehat{\chi_{\Gamma}}(\mathbf{x}-\mathbf{x}^{\prime})$
with
$\widehat{\chi_{\Gamma}}(\mathbf{x})=(2\pi)^{-n}\int_{\Gamma}d\mathbf{p}\,e^{i\mathbf{x}\cdot\mathbf{p}}$
being the inverse Fourier transform of $\chi_{\Gamma}$.

In order to restrict the state $\rho_{\Gamma}$ to the region $\Omega$ we
project the Fermi projector $P_{\Gamma}$ onto $L^{2}(\Omega)$ with
$Q_{\Omega}=\chi_{\Omega}$. This gives the reduced one-particle density matrix
$\varrho_{\Omega,\Gamma}=Q_{\Omega}P_{\Gamma}Q_{\Omega}$. The entanglement
entropy, $S(\Omega,\Gamma)$, of the many particle system in the ground state
$\rho_{\Gamma}$ restricted to the region $\Omega$ is then defined as the grand
canonical entropy of $\varrho_{\Omega,\Gamma}$, that is,
$S(\Omega,\Gamma)=\operatorname{{\mathrm{t}r}}\eta(\varrho_{\Omega,\Gamma})$
with $\eta(t)=-t\log{(t)}-(1-t)\log{(1-t)}$ for $0<t<1$. For details see [8,
Section 4]. We are interested here in the behaviour of this entropy for fixed
$\Gamma$ but large $\Omega$. To this end, we also fix $\Omega$ and study the
asymptotic behaviour of $S(R\Omega,\Gamma)$ as $R\to\infty$.

Our main result is the asymptotic computation of
$\operatorname{{\mathrm{t}r}}[\varrho_{R\Omega,\Gamma}(1-\varrho_{R\Omega,\Gamma})]$
as $R\to\infty$:

$\displaystyle\operatorname{{\mathrm{t}r}}[\varrho_{R\Omega,\Gamma}(1-\varrho_{R\Omega,\Gamma})]=\left(\frac{R}{2\pi}\right)^{n-1}\frac{\log(R)}{4\pi^{2}}\int_{\partial\Omega\times\partial\Gamma}dA(\mathbf{x})dA(\mathbf{p})\,|\mathbf{n}_{\mathbf{x}}\cdot\mathbf{n}_{\mathbf{p}}|,$
(1)

up to terms that grow slower in $R$. Here, $\mathbf{n}_{\mathbf{x}}$ denotes
the unit normal vector at $\mathbf{x}\in\partial\Omega$, $dA(\mathbf{x})$ is
the surface measure on $\partial\Omega$, and similarly for
$\mathbf{n}_{\mathbf{p}}$ and $dA(\mathbf{p})$.

Since $\eta(t)\geq\log(2)4t(1-t)$ for $0<t<1$, we obtain the asymptotic lower
bound on the entanglement entropy,

$S(R\Omega,\Gamma)\geq\frac{\log(2)}{\pi^{2}}\left(\frac{R}{2\pi}\right)^{n-1}\log(R)\int_{\partial\Omega\times\partial\Gamma}dA(\mathbf{x})dA(\mathbf{p})\,|\mathbf{n}_{\mathbf{x}}\cdot\mathbf{n}_{\mathbf{p}}|.$
(2)

Gioev and Klich conjectured in [5] that the exact scaling of
$S(R\Omega,\Gamma)$ is obtained if we replace the factor $\log{(2)}/\pi^{2}$
in (2) by $\frac{1}{12}$. This remains an open problem.

##  2 Computation of
$\operatorname{{\mathrm{t}r}}[\varrho_{R\Omega,\Gamma}(1-\varrho_{R\Omega,\Gamma})]$

The trace of $\varrho_{R\Omega,\Gamma}$ is simply equal to
$\left(\frac{R}{2\pi}\right)^{n}|\Omega||\Gamma|$, where $|\cdot|$ denotes the
$n$-dimensional Lebesgue volume. The trace of $\varrho_{R\Omega,\Gamma}^{2}$
equals

$\displaystyle\operatorname{{\mathrm{t}r}}(Q_{R\Omega}P_{\Gamma}Q_{R\Omega}P_{\Gamma})\\!\\!\\!$
$\displaystyle=$
$\displaystyle\\!\\!\\!\int_{R\Omega}\\!\\!\\!d\mathbf{x}\int_{R\Omega}\\!\\!\\!d\mathbf{x}^{\prime}\,|\widehat{\chi_{\Gamma}}(\mathbf{x}-\mathbf{x}^{\prime})|^{2}=\int_{R(\Omega-\Omega)}\\!\\!\\!d\mathbf{v}\,|\widehat{\chi_{\Gamma}}(\mathbf{v})|^{2}\,|R\Omega\cap(R\Omega-\mathbf{v})|\,.$
(3)

In the last line we have changed the variables $\mathbf{x}$ and
$\mathbf{x}^{\prime}$ to $\mathbf{u}=\mathbf{x}$ and
$\mathbf{v}=\mathbf{x}-\mathbf{x}^{\prime}$. Then we expand the volume
$|R\Omega\cap(R\Omega-\mathbf{v})|$ to first order in $\mathbf{v}$ (cf. [9,
Theorem 2.1]),

$|R\Omega\cap(R\Omega-\mathbf{v})|=R^{n}|\Omega|+R^{n-1}\\!\\!\\!\int_{\partial\Omega}\\!\\!\\!dA(\mathbf{x})\,\max(0,\mathbf{v}\cdot\mathbf{n}_{\mathbf{x}})+R^{n-2}O(|\mathbf{v}|^{2}).$
(4)

Let us first look at the contribution of $R^{n}|\Omega|$ to the trace of
$\varrho_{R\Omega,\Gamma}^{2}$. The function
$\mathbf{v}\mapsto\widehat{\chi_{\Gamma}}(\mathbf{v})$ decays like
$|\mathbf{v}|^{-(n+1)/2}$ for large $|\mathbf{v}|$, see (7). At the cost of an
order $R^{n-1}$-term we may therefore extend the $\mathbf{v}$-integration to
all of ${\mathbb{R}}^{n}$. By the Plancherel formula this integral gives
$(2\pi)^{-n}|\Gamma|$ and cancels with
$\operatorname{{\mathrm{t}r}}\varrho_{R\Omega,\Gamma}$. When integrated over
$\mathbf{v}$, the remainder term $R^{n-2}O(|\mathbf{v}|^{2})$ is also seen to
yield a term of the order $R^{n-1}$ by using again the above mentioned decay
of $\widehat{\chi_{\Gamma}}$.

Thus, our bound on the entanglement entropy will come from the second term in
(4). Here, we write
$\max(0,\mathbf{v}\cdot\mathbf{n}_{\mathbf{x}})=\chi_{[0,\infty)}(\mathbf{v}\cdot\mathbf{n}_{\mathbf{x}})\,\mathbf{v}\cdot\mathbf{n}_{\mathbf{x}}$
and use the Gauß Theorem so that

$(2\pi)^{n}\,\mathbf{v}\,\widehat{\chi_{\Gamma}}(\mathbf{v})=-i\int_{\partial\Gamma}dA(\mathbf{p})\,\mathbf{n}_{\mathbf{p}}\,e^{i\mathbf{v}\cdot\mathbf{p}}.$
(5)

So it remains to show that for $\mathbf{p}\in\partial\Gamma$,

$\Big{|}\int_{R(\Omega-\Omega)}d\mathbf{v}\,\chi_{[0,\infty)}(\mathbf{v}\cdot\mathbf{n}_{\mathbf{x}})\,\widehat{\chi_{\Gamma}}(-\mathbf{v})\,e^{i\mathbf{v}\cdot\mathbf{p}}+(2\pi
i)^{-1}\operatorname{{\mathrm{s}gn}}({\bm{n}}_{\mathbf{x}}\cdot{\bm{n}}_{\mathbf{p}})\log{(R)}\Big{|}=o(R)\,.$
(6)

Let us now consider the function
$\mathbf{v}\mapsto\widehat{\chi_{\Gamma}}(-\mathbf{v})$ in detail. We use the
representation from (5), that is,
$(2\pi)^{-n}\widehat{\chi_{\Gamma}}(-\mathbf{v})=\frac{i\mathbf{v}}{|\mathbf{v}|^{2}}\cdot\int_{\partial\Gamma}dA(\mathbf{p}^{\prime})\,\mathbf{p}^{\prime}\,e^{-i\mathbf{v}\cdot\mathbf{p}^{\prime}}$.
Then we introduce a coordinate system where $\mathbf{v}=(0,\ldots,0,v)$ and
where the boundary $\partial\Gamma$ is locally written as the graph of a
function $f\colon U\subset{\mathbb{R}}^{n-1}\to{\mathbb{R}}$, that is,
$\mathbf{p}^{\prime}=(\mathbf{t},f(\mathbf{t}))$ and
$dA(\mathbf{p}^{\prime})=\sqrt{1+|\nabla f|^{2}}d\mathbf{t}$. The unit normal
vector is
${\bm{n}}_{\bm{p}^{\prime}}=\operatorname{{\mathrm{s}gn}}(\mathbf{v}\cdot\mathbf{p}^{\prime})(-\nabla
f,1)/{\sqrt{1+|\nabla f|}}$. Then,

$\int_{f(U)}dA(\mathbf{p}^{\prime})\,({\bm{n}}_{\bm{p}^{\prime}})_{n}\,e^{-i\mathbf{v}\cdot\mathbf{p}}=-\frac{1}{v}\int_{U}d\mathbf{t}\,\operatorname{{\mathrm{s}gn}}(f(\mathbf{t}))\,e^{-ivf(\mathbf{t})}.$

In order to find the leading asymptotic behaviour of this
$\mathbf{t}$-integral for large $v$ we apply the method of stationary phase.
Let
${\mathbf{k}_{a}}={\mathbf{k}_{a}}(\mathbf{v})=(\mathbf{t}_{a}(\mathbf{v}),f(\mathbf{t}_{a}(\mathbf{v})))$
be the collection of all stationary points of such local functions $f$, that
is, $\nabla f(\mathbf{t}_{a})=0$; in other words, the points
${\mathbf{k}_{a}}\in\partial\Gamma$ are such that the unit normal vector
$\bm{n}_{{\mathbf{k}_{a}}}$ at ${\mathbf{k}_{a}}$ is parallel to $\mathbf{v}$.
Thus,

$\displaystyle\widehat{\chi_{\Gamma}}(-\mathbf{v})=-i(2\pi
v)^{-(n+1)/2}\sum_{{\mathbf{k}_{a}}}\frac{\operatorname{{\mathrm{s}gn}}(\mathbf{v}\cdot{\mathbf{k}_{a}})}{\sqrt{|\det
f_{ij}(\mathbf{t}_{a})}|}\,e^{-i\mathbf{v}\cdot{\mathbf{k}_{a}}-i\frac{\pi}{4}\operatorname{{\mathrm{s}gn}}(f_{ij}(\mathbf{t}_{a}))}+o(v^{-(n+1)/2}).$
(7)

Here, $f_{ij}(\mathbf{t}_{a})$ denotes the Hessian of $f$ at $\mathbf{t}_{a}$,
$\operatorname{{\mathrm{s}gn}}(f_{ij}(\mathbf{t}_{a}))$ the signum of this
Hessian. The determinant, $\det f_{ij}(\mathbf{t}_{a})$, equals the Gaußian
curvature of $\partial\Gamma$ at ${\mathbf{k}_{a}}$.

Using (7) we return to the oscillatory integral in (6), and employ once more
the method of stationary phase. The composite phase from (5) and (7) is equal
to $\mathbf{v}\cdot(\mathbf{p}-{\mathbf{k}_{a}}(\mathbf{v}))$. Next, we
introduce generalised spherical coordinates for $\mathbf{v}$ as
$\mathbf{v}=\rho(\mathbf{u},h(\mathbf{u}))$, where the map $h\colon
V\subset{\mathbb{R}}^{n-1}\to{\mathbb{R}}$ locally parametrises the boundary
$\partial(\Omega-\Omega)\ni(\mathbf{u},h(\mathbf{u}))$ and $\rho\in[0,R]$ is a
radial coordinate. The stationary point ${\mathbf{k}_{a}}$ is now a function
of $\mathbf{u}$. We have the freedom to assume that
$\mathbf{n}_{\mathbf{p}}=(0,\ldots,0,1)$ such that
${\mathbf{k}_{a}}((0,h(0)))=\mathbf{p}$ for one index $a$ and that $h$ has its
extremum at the origin $\mathbf{u}=0$. Instead of integrating $\rho$ from 0 to
$R$ we will actually integrate $\rho$ only over $[1,R]$ thereby making an
irrelevant error independent of $R$.

To express ${\mathbf{k}_{a}}(\mathbf{v})$ as a function of $\mathbf{u}$ we
equate $\mathbf{v}/v=\bm{n}_{{\bm{k}}_{a}(\mathbf{v})}$, that is,

$\frac{(\mathbf{u},h(\mathbf{u}))}{\sqrt{\mathbf{u}^{2}+h(\mathbf{u})^{2}}}=\operatorname{{\mathrm{s}gn}}(\mathbf{v}\cdot\mathbf{n}_{\mathbf{k}_{a}})\frac{(-\nabla_{\mathbf{t}}f(\mathbf{t}),1)}{\sqrt{1+|\nabla_{\mathbf{t}}f(\mathbf{t})|^{2}}}.$
(8)

Taking derivatives and evaluating at $\mathbf{u}=0$, we find $\frac{\partial
t_{j}}{\partial u_{i}}(0)=-f_{ij}^{-1}({\mathbf{k}_{a}}(0))/h(0)$, where
${\mathbf{k}_{a}}(0)$ is short for ${\mathbf{k}_{a}}(0,h(0))$ and
$f_{ij}^{-1}$ is the matrix inverse of the Hessian of $f$. With this, we can
expand the phase of the remaining $\mathbf{v}$-integral to second order as

$\mathbf{v}\cdot(\mathbf{p}-{\mathbf{k}_{a}}(\mathbf{v}))=\rho
h(0)(\mathbf{p}-{\mathbf{k}_{a}}(0))_{n}+\rho\frac{f_{ij}^{-1}({\mathbf{k}_{a}}(0))}{2h(0)}u_{i}u_{j},$
(9)

The volume element is given by $d\mathbf{v}=\rho^{n-1}h(\mathbf{u})d\rho
d\mathbf{u}$. The stationary phase integral over the $n-1$ coordinates
$\mathbf{u}$ yields $(2h(0)/\rho)^{(n-1)/2}\sqrt{|\det f_{ij}^{-1}|}\exp(i\rho
h(0)(\mathbf{p}-{\mathbf{k}_{a}}(0))_{n})+i\frac{\pi}{4}\operatorname{{\mathrm{s}gn}}(f_{ij}(\mathbf{t}_{a}))$.
This, surprisingly, reduces the remaining $\rho$-integral to
$\int_{1}^{R}d\rho\;e^{i\rho h(0)(\mathbf{p}-{\mathbf{k}_{a}}(0))_{n}}/\rho$.
As noted above, for one index $a$, the exponent vanishes as
$\mathbf{p}={\mathbf{k}_{a}}(0)$ and the integral is the desired $\log(R)$.
If, however, $\mathbf{p}\neq{\mathbf{k}_{a}}(0)$ the integral is bounded for
large $R$ and does not contribute to leading order.

Collecting all terms we have thus proved the lower bound (1) which grows
faster than the area law scaling $R^{n-1}$ by a factor of $\log(R)$. Note that
the stationary phase integrals localises $\mathbf{v}$ and
$\mathbf{p}^{\prime}$ such that there are only contributions from
$\mathbf{p}$, $\mathbf{n}_{\mathbf{v}}$, ${\bm{n}}_{\bm{p}^{\prime}}$ all
being parallel.

##  3 Discussion

Above, we presented a lower bound for the entanglement entropy which violates
an $R^{n-1}$ area law scaling. The expression (2) is in terms of integrals
over the boundaries $\partial\Omega$ and $\partial\Gamma$ and is thus still a
“boundary effect”.

We find it most curious that although we have used stationary phase methods,
which in general depend on (the existence of) second order derivatives at the
stationary points, all these curvature terms involving $f_{ij}$ and $h_{ij}$
eventually cancel out and the integrand in the bound does not contain
derivatives. However, one expects [4, 10] that a fractal boundary
$\partial\Omega$ of dimension $n-1+\alpha$ with $0<\alpha<1$ leads to a
scaling of the entanglement entropy of at least of the order $R^{n-1+\alpha}$
and, presumably, without a $\log{(R)}$ correction.

For $n=1$, if $\Omega$ and $\Gamma$ are a disjoint union of $k$ and $\ell$
compact intervals of finite length respectively, then our lower bound (2) for
the entanglement entropy gives $\log(R)\log(2)4k\ell/\pi^{2}$. In this one-
dimensional case, the precise scaling has been proved, namely,
$\log(R)k\ell/3$. If we then consider the hypercubes, say
$\Omega=\Gamma=[0,1]^{n}$, it is not difficult to derive the exact asymptotic
scaling of $S(R\Omega,\Gamma)$ to be $(R/(2\pi))^{n-1}\log{(R)}n^{2}/3$, which
is in agreement with the conjecture by Gioev and Klich mentioned at the end of
Section 1. Our method of proof requires that the surfaces $\partial\Omega$ and
$\partial\Gamma$ are $C^{3}$. Hence, hypercubes are not included. For a
$C^{3}$ surface, $\partial\Gamma$, it was crucial that
$\widehat{\chi_{\Gamma}}(\mathbf{v})$ behaves like $|\mathbf{v}|^{-(n+1)/2}$
for large $|\mathbf{v}|$. This is not the case for a non-smooth surface such
as the hypercube, where
$\widehat{\chi_{[0,1]^{n}}}(\mathbf{v})=\prod_{i=1}^{n}\sin(v_{i})/(\pi
v_{i})$.

It should be noted as well that the discontinuity of $\chi_{\Gamma}$ is
crucial for the decay of its Fourier transform. For example, for the
equilibrium state at positive temperature, the entanglement entropy scales
like the volume $R^{n}$, see [5]. We did not have to introduce an ultraviolet
regulator since momentum integrations are limited to the compact region
$\Gamma$.

## Acknowledgments

We are grateful to Hajo Leschke with whom the work in [8] was performed on
which is text is based. Furthermore, RCH would like to thank the Elitenetwork
of Bavaria for financial support and Jacobs University Bremen, where this work
was started. We also thank Urs Frauenfelder for discussions.

## References

  * [1] M. Srednicki, Phys. Rev. Lett. 71(5), 666–669 (1993).
  * [2] L. Bombelli, R. K. Koul, J. Lee, and R. D. Sorkin, Phys. Rev. D 34(2), 373–383 (1986).
  * [3] J. Eisert, M. Cramer, and M. B. Plenio, Area laws for the entanglement entropy - a review, 2008, arXiv.org:0808.3773.
  * [4] D. Gioev, Int. Math. Res. Not. 2006(O95181), 95181–23 (2006).
  * [5] D. Gioev and I. Klich, Phys. Rev. Lett. 96(10), 100503 (2006).
  * [6] H. Widom, Trans. AMS 94(1), 170–180 (1960).
  * [7] H. Widom, J. Funct. Anal. 88, 166–193 (1990).
  * [8] R. C. Helling, H. Leschke, and W. L. Spitzer, A special case of a conjecture by Widom with implications to fermionic entanglement entropy, 2009, arXiv.org:0906.4946.
  * [9] R. Roccaforte, Trans. AMS 285(2), 581–602 (1984).
  * [10] M. Fannes, B. Haegeman, and M. Mosonyi, J. Math. Phys. 44(12), 6005–6019 (2003).


# Lighting, Reflectance and Geometry Estimation from 360∘ Panoramic Stereo

Junxuan Li1,2 Hongdong Li1 Yasuyuki Matsushita3
1Australian National University 2Data61-CSIRO  Australia 3Osaka University
Japan
{junxuan.li<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We propose a method for estimating high-definition spatially-varying lighting,
reflectance, and geometry of a scene from 360∘ stereo images. Our model takes
advantage of the 360∘ input to observe the entire scene with geometric detail,
then jointly estimates the scene’s properties with physical constraints. We
first reconstruct a near-field environment light for predicting the lighting
at any 3D location within the scene. Then we present a deep learning model
that leverages the stereo information to infer the reflectance and surface
normal. Lastly, we incorporate the physical constraints between lighting and
geometry to refine the reflectance of the scene. Both quantitative and
qualitative experiments show that our method, benefiting from the 360∘
observation of the scene, outperforms prior state-of-the-art methods and
enables more augmented reality applications such as mirror-objects insertion.

##  1 Introduction

Intrinsic decomposition of scene properties is a long-standing and essential
task in computer vision. It includes the estimation of lighting, geometry, and
reflectance of an arbitrary scene. Inferring the above properties of a scene
enables us to develop various novel applications, especially in augmented
reality, such as object insertion and scene modification. It is a challenging
and extremely under-constrained problem because of the complexity of light
transportation on complicated geometry and various material reflectances in
real-world. The majority of previous methods used perspective cameras for
solving this problem. However, the limited field of view of a perspective
camera results in the lack of observation of the entire scene, making this
inverse problem even more intractable.

To overcome the problem, we propose a method that uses a pair of 360∘ images
under equirectangular projection as input. Our method utilizes this input to
bring up many advantages that the perspective approach does not. Firstly, the
360∘ image captures the entire scene at once, offering us an adequate
observation for lighting estimation. Secondly, the stereo input naturally
encodes the depth information, making the geometry estimation possible.
Furthermore, by jointly leveraging the physical constraints between lighting
and geometry, the reflectance can be revealed. Figure. 1 illustrates the
camera setting for capturing 360∘ stereo input and the estimated results of
our method.

Figure 1: Our system consists of two 360∘ cameras in a top-bottom setting. We
present the predicted reflectance and surface normal of the scene at the
bottom-left of this figure. The bottom-right are virtual mirror-objects
relighted by our illumination map at two different locations. Our method can
recognize the geometric difference among 3D locations and preserve high-
frequency information of the lighting, enabling us to insert mirror-objects
around the entire scene with appealing reflection effects.

Leveraging 360∘ stereo input, we achieve two strengths in lighting estimation:
(i) our lighting is spatially-varying and 3D coherent, which means the
lighting will be different and changing smoothly for different 3D locations
condition on the scene geometry; (ii) our lighting is in high-definition,
which means it is generated in high-resolution and contains high-frequency
details of the scene to enable mirror-like objects insertion. The lighting
estimated by perspective methods rarely has these properties. They either
estimate the lighting globally [20, 16] or per-pixel-individually [24, 11,
18], having no consistency between different locations. In addition, because
of the limited field of view in perspective images, prior works have
difficulties in ‘inferring’ the unseen regions of the scene in high-
definition. But, our method naturally avoids this problem.

The reflectance estimation is also an ill-posed problem under the perspective
cameras [4]. However, with the 360∘ stereo images, the input contains more
information about the scene’s lighting and geometry, giving us substantial
leads and more constraints to infer both reflectance and normal.

In this paper, we present a method that utilizes the strengths of the 360∘
input to jointly estimate the high-definition and spatially-varying lighting,
reflectance, and geometry of the entire scene. Our contributions are:

  1. 1.

A near-field environment light that can generate spatially-varying and 3D
coherent high-definition illumination maps when given any 3D location within
the scene.

  2. 2.

A deep learning model that can estimate the reflectance and surface normal of
the entire scene.

  3. 3.

A rendering and refinement model that leverages the physical constraints
between lighting and geometry to jointly estimate a finer reflectance.

##  2 Related Work

#### Lighting Estimation

Inferring the lighting of the environment enables us to render a synthetic
object into real-world. Debevec [7] used a light probe to measure an HDR
illumination map. Though a mirror-ball is accurate in lighting estimation, it
is not suitable for estimating lighting at different locations. Many recent
studies prefer spatially-varying lighting for indoor scenarios, which predicts
different lighting given different locations within the scene. [10, 9] use
deep learning models to explicitly predict the location and intensity of the
primary light sources; [11, 28] adopt the spherical harmonics lighting model
for fast estimation; [27] assumes indoor objects located in a six-faces-box;
[25] uses a deep model to represent the scene in lighting volume; [24] warps
the seen scene points into the target illumination map based on geometry
estimation, then per-pixel-independently completes the unseen region by a
neural network.

Due to the limited field of view of perspective images, all the previous
methods either use simplified lighting models; or simplify environment models;
or hallucinate the unobserved scene’s geometry and appearance, leading to lost
or inconsistent lighting. It is also why previous works are unsuitable for
inserting mirror-objects, which requires high-definition illumination maps and
detailed lighting from the scene.

#### Intrinsic Image Decomposition

The studies in the intrinsic image decomposition can be divided into two folds
by its input types: the object-scale and the scene-scale. An object-scale
intrinsic decomposition method usually assumes global illumination. The
problem can be solved using carefully designed handcrafted priors [3], or
recently developed deep learning models [23].

For scene-scale decomposition problem, Barron  _et al_ [2] takes RGB-D as
input to estimate the reflectance, lighting and normal by applying handcrafted
priors. [5] proposes an $\ell_{1}$ norm for constraint the reflectance to be
piecewise flattening. The prevailing deep learning methods also show its
effectiveness in this task: [21] proposes a CNN trained by a synthetic
dataset. The subsequent studies [8, 19, 30] enlarge the training datasets and
enrich the designs of network architectures and loss functions. Li  _et al_
[18] proposes a framework that jointly reasons shape, lighting and SVBRDF from
a single perspective image. However, they simplify the lighting model and fail
to consider geometry constraint between different locations within the same
scene. Our method, taking the 360∘ input to fully observe the lighting and
geometry of the entire scene, estimates the scene-level reflectance, normal,
and lighting with physical constraints.

#### 360∘ Panoramic Imaging

Many studies focus on geometry estimation by 360∘ panoramic images. Li  _et
al_ [17] captures the 360∘ stereo by fixing and rotating two concentric
cameras for depth estimation. Kim and Hilton [14] proposes a 3D mesh modeling
method using multiple pairs of spherical images captured by a line scan camera
at different locations. Recently, consumer-level 360∘ cameras are used for
depth estimation from the video clip [13] and a stereo-pair [26]. Other than
depth estimation, Banterle  _et al_ [1] takes an annotated high dynamic range
(HDR) panoramic environment map for local illumination recovery. Our work
fills a literature gap by estimating the lighting, geometry, and reflectance
from the 360∘ stereo input.

##  3 Method Overview

Figure 2: System overview. We first estimate the depth of a stereo 360∘ input.
Then a near-field environment light is reconstructed from the input images and
estimated depth for estimating illumination map later. We parallelly apply an
RN-Net on inputs and estimated depth for reflectance and normal estimation.
Finally, we use the illumination map and normal map to render the shading,
which then jointly refines the reflectances.

Our method uses two 360∘ images taken from a 360∘ stereo camera to estimate
the target scene’s lighting, geometry, and reflectance. It consists of four
modules, as illustrated in Fig. 2. The first module shows our 360∘ camera
setting and depth estimation from the stereo input (see Fig. 2 gray part, Sec.
4.1). The second module is the lighting estimation that computes a near-field
environment light from the input images and estimated depth. We treat each
scene point as a light source in the 3D space and build a near-field
environment light. With this near-field environment light, given any 3D
location within the scene, our method can reconstruct the corresponding
illumination map in high-definition for object insertion and relighting (see
Fig. 2 green part, Sec. 4.2). The third module is reflectance and normal
estimation. It is a deep learning model, named RN-Net, for estimating the
reflectance map and surface normal of the entire scene. To preserve the high-
frequency information from the input, we take the input with the resolution to
be $512\times 1024$. To tackle the large input size, we proposed a pyramid
structure for RN-Net to estimate the reflectance and surface normal from small
to large (see Fig. 2 orange part, Sec. 4.3).

With these three modules, our method can obtain the scene’s lighting,
reflectance, and geometry at a certain granularity. To obtain more refined
estimates of shadings and reflectances, we use the fourth module that performs
physically-based rendering and refinement that aims at minimizing the
reconstruction loss with the input (see Sec. 4.4 blue part).

##  4 Proposed Method

###  4.1 Camera Setting and Depth Estimate

Our imaging setup consists of two 360∘ cameras in a top-bottom setting as
shown in Fig. 3. A similar setup has also been used by [26]. This top-bottom
arrangement ensures only the vertical disparities between two 360∘ images. The
captured 360∘ image is in equirectangular projection, allowing us to capture
the entire scene at once, while the stereo images enable us to estimate the
geometry at a low cost.

Figure 3: The system comprises two Samsung Gear 360∘ cameras. Once calibrated,
a point $\mathbf{x}$ in the scene will be aligned in the 360∘ images only with
a vertical angular disparity $\Delta\theta$.

As illustrated in Fig. 3, for a point
$\mathbf{x}=[x,y,z]^{\top}\in\mathbb{R}^{3}$ in the 3D space, let its
projection on the top and bottom images be
$\mathbf{u}_{t}=[u_{t},v_{t}]^{\top}\in\mathbb{R}^{2}$ and
$\mathbf{u}_{b}=[u_{b},v_{b}]^{\top}\in\mathbb{R}^{2}$, respectively. When the
two cameras are aligned vertically with the same $v$-axis, the displacement
$\Delta\theta\in\mathbb{R}$ of $\mathbf{x}$ on two images can be given by

$\displaystyle\Delta\theta=\theta_{b}-\theta_{t}=\frac{\pi}{h}\left(v_{b}-v_{t}\right).$
(1)

$h$ is the height of image, $v_{b}$ and $v_{t}$ are the $v$-coordinates of the
projected image points $\mathbf{u}_{t}$ and $\mathbf{u}_{b}$ respectively. The
distance between $\mathbf{x}$ and the top camera, $d_{t}$ is given by
triangulation as:

$\displaystyle
d_{t}=b\left(\frac{\sin{\theta_{t}}}{\tan{\Delta\theta}}+\cos{\theta_{t}}\right),$
(2)

where $b\in\mathbb{R}_{+}$ is the baseline between the two cameras.

Therefore, with a stereo matching method to find matchings along the vertical
direction, we can obtain the given point’s angular disparity and depth by the
above Eqs. (1) and (2). In this paper, we use a recent CNN-based stereo
method, 360SD-Net [26] for depth estimation.

###  4.2 Lighting Estimation

To preserve the high-frequency lighting and geometry information of the 360∘
environment, we propose the second module to reconstruct a near-field
environment light for estimating the high-definition spatially-varying
illumination map given an arbitrary 3D location in the scene.

#### Near-field Environment Light

We assume that all the observed scene materials are diffuse when estimating
the lighting. It is also a convention, and a good approximation, to treat
scene material as diffuse when doing lighting estimation [27, 30].

For simplicity, we use the top camera as the reference camera to omit the
subscript of the notations. Following the notation above,
$(\mathbf{c},\mathbf{u},d)$ represents a pixel in the 360∘ image, where
$\mathbf{c}\in\mathbb{R}^{3}$ is the RGB observation of a point in the camera,
_i.e_ , the pixel value with three channels; and $\mathbf{u}$ is its position
on the reference camera with coordinates $[u,v]^{T}$; $d\in\mathbb{R}_{+}$ is
the estimated depth from the previous step.

We define $f(\cdot)$ as the projection function between a pixel in 360∘ image
under the equirectangular projection to the world coordinates:
$(\mathbf{c},\mathbf{x})=f(\mathbf{c},\mathbf{u},d).$ Applying the projection
$f(\cdot)$ to all pixels on the 360∘ image will give us the representation of
the near-field environment light $\mathbf{E}\in\mathbb{R}^{n\times 6}$, where
$n$ is the number of points, as:

$\displaystyle\mathbf{E}=\left\\{(\mathbf{c}_{i},\mathbf{x}_{i})\;|\;i\in\text{pixels}\right\\}.$
(3)

Each point in the scene is treated as a light source with intensity
$\mathbf{c}$ and its position $\mathbf{x}$.

#### Illumination Map

Given an arbitrary 3D point $\mathbf{x}^{\prime}$ in the scene, we re-project
the near-field environment light $\mathbf{E}$ to the new point to generate an
illumination map for $\mathbf{x}^{\prime}$ by:

$\displaystyle\left\\{(\mathbf{c}^{\prime}_{i},\mathbf{u}^{\prime}_{i},d^{\prime}_{i})\;|\;i\in\text{pixels}\right\\}=g(\mathbf{E},\mathbf{x}^{\prime}),$
(4)

where $g(\cdot)$ projects the coordinates from 3D to 2D illumination map.
However, due to the sparsity of the near-field environment light, the
reconstructed illumination map contains pixels without any projection of the
lights, while some pixels may have many lights fall into. Hence, we need to
refine the illumination map to sort out empty and overlapped pixels. The
refinement function $r(\cdot)$ is defined as that for the position
$\mathbf{u}^{\prime}_{i}$ with many projected lights, only select the light
with the minimum depth value $d^{\prime}$. This manner simulates the occlusion
effect in the real world, where one may obstruct another light. We use the
nearest interpolation method to extrapolate those empty pixels. In summary,
our reconstructed illumination map is given by applying functions $g(\cdot)$
and $r(\cdot)$ in sequence. To simplify, we use function $w$ to denote a
composition of functions $r\circ g$. The reconstructed illumination map
$\mathbf{M}^{\prime}\in\mathbb{R}^{n\times 6}$ at position
$\mathbf{x}^{\prime}$ is:

$\displaystyle\mathbf{M}^{\prime}=\left\\{(\mathbf{c}^{\prime}_{i},\mathbf{u}^{\prime}_{i},d^{\prime}_{i})\;|\;i\in\text{pixels}\right\\}$
$\displaystyle=w(\mathbf{E},\mathbf{x}^{\prime}).$ (5)

Figure 4: The architecture of RN-Net. It follows a U-Net structure. Please see
supplementary material for more details.

###  4.3 Reflectance and Normal Estimation

The third module is a convolutional neural network, named RN-Net, for
predicting the reflectance and surface normal of the entire scene at a large
resolution.

As shown in Fig. 2, it takes the reference 360∘ image and estimated depth as
the input to infer the reflectance and normal. Figure. 4 shows detailed
network architecture. It processes the input with four encoder-blocks. Each
block consists of two convolutional layers and a short skip connection between
input and output and will down-sample the feature size into half, similar to
ResNet [12]. Then, for normal and reflectance estimation, we apply another
four decoder-blocks for each task. The decoder is similar to the encoder but
will up-sample the feature size twice larger at the output. Besides the short
skip connection within each block, we also add a long skip connection between
each layer, as shown in Fig. 4.

To tackle the large input size of the 360∘ images, we apply a pyramid
structure to the RN-Net. The input is first scaled to a different size, then
feed into different RN-Net for training. In the end, we up-sample all the
results to the original resolution and add them together to get the
reflectance and normal estimation. In this paper, our network takes RGB image
with estimated depth as the input $\mathbf{I}_{1}\in\mathbb{R}^{512\times
1024\times 4}$ and scales it to four times smaller to be
$\mathbf{I}_{\frac{1}{4}}\in\mathbb{R}^{128\times 256\times 4}$. The overall
structure can be illustrated as:

$\displaystyle\left\\{\begin{array}[]{lcl}\mathbf{R}_{1},\mathbf{N}_{1}&=&\Phi_{1}(\mathbf{I}_{1}),\\\
\mathbf{R}_{\frac{1}{4}},\mathbf{N}_{\frac{1}{4}}&=&\Phi_{\frac{1}{4}}(\mathbf{I}_{\frac{1}{4}}),\\\
\mathbf{R},\mathbf{N}&=&\text{up}(\mathbf{R}_{\frac{1}{4}})+\mathbf{R}_{1},\;\text{up}(\mathbf{N}_{\frac{1}{4}})+\mathbf{N}_{1},\end{array}\right.$
(9)

where $\mathbf{R}_{1}$, $\mathbf{N}_{1}$, $\mathbf{R}_{\frac{1}{4}}$, and
$\mathbf{N}_{\frac{1}{4}}$ are the reflectance map and normal map at the
original size and $\frac{1}{4}$ size respectively;
$\Phi_{1}(\cdot)and\Phi_{\frac{1}{4}}(\cdot)$ are the RN-Nets in different
scales; $\text{up}(\cdot)$ operator represents the bilinear up-sampling;
$\mathbf{R},\mathbf{N}$ are the estimated results of this pyramid structure.

We use a scale-invariant loss for training reflectance:

$\displaystyle\mathcal{L}_{\mathbf{R}}=\left\|s\mathbf{R}-\mathbf{R^{*}}\right\|_{2}^{2}+\left\|s\nabla\mathbf{R}-\nabla\mathbf{R^{*}}\right\|_{1},$
(10)

where $\mathbf{R^{*}}$ is the ground truth of the reflectance map; $s$ is a
scale factor computed by applying least square regression between $\mathbf{R}$
and $\mathbf{R^{*}}$; $\nabla\mathbf{R}$ is the gradient of $\mathbf{R}$. For
the training of surface normal, we define the loss as:

$\displaystyle\mathcal{L}_{\mathbf{N}}=-\mathbf{N}^{T}\mathbf{N^{*}}+\left\|\nabla\mathbf{N}-\nabla\mathbf{N^{*}}\right\|_{1},$
(11)

where the $\mathbf{N^{*}}$ is the ground truth of surface normal. Here, the
first term of $\mathcal{L}_{\mathbf{N}}$ is the cosine loss between normal,
while the second is gradient loss. Both of the reflectance and normal loss
adopt the gradient loss, which makes results piecewise smooth, as shown in
many previous works [19, 8, 30]. The total training loss is
$\mathcal{L}=\mathcal{L}_{\mathbf{R}}+\mathcal{L}_{\mathbf{N}}$.

Figure 5: The 360∘ input and results from our method on the synthetic scene
‘barbershop.’ We use our estimated illumination maps to virtually relight
mirror-objects at four different locations within the scene. The last two rows
present the corresponding estimated illumination maps and ground truth at each
location. Please notice how the inserted ‘cow’ at location ‘A’ and ‘B’
consistently reflect the surroundings, demonstrating the spatially-coherency
of our lighting. The changes between the highlighted areas of the objects
inserted at ‘X’ and ‘Y’ also indicate the variance of our lighting at
different locations. By comparing with the ground truth, our method can
reconstruct the illumination map with high-frequency details and accurately
reflect the geometry of the scene.

###  4.4 Rendering and Refinement

The 360∘ images observe the entire scene at once to provide lighting and
geometry of the environment as constraints for solving this ill-posed
reflectance estimation problem. Our last module utilizes the estimated
illumination maps and surface normal from previous steps, incorporating
physical insights, to render and refine the shading and reflectance map.

#### Shading Rendering

For each pixel with index $i$ and its image location $\mathbf{u}_{i}$ in the
360∘ image $\mathbf{I}$, we first compute the corresponding 3D location
$\mathbf{x}_{i}$ and estimate an illumination map centered at the 3D location
by $\mathbf{M}_{i}=w(\mathbf{E},\mathbf{x}_{i})$. As we assume all the scene
material to be diffuse, the $i$-th pixel’s shading value
$\mathbf{S}_{i}\in\mathbb{R}^{3}$ can be computed by the integration of the
illumination map and dot product between light direction and surface normal.
Hence, the $i$-th pixel’s shading value is given by:

$\displaystyle\mathbf{S}_{i}=\sum_{j\in\mathbf{M}_{i}}\mathbf{c}_{j}\max\left({\mathbf{l}_{j}}^{T}\mathbf{N}_{i},0\right),$
(12)

where $\mathbf{c}_{j}$ and $\mathbf{l}_{j}\in\mathbb{R}^{3}$ are the light
intensity and light direction of light at $j$-th pixel of illumination map
$\mathbf{M}_{i}$, respectively; and $\mathbf{N}_{i}\in\mathbb{R}^{3}$ is the
estimated surface normal of the $i$-th pixel.

To render the whole shading map $\mathbf{S}\in\mathbb{R}^{512\times 1024\times
3}$, we iterate every pixel $i$ in the input image, reconstruct the
corresponding illumination map $\mathbf{M}_{i}$ and compute every shading
value by Eq. (12).

#### Refinement Using TV Regularization

The input RGB image $\mathbf{I}$ can now be reconstructed by taking the
product between shading map $\mathbf{S}$ and reflectance map $\mathbf{R}$.
However, errors and noise will inevitably occur at every step of the process.
Here, we refine the results from previous steps by optimizing a target energy
function:

$\displaystyle\mathcal{L}_{TV}=||\mathbf{I}-s\mathbf{R}\odot\mathbf{S}||_{2}^{2}+\lambda_{1}||\nabla\mathbf{R}||_{1}+\lambda_{2}||\nabla\mathbf{S}||_{2}^{2},$
(13)

where $\odot$ denotes the Hadamard product; $s$ is a scale factor computed by
applying least square regression between $\mathbf{I}$ and
$\mathbf{R}\odot\mathbf{S}$. Our energy loss takes a similar form to total
variation regularization [22] and serves in a similar task, _i.e_ , to
minimize noise and reject outliers.

The first term can be interpreted as the reconstruction loss between the input
and our reconstructed input image. The second term, applying $\ell_{1}$ loss
on the gradient of reflectance map, aims to constrain reflectance map being
piecewise smooth. While the third term, $\ell_{2}$ loss on the gradient of
shading, aims to suppress the abrupt changes in shading effects. In this
paper, we set the hyper parameters as $\lambda_{1}=0.1,\lambda_{2}=10$.

We use the reflectance map $\mathbf{R}$ and shading map $\mathbf{S}$ from the
previous step as the initialization. Then update the two maps based on the
gradient descend algorithm.

$\displaystyle\mathbf{S}^{\prime}=\mathbf{S}-\gamma\dfrac{\partial\mathcal{L}_{TV}}{\partial\mathbf{S}},\quad\mathbf{R}^{\prime}=\mathbf{R}-\gamma\dfrac{\partial\mathcal{L}_{TV}}{\partial\mathbf{R}}.$
(14)

##  5 Implementation Details and Dataset

###  5.1 Training Details

#### Training Data for RN-Net

We use a public dataset Structured3D [29] as our training data for RN-Net. The
Structured3D is a synthetic indoor dataset with rich layouts and interior
designs. It has $21835$ images of different rooms with $512\times 1024$
resolution in equirectangular projection. It also provides us with ground
truth surface normal and reflectance map for supervised learning. The only
inconsistency between this dataset and our requirements is that it does not
provide stereo inputs. To overcome this, when training RN-Net, we take the
ground truth depth map and added with smoothed Gaussian noise as input to
simulate the errors in depth estimation.

#### Training of RN-Net

Our model is trained from scratch with Adam [15] optimizer. We first train the
small scale of RN-Net $\Phi_{\frac{1}{4}}$ with batch size to be $32$ and the
learning rate to be $10^{-3}$. After $100$ epochs, we combine both RN-Nets to
further train it with the large scale images. Batch size is $12$ for another
$50$ epochs in the large size. The learning rate starts from $10^{-3}$, then
being half every $10$ epochs. The RN-Net is implemented by PyTorch; trained on
a single GPU NVIDIA GTX 1080Ti for around $20$ hours.

#### Total Variation Refinement

For optimization on $\mathcal{L}_{TV}$, we adopt Adam [15] optimizer with
fixed learning rate $10^{-4}$. In addition, to avoid overfitting, we apply a
$\ell_{2}$ loss between the initialization and optimized results, acting as
the weight-decay. The model is implemented in PyTorch and converges after
$1000$ iterations in 2 minutes.

###  5.2 Testing Data

We prepare two datasets for testing. A synthetic dataset rendered by Blender
[6], and a real dataset captured by our 360∘ cameras.

Synthetic data. We create a synthetic dataset by Blender to simulating the
complex light transportation of an indoor scene. Each scene consists of two
vertically-aligned cameras to capture the whole environment in an
equirectangular projection. The renderer provides us with the ground truth of
reflectance, normal, and illumination map for quantitative evaluation. We
showcase some results from our synthetic scenes: ‘school’ in Fig. 1;
‘barbershop’ in Fig. 5; ‘classroom’ in Fig. 6; and ‘bedroom’ in Fig. 7.

Li  _et al_ [18] Lighthouse [25] Ours Ground Truth

Figure 6: Comparison to previous works on inserted mirror-objects and
estimated illumination maps. The value of peak signal-to-noise ratio (higher
is better) between each estimated illumination map and ground truth is shown
at the bottom of each image. Our method outperforms all the previous works by
providing illumination maps with rich details close to the ground truth.

Real data. As shown in Fig. 3, we connect and fix two Samsung Gear 360∘
cameras in a top-bottom manner. The two cameras are calibrated to vertically
align two images and measure the baseline. We present the captured data
‘office’ in Fig. 8. Following the convention on many 360∘ datasets [29, 26],
both the real and synthetic images are captured with the input resolution to
be $512\times 1024$.

Pre-process for Comparison to Previous Works. We choose two recently published
works for our lighting estimation comparison [18, 25]; and also two for
reflectance and surface normal comparison [2, 18]. Lighthouse [25] takes
perspective stereo images as input to estimate the environment lighting given
a position within the scene. Li  _et al_ [18] takes a single image while
Barron  _et al_ [2] takes a RGB-D input to estimate the reflectance, normal,
depth, and lighting. All the above methods only take input with the resolution
to be $240\times 320$. To satisfy their input requirement, we crop the middle
region of our 360∘ stereo input into small patches with the target resolution.
The top and bottom regions of the input are discarded to avoid the distortions
and simulate the views in perspective projection. Then we feed the stereo
images to Lighthouse [25]; a single image to Li  _et al_ [18]; and a single
image with depth estimation to Barron  _et al_ [2].

##  6 Experiments

Since there are no previous works that use a similar setup for this task, we
only compare our work with those using perspective images. The comparisons
here aim at demonstrating the strengths of 360∘ images over the perspective
images in the tasks of lighting estimation and intrinsic decomposition of the
scene properties.

###  6.1 Lighting Estimation

In Fig. 5, we present the inserted mirror-objects at different locations
within the scene. Our method estimates the lighting with elaborate high-
frequency details. Our inserted objects correctly reflect the changes of
lighting among different locations, demonstrating the spatially-coherency of
our lighting model. The two strengths above jointly contribute to the
appealing reflection effects of a virtual mirror-object.

We provide quantitative comparisons between our estimated illumination maps
and previous methods [18, 25] in Fig. 6. As a result of a simplified lighting
model, Li [18]’s illumination map only contains low-frequency information,
leading a mirror-object looks diffuse. Lighthouse [25], instead, based on a
simplified model of the scene’s geometry, can recover part of the scene in low
definition. However, due to the limited field of view of their perspective
input, their hallucination on the unobserved scene is far from satisfactory.
By the virtue of the 360∘ input, our near-field environment light provides an
accurate representation of the lighting and geometry of the scene. Hence, our
illumination maps are all estimated in high-definition and very close to the
ground truth.

###  6.2 Reflectance and Normal Estimation

GT Ours Li  _et al_ [18] Barron  _et al_ [2] Input

Figure 7: Estimated reflectance and normal on our synthetic scene ‘bedroom.’ The sMSE and MAE are shown at the bottom.  Methods | Reflectance | Normal
---|---|---
Barron  _et al_ [2] | 0.108 | 70.6
Li  _et al_ [18] | 0.084 | 34.0
Ours | 0.073 | 24.6
Table 1: Quantitative comparison on our synthetic scenes. We use sMSE for
reflectance and MAE in degrees for normal.

We showcase the quality of our estimated reflectance map and surface normal
map in the first row of Fig. 5. We also present quantitative results in Fig. 7
and Table 1. We use scale-invariant mean-square-error (sMSE) for reflectance
and mean angular error (MAE) in degrees for normal. Lower is better for both
the metrics. As mentioned in Sec. 5.2, all the competing methods [2, 18] take
small resolution perspective input. To avoid the heavy distortion on 360∘
images, we crop the middle region of our 360∘ image into four $240\times 320$
image patches as the input for others. Then we merge the results for viewing
and comparison. The discarded regions are displayed in black in their results.
Although our method can estimate the entire scene, the quantitative evaluation
shown in Fig. 7 and Table 1 is only computed on those regions that are both
generated by all the methods for a fair comparison.

Li  _et al_ [18] estimates the reflectance from prior knowledge and past data.
They present a wrong estimation on the ‘wall’ region in Fig. 7, which is
likely caused by the overfitting on a ‘white wall.’ Our method takes the 360∘
full observation of the scene to jointly reason lighting, geometry, and
reflectance by physical insights. Therefore, our estimation is of higher
accuracy.

###  6.3 Testing on Real Data

Our results on the real data are shown in Fig. 8. The top-bottom camera
setting has difficulties in capturing the ceiling and floor regions with
accurate depth, as they are either occluded by the tripod or containing severe
distortions that may lead to wrong disparities. Hence, our method fail to
estimate the normal at the part of the ceiling region in this scene. This kind
of noise hardly occurs in the synthetic data, which does not have the
occlusion problem and noises.

Figure 8: Results of real images. Both the reflectance and normal are of high
quality except the ceiling region. We relight three objects by our
illumination map at different locations. Note how the three inserted mirror-
objects correctly reflect the surroundings at each location: the orange cloth
reflected on the ‘duck’; the ceiling light reflected on the ‘cow’; the scene
reflected on the ‘teapot.’

###  6.4 Ablation Study

| $\Phi_{1}$ | $\Phi_{1+\frac{1}{4}}$ | Render and Refine
---|---|---|---
Reflectance | 0.086 | 0.080 | 0.073
Normal | 47.3 | 24.6 | -
Table 2: Quantitative results on ablated versions of our model. We use sMSE
for reflectance and MAE for normal.

Table 2 shows the quantitative results on the ablated versions of our method.
From left to right, the columns denote the origin RN-Net, pyramid RN-Net, and
our full method with the rendering and total variation refinement,
respectively. It shows that the pyramid structure improves reflectance and
normal. We also observe that the rendering and refinement module can
effectively reduce the noise and outliers to provide more plausible
reflectance maps.

##  7 Conclusion

We have presented a method that takes a 360∘ panoramic stereo as input, and
jointly estimates spatially-varying and 3D coherent lighting in high-
definition, reflectance, and geometry of the entire scene. Instead of using a
regular camera with a limited field of view, we demonstrate the advantages of
360∘ input in observing and estimating the whole scene. Our lighting model
accurately reconstructs 3D illumination maps, enabling mirror-like objects to
be inserted in the scene with realistic effect. We also leverage the physical
constraints between the lighting and geometry to infer both surface
reflectances and normals of the environment. Our results outperform previous
state-of-the-art, both quantitatively and qualitatively. Results on synthetic
and real images confirm the effectiveness and practicability of our method, by
a simple 360∘ stereo setup.

Acknowledgement. This research is funded in part by the ARC Centre of
Excellence for Robotics Vision (CE140100016) and ARC-Discovery (DP 190102261).
Yasuyuki Matsushita is supported by JSPS KAKENHI Grant Number JP19H01123.

## References

  * [1] Francesco Banterle, Marco Callieri, Matteo Dellepiane, Massimiliano Corsini, Fabio Pellacini, and Roberto Scopigno.  Envydepth: An interface for recovering local natural illumination from environment maps.  In Computer Graphics Forum, volume 32, pages 411–420. Wiley Online Library, 2013.
  * [2] Jonathan T Barron and Jitendra Malik.  Intrinsic scene properties from a single rgb-d image.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 17–24, 2013.
  * [3] Jonathan T Barron and Jitendra Malik.  Shape, illumination, and reflectance from shading.  IEEE transactions on pattern analysis and machine intelligence, 37(8):1670–1687, 2014.
  * [4] Sean Bell, Kavita Bala, and Noah Snavely.  Intrinsic images in the wild.  ACM Transactions on Graphics (TOG), 33(4):1–12, 2014.
  * [5] Sai Bi, Xiaoguang Han, and Yizhou Yu.  An l 1 image transform for edge-preserving smoothing and scene-level intrinsic decomposition.  ACM Transactions on Graphics (TOG), 34(4):1–12, 2015.
  * [6] Blender Online Community.  Blender - a 3d modelling and rendering package, 2020.
  * [7] Paul Debevec.  Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography.  In ACM SIGGRAPH 2008 classes, pages 1–10. 2008.
  * [8] Qingnan Fan, Jiaolong Yang, Gang Hua, Baoquan Chen, and David Wipf.  Revisiting deep intrinsic image decompositions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8944–8952, 2018.
  * [9] Marc-André Gardner, Yannick Hold-Geoffroy, Kalyan Sunkavalli, Christian Gagné, and Jean-Francois Lalonde.  Deep parametric indoor lighting estimation.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7174–7182. IEEE.
  * [10] Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and Jean-François Lalonde.  Learning to predict indoor illumination from a single image.  ACM Transactions on Graphics (TOG), 36(6):1–14, 2017.
  * [11] Mathieu Garon, Kalyan Sunkavalli, Sunil Hadap, Nathan Carr, and Jean-François Lalonde.  Fast spatially-varying indoor lighting estimation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6908–6917, 2019.
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [13] Sunghoon Im, Hyowon Ha, François Rameau, Hae-Gon Jeon, Gyeongmin Choe, and In So Kweon.  All-around depth from small motion with a spherical panoramic camera.  In European Conference on Computer Vision, pages 156–172. Springer, 2016.
  * [14] Hansung Kim and Adrian Hilton.  3d scene reconstruction from multiple spherical stereo pairs.  International journal of computer vision, 104(1):94–116, 2013.
  * [15] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [16] Chloe LeGendre, Wan-Chun Ma, Graham Fyffe, John Flynn, Laurent Charbonnel, Jay Busch, and Paul Debevec.  Deeplight: Learning illumination for unconstrained mobile mixed reality.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5918–5928, 2019.
  * [17] Yin Li, Heung-Yeung Shum, Chi-Keung Tang, and Richard Szeliski.  Stereo reconstruction from multiperspective panoramas.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(1):45–62, 2004.
  * [18] Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker.  Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2475–2484, 2020.
  * [19] Zhengqi Li and Noah Snavely.  Cgintrinsics: Better intrinsic image decomposition through physically-based rendering.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 371–387, 2018.
  * [20] Stephen Lombardi and Ko Nishino.  Reflectance and illumination recovery in the wild.  IEEE transactions on pattern analysis and machine intelligence, 38(1):129–141, 2015.
  * [21] Takuya Narihira, Michael Maire, and Stella X Yu.  Direct intrinsics: Learning albedo-shading decomposition by convolutional regression.  In Proceedings of the IEEE international conference on computer vision, pages 2992–2992, 2015.
  * [22] Leonid I Rudin, Stanley Osher, and Emad Fatemi.  Nonlinear total variation based noise removal algorithms.  Physica D: nonlinear phenomena, 60(1-4):259–268, 1992.
  * [23] Jian Shi, Yue Dong, Hao Su, and Stella X Yu.  Learning non-lambertian object intrinsics across shapenet categories.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1685–1694, 2017.
  * [24] Shuran Song and Thomas Funkhouser.  Neural illumination: Lighting prediction for indoor environments.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6918–6926, 2019.
  * [25] Pratul P Srinivasan, Ben Mildenhall, Matthew Tancik, Jonathan T Barron, Richard Tucker, and Noah Snavely.  Lighthouse: Predicting lighting volumes for spatially-coherent illumination.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8080–8089, 2020.
  * [26] Ning-Hsu Wang, Bolivar Solarte, Yi-Hsuan Tsai, Wei-Chen Chiu, and Min Sun.  360sd-net: 360 stereo depth estimation with learnable cost volume.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 582–588. IEEE, 2020.
  * [27] Guanyu Xing, Yanli Liu, Haibin Ling, Xavier Granier, and Yanci Zhang.  Automatic spatially varying illumination recovery of indoor scenes based on a single rgb-d image.  IEEE Transactions on Visualization and Computer Graphics, 2018.
  * [28] Yiqin Zhao and Tian Guo.  Pointar: Efficient lighting estimation for mobile augmented reality.  arXiv preprint arXiv:2004.00006, 2020.
  * [29] Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou.  Structured3d: A large photo-realistic dataset for structured 3d modeling.  arXiv preprint arXiv:1908.00222, 2019.
  * [30] Hao Zhou, Xiang Yu, and David Jacobs.  Glosh: Global-local spherical harmonics for intrinsic image decomposition.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7819–7828. IEEE.