* Skillman et al. (2011) Skillman, S. W., Hallman, E. J., O’Shea, B. W., et al. 2011, ApJ, 735, 96
  * Skillman et al. (2008) Skillman, S. W., O’Shea, B. W., Hallman, E. J., Burns, J. O., & Norman, M. L. 2008, ApJ, 689, 1063
  * Skillman et al. (2013) Skillman, S. W., Xu, H., Hallman, E. J., et al. 2013, ApJ, 765, 21
  * Smith et al. (2001) Smith, R. K., Brickhouse, N. S., Liedahl, D. A., & Raymond, J. C. 2001, ApJ, 556, L91
  * Springel (2010) Springel, V. 2010, MNRAS, 401, 791
  * Springel et al. (2018) Springel, V., Pakmor, R., Pillepich, A., et al. 2018, MNRAS, 475, 676
  * Springel et al. (2001) Springel, V., White, S. D. M., Tormen, G., & Kauffmann, G. 2001, MNRAS, 328, 726
  * Stevens et al. (2019) Stevens, A. R. H., Diemer, B., Lagos, C. d. P., et al. 2019, MNRAS, 483, 5334
  * Stuardi et al. (2019) Stuardi, C., Bonafede, A., Wittor, D., et al. 2019, MNRAS, 489, 3905
  * Tinker et al. (2008) Tinker, J., Kravtsov, A. V., Klypin, A., et al. 2008, ApJ, 688, 709
  * van Weeren et al. (2017) van Weeren, R. J., Andrade-Santos, F., Dawson, W. A., et al. 2017, Nature Astronomy, 1, 0005
  * van Weeren et al. (2011a) van Weeren, R. J., Brüggen, M., Röttgering, H. J. A., & Hoeft, M. 2011a, MNRAS, 418, 230
  * van Weeren et al. (2011b) van Weeren, R. J., Brüggen, M., Röttgering, H. J. A., et al. 2011b, A&A, 533, A35
  * van Weeren et al. (2016) van Weeren, R. J., Brunetti, G., Brüggen, M., et al. 2016, ApJ, 818, 204
  * van Weeren et al. (2019) van Weeren, R. J., de Gasperin, F., Akamatsu, H., et al. 2019, Space Sci. Rev., 215, 16
  * van Weeren et al. (2013) van Weeren, R. J., Fogarty, K., Jones, C., et al. 2013, ApJ, 769, 101
  * van Weeren et al. (2010) van Weeren, R. J., Röttgering, H. J. A., Brüggen, M., & Hoeft, M. 2010, Science, 330, 347
  * van Weeren et al. (2021) van Weeren, R. J., Shimwell, T. W., Botteon, A., et al. 2021, A&A, 651, A115
  * Vazza et al. (2012) Vazza, F., Brüggen, M., van Weeren, R., et al. 2012, MNRAS, 421, 1868
  * Vazza et al. (2009) Vazza, F., Brunetti, G., & Gheller, C. 2009, MNRAS, 395, 1333
  * Vazza et al. (2015) Vazza, F., Eckert, D., Brüggen, M., & Huber, B. 2015, MNRAS, 451, 2198
  * Vazza et al. (2023a) Vazza, F., Wittor, D., Brüggen, M., & Brunetti, G. 2023a, Galaxies, 11, 45
  * Vazza et al. (2023b) Vazza, F., Wittor, D., Di Federico, L., et al. 2023b, A&A, 669, A50
  * Vogelsberger et al. (2018) Vogelsberger, M., Marinacci, F., Torrey, P., et al. 2018, MNRAS, 474, 2073
  * Weinberger et al. (2017) Weinberger, R., Springel, V., Hernquist, L., et al. 2017, MNRAS, 465, 3291
  * Wittor et al. (2021) Wittor, D., Ettori, S., Vazza, F., et al. 2021, MNRAS, 506, 396
  * Wittor et al. (2019) Wittor, D., Hoeft, M., Vazza, F., Brüggen, M., & Domínguez-Fernández, P. 2019, MNRAS, 490, 3987
  * Yoon et al. (2020) Yoon, M., Lee, W., Jee, M. J., et al. 2020, ApJ, 903, 151
  * Zhang et al. (2020) Zhang, C., Churazov, E., Dolag, K., Forman, W. R., & Zhuravleva, I. 2020, MNRAS, 494, 4539
  * Zhang et al. (2019) Zhang, C., Churazov, E., Forman, W. R., & Lyskova, N. 2019, MNRAS, 488, 5259
  * ZuHone et al. (2010) ZuHone, J. A., Markevitch, M., & Johnson, R. E. 2010, ApJ, 717, 908

##  Appendix A Resolution tests

Figure 17: Volume-averaged energy dissipation rate in shocks simulated in the
various runs of the IllustrisTNG. The TNG-Cluster simulation has the same
resolution as the TNG300-1 and TNG100-2 runs. The energy dissipated by weak
shocks ($\mathcal{M}<10$) converges in the simulation runs if the resolution
is better than that of TNG300-2, while the energy of the strong shocks
($\mathcal{M}>10$) increases with increasing resolution.  Figure 18:
Projection map of mass (left) and radio surface brightness (right column) of
the same cluster merger in TNG300-1 (top), TNG300-2 (middle), and TNG300-3 run
(bottom row). The figure is centered on the center of mass. The dotted circle
marks the $R_{\rm 500c}$. The annotated $M_{\rm 200c}$ and $P_{\rm
1.4\leavevmode\nobreak\ GHz}$ have the unit of $\rm M_{\odot}$ and $\rm
W\leavevmode\nobreak\ Hz^{-1}$, respectively. The radio relic morphology is
consistent between the TNG300-2 and TNG300-1 runs, whereas the radio relics
are not resolved in the TNG300-3 run.

We test the dependence of our results on numerical resolution by using the
IllustrisTNG series. We remind the readers that the TNG-Cluster employs the
same TNG galaxy formation model and has a mass resolution identical to the
TNG300-1 and TNG100-2 runs. Therefore, we can use the TNG runs (see e.g.,
summary tables in Nelson et al. 2019b) to quantify the convergence of
resolution of our findings, at least in lower-density environments and for
fewer and less massive systems.

Figure 17 presents the distribution of the energy dissipation rate of the
cosmological shockwaves in the unit comoving volumes as a function of the Mach
number. Weak shocks with $\mathcal{M}<10$ dominate the dissipated-energy
budget by an order of magnitude compared to stronger shocks with
$\mathcal{M}>10$. According to the dissipation rate (e.g., Kang et al. 2007),
the kinetic energy flux of the weak shocks is two orders of magnitude higher
than that of the strong shocks, which is consistent with the findings of
previous studies (e.g., Ryu et al. 2003; Skillman et al. 2008; Vazza et al.
2009; Schaal & Springel 2015).

According to this comparison, the energy dissipation rate of the
low-$\mathcal{M}$ shockwaves is reasonably well converged at the TNG300-1
resolution, i.e., in TNG-Cluster. In fact, the distribution of the dissipated
energy rate in weak shocks is consistent across all these runs of the TNG
series except for TNG300-3, which has the worst resolution. The shock
strengths with the maximum energy dissipation rates are slightly lower in
TNG300-1 than in TNG100-2, likely due to different box sizes.

On the other hand, the energy dissipated by $\mathcal{M}>10$ shocks increases
by a factor of $\raise 0.81805pt\hbox{$\scriptstyle\sim$}2$ for every
improvement in the numerical resolution, namely from TNG300-2 to TNG300-1, and
from TNG300-1 to TNG100-1. This implies that the properties of strong shocks
($\mathcal{M}>10$) may not be converged at the resolution of the TNG-Cluster.
However, studies have found that these shockwaves are dominated mainly by
accretion shocks (e.g., Ryu et al. 2003), which has negligible contribution to
the diffuse radio features (see Section 3.2).

We further check the impact of resolution on the radio relic morphology and
properties. In Figure 18, we show the mass and radio maps of a double radio
relic system in the TNG300 series, from the higher-resolution TNG300-1 to
TNG300-3 from top to bottom. The position of the radio relics and overall
morphology are consistent between the TNG300-2 and TNG300-1 runs. TNG300-3
presents an entirely different morphology in both dark matter surface density
and radio surface brightness maps, which we suspect is the result of the
different phases of cluster merger by the sparser timesteps of TNG300-3.

The physical properties of the radio relics appear reliable at the resolution
of TNG300-1, giving us confidence in our results with the TNG-Cluster. In
particular, we extract all the radio relic regions using the method described
in Section 4.1 by grouping the radio structures with a larger bin size for the
lower-resolution simulation runs: 40 and $80\rm\leavevmode\nobreak\ kpc$ for
the TNG300-2 and TNG300-3 runs, respectively. The shock strength
($\mathcal{M}\sim 3$) of the relic cells is converged at the resolution of
TNG300-2 in all identified radio relics. On the other hand, the radio
luminosity may vary with the resolution. The total radio luminosity of double
radio relics is, on average, consistent between TNG300-1 and TNG300-2 ($P_{\rm
1.4\leavevmode\nobreak\ GHz}\sim 10^{24}\rm\leavevmode\nobreak\
W\leavevmode\nobreak\ Hz^{-1}$). However, the luminosity of the individual
radio relics may change across runs. The luminosity of the brightest radio
relic in the TNG300 series increases by $\raise
0.81805pt\hbox{$\scriptstyle\sim$}60\%$ from TNG300-2 to TNG300-1, whereas
that of the double radio relic pair of Figure 18 decreases by a factor of
$\raise 0.81805pt\hbox{$\scriptstyle\sim$}3.5$. We find a similar resolution
dependence in the single radio relic system of Figure 19, where the position,
linear extension, and shock strength ($\mathcal{M}\sim 4$) are consistent
between TNG300-1 and TNG300-2, while the luminosity increased by a factor of
$\raise 0.81805pt\hbox{$\scriptstyle\sim$}2.8$ with resolution improvement.

Figure 19: Same as Figure 18 of the single radio relic system.

Based on these resolution tests, we conclude that the overall properties of
the radio relics, such as their position from the cluster, morphology, and
shock strength, converge at the resolution of the TNG-Cluster simulation.
Radio luminosity varies between TNG300-1 and TNG300-2, so caution is required
when interpreting radio relic luminosity. Nevertheless, we considered the
luminosity statistically consistent since the total energy budget from the
cosmological shockwaves converged in TNG300-1.

##  Appendix B Table of observed radio relics

Table 1 summarises the properties of all radio relics observed so far and
their reference. We have referred to a few of them in the main body of the
manuscript.

We quote $M_{\rm 500c}$ of the cluster based on SZ observations when
available. In other cases, we derive the X-ray luminosity assuming the mass-
luminosity relation of Pratt et al. (2009) or interpolate the weak-lensing
based mass, assuming an NFW profile (Navarro et al. 1996) and the Duffy et al.
(2008) M-c relation. We also include radio relics that are detected only at
low-frequency radio wavelengths. As we do not have information on the spectral
index, we derive the $1.4\rm\leavevmode\nobreak\ GHz$ radio luminosity
assuming the spectrum follows $S\propto\nu^{-1.5}$, except Abell 2108
($\propto\nu^{-2}$, Schellenberger et al. 2022).

Name | $z$ | $M_{\rm 500c}$ | Pos. | $P_{\rm 1.4GHz}$ | $LLS$ | Ref. | Name | $z$ | $M_{\rm 500c}$ | Pos. | $P_{\rm 1.4GHz}$ | $LLS$ | Ref.
---|---|---|---|---|---|---|---|---|---|---|---|---|---
|  |  [$10^{14}\leavevmode\nobreak\ \rm M_{\odot}$] |  | $[10^{24}\rm\leavevmode\nobreak\ W\leavevmode\nobreak\ Hz^{-1}]$ | $[\rm Mpc]$ |  |  |  |  [$10^{14}\leavevmode\nobreak\ \rm M_{\odot}$] |  | $[10^{24}\rm\leavevmode\nobreak\ W\leavevmode\nobreak\ Hz^{-1}]$ | $[\rm Mpc]$ |
Abell 1656I | 0.023 | 7.17 |  | 0.31 | 0.85 | 1 |  PSZ G087∗ | 0.222 | 3.83 |  | 0.03 | 1.0 | 15
PSZ2G146∗,I | 0.03 | 1.8 |  | 0.02 | 0.37 | 2 |  SPT2023-5535I | 0.23 | 6.49 |  | 3.4 | 0.5 | 23,24
Abell 168 | 0.045 | 1.24 |  | 0.14 | 0.8 | 3 | Abell 746 | 0.232 | 5.34 | W | 4.25 | 1.8 | 25
Abell 3376 | 0.046 | 3.64 | E | 0.54 | 1.6 | 4 |  |  |  | N | 0.18 | 0.38 | 25
|  |  | W | 0.40 | 0.96 | 4 |  |  |  | E | 0.19 | 0.95 | 25
Abell 3667 | 0.056 | 5.77 | NW | 15.1 | 2.3 | 5 |  PSZ2 G181∗ | 0.240 | 4.23 | S | 0.48 | 1.66 | 15
|  |  | SE | 2.6 | 1.6 | 5 |  |  |  | N | 0.20 | 1.39 | 15
Abell 3266I | 0.059 | 6.64 |  | 0.24 | 0.58 | 6 | RXCJ1314 | 0.244 | 6.15 | W | 3.90 | 0.91 | 13
CIZA J0649 | 0.064 | 3.3 |  | 1.2 | 0.80 | 7,8 |  |  |  | E | 1.68 | 1.13 | 13
ClG 0217+70 | 0.066 | 10.6 | SE | 0.99 | 3.5 | 9 |  MCXC J0929∗ | 0.247 | 3.9 | SW | 0.3 | 0.30 | 17
|  |  | SE | 1.27 | 5.1 | 9 |  |  |  | SE | 0.2 | 0.35 | 17
|  |  | W | 1.85 | 2.3 | 9 | Abell 521 | 0.247 | 7.26 | SE | 3.09 | 0.93 | 26
RXC J1053 | 0.070 | 1.1 |  | 0.2 | 0.6 | 7,8 | ZwCl2341 | 0.27 | 5.15 | S | 4.09 | 1.23 | 13
Abell 1904∗ | 0.070 | 1.83 | N | 0.32 | 0.23 | 10 |  |  |  | N | 1.82 | 0.57 | 13
|  |  | S | 0.005 | 0.17 | 10 |  Abell 1758S∗ | 0.280 | 8.22 |  | 0.22 | 0.54 | 15
|  |  | NE | 0.01 | 0.9 | 10 | SPT2023-5627 | 0.284 | 5.74 | SE | 1.40 | 0.73 | 27
Abell 2061 | 0.078 | 3.59 |  | 0.45 | 0.68 | 7 |  |  |  | NW | 0.91 | 0.86 | 27
Abell 2255 | 0.081 | 5.38 |  | 0.18 | 0.70 | 7 |  Abell 959∗ | 0.289 | 5.08 |  | 0.22 | 0.81 | 15
Abell 2249 | 0.084 | 3.73 |  | 0.41 | 1.3 | 11 | 1E 0657-55.8 | 0.296 | 11.4 | E | 43.5 | 0.99 | 28
Abell 2018∗ | 0.088 | 2.51 |  | 0.04 | 1.1 | 10 |  |  |  | NNW | 0.6 | 0.90 | 28
Abell 2108∗ | 0.092 | 1.8 |  | 0.01 | 0.2 | 12 | Abell 781 | 0.295 | 6.13 |  | 4.47 | 0.44 | 1
Abell 3365 | 0.093 | 1.7 | E | 0.74 | 0.55 | 13,14 |  PSZ2 G198∗ | 0.299 | 5.5 |  | 0.51 | 1.49 | 15
|  |  | W | 0.09 | 0.29 | 13,14 | PSZ1G097 | 0.3 | 4.7 | S | 3.12 | 1.42 | 13
Abell 523 | 0.10 | 1.7 |  | 1.7 | 1.35 | 7,8 |  |  |  | N | 1.52 | 0.88 | 13
ZwCl0008 | 0.103 | 3.30 | E | 1.54 | 1.41 | 13 | Abell 1300 | 0.3072 | 8.97 |  | 5.75 | 0.70 | 1
|  |  | W | 0.30 | 0.30 | 13 | Abell 2744 | 0.308 | 9.84 | NE | 4.37 | 1.5 | 29
Abell 1925∗ | 0.105 | 2.81 | N | 0.12 | 1.71 | 15 |  |  |  | SE | 0.96 | 1.15 | 29
|  |  | S | 0.08 | 1.09 | 15 | MACS J0258 | 0.322 |  |  |  | 0.39 | 16
Abell 2034 | 0.113 | 5.85 |  | 0.89 | 0.22 | 7 |  Abell 1943∗ | 0.336 | 8.14 |  | 0.03 | 0.37 | 17
Abell 3186 | 0.127 | 6.44 |  | 2.5 | 2.0 | 16 |  PSZ2 G121∗ | 0.344 | 5.69 |  | 0.21 | 0.54 | 15
WHL J1013∗ | 0.146 | 2.49 |  | 0.01 | 0.31 | 17 |  PSZ2 G096∗ | 0.350 | 5.39 |  | 0.60 | 1.48 | 15
PSZ2 G278 | 0.158 | 3.6 | N | 1 | 0.66 | 18 |  PSZRX G095∗ | 0.362 | 3.34 |  | 0.8 | 1.30 | 17
|  |  | S | 0.7 | 1.64 | 18 |  PSZ2 G092∗ | 0.362 | 6.31 |  | 0.32 | 1.53 | 15
Abell 1240 | 0.159 | 3.71 | S | 0.73 | 1.28 | 13 | MACS J1752 | 0.366 | 6.96 | NE | 33.1 | 1.19 | 13
|  |  | N | 0.43 | 0.87 | 13 |  |  |  | SW | 15.0 | 0.69 | 13
Abell 3411 | 0.169 | 6.59 |  | 5.0 | 1.9 | 19 |  PSZ2 G114∗ | 0.371 | 7.58 | N | 3.40 | 1.17 | 15
PSZ2 G109∗ | 0.173 | 3.26 |  | 0.50 | 1.17 | 15 |  |  |  | S | 0.83 | 1.48 | 15
PSZRX G182∗ | 0.175 | 2.4 |  | 0.3 | 1.00 | 17 | ZwCl1447 | 0.376 | 3.4 | SW | 1.11 | 1.2 | 30
Abell 2345 | 0.177 | 5.92 | W | 2.85 | 0.83 | 13 |  |  |  | NE | 0.51 | 0.3 | 30
|  |  | E | 2.66 | 1.57 | 13 |  PSZ2 G187∗ | 0.378 | 6.84 |  | 0.27 | 0.72 | 15
Abell 1612 | 0.179 | 4.42 |  | 7.9 | 0.78 | 7 | PSZ1 G287 | 0.39 | 13.89 | NW | 23.3 | 2.48 | 13
Abell 1697∗,I | 0.183 | 4.34 |  | 0.38 | 0.7 | 10 |  |  |  | SE | 9.71 | 1.58 | 13
WHL J1721∗ | 0.184 | 2.29 |  | 0.08 | 0.60 | 17 |  WHL J1305∗ | 0.396 | 3.29 |  | 0.06 | 1.40 | 17
Abell 1889∗ | 0.185 | 2.58 | NW | 0.50 | 1.65 | 17 |  PSZ2 G117∗ | 0.396 | 7.61 |  | 0.32 | 0.56 | 15
|  |  | SE | 0.40 | 1.65 | 17 |  MCXC J0943∗ | 0.406 | 4.75 |  | 0.4 | 0.53 | 17
PSZ2 G145∗ | 0.190 | 4.25 |  | 0.04 | 0.77 | 15 |  PSZ2 G206∗ | 0.447 | 7.39 | S | 0.25 | 0.66 | 15
CIZAJ2243 | 0.192 | 16.2 | N | 15.0 | 2.00 | 13,20 |  |  |  | N | 0.18 | 0.66 | 15
|  |  | S | 1.95 | 1.65 | 13,20 |  PSZ2 G191∗ | 0.488 | 5.55 |  | 2.49 | 0.57 | 15
Abell 115 | 0.197 | 7.65 |  | 2.7 | 2.5 | 21 | MACS J1149 | 0.544 | 8.55 | W | 6.03 | 0.70 | 13
Abell 2163 | 0.203 | 16.1 |  | 2.09 | 0.48 | 1 |  |  |  | E | 5.33 | 0.78 | 13
1RXS J0603 | 0.22 | 10.76 | N | 65.0 | 1.9 | 22 | PSZ2 G069 | 0.762 | 5.69 |  | 3.75 | 1.49 | 15
|  |  | E | 2.5 | 1.1 | 22 | PSZ2 G092 | 0.822 | 7.4 |  | 5.25 | 0.9 | 31
|  |  | S | 1.4 | 0.25 | 22 | ACT J0102 | 0.87 | 8.80 | NW | 29.6 | 0.93 | 13
PLCK G201 | 0.22 | 2.7 | E | 3.9 | 1.19 | 16 |  |  |  | SE | 4.48 | 0.46 | 13
Table 1: Properties of the observed radio relics. Column 1: cluster name.
Column 2: cluster redshift. Column 3: radio luminosity at
$1.4\rm\leavevmode\nobreak\ GHz$. We derive the luminosity assuming the radio
spectrum of $S\propto\nu^{-1.5}$ for the radio relics only detected at a low
frequency. These radio relics are annotated with ∗. The inverted radio relics
are marked with I. Column 4: largest linear size (LLS). Column 5: reference
for the cluster radio luminosity. References: 1. Feretti et al. (2012); 2.
Botteon et al. (2021); 3. Dwarakanath et al. (2018); 4. Chibueze et al.
(2023); 5. de Gasperin et al. (2022); 6. Riseley et al. (2022); 7. van Weeren
et al. (2011b); 8. Pratt et al. (2009); 9. Hoang et al. (2021); 10. van Weeren
et al. (2021); 11. Locatelli et al. (2020); 12. Schellenberger et al. (2022);
13. de Gasperin et al. (2014); 14. Lovisari & Reiprich (2019); 15. Botteon et
al. (2022); 16. Knowles et al. (2022); 17. Hoang et al. (2022); 18. Koribalski
et al. (2023); 19. van Weeren et al. (2013); 20. Jee et al. (2015); 21. Govoni
et al. (2001b); 22. van Weeren et al. (2016); 23. HyeongHan et al. (2020); 24.
Bulbul et al. (2019); 25. Rajpurohit et al. (2023); 26. Giacintucci et al.
(2006); 27. Duchesne et al. (2021); 28. Sikhosana et al. (2023); 29. Pearce et
al. (2017); 30. Lee et al. (2022); 31. Di Gennaro et al. (2023)


OSTeC: One-Shot Texture Completion

Baris Gecer,   Jiankang Deng,   and Stefanos Zafeiriou

Imperial College London

{b.gecer, j.deng16<EMAIL_ADDRESS>
figureFace frontalization and UV texture completion by our approach. The first row is the input, the second row is the frontalization result, and the third row is the completed UV texture. The proposed method can produce photo-realistic and identity-preserved full UV textures even under extreme poses.

The last few years have witnessed the great success of non-linear generative models in synthesizing high-quality photorealistic face images. Many recent 3D facial texture reconstruction and pose manipulation from a single image approaches still rely on large and clean face datasets to train image-to-image Generative Adversarial Networks (GANs). Yet the collection of such a large scale high-resolution 3D texture dataset is still very costly and difficult to maintain age/ethnicity balance. Moreover, regression-based approaches suffer from generalization to the in-the-wild conditions and are unable to fine-tune to a target-image. In this work, we propose an unsupervised approach for one-shot 3D facial texture completion that does not require large-scale texture datasets, but rather harnesses the knowledge stored in 2D face generators. The proposed approach rotates an input image in 3D and fill-in the unseen regions by reconstructing the rotated image  in  a  2D  face  generator, based  on  the visible parts. Finally, we stitch the most visible textures at different angles in the UV image-plane. Further, we frontalize the target image by projecting the completed texture into the generator. The qualitative and quantitative experiments demonstrate that the completed UV textures and frontalized images are of high quality, resembles the original identity, can be used to train a texture GAN model for 3DMM fitting and improve pose-invariant face recognition.[Project Page: <https://github.com/barisgecer/OSTeC>]

§ INTRODUCTION

The problem of 3D face texture completion (as shown in Fig. <ref>) refers generally to the problem of recovering near ear-to-ear visible and non-visible colour from a single image <cit.> in a “canonical”, deformation-free parameterization of the face surface (usually referred as UV-space). A very similar problem is that of producing arbitrary face rotations from a single image <cit.>. Both of the above problems have important applications in many different domains of face analysis such as pose-invariant face recognition <cit.>, as well developing of 3D Morphable Model (3DMM) algorithms <cit.> and creating complete head avatars from single images <cit.>. That is why 3D face texture completion, as well as, producing face rotations has been very popular in the intersection of machine learning and computer vision, offering an important application domain to the advancements of machine learning in each era (from robust component analysis <cit.> to modern deep learning <cit.>).

The problem of predicting the missing colour in the texture coordinated of the UV space or predicting a new view from a single image has been the application domain of many machine learning algorithms starting from simple nearest-neighbour interpolation, (Fig. <ref>), regression techniques using linear-statistical priors (e.g., Robust Principal Component Analysis <cit.>) to modern deep learning regression techniques such as image-to-image translation models using conditional Generative Adversarial Networks (GANs) <cit.>. The problem has been modeled as fully supervised, the regression model was trained with pairs of missing and complete 3D facial texture <cit.>, or recently using self-supervised methods and image rendering <cit.>. Nevertheless, fully-supervised or self-supervised, to the best of our knowledge, all current methods belong in the family of regression techniques.

Contrary to the above, we take a radically different line of work in this paper: We propose to re-think the 3D facial texture prediction and rotation generation as an optimisation problem and design our method as a one-shot texture completion approach. One of the key problems of regression-based approaches such as <cit.> is that they may lose the identity because the function they learn is quite generic. Contrary, our approach optimises, along-side many other functions, identity-related features. Our method produces visually stunning results in both 3D texture completion as well as frontalization (for some results please inspect Fig. <ref>). Another by-product of our method is a 3D texture model learned from in-the-wild images that, as we show, can be used for training state-of-the-art 3D face reconstruction algorithms such as GANFit <cit.> (which was trained with around 10K 3D faces captured in well-controlled conditions which are not released to the public).

In short, the contributions of our paper are as follows:

* We re-design the problem of 3D facial texture completion as a one-shot optimisation-based approach. We propose a well-engineered novel methodology and cost function suitable for the task.

* We capitalize on the power of 2D face generators to recover unseen part of 2D face by rotating it in 3D. So that, there would no need for 3D data collection.

* We show the effectiveness of the proposed approach in qualitative and quantitative experiments. Additionally, we apply the method to many in-the-wild images in order to train a large-scale prior of the 3D facial texture which we use to train state-of-the-art 3D face reconstruction algorithms.

§ RELATED WORK

Face Generation, Manipulation & Rotation :
In just a few years, the quality of face generations by GANs have improved incredibly <cit.>. The recently proposed StyleGANv2 <cit.> has shown high-quality 2D face generations up to $1024 \times 1024$ by eliminating artefacts that appear in the previous results. Many follow up works <cit.> could successfully project real images over its latent space and perform semantic manipulation. This indicates that one can utilize StyleGAN generator as a 2D facial texture prior. In this study, we exploit this finding for image inpainting to recover the unseen part of a 2D face.

One of the commonly manipulated facial attributes is the pose, especially to a frontal view for its applications in face recognition and normalization. Unfortunately, above mentioned latent space manipulation methods are either struggling to disentangle other attributes from the latent parameters or having difficulty to project an in-the-wild image to this space. Even if it is possible to achieve excellent reconstruction by projecting to the extended latent space ($\mathbb{R}^{18 \times 512}$) of StyleGAN <cit.>, this enforcement exhaust its semantic meaning, therefore, become non-functional for frontalization. In fact, one can project a cat image to a StyleGAN trained on human faces by these approaches.

A large body of work addresses this problem by image-to-image translation GANs <cit.>. Many of these approaches utilize paired datasets in a supervised setting which does not generalize well to in-the-wild settings. A recent work <cit.> proposed a self-supervised training approach which perturbs images by 3D rotation to generate training pairs automatically. Nevertheless, these regression-based methods suffer from generalization and fall behind the optimization-based approaches which can fine-tune for any target image.

3D Texture Completion :
Modelling and synthesis of faces have been extensively studied in 3D as well <cit.>. Nevertheless, generations from these models have been far from being photorealistic. Therefore, there have been some works that proposed to complete a partially visible appearance of 2D images to a 3D appearance maps <cit.>. The most recent one <cit.> trains an image-to-image translation network supervised by a set of controlled datasets, failing to generate high-quality images for in-the-wild settings. Although the proposed approach tackles the problem of texture completion, it brings a new perspective which is formulating texture completion as an optimization-based inpainting problem fortified by 2D StyleGAN and 3D geometry priors.

Unsupervised 3D Face Model : There have been some studies to build 3d face model directly from 2D images such as <cit.> which learns a non-linear model from in-the-wild images and <cit.> learns a complete model from videos. As a side-product of this approach, we attempt to build a texture model from a set of complete texture UV-maps of 2D images and compare it to GANFit <cit.> model which is trained by $\sim$ 10,000 high-quality 3D textures.

§ UNSUPERVISED UV COMPLETION

The key insight of our work is to utilize 2D face generator networks and 3D geometry in a progressive one-shot optimization procedure for texture completion and frontalization. Basically, our approach rotates an input image in 3D and fill-in the unseen regions by reconstructing the rotated image in a 2D face generator, based on the visible parts. This 2D reconstruction is performed by an optimization in the latent space of the generator. Finally, textures acquired from these generations are collected progressively to build a coherent texture UV-map. In this section, we explain the details of our method.

§.§ 3DMM Fitting & Input Texture Acquisition

Input image ($\mathbf{I}_0$)

3D recons-

Interpolated UV map($\mathbf{T}_0$)

A different view ($\mathbf{I}_i$)

3DMM Fitting and texture completion by nearest-neighbour interpolation. As can be seen in (d), interpolation method produces artefacts for different camera views.

For a given 2D face image $\mathbf{I}_0$, our approach relies on a rough estimation of its dense landmarks by a 3D reconstruction method. Therefore, we begin by fitting an off-the-shelf 3DMM algorithm to estimate its geometry[Please note that no texture reconstruction from the 3DMM fitting algorithm is passed to the next stages.]  $\mathbf{S} \in \mathbb{R}^{n \times 3}$ and camera parameters $\mathbf{c}=[f,r_x,r_y,r_z,t_x,t_y,t_z]$. Let us define a 2D projection operation by a pinhole camera model with the function $\mathcal{P}(\mathbf{S}, \mathbf{c}) : \mathbb{R}^{n\times3}, \mathbb{R}^{7} \rightarrow \mathbb{R}^{n\times2}$, the geometry is then projected onto 2D image plane, dense landmarks, by $
\mathbf{S'} = \mathcal{P}(\mathbf{S}, \mathbf{c})$.

Traditionally, high-quality 3D texture information can be stored in UV maps which assign 3D texture data into 2D planes with a universal per-pixel alignment for all textures. Each vertex of the geometry has a texture coordinate $t_{coord} \in \mathbb{R}^{n\times2}$ in the UV image plane in which the texture information is stored. In our approach, starting from the texture available in the input image, we progressively complete the texture in the UV space.

Given a set of 2D vertex coordinates, a texture UV map $\mathbf{T} \in \mathbb{R}^{w \times h \times 3}$ and texture coordinates, one can render a textured geometry by performing rasterization with barycentric interpolation expressed as $\mathcal{R} : (\mathbb{R}^{n\times2}, \mathbb{R}^{w \times h \times 3}, \mathbb{R}^{n\times2}) \rightarrow \mathbb{R}^{w' \times h' \times 3}$.

In order to acquire the visible part of the texture from the input image ($\mathbf{I}_0$), we perform a similar rendering by swapping vertex coordinates with texture coordinates and the texture UV map with the input image (, image-to-UV rendering). In other words, the dense landmarks ($\mathbf{S'}$) from 3DMM fitting replace texture coordinates where the texture is actually the original image ($\mathbf{I}_0$). So, we unfold the input image into the UV space by giving the actual $t_{coord}$ of our topology as the vertex coordinates to be rendered. Consequently, the rendering is performed by the following:

\begin{align}
\mathbf{T}_0 = \mathcal{R}'(t_{coord}, \mathbf{I}_0, \mathbf{S'})
\end{align}

in which image-to-UV rendering ($\mathcal{R}'$) is essentially same operation as UV-to-image rendering ($\mathcal{R}$), however, we denote them differently to avoid confusion.

An obvious motivation of this work can be seen in the illustration of this operation in Fig. <ref>. After acquisition of the visible texture from the input image, we can see huge artefacts at invisible and narrow-angled parts of the geometry. Therefore, we explain how to detect and inpaint these regions by slowly building on top of the visible texture from the input image.

§.§ Re-Rendering of the Mesh

In order to fill-in the less-visible parts of the texture acquired from the original image, we rotate and render the fitted mesh by certain angles. We take the textured geometry as described in Sec. <ref> and render it with a set of predefined camera parameters. The perspectives of these novel views are defined to maintain best visibility of every part of the face with a near-perpendicular view.

Given $\mathbf{c}_i$ ($i>0$) as the ith novel camera parameters, we project geometry to the image plane and render texture geometry under this new perspective by the followings[The term $\overline{\mathbf{T}}_{i-1}$ refer to progressive texture of the previos iteration. It is explained in Sec. <ref>]:

\begin{align}
\mathbf{S'}_i &= \mathcal{P}(\mathbf{S}, \mathbf{c}_i) \\
\mathbf{I}_i &= \mathcal{R}(\mathbf{S'}_i, \overline{\mathbf{T}}_{i-1}, t_{coord})
\label{eq:render}
\end{align}

§.§.§ Building a Visibility Index

Each of the novel perspectives dominates certain part of the texture map in terms of clarity and visibility, bottom view is best for under-chin and side views are for cheeks. This visibility score can be defined in terms of the angle between the normal of each triangle and its vector pointing towards the camera. Meaning that, the acquired texture would have higher resolution and less artefact with lower angles between the two vectors, for triangles that are facing towards the camera. For each perspective ($\mathbf{c}_i$), we extract a visibility UV map $\mathbf{V}_i$, ranging between $(-1,1)$ where $1$ indicates that the triangles around the vertex are facing towards the camera in average and $-1$ is facing the opposite direction. This process can be formulated by applying camera $\mathbf{c}_i$ to the geometry $\mathbf{S}$, and taking a dot product between vertex coordinates with respect to the camera and vertex normals.

\begin{align}
\mathbf{V}_i &= \text{diag}( \frac{[\mathbf{S}'_i, \mathbf{h}]}{||[\mathbf{S}'_i, \mathbf{h}]||_2} \cdot \mathcal{N}(\mathbf{S}_i)^T)
\end{align}

where $\mathbf{h} \in \mathbb{R}^{n \times 1}$ stands for a vector of ones to make $\mathbf{S}'_i$ homogeneous. And $\mathcal{N}$ denotes the calculation the normals of the vertices. Some visibility score UV maps can be seen in Fig. <ref> for different camera settings. Fig. <ref> illustrates dominance map of all visibility scores, which we call visibility index and use it for stitching texture maps that are generated from the optimization of different views. The binary masks of visibility index can be formulated as the following:

\begin{align}
\overline{\mathbf{V}}_i =   \bigcap\limits_{i \ne j} (\mathbf{V}_i > \mathbf{V}_j) \
\end{align}

Input view

Input view

$\!$Visibility Ind.$\!\!\!\!$

Visibility scores are to measure optimal camera angles with respect to facial surface in UV-map.(a) $\mathbf{V}_0$. of input image (b-g) $\mathbf{V}_i$ of different views. (h) Visibility index ($\overline{\mathbf{V}}_i$): an index of optimal angles for texture acquisition.

§.§ Inpainting by Projection

Overview of the method. The proposed approach iteratively optimizes the texture UV-maps for different re-rendered images with their masks. At the end of each optimization, generated images are used to acquire partial UV images by dense landmarks. Finally, the completed UV images are fed to the next iteration for progressive texture building.

The main assumption of this work is that we can utilize a generator network trained by 2D images as a prior appearance model for inpainting. Since we extracted a part of texture from the original image in Sec. <ref>, we can now use it for conditional projection to styleGAN model to generate high quality and consistent faces for the invisible part.

§.§.§ Masking

In order to separate visible and invisible regions, for each novel view, binary masks are generated from the visibility scores ($\mathbf{V}_i$). We empirically found that intersection of two masks gives the best results: 1) regions where the visibility score of the original camera is higher than a threshold ($\mathbf{V}_0 > t_1$), and 2) regions where the visibility score of the original camera perspective is higher than the target camera[Other cameras are handicapped by a factor of 2 to enlarge texture from the original image]. Additionally, we progressively enlarge this mask by the dominant regions of all previously processed camera views, which is explained in Sec. <ref>.

The mask as explained above would give a UV mask which is then rendered by the current camera parameters $\mathbf{c}_i$ (similar to Eq. <ref>). The whole mask extraction process can be formulated as the following:

\begin{align}
\mathbf{M}^{UV}_i &=\big((\mathbf{V}_0 > t_1) \cap (2\mathbf{V}_0 > \mathbf{V}_i)\big) \cup \displaystyle\bigcup\limits_{i>j} \overline{\mathbf{V}}_j\\
\mathbf{M}_i &= \mathcal{R}(\mathbf{S'}_i, \mathbf{M}^{UV}_i, t_{coord})
\end{align}

§.§.§ Face Generation

The proposed approach requires a good quality generator that can synthesize face images from an arbitrary noise vector. Therefore, we borrow one of the state-of-the-art GAN network: StyleGANv2 <cit.> for this task. The StyleGAN or StyleGANv2 generators are particularly practical for this task as they consist of a mapping network that adds flexibility for manipulation and better projection. The mapping network ($\mathcal{G}_M: \mathbb{R}^{1 \times 512} \rightarrow \mathbb{R}^{18 \times 512}$) inputs a noise vector $\mathbf{z} \in \mathbb{R}^{1 \times 512}$ and generates an extended latent parameters $\mathbf{W} \in \mathbb{R}^{18 \times 512}$. The generator network can synthesize face images from this extended latent parameters fed into its different layers, $\mathcal{G}: \mathbb{R}^{18 \times 512} \rightarrow \mathbb{R}^{h' \times w' \times 3}$. In this work, we optimize only based on $\mathbf{W}$ which we call latent parameters and ignore the mapping network.

During the forward pass of the optimization, we generate an image $\mathbf{G}_i^*$ by the generator network $\mathcal{G}(\mathbf{W}_i^*)$ and extract a set of features for the energy terms that we explain below. As explained in Sec. <ref>, the loss is backpropagated to find a good generation by updating the latent parameters $\mathbf{W}$.

§.§.§ Energy Functions

Photometric Loss :
Obviously, one of the simplest form of supervision is photometric loss which  encourages low-level similarity at the visible part of the image. Although simpler form of photometric loss can be defined as pixel-wise mean absolute difference between two images, we empirically find that log-cosh loss provides smoother convergence. Log-cosh loss can be defined as the following:

\begin{align}
\mathcal{L}_p = \frac{1}{w' \times h' \times 3} \sum^{w' \times h' \times 3} \log \Big( \cosh \big( \mathbf{M}_i \odot (\mathbf{I}_i - \mathbf{G}_i) \big)  \Big)
\end{align}

where $\odot$ stands for element-wise multiplication.

Identity Loss :
Since photometric loss is only concerned by the low-level similarity, it struggles to achieve smooth convergence. Following <cit.>, we exploit identity features from a pretrained face recognition network <cit.> in order to capture good identity resemblance with the original image. Given a network $\mathcal{F}: \mathbb{R}^{h' \times w' \times c} \rightarrow \mathbb{R}^{512}$, we calculate the cosine distance between the identity features of the generated image and the input image as following:

\begin{align}
\mathcal{L}_{id} = 1 - \frac{\mathcal{F}(\mathbf{I}_0) \cdot \mathcal{F}( \mathbf{G}_i))}{||\mathcal{F}(\mathbf{I}_0)||_2 ||\mathcal{F}(\mathbf{G}_i))||_2}
\label{eq:id_loss}
\end{align}

Perceptual Loss :
Following the previous studies <cit.>, we exploit high-level similarity features, known as perceptual loss, to regularize convergence. We empirically choose 9th layer of a VGG-16 network that is pretrained as an ImageNet classifier as below:

\begin{align}
\mathcal{L}_{per} = \sum \log \bigg( \cosh \Big( \mathbf{M}_i \odot \big(\text{VGG}(\mathbf{I}_i) - \text{VGG}(\mathbf{G}_i) \big)\Big)\bigg)
\end{align}

Landmark Loss :
All previous objectives are segmented by the visibility mask that covers the face partially. Therefore, invisible parts become totally relaxed, which leads to ill-aligned generations with the rendered dense landmarks ($\mathbf{S}'_i$). To this end, we propose to minimize the landmark distance between $\mathbf{I}_i$ and $\mathbf{G}_i$. As we rotate 3D mesh with a fixed topology, sparse landmark locations of the rendered images can be easily obtained from the mesh with pre-defined landmark indices ($l \in \mathbb{N}^{68}, l<n$), ($\mathbf{S}'_i(l)$).
In order to extract landmarks of the generated image ($\mathbf{G}_i$) during the optimization [In order to flow the gradient from landmark loss, landmarks need to be computed by differentiable connections. To the best of our knowledge, this is the first attempt of such point-based supervision to a 2D image.], we employ a differentiable landmark estimator <cit.> defined as $\mathcal{K} : \mathbb{R}^{w' \times h' \times 3} \rightarrow \mathbb{R}^{68 \times 2}$. And the loss is expressed as:

\begin{align}
\mathcal{L}_{lan} = \frac{1}{68}\sum^{68} ||\mathcal{K}(\mathbf{G}_i) - \mathbf{S}'_i(l) ||_2
\end{align}

§.§.§ Projection

Initialization by Regression :
Following <cit.>, we train a regressor CNN network $\mathcal{E}: \mathbb{R}^{h' \times w' \times 3} \rightarrow \mathbb{R}^{18 \times 512}$ from random styleGAN generated images ($\mathcal{G}(\mathcal{G}_M(\mathbf{z})), \mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$) to predict their latent parameters ($\mathbf{W} = \mathcal{G}_M(\mathbf{z})$). We initialize $\mathbf{W}$ by the regression of this network for the rendered images, $\mathbf{W}^* = \mathcal{E}(\mathbf{I}_i)$. Initializing the latent parameters with this regression not only accelerate the convergence but also assist optimizer to avoid local minimas.

Optimization :
Given a rendered image $\mathbf{I}_i$, its respective mask $\mathbf{M}_i$, and dense landmarks $\mathbf{S}'_i$, our goal is to find the best latent parameters ($\mathbf{W}_i$) to reconstruct $\mathbf{I}_i$ by a pretrained StyleGANv2 generator $\mathcal{G}$. To this end, we first align $\mathbf{I}_i$, $\mathbf{M}_i$, and $\mathbf{S}'_i$ to the alignment template of StyleGANv2. And, we perform gradient descent optimization by ADAM optimizer <cit.> with a weighted sum of loss functions defined above:

\begin{equation}
\begin{aligned}
\min_{\mathbf{W}_i} \mathcal{L}_{total} (\mathbf{W}_i) = \lambda_p \mathcal{L}_p + \lambda_{id} \mathcal{L}_{id} + \lambda_{per} \mathcal{L}_{per} +  \lambda_{lan} \mathcal{L}_{lan}
\end{aligned}
\label{eq:optimization}
\end{equation}

After convergence, we synthesize a face image with the novel view $\mathbf{c}_i$ by $\mathbf{G}_i = \mathcal{G}(\mathbf{W}_i)$. Finally, we can acquire partial texture in the same way as input texture acquisition in Sec. <ref> by $\mathbf{T}_i = \mathcal{R}'(t_{coord}, \mathbf{G}_i, \mathbf{S'}_i)$.

§.§ Progressive Texture Building for Consistency

In order to generate globally consistent texture maps, we run the optimization for each of the camera views iteratively to progressively improve the texture UV map. After every iteration, we blend the generated UV map ($\mathbf{T}_i$) into the current UV map at the dominated pixels ($\overline{\mathbf{V}}_i$) by that particular camera settings $\mathbf{c}_i$.

\begin{align}
\overline{\mathbf{T}}_i = \overline{\mathbf{V}}_i \odot \mathbf{T}_i +  (1 - \overline{\mathbf{V}}_i) \odot \overline{\mathbf{T}}_{i-1}
\end{align}

Blending: Texture UV-maps are stitched by alpha blending for smooth shift between different UV maps. Also, they are RGB normalized by Gaussian statistics at the intersection of visibility indices $\overline{\mathbf{V}}_0$ and $\overline{\mathbf{V}}_i$ [Normally, these two indices do not overlap, however we build $\overline{\mathbf{V}}_i$ without the handicap to find out true dominated regions. And $\overline{\mathbf{V}}_0$ is from the previous index in which it is given advantage by a factor of 2].

Face Frontalization: Finally, with the complete UV map $\overline{\mathbf{T}}$, we render it once more by a frontal camera and perform a final optimization as in Eq. <ref>
to generate the frontal image of the input image.

§ EXPERIMENTS

We implement the proposed approach in Tensorflow framework <cit.> and it takes around 5 minutes to UV-complete and frontalize an input image. Unfortunately, some of the preprocessing steps are CPU-intensive, therefore is a room for further efficiency. We have used geometry fitting pipeline of GANFit <cit.> as a preprocessing step. Other than pretrained networks for the loss function, the method itself does not require any additional training data. In the following, we illustrate some qualitative and quantitative results of our method.

Comparison of our frontalization to others: Rotate&Render <cit.>, FNM <cit.>, CAPG-GAN <cit.>, FF-GAN <cit.>, HF-PIM <cit.>, HPEN <cit.>, LFW-3D <cit.>

§.§ Unsupervised Texture Model: UTEM

Many 3D texture reconstruction approaches rely on large-scale high-quality 3D appearance data which is costly to collect, difficult to maintain diversity (ethnicity, age) and often kept private due licensing issues. On the other hand, large-scale high-quality 2D face datasets are widely available <cit.> for all. As a by-product of our approach, we build a 3D texture model by completing texture UV-maps for $\sim$1,500 images from CelebA-HQ <cit.> (as can be seen in Fig. <ref> a), without any 3D data collection. After the completion, we train a GAN <cit.> as a GAN-based texture model and perform 3DMM fitting similar to <cit.>. We call this model UTEM and show some generated samples in Fig. <ref> b. The 3DMM fitting results by the original GANFit <cit.> and the one with UTEM texture model can be seen in the last two rows of Fig. <ref>. The reconstructed textures show similar identity recovery and quality as GANFit textures, and it will be available for all.

§.§ Qualitative Results

We run our algorithm on some images in comparison with the recent state-of-the-art approaches, as shown in Fig. <ref>,<ref> and <ref>. Fig. <ref> shows better quality and semantically meaningful UV-maps compared to UV-GAN <cit.> and GANFit <cit.>. Frontalization results in both Fig. <ref> and <ref> look superior to other previous methods in terms of identity-resemblance, artefacts and resolution.

ȷin Julianne_Moore_0012
ıin input,none,pixel,land,perc,all

ıin none,pixel,land,perc,all

ıin input,none,pixel,land,perc,all

Additive ablation study. `+' refers to addition of that loss term compared to the column on the left. (f) refers to all loss term used in this paper, Ours.

ıin 1,2,3,4,5,6,7

Training Samples from OSTeC (Ours)
ıin 1,2,3,4,5,6,7

Generated samples from Unsupervised Texture Model (Ours)
We build a texture model from completed textures from 2D image and train a GAN similar to GANFit <cit.> approach for high-quality texture modeling.

Qualitative results in comparison with other state-of-the-art methods (UV-GAN <cit.>, Rotate&Render <cit.> and GANFit <cit.>). (From Top to down) First block shows input images, second block UV-completion, third block frontalization, and the fourth block texture completion/reconstruction results.

§.§ Quantitative Results

UV Texture Completion.
For the quantitative evaluation of UV texture completion, we employ the UVDB (Multi-PIE <cit.>) dataset released by <cit.>. Following <cit.>, we skip the first 200 subjects, as there is no training, and test on the remaining 137 subjects. We employ two metrics namely peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), which are computed between the predicted UV texture and the ground truth. In Tab. <ref>, the proposed method shows great priority over UV-GAN <cit.> and CE <cit.>, especially for the profile faces.

Methods     Metric     $0^{\circ}$     $\pm30^{\circ}$      $\pm60^{\circ}$     $\pm90^{\circ}$

2* CE <cit.>      PSNR     23.03      21.93      20.27      19.63

SSIM     0.9201     0.8920     0.8881     0.7179

2* UV-GAN <cit.>     PSNR     23.36      22.25      20.53      19.83

SSIM     0.9241     0.8971     0.8919     0.7250

2* Ours       PSNR     23.95     22.54      21.04      20.44

SSIM     0.9282     0.9018     0.8979     0.7462

Quantitative evaluations of UV texture completion on the MultiPIE dataset <cit.> under view changes.

Method     Frontal-Frontal     Frontal-Profile

Human                                                  96.24 $\pm$ 0.67     94.57 $\pm$ 1.10

DR-GAN <cit.>                      97.84 $\pm$ 0.79     93.41 $\pm$ 1.17

DR-GAN+ <cit.>                   98.36 $\pm$ 0.75     93.89 $\pm$ 1.39

PIM <cit.>                              99.44$\pm$0.36       93.10 $\pm$ 1.01

HF-PIM <cit.>                            -                    94.71 $\pm$ 0.83

UVGAN <cit.>                                 98.83 $\pm$ 0.27     93.09  $\pm$ 1.72

+Profile2Frontal                                       -                    93.55  $\pm$ 1.67

+Frontal2Profile                                       -                    93.72  $\pm$ 1.59

+Set2set                                               -                    94.05  $\pm$ 1.73

CASIA-R18-ArcFace                                       99.34 $\pm$ 0.49     93.69 $\pm$ 1.33

+Profile2Frontal                                       -                     94.87 $\pm$ 0.96

+Frontal2Profile                                       -                     95.68 $\pm$ 0.91

+Set2set                                               -                     95.92 $\pm$ 0.87

MS1M-R18-ArcFace                                        99.68  $\pm$0.29       96.14 $\pm$ 1.06

+Profile2Frontal                                       -                       97.06 $\pm$ 0.74

+Frontal2Profile                                       -                       97.43 $\pm$ 0.61

+Set2set                                               -                       97.85 $\pm$ 0.57

Verification accuracy($\%$) comparison on the CFP dataset <cit.>.

Pose-invariant Face Matching.
We evaluate the performance of frontalization of our work on pose-invariant face recognition in the wild. We choose the widely used dataset CFP <cit.>, which focuses on extreme pose face verification. We employ the ArcFace loss <cit.> to train the ResNet-18 networks <cit.> on CASIA-WebFace <cit.> and the refined version of MS1M <cit.>. Note that the backbone of our embedding network is smaller than LightCNN-29 <cit.> used by HF-PIM <cit.> and ResNet-27 used by UV-GAN <cit.>.
As shown in Tab. <ref>, synthesising frontal faces from profile faces significantly improves the accuracy by $1.18\%$ and $0.92\%$ for the ArcFace models trained on CASIA and MS1M, respectively.
Since face frontalization is a very challenging problem, we also synthesise profile faces from frontal faces following <cit.>, which leads to even better results, $95.68\%$ for the CASIA model and $97.43\%$ for the MS1M model.
In addition, we use a view interpolation of $15^\circ$ to generate a set of images for each test face. Then, we use the generated set centres to conduct verification. The accuracy further improves to $95.92\%$ for the CASIA model and $97.85\%$ for the MS1M model, both surpassing recent state-of-art methods (HF-PIM <cit.> and UV-GAN <cit.>) by a large margin.

§.§ Ablation Study

We performed an ablation study to explore the contribution of each loss terms in Fig. <ref>. The study shows that encoder $\mathcal{E}$ starts with a good initialization. $\mathcal{L}_p$ helps to match some low-level features. $\mathcal{L}_{lan}$ aligns generated images to the input geometry, background leakage around the neck. $\mathcal{L}_{per}$ matches mid-level features and finally $\mathcal{L}_{per}$ shows the biggest contribution by precise identity recovery.

§ CONCLUSION

In this paper, we propose an optimization-based one-shot 3D texture completion and frontalization approach by exploiting pretrained 2D image generation networks. Our approach can generate visually remarkable, accurate and identity-resembling complete texture maps and frontalized faces. The experiments show its superiority over other methods by accuracy and face matching at extreme poses.

§ ALGORITHM

We summarize our method in Algorithm <ref> where the notations are described in the original paper.

RGB Face Image: $\mathbf{I}_0$
3DMM Fitting: $(\mathbf{S}, \mathbf{c})$
Novel Camera Views: $\{\mathbf{c}_i\}, i \in \{0 \dots n\}$
Completed UV Texture Map: $\overline{\mathbf{T}}_n$

$\mathbf{T}_0 \leftarrow \mathcal{R}'(t_{coord}, \mathbf{I}_0, \mathbf{S'})$

$i\gets0$ $n$
$\mathbf{S'}_i \leftarrow \mathcal{P}(\mathbf{S}, \mathbf{c}_i)$

$\mathbf{V}_i \leftarrow \text{diag}( \frac{[\mathbf{S}'_i, \mathbf{h}]}{||[\mathbf{S}'_i, \mathbf{h}]||_2} \cdot \mathcal{N}(\mathbf{S}_i)^T)$

$i\gets0$ $n$
$\overline{\mathbf{V}}_i \leftarrow   \bigcap\limits_{i \ne j} (\mathbf{V}_i > \mathbf{V}_j)$

$\mathbf{M}^{UV}_i \leftarrow \big((\mathbf{V}_0 > t_1) \cap (2\mathbf{V}_0 > \mathbf{V}_i)\big) \cup \displaystyle\bigcup\limits_{i>j} \overline{\mathbf{V}}_j$

$\mathbf{M}_i \leftarrow  \mathcal{R}(\mathbf{S'}_i, \mathbf{M}^{UV}_i, t_{coord})$

$\mathbf{I}_i \leftarrow \mathcal{R}(\mathbf{S'}_i, \overline{\mathbf{T}}_{i-1}, t_{coord})$

$\mathbf{W}_i \leftarrow \mathcal{E}(\mathbf{I}_i)$

$\mathbf{W}_i \leftarrow  \argmin_{\mathbf{W}_i} \mathcal{L}_{total} (\mathbf{I}_i, \mathbf{M}_i, \mathbf{W}_i) $

$\mathbf{G}_i \leftarrow \mathcal{G}(\mathbf{W}_i)$

$\mathbf{T}_i \leftarrow \mathcal{R}'(t_{coord}, \mathbf{G}_i, \mathbf{S'}_i)$

$\overline{\mathbf{T}}_i \leftarrow \overline{\mathbf{V}}_i \odot \mathbf{T}_i +  (1 - \overline{\mathbf{V}}_i) \odot \overline{\mathbf{T}}_{i-1} $

One-Shot Texture Completion

§ POSE-INVARIANT FACE MATCHING: MULTIPIE DATASET

For the evaluation in under-controlled scenario, we compare our method with recent state-of-the-art studies, CPF <cit.>, DR-GAN <cit.>, FF-GAN <cit.>, TP-GAN <cit.>, CAPG-GAN <cit.>, PIM <cit.>, HF-PIM <cit.> and Rotate $\&$ Render <cit.>, on the Multi-PIE dataset <cit.>.
The performances are reported following the protocol of the setting 2 <cit.> provided by the Multi-PIE dataset.
Each testing identity has one gallery image from the first appearance. Hence, there are 72,000 and 137 images in the probe and gallery sets, respectively.
In Tab. <ref>, results are reported across different poses.
We employ the strategy of “recognition via generation”  and faces at any pose are first frontalized by our model.
After the face frontalization, the pre-trained ArcFace model trained on MS1M is employed as the feature extractor.
Here, we refer to <cit.> to train ResNet-18, which is slightly smaller than LightCNN-29 <cit.> used by <cit.>.
For those poses less than \({60^{\circ}}\), the performances of most methods are quite good whereas our method almost achieves zero failure rate.
However, the profiles with extreme poses ($> 60^{\circ}$) on are very challenging. For those extreme poses, our method obviously outperforms
other methods, surpassing the “Rotate $\&$ Render” method <cit.> by $0.84\%$ under the pose of $90^{\circ}$.
This impressive recognition performance undoubtedly confirms the effectiveness of the proposed identity-preserved UV texture completion.

Method    \(\pm 15^{\circ}\)     \(\pm 30^{\circ}\)     \(\pm 45^{\circ}\)     \(\pm 60^{\circ}\)     \(\pm 75^{\circ}\)     \(\pm 90^{\circ}\)

CPF <cit.>     95.0    88.5      79.9     61.9      -     -

DR-GAN <cit.>    94.9     91.1     87.2     84.6     -         -

FF-GAN <cit.>    94.6     92.5     89.7     85.2     77.2     61.2

TP-GAN <cit.>    98.7     98.1     95.4     87.7     77.4     64.6

CAPG-GAN <cit.>     99.8     99.6     97.3     90.3     83.1     66.1

PIM <cit.>     99.3     99.0     98.5     98.1     95.0     86.5

HF-PIM <cit.>     99.99     99.98     99.88     99.14     96.40    92.32

R$\&$R <cit.>     -      100      100     99.7     99.3     94.4

Baseline       99.98    99.86        99.80      98.50     96.19     92.06

Ours           100       100          99.88      99.62      99.35     95.24

Rank-1 recognition rates (%) across views on the Multi-PIE dataset <cit.>. The baseline model is ResNet-18 trained on MS1M with the ArcFace loss. Our method further employs face finalization to improve the accuracy.

§ PERFORMANCE ON `IN-THE-WILD' SCENARIO

Following `Pose-Invariant Face Matching' experiment in the original paper, we visualize some of the frontal-profile pairs from CFP dataset <cit.> to evaluate and verify quantitative experiments qualitatively. Figures <ref>,<ref>,<ref> show many pairs of frontal and profile images of the same identity, completed texture UV maps by our method, its rendering, frontalization by our method and cosine similarity scores. The scores are obtained by a ResNet-18 networks <cit.> on CASIA-WebFace <cit.> for:

* `Org.': the pairs of original images

* `UV.': original frontal image and rendered geometry with a completed UV map by our method

* `Frontalized': original frontal image and frontalized image by our method

As can be seen in the figures both qualitatively and quantitatively, our approach can generate excellent quality frontal images and UV texture maps with preserved identity, even under low resolution, extreme pose, occlusion, lighting and expression variations. The cosine similarity scores are mostly improved by the generations of our method compared to the original profile images which verifies the qualitative results.

ıin 004_11-0.2721-0.5243-0.4069,

ıin 095_14-0.2467-0.3714-0.5629,

Qualitative verification of the Pose-Invariant Face Matching experiment. Each block respectively consists of: (1) Original frontal image, (2) Original profile image, (3) Rendered geometry with our completed texture map, (4) Our completed texture map, (5) Our frontalized image.

ıin 212_12-0.3270-0.4017-0.4985,

ıin 310_14-0.3251-0.1906-0.4698,

Qualitative verification of the Pose-Invariant Face Matching experiment. Each block respectively consists of: (1) Original frontal image, (2) Original profile image, (3) Rendered geometry with our completed texture map, (4) Our completed texture map, (5) Our frontalized image.

ıin 438_11-0.2931-0.3546-0.4556,

Successful cases

ıin 005_12-0.2858-0.1868-0.3251,

Failure cases

Qualitative verification of the Pose-Invariant Face Matching experiment. Each block respectively consists of: (1) Original frontal image, (2) Original profile image, (3) Rendered geometry with our completed texture map, (4) Our completed texture map, (5) Our frontalized image.

§ MANIPULATING FRONTALIZED FACES

Frontalization by our approach is achieved by rendering the geometry that is textured by the completed UV map and reconstructing it in StyleGAN <cit.> latent space. Therefore the frontalized images can be manipulated by common StyleGAN manipulation techniques such as interpolation between different identities and changing/adding some facial attributes.

Fig. <ref> illustrates some interpolations performed between the original and the frontalized projections that slowly shift from various poses to the frontal pose. Fig. <ref> shows interpolation between different identities, both in the frontalized and the original projections. Please note that, the frontalized interpolation maintain smoother transition between the identities, whereas the original image projections generates artefacts at the intermediate generations due to exhausted latent parameters. Lastly, Fig. <ref> illustrates attribute manipulation by extracting some attribute directions with <cit.> such as age, gender and expression.

ıin teaser_3,

Interpolation between the original poses and the frontalized versions. First column is the original image. Second column is its projection to the StyleGAN space. Last column is the frontalized generated image by our approach. And other columns are the interpolation in-between.

ıin teaser_3_Ann_Veneman_0010,

Interpolations between different identities. First and Last colums are the original images and other columns are interpolations. Odd rows are interpolating frontal projections and even rows are interpolating the original image projections. Please note that, the frontalized interpolation maintain smoother transition between the identities, whereas the original image projections generates artefacts at the intermediate generations due to exhausted latent parameters.

ıin age,gender,smile

Attribute manipulation by <cit.> can be performed on the frontalized images.

§ LIMITATIONS AND FAILURE CASES

The biggest strength and the biggest weakness of our approach is being an optimization-based method. Usually, the running time takes around 5-10 minutes depending on the convergence speed. This is mainly due to CPU intensive visibility mask and 3D mesh rendering over the iterations. We believe that the code might be optimized to run under 1 minute which is a reasonable running time for an optimization-based method.

Another limitation of an optimization-based method is the danger of local minima. We observed in some cases, optimization gets stuck at local minima, failing to find a good texture completion and frontalization. This is partially addressed by the encoder network $\mathcal{E}$, but emprically we can still observe this behaviour as can be seen in Fig. <ref>.

Another drawback of our approach is that it heavily relies on 3D face reconstruction. Therefore, our method is limited by the accuracy and performance of the 3D reconstruction. That is to say, if some part of the identity cannot be captured by the reconstruction, our method might struggle to compensate. Some of such failure cases are illustrated in Fig. <ref>.