# Patterns for ghost imaging generated by the Ising model

Hamidreza Oliaei-Moghadam
Department of Physics, College of Sciences
Shiraz University
Shiraz 71946-84795, Iran
<EMAIL_ADDRESS>
Chané Moodley
School of Physics
University of the Witwatersrand
Johannesburg 2000, South Africa
<EMAIL_ADDRESS>
Mahmood Hosseini-Farzad
Department of Physics, College of Sciences
Shiraz University
Shiraz 71946-84795, Iran
<EMAIL_ADDRESS>

###### Abstract

We offer new illumination patterns for imaging in all-digital ghost imaging
(GI) systems. The digital patterns, written as computer generated holograms on
spatial light modulators (SLM), are generated by the Ising model, a well-known
ferromagnetic model. In general, Ising patterns are affected by the system
temperature and magnetic field. For a modest number of measurements in GI, our
simulations suggest that low-temperature Ising patterns outperform random
patterns. We also experimentally confirm that Ising masks can be used in
quantum GI. Moreover, we present a strategy for applying Ising patterns of
different temperatures to obtain high-quality images with the number of
measurements approximately equal to the number of pixels in the image.

##  1 Introduction

GI is an alternative imaging technique which can be used for imaging in both
quantum [1, 2, 3] and classical regimes [4, 5, 6, 7]. GI has received much
attention in the last decades due to its advantages over conventional imaging,
such as super-resolution [8, 9, 1, 10], better image quality in a harsh
optical environment [11, 12, 13], and higher detection sensitivity [14, 15].
GI utilizes correlation information between two beams, the object and the
reference beam, to reconstruct an image of the object or scene. The object
beam illuminates the object (scene), then a single-pixel detector monitors all
the passing light. Meanwhile, a spatially resolving detector monitors the
reference beam that does not see the object. None of the detectors can
determine the object’s spatial information on their own. Since the two beams
are spatially correlated, one can reconstruct the image of the object using
information obtained from both the detectors[16, 17]. The correlation of the
object beam and reference beam can be driven by either a quantum source
(entangled photons) or a classical source (pseudo-thermal light) [2, 7].
Although GI is thought to have a wide range of applications, many of them have
yet to be realized in practice due to the complexity of conventional GI’s
optical path structure.

Fortunately, Shapiro improved on the classical GI by omitting the reference
beam and the spatially resolving detector, resulting in a more straightforward
experimental setup. In his novel scheme, the correlation is now between the
signal of the single-pixel detector located behind the object and pre-computed
patterns that are encoded onto a SLM. This scheme is known as “computational
ghost imaging" (CGI) [18], and it is closely related to the single-pixel
imaging field [19, 20]. The ability to image at wavelengths outside the
visible spectrum, where focal plane detector arrays are unavailable or
expensive, is a significant advantage of using single-pixel imaging systems
[21, 22, 23, 24, 25].

SLMs were soon employed in quantum GI systems as well [26, 27]. In this novel
quantum imaging protocol, the object beam (signal photon) interacts with the
object, while the reference beam (idler photon) interacts with a SLM. Two-
photon detection of the signal and idler photons provide information about the
object’s similarity to the pattern displayed on the SLM. This method suggests
a more cost-effective use of photons for quantum imaging.

Despite all the benefits, GI has a significant drawback. Numerous measurements
are required to reconstruct the image, implying that imaging speed is
unsatisfactory. There have been many attempts to improve GI speed by improving
imaging strategies and image reconstruction algorithms [28, 29, 30, 31, 32,
33, 34, 35, 36, 37, 38, 39, 40]. Machine learning algorithms have also been
used to improve picture quality using denoising processes and to decrease
image reconstruction time [41, 42, 43, 44, 45].

In this paper, we introduce new illumination patterns for single-pixel imaging
systems. These patterns are generated by the Ising model, a well-known
ferromagnetic model [46, 47]. We show that employing Ising patterns instead of
traditional random patterns can enhance imaging speed. The main statistical
characteristics of these patterns are determined by two variables: temperature
and magnetization field. Random patterns are, in fact, a subset of Ising
patterns in which the temperature tends to infinity.

##  2 Theory

Imaging using the GI technique requires a large number of measurements because
each measurement can only retrieve a small portion of the object’s spatial
information. An expanded light beam with an intensity cross-section of
$I_{i}(x,y)$ illuminates the object in each measurement. Then a bucket
detector collects and measures all the passing light through the object. The
measured intensity is proportional to

$c_{i}=\int\int dx\;dy\;I_{i}(x,y)T(x,y),$ (1)

where $T(x,y)$ is the transmission function of the object. In traditional GI,
a beam with speckle-pattern cross-sections is used and the image estimation
(ghost image) of the object can be obtained by [48, 4]

$T^{\prime}(x,y)=\frac{1}{M}\sum_{i=1}^{M}\left(c_{i}-\left\langle{c}\right\rangle\right)\mathbf{I}_{i}(x,y),$
(2)

where $\left\langle c\right\rangle=\frac{1}{M}\sum_{i=1}^{M}c_{i}$ is the
ensemble average over M measurements. In SLM-based GI systems, beams with the
desired cross-section can be obtained by properly turning the modulator pixels
on and off. For example, speckle patterns could be simulated with random
patterns [28, 32]. We model each pixel into a magnetic dipole. In this
scenario, pixels on or off indicate whether the spin direction is up or down.
As a result, a pixel array can be viewed as a spin array that can be studied
using appropriate statistical models. In actual images, spatially nearby
pixels tend to have similar values. With this in mind, we employ the Ising
model, which takes into account the interaction of the nearest neighbors,
forcing nearby spins to orient.

###  2.1 Ising model

The Ising model is a well-known model of ferromagnetism in statistical
mechanics. The 2D Ising model includes discrete variables $(s_{i}=\pm 1)$ that
are defined on a quadratic lattice site and represent the spins. Assuming the
existence of an external magnetic field $H$ and considering the interaction of
each spin with its nearest neighbor, the energy of a configuration
$\left\\{s_{1},s_{2},...,s_{N}\right\\}$ is defined as [49, 50]

$E=-J\sum_{\left\langle i,j\right\rangle}s_{i}s_{j}-H\sum_{i=1}^{N}s_{i},$ (3)

where $J>0$ is spin-spin coupling, and $\left\langle i,j\right\rangle$ denotes
that only interactions between neighbors are taken into account. The partition
function is

$Z=\sum_{\left\\{s_{i}=\pm 1,i=1,...,N\right\\}}e^{-\frac{E}{k_{B}T}},$ (4)

where $k_{B}$ indicates the Boltzmann constant and $T$ denotes the
temperature. From the partition function, one can calculate the magnetization
per dipole as follows:

$m=\left\langle s_{j}\right\rangle=\frac{1}{Z}\sum_{\left\\{s_{i}=\pm
1,i=1,...,N\right\\}}s_{j}e^{-\frac{E}{k_{B}T}}.$ (5)

At very low temperatures ($T\ll J$ ), even if $H=0$, it is expected that most
of the spins will have the same direction . At very high temperatures ($T\gg
J$ ), on the other hand, the interaction between the spins becomes
insignificant, and their orientation becomes random $m=0$ . Therefore,
temperature can be regarded as a cause of disorder. In the absence of the
magnetic field, the Ising model has global $Z_{2}$ symmetry under the
transformation $s_{i}\rightarrow-s_{i}$ at all sites. Ising symmetry exists
only in the paramagnetic phase with $m=0$. When the system phase changes from
the paramagnetic phase to the ferromagnetic phase with $m\neq 0$, symmetry
breaks spontaneously. The magnetization $m$, which behaves differently in two
phases, is called the order parameter, and the temperature at which the phase
transition occurs is called the critical temperature. L. Onsager showed that
this phase transition occurs at $k_{B}T_{c}/J=2.269$[51].

In this paper we work in the regime of $T_{c}<T<\infty$. For simplicity, it is
more convenient to work with the inverse temperature instead of $T$. By
defining the inverse temperature parameter $\beta=\frac{1}{k_{B}T}$, the
desired regime is $0<\beta<\beta_{c}$. The spin configuration is completely
random if $\beta=0$ (Fig. 1 (1(a))), but magnetic domains can be formed if
$\beta>0$. As $\beta$ increases, so do the number and size of the magnetic
domains. When the system reaches its critical point $\beta_{c}$, the magnetic
domains appear at all scales (Fig. 1(d)). We can scale $\beta$ to $\beta_{c}$
to get dimensionless parameter $\beta^{\prime}=\frac{\beta}{\beta_{c}}$. To
illustrate system behavior by increasing $\beta^{\prime}$ from 0 to 1, we
plotted spin configurations of a square lattice of size $64\times 64$ for
several values of $\beta^{\prime}=$0, 0.75, 0.9, 1 (Fig. 1).

(a) $\beta^{\prime}=0$

(b) $\beta^{\prime}=0.75$

(c) $\beta^{\prime}=0.9$

(d) $\beta^{\prime}\simeq 1$

Figure 1: Ising patterns at different value of $\beta^{\prime}$. The size of
the speckles grows as $\beta^{\prime}$ is increased.

These spin configurations are generated using the Metropolis method, the most
commonly used Monte Carlo approach for calculating Ising model estimations. By
converting each spin to $s_{i}\rightarrow\frac{s_{i}+1}{2}$, we can obtain
logical patterns from the array of spins. We refer to these logical patterns
as “Ising patterns." Ising patterns with $H=0$ and $\beta^{\prime}<1$
conditions leads to unbiased patterns in pixel values. It should be mentioned
that biased Ising patterns can also be generated if $H\neq 0$ or
$\beta^{\prime}>1$:

  * •

For $\beta^{\prime}<1$, negative (positive) fields cause sparse (dense) Ising
patterns.

  * •

For $\beta^{\prime}>1$ and $H=0$, the generated Ising pattern is dense or
sparse with equal probability.

In general, in GI, speckle patterns with small speckle sizes are appropriate
for measuring image detail, whereas speckle patterns with large speckle sizes
are appropriate for measuring image outline. Ising patterns have speckles of
different sizes and shapes. This feature of Ising patterns makes it possible
to measure the details and outline of the image simultaneously.

##  3 Experimental Realization

Fig. 2 shows a sketch of the quantum ghost imaging optical setup implemented
for this work.

Figure 2: Schematic diagram of the implemented quantum ghost imaging setup.
Entangled bi-photons are produced via spontaneous parametric down-conversion
in the non-linear crystal. The entangled photons are spatially separated and
imaged on two SLMs. Coincidence measurements are made between both paths, and
the coincidences are used to reconstruct an image of the object.

Light from a diode laser (wavelength $\lambda=$ 405 nm) was used to pump a
temperature-controlled type 1 PPKTP non-linear crystal (NLC). The optimal
temperature was set to obtain co-linear emission of entangled bi-photons at
810 nm via spontaneous parametric down-conversion (SPDC). Any unconverted pump
photons were filtered out using a band-pass filter centered at 810 nm (F).
Vertically polarized down-converted photons emitted from the NLC were then
rotated to horizontal polarization by a half-waveplate (HWP). The polarization
rotation was done to ensure optimal modulation of the impinging photons by the
SLMs situated in each arm. The entangled photon beam was separated into two
paths by a 50:50 beam splitter (BS), and each arm was then directed to an SLM,
where the required holograms were displayed. The first SLM displayed the
digital object to be imaged, while the second SLM displayed the Ising mask
patterns. A blazed grating was added to both holograms to separate the first
diffracted order from the zeroth, unmodulated order. Photons from each first
diffracted order were then coupled into multimode optical fibers (MMFs)
connected to avalanche photo-diodes (APDs) for photon detection. Coincidences
were measured within a 25 ns window by a photon-counting device (CC). The
crystal, SLMs, and fiber entrances are placed in conjugate planes. After
removing the DC component, the image is reconstructed as a linear mixture of
the patterns, weighted by the measured coincidences between both arms. LabVIEW
was used to read Ising patterns and show the holograms on each SLM, capture
photon correlations, and reconstruct the real-time image using a linear
combination of patterns weighted by coincidences.

##  4 Results and Discussion

We have generated $64\times 64$ patterns for several $\beta^{\prime}=$ 0, 0.3,
0.5, 0.75 and 0.9 as a pre-measurement step. For simulation results, we used M
= 0.01 $\times$ N, 0.05 $\times$ N, 0.25 $\times$ N, 0.5 $\times$ N, N, 2
$\times$ N (where N = 64 $\times$ 64) samples to construct ghost images of two
different objects, and we used Pearson’s correlation coefficient to check the
quality of the reconstructed images. We also ran an experiment in order to
evaluate the use of Ising patterns in a quantum ghost imaging optical setup.
The experimental results were obtained using M=2 $\times$ N samples of
$\beta^{\prime}=0.75$. Simulation and experimental ghost images of
$\beta^{\prime}=0.75$ are shown in Fig. 3. We can see from these results that
using high $\beta^{\prime}$ Ising patterns instead of random patterns
($\beta^{\prime}=0$) leads to ghost images with a darker background and less
noise. Also, the quality factor graphs ( see Fig. 4 ) show that, given a small
number of measurements, Ising patterns with high $\beta^{\prime}$ are
beneficial to random patterns.

Figure 3: Experimental and simulation ghost images of two objects. We can see
that background of the Ising ghost images with ($\beta^{\prime}=0.75$) is
darker than random ghost images with ($\beta^{\prime}=0$). Figure 4: Quality
factor of simulated ghost images. In a small number of samples ( $M<1024$ ),
pattern set with $\beta^{\prime}=0.9$ gives a higher correlation coefficient.
But when the number of samples increases, pattern sets with lower $\beta$ give
better results.

However, as the number of measurements increases, the advantage of the high
$\beta^{\prime}$ patterns over random patterns decreases, and the images
become blurry. For this reason, it is inconvenient to employ patterns of one
specific $\beta^{\prime}$. A multi-$\beta^{\prime}$ strategy can be utilized
instead. One can use higher $\beta^{\prime}$ patterns to estimate basic
information about the scene, such as the size and location of the items within
the scene. Then, by using this basic information and lower $\beta^{\prime}$
patterns, we can generate new masks. The algorithm for applying the
multi-$\beta^{\prime}$ strategy is as follows:

  1. 1.

Estimate the image using $M_{0}$ samples of $\beta^{\prime}_{0}$ patterns.

  2. 2.

Binarize the estimated image using the Otsu threshold[52].

  3. 3.

Generate new patterns by element-wise multiplication of the binarized image
and patterns with $\beta^{\prime}_{i}<\beta^{\prime}_{0}$.

  4. 4.

Use $M_{i}$ new patterns to reconstruct the new estimation of the image.

  5. 5.

Repeat 2-4 with condition $\beta^{\prime}_{i+1}<\beta^{\prime}_{i}$ (See Fig.
5 )

Figure 5: Diagram for a typical step of applying the multi-$\beta^{\prime}$
strategy.

An example of applying this strategy is shown in Fig. 6. The number of samples
used for this example are

  * •

Fig. 6(6(a)): M = 410 ($\beta^{\prime}=0.9$)+ 614 ($\beta^{\prime}=0.75$) =
1024 (0.25 $\times$ N)

  * •

Fig. 6(6(b)): M = 1024 + 819 ($\beta^{\prime}=0.5$) = 1843 (0.45 $\times$ N)

  * •

Fig. 6(6(c)): M = 1843 + 1229 ($\beta^{\prime}=0.3$) = 3072 (0.75 $\times$ N)

  * •

Fig. 6(6(d)): M = 3072 + 2048 ($\beta^{\prime}=0$) = 5120 (1.25 $\times$ N)

(a) 0.25 $\times$ N

(b) 0.45 $\times$ N

(c) 0.75 $\times$ N

(d) 1.25 $\times$ N

Figure 6: Multi-$\beta^{\prime}$ ghost images. The correlation coefficients
for (a)-(d) is equal to 0.797, 0.896, 0.956, 0.993 respectively.

As can be seen from the Figs. 6(6(c)) and 6(6(c)), we can achieve a high
quality image using $M\simeq N$ samples.

Here, we also like to mention the caustic pattern imaging approach. Caustic
patterns can be formed using slowly varying random phase masks and can be used
for imaging over a wide variety of length scales and wavelengths [53]. Caustic
patterns become random speckle patterns in the far-field, similar to Ising
patterns that become random at the infinite temperature ($\beta^{\prime}=0$).

When employing a small number of samples, the caustic and Ising approaches are
superior to random patterns in different ways. Caustic patterns can estimate
binary images with enhanced edges, while Ising patterns can be used to
estimate binary images with enhanced contrast.

##  5 Conclusion

Ghost imaging offers several advantages over conventional imaging, such as
improved image quality in a harsh optical environment, but it has a slow
imaging speed. We introduced new pattern sets for SLM-based ghost imaging
systems called Ising patterns to speed up the imaging process. Ising patterns
are generally determined by two parameters: the magnetic field $H$ and the
inverse temperature $\beta$. In this paper, we set $H=0$ and used Ising
patterns of several $\beta$ values to reconstruct images of two objects. Our
simulations results showed that Ising patterns with higher $\beta$ are
beneficial to random patterns when the number of measurements is low and our
experimental results verified the usage of Ising masks in quantum GI. We also
introduced a multi-$\beta$ strategy for implementing Ising patterns. Using
this strategy, we were able to reconstruct high-quality images with $M\simeq
N$ samples.

##  6 acknowledgments

The authors would like to thank Professor A. Forbes and Dr. J. T. Francis for
helpful discussions and their generous support during the preparation of this
manuscript.

## References

  * [1] H. Wang, S. Han, and M. I. Kolobov, Opt. Express 20, 23235 (2012).
  * [2] R. S. Bennink, S. J. Bentley, R. W. Boyd, and J. C. Howell, Phys. Rev. Lett. 92, 033601 (2004).
  * [3] T. B. Pittman, Y. H. Shih, D. V. Strekalov, and A. V. Sergienko, Phys. Rev. A 52, R3429 (1995).
  * [4] A. Valencia, G. Scarcelli, M. D’Angelo, and Y. Shih, Phys. Rev. Lett. 94, 063601 (2005).
  * [5] L. Basano and P. Ottonello, Appl. Phys. Lett. 89, 091109 (2006).
  * [6] Z. Ye, D. Sheng, Z. Hao, H. B. Wang, J. Xiong, X. Wang, and W. Jin, Appl. Phys. Lett. 117, 091103 (2020).
  * [7] A. Gatti, E. Brambilla, M. Bache, and L. A. Lugiato, Phys. Rev. Lett. 93, 093602 (2004).
  * [8] Z. Chen, J. Shi, Y. Li, Q. Li, and G. Zeng, Eur. Phys. J. - Appl. Phys. 67 (2014).
  * [9] W. Gong and S. Han, Phys. Lett. A 376, 1519 (2012).
  * [10] F.-H. Kong, L.-A. Wu, Q. Fu, S.-Y. Meng, and X.-H. Chen, Opt. Lett. 42, 5290 (2017).
  * [11] C. Fan, C. Qiao, D. Shi, H. Shen, J. Zhang, P. Zhang, and Y. Wang, Opt. Express 21, 2050 (2013).
  * [12] R. E. Meyers, K. S. Deacon, and Y. Shih, Appl. Phys. Lett. 98, 111115 (2011).
  * [13] C. A. Osorio Quero, D. Durini, J. Rangel-Magdaleno, and J. Martinez-Carranza, Rev. Sci. Instrum. 92, 111501 (2021)
  * [14] A.-X. Zhang, B.-B. Wang, L.-M. Chen, L.-A. Wu, and Y.-H. He, Opt. 5, 374 (2018).
  * [15] Y. Zhang, W. Li, H. Wu, Y. Chen, X. Su, Y. Xiao, Z. Wang, and Y. Gu, Opt. Commun. 441, 45 (2019).
  * [16] J. H. Shapiro and R. W. Boyd, Quantum Inf. Process. 11, 949 (2012).
  * [17] M. J. Padgett and R. W. Boyd, Philos. Trans. R. Soc. A Math. Phys. Eng. Sci. 375, 20160233 (2017).
  * [18] J. H. Shapiro, Phys. Rev. A 78, 061802(R) (2008).
  * [19] G. M. Gibson, S. D. Johnson, and M. J. Padgett, Opt. Express 28, 28190 (2020).
  * [20] M. P. Edgar, G. M. Gibson, and M. J. Padgett, Nat. Photonics 13, 13 (2018).
  * [21] G. M. Gibson, B. Sun, M. P. Edgar, D. B. Phillips, N. Hempler, G. T. Maker, G. P. A. Malcolm, and M. J. Padgett, Opt. Express 25, 2998 (2017).
  * [22] M. F. Duarte, M. A. Davenport, D. Takbar, J. N. Laska, T. Sun, K. F. Kelly, and R. G. Baraniuk, IEEE Signal Process. Mag. 25, 83 (2008).
  * [23] W. L. Chan, K. Charan, D. Takhar, K. F. Kelly, R. G. Baraniuk, and D. M. Mittleman, Appl. Phys. Lett. 93, 121105 (2008).
  * [24] D. Pelliccia, A. Rack, M. Scheel, V. Cantelli, and D. M. Paganin, Phys. Rev. Lett. 117, 113902 (2016).
  * [25] B. Sun, M. P. Edgar, R. Bowman, L. E. Vittert, S. Welsh, A. Bowman, and M. J. Padgett, Science. 340, 844 (2013).
  * [26] P. Zerom, K. W. C. Chan, J. C. Howell, and R. W. Boyd, Phys. Rev. A 84, 061804(R) (2011).
  * [27] G. A. Howland and J. C. Howell, Phys. Rev. X 3, 011013 (2013).
  * [28] O. Katz, Y. Bromberg, and Y. Silberberg, Appl. Phys. Lett. 95, 131110 (2009).
  * [29] F. Ferri, D. Magatti, L. A. Lugiato, and A. Gatti, Phys. Rev. Lett. 104, 253603 (2010).
  * [30] K. W. C. Chan, M. N. O’Sullivan, and R. W. Boyd, Opt. Lett. 34, 3343 (2009).
  * [31] C. Zhao, W. Gong, M. Chen, E. Li, H. Wang, W. Xu, and S. Han, Appl. Phys. Let. 101, 141123 (2012).
  * [32] B. Sun, S. S. Welsh, M. P. Edgar, J. H. Shapiro, and M. J. Padgett , Opt. Express 20, 16892 (2012).
  * [33] S. Sun, W. T. Liu, H. Z. Lin, E. F. Zhang, J. Y. Liu, Q. Li, and P. X. Chen, Sci. Rep. 6, 1 (2016).
  * [34] W. Wang, Y. P. Wang, J. Li, X. Yang, and Y. Wu, Opt. Lett. 39, 5150 (2014).
  * [35] W.-K. Yu, M.-F. Li, X.-R. Yao, X.-F. Liu, L.-A. Wu, and G.-J. Zhai, Opt. Express 22, 7133 (2014).
  * [36] L. Wang and S. Zhao, Photonics Res. 4, 240 (2016).
  * [37] Z. Zhang, X. Wang, G. Zheng, and J. Zhong, Opt. Express 25, 19619 (2017).
  * [38] B. Luo, G. Wu, H. Guo, L. Yin, and P. Yin, Opt. Express 26, 23093 (2018).
  * [39] R. Zhu, G. Li, and Y. Guo, Int. J. Theor. Phys. 58, 1215 (2019).
  * [40] L. Wang and S. Zhao, Opt. Lasers Eng. 139, 106473 (2021).
  * [41] M. Lyu, W. Wang, H. H. H. Wang, H. H. H. Wang, G. Li, N. Chen, and G. Situ, Sci. Rep. 7, 1 (2017).
  * [42] S. Rizvi, J. Cao, K. Zhang, and Q. Hao, Sci. Rep. 10, 1 (2020).
  * [43] N. Radwell, S. D. Johnson, M. P. Edgar, C. F. Higham, R. Murray-Smith, and M. J. Padgett, Appl. Phys. Lett. 115, 231101 (2019).
  * [44] C. Moodley, B. Sephton, V. Rodríguez-Fajardo, and A. Forbes, Sci. Rep, 11, 1 (2021).
  * [45] V. Rodríguez-Fajardo, J. Pinnell, and A. Forbes, J. Mod. Opt. 67, 1176 (2020).
  * [46] G. F. Newell and E. W. Montroll, Rev. Mod. Phys. 25, 353 (1953).
  * [47] B. A. Cipra, Am. Math. Mon. 94, 937 (2018).
  * [48] D. S. Simon, G. Jaeger, and A. V. Sergienko, Quantum Metrol. Imaging, Commun. (Springer, Cham, 2017) pp. 131–158.
  * [49] R. Pathria and P. D. Beale, Stat. Mech. 4th ed. (Academic Press, 2022) , pp. 417–486.
  * [50] I. Herbut, A Mod. Approach to Crit. Phenom. (Cambridge University Press, 2007) pp. 1–22.
  * [51] L. Onsager, Phys. Rev. 65, 117 (1944).
  * [52] N. Otsu, IEEE Trans. Syst. Man Cybern. Syst. 9, 62 (1979).
  * [53] E. Toninelli, D. Stellinga, B. Sephton, A. Forbes, and M. J. Padgett, Sci. Re. 10, 1 (2020).


# Investigation of Architectures and Receptive Fields for Appearance-based
Gaze Estimation

Yunhan Wang1 Xiangwei Shi1 Shalini De Mello2 Hyung Jin Chang3 Xucong Zhang1
1Computer Vision Lab, Delft University of Technology 2NVIDIA
3School of Computer Science, University of Birmingham

###### Abstract

With the rapid development of deep learning technology in the past decade,
appearance-based gaze estimation has attracted great attention from both
computer vision and human-computer interaction research communities.
Fascinating methods were proposed with variant mechanisms including soft
attention, hard attention, two-eye asymmetry, feature disentanglement,
rotation consistency, and contrastive learning. Most of these methods take the
single-face or multi-region as input, yet the basic architecture of gaze
estimation has not been fully explored. In this paper, we reveal the fact that
tuning a few simple parameters of a ResNet architecture can outperform most of
the existing state-of-the-art methods for the gaze estimation task on three
popular datasets. With our extensive experiments, we conclude that the stride
number, input image resolution, and multi-region architecture are critical for
the gaze estimation performance while their effectiveness dependent on the
quality of the input face image. We obtain the state-of-the-art performances
on three datasets with 3.64 on ETH-XGaze, 4.50 on MPIIFaceGaze, and 9.13 on
Gaze360 degrees gaze estimation error by taking ResNet-50 as the backbone.

##  1 Introduction

Eye gaze can serve as a cue to model a person’s cognitive process and analyze
human visual attention [44]. Various eye-tracking applications have been
proposed, such as diagnostic interpretation [3], human-computer interaction
[35, 45], visual marketing [40], and augmented and virtual reality [32, 1].
Appearance-based gaze estimation alleviates the requirement of accurate 3D
model fitting by directly regressing from the input eye/face image to the gaze
target [36]. With the recent development of deep learning methods, appearance-
based gaze estimation has attracted large attention in the computer vision
community [15, 11].

The early works usually take a single-eye image [27, 34, 14, 48, 50] or a
combination of two eyes [20, 19] as the input. Later methods demonstrate the
multi-region method, i.e. two eye patches and single face region, is more
effective than the eye-region only methods [25]. Taking a single-face image
could further improve the gaze estimation performance, yet enlarging input
image resolution plays an important role [49]. Variant methods have been
proposed for the gaze estimation, such as generative model [37], pictorial
representation [30], unsupervised learning [42], hard attention [47], two-eye
asymmetry [12], coarse-to-fine [8], weakly supervision [23], rotation
consistency [2], self-adversarial [7], and vision transformer [9]. Some of the
previous works imply the input image resolution could be critical for the
final gaze estimation performance [49, 43, 4], and whether to take eye, face,
or multi-region as the input is still a mystery. It is not clear which method
we should choose when dealing with real-world settings given the specific
devices, applications, and environments. In addition, while these deep
learning approaches have outperformed traditional methods across datasets,
there still exists a considerable gap towards a “perfect gaze estimator”, as
the general gaze estimation error is around four to five degrees on high-
resolution controlled laboratory setting ETH-XGaze datasets [43], four degrees
on the real-world laptop setting MPIIFaceGaze dataset [49], two centimeters on
the cellphone screen GazeCapture [25], and ten degrees on the challenging
outdoor setting Gaze360 [21].

In this paper, we improve the gaze estimation performance on multiple datasets
by examining the basic strategy of taking high and low-resolution images as
input, changing the stride of the first convolutional layer, taking a single-
face image as input, and taking multi-region as input. Taking single-face or
multi-region as input is a popular choice for the current gaze estimation
methods, yet there is no conclusion about which one we should pick. The input
image resolution and stride both affect the reception field of the neural
networks and, thus, have an impact on the final gaze estimation performances.

Our main findings are:

  * •

Decreasing the stride of a CNN’s first convolutional layer effectively
improves performance on high-resolution datasets.

  * •

Increasing input image resolution effectively improves performance on high-
resolution datasets.

  * •

Multi-region architecture (left eye, right eye, and face images each with a
CNN backbone) performs well on high-resolution datasets, while not on low-
resolution datasets.

##  2 Related work

There are two main categories of gaze estimation methods: model-based and
appearance-based [17]. Model-based methods employ geometric eye models and
detect geometric features to estimate gaze [33, 51, 6]. However, the accuracy
of model-based methods can suffer from in-the-wild settings [48] and sometimes
require a time-consuming process of collecting subject-specific parameters,
such as cornea radii, cornea center, and kappa angles [15].

Appearance-based gaze estimation attempts to regress gaze direction from eye
or face images. Most of the recent appearance-based approaches adopt a CNN-
based architecture. Zhang _et al_. proposed the in-the-wild MPIIGaze dataset
and demonstrated the exceptional performance of CNN-based model in this
setting [48]. Krafka _et al_. proposed the large-scale GazeCapture dataset and
the multi-region multi-branch iTracker framework that takes left eye, right
eye, face, and face grid location as input to separate branches of CNN to
estimate gaze [25]. Zhang _et al_. implemented a model that takes only full-
face images as input [49]. Park _et al_. proposed a pictorial gaze estimation
model that first regresses the image to an intermediate gazemap and then
estimates gaze from that [31]. Kellnhofer _et al_. provide an LSTM-based model
with pinball loss and the Gaze360 dataset in unconstrained indoor and outdoor
environments with a wide range of head poses [21]. Zhang _et al_. developed
the large-scale high-resolution ETH-XGaze dataset in a constraint environment
with extreme head poses and gaze variations [43]. Furthermore, various gaze
estimation approaches are proposed, such as few-shot learning [29], bayesian
learning [38], unsupervised learning [42], weakly-supervised learning [24],
contrastive learning [39]. Recent research has demonstrated that methods based
on Vision Transformers [13, 26, 5] achieve exceptional performance compared to
previous CNN-based methods. Cheng _et al_. proposed the first transformer-
based architecture for gaze estimation [10].

##  3 Background

The gaze estimation task is strongly relevant to the eye region since eyeball
rotation is the only factor that determines the gaze direction. Unfortunately,
accurate eyeball rotation is difficult so it becomes crucial to include the
rest of the face region for the appearance-based gaze estimation methods.
However, the balance between the eye region and the rest of the face region
has not been fully explored before.

In this section, we describe the strategy of stride and resolution selection
for gaze estimation and a multi-region multi-branch architecture.

###  3.1 Receptive filed

In deep convolutional neural networks (CNNs), a basic concept is the receptive
field, also known as the field of view, which refers to the region of the
input that influences a unit in a specific layer of the network. Unlike fully
connected networks, where each unit’s value depends on the entire input, a
unit in convolutional networks is influenced by a localized region of the
input corresponding to the unit’s receptive field [28]. In various tasks,
particularly in dense prediction tasks such as semantic image segmentation,
stereo, and optical flow estimation, where predictions are made for individual
pixels in the input image, it is crucial to ensure that each output pixel has
a substantial receptive field. This ensures that no vital information is
overlooked during the prediction process.

In general, there are multiple ways to change the receptive field inside CNNs.
We take two ways of them, stride and input image size, for gaze estimation.

###  3.2 Stride

As a parameter of the neural network’s filter, stride determines the amount of
shifting applied to the input image or feature. Thus, the smaller stride
results in a larger receptive field. By changing the stride of the first
convolutional layer of a gaze estimation network, we enlarge the receptive
field for the units in subsequent layers until the bottom layer’s units.
Furthermore, we investigate changing the stride of the sliding patch of a
transformer-like model, Poolformer [41], where the self-attention layers in
the transformer were replaced with pooling layers.

###  3.3 Input image resolution

Enlarging input image resolution can consequently increase the receptive
fields of CNNs. A simple way to enlarge the input image is the interpolation
of existing pixel values that do not add any information to the input image.
Instead, we enlarge the input image during the data normalization process [46]
in the image warping step. The total amount of information contained by the
input image is limited by the raw image resolution, i.e. there would be no
benefits of enlarging the input image if the size becomes larger than the face
in the original raw image. Enlarging the input image brings heavy computation
costs due to more convolution operations and longer feature vectors before the
output layer. In this study, we only experiment with two image resolutions as
224$\times$224 and 448$\times$448 pixels.

###  3.4 Gaze estimation model

Theoretically, eyeball rotation is the only factor to determine the gaze
direction and the eye region is the only required input for the gaze
estimation model. Nonetheless, taking the full face instead of the eye as
input improves the gaze estimation performance empirically [49]. Since the eye
region should be the most critical part of the gaze estimation task compared
to the rest of the face, the eye region should have a larger receptive field
than the rest of the face. The multi-region method reflects such an intuition
by cropping and enlarging the two eye regions as parallel input for the gaze
estimator [25].

In this paper, we explore the potential of full-face and multi-region gaze
estimation. For the multi-region method, the model delegates a network branch
for the left eye, right eye, and full face, and adopts a fully connected layer
to regress the outputs from the three branches to the gaze direction.
ResNet-50 serves as the backbone model for each branch. This model has a
similar fashion to iTracker [25], yet it does not share model weights for eye
regions and excludes the branch of the face grid location. We further study
the necessity of using different backbones for different regions.

##  4 Experiments

Input Image Resolution |  Methods Datasets | ETH-XGaze | MPIIFaceGaze | Gaze360
---|---|---|---|---
224$\times$224 | Res50 (stride 2) | $4.50^{\circ}$ | $4.71^{\circ}$ | $9.21^{\circ}$
| Res50 (stride 1) | $4.00^{\circ}$ | $4.51^{\circ}$ | $9.19^{\circ}$
448$\times$448 | Res50 (stride 2) | $3.95^{\circ}$ | $4.53^{\circ}$ | $9.13^{\circ}$
| Res50 (stride 1) | $3.76^{\circ}$ | $4.50^{\circ}$ | $9.56^{\circ}$
Table 1: Comparison of methods on datasets in $224\times 224$ and $448\times
448$ pixels resolutions. ETH-XGaze has the highest raw image resolution,
followed by MPIIFaceGaze then Gaze360. As we change the stride from two to
one, the gaze error improves by a greater scale on the dataset with higher raw
image resolution.

We experimented with manipulating strides, input resolutions, and model
architectures across three datasets, ETH-XGaze [43], MPIIFaceGaze [49], and
Gaze360 [21]. We also tested using self-attention in a transformer-like model
and variant architectures designed for multi-region CNN.

###  4.1 Settings

We applied the data normalization method introduced in [46] to cancel out the
geometric variability caused by various head poses and distances to the camera
by converting input images and gaze ground truth to a normalized space. For
training CNN-based models, we implemented ResNet-50 [18] as the CNN backbone.
We use the Adam optimizer [22] with the initial learning rate set to 0.0001.
For different model architectures, the batch sizes were set according to the
limitation of GPU memory. The model was trained for 30 epochs and we divide
the learning rate by 10 for every 10 epochs. We pick the results of the 30th
epoch for ETH-XGaze, and the 25th epoch for the MPIIFaceGaze according to the
previous paper settings. The results of Gaze360 are tested based on the
validation performance. We did not observe significant performance differences
between the 30th and 25th epochs. For training transformer-based models, we
adopted the ideas introduced in [10] to train the model with 50 epochs, and an
initial learning rate of 0.0005. The learning rate decay factor is set to 0.5
for every 10 epochs. We implemented a gradual learning rate warm-up procedure
[16] in the first three epochs. The batch size was set to 100. If the batch
size could not fit the model in an NVIDIA A40 GPU, we set it to the largest
number that can fit the model on A40.

###  4.2 Datasets

ETH-XGaze [43] contains 1.1 million images and the raw image resolution is
$6000\times 4000$. The images were collected in the laboratory and with
extreme head pose, gaze variations, and 16 illumination conditions. For
person-independent gaze estimation, 80 subjects are set for training and 15
for testing. We obtained the face and eye input images using the normalization
method [46] on raw images.

MPIIFaceGaze [49] contains 214 thousand images and the raw image resolution is
$1280\times 720$. The images were collected in the wild and with daily life
illustrations. MPIIFaceGaze contains 15 subjects and the typical evaluation
procedure is to conduct a cross-subject 15-fold evaluation. For the single
region training, the original input image size is $448\times 488$ and we
downsampled them to be $224\times 224$ for later usage. For the multi-region
training, we use the normalization method [46] to obtain the face and two-eye
images. Note that, the coordinate system of gaze labels after normalization is
not aligned with the one for single region training, but it is aligned with
the one from the ETH-XGaze dataset.

Gaze360 [21] contains 172 thousand images and the raw head image resolution
span from around $100\times 100$ to $500\times 500$. Gaze360 is a diverse
dataset containing 238 subjects in unconstrained indoor and outdoor
environments with a wide range of head poses. We performed the aforementioned
data normalization method [46] on Gaze360 and obtain face and eye input
images. The dataset is split into train, validation, test and unused subsets.

###  4.3 Results

To gain a deeper understanding of the effect of our design choices, we assess
the performance of our approach across diverse settings and configurations.

Model Dataset | ETH-XGaze
---|---
PoolFormer-24 (stride 4, with self-attention) | $4.73^{\circ}$
PoolFormer-24 (stride 4) | $4.56^{\circ}$
PoolFormer-24 (stride 2) | $3.98^{\circ}$
PoolFormer-24 (stride 1) | $3.67^{\circ}$
Table 2: Comparison of PoolFormer [41] with or without self-attention and with
different strides on ETH-XGaze of $224\times 224$ pixels input image
resolution.

####  4.3.1 Stride

The default stride parameter in the first convolutional layer in the ResNet
series is set to be two to reduce the size of the receptive field. However, it
may not be applicable to regression tasks such as gaze estimation. As shown in
the first and second rows of Tab. 1, by changing the stride from two to one
for the input image resolution $224\times 224$ pixels, an 11.1% performance
improvement ($4.50^{\circ}\rightarrow 4.00^{\circ}$) is achieved on ETH-XGaze,
a 4.2% performance improvement ($4.71^{\circ}\rightarrow 4.51^{\circ}$) is
achieved on MPIIFaceGaze. A similar trend can be observed for the input image
resolution $448\times 448$ pixels settings for the ETH-XGaze. However, there
is no significant performance change on Gaze360 with either input image
resolution settings and MPIIFaceGaze with $448\times 448$ pixels input image
setting. We conclude that percentages of performance improvements on these
three datasets are aligned with raw image resolution increases, _i.e_. the
higher raw image resolution, the higher performance improvement by decreasing
the stride number.

To valid the effect of stride on gaze estimation with different architecture,
we also conducted experiments with PoolFormer-24 [41] that collects input
patches with different strides. Note we only changed the stride in the first
stage of the PoolFormer architecture, which is used to slide image patches
from patch embedding. As shown in Tab. 2, PoolFormer yields a gaze error of
$4.56^{\circ}$ with the default stride as four, and decreasing stride number
yields a drastic improvement of 19.5% ($4.56^{\circ}\rightarrow
3.67^{\circ}$). It shows that stride is critical for the gaze estimation task
for different architectures. The main reason could be that the model needs to
extract fine-level features around the eye region. Since vision transformers
are famous for their self-attention modules, we replaced the pooling layers
with self-attention layers in the top two out of four stages in PoolFormer-24
to better capture global information at early stages. However, we observe an
increase in gaze error in the setting with a self-attention based module. It
could be that the amount of training data is insufficient to train the self-
attention modules [13].

Methods Dataset | ETH-XGaze | MPIIFaceGaze | Gaze360
---|---|---|---
No shared eye net (stride 2) | $3.88^{\circ}$ | $4.62^{\circ}$ | $9.26^{\circ}$
No shared eye net (stride 1) | $3.64^{\circ}$ | $4.61^{\circ}$ | $9.26^{\circ}$
Shared eye nets (stride 2) | $3.70^{\circ}$ | $4.51^{\circ}$ | $9.28^{\circ}$
Shared eye nets (stride 1) | $3.69^{\circ}$ | $4.62^{\circ}$ | $9.13^{\circ}$
Table 3: Comparison of variants of multi-region ResNet-50 on different
datasets in $224\times 224$ resolution. We experimented with sharing or not
sharing the eye net, and different stride numbers.

####  4.3.2 Input image resolution

Besides the stride, input image resolution can also change the size of the
receptive field. We examined ResNet-50 with two different image resolutions as
$224\times 224$ and $448\times 448$ pixels. To achieve different image
resolutions, we change the parameters during the data normalization procedure.
As seen in first the third rows of Tab 1, by increasing input resolution
($224\times 224\rightarrow 448\times 448$), ResNet-50 with a stride of two in
the first convolutional layer can yield improvements of $12.2\%$
($4.50^{\circ}\rightarrow 3.95^{\circ}$) on XGaze and $3.8\%$
($4.71^{\circ}\rightarrow 4.53^{\circ}$) MPIIFaceGaze, respectively. It shows
that increasing image resolution can improve gaze estimation performance based
on the raw image resolution of the dataset. The raw face size in the original
image is around $1000\times 1000$ pixels and $500\times 500$ pixels on XGaze
and MPIIFaceGaze datasets, respectively. We expect even more improvement on
the ETH-XGaze dataset when increasing the input image resolution to the
networks to be $1000\times 1000$ pixels. However, such a model would require
more energy and memory to train.

We further combine our finding of the stride effect that we changed the stride
of the first convolutional layer in ResNet-50 from two to one. By comparing
results in the third and fourth rows in Tab 1, we can see that improvements by
decreasing stride yield a small improvement ($4.8\%$, $3.95^{\circ}\rightarrow
3.76^{\circ}$) on XGaze and no improvement on MPIIFaceGaze. Note decreasing
the stride from two to one results in worse performance on Gaze360
($9.13^{\circ}\rightarrow 9.56^{\circ}$). It could be that the faces crops on
Gaze360 are much smaller than the $448\times 448$ pixels that the enlarging
input image resolution introduces noise with interpolation.

In general, we found the input image resolution and stride number could
improve the gaze estimation performance while the relative improvements are
dependent on the image resolution of the raw input frame.

####  4.3.3 Multi-region CNN

With the assumption that gaze estimation performance improves by increasing
the receptive field from extracting detailed features on eye regions, the
multi-region method is assumed to perform better than the single-face input
methods.

We conducted the experiment of the multi-region CNN model that consists of
three separated ResNet-50 networks for the left eye, right eye, and face
patches, respectively. For the sake of computation time, we use $224\times
224$ pixels resolution input image size. The cropped left and right eye
patches are obtained by the data normalization method [46] on the raw images.
The features of the three networks are concatenated and fed into the last
fully connected layer to regress to the gaze direction.

As shown in the first row of Tab. 3, the multi-region architecture achieves
3.88 degrees gaze error on ETH-XGaze, which is significantly better than the
single-face input method (4.5 degrees). We further investigated variations of
the multi-region architectures. We change the stride of the first
convolutional layers from two to one, which achieves $6.2\%$ improvement
($3.88^{\circ}\rightarrow 3.64^{\circ}$). It is the same conclusion in 4.3.1
that decreasing the stride can improve the performance of gaze estimation.

Since previous works have variations of sharing the network for left and right
eye patches or not, we implemented a variant of the multi-region method in
that the left and right eye nets are shared. It can be seen from the table
that the sharing eye nets result in better performance
($3.88^{\circ}\rightarrow 3.70^{\circ}$) on ETH-XGaze. To explore the
limitations of different strategies, we experimented with a multi-region of
shared eye nets and the stride to be one in the first convolutional layer.
However, the result on the ETH-XGaze dataset does not show an improvement
compared to the model with the stride one ($3.70^{\circ}$ vs. $3.69^{\circ}$).
It could be that the feature extraction with cropped eye region and stride one
would be similar to the eye region size in the raw images. Therefore, changing
the stride number cannot yield performance improvement.

A similar trend can be observed on the MPIIFaceGaze dataset as switching from
a single face to a multi-region model with input image resolution $224\times
224$ results in a performance improvement ($4.71^{\circ}$ vs. $4.62^{\circ}$,
stride with two), and the shared eye net can further lower the gaze error to
$4.51^{\circ}$. However, there is no noticeable improvement with changing the
stride.

For the Gaze360, we do not observe the significant changes with different
architectures as shown in the last column of Tab. 3. It could be that the raw
face sizes on Gaze360 are relatively small, thus, fine-level feature
extraction could not boost the performance.

##  5 Conclusion

In this study, we have investigated the substantial potential of optimization
of fundamental parameters, namely input image resolution, stride, and input
patches to achieve exceptional performance for the gaze estimation task.
Through a series of extensive experiments, we have derived decreasing the
stride of the first convolutional layer, increasing the input image
resolution, and switching to multi-region architecture lead to a noticeable
improvement in performance when dealing with high-resolution datasets.
However, while these strategies exhibit strong performance on high-resolution
datasets, their effectiveness diminishes when applied to low-resolution
datasets. These findings collectively highlight the importance of parameter
selection, including input image resolution, stride, and the choice of
architecture, in optimizing performance for the gaze estimation task across
different dataset resolutions. With our optimized yet simple architecture, we
successfully achieved state-of-the-art gaze estimation performances on three
popular datasets.

## References

  * [1] RT Azuma.  A survey of augmented reality presence teleoperators virtual environ., 6.  1997\.
  * [2] Yiwei Bao, Yunfei Liu, Haofei Wang, and Feng Lu.  Generalizing gaze estimation with rotation consistency.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4207–4216, June 2022.
  * [3] Tad T Brunyé, Trafton Drew, Donald L Weaver, and Joann G Elmore.  A review of eye tracking for understanding and improving diagnostic interpretation.  Cognitive research: principles and implications, 4:1–16, 2019.
  * [4] Xin Cai, Boyu Chen, Jiabei Zeng, Jiajun Zhang, Yunjia Sun, Xiao Wang, Zhilong Ji, Xiao Liu, Xilin Chen, and Shiguang Shan.  Gaze estimation with an ensemble of four architectures.  arXiv, 2021.
  * [5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In ECCV, pages 213–229. Springer, 2020.
  * [6] Jixu Chen and Qiang Ji.  3d gaze estimation with a single camera without ir illumination.  In 2008 19th International Conference on Pattern Recognition, pages 1–4. IEEE, 2008.
  * [7] Yihua Cheng, Yiwei Bao, and Feng Lu.  Puregaze: Purifying gaze feature for generalizable gaze estimation.  Proc. AAAI, 2022.
  * [8] Yihua Cheng, Shiyao Huang, Fei Wang, Chen Qian, and Feng Lu.  A coarse-to-fine adaptive network for appearance-based gaze estimation.  In Proc. AAAI, 2020.
  * [9] Yihua Cheng and Feng Lu.  Gaze estimation using transformer.  In Proc. ICPR, 2022.
  * [10] Yihua Cheng and Feng Lu.  Gaze estimation using transformer.  In ICPR, pages 3341–3347. IEEE, 2022.
  * [11] Yihua Cheng, Haofei Wang, Yiwei Bao, and Feng Lu.  Appearance-based gaze estimation with deep learning: A review and benchmark.  arXiv preprint arXiv:2104.12668, 2021.
  * [12] Yihua Cheng, Xucong Zhang, Feng Lu, and Yoichi Sato.  Gaze estimation by exploring two-eye asymmetry.  IEEE TIP, 29:5259–5272, 2020.
  * [13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [14] Kenneth Alberto Funes Mora, Florent Monay, and Jean-Marc Odobez.  EYEDIAP: A database for the development and evaluation of gaze estimation algorithms from rgb and rgb-d cameras.  In Proc. ACM Symp. on Eye Tracking Research and Applications, pages 255–258, 2014.
  * [15] Shreya Ghosh, Abhinav Dhall, Munawar Hayat, Jarrod Knibbe, and Qiang Ji.  Automatic gaze analysis: A survey of deep learning based approaches.  arXiv preprint arXiv:2108.05479, 2021.
  * [16] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He.  Accurate, large minibatch sgd: Training imagenet in 1 hour.  arXiv preprint arXiv:1706.02677, 2017.
  * [17] Dan Witzner Hansen and Qiang Ji.  In the eye of the beholder: A survey of models for eyes and gaze.  IEEE transactions on pattern analysis and machine intelligence, 32(3):478–500, 2009.
  * [18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [19] Qiuhai He, Xiaopeng Hong, Xiujuan Chai, Jukka Holappa, Guoying Zhao, Xilin Chen, and Matti Pietikäinen.  Omeg: Oulu multi-pose eye gaze dataset.  In Image Analysis, pages 418–427. 2015.
  * [20] Qiong Huang, Ashok Veeraraghavan, and Ashutosh Sabharwal.  Tabletgaze: Unconstrained appearance-based gaze estimation in mobile tablets.  2017\.
  * [21] Petr Kellnhofer, Adria Recasens, Simon Stent, Wojciech Matusik, and Antonio Torralba.  Gaze360: Physically unconstrained gaze estimation in the wild.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6912–6921, 2019.
  * [22] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [23] Rakshit Kothari, Shalini De Mello, Umar Iqbal, Wonmin Byeon, Seonwook Park, and Jan Kautz.  Weakly-supervised physically unconstrained gaze estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9980–9989, 2021.
  * [24] Rakshit Kothari, Shalini De Mello, Umar Iqbal, Wonmin Byeon, Seonwook Park, and Jan Kautz.  Weakly-supervised physically unconstrained gaze estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9980–9989, 2021.
  * [25] Kyle Krafka, Aditya Khosla, Petr Kellnhofer, Harini Kannan, Suchendra Bhandarkar, Wojciech Matusik, and Antonio Torralba.  Eye tracking for everyone.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2176–2184, 2016.
  * [26] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In ICCV, pages 10012–10022, 2021.
  * [27] Feng Lu, Yusuke Sugano, Takahiro Okabe, and Yoichi Sato.  Head pose-free appearance-based gaze sensing via eye image synthesis.  In ICPR, pages 1008–1011, 2012.
  * [28] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel.  Understanding the effective receptive field in deep convolutional neural networks.  Advances in neural information processing systems, 29, 2016.
  * [29] Seonwook Park, Shalini De Mello, Pavlo Molchanov, Umar Iqbal, Otmar Hilliges, and Jan Kautz.  Few-shot adaptive gaze estimation.  In International Conference on Computer Vision (ICCV), 2019.
  * [30] Seonwook Park, Adrian Spurr, and Otmar Hilliges.  Deep pictorial gaze estimation.  In Proc. ECCV, 2018.
  * [31] Seonwook Park, Adrian Spurr, and Otmar Hilliges.  Deep pictorial gaze estimation.  In Proceedings of the European conference on computer vision (ECCV), pages 721–738, 2018.
  * [32] Anjul Patney, Joohwan Kim, Marco Salvi, Anton Kaplanyan, Chris Wyman, Nir Benty, Aaron Lefohn, and David Luebke.  Perceptually-based foveated virtual reality.  In ACM SIGGRAPH 2016 emerging technologies, pages 1–2. 2016.
  * [33] Sheng-Wen Shih and Jin Liu.  A novel approach to 3-d gaze tracking using stereo cameras.  IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 34(1):234–245, 2004.
  * [34] Yusuke Sugano, Yasuyuki Matsushita, and Yoichi Sato.  Learning-by-synthesis for appearance-based 3d gaze estimation.  In Proc. CVPR, 2014.
  * [35] Yusuke Sugano, Xucong Zhang, and Andreas Bulling.  Aggregaze: Collective estimation of audience attention on public displays.  In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, pages 821–831, 2016.
  * [36] Kar-Han Tan, David J Kriegman, and Narendra Ahuja.  Appearance-based eye gaze estimation.  In Proc. IEEE Workshop Applications of Computer Vision, pages 191–195, 2002.
  * [37] Kang Wang, Rui Zhao, and Qiang Ji.  A hierarchical generative model for eye image synthesis and eye gaze estimation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 440–448, 2018.
  * [38] Kang Wang, Rui Zhao, Hui Su, and Qiang Ji.  Generalizing eye tracking with bayesian adversarial learning.  In CVPR, pages 11907–11916, 2019.
  * [39] Yaoming Wang, Yangzhou Jiang, Jin Li, Bingbing Ni, Wenrui Dai, Chenglin Li, Hongkai Xiong, and Teng Li.  Contrastive regression for domain adaptation on gaze estimation.  In CVPR, pages 19376–19385, June 2022.
  * [40] Michel Wedel.  Attention research in marketing: A review of eye tracking studies.  Robert H. Smith School Research Paper No. RHS, 2460289, 2013.
  * [41] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan.  Metaformer is actually what you need for vision.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10819–10829, 2022.
  * [42] Yu Yu and Jean-Marc Odobez.  Unsupervised representation learning for gaze estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7314–7324, 2020.
  * [43] Xucong Zhang, Seonwook Park, Thabo Beeler, Derek Bradley, Siyu Tang, and Otmar Hilliges.  Eth-xgaze: A large scale dataset for gaze estimation under extreme head pose and gaze variation.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16, pages 365–381. Springer, 2020.
  * [44] Xucong Zhang, Seonwook Park, and Anna Maria Feit.  Eye gaze estimation and its applications.  Artificial Intelligence for Human Computer Interaction: A Modern Approach, pages 99–130, 2021.
  * [45] Xucong Zhang, Yusuke Sugano, and Andreas Bulling.  Everyday eye contact detection using unsupervised gaze target discovery.  In Proceedings of the 30th annual ACM symposium on user interface software and technology, pages 193–203, 2017.
  * [46] Xucong Zhang, Yusuke Sugano, and Andreas Bulling.  Revisiting data normalization for appearance-based gaze estimation.  In Proceedings of the 2018 ACM symposium on eye tracking research & applications, pages 1–9, 2018.
  * [47] Xucong Zhang, Yusuke Sugano, Andreas Bulling, and Otmar Hilliges.  Learning-based region selection for end-to-end gaze estimation.  In Proc. BMVC, 2020.
  * [48] Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling.  Appearance-based gaze estimation in the wild.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4511–4520, 2015.
  * [49] Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling.  It’s written all over your face: Full-face appearance-based gaze estimation.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 51–60, 2017.
  * [50] Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling.  Mpiigaze: Real-world dataset and deep appearance-based gaze estimation.  IEEE transactions on pattern analysis and machine intelligence, 41(1):162–175, 2017.
  * [51] Zhiwei Zhu and Qiang Ji.  Eye gaze tracking under natural head movements.  In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 1, pages 918–923. IEEE, 2005.


$\Lambda_\oo\simeq \Xi[\tau_n^{-1}|\, n\geq 0] $ is identified with the localization of $\Xi$ with respect to
the morphisms $\tau_n$, $n\geq 0$.

In fact, the powers $\tau_n^{n+1}$ forming a central system (a natural transformation from
$\Id_\Xi$ to itself),
it is easy to see that the $\oo$-categorical localization $ \Xi[\tau_n^{-1}|\, n\geq 0]_\oo$ is also identified with $\Lambda_\oo$.
In particular, $\Xi$, like $\Lambda_\oo$, is generated by the coface and codegeneracy morphisms

\[
\begin{gathered}
\delta^{(n)}_i: \langle n-1\rangle \to \cn, \quad (n\geq 1, \,\, 0\leq i\leq n),
\\
\sigma^{(n)}_i: \cn \to \langle n-1\rangle, \quad (n\geq 1, \,\, 0\leq i\leq n),
\end{gathered}
\]

satisfying the same quadratic relations as in $\Lambda_\oo$. The morphisms

\[
\delta^{(n)}_i, \,\, 0\leq i\leq n, \quad \sigma^{(n)}_i,\,\, 0\leq i\leq n-1,
\]

generate the simplex category $\Delta\subset\Xi$. The shift map is expressed as $\tau_n=\delta^{(n-1}_0 \sigma^{(n)}_{n-1}$.
Accordingly, a duplicial object $Y_\bullet$ in a category $\C$
can be identified with a sequence of objects $X_0, X_1, \dots$
equipped with face and degeneracy maps

\[
\begin{gathered}
\del_i: X_n \lra X_{n-1}\quad \text{($n \ge 1$, $0 \le i \le n$)},
\\
s_i: X_{n-1} \lra X_{n}\quad \text{($n \ge 1$, $0 \le i \le n$)}
\end{gathered}
\]

subject to   relations dual to those among the $\delta^{(n)}_i, \sigma^{(n)}_i$. The action of $\tau_n$ is then
$t_n = s_{n+1}\del_0: Y_n\to Y_n$. A paracyclic object is a duplicial object such that all the $t_n$ are isomorphisms.

Following Dwyer-Kan, we  call a connective ducomplex in $\A$ a diagram

\[
\xymatrix{
\cdots \ar@/^1ex/[r]^{\delta}  & \ar@/^1ex/[l]^{d} B^{-2}  \ar@/^1ex/[r]^{\delta}  & \ar@/^1ex/[l]^{d} B^{-1}
\ar@/^1ex/[r]^{\delta}  & \ar@/^1ex/[l]^{d} B^0
\]

satisfying $d^2=0$, $\delta^2=0$ and no further relations. We denote $\DC^{\leq 0}(\A)$ the category
of connective ducoplexes in $\A$.

(a) 	There is an equivalence of categories

\[
\A_{\Xi} \overset{\simeq}{\lra} \DC^{\leq 0}(\A)
\]

given by associating to a duplicial abelian group $A_{\bullet}$ the ducomplex $B^{\bullet}$

\begin{align*}
B^{-n}  & = \bigcap_{ i=1}^n \,  \Ker\{\del_i: A_n\to A_{n-1}\},  \quad n\geq 0,
\\
d & =  \del_0: B^{-n} \to B^{-n+1}, \\
\delta & = \sum_{i=0}^{n} (-1)^i s_i : B^{-n+1} \to B^{-n}.
\end{align*}

(b) Under this equivalence, paracyclic objects correspond to ducomplexes satisfying:

\[
\Id_{B^{-n}} + (-1)^n (d\delta-\delta d): B^{-n} \lra B^{-n} \quad \text {is invertible for any } n\geq 0.
\]

Part (a) is Theorem 3.5 of [14]. Part (b) follows from the interpretation of $t_n^{n+1}$
in terms of ducomplexes given in Proposition 6.5 of [14].

The equivalence between the descriptions of the category of perverse sheaves on $(\CC,\{0\})$ from
Proposition <ref> and Proposition <ref>, respectively, is then a
consequence of restricting the equivalence from Theorem <ref> to paracyclic Segal objects.

§ $\OO$-CATEGORICAL PRELIMINARIES

§.§ Generalities on $\oo$-categories

In the rest of the paper we will use freely the language of $\oo$-categories [31]. The
following is intended to fix the terminology and notation and to recall the main tools that will be

We denote by $\sSet$ the category of simplicial sets. For a simplicial set $S=(S_n)_{n\geq 0}$ we

\[
\partial_i: S_n\to S_{n-1}, \,\,\, 0\leq i\leq n
\]

the simplicial face maps. By $\Delta^n\in\sSet$ we denote the standard $n$-simplex.

Following [31] we will use the term $\infty$-category for a weak Kan complex. Thus an
$\infty$-category $\C$ is a simplicial set $(\C_n)_{n\geq 0}$ satisfying the lifting condition for
intermediate horns $\Lambda^n_i \subset \Delta^n$, $0<i<n$.
Any ordinary category can be considered as an $\infty$-category by passing to the nerve.

Each $\infty$-category $\C$ contains the maximal Kan subcomplex $\C^{\on{Kan}}\subset\C$, which is
can be interpreted as “the $\infty$-groupoid of equivalences in $\C$".

We follow the usual notation and terminology: $0$-simplices of $\C$ are called objects, and we
denote $\Ob(\C)=\C_0$, while $1$-simplices are called morphisms and we denote $\Mor(\C)=\C_1$.
For any two $x,y\in\Ob(\C)$ we denote $\Hom_\C(x,y)\subset\Mor(\C)$ the set of 1-simplices $f$ such
that $\partial_1(f)=x$ and $\partial_0(f)=y$.

To any $\infty$-category $\C$ one associates its homotopy category $\Ho(\C)$ which is an
ordinary category with the set of objects $\Ob(\C)$ and $\Hom_{\Ho(\C)}(x,y)$ defined as the
quotient of $\Hom_\C(x,y)$ by the homotopy relation: $f\sim g$ if there is $\sigma\in \C_2$ with
$\partial_1(\sigma)=f$ and $\partial_2(\sigma)=g$. An equivalence in $\C$ is a morphism which
becomes an isomorphism in $\Ho(\C)$.

For an $\infty$-category $\C$ and any $x,y\in\Ob(\C)$,  the set $\Hom_\C(x,y)$ can be upgraded to a
simplicial set $\Map_\C(x,y)$  (the mapping space), in such a way as to make out of $\C$ a
category enriched in simplicial sets. See [31] 1.2.2 for details.

This leads to another point of view on $\infty$-categories: as categories enriched in topological
spaces (or simplicial sets). Several $\infty$-categorical concepts can be formulated in this
For example, an initial object of an $\infty$-category $\C$ is an object $0$ such that, for
each $x\in\Ob(\C)$ the space $\Map_\C(0,x)$ is contractible.

[(Kan simplicial sets as an $\oo$-category)]  Any Kan simplicial set is an $\oo$-category.
The $\oo$-category $\Sp$ of spaces is defined as the simplicial nerve of the category
of Kan simplicial sets <cit.>.

§.§ Dg-categories

We denote by $\Ab$ the category of abelian groups and by
$C(\Ab)$ the category of cochain complexes of abelian groups, with its standard
symmetric monoidal structure.
By a dg-category we mean a category $\Ac$ enriched in $C(\Ab)$. For such $\Ac$ we have
the ordinary categories $Z^0(\Ac)$,  $H^0(\Ac)$ with the same objects as $\Ac$ and

\[
\Hom_{Z^0(\Ac)}(x,y) \,=\, Z^0 \Hom^\bullet_\Ac(x,y) , \quad
\Hom_{H^0(\Ac)}(x,y) \,=\, H^0 \Hom^\bullet_\Ac(x,y).
\]

Here $Z^0$ is the subgroup of $0$-cocycles in the $\Hom$-complex.

A dg-category $\Ac$ gives an $\infty$-category
$N_\dg(\Ac)$   known as the dg-nerve of $\Ac$.
As a simplicial set, $N_\dg(\Ac)$ was introduced in [25].
For a given $n\geq 0$ the set
$N_\dg(\Ac)_n$ consists of weakly commutative $n$-simplices in $\Ac$ (called Sugawara simplices  in [25]),
which are data

\[
\begin{gathered}
x_0, \cdots, x_n\in \Ob(\Ac);
\\
u_{ij}\in\Hom^0_\Ac(x_i, x_i), \,\,\, d(u_{ij})=0, \quad i<j;
\\u_{ijk}\in\Hom^{-1}_\Ac(x_i, x_k), \,\,\, d(u_{ijk}) = u_{jk}u_{ij}-u_{ik}, \quad i<j<k; \\
\text{ and so on.}
\end{gathered}
\]

It was shown in [32] that $N_\dg(\Ac)$ is in fact a $\oo$-category.  By construction, we have

\[
\Ho(N_\dg(\Ac)) \,\simeq \, H^0(\Ac).
\]

§.§ The derived $\oo$-category of an abelian category

Let $\A$ be a Grothendieck abelian category. In particular, $\A$ has enough injectives.
Denote by $\Co(\A)$  the dg-category of all cochain complexes over $\A$.
Thus $Z^0(\Co(\A))$ is the “usual” category of complexes (morphsims = morphisms of complexes) and
$H^0(\Co(\A))$ is the homotopy category.
The classical (unbounded) derived category of $\A$, denoted $\D(\A)$,  is defined as the categorical localization
of $H^0(\Co(\A))$ by the class of quasi-isomorphisms. It is a triangulated category.

The  (unbounded) derived $\oo$-category of $\A$, denoted $\D(\A)$,  can be  defined
in one of two equivalent ways, see [32] 1.3.5, esp. Prop. 1.3.5.16 and before.

(i) As the $\oo$-categorical localization of the  usual (abelian) category  $Z^0(C(\A))$ by the
class of quasi-isomorphisms.

(ii) As the full $\oo$-subcategory in $N_\dg(\Co(\A))$ spanned by fibrant complexes.
A fibrant complex is a possibly unbounded
complex of injective objects with some additional properties, see  [32] 1.3.5 and [36].

We have

\[
\Ho (\D_\oo(\A)) \,\simeq \, \D(\A).
\]

§.§ Stable $\infty$-categories

The derived $\infty$-categories from <ref> are examples of stable $\infty$-categories.
Here, we recall the definition of a stable $\infty$-category from [32] and discuss some basic
results that we will use.
Let $\D$ be a pointed $\infty$-category and consider a square

\begin{equation}\label{eq:stablesquare}
\begin{tikzcd}
X \ar{r}{f} \ar{d} & Y \ar{d}{g}  \\
0 \ar{r} & Z
\end{tikzcd}
\end{equation}

in $\D$ where $0$ is a zero object. The square is called a fiber sequence if it is a pullback
square. In this case, the morphism $f$ is called a fiber of $g$. Dually, the square is called a
cofiber sequence if it is a pushout square. In this case, we say that $g$ is a cofiber of
$f$. The category $\D$ is called stable if

* every morphism admits a fiber and a cofiber,

* a square of the form (<ref>) is a fiber sequence if and only if it is
a cofiber sequence.

We collect some basic results about stable $\infty$-categories (cf. [32]):

Let $\D$ be a stable $\infty$-category. Then:

* $\D$ admits finite limits and colimits.

* The homotopy category of $\D$ admits a triangulated structure.

*  A square

\[
\begin{tikzcd}
X \ar{r} \ar{d} & Y \ar{d}  \\
X' \ar{r} & Y'
\end{tikzcd}
\]

in $\D$ is Cartesian if and only if it is coCartesian.

Cartesian squares in a stable $\infty$-category (which are hence also coCartesian) will be called
biCartesian squares. The statement of Proposition <ref> <ref> has a useful generalization to
higher–dimensional cubes: Let $\D$ be a stable $\infty$-category, $n \ge 1$, and let
$\P(\{1,\dots,n\})$ be the poset of all subsets of the set $\{1,\dots,n\}$. Consider a

\[
q: \N(\P(\{1,\dots,n\})) \lra \D
\]

which, due to the apparent isomorphism $\N(\P(\{1,\dots,n\})) \cong (\Delta^1)^n$, has the shape of
an $n$-dimensional cube. Note that, we may either interpret $q$ as

* a cone over the diagram $q|\N(\P(\{1,\dots,n\}) \setminus \{\emptyset\})$, or

* a cone under the diagram $q|\N(\P(\{1,\dots,n\}) \setminus \{\{1,\dots,n\}\})$.

If the first cone is a limit cone, then we call $q$ Cartesian, if the second cone is a colimit cone,
then we call $q$ coCartesian. We recall some results from [32]:

A cube $q$ is Cartesian if and only if it is coCartesian.

We will refer to cubes in a stable $\infty$-category which are Cartesian (and hence coCartesian) as
biCartesian generalizing the above terminology in the case $n=2$. We further recall:

An $n$-cube is biCartesian if and only if the $(n-1)$-cube,
obtained by passing to cofibers along all morphisms parallel to one coordinate axis, is

We also note the following immediate consequences of Proposition <ref>.

Suppose we are given an $n$-cube $q$ with one face $f$ biCartesian.
Then $q$ is biCartesian if and only if the face parallel to $f$ is also biCartesian.

The property for cubes being biCartesian satisfies the two-out-of-three property with
respect to pasting of cubes.

§.§ Limits and Kan extensions

Let $\C$ be an $\oo$-category. As usual, a ($\oo$-) functor $F: I\to\C$ where $I$ is a (small) $\oo$-category, will
be called a diagram in $\C$. We will also use the notation $(F_i)_{i\in I}$ for such a diagram, with $F_i=F(i)$, $i\in\Ob(I)$.
The $\oo$-categorical limit and colimit of $(F_i)_{i\in I}$ (when they exist)  will be denoted by

\[
\varprojlim_{i\in I}{}^\C \, F_i, \quad\quad  \varinjlim_{i\in I} {}^\C\,  F_i,
\]

or, in the functor notation, simply $\varprojlim F$, $\varinjlim F$.

Let $\alpha: I\to J$ be a functor of small $\oo$-categories and $F: I\to\C$ be another functor.  In
this case we can speak about the left and right Kan extensions which are functors

\[
\alpha_! F, \alpha_* F: J\lra \C
\]

characterized by universal properties. More precisely, the functors (when they exist)

\[
\alpha_!: \Fun(I,\C)\to \Fun(J, \C), \,\, F\mapsto \alpha_!F, \quad  \alpha_*:  \Fun(I,\C)\to \Fun(J, \C), \,\, F\mapsto \alpha_* F
\]

are, resectively, left and right adjoints to the pullback functor

\[
\alpha^*: \Fun(J,\C) \lra \Fun(I,\C), \quad G\mapsto \alpha^*G := G\circ \alpha.
\]

see [31] 4.3.  While the general concept of adjunction in the $\oo$-categorical context is
somewhat subtle, see [31] 5.2,  it implies  identifcations (weak equivalences) of mapping
spaces having the familiar shape, which in our case read:

\begin{equation}
\label{eq:kan-adj}
\Map_{\Fun(J,\C)} (G,\alpha_*F) \,\simeq\, \Map_{\Fun(I,\C)} (\alpha^* G, F), \quad \Map  (\alpha_!F, G) \,\simeq \,
\Map  (F, \alpha^*G)
\end{equation}

for any $F\in \Fun(I,\C)$, $G\in\Fun(J, \C)$.

We recall the pointwise formulas for Kan extensions which describe their values on an object
$j\in J$. More precisely, assuming the existence of all the relevant (co)limits, we have

\begin{equation}
\label{eq:kan-pointwise}
(\alpha_! F)(j) \,=\, \varinjlim{}_{i\in \alpha/j}^\C \, \,F(j), \quad \quad
(\alpha_*(F)_j \,=\,  \varprojlim{}_{i\in j/\alpha }^\C  \,\,F(i),
\end{equation}

where $\alpha/j$ resp. $j/\alpha$ are the overcategory and undercategory, whose objects
are pairs

\[
\bigl(i\in \Ob(I), a: \alpha(i)\to j\bigr), \text{ resp. }  \bigl(i\in\Ob(I), b: j\to \alpha(i)\bigr),
\]

see [31] 1.2.9.

Recall, further, that $\alpha$ is called $\oo$-cofinal, resp. $\oo$-coinitial, if any
resp. $\alpha/j$ is contractible, (i.e., its nerve is a contractible simplicial set). If this is the case, then
for any $G; J\to\C$ we have equivalences

\[
\varinjlim \alpha^*G \,\simeq \, \varinjlim \, G, \quad \text{resp.} \quad
\varprojlim \alpha^*G \,\simeq \, \varprojlim \, G.
\]

in $\C$.

The following result <cit.> will be a fundamental tool for us to establish equivalences of $\oo$-categories.

Let $J$ be an $\infty$-category and $I \subset J$ a full subcategory. Let $\C$ be an
$\infty$-category with colimits let $\E \subset \Fun(I,\C)$ be a full subcategory and let $\Ec^!$  be the full
subcategory in $ \Fun(J,\C)$ spanned by functors of the form $\alpha_! F$ for $F$ in $\Ec$.
Then the restriction functor $\Ec^! \to \Ec$ is an equivalence. The analogous statement
holds for right Kan extensions if $\C$ has limits.

We also note, for future use, the following fact <cit.>.

Let $K$ be a weakly contractible simplicial set, and let $\C$ be an $\infty$-category.

\[
F: K \to \C
\]

be a diagram sending every edge of $K$ to an equivalence in $\C$. Then a cone

\[
F^+: K^{\triangleright} \lra \C
\]

is a colimit cone if and only if every edge from a vertex in $K$ to the cone vertex $*$ is
mapped to an equivalence in $\C$. In particular, for every vertex $k$ of $K$, the value
$F(k)$ is a colimit of $F$.

§.§ $\oo$-categorical localization

Let $\C$ be an $\oo$-category and $W\subset\Mor(\C)$ be a set of $1$-morphisms.
For any $\oo$-category $\E$ we denote

\[
\Fun(\C, \E)_{[W\to\Eq]} \,\subset \, \Fun(\C,\E)
\]

the full $\oo$-subcategory spanned by ($\oo$-)functors that take elements of $W$ to
equivalences in $\E$.

Let $\pi:\C\to\Di$ be an $\oo$-functor. We say that $\pi$ exhibits $\Di$ an an $\oo$-categorical
localization of $\C$ by $W$, or, simply, that $\pi$ is an $\oo$-localization of $\C$ by $W$,

(1) $\pi \in \Fun(\C, \Di)_{[W\to\Eq]}$.

(2) For any $\oo$-category $\E$, composition with $\pi$ gives an equivalence

\[
\pi^*: \Fun(\Di,\E) \lra \Fun(\C, \E)_{[W\to\Eq]}.
\]

Given $\C$ and $W$, the datum $(\Di, \pi)$ as above, is known to exist and be unique up to
a contractible space of choices, see [32], 5.2.7. We will therefore  denote such $\Di$ by

We will be particularly interested in the case when $\C$ is a usual category. In this case
$\C[W^{-1}]_\oo$ is the $\oo$-categorical analog of the Dwyer-Kan simplicial localization
[13, 12]. In particular,

\[
\Ho \,\C[W^{-1}]_\oo \,= \, \C[W^{-1}]
\]

is the usual categorical localization of $\C$ by $W$.

We will be further interested in the cases when $\C[W^{-1}]_\oo$ is equivalent to a usual
category, i.e., reduces to $\C[W^{-1}]$.

Given a functor $\pi: \C\to\Di$ of usual categories and an object $d\in\Di$, we denote by

\[
j_d: (d/\pi)^\iso \hookrightarrow d/\pi
\]

the embedding of the full subcategory spanned by pairs $(c\in\C, a: d\to\pi(c))$
for which $a$ is an isomorphism in $\Di$. Then we have (cf. [39]):

Let $\pi: \C\to\Di$ be a functor of usual categories and $W\subset\Mor(\C)$. Suppose that:

(1) Elements of $W$ are precisely the morphims of $\C$ sent by $\pi$ into isomorphisms.

(2) For any $d\in\Di$ the category $(d/\pi)^\iso$ is contractible.

(3) For any $d\in\Di$ the functor $j_d$ is $\oo$-coinitial.

Then $\pi$ exhibits $\Di$ as the $\oo$-categorical localization of $\C$ by $W$.

Step 1. Let $\Ec$ be any $\oo$-category with limits and $G: \Di\to\E$
be an $\oo$-functor. Then the natural transformation
$G\to \pi_*\pi^*G$ is an equivalence.
Indeed, by the pointwise formula for Kan extensions and $\oo$-coinitiality of $j_d$, we have

\[
(\pi_*\pi^* G)(d) \,=\,\varprojlim_{c\in d/\pi} G(\pi(c)) \,=\,\varprojlim_{c\in (d/pi)^\iso} G(\pi(c)).
\]

But $(d/\pi)^\iso$ consists of isomorphisms $d\to\pi(c)$, so by inverting them, we can say that it
consists of isomorphisms $\pi(c)\buildrel\simeq \over \to d$. So we have

\[
(\pi_*\pi^* G)(d) \,=\,\varprojlim_{\{\pi(c)\buildrel\simeq \over \to d\}} G(\pi(c)) \,=\, G(d),
\]

since the last limit is taken over a cone-shaped diagram (one with an initial object).

Step 2. Further, let $F: \C\to\E$ be any $\oo$-functor which takes elements
of $W$ into equivalences. Then the natural transformation $\pi^* \pi_* F\to F$ is an equivalence.
Indeed, as above, for any $c\in\C$,

(π^*π_* F)(c)  = _c'∈π(c)/π F(c')   =
_c'∈(π(c)/π)^ F(c').
But $ (\pi(c)/\pi)^\iso$ has, as objects, isomorphisms $\pi(c)\buildrel b\over\to \pi(c')$, while a morphism

\[
[\pi(c)\buildrel b\over\to \pi(c'_1)] \,\lra \, [\pi(c)\buildrel b\over\to \pi(c'_2)]
\]

between two such objects is a morphism $u: c'_1\to c'_2$ in $\C$ such that the diagram

\[
\xymatrix{
& \pi(c'_1) \ar[d]^{\pi(u)}
\\
\pi(c) \ar[r]_{b_2}
\ar[ur]^{b_1}& \pi(c'_2)
\]

commutes. This means that $\pi(u)$ is an isomorphism and so $u\in W$ by the assumption (1).
Thus $F(u)$ is an equivalence. So the limit in (<ref>) is a limit of a diagram of
equivalences parametrized by a category that is contractible by assumption (3).
So (e.g., by Lemma <ref> for limits instead of colimits) it is identified with any term
of the diagram, in particular, the natural map from this limit to $F(c)$ is an equivalence.

Step 3.  Now consider the pullback functor

\[
\pi^*: \Fun(\Di,\E) \lra\Fun(\C, \E).
\]

Step 1 implies that $\pi^*$ is fully faithful (the embedding of a full $\oo$-subcategory).
Step 2 means that the essential image of $F$ is $\Fun(\C, \E)_{[W\to\Eq]}$. This means that
$\pi$ satisfies the condition (2) of Definition <ref> for any $\E$ with limits.

Finally, we note that in the above reasoning it is not necessary to require that $\E$
has all limits as all the limits we need, automatically exist and are explicitly identified.
This proves Proposition <ref>.

Fir future use, we note a dual version of  Proposition <ref>.
For a functor $\pi: \C\to\Di$  of usual categories and an object $d\in\Di$, we consider the embedding

\[
j^d: (\pi/d)^\iso \hookrightarrow \pi/d
\]

where $(\pi/d)^\iso$ is the full subcategory of $\pi/d$ formed by pairs
$(c\in\C, b: \pi(c)\to d)$ for which $b$ is an isomorphism.

Let $\pi: \C\to\Di$ be a functor  of usual categories. Suppose that:

(1) Elements of $W$ are precisely the morphims of $\C$ sent by $\pi$ into isomorphisms.

(2) For any $d\in\Di$ the category $(\pi/di)^\iso$ is contractible.

(3) For any $d\in\Di$ the functor $j^d$ is $\oo$-cofinal.

(a)  $\pi$ exhibits $\Di$ as the $\oo$-categorical localization of $\C$ by $W$.

(b) For any $\oo$-category $\E$ with colimits and any functor $G: \Di\to\E$
the natural transformaton $\pi_!\pi^* G\to \G$ is an equivalence.

(c) For any functor $F: \C\to \E$ sending elements of $W$ to equivalences,
the natural transformation $F\to \pi^*\pi_!F$ is an equivaence.

Obtained from that of Proposition <ref> by dualization.

§.§ A covering lemma

We recall the following lemma which generalizes various classical statements of the kind that a
space is homotopy equivalent to the nerve of its sufficiently fine open covering.

Let $T$ be a small category. Let $E$ be a topological space and let $\Op(E)$
denote the poset of open subsets of $E$. Let

\[
\chi: T\lra \Op(E))
\]

be a functor. For any $e\in E$ let $\chi^{-1}(e)\subset T$ be full subcategory   spanned by $t$ such that
$e \in \chi(t)$. Suppose that:

* for every $t \in T$, the open  set $\chi(t)$ is contractible,

* for every $e \in E$, the  category $\chi^{-1}(e)$ is  contractible.

Then there is a weak homotopy equivalence $|\N(T)| \simeq E$.

Note that the assumption (2) implies, in particular, that the $\chi(t)$ form an open covering of $E$,
as a contractible category is nonempty.

Let $\pi: K \to \N(T)$ be the relative nerve (<cit.>) associated to the functor

\[
\N(T) \to \Sp,\; t \mapsto \Sing(\chi(t)).
\]

Then $\pi$ is a left fibration whose fibers are, by assumption, contractible Kan complexes.
By <cit.> it is a trivial Kan fibration so that $|K| \simeq |N(T)|$. On
the other hand, $\Sing(|K|)$ is a model for the colimit of $\pi$ (<cit.>) which
by Lurie's Seifert-van Kampen Theorem <cit.> is weakly equivalent to

ayala-francis D. Ayala, J. Francis. Factorization homology of
topological manifolds.  J. Topol.  8   (2015) 1045-1084.

[1] A. Beilinson.
How to glue perverse sheaves. In: $K$-theory, arithmetic and geometry (Moscow, 1984),
Lecture Notes in Math. 1289,
Springer-Verlag, 1987, 42 - 51.

[2] A. Beilinson, I. Bernstein, P. Deligne.
Faisceaux Pervers, Astérisque 100, 1982.

BD A. Beilinson, V. Drinfeld. Chiral Algebras.
Amer. Math. Soc. 2004.

[3] J. Bergner. The Homotopy Theory of  $(\oo,1)$-Categories.
Cambridge Univ. Press,  2018.

[4] A. I. Bondal, M. M. Kapranov. Enhanced triangulated categories.
Math. USSR-Sb.  70  (1991)  93-107.

borsuk-ulam K. Borsuk, S. Ulam. On symmetric products of topological spaces.
Bull. Amer. Math. Soc.  37 (1931) 875-882.

[5] R. Bott. On the third symmetric potency of $S_1$. Fundamenta Math.
36 (1949) 236-244.

[6] A. Cepek. On configuration categories. 2019 PhD Thesis

[7] A. Cepek. Configuration spaces of $\RR^n$ by way of exit-path $\infty$-categories. arXiv:1910.11980

[8]
A. Connes.
Noncommutative Geometry.
Academic Press,  1994.

[9] P. Deligne. La formule de dualité globale. SGA 4, Exp. XVIII.

[10]
V.  Drinfeld.
On the notion of geometric realization.
Mosc. Math. J.   4  (2004)  619-626 .

[11] D. Dugger, S. Hollander, D. C. Isaksen. Hypercovers and
simplicial presheaves. arXiv:math/0205027.

[12]
W. G. Dwyer, P. S. Hirschhorn, D. M. Kan, and J. H. Smith.
Homotopy Limit Functors on Model Categories and Homotopical
Amer. Math.  Soc.  Providence, RI, 2004.

[13]
W. G. Dwyer and D. M. Kan.
Simplicial localizations of categories.
J. Pure Appl. Alg. 17 (1980) 267-294.

[14] W. G. Dwyer, D. M. Kan. Normalizing the cyclic modules of Connes.
Comm. Math. Helv. 60 (1985) 582-600.

[15]  T. Dyckerhoff. A categorified Dold-Kan correspondence.

[16] T. Dyckerhoff, M. Kapranov.
Crossed simplicial groups and structured surfaces, in:
“Stacks and categories in geometry, topology, and algebra”, p. 37-110,
Contemp. Math.  643, Amer. Math. Soc., Providence, RI, 2015.

[17] T. Dyckerhoff, M. Kapranov. Higher Segal Spaces. Lect. Notes in Math.2244, Springer-Verlag, 2019.

[18] A. D. Elmendorf. A simple formula for cyclic duality.
Proc. AMS, 118 (1993) 709-711.

[19] B. Farb,  D. Margalit. A Primer on Mapping Class Groups. Princeton Univ. Press,

[20]
P. Gabriel, M. Zisman.
Calculus of Fractions and Homotopy Theory,
Springer-Verlag,  1967.

[21] A. Galligo, M. Granger, P.  Maisonobe.
$\Dc$-modules et faisceaux pervers dont le support singulier
est un croisement normal. Ann. Inst. Fourier Grenoble,
35 (1985) 1-48.

[22] P. G. Goerss,  J. F. Jardine.  Simplicial Homotopy Theory.  Birkhäuser,
Boston, 1999.

[23] M. Goresky, R. MacPherson. Intersection homology II.
Invent. Math.  72 (1983) 77-130.

[24] M. Goresky, R. MacPherson. Stratified Morse Theory. Springer-Verlag, 1988.

[25]  V. A. Hinich, V. V.  Schechtman.
On homotopy limit of homotopy algebras. K-theory, arithmetic and geometry (Moscow, 1984-1986), p. 240-264,
Lecture Notes in Math.  1289, Springer-Verlag, 1987.

[26] J. F. Jardine. Simplicial presheaves.  J. of Pure and Appl. Alg.
47 (1987) 35-87.

[27]
N. Johnson,  A. M. Osorno. Modeling stable one-types.
Theory Appl. Categ.  26 (2012) 520–537.

kapranov-schechtman:schobers M. Kapranov, V. Schechtman.
Perverse Schobers. arXiv:1411.2772.

kapranov-schechtman:graphs M. Kapranov, V. Schechtman.
Perverse sheaves and graphs on surfaces.  arXiv:1601.01789.

[28] M. Kashiwara, P. Schapira. Sheaves on Manifolds. Springer-Verlag, 1991.

[29] K. Kuratowski. Topology vol. I, Academic Press, 2014.

[30] G. Laumon. Vanishing cycles over a base of dimension $\geq 1$, in:
“Algebraic Geometry”, Lecture Notes in Math. 1016, p. 143-150, Springer-Verlag, 1983.

loday J.-L. Loday. Cyclic Homology. Springer-Verlag, 1992.

[31] J. Lurie. Higher Topos Theory. Princeton Univ. Press, 2009.

[32] J. Lurie. Higher Algebra. Available at:
$<$http://people.math.harvard.edu/ lurie/papers/HA.pdf$>$

lurie:SAG J. Lurie. Spectral Algebraic Geometry. Available at:
$<$https://www.math.ias.edu/ lurie/papers/SAG-rootfile.pdf$>$

macpherson:intersection R. D. MacPherson.
Intersection Homology and Perverse Sheaves, Report December 15, 1990.
Unpublished Notes: http://faculty. tcu. edu/gfriedman/notes/ih. pdf

[33] J. Mostovoy. Lattices in $\CC$ and finite subsets of a circle.
Amer. Math. Monthly 111 (2004) 357-360.

[34] B. Noohi. Notes on 2-groupoids, 2-groups and crossed modules.
Homology Homotopy Appl.
9  (2007), 75-106.

[35] C. Rezk.  A model for the homotopy theory of  homotopy theory.
Trans. Amer. Math. Soc.  353 (2001)  973-1007.

[36] N. Spaltenstein. Resolutions of unbounded complexes.
Compositio Math.  65  (1988) 121-154.

[37] D. Treumann. Exit paths and constructible stacks.
Compositio Math. 145 (2009) 1504-1532.

[38] C. Tuffley. Finite subset spaces of $S^1$.
Alg. and Geom. Topology, 2 (2002) 1119-1145.

[39] T. Walde. Walde, 2-Segal spaces as invertible infinity-operads. Algebraic & Geometric Topology, to appear

[40]
F. Waldhausen. Algebraic K-theory of spaces, in: “ Algebraic and geometric topology” (New Brunswick, N.J., 1983),
Lecture Notes in Math. 1126,
Springer-Verlag, 1985.

T.D.:  Universität Hamburg,
Fachbereich Mathematik,
Bundesstrasse 55,
20146 Hamburg, Germany. Email:

M.K.: Kavli IPMU, 5-1-5 Kashiwanoha, Kashiwa, Chiba, 277-8583 Japan. Email:

Y.S.:  Dept.  Math.,  Kansas State University, Manhattan, KS 66506 USA.
Email<EMAIL_ADDRESS>