††thanks: These authors contributed equally to this work††thanks: These
authors contributed equally to this work

#  Observation of period-doubling Bloch oscillations

Naveed Khan  School of Physics and Astronomy, Shanghai Jiao Tong University,
Shanghai 200240, China  Peng Wang  School of Physics and Astronomy, Shanghai
Jiao Tong University, Shanghai 200240, China  Qidong Fu  Corresponding author:
<EMAIL_ADDRESS>School of Physics and Astronomy, Shanghai Jiao Tong
University, Shanghai 200240, China  Ce Shang  Physical Science and Engineering
Division (PSE), King Abdullah University of Science and Technology (KAUST),
Thuwal 23955-6900, Saudi Arabia  Fangwei Ye  Corresponding author:
<EMAIL_ADDRESS>School of Physics and Astronomy, Shanghai Jiao Tong
University, Shanghai 200240, China

###### Abstract

Bloch oscillations refer to the periodic oscillation of a wavepacket in a
lattice under a constant force. Typically, the oscillation has a fundamental
period that corresponds to the wavepacket traversing the first Brillouin zone
once. Here we demonstrate, both theoretically and experimentally, the optical
Bloch oscillations where the wavepacket must traverse the first Brillouin zone
twice to complete a full cycle, resulting in a period of oscillation that is
two times longer than that of usual Bloch oscillations. The unusual Bloch
oscillations arise due to the band crossing of valley-Hall topological edge
states at the Brillouin boundary for zigzag domain walls between two staggered
honeycomb lattices with inverted on-site energy detuning, which are protected
by the glide-reflection symmetry of the underlying structures. Our work sheds
light on the direct detection of band crossings resulting from intrinsic
symmetries that extend beyond the fundamental translational symmetry in
topological systems.

Bloch oscillations (BOs) are periodic oscillations exhibited by a wavepacket
in a periodic lattice under a constant force. The oscillations have a period
that takes the wavepacket to transverse the first Brillouin zone once. As
such, BOs offer a valuable tool for revealing crucial information about the
band structure and transport properties of periodic systems. Initially
predicted for electrons in crystalline lattices [1, 2] and shortly after the
observation of Wannier-Stark ladders [3, 4], BOs were observed first in
semiconductor superlattices [5, 6]. Since then, they have been observed in
various systems including ultracold atoms [7], Bose-Einstein condensates(BECs)
[8, 9], waveguide arrays [10, 11, 13, 14, 12], optically induced lattices [15,
16], plasmonic waveguides [17], and systems with parity-time symmetry [18].
BOs have also been observed in strongly correlated quantum systems, but with a
doubled frequency [19, 20, 21].

BOs occur in topological materials too [22]. These materials have a nontrivial
topological structure in their Bloch bands, defined by certain invariants, and
their accurate measurement is crucial to characterize the topological
materials. In this regard, BOs have proven to be a powerful tool for measuring
these invariants, such as Chern numbers or Zak/Berry phases [23, 24, 25, 26].
Interestingly, in truncated topological structures [27, 28, 29] or for spin-
orbit coupled atoms loaded in a one-dimensional (1D) Zeeman lattice [30], it
is predicted that completing one BO cycle requires scanning the Brillouin zone
twice, resulting in a period doubling that of usual BOs [27, 28, 29]. However,
realizing these predictions is challenging due to the need for strong time-
reversal-symmetry breaking, and the practical difficulties in implementing
them in 1D system [30] or transforming edge states with delocalized bulk
modes. Additionally, by bypassing constraints in real space, indirectly
achieving period-doubling BOs for BECs on a synthetic topological Hall
cylinder has been mentioned [31]. However, this was done in a virtual,
synthetic space, hence a direct observation of period-doubling BOs has not
been achieved yet.

Here we investigate both theoretically and experimentally, the optical BOs
with a doubling period without breaking time-reversal symmetry. The BOs occur
on valley-Hall edge states (VHESs) at a domain wall between two inverted
staggered photonic honeycomb lattices. The system preserves the time-reversal
symmetry, enabling wavepackets to move in either direction within the same
domain wall. We consider two types of domain walls: zigzag and bearded. While
both types support a pair of localized states due to the valley-Hall effect,
the zigzag domain wall has energy band crossings for VHESs, resulting in the
need to scan the Brillouin zone twice for the BOs of zigzag VHESs. This is in
contrast to the bearded domain wall or other usual BOs, where one cycle is
achieved by traversing the Brillouin zone once. Unlike previous proposals on
period doubling of BOs [27, 28, 29, 30], our study considers BOs in a 2D
system, where the back-and-forth oscillation occurs within the same domain
wall without involving bulk mode transformations. This facilitates the
experimental measurement of the wavepacket drift in real space. By tracking
the “center of mass” (COM) of the VHESs, we directly observed, to the best of
our knowledge, the first period-doubling BOs in photonic systems or any other
physical systems.

We analyze the light propagation along the $z$-axis in a photorefractive
medium, specifically strontium barium niobate (SBN: 61) that is described by
the following dimensionless equation for the light amplitude $\Psi$:

$i\frac{\partial\Psi}{\partial z}=-\frac{1}{2}(\frac{\partial^{2}}{\partial
x^{2}}+\frac{\partial^{2}}{\partial
y^{2}})\Psi-\frac{E_{0}}{1+I_{L}(x,y)+I_{G}(y)}\Psi,$ (1)

where the applied dimensionless d.c. field is set to $E_{0}=-30$,
corresponding to an electric field $E$ of 2.4$\times$105 Vm-1. $I_{L}(x,y)$
represents a 2D lattice wave field that consists of a domain wall and two
staggered honeycomb lattices with inverted on-site potentials surrounding it.
Two types of domain walls are considered: zigzag [Fig. 1(a)] and bearded [Fig.
1(c)] (see [34] on how these lattices are constructed in modelling and in
experiment). To induce BOs, a potential gradient is created along the domain
walls by illuminating the sample laterally with a gradient white light,
represented by $I_{G}=I_{0}$[1+tanh($y/\eta$)], where $I_{0}$ is the intensity
of the white light, and $\eta$ characterizes the width of its intensity
gradient that controls the rate $\beta$ of a linear increase in the refractive
index in the $y$ axis [34]. Note that the form in which the lattice intensity
$I_{L}(x,y)$ and gradient light intensity $I_{G}(y)$ enter Eq. (1) is
determined by the mechanism of the photorefractive response [32, 33].

To investigate the impact of the Floquet-Bloch spectrum on the dynamics of
BOs, we first set the gradient $I_{G}$ to zero and search for Bloch eigenmodes
in the form of $\Psi(x,y,z)=\psi(x,y)e^{iEz}=u(x,y)e^{ik_{y}y+iEz}$, where
$E(k_{y})$ represents the energy bands, $k_{y}\in[-K/2,K/2]$ is the Bloch
momentum along the $y$ axis, $K=2\pi/a_{y}$ is the width of Brillouin zone,
$a_{y}$ is the lattice’s y-periodicity, and $u(x,y)\,=\,u(x,y+a_{y})$ is the
periodic part of the Bloch function $\psi$.

The spectrum $E(k_{y})$ is displayed in Fig. 1 (b) and (d). As expected, due
to the breaking of the spatial inversion symmetry in the staggered honeycomb
lattices, a gap opens and a pair of VHESs emerge (indicated by the red curves)
[35]. However, the VHESs exhibit essentially distinct spectra rooted in the
lattice symmetries. The lattice with the bearded domain wall has mirror
symmetry $\mathcal{M}_{x}(x\rightarrow-x)$ along the $x$-axis [Fig.1(c)].
Applying $\mathcal{M}_{x}$ twice returns a field component to its original,
namely, $\mathcal{M}_{x}^{2}\psi(x,y)=\psi(x,y)$. Correspondingly,
$\mathcal{M}_{x}\psi(x,y)=\pm\psi(x,y)$, resulting in the gapped VHESs, as
shown in Fig.1(d). On the other hand, the lattice with the zigzag domain wall
lacks a straightforward mirror symmetry $\mathcal{M}_{x}$ but has a glide-
reflection symmetry denoted as $G=\mathcal{M}_{x}T_{y}(a_{y}/2)$, namely, the
lattice is invariant under $\mathcal{M}_{x}$ followed by a translation along
the $y$ direction by half a lattice constant. When $G$ is applied twice, it
results in a full-lattice-constant translation, i.e.,
$G^{2}\psi(x,y)=\psi(x,y+a_{y})$. By Bloch’s theorem,
$G\psi(x,y)=e^{ik_{y}a_{y}/2}\psi(x,y)$. Considering $k_{y}=\pm K/2$, such
that $G^{2}=-1$, one finds that the eigenvalues of $G$ are $\pm i$ [36], and
consequently, bands must cross by pairs at $k=\pm K/2$, as confirmed in Fig.
1(b). This band crossing is protected by the glide-reflection symmetry and is
insensitive to a limited continuous deformation of the lattice that preserves
the symmetry [37, 38]. Note that, band crossings can also be achieved in
tilted lattices by delicately balancing the coupling between sites and
external forces [39, 40, 41]. However, symmetry-protected band crossing
facilitates experimental observations of the associated band dynamics, with
BOs being a salient example, as will be shown below.

Figure 1:  (a, c) Domain walls of zigzag-type (a) and bearded-type (c),
between two inverted, staggered honeycomb lattices. The red-shaded regions
represent the domain walls, and the rectangular dashed yellow boundaries on
the left and right of the domain walls are used to illustrate the glide-
reflection symmetry in (a) and mirror symmetry in (c). (b, d) Band structures
of the lattices corresponding to (a, c). Red curves represent the VHESs, while
the black curves represent bulk states. Blue dots on the red curves represent
VHESs at the center ($k_{y}=0$) or boundaries ($k_{y}=\pm K/2$) of the
Brillouin zone respectively. Note that VHESs (red curves) are localized in the
$x$ direction but the bulk states (black curves) are delocalized in the $x$
direction, and in what follows we are going to excite only the localized
states.

With a potential gradient $I_{G}$ present in Eq. 1, the VHES is subject to a
constant force along $y$. If the force is small, the evolution of the state is
adiabatic, so that it remains in the same energy band and doesn’t transition
between bands. The Bloch momentum of the VHES increases linearly with
$k_{y}(z)=k_{y}^{0}+\beta z$, $k_{y}^{0}$ being the initial momentum. For
VHESs in a bearded domain with initially zero momentum, they follows either
the path A$\rightarrow$B$\rightarrow$C$\rightarrow$A or
D$\rightarrow$E$\rightarrow$F$\rightarrow$D [Fig. 1(d)], scanning the whole
Brillouin zone once and returning to the initial state. Therefore, the BOs
period in momentum space for the bearded state is $K$, corresponding to a
distance $Z=K/\beta$ in real space. This is the typical BOs scenario.

The situation changes significantly when it comes to the zigzag VHESs due to
band crossing that directly results in the period doubling of BOs. Indeed,
consider a VHES with zero momentum $k_{y}=0$ in the lower band at point A in
Fig. 1(b). Due to the constant force, it moves up to point B at the Brillouin
border where the lower band crosses with the upper band. Upon crossing, it
reappears at the opposite border C, but now on the upper band. It does so
because the phase distributions of the VHESs or the pseudospin, as defined in
relation to it, precisely match between the upper and lower bands at the
Brillouin zone border [34]. Conversely, the phase or pseudospin of the VHESs
at the two edges of the same band is completely opposite (orthogonal),
preventing movement along the same band across the Brillouin zone.

Therefore, after the band crossing, the VHES at point C continues moving up
along the upper band until it reaches the band edge at point B, where it
undergoes another band crossing and moves back to the lower band, and
ultimately return to its initial position A. Thus, unlike the bearded domain
case, where only one band is involved throughout the BOs, the zigzag domain
involves sequential traversal of two bands. This means that they must traverse
the Brillouin zone twice, or a total distance of $2K/\beta$ to complete a full
BOs.

The aforementioned analysis was confirmed through direct simulations of
propagation using Eq. (1), with an excitation condition being a VHES that is
horizontally localized and vertically modulated by a broad Gaussian envelope
of width $w=8$ and centered at, say, $y=y_{0}=-6$, expressed as
$\Psi(x,y)|_{z=0}=\psi(x,y;k_{y})\exp[-({y-y_{0}})^{2}/w^{2}]$. Specifically,
we selected the Bloch wave with momentum $k_{y}=0$ corresponding to point A in
Fig. 1(b) and (d). Such selected VHESs initially move along the domain wall in
the positive $y$ direction [Fig. 2(a), (b)]. After reaching their maximum
displacement, they return to their initial positions, and a new cycle starts.
To quantify the oscillation, the variation of COM of the wavepacket, defined
by $y_{c}=\iint{\Psi}^{*}y{\Psi}dxdy$, was monitored [Fig. 2(c)]. Our results
clearly show that while the bearded state completes one full BOs at $z=27$,
the counterpart wavepacket in the zigzag domain only reaches its maximum
displacement, i.e., completing only half of its BO cycle. Subsequently, it
takes an additional distance of $27$ for the wavepacket to return to its
initial position and complete one full oscillation cycle. By this distance,
the wavepacket in the bearded domain has already completed two full
oscillations. Note also, as the amplitude of BOs is proportional to the width
of the involved band(s)(divided by the force $\beta$), the amplitude of two-
band BOs in the zigzag domain wall is much larger (around 4 times larger in
this case) than that of one-band BOs in the bearded one.

Figure 2: Simulation results for the VHESs evolving at the zigzag (a), and (b)
bearded domain walls, respectively. In both figures, the horizontal green
lines indicate the location of COM ($y_{c}$) of the light beam, that are
further shown in (c). The dashed vertical blue lines shown in (c) indicate the
point where the light beam along the zigzag domain wall attains its maximum
displacement, which is equivalent to half of its period of BOs. $E_{0}=-30$,
$\eta=35,I_{0}=0.25$, $w=8$. Figure 3: (a) Schematic diagram of the
experimental setup. $\lambda/2$: half-wave plate; PBS: polarization beam
splitter; BS: beam splitter; BE: beam expander; RD: rotating diffuser; ID0:
iris diaphragm; ID1: Fourier filter with six holes; ID2: phase mask; SLM:
spatial light modulator. The panel at the bottom-right corner shows an example
of a refractive index ramp $\delta n$ with a width $\eta=120~{}\mu\text{m}$.
(b, c) The lattice-forming light field measured at the output of the SBN,
showing staggered honeycomb lattice profiles with zigzag (b) and bearded (c)
domain walls. The domain walls are depicted within green rectangular-dashed
boundaries. The brightness of the color in the lattices is proportional to the
local light intensity. The insets at the bottom-right corners of (b) and (c)
show the designed structures on the amplitude mask, while the inset at the
top-right of (b) displays the probe light beam at the front facet of the
crystal.

We experimentally investigated BOs with the setup in Fig. 3(a). A Nd: YAG
laser beam ($\lambda=532$ nm) was split into two beams: beam $a$, with
ordinary polarization, writing the required lattice structure into SBN, while
beam $b$, with extraordinary polarization, probing the lattice and studying
the BOs. Note that the crystal’s large electro-optic anisotropy was utilized
in the polarization configuration, to ensure that the probe experiences strong
modulation of the lattice, but the induced lattice remains undistorted during
propagation [32, 33]. The writing beam passed through a rotating diffuser,
which made it partially spatially incoherent, and then went through an
amplitude mask that has a pattern matching the structure of the desired
writing beam, as shown in the bottom-right corner of Fig. 3 (b) and (c). The
mask was then imaged onto the input facet of the crystal through a 4f optical
system with an appropriate filter at the Fourier plane (see the experimental
details in [34]). This process generated an unconventional lattice-forming
beam that remains nondiffracting over a distance of at least $2$ cm in the
sample [43, 42]. Furthermore, to introduce a ramp of refractive index across
the sample, a white light diffracted by a sharp razor blade was utilized to
laterally illuminate the SBN [15, 16, 44].

Figures 3(b) and 3(c) show the optically induced staggered honeycomb lattices
with zigzag and bearded domains, overlaid with a refractive index gradient.
The total induced refractive index change is given by $\delta
n=-\gamma/(1+I_{L}+I_{G})$, where $\gamma$ = ${n_{e}^{3}}{r_{33}}E/2$ =
$3.56\times$10-4 and ${r_{33}}$ is the nonlinear electro-optic coefficient.
The intensity of the lattice-writing beam is $\ I_{L}\approx$
$2.5~{}\text{mW}/\text{cm}^{2}$, and the maximum intensity of the white light
is $I_{G}\approx$ $4.2~{}\text{mW}/\text{cm}^{2}$. An example of the
refractive index ramp $\delta n_{g}$ is given in Fig. 3 (a). As shown, the
index change over one period $a_{y}=55~{}\mu$m due to the ramp is $\sim
10^{-5}$, one order of magnitude smaller than the variation of index of the
periodic lattice $\sim 10^{-4}$. This implies that the influence of the
gradient on the profiles of Bloch modes is negligible, and its main effect is
to initiate the variation of Bloch momentum of the wave packet in the
Brillouin zone.

The lattices are excited by launching into both domain walls a Gaussian laser
beam with an elliptical shape that corresponds to the numerical simulations.
The beam has a width of $90~{}\mu\text{m}$ along the domain wall extension
($y$-axis), and $30~{}\mu\text{m}$ across the domain wall ($x$-axis). Its
elliptical shape ensures that it is wide enough along the $y$ direction so
that only one Bloch mode at $k_{y}=0$ is excited, yet narrow enough to fit
into the width of the domain wall. Thanks to the topological protection of the
valley Hall effect, both types of the VHESs are efficiently excited, as
evidenced by their quick reshaping into the “zigzag” and “bearded” profile,
and remain well-trapped in the domain wall while moving along it, without
radiating into the lattice bulk [Fig. 4 (a), (b)]. Under the effect of the
index ramp, both states are observed to drift as a whole along the domain
wall. The zigzag states exhibit a more profound drift than that in the bearded
domain, yet the overall shift for the latter is still measurably observable in
term of their COM which we extracted from their intensity distribution. As the
plots show, initially, both VHESs move in the positive direction of the
$y$-axis and, at a certain distance, arrive at their maximum displacements
before moving back. Our results clearly show that the maximum displacement for
the light beam in the zigzag domain wall appears much later than in the
bearded domain. Comparing their COM finds that when the bearded state
completes a full cycle at $z\approx 16~{}\text{mm}$, the zigzag state has only
just reached its maximum displacement. This observation clearly indicates that
the period of the BOs for zigzag states is twice that of bearded states, as
theoretically predicted.

Figure 4: Experimental observation of BOs at (a) zigzag and (b) bearded domain
walls. The horizontal green lines indicate the position of COM of the light
beam. The index gradient applied in the lattices increases in the downward
direction, and for the shown particular case, the width of the index gradient
is $\eta=120~{}\mu\text{m}$. Note in the bearded state case (b), the close
proximity of two horizontal potential maxima makes the light spots between
them unresolvable [cf Fig. 2(b)]. (c) Displacement of the COM versus
propagation distance $z$ under three different index gradients.

This period-doubling effect is further demonstrated in Fig. 4 (c), where we
compare the evolution of the COM for both VHESs at three different widths of
the induced index ramp $\eta$, i.e., $\eta=120,90,60~{}\mu\text{m}$,
corresponding to three different gradients $\beta$ of refractive index [34].
For each $\eta$, the experiment was conducted five times under the same
parameter settings, thus each curve in Fig. 4(c) is the average of the
outcomes obtained from five experimental repetitions.

As expected, a larger gradient of refractive index leads to faster BOs of the
wavepackets. Therefore, the period of BOs for zigzag states is about
$32~{}\text{mm}$ at $\eta=120~{}\mu\text{m}$, but it is reduced to around
$20~{}\text{mm}$ at $\eta=60~{}\mu\text{m}$. This allows us to observe a
complete cycle of BOs within the $2~{}\text{cm}$ long sample [Fig. 4(c),
right]. However, it is worth noting that the period at $\eta=60~{}\mu\text{m}$
is not exactly halved compared to $\eta=120~{}\mu\text{m}$, as would be
expected in the ideal case. This discrepancy is attributed to the relatively
narrow width of the index ramp ($\eta=60~{}\mu\text{m}$) compared to the width
of the wavepacket, which has a fixed $y$-width of $90~{}\mu\text{m}$. Due to
this, the light beam experiences a slightly shallower “local” index ramp when
oscillating near the ramp boundary, compared to oscillating in the ramp
center. As a result, the effective ramp experienced by the light wavepacket is
reduced, leading to a slightly longer period. Among others [34], the slowly
varying gradient of the index also explains the unevenness in the overall
oscillation seen in the right two panels of Fig. 4(c), compared to the more
ideal oscillation observed in the first panel. Surprisingly, however, although
the oscillation speed may slowly change during the oscillation, which could
potentially affect the period and amplitude of oscillation, the relative
relationship of period doubling between zigzag and bearded domain walls
remains fixed. This is specifically illustrated by the vertical dashed blue
lines, which represent the distance the light beam at the zigzag domains
travels to reach the maximum displacement (half of its oscillation period), is
found to match well with the distance at which the light beam at the bearded
domains completes one full oscillation cycle!

Finally, Fig. 4(c) also reveals a remarkable relationship between the
amplitude of BOs for the two VHESs. As their oscillation periods decrease with
the decreasing of $\eta$ (or increasing $\beta$ ), their amplitudes also
decrease. However, in all the shown cases, the amplitude of oscillation in
zigzag states is significantly larger (two times larger) compared to the
bearded states, reflecting that the former BOs involve two bands while the
latter involves only one.

Before closing, it is worth noting that the glide-reflection symmetry, which
originates in the band degeneracy and period-doubling of BOs in this study, is
just one aspect of broader nontrivial point-group $G$ [45]. Apart from the
symmetries of glide of reflection, other symmetries of $G$, including the
symmetry of rotations, inversion, and screw rotations, are also often present,
that in general indicates BOs with $\upmu$-multiple periods [46]. Hence, our
experimental observation of period-doubling ($\upmu=2$) marks the first
measurement of an entire family of topological invariants, and is expected to
promote general measurements of the topological invariants $\upmu$
characterizing the relevant topological materials.

###### Acknowledgements.

N. K. and F. Y. acknowledge the support of Shanghai Jiao Tong University
Scientific and Technological Innovation Funds, and Shanghai Outstanding
Academic Leaders Plan (No. 20XD1402000). P. W. acknowledges funding from the
National Natural Science Foundation (No. 12304366) and China Postdoctoral
Science Foundation (No. BX20230217). Q. F. acknowledges the support of China
Postdoctoral Science Foundation (No. BX20230218).

## References

  * [1] F. Bloch, Uber die quantenmechanik der elektronen in kristallgittern, Z. Phys. 52, 555 (1929).
  * [2] C. Zener, A theory of the electrical breakdown of solid dielectrics, Proc. R. Soc. London. Ser. A, Containing Papers of a Mathematical and Physical Character 145, 523 (1934).
  * [3] E. E. Mendez, F. Agulló-Rueda, and J. M. Hong, Stark localization in GaAs-GaAlAs superlattices under an electric field, Phys. Rev. Lett. 60, 2426 (1988).
  * [4] P. Voisin, J. Bleuse, C. Bouche, S. Gaillard, C. Alibert, and A. Regreny, Observation of the Wannier-Stark quantization in a semiconductor superlattice, Phys. Rev. Lett. 61, 1639 (1988).
  * [5] J. Feldmann, K. Leo, J. Shah, D. A. B. Miller, J. E. Cunningham, T. Meier, G. Von Plessen, A. Schulze, P. Thomas, and S. Schmitt-Rink, Optical investigation of Bloch oscillations in a semiconductor superlattice, Phys. Rev. B 46, 7252 (1992).
  * [6] C. Waschke, H. G. Roskos, R. Schwedler, K. Leo, H. Kurz, and K. Köhler, Coherent submillimeter-wave emission from Bloch oscillations in a semiconductor superlattice, Phys. Rev. Lett. 70, 3319 (1993).
  * [7] M. B. Dahan, E. Peik, J. Reichel, Y. Castin, and C. Salomon, Bloch oscillations of atoms in an optical potential, Phys. Rev. Lett. 76, 4508 (1996).
  * [8] O. M. J. H. Morsch, J. H. Müller, M. Cristiani, D. Ciampini, and E. Arimondo, Bloch oscillations and mean-field effects of Bose-Einstein condensates in 1D optical lattices, Phys. Rev. Lett. 87, 140402 (2001).
  * [9] B. P. Anderson, and M. A. Kasevich, Macroscopic quantum interference from atomic tunnel arrays, Science 282, 1686 (1998).
  * [10] U. Peschel, T. Pertsch, and F. Lederer, Optical Bloch oscillations in waveguide arrays, Opt. Lett. 23, 1701 (1998).
  * [11] T. Pertsch, P. Dannberg, W. Elflein, A. Bräuer, and F. Lederer, Optical Bloch oscillations in temperature tuned waveguide arrays, Phys. Rev. Lett. 83, 4752 (1999).
  * [12] F. Dreisow, A. Szameit, M. Heinrich, T. Pertsch, S. Nolte, A. Tunnermann, and S. Longhi, Bloch-Zener oscillations in binary superlattices, Phys. Rev. Lett. 102, 076802 (2009).
  * [13] S. Stützer, Y. V. Kartashov, V. A. Vysloukh, V. V. Konotop, S. Nolte, L. Torner, and A. Szameit, Hybrid Bloch-Anderson localization of light, Opt. Lett. 38, 1488 (2013).
  * [14] A. Joushaghani, R. Iyer, J. K. Poon, J. S. Aitchison, C. M. De Sterke, J. Wan, and M. M. Dignam, Quasi-Bloch oscillations in curved coupled optical waveguides, Phys. Rev. Lett. 103, 143903 (2009).
  * [15] H. Trompeter, W. Krolikowski, D. N. Neshev, A. S. Desyatnikov, A. A. Sukhorukov, Y. S. Kivshar, T. Pertsch, U. Peschel, and F. Lederer, Bloch oscillations and Zener tunneling in two-dimensional photonic lattices, Phys. Rev. Lett. 96, 053903 (2006).
  * [16] Y. Sun, D. Leykam, S. Nenni, D. Song, H. Chen, Y. D. Chong, and Z. Chen, Observation of valley Landau-Zener-Bloch oscillations and pseudospin imbalance in photonic graphene, Phys. Rev. Lett. 121, 033904 (2018).
  * [17] A. Block, C. Etrich, T. Limboeck, F. Bleckmann, E. Soergel, C. Rockstuhl, and S. Linden, Bloch oscillations in plasmonic waveguide arrays, Nat. Commun. 5, 3843 (2014).
  * [18] M. Wimmer, M. A. Miri, D. Christodoulides, and U. Peschel, Observation of Bloch oscillations in complex PT-symmetric photonic lattices, Sci. Rep. 5, 1 (2015).
  * [19] W. Dias, E. Nascimento, M. Lyra, and F. de Moura, Frequency doubling of Bloch oscillations for interacting electrons in a static electric field, Phys. Rev. B 76, 155124 (2007).
  * [20] P. Preiss, R. Ma, M. Tai, A. Lukin, M. Rispoli, P. Zupancic, Y. Lahini, R. Islam, and M. Greiner, Strongly correlated quantum walks in optical lattices, Science 347, 1229 (2015).
  * [21] G. Corrielli, A. Crespi, G. D. Valle, S. Longhi, and R. Osellame, Fractional Bloch oscillations in photonic lattices, Nat. Comm. 4, 1555 (2013).
  * [22] L. Lu, J. D. Joannopoulos, and M. Soljačić, Topological photonics, Nat. photon. 8, 821 (2014).
  * [23] M. Atala, M. Aidelsburger, J. T. Barreiro, D. Abanin, T. Kitagawa, E. Demler, and I. Bloch, Direct measurement of the Zak phase in topological Bloch bands, Nat. Phys. 9, 795 (2013).
  * [24] M. Aidelsburger, M. Lohse, C. Schweizer, M. Atala, J. T. Barreiro, S. Nascimbène, N. R. Cooper, I. Bloch, and N. Goldman, Measuring the Chern number of Hofstadter bands with ultracold bosonic atoms, Nat. Phys. 11, 162 (2015).
  * [25] G. Jotzu, M. Messer, R. Desbuquois, M. Lebrat, T. Uehlinger, D. Greif, and T. Esslinger, Experimental realization of the topological Haldane model with ultracold fermions, Nature 515, 237 (2014).
  * [26] D. Xiao, M.-C. Chang, and Q. Niu, Berry phase effects on electronic properties, Rev. Mod. Phys. 82 1959 (2010).
  * [27] I. Yu. Chesnokov and A. R. Kolovsky, Landau-Stark states in finite lattices and edge-induced Bloch oscillations, Europhys. Lett. 106, 5001(2014).
  * [28] C. Li, W. Zhang, Y. V. Kartashov, D. V. Skryabin, and F. Ye, Bloch oscillations of topological edge modes, Phys. Rev. A 99, 053814 (2019).
  * [29] Y. Plotnik, M. A. Bandres, Y. Lumer, M. Rechtsman, and M. Segev, Topological control of Bloch oscillations of edge modes in photonic lattices, in Conference on Lasers and Electro-Optics (CLEO) (Optical Society of America, San Jose, 2015) paper FTu2C–4.
  * [30] Y. V. Kartashov, V. V. Konotop, D. A. Zezyulin, and L. Torner, Bloch oscillations in optical and Zeeman lattices in the presence of spin-orbit coupling, Phys. Rev. Lett. 117, 215301 (2016).
  * [31] C-H Li, Y. Yan, S-W Feng, S. Choudhury, D. B. Blasing, Q. Zhou, and Y. P. Chen, Bose-Einstenin condensate on a synthetic topological Hall cylinder, PRX Quantum 3, 010316 (2022).
  * [32] N. K. Efremidis, S. Sears, D. N. Christodoulides, J. W. Fleischer, and M.Segev, Discrete solitons in photorefractive optically induced photonic lattices, Phys. Rev. E 66, 046602 (2002).
  * [33] J. W. Fleischer, M. Segev, N. K. Efremidis, and D. N. Christodoulides, Observation of two-dimensional discrete solitons in optically induced nonlinear photonic lattices, Nature 422, 147 (2003).
  * [34] See the details in the Supplemental Materials.
  * [35] J. Noh, S. Huang, K. P. Chen, and M. C. Rechtsman, Observation of photonic topological valley Hall edge states, Phys. Rev. Lett. 120, 063902 (2018).
  * [36] A. Mock, L. Lu, and J. O’Brien, Space group theory and Fourier space analysis of two-dimensional photonic crystal waveguides, Phys. Rev. B 81, 155115 (2010).
  * [37] J. Martinez, N. Laforge, M. Kadic, and V. Laude, Topological waves guided by a glide-reflection symmetric crystal interface, Phys. Rev. B 106, 064304 (2022).
  * [38] An alternative viewpoint in the interpretation of the band crossing in the zigzag Valley Hall edge states is that the zigzag domain wall itself can be regarded as a standard 1D Su-Schrieffer-Heeger (SSH) model, where the intercell and intracell couplings are equal.
  * [39] A. R. Kolovsky and E. N. Bulgakov, Wannier-Stark states and Bloch oscillations in the honeycomb lattice, Phys. Rev. A 87, 033602 (2013).
  * [40] A. R. Kolovsky, Topological phase transitions in tilted optical lattices, Phys. Rev. A 98, 013603 (2018).
  * [41] A. R. Kolovsky, A. Ramachandran, and S. Flach, Topological flat Wannier-Stark bands, Phys. Rev. B 97, 045120 (2018).
  * [42] Z. Chen, and K. McCarthy, Spatial soliton pixels from partially incoherent light, Opt. Lett. 27, 2019 (2002).
  * [43] H. Martin, E. D. Eugenieva, Z. Chen, and D. N. Christodoulides, Discrete solitons and soliton-induced dislocations in partially coherent photonic lattices, Phys. Rev. Lett. 92, 123902 (2004).
  * [44] B. Terhalle, A. S. Desyatnikov, D. N. Neshev, W. Krolikowski, C. Denz, and Y. S. Kivshar, Dynamic diffraction and interband transitions in two-dimensional photonic lattices, Phys. Rev. Lett. 106, 083902 (2011).
  * [45] H. Hiller, Crystallography and cohomology of groups, Am. Math. Mon. 93, 765 (1986).
  * [46] J. Hoeller, and A. Alexandradinata, Topological Bloch oscillations, Phys. Rev. B 98, 024310 (2018).


* [96] R. S. de Jong, O. Agertz, A. Agudo Berbel, J. Aird, D. A. Alexander, A. Amarsi, F. Anders, R. Andrae, B. Ansarinejad, W. Ansorge, P. Antilogus, H. Anwand -Heerwart, A. Arentsen, A. Arnadottir, M. Asplund, M. Auger, N. Azais, D. Baade, G. Baker, S. Baker, E. Balbinot, I. K. Baldry, M. Banerji, S. Barden, P. Barklem, E. Barthélémy-Mazot, C. Battistini, S. Bauer, C. P. M. Bell, O. Bellido-Tirado, S. Bellstedt, V. Belokurov, T. Bensby, M. Bergemann, J. M. Bestenlehner, R. Bielby, M. Bilicki, C. Blake, J. Bland-Hawthorn, C. Boeche, W. Boland, T. Boller, S. Bongard, A. Bongiorno, P. Bonifacio, D. Boudon, D. Brooks, M. J. I. Brown, R. Brown, M. Brüggen, J. Brynnel, J. Brzeski, T. Buchert, P. Buschkamp, E. Caffau, P. Caillier, J. Carrick, L. Casagrande, S. Case, A. Casey, I. Cesarini, G. Cescutti, D. Chapuis, C. Chiappini, M. Childress, N. Christlieb, R. Church, M. R. L. Cioni, M. Cluver, M. Colless, T. Collett, J. Comparat, A. Cooper, W. Couch, F. Courbin, S. Croom, D. Croton, E. Daguisé, G. Dalton, L. J. M. Davies, T. Davis, P. de Laverny, A. Deason, F. Dionies, K. Disseau, P. Doel, D. Döscher, S. P. Driver, T. Dwelly, D. Eckert, A. Edge, B. Edvardsson, D. El Youssoufi, A. Elhaddad, H. Enke, G. Erfanianfar, T. Farrell, T. Fechner, C. Feiz, S. Feltzing, I. Ferreras, D. Feuerstein, D. Feuillet, A. Finoguenov, D. Ford, S. Fotopoulou, M. Fouesneau, C. Frenk, S. Frey, W. Gaessler, S. Geier, N. Gentile Fusillo, O. Gerhard, T. Giannantonio, D. Giannone, B. Gibson, P. Gillingham, C. González-Fernández, E. Gonzalez-Solares, S. Gottloeber, A. Gould, E. K. Grebel, A. Gueguen, G. Guiglion, M. Haehnelt, T. Hahn, C. J. Hansen, H. Hartman, K. Hauptner, K. Hawkins, D. Haynes, R. Haynes, U. Heiter, A. Helmi, C. Hernandez Aguayo, P. Hewett, S. Hinton, D. Hobbs, S. Hoenig, D. Hofman, I. Hook, J. Hopgood, A. Hopkins, A. Hourihane, L. Howes, C. Howlett, T. Huet, M. Irwin, O. Iwert, P. Jablonka, T. Jahn, K. Jahnke, A. Jarno, S. Jin, P. Jofre, D. Johl, D. Jones, H. Jönsson, C. Jordan, I. Karovicova, A. Khalatyan, A. Kelz, R. Kennicutt, D. King, F. Kitaura, J. Klar, U. Klauser, J. Kneib, A. Koch, S. Koposov, G. Kordopatis, A. Korn, J. Kosmalski, R. Kotak, M. Kovalev, K. Kreckel, Y. Kripak, M. Krumpe, K. Kuijken, A. Kunder, I. Kushniruk, M. I Lam, G. Lamer, F. Laurent, J. Lawrence, M. Lehmitz, B. Lemasle, J. Lewis, B. Li, C. Lidman, K. Lind, J. Liske, J. L. Lizon, J. Loveday, H. G. Ludwig, R. M. McDermid, K. Maguire, V. Mainieri, S. Mali, H. Mandel, K. Mandel, L. Mannering, S. Martell, D. Martinez Delgado, G. Matijevic, H. McGregor, R. McMahon, P. McMillan, O. Mena, A. Merloni, M. J. Meyer, C. Michel, G. Micheva, J. E. Migniau, I. Minchev, G. Monari, R. Muller, D. Murphy, D. Muthukrishna, K. Nandra, R. Navarro, M. Ness, V. Nichani, R. Nichol, H. Nicklas, F. Niederhofer, P. Norberg, D. Obreschkow, S. Oliver, M. Owers, N. Pai, S. Pankratow, D. Parkinson, I. Parry, J. Paschke, R. Paterson, A. Pecontal, D. Phillips, A. Pillepich, L. Pinard, J. Pirard, N. Piskunov, V. Plank, D. Plüschke, E. Pons, P. Popesso, C. Power, J. Pragt, A. Pramskiy, D. Pryer, M. Quattri, A. B. de Andrade Queiroz, A. Quirrenbach, S. Rahurkar, A. Raichoor, S. Ramstedt, A. Rau, A. Recio-Blanco, R. Reiss, F. Renaud, Y. Revaz, P. Rhode, J. Richard, A. D. Richter, H. W. Rix, A. S. G. Robotham, R. Roelfsema, M. Romaniello, D. Rosario, F. Rothmaier, B. Roukema, G. Ruchti, G. Rupprecht, J. Rybizki, N. Ryde, A. Saar, E. Sadler, M. Sahlén, M. Salvato, B. Sassolas, W. Saunders, A. Saviauk, L. Sbordone, T. Schmidt, O. Schnurr, R. D. Scholz, A. Schwope, W. Seifert, T. Shanks, A. Sheinis, T. Sivov, Á. Skúladóttir, S. Smartt, S. Smedley, G. Smith, R. Smith, J. Sorce, L. Spitler, E. Starkenburg, M. Steinmetz, I. Stilz, J. Storm, M. Sullivan, W. Sutherland, E. Swann, A. Tamone, E. N. Taylor, J. Teillon, E. Tempel, R. ter Horst, W. F. Thi, E. Tolstoy, S. Trager, G. Traven, P. E. Tremblay, L. Tresse, M. Valentini, R. van de Weygaert, M. van den Ancker, J. Veljanoski, S. Venkatesan, L. Wagner, K. Wagner, C. J. Walcher, L. Waller, N. Walton, L. Wang, R. Winkler, L. Wisotzki, C. C. Worley, G. Worseck, M. Xiang, W. Xu, D. Yong, C. Zhao, J. Zheng, F. Zscheyge, and D. Zucker.  4MOST: Project overview and information for the First Call for Proposals.  arXiv e-prints, page arXiv:1903.02464, Mar 2019.
  * [97] B. Joachimi, M. Cacciato, T. D. Kitching, A. Leonard, R. Mandelbaum, B. M. Schäfer, C. Sifón, H. Hoekstra, A. Kiessling, D. Kirk, and A. Rassat.  Galaxy Alignments: An Overview.  Space Sci. Rev., 193:1–65, November 2015.
  * [98] M. A. Troxel and Mustapha Ishak.  The Intrinsic Alignment of Galaxies and its Impact on Weak Gravitational Lensing in an Era of Precision Cosmology.  Phys. Rept., 558:1–59, 2014.
  * [99] D. Kirk, A. Rassat, O. Host, and S. Bridle.  The cosmological impact of intrinsic alignment model choice for cosmic shear.  MNRAS, 424:1647–1657, August 2012.
  * [100] Elisabeth Krause, Tim Eifler, and Jonathan Blazek.  The impact of intrinsic alignment on current and future cosmic shear surveys.  MNRAS, 456:207–222, February 2016.
  * [101] Jonathan A. Blazek, Niall MacCrann, M. A. Troxel, and Xiao Fang.  Beyond linear galaxy alignments.  Phys. Rev. D, 100:103506, Nov 2019.
  * [102] J. Yao, M. Ishak, W. Lin, and M. Troxel.  Effects of self-calibration of intrinsic alignment on cosmological parameter constraints from future cosmic shear surveys.  J. Cosmology Astropart. Phys., 10:056, October 2017.
  * [103] Daniel Martens, Christopher M. Hirata, Ashley J. Ross, and Xiao Fang.  A radial measurement of the galaxy tidal alignment magnitude with BOSS data.  MNRAS, 478(1):711–732, July 2018.
  * [104] Nora Elisa Chisari, Cora Dvorkin, Fabian Schmidt, and David N. Spergel.  Multitracing anisotropic non-Gaussianity with galaxy shapes.  Phys. Rev. D, 94:123507, December 2016.
  * [105] B. Joachimi, R. Mandelbaum, F. B. Abdalla, and S. L. Bridle.  Constraints on intrinsic alignment contamination of weak lensing surveys using the MegaZ-LRG sampl e.  A&A, 527:A26, March 2011.
  * [106] Harry Johnston, Christos Georgiou, Benjamin Joachimi, Henk Hoekstra, Nora Elisa Chisari, Daniel Farrow, Maria Cristina Fortuna, Catherine Heymans, Shahab Joudaki, Konrad Kuijken, and Angus Wright.  KiDS+GAMA: Intrinsic alignment model constraints for current and future weak lensing cosmology.  arXiv e-prints, page arXiv:1811.09598, November 2018.
  * [107] A. Tenneti, R. Mandelbaum, and T. Di Matteo.  Intrinsic alignments of disk and elliptical galaxies in the MassiveBlack-II and Illustris simulati ons.  ArXiv e-prints, October 2015.
  * [108] N. Chisari, S. Codis, C. Laigle, Y. Dubois, C. Pichon, and J. Devriendt.  Intrinsic alignments of galaxies in the Horizon-AGN cosmological hydrodynamical simulation.  MNRAS, 454:2736–2753, December 2015.
  * [109] Maria Cristina Fortuna, Henk Hoekstra, Benjamin Joachimi, Harry Johnston, Nora Elisa Chisari, Christos Georgiou, and Constance Mahony.  The halo model as a versatile tool to predict intrinsic alignments.  MNRAS, 501(2):2983–3002, February 2021.
  * [110] Harry Johnston, Christos Georgiou, Benjamin Joachimi, Henk Hoekstra, Nora Elisa Chisari, Daniel Farrow, Maria Cristina Fortuna, Catherine Heymans, Shahab Joudaki, Konrad Kuijken, and Angus Wright.  KiDS+GAMA: Intrinsic alignment model constraints for current and future weak lensing cosmology.  A&A, 624:A30, Apr 2019.
  * [111] S. Singh, R. Mandelbaum, and S. More.  Intrinsic alignments of SDSS-III BOSS LOWZ sample galaxies.  MNRAS, 450:2195–2216, June 2015.
  * [112] N. E. Chisari, R. Mandelbaum, M. A. Strauss, E. M. Huff, and N. A. Bahcall.  Intrinsic alignments of group and cluster galaxies in photometric surveys.  MNRAS, 445:726–748, November 2014.
  * [113] K. Kraljic, S. Arnouts, C. Pichon, C. Laigle, S. de la Torre, D. Vibert, C. Cadiou, Y. Dubois, M. Treyer, C. Schimd, S. Codis, V. de Lapparent, J. Devriendt, H. S. Hwang, D. Le Borgne, N. Malavasi, B. Milliard, M. Musso, D. Pogosyan, M. Alpaslan, J. Bland-Hawthorn, and A. H. Wright.  Galaxy evolution in the metric of the cosmic web.  MNRAS, 474:547–571, February 2018.
  * [114] S. Codis, A. Jindal, N. E. Chisari, D. Vibert, Y. Dubois, C. Pichon, and J. Devriendt.  Galaxy orientation with the cosmic web across cosmic time.  MNRAS, 481:4753–4774, December 2018.
  * [115] Knut Olsen, Marcella Di Criscienzo, R. Lynne Jones, Megan E. Schwamb, Hsing Wen ”Edward” Lin, Humna Awan, Phil Marshall, Eric Gawiser, Adam Bolton, and Daniel Eisenstein.  A Big Sky Approach to Cadence Diplomacy.  arXiv e-prints, page arXiv:1812.02204, Dec 2018.
  * [116] F. Schmidt, N. E. Chisari, and C. Dvorkin.  Imprint of inflation on galaxy shape correlations.  ArXiv e-prints, June 2015.
  * [117] D. E. Applegate, A. von der Linden, P. L. Kelly, M. T. Allen, S. W. Allen, P. R. Burchat, D. L. Burke, H. Ebeling, A. Mantz, and R. G. Morris.  Weighing the Giants - III. Methods and measurements of accurate galaxy cluster weak-lensing masses.  MNRAS, 439:48–72, March 2014.
  * [118] T. McClintock, T. N. Varga, D. Gruen, E. Rozo, E. S. Rykoff, T. Shin, P. Melchior, J. DeRose, S. Seitz, J. P. Dietrich, E. Sheldon, Y. Zhang, A. von der Linden, T. Jeltema, A. B. Mantz, A. K. Romer, S. Allen, M. R. Becker, A. Bermeo, S. Bhargava, M. Costanzi, S. Everett, A. Farahi, N. Hamaus, W. G. Hartley, D. L. Hollowood, B. Hoyle, H. Israel, P. Li, N. MacCrann, G. Morris, A. Palmese, A. A. Plazas, G. Pollina, M. M. Rau, M. Simet, M. Soares-Santos, M. A. Troxel, C. Vergara Cervantes, R. H. Wechsler, J. Zuntz, T. M. C. Abbott, F. B. Abdalla, S. Allam, J. Annis, S. Avila, S. L. Bridle, D. Brooks, D. L. Burke, A. Carnero Rosell, M. Carrasco Kind, J. Carretero, F. J. Castander, M. Crocce, C. E. Cunha, C. B. D’Andrea, L. N. da Costa, C. Davis, J. De Vicente, H. T. Diehl, P. Doel, A. Drlica-Wagner, A. E. Evrard, B. Flaugher, P. Fosalba, J. Frieman, J. García-Bellido, E. Gaztanaga, D. W. Gerdes, T. Giannantonio, R. A. Gruendl, G. Gutierrez, K. Honscheid, D. J. James, D. Kirk, E. Krause, K. Kuehn, O. Lahav, T. S. Li, M. Lima, M. March, J. L. Marshall, F. Menanteau, R. Miquel, J. J. Mohr, B. Nord, R. L. C. Ogando, A. Roodman, E. Sanchez, V. Scarpine, R. Schindler, I. Sevilla-Noarbe, M. Smith, R. C. Smith, F. Sobreira, E. Suchyta, M. E. C. Swanson, G. Tarle, D. L. Tucker, V. Vikram, A. R. Walker, and J. Weller.  Dark Energy Survey Year 1 results: weak lensing mass calibration of redMaPPer galaxy clusters.  MNRAS, 482:1352–1378, January 2019.
  * [119] A. Molino, N. Benítez, B. Ascaso, D. Coe, M. Postman, S. Jouvel, O. Host, O. Lahav, S. Seitz, E. Medezinski, P. Rosati, W. Schoenell, A. Koekemoer, Y. Jimenez-Teja, T. Broadhurst, P. Melchior, I. Balestra, M. Bartelmann, R. Bouwens, L. Bradley, N. Czakon, M. Donahue, H. Ford, O. Graur, G. Graves, C. Grillo, L. Infante, S. W. Jha, D. Kelson, R. Lazkoz, D. Lemze, D. Maoz, A. Mercurio, M. Meneghetti, J. Merten, L. Moustakas, M. Nonino, S. Orgaz, A. Riess, S. Rodney, J. Sayers, K. Umetsu, W. Zheng, and A. Zitrin.  CLASH: accurate photometric redshifts with 14 HST bands in massive galaxy cluster cores.  MNRAS, 470:95–113, September 2017.
  * [120] A. Diaferio.  Mass estimation in the outer regions of galaxy clusters.  MNRAS, 309:610–622, November 1999.
  * [121] K. Rines, M. J. Geller, M. J. Kurtz, and A. Diaferio.  CAIRNS: The Cluster and Infall Region Nearby Survey. I. Redshifts and Mass Profiles.  AJ, 126:2152–2170, November 2003.
  * [122] M. Falco, G. A. Mamon, R. Wojtak, S. H. Hansen, and S. Gottlöber.  Dynamical signatures of infall around galaxy clusters: a generalized Jeans equation.  MNRAS, 436:2639–2649, December 2013.
  * [123] J. Arthur, F. R. Pearce, M. E. Gray, P. J. Elahi, A. Knebe, A. M. Beck, W. Cui, D. Cunnama, R. Davé, S. February, S. Huang, N. Katz, S. T. Kay, I. G. McCarthy, G. Murante, V. Perret, C. Power, E. Puchwein, A. Saro, F. Sembolini, R. Teyssier, and G. Yepes.  nIFTy galaxy cluster simulations - V. Investigation of the cluster infall region.  MNRAS, 464:2027–2038, January 2017.
  * [124] K. J. Rines, M. J. Geller, A. Diaferio, H. S. Hwang, and J. Sohn.  HeCS-red: Dense Hectospec Surveys of redMaPPer-selected Clusters.  ApJ, 862:172, August 2018.
  * [125] Y. Zu, D. H. Weinberg, E. Jennings, B. Li, and M. Wyman.  Galaxy infall kinematics as a test of modified gravity.  MNRAS, 445:1885–1897, December 2014.
  * [126] Kevork N. Abazajian et al.  CMB-S4 Science Book, First Edition.  10 2016.
  * [127] K. Abazajian et al. (CMB-S4 Collaboration).  CMB-S4 Science Case, Reference Design and Project Plan.  2019\.
  * [128] R. A. Sunyaev and Ya. B. Zeldovich.  Small scale fluctuations of relic radiation.  Astrophys. Space Sci., 7:3–19, 1970.
  * [129] J. E. Carlstrom, G. P. Holder, and E. D. Reese.  Cosmology with the Sunyaev-Zel’dovich Effect.  ARA&A, 40:643–680, 2002.
  * [130] T. Baldauf, R. E. Smith, U. Seljak, and R. Mandelbaum.  Algorithm for the direct reconstruction of the dark matter correlation function from weak lensing and galaxy clustering.  Phys. Rev. D, 81(6):063531, March 2010.
  * [131] R. Mandelbaum, A. Slosar, T. Baldauf, U. Seljak, C. M. Hirata, R. Nakajima, R. Reyes, and R. E. Smith.  Cosmological parameter constraints from galaxy-galaxy lensing and galaxy clustering with the SDSS DR7.  MNRAS, 432:1544–1575, June 2013.
  * [132] R. Reyes, R. Mandelbaum, U. Seljak, T. Baldauf, J. E. Gunn, L. Lombriser, and R. E. Smith.  Confirmation of general relativity on large scales from weak lensing and galaxy velocities.  Nature, 464:256–258, March 2010.
  * [133] P. Zhang, M. Liguori, R. Bean, and S. Dodelson.  Probing Gravity at Cosmological Scales by Measurements which Test the Relationship between Gravitational Lensing and Matter Overdensity.  Physical Review Letters, 99(14):141302, October 2007.
  * [134] P. Capak, D. Sconlic, J-C. Cuillandre, F. Castander, A. Bolton, R. Bowler, C. Chang, A. Dey, T. Eifler, D. Eisenstein, C. Grillmair, P. Gris, N. Hernitschek, I. Hook, C. Hirata, B. Jain K. Kuijken, M. Lochner, J. Newman, P. Oesch, K. Olsen, J. Rhodes, B. Robertson, D. Rubin, C. Scarlata, J. Silverman, S. Wachter, Y. Wang, and The Tri-Agency Working Group.  Mini-survey of the northern sky to Dec ¡+30.  arXiv e-prints, page arXiv:1904.10438, April 2019.
  * [135] David Schlegel, Juna A. Kollmeier, and Simone Ferraro.  The MegaMapper: a z¿2 spectroscopic instrument for the study of Inflation and Dark Energy.  In Bulletin of the American Astronomical Society, volume 51, page 229, September 2019.
  * [136] Alexis Hill, Nicolas Flagey, Alan McConnachie, Kei Szeto, Andre Anthony, Javier Ariño, Ferdinand Babas, Gregoire Bagnoud, Gabriella Baker, Gregory Barrick, Steve Bauman, Tom Benedict, Christophe Berthod, Armando Bilbao, Alberto Bizkarguenaga, Alexandre Blin, Colin Bradley, Denis Brousseau, Rebecca Brown, Jurek Brzeski, Walter Brzezik, Patrick Caillier, Ramón Campo, Pierre-Henri Carton, Jiaru Chu, Vladimir Churilov, David Crampton, Lisa Crofoot, Laurie Dale, Lander de Bilbao, Markel Sainz de la Maza, Daniel Devost, Michael Edgar, Darren Erickson, Tony Farrell, Pascal Fouque, Paul Fournier, Javier Garrido, Mike Gedig, Nicolas Geyskens, James Gilbert, Peter Gillingham, Guillermo González de Rivera, Greg Green, Eric Grigel, Patrick Hall, Kevin Ho, David Horville, Hongzhuan Hu, David Irusta, Sidik Isani, Farbod Jahandar, Manoj Kaplinghat, Collin Kielty, Neelesh Kulkarni, Leire Lahidalga, Florence Laurent, Jon Lawrence, Mary Beth Laychak, Jooyoung Lee, Zhigang Liu, Nathan Loewen, Fernando López, Thomas Lorentz, Guillaume Lorgeoux, Billy Mahoney, Slavko Mali, Eric Manuel, Sofía Martínez, Celine Mazoukh, Younès Messaddeq, Jean-Emmanuel Migniau, Shan Mignot, Stephanie Monty, Steeve Morency, Yves Mouser, Ronny Muller, Rolf Muller, Gaizka Murga, Rick Murowinski, Victor Nicolov, Naveen Pai, Rafal Pawluczyk, John Pazder, Arlette Pécontal, Andreea Petric, Francisco Prada, Corinne Rai, Coba Ricard, Jennifer Roberts, J. Michael Rodgers, Jane Rodgers, Federico Ruan, Tamatea Russelo, Derrick Salmom, Justo Sánchez, Will Saunders, Case Scott, Andy Sheinis, Douglas Simons, Scott Smedley, Zhen Tang, Jose Teran, Simon Thibault, Sivarani Thirupathi, Laurence Tresse, Mitchell Troy, Rafael Urrutia, Emile van Vuuren, Sudharshan Venkatesan, Kim Venn, Tom Vermeulen, Eva Villaver, Lew Waller, Lei Wang, Jianping Wang, Eric Williams, Matt Wilson, Kanoa Withington, Christophe Yèche, David Yong, Chao Zhai, Kai Zhang, Ross Zhelem, and Zengxiang Zhou.  The Maunakea Spectroscopic Explorer Book 2018.  arXiv e-prints, page arXiv:1810.08695, Oct 2018.
  * [137] Richard Ellis and Kyle Dawson.  SpecTel: A 10-12 meter class Spectroscopic Survey Telescope.  In Bulletin of the American Astronomical Society, volume 51, page 45, September 2019.
  * [138] Kyle Dawson, Andrew Hearin, Katrin Heitmann, Mustapha Ishak, Johannes Ulf Lange, Martin White, and Rongpu Zhou.  Snowmass2021 Cosmic Frontier White Paper: High Density Galaxy Clustering in the Regime of Cosmic Acceleration.  arXiv e-prints, page arXiv:2203.07291, March 2022.
  * [139] Simone Ferraro, Noah Sailer, Anze Slosar, and Martin White.  Snowmass2021 Cosmic Frontier White Paper: Cosmology and Fundamental Physics from the three-dimensional Large Scale Structure.  arXiv e-prints, page arXiv:2203.07506, March 2022.
  * [140] Keith Bechtol, Simon Birrer, Francis-Yan Cyr-Racine, Katelin Schutz, Susmita Adhikari, Arka Banerjee, Simeon Bird, Nikita Blinov, Kimberly K. Boddy, Celine Boehm, Kevin Bundy, Malte Buschmann, Sukanya Chakrabarti, David Curtin, Liang Dai, Alex Drlica-Wagner, Cora Dvorkin, Adrienne L. Erickcek, Daniel Gilman, Saniya Heeba, Stacy Kim, Vid Iršič, Alexie Leauthaud, Mark Lovell, Zarija Lukić, Yao-Yuan Mao, Sidney Mau, Andrea Mitridate, Philip Mocz, Julian B. Muñoz, Ethan O. Nadler, Annika H. G. Peter, Adrian Price-Whelan, Andrew Robertson, Nashwan Sabti, Neelima Sehgal, Nora Shipp, Joshua D. Simon, Rajeev Singh, Ken Van Tilburg, Risa H. Wechsler, Axel Widmark, and Hai-Bo Yu.  Snowmass2021 Cosmic Frontier White Paper: Dark Matter Physics from Halo Measurements.  arXiv e-prints, page arXiv:2203.07354, March 2022.
  * [141] Sukanya Chakrabarti, Alex Drlica-Wagner, Ting S. Li, Neelima Sehgal, Joshua D. Simon, Simon Birrer, Duncan A. Brown, Rebecca Bernstein, Alberto D. Bolatto, Philip Chang, Kyle Dawson, Paul Demorest, Daniel Grin, David L. Kaplan, Joseph Lazio, Jennifer Marshall, Eric J. Murphy, Scott Ransom, Brant E. Robertson, Rajeev Singh, Anže Slosar, Tommaso Treu, Yu-Dai Tsai, and Benjamin F. Williams.  Snowmass2021 Cosmic Frontier White Paper: Observational Facilities to Study Dark Matter.  arXiv e-prints, page arXiv:2203.06200, March 2022.


11institutetext: Electrical and Computer Engineering, Purdue University, USA
11email<EMAIL_ADDRESS>
22institutetext: Sensetime Research, China
22email<EMAIL_ADDRESS>
Corresponding author

# Adaptive Supervised PatchNCE Loss for Learning H&E-to-IHC Stain Translation
with Inconsistent Groundtruth Image Pairs

Fangda Li1* Zhiqiang Hu2 Wen Chen2 Avinash Kak1

###### Abstract

Immunohistochemical (IHC) staining highlights the molecular information
critical to diagnostics in tissue samples. However, compared to H&E staining,
IHC staining can be much more expensive in terms of both labor and the
laboratory equipment required. This motivates recent research that
demonstrates that the correlations between the morphological information
present in the H&E-stained slides and the molecular information in the IHC-
stained slides can be used for H&E-to-IHC stain translation. However, due to a
lack of pixel-perfect H&E-IHC groundtruth pairs, most existing methods have
resorted to relying on expert annotations. To remedy this situation, we
present a new loss function, Adaptive Supervised PatchNCE (ASP), to directly
deal with the input to target inconsistencies in a proposed H&E-to-IHC image-
to-image translation framework. The ASP loss is built upon a patch-based
contrastive learning criterion, named Supervised PatchNCE (SP), and augments
it further with weight scheduling to mitigate the negative impact of noisy
supervision. Lastly, we introduce the Multi-IHC Stain Translation (MIST)
dataset, which contains aligned H&E-IHC patches for 4 different IHC stains
critical to breast cancer diagnosis. In our experiment, we demonstrate that
our proposed method outperforms existing image-to-image translation methods
for stain translation to multiple IHC stains. All of our code and datasets are
available at https://github.com/lifangda01/AdaptiveSupervisedPatchNCE.

###### Keywords:

Generative Adversarial Network Contrastive Learning H&E-to-IHC Stain
Translation.

##  1 Introduction

Immunohistochemical (IHC) staining is a widely used technique in pathology for
visualizing abnormal cells that are often found in tumors. IHC chromogens
highlight the presence of certain antigens or proteins by staining their
corresponding antibodies. For instance, the HER2 (human epidermal growth
factor receptor 2) biomarker is associated with aggressive breast tumor
development and is essential in forming a precise treatment plan. Despite its
capability to provide highly valuable diagnostic information, the process of
IHC staining is very labor-intensive, time-consuming and requires specialized
histotechnologists and laboratory equipments [2]. Such factors hinder the
general availability of IHC staining in histopathological applications.

At the other end of the spectrum, H&E (Hematoxylin and Eosin) staining, as the
gold standard in histological staining, highlights the tissue structures and
cell morphology. In routine diagnostics, on account of its much lower cost, an
H&E-stained slide is prepared by pathologists in order to determine whether or
not to also apply the IHC stains for a more precise assessment of the disease.
Therefore, it is of great interest to have an algorithm that can automatically
translate an H&E-stained slide into one that could be considered to have been
stained with IHC while accurately predicting the target expression levels.

To that end, researchers have recently proposed to use GAN-based Image-to-
Image Translation (I2IT) algorithms for transforming H&E-stained slides into
IHC. Despite the progress, the outstanding challenge in training such I2IT
frameworks is the lack of aligned H&E-IHC image pairs, or in other words, the
inconsistencies in the H&E-IHC groundtruth pairs. To explain, since re-
staining a slice is physically infeasible, a matching pair of H&E-IHC slices
are taken from two depth-wise consecutive cuts of the same tissue then stained
and scanned separately. This inevitably prevents pixel-perfect image
correspondences due to the slice-to-slice changes in cell morphology,
staining-induced degradation (e.g. tissue-tearing), imaging artifacts that may
vary among slices (e.g. camera out-of-focus) and multi-slice registration
errors. An example pair of patches is shown in Fig. 1 and another pair with
significant inconsistencies is shown in Fig. 2(a)(c). In the latter, comparing
the groundtruth IHC image to the input H&E image, one can clearly see the
inconsistencies – nearly the entire left half of the tissue present in the H&E
image is missing.

As a result, recent advances in H&E-to-IHC I2IT have mostly avoided using the
inconsistent GT pairs and instead have imposed the cycle-consistency
constraint [6, 8, 13]. Moreover, existing approaches have also exploited using
expert annotations such as per-cell labels [9], semantic masks [8] and patch-
level labels [8, 13]. As for the prior works that directly utilize the H&E-IHC
pairs for supervision, a variant of Pix2Pix [4] that uses a Gaussian Pyramid
based reconstruction loss to accommodate the noisy GT is proposed in [7].
However, the robustness of such approaches that punish absolute errors in the
generated image to dealing with GT inconsistencies remains unclear.

In this paper, we argue that the IHC slides, despite the disparities vis-a-vis
their H&E counterparts, can still serve as useful targets for stain
translation. The work we present in this paper is based on the important
realization that even when pairs of consecutive tissue slices do not yield
images that are pixel-perfect aligned, it is highly likely that the
corresponding patches in the two stains share the same diagnostic label. For
example, if the levels of expression in a region of the HER2 slide are high,
the corresponding region in the H&E slide is highly likely to contain a high
density of cancerous cells. Therefore, we set our goal to meaningfully
leverage such correlations to benefit the H&E-to-IHC I2IT while being
resilient to any inconsistencies.

Toward this goal, we propose a supervised patchwise contrastive loss named the
Adaptive Supervised PatchNCE (ASP) loss. Our formulation of this loss was
inspired by the recent research findings that contrastive loss benefits model
robustness under label noise [12, 3]. Furthermore, based on the observation
that any dissimilarity between the patch embeddings at corresponding locations
in the generated and groundtruth IHC images is indicative to the level of
inconsistency of the GT at that location, we employ an adaptive weighting
scheme in ASP. By down-weighting the contrastive loss at locations with low
similarities, i.e. high inconsistencies, our proposed ASP loss helps the
network learn more robustly.

Lastly, to support further research in virtual IHC-restaining, we present the
Multi-IHC Stain Translation (MIST) as a new public dataset. The MIST dataset
contains 4k+ training and 1k testing aligned H&E-IHC patches for each of the
following IHC stains that are critical for breast cancer diagnostics: HER2,
Ki67, ER (Estrogen Receptor) and PR (Progesterone Receptor). We evaluated
existing I2IT methods and ours for multiple IHC stains and demonstrate the
superior performance achieved by our method both qualitatively and
quantitatively.

##  2 Method Description

###  2.1 The Supervised PatchNCE (SP) Loss

Figure 1:  Illustration of the PatchNCE loss from [11] and the Supervised
PatchNCE (SP) loss. The patch embeddings $\boldsymbol{z}$ are extracted by a
shared network $F$.

Before getting to our ASP loss, we need to first introduce the SP loss as a
robust means to learning from inconsistent GT image pairs. The SP loss was
inspired by the findings in recent literature that demonstrate the positive
effect of contrastive learning on boosting model robustness against label
noise [12, 3, 14]. It takes the same form as the PatchNCE loss as introduced
in [11], except that it is applied on the generated-GT image pair (instead of
the input-generated pair).

The goal of the PatchNCE loss is to ensure the content is consistent across
translation by maximizing the mutual information between the input and the
corresponding output. It does so by minimizing a patch-based InfoNCE loss
[10], which encourages the network to associate the corresponding patches with
each other in the learned embedding space, while disassociating them from the
noncorresponding ones. Mathematically, the InfoNCE loss takes the form:

$\mathcal{L}_{\text{InfoNCE}}(\boldsymbol{v},\boldsymbol{v}^{+},\boldsymbol{v}^{-})=-\log\left[\frac{\exp{(\boldsymbol{v}\cdot\boldsymbol{v}^{+}/\tau)}}{\exp{(\boldsymbol{v}\cdot\boldsymbol{v}^{+}/\tau)}+\sum_{n=1}^{N}\exp{(\boldsymbol{v}\cdot\boldsymbol{v}_{n}^{-}/\tau)}}\right],$
(1)

where $\boldsymbol{v}$, $\boldsymbol{v}^{+}$ and $\boldsymbol{v}^{-}$ are the
embeddings of the anchor, positive and negative samples, respectively. With
InfoNCE, the PatchNCE loss is set up as follows: given the anchor embedding
$\boldsymbol{\hat{z}}_{Y}$ of a patch in the output image, the positive
$\boldsymbol{z}_{X}$ is the embedding of the corresponding patch from the
input image, while the negatives $\boldsymbol{\widetilde{z}}_{X}$ are
embeddings of the non-corresponding ones, i.e.
$\mathcal{L}_{\text{PatchNCE}}=\mathcal{L}_{\text{InfoNCE}}(\boldsymbol{\hat{z}}_{Y},\boldsymbol{z}_{X},\boldsymbol{\widetilde{z}}_{X})$.

As for the SP loss, given the embedding of an output patch
$\boldsymbol{\hat{z}}_{Y}$ as anchor, we now designate the embedding of the
corresponding patch in the groundtruth image $\boldsymbol{z}_{Y}$ as the
positive and the embeddings of the non-corresponding ones
$\boldsymbol{\widetilde{z}}_{Y}$ as the negatives. We then use the same
InfoNCE-based contrastive learning objective, i.e.
$\mathcal{L}_{\text{SP}}=\mathcal{L}_{\text{InfoNCE}}(\boldsymbol{\hat{z}}_{Y},\boldsymbol{z}_{Y},\boldsymbol{\widetilde{z}}_{Y})$.
A depiction of both the PatchNCE loss and the SP loss is given in Fig. 1. It
is worthy to note that, although a similar patchwise contrastive loss has been
proposed in [1] for supervised I2IT, it is one of our contributions in this
paper to explicitly exploit the robustness of this contrastive loss in the
context of H&E-to-IHC translation where the GT pairs can be highly
inconsistent for reasons mentioned previously. We think that the key factor
behind the robustness of $\mathcal{L}_{\text{SP}}$ towards inconsistent GT
compared to, say, the MSE loss, is its relativeness. Instead of using an
absolute loss term that may not work well on inconsistent groundtruth pairs,
$\mathcal{L}_{\text{SP}}$ punishes dissimilarities between the anchor and the
positive in a learned latent space, relative to those between the anchor and
the negatives.

###  2.2 The Adaptive Supervised PatchNCE (ASP) Loss

Figure 2:  (a) Input H&E image $\boldsymbol{x}$, (b) generated IHC image
$\boldsymbol{\hat{y}}$, (c) groundtruth IHC image $\boldsymbol{y}$, and (d)
heat map of the anchor-positive cosine similarities produced by a trained
network at corresponding locations:
$C_{s}=\boldsymbol{z}_{\boldsymbol{\hat{y}}}^{s}\cdot\boldsymbol{z}_{\boldsymbol{y}}^{s}$,
where $s$ is index of the spatial location.

To learn selectively from more consistent groundtruth locations, we further
propose to augment the Supervised PatchNCE loss in an adaptive manner. The key
idea here is to automatically recognize patch locations that are inconsistent
and adapt the SP loss so that the severely inconsistent patch locations will
have lesser effects on training. To measure the consistency at a given patch
location, we use the cosine similarity between the embeddings of the generated
IHC patch and the corresponding GT patch. In Fig. 2, we show an example pair
of generated vs GT IHC images that contain significant inconsistencies and
their anchor-positive similarity heat map. For pairs of embeddings produced by
a trained network, a high similarity value indicates good correspondence
between the groundtruth patches while a low similarity value indicates
inconsistencies.

Directly motivated by this observation, we first propose a weighting scheme
for the SP loss. More specifically, we assign lower weights to patch locations
that have low anchor-positive similarity values to alleviate the negative
impacts the inconsistent targets may have on training. At training time $t$,
the weight is a function of the anchor-positive cosine similarity. Examples of
the weight function $h(\cdot)$ are shown in Fig. 3(b). The weight functions
are monotonic increasing so that the more confident patch locations are always
treated with more importance.

Figure 3:  (a) Histograms of the anchor-positive similarity values at
different epochs; (b) The weight $h(\cdot)$ as a function of $C_{s}$; (c) The
scheduling function $g(\cdot)$.

In order to make the weighting scheme work in practice, we must also account
for the phase of training. The intuition is that, during the initial phase of
training, the network is not going to be able to discriminate between
consistent patch locations from those that are inconsistent. Additionally, as
shown in Fig. 3(a), the histograms of the anchor-positive similarity evolve
rather slowly over the training epochs. Therefore, it would not make sense to
reinforce the weighting function in the beginning of the training as much as
near the end of the training.

Figure 4:  Left to right: (a) Input H&E image; (b) Groundtruth IHC image; (c)
Generated image without $\mathcal{L}_{\text{SP}}$; (d) With
$\mathcal{L}_{\text{SP}}$; (e) With
$\mathcal{L}_{\text{ASP}}^{\text{(lambda,linear)}}$.

To that end, we further augment the weight so that it is also a function of
the training iterations. Such scheduling of the weights is done so that in the
beginning of the training, the weights are uniform in order not to wrongly
bias the network when the embeddings are still indiscriminative. And as
training progresses, the selective weighting scheme is gradually enforced so
that the inconsistent patch locations are treated with reduced weights. We
call this gradual process of shifting the learning focus weight scheduling.
Let $t$ denote the current iteration and $T$ the total number of training
iterations. Then weight scheduling is achieved by using a scheduling function
$g(\frac{t}{T})$. Various options of $g(\cdot)$ are shown in Fig. 3(c).
Subsequently, combining the weighting function with the scheduling function,
we can write the following formula for the final weight:

$w_{t}(\boldsymbol{v},\boldsymbol{v}^{+})=\left(1-g\left(\frac{t}{T}\right)\right)\times
1.0+g\left(\frac{t}{T}\right)\times h(\boldsymbol{v}\cdot\boldsymbol{v}^{+}).$
(2)

We refer to the new augmented Supervised PatchNCE loss as the Adaptive
Supervised PatchNCE (ASP) loss, which can be expressed as:

$\mathcal{L}_{\text{ASP}}(G,H,X,Y,t)=\mathbb{E}_{(\boldsymbol{x},\boldsymbol{y})\sim(X,Y)}\sum_{l=1}^{L}\sum_{s=1}^{S_{l}}\frac{w_{t}(\boldsymbol{\hat{z}}^{l,s}_{Y},\boldsymbol{z}^{l,s}_{Y})}{W_{t}^{l}}\cdot\mathcal{L}_{\text{InfoNCE}}(\boldsymbol{\hat{z}}^{l,s}_{Y},\boldsymbol{z}^{l,s}_{Y},\boldsymbol{\widetilde{z}}^{l,s}_{Y}),$
(3)

where $W_{t}^{l}=\sum_{s}w_{t}^{l,s}$ is a normalization factor that maintains
the total magnitude of the loss after applying the weights. Finally, the
overall learning objective for our generator is as follows:

$\displaystyle\mathcal{L}_{\text{adv}}+\lambda_{\text{PatchNCE}}\mathcal{L}_{\text{PatchNCE}}+\lambda_{\text{ASP}}\mathcal{L}_{\text{ASP}}+\lambda_{\text{GP}}\mathcal{L}_{\text{GP}},$
(4)

where $\mathcal{L}_{\text{GP}}$ is the Gaussian Pyramid based reconstruction
loss from [7].

##  3 Experiments

Table 1:  Quantitative evaluations on all three datasets using both paired and unpaired metrics. The best values are highlighted. Ours is with $\mathcal{L}_{\text{ASP}}^{\text{(lambda,linear)}}$.  Dataset | Method | SSIM$\uparrow$ |  $\text{PHV}_{T=0.01}$$\downarrow$ | FID$\downarrow$ | KID$\downarrow$
---|---|---|---|---|---
layer1 | layer2 | layer3 | layer4 | avg.
$\text{BCI}_{\text{HER2}}$ | CycleGAN | 0.4424 | 0.4264 | 0.3924 | 0.2610 | 0.7262 | 0.4515 | 63.5 | 10.7
CUT+$\mathcal{L}_{\text{GP}}$ | 0.4802 | 0.4263 | 0.3784 | 0.2364 | 0.7328 | 0.4435 | 65.0 | 10.9
Pix2Pix | 0.4372 | 0.5121 | 0.4531 | 0.2953 | 0.7484 | 0.5022 | 100.0 | 44.6
PyramidP2P | 0.5001 | 0.4531 | 0.3826 | 0.2618 | 0.7293 | 0.4567 | 113.6 | 79.4
Ours ($\mathcal{L}_{\text{ASP}}$) | 0.5032 | 0.4308 | 0.3670 | 0.2235 | 0.7210 | 0.4356 | 65.1 | 9.9
$\text{MIST}_{\text{HER2}}$ | CycleGAN | 0.1914 | 0.5633 | 0.6346 | 0.4695 | 0.8871 | 0.6386 | 240.3 | 311.1
CUT+$\mathcal{L}_{\text{GP}}$ | 0.1810 | 0.5321 | 0.4826 | 0.3060 | 0.8323 | 0.5383 | 66.8 | 19.0
Pix2Pix | 0.1559 | 0.5516 | 0.5070 | 0.3253 | 0.8511 | 0.5588 | 137.3 | 82.9
PyramidP2P | 0.2078 | 0.4787 | 0.4524 | 0.3313 | 0.8423 | 0.5262 | 104.0 | 61.8
Ours ($\mathcal{L}_{\text{ASP}}$) | 0.2004 | 0.4534 | 0.4150 | 0.2665 | 0.8174 | 0.4881 | 51.4 | 12.4
$\text{MIST}_{\text{ER}}$ | CycleGAN | 0.1982 | 0.5175 | 0.5092 | 0.3710 | 0.8672 | 0.5662 | 125.7 | 95.1
CUT+$\mathcal{L}_{\text{GP}}$ | 0.2217 | 0.4531 | 0.4079 | 0.2725 | 0.8194 | 0.4882 | 43.7 | 8.7
Pix2Pix | 0.1500 | 0.5818 | 0.5282 | 0.3700 | 0.8620 | 0.5855 | 128.1 | 79.0
PyramidP2P | 0.2172 | 0.4767 | 0.4538 | 0.3757 | 0.8567 | 0.5407 | 107.4 | 84.2
Ours ($\mathcal{L}_{\text{ASP}}$) | 0.2061 | 0.4336 | 0.4007 | 0.2649 | 0.8205 | 0.4799 | 41.4 | 5.8
Note that KID values multiplied by 1000 are shown. CUT is from [11].
Table 2:  Ablation studies comparing different adaptive strategies for the SP loss. The best and second best values are highlighted. Note that $\mathcal{L}_{\text{SP}}$ is $\mathcal{L}_{\text{ASP}}^{\text{(zero,uniform)}}$.  Dataset | Method | SSIM$\uparrow$ |  $\text{PHV}_{T=0.01}$$\downarrow$ | FID$\downarrow$ | KID$\downarrow$
---|---|---|---|---|---
layer1 | layer2 | layer3 | layer4 | avg.
$\text{BCI}_{\text{HER2}}$ | $\mathcal{L}_{\text{SP}}$ | 0.5094 | 0.4413 | 0.3771 | 0.2273 | 0.7202 | 0.4415 | 62.7 | 11.4
$\mathcal{L}_{\text{ASP}}^{\text{(lambda,top)}}$ | 0.5206 | 0.4411 | 0.3813 | 0.2282 | 0.7244 | 0.4438 | 62.1 | 12.8
$\mathcal{L}_{\text{ASP}}^{\text{(lambda,lin.)}}$ | 0.5032 | 0.4308 | 0.3670 | 0.2235 | 0.7210 | 0.4356 | 65.1 | 9.9
$\mathcal{L}_{\text{ASP}}^{\text{(sigmoid,top)}}$ | 0.5236 | 0.4503 | 0.3877 | 0.2331 | 0.7292 | 0.4501 | 65.9 | 12.3
$\mathcal{L}_{\text{ASP}}^{\text{(linear,top)}}$ | 0.4890 | 0.4327 | 0.3828 | 0.2305 | 0.7318 | 0.4445 | 61.9 | 9.8
$\text{MIST}_{\text{HER2}}$ | $\mathcal{L}_{\text{SP}}$ | 0.2159 | 0.4712 | 0.4243 | 0.2611 | 0.8129 | 0.4924 | 55.6 | 20.8
$\mathcal{L}_{\text{ASP}}^{\text{(lambda,top)}}$ | 0.2035 | 0.4451 | 0.4068 | 0.2554 | 0.8117 | 0.4798 | 51.2 | 16.7
$\mathcal{L}_{\text{ASP}}^{\text{(lambda,lin.)}}$ | 0.2004 | 0.4534 | 0.4150 | 0.2665 | 0.8174 | 0.4881 | 51.5 | 12.4
$\mathcal{L}_{\text{ASP}}^{\text{(sigmoid,top)}}$ | 0.2086 | 0.4655 | 0.4191 | 0.2581 | 0.8138 | 0.4891 | 45.2 | 11.5
$\mathcal{L}_{\text{ASP}}^{\text{(linear,top)}}$ | 0.1809 | 0.4766 | 0.4262 | 0.2667 | 0.8178 | 0.4968 | 68.8 | 28.9
$\text{MIST}_{\text{ER}}$ | $\mathcal{L}_{\text{SP}}$ | 0.2236 | 0.4517 | 0.4117 | 0.2714 | 0.8208 | 0.4889 | 46.4 | 12.5
$\mathcal{L}_{\text{ASP}}^{\text{(lambda,top)}}$ | 0.2096 | 0.4388 | 0.4052 | 0.2676 | 0.8215 | 0.4833 | 42.4 | 8.1
$\mathcal{L}_{\text{ASP}}^{\text{(lambda,lin.)}}$ | 0.2061 | 0.4336 | 0.4007 | 0.2649 | 0.8205 | 0.4799 | 41.4 | 5.8
$\mathcal{L}_{\text{ASP}}^{\text{(sigmoid,top)}}$ | 0.2192 | 0.4376 | 0.3965 | 0.2684 | 0.8215 | 0.4810 | 43.6 | 8.1
$\mathcal{L}_{\text{ASP}}^{\text{(linear,top)}}$ | 0.1981 | 0.4581 | 0.4072 | 0.2706 | 0.8217 | 0.4894 | 46.9 | 10.7

Datasets. The following datasets are used in our experiments: the Breast
Cancer Immunohistochemical (BCI) challenge dataset [7] and our own MIST
dataset that is now in the public domain. The publicly available portion of
BCI contains 3396 H&E-HER2 patches for training and 500 of the same for
testing. Note that we have additionally normalized the brightness levels of
all BCI images to the same level. Due to the page limit, from the MIST
dataset, here we only present detailed results on HER2 and ER. For
$\text{MIST}_{\text{HER2}}$, we extracted 4642 paired patches for training and
1000 for testing from 64 WSIs. And for $\text{MIST}_{\text{ER}}$, we extracted
4153 patches for training, and 1000 for testing from 56 WSIs. All patches are
of size $1024\times 1024$ and non-overlapping. Additional results on
$\text{MIST}_{\text{Ki67}}$ and $\text{MIST}_{\text{PR}}$ are provided in the
Supplementary Materials.

Implementation Details. For all of our models, we used ResNet-6Blocks as the
generator and a 5-layer PatchGAN as the discriminator. We trained our networks
with random $512\times 512$ crops and a batch size of one. The Adam optimizer
[5] was used with a linear decay scheduler (as shown in Fig. 3(c)) and an
initial learning rate of $2\times 10^{-4}$. The hyperparameters in Eq. 4 are
set as: $\lambda_{\text{PatchNCE}}=10.0$, $\lambda_{\text{ASP}}=10.0$ and
$\lambda_{\text{GP}}=10.0$.

Evaluation Metrics. We compare the methods using both paired and unpaired
evaluation metrics. To compare a pair of images, generated and groundtruth, we
use the standard SSIM (Structural Similarity Index Measure) and PHV
(Perceptual Hash Value) as described in [8]. As for the unpaired metrics, we
use the FID (Fréchet Inception Distance) and the KID (Kernel Inception
Distance).

Qualitative Evaluations. In Fig. 5, we compare visually the generated IHC
images by our framework. It can be observed that by using either
$\mathcal{L}_{\text{SP}}$ or $\mathcal{L}_{\text{ASP}}$, the pathological
representations in the generated images are significantly more accurate. And
by using $\mathcal{L}_{\text{ASP}}$, such representations appear to be more
consistent.

Quantitative Evaluations. The full results comparing existing I2IT methods to
ours are tabulated in Tab. 1. Overall, it can be observed that the proposed
framework with the ASP loss consistently outperforms existing methods across
all three datasets. Subsequently, in Tab. 2, we further provide results using
different weighting and scheduling functions in our proposed
$\mathcal{L}_{\text{ASP}}$. With $\mathcal{L}_{\text{SP}}$ already being a
strong baseline, using different adaptive strategies can provide further gains
in performance. It is also worth noting that if reinforced prematurely,
adaptive weighting can lead to inferior convergence, e.g.
$\mathcal{L}_{\text{ASP}}^{\text{(linear,top)}}$.

##  4 Conclusion

In this paper, we have proposed the Adaptive Supervised PatchNCE (ASP) loss
for learning H&E-to-IHC stain translation with inconsistent GT image pairs.
The adaptive logic in ASP is based on the intuition that inconsistent patch
locations should contribute less to learning. We demonstrated that our
proposed framework is able to achieve significant improvements both
qualitatively and quantitatively over the existing approaches for translations
to multiple IHC stains. Finally, we are making public our Multi-IHC Stain
Translation dataset with the hope to assist further research towards accurate
H&E-to-IHC stain translation.

## References

  * [1] Andonian, A., Park, T., Russell, B., Isola, P., Zhu, J.Y., Zhang, R.: Contrastive feature loss for image prediction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1934–1943 (2021)
  * [2] Anglade, F., Milner Jr, D.A., Brock, J.E.: Can pathology diagnostic services for cancer be stratified and serve global health? Cancer 126, 2431–2438 (2020)
  * [3] Ghosh, A., Lan, A.: Contrastive learning improves model robustness under label noise. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2703–2708 (2021)
  * [4] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1125–1134 (2017)
  * [5] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [6] Lin, Y., Zeng, B., Wang, Y., Chen, Y., Fang, Z., Zhang, J., Ji, X., Wang, H., Zhang, Y.: Unpaired multi-domain stain transfer for kidney histopathological images. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 1630–1637 (2022)
  * [7] Liu, S., Zhu, C., Xu, F., Jia, X., Shi, Z., Jin, M.: Bci: Breast cancer immunohistochemical image generation through pyramid pix2pix. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1815–1824 (2022)
  * [8] Liu, S., Zhang, B., Liu, Y., Han, A., Shi, H., Guan, T., He, Y.: Unpaired stain transfer using pathology-consistent constrained generative adversarial networks. IEEE Transactions on Medical Imaging 40(8), 1977–1989 (2021)
  * [9] Liu, Y., Li, X., Zheng, A., Zhu, X., Liu, S., Hu, M., Luo, Q., Liao, H., Liu, M., He, Y., et al.: Predict ki-67 positive cells in h&e-stained images using deep learning independently from ihc-stained images. Frontiers in Molecular Biosciences 7, 183 (2020)
  * [10] Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)
  * [11] Park, T., Efros, A.A., Zhang, R., Zhu, J.Y.: Contrastive learning for unpaired image-to-image translation. In: European Conference on Computer Vision. pp. 319–345. Springer (2020)
  * [12] Xue, Y., Whitecross, K., Mirzasoleiman, B.: Investigating why contrastive learning benefits robustness against label noise. In: International Conference on Machine Learning. pp. 24851–24871. PMLR (2022)
  * [13] Zeng, B., Lin, Y., Wang, Y., Chen, Y., Dong, J., Li, X., Zhang, Y.: Semi-supervised pr virtual staining for breast histopathological images. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part II. pp. 232–241. Springer (2022)
  * [14] Zheltonozhskii, E., Baskin, C., Mendelson, A., Bronstein, A.M., Litany, O.: Contrast to divide: Self-supervised pre-training for learning with noisy labels. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 1657–1667 (2022)

##  5 Supplementary Materials

Table 3:  Additional quantitative evaluations on $\text{MIST}_{\text{Ki67}}$ and $\text{MIST}_{\text{PR}}$. Ours is with $\mathcal{L}_{\text{ASP}}^{\text{(lambda,linear)}}$.  Dataset | Method | SSIM$\uparrow$ |  $\text{PHV}_{T=0.01}$$\downarrow$ | FID$\downarrow$ | KID$\downarrow$
---|---|---|---|---|---
layer1 | layer2 | layer3 | layer4 | avg.
$\text{MIST}_{\text{Ki67}}$ | CycleGAN | 0.3875 | 0.8274 | 0.8275 | 0.6081 | 0.9038 | 0.7917 | 343.9 | 317.9
CUT+$\mathcal{L}_{\text{GP}}$ | 0.1909 | 0.5426 | 0.4739 | 0.3160 | 0.8415 | 0.5435 | 76.1 | 43.5
Pix2Pix | 0.1819 | 0.5468 | 0.4905 | 0.3415 | 0.8496 | 0.5571 | 147.0 | 142.4
PyramidP2P | 0.2286 | 0.4533 | 0.4222 | 0.3360 | 0.8363 | 0.5120 | 94.4 | 78.0
Ours ($\mathcal{L}_{\text{ASP}}$) | 0.2410 | 0.4472 | 0.4001 | 0.2701 | 0.8128 | 0.4826 | 51.0 | 19.1
$\text{MIST}_{\text{PR}}$ | CycleGAN | 0.2232 | 0.5334 | 0.5554 | 0.3867 | 0.8654 | 0.5852 | 96.1 | 96.6
CUT+$\mathcal{L}_{\text{GP}}$ | 0.2153 | 0.4656 | 0.4128 | 0.2724 | 0.8154 | 0.4916 | 54.6 | 20.1
Pix2Pix | 0.1617 | 0.6027 | 0.5569 | 0.4043 | 0.8601 | 0.6060 | 183.8 | 148.1
PyramidP2P | 0.2403 | 0.5078 | 0.4682 | 0.3509 | 0.8446 | 0.5429 | 98.8 | 59.5
Ours ($\mathcal{L}_{\text{ASP}}$) | 0.2159 | 0.4484 | 0.3898 | 0.2564 | 0.8080 | 0.4757 | 44.8 | 10.2
Note that KID values multiplied by 1000 are shown.
Table 4:  Additional ablation studies on $\text{MIST}_{\text{Ki67}}$ and $\text{MIST}_{\text{PR}}$.  Dataset | Method | SSIM$\uparrow$ |  $\text{PHV}_{T=0.01}$$\downarrow$ | FID$\downarrow$ | KID$\downarrow$
---|---|---|---|---|---
layer1 | layer2 | layer3 | layer4 | avg.
$\text{MIST}_{\text{Ki67}}$ | $\mathcal{L}_{\text{SP}}$ | 0.2089 | 0.4631 | 0.4014 | 0.2686 | 0.8200 | 0.4883 | 41.3 | 8.3
$\mathcal{L}_{\text{ASP}}^{\text{(lambda,top)}}$ | 0.2078 | 0.4507 | 0.3938 | 0.2642 | 0.8103 | 0.4798 | 50.4 | 21.5
$\mathcal{L}_{\text{ASP}}^{\text{(lambda,linear)}}$ | 0.2410 | 0.4472 | 0.4001 | 0.2701 | 0.8128 | 0.4826 | 51.0 | 19.1
$\mathcal{L}_{\text{ASP}}^{\text{(sigmoid,top)}}$ | 0.2116 | 0.4403 | 0.3940 | 0.2634 | 0.8132 | 0.4777 | 41.4 | 11.0
$\mathcal{L}_{\text{ASP}}^{\text{(linear,top)}}$ | 0.1819 | 0.4483 | 0.4067 | 0.2762 | 0.8154 | 0.4867 | 49.5 | 21.4
$\text{MIST}_{\text{PR}}$ | $\mathcal{L}_{\text{SP}}$ | 0.2243 | 0.4511 | 0.3996 | 0.2682 | 0.8163 | 0.4838 | 50.7 | 11.2
$\mathcal{L}_{\text{ASP}}^{\text{(lambda,top)}}$ | 0.2169 | 0.4495 | 0.3940 | 0.2604 | 0.8130 | 0.4792 | 43.6 | 11.4
$\mathcal{L}_{\text{ASP}}^{\text{(lambda,linear)}}$ | 0.2159 | 0.4484 | 0.3898 | 0.2564 | 0.8080 | 0.4757 | 44.8 | 10.2
$\mathcal{L}_{\text{ASP}}^{\text{(sigmoid,top)}}$ | 0.2056 | 0.4725 | 0.4105 | 0.2703 | 0.8150 | 0.4921 | 48.6 | 11.6
$\mathcal{L}_{\text{ASP}}^{\text{(linear,top)}}$ | 0.2164 | 0.4569 | 0.4017 | 0.2655 | 0.8130 | 0.4843 | 45.5 | 10.9
Figure 5:  Left to right: (a) Input H&E image; (b) Groundtruth IHC image; (c)
Generated image without $\mathcal{L}_{\text{SP}}$; (d) With
$\mathcal{L}_{\text{SP}}$; (e) With
$\mathcal{L}_{\text{ASP}}^{\text{(lambda,linear)}}$.