# Programmable access to microresonator solitons with modulational sideband
heating

Huamin Zheng  College of Electronics and Information Engineering, Shenzhen
University, Shenzhen 518000, China  International Quantum Academy, Shenzhen
518048, China  Wei Sun<EMAIL_ADDRESS>International Quantum Academy,
Shenzhen 518048, China  Xingxing Ding  Key Laboratory of Specialty Fiber
Optics and Optical Access Networks, Shanghai University, Shanghai 200444,
China  Haoran Wen  International Quantum Academy, Shenzhen 518048, China
School of Science and Engineering, CUHK(SZ), Shenzhen 518100, China  Ruiyang
Chen  International Quantum Academy, Shenzhen 518048, China  Shenzhen
Institute for Quantum Science and Engineering, Southern University of Science
and Technology, Shenzhen 518055, China  Baoqi Shi  International Quantum
Academy, Shenzhen 518048, China  Department of Optics and Optical Engineering,
University of Science and Technology of China, Hefei 230026, China  Yi-Han Luo
International Quantum Academy, Shenzhen 518048, China  Shenzhen Institute for
Quantum Science and Engineering, Southern University of Science and
Technology, Shenzhen 518055, China  Jinbao Long  International Quantum
Academy, Shenzhen 518048, China  Chen Shen  International Quantum Academy,
Shenzhen 518048, China  Shan Meng  College of Electronics and Information
Engineering, Shenzhen University, Shenzhen 518000, China  Hairun Guo  Key
Laboratory of Specialty Fiber Optics and Optical Access Networks, Shanghai
University, Shanghai 200444, China  Junqiu Liu<EMAIL_ADDRESS>International
Quantum Academy, Shenzhen 518048, China  Hefei National Laboratory, University
of Science and Technology of China, Hefei 230088, China

Dissipative Kerr solitons formed in high-$Q$ optical microresonators provide a
route to miniaturized optical frequency combs that can revolutionize precision
measurements, spectroscopy, sensing, and communication. In the last decade, a
myriad of integrated material platforms have been extensively studied and
developed to create photonic-chip-based soliton combs. However, the photo-
thermal effect in integrated optical microresonators has been a major issue
preventing simple and reliable soliton generation. Several sophisticated
techniques to circumvent the photo-thermal effect have been developed. In
addition, instead of the single-soliton state, emerging applications in
microwave photonics and frequency metrology prefer multi-soliton states. Here
we demonstrate an approach to manage the photo-thermal effect and facilitate
soliton generation. The approach is based on a single phase-modulated pump,
where the generated blue-detuned sideband synergizes with the carrier and
thermally stabilizes the microresonator. We apply this technique and
demonstrate deterministic soliton generation of 19.97 GHz repetition rate in
an integrated silicon nitride microresonator. Furthermore, we develop a
program to automatically address to target $N-$soliton state, in addition to
the single-soliton state, with near 100% success rate and as short as 10 s
time consumption. Our method is valuable for soliton generation in essentially
any platforms even with strong photo-thermal effect, and can promote wider
applications of soliton frequency comb systems for microwave photonics,
telecommunication and frequency metrology.

Dissipative Kerr solitons formed in high-$Q$ optical microresonators Herr  _et
al._ (2013); Kippenberg  _et al._ (2018) constitute miniaturized optical
frequency combs with broad bandwidths and repetition rates in the microwave to
millimeter-wave domain. Commonly referred to as “soliton microcombs”, they
have been already used in many system-level information and metrology
applications, such as coherent telecommunication Marin-Palomo  _et al._
(2017); Corcoran  _et al._ (2020); Mazur  _et al._ (2021), ultrafast ranging
Trocha  _et al._ (2018); Suh and Vahala (2018), astronomical spectrometer
calibration Obrzud  _et al._ (2019); Suh  _et al._ (2019), dual-comb
spectroscopy Dutt  _et al._ ; Yang  _et al._ (2019), low-noise microwave
generation Liang  _et al._ (2015); Liu  _et al._ (2020a), photonic neural
networks Feldmann  _et al._ (2021); Xu  _et al._ (2021), datacenter circuit
switch Raja  _et al._ (2021), microwave photonics Wu  _et al._ (2018); Hu  _et
al._ (2020), frequency synthesizers Spencer  _et al._ (2018), and optical
atomic clocks Newman  _et al._ (2019). Critical to the rapid progress of
soliton microcomb technology is the development and continuous optimization of
various photonic integrated platforms including silica Yi  _et al._ (2015);
Yang  _et al._ (2018), silicon nitride (Si3N4) Brasch  _et al._ (2016a); Joshi
_et al._ (2016), high-index doped silica (Hydex) Bao  _et al._ (2019); Wang
_et al._ (2021), aluminium nitride (AlN) Liu  _et al._ (2021); Weng  _et al._
(2021), lithium niobate (LiNbO3) He  _et al._ (2019); Gong  _et al._ (2020),
tantala pentaoxide (Ta2O5) Jung  _et al._ (2021), silicon carbide (SiC) Guidry
_et al._ (2022); Wang  _et al._ (2022), chalcogenide Xia  _et al._ (2022),
aluminium gallium arsenide (AlGaAs) Pu  _et al._ (2016); Moille  _et al._
(2020) and gallium phosphide (GaP) Nardi  _et al._ (2023). Besides, the
introduction and successful implementation of hybrid and heterogeneous
integration Stern  _et al._ (2018); Liu  _et al._ (2020b); Voloshin  _et al._
(2021); Xiang  _et al._ (2021); Churaev  _et al._ (2023); Liu  _et al._
(2022); de Beeck  _et al._ (2020) further enable complex control schemes,
extra nonlinearity and efficient amplification for integrated soliton
microcombs.

Despite these advances, one issue that currently prevents a wider deployment
of soliton microcombs is to deterministically access and control the soliton
state. This issue is caused by the photo-thermal effect, arising from light
absorption and thermal-optic effect in optical microresonators Li  _et al._
(2017); Gao  _et al._ (2022), particularly for those built on integrated
platforms. When the CW pump’s frequency is scanned through a microresonator
resonance, from the blue-detuned side to the red-detuned side, a self-
organized pulse waveform (i.e. a soliton state), is formed Herr  _et al._
(2013). However, the photo-thermal effect leads to serious thermal instability
that often annihilates the soliton state immediately. Therefore, sophisticated
techniques to manage this effect have been developed, such as power kicking
Brasch  _et al._ (2016b); Yi  _et al._ (2016), single-sideband suppressed-
carrier frequency shifters Stone  _et al._ (2018), dual-laser pump Zhou  _et
al._ (2019); Zhang  _et al._ (2019); Drake  _et al._ (2020), pump modulation
Jang  _et al._ (2015); Wildi  _et al._ (2019); Nishimoto  _et al._ (2022);
Cole  _et al._ (2018); Miao  _et al._ (2022), pulse pumping Obrzud  _et al._
(2017), or laser self-cooling Lei  _et al._ (2022); Grassani  _et al._ (2022).
Besides, cryogenic operation can be helpful Moille  _et al._ (2020). In
addition, instead of the single-soliton state, efforts have been also made on
employing multi-soliton states for reconfigurable photonic microwave filters
Hu  _et al._ (2020) and synthesis of terahertz frequency Lu  _et al._ (2021).
For frequency metrology applications, the local comb line power enhancement in
multi-soliton state further facilitates the realization of self-referencing
Brasch  _et al._ (2017).

Figure 1:  Principle and numerical simulation of soliton generation in optical
microresonators assisted with sideband heating. (a). Comparison of soliton
step length without (top row) and with (bottom row) sideband heating. Left
column shows the photos of the same Si3N4 microresonator of 19.97 GHz FSR. Red
and blue arc arrows represent the CW carrier pump and the blue-detuned
sideband. Middle column show the intra-cavity power dynamics (solid black
line) and the “cold” resonance (dashed black line). Right column shows the
intra-cavity temperature dynamics. If the scanning pump (red line) stops on
the soliton step, a soliton state is formed (red spot). Such a process can be
facilitated by the blue-detuned sideband (blue line), which locks the “hot”
resonance, avoid its shift, and extend the soliton step. (b). Numerical
simulation of the intra-cavity power (blue) and temperature(red), without
(left) and with (right) the blue-detuned sideband. The simulation reveals
soliton step extension due to the blue-detuned sideband.

Here we demonstrate an approach to manage the photo-thermal effect, and to
automatically and deterministically access to a soliton state. This method,
employing single-sideband heating, overcomes the photo-thermal effect in
integrated microresonators. We apply this method to demonstrate that,
$N-$soliton states of 19.97 GHz repetition rate can be deterministically
generated for any assigned value of $N\leq 5$.

Soliton step extension via sideband heating. The principle to excite a soliton
state in a high-$Q$ optical microresonator with a single continuous-wave (CW)
pump laser is shown in Fig. 1(a) top. As the pump laser scans through a
resonance from its blue-detuned to red-detuned side, the resonance is
distorted from a Lorentzian to a triangular profile Carmon  _et al._ (2004).
This change is due to the increasing intra-cavity power, which, via the the
photo-thermal effect, levitates the microresonator’s temperature and extends
its total optical length. Therefore, the “hot” resonance shifts following the
direction of laser scanning. In this case, solitons can form at the red-
detuned side of the effective resonance. A signature indicating soliton states
is a series of step features (“soliton step”) in the intra-cavity power Guo
_et al._ (2017) as shown in Fig. 1(a). However, once a soliton state is
reached, the intra-cavity power, as well as the microresonator temperature,
drops suddenly. This leads to a sudden blue-shift of the resonance, which in
return increases the effective soliton-pump detuning. Therefore, this
transition often compromises the soliton stability, causes extremely short or
even negligible soliton steps, and often annihilates solitons immediately.

Figure 2:  Experimental demonstration of soliton step extension using sideband
heating. (a). Experimental setup. ECDL, external-cavity diode laser. AFG,
arbitrary function generator. PM, phase modulator. SG, signal generator. EDFA,
erbium-doped fiber amplifier. BPF, bandpass filter. MR, Si3N4 microresonator.
FBG, fiber Bragg grating. PD, photodiode. OSA, optical spectrum analyzer. OSC,
oscilloscope. (b). Experimentally detected light generation from the
microresonator, showing soliton step extension with the sideband. $\nu_{0}$,
pump laser frequency. $\nu_{+}$, blue-detuend sideband frequency. $\nu_{-}$,
red-detuend sideband frequency, which is idle in soliton generation.
$\nu_{\pm}$ are generated by phase-modulating the pump laser with 396 MHz RF
frequency and 11 dBm RF power on phase modulator. The arrows mark that the
laser frequency is decreasing. The red-colored segments are soliton steps. In
the right panel, with the sideband, the small triangular peak is due to
$\nu_{-}$, the major triangular peak is due to $\nu_{0}$, and the soliton step
is extended due to $\nu_{+}$. (c). Statistical analysis of 150 measured
soliton step length values without (left) and with (right) the sideband
$\nu_{+}$. The mean soliton step length is increased from 5.2 MHz (left) to
42.7 MHz (right). (d). Left: With 11 dBm RF power, the dependence of soliton
step length on RF frequency. Right: The maximum value of step length for a
given RF power with varying RF frequency. The best combination found in our
experiment is 352 MHz RF frequency and 13 dBm RF power.  Figure 3:  Criteria
and program for automatic $N-$soliton state addressing. (a). Generated light
power dynamics labeled with different zones. The red curve shows that the
laser frequency $\nu_{0}$ is decreasing from $\nu_{s}$ to $\nu_{e}$. The blue
curves with different tones show the normalized power traces detected by the
PD and recorded by the OSC. The traces together show the process to access a
soliton state by translating $(\nu_{s},\nu_{e})$ with step $\Delta\nu$ as
shown in Panel (c). Zone 1/2/3 are marked, and only Zone 2 supports soliton
formation. (b). Using the backward-tuning process, the power trace measured by
the PD shows a stair feature, which contains many steps corresponding to
soliton states with different soliton number $N$. The arrow marks that the
laser frequency is increasing. (c). The backward-tuning soliton generation
process is repeated 20 times, and the average PD voltage $\overline{V}$ for
each step are fitted with the soliton number $N$, resulting in a linear
relation of $\overline{V}=a\times N+b$ ($a=0.0609,b=0.0265$). (d). The logic
codes and algorithm flowchart for computer-controlled automatic $N-$soliton
state generation. The blue-colored block is for soliton generation, and the
red-colored block is for $N-$soliton state addressing. The solid lines with
arrows represents successful addressing and the dashed line with arrows
represents failed addressing.

To circumvent such thermally induced instability, here we develop a blue-
detuned sideband heating scheme to balance the sudden cooling of the
microresonator during soliton formation. The principle is depicted in Fig.
1(a) bottom. With proper frequency and power differences between the pump
laser and its sideband, wherever the resonance moves, the sideband will drag
the resonance back. For example, when the resonance experiences a sudden blue-
shift during the transition from modulation instability to a soliton state,
more optical power of the sideband enters into the resonance, which levitates
the microresonator’s temperature to counteract the cooling. In such a way, the
microresonator’s intra-cavity power and temperature are locked, and the
soliton step is extensively extended, facilitating soliton formation.

A modified Lugiato-Lefever Equation (LLE) model to simulate soliton formation
is considered as below,

$\displaystyle\frac{\partial}{\partial t}A(t,\varphi)$
$\displaystyle=-\frac{\kappa}{2}A(t,\varphi)-i[\delta_{\omega}+g_{\omega}|A|^{2}+\mathcal{D}_{\omega}+T]A(t,\varphi)$
(1) $\displaystyle+\sqrt{\kappa_{\text{ex}}}A_{\text{in}},$

where $A(t,\phi)$ is the intra-cavity field amplitude,
$\kappa=\kappa_{0}+\kappa_{\text{ex}}$ is the total optical loss rate (with
$\kappa_{0}$ being the intrinsic loss rate and $\kappa_{\text{ex}}$ being the
external coupling rate with the input pump laser), $\delta_{\omega}$ is the
soliton detuning, $g_{\omega}$ is the nonlinear coefficient,
$\mathcal{D}_{\omega}$ is the microresonator dispersion, $T$ is the
microresonator temperature,
$A_{\text{in}}=\sqrt{P_{\text{in}}}(1+\delta_{\text{m}}\text{exp}(+i2\pi
f_{\text{m}}t))$ is the input laser amplitude, $P_{\text{in}}$ is the input
pump laser power, $P_{\text{in}}\delta_{\text{m}}^{2}$ is the input sideband
power, $f_{\text{m}}$ is the frequency difference between the pump laser and
the sideband. The microresonator temperature $T$ evolves following the
equation

$\frac{\partial T}{\partial
t}=\frac{KT}{\tau_{\text{r}}}|A(t,\varphi)|^{2}-\frac{T}{\tau_{\text{r}}},$
(2)

where $KT$ is the thermal induced resonance shift and $\tau_{\text{r}}$ is the
thermal relaxation time. With a proper set of parameters (see Supplementary
Information), the simulation shows prominent efficacy for thermal
stabilization and soliton step extension, as shown in Fig. 1(b).

Experimentally, we create the sideband by phase-modulation of the carrier pump
laser. The experimental setup is shown in Fig. 2(a). The pump laser is an
external-cavity diode laser (ECDL, Toptica CTL 1550) with its frequency
$\nu_{0}$ tuned by a piezo which is actuated by an arbitrary function
generator (AFG, Rigol DG4062). A phase modulator (PM, iXblue LPZ-LN-10)
creates two sidebands $\nu_{-}$ and $\nu_{+}$ from the carrier pump laser. The
modulated laser is then power-amplified by an erbium-doped fiber amplifier
(EDFA, KEOPSYS CEFA-C-PB-HP). A band-pass filter is applied to reject
amplifier spontaneous emission (ASE) noise from the EDFA. After that, the
laser power is coupled into an integrated silicon nitride (Si3N4) optical
microresonator fabricated using a DUV subtractive process Ye  _et al._ (2023).
The Si3N4 microresonator features 19.97 GHz free spectral range (FSR) and an
intrinsic quality factor of $Q_{0}\approx 1\times 10^{7}$, characterized using
a vector spectrum analyzer Luo  _et al._ (2023) (see Supplementary
Information). After the Si3N4 microresonator, the pump laser is blocked by two
fiber Bragg gratings (FBG) providing 30 dB total attenuation. Finally, the
output optical spectrum is monitored by an optical spectrum analyzer (OSA,
Yokogava AQ6370D). The optical power is detected by a photodiode (PD, Thorlabs
PDA10CS2) and monitored by an oscilloscope (OSC, Rigol DHO4404).

Figure 4:  single- and multi-soliton spectra, and the time consumption to
reach these states. (a). Optical spectra for $N=5,4,3,2,1-$soliton states of
19.97 GHz mode spacing. Insets show the relative soliton positions of the
$N-$soliton state in the microresonator. The soliton numbers $N$ are retrieved
by inverse Fourier transform on the spectrum. The single-soliton spectrum is
fitted with a sech2 function (dashed red line). (b). Statistics of time
consumption for $N=3,2,1-$soliton state addressing with 100 repeated processes
among which 93/99/100 processes come into successful state addressing.

The wavelength and power of the pump laser without phase modulation are
$1555.7$ nm and 730 mW, respectively. The pump laser’s frequency scans through
a resonance from the effectively blue-detuned to red-detuned side (“forward
tunining” Herr  _et al._ (2013)), with the direction marked by arrows in Fig.
2(b). The light detected by the PD and recorded by the OSC corresponds to the
generated light from Kerr parametric oscillation in the microresonator. The
soliton step is marked with red color within the total 1.5 GHz frequency
range. Due to the photo-thermal effect in our Si3N4, the soliton step is so
short that is barely seen, impossible to access a soliton state. However, when
phase-modulating the pump laser with modulation frequency of
$f_{\text{m}}=374$ MHz and RF power applied on the PM of 11 dBm, a pair of
sidebands $\nu_{\pm}=\nu_{0}\pm f_{\text{m}}$ are created. The carrier pump
laser’s power is $8.3$ times higher than that of one sideband. Only the blue
sideband $\nu_{+}$ is used to counteract the thermal effect Wildi  _et al._
(2019); Nishimoto  _et al._ (2022), while the red sideband $\nu_{-}$ is idle.
Prominent soliton step extension is observed with the phase modulation, as
shown in Fig. 2(b). We repeatedly scan the laser through the resonance for 150
times, and statistically analyze the soliton step length as shown in Fig.
2(c). The statistical mean values are 5.2 and 42.7 MHz, respectively,
corresponding to 8-times extension due to sideband heating.

We further investigate the optimal value of $f_{\text{m}}$ for maximum soliton
step extension, as shown in Fig. 2(d) left. The optimal value under 11 dBm RF
power is $f_{\text{m}}=374$ MHz. The effect of RF power on soliton step length
is also investigated as shown in Fig. 2(d) right.

Soliton number-state addressing. Next, we realize deterministic generation of
$N-$soliton state using the sideband heating method. Figure 3(a) shows the
dynamic PD voltage, i.e. the dynamic generated light power, where the soliton
step sits in Zone 2. Supplementary Information shows that Zone 2 covers the
full range of detected voltages for all possible soliton steps in 50 times
measurements. Zone 1 indicates that the laser frequency is effectively blue-
detuned, and Zone 3 indicates far red-detuned, neither of which allows soliton
existence.

To distinguish the soliton number, we use the backward tuning scheme by
increasing the laser frequency Guo  _et al._ (2017). The soliton power trace
probed by the PD is converted to a voltage trace that shows clear stair
feature in Fig. 3(b). Each step in the stair corresponds to a soliton state
with different $N$. We measure the mean voltage $\overline{V}$ values of each
step and classify them into different soliton number $N$. Figure 3(c) shows
that, after repeating 20 times, a linear relation between the voltage and
soliton number is found, i.e. $\overline{V}=a\times N+b$, with $a=0.0609$ and
$b=0.0265$.

Previously, program-controlled single-soliton generation has been realized on
integrated microresonators with repetition rates above 49 GHz Wang  _et al._
(2021); Zhou  _et al._ (2022) that are difficult to detect with normal fast
photodetectors. Here, with calibrated zones and $(a,b)$ parameters, we
demonstrate a program for deterministic generation of target $N-$soliton state
in our Si3N4 microresonator of 19.97 GHz FSR, in addition to the single-
soliton state. The program logic is shown in Fig. 3(d) which consists two
blocks. The first block (blue) is to land on a soliton step, and the second
block (red) is to produce the target $N-$soliton state. Initially, the laser
frequency scans from $\nu_{s}$ to $\nu_{e}$ ($\nu_{s}>\nu_{e}$) close to the
resonance, such that the final PD voltage lands in Zone 3, as shown by the
lightest blue trace in Fig. 3(a). The scanning window is then translated by
$\Delta\nu$, i.e. the laser re-scanned from $\nu_{s}+\Delta\nu$ to
$\nu_{e}+\Delta\nu$. After that, if the PD voltage lands in Zone 1, meaning
that the laser frequency is blue-detuned and the soliton state is not yet
reached, then the system is re-initiated. If the PD voltage still lands in
Zone 3, the laser frequency scanning window is further translated by
$\Delta\nu$. The process continues until the PD voltage lands in Zone 2,
signalling that a soliton state is reached, as the darkest blue trace showing
in Fig. 3(a).

When a soliton state is reached, the program’s second block starts. Using
$(a,b)$ parameters and the PD voltage, the current soliton number is estimated
by $(V-b)/a$. For a target soliton number $N$, if $(V-b)/a<N-0.5$, the target
$N-$soliton generation fails and the system re-initiated. If $(V-b)/a>N+0.5$,
the laser frequency is increased with step $\Delta\nu^{\prime}$ until that
$(V-b)/a\in[N-0.5,N+0.5]$ where $N-$soliton state is successfully produced.
Parameters $(\Delta\nu,\Delta\nu^{\prime})$ are self-adaptive modified to
accelerate the sequence (see Supplementary Information).

We repeatedly apply the program for automatic $N-$soliton generation for 100
times. The soliton number $N$ can be retrieved by inverse Fourier transform on
the soliton’s optical spectrum on OSA, as shown in Fig 4(a). We verify the
soliton number of every addressed $N=5,4,3,2,1-$soliton state. The statistic
charts of time consumption for the $N=3,2,1-$soliton state addressing are
shown in Fig. 4(b). The success rates are 93%, 99% and 100% for
$N=3,2,1-$soliton production, respectively. Table. 1 summarizes the data of up
to 5-soliton production. The existence of $N-$soliton state is monitored by
sampling the PD voltage $V$ at 10 Hz rate.

Table 1: Statistics of success rate and time consumption to produce $N-$soliton states. Soliton number | Success rate | Probability | Probability | Mean time | Shortest time
---|---|---|---|---|---
$N$ |  | < 15 s | < 20 s | (s) | (s)
1 | 100% | 59% | 96% | 14.6 | 10.6
2 | 99% | 65% | 88% | 14.7 | 10.3
3 | 93% | 63% | 96% | 14.3 | 10.1
4 | 96% | 45% | 72% | 17.5 | 9.9
5 | 95% | 47% | 80% | 16.7 | 2.8

In conclusion, we have demonstrated a novel experimental approach to avoid
photo-thermal effect commonly encountered in the generation of dissipative
Kerr solitons in optical microresonators. The approach is based on phase-
modulation of the CW pump, where the generated blue sideband synergizes with
the carrier pump and thermally stabilizes the microresonator. The sideband
provides immediate heating when the intra-cavity power transits to a soliton
state and the microresonator experiences a sudden cooling. Using a 19.97-GHz-
FSR Si3N4 microresonator, we successfully demonstrate soliton step extension
by 8 times with the sideband heating. Furthermore, we have developed a program
to automatically, deterministically and quickly address to target $N-$soliton
state, including the single-soliton state. We show near 100% success rate and
as short as 10 s time consumption for $N-$soliton state addressing.

While demonstrated on Si3N4, our method is also useful for soliton generation
in other platforms suffering from stronger photo-thermal effect. The program-
controlled automatic addressing of $N-$soliton state can be critical for
several applications in microwave photonics and frequency comb metrology.
Together, our method facilitates a wider applications of soliton microcomb
systems out of laboratory environments.

Funding Information: J. Liu acknowledges support from the National Natural
Science Foundation of China (Grant No.12261131503), Shenzhen-Hong Kong
Cooperation Zone for Technology and Innovation (HZQB-KCZYB2020050), and from
the Guangdong Provincial Key Laboratory (2019B121203002). Y.-H. L.
acknowledges support from the China Postdoctoral Science Foundation (Grant No.
2022M721482). Hairun Guo acknowledges support from the National Natural
Science Foundation of China (Grant No.11974234).

Acknowledgments: Silicon nitride chips were fabricated by Qaleido Photonics.
We thank Yuan Chen, Sanli Huang and Zhongkai Wang for assistance in the
experiment, Suwan Sun for assistance in simulation, Jijun He for the fruitful
discussion.

Author contributions: W. S. and J. Liu conceived the experiment. H. Z., W. S.,
H. W., B. S. and J. Long performed the experiment, assisted by S. M.. X. D.,
R. C. and Y.-H. L. performed the simulation, supervised by H. G.. C. S.
assisted in fabricating Si3N4 chips. W. S., Y.-H. L., B. S. and J. Liu
prepared the manuscript. J. Liu supervised the project and managed
collaboration.

Disclosures: W. S., J. Long and J. Liu filed a patent application related to
this work. Others declare no conflicts of interest.

Data Availability Statement: The code and data used to produce the plots
within this work will be released on the repository Zenodo upon publication of
this preprint.

## References

  * Herr  _et al._ (2013) T. Herr, V. Brasch, J. D. Jost, C. Y. Wang, N. M. Kondratiev, M. L. Gorodetsky, and T. J. Kippenberg, Nature Photonics 8, 145 (2013).
  * Kippenberg  _et al._ (2018) T. J. Kippenberg, A. L. Gaeta, M. Lipson, and M. L. Gorodetsky, Science 361, eaan8083 (2018).
  * Marin-Palomo  _et al._ (2017) P. Marin-Palomo, J. N. Kemal, M. Karpov, A. Kordts, J. Pfeifle, M. H. P. Pfeiffer, P. Trocha, S. Wolf, V. Brasch, M. H. Anderson, R. Rosenberger, K. Vijayan, W. Freude, T. J. Kippenberg, and C. Koos, Nature 546, 274 (2017).
  * Corcoran  _et al._ (2020) B. Corcoran, M. Tan, X. Xu, A. Boes, J. Wu, T. G. Nguyen, S. T. Chu, B. E. Little, R. Morandotti, A. Mitchell, and D. J. Moss, Nature Communications 11, 2568 (2020).
  * Mazur  _et al._ (2021) M. Mazur, M.-G. Suh, A. Fülöp, J. Schröder, V. Torres-Company, M. Karlsson, K. Vahala, and P. Andrekson, Journal of Lightwave Technology 39, 4367 (2021).
  * Trocha  _et al._ (2018) P. Trocha, M. Karpov, D. Ganin, M. H. P. Pfeiffer, A. Kordts, S. Wolf, J. Krockenberger, P. Marin-Palomo, C. Weimann, S. Randel, W. Freude, T. J. Kippenberg, and C. Koos, Science 359, 887 (2018).
  * Suh and Vahala (2018) M.-G. Suh and K. J. Vahala, Science 359, 884 (2018).
  * Obrzud  _et al._ (2019) E. Obrzud, M. Rainer, A. Harutyunyan, M. H. Anderson, J. Liu, M. Geiselmann, B. Chazelas, S. Kundermann, S. Lecomte, M. Cecconi, A. Ghedina, E. Molinari, F. Pepe, F. Wildi, F. Bouchy, T. J. Kippenberg, and T. Herr, Nature Photonics 13, 31 (2019).
  * Suh  _et al._ (2019) M.-G. Suh, X. Yi, Y.-H. Lai, S. Leifer, I. S. Grudinin, G. Vasisht, E. C. Martin, M. P. Fitzgerald, G. Doppmann, J. Wang, D. Mawet, S. B. Papp, S. A. Diddams, C. Beichman, and K. Vahala, Nature Photonics 13, 25 (2019).
  * (10) A. Dutt, C. Joshi, X. Ji, J. Cardenas, Y. Okawachi, K. Luke, A. L. Gaeta, and M. Lipson, Science Advances 4, e1701858.
  * Yang  _et al._ (2019) Q.-F. Yang, B. Shen, H. Wang, M. Tran, Z. Zhang, K. Y. Yang, L. Wu, C. Bao, J. Bowers, A. Yariv, and K. Vahala, Science 363, 965 (2019).
  * Liang  _et al._ (2015) W. Liang, D. Eliyahu, V. S. Ilchenko, A. A. Savchenkov, A. B. Matsko, D. Seidel, and L. Maleki, Nature Communications 6, 7957 (2015).
  * Liu  _et al._ (2020a) J. Liu, E. Lucas, A. S. Raja, J. He, J. Riemensberger, R. N. Wang, M. Karpov, H. Guo, R. Bouchand, and T. J. Kippenberg, Nature Photonics 14, 486 (2020a).
  * Feldmann  _et al._ (2021) J. Feldmann, N. Youngblood, M. Karpov, H. Gehring, X. Li, M. Stappers, M. Le Gallo, X. Fu, A. Lukashchuk, A. S. Raja, J. Liu, C. D. Wright, A. Sebastian, T. J. Kippenberg, W. H. P. Pernice, and H. Bhaskaran, Nature 589, 52 (2021).
  * Xu  _et al._ (2021) X. Xu, M. Tan, B. Corcoran, J. Wu, A. Boes, T. G. Nguyen, S. T. Chu, B. E. Little, D. G. Hicks, R. Morandotti, A. Mitchell, and D. J. Moss, Nature 589, 44 (2021).
  * Raja  _et al._ (2021) A. S. Raja, S. Lange, M. Karpov, K. Shi, X. Fu, R. Behrendt, D. Cletheroe, A. Lukashchuk, I. Haller, F. Karinou, B. Thomsen, K. Jozwik, J. Liu, P. Costa, T. J. Kippenberg, and H. Ballani, Nature Communications 12, 5867 (2021).
  * Wu  _et al._ (2018) J. Wu, X. Xu, T. G. Nguyen, S. T. Chu, B. E. Little, R. Morandotti, A. Mitchell, and D. J. Moss, IEEE Journal of Selected Topics in Quantum Electronics 24, 1 (2018).
  * Hu  _et al._ (2020) J. Hu, J. He, J. Liu, A. S. Raja, M. Karpov, A. Lukashchuk, T. J. Kippenberg, and C.-S. Brès, Nature Communications 11, 4377 (2020).
  * Spencer  _et al._ (2018) D. T. Spencer, T. Drake, T. C. Briles, J. Stone, L. C. Sinclair, C. Fredrick, Q. Li, D. Westly, B. R. Ilic, A. Bluestone, N. Volet, T. Komljenovic, L. Chang, S. H. Lee, D. Y. Oh, M.-G. Suh, K. Y. Yang, M. H. P. Pfeiffer, T. J. Kippenberg, E. Norberg, L. Theogarajan, K. Vahala, N. R. Newbury, K. Srinivasan, J. E. Bowers, S. A. Diddams, and S. B. Papp, Nature 557, 81 (2018).
  * Newman  _et al._ (2019) Z. L. Newman, V. Maurice, T. Drake, J. R. Stone, T. C. Briles, D. T. Spencer, C. Fredrick, Q. Li, D. Westly, B. R. Ilic, B. Shen, M.-G. Suh, K. Y. Yang, C. Johnson, D. M. S. Johnson, L. Hollberg, K. J. Vahala, K. Srinivasan, S. A. Diddams, J. Kitching, S. B. Papp, and M. T. Hummon, Optica 6, 680 (2019).
  * Yi  _et al._ (2015) X. Yi, Q.-F. Yang, K. Y. Yang, M.-G. Suh, and K. Vahala, Optica 2, 1078 (2015).
  * Yang  _et al._ (2018) K. Y. Yang, D. Y. Oh, S. H. Lee, Q.-F. Yang, X. Yi, B. Shen, H. Wang, and K. Vahala, Nature Photonics 12, 297 (2018).
  * Brasch  _et al._ (2016a) V. Brasch, M. Geiselmann, T. Herr, G. Lihachev, M. H. P. Pfeiffer, M. L. Gorodetsky, and T. J. Kippenberg, Science 351, 357 (2016a).
  * Joshi  _et al._ (2016) C. Joshi, J. K. Jang, K. Luke, X. Ji, S. A. Miller, A. Klenner, Y. Okawachi, M. Lipson, and A. L. Gaeta, Opt. Lett. 41, 2565 (2016).
  * Bao  _et al._ (2019) H. Bao, A. Cooper, M. Rowley, L. Di Lauro, J. S. Totero Gongora, S. T. Chu, B. E. Little, G.-L. Oppo, R. Morandotti, D. J. Moss, B. Wetzel, M. Peccianti, and A. Pasquazi, Nature Photonics 13, 384 (2019).
  * Wang  _et al._ (2021) X. Wang, P. Xie, W. Wang, Y. Wang, Z. Lu, L. Wang, S. T. Chu, B. E. Little, W. Zhao, and W. Zhang, Photon. Res. 9, 66 (2021).
  * Liu  _et al._ (2021) X. Liu, Z. Gong, A. W. Bruch, J. B. Surya, J. Lu, and H. X. Tang, Nature Communications 12, 5428 (2021).
  * Weng  _et al._ (2021) H. Weng, J. Liu, A. A. Afridi, J. Li, J. Dai, X. Ma, Y. Zhang, Q. Lu, J. F. Donegan, and W. Guo, Photon. Res. 9, 1351 (2021).
  * He  _et al._ (2019) Y. He, Q.-F. Yang, J. Ling, R. Luo, H. Liang, M. Li, B. Shen, H. Wang, K. Vahala, and Q. Lin, Optica 6, 1138 (2019).
  * Gong  _et al._ (2020) Z. Gong, X. Liu, Y. Xu, and H. X. Tang, Optica 7, 1275 (2020).
  * Jung  _et al._ (2021) H. Jung, S.-P. Yu, D. R. Carlson, T. E. Drake, T. C. Briles, and S. B. Papp, Optica 8, 811 (2021).
  * Guidry  _et al._ (2022) M. A. Guidry, D. M. Lukin, K. Y. Yang, R. Trivedi, and J. Vučković, Nature Photonics 16, 52 (2022).
  * Wang  _et al._ (2022) C. Wang, J. Li, A. Yi, Z. Fang, L. Zhou, Z. Wang, R. Niu, Y. Chen, J. Zhang, Y. Cheng, J. Liu, C.-H. Dong, and X. Ou, Light: Science & Applications 11, 341 (2022).
  * Xia  _et al._ (2022) D. Xia, Z. Yang, P. Zeng, B. Zhang, J. Wu, Z. Wang, J. Zhao, J. Huang, L. Luo, D. Liu, S. Yang, H. Guo, and Z. Li, Laser & Photonics Reviews n/a, 2200219 (2022).
  * Pu  _et al._ (2016) M. Pu, L. Ottaviano, E. Semenova, and K. Yvind, Optica 3, 823 (2016).
  * Moille  _et al._ (2020) G. Moille, L. Chang, W. Xie, A. Rao, X. Lu, M. Davanço, J. E. Bowers, and K. Srinivasan, Laser & Photonics Reviews 14, 2000022 (2020).
  * Nardi  _et al._ (2023) A. Nardi, A. Davydova, N. Kuznetsov, M. H. Anderson, C. Möhl, J. Riemensberger, P. Seidler, and T. J. Kippenberg, arXiv 2304.12968 (2023).
  * Stern  _et al._ (2018) B. Stern, X. Ji, Y. Okawachi, A. L. Gaeta, and M. Lipson, Nature 562, 401 (2018).
  * Liu  _et al._ (2020b) J. Liu, H. Tian, E. Lucas, A. S. Raja, G. Lihachev, R. N. Wang, J. He, T. Liu, M. H. Anderson, W. Weng, S. A. Bhave, and T. J. Kippenberg, Nature 583, 385 (2020b).
  * Voloshin  _et al._ (2021) A. S. Voloshin, N. M. Kondratiev, G. V. Lihachev, J. Liu, V. E. Lobanov, N. Y. Dmitriev, W. Weng, T. J. Kippenberg, and I. A. Bilenko, Nature Communications 12, 235 (2021).
  * Xiang  _et al._ (2021) C. Xiang, J. Liu, J. Guo, L. Chang, R. N. Wang, W. Weng, J. Peters, W. Xie, Z. Zhang, J. Riemensberger, J. Selvidge, T. J. Kippenberg, and J. E. Bowers, Science 373, 99 (2021).
  * Churaev  _et al._ (2023) M. Churaev, R. N. Wang, A. Riedhauser, V. Snigirev, T. Blésin, C. Möhl, M. H. Anderson, A. Siddharth, Y. Popoff, U. Drechsler, D. Caimi, S. Hönl, J. Riemensberger, J. Liu, P. Seidler, and T. J. Kippenberg, Nature Communications 14, 3499 (2023).
  * Liu  _et al._ (2022) Y. Liu, Z. Qiu, X. Ji, J. He, J. Riemensberger, A. S. Raja, W. Weng, J. Liu, and T. J. Kippenberg, to appear (2022).
  * de Beeck  _et al._ (2020) C. O. de Beeck, B. Haq, L. Elsinger, A. Gocalinska, E. Pelucchi, B. Corbett, G. Roelkens, and B. Kuyken, Optica 7, 386 (2020).
  * Li  _et al._ (2017) Q. Li, T. C. Briles, D. A. Westly, T. E. Drake, J. R. Stone, B. R. Ilic, S. A. Diddams, S. B. Papp, and K. Srinivasan, Optica 4, 193 (2017).
  * Gao  _et al._ (2022) M. Gao, Q.-F. Yang, Q.-X. Ji, H. Wang, L. Wu, B. Shen, J. Liu, G. Huang, L. Chang, W. Xie, S.-P. Yu, S. B. Papp, J. E. Bowers, T. J. Kippenberg, and K. J. Vahala, Nature Communications 13, 3323 (2022).
  * Brasch  _et al._ (2016b) V. Brasch, M. Geiselmann, M. H. P. Pfeiffer, and T. J. Kippenberg, Opt. Express 24, 29312 (2016b).
  * Yi  _et al._ (2016) X. Yi, Q.-F. Yang, K. Y. Yang, and K. Vahala, Opt. Lett. 41, 2037 (2016).
  * Stone  _et al._ (2018) J. R. Stone, T. C. Briles, T. E. Drake, D. T. Spencer, D. R. Carlson, S. A. Diddams, and S. B. Papp, Phys. Rev. Lett. 121, 063902 (2018).
  * Zhou  _et al._ (2019) H. Zhou, Y. Geng, W. Cui, S.-W. Huang, Q. Zhou, K. Qiu, and C. Wei Wong, Light: Science & Applications 8, 50 (2019).
  * Zhang  _et al._ (2019) S. Zhang, J. M. Silver, L. D. Bino, F. Copie, M. T. M. Woodley, G. N. Ghalanos, A. Ø. Svela, N. Moroney, and P. Del’Haye, Optica 6, 206 (2019).
  * Drake  _et al._ (2020) T. E. Drake, J. R. Stone, T. C. Briles, and S. B. Papp, Nature Photonics 14, 480 (2020).
  * Jang  _et al._ (2015) J. K. Jang, M. Erkintalo, S. Coen, and S. G. Murdoch, Nature Communications 6, 7370 (2015).
  * Wildi  _et al._ (2019) T. Wildi, V. Brasch, J. Liu, T. J. Kippenberg, and T. Herr, Opt. Lett. 44, 4447 (2019).
  * Nishimoto  _et al._ (2022) K. Nishimoto, K. Minoshima, T. Yasui, and N. Kuse, Opt. Lett. 47, 281 (2022).
  * Cole  _et al._ (2018) D. C. Cole, J. R. Stone, M. Erkintalo, K. Y. Yang, X. Yi, K. J. Vahala, and S. B. Papp, Optica 5, 1304 (2018).
  * Miao  _et al._ (2022) R. Miao, C. Zhang, X. Zheng, X. Cheng, K. Yin, and T. Jiang, Photon. Res. 10, 1859 (2022).
  * Obrzud  _et al._ (2017) E. Obrzud, S. Lecomte, and T. Herr, Nature Photonics 11, 600 (2017).
  * Lei  _et al._ (2022) F. Lei, Z. Ye, and V. Torres-Company, Opt. Lett. 47, 513 (2022).
  * Grassani  _et al._ (2022) D. Grassani, H. E. Dirani, F. A. Sabattoli, L. Youssef, C. Petit-Etienne, S. Kerdiles, E. Pargon, M. Liscidini, C. Sciancalepore, D. Bajoni, and M. Galli, Opt. Continuum 1, 1516 (2022).
  * Lu  _et al._ (2021) Z. Lu, H.-J. Chen, W. Wang, L. Yao, Y. Wang, Y. Yu, B. E. Little, S. T. Chu, Q. Gong, W. Zhao, X. Yi, Y.-F. Xiao, and W. Zhang, Nature Communications 12, 3179 (2021).
  * Brasch  _et al._ (2017) V. Brasch, E. Lucas, J. D. Jost, M. Geiselmann, and T. J. Kippenberg, Light: Science & Applications 6, e16202 (2017).
  * Carmon  _et al._ (2004) T. Carmon, L. Yang, and K. J. Vahala, Opt. Express 12, 4742 (2004).
  * Guo  _et al._ (2017) H. Guo, M. Karpov, E. Lucas, A. Kordts, M. H. P. Pfeiffer, V. Brasch, G. Lihachev, V. E. Lobanov, M. L. Gorodetsky, and T. J. Kippenberg, Nature Physics 13, 94 (2017).
  * Ye  _et al._ (2023) Z. Ye, H. Jia, Z. Huang, C. Shen, J. Long, B. Shi, Y.-H. Luo, L. Gao, W. Sun, H. Guo, J. He, and J. Liu, Photon. Res. 11, 558 (2023).
  * Luo  _et al._ (2023) Y.-H. Luo, B. Shi, W. Sun, R. Chen, S. Huang, Z. Wang, J. Long, C. Shen, H. Jia, Z. Ye, H. Guo, and J. Liu, arXiv 2304.04295 (2023).
  * Zhou  _et al._ (2022) L. Zhou, Y. Shen, C. Xi, X. Huang, and G. He, Opt. Continuum 1, 161 (2022).


# SASFormer: Transformers for Sparsely Annotated Semantic Segmentation

###### Abstract

Semantic segmentation based on sparse annotation has advanced in recent years.
It labels only part of each object in the image, leaving the remainder
unlabeled. Most of the existing approaches are time-consuming and often
necessitate a multi-stage training strategy. In this work, we propose a simple
yet effective sparse annotated semantic segmentation framework based on
segformer, dubbed SASFormer, that achieves remarkable performance.
Specifically, the framework first generates hierarchical patch attention maps,
which are then multiplied by the network predictions to produce correlated
regions separated by valid labels. Besides, we also introduce the affinity
loss to ensure consistency between the features of correlation results and
network predictions. Extensive experiments showcase that our proposed approach
is superior to existing methods and achieves cutting-edge performance. The
source code is available at https://github.com/su-hui-zz/SASFormer.

Index Terms—  semantic segmentation, weakly supervised, sparsely annotated,
scribble-supervised, vision transformer

##  1 Introduction

Semantic segmentation is an essential problem in computer vision, which seeks
to identify each pixel in an image. Although semantic segmentation has
observed ongoing improvements in recent years, impact of semantic segmentation
demands a lot of time and labor to conduct pixel by pixel annotation. Due to
the high cost of labeling, researchers have to examine techniques to maintain
adequate segmentation performance while only making partial labeling. Sparsely
annotated semantic segmentation (SASS) comes into existence, which provides
sparse annotations for each object in an image [1], such as point-wise [2, 3]
and scribble-wise [4, 5] supervision.

Fig. 1: Semantic segmentation with sparse annotation. The baseline trained
only with sparse annotations is incapable of recognizing the full object. In
the majority of earlier efforts, auxiliary networks containing information
about other tasks or previous stage. Our SASFormer utilizes inherent global
dependencies of the transformer to achieve state-of-the-art performance
without developing a complicated framework.

Sparse annotation semantic segmentation is a kind of weakly supervised
semantic segmentation (WSSS) [6]. It marks just a piece of each visual item,
leaving the rest unidentified. Due to the absence of a precise object
boundary, segmentation performance is substantially impaired. To more
precisely estimate unlabeled pixels, it is required to discover more hidden
information.

Existing methodologies for SASS can be categorized as regularization loss,
multitask auxiliary, consistent learning, and the pseudo-label method.
Regularization losses [7, 8] utilize MRF/CRF potentials to implement
clustering of low-dimensional information. Multitask assistance [9] increases
segmentation performance by adding an auxiliary counterpart, for example
boundary detection. Consistency learning [10, 11] introduces multiple networks
to compensate for insufficient information provided in sparse annotations. The
pseudo-label approach typically involves a multi-stage training procedure.
Most of the above methods necessitate the use of additional complex frameworks
to facilitate training, resulting in a complicated and time-consuming training
process.

In recent years, vision transformers [12] have steadily evolved in various
fields [13, 14]. The self-attention mechanism in the vision transformer can
capture effective relationships between pixels, which is beneficial for
information propagation from labeled pixels to unlabeled pixels. Inspired by
this, We investigate single-stage effective sparsely annotated semantic
segmentation with vision transformer, aiming to achieve the state-of-the-art
results in a straightforward manner, as shown in Fig 1.

In this work, we propose SASFormer, a simple yet effective framework based on
segformer[12], as the first effort at sparsely annotated semantic segmentation
with Vision Transformer. With the inherent patch-to-patch attention of
cascaded transformer blocks, we first create patch attention maps at multiple
levels. The patch-level attention maps are then multiplied by network
predictions in order to correlate labeled and unlabeled image regions.
Following this, an affinity loss function is applied to assure the consistency
of the correlation statistics and network predictions. We finally consider
embedding priors from unlabeled areas to boost the performance by combining
affinity loss and conventional segmentation loss. Our contributions can be
summarized in the following manner:

  * •

We propose SASFormer as the first trustworthy baseline for SASS with a visual
transformer to model relationships between distinct areas and offer category
recommendations to unlabeled regions.

  * •

We provide a novel affinity loss function to assure the similarity prior in
SASS: areas of identical objects share similarity in both low-dimensional and
high-dimensional feature space.

  * •

The proposed SASFormer manifests remarkable performances in both point- and
scribble-annotated SASS assignments on PASCAL VOC.

##  2 Related Works

###  2.1 Sparsely Annotated Semantic Segmentation

Sparsely annotated semantic segmentation aims to address the segmentation
problem using minimally labeled visual areas. What’s the point[2] brought
point-level labels to the semantic segmentation problem for the first time,
which boost the performance by introducing objectiveness priors. After that,
ScribbleSup[4] extends point-level labels into the scribbling ones in order to
bridge the performance gap between fully annotated methods and unlabeled ones.
ScribeSup constructs a graphical model to transmit information from scribbles
to unknown pixels, hence providing semantic guidance for unknown pixels. Since
then, approaches for sparse annotated semantic segmentation have emerged and
developed. In an attempt to enhance performance at the lowest feasible cost,
researchers have begun investigating the association between labeled and
unlabeled regions. For example, Tang et al. [7] offered a variety of
regularized losses for sparsely supervised segmentation based on dense CRF and
kernel cut. BPG [9] tries to integrate semantic characteristics and textural
information by constructing a prediction refinement network, whilst a boundary
regression network is introduced to facilitate performance by generating
clearly defined semantically separate parts. Recently, SPML [11] tackles this
challenge by developing a semi-supervised metric learning approach with four
unique forms of attraction and repulsion relationships TEL [1] presents a tree
energy loss, in which minimal spanning trees are constructed to represent low-
level and high-level pair-wise affinities. The majority of these techniques
either use multi-stage processes for progressive inference or include
intricate frameworks for supplementary training. While in this work, we make
the first attempt to model relationships between distinct areas and offer
category recommendations to unlabeled regions based on vision transformer.

###  2.2 Transformer

The Transformer was initially proposed to model long-term dependencies in
natural language processing tasks [15]. In 2020, Alexey Dosovitskiy introduced
the pure Transformer architecture, which achieved remarkable results in image
classification [16]. Subsequently, the Transformer has been widely employed in
various computer vision tasks, such as object detection [17, 18, 19], semantic
segmentation [12, 20] and video processing [21, 22]. Recently, researchers
have started to utilize the long-term dependencies mechanism of the
Transformer to capture the relationship between image categories and local
image features, aiming to optimize weakly supervised problems [23]. However,
these methods mainly focus on utilizing the correlation between class tokens
and patch tokens, while few works consider the optimization and utilization of
the correlation among different patch tokens. Our work takes an early attempt
to delve deeply into the issue of imprecise correlation among different patch
tokens in the Transformer block and proposes a simple yet effective solution.

##  3 Methods

Fig. 2: An overview of the proposed Sparsely Annotated Segmentation
Transformer(SasFormer)

###  3.1 Overview

As depicted in Fig 2, given an image $I\in\mathbb{R}^{W\times H\times 3}$, we
partition it into $p\times p$ resolution patches. These patches are fed into
hierarchical transformer encoder to provide multilevel characteristics. We
pass multi-level features into the decoder block in order to obtain
segmentation probabilities $\mathbf{P}$ with resolution $M\times C$, where $M$
is the patch number and $C$ is the number of categories. On the one hand, we
build multi-level patch attention maps with transformer layers in encoder
blocks during the training phase. In addition, segmentation probabilities are
interpolated into resolutions corresponding to the width of patch attention
maps. Patch attention maps are then multiplied by interpolated segmentation
probabilities, yielding propagated segmentation probabilities $\mathbf{Y}$.
For labeled regions of interest, we perform standard cross entropy loss
function $L_{seg}$ to supervise segmentation probabilities with ground truth
labels. For unlabeled regions, we introduce an affinity loss function
$L_{aff}$ to match segmentation probabilities to propagated segmentation
probabilities. During testing, segmentation probabilities are immediately
assigned to the class with the maximum probability at each pixel, resulting in
the final segmentation map. The overall loss function is defined as follows:

$L=L_{seg}+\alpha*L_{aff}$ (1)

###  3.2 Patch Attention Map Generation

We introduce Segformer [12] as our backbone in order to describe our method
more conveniently. More results of different backbones can be referred to Sec
4.3. Hierarchical transformer encoder contains $L=4$ encoder blocks. Each
encoder block contains $N_{l}$ consecutive transformer layers. We get the
patch attention map with the efficient self-attention of transformer layer,
which is defined as follows:

$\mathbf{A}_{l,n}=softmax(\frac{\mathbf{Q}_{l,n}\mathbf{K}_{l,n}^{\mathbf{T}}}{\sqrt{D}})$
(2)

where $\mathbf{Q}_{l,n}\in\mathbb{R}^{M_{l}\times D}$ and
$\mathbf{K}_{l,n}\in\mathbb{R}^{M^{\prime}_{l}\times D}$ are the query and key
representations of $n$-th transformer layer in $l$-th encoder block,
respectively.
$M_{l}\in\\{\frac{W*H}{4},\frac{W*H}{8},\frac{W*H}{16},\frac{W*H}{32}\\}$
denotes resolution size in different encoder blocks. $M^{\prime}_{l}$ denotes
resolution size from $M_{l}$ after sequence reduction process[12] to be more
efficient. $D$ indicates dimension of patch embeddings. $\mathbf{T}$ is the
transpose operator.

Efficient attention map $\mathbf{A}_{l,n}\in\mathbb{R}^{M_{l}\times
M^{\prime}_{l}}$ records dependency of each patch token of $n$-th transformer
layer in $l$-th encoder block. We aggregate these attention maps over
transformer layers in each decoder block as the following:

$\mathbf{A}_{l}=\frac{1}{N}\sum_{n=1}^{N}\mathbf{A}_{l,n}$ (3)

where $\mathbf{A}_{l}\in\mathbb{R}^{M_{l}\times M^{\prime}_{l}}$ means patch
attention map in $l$-th encoder block.

To better comprehend what information patch attention maps capture at various
levels, we randomly draw a point with a red pentagram in the inputs and
analyze the dependency between the reference point and other pixels on patch
attention maps of various encoder blocks, as depicted in the baseline of Fig
3. $\mathbf{A}^{i,j}_{l,n}$ means the amount of $j$-th input token embedding
contributes to $i$-th output token of $n$-th transformer layer in $l$-th
encoder block. Likewise, $\mathbf{A}^{i}_{l}$ means the aggregated amount of
each input token embedding contributes to $i$-th output token in $l$-th
encoder block. We construct $\mathbf{A}^{r}_{1}$, $\mathbf{A}^{r}_{2}$,
$\mathbf{A}^{r}_{3}$ and $\mathbf{A}^{r}_{4}$ to illustrate dependency between
the reference point and each pixel in the first, second, third, forth encoder
block, respectively.

It can be seen that $\mathbf{A}_{1}$ and $\mathbf{A}_{2}$ prefer to assign
larger weights to patches that have the same color and texture as the
reference point. For instance, $\mathbf{A}^{r}_{1}$ and $\mathbf{A}^{r}_{2}$
of the first input emphasize the white color of the bird, while the second
input highlights the dark brown color of the material. $\mathbf{A}_{3}$ and
$\mathbf{A}_{4}$ are more concerned with semantic consistency, although
$\mathbf{A}_{4}$ is more class-specific. Patch attention maps establish
correlations between pixels in color space and high-level characteristics,
facilitating the propagation of segmentation information from labeled to
unlabeled pixels. However, transformer tends to distract attention from
targets to background area[18], which can be clearly seen in those of
baseline($\mathbf{A}_{2}$) and baseline($\mathbf{A}_{3}$). We hypothesize that
the inability of attention maps to capture dependence, which impacts the
transfer of information across pixels, is partly manifested in distractions
from irrelevant background.

###  3.3 Affinity Loss Function

Fig. 3: Visualization of inputs with reference points (red pentagram) and
patch attention maps at various levels. The baseline indicates the Segformer
[12] approach. $\mathbf{A}^{r}_{1}$, $\mathbf{A}^{r}_{2}$,
$\mathbf{A}^{r}_{3}$ and $\mathbf{A}^{r}_{4}$ illustrate the dependency
between the reference point and each pixel of patch attention map in the
first, second, third and forth encoder block, respectively.

For each encoder block, we interpolate segmentation probabilities $\mathbf{P}$
into the resolution of $M^{\prime}_{l}$ and $M_{l}$, yielding
$\mathbf{P}_{l}^{{}^{\prime}}$ and $\mathbf{P}_{l}$ respectively. Then we
transfer segmentation predictions from labeled pixels to unlabeled pixels by
multiplying patch attention map $\mathbf{A}_{l}$ and interpolated segmentation
probabilities $\mathbf{P}_{l}^{{}^{\prime}}$, which is described as follows:

$\mathbf{Y}_{l}=\mathbf{A}_{l}\otimes\mathbf{P}_{l}^{{}^{\prime}}$ (4)

where $\mathbf{Y}_{l}$ is propagated segmentation probabilities, $\otimes$
denotes matrix multiplication.

$\mathbf{Y}_{l}$ and $\mathbf{P}_{l}$ are normalized along the category
dimension, which is defined as:

$\mathbf{Y}^{*(m_{l},c)}_{l}=\frac{\exp(\mathbf{Y}^{(m_{l},c)}_{l})}{\sum_{c=1}^{C}{\exp(\mathbf{Y}^{(m_{l},c)}_{l})}}\qquad
1\leq m_{l}\leq M_{l}$ (5)
$\mathbf{P}^{*(m_{l},c)}_{l}=\frac{\exp(\mathbf{P}^{(m_{l},c)}_{l})}{\sum_{c=1}^{C}{\exp(\mathbf{P}^{(m_{l},c)}_{l})}}\qquad
1\leq m_{l}\leq M_{l}$ (6)

where $m_{l}$ and $c$ denote patch index and class index.
$\mathbf{Y}^{(m_{l},c)}_{l}$ and $\mathbf{P}^{(m_{l},c)}_{l}$ mean the
position of row $m_{l}$, column $c$ of matrix $\mathbf{Y}_{l}$ and
$\mathbf{P}_{l}$.

The affinity loss $L_{aff}$ attempts to maximize the similarity between
propagated probabilities and the interpolated segmentation probabilities over
all encoder blocks. Formally, the affinity loss is defined as:

$L_{aff}=\frac{1}{L}\sum_{l=1}^{L}{\|\mathbf{Y}_{l}^{*}-\mathbf{P}_{l}^{*}\|_{1}}$
(7)

where $\|\|_{1}$ and L denote L1 regularization term and encoder block number,
respectively.

In the process of segmentation training, we obtain additional bonus
information, as illustrated in Fig 3. Under the constraint of consistency
between propagated and interpolated segmentation probabilities, pair-wise
dependencies between patch tokens with different labels on patch attention
maps are simultaneously deteriorated, while pair-wise dependencies between
patch tokens with the same label are enhanced. By comparing to the baseline in
Fig 3, it is evident that the background noise in the patch attention map has
been suppressed in our proposed SASFormer.

##  4 Experiments

###  4.1 Datasets and Settings

The point-wise annotation [2] and scribble-wise annotation [4] of Pascal VOC
2012 dataset [24] are utilized for point-supervised and scribble-supervised
settings, respectively.

Table 1: Experimental reults for point-wise annotations on Pascal VOC 2012 validation set. Methods | Backbone | multi- | mIoU(%)
---|---|---|---
stage
DenseCRF Loss [7] | ResNet101 | $\checkmark$ | 57.00
A2GNN [25] | ResNet101 | $\checkmark$ | 66.80
Seminar [10] | ResNet101 | $\checkmark$ | 72.51
What’s the point [2] | VGG16 | - | 43.40
TEL [1] | ResNet101 | - | 68.40
Baseline* | Segformer | - | 64.21
TEL* [1] | Segformer | - | 69.33
Sasformer (Ours) | Segformer | - | 73.13

We adopt the segformer with pre-trained weights on ImageNet as our backbone
for experiments. Our model is trained for 80k training epochs with initial
learning rate 0.001, batch size 16 and input resolution 512x512. The SGD
optimizer is used with momentum 0.9 and weight decay 1e-4. Random horizontal
flip, random brightness in [-10, 10], random resize in [0.5, 2.0] and random
crop are all employed in data augmentation. In practice, we set $\alpha$ in
Eq.1 1.2 for scribble setting and 0.2 for point setting. All experiments are
conducted on PyTorch with four 32GB V100 GPUs. * indicates that we reproduce
the results using the provided codes, unless otherwise specified.

###  4.2 Comparison with State-of-the-art Methods

Table 2: Experimental reults for scribble-wise annotations on Pascal VOC 2012 validation set. Methods | Backbone | multi- | mIoU (%)
---|---|---|---
stage
ScribbleSup [4] | VGG16 | $\checkmark$ | 63.10
DenseCRF Loss [7] | ResNet101 | $\checkmark$ | 75.00
GridCRF Loss [8] | ResNet101 | $\checkmark$ | 72.80
URSS [26] | ResNet101 | $\checkmark$ | 76.10
Seminar [10] | ResNet101 | $\checkmark$ | 76.20
A2GNN [25] | ResNet101 | $\checkmark$ | 76.20
BPG [9] | ResNet101 | - | 76.00
SPML [11] | ResNet101 | - | 76.10
PSI [5] | ResNet101 | - | 74.90
TEL [1] | ResNet101 | - | 77.30
Baseline* | Segformer | - | 73.37
TEL* [1] | Segformer | - | 78.53
Sasformer (Ours) | Segformer | - | 79.49
Fig. 4: Segmentation results on Pascal VOC 2012 dataset. The point and
scribble indicate the point and scribble supervision. The baseline and TEL
indicate the Segformer [12] and TEL [1] approach, respectively.

Tab 1 and Tab 2 present the experimental results on the Pascal VOC 2012 for
point-wise and scribble-wise annotations, respectively. In Tab 1 for point-
wise annotations, the baseline based on segformer-B4 employs the cross-entropy
loss function only can reach 64.2% mIoU. TEL [1] improves mIoU performance by
5.13%. Our proposed SASFormer outperforms TEL by 4.43% with the same backbone,
yielding a mIoU of 72.78%. As for scribble-wise annotations in Tab 2, the
baseline framework achieves a mIoU of 73.3%. SASFormer produces the best
performance with the mIoU of 79.49%, surpassing the baseline by 6.19%. It
demonstrates that SASFormer achieves remarkable performance compared to all
other state-of-the-art approaches, which adopt a single-stage training
strategy without any additional data. The qualitative results for point-wise
annotations and scribble-wise annotations showcase that not only does our
technique capture object outlines more accurately, but it also minimizes
category misidentification, as shown in Fig 4.

###  4.3 Ablations

Table 3: Effect of patch attention map in different encoder blocks. | $L_{aff}^{1}$ | $L_{aff}^{2}$ | $L_{aff}^{3}$ | $L_{aff}^{4}$ | mIoU (%)
---|---|---|---|---|---
exp1 | - | - | - | - | 73.37
exp2 | $\checkmark$ | - | - | - | 77.38
exp3 | - | $\checkmark$ | - | - | 76.97
exp4 | - | - | $\checkmark$ | - | 75.98
exp5 | - | - | - | $\checkmark$ | 74.10
exp6 | $\checkmark$ | $\checkmark$ | - | - | 79.06
exp7 | $\checkmark$ | $\checkmark$ | $\checkmark$ | - | 79.24
exp8 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 79.49

Single scale vs. Multi scale. We examine the influence of patch attention maps
in various encoder blocks on segmentation performance in Tab 3, where
$L_{aff}^{l}$ denotes the affinity loss generated by the patch attention map
of the $l$-th encoder block. The optimal segmentation performance is achieved
when patch attention maps are incorporated into the generation of affinity
loss in different encoder blocks.

Table 4: The effect of different loss configurations. Configuration | KL | CE | L2 | L1
---|---|---|---|---
mIoU (%) | 78.62 | 76.80 | 77.25 | 79.49

The impact of distance metrics. For the similarity between propagated and
interpolated segmentation probabilities, in addition to L1 distance in Eq.7,
cross entropy, L2 distance, and KL divergence may also be considered. To
evaluate the performance of our SASFormer, we evaluate different loss
configurations. As seen in Tab 4, different forms of affinity loss can also
improve performance. We choose L1 distance as the final implementation of our
SASFormer since it produces the best results (79.49% mIoU).

Table 5: The flexibility of our proposed approach. Backbone | Ours | mIoU (%)
---|---|---
Segmentor [27] | w/o | 72.75
w/ | 76.11 (+3.36)
SETR [28] | w/o | 70.47
w/ | 76.04 (+5.57)
segformer [12] | w/o | 73.37
w/ | 79.49 (+6.12)

The flexibility among backbones. To evaluate the flexibility of our technique
across a variety of transformer-based segmentation networks, we also present
the popular segmentor [27] and SETR [28] in order to compare segmentation
performance with and without affinity loss. As shown in Tab 5, our affinity
loss is robust across a variety of segmentation networks.

##  5 Conclusions

In this work, we propose an efficient framework, dubbed SASFormer, to deal
with the issue of semantic segmentation under sparsely annotated supervision.
Most of the existing approaches are always time-consuming and often
necessitate a multi-stage training strategy. While the proposed SASFormer
adopts an end-to-end scheme that leverages hierarchical patch attention maps.
In addition, we also introduce the affinity loss to capture the consistency
between the features of correlation results and network predictions.
Exhaustive experiments validate the effectiveness of the proposed approach and
showcase remarkable performance compared to state-of-the-art approaches.

## Acknowledgments

This work is partially supported by the National Natural Science Foundation of
China (Grant No. 62106235), by the Exploratory Research Project of Zhejiang
Lab(2022PG0AN01), by the Zhejiang Provincial Natural Science Foundation of
China (LQ21F020003).

## References

  * [1] Zhiyuan Liang, Tiancai Wang, Xiangyu Zhang, Jian Sun, and Jianbing Shen,  “Tree energy loss: Towards sparsely annotated semantic segmentation,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 16907–16916.
  * [2] Amy Bearman, Olga Russakovsky, Vittorio Ferrari, and Li Fei-Fei,  “What’s the point: Semantic segmentation with point supervision,”  in European conference on computer vision. Springer, 2016, pp. 549–565.
  * [3] Rui Qian, Yunchao Wei, Honghui Shi, Jiachen Li, Jiaying Liu, and Thomas Huang,  “Weakly supervised scene parsing with point-based distance metric learning,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2019, vol. 33, pp. 8843–8850.
  * [4] Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun,  “Scribblesup: Scribble-supervised convolutional networks for semantic segmentation,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 3159–3167.
  * [5] Jingshan Xu, Chuanwei Zhou, Zhen Cui, Chunyan Xu, Yuge Huang, Pengcheng Shen, Shaoxin Li, and Jian Yang,  “Scribble-supervised semantic segmentation inference,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 15354–15363.
  * [6] Dingwen Zhang, Wenyuan Zeng, Guangyu Guo, Chaowei Fang, Lechao Cheng, and Junwei Han,  “Weakly supervised semantic segmentation via alternative self-dual teaching,”  arXiv preprint arXiv:2112.09459, 2021.
  * [7] Meng Tang, Federico Perazzi, Abdelaziz Djelouah, Ismail Ben Ayed, Christopher Schroers, and Yuri Boykov,  “On regularized losses for weakly-supervised cnn segmentation,”  in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 507–522.
  * [8] Dmitrii Marin, Meng Tang, Ismail Ben Ayed, and Yuri Boykov,  “Beyond gradient descent for regularized segmentation losses,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 10187–10196.
  * [9] Bin Wang, Guojun Qi, Sheng Tang, Tianzhu Zhang, Yunchao Wei, Linghui Li, and Yongdong Zhang,  “Boundary perception guidance: A scribble-supervised semantic segmentation approach,”  in IJCAI International joint conference on artificial intelligence, 2019.
  * [10] Hongjun Chen, Jinbao Wang, Hong Cai Chen, Xiantong Zhen, Feng Zheng, Rongrong Ji, and Ling Shao,  “Seminar learning for click-level weakly supervised semantic segmentation,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 6920–6929.
  * [11] Tsung-Wei Ke, Jyh-Jing Hwang, and Stella X Yu,  “Universal weakly supervised segmentation by pixel-to-segment contrastive learning,”  arXiv preprint arXiv:2105.00957, 2021.
  * [12] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo,  “Segformer: Simple and efficient design for semantic segmentation with transformers,”  in Neural Information Processing Systems (NeurIPS), 2021.
  * [13] Chaowei Fang, Dingwen Zhang, Liang Wang, Yulun Zhang, Lechao Cheng, and Junwei Han,  “Cross-modality high-frequency transformer for mr image super-resolution,”  arXiv preprint arXiv:2203.15314, 2022.
  * [14] Mengqi Xue, Qihan Huang, Haofei Zhang, Lechao Cheng, Jie Song, Minghui Wu, and Mingli Song,  “Protopformer: Concentrating on prototypical parts in vision transformers for interpretable image recognition,”  arXiv preprint arXiv:2208.10431, 2022.
  * [15] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.,  “An image is worth 16x16 words: Transformers for image recognition at scale,”  arXiv preprint arXiv:2010.11929, 2020.
  * [17] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko,  “End-to-end object detection with transformers,”  in European conference on computer vision. Springer, 2020, pp. 213–229.
  * [18] Hui Su, Yue Ye, Zhiwei Chen, Mingli Song, and Lechao Cheng,  “Re-attention transformer for weakly supervised object localization,”  arXiv preprint arXiv:2208.01838, 2022.
  * [19] Tian Qiu, Linyun Zhou, Wenxiang Xu, Lechao Cheng, Zunlei Feng, and Mingli Song,  “Team-detr: Guide queries as a professional team in detection transformers,”  arXiv preprint arXiv:2302.07116, 2023.
  * [20] Hao Li, Dingwen Zhang, Nian Liu, Lechao Cheng, Yalun Dai, Chao Zhang, Xinggang Wang, and Junwei Han,  “Boosting low-data instance segmentation by unsupervised pre-training with saliency prompt,”  arXiv preprint arXiv:2302.01171, 2023.
  * [21] Hao Zhang, Yanbin Hao, and Chong-Wah Ngo,  “Token shift transformer for video classification,”  in Proceedings of the 29th ACM International Conference on Multimedia, 2021, pp. 917–925.
  * [22] Hao Zhang, Lechao Cheng, Yanbin Hao, and Chong-wah Ngo,  “Long-term leap attention, short-term periodic shift for video classification,”  in Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 5773–5782.
  * [23] Wei Gao, Fang Wan, Xingjia Pan, Zhiliang Peng, Qi Tian, Zhenjun Han, Bolei Zhou, and Qixiang Ye,  “Ts-cam: Token semantic coupled attention map for weakly supervised object localization,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 2886–2895.
  * [24] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman,  “The pascal visual object classes (voc) challenge,”  International journal of computer vision, vol. 88, no. 2, pp. 303–338, 2010.
  * [25] Bingfeng Zhang, Jimin Xiao, Jianbo Jiao, Yunchao Wei, and Yao Zhao,  “Affinity attention graph neural network for weakly supervised semantic segmentation,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [26] Zhiyi Pan, Peng Jiang, Yunhai Wang, Changhe Tu, and Anthony G Cohn,  “Scribble-supervised semantic segmentation by uncertainty reduction on neural representation and self-supervision on neural eigenspace,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 7416–7425.
  * [27] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid,  “Segmenter: Transformer for semantic segmentation,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 7262–7272.
  * [28] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H.S. Torr, and Li Zhang,  “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,”  in CVPR, 2021.

##  Appendix A supplementary material

Table 6: Experimental results on Cityscapes and ADE20k datasets with different sparsity levels. Method | Params | Cityscapes | ADE20k
---|---|---|---
10% | 20% | 50% | 10% | 20% | 50%
DenseCRF Loss [7] | 70.0M | 57.4 | 61.8 | 70.9 | 31.9 | 33.8 | 38.4
TEL [1] | 70.0M | 61.9 | 66.9 | 72.2 | 33.8 | 35.5 | 40.0
Segformer | 61.4M | 58.4 | 60.5 | 70.4 | 39.8 | 41.4 | 45.3
Ours | 61.4M | 64.6 | 70.0 | 75.6 | 41.8 | 44.6 | 46.8
Fig. 5: Cityscapes val set segmentation results at 10%, 20%, 50% label
proportions.

To further demonstrate the effective of our approach, we conduct additional
experiments on the Cityscapes and ADE20k datasets, which have segmentation
annotations with varying levels of sparsity, including 10%, 20%, and 50% of
full labels. As shown in Table 6, our SASFormer outperforms state-of-the-art
methods at all sparsity levels. Moreover, we include quantitative results on
the Cityscapes validation set in Fig. 5.

Fig. 6: Visualization of inputs with reference points (red pentagram) and
patch attention maps at various levels.