sensor, as discussed in Sec. 3.4.1 and 3.4.2. Moreover, heat transfer by
athermal processes is greatly suppressed. As a result, the typical rise and
decay times of heat signals in the CUORE detectors are relatively slow, on the
order of a few hundred milliseconds and several seconds, respectively. This
inefficient heat flow was intended to function as an efficient low-pass filter
of the detector signals because it can help increase the energy resolution of
such a massive detector. Thermistors are a suitable choice for CUORE detectors
also because they have a large dynamic range of several MeV. On the other
hand, the slow response of the thermistor makes the detector particularly
vulnerable to microphonic noise, such as that caused by mechanical vibration,
typically with $1/f$ characteristics. Thus, it is critical to minimize the
vibration of the cryostat for CUORE. After all these considerations, the CUORE
detectors have demonstrated high energy resolution in their ROI (7.7 keV FWHM
near 2.5 MeV) [232] .

The CUORICINO and CUORE experiments resulted in the best detection limit for
$0\nu\beta\beta$ decay of 130Te to date (a half-life of 3.2$\times$1025 years,
which corresponds to the Majorana neutrino mass of 75 $<$ $m_{\beta\beta}$ $<$
350 meV [234]). The next-generation experiment CUPID aims not only to employ
scintillating crystals to improve the background through PID but also to
investigate other isotopes, such as 82Se and 100Mo. In particular, the CUPID-
Mo experiment based on Li2MoO4 crystals with NTD Ge thermistors has been
carried out to demonstrate the applicability of CUPID [235]. Together with
AMoRE, CUPID is expected to be a key player in the $0\nu\beta\beta$ searches
for the next decade.

####  4.2.3 Superconducting light detectors for $0\nu\beta\beta$ experiments

The PID capability is critical in $0\nu\beta\beta$ searches to efficiently
reject background signals. Although PID using only the heat channel has been
demonstrated in a few scintillating crystals [222, 236], in general,
simultaneous measurement of heat and light signals using a separate light
detector is superior to the single-channel-based PID. A separate light
detector may also enable dual-channel-based PID even with a nonscintillating
crystal such as TeO2 [181] because energetic electrons and ions in a
dielectric crystal generate a different amount of Cherenkov light. This
capability will broaden the selection of crystals and facilitate searching for
$0\nu\beta\beta$ in multiple isotopes. Moreover, having a light detector is
beneficial not only during $0\nu\beta\beta$ data acquisition but also during
detector development and characterization in the R&D stages. This is why many
of the current $0\nu\beta\beta$ search experiments (e.g., AMoRE, CUPID, and
CUPID-Mo) have adopted scintillating crystals and light detectors. However,
the total energy released in the light channel is often very weak. For
example, only a few percent of the total energy is carried away by
scintillation light from scintillating crystals, or in the case of a
nonscintillating crystal, Cherenkov photons with a total energy of 100–200 eV
are created by a few MeV electrons absorbed in the nonscintillating crystal
[181]. Therefore, the light detector should be carefully designed to maximize
the size of the light signals.

The light detector should be implemented with a sensor with high energy
resolution and high sensitivity. Superconducting sensors such as TES, MMC, and
KID are a good choice [237, 238, 239]. These sensors are also advantageous
because their intrinsic response time is very fast, and the overall response
time can be accelerated by making direct contact between the sensor and the
light absorber or by placing a phonon collector film between them, which makes
these detectors sensitive to both athermal and thermal phonons. The athermal
phonon signal contributes a large portion of the total heat signal in the
first few milliseconds and can be utilized to extract extra information that
the thermal phonon signal alone cannot reveal.

In addition, if the light signal is still too weak even with a high-
sensitivity sensor, NTL phonon amplification can be applied to the light
detectors, especially when the absorber is a Si or Ge wafer, as discussed in
Sec. 3.6. Large-gain signal amplification has been demonstrated with TES [240]
and MMC [183] as well as NTD Ge [181] devices.

###  4.3 Noncalorimetric approaches for wave-like DM

As axions and ALPs emerge as strong candidates for wave-like DM of sub-eV
mass, many detection technologies have been developed to search for these
hypothetical particles. One powerful approach is to detect excess
electromagnetic (photon) signals of a certain frequency converted from axions
or ALPs in a strong magnetic field, the so-called inverse Primakoff effect, as
illustrated in Figure 13. Since the possible mass range of wave-like DM spans
orders of magnitude, different detection schemes are required depending on the
target mass (frequency) of the DM. For the detection of high-frequency DM, it
is desirable to use a microwave cavity and quantum-limited amplifiers, which
are often composed of superconductors. However, for ultralow-mass (low
frequency) DM, the dimension of a microwave cavity that matches the frequency
of the excess photons becomes impractically large. In this case, a
superconducting lumped element circuit can be adopted to directly sense the
electromagnetic signals converted from ultralow-mass DM. Below, we review both
microwave cavity-based and lumped circuit-based detection methods.

####  4.3.1 Axion searches with microwave cavities

Figure 21: Schematic diagram of an axion haloscope based on a microwave
cavity. Under a strong magnetic field, an axion interacts with a virtual
photon (a quantum fluctuation of the magnetic field) and is converted to a
real photon due to the inverse Primakoff effect. The conversion rate depends
on the axion-photon coupling constant $g_{a\gamma\gamma}$. When the frequency
of the photon created inside the microwave cavity matches the resonance
frequency of the cavity, the detection sensitivity is maximized. Quantum-
measurement-based superconducting devices such as SQUIDs and JPAs can be
adopted for the frond-end amplifier.

The Primakoff effect describes the resonant production of neutral pseudoscalar
particles by photons interacting with an atomic nucleus [241]. Its reverse
process, the inverse Primakoff effect/scattering, provides an excellent way to
detect axions; an axion can decay into two real photons or be converted to a
photon through interaction with a virtual photon in an electromagnetic field.
The decay rate can be enhanced by a strong magnetic field.

The current most sensitive axion detectors were developed based on the axion
haloscope proposed by Sikivie [115]. As shown in Figure 21, in an axion
haloscope, a microwave cavity is adopted to detect photons produced via the
inverse Primakoff effect. The detection sensitivity of the axion-photon
conversion is enhanced when the resonant frequency of a microwave cavity
matches the axion mass. The resonant frequency of the cavity is adjustable
with a tuning rod. An antenna placed in the cavity detects the induced
microwave power in the cavity and transfers the signal into the cryogenic
receiver chain.

Because an extremely low conversion rate is expected, several experimental
strategies should be used in an axion haloscope. For instance, a microwave
cavity can be implemented with a large superconducting magnet to increase the
axion-photon conversion rate. A longer measurement time would increase the
chance of detecting rare conversions. Moreover, the signal-to-noise ratio
(SNR) of photon detection can be increased by decreasing the temperature of
the microwave cavity to 100 mK or even lower, thereby reducing the thermal
noise in the measurement system, or by adopting a high-gain amplifier.

Because the major noise in an axion haloscope is thermal noise, the SNR can be
described by the Dicke radiometer equation [242, 243, 244]:

$\mathrm{SNR}=\frac{P}{k_{\mathrm{B}}T_{\mathrm{sys}}}\sqrt{\frac{t}{b_{a}}},$
(19)

where $P$ is the signal power, $k_{\mathrm{B}}$ is the Boltzmann constant,
$T_{\mathrm{sys}}$ is the system noise temperature, $t$ is the integration
time, and $b_{a}$ is the axion signal bandwidth. For given experimental
parameters, the signal power $P$ is expressed as

$P\propto g_{a\gamma\gamma}^{2}B^{2}VQ_{\mathrm{L}},$ (20)

where $g_{a\gamma\gamma}$, $B$, $V$, and $Q_{\mathrm{L}}$ are the model-
dependent axion-photon coupling constant, magnetic field, cavity volume, and
loaded quality factor of the cavity, respectively [243]. The axion signal
bandwidth is determined by the energy dispersion of the axion and can be
expressed as $b_{a}=m_{a}v\delta v/c^{2}=m_{a}/Q_{a}$, where $m_{a}$ is the
axion mass, $v$ is the axion velocity, $\delta v$ is the velocity dispersion,
and $Q_{a}$ is the axion “quality factor” on the order of $10^{6}$ [244]. If
$Q_{L}$ is less than $Q_{a}$, the number of signal bandwidths that can be
simultaneously scanned is $N=Q_{a}/Q_{\mathrm{L}}$, and the cavity bandwidth
$b_{c}$ is $Nb_{a}$. Then, the time $\Delta t$ required to scan a frequency
range $\Delta f$ to achieve a given SNR is determined by

$\Delta t=\frac{\Delta f}{b_{c}}t\propto\Delta
f\frac{\mathrm{SNR}^{2}k_{\mathrm{B}}T_{\mathrm{sys}}^{2}}{{g_{a\gamma\gamma}}^{4}B^{4}V^{2}Q_{\mathrm{L}}Q_{a}}.$
(21)

This equation shows that the scan rate $r=\Delta f/\Delta t$ for a target SNR
is proportional to ${g_{a\gamma\gamma}}^{4}$, or equivalently, the sensitivity
of the experiment (the minimum $g_{a\gamma\gamma}$ that can be detected)
improves only with $r^{1/4}$, showing the difficulty of the experiment. Thus,
it is highly desired to also improve other experimental parameters in Eq.
(21), namely, the system noise temperature, the magnetic field strength, the
cavity volume, and the quality factor of the cavity.

The system noise temperature $T_{\mathrm{sys}}$ is approximately given by the
following:

$T_{\mathrm{sys}}=T_{\mathrm{phy}}+T_{A1}+\frac{T_{A2}}{G_{1}}+\frac{T_{A3}}{G_{1}G_{2}}+...,$
(22)

where $T_{\mathrm{phy}}$ is the physical temperature of the cavity and
$T_{Ai}$ and $G_{i}$ are the noise temperature and the gain of the $i$-th
stage amplifier in the receiver chain, respectively [86]. The noise from the
amplifiers, particularly that from the front-end amplifier, is a major
contributor to $T_{\mathrm{sys}}$. Thus, for efficient reduction in
$T_{\mathrm{sys}}$, a voltage-tunable microstrip SQUID amplifier (MSA) is
often employed as a front-end amplifier, whose resonance frequency can be
adjusted between 0.1 and 0.8 GHz with the length of the microstrip and the
tuning voltage [245]. The noise temperature of MSAs can reach as low as 30 mK
[85]. For higher-frequency measurements, Josephson parametric amplifiers
(JPAs) can be used. With a modulation frequency of twice the measuring
resonance frequency, phase-preserving amplification can be realized in a wide
frequency range up to 10 GHz [96].

The current axion-photon coupling limits of several axion haloscopes are shown
in Figure 5. Below, we introduce some of the major axion haloscope projects.

  * •

RBF and UF:
In the early period of experimental axion searches, Rochester-Brookhaven-
Fermilab (RBF) [88, 246] and the University of Florida (UF) [89] conducted
axion searches using axion haloscopes based on microwave cavities. The RBF
experiment probed the frequency range of 1.09–3.93 GHz with a 10 L copper
cavity, a 6 T magnet, and a cryogenic amplifier based on a GaAs heterojunction
field effect transistor (HFET), also known as a high-electron-mobility
transistor (HEMT). The UF experiment probed a similar frequency range with a 7
L copper cavity and a 7.5 T magnet. Using cryogenic HFET amplifiers with noise
temperatures as low as 3 K, the detection sensitivity was improved by more
than 10 times compared to that of the RBF experiment. Although the
$g_{a\gamma\gamma}$ limits obtained by these experiments were far above the
values predicted by the KSVZ or DFSZ models, they established the foundation
of axion search experiments using axion haloscopes.

  * •

ADMX:
Based on the successful demonstration by the two axion haloscope experiments
of RBF and UF, the Axion Dark Matter eXperiment (ADMX) collaboration built an
axion haloscope with sub-GHz target frequencies using a large 200 L cavity and
a 7.6 T magnet at Lawrence Livermore National Laboratory (LLNL) in 1995, and
this device operated until 2010 [247, 84]. Initially, HFETs with a noise
temperature slightly above 2 K were used as the first-stage amplifier, but
later, MSAs, whose noise temperatures were well below 100 mK, were adopted.
From this first-phase experiment, KSVZ axion-photon couplings between 461 and
860 MHz (1.9–3.53 $\mu$eV) were excluded at a 90% confidence level.
In 2010, ADMX moved to the Center for Experimental Physics and Astrophysics
(CENPA) at the University of Washington and achieved several major detector
upgrades [85, 87, 248]. One of the major upgrades was a new cooling system. In
the previous phase, the ADMX microwave cavity was cooled to approximately 2 K
by a pumped liquid helium cryostat. The new setup used a dilution refrigerator
(DR), which cooled the cavity and the first-stage amplifier to the 100 mK
range. Another major upgrade was the replacement of the first amplifier MSAs
with JPAs with a tunable resonance, which enabled searches for axions in the
675–800 MHz (2.79–3.31 $\mu$eV) range, extended from the $\sim 650$ MHz ($\sim
2.69$ $\mu$eV) region allowed when using the MSAs. The sensitivity of the ADMX
with a 140 L cavity in these frequency regions reached the DFSZ level.
Recently, the ADMX collaboration has developed a new “sidecar” cavity to
search for axions at higher frequencies (4–7 GHz) that are inaccessible with
the main cavity [86]. To increase the resonant frequency, the sidecar cavity
has a tiny volume of 0.38 L. The sidecar cavity experiment published new
limits in this high-frequency region [86] and is planning to broaden their
axion search range with several upgrades in their setup, including the
adoption of quantum-limited amplifiers, to be operated in tandem with the main
ADMX experiment.

  * •

HAYSTAC:
The Haloscope at Yale Sensitive to Axion CDM (HAYSTAC) branched off of ADMX to
probe high-frequency axions; HAYSTAC was initially called ADMX-HF. They
utilize a 2 L cavity, which is much smaller than ADMX’s 140 L cavity.
Recently, they reported a near-quantum-limited sensitivity of approximately
5.75 GHz ($\sim 23.8$ $\mu$eV) using a JPA [249] and a DR. Their first result
was near that of the KSVZ prediction except for ADMX [91]. Recently, the
detection sensitivity was further improved by using a quantum squeezed-state
receiver (SSR), a new technique that circumvents the quantum limit [250].
Using the SSR, the scan rate was doubled, and a detection sensitivity almost
reaching the KSVZ prediction was achieved at around 4.14 GHz ($\sim 17.1$
$\mu$eV).

  * •

CAPP:
The Center for Axion and Precision Physics (CAPP) at the Institute for Basic
Science (IBS) was established in 2013 to search for axions in a wide frequency
range of 1–10 GHz. Recently, CAPP published results from a series of pilot
experiments using three different microwave cavities and HEMTs. Lee et al.
reported the highest sensitivity limit at around 1.62 GHz ($\sim 6.7$ $\mu$eV)
using a 3.5 L cavity, a superconducting 8 T magnet, and a DR [92]. Jeong et
al. adopted an interesting cavity design called a multiple-cell cavity, a
highly efficient way to probe high-frequency regions with a large volume and a
relatively simple setup [93]. Using a double-cell cavity, a 9 T magnet and
He-3 cryogenic system, they obtained the highest sensitivity limit at around
3.25 GHz ($\sim 13.5$ $\mu$eV). More recently, Kwon et al. reported promising
results in the 2.46–2.75 GHz range using a setup composed of two half-
cavities, one with a volume of 0.59 L and the other with a volume of 1.12 L,
in the same cryostat with an 8 T magnet and a DR [94]. Especially at around
2.59 GHz ($\sim 10.7$ $\mu$eV), the limit of this setup was just above the
KSVZ prediction.

There are several ongoing R&D projects within CAPP to enhance the detection
sensitivity. These efforts include employing a much stronger magnet, further
advancing the multiple-cell cavity approach, and developing a superconducting
cavity and quantum-limited amplifiers [251]. For the stronger magnet, they
have been testing a new 12 T low-temperature superconducting (LTS) magnet with
a bore size of 32 cm and an 18 T high-temperature superconducting (HTS) magnet
with a bore size of 7 cm [252]. Although their 18 T magnet is the strongest
magnet ever adopted for axion haloscopes, its relatively small bore limits the
size of the cavity. They plan to upgrade the magnet to a 25 T HTS magnet with
a 10 cm bore that is under development [253, 254]. For the superconducting
cavity, they demonstrated a cavity quality factor 6 times higher than that of
their copper cavity using a polygon-shaped cavity with commercial YBCO tapes
covering the entire inner wall [255]. For the quantum-limited amplifiers, R&D
on JPAs is ongoing [256]. Employing JPAs as first-stage amplifiers instead of
HEMTs will decrease the noise temperature to under 1 K and improve the
sensitivity. With all the planned upgrades, CAPP is expected to reach the DFSZ
limit in the 0.7–3 GHz range using the 12 T magnet and the KSVZ limit at
frequencies up to 10 GHz using the 25 T magnet [251].

####  4.3.2 Direct detection with superconducting lumped element circuits

For the search for axions in the mass range below 1 $\mu$eV (or 0.2 GHz), the
use of microwave cavities is not practical because the dimensions of the
cavities with such a low resonance frequency become too large to allow
installation under a high magnetic field and cooling to cryogenic
temperatures. Instead, a lumped superconducting circuit can be employed to
directly sense the electromagnetic signal induced by the photons converted
from axions via the inverse Primakoff effect.

Figure 22 describes an axion detection scheme using a resonator circuit. A
superconducting pickup loop is placed in a strong magnetic field to boost
axion-photon conversion, as shown in Eq. (21). This scheme overcomes the
practical limit of placing a large microwave cavity inside the bore of a
fixed-size high-field magnet to search for low-mass (frequency) axions. The
other end of the pickup loop is coupled to a magnetometer such as a SQUID.
Since the coupling constant $g_{a\gamma\gamma}$ predicted by the KSVZ and DFSZ
models becomes weaker as the axion mass becomes lighter, the magnetometer
should be extremely sensitive for ultralow mass axion searches. The
technological maturity of superconducting sensors and superconducting
electronics may provide the required sensitivity over a wide frequency region
between DC and a few hundred MHz.

We introduce three projects searching for low-mass axion DM based on
superconducting circuits and superconducting sensor technologies.

Figure 22: A simplified measurement circuit for axion DM using a
superconducting LC circuit. The small coil of the circuit is coupled to a dc-
SQUID. The figure was taken from [257].

  * •

ABRACADABRA:
A Broadband/Resonant Approach to Cosmic Axion Detection with an Amplifying
B-field Ring Apparatus (ABRACADABRA) uses a toroidal magnet that induces a
large circular magnetic field and a superconducting pickup loop placed in the
center of the toroidal structure. The pickup coil is meant to measure
electromagnetic waves along the circular axial volume due to axion-photon
conversion under the magnetic field. A sensitive dc-SQUID is inductively
coupled to the pickup loop with a superconducting flux transformer to match
the inductance of the SQUID input coil and the pickup loop. The detector
apparatus also includes a current-injection coil along the axial line of the
toroidal coil for calibration purposes. Recently, ABRACADABRA-10 cm, a
prototype detector, was built with an active volume of 890 cm3 and a 10 cm 1 T
toroidal magnet, and its performance was tested. Broad-band detection was
carried out in the $m_{a}$ range of 0.3–8.1 neV, resulting in a
$g_{a\gamma\gamma}$ sensitivity as low as approximately 10-10 GeV-1 [258].

  * •

ADMX SLIC:
ADMX Superconducting LC Circuit Investigating Cold Axions (ADMX SLIC) is
another project using a superconducting lumped element circuit for axion DM
searches. The idea was proposed in Ref. [257], and the first measurement in
this project was carried out recently [259]. A superconducting LC circuit was
introduced into an ADMX-type 7 T magnet, which is a cylindrical magnet with a
large bore, as shown in Fig. 22. A loop antenna was placed inside the magnet
to measure the electromagnetic signals induced by the photons converted from
axions. The LC resonance circuit was equipped with a piezoelectric-driven
capacitor placed at the low-temperature stage to be able to tune the resonance
frequency of the circuit. The first measurement resulted in
$g_{a\gamma\gamma}$ sensitivities below 10-12 GeV-1 in the narrow ranges of
(1.7498–1.7519) $\times$ 10-7 eV, (1.7734–1.7738) $\times$ 10-7 eV, and
(1.8007–1.8015) $\times$ 10-7 eV [259].

  * •

DM Radio:
Dark Matter Radio (DM Radio) is another DM detector based on a tunable lumped-
element LC resonator [260]. The inductor is a superconducting coil wrapped
around a pickup sheath composed of Nb. The capacitor is tunable using movable
sapphire dielectrics placed inside a Nb capacitor. The excess power induced by
axion-photon conversion above the thermal noise level is sensed by dc-SQUIDs,
ac-SQUIDs, or parametric amplifiers depending on the target frequency. The
overall detection scheme is similar to that in other projects based on a
superconducting LC circuit, but the resonator of DM Radio is enclosed in a Nb
superconducting shield, which keeps the active detector volume free from
electromagnetic noise from the surroundings while having no effect on the
axions. This setup allows DM Radio to detect axions as well as ultralight
hidden photons (or dark photons). A hidden photon is a hypothetical particle
that interacts with normal photons via kinetic mixing with an extremely small
mixing angle $\epsilon$ [261]. Hidden photons have many phenomenological
similarities with axions, but unlike axions, their detection does not require
a strong magnetic field.

A proof-of-principle detector of 0.1 L was designed with an LC circuit with a
fixed resonance frequency of 492 kHz and a quality factor of $\sim 40,000$.
Since this detector did not have a magnet for axion-photon conversion, it was
not sensitive to axions but only to hidden photons. A test run with a total
integration time of 5.14 hours resulted in an upper limit on the hidden photon
mixing angle $\epsilon$ of $\sim$ 10-9 at around 2 neV [262]. As a next step,
a small-scale ($\sim 0.7$ L) detector called DM Radio-Pathfinder was built
with a tunable LC resonator with a quality factor of 148,000 [263]. A 1-year
scan of the DM Radio-Pathfinder over the 100 kHz to 10 MHz range is expected
to set an upper limit on $\epsilon$ far below $10^{-10}$ [262].

Based on the success of the pilot projects of ABRACADABRA and DM Radio, the
two projects are collaboratively working on a new DM Radio-50L detector. The
new detector will be operated under a magnetic field in a dilution
refrigerator. After a 1-year scan with a resonator quality factor of $10^{6}$
and a 0.5 T magnetic field over a wide range between 40 peV and 40 neV (10
kHz–10 MHz), the projected sensitivities for axions and hidden photons are
$g_{a\gamma\gamma}\sim 1\times 10^{-15}$ GeV-1 and
$\epsilon=10^{-15}$–$10^{-12}$, respectively [263]. A future generation
detector, DM Radio-M3, with a 1 m3 volume and a 4 T magnet is also proposed.
The projected axion sensitivity after a 3.5-year scan may reach the KSVZ
prediction above 3 MHz and the DFSZ prediction above 10 MHz [263].

##  5 Conclusion

The two open tasks of the direct detection of DM and the search for double
beta decay in neutrinoless mode are among the most important and immediate
experiments that deal with fundamental physics questions but have not received
plausible answers. Superconducting detectors are a type of detector with
extremely high sensitivity that have the potential to lead to groundbreaking
discoveries related to these tasks. These novel detectors are based on the
measurement of low-temperature quantum phenomena such as superconducting
transitions, quasiparticles, or spin dynamics. These detectors also rely on
state-of-the-art superconducting electronics such as SQUIDs and parametric
amplifiers to improve their signal-to-noise ratio, which is essential in rare
event search experiments. Many international experiments have been carried out
recently and are planned to be built with the superconducting detector
technologies. Many international experiments have been carried out recently
and are planned to be conducted with superconducting detector technologies.

Direct DM detection experiments can be divided into two groups depending on
the mass of the DM particles, which determines the type of detector technology
to be used. For particle-like DM heavier than $\sim$ 1 eV, superconducting
detectors look for a signal induced by the absorption or scattering of DM
particles in a target material. To date, the main goal of this approach has
been increasing the total mass of the target material and the energy
resolution of the detector. However, the focus is shifting toward the
detection of low-mass DM particles by lowering the detection energy threshold.
For much lighter (wave-like) DM particles, low-temperature quantum amplifiers
are adopted, where the wave-like DM particles are converted to photons or
induced in a microwave cavity or superconducting circuit.

In the search for the $0\nu\beta\beta$ process, the energy resolution and
timing resolution are important parameters for a large-scale low-temperature
experiment with crystal targets. In these experiments, superconducting
detectors are often equipped with two different sensors to provide dual-
channel detection of phonon and scintillation signals, an essential feature to
select the $0\nu\beta\beta$ signals from unwanted background signals.

In this review, we discussed the physics of the superconducting detectors used
in DM detection and $0\nu\beta\beta$ search experiments and provided brief
introductions to the history, present status, and future plans of several
selected experiments. This review should be used not only to understand and
utilize the technologies specific to the superconducting detectors but also to
inspire the development of other detection methods based on newly developed
novel characteristics and phenomena at low temperatures; the existing
superconducting detector technologies might be a barrier to the desired break-
through discoveries.

We would like to conclude with several examples of other experimental topics
in the field of astroparticle physics in which superconducting detectors have
played and will continue to play a major role. Several coherent elastic
neutrino-nucleus scattering (CE$\nu$NS) experiments have adopted TESs as their
main detection technology, leveraging their experience in developing detectors
for direct DM search experiments [264, 265]. In the direct measurement of
neutrino mass through electron capture, two active projects, namely, Electron
Capture 163Holmium experiment (ECHo) [266] and HOLMES [267], have adopted MMCs
and TESs, respectively. In the search for keV-scale sterile neutrinos as DM
candidates, the Beryllium Electron capture in Superconducting Tunnel junctions
(BeEST) experiment has adopted superconducting tunnel junction (STJ) detectors
[268], another type of superconducting detector with a less superior energy
resolution but much faster response than TESs and MMCs in general. Finally,
MMCs have also been also proposed as next-generation detectors of axion or ALP
search experiments in the keV range [269].

## References

## References

  * [1] Enss C (ed) 2005 Cryogenic Particle Detection (Springer-Verlag, Berlin Heidelberg)
  * [2] Mitsuda K et al. 2018 J. Low Temp. Phys. 193 85
  * [3] Faverzani M et al. 2020 J. Low Temp. Phys. 199
  * [4] Lita A E et al. 2008 Optics express 16 3032–3040
  * [5] Natarajan C M et al. 2012 Superconductor science and technology 25 063001
  * [6] Barcons X et al. 2015 Athena: The X-ray observatory to study the hot and energetic Universe vol 610
  * [7] Gaskin J A et al. 2019 Journal of Astronomical Telescopes, Instruments, and Systems 5 021001
  * [8] Ullom J N and Bennett D A 2015 Supercond. Sci. Technol. 28 084003
  * [9] Horansky R D et al. 2010 J. Appl. Phys. 107 044512
  * [10] Kim I et al. 2017 Supercond. Sci. Technol. 30 094005
  * [11] Lee S J et al. 2010 J. Phys. G: Nucl. Part. Phys. 37 055103
  * [12] Loidl M et al. 2014 Appl. Radiat. Isotopes 87 302–305
  * [13] Ranitzsch P O et al. 2019 J. Low Temp. Phys. 1–10
  * [14] Monfardini A et al. 2006 Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 559 346–348
  * [15] Gastaldo L et al. 2014 J. Low Temp. Phys. 176 876–884
  * [16] Alpert B et al. 2015 The European Physical Journal C 75 112
  * [17] Rubin V C and Ford Jr W K 1970 Astrophys. J. 159 379–403
  * [18] Rubin V C et al. 1980 Astrophys. J. 238 471
  * [19] Massey R et al. 2010 Rept. Prog. Phys. 73 086901 (Preprint 1001.1739)
  * [20] Hinshaw G et al. 2013 The Astrophysical Journal Supplement Series 208 19
  * [21] Aghanim N et al. 2020 Astronomy & Astrophysics 641 A6 URL https://doi.org/10.1051/0004-6361201833910
  * [22] Markevitch M et al. 2004 The Astrophysical Journal 606 819
  * [23] Allen S W et al. 2011 Annual Review of Astronomy and Astrophysics 49 409–470 (Preprint https://doi.org/10.1146/annurev-astro-081710-102514) URL https://doi.org/10.1146/annurev-astro-081710-102514
  * [24] Agnese R et al. (SuperCDMS) 2018 Phys. Rev. Lett. 120 061802 (Preprint 1708.08869)
  * [25] Abdelhameed A H et al. (CRESST) 2019 Phys. Rev. D 100 102002 (Preprint 1904.00498)
  * [26] Majorana E and Maiani L 2006 A symmetric theory of electrons and positrons (Berlin, Heidelberg: Springer Berlin Heidelberg) pp 201–233 ISBN 978-3-540-48095-2 URL https://doi.org/10.1007/978-3-540-48095-2_10
  * [27] Bertone G and Hooper D 2018 Rev. Mod. Phys. 90 045002 (Preprint 1605.04909) URL https://link.aps.org/doi/10.1103/RevModPhys.90.045002
  * [28] Kapteyn J C 1922 Astrophys. J. 55 302–328
  * [29] Oort J H 1932 Bulletin of the Astronomical Institutes of the Netherlands 6 249
  * [30] Zwicky F 1933 Helv. Phys. Acta 6 110–127
  * [31] Babcock H W 1939 Lick Obs. Bull. 41 498
  * [32] Oort J H 1940 Astrophysical Journal 91 273
  * [33] Zwicky F 1933 Physical Review 43 147
  * [34] Zwicky F 1937 The Astrophysical Journal 86 217
  * [35] Corbelli E and Salucci P 2000 Monthly Notices of the Royal Astronomical Society 311 441–447 ISSN 0035-8711 (Preprint https://academic.oup.com/mnras/article-pdf/311/2/441/2881340/311-2-441.pdf) URL https://doi.org/10.1046/j.1365-8711.2000.03075.x
  * [36] Mario De Leo 2018 Rotation curve of spiral galaxy messier 33 (triangulum) [Online; accessed: 04-28-2021] URL https://en.wikipedia.org/wiki/Galaxy_rotation_curve
  * [37] Freeman K 1970 Astrophys. J. 160 811
  * [38] Primack J R 1997 Dark matter and structure formation Midrasha Mathematicae in Jerusalem: Winter School in Dynamical Systems (Preprint astro-ph/9707285)
  * [39] Battaglieri M et al. 2017 arXiv preprint arXiv:1707.04591
  * [40] Hlozek R et al. 2015 Phys. Rev. D 91 103512 (Preprint 1410.2896)
  * [41] Armengaud E et al. 2017 Mon. Not. Roy. Astron. Soc. 471 4606–4614 (Preprint 1703.09126)
  * [42] Nori M et al. 2019 Mon. Not. Roy. Astron. Soc. 482 3227–3243 (Preprint 1809.09619)
  * [43] Nadler E O et al. 2019 Astrophys. J. Lett. 878 32 [Erratum: Astrophys.J.Lett. 897, L46 (2020), Erratum: Astrophys.J. 897, L46 (2020)] (Preprint 1904.10000)
  * [44] Monroy-Rodríguez M A and Allen C 2014 The Astrophysical Journal 790 159
  * [45] Brandt T D 2016 Astrophys. J. Lett. 824 L31 (Preprint 1605.03665)
  * [46] Jaeckel J and Ringwald A 2010 Ann. Rev. Nucl. Part. Sci. 60 405–437 (Preprint 1002.0329)
  * [47] Zel’dovich I D Y B N 1967 Soviet Astron. AJ (Engl. Transl. ), 10 602
  * [48] Hawking S 1971 Mon. Not. Roy. Astron. Soc. 152 75
  * [49] Pagels H and Primack J R 1982 Phys. Rev. Lett. 48 223
  * [50] Goldberg H 1983 Phys. Rev. Lett. 50 1419 [Erratum: Phys.Rev.Lett. 103, 099905 (2009)]
  * [51] Ellis J R et al. 1984 Nucl. Phys. B 238 453–476
  * [52] Jungman G et al. 1996 Phys. Rept. 267 195–373 (Preprint hep-ph/9506380)
  * [53] Steffen F D 2009 Eur. Phys. J. C 59 557–588 (Preprint 0811.3347)
  * [54] Bertone G et al. 2005 Phys. Rept. 405 279–390 (Preprint hep-ph/0404175)
  * [55] Kane G and Watson S 2008 Mod. Phys. Lett. A 23 2103–2123 (Preprint 0807.2244)
  * [56] Trevisani N (ATLAS, CMS) 2018 Universe 4 131
  * [57] Agnese R et al. (SuperCDMS) 2019 Phys. Rev. D 99 062001 (Preprint 1808.09098)
  * [58] Aprile E et al. (XENON) 2018 Phys. Rev. Lett. 121 111302 (Preprint 1805.12562)
  * [59] Bernabei R et al. 2013 The European Physical Journal C 73 1–11
  * [60] Benoit A et al. 2002 Physics Letters B 545 43–49
  * [61] Akerib D et al. 2004 Physical Review Letters 93 211301
  * [62] Angloher G et al. 2005 Astroparticle Physics 23 325–339
  * [63] Akerib D S et al. (LUX) 2017 Phys. Rev. Lett. 118 251302 (Preprint 1705.03380)
  * [64] Cui X et al. (PandaX-II) 2017 Phys. Rev. Lett. 119 181302 (Preprint 1708.06917)
  * [65] Adhikari G et al. 2018 Nature 564 83–+
  * [66] Ellis J and Olive K A 2010 Supersymmetric dark matter candidates (Cambridge University Press) p 142-163
  * [67] Servant G 2010 Dark matter at the electroweak scale: non-supersymmetric candidates (Cambridge University Press) p 164-189
  * [68] Alkhatib I et al. (SuperCDMS) 2021 Phys. Rev. Lett. 127(6) 061801 (Preprint 2007.14289) URL https://link.aps.org/doi/10.1103/PhysRevLett.127.061801
  * [69] Angloher G et al. (CRESST) 2017 Eur. Phys. J. C 77 637 (Preprint 1707.06749)
  * [70] Aprile E et al. (XENON) 2019 Phys. Rev. Lett. 123 251801 (Preprint 1907.11485)
  * [71] Agnes P et al. (DarkSide) 2018 Phys. Rev. Lett. 121 081307 (Preprint 1802.06994)
  * [72] Billard J et al. 2021 arXiv preprint arXiv:2104.07634
  * [73] CDMS 2020 Dark matter limit plotter URL https://supercdms.slac.stanford.edu/dark-matter-limit-plotter
  * [74] Duan G H et al. 2018 Phys. Lett. B 778 296–302 (Preprint 1711.03893)
  * [75] Kaplan D B 1992 Phys. Rev. Lett. 68 741–743
  * [76] Kaplan D E et al. 2009 Phys. Rev. D 79 115016 (Preprint 0901.4117)
  * [77] Petraki K and Volkas R R 2013 Int. J. Mod. Phys. A 28 1330028 (Preprint 1305.4939)
  * [78] Feng J L and Kumar J 2008 Phys. Rev. Lett. 101 231301 (Preprint 0803.4196)
  * [79] Gelmini G B 2017 Rept. Prog. Phys. 80 082201 (Preprint 1612.09137)
  * [80] Adhikari R et al. 2017 Journal of cosmology and astroparticle physics 2017 025
  * [81] Aguilar A et al. 2001 Phys. Rev. D 64 112007
  * [82] Aguilar-Arevalo A A et al. 2018 Physical review letters 121 221801
  * [83] Boyarsky A et al. 2019 Progress in Particle and Nuclear Physics 104 1–45
  * [84] Asztalos S J et al. (ADMX) 2010 Phys. Rev. Lett. 104(4) 041301 URL https://link.aps.org/doi/10.1103/PhysRevLett.104.041301
  * [85] Du N et al. (ADMX) 2018 Phys. Rev. Lett. 120(15) 151301 URL https://link.aps.org/doi/10.1103/PhysRevLett.120.151301
  * [86] Boutan C et al. (ADMX) 2018 Phys. Rev. Lett. 121 261302 (Preprint 1901.00920)
  * [87] Braine T et al. (ADMX) 2020 Phys. Rev. Lett. 124(10) 101303 URL https://link.aps.org/doi/10.1103/PhysRevLett.124.101303
  * [88] DePanfilis S et al. 1987 Phys. Rev. Lett. 59(7) 839–842 URL https://link.aps.org/doi/10.1103/PhysRevLett.59.839
  * [89] Hagmann C et al. 1990 Phys. Rev. D 42(4) 1297–1300 URL https://link.aps.org/doi/10.1103/PhysRevD.42.1297
  * [90] Backes K M et al. (HAYSTAC) 2021 Nature 590 238–242 (Preprint 2008.01853)
  * [91] Brubaker B M et al. 2017 Phys. Rev. Lett. 118(6) 061302 URL https://link.aps.org/doi/10.1103/PhysRevLett.118.061302
  * [92] Lee S et al. (CAPP) 2020 Phys. Rev. Lett. 124(10) 101802 URL https://link.aps.org/doi/10.1103/PhysRevLett.124.101802
  * [93] Jeong J et al. (CAPP) 2020 Phys. Rev. Lett. 125 221302 (Preprint 2008.10141)
  * [94] Kwon O et al. (CAPP) 2021 Phys. Rev. Lett. 126 191802 (Preprint 2012.10764)
  * [95] Alesini D et al. (QUAX–$a\gamma$) 2019 Phys. Rev. D 99 101101 (Preprint 1903.06547)
  * [96] Alesini D et al. 2021 Phys. Rev. D 103 102004 (Preprint 2012.09498)
  * [97] McAllister B T et al. (ORGAN) 2017 Phys. Dark Univ. 18 67–72 (Preprint 1706.00209)
  * [98] Gramolin A V et al. (SHAFT) 2021 Nature Phys. 17 79–84 (Preprint 2003.03348)
  * [99] Salemi C P et al. (ABRACADABRA) 2021 (Preprint 2102.06722)
  * [100] Anastassopoulos V et al. (CAST) 2017 Nature Phys. 13 584–590 (Preprint 1705.02290)
  * [101] Kim J E 1979 Phys. Rev. Lett. 43 103
  * [102] Shifman M A et al. 1980 Nucl. Phys. B 166 493–506
  * [103] Dine M et al. 1981 Phys. Lett. B 104 199–202
  * [104] Zhitnitsky A R 1980 Sov. J. Nucl. Phys. 31 260
  * [105] O’Hare C 2020 cajohare/axionlimits: Axionlimits URL https://doi.org/10.5281/zenodo.3932430
  * [106] Christenson J H et al. 1964 Physical Review Letters 13 138
  * [107] Alavi-Harati A et al. 1999 Physical Review Letters 83 22
  * [108] Aubert B et al. 2001 Physical review letters 87 091801
  * [109] Peccei R D and Quinn H R 1977 Phys. Rev. Lett. 38 1440–1443
  * [110] Peccei R D and Quinn H R 1977 Phys. Rev. D 16 1791–1797
  * [111] Wilczek F 1978 Phys. Rev. Lett. 40 279–282
  * [112] Weinberg S 1978 Phys. Rev. Lett. 40 223–226
  * [113] Lamoreaux S 2006 Nature 441 31–32
  * [114] Kim J E 1987 Physics Reports 150 1–177
  * [115] Sikivie P 1983 Phys. Rev. Lett. 51(16) 1415–1417 URL https://link.aps.org/doi/10.1103/PhysRevLett.51.1415
  * [116] O’Kelley S R et al. 2020 The microstrip squid amplifier in admx Microwave Cavities and Detectors for Axion Research (Springer) pp 23–36
  * [117] Kutlu c et al. 2021 Supercond. Sci. Technol. 34 085013 (Preprint 2101.08496)
  * [118] Choi K et al. 2020 arXiv:2012.05029
  * [119] Cowan C L et al. 1956 Science 124 103–104
  * [120] Fukuda Y et al. (Super-Kamiokande Collaboration) 1998 Phys. Rev. Lett. 81(8) 1562–1567
  * [121] Cleveland B T et al. 1998 The Astrophysical Journal 496 505
  * [122] Ahmad Q R et al. (SNO Collaboration) 2001 Phys. Rev. Lett. 87(7) 071301
  * [123] Fogli G L et al. 2012 Phys. Rev. D 86(1) 013012
  * [124] Goeppert-Mayer M 1935 Physical Review 48 512
  * [125] Barabash A 2015 Nuclear Physics A 935 52–64
  * [126] Majorana E 1937 Il Nuovo Cimento 14 171
  * [127] Racah G 1937 Il Nuovo Cimento 14 322
  * [128] Furry W H 1939 Phys. Rev. 56(12) 1184–1193
  * [129] Avignone F T et al. 2008 Rev. Mod. Phys. 80(2) 481–516
  * [130] Chun E J et al. 2018 International Journal of Modern Physics A 33 1842005
  * [131] Kotila J and Iachello F 2012 Phys. Rev. C 85(3) 034316
  * [132] Engel J and Menéndez J 2017 Reports on Progress in Physics 80 046301
  * [133] Deppisch F F et al. 2012 Journal of Physics G: Nuclear and Particle Physics 39 124007
  * [134] Fano U 1947 Phys. Rev. 72(1) 26–29 URL http://link.aps.org/doi/10.1103/PhysRev.72.26
  * [135] Steinbauer E et al. 1994 Nucl. Instr. and Meth. A 85 642–649
  * [136] de Wit M et al. 2020 Optimizing the geometry of AC-biased Ti/Au X-ray calorimeters applied Superconductivity Conference 2020
  * [137] Smith S J et al. 2012 Journal of Low Temperature Physics 167 168–175
  * [138] Kempf S et al. 2018 Journal of Low Temperature Physics 193 365–379 ISSN 1573-7357 URL https://doi.org/10.1007/s10909-018-1891-6
  * [139] Fleischmann A et al. 2005 Topics in Applied Physics 99 151
  * [140] Kivelson S and Rokhsar D 1990 Physical Review B 41 11693
  * [141] Colling P et al. 1995 Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 354 408–416
  * [142] Martinis J M et al. 2000 Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 444 23–27
  * [143] Lee S et al. 2015 Applied Physics Letters 107 223503
  * [144] Yoho M D et al. 2020 Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 969 164056
  * [145] Hoover A S et al. 2015 Analytical chemistry 87 3996–4000
  * [146] Kim Y H et al. 2004 Nucl. Instr. and Meth. A 520 208–211
  * [147] Lee S J et al. 2011 Astropart. Phys. 34 732 – 737
  * [148] Kim G B et al. 2017 Astropart. Phys. 91 105–112
  * [149] Nagler P et al. 2012 Journal of Low Temperature Physics 167 455–460
  * [150] Enss C et al. 2000 J. Low Temp. Phys. 121 137–176
  * [151] Fleischmann A et al. 2009 Metallic magnetic calorimeters AIP Conference Proceedings vol 1185 (AIP) pp 571–578
  * [152] Kim S G et al. 2021 IEEE Trans. Appl. Superconductivity 31 1–5
  * [153] Fleischmann A et al. 2000 Nucl. Instr. and Meth. A 444 100–103
  * [154] Rotzinger H et al. 2008 J. Low Temp. Phys. 151 351–356
  * [155] Fleischmann A et al. 2009 Metallic magnetic calorimeters AIP Conference Proceedings vol 1185 (American Institute of Physics) pp 571–578
  * [156] Chernyak D et al. 2012 The European Physical Journal C 72 1–6
  * [157] Day P K et al. 2003 Nature 425 817–821
  * [158] Mazin B A 2020 arXiv preprint arXiv:2004.14576
  * [159] O’Connor E et al. 2019 New Astronomy Reviews 87 101526
  * [160] Lee K et al. 2020 Journal of Low Temperature Physics 200 384–391
  * [161] Moore D C et al. 2012 Appl. Phys. Lett. 100 232601
  * [162] Chang Y Y et al. 2018 J. Low Temp. Phys. 193 1199–1205
  * [163] Colantoni I et al. 2020 Journal of Low Temperature Physics 1–5
  * [164] Cardani L et al. 2021 arXiv preprint arXiv:2104.06850
  * [165] Éfros A L and Shklovskii B I 1975 Journal of Physics C: Solid State Physics 8 L49
  * [166] McCammon D 2005 Topics in Applied Physics 99 35
  * [167] Silver E et al. 2005 Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 545 683–689
  * [168] Porter F et al. 2006 Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 559 436–438
  * [169] Sikorsky T et al. 2020 Physical Review Letters 125(14) 142503 URL https://link.aps.org/doi/10.1103/PhysRevLett.125.142503
  * [170] Swartz E T and Pohl R O 1989 Rev. Mod. Phys. 61 605
  * [171] Little W A 1959 Canadian Journal of Physics 37 334–349
  * [172] Cheeke J et al. 1976 Canadian Journal of Physics 54 1749–1771
  * [173] Pobell F 2007 Matter and methods at low temperatures vol 2 (Springer)
  * [174] Schoenlein R et al. 1987 Physical Review Letters 58 1680
  * [175] Allen P B 1987 Phys. Rev. Lett. 59 1460
  * [176] Maris H J and Tamura S I 1993 Physical Review B 47 727
  * [177] Tamura S I and Maris H 1985 Physical Review B 31 2595
  * [178] Brandt D et al. 2012 Journal of Low Temperature Physics 167 485–490
  * [179] Isaila C et al. 2012 Physics Letters B 716 160–164
  * [180] Defay X et al. 2016 Journal of Low Temperature Physics 184 274–279
  * [181] Artusa D R et al. 2017 Physics Letters B 767 321–329
  * [182] Romani R et al. 2018 Applied Physics Letters 112 043501
  * [183] Jeon J et al. 2020 Journal of Low Temperature Physics 199 883–890
  * [184] Ahmed Z et al. 2010 Science 327 1619
  * [185] Armengaud E et al. 2010 Physics Letters B 687 294–298
  * [186] Agnese R et al. 2018 Physical review letters 121 051301
  * [187] Tretyak V 2010 Astroparticle Physics 33 40–53
  * [188] Abrams D et al. (CDMS) 2002 Phys. Rev. D 66 122003 (Preprint astro-ph/0203500)
  * [189] Akerib D et al. 2005 Physical Review D 72 052009
  * [190] Shutt T et al. 2000 Nucl. Instrum. Meth. A 444 340–344
  * [191] Agnese R et al. (SuperCDMSSoudan) 2013 Appl. Phys. Lett. 103 164105 (Preprint 1305.2405)
  * [192] Fink C et al. 2021 Applied Physics Letters 118 022601
  * [193] Petricca F et al. 2020 First results on low-mass dark matter from the CRESST-III experiment Journal of Physics: Conference Series vol 1342 (IOP Publishing) p 012076
  * [194] Ferger P et al. 1994 Physics Letters B 323 95–98
  * [195] Berman R et al. 1955 Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences 231 130–144
  * [196] Seidel W et al. 1990 Physics Letters B 236 483–487
  * [197] Meier O et al. 2000 Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 444 350–352
  * [198] Altmann M et al. 2001 arXiv preprint astro-ph/0106314
  * [199] Angloher G et al. 2002 Astroparticle Physics 18 43–55
  * [200] Meunier P et al. 1999 Applied physics letters 75 1335–1337
  * [201] Angloher G et al. 2009 Astroparticle Physics 31 270–276
  * [202] Brown A et al. 2012 Physical Review D 85 021301
  * [203] Angloher G et al. 2012 The European Physical Journal C 72 1971
  * [204] Angloher G et al. 2016 The European Physical Journal C 76 1–8
  * [205] Angloher G et al. 2019 The European Physical Journal C 79 1–7
  * [206] Abdelhameed A et al. 2019 Physical Review D 100 102002
  * [207] Sanglard V et al. 2005 Physical Review D 71 122002
  * [208] Armengaud E et al. 2011 Physics Letters B 702 329–335
  * [209] Gascon J et al. 2020 Low-mass WIMP searches with the EDELWEISS experiment Journal of Physics: Conference Series vol 1342 (IOP Publishing) p 012064
  * [210] Defay X et al. 2008 Journal of Low Temperature Physics 151 896–901
  * [211] Broniatowski A et al. 2008 Journal of Low Temperature Physics 151 830–834
  * [212] Smith C 1996 Reports on Progress in Physics 59 235
  * [213] Guarcello C et al. 2019 Physical Review Applied 11 054074
  * [214] Lee G H et al. 2020 Nature 586 42–46
  * [215] Walsh E D et al. 2021 Science 372 409–412
  * [216] Geim A K 2009 science 324 1530–1534
  * [217] Kim D et al. 2020 arXiv e-prints arXiv–2002
  * [218] Echternach P et al. 2018 Nature Astronomy 2 90–97
  * [219] Dolinski M J et al. 2019 Annual Review of Nuclear and Particle Science 69 219–251
  * [220] Kim Y H 2020 arXiv:2004.02510
  * [221] Kim G B et al. 2015 Advances in High Energy Physics 2015 817530
  * [222] Kim G B et al. 2017 Astroparticle Physics 91 105 – 112
  * [223] Kim G B et al. 2014 J. Low Temp. Phys. 176 637–643
  * [224] Lee C et al. 2018 Journal of Low Temperature Physics 193 786–792
  * [225] Alenkov V et al. 2019 The European Physical Journal C 79 791 ISSN 1434-6052
  * [226] Seo K 2019 The Status of AMoRE 0$\nu\beta\beta$ Experiment Talk at TAUP-2019 Toyama, Japan
  * [227] Tetsuno K et al. 2020 Status of 48Ca double beta decay search and its future prospect in CANDLES Journal of Physics: Conference Series vol 1468 (IOP Publishing) p 012132
  * [228] Li X et al. 2020 Study of a large CaF2(Eu) scintillating bolometer for neutrinoless double beta decay Journal of Physics: Conference Series vol 1468 (IOP Publishing) p 012116
  * [229] Fiorini E and Niinikoski T 1984 Nuclear Instruments and Methods in Physics Research 224 83–88
  * [230] Arnaboldi C et al. 2003 Astroparticle Physics 20 91–110
  * [231] Gorla P et al. 2006 Nuclear Physics B-Proceedings Supplements 150 214–218
  * [232] Alduino C et al. 2018 Phys. Rev. Lett. 120 132501
  * [233] Beretta M and Pagnanini L 2021 Applied Sciences 11 1606
  * [234] Adams D et al. 2020 Physical review letters 124 122501
  * [235] Armengaud E et al. 2021 Physical review letters 126 181802
  * [236] Armengaud E et al. 2020 The European Physical Journal C 80 1–15
  * [237] Rothe J et al. 2018 Journal of Low Temperature Physics 193 1160–1166
  * [238] Lee H J et al. 2015 Nucl. Instr. and Meth. A 784 508–512
  * [239] Casali N et al. 2019 The European Physical Journal C 79 1–7
  * [240] Stark M et al. 2005 Nucl. Instr. and Meth. A 545 738 – 743 ISSN 0168-9002 URL http://www.sciencedirect.com/science/article/pii/S0168900205007679
  * [241] Primakoff H 1951 Phys. Rev. 81(5) 899–899 URL https://link.aps.org/doi/10.1103/PhysRev.81.899
  * [242] Dicke R H 1946 Review of Scientific Instruments 17 268–275 URL https://doi.org/10.1063/1.1770483
  * [243] Stern I et al. 2015 Rev. Sci. Instrum. 86 123305 (Preprint 1603.06990)
  * [244] Sikivie P 2021 Rev. Mod. Phys. 93 015004 (Preprint 2003.02206)
  * [245] Clarke J et al. 2001 NATO Sci. Ser. E 375 473–504
  * [246] Wuensch W et al. 1989 Physical Review D 40 3153
  * [247] Hagmann C et al. (ADMX) 1998 Phys. Rev. Lett. 80 2043–2046 (Preprint astro-ph/9801286)
  * [248] Bartram C et al. (ADMX) 2021 Phys. Rev. D 103 032002 (Preprint 2010.06183)
  * [249] Zhong L et al. 2018 Physical Review D 97 092001
  * [250] Backes K et al. 2021 Nature 590 238–242
  * [251] Semertzidis Y K et al. 2019 arXiv preprint arXiv:1910.11591
  * [252] Kim J et al. 2020 Rev. Sci. Instrum. 91 023314
  * [253] Kim J G et al. 2019 IEEE Transactions on Applied Superconductivity 29 1–5
  * [254] Gupta R et al. 2019 IEEE Transactions on Applied Superconductivity 29 1–5
  * [255] Ahn D et al. 2020 arXiv preprint arXiv:2002.08769
  * [256] Kutlu Ç et al. 2021 Superconductor Science and Technology
  * [257] Sikivie P et al. 2014 Physical review letters 112 131301
  * [258] Ouellet J L et al. 2019 Phys. Rev. Lett. 122 121802
  * [259] Crisosto N et al. 2020 Phys. Rev. Lett. 124 241101
  * [260] Silva-Feaver M et al. 2016 IEEE Trans. Appl. Superconductivity 27 1–4
  * [261] Graham P W et al. 2016 Physical Review D 93 103520
  * [262] Phipps A et al. 2020 Exclusion limits on hidden-photon dark matter near 2 neV from a fixed-frequency superconducting lumped-element resonator Microwave Cavities and Detectors for Axion Research (Springer) pp 139–145
  * [263] Chaudhuri S 2019 The dark matter radio: a quantum-enhanced search for QCD axion dark matter Dissertation Stanford University Stanford
  * [264] Billard J et al. 2017 Journal of Physics G: Nuclear and Particle Physics 44 105101
  * [265] Rothe J et al. 2020 J. Low Temp. Phys. 199 433–440
  * [266] Gastaldo L et al. 2014 J. Low Temp. Phys. 176 876–884
  * [267] Alpert B et al. 2015 The European Physical Journal C 75 1–11
  * [268] Friedrich S et al. 2021 Phys. Rev. Lett. 126 021803
  * [269] Unger D et al. 2021 Journal of Instrumentation 16 P06006


# ConvFormer: Combining CNN and Transformer for Medical Image Segmentation

###### Abstract

Convolutional neural network (CNN) based methods have achieved great successes
in medical image segmentation, but their capability to learn global
representations is still limited due to using small effective receptive fields
of convolution operations. Transformer based methods are capable of modelling
long-range dependencies of information for capturing global representations,
yet their ability to model local context is lacking. Integrating CNN and
Transformer to learn both local and global representations while exploring
multi-scale features is instrumental in further improving medical image
segmentation. In this paper, we propose a hierarchical CNN and Transformer
hybrid architecture, called ConvFormer, for medical image segmentation.
ConvFormer is based on several simple yet effective designs. (1) A feed
forward module of Deformable Transformer (DeTrans) is re-designed to introduce
local information, called Enhanced DeTrans. (2) A residual-shaped hybrid stem
based on a combination of convolutions and Enhanced DeTrans is developed to
capture both local and global representations to enhance representation
ability. (3) Our encoder utilizes the residual-shaped hybrid stem in a
hierarchical manner to generate feature maps in different scales, and an
additional Enhanced DeTrans encoder with residual connections is built to
exploit multi-scale features with feature maps of different scales as input.
Experiments on several datasets show that our ConvFormer, trained from
scratch, outperforms various CNN- or Transformer-based architectures,
achieving state-of-the-art performance.

##  1 Introduction

Image segmentation is a central problem in medical image analysis.
Convolutional neural networks (CNNs), especially fully convolutional networks
(FCNs), have become predominant approaches for this problem (e.g., U-Net [1],
UNet 3+ [2], etc). Despite their successes, CNNs still have yet to address the
limitation in learning long-range dependencies (global information) to see a
“big picture” due to limited effective receptive fields (ERFs) of convolution
(Conv) operations. Attempts were made to enlarge ERFs, such as utilizing
atrous Convs with different dilated rates (e.g., [3]), applying pyramid
pooling (e.g., [4]), and designing large kernels (e.g., [5]). Although these
methods enlarged ERFs to some extent, they still suffered from limited ERFs,
yielding sub-optimal image segmentation accuracy.

Recently, Transformers (e.g., vision Transformer [6]) became a de-facto choice
for modelling long-range dependencies in computer vision, inspired by their
success with self-attention mechanism in natural language processing. Compared
to CNN methods, Transformer models have larger receptive fields and excel at
learning global information. But, they also have drawbacks, e.g., high
computation cost, slow convergence, and short of CNN’s inductive biases. Two
types of methods attempted to reduce their computation cost. (1) Limiting
self-attention to local windows (e.g., [7, 8]). (2) Downsampling the key and
value feature maps (e.g., [9]). Though effective in capturing global
information, these Transformer-based methods still yielded unsatisfactory
performance due to deficiency in learning local information.

In medical image segmentation, efforts were made to combine Transformer and
CNN to learn both local and global representations. In [10], Transformer was
utilized as a supplement, appended to the last Conv block of the CNN encoder
to learn global information. MedT [11] exploited local and global information
by employing two branches, where a gated axial Transformer was applied to
explore global information and CNN was leveraged to learn local information.
CoTr [12] employed Deformable Transformer (DeTrans) as an additional encoder
for exploring multi-scale information from multi-scale feature maps for 3D
medical image segmentation. MISSFormer [13] introduced a Conv layer to
Transformer to enhance its capability to learn local information. UNETR [14]
proposed a U-shaped encoder-decoder network, where Transformer was used as the
encoder to capture global multi-scale information and a CNN decoder was used
to compute final segmentation output. Although achieving promising
performances, these methods still incurred several drawbacks. (1) Limited
capability to learn both local and global representations due to ineffective
integration of CNN and Transformer. For example, CoTr [12] did not apply Conv
to DeTrans for effectively learning local information. TransUNet [10] simply
appended Transformer to a Conv block in the encoder. UNETR [14] did not
leverage Conv in the encoder. (2) Deficiency in capturing multi-scale
information. For instance, MedT [11] did not explore multi-scale information
in its global Transformer branch, and such drawbacks in learning both local
and global representations and capturing multi-scale information led to sub-
optimal segmentation.

In this paper, we propose a new hierarchical CNN and Transformer hybrid
architecture, called ConvFormer, to capture both local and global
representations while exploiting multi-scale features for medical image
segmentation. Specifically, ConvFormer is based on several key designs. (1) We
re-design the feed forward module of DeTrans to introduce local information,
called Enhanced DeTrans. (2) We develop a residual-shaped hybrid stem based on
a combination of Convs and Enhanced DeTrans to capture both local and global
representations to enhance representation ability. (3) The residual-shaped
hybrid stem is utilized in the encoder in a hierarchical manner to generate
feature maps in different scales, and an additional Enhanced DeTrans encoder
with residual connections is built to explore multi-scale features with
feature maps of different scales as input. (4) An enhanced positional encoding
(EPE) introduces a Conv layer to the fixed sinusoidal encoding method [15] to
improve adaptability and flexibility.

Extensive experiments on two public datasets (2017 ISIC Skin Lesion
segmentation (2D) [16] and MM-WHS CT (3D) [17]) and one in-house dataset
(lymph node segmentation (2D)) show that our ConvFormer, trained from scratch,
outperforms various known CNN-based or Transformer-based methods, achieving
state-of-the-art performance.

Fig. 1: (a) An overview of our Enhanced DeTrans with the proposed enhanced
positional encoding (EPE) and Conv-based feed forward module (FFM) (shown in
gray color). (b) Our ConvFormer: The two big boxes on the left are for the
encoder and decoder, respectively (best viewed in color).

##  2 Method

Fig. 1 shows the architecture of our ConvFormer that contains three main
components: (1) Enhanced DeTrans that captures global representations with
local information, using a Conv-based feed forward module (FFM); (2) the
residual-shaped hybrid stem that extracts local and global representations in
different scales that are fed to an additional Enhanced DeTrans encoder to
explore multi-scale features; (3) enhanced positional encoding (EPE) that
improves the adaptability and flexibility of the fixed sinusoidal encoding
method [15].

###  2.1 Enhanced DeTrans

FFM is an essential component of Transformer to enhance the representation
ability. The FFM proposed in DeTrans [8, 12] consists of a multilayer
perceptron (MLP) composing of two linear layers separated by GELU, as follows:

$\text{FFM}(x)=\text{LN}\left(\text{GELU}(xW_{1}+b_{1})W_{2}+b_{2}+x\right),$
(1)

where $x$ is an input feature map, $W_{1}$ and $W_{2}$ are weights of the two
linear layers respectively, $b_{1}$ and $b_{2}$ are bias terms, and LN denotes
layer norm. For simplicity, Dropouts are omitted. Though effective, this
design does not have good capability to learn local information, which is
critical in dense predictions.

To overcome this limitation and enable FFM to introduce local information to
the global representations captured by multi-scale multi-head self attention
(MS-MHSA), we resort to a Conv layer. Specifically, we insert depth-wise
convolution (DW Conv) to the end of FFM. The resulted FFM is called Conv-based
FFM. Note that we leverage DW Conv instead of Conv for reducing computation
cost. As a result, the proposed Conv-based FFM inherits the merit of both CNN
and DeTrans in learning local and global representations.

To adapt DW Conv to FFM, we first reshape the input 1D sequences captured by
FFM to 2D/3D feature maps, apply DW Conv to the reshaped feature maps to learn
local information, and reshape the result back to 1D sequences as output:

$\displaystyle\text{Conv-based FFM}(x)$
$\displaystyle=\text{reshape}\left(\text{DW
Conv}\left(\text{reshape}(\text{FFM}(x))\right)\right),$ (2)

where $x$ is an input feature map. Note that our Conv-based FFM is capable of
dealing with multi-scale feature map input, which is important for accurate
segmentation. DW Conv is shared when processing multi-scale feature maps.

We replace the FFM of DeTrans by our Conv-based FFM, and the resulted DeTrans
is called Enhanced DeTrans. Enhanced DeTrans is capable of capturing both
local and global information by combining Conv and Transformer.

###  2.2 Residual-shaped Hybrid Stem

Stacking Transformer to the last Conv block as a global feature extractor is a
common way to combine CNN/Transformer in encoder to learn local and global
representations. For example, in TransUNet [10] and CoTr [12], Transformer was
followed by a Conv block to capture global information for local features
obtained by Conv blocks. Although such methods might enhance the models’
representation ability on learning local and global information to some
extent, they did not exploit both local and global features effectively. More
importantly, they did not generate and explore multi-scale local and global
features, which are highly important to handle datasets with large variations
in object size, shape, and texture, which are common characteristics of
medical image datasets.

To effectively capture both local and global representations, we propose the
residual-shaped hybrid stem, which consists of two key designs (see the
leftmost box in Fig. 1(b)). (1) Residual connections are introduced to the
stacked Convs for feature diversity and delivery. (2) Local representations
captured by the stacked Convs are combined with global representations learned
by Enhanced DeTrans via an add operation. These two designs offer two
compelling advantages: (i) The introduced residual connections are critical
for avoiding feature collapse (i.e., feature diversity is continuously reduced
as the layer depth increases); (ii) the captured local and global
representations can be better fused via add operation. The improvement (e.g.,
in Table 4, F1 improved by $1.0\%$, $p<0.05$, t-test) shows the importance of
the introduced residual connections.

We use the residual-shaped hybrid stem in the encoder in a hierarchical
manner, so that it can generate global and local representations in different
scales. Given an input image, we can obtain hierarchical global and local
representations with different resolutions, which are then fed to an
additional Enhanced DeTrans encoder to exploit multi-scale representations.
Different from CoTr [12] which utilized DeTrans to exploit multi-scale
features, we add residual connections to our Enhanced DeTrans to reuse the
local representations captured by the stacked Convs and consolidate the
captured local and global representations.

The architecture of our ConvFormer is shown in Fig. 1. Following the
architecture of CoTr [12], ConvFormer has four encoder and decoder stages,
with a Conv stem and three residual-shaped hybrid stems in encoder, a
deconvolution (DeConv) stem and three decoder stems in decoder, and an
additional Enhanced DeTrans encoder.

###  2.3 Enhanced Positional Encoding (EPE)

In Transformer, we first flatten input feature maps into 1D sequences. But,
this flattening process may cause loss of spatial information that is critical
for segmentation. Known methods [12, 15] employed fixed sinusoidal encoding to
supplement flattened sequences with position information. In particular, sine
and cosine functions of different frequencies were used to compute the
positional coordinates of each dimension:

$\displaystyle\text{PE}_{(pos,2i)}$
$\displaystyle=\sin\left(pos/10000^{2i/C}\right),$ (3)
$\displaystyle\text{PE}_{(pos,2i+1)}$
$\displaystyle=\cos\left(pos/10000^{2i/C}\right),$ (4)

where $pos$ is for position, $i$ is the dimension, and $C$ is a constant.
However, such a fixed sinusoidal encoding method lacks adaptability and
flexibility, as the code (e.g., frequencies) is predefined.

To improve adaptability and flexibility, we introduce DW Conv to the fixed
sinusoidal encoding [15], since a Conv kernel naturally encodes pixel locality
and semantic continuity with adaptability and flexibility. The resulted
positional encoding method is called enhanced positional encoding (EPE).
Specifically, it uses two branches: one branch applies the fixed sinusoidal
encoding [15] to learn position information, and the other uses DW Conv to
capture position information. The two branches are combined by an add
operation:

$x^{\prime}=\text{fixed sinusoidal encoding}(x)+\text{ReLU}(\text{BN}(\text{DW
Conv}(x))),$ (5)

where $x$ is an input feature map, $x^{\prime}$ denotes the output feature map
embedded with position information, and BN is batch normalization. The
improvement (in Table 4, F1 improved by $0.4\%$, $p<0.05$, t-test)
demonstrates the importance of EPE.

##  3 Experiments and Results

Datasets. (1) The lymph node segmentation ultrasound dataset: This in-house
dataset is for segmenting lymph nodes in 2D ultrasound images. It contains 137
training and 100 test images. (2) The 2017 MM-WHS CT dataset: This public
dataset [17] is for segmenting seven cardiac structures, the left/right
ventricle blood cavity (LV/RV), left/right atrium blood cavity (LA/RA),
myocardium of the left ventricle (LV-myo), ascending aorta (AO), and pulmonary
artery (PA). It contains 20 unpaired 3D CT images, which are randomly split
into 16 images and 4 images for training and testing, following [18]. (3) The
2017 ISIC skin lesion segmentation dataset: This public dataset [16] is for
segmenting lesion boundaries in 2D dermoscopic images. It contains 2000
training, 150 validation, and 600 test images.

Implementation Details. Our ConvFormer is implemented with PyTorch, and is
trained on an NVIDIA Tesla P100 Graphics Card with 16GB GPU memory using the
AdamW [19] optimizer with weight decay $0.005$. We apply the “poly” learning
rate policy with an initial learning rate of $2e-4$. The maximum number of
iterations is $100$k for the lymph node and 2017 ISIC skin lesion datasets,
and $240$k for the 2017 MM-WHS CT dataset (using about $34$, $48$, and $96$
training hours, respectively). To avoid overfitting, standard data
augmentation (e.g., random flip, crop, etc) is applied.

Table 1: Quantitative results of various models on the lymph node dataset. The
reported values are average$\pm$STD for 3 runs with different random seeds.
The best results are in bold.

Method | IoU | Precision | Recall | F1 Score
---|---|---|---|---
U-Net [1] | 0.661 | 0.834 | 0.761 | 0.796
Zhang et al. [20] | 0.810 | 0.901 | 0.889 | 0.895
$k$CBAC-Net [21] | 0.829 | 0.909 | 0.904 | 0.906
ConvFormer (Ours) |  0.845$\pm$0.002 |  0.925$\pm$0.002 |  0.907$\pm$0.002 |  0.916$\pm$0.002

Experimental Results. Table 1 reports quantitative results on the lymph node
dataset, showing that our ConvFormer outperforms the known methods by a clear
margin in all the evaluation measures. In particular, ConvFormer outperforms
the state-of-the-art (SOTA) method, $k$CBAC-Net [21], by 1.6% and 1.0% in IoU
and F1, respectively, achieving new SOTA performances. Table 2 shows
quantitative results on the 2017 MM-WHS CT 3D dataset. Our ConvFormer
outperforms the best-known CNN-based (HFA-Net [18], $k$CBAC-Net [21]) and
Transformer-based (CoTr [12], UNETR [14]) methods, achieving new SOTA
performances. Specifically, ConvFormer outperforms HFA-Net [18], $k$CBAC-Net
[21], CoTr [12], and UNETR [14] by 1.6%, 0.6%, 1.4%, and 1.0% in Dice score on
average, respectively. Table 3 gives quantitative results on the 2017 ISIC
skin lesion dataset. Our ConvFormer attains accuracy gain of 3.2% in Jaccard
Index over the 2017 ISIC Challenge winner [22] and slightly outperforms the
SOTA method, $k$CBAC-Net [21], achieving new SOTA performances. Note that, (1)
our ConvFormer shows excellent generalization on both 2D and 3D datasets, on
ultrasound, dermoscopic, and CT images, outperforming previous SOTA methods;
(2) ConvFormer achieves competitive performances without pre-training,
different from most of the known Transformer-based models. All these suggest
that our ConvFormer is a promising and robust method suitable for medical
image segmentation tasks on datasets of different imaging characteristics and
modalities, and it is capable of learning effective local, global, and multi-
scale representations. Fig. 2 shows some qualitative results.

Table 2: Quantitative results of different models on the 2017 MM-WHS CT
dataset. “—” means that the results were not reported by the original papers,
“Para.” means the number of parameters of the model, and “HDF” represents
Hausdorff.

Method | Para. | Metric | LV | RV | LA | RA |  LV -myo | AO | PA | Mean
---|---|---|---|---|---|---|---|---|---|---
Payer et al. [23] | — | Dice | 0.918 | 0.909 | 0.929 | 0.888 | 0.881 | 0.933 | 0.840 | 0.900
Dou et al. [24] | — | Dice | 0.888 | — | 0.891 | — | 0.733 | 0.813 | — | —
Chen et al. [25] | — | Dice | 0.919 | — | 0.911 | — | 0.877 | 0.927 | — | 0.909
HFA-Net [18] | — | Dice | 0.946 | 0.893 | 0.925 | 0.897 | 0.910 | 0.964 | 0.830 | 0.909
IoU | 0.898 | 0.810 | 0.861 | 0.816 | 0.836 | 0.930 | 0.722 | 0.839
HDF | 7.148 | 33.128 | 42.173 | 22.903 | 36.954 | 12.075 | 37.845 | 27.461
ADB | 0.076 | 0.562 | 0.210 | 0.334 | 0.225 | 0.103 | 1.685 | 0.456
$k$CBAC-Net [21] | 134M | Dice | 0.951 | 0.902 | 0.938 | 0.911 | 0.922 | 0.974 | 0.837 | 0.919
IoU | 0.907 | 0.825 | 0.883 | 0.838 | 0.855 | 0.949 | 0.734 | 0.856
HDF | 5.500 | 14.940 | 12.403 | 15.081 | 7.337 | 6.848 | 32.499 | 13.516
ADB | 0.074 | 0.285 | 0.163 | 0.248 | 0.119 | 0.059 | 1.403 | 0.336
CoTr [12] | 42M | Dice | 0.944 | 0.896 | 0.934 | 0.898 | 0.908 | 0.953 | 0.845 | 0.911
IoU | 0.895 | 0.816 | 0.876 | 0.816 | 0.832 | 0.912 | 0.742 | 0.841
HDF | 5.905 | 15.819 | 12.746 | 15.767 | 10.346 | 12.330 | 31.539 | 14.922
ADB | 0.084 | 0.352 | 0.159 | 0.285 | 0.131 | 0.118 | 1.391 | 0.360
UNETR [14] | 93M | Dice | 0.950 | 0.888 | 0.939 | 0.903 | 0.919 | 0.964 | 0.843 | 0.915
IoU | 0.896 | 0.806 | 0.868 | 0.818 | 0.854 | 0.930 | 0.744 | 0.845
HDF | 5.869 | 15.698 | 11.769 | 14.080 | 8.816 | 11.864 | 33.392 | 14.498
ADB | 0.080 | 0.349 | 0.151 | 0.278 | 0.115 | 0.103 | 1.404 | 0.354
ConvFormer (Ours) | 61M | Dice | 0.955 | 0.910 | 0.943 | 0.914 | 0.926 | 0.980 | 0.844 | 0.925
IoU | 0.914 | 0.836 | 0.885 | 0.835 | 0.857 | 0.957 | 0.740 | 0.861
HDF | 5.089 | 12.873 | 11.745 | 13.360 | 7.000 | 6.823 | 34.560 | 13.064
ADB | 0.069 | 0.264 | 0.175 | 0.242 | 0.120 | 0.030 | 1.359 | 0.323

Table 3: Quantitative results of different models on the 2017 ISIC skin lesion
dataset.

Method | Jaccard Index | Dice | Sensitivity | Specificity
---|---|---|---|---
Yuan et al. [22] | 0.765 | 0.849 | 0.825 | 0.975
Li et al. [26] | 0.765 | 0.866 | 0.825 | 0.984
Lei et al. [27] | 0.771 | 0.859 | 0.835 | 0.976
Mirikharaji et al. [28] | 0.773 | 0.857 | 0.855 | 0.973
Xie et al. [29] | 0.788 | 0.868 | 0.884 | 0.957
$k$CBAC-Net [21] | 0.794 | 0.887 | 0.847 | 0.984
ConvFormer (Ours) |  0.797$\pm$0.003 |  0.889$\pm$0.002 |  0.846$\pm$0.002 |  0.986$\pm$0.001

Fig. 2: Some visual qualitative results on the lymph node dataset (a) and the
2017 MM-WHS CT dataset (b), demonstrating the capability of our ConvFormer.
Table 4: Ablation study of the effects of different components in ConvFormer
using the lymph node dataset.

DeTrans |  Conv-based FFM |  Addi. E-DeTrans | EPE | Method | IoU | Precision | Recall | F1
---|---|---|---|---|---|---|---|---
|  |  |  |  ConvFormer w/o DeTrans | 0.673 | 0.833 | 0.782 | 0.807
$\surd$ |  |  |  |  ConvFormer w DeTrans | 0.825 | 0.906 | 0.900 | 0.903
$\surd$ | $\surd$ |  |  |  ConvFormer w/o Additional Enhanced DeTrans | 0.830 | 0.908 | 0.906 | 0.907
$\surd$ | $\surd$ | $\surd$ |  |  ConvFormer w/o EPE | 0.839 | 0.912 | 0.910 | 0.912
$\surd$ | $\surd$ | $\surd$ | $\surd$ | ConvFormer | 0.845 | 0.925 | 0.907 | 0.916
ConvFormer w/o residual connections | 0.829 | 0.910 | 0.902 | 0.906

Ablation Study. We conduct an ablation study to examine the effects of
different components in our ConvFormer using the lymph node dataset. As Table
4 shows, (1) when using the fixed sinusoidal encoding [15] for positional
encoding (the resulted ConvFormer is denoted by ConvFormer w/o EPE), the F1
score drops by 0.4%; (2) when removing the additional Enhanced DeTrans encoder
that aims to explore multi-scale information from ConvFormer w/o EPE (denoted
by ConvFormer w/o Additional Enhanced DeTrans), the F1 score drops by 0.5%;
(3) when removing Conv-based FFM from ConvFormer w/o Additional Enhanced
DeTrans (denoted by ConvFormer w DeTrans), the F1 score drops by 0.4%; (4)
when completely removing DeTrans (the resulted network is a pure CNN baseline,
denoted by ConvFormer w/o DeTrans), the F1 score drops by 9.6%. These effects
demonstrate the importance of our proposed components for better capturing
local, global, and multi-scale information for accurate image segmentation.
Besides, we remove the residual connections of the residual-shaped hybrid stem
(denoted by ConvFormer w/o residual connections), and the F1 score drops by
1.0%. This validates the importance of the residual connections.

##  4 Conclusions

In this paper, we proposed a new hierarchical CNN and Transformer hybrid
architecture, ConvFormer, for medical image segmentation. ConvFormer is
capable of well learning local, global, and multi-scale representations by
introducing Conv-based FFM, residual-shaped hybrid stem, and an additional
Enhanced DeTrans encoder with residual connections. Moreover, we presented an
enhanced positional encoding to improve the adaptability and flexibility of
the fixed sinusoidal encoding method. Experiments on 2D and 3D datasets of
different imaging characteristics and modalities demonstrated the
effectiveness of ConvFormer.

##  5 Compliance with ethical standards

This research study was conducted retrospectively using human subject data
made available in open access by two publicly available datasets [17, 16] and
one in-house dataset. Ethical approval was not required as confirmed by the
licenses attached with the open access datasets.

## References

  * [1] O. Ronneberger, P. Fischer, and T. Brox,  “U-Net: Convolutional networks for biomedical image segmentation,”  in MICCAI, 2015, pp. 234–241.
  * [2] H. Huang, L. Lin, R. Tong, H. Hu, Q. Zhang, Y. Iwamoto, X. Han, Y.-W. Chen, and J. Wu,  “UNet 3+: A full-scale connected unet for medical image segmentation,”  in ICASSP, 2020, pp. 1055–1059.
  * [3] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille,  “DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs,”  IEEE PAMI, vol. 40, no. 4, pp. 834–848, 2018.
  * [4] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia,  “Pyramid scene parsing network,”  in CVPR, 2017, pp. 6230–6239.
  * [5] C. Peng, X. Zhang, G. Yu, G. Luo, and J. Sun,  “Large kernel matters – improve semantic segmentation by global convolutional network,”  in CVPR, 2017, pp. 1743–1751.
  * [6] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby,  “An image is worth 16x16 words: Transformers for image recognition at scale,”  in ICLR, 2021.
  * [7] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo,  “Swin Transformer: Hierarchical vision transformer using shifted windows,”  in ICCV, 2021, pp. 10012–10022.
  * [8] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai,  “Deformable DETR: Deformable transformers for end-to-end object detection,”  in ICLR, 2021.
  * [9] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao,  “Pyramid Vision Transformer: A versatile backbone for dense prediction without convolutions,”  in ICCV, 2021, pp. 568–578.
  * [10] J. Chen, Y. Lu, Q. Yu, X. Luo, E. Adeli, Y. Wang, L. Lu, A. L. Yuille, and Y. Zhou,  “TransUNet: Transformers make strong encoders for medical image segmentation,”  arXiv preprint arXiv:2102.04306, 2021.
  * [11] J. M. J. Valanarasu, P. Oza, I. Hacihaliloglu, and V. M. Patel,  “Medical Transformer: Gated axial-attention for medical image segmentation,”  in MICCAI, 2021, pp. 36–46.
  * [12] Y. Xie, J. Zhang, C. Shen, and Y. Xia,  “CoTr: Efficiently bridging cnn and transformer for 3D medical image segmentation,”  in MICCAI, 2021, pp. 171–180.
  * [13] X. Huang, Z. Deng, D. Li, and X. Yuan,  “MISSFormer: An effective medical image segmentation transformer,”  arXiv preprint arXiv:2109.07162, 2021.
  * [14] A. Hatamizadeh, Y. Tang, V. Nath, D. Yang, A. Myronenko, B. Landman, H. R. Roth, and D. Xu,  “UNETR: Transformers for 3D medical image segmentation,”  in WACV, 2022, pp. 1748–1758.
  * [15] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin,  “Attention is all you need,”  in NIPS, 2017, pp. 5998–6008.
  * [16] N. C. Codella, D. Gutman, M. E. Celebi, B. Helba, M. A. Marchetti, S. W. Dusza, A. Kalloo, K. Liopyris, N. Mishra, H. Kittler, and A. Halpern,  “Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC),”  in IEEE, ISBI, 2018, pp. 168–172.
  * [17] X. Zhuang and J. Shen,  “Multi-scale patch and multi-modality atlases for whole heart segmentation of MRI,”  Medical Image Analysis, vol. 31, pp. 77–87, 2016.
  * [18] H. Zheng, L. Yang, J. Han, Y. Zhang, P. Liang, Z. Zhao, C. Wang, and D. Z. Chen,  “HFA-Net: 3D cardiovascular image segmentation with asymmetrical pooling and content-aware fusion,”  in MICCAI, 2019, pp. 759–767.
  * [19] I. Loshchilov and F. Hutter,  “Decoupled weight decay regularization,”  in ICLR, 2019.
  * [20] Y. Zhang, M. T. Ying, and D. Z. Chen,  “Decompose-and-integrate learning for multi-class segmentation in medical images,”  in MICCAI, 2019, pp. 641–650.
  * [21] P. Gu, H. Zheng, Y. Zhang, C. Wang, and D. Z. Chen,  “$k$CBAC-Net: Deeply supervised complete bipartite networks with asymmetric convolutions for medical image segmentation,”  in MICCAI, 2021, pp. 337–347.
  * [22] Y. Yuan and Y.-C. Lo,  “Improving dermoscopic image segmentation with enhanced convolutional-deconvolutional networks,”  IEEE JBHI, vol. 23, no. 2, pp. 519–526, 2019.
  * [23] C. Payer, D. Štern, H. Bischof, and M. Urschler,  “Multi-label whole heart segmentation using CNNs and anatomical label configurations,”  in International Workshop, STACOM, 2017, pp. 190–198.
  * [24] Q. Dou, C. Ouyang, C. Chen, H. Chen, and P.-A. Heng,  “Unsupervised cross-modality domain adaptation of ConvNets for biomedical image segmentations with adversarial loss,”  in IJCAI, 2018, pp. 691–697.
  * [25] C. Chen, Q. Dou, H. Chen, J. Qin, and P. A. Heng,  “Unsupervised bidirectional cross-modality adaptation via deeply synergistic image and feature alignment for medical image segmentation,”  IEEE TMI, vol. 39, no. 7, pp. 2494–2505, 2020.
  * [26] H. Li, X. He, F. Zhou, Z. Yu, D. Ni, S. Chen, T. Wang, and B. Lei,  “Dense deconvolutional network for skin lesion segmentation,”  IEEE JBHI, vol. 23, no. 2, pp. 527–537, 2019.
  * [27] B. Lei, Z. Xia, F. Jiang, X. Jiang, Z. Ge, Y. Xu, J. Qin, S. Chen, T. Wang, and S. Wang,  “Skin lesion segmentation via generative adversarial networks with dual discriminators,”  Medical Image Analysis, vol. 64, pp. 101716, 2020.
  * [28] Z. Mirikharaji and G. Hamarneh,  “Star shape prior in fully convolutional networks for skin lesion segmentation,”  in MICCAI, 2018, pp. 737–745.
  * [29] Y. Xie, J. Zhang, H. Lu, C. Shen, and Y. Xia,  “SESV: Accurate medical image segmentation by predicting and correcting errors,”  IEEE TMI, vol. 40, no. 1, pp. 286–296, 2021.