11institutetext: Departamento de Física, Universidad Técnica Federico Santa
María, Av. España 1680, Valparaíso, Chile
11email<EMAIL_ADDRESS><EMAIL_ADDRESS>22institutetext: Millennium Nucleus for Planet Formation, Valparaíso, Chile
33institutetext: Anton Pannekoek Institute for Astronomy, University of
Amsterdam, NL-1090 GE Amsterdam, the Netherlands

# Period bouncers as detached magnetic cataclysmic variables

Matthias R. Schreiber 1.21.2 Diogo Belloni 11 Jan van Roestel 33

###### Abstract

Context. The general prediction that more than half of all cataclysmic
variables (CVs) have evolved past the period minimum is in strong disagreement
with observational surveys, which show that the relative number of these
objects is just a few per cent.

Aims. Here, we investigate whether a large number of post-period minimum CVs
could detach because of the appearance of a strong white dwarf magnetic field
potentially generated by a rotation- and crystallization-driven dynamo.

Methods. We used the MESA code to calculate evolutionary tracks of CVs
incorporating the spin evolution and cooling as well as compressional heating
of the white dwarf. If the conditions for the dynamo were met, we assumed that
the emerging magnetic field of the white dwarf connects to that of the
companion star and incorporated the corresponding synchronization torque,
which transfers spin angular momentum to the orbit.

Results. We find that for CVs with donor masses exceeding $\sim 0.04$M⊙,
magnetic fields are generated mostly if the white dwarfs start to crystallize
before the onset of mass transfer. It is possible that a few white dwarf
magnetic fields are generated in the period gap. For the remaining CVs, the
conditions for the dynamo to work are met beyond the period minimum, when the
accretion rate decreased significantly. Synchronization torques cause these
systems to detach for several gigayears even if the magnetic field strength of
the white dwarf is just one MG.

Conclusions. If the rotation- and crystallization-driven dynamo —which is
currently the only mechanism that can explain several observational facts
related to magnetism in CVs and their progenitors— or a similar temperature-
dependent mechanism is responsible for the generation of magnetic field in
white dwarfs, most CVs that have evolved beyond the period minimum must detach
for several gigayears at some point. This reduces the predicted number of
semi-detached period bouncers by up to $\sim 60-80$ per cent.

###### Key Words.:

binaries: close – methods: numerical – stars: evolution – stars: magnetic
fields – white dwarfs

##  1 Introduction

According to the standard theory for the formation and evolution of
cataclysmic variables (CVs, Belloni & Schreiber 2023; Knigge et al. 2011), and
independent of the details of simulating CV evolution, a large number of all
CVs should already have passed the period minimum, that is, the population of
CVs should be dominated by systems evolving toward longer orbital periods
consisting of a cool white dwarf with a brown dwarf companion. The predicted
fraction of these so-called period bouncers among CVs was estimated decades
ago to be 70 per cent (Kolb, 1993), a value roughly in agreement with more
recent calculations, which predict $38-60$ per cent (Goliasch & Nelson, 2015)
and $\sim 75$ per cent (Belloni et al., 2018).

However, despite significant observational efforts, very few period bouncers
have so far been identified. Patterson (2011) heard some murmurs from the
period bounce by identifying 22 candidates among the CVs known at that time.
More recently, Pala et al. (2020) found the fraction of period bouncers to be
$7-14$ per cent in a volume-limited sample of CVs. Dedicated surveys for CVs
in the data releases of the Sloan Digital Sky Survey (SDSS) confirm that the
fraction of period bouncers does not exceed a few per cent (Inight et al.,
2023a, b). This large discrepancy between theoretical prediction and
observations might be the most important problem in our understanding of CV
evolution.

A potentially important ingredient in CV evolution —that was ignored in binary
population models of CVs for decades— is the possible impact of white dwarf
magnetic fields, despite the fact that more than one-third of all CVs in
volume-limited samples host a magnetic white dwarf (Pala et al., 2020).
Observations of both single white dwarfs (Bagnulo & Landstreet, 2021) and
detached white dwarf binaries with M-dwarf companions (Parsons et al., 2021)
clearly show that the vast majority of white dwarf magnetic fields appear when
the white dwarf has significantly cooled, that is, either the magnetic field
is buried for several gigayears (Gyr) after the white dwarf formation, or the
conditions for generating a magnetic field are met only in cool white dwarfs.

Based on these observational facts and the early work by Isern et al. (2017),
we developed the idea of a rotation- and crystallization-driven magnetic
dynamo that generates strong white dwarf magnetic fields during CV evolution
(Schreiber et al., 2021). According to this model, white dwarfs that
crystallize before mass-transfer starts generate a strong magnetic field as
soon as accretion has spun up the white dwarf. If the emerging magnetic field
connects with that of the donor star, spin angular momentum is transferred to
the orbit, which causes the separation of both stars to increase and the CV to
convert into a detached binary. Loss of angular momentum through gravitational
radiation and/or reduced magnetic braking (Belloni et al., 2020) then slowly
shrinks the orbit until the donor fills its Roche lobe and the system becomes
a magnetic CV.

While the assumption of magnetic field generation through the dynamo is rather
speculative (as other ideas for the origin of magnetic fields in white dwarfs
are), the described evolutionary scenario can reproduce a number of
observations that otherwise remain inexplicable: (i) the fact that so far all
observed detached close binaries containing a strongly magnetic white dwarf
are close to Roche-lobe filling and typically have periods of between $3$ and
$5$ hr; (ii) the existence of the radio pulsing white dwarf binaries AR Sco
(Marsh et al., 2016) and eRASSU J191213.9–441044 (Pelisoli et al., 2023);
(iii) the small number of high-accretion-rate magnetic CVs in globular
clusters (Belloni et al., 2021); and (iv) the observed magnetic fields in
detached double white dwarfs (Schreiber et al., 2022).

Here we focus on magnetic field generation and angular momentum transfer for
systems that evolve toward and beyond the period minimum as nonmagnetic CVs.
In these systems, the mass-transfer rate becomes small enough to allow the
core of the white dwarf to crystallize, and according to the dynamo idea, the
white dwarf should develop a magnetic field. The corresponding synchronization
torques should then cause the system to detach for several Gyr, thereby
reducing the number of predicted period bouncers significantly.

##  2 The model

We used the MESA code (Paxton et al., 2011, 2013, 2015, 2018, 2019; Jermyn et
al., 2023, r15140) to compute the evolution of CVs as in Schreiber et al.
(2021). The initial conditions for all our simulations are detached post-CE
binaries with different initial donor star and white dwarf masses and orbital
periods.

We assumed angular momentum loss through magnetic braking according to
Rappaport et al. (1983) and through gravitational radiation, and that magnetic
braking gets disrupted when the donor star becomes fully convective. We used
normalization factors to calibrate the strength of angular momentum loss. We
also considered the empirical prescription for consequential angular momentum
loss found by Schreiber et al. (2016), which is required to bring the observed
and predicted white dwarf mass distributions into agreement.

Angular momentum accretion and spin-up of the white dwarf during mass transfer
was calculated as in Schreiber et al. (2021), that is, largely following King
et al. (1991). The potential connection between the emerging white dwarf
magnetic field and that of the donor star, as well as the resulting angular
momentum transfer of spin angular momentum to orbital angular momentum, were
also treated as in Schreiber et al. (2021).

In line with Ginzburg et al. (2022), we assumed the critical spin period to be
one hour, which is significantly longer than the values assumed by Schreiber
et al. (2021). We also updated our model by incorporating a diffusion
timescale that reflects the time the field requires to reach the surface of
the white dwarf after the conditions for magnetic field generation are met in
the core (Ginzburg et al., 2022).

Crystallization depends on the core temperature of the white dwarf, which is
therefore of fundamental importance for magnetic field generation in our
model. We assumed the critical temperature for the dynamo to work to be the
temperature corresponding to the onset of crystallization, as provided by the
thick hydrogen atmosphere models of Bédard et al. (2020). The core temperature
of cooling single white dwarfs can be reliably determined from these
evolutionary sequences.

The most important update compared to our previous modeling of CV evolution
including magnetic field generation through the rotation- and crystallization-
driven dynamo concerns the heating of the white dwarf core through accretion.
Accretion can slowly (on a timescale of a few 100 million years) heat up the
core until an equilibrium at the base of the radiative layer is reached.

Incorporating the full details of cooling and heating of accreting white
dwarfs in our simulations is beyond the scope of this paper. However, based on
published results (Townsley & Bildsten, 2004; Epelstain et al., 2007; Townsley
& Gänsicke, 2009), we developed a simple prescription that should cover the
main physics. Briefly, we used the cooling temperature for nonaccreting white
dwarfs while for accreting white dwarfs we slowly increased the core
temperature until the equilibrium core temperature (governed by the mass
accretion rate) was reached.

We are aware that our modeling approach is based on several assumptions, such
as the stellar field strengths, the accretion efficiency of angular momentum,
and the synchronization or diffusion timescales. In addition, our prescription
for determining the white dwarf core temperature during accretion is clearly a
simplified representation of reality. However, the general idea of the
scenario is independent of the exact values and procedures that we assumed in
our modeling. More details can be found in Schreiber et al. (2021) and
Appendix A. The code we used to calculate the evolutionary tracks can be found
here: https://zenodo.org/records/10008722.

##  3 Evolutionary pathways

Figure 1: Evolutionary tracks for two systems starting mass transfer above the
gap. The initial parameters are $M_{\rm WD}=0.8$ M⊙, $M_{2}=0.6$ M⊙, and
$P_{\mathrm{orb}}=0.25$ (black) and $1.0$ days (red). The horizontal gray line
in the bottom panel corresponds to $\log(Tc)=6.742366$, which is the core
temperature at the onset of crystallization. The vertical lines refer to the
period minimum of $P_{\mathrm{min}}=76-82$ minutes, and the period gap
($2.15-3.18$ hours). While the system represented by the red track detaches
above the gap because the white dwarf crystallizes before mass transfer
starts, CVs that start mass transfer with a noncrystallizing white dwarf
detach after they have passed the period minimum (black track). We note that
the period minimum is at shorter periods for systems that previously generated
a white dwarf magnetic field. This is because magnetic braking is reduced as
some of the open field lines of the donor star are affected by the white dwarf
magnetic field.

Figure 1 shows two evolutionary tracks for CVs that start mass transfer above
the orbital period gap, together with observed period bouncers and the
detached magnetic white dwarf with a brown dwarf companion SMSSJ
160639.78$-$100010.7 (see Appendix B for more details on the observed
systems). The red dashed lines represent a case similar to those described in
Schreiber et al. (2021), that is, mass transfer starts when the white dwarf
core has already started to crystallize. The magnetic field is generated in
the core of the white dwarf when the critical rotation rate is reached. The
field then diffuses outward and when it emerges at the surface, it can connect
with the field of the secondary star and synchronization torques lead to the
transfer of spin angular momentum to the orbit, and the binary becomes a
detached system. The model predicts the detached binary to first resemble AR
Sco (Marsh et al., 2016) and later, when spin and orbit are synchronized, to
become a pre-polar (Parsons et al., 2021).

The black solid line corresponds to the case of a white dwarf that is still
far from crystallizing when mass transfer starts. This prevents the system
from generating a magnetic field above the gap because accretion keeps the
core temperature above the crystallization limit. The system detaches at the
upper edge of the orbital period gap as predicted by standard CV evolution
theory. While crossing the gap, the white dwarf cools but not enough to start
crystallizing because with residual magnetic braking, the evolution through
the gap takes only $\sim 300$ Myr. When mass transfer resumes below the gap,
accretion determines the core temperature. As the mass-accretion rate
decreases significantly when the donor converts into a brown dwarf and the
system passes the orbital period minimum, the core temperature of the white
dwarf inevitably reaches the crystallization limit at some point. This causes
the dynamo to generate a magnetic field which, after it has diffused to the
white dwarf surface, connects with that of the donor and causes the secondary
to detach from its Roche lobe.

The generated detached phase takes several Hubble times as the spin period has
reached very small values because of the large amount of accreted angular
momentum and because the remaining angular-momentum-loss mechanism is
gravitational radiation (residual magnetic braking is suppressed through the
connection of both magnetic fields; e.g. Belloni et al. 2020). The field
strength needed to connect with the secondary star is much smaller than above
the gap because the binary separation and mass-transfer rate are much smaller
(see Schreiber et al., 2021, their eq. 9). As little as $\sim 1$ MG is
sufficient to generate a long detached phase.

In full analogy to what happens in the case where strong fields are generated
in the 3-5 hour orbital period range, the system appears first as a radio-
pulsating fast-spinning white dwarf with a brown dwarf companion, before it
synchronizes and becomes a pre-polar. Given that the spin angular momentum is
typically high enough for the system to quickly evolve to periods of up to
five hours, the age of the Galaxy is not sufficient for these detached systems
to evolve back into a semi-detached configuration.

The tracks discussed above are representative of the majority of CVs, as most
are born above the gap. If the core crystallizes before the onset of mass
transfer, the magnetic field appears above or in the period gap. If, on the
other hand, the white dwarf does not crystallize before the onset of mass
transfer, it will only do so when the mass-transfer rate significantly drops
after the system has passed the period minimum.

The situation is similar for CVs that start mass transfer in or below the gap.
Figure 2 shows two tracks that describe the possible evolutionary pathways for
such objects. The red dashed line corresponds to an evolution where the core
of the white dwarf crystallizes before the onset of mass transfer, which
causes the field to appear when the rotation criterion is met and the field
has had time to diffuse to the white dwarf surface. This causes a relatively
short ($\sim 300$ Myr) detached phase that occurs at a period similar to that
of the onset of mass transfer (and before the period minimum is reached).

If the core does not crystallize before the onset of mass transfer (black line
in Fig. 2), compressional heating of the white dwarf core during the mass
transfer phase prevent crystallization until the mass accretion rate drops,
which occurs after the binary has passed the period minimum. Compared to the
case of CVs that started mass transfer above the gap with a noncrystallizing
white dwarf, the maximum period reached during the detached phase is shorter
because less angular momentum has been accreted, but it still takes longer
than a Hubble time to evolve back to mass transfer.

While the exact evolution of a given system depends on the masses and orbital
period of the detached post-common-envelope binary, virtually all CVs are
predicted to evolve according to one of the four scenarios described above.
The exceptions are CVs that become magnetic in the period gap. This might
occur when the core of the white dwarf cools substantially before mass
transfer but does not crystallize and accretion does not fully heat up the
white dwarf core. To match all these conditions, some fine-tuning is required
and therefore we expect relatively few systems to evolve in this way.

Figure 2: Examples of evolutionary tracks for CVs starting mass transfer below
the gap. The initial donor mass is assumed to be ${M_{2}=0.15}$ M⊙ and with
initial post-common-envelope periods of $0.10$ (black) and $0.25$ days (red).
If a CV is born with a crystallizing white dwarf, the magnetic field is
generated before the period minimum is reached and the system detaches for a
relatively short period of time ($\sim 130$ Myr) and then evolves as a
magnetic CV through the period minimum (red dashed track). If the white dwarf
does not start to crystallize before the onset of mass transfer, the binary
detaches after it has passed the period minimum when the accretion rate became
low enough to allow the white dwarf to crystallize (black solid track).

##  4 Predictions

Schreiber et al. (2021) invented the rotation- and crystallization-driven
dynamo to explain the existence of detached post-common-envelope binaries with
cool magnetic white dwarfs close to Roche-lobe filling in the period range of
3 to 5 hr, the radio pulsing white dwarf binary AR Sco, and the fact that a
large number of CVs contain a magnetic white dwarf. We here focus on the
predictions of the model with respect to the evolution and magnetic field
generation for CVs below the orbital period gap.

The relative number of CVs that are predicted to generate a magnetic field
before the period minimum is reached, that is, systems that either crystallize
before mass transfer starts or in the period gap, depends on the initial mass-
ratio distribution, common envelope efficiency, and the strengths of magnetic
braking. Performing detailed binary population synthesis is beyond the scope
of this paper. However, assuming that a set of parameters for common envelope
evolution, magnetic braking, and the initial mass ratio distribution exists
that can explain the measured fraction of magnetic CVs, which is in the range
of $20-40$ per cent (Pala et al., 2020; Inight et al., 2023a, b), it is
straightforward to make testable predictions.

If the dynamo is fully responsible for magnetic field generation in CVs, $\sim
20-40$ per cent of all CVs either hosted a crystallizing white dwarf at the
onset of mass transfer or the white dwarf crystallized in the period gap.
These systems will pass the period minimum as magnetic CVs. The remaining
$\sim 60-80$ per cent of all CVs evolve toward the period minimum as
nonmagnetic CVs. Only after evolving through the period minimum can the core
crystallize which causes the magnetic field to emerge and the binary to become
a detached binary consisting of a magnetic white dwarf ($B\geq 1$ MG) with a
brown dwarf companion. This reduces the fraction of expected period bouncers
from $38-75$ per cent (Kolb, 1993; Goliasch & Nelson, 2015; Belloni et al.,
2018) to $\sim 8-30$ per cent if we ignore the post-period-minimum evolution
before the detached phase. This rough estimate represents an upper limit. If
nonmagnetic CVs spend a significant time span as accreting CVs after having
passed the period minimum and before the accretion rate sufficiently drops for
the core to crystallize, the reduction of period bouncers might be
significantly smaller than $60-80$ per cent. How long a given system spends as
an accreting CV after having passed the period minimum depends on the
crystallization temperature (which depends on the white dwarf mass) and the
strength of angular momentum loss at this stage, which is not well known.

While detailed binary population synthesis is required to provide the exact
numbers, we conclude that the dynamo, which was invented to explain the
magnetic nature of CVs and detached systems with periods longer than $3$ hr,
may bring the predicted relative numbers of period bouncers into agreement
with the currently available observational constraints. The trade-off of this
result is that the model predicts a large number of cool magnetic white dwarfs
with brown dwarf companions, with a predicted space density similar to that of
CVs with stellar companions. These systems might be difficult to detect
because of the faintness of both stellar components and the absence of
accretion. So far, there is only one known magnetic white dwarf that has a
brown dwarf companion in a clearly detached binary that could be a detached CV
(Kawka et al., 2021).

##  5 Conclusion

If a rotation- and crystallization-driven dynamo generates the magnetic field
and the explanation for the existence of AR Sco and the detached magnetic
white dwarf binaries in the orbital period range of $3-5$ hr presented by
Schreiber et al. (2021) is correct, as a natural consequence, a large fraction
of period bouncers must detach as soon as the accretion rate drops
sufficiently and the white dwarf starts to crystallize. This may in principle
explain the low number of period bouncers identified so far.

Even if the dynamo idea is found to be unfeasible, for example because the
energy carried as kinetic energy in the convection zone is found to be
insufficient to drive an efficient dynamo (Fuentes et al., 2023), the
evolutionary scenario outlined in Schreiber et al. (2021) and in this letter
remains plausible as long as the magnetic field appearance in white dwarfs
depends on their core temperature. There is indeed overwhelming evidence for
the increased occurrence of magnetic fields in cool white dwarfs (Parsons et
al., 2021; Bagnulo & Landstreet, 2021).

Unequivocal proof for a temperature-dependent magnetic field generation in
white dwarfs would be provided by the detection of a large number of detached
binaries consisting of a cool
($\raisebox{-1.72218pt}{$\stackrel{{\scriptstyle<}}{{\scriptstyle\sim}}$}10\,000$
K) magnetic
($\raisebox{-1.72218pt}{$\stackrel{{\scriptstyle>}}{{\scriptstyle\sim}}$}1$
MG) white dwarf with a brown dwarf companion and orbital periods of between 80
minutes and a few hours. At present, just one such system is known (Kawka et
al., 2021).

###### Acknowledgements.

We are very grateful to the Kavli Institute for Theoretical Physics (KITP) for
hosting the program “White Dwarfs as Probes of the Evolution of Planets,
Stars, the Milky Way and the Expanding Universe”. This research was supported
in part by the National Science Foundation under Grant No. NSF PHY-1748958 and
by the Munich Institute for Astro-, Particle and BioPhysics (MIAPbP) which is
funded by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) under Germany’s Excellence Strategy – EXC-2094 – 390783311. MRS
was supported from FONDECYT (grant number 1221059) and ANID, – Millennium
Science Initiative Program – NCN19_171. DB acknowledges financial support from
FONDECYT (grant number 3220167).

## References

  * Amantayeva et al. (2021) Amantayeva, A., Zharikov, S., Page, K. L., et al. 2021, ApJ, 918, 58
  * Angulo et al. (1999) Angulo, C., Arnould, M., Rayet, M., et al. 1999, Nuclear Physics A, 656, 3
  * Bagnulo & Landstreet (2021) Bagnulo, S. & Landstreet, J. D. 2021, MNRAS, 507, 5902
  * Bédard et al. (2020) Bédard, A., Bergeron, P., Brassard, P., & Fontaine, G. 2020, ApJ, 901, 93
  * Belloni & Schreiber (2023) Belloni, D. & Schreiber, M. R. 2023, in Handbook of X-ray and Gamma-ray Astrophysics. Edited by Cosimo Bambi and Andrea Santangelo, 129
  * Belloni et al. (2020) Belloni, D., Schreiber, M. R., Pala, A. F., et al. 2020, MNRAS, 491, 5717
  * Belloni et al. (2021) Belloni, D., Schreiber, M. R., Salaris, M., Maccarone, T. J., & Zorotovic, M. 2021, MNRAS, 505, L74
  * Belloni et al. (2018) Belloni, D., Schreiber, M. R., Zorotovic, M., et al. 2018, MNRAS, 478, 5626
  * Buchler & Yueh (1976) Buchler, J. R. & Yueh, W. R. 1976, ApJ, 210, 440
  * Cassisi et al. (2007) Cassisi, S., Potekhin, A. Y., Pietrinferni, A., Catelan, M., & Salaris, M. 2007, ApJ, 661, 1094
  * Chugunov et al. (2007) Chugunov, A. I., Dewitt, H. E., & Yakovlev, D. G. 2007, Phys. Rev. D, 76, 025028
  * Cyburt et al. (2010) Cyburt, R. H., Amthor, A. M., Ferguson, R., et al. 2010, ApJS, 189, 240
  * Epelstain et al. (2007) Epelstain, N., Yaron, O., Kovetz, A., & Prialnik, D. 2007, MNRAS, 374, 1449
  * Fuentes et al. (2023) Fuentes, J. R., Cumming, A., Castro-Tapia, M., & Anders, E. H. 2023, ApJ, 950, 73
  * Fuller et al. (1985) Fuller, G. M., Fowler, W. A., & Newman, M. J. 1985, ApJ, 293, 1
  * Ginzburg et al. (2022) Ginzburg, S., Fuller, J., & Kawka, A. 2022, MNRAS, 514, 4111
  * Goliasch & Nelson (2015) Goliasch, J. & Nelson, L. 2015, ApJ, 809, 80
  * Guidry et al. (2021) Guidry, J. A., Vanderbosch, Z. P., Hermes, J. J., et al. 2021, ApJ, 912, 125
  * Hakala et al. (2022) Hakala, P., Parsons, S. G., Marsh, T. R., et al. 2022, MNRAS, 513, 3858
  * Hurley et al. (2002) Hurley, J. R., Tout, C. A., & Pols, O. R. 2002, MNRAS, 329, 897
  * Iglesias & Rogers (1993) Iglesias, C. A. & Rogers, F. J. 1993, ApJ, 412, 752
  * Iglesias & Rogers (1996) Iglesias, C. A. & Rogers, F. J. 1996, ApJ, 464, 943
  * Inight et al. (2023a) Inight, K., Gänsicke, B. T., Breedt, E., et al. 2023a, MNRAS, 524, 4867
  * Inight et al. (2023b) Inight, K., Gänsicke, B. T., Schwope, A., et al. 2023b, MNRAS, 525, 3597
  * Irwin (2004) Irwin, A. W. 2004, The FreeEOS Code for Calculating the Equation of State for Stellar Interiors
  * Isern et al. (2017) Isern, J., García-Berro, E., Külebi, B., & Lorén-Aguilar, P. 2017, ApJ, 836, L28
  * Itoh et al. (1996) Itoh, N., Hayashi, H., Nishikawa, A., & Kohyama, Y. 1996, ApJS, 102, 411
  * Jermyn et al. (2023) Jermyn, A. S., Bauer, E. B., Schwab, J., et al. 2023, ApJS, 265, 15
  * Jermyn et al. (2021) Jermyn, A. S., Schwab, J., Bauer, E., Timmes, F. X., & Potekhin, A. Y. 2021, ApJ, 913, 72
  * Kawka et al. (2021) Kawka, A., Vennes, S., Ferrario, L., et al. 2021, MNRAS, 507, L30
  * King et al. (1991) King, A. R., Regev, O., & Wynn, G. A. 1991, MNRAS, 251, 30P
  * Knigge et al. (2011) Knigge, C., Baraffe, I., & Patterson, J. 2011, ApJS, 194, 28
  * Kolb (1993) Kolb, U. 1993, A&A, 271, 149
  * Langanke & Martínez-Pinedo (2000) Langanke, K. & Martínez-Pinedo, G. 2000, Nuclear Physics A, 673, 481
  * Littlefair et al. (2008) Littlefair, S. P., Dhillon, V. S., Marsh, T. R., et al. 2008, MNRAS, 388, 1582
  * Littlefair et al. (2006) Littlefair, S. P., Dhillon, V. S., Marsh, T. R., et al. 2006, Science, 314, 1578
  * Marsh et al. (2016) Marsh, T. R., Gänsicke, B. T., Hümmerich, S., et al. 2016, Nature, 537, 374
  * McAllister et al. (2019) McAllister, M., Littlefair, S. P., Parsons, S. G., et al. 2019, MNRAS, 486, 5535
  * McAllister et al. (2017) McAllister, M. J., Littlefair, S. P., Dhillon, V. S., et al. 2017, MNRAS, 467, 1024
  * Muñoz-Giraldo et al. (2023) Muñoz-Giraldo, D., Stelzer, B., de Martino, D., & Schwope, A. 2023, A&A, 676, A7
  * Neustroev & Mäntynen (2023) Neustroev, V. V. & Mäntynen, I. 2023, MNRAS, 523, 6114
  * Neustroev et al. (2017) Neustroev, V. V., Marsh, T. R., Zharikov, S. V., et al. 2017, MNRAS, 467, 597
  * Oda et al. (1994) Oda, T., Hino, M., Muto, K., Takahara, M., & Sato, K. 1994, Atomic Data and Nuclear Data Tables, 56, 231
  * Pala et al. (2022) Pala, A. F., Gänsicke, B. T., Belloni, D., et al. 2022, MNRAS, 510, 6110
  * Pala et al. (2020) Pala, A. F., Gänsicke, B. T., Breedt, E., et al. 2020, MNRAS, 494, 3799
  * Pala et al. (2018) Pala, A. F., Schmidtobreick, L., Tappert, C., Gänsicke, B. T., & Mehner, A. 2018, MNRAS, 481, 2523
  * Parsons et al. (2021) Parsons, S. G., Gänsicke, B. T., Schreiber, M. R., et al. 2021, MNRAS, 502, 4305
  * Patterson (2011) Patterson, J. 2011, MNRAS, 411, 2695
  * Paxton et al. (2011) Paxton, B., Bildsten, L., Dotter, A., et al. 2011, ApJS, 192, 3
  * Paxton et al. (2013) Paxton, B., Cantiello, M., Arras, P., et al. 2013, ApJS, 208, 4
  * Paxton et al. (2015) Paxton, B., Marchant, P., Schwab, J., et al. 2015, ApJS, 220, 15
  * Paxton et al. (2018) Paxton, B., Schwab, J., Bauer, E. B., et al. 2018, ApJS, 234, 34
  * Paxton et al. (2019) Paxton, B., Smolec, R., Schwab, J., et al. 2019, ApJS, 243, 10
  * Pelisoli et al. (2023) Pelisoli, I., Marsh, T. R., Buckley, D. A. H., et al. 2023, Nature Astronomy, 7, 931
  * Potekhin & Chabrier (2010) Potekhin, A. Y. & Chabrier, G. 2010, Contributions to Plasma Physics, 50, 82
  * Rappaport et al. (1983) Rappaport, S., Verbunt, F., & Joss, P. C. 1983, ApJ, 275, 713
  * Rogers & Nayfonov (2002) Rogers, F. J. & Nayfonov, A. 2002, ApJ, 576, 1064
  * Saumon et al. (1995) Saumon, D., Chabrier, G., & van Horn, H. M. 1995, ApJS, 99, 713
  * Schreiber et al. (2021) Schreiber, M. R., Belloni, D., Gänsicke, B. T., Parsons, S. G., & Zorotovic, M. 2021, Nature Astronomy, 5, 648
  * Schreiber et al. (2022) Schreiber, M. R., Belloni, D., Zorotovic, M., et al. 2022, MNRAS, 513, 3090
  * Schreiber et al. (2016) Schreiber, M. R., Zorotovic, M., & Wijnen, T. P. G. 2016, MNRAS, 455, L16
  * Timmes & Swesty (2000) Timmes, F. X. & Swesty, F. D. 2000, ApJS, 126, 501
  * Townsley & Bildsten (2004) Townsley, D. M. & Bildsten, L. 2004, ApJ, 600, 390
  * Townsley & Gänsicke (2009) Townsley, D. M. & Gänsicke, B. T. 2009, ApJ, 693, 1007
  * Unda-Sanzana et al. (2008) Unda-Sanzana, E., Marsh, T. R., Gänsicke, B. T., et al. 2008, MNRAS, 388, 889
  * van Spaandonk et al. (2010) van Spaandonk, L., Steeghs, D., Marsh, T. R., & Parsons, S. G. 2010, ApJ, 715, L109
  * Webbink & Wickramasinghe (2002) Webbink, R. F. & Wickramasinghe, D. T. 2002, MNRAS, 335, 1
  * Zharikov et al. (2008) Zharikov, S. V., Tovmassian, G. H., Neustroev, V. V., et al. 2008, A&A, 486, 505
  * Zorotovic et al. (2011) Zorotovic, M., Schreiber, M. R., & Gänsicke, B. T. 2011, A&A, 536, A42

##  Appendix A Modeling details

The equations describing most of the ingredients of the model, such as the
white dwarf spin-up and orbital angular momentum loss, are given in Schreiber
et al. (2021). We report all parameters and revised assumptions below.

###  A.1 MESA

We used the MESA code (Paxton et al., 2011, 2013, 2015, 2018, 2019; Jermyn et
al., 2023, r15140) to compute the evolution of CVs and their progenitors. The
MESA equation of state is a blend of the OPAL (Rogers & Nayfonov, 2002), SCVH
(Saumon et al., 1995), FreeEOS (Irwin, 2004), HELM (Timmes & Swesty, 2000), PC
(Potekhin & Chabrier, 2010) and Skye (Jermyn et al., 2021) equations of state.
Nuclear reaction rates are a combination of rates from NACRE (Angulo et al.,
1999), JINA REACLIB (Cyburt et al., 2010), plus additional tabulated weak
reaction rates (Fuller et al., 1985; Oda et al., 1994; Langanke & Martínez-
Pinedo, 2000). Screening is included via the prescription of Chugunov et al.
(2007) and thermal neutrino loss rates are from Itoh et al. (1996). Electron
conduction opacities are from Cassisi et al. (2007) and radiative opacities
are primarily from OPAL (Iglesias & Rogers, 1993, 1996), with the high-
temperature Compton-scattering-dominated regime calculated using the equations
of Buchler & Yueh (1976).

###  A.2 Spin-up of the white dwarf

The spin up of accreting white dwarfs was calculated as in Schreiber et al.
(2021, their eq. 1) assuming an initial spin period of one year and the
maximum spin-up efficiency of $\alpha=1$. The angular momentum loss parameter
of the white dwarf due to nova explosions was set to $\eta=0.833$ which
corresponds to a slightly nonspherical ejection. The critical spin period for
the generation of a magnetic field was set to one hour. We assumed the white
dwarf mass to be constant during CV-evolution, that is, we assume the accreted
mass to be ejected during nova eruptions.

###  A.3 Orbital angular momentum loss

For secondary stars with a radiative core and convective envelope, we assumed
angular momentum loss through magnetic braking according to Rappaport et al.
(1983, with $\gamma=3$) with a normalization factor of $0.6$ (Knigge et al.,
2011). In case the magnetic field of the white dwarf connects with that of the
secondary star, we assumed that magnetic braking is reduced as described in
Webbink & Wickramasinghe (2002) and Belloni et al. (2020).

Angular momentum loss through gravitational radiation was calculated using the
weak-field approximation of general relativity as in Hurley et al. (2002). We
assumed residual magnetic braking for fully convective donor stars and
approximated it by multiplying angular momentum loss through gravitational
radiation by $1.5$ which provides accretion rates below the gap in agreement
with the observations (Knigge et al., 2011). We assumed the empirical
consequential angular momentum loss prescription of Schreiber et al. (2016,
their eq. 5) with a normalization factor of $C=0.35$.

###  A.4 Cooling and heating of the white dwarf

We follow the cooling of nonaccreting white dwarfs using the models by Bédard
et al. (2020). Incorporating the full details of cooling and heating of
accreting white dwarfs in our simulations is beyond the scope of this paper.
We instead incorporated a simple prescription based on published results
(Townsley & Bildsten, 2004; Epelstain et al., 2007; Townsley & Gänsicke,
2009).

In cases where the cooling temperature of the white dwarf at the onset of mass
transfer is below the temperature generated by compressional heating (Townsley
& Gänsicke, 2009, their equation 1), the white dwarf core temperature is
heated on timescales of a few hundred million years until the equilibrium
luminosity ${L_{\mathrm{eq}}\simeq\langle\dot{M}\rangle T_{\mathrm{c}}/\mu
m_{\mathrm{p}}=4\pi R^{2}\sigma T_{\mathrm{eff}}^{4}}$ is reached. Here,
$T_{\mathrm{c}}$ is the core temperature assumed to be equal to that at the
base of the radiative layers, $R$ is the white dwarf radius, $\mu$ is the mean
molecular weight of the accreted material, $m_{\mathrm{p}}$ is the mass of the
proton, and $\sigma$ is the Stefan–Boltzmann constant (see Townsley &
Gänsicke, 2009, for more details).

Throughout this work we assume that the timescale required to reach the
equilibrium temperature is $200$ Myr. When mass transfer starts and the
compressional heating temperature exceeds the cooling temperature of the white
dwarf, we slowly increase the core temperature interpolating between the
cooling temperature and the equilibrium temperature using a cubic equation.
The latter makes sure that the increase in temperature is slow at the
beginning (Epelstain et al., 2007). If mass transfer stops, the white dwarf
temperature is assumed to follow again the cooling models from Bédard et al.
(2020) but starting from the core temperature reached during accretion.

In the case where the cooling temperature of the white dwarf is hotter than
the temperature derived from compressional heating, the cooling temperature of
course should represent the true white dwarf core temperature. In this case,
we therefore assume the usual cooling of a nonaccreting white dwarf to
determine the timescale on which the equilibrium between compressional heating
and core temperature is reached.

###  A.5 Dynamo conditions and synchronization

We assumed the generation of a magnetic field to require the core temperature
of the white dwarf to have reached the onset of crystallization according to
the cooling models by Bédard et al. (2020) and a spin period shorter than one
hour. Once the conditions for the dynamo to work are met, we assume a delay
for the appearance of the magnetic field on the surface of the white dwarf.
This timescale depends on the mass and the temperature of the white dwarf
(Ginzburg et al., 2022, their fig. 3). Here, we only show tracks for white
dwarfs of the canonical mass for white dwarfs in CVs ($0.8$ M⊙, Zorotovic et
al. 2011) and assume a diffusion timescale of $10^{8}$ yr.

We assumed a magnetic field of $1$ kG for the donor star independent of its
mass and the generated white dwarf magnetic field to be of $60$ MG. The
synchronization timescale as defined in Schreiber et al. (2021) was set to $1$
Myr.

###  A.6 Other parameters

We assumed a maximum evolutionary time of $10$ Gyr. We start all our
simulations with a detached post-common-envelope binary assuming that the
formation of the white dwarf takes $300$ Myr. We calculated the evolution for
a white dwarf mass of $0.8$ M⊙ and donor masses of $0.15$ and $0.6$ M⊙. The
initial orbital periods of the post common envelope binary were assumed to be
between $0.1-1$ days.

##  Appendix B Observed period bouncers

In order to be able to compare our evolutionary tracks with observed systems,
we compiled a list of observed period bouncers (Table 1). We only included
systems with measured orbital period as well as donor mass and/or accretion
rate.

Three out of $20$ systems contain a magnetic white dwarf which corresponds to
a fraction of $15\pm 7$ per cent in rough agreement with the number of
magnetic CVs in the SDSS sample (Inight et al., 2023a, b) but smaller than in
the volume limited 150 pc sample (Pala et al., 2020).

We found one system that could perhaps be a detached magnetic CV: SMSSJ
160639.78$-$100010.7 consists of a magnetic white dwarf with a brown dwarf
companion orbiting their common center of mass in $92$ minutes (Kawka et al.,
2021). While for the measured white dwarf mass, the temperature seems to be
slightly too hot for the white dwarf to be crystallizing, the system could be
explained by a CV with a white dwarf that crystallized before the onset of
mass transfer. In this scenario, the mass transfer must have started below the
gap with a low-mass M-dwarf companion
($\raisebox{-1.72218pt}{$\stackrel{{\scriptstyle<}}{{\scriptstyle\sim}}$}0.1$
M⊙). Two promising candidate detached magnetic period bouncers are ZTF
J014635.73+491443.1 and SDSS J121209.31+013627.7 (Hakala et al., 2022; Guidry
et al., 2021). We do not list these candidates in Table 1 because of
uncertainties related to the measured period or the nature of the donor star.
Interestingly, ZTF J014635.73+491443.1 is nearby (56 pc), which might indicate
that such systems are not rare.

Table 1: Properties of confirmed and candidate period bouncers.

System | $P_{\rm orb}$ | $\langle\dot{M}_{\rm d}\rangle$ | $M_{\rm d}$ | ref.
---|---|---|---|---
(min) | ($10^{-10}$ M⊙ yr-1) | (M⊙) |
SDSS J143544.02$+$233638.7 | $78.00$ | $0.32^{+0.08}_{-0.05}$ |  | 1, 2
SDSS J143317.78$+$101123.3 | $78.10$ | $0.40^{+0.20}_{-0.10}$ | $0.0571\pm 0.001$ | 1, 2
V455 And | $81.08$ | $0.22^{+0.10}_{-0.07}$ |  | 1, 2
SDSS J150137.22$+$550123.3 | $81.85$ | $0.90\pm 0.30$ | $0.061\pm 0.004$ | 1, 2
SDSS J103533.02$+$055158.4 | $82.22$ | $0.18^{+0.06}_{-0.04}$ | $0.0475\pm 0.001$ | 3, 2
SDSS J150240.98$+$333423.9 | $84.83$ | $0.6^{+0.3}_{-0.2}$ |  | 4
EG Cnc | $86.36$ | $0.2^{+0.03}_{-0.02}$ |  | 4
1RXS J105010.8$-$140431 | $88.56$ | $0.332^{+0.028}_{-0.018}$ |  | 4
QZ Lib | $92.36$ | $0.27^{+0.15}_{-0.09}$ | $0.032\pm 0.012$ | 4
GD 552 | $102.73$ | $0.243^{+0.023}_{-0.018}$ |  | 5, 6
BW Scl | $78.23$ | $0.483^{+0.014}_{-0.012}$ | $0.051\pm 0.006$ | 7, 6
WZ Sge | $81.63$ | $0.52^{+0.014}_{-0.01}$ |  | 8, 6
SDSS J080434.20$+$510349.2 | $84.97$ | $0.32\pm 0.03$ |  | 8, 6
SDSS J105754.25$+$275947.5 | $90.42$ | $0.60\pm 0.20$ | $0.0436\pm 0.0020$ | 9
GW Lib | $76.78$ |  | $0.050\pm 0.007$ | 10
EZ Lyn | $84.66$ |  | $0.042\pm 0.014$ | 11
CRTS J122221.6$-$311525 | $109.80$ |  | $0.036\pm 0.010$ | 12
SDSS J125044.42$+$154957.4aa$a$Magnetic systems that are presumably period bouncers and the detached magnetic white dwarf plus brown dwarf binary SMSSJ 160639.78$-$100010.7 which might be a detached CV. | $86.30$ | $1.7\times 10^{-4}$ |  | 13
V379 Viraa$a$Magnetic systems that are presumably period bouncers and the detached magnetic white dwarf plus brown dwarf binary SMSSJ 160639.78$-$100010.7 which might be a detached CV. | $88.40$ | $5.3\times 10^{-4}$ |  | 13
SDSSJ 151415.65$+$074446.5aa$a$Magnetic systems that are presumably period bouncers and the detached magnetic white dwarf plus brown dwarf binary SMSSJ 160639.78$-$100010.7 which might be a detached CV. | $88.70$ | $5.9\times 10^{-4}$ |  | 13
SMSSJ 160639.78$-$100010.7aa$a$Magnetic systems that are presumably period bouncers and the detached magnetic white dwarf plus brown dwarf binary SMSSJ 160639.78$-$100010.7 which might be a detached CV. | $92.00$ |  | $0.065\pm 0.005$ | 14
111

(1) Littlefair et al. (2008) (2) McAllister et al. (2019); (3) Littlefair et
al. (2006); (4) Pala et al. (2018); (5) Unda-Sanzana et al. (2008); (6) Pala
et al. (2022); (7) Neustroev & Mäntynen (2023); (8) Zharikov et al. (2008);
(9) McAllister et al. (2017); van Spaandonk et al. (2010); (11) Amantayeva et
al. (2021); (12) Neustroev et al. (2017); (13) Muñoz-Giraldo et al. (2023);
(14) Kawka et al. (2021)


# Radio Frequency Signal based Human Silhouette Segmentation: A Sequential
Diffusion Approach

Penghui Wen2,1, Kun Hu2,2,*, Dong Yuan3,3, Zhiyuan Ning3,4 Changyang Li4,5 and
Zhiyong Wang2,6 * Corresponding author.  2School of Computer Science, The
University of Sydney, Darlinton, NSW, Australia
3School of Electrical and Computer Engineering, The University of Sydney,
Darlinton, NSW, Australia
4Sydney Polytechnic Institute, Haymarket, NSW, Australia
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Radio frequency (RF) signals have been proved to be flexible for human
silhouette segmentation (HSS) under complex environments. Existing studies are
mainly based on a one-shot approach, which lacks a coherent projection ability
from the RF domain. Additionally, the spatio-temporal patterns have not been
fully explored for human motion dynamics in HSS. Therefore, we propose a two-
stage Sequential Diffusion Model (SDM) to progressively synthesize high-
quality segmentation jointly with the considerations on motion dynamics.
Cross-view transformation blocks are devised to guide the diffusion model in a
multi-scale manner for comprehensively characterizing human related patterns
in an individual frame such as directional projection from signal planes.
Moreover, spatio-temporal blocks are devised to fine-tune the frame-level
model to incorporate spatio-temporal contexts and motion dynamics, enhancing
the consistency of the segmentation maps. Comprehensive experiments on a
public benchmark - HIBER demonstrate the state-of-the-art performance of our
method with an IoU 0.732. Our code is available at
https://github.com/ph-w2000/SDM.

###### Index Terms:

Wireless sensing, semantic segmentation, diffusion model, radio frequency

##  I Introduction

Human Silhouette Segmentation (HSS) aims to identify and separate human
figures from the background in images. HSS has been used in a wide range of
applications including human re-identification [1], fall detection [2] and
gait recognition [3, 4]. The primary methodology involved the use of optical
cameras for the segmentation of the human silhouette from RGB images [5, 6,
7]. With the great success of deep learning methods across various domains [8,
9, 10, 11], such techniques have been adopted for HSS as well using optical
cameras, where they have demonstrated encouraging performance [12]. However,
the inherent physical limitations result in significant performance
degradation in capturing fine-grained human information, especially in low
illumination and occlusion conditions. Moreover, optical cameras may give rise
to privacy infringement issues [13]. Therefore, researchers delved into
privacy-preserving wireless sensors as potential solutions in such intricate
environments.

Figure 1: Human silhouette segmentation based on RF signals. Processed a)
horizontal and b) vertical heatmaps converge to generate the c) silhouette
map; and d) a 3D space represents an overall procedure of the RF-based
segmentation task.

With the advancement of wireless technologies, wireless signals have found
extensive applications in recognising human movement. For instance, in the
domain of gesture recognition, a WiFi-based zero-effort cross-domain dataset
was developed by [13] for gesture recognition, while a mmWave data
augmentation framework was devised by [14] for their gesture recognition
system. In the field of pose estimation, the pioneer work proposed by [15]
attempted to perceive human and estimate 2D human pose using WiFi signals.
Subsequently, the use of WiFi signals was extended by [16] to achieve dense 3D
human pose estimation through deep learning architectures. MobiRFPose [17] was
proposed to use radio frequency (RF) signal to estimate 3D human pose with a
lightweight model and OCHID-Fi [18] introduced the first RF-based 3D hand pose
estimation method. More recently, RFMask [19] framework has been introduced,
which is the initial endeavor to generate human silhouette maps based on
wireless RF signals.

As shown in Fig. 1, in this study, HSS is conducted using horizontal and
vertical heatmaps processed from the collected RF signals. HSS characterizes
human motion information including position, orientation and shape, and
generates silhouette mask on the imaging plane orthogonal to the two signal
heatmap planes. Existing methods like RFMask endeavors to generate silhouette
maps in a single forward process, resulting in two major issues: 1) such one-
shot way is hard to preserve a coherent projection of silhouette patterns from
the two signal planes; and 2) the temporal patterns are not adequately
explored, which are crucial for ensuring the consistency and precision
regarding the human motions.

Therefore, we propose a two-stage deep learning framework for HSS, with a
sequential diffusion model (SDM) for progressively synthesising high-quality
HSS results. To condition on the diffusion process, multi-scale guidance is
introduced to characterize foreground human patterns, which encodes the
horizontal and vertical view heatmaps that are processed from RF signals.
Then, cross-view transformation blocks are devised to inject the multi-scale
patterns into the UNet of the diffusion pipeline such as directional
projection from RF signals. Moreover, conventional diffusion methods typically
focus on singleton image generation. To conduct a sequential HSS to involve
continuous human motion dynamics, the two-stage learning strategy first
optimize the silhouette diffusion pipeline at frame-level, and next a fine-
tuning step is adopted for enhancing the consistency regarding the contexts at
sequence-level with our proposed spatio-temporal blocks. Extensive experiments
on a public benchmark - HIBER demonstrate the effectiveness of the proposed
method, showcasing the state-of-the-art performance with an IoU 0.732.

The primary contributions of this study include:

  * •

The first diffusion approach for human silhouette segmentation task from RF
based multi-view heatmaps with a novel two-stage learning strategy.

  * •

A cross-view transformation block (CTB) to guide the diffusion pipeline in a
multi-scale manner to inject patterns such as directional projection from RF
signals.

  * •

A spatio-temporal block (STB), to fine-tune a singleton frame-based model to
involve sequential context and motion dynamics.

##  II RELATED WORK

Firstly, we introduce both optical camera-based and wireless sensor-based HSS.
Subsequently, we delve into the evolution of diffusion models.

###  II-A Optical Cameras based HSS

Over the past decade, optical based methods with cameras have shown
encouraging outcomes in HSS. For example, Mask R-CNN [20] utilised a region-
based convolution method to perform HSS and PointRend [5] proposed a
subdivision strategy for computing high-resolution silhouette maps
efficiently. Segformer [12] eliminated the need for positional embedding and
intricate decoder design used in conventional Transformer structure.
Mask2Former [21] introduced masked attentions to limit attention within the
foreground region of the predicted mask, leading to more efficient computing.
Recently, the Segment Anything Model (SAM) [6] demonstrated a significantly
potent segmentation capability and propelled image segmentation into the era
of foundation models.

###  II-B Wireless Sensors based HSS

Unlike optical cameras, wireless signals including LiDAR point clouds, WiFi
signals, and radio frequency (RF) signals are minimally affected by complex
environmental factors such as low illumination and occlusion. An efficient and
scalable LiDAR-based online mapping method was studied [22], and [23]
introduced a latent memory mechanism to leverage temporal context information
from sequential LiDAR inputs. In contrast to LiDAR which offers high
definition but comes with an expensive cost and susceptibility to weather,
WiFi is cheaper and power efficient [24]. The first study to detect
individuals using pervasive WiFi signals was proposed in [15]. A 3D skeleton-
based human pose estimation system was explored in [25] and Wi-Mesh [26]
leveraged WiFi signals to construct 3D human mesh. RF signals are with broader
bandwidth can provide more spatial information for more precise segmentation.
RFMask [19] performs HSS in an orthogonal plane using two signal planes in a
single forward process. However, its one-shot generation process could miss
fine-grained patterns and struggle to preserve coherent transformation from
signal planes to image plane.

###  II-C Diffusion Methods

Diffusion models [27, 28, 29, 30], have emerged as powerful generative models
that are capable of synthesizing high-quality images. Following their success
in visual content generation, these models have been adapted for segmentation
tasks, such as DDP [31], the aggregation & decoupling framework [32] and LD-
ZNet[33]. Despite this progress, diffusion models still face limitations when
it comes to segmenting wireless signals (in heatmaps) from multi-views.
Therefore, our work aims to pioneer a novel diffusion-based method
specifically tailored for HSS task using RF-based multi-view heatmaps. We also
introduce a cross-view transformation block to guide the diffusion pipeline in
a multi-scale manner for characterizing human related patterns and a spatio-
temporal block to fine-tune the frame-level model to incorporate spatio-
temporal contexts and motion dynamics.

##  III METHODOLOGY

Figure 2: Illustration of the proposed architecture: sequential diffusion
model (SDM). SDM is a UNet-based network composed of a ortho-cross encoder
$H_{\text{OCH}}$ and a silhouette diffusion network $H_{\text{SDN}}$ with
multiple cross-view transformation blocks (CTBs) and spatio-temporal blocks
(STBs). The ortho-cross encoder $H_{\text{OCH}}$ encodes human motion patterns
from the given paired horizontal heatmap and vertical heatmaps with residual
blocks. To transfer rich multi-scale features extracted from $H_{\text{OCH}}$
to the $H_{\text{SDN}}$ , we propose to use cross-attention based cross-view
transformation blocks (CTBs) that are embedded in different layers of
$H_{\text{SDN}}$. During the Sequence-Level Silhouette Fine-Tuning stage, we
fix the weights of the components in the frame-level stage and insert an
spatio-temporal block (STB) after every CTB block for fine-tuning, to
formulate thespatio-temporal cohension.

As illustrated in Fig. 2, a diffusion model
$p_{\theta}(\mathbf{y}|\mathbf{c}_{\text{\text{hor}}},\mathbf{c}_{\text{ver}})$
is devised conditioning on a given horizontal heatmap
$\mathbf{c}_{{\text{hor}}}$ and a vertical heatmap
$\mathbf{c}_{{\text{ver}}}$. The desired output silhouette map
$\mathbf{\hat{y}}$ is expected to adhere to the requirement of matching human
motions information of the ground truth $\mathbf{y}$. In particular, the
predicted $i^{\text{th}}$ pixel label $\hat{y}_{i}$ is expected to align with
the corresponding ground truth $y_{i}$. The overall pipeline consists of two
stages: frame-level silhouette diffusion stage and sequence-level silhouette
fine-tuning stage. The frame-level silhouette diffusion phase aims to perform
silhouette segmentation based on individual frames, without taking into
account the temporal relationships between consecutive frames. Meanwhile, the
sequence-level silhouette fine-tuning phase is devised to equip our model to
address silhouette segmentation in a temporal context.

###  III-A Preliminary

The foundation of our SDM approach relies on the denoising diffusion
probabilistic model (DDPM) [27]. Denoting $\mathbf{y}_{0}=\mathbf{y}$, DDPM
involves creating a diffusion process that incrementally introduces noise to
data sampled from the target distribution $\mathbf{y}_{0}\sim
q(\mathbf{y}_{0})$, while the reverse denoising process is designed to learn
an inverse mapping. The denoising process ultimately transforms the isotropic
Gaussian noise $\mathbf{y}_{\text{t}}\sim\mathcal{N}(0,\mathbf{I})$ into the
target data distribution over $t$ steps. In essence, this approach breaks down
a challenging distribution-modelling task into a series of simple denoising
problems. The forward diffusion path of DDPM forms a Markov chain
characterized by the following conditional distribution:

$\displaystyle
q(\mathbf{y}_{\text{t}}\mid\mathbf{y}_{\text{t-1}})=\mathcal{N}(\mathbf{y}_{\text{t}};\sqrt{1-\beta_{\text{t}}}\mathbf{y}_{\text{t-1}},\beta_{\text{t}}\mathbf{I}),$
(1)

where $t\in[1,T]$ and $\beta_{1}$,$\beta_{2}$,…,$\beta_{\text{t}}$ follow a
fixed variance schedule with $\beta_{\text{t}}\in(0,1)$. With the notation
${\alpha_{\text{t}}}=1-{\beta_{\text{t}}}$ and
$\hat{\alpha}_{\text{t}}=\prod_{s=0}^{t}\alpha_{s}$, we can derive samples
from $q(\mathbf{y}_{\text{t}}|\mathbf{y}_{0})$ in a closed form at any given
timestep $t$:
$\mathbf{y}_{\text{t}}=\sqrt{{\hat{\alpha}_{\text{t}}}}\mathbf{y}_{0}+\sqrt{{1-\hat{\alpha}_{\text{t}}}}\epsilon$,
where $\epsilon\sim\mathcal{N}(0,\mathbf{I})$ [29]. A deep neural network can
approximate the true posterior
$q(\mathbf{y}_{\text{t-1}}|\mathbf{y}_{\text{t}})$ by predicting the mean and
variance of $\mathbf{y}_{\text{t-1}}$. Specifically, two additional heatmaps
$\mathbf{c}_{\text{hor}}$ and $\mathbf{c}_{\text{ver}}$ are used as conditions
in this HSS task. The process can be achieved through the following
parameterization:

$\displaystyle\begin{aligned}
p_{\theta}(\mathbf{y}_{\text{t-1}}|\mathbf{y}_{\text{t}},\mathbf{c}_{\text{hor}},\mathbf{c}_{\text{ver}})=&\mathcal{N}(\mathbf{y}_{\text{t-1}};\mu_{\theta}(\mathbf{y}_{\text{t}},t,\mathbf{c}_{\text{hor}},\mathbf{c}_{\text{ver}}),\\\
&\Sigma_{\theta}(\mathbf{y}_{\text{t}},t,\mathbf{c}_{\text{hor}},\mathbf{c}_{\text{ver}})),\end{aligned}$
(2)

where $\mu_{\theta}$ predicts the mean value and $\Sigma_{\theta}$ predicts
the variance value of $\mathbf{y}_{\text{t-1}}$

###  III-B Frame-Level Silhouette Diffusion

The frame-level silhouette diffusion stage consists of an ortho-cross heatmap
encoder $H_{\text{OCH}}$ and a UNet-based silhouette diffusion network (SDN)
$H_{\text{SDN}}$. $H_{\text{OCH}}$ aims to formulate multi-scale features from
the given horizontal and vertical heatmaps, while $H_{\text{SDN}}$ leverages
the multi-scale features to generate silhouette segmentation maps. The details
of the two components are discussed as follows.

####  III-B1 Ortho-Cross Heatmap Encoder

The ortho-cross encoder $H_{\text{OCH}}$ encodes human motion patterns from
the given paired horizontal heatmap $\mathbf{c}_{{\text{hor}}}$ and vertical
heatmap $\mathbf{c}_{{\text{ver}}}$ with residual blocks. The encoded features
extracted from the two heatmaps are subsequently combined and processed with a
self-attention mechanism in pursuit of the pairwise consistency across the two
planes. As different layers of the encoder observes heatmap associated
patterns in different scales, leveraging their feature maps as multi-scale
features is expected to obtain high-quality silhouette maps, particularly with
regard to human limbs. Therefore, we construct a stacked feature
representation
$\mathbf{F}_{\text{cond}}=[\mathbf{F}^{1}_{\text{cond}},\mathbf{F}^{2}_{\text{cond}},...,\mathbf{F}^{\text{M}}_{\text{cond}}]$,
where M represents the number of scales of $H_{\text{OCH}}$ output. To this
end, the multi-scale features $\mathbf{F}_{\text{cond}}$ are ready to be used
for guiding the silhouette diffusion network for generating segmentation
masks.

####  III-B2 Discrete Segmentation Embedding

Diffusion models are typically designed for continuous data, whilst silhouette
segmentation map involves binary discrete labels. To address this, we adopt
class embeddings, which employ a learnable embedding layer to map discrete
labels $\mathbf{y}_{\text{0}}$ into a high-dimensional continuous space,
normalized using a sigmoid function. The embedded features
$\mathbf{y}^{\prime}_{\text{0}}$ will then be processed by our diffusion
model. In the final stage of the diffusion model, a convolution layer is
employed to convert the embeded features $\mathbf{y}^{\prime}_{\text{0}}$ into
binary discrete labels $\mathbf{y}_{\text{0}}$ . For notation simplicity, we
do not differentiate $\mathbf{y}_{\text{0}}$ and
$\mathbf{y}^{\prime}_{\text{0}}$ in the following discussions.

####  III-B3 Silhouette Diffusion Network

We predict the noise
$\epsilon_{\theta}(\mathbf{y}_{\text{t}},t,\mathbf{c}_{\text{hor}},\mathbf{c}_{\text{ver}})$
using the silhouette diffusion network $H_{\text{SDN}}$. $t$ is the diffusion
timestep, and the noisy frame $\mathbf{y}_{\text{t}}$ is passed through
$H_{\text{SDN}}$ to predict the noise added at step $t$. $H_{\text{SDN}}$ is a
UNet-based network and additionally with our cross-view transformation blocks
(CTBs). We insert a CTB between every two blocks of original UNet. CTB aims to
guide this denoising process to inject patterns such as directional projection
from RF signals based on cross-attentions, which inject the obtained
$\mathbf{F}_{\text{cond}}$ into $H_{\text{SDN}}$ to embed multi-scale features
in every layer of $H_{\text{SDN}}$.

Formally, let $\mathbf{F}^{\text{b}}_{\text{frame}}$ be the input feature map
regarding HSS in the $b^{\text{th}}$ UNet block of $H_{\text{SDN}}$. For the
corresponding CTB, the keys $\mathbf{K}$ and values $\mathbf{V}$ are based on
$\mathbf{F}_{\text{cond}}$, while queries $\mathbf{Q}$ are obtained from
$\mathbf{F}^{\text{b}}_{\text{frame}}$. Formally, the computations of CTB can
be formulated as:

$\displaystyle\begin{aligned}
&\mathbf{Q}=\textbf{W}^{\text{b}}_{\text{Q}}\circledast\mathbf{F}^{\text{b}}_{\text{frame}},\mathbf{K}=\textbf{W}^{\text{b}}_{\text{K}}\circledast\mathbf{F}^{\text{b}}_{\text{cond}},\mathbf{V}=\textbf{W}^{\text{b}}_{\text{V}}\circledast\mathbf{F}^{\text{b}}_{\text{cond}},\\\
&\bar{\mathbf{F}}_{\text{frame}}^{\text{b}}=\mathbf{W}^{\text{b}}_{\text{CTB}}\text{softmax}(\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d_{\text{b}}}})\mathbf{V}+\mathbf{F}^{\text{b}}_{\text{frame}},\end{aligned}$
(3)

where $\textbf{W}^{\text{b}}_{\text{Q}}$, $\textbf{W}^{\text{b}}_{\text{K}}$
and $\textbf{W}^{\text{b}}_{\text{V}}$ are learnable $1\times 1$ convolution
filters; and $\mathbf{W}^{\text{b}}_{\text{CTB}}$ is a learned weight matrix
and $d_{\text{b}}$ is the vectorized dimension of these variables. Note that
to compute $\bar{\mathbf{F}}_{\text{frame}}^{\text{b}}$, $\mathbf{Q}$,
$\mathbf{K}$, and $\mathbf{V}$ are reshaped as matrices where each row
indicates the channels of a pixel, and afterwards they are reshaped back to
the original shape. The final output
$\bar{\mathbf{F}}_{\text{frame}}^{\text{b}}$ is then processed by the
$(b+1)^{\text{th}}$ UNet block in $H_{\text{SDN}}$.

####  III-B4 Frame-Level Optimization

For the training of the denoising process, we generate a noisy sample
$\mathbf{y}_{\text{t}}\sim q(\mathbf{y}_{\text{t}}|\mathbf{y}_{0})$ by
introducing a Gaussian noise $\epsilon$ to the ground truth $\mathbf{y}_{0}$.
Subsequently,
$\epsilon_{\theta}(\mathbf{y}_{\text{t}},t,\mathbf{c}_{\text{hor}},\mathbf{c}_{\text{ver}})$
is trained to forecast the introduced noise. A mean squared error (MSE) loss
$\mathcal{L}_{\mathrm{MSE}}$ is adopted for the optimization:

$\displaystyle\mathcal{L}_{\mathrm{MSE}}=\mathbb{E}_{t\sim[1,T],\mathbf{y}_{0}\sim
q(\mathbf{y}_{\mathbf{0}}),\epsilon}\|\epsilon-\epsilon_{\theta}(\mathbf{y}_{\text{t}},t,\mathbf{c}_{\text{hor}},\mathbf{c}_{\text{ver}})\|^{2}.$
(4)

The quality of the generated silhouette maps is further assessed using a
cross-entropy loss $\mathcal{L}_{\mathrm{CE}}$, and a dice loss
$\mathcal{L}_{\mathrm{DICE}}$.

$\displaystyle\mathcal{L}_{\mathrm{CE}}(\mathbf{\hat{y}},\mathbf{y})=-\sum_{i}y_{i}\log(\hat{y}_{i}),$
(5)
$\displaystyle\mathcal{L}_{\mathrm{DICE}}(\mathbf{\hat{y}},\mathbf{y})=1-\frac{2\sum_{i}y_{i}\hat{y}_{i}}{\sum_{i}y_{i}^{2}+\sum_{i}\hat{y}_{i}^{2}}.$
(6)

Note that we additionally adopt a loss function $\mathcal{L}_{\mathrm{VIB}}$,
which aims to facilitate the learning regarding the variance
$\Sigma_{\theta}$, following the existing practice [29]. To this end, the
overall loss $\mathcal{L}$ is outlined as follows:

$\displaystyle\mathcal{L}=\lambda_{1}\mathcal{L}_{\mathrm{MSE}}+\lambda_{2}\mathcal{L}_{\mathrm{VIB}}+\lambda_{3}\mathcal{L}_{\mathrm{CE}}+\lambda_{4}\mathcal{L}_{\mathrm{DICE}},$
(7)

where $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$ and $\lambda_{4}$ are hyper-
parameters to balance the weights of different loss terms.

###  III-C Sequence-Level Silhouette Fine-Tuning

In the sequence-level silhouette fine-tuning stage, our model is devised to
accept sequential heatmaps as conditions to generate a sequence of silhouette
maps with motion dynamics. In this stage, the weights of the components in the
frame-level stage are fixed, and we additionally introduce spatio-temporal
blocks (STBs) $H_{\text{STB}}$ based on cross-attentions. We insert an STB
after every CTB block for fine-tuning, to formulate spatio-temporal cohesion
regarding the motion dynamics.

Due to the minimal differences between consecutive frames and the high
computational complexity of an attention mechanism for all frames, we apply
cross-attentions in a two-step manner. Let
$\mathbf{F}^{\text{b}}_{\text{seq,n}}$ be the output feature map for the
$n^{\text{th}}$ frame of the $b^{\text{th}}$ CTB block in $H_{\text{SDN}}$,
where $\text{n}\in[2,\text{N}]$. First, a cross-attention is adopted between a
given frame $\mathbf{F}^{\text{b}}_{\text{seq,n}}$ and its two preceding
frames, $\mathbf{F}^{\text{b}}_{\text{seq},1}$ and
$\mathbf{F}^{\text{b}}_{\text{seq,n-1}}$. The leading frame
$\mathbf{F}^{\text{b}}_{\text{seq},1}$ serves to define the target’s position
and orientation, while the former frame
$\mathbf{F}^{\text{b}}_{\text{seq,n-1}}$ contributes to detailing the recent
motion and shape.

Formally, the query embedding is formed by
$\mathbf{F}^{\text{b}}_{\text{seq,n}}$, the key and value embeddings from
$\mathbf{F}^{\text{b}}_{\text{seq},1}$ and
$\mathbf{F}^{\text{b}}_{\text{seq,n-1}}$ . The cross-attention can be
formulated as:

$\displaystyle\begin{aligned}
&\mathbf{Q}=\textbf{W}^{\text{b}}_{\text{Q}}\circledast\mathbf{F}^{\text{b}}_{\text{seq,n}},\quad\mathbf{K}=\textbf{W}^{\text{b}}_{\text{K}}\circledast([\mathbf{F}^{\text{b}}_{\text{seq},1}\cdot\mathbf{F}^{\text{b}}_{\text{seq,n-1}}]),\\\
&\mathbf{V}=\textbf{W}^{\text{b}}_{\text{V}}\circledast([\mathbf{F}^{\text{b}}_{\text{seq},1}\cdot\mathbf{F}^{\text{b}}_{\text{seq,n-1}}]),\\\
&\bar{\mathbf{F}}^{\text{b}}_{\text{seq,n}}=\operatorname{softmax}(\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d}})\mathbf{V},\end{aligned}$
(8)

where [$\cdot$] is a concatenation operation through the channel dimension and
$d$ is the dimension of the associated variables.
$\bar{\mathbf{F}}^{\text{b}}_{\text{seq,n}}$ is with a lower-dimension now,
and a general sequential self-attention can be further adopted in the second
step. The final output feature map is processed by the $(b+1)^{\text{th}}$
UNet block in $H_{\text{SDN}}$.

##  IV Experiments

###  IV-A Dataset & Implementation Details

To demonstrate the effectiveness of the proposed method, we followed the same
evaluation protocol and adopted HIBER dataset, the only publicly available
dataset in RF-based HSS. It consists of WALK and MULTI subsets [19]. In total,
152 group sequences in WALK set (walking with only one person in the scene)
and 120 group sequences in MULTI set (walking with multiple persons in the
scene) were collected. Each group sequence contains 590 silhouette frames with
590 horizontal and 590 vertical heatmaps. Considering a significant inter-
group difference (between environment number 1 and others) in the distribution
of the correlation between heatmap highlights and the ground truth human
position, we eliminated these problematic data from our experiments. The
evaluation was conducted both quantitatively and qualitatively, and
intersection over union (IoU) was adopted as metrics.

For the training of the first stage - frame-level silhouette diffusion, an
AdamW optimizer was adopted with a learning rate $2\times 10^{-4}$. Training
in this stage is conducted on one NVIDIA RTX3090, with a batch size 16. The
multi-scale features $\mathbf{F}_{\text{cond}}$ consist of three scales, i.e.,
$5\times 6$, $10\times 12$ and $20\times 25$. We set the values for
$\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$ and $\lambda_{4}$ as 3, 1, 1 and
1. For the second stage - sequence-level silhouette fine-tuning, we employed
two settings regarding the sequence length: 4 and 12, where the first setting
was trained on one NVIDIA RTX3090 with a batch size 6 and the second setting
was trained on one NVIDIA RTXA6000 with a batch size 5. The two stages were
trained with 500 noising steps and a linear noise schedule.

###  IV-B Overall Performance

Table I lists quantitative comparisons between our proposed SDM and the state-
of-the-art RFMask method. Note that SDM with sequence length 1 is with a
setting only involving the first stage and processes singleton frames. The
observation reveals that as the sequence length increases, the performance of
SDM improves on both the WALK and MULTI sets, and SDM achieves the highest IoU
score for both sets with a sequence length of 12. Compared to RFMask with the
same sequence length, SDM outperforms RFMask on both sets. This suggests that
SDM is capable of generating silhouette maps of superior quality.

TABLE I: Quantitative comparison with RFMask. Model | Seq. Length | WALK | MULTI
---|---|---|---
RFMask | 4 | 0.681 | 0.682
RFMask | 12 | 0.706 | 0.711
SDM | 1 | 0.689 | 0.686
SDM | 4 | 0.714 | 0.702
SDM | 12 | 0.732 | 0.716
Figure 3: Qualitative comparison with RFMask.

Fig. 3 provides a detailed visual comparison between our approach and RFMask
on the WALK and MULTI sets of the HIBER dataset. The RFMask results are
sourced from the authors’ paper. It can be observed that SDM exhibits a better
consistency preservation in human shape. For instance, regarding the example
in the first row, the RFMask result resembles an alien-looking and produces
implausible human leg volumes, whilst our result is more close to realistic
human. In the second example, our model is capable of generating a slightly
more realistic neck, whereas the RFMask result akins to a goitre human.

###  IV-C Ablation Study

Figure 4: Ablation studies on CTB. The first example is from the WALK set,
while the second is from MULTI set.

We conducted ablation studies to demonstrate the effectiveness of the proposed
mechanisms. Table II illustrates the impact of CTB and STB modules. For the
base scenario, neither CTB units nor STB is used. This setup simply
concatenates multi-scale features $\mathbf{F}_{\text{cond}}^{\text{b}}$ with
$\mathbf{F}^{\text{b}}_{\text{frame}}$ and produces singleton silhouette
frames. We integrate CTB into the base, denoted as base + CTB. In comparison
with the base, base + CTB exhibits improvements in both sets regarding IoU. As
shown in Fig. 4, base without CTB leads to inaccurate inter-plane projection,
causing a frontal display instead of the intended side display. Conversely,
when CTB is employed, the outcomes clearly demonstrate the rightward movement
of the target. Therefore, the results indicate CTB is capable of modelling the
intricate directional information from signal planes to silhouette plane.

Figure 5: Ablation studies on STB. The first example is from the WALK set,
while the second is from MULTI set.

Next, we integrate our STB into base + CTB, denoted as base + CTB + STB, which
takes 12 sequence frames as input. The results indicate an enhancement in both
sets quantitatively. As shown in Fig. 5, consistency issue occurs for base +
CTB that the fourth frames are erroneously generated with an incorrect
direction, while employing base + CTB + STB maintains consistent directional
information. Therefore, our STB attention module can effectively enhance the
consistency of sequential context and motion dynamics to improve the quality
of silhouette maps.

TABLE II: Ablation studies on the proposed mechanisms. Methods | Seq. length | WALK | MULTI
---|---|---|---
Base | 1 | 0.669 | 0.671
Base + CTB | 1 | 0.689 | 0.686
Base + CTB + STB | 12 | 0.732 | 0.716

##  V Conclusion

We present a two-stage diffusion-based HSS approach - SDM using RF signals. We
introduced CTB and STB modules to formulate frame-level and sequence-level
patterns. These allow SDM progressively synthesizing high-quality segmentation
maps considering motion dynamics. The main limitation of SDM is the coarse
segmentation of each person’s limbs in scenes with multiple persons. We are
considering the use of a region proposal network to treat each person
individually as a potential solution in the future. Additionally, addressing
issues such as rough body edges and missing arms is essential to achieve more
realistic human generation.

## Acknowledgement

This study was partially supported by Australian Research Council (ARC) grant
DP210102674.

## References

  * [1] Yuxiang Guo, Cheng Peng, Chun Pong Lau, and Rama Chellappa,  “Multi-modal human authentication using silhouettes, gait and rgb,”  in FG. IEEE, 2023, pp. 1–7.
  * [2] Miao Yu, Liyun Gong, and Stefanos Kollias,  “Computer vision based fall detection by a convolutional neural network,”  in ICMI, 2017, pp. 416–420.
  * [3] Kun Hu, Zhiyong Wang, Kaylena Ehgoetz Martens, and Simon Lewis,  “Vision-based freezing of gait detection with anatomic patch based representation,”  in ACCV, 2019.
  * [4] Kun Hu, Shaohui Mei, Wei Wang, Kaylena A Ehgoetz Martens, Liang Wang, Simon JG Lewis, David D Feng, and Zhiyong Wang,  “Multi-level adversarial spatio-temporal learning for footstep pressure based fog detection,”  Journal of Biomedical and Health Informatics, 2023.
  * [5] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick,  “Pointrend: Image segmentation as rendering,”  in CVPR, 2020, pp. 9799–9808.
  * [6] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.,  “Segment anything,”  arXiv preprint arXiv:2304.02643, 2023.
  * [7] Tomer Amit, Tal Shaharbany, Eliya Nachmani, and Lior Wolf,  “Segdiff: Image segmentation with diffusion probabilistic models,”  arXiv preprint arXiv:2112.00390, 2021.
  * [8] Penghui Wen, Kun Hu, Wenxi Yue, Sen Zhang, Wanlei Zhou, and Zhiyong Wang,  “Robust audio anti-spoofing with fusion-reconstruction learning on multi-order spectrograms,”  in INTERSPEECH, 2023.
  * [9] Peggy Tang, Kun Hu, Rui Yan, Lei Zhang, Junbin Gao, and Zhiyong Wang,  “OTextsum: extractive text summarisation with optimal transport,”  arXiv preprint arXiv:2204.10086, 2022.
  * [10] Gang Li, Xiang Li, Shanshan Zhang, and Jian Yang,  “Towards more reliable evaluation in pedestrian detection by rethinking “ignore regions”,”  Visual Intelligence, vol. 2, no. 1, pp. 1–13, 2024.
  * [11] Jiyuan Chen, Changxin Gao, Li Sun, and Nong Sang,  “Ccsd: cross-camera self-distillation for unsupervised person re-identification,”  Visual Intelligence, vol. 1, no. 1, pp. 27, 2023.
  * [12] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo,  “Segformer: Simple and efficient design for semantic segmentation with transformers,”  2021, vol. 34, pp. 12077–12090.
  * [13] Yi Zhang, Yue Zheng, Kun Qian, Guidong Zhang, Yunhao Liu, Chenshu Wu, and Zheng Yang,  “Widar3. 0: Zero-effort cross-domain gesture recognition with wi-fi,”  IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 11, pp. 8671–8688, 2021.
  * [14] Yadong Li, Dongheng Zhang, Jinbo Chen, Jinwei Wan, Dong Zhang, Yang Hu, Qibin Sun, and Yan Chen,  “Towards domain-independent and real-time gesture recognition using mmwave signal,”  Trans. Mobile Comput, 2022.
  * [15] Fei Wang, Sanping Zhou, Stanislav Panev, Jinsong Han, and Dong Huang,  “Person-in-WiFi: Fine-grained person perception using WiFi,”  in ICCV, 2019, pp. 5452–5461.
  * [16] Jiaqi Geng, Dong Huang, and Fernando De la Torre,  “Densepose from WiFi,”  arXiv preprint arXiv:2301.00250, 2022.
  * [17] Cong Yu, Dongheng Zhang, Zhi Wu, Chunyang Xie, Zhi Lu, Yang Hu, and Yan Chen,  “Mobirfpose: Portable rf-based 3d human pose camera,”  IEEE Trans. Multimedia, 2023.
  * [18] Shujie Zhang, Tianyue Zheng, Zhe Chen, Jingzhi Hu, Abdelwahed Khamis, Jiajun Liu, and Jun Luo,  “Ochid-fi: Occlusion-robust hand pose estimation in 3d via rf-vision,”  in ICCV, 2023, pp. 15112–15121.
  * [19] Zhi Wu, Dongheng Zhang, Chunyang Xie, Cong Yu, Jinbo Chen, Yang Hu, and Yan Chen,  “Rfmask: A simple baseline for human silhouette segmentation with radio signals,”  IEEE Trans. Multimedia, 2022.
  * [20] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick,  “Mask r-cnn,”  in ICCV, 2017, pp. 2961–2969.
  * [21] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar,  “Masked-attention mask transformer for universal image segmentation,”  in CVPR, 2022, pp. 1290–1299.
  * [22] David Droeschel and Sven Behnke,  “Efficient continuous-time slam for 3d lidar-based online mapping,”  in ICRA. IEEE, 2018, pp. 5000–5007.
  * [23] Enxu Li, Sergio Casas, and Raquel Urtasun,  “Memoryseg: Online lidar semantic segmentation with a latent memory,”  in ICCV, 2023, pp. 745–754.
  * [24] Jingzhi Hu, Tianyue Zheng, Zhe Chen, Hongbo Wang, and Jun Luo,  “Muse-fi: Contactless muti-person sensing exploiting near-field wi-fi channel variation,”  in MobiCom, 2023, pp. 1–15.
  * [25] Yili Ren, Zi Wang, Yichao Wang, Sheng Tan, Yingying Chen, and Jie Yang,  “Gopose: 3d human pose estimation using WiFi,”  IMWUT, vol. 6, no. 2, pp. 1–25, 2022.
  * [26] Yichao Wang, Yili Ren, Yingying Chen, and Jie Yang,  “Wi-mesh: A wifi vision-based approach for 3d human mesh construction,”  in SenSys, 2022, pp. 362–376.
  * [27] Jonathan Ho, Ajay Jain, and Pieter Abbeel,  “Denoising diffusion probabilistic models,”  NeurIPS, vol. 33, pp. 6840–6851, 2020.
  * [28] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet,  “Video diffusion models,”  arXiv:2204.03458, 2022.
  * [29] Alexander Quinn Nichol and Prafulla Dhariwal,  “Improved denoising diffusion probabilistic models,”  in ICML, 2021, pp. 8162–8171.
  * [30] Zhuqiang Lu, Kun Hu, Chaoyue Wang, Lei Bai, and Zhiyong Wang,  “Autoregressive omni-aware outpainting for open-vocabulary 360-degree image generation,”  in AAAI, 2024, vol. 38, pp. 14211–14219.
  * [31] Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo,  “Ddp: Diffusion model for dense visual prediction,”  in CVPR, 2023, pp. 21741–21752.
  * [32] Haonan Wang and Xiaomeng Li,  “Towards generic semi-supervised framework for volumetric medical image segmentation,”  NeurIPS, vol. 36, 2024.
  * [33] Koutilya PNVR, Bharat Singh, Pallabi Ghosh, Behjat Siddiquie, and David Jacobs,  “Ld-znet: A latent diffusion approach for text-based image segmentation,”  in ICCV, October 2023, pp. 4157–4168.


Bacon et al

*Roland Bacon

Univ Lyon, Univ Lyon1, Ens de Lyon, CNRS, Centre de Recherche Astrophysique de
Lyon UMR5574, F-69230, Saint-Genis- Laval, France

# Detector System Challenges of the Wide-field Spectroscopic Survey Telescope
(WST)

Roland Bacon  Martin M. Roth  Paola Amico  Eloy Hernandez  The WST consortium
Univ. Lyon 1, CNRS, France  innoFSPEC, Leibniz-Institut für Astrophysik
Potsdam (AIP), Germany  Institut für Physik und Astronomie, Universität
Potsdam, Germany  European Southern Observatory, Germany
https://www.wstelescope.com<EMAIL_ADDRESS>
(01 November 2022)

###### Abstract

The wide-field spectroscopic survey telescope (WST) is proposed to become the
next large optical/near infrared facility for the European Southern
Observatory (ESO) once the Extremely Large Telescope (ELT) has become
operational. While the latter is optimized for unprecedented sensitivity and
adaptive-optics assisted image quality over a small field-of-view, WST
addresses the need for large survey volumes in spectroscopy with the light-
collecting power of a 10 m class telescope. Its unique layout will feature the
combination of multi-object and integral field spectroscopy simultaneously.
For the intended capacity of this layout a very large number of detectors is
needed. The complexity of the detector systems presents a number of challenges
that are discussed with a focus on novel approaches and innovative detector
designs that can be expected to emerge over the anticipated 20-year timeline
of this project.

###### keywords:

multi-object spectroscopy, integral field spectroscopy, spectroscopic surveys,
CCD, CMOS

††articletype: Article Type

##  1 Introduction

WST is proposed as an innovative 10-m class wide-field spectroscopic survey
telescope with a large field-of-view (5 sq. degree) providing simultaneous
operation of a high multiplex (20,000) multi-object spectrograph facility with
both medium and high resolution modes (MOS), and a giant panoramic integral
field spectrograph (IFS). Through spectroscopic characterization, WST will
leverage the understanding of the data from many other huge imaging surveys
that merely detect and classify sources. WST will achieve transformative
results in most areas of astrophysics (Fig. 1): e.g. the nature and expansion
of the dark Universe, the formation of first stars and galaxies and their role
in the cosmic reionisation, the study of the dark and baryonic material in the
cosmic web, the baryon cycle in galaxies, the formation history of the Milky
Way and dwarf galaxies in the Local Group, characterization of exoplanet
hosts, and the characterization of transient phenomena. The WST telescope and
instruments will be designed as an integrated system and mostly use existing
technology, aiming to minimize its carbon footprint and impact on local
environment. The ambition is to make WST the next large ESO project after the
39m ELT. ESO is the major inter-governmental organization for European
astronomy with a unique suite of large telescopes in Chile. A large
consortium, representing 9 European countries and Australia, has formed to
start a conceptual design study. In the following, we will present the
scientific rationale, the intended layout of the WST, and discuss the specific
challenges for the detector subsystems of this complex facility.

##  2 Science case

Figure 1: WST will be a revolutionary spectroscopic facility addressing many
open questions in astrophysics over a large range in physical scales: from the
formation of the large-scale structures in the early universe (100s of Mpc),
to the interplay of galaxies in the cosmic web (10s of Mpc), to the formation
of our own Galaxy (kpc scales), to the evolution of stars and the formation of
planets around them (sub-pc scales).

Astrophysics is witnessing a golden era with a number of breakthrough
discoveries achieved in the last decade or so, and a large variety of new
instrumentation and programs planned for the next decade. Nonetheless, a
comprehensive understanding of the formation and evolution of structures in
the Universe is still missing. Many key quests in modern astrophysics and
cosmology are still in their infancy and require new data to address them:

  * •

Is the accelerated expansion of the Universe due to an unknown form of energy
or to a modification of General Relativity on large scales?

  * •

What is the interplay between dark, stellar, and gaseous material in galaxies
and how does primordial and metal-enriched gas flow in and out of galaxies at
various scales?

  * •

What is the detailed formation history of our own Galaxy, the Milky Way and of
its satellites?

  * •

What is the origin of the various chemical elements that are crucial to trace
galactic evolution?

  * •

What are the conditions that drive the formation and evolution of extra-solar
planets?

  * •

What are the extreme physical conditions that govern transient events
(explosions, eruptions, and disruptions)?

A vital element to answering these questions will be the availability of vast
all-sky spectroscopic datasets of nearby and distant astrophysical sources at
medium and high resolution delivering quantitative measurements of key
physical parameters (examples include the redshift of galaxies, z -an
indicator of a galaxy distance), the kinematics of galaxies and stars, the
physical and evolutionary properties of galaxies, the metal content and
detailed chemical composition of stars, etc.).

Over the next decades, we expect a deluge of high-quality data delivered by
the upcoming large ground-based (e.g., LSST/VRO, SKA, CTA) and space (e.g.,
JWST, Euclid, Roman Space Telescope, Athena) telescopes. In parallel the Gaia
astrometric mission will have its final release. The imaging capabilities of
some of these new facilities will detect and classify a huge number of
objects. However, to learn what they are, spectroscopic follow-up on the main
representatives as well as rare cases at adequate spectral resolution and
cadence is required. Given the expected number of sources (e.g., 20 billion
galaxies and 17 billion stars down to R 27.5 for the VRO alone), only a
dedicated spectroscopic facility will be able to fully realise the scientific
potential of these wide-field imaging surveys. This has been ably demonstrated
by the many previous major stellar and extragalactic surveys that have had a
huge scientific impact and strong legacy value thanks to their spectroscopic
follow-up on photometric and astrometric observations.

Spectroscopic surveys are usually performed with either multi-object
spectroscopy (MOS) instruments based on fibre technology (e.g., FLAMES at VLT,
Pasquini . 2004) or multi-slit technology (e.g., VIMOS at VLT, Le Fèvre .
2003). New MOS facilities using dedicated 4m-class telescopes are coming on-
line (e.g., DESI, 4MOST, WEAVE). In parallel, new MOS instruments will be
installed on existing 10m-class multi-purpose telescopes (e.g., PFS at Subaru,
MOONS at VLT). These new capabilities will only be able to perform large
surveys of bright sources (using the 4m-class facilities full time) or small
surveys of faint sources (using part-time access to the 10m-class MOS). The
upcoming giant telescopes (ELT, TMT, GMT) are also limited, by technical
feasibility, to a small field-of-view and a correspondingly small multiplex
capability (e.g., a multiplex of about 300 over 0.12 square degree for MOSAIC
at the ELT). Consequently, only a new spectroscopic survey facility on a
dedicated wide field-of-view 10m-class telescope equipped with a very high-
multiplex MOS will be able to follow-up on the development of upcoming massive
multi-wavelength imaging and astrometric capabilities and datasets, as well as
conducting efficient follow-up of transients and variable phenomena.

Separately, the advent of panoramic integral-field spectrographs (IFS) on
10m-class telescopes (e.g., MUSE, Bacon . 2010) has been made possible by
well-established IFS expertise in Europe and in particular technological
advances in large-format glass slicers. The field of view of the latest IFS,
combined with their exquisite sensitivity, is now sufficiently large (e.g., 1
arcmin2 for MUSE) to enable very deep spectroscopic surveys in small patches
of the sky. The MOS and IFS approaches to spectroscopy are highly
complementary as they probe the sky in very different ways: sparse sampling of
sources over a wide area for the MOS and densely packed faint sources in a
small area for the IFS. A panoramic IFS also has the unique property of not
requiring source pre-selection as needed for the MOS and of providing a 2D
continuous sampling of the sky for studies of faint extended spectral
signatures (e.g., circumgalactic gas emission). The main limitation of the
current panoramic IFS is their relatively small field-of-view. An IFS with a
larger-field of view is highly desirable, but given the very large number of
spectrographs involved, is not compatible with the existing 10m-class
telescope infrastructures. Instead, a new dedicated facility with the right
infrastructure is the most appropriate way forward.

The demand for a 10m-class telescope dedicated to spectroscopic surveys is an
interest shared worldwide and figures explicitly in many national strategic
science plans (e.g., US 2020 decadal survey, 2016-2025 decadal plan for
Australian astronomy, Canadian astronomy long range plan 2020-2030). In Europe
in particular, a recent poll among the ESO users showed that 75% of that
community identified such a facility as the most crucial one for the future
(Mérand ., 2021). The concept of combining both MOS and IFS capabilities in a
dedicated wide-field 10m-class spectroscopic telescope would fill a clear gap
in the European research infrastructures as described above and ensure full
exploitation of the imaging data from upcoming instrumentation. The ambitious
WST top-level requirements (Fig. 2) place it far ahead of existing and planned
facilities. In just 5 years of operation, the WST MOS will target 250 million
galaxies and 25 million stars at medium resolution + 2 million stars at high
resolution, and 4 billion spectra with the WST IFS. Such a facility will have
a very broad user community, as reflected by the current scientific
productivity of VLT instruments. Most importantly, it will enable the
scientific community to comprehensively address the questions outlined above
and lead to transformative science in a number of key areas, from cosmology to
the formation of planets.

Figure 2: WST preliminary top level requirements. See also Fig.4

##  3 Conceptual Design

The consortium has a wealth of experience in the design and construction of
similar telescope and instrumentation capabilities. Our initial assessment is
that almost all the building blocks are in place. The telescope technologies
are available through ELT and other segmented primary 10m-class telescopes
worldwide.

Figure 3: WST preliminary layout (Pasquini ., 2018).

The partners in this proposal have built a massive panoramic IFS (MUSE) and
many fibre MOS instruments and we know we can build individual positioners to
the scale needed. We already have a preliminary optical design for the
telescope that fulfills the major requirements of aperture and field-of-view
(Fig. 3). The main challenge is to move to relatively large-scale production
of instrumentation, especially spectrographs, for which the detectors are a
critical item.

Figure 4: WST field-of-view.

We will develop close cooperation with industry in the following key areas
requiring development to deliver cost effectively in the quantities we need:
detectors (see next section), innovative fabrication of image slicers,
innovative photopolymer-based volume phase holographic gratings, massive and
cost-effective mass production of spectrographs and miniature robots for the
positioner.

| Spectro. | Chan. | Detec. | Format
---|---|---|---|---
IFS | 144 | 2 | 288 | 4k$\times$4k 15 $\mu$m
MOS MR | 26 | 2 | 52 | 6k$\times$6k 15 $\mu$m
MOS HR | 4 | 4 | 16 | 9k$\times$9k 10 $\mu$m
Total | 174 |  | 365 |
Table 1: Estimated number of spectrographs, channels and detectors (with
desired formats) for WST.

##  4 The WST detector systems

The combination of a large aperture telescope with wide field-of-view, massive
multiplex, medium to high spectral resolution and large contiguous spectral
range is challenging. To maintain the number of spectrographs in an acceptable
range, one needs to use spectrographs with a very fast camera (e.g. f/1). Such
spectrographs are usually difficult to build, have a low throughput because of
the large number of optical elements, and are very expensive. An elegant
solution is to use curved detectors. Curved detectors can greatly simplify the
spectrograph camera optics, improving its throughput and reducing its cost
(Hugot ., 2019). Curved detectors are currently at a low Technology Readiness
Level (TRL). However, a radius of curvature < 200mm for a 4k$\times$4k
15$\mu$m sensor would have very significant cost benefits for the optics.

Figure 5: Top: MUSE at the Nasmyth platform of the VLT. Bottom: artist’s view
of a possible configuration of a total of 12 MUSE-equivalent spectrograph and
detector modules.

Even with fast optics, a large number of spectrographs and an even larger
number of detectors is needed (Tab.1) to achieve the top level requirements.
Given the expected large number of detectors, we will also consider the use of
CMOS (whether flat or curved) over traditional CCDs. Thus, we will seek
industrial studies to understand the current TRL of these devices and future
availability plus costs. In practice, there will be a number of challenging
tasks that will be addressed in an upcoming conceptual design study:

  * •

Detector selection trade-off: flat versus curved, CMOS versus CCD. Challenges:
unproven technologies.

  * •

Detector electronics. Challenges: current controller technologies will be
outdated, long lead time.

  * •

Detector cryogenics. Challenges: complex system for very large number of
detectors. LN2 vs. cryocooler.

  * •

Detector procurement and characterization plan. Challenges: massive task,
involving industry.

  * •

Data acquisition system. Challenges: data rates, storage.

  * •

Data preprocessing. Challenges: data rates.

  * •

Thermal management. Challenges: spectrograph thermal stability, potentially
high dissipation of heat.

  * •

Mechanical layout and cable routing. Challenges: complexity, interference.

  * •

Detector QC management. Challenges: complexity, interference.

Fig. 5 is an illustration to support the notion that with technologies already
available today, WST can indeed be built and operated: the photograph on top
shows the existing and highly successful MUSE instrument on the Nasmyth
platform of the VLT UT4 that has been in operation at Paranal Observatory
since 2014. The rough sketch on the bottom imagines how a total of 12 copies
of MUSE modules might look like in a possible configuration for the WST IFU,
assuming the number of detectors listed in Tab. 1. In fact, the completion of
the VIRUS instrument for HETDEX has demonstrated that it is possible to build
a complex IFU with as many as 156 detectors (Hill ., 2021).

While it is not reasonable to assume that identical copies of a technology
that was designed and built between 2003 and 2014 would serve a significantly
more complex tasks two decades later, it is entirely plausible to assume that
an evolution towards more compact, replicable units is feasible and
affordable. Lessons learned with the VIRUS detector system (Hill ., 2016,
2021) have prompted us to pay particular attention to technical details that
cannot be simply scaled up by number, and that are listed above.

For example, the complex tubing of the vacuum and cryogenic system for MUSE
that can be seen in Fig. 5 will almost certainly shrink to a more compact
design with new emerging cryocooler technologies that are currently becoming
available on the market.

Interesting trends of CMOS image sensors with detector controller and pre-
processor architectures on the same chip may point to revolutionary avenues of
creating compact detector modules that would be very different from the bulky
CCD detector and controller subsystems of today.

New concepts of distributed data processing units, allocated to individual
detectors, similar to ongoing data pre-processing activities for the massive
data rates of SKAO (Bonaldi ., 2021), might significantly reduce data rates,
storage capacity, and power consumption - considerations that will play an
increasingly important role for green technologies of the future.

These and other issues will be carefully investigated in an upcoming
conceptual design study. The benefits of replicable system modules in terms of
cost and reliability, as demonstrated successfully with MUSE, and considered
to form the core of the WST layout, have been highlighted by Hill (2014).

##  5 Conclusions

The Wide field Spectroscopic Telescope facility is foreseen as a strong
candidate for the post-ELT major project at ESO. The detector system will be a
major item of WST both in quantity ($\sim$500 sensors) and characteristics
(curved sensors). The complexity of this system will likely drive research and
development concerning detectors and detector electronics, as well as
cryogenics, house-keeping and quality control, data acquisition and data
processing to new levels, that potentially could result in a paradigm change
and a disruptive innovation for detector systems.

## Acknowledgments

This work was supported by BMBF under Contract No. 03Z22AN11 and03Z22AB1A .

## References

  * Bacon . (2010) Bacon2010Bacon, R., Accardo, M., Adjali, L. et al.  201007,  The MUSE second-generation VLT instrument The MUSE second-generation VLT instrument.  IS. McLean, SK. Ramsay H. Takami (), Ground-based and Airborne Instrumentation for Astronomy III Ground-based and Airborne Instrumentation for Astronomy III 7735, 773508.  10.1117/12.856027
  * Bonaldi . (2021) Bonaldi2021Bonaldi, A., An, T., Brüggen, M. et al.  202101,  MNRAS50033821-3837.  10.1093/mnras/staa3023
  * Hill (2014) Hill2014Hill, GJ.  201406,  Advanced Optical Technologies33265-278.  10.1515/aot-2014-0019
  * Hill . (2021) Hill2021Hill, GJ., Lee, H., MacQueen, PJ. et al.  202112,  AJ1626298.  10.3847/1538-3881/ac2c02
  * Hill . (2016) Hill2016Hill, GJ., Tuttle, SE., Vattiat, BL. et al.  201608,  VIRUS: first deployment of the massively replicated fiber integral field spectrograph for the upgraded Hobby-Eberly Telescope VIRUS: first deployment of the massively replicated fiber integral field spectrograph for the upgraded Hobby-Eberly Telescope.  CJ. Evans, L. Simard H. Takami (), Ground-based and Airborne Instrumentation for Astronomy VI Ground-based and Airborne Instrumentation for Astronomy VI 9908, 99081H.  10.1117/12.2231064
  * Hugot . (2019) Hugo20191Hugot, E., Lombardo, S., Behaghel, T. et al.  201907,  Curved sensors: experimental performance of CMOS prototypes and wide field related imagers Curved sensors: experimental performance of CMOS prototypes and wide field related imagers.  Z. Sodnik, N. Karafolas B. Cugny (), International Conference on Space Optics — ICSO 2018 International Conference on Space Optics — ICSO 2018 11180, 111802Y.  10.1117/12.2536025
  * Le Fèvre . (2003) Lefevre2003Le Fèvre, O., Saisse, M., Mancini, D. et al.  200303,  Commissioning and performances of the VLT-VIMOS instrument Commissioning and performances of the VLT-VIMOS instrument.  M. Iye AFM. Moorwood (), Instrument Design and Performance for Optical/Infrared Ground-based Telescopes Instrument Design and Performance for Optical/Infrared Ground-based Telescopes 4841, 1670-1681.  10.1117/12.460959
  * Mérand . (2021) Merand2021Mérand, A., Andreani, P., Cirasuolo, M. et al.  202109,  The Messenger1848-14.  10.18727/0722-6691/5238
  * Pasquini . (2004) Pasquini2004Pasquini, L., Castillo, R., Dekker, H. et al.  200409,  Performance of FLAMES at the VLT: one year of operation Performance of FLAMES at the VLT: one year of operation.  AFM. Moorwood M. Iye (), Ground-based Instrumentation for Astronomy Ground-based Instrumentation for Astronomy 5492, 136-147.  10.1117/12.550437
  * Pasquini . (2018) Pasquini2018Pasquini, L., Delabre, B., Ellis, RS., Marrero, J., Cavaller, L. de Zeeuw, T.  201808,  ESO Spectroscopic Facility ESO Spectroscopic Facility.  C. Chiappini, I. Minchev, E. Starkenburg . (), Rediscovering Our Galaxy Rediscovering Our Galaxy 334, 242-247.  10.1017/S174392131700847X