# Unconventional charge density wave and photoinduced lattice symmetry change
in Kagome Metal CsV3Sb5 probed by time-resolved spectroscopy

Z. X. Wang  International Center for Quantum Materials, School of Physics,
Peking University, Beijing 100871, China  Q. Wu  International Center for
Quantum Materials, School of Physics, Peking University, Beijing 100871, China
Q. W. Yin  Department of Physics and Beijing Key Laboratory of Opto-electronic
Functional Materials and Micro-nano Devices, Renmin University of China,
Beijing 100872, China  Z. J. Tu  Department of Physics and Beijing Key
Laboratory of Opto-electronic Functional Materials and Micro-nano Devices,
Renmin University of China, Beijing 100872, China  C. S. Gong  Department of
Physics and Beijing Key Laboratory of Opto-electronic Functional Materials and
Micro-nano Devices, Renmin University of China, Beijing 100872, China  T. Lin
International Center for Quantum Materials, School of Physics, Peking
University, Beijing 100871, China  Q. M. Liu  International Center for Quantum
Materials, School of Physics, Peking University, Beijing 100871, China  L. Y.
Shi  International Center for Quantum Materials, School of Physics, Peking
University, Beijing 100871, China  S. J. Zhang  International Center for
Quantum Materials, School of Physics, Peking University, Beijing 100871, China
D. Wu  International Center for Quantum Materials, School of Physics, Peking
University, Beijing 100871, China  H. C. Lei  Department of Physics and
Beijing Key Laboratory of Opto-electronic Functional Materials and Micro-nano
Devices, Renmin University of China, Beijing 100872, China  T. Dong
International Center for Quantum Materials, School of Physics, Peking
University, Beijing 100871, China  N. L. Wang  International Center for
Quantum Materials, School of Physics, Peking University, Beijing 100871, China
Beijing Academy of Quantum Information Sciences, Beijing 100913, China

###### Abstract

Recently, kagome lattice metal AV3Sb5 (A = K, Rb, Cs) family has received wide
attention due to its presence of superconductivity, charge density wave (CDW)
and peculiar properties from topological nontrivial electronic structure. With
time-resolved pump-probe spectroscopy, we show that the excited quasiparticle
relaxation dynamics can be explained by formation of energy gap below the
phase transition being similar to a usual second-order CDW condensate, by
contrast, the structure change is predominantly first order phase transition.
Furthermore, no CDW amplitude mode is identified in the ordered phase. The
results suggest that the CDW order is very different from the traditional CDW
condensate. We also find that weak pump pulse can non-thermally melt the CDW
order and drive the sample into its high temperature phase, revealing the fact
that the difference in lattice potential between those phases is small.

Control over physical properties or electronic phases in quantum materials via
external tuning parameters is a central topic in condensed matter physics.
Traditional ways of tuning external parameters are to change the temperature,
pressure, electric/magnetic field, or doping level of the material systems. In
the last two decades, photo-excitation has emerged as a new way to probe and
manipulate the properties of quantum materials, enabling an ultra-fast control
over the material properties or quantum phases Cavalleri  _et al._ (2001);
Tomimoto  _et al._ (2003); Okamoto  _et al._ (2004); Takubo  _et al._ (2005);
Matsubara  _et al._ (2007); Tomeljak  _et al._ (2009); Wall  _et al._ (2012);
Stojchevska  _et al._ (2014); Giannetti  _et al._ (2016); Zhang  _et al._
(2019); Sie  _et al._ (2019); Nova  _et al._ (2019); Li  _et al._ (2019);
McLeod  _et al._ (2020); Kogar  _et al._ (2020); Liu  _et al._ (2021). The
ultrashort laser pulses can create a far from equilibrium distribution of the
energy among the different degrees of freedom, triggering the formation of
thermal or non-thermal metastable or even stable phases. Among different
systems, correlated electronic systems with rich phase diagrams have been the
most explored materials for practical manipulation, since the complex
interactions or competition among different degrees of freedom make them very
susceptible to external perturbation or stimuli Giannetti  _et al._ (2016).

Recently, a new family of kagome metals AV3Sb5 (A = Cs, Rb, K), composed of
vanadium layers, antimony layers and alkali ions sandwiched between the two
layers, has attracted tremendous attention in the community Ortiz  _et al._
(2019, 2020, 2021); Yin  _et al._ (2021). These materials undergo a charge
density wave (CDW) phase transition at $T_{\rm{CDW}}=80-100$ K with a
2$\times$2$\times$2 superlattice formation and a superconducting transition at
$T_{\rm{c}}=0.9-3.5$ K. High-pressure measurements revealed the competition
between the charge order and superconductivity by observing a double-dome
superconductivityZhao  _et al._ (2021); Chen  _et al._ (2021a); Du  _et al._
(2021); Chen  _et al._ (2021b); Zhang  _et al._ (2021). More intriguingly, the
topological nontrivial electronic structure was found in AV3Sb5 family (even
in CDW state) Ortiz  _et al._ (2019, 2020, 2021); Yin  _et al._ (2021); Wang
_et al._ (2020); Tan  _et al._ (2021); Li  _et al._ (2021); Zhao  _et al._
(2021); Chen  _et al._ (2021c); Ni  _et al._ (2021); Jiang  _et al._ (2020). A
giant anomalous Hall effect Yang  _et al._ (2020); Yu  _et al._ (2021) was
observed even in the absence of magnetic ordering Ortiz  _et al._ (2020);
Kenney  _et al._ (2020). Furthermore, transport and scanning tunnelling
microscopy (STM) measurements revealed that the 6-fold rotation $C_{6}$
symmetry is further broken and reduced to $C_{2}$ symmetry at low temperature
below 60 K Xiang  _et al._ (2021); Zhao  _et al._ (2021); Chen  _et al._
(2021c); Li  _et al._ ; Ratcliff  _et al._ (2021). A recent coherent phonon
spectroscopy measurement Ratcliff  _et al._ (2021) revealed appearance of new
phonon modes below 94 K and 60 K, respectively, implying structural phase
transitions. Thus, the AV3Sb5 series provide a new opportunity to study the
competition between different phases and to understand the unconventional
correlated physics emerging from the itinerant kagome lattice electrons.

In this work, we perform ultrafast pump-probe reflectivity experiment on
CsV3Sb5, aiming to investigate the CDW order across the phase transitions and
possible photoexcitation control of different orders in the compound. Our
measurements show that, while the quasiparticle relaxation dynamics upon weak
pumping can be described by formation of energy gap below the phase
transition, the structure change is characterized by an abrupt change in the
number of coherent phonon modes without showing clear softening at the CDW
phase transition. Furthermore, no CDW amplitude mode can be identified in the
ordered phase. Those results suggest that the CDW order is predominantly first
order phase transition, and is very different from the traditional CDW
condensate. More intriguingly, we show that even small pumping fluence can
non-thermally melt the $C_{2}$ ground state order and $C_{6}$ CDW order
successionally and drive the compound into high temperature phase. We shall
discuss the implication of the results.

Single crystals of CsV3Sb5 were grown from Cs ingot (purity 99.9%), V powder
(purity 99.9%) and Sb grains (purity 99.999%) using the self-flux method,
similar to the growth of RbV3Sb5 Yin  _et al._ (2021). We used an amplified
Ti:sapphire laser system with 800 nm wavelength and 35 fs pulse duration
operating at 1 kHz repetition frequency as the light source for pump-probe
measurement. The fluence of probe beam is set below 3 $\mu$J cm-2, weaker than
that of pump beam. To reduce the noise caused by stray light, the pump and
probe pulses were set to be cross polarized and an extra polarizer was mounted
just before the detector.

Figure 1: Temperature-dependent dynamics for CsV3Sb5. (a) $\Delta R/R$ in the
temperature range of 5-200 K. With increasing temperature, the absolute value
of $\Delta R/R$ increases first, then decreases and finally it changes the
sign at $T\approx 98$ K (above $T_{\rm{CDW}}=94$ K). The lines in high
temperature phase are scaled by a five-fold factor. The dash lines are the
double-exponential fitting curve: $\Delta
R/R(t)=A_{1}\rm{e}^{-t/\tau_{1}}+A_{2}\rm{e}^{-t/\tau_{2}}$. (b) (c) A fast
($\tau_{1}\sim 0.5$ ps) and a slow ($\tau_{2}\sim 10$ ps) relaxation process,
respectively. The amplitude $A_{1}$, $A_{2}$ (orange squares) and decay time
$\tau_{1}$, $\tau_{2}$ (blue circles) of the reflectivity transients extracted
from fits to the double-exponential. We notice the amplitude of the fast
relaxation process $A_{1}$ becomes 0 in high temperature phase, which means
the fast relaxation process is absent in high temperature phase and the
double-exponential function degrades into single-exponential function. Error
bars represent the standard deviation of the fit. The orange and blue lines
are the RT model fits.

Figure 1 presents the photoinduced reflectivity change of CsV3Sb5 as a
function of time delay at different temperatures under rather weak pumping
excitation $\sim$5 $\mu$J cm-2. Very prominently, $\Delta R/R$ changes sign,
from positive to negative when temperature decreases across the CDW transition
temperature $T_{\rm{CDW}}$. At low temperature, the pump pulse induces an
abrupt drop in reflectivity followed by a fast decay within a few picoseconds
and a slower recovery process in the order of 10 ps, while there is only a
slow decay dynamics at high temperature. We use double-exponential function to
fit the decay process: $\Delta
R/R=A_{1}\rm{e}^{-t/\tau_{1}}+A_{2}\rm{e}^{-t/\tau_{2}}$, where
$A_{1}\rm{e}^{-t/\tau_{1}}$ is the fast decay process and
$A_{2}\rm{e}^{-t/\tau_{2}}$ presents the slower recovery process,
respectively. The best fitting results are shown as dashed lines in Fig.1 (a).
We note the fast decay process becomes invisible above 95 K, which indicates
the strong correlation with CDW energy gap. Fitting parameters of the double-
exponential function are shown in Fig. 1 (b) and (c). For the fast decay
dynamics, the decay lifetime $\tau_{1}$ diverges and the absolute value of
amplitude $A_{1}$ drops to zero precipitously, as the temperature rising close
to $T_{\rm{CDW}}$. These behaviors indicate that an energy gap is closing
toward critical temperature, which can be roughly described by Rothwarf-Taylor
(R-T) model, showing that the opening of a CDW energy gap would significantly
impede the relaxation of the photoinduced quasiparticle Kabanov  _et al._
(1999). Similar to the situation of superconductivity from the condensate of
electron-electron Cooper pairs, CDW is a condensate from electron-hole (e-h)
pairs. The pump pulse serves as an excitation to break e-h pairs across energy
gap of CDW condensate, the recombination of e-h pairs is accompanied by the
emission of phonons with energy higher than the energy gap $\Delta(T)$.
$\tau_{1}$ is the characteristic time for this recombination, since these
excited phonons will in turn break additional e-h pairs, decay time $\tau_{1}$
is ultimately determined by the time required for these phonons anharmonically
decaying into phonons with the energy less than $\Delta$. The amplitude $A$
represents the population of excited quasiparticles by pump pulse. The change
of relaxation time and the amplitude of the photoinduced reflectivity signal
near the transition temperature in R-T model is given by following equations
Kabanov  _et al._ (1999); Lin  _et al._ (2020):

$\displaystyle\tau(T)$
$\displaystyle\propto\frac{\ln{\left[g+\textrm{e}^{-\Delta(T)/k_{B}T}\right]}}{\Delta(T)^{2}}$
(1) $\displaystyle A(T)$
$\displaystyle\propto\frac{\Phi/(\Delta(T)+k_{B}T/2)}{1+\Gamma\sqrt{2k_{B}T/\pi\Delta(T)}\textrm{e}^{-\Delta(T)/k_{B}T}}$

where a BCS-type energy band is assumed here as
$\Delta(T)=\Delta_{0}\sqrt{1-T/T_{\rm{CDW}}}$ with $T_{\rm{CDW}}=94$ K and the
CDW gap at 0 K $\Delta_{0}=40$ meV Wang  _et al._ (2021). Qualitatively, the
equations can reproduce the measurement results. The best fits yield the
phenomenological fitting parameter $g=0.18$ and $\Gamma=17$ as shown in Fig. 2
(b). Although there appears an increase of the decay time $\tau_{1}$ below
$T^{*}=60$ K, no significant change can be identified from the relaxation
process when the symmetry is further broken from $C_{6}$ to $C_{2}$.

Figure 2: Temperature-dependent coherent phonon spectroscopy for CsV3Sb5. (a)
Coherent phonon oscillation in time-domain, where the decay background is
subtracted. (b) The Fast Fourier transformation of the data in (a). (c)
Temperature dependence waterfall map extracted from (b). The 4.1 THz coherent
phonon is present at all temperature through phase change. The 1.3 THz phonon
can be only detected below $T_{\rm{CDW}}$, while the 3.1 THz phonon disappears
at the temperature above $T^{*}=30\sim 60$ K.
Figure 3:  Fluence-dependent pump probe reflectivity signal of CsV3Sb5. (a)
Photo-induced reflectivity $\Delta R/R$ as a function of time delay versus
pump fluence in range from 5 to 300 $\mu$J cm-2 of CsV3Sb5 at 20 K. The data
of the three highest fluence is scaled by a factor of 0.3. (b) Fluence
dependence of decay time $\tau$ of the reflectivity transients extracted from
fits to the double exponential decay function. (c) The Fast Fourier
transformation of the oscillation parts by subtracting the decay background of
$\Delta R/R$ in (a). Three resonance peaks at 1.3, 3.1 and 4.1 THz can be seen
at the lowest fluence 5 $\mu$J cm-2. (d) Two-dimensional waterfall map of
these three resonance peaks. 3.1 THz phonon can be survived below the fluence
of 23 $\mu$J cm-2, the phonon at 1.3 THz is killed by 55 $\mu$J cm-2 pulses,
while the 4.1 THz phonon is coherently enhanced even under an intense fluence
higher than 300 $\mu$J cm-2.

After subtracting the dynamics background, we obtained the coherent phonon
oscillations, as shown in Fig. 2 (a). The fast Fourier transform result is
presented in Fig. 2 (b). A two dimensional intensity map as a function of
temperature and frequency is displayed in Fig. 2 (c). The coherent phonon
spectroscopy is similar to previous study on CsV3Sb5 Ratcliff  _et al._
(2021). Above $T_{\rm{CDW}}=94$ K, there is only one peak at 4.1 THz, which is
in good agreement with the density functional theory (DFT) calculations for
the $A_{1g}$ mode of phonon spectra Li  _et al._ (2021); Ratcliff  _et al._
(2021). Below $T_{\rm{CDW}}$, a new mode at 1.3 THz peak abruptly appears,
yielding evidence for the CDW structural modulation. Below 60 K, a weak and
broad feature emerges near 3.1 THz, which evolves into a sharp and strong peak
below 30 K. This mode was suggested to be linked to the uniaxial order
observed by STM experiments which breaks the $C_{6}$ rotational symmetry
Ratcliff  _et al._ (2021). It deserves to remark that, for conventional CDW
condensate, the CDW amplitude mode usually appears as the strongest
oscillation in the pump-probe measurement. As the temperature increases to the
transition temperature $T_{\rm{CDW}}$, the mode frequency softens and behaves
like an order parameter Chen  _et al._ (2017). However, no such CDW amplitude
mode could be identified from the above spectra, suggesting that the CDW order
in CsV3Sb5 is very different from the conventional CDW condensate. The abrupt
change of the coherent mode near 1.3 THz further indicates that the phase
transition is predominantly first-order phase transition. On the other hand,
the symmetry change from $C_{6}$ to $C_{2}$ is a crossover behavior. A short
range of $C_{2}$ nematic order, corresponding the broad feature, is seen
roughly below 60 K which gradually evolves into a static $C_{2}$ order below
30 K.

The above analysis reveals a rather peculiar situation: while the excited
electron relaxation dynamics upon weak pumping can be described by formation
of energy gap below the phase transition being similar to a usual second-order
CDW condesate, the structure change is primarily first order phase transition,
being characterized by an abrupt change in the number of coherent phonon modes
without showing clear softening at the CDW phase transition. Additionally, no
CDW amplitude mode is present in the ordered phase. Those results suggest that
the CDW order is rather unconventional. We notice that nuclear magnetic
resonance (NMR) measurements on CsV3Sb5 indicated similar situation Song  _et
al._ (2021). A first-order structural phase transition associated with orbital
ordering is seen as the sudden splitting of orbital shift in 51V NMR spectrum
at TCDW, by contrast, a typical second order transition behavior is seen in
the quadrupole splitting which appears gradually below TCDW. The NMR
measurement also suggests that the CDW order is a secondary electronic order.
The seemingly week coupling between electron and lattice degree of freedoms
needs to be further explored.

In addition to varying temperature, we also performed measurement of the
photoinduced reflectivity change at different pump fluences in CDW phase. The
measurement results at 20 K are shown in Fig.3. We found that a surprisingly
small pump fluence can melt the $C_{2}$ nematic and then the $C_{6}$ CDW
order. As shown in Fig.3 (a), at 5 $\mu$J cm-2 fluence, the fast decay process
shows a lifetime of hundreds of femtoseconds. With pump fluence incresing, the
absolute value of negative $\Delta R/R$ further grows, and reaches the maximum
value at around $F^{\rm{melt}}\sim$55 $\mu$ J cm-2. Then it turns back and
goes into the positive direction, passes through zero, and gradually grows
larger. This behavior is very similar to the warming process of temperature
dependent measurement. From the two exponential decay approach, we can extract
the relaxation time of fast decay dynamics, as shown in Fig.3 (b). As the
fluence increases to $F^{\rm{melt}}$, the relaxation time sharply increases.
Similar to the above analysis for temperature variation, we assume that an CDW
energy gap is present up to the melting fluence as
$\Delta=\Delta_{0}(1-F/F^{\rm{melt}})$, and the decay time is inversely
proportional to the energy gap near the critical melting fluence $\tau\propto
1/\Delta$. Indeed, the fast decay time can be reasonably reproduced by the
relation as shown in Fig.3 (b). The fluence dependent relaxation dynamics
suggests that a pump flence as small as 55 $\mu$J cm-2 can melt the CDW order
in CsV3Sb5.

The coherent phonon spectroscopy provides more direct evidence for ultrafast
optical melting of those orders in CsV3Sb5. We performed Fourier
transformation of the pump-probe measurement after subtracting the relaxation
background and obtained the coherent phonon spectra for different fluences, as
displayed in Fig.3 (c). The intensity plot as function of both frequency and
pump fluence is shown in Fig.3 (d). At small fluence, for example, at 5 $\mu$J
cm-2, three resonance peaks centered at 1.3, 3.1 and 4.1 THz can be observed.
As the pump fluence rises to 23 $\mu$J cm-2, the peak at 3.1 THz being
associated with $C_{2}$ symmetryRatcliff  _et al._ (2021) disappears. As the
fluence increases beyond 55 $\mu$J cm-2, the phonon peak at 1.3 THz, which is
related to CDW order Ratcliff  _et al._ (2021), is not visible. The critical
pump fluence is in agreement with the analysis form the relaxation dynamics.
The 4.1 THz resonance is present in all fluences. The strength of this mode is
further enhanced up to the highest used fluence 300 $\mu$J cm-2. The behavior
is expected normally for the coherent phonon generation in pump-probe
experiment.

We emphasize that the melting of the low temperature orders cannot be
attributed to trivial thermal effect owning to the very small pump fluence.
The temperature rise $\Delta T$ can be phenomenologically estimated by energy
conservation law $S\delta_{0}\rho/M\int_{T_{0}}^{T_{0}+\Delta
T}C_{p}(T)\textrm{d}T=(1-R)FS$, where $S$ is the excitation area, the mass
density $\rho=5.2$ g/cm3, the molar mass $M=894.5$ g/mol, the penetration
depth $\delta_{0}\approx 64$ nm and the reflectivity $R=0.42$ at 800 nm for
CsV3Sb5 from our own reflectance measurement by Fourier transform infrared
spectrometer. With the initial temperature $T_{0}=20$ K and the temperature-
dependent thermal capacity $C_{p}(T)$ is extracted from previous data Ortiz
_et al._ (2020), the values of $\Delta T$ are calculated to be approximately
10 and 19 K at the fluence of 23 and 55 $\mu$J cm-2, respectively. From this
estimation, the possibility of thermal effect can be unambiguously ruled out.

The nonthermal melting at such small pump fluence is the key finding in this
work. Although photoinduced lattice symmetry change or phase transitions have
been observed in many different systems, the threshold fluences for completely
changing the lattice potentials are usually much higher than the present
measurement. For example, VO2 undergoes a first order structural phase
transition on cooling from a rutile R-phase to a monoclinic M1-phase at 343 K.
Photoexcitation at room temperature can induce the lattice symmetry change
from the M1-phase to rutile R-phase based on similar coherent phonon
measurement Wall  _et al._ (2012). However, the required pump threshold
fluence is $\sim$7 mJ cm-2 for 800 nm pump-probe measurement with similar
pulse duration $<$40 fs. Our measurement suggests that the lattice potential
difference between those different phases are relatively small. The weak pump
pulses can excite a sufficient number of electrons and their perturbation to
the lattice potential is large enough to modify the symmetry. As a result, it
drives the structural phase transitions non-thermally. We noticed that a
recent X-ray scattering measurement Li  _et al._ (2021) on CsV3Sb5 revealed
that, while the CDW is long-range ordered, the integrated CDW superlattice
peak intensity that is proportional to the CDW order parameter is extremely
small. Comparing with fundamental Bragg peaks, the CDW peak intensity is
3$\sim$5 orders of magnitude weaker, demonstrating small lattice distortions.
This observation appears to be correlated to our result.

To summarize, laser pulses serve as a tool not only to probe the dynamic
behavior but also to drive phase transitions in kagome metal compound CsV3Sb5.
Our measurement reveals a peculiar CDW phase transition, _i.e._ the
quasiparticle relaxation dynamics can be explained by formation of energy gap
below the phase transition being similar to a usual second-order CDW
condensate, the structure change is predominantly first order phase
transition. Furthermore, no CDW amplitude mode can be identified in the
ordered phase. We also show that even small pumping fluence can non-thermally
melt the $C_{2}$ ground state order and then $C_{6}$ CDW order seccessionally
and drive the compound into high temperature phase, suggesting that the
lattice potential difference between those different phases is relatively
small.

ACKNOWLEDGMENTS

This work was supported by National Natural Science Foundation of China (No.
11888101, 11822412 and 11774423), the National Key Research and Development
Program of China (No. 2017YFA0302904, 2018YFE0202600) and Beijing Natural
Science Foundation (Grant No. Z200005)

## References

  * Cavalleri  _et al._ (2001) A. Cavalleri, C. Tóth, C. W. Siders, J. A. Squier, F. Ráksi, P. Forget, and J. C. Kieffer, Phys. Rev. Lett. 87, 237401 (2001).
  * Tomimoto  _et al._ (2003) S. Tomimoto, S. Miyasaka, T. Ogasawara, H. Okamoto, and Y. Tokura, Phys. Rev. B 68, 035106 (2003).
  * Okamoto  _et al._ (2004) H. Okamoto, Y. Ishige, S. Tanaka, H. Kishida, S. Iwai, and Y. Tokura, Phys. Rev. B 70, 165202 (2004).
  * Takubo  _et al._ (2005) N. Takubo, Y. Ogimoto, M. Nakamura, H. Tamaru, M. Izumi, and K. Miyano, Phys. Rev. Lett. 95, 017404 (2005).
  * Matsubara  _et al._ (2007) M. Matsubara, Y. Okimoto, T. Ogasawara, Y. Tomioka, H. Okamoto, and Y. Tokura, Phys. Rev. Lett. 99, 207401 (2007).
  * Tomeljak  _et al._ (2009) A. Tomeljak, H. Schäfer, D. Städter, M. Beyer, K. Biljakovic, and J. Demsar, Physical Review Letters 102, 066404 (2009), arXiv:0901.1951 .
  * Wall  _et al._ (2012) S. Wall, D. Wegkamp, L. Foglia, K. Appavoo, J. Nag, R. F. Haglund, J. StäCurrency Signhler, and M. Wolf, Nature Communications 3 (2012), 10.1038/ncomms1719, arXiv:1012.1468 .
  * Stojchevska  _et al._ (2014) L. Stojchevska, I. Vaskivskyi, T. Mertelj, P. Kusar, D. Svetin, S. Brazovskii, and D. Mihailovic, Science 344, 177 (2014).
  * Giannetti  _et al._ (2016) C. Giannetti, M. Capone, D. Fausti, M. Fabrizio, F. Parmigiani, and D. Mihailovic, Advances in Physics 65, 58 (2016), arXiv:1601.07204 .
  * Zhang  _et al._ (2019) M. Y. Zhang, Z. X. Wang, Y. N. Li, L. Y. Shi, D. Wu, T. Lin, S. J. Zhang, Y. Q. Liu, Q. M. Liu, J. Wang, T. Dong, and N. L. Wang, Phys. Rev. X 9, 021036 (2019).
  * Sie  _et al._ (2019) E. J. Sie, C. M. Nyby, C. D. Pemmaraju, S. J. Park, X. Shen, J. Yang, M. C. Hoffmann, B. K. Ofori-Okai, R. Li, A. H. Reid, S. Weathersby, E. Mannebach, N. Finney, D. Rhodes, D. Chenet, A. Antony, L. Balicas, J. Hone, T. P. Devereaux, T. F. Heinz, X. Wang, and A. M. Lindenberg, Nature 565, 61 (2019).
  * Nova  _et al._ (2019) T. F. Nova, A. S. Disa, M. Fechner, and A. Cavalleri, Science 364, 1075 (2019), https://science.sciencemag.org/content/364/6445/1075.full.pdf .
  * Li  _et al._ (2019) X. Li, T. Qiu, J. Zhang, E. Baldini, J. Lu, A. M. Rappe, and K. A. Nelson, Science 364, 1079 (2019), https://science.sciencemag.org/content/364/6445/1079.full.pdf .
  * McLeod  _et al._ (2020) A. S. McLeod, J. Zhang, M. Q. Gu, F. Jin, G. Zhang, K. W. Post, X. G. Zhao, A. J. Millis, W. B. Wu, J. M. Rondinelli, R. D. Averitt, and D. N. Basov, Nature Materials 19, 397 (2020), arXiv:1910.10361 .
  * Kogar  _et al._ (2020) A. Kogar, A. Zong, P. E. Dolgirev, X. Shen, J. Straquadine, Y.-Q. Bie, X. Wang, T. Rohwer, I.-C. Tung, Y. Yang, R. Li, J. Yang, S. Weathersby, S. Park, M. E. Kozina, E. J. Sie, H. Wen, P. Jarillo-Herrero, I. R. Fisher, X. Wang, and N. Gedik, Nature Physics 16, 159 (2020).
  * Liu  _et al._ (2021) Q. M. Liu, D. Wu, Z. A. Li, L. Y. Shi, Z. X. Wang, S. J. Zhang, T. Lin, T. C. Hu, H. F. Tian, J. Q. Li, T. Dong, and N. L. Wang, Nature Communications 12, 1 (2021).
  * Ortiz  _et al._ (2019) B. R. Ortiz, L. C. Gomes, J. R. Morey, M. Winiarski, M. Bordelon, J. S. Mangum, I. W. Oswald, J. A. Rodriguez-Rivera, J. R. Neilson, S. D. Wilson, E. Ertekin, T. M. McQueen, and E. S. Toberer, Physical Review Materials 3, 94407 (2019).
  * Ortiz  _et al._ (2020) B. R. Ortiz, S. M. L. Teicher, Y. Hu, J. L. Zuo, P. M. Sarte, E. C. Schueller, A. M. M. Abeykoon, M. J. Krogstad, S. Rosenkranz, R. Osborn, R. Seshadri, L. Balents, J. He, and S. D. Wilson, Phys. Rev. Lett. 125, 247002 (2020).
  * Ortiz  _et al._ (2021) B. R. Ortiz, P. M. Sarte, E. M. Kenney, M. J. Graf, S. M. L. Teicher, R. Seshadri, and S. D. Wilson, Physical Review Materials 5, 1 (2021).
  * Yin  _et al._ (2021) Q. Yin, Z. Tu, C. Gong, Y. Fu, S. Yan, and H. Lei, Chinese Physics Letters 38, 37403 (2021).
  * Zhao  _et al._ (2021) H. Zhao, H. Li, B. R. Ortiz, S. M. L. Teicher, T. Park, M. Ye, Z. Wang, L. Balents, S. D. Wilson, and I. Zeljkovic,  (2021), arXiv:2103.03118 .
  * Chen  _et al._ (2021a) K. Y. Chen, N. N. Wang, Q. W. Yin, Z. J. Tu, C. S. Gong, J. P. Sun, H. C. Lei, Y. Uwatoko, and J. G. Cheng, arXiv e-prints , arXiv:2102.09328 (2021a), arXiv:2102.09328 [cond-mat.supr-con] .
  * Du  _et al._ (2021) F. Du, S. Luo, B. R. Ortiz, Y. Chen, W. Duan, D. Zhang, X. Lu, S. D. Wilson, Y. Song, and H. Yuan, arXiv e-prints , arXiv:2102.10959 (2021), arXiv:2102.10959 [cond-mat.supr-con] .
  * Chen  _et al._ (2021b) X. Chen, X. Zhan, X. Wang, J. Deng, X.-b. Liu, X. Chen, J.-g. Guo, and X. Chen, arXiv e-prints , arXiv:2103.13759 (2021b), arXiv:2103.13759 [cond-mat.supr-con] .
  * Zhang  _et al._ (2021) Z. Zhang, Z. Chen, Y. Zhou, Y. Yuan, S. Wang, L. Zhang, X. Zhu, Y. Zhou, X. Chen, J. Zhou, and Z. Yang, arXiv e-prints , arXiv:2103.12507 (2021), arXiv:2103.12507 [cond-mat.supr-con] .
  * Wang  _et al._ (2020) Y. Wang, S. Yang, P. K. Sivakumar, B. R. Ortiz, S. M. L. Teicher, H. Wu, A. K. Srivastava, C. Garg, D. Liu, S. S. P. Parkin, E. S. Toberer, T. McQueen, S. D. Wilson, and M. N. Ali, arXiv e-prints , arXiv:2012.05898 (2020), arXiv:2012.05898 [cond-mat.supr-con] .
  * Tan  _et al._ (2021) H. Tan, Y. Liu, Z. Wang, and B. Yan, arXiv e-prints , arXiv:2103.06325 (2021), arXiv:2103.06325 [cond-mat.supr-con] .
  * Li  _et al._ (2021) H. X. Li, T. T. Zhang, Y. Y. Pai, C. Marvinney, A. Said, T. Yilmaz, Q. Yin, C. Gong, Z. Tu, E. Vescovo, R. G. Moore, S. Murakami, H. C. Lei, H. N. Lee, B. Lawrie, and H. Miao, arXiv e-prints , arXiv:2103.09769 (2021), arXiv:2103.09769 [cond-mat.supr-con] .
  * Zhao  _et al._ (2021) H. Zhao, H. Li, B. R. Ortiz, S. M. L. Teicher, T. Park, M. Ye, Z. Wang, L. Balents, S. D. Wilson, and I. Zeljkovic, arXiv e-prints , arXiv:2103.03118 (2021), arXiv:2103.03118 [cond-mat.supr-con] .
  * Chen  _et al._ (2021c) H. Chen, H. Yang, B. Hu, Z. Zhao, J. Yuan, Y. Xing, G. Qian, Z. Huang, G. Li, Y. Ye, Q. Yin, C. Gong, Z. Tu, H. Lei, S. Ma, H. Zhang, S. Ni, H. Tan, C. Shen, X. Dong, B. Yan, Z. Wang, and H.-J. Gao, arXiv e-prints , arXiv:2103.09188 (2021c), arXiv:2103.09188 [cond-mat.supr-con] .
  * Ni  _et al._ (2021) S. Ni, S. Ma, Y. Zhang, J. Yuan, H. Yang, Z. Lu, N. Wang, J. Sun, Z. Zhao, D. Li, S. Liu, H. Zhang, H. Chen, K. Jin, J. Cheng, L. Yu, F. Zhou, X. Dong, J. Hu, H.-J. Gao, and Z. Zhao, arXiv e-prints , arXiv:2104.00374 (2021), arXiv:2104.00374 [cond-mat.supr-con] .
  * Jiang  _et al._ (2020) Y.-X. Jiang, J.-X. Yin, M. M. Denner, N. Shumiya, B. R. Ortiz, G. Xu, Z. Guguchia, J. He, M. Shafayat Hossain, X. Liu, J. Ruff, L. Kautzsch, S. S. Zhang, G. Chang, I. Belopolski, Q. Zhang, T. A. Cochran, D. Multer, M. Litskevich, Z.-J. Cheng, X. P. Yang, Z. Wang, R. Thomale, T. Neupert, S. D. Wilson, and M. Zahid Hasan, arXiv e-prints , arXiv:2012.15709 (2020), arXiv:2012.15709 [cond-mat.supr-con] .
  * Yang  _et al._ (2020) S. Y. Yang, Y. Wang, B. R. Ortiz, D. Liu, J. Gayles, E. Derunova, R. Gonzalez-Hernandez, L. Šmejkal, Y. Chen, S. S. Parkin, S. D. Wilson, E. S. Toberer, T. McQueen, and M. N. Ali, Science Advances 6, 1 (2020).
  * Yu  _et al._ (2021) F. H. Yu, T. Wu, Z. Y. Wang, B. Lei, W. Z. Zhuo, J. J. Ying, and X. H. Chen, arXiv e-prints , arXiv:2102.10987 (2021), arXiv:2102.10987 [cond-mat.str-el] .
  * Kenney  _et al._ (2020) E. M. Kenney, B. R. Ortiz, C. Wang, S. D. Wilson, and M. J. Graf, arXiv (2020), 10.1088/1361-648x/abe8f9, arXiv:2012.04737 .
  * Xiang  _et al._ (2021) Y. Xiang, Q. Li, Y. Li, W. Xie, H. Yang, Z. Wang, Y. Yao, and H.-H. Wen, arXiv e-prints , arXiv:2104.06909 (2021), arXiv:2104.06909 [cond-mat.supr-con] .
  * (37) H. Li, H. Zhao, B. R. Ortiz, T. Park, M. Ye, L. Balents, Z. Wang, S. D. Wilson, and I. Zeljkovic, 17.
  * Ratcliff  _et al._ (2021) N. Ratcliff, L. Hallett, B. R. Ortiz, S. D. Wilson, and J. W. Harter, arXiv e-prints , arXiv:2104.10138 (2021), arXiv:2104.10138 [cond-mat.str-el] .
  * Kabanov  _et al._ (1999) V. V. Kabanov, J. Demsar, B. Podobnik, and D. Mihailovic, Phys. Rev. B 59, 1497 (1999).
  * Lin  _et al._ (2020) T. Lin, L. Y. Shi, Z. X. Wang, S. J. Zhang, Q. M. Liu, T. C. Hu, T. Dong, D. Wu, and N. L. Wang, Phys. Rev. B 101, 205112 (2020).
  * Wang  _et al._ (2021) Z. Wang, S. Ma, Y. Zhang, H. Yang, Z. Zhao, Y. Ou, Y. Zhu, S. Ni, Z. Lu, H. Chen, K. Jiang, L. Yu, Y. Zhang, X. Dong, J. Hu, H.-J. Gao, and Z. Zhao,  (2021), arXiv:2104.05556 .
  * Chen  _et al._ (2017) R. Y. Chen, S. J. Zhang, M. Y. Zhang, T. Dong, and N. L. Wang, Phys. Rev. Lett. 118, 107402 (2017).
  * Song  _et al._ (2021) D. W. Song, L. X. Zheng, F. H. Yu, J. Li, L. P. Nie, M. Shan, D. Zhao, S. J. Li, B. L. Kang, Z. M. Wu, Y. B. Zhou, K. L. Sun, K. Liu, X. G. Luo, Z. Y. Wang, J. J. Ying, X. G. Wan, T. Wu, and X. H. Chen, arXiv:2104.09173 (2021), arXiv:2104.09173 .


# Real-time Lane-wise Traffic Monitoring in Optimal ROIs

Mei Qiu, Wei Lin, Lauren Ann Christopher, Stanley Chien∗, Yaobin Chen, Shu
HuPurdue University Indianapolis, IN, USA ∗Corresponding Author

###### Abstract

In the US, thousands of Pan, Tilt, and Zoom (PTZ) traffic cameras monitor
highway conditions. There is a great interest in using these highway cameras
to gather valuable road traffic data to support traffic analysis and decision-
making for highway safety and efficient traffic management. However, there are
too many cameras for a few human traffic operators to effectively monitor, so
a fully automated solution is desired. This paper introduces a novel system
that learns the locations of highway lanes and traffic directions from these
camera feeds automatically. It collects real-time, lane-specific traffic data
continuously, even adjusting for changes in camera angle or zoom. This
facilitates efficient traffic analysis, decision-making, and improved highway
safety.

##  I Introduction

Numerous Pan, Tilt, and Zoom (PTZ) traffic cameras are installed along
highways in the USA, allowing operators to monitor traffic conditions.
However, human operators oversee hundreds of cameras, making it impossible to
watch them all simultaneously. Thus, there is interest in using Artificial
Intelligence (AI) to analyze footage in real time, providing valuable traffic
data and alerting operators to issues.

Traffic monitoring systems come in various types, broadly classified into two
groups: traditional systems and intelligent management systems [1, 2]. AI and
big data analytics have revolutionized traffic monitoring and management.
Intelligent Transportation Systems (ITS), equipped with advanced sensors,
radars, and license plate recognition cameras play a vital role in detecting
and deterring traffic rule violations [3, 4]. These systems also aid in
alleviating traffic congestion by managing traffic scenarios through real-time
data visualization [5, 6].

However, these existing traffic monitoring systems need to annotate images
from a large video surveillance data. Most of them can monitor two directions
of traffic status but are not lane-specific. There is some work with lane-wise
counting, but others rely on road lane marking detection, which fails in
various lighting, weather, and ground conditions [7]. Vehicle motion
trajectories can be used to learn lanes [8] but the performance depends on
vehicle detection and long tracking accuracy. The innovation in this work is
to perform detection and tracking only in optimal regions of the road. Our new
result also solves the problem of existing traffic flow estimation that
requires fixed zoom and viewing angles [9, 7, 10].

Figure 1:  (Top) Traditional System: Vehicles are detected, tracked, and
counted across the entire frame, with manually defined counting lines, leading
to suboptimal performance. (Bottom) Our system: Vehicle detection is performed
across the entire frame. Vehicle tracking is concentrated in several
automatically learned Regions of Interest (ROIs), optimizing detection and
tracking results. Counting is lane-specific, with each vehicle assigned a
unique LaneID, moving away from bi-directional counting. Best view in color.

This paper introduces a Real-Time Automatic Lane Learning and Lane-Specific
Traffic Status Monitoring System, building upon our prior research [11, 12].
In our previous work, we developed a method for automatically identifying
highway locations, lane boundaries, and traffic flow directions from video
footage captured by cameras. Leveraging this acquired knowledge, the current
study focuses on the real-time collection and reporting of traffic data for
each lane, including metrics such as vehicle counts (illustrated in Fig. 1),
flow rates, and congestion estimates derived from vehicle count statistics.
Furthermore, this system is equipped to autonomously detect changes in the
camera’s angle or zoom level, subsequently reinitiating the road and lane
learning process to maintain continuous and accurate traffic status
monitoring.

The contributions of our work can be summarized as follows:

  * •

Building on our earlier contributions in LCD[11] and MRLL[12], this paper
introduces an advanced, adaptive system capable of lane-wise vehicle counting,
flow rate calculation, and traffic status detection that operates
continuously, around the clock. This system is designed to accommodate cameras
of varying resolutions and frame rates and is robust against a wide range of
weather and traffic conditions, utilizing real-time video streams as its
primary input source.

  * •

We have developed a novel, standalone module named “Camera View Checking” that
operates continuously to monitor for any changes in the camera’s angle or
view. It accomplishes this by comparing the current camera view with an
initial, well-defined reference view. Should any deviation in the camera angle
or view be detected, this module automatically triggers the system to re-
initiate the learning process for road and lane parameters.

  * •

We devised a novel approach, termed “Video Rate-Computer Speed
Synchronization” to dynamically adjust the input frame rate in accordance with
the system’s processing capabilities, thereby ensuring the maintenance of
real-time performance.

  * •

We have developed a region-based tracking system leveraging an enhanced
DeepSort architecture, specifically designed for real-time tracking in the
most effective Regions of Interest (ROIs) to ensure optimal vehicle detection.
Key improvements include the integration of our ”Video Rate-Computer Speed
Synchronization” method, which dynamically adjusts input frame rates based on
processing speed to sustain real-time operations. We have also transitioned
from using the traditional Intersection Over Union (IOU) metric to the more
advanced Complete IOU (CIOU) distance, significantly improving the precision
of detection and tracking data association. Furthermore, the introduction of
an adaptive matching threshold has been instrumental in optimizing the
system’s tracking accuracy.

  * •

We have thoroughly evaluated the effectiveness of our proposed lane-wise
vehicle counting system by comparing it with manual counting in 9 videos.

Figure 2:  The system we designed is running on every local computer with a
single GPU. This system gets real-time stream input from a single camera.
Since cameras are not fixed on highways, the Environment Learning module will
learn the road and lane information based on vehicle motion trajectories (the
details can be found in our previous work: [11, 13, 12]). When these
parameters about road and lanes are learned for a particular view (can be
defined by customer), the system will go to the next stage, to do Road
Condition and Traffic Status Detection via each lane. An independent module
(i.e., Camera View Checking) keeps running all the time to check camera angle
changes or not by comparing with the first, well-defined view. Once the system
detects the camera angle/view is changed, the Road Condition and Traffic
Status Detection module stops working immediately and the system goes to the
Environment Learning to learn a new set of road and lane information
parameters. More details are explained in Section III.

##  II Related Works

###  II-A Object Detection

Traditional object detection methods like background subtraction [14] and
image difference [15] have been replaced by deep learning, due to superior
detection accuracy and speed. Deep learning methods fall into two categories:
two-stage detectors (e.g., Faster R-CNN, Mask R-CNN [16, 17]) and one-stage
detectors (e.g., YOLO, YOLOv4 [18, 19]). Two-stage detectors offer superior
accuracy, while one-stage detectors provide faster inference without pre-
generated region proposals. These deep learning-based object detection
algorithms are extensively employed in traffic surveillance videos, delivering
satisfactory performance [20, 21].

###  II-B Multiple Object Tracking

Most object-tracking algorithms fall into two categories: detection-based
tracking (tracking-by-detection) and detection-free tracking. In recent years,
tracking-by-detection methods have been predominant. These methods involve
performing object detection on each frame to obtain detection results, which
are then associated with adjacent frames to form trajectories. Popular data
association methods include the Hungarian algorithm and the Kuhn-Munkres
algorithm [22]. Post-processing techniques like soft non-maximum suppression
(NMS) [23] are often applied to smooth and refine the trajectories. The
Intersection Over Union (IOU) [24] metric is commonly used to associate
detection results based on bounding box overlaps between frames. SORT [25] and
Deep SORT [26] are two widely used tracking methods in traffic surveillance
videos [27, 28]. SORT combines Kalman Filter with the Hungarian algorithm and
relies on bounding box size and position for motion estimation and data
association. Deep SORT utilizes a CNN to extract appearance information,
enhancing the association metric with motion information. This allows Deep
SORT to track objects more effectively during occlusions, reducing ID switches
but also increasing computational costs.

##  III Method

This section summarizes our system, described in [11, 12], that analyzes
highway surveillance camera footage in real-time to detect changes in camera
angles or zoom levels. It then learns the number of lanes and their locations,
providing lane-wise vehicle counting, flow rate, and congestion estimation in
optimal ROIs with best vehicle detection. The system, depicted in Fig. 2, runs
on a local computer with a single GPU and comprises two main components:
Environment Learning for adapting to unpredictable camera settings and Road
Condition and Traffic Status Detection for analyzing lane-based traffic
information.

###  III-A Environment Learning

The goal of this component includes finding the locations of the lanes and
optimal locations to detect and count vehicles on the lanes. Instead of
relying on non-robust lane-markings in highways, we adopted a method that uses
the vehicle motion information on the video to find the lane locations because
most vehicles on the highway are driven within their lanes [11, 12].

Camera View Checking The program monitors for changes in camera angle or zoom
level which would invalidate learned lane information. The efficient ImageHash
algorithm [29] is used to assess significant and rapid background changes in
the top fifth of the image frame (the sky and far side of the road), to signal
potential camera adjustments. If the Hash Distance value exceeds a certain
threshold, it is considered a camera angle or zoom level change, prompting a
relearning process. This check occurs every 50 frames.

The algorithm is proven to be very robust in various camera motions and
lighting conditions. The example testing cases are shown in Fig. LABEL:fig4,
such as when the cloud background changed significantly, when the two input
videos were taken over a long time apart, or when the camera zoomed out or
zoom in, under all of these tough PTZ camera situations, our algorithm gives
the correct detection with the default Hash threshold. This algorithm detects
the large camera angle shifts and also detects those less apparent camera
parameter changes. There are some cases that our algorithm cannot handle well,
as shown in Fig. LABEL:fig5. In summary, our camera viewing checking algorithm
works well in the majority of our ITS environments. The system monitors the
camera’s viewing angle. If it detects changes in angle or zoom level, it halts
and restarts the environment learning stage to relearn the road and lane
parameters from the video.

###  III-B Road Condition and Traffic Status Detection

Learning the ROI and lane boundaries makes lane-based traffic status detection
possible. The issues faced and solutions in lane-based traffic status
detection are described in the following subsections.

####  III-B1 Video Rate-Computer Speed Synchronization

The goal of this system development is that it can be used for all highway
surveillance cameras. Since the frame rate (frame per second: $fps$) and
resolution of these cameras can be quite different, the processing time for
image processing can vary significantly. We reduce the $fps$ processed based
on the computer computation speed(assume the time cost of each frame is
$\delta t$ with the unit of second) by adaptively skipping $K$ frames for
every frame processed during the process following Equation (1):

$K=\begin{cases}fps-\frac{1}{\delta t}&\text{if $\frac{1}{\delta t}-fps<0$}\\\
0&\text{otherwise}\end{cases}$ (1)

We designed a Video Rate-Computer Speed Synchronization method that observes
the computer processing time for each input frame and the incoming frames per
second. Then we use this information to determine how many frames should be
skipped for each frame processed so there will be no backlogged frames. Since
the processing time for each frame is affected by the vehicles captured and
processed in the frame, the number of frames skipped fluctuates over time.

####  III-B2 Vehicle Detection

The pre-trained YOLOv4 is used in this work and the input to vehicle detection
is no longer continuous but with $K$ frames skipped.

####  III-B3 Online Multi-target Tracking with Adaptive Kalman Filter

Before tracking, we filter out invalid detected vehicles, removing those with
bounding box sizes more than twice the median truck size and those outside any
ROI or roads. NMS is applied to eliminate redundant bounding box overlaps.
DeepSort [26] is then employed for online multiple vehicle tracking on the
remaining valid detection in each frame. However, the classic tracking
pipeline has limitations in our application. Firstly, real-time performance on
a single GPU is essential, but feature extraction from a deep learning model
at each frame is time-consuming. Secondly, our proposed skipping frame
strategy renders the IOU distance and fixed threshold in DeepSort inadequate,
as detection and tracklets may not overlap. Therefore, we modify DeepSort to
meet our application’s requirements.

We modified Deepsort: By removing object appearance feature matching and using
CIOU distance instead of IOU distance in Cascade Matching step in DeepSort
By importing the frame-skipping strategy, the whole system achieves real-time
processing status without much tracking accuracy loss. However, in the
default, Deepsort, the IOU distance used in the last step for matching
detected vehicles and tracks will not meet the requirement if the new detected
vehicles do not overlap. So the distance between the bounding box’s centers
and the consistency of the aspect ratio has the potential to solve this
problem. Based on the work CIOU loss [30], we replace the IOU distance with
CIOU distance. The original IOU distance is defined as Equation (2) and IOU is
the intersection-over-union of bounding boxes in detections and tracks. The
modified CIOU distance is defined as shown in Equation (3).

$d_{IOU}=1-IOU$ (2) $d_{CIOU}=1-IOU+D+\alpha*V$ (3)

In CIOU, the normalized central point distance $D$ is designed to measure the
distance of two boxes as calculated in Equation (4),

$D=\frac{\rho^{2}(p^{d},p^{t})}{c^{2}}$ (4)

where $p^{d}=[x_{d},y_{d}]^{T}$ and $p^{t}=[x_{t},y_{t}]^{T}$ are the central
points of boxes in detections and tracks , $c$ is the diagonal length of each
box in the track, and $\rho$ is specified as the Euclidean distance function.
$V$ is defined as the consistency of the aspect ratio and calculated as:
$\frac{4}{pi^{2}}(\arctan\frac{w^{d}}{h^{d}}-\arctan\frac{w^{t}}{h^{t}})$. The
trade-off parameter $\alpha$ is defined the same as with the default CIOU. The
CIOU distance mitigates the missing match when detections are not overlapping
with predicted tracks and are more robust to the scale of bounding boxes.

The matching distance threshold was adjusted with the skipped frame number $K$
in Cascade Matching step in DeepSort
Based on the above analysis, a fixed IOU distance threshold does not work well
because of different frames are skipped in each iteration. So we adjust the
CIOU distance threshold to $pre_{thre}*(K+1)$, where $K$ donates the skipped
frame number and $pre_{thre}$ is the default fixed matching threshold in
Deepsort, if no frame is skipped, $K$ equals 0.

After tracking, each vehicle passing through the ROI is assigned a unique
$Fid$.

####  III-B4 Lane-wise Vehicle Counting

The result of vehicle counting with time information can be proportionally
converted to flow rate and valuable traffic status information. To realize
lane-wise vehicle counting, we associate every tracked vehicle $i$ in the ROI
with $Fid_{i}$ to a unique lane ID $Lid_{i}$ using the lane boundary
information obtained in the environment learning stage. When assigning the
lane ids to each vehicle in the current frame, only the previous frame is
compared. This strategy will not only reduce the time cost and also reduce the
id switch when we assign the lane ids if the id switch happens in FIDs in a
longer trace. If the center of a tracked vehicle is within the boundary of a
lane and within the average car length at the baseline, the vehicle is counted
for that lane. A tracked vehicle is only counted one time.

####  III-B5 Traffic Status Detection

The system can generate traffic status and incident reports for the users in
three steps: (i) determine the flowrate of each lane based on the vehicle
counts over a specific time interval periodically (ii) estimate the percentage
of pixels occupied by all vehicles on each lane within the ROI as occupancy
rate as shown in the Equation (5), and (iii) traffic status checking based on
the combination of the flow rate and occupancy.

$Occp_{l}=\frac{\sum(h_{l})}{H}$ (5)

, here $l$ is the lane ID, and $h_{l}$ is the sum of all vehicles’ bounding
box height in lane $l$. For every $T$ minute, the instantaneous flow rate
$Fr_{l}$(vehicle per hour) of lane $l$, is calculated based on the vehicle
counting $C_{l}$ of lane $l$ during $T$: $Fr_{l}=\frac{C_{l}*60}{T}$. The time
interval for the flow rate calculation cannot be too long since it will not
show the short-term traffic status changes. On the other hand, the time
interval for the flow rate calculation cannot be too short since it will
fluctuate too much to understand its meaning.

Figure 4: 9 ITS traffic scenes recorded in sunny, rainy, snowy, nighttime, and
congestion traffic conditions.

##  IV Experiments

###  IV-A Experimental Settings

Datasets. For vehicle count testing purpose, we created another 9 videos and
each video lasts about 2 minutes. These testing videos have unique camera
views and various weather, traffic density and visibility as shown in Fig. 4.
We assume each video has finished the ROIs and lane learning including: lane
center locations, lane directions, lane boundaries in all the ROIs.

Evaluation Metrics. To estimate our system’s performance in vehicle counting
in each lane, we manually counted 9 videos from four scenes including sunny,
rainy, night, and congested traffic, and each video lasts 2 minutes. We
counted vehicles in each lane separately when every vehicle passes the
baseline. The flowrate of ground truth is also estimated with the counting
ground truth. The total counting accuracy of one video is defined as the
corrected counting percentage: $\frac{total\quad system\quad count}{total\quad
ground\quad truth\quad count}*\%$. We consider the smaller count as the
numerator and the higher the value means the higher the counting accuracy. We
design two other matrices to estimate the road level flow rate accuracy
compared with the flow rate generated from the ground truth counting: MEA(mean
estimation accuracy): $\frac{\sum(Fr_{l})}{\sum(FrGT_{l})}$ and RMSE(root mean
square error): $\sqrt{\frac{\sum{(Fr_{l}-FrGT_{l})^{2}}}{m}}$. We define the
$FrGT_{l}$ as the ground truth flow rate at lane $l$ and $m$ as the total
number of lanes. The MAE has a similar definition of counting accuracy, we
regard the smaller value as the numerator and the higher value means the
estimation is closer to the ground truth. The RMSE estimates the whole error
of estimation, and the lower value means better counting and flow rate
estimation. To check the traffic status, we defined an estimate rule based on
the flow rate and occupancy rate of each lane as shown in Equation (6). Based
on the rules we defined, our system gets 100% accurate traffic status (normal,
slow, and jam) reports of all the lanes of total 9 videos.

$status\\_{l}=\begin{cases}Jam&\begin{subarray}{c}\text{if $Fr_{l}<600$ and
$Occp_{l}>0.6$}\end{subarray}\\\ Slow&\begin{subarray}{c}\text{if
$600<Fr_{l}<900$ and $0.4<Occp_{l}<0.6$}\end{subarray}\\\
Normal&\text{otherwise}\end{cases}$ (6)

Manual Count. We hire 2 master students to count the actual vehicles crossing
the counting line manually. For each video, they count the vehicles in each
video on a regular interval which is 30 seconds and totally count 4 intervals.
And we also record flow rate value, and occupancy in each lane during the same
interval calculated automatically by our designed system. Finally, we analyze
how these two indicators change over time and do the comparison.

Implementation Details. We utilized YOLOv4 [19], a popular deep learning
object detector, to detect vehicles in camera frame streams. During the
reference phase, we set detection confidence score thresholds to 0.25 and IOU
threshold to 0.45. For tracking, we employed Deep Sort framework, excluding
the CNN feature extractor. Cosine distance metric facilitated track
association in each frame, with an initial IOU distance threshold of 0.35.
Parameters like $max\\_age$ (30) and $n\\_init$ (3) were set to control track
deletion and initialization phases, respectively. Coupled with our frame-
skipping strategy, the entire system ran in real-time on a single NVIDIA
Quadro RTX 5000 GPU.

###  IV-B Results

Figure 5: The overall count percentage for all vehicles from 9 cases by our
counting system. Lower than 100% means miss count, larger than 100% means over
count, and 100% means count by our system is totally the same as the ground
truth.

Lane-wise Real-time Vehicle Counting. The lane detection in this system
performs with acceptable accuracy when not in totally dark conditions.
Experimental results on our own collected highway data demonstrates the
effectiveness of our proposed framework in lane-wise vehicle counting by
comparing with manual ground truth. The system count of 7 of 9 videos is close
to the ground truth, and lowest counting error is 2% as shown in Fig.5.
However, the performance decreases in low lighting conditions, poor viewing
angles, or with occlusions.

Lane-wise Flow Rate Estimation. When estimating our system’s lane-wise flow
rate estimation accuracy, we only consider the cases in which all the lanes
are correctly learned. The flow rate estimation accuracy and error results of
the 8 videos are shown in TABLE I.

These results prove our system achieves good flow rate estimation accuracy and
the highest MAE reaches 0.95 and it reaches 0.86 even in night cases. The
average RMSE is lowest in the sunny and daytime cases which means the lighting
is still the major factor affecting the detection and counting.

Road Condition and Traffic Status Detection. Based on the rules we defined,
our system gets 100% accurate traffic status (normal, slow, and jam) reports
of all the lanes of all 9 videos.

TABLE I: Accuracy of lane-wise flow rate estimation in various scenarios. Video Name | MEA $\uparrow$ | RMSE $\downarrow$
---|---|---
Sunny 1 | 0.95 | 76.49
Sunny 2 | 0.93 | 54.38
Rainy 1 | 0.93 | 47.43
Rainy 2 | 0.77 | 228.47
Night 1 | 0.86 | 118.59
Night 2 | 0.93 | 36.74
Congestion 1 | 0.86 | 354.96
Congestion 2 | 0.9 | 129.03
Average | 0.89 | 130.63

##  V Conclusion

Highway surveillance cameras face challenges due to unpredictable changes in
viewing direction and zoom level. To address this, a lane-based automatic
traffic monitoring system with a lane learning component has been developed
and proved its success using real-time Indiana Highway data. The paper
outlines the system’s architecture and highlights the importance of vehicle
motion for lane detection, vehicle counting and traffic status estimation.

Limitations. The performance of our system depends critically on the accuracy
of its various sub-modules, which include lane learning, vehicle detection,
tracking, LaneID assignment, and counting. Specifically, lane learning, which
is primarily informed by video input and vehicle motion, tends to be less
effective in situations such as traffic jams or when vehicles remain
stationary. The task of vehicle detection faces significant challenges in the
Intelligent Transportation System (ITS) environment, where adverse weather
conditions and low-light conditions at night can significantly impair
detection capabilities.

In Multi-Object Tracking (MOT), one of the major hurdles is dealing with
occlusions, particularly in dense traffic scenarios where bounding boxes of
different vehicles overlap, making it difficult to track individual vehicles
accurately. Moreover, the use of heavy-weight computational architectures for
tracking hampers the ability to perform real-time tracking, necessitating the
development of lighter-weight alternatives that do not sacrifice accuracy.

To accurately assess the overall performance of our system, it is essential to
conduct tests across a broader range of scenarios to validate the system’s
robustness. This entails incorporating more diverse test cases that can
effectively simulate the wide variety of real-world conditions under which the
system must operate.

Future Work. To improve the overall accuracy of the system, it is crucial to
make incremental improvements to each individual module. Additionally,
employing a more diverse collection of test videos will be instrumental in
accurately estimating the system’s performance.

Acknowledgment. This work was supported by the Joint Transportation Research
Program (JTRP), administered by the Indiana Department of Transportation and
Purdue University, Grant SPR-4436. The authors would like to thank all INDOT
Study Advisory Committee members, including Jim Sturdevant, Ed Cox, Tim Wells,
for their guidance and advice throughout the project. The authors would also
thank Zhengming Ding, Stephan Gerve, Upadhyay, Aniket Pankaj, Prathamesh
Somkant Panat, Anup Atul Mulay, Baudouin Ramsey, Kunal Mandil, Arjun
Narukkanchira Anilkumar, and Kavya Prasad for their contributions in various
part of this project.

## References

  * [1] Tamer Nadeem et al.  Trafficview: A scalable traffic monitoring system.  In IEEE International Conference on Mobile Data Management, 2004. Proceedings. 2004, pages 13–26. IEEE, 2004.
  * [2] Neeraj Kumar Jain et al.  A review on traffic monitoring system techniques.  Soft computing: Theories and applications: Proceedings of SoCTA 2017, pages 569–577, 2019.
  * [3] Mahmoud Abbasi et al.  Deep learning for network traffic monitoring and analysis (ntma): A survey.  Computer Communications, 170:19–41, 2021.
  * [4] Igor Bisio et al.  A systematic review of drone based road traffic monitoring system.  IEEE Access, 2022.
  * [5] Pallavi A Mandhare et al.  Intelligent road traffic control system for traffic congestion: a perspective.  International Journal of Computer Sciences and Engineering, 6(07):2018, 2018.
  * [6] Vishal Mandal et al.  Artificial intelligence-enabled traffic monitoring system.  Sustainability, 12(21):9177, 2020.
  * [7] Xingchen Zhang et al.  Monocular visual traffic surveillance: a review.  IEEE TITS, 2022.
  * [8] Adriel Isaiah Valeroso Amoguis et al.  Road lane segmentation using vehicle trajectory tracking and lane demarcation lines.  In Proceedings of the 2023 6th International Conference on Machine Vision and Applications, pages 64–71, 2023.
  * [9] Aleksandr Fedorov et al.  Traffic flow estimation with data from a video surveillance camera.  Journal of Big Data, 6:1–15, 2019.
  * [10] Li Li et al.  Trajectory data-based traffic flow studies: A revisit.  Transportation Research Part C: Emerging Technologies, 114:225–240, 2020.
  * [11] Mei Qiu et al.  Intelligent highway lane center identification from surveillance camera video.  In ITSC, pages 2506–2511. IEEE, 2021.
  * [12] Mei Qiu et al.  Intelligent highway adaptive lane learning system in multiple rois of surveillance camera video.  IEEE TITS, 2024.
  * [13] Mei Qiu et al.  Attention mechanism improves yolov5x for detecting vehicles on surveillance videos.  In 2022 IEEE AIPR, pages 1–8. IEEE, 2022.
  * [14] Quin Cai et al.  Tracking human motion in an indoor environment.  In ICIP, volume 1, pages 215–218. IEEE, 1995.
  * [15] Daniel J Dailey et al.  An algorithm to estimate mean traffic speed using uncalibrated cameras.  TITS, 1(2):98–107, 2000.
  * [16] Shaoqing Ren et al.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [17] Kaiming He et al.  Mask r-cnn.  In ICCV, pages 2961–2969, 2017.
  * [18] Joseph Redmon et al.  You only look once: Unified, real-time object detection.  In CVPR, pages 779–788, 2016.
  * [19] Alexey Bochkovskiy et al.  Yolov4: Optimal speed and accuracy of object detection.  arXiv preprint arXiv:2004.10934, 2020.
  * [20] Hang Shi et al.  Anomalous driving detection for traffic surveillance video analysis.  In ICIST, pages 1–6. IEEE, 2021.
  * [21] Soumi Mitra et al.  Towards an optimised vehicle detection algorithm for multi-object tracking in traffic surveillance.  In AICS, pages 200–211, 2021.
  * [22] Harold W Kuhn.  The hungarian method for the assignment problem.  Naval research logistics quarterly, 2(1-2):83–97, 1955.
  * [23] Navaneeth Bodla et al.  Soft-nms–improving object detection with one line of code.  In ICCV, pages 5561–5569, 2017.
  * [24] Erik Bochinski, Volker Eiselein, and Thomas Sikora.  High-speed tracking-by-detection without using image information.  In 2017 14th IEEE international conference on advanced video and signal based surveillance (AVSS), pages 1–6. IEEE, 2017.
  * [25] Alex Bewley et al.  Simple online and realtime tracking.  In 2016 IEEE ICIP, pages 3464–3468. IEEE, 2016.
  * [26] Nicolai Wojke et al.  Simple online and realtime tracking with a deep association metric.  In 2017 IEEE ICIP, pages 3645–3649. IEEE, 2017.
  * [27] Chong Liu et al.  City-scale multi-camera vehicle tracking guided by crossroad zones.  In CVPR, pages 4129–4137, 2021.
  * [28] Lijun Yu et al.  Traffic danger recognition with surveillance cameras without training data.  In AVSS, pages 1–6. IEEE, 2018.
  * [29] David Marr and Ellen Hildreth.  Theory of edge detection.  Proceedings of the Royal Society of London. Series B. Biological Sciences, 207(1167):187–217, 1980.
  * [30] Zhaohui Zheng et al.  Enhancing geometric factors in model learning and inference for object detection and instance segmentation.  IEEE Trans. Cybern., 52(8):8574–8586, 2021.


# Room Transfer Function Reconstruction Using Complex-valued Neural Networks
and Irregularly Distributed Microphones

###### Abstract

Reconstructing the room transfer functions needed to calculate the complex
sound field in a room has several important real-world applications. However,
an unpractical number of microphones is often required. Recently, in addition
to classical signal processing methods, deep learning techniques have been
applied to reconstruct the room transfer function starting from a very limited
set of room transfer functions measured at scattered points in the room. In
this study, we employ complex-valued neural networks to estimate room transfer
functions in the frequency range of the first room resonances, using a few
irregularly distributed microphones. To the best of our knowledge, this is the
first time complex-valued neural networks are used to estimate room transfer
functions. To analyze the benefits of applying complex-valued optimization to
the considered task, we compare the proposed technique with a state-of-the-art
real-valued neural network method and a state-of-the-art kernel-based signal
processing approach for sound field reconstruction, showing that the proposed
technique exhibits relevant advantages in terms of phase accuracy and overall
quality of the reconstructed sound field.

Index Terms—  sound field reconstruction, RIR interpolation, complex-valued
neural network, room acoustics

##  1 Introduction

Immersive audio plays a crucial role in virtual and augmented reality
applications, leading to a growing interest in the navigation of the acoustic
scene [1, 2], often referred to as 6 Degrees of Freedom (6DOF). To develop
such solutions in an effective way, it is necessary to reconstruct the sound
field in a large area, with the availability of room transfer functions
measured in just a few points in the room.

Solutions for sound field reconstruction incorporate both model-based and
data-driven approaches. Model-based strategies leverage existing knowledge of
acoustic principles to address the task of sound field reconstruction. These
can be categorized into parametric [3, 4, 5, 6, 7], and non-parametric models
[8, 9, 10, 11]. Parametric models represent the sound scene using a limited
set of parameters, such as source position[12] and directivity, [3], to
effectively convey spatial audio information. Non-parametric models, on the
other hand, exploit combinations of plane waves [10] or spherical waves [13,
11] to accurately reconstruct the acoustic field. Typically, this class of
techniques relies on compressed sensing [14], which is often applied in
combination with the equivalent source model [9, 15]. Some non-parametric
solutions adopt as a prior the Helmholtz equation to derive the kernel of the
interpolation. Caviedes et. al. [16] explore the use of Gaussian Process (GP)
regression for sound field interpolation, while kernel-based interpolation
approach have been proposed in Ueno et.al. [17, 18] and Ribeiro et. al.[19,
20].

Mainly motivated by the encouraging outcomes observed within the acoustic
domain [21, 22, 23, 24], data-driven approaches have found application also in
addressing the challenge of sound field reconstruction. The method proposed in
Pezzoli et. al. [25] exploits the regularization of a Convolutional Neural
Network (CNN) to estimate the Room Impulse Responses (RIRs) without relying on
any assumptions about the acoustic model. Fernandez et. al. [26] and Pezzoli
et. al. [27] propose approaches that incorporate physics-informed neural
networks leveraging prior information from the wave equation to guide the
reconstruction of RIRs according to the governing equation of the system.
Despite operating in the time domain, the primary drawback of these techniques
lies in the necessity to retrain the model for every set of measurements.
Following the paradigm of image inpainting [28, 29], Lluis et. al. [30]
propose a CNN trained on an extensive dataset of Room Transfer Functions
(RTFs). This approach forces the network to learn features from RTFs under
various acoustic conditions, enabling its applicability to unseen rooms with
similar acoustic conditions. However, the majority of learning-based
approaches primarily handle real-valued features [31], leaving the inference
of the phase, if needed, to other processing blocks, such as Griffin-Lim [32],
leading to suboptimal results where the phase is key for good results. In this
context, the adoption of Complex-Valued Neural Networks (CVNNs) [33] is a
convenient choice, due to their ability to directly handle complex-valued data
and optimization. This makes them well-suited for addressing various audio
signal processing problems such as echo cancellation [34], speech enhancement
[35], speech separation [36], beamforming [37], and sound field reproduction
[38].

Inspired by the complex-valued representation of RTFs, this paper presents a
novel approach based on CVNNs and irregularly distributed microphones for
addressing the RTF reconstruction problem. Similarly to what done in [30], the
proposed method approaches the problem as an image inpainting task. To the
best of our knowledge, this is the first application of CVNNs in the context
of RTF reconstruction. Additionally, the approach showcases its efficacy by
successfully handling the reconstruction task with a minimal number of
irregularly distributed microphones. We consider a 2D grid of positions
deployed on a plane where we aim to estimate the RTFs. The input of the CVNN
is a sparse set of RTFs measured at a few points on the grid, while the
outputs are the RTFs estimated at all the grid points. Through a simulation
study, we compare the proposed model with the approaches proposed by LLuis et.
al. [30] and by Ueno et.al. [18], demonstrating the benefits of incorporating
CVNNs into the RTF reconstruction problem. The code used to train the proposed
model and compute the results is publicly available on GitHub
111https://github.com/RonFrancesca/complex-sound-field.

Fig. 1: Schematic representation of the proposed CVNN.

##  2 Data Model and Problem formulation

In the scope of this study, we consider an acoustic source positioned at
$\mathbf{s}\in\mathbb{R}^{3}$ and an array of microphones deployed on a two-
dimensional $W\times H$ grid in a shoebox room of dimension $L_{x}\times
L_{y}\times L_{z}$. We define the coordinates of a point $\mathbf{r}_{w,h}$ on
the grid as:

$\mathbf{r}_{w,h}=[w[L_{x}/(W-1)],h[L_{y}/(H-1)],\overline{z}]^{T},$ (1)

where $w=0,\ldots,W-1$ and $h=0,\ldots,H-1$ are the indexes of the microphones
on the grid and $\overline{z}$ is a fixed value on the z-axis. The RTF from
source $\mathbf{s}$ to microphone $\mathbf{r}_{w,h}$ in a lightly damped
shoebox room can be computed using an infinite summation of room modes,
expressed as:

$G(\mathbf{r}_{w,h}|\mathbf{s},\omega)=-\frac{1}{V}\sum_{n}^{\infty}\frac{\Psi_{n}(\mathbf{r}_{wh})\Psi_{n}(\mathbf{s})}{(\omega/c)^{2}-(\omega_{n}/c)^{2}-j\omega/\tau_{n}},$
(2)

where $\omega$ denotes the angular frequency, $c$ the speed of sound,
$\tau_{n}$ the decay time of the considered mode and $\Psi(\cdot)$ the
corresponding mode shape. For notational simplicity, we denote the mode,
identifed through the 3-dimensional index $[n_{x},n_{y},n_{z}]$ with the index
$n$, which describes its position on the frequency axis. The RTFs from source
$\mathbf{s}$ to points $\mathbf{r}_{w,h}$ at frequency $\omega$ are collected
in $\mathbf{G}\in\mathbb{C}^{W\times H}$ such that:

$[\mathbf{G}]_{w,h}(\mathbf{s},\omega)=G(\mathbf{r}_{w,h}|\mathbf{s},\omega).$
(3)

We define the set

$\mathcal{I}_{M}=\\{(\tilde{w},\tilde{h})|0<\tilde{w}<W-1,0<\tilde{h}<H-1\\},$
(4)

corresponding to positions on the 2D grid where no microphones are deployed.
The corresponding complex-valued RTFs measured on such incomplete grid can be
then defined as:

$[\tilde{\mathbf{G}}]_{w,h}(\mathbf{s},\omega)=\begin{cases}0,&\text{if}\
(w,h)\in\mathcal{I}_{M}\\\
[\mathbf{G}]_{w,h}(\mathbf{s},\omega),&\text{otherwise}\end{cases}.$ (5)

The RTF reconstruction problem is thus formulated as finding the function
$\mathcal{U}(\cdot)$ that provides an estimate $\hat{\mathbf{G}}$ of the RTF
matrix imposing that

$\hat{\mathbf{G}}=\arg\min_{\mathcal{U(\cdot)}}\sum_{w=0}^{W-1}\sum_{h=0}^{H-1}|[\mathbf{G}]_{w,h}-[\mathcal{U(\tilde{\mathbf{G}})}]_{w,h}|^{2},$
(6)

omitting the frequency dependence for simplicity.

##  3 Complex-valued Neural Network for room transfer function reconstruction

###  3.1 Complex-valued networks prerequisites

CVNNs operate with tensors containing complex values, necessitating a
redefinition of primary operations and activation functions.

Given a matrix of complex-valued weights $\mathbf{W}$ and a complex-valued
input tensor $\mathbf{X}$, we can define the convolution operation in CVNN as

$\mathbf{W}*\mathbf{X}=(\Re(\mathbf{W})+j\Im(\mathbf{W}))*(\Re(\mathbf{X})+j\Im(\mathbf{X})),$
(7)

which, by applying the distributive property of convolution, can be
conveniently expressed in matrix form [33].

Similarly, most real-valued activation functions have a complex-valued
equivalent [39]. In this study, we will use the CPReLU activation [40], which
can be defined as:

$\mathrm{CPReLU}(\mathbf{X})=\mathrm{PReLU}(\Re(\mathbf{X}))+j\mathrm{PReLU}(\Im(\mathbf{X})).$
(8)

During the training of a CVNN, weight updates involve complex-valued
gradients. The condition for a function to be differentiable (i.e.
holomorphic) in the complex domain is more stringent compared to the real
domain, since it is needed to satisfy also the Cauchy–Riemann equations. To
extend complex-valued calculus to non-holomorphic functions, Wirtinger
calculus [41] is applied to extract the gradient. Given a complex-valued
function $f(z)$, where $z=a+jb$ and $a,b\in\mathbb{R}$, the partial
derivatives w.r.t. $z$ and its conjugate $z^{*}$ can be computed as:

$\frac{\partial{f}}{\partial{z}}=1/2\left(\frac{\partial{f}}{\partial{a}}-j\frac{\partial{f}}{\partial{b}}\right),~{}\frac{\partial{f}}{\partial{z}^{*}}=1/2\left(\frac{\partial{f}}{\partial{a}}+j\frac{\partial{f}}{\partial{b}}\right).$
(9)

Given a real-valued loss function
$\mathcal{L}:\mathbb{C}\rightarrow\mathbb{R}$, the complex-gradient used to
perform backpropagation is calculated as introduced in [42, 43]:

$\nabla_{z^{*}}\mathcal{L}=2\frac{\partial{\mathcal{L}}}{\partial{z^{*}}}.$
(10)

$31.6$$39.8$$50.1$$63.1$$79.5$$100$$126$$159$$200$$251$$-25$$-20$$-15$$-10$$-5$$0$Frequency
$\left[$\mathrm{Hz}$\right]$$\mathrm{NMSE_{abs}}$
$\left[$\mathrm{dB}$\right]$$m=5$$m=15$$m=35$$m=55$

(a)

$31.6$$39.8$$50.1$$63.1$$79.5$$100$$126$$159$$200$$251$$-20$$-15$$-10$$-5$$0$Frequency
$\left[$\mathrm{Hz}$\right]$$\mathrm{NMSE_{complex}}$
$\left[$\mathrm{dB}$\right]$$m=5$$m=15$$m=35$$m=55$

(b)

Fig. 2: Normalized Mean Squared Error (NMSE) for different number of
microphones measured over the reconstructed magnitude (a) and the complex
pressure field (b). Thick lines correspond to results obtained with the
proposed method, dashed lines to the ones related to the technique proposed by
Lluis et.al. [30], and dash-dotted lines correspond to the technique proposed
by Ueno et.al. [18].

###  3.2 Input representation

To properly condition the proposed network model on reliable measurement
positions, we concatenate the incomplete RTF matrix $\tilde{\mathbf{G}}$ with
a binary mask $\mathbf{M}\in\mathbb{Z}_{2}^{W\times H}$ to create the input.
Given the index matrix $\mathcal{I}_{M}$, the binary mask $[M]_{w,h}$ is
obtained as

$[M]_{w,h}(\mathbf{s},\omega)=\begin{cases}0,&\text{if}\
(w,h)\in\mathcal{I}_{M}\\\ 1,&\text{otherwise}\end{cases}.$ (11)

Subsequently, denoting $K$ as the number of frequencies considered, the
network’s input is represented by the matrix
$\mathbf{G}_{\text{input}}\in\mathbb{C}^{W\times H\times 2K}$ obtained as:

$\tilde{\mathbf{G}}_{\text{input}}=[\tilde{\mathbf{G}}~{}~{}\mathbf{M}].$ (12)

In this context, the notation $[\cdot]$ denotes tensor concatenation along the
frequency axis.

###  3.3 Network Architecture

The CVNN proposed in this study adopts a U-Net-like [44] network architecture.
Specifically, the encoder is composed of four complex-valued convolutional
layers with filter counts of i) 128, ii) 256, iii) 512, and iv) 1024. The
decoder consists of five complex-valued convolutional layers with filter
counts of v) 512, vi) 256, vii) 128, viii) 80, and ix) 80. All encoder layers
employ a stride of $2\times 2$. In contrast, all decoder layers, except layer
ix), are preceded by a complex-valued upsampling operation with a factor of
$2\times 2$. Each complex-valued convolutional layer but layer ix) is followed
by CPReLU and complex-batch normalization [33], except layers i), viii), and
ix). The kernel size for all convolutional layers is $3\times 3$, except layer
ix), which has a kernel size of $1\times 1$. Four skip connections are
implemented by concatenating the inputs of layers v), vi), vii), and viii)
with the inputs of layers iv), iii), ii), and i), respectively. A schematic
representation of the proposed architecture is shown in Fig. 1.

###  3.4 Training Procedure

In the training phase, the matrix $\tilde{\mathbf{G}}_{\text{input}}$ serves
as the input for the CVNN $\mathcal{U}(\cdot)$. This network produces an
estimate $\hat{\mathbf{G}}$ of the ground truth complex-valued RTFs
$\mathbf{G}$.

The loss $\mathcal{L}:\mathbb{C}\rightarrow\mathbb{R}$ employed for
backpropagation is computed as the $\ell_{1}$ norm, expressed as follows:

$\mathcal{L}(\mathbf{G},\hat{\mathbf{G}})=\sum_{w=0}^{W-1}\sum_{h=0}^{H-1}|[\mathbf{G}]_{w,h}-[\hat{\mathbf{G}}]_{w,h}|,$
(13)

where, for simplicity, the batch index, frequency dependence, and active
source $\mathbf{s}$ have been omitted.

##  4 Experimental Validation

(a)

(b)

(c)

(d)

(e)

Fig. 3: Magnitude of the sound field, obtained using the proposed method (b),
Llluis et al. [30] (c), Ueno et al. [18] (d) using the $m=15$ microphone
configuration depicted in (e). Ground truth magnitude is shown in (a). Top
row: $[4.8~{}\mathrm{m}\times 5.4~{}\mathrm{m}\times 26.4~{}\mathrm{m}]$ room
with a $100~{}\mathrm{Hz}$ active source $\mathbf{s}$ positioned at
$[2.1484~{}\mathrm{m},2.0132~{}\mathrm{m},2.4~{}\mathrm{m}]^{T}$. Bottom row:
$[4.4~{}\mathrm{m}\times 9.3~{}\mathrm{m}\times 41.4~{}\mathrm{m}]$ room with
a $200~{}\mathrm{Hz}$ active source $\mathbf{s}$ positioned at
$[3.3481~{}\mathrm{m},5.1397~{}\mathrm{m},2.4~{}\mathrm{m}]^{T}$

(a)

(b)

(c)

Fig. 4: Phase of the soundfield obtained using the same configuration
considered in Fig. 3 using the proposed method (a), Ueno et al. [18] (b)
ground truth is shown in (b).

In this section, we present results from simulations designed to demonstrate
the effectiveness of employing complex-valued networks for RTF reconstruction.
In particular, we compare the proposed method with a data-driven approach
proposed by LLuis et. al. [30], as well as a signal processing kernel-based
interpolation technique proposed by Ueno et.al. [18].

###  4.1 Setup

To train the network-based models, we generated a dataset of $5000$ rooms,
divided with a 75/25% split between the training and validation sets,
respectively. The dataset has been generated following the procedure proposed
by Lluis et.al. [30]. Room dimensions were chosen based on the ITU-R BS.1116-3
standard for listening rooms, and the reverberation time is fixed to
$0.6~{}\mathrm{s}$. All methods have been tested on a separate dataset of
30000 rooms.

For this study, a significantly small set of irregularly placed microphones
has been considered. During the training of the network, the number of
microphones was randomly selected from a set of $5$, $10$, $15$, $35$, and
$55$ microphones. The selected microphones were irregularly distributed in the
considered room. The same has been done for Lluis et.al [30] architecture.

The proposed CVNN has been trained for $400$ epochs using a mini-batch of size
$16$ using the Adam optimizer [45] with the default configuration and a
learning rate of $0.001$. Lluis et. al. [30] model was trained for $400$
epochs using a mini-batch size of $32$ and the Adam optimizer with learning
rate $0.0001$. Ueno et al. [18] technique was used with default parameters and
a regularization factor of $0.1$. We considered $K=40$ frequencies in the
range between $30~{}\mathrm{Hz}$ and $300~{}\mathrm{Hz}$, as proposed by Lluis
et.al.[30].

###  4.2 Evaluation Metric

The models have been evaluated using two Normalized Mean Squared Error (NMSE)
metrics. When considering only the magnitude of the RTFs, the metric is
defined as:

$\mathrm{NMSE}_{\mathrm{abs}}=10\log_{10}\frac{\sum_{w=0}^{W-1}\sum_{h=0}^{H-1}||[\hat{\mathbf{G}}]_{w,h}|-|[\mathbf{G}]_{w,h}||^{2}}{\sum_{w=0}^{W-1}\sum_{h=0}^{H-1}|[\mathbf{G}]_{w,h}|^{2}},$
(14)

The second NMSE, which consider the entire complex field, is defined as:

$\mathrm{NMSE}_{\mathrm{complex}}=10\log_{10}\frac{\sum_{w=0}^{W-1}\sum_{h=0}^{H-1}|[\hat{\mathbf{G}}]_{w,h}-[\mathbf{G}]_{w,h}|^{2}}{\sum_{w=0}^{W-1}\sum_{h=0}^{H-1}|[\mathbf{G}]_{w,h}|^{2}}.$
(15)

###  4.3 Results

Fig. 2(a) illustrates the performances of $\mathrm{NMSE}_{\text{abs}}$, while
Fig. 2(b) showcases the outcomes for $\mathrm{NMSE}_{\text{complex}}$. Fig.
2(a) show that, when comparing the methods for each selected number of
microphones, the proposed method excels at lower frequencies, whereas Lluis et
al. [30] reach better performance at higher frequencies. However, the
advantage at high frequencies decreases as the considered number of
microphones increases. For $m=55$, the proposed method outperforms the
alternatives across almost the entire frequency range. This behavior can be
attributed to both the selected loss function, which tends to underperform at
higher frequencies, and the observation that the proposed model extracts more
information compared to Lluis et al. [30], while being evaluated on a smaller
set of measurements in this case. More specifically, this implies that the
proposed method undergoes a more intricate optimization process, wherein the
entire complex sound field must be reconstructed, while Lluis et al. [30]
concentrates on the magnitude. When compared with Ueno et al. [18], the
proposed CVNN consistently succeeded.

Fig. 2(b) report the results related to the $\mathrm{NMSE}_{\text{complex}}$.
In this case, it is not possible to compare the proposed method with Lluis et
al. [30], since the latter method only reconstructs the magnitude of the
pressure field. From the figure, it is possible to observe that the error
increases together with the frequency, but it never reaches $0~{}\mathrm{dB}$.
As expected, also in this case, the error decreases as the number of
microphones increases, likely due to the greater amount of input information.

In Fig. 3, an example of the reconstructed magnitude is displayed, obtained
using the considered neural networks with a configuration of $15$ microphones
at a frequency of $100~{}Hz$. As it is possible to observe, while both methods
can give a rough idea of the mode behavior present in the room at the
considered frequency, the proposed CVNN shows more precise performances.

Fig. 4 shows the reconstructed phase using the same setup. Through visual
examination, it is noticeable that the RTF obtained using the proposed method
is nearly indistinguishable from the ground-truth.

##  5 Conclusion

This paper proposed a novel application of complex-valued neural networks to
the problem of room transfer function reconstruction, considering a
significantly small set of irregularly placed microphones, overcoming the need
for an unpractical number of microphones. We compare the magnitude
reconstruction performance of the proposed model with a state-of-the-art data-
driven technique, showing better performances at low frequencies while
underperforming at higher frequencies, although the gap becomes smaller as the
number of microphones increases. The proposed CVNN is also compared with a
kernel interpolation method, outperforming the latter both at low and high
frequencies. The proposed architecture is also able to reconstruct the phase
of the RTFs, differently from most learning-based approaches. The results
obtained motivate deeper investigation into incorporating CVNN in addressing
RTF reconstruction challenges. Future work aims to enhance the performance of
the proposed model by considering more suitable loss functions, including
direct considerations of phase handling. Additionally, future work will focus
on analyzing the behavior of the proposed model across frequency ranges beyond
the modal range.

## References

  * [1] J. G. Tylka and E. Y. Choueiri, “Fundamentals of a parametric method for virtual navigation within an array of ambisonics microphones,” JAES, vol. 68, no. 3, pp. 120–137, 2020.
  * [2] M. Cobos, J. Ahrens, K. Kowalczyk, and A. Politis, “An overview of machine learning and other data-based methods for spatial audio capture, processing, and reproduction,” Eurasip J. Audio Speech Music Process., vol. 2022, no. 1, pp. 1–21, 2022.
  * [3] M. Pezzoli, F. Borra, F. Antonacci, S. Tubaro, and A. Sarti, “A parametric approach to virtual miking for sources of arbitrary directivity,” IEEE/ACM Trans. Acoust., Speech, Signal Process., vol. 28, pp. 2333–2348, 2020\.
  * [4] M. Pezzoli, F. Borra, F. Antonacci, A. Sarti, and S. Tubaro, “Reconstruction of the virtual microphone signal based on the distributed ray space transform,” in 26th Eur. Signal Process. Conf., pp. 1537–1541, IEEE, 2018.
  * [5] L. McCormack, A. Politis, R. Gonzalez, T. Lokki, and V. Pulkki, “Parametric ambisonic encoding of arbitrary microphone arrays,” IEEE/ACM Trans. Acoust., Speech, Signal Process., vol. 30, pp. 2062–2075, 2022.
  * [6] L. McCormack, A. Politis, T. McKenzie, C. Hold, and V. Pulkki, “Object-based six-degrees-of-freedom rendering of sound scenes captured with multiple ambisonic receivers,” JAES, vol. 70, no. 5, pp. 355–372, 2022.
  * [7] V. Pulkki, S. Delikaris-Manias, and A. Politis, Parametric time-frequency domain spatial audio.  Wiley Online Library, 2018.
  * [8] S. Koyama and L. Daudet, “Sparse representation of a spatial sound field in a reverberant environment,” IEEE J. Sel. Top. Signal Process., vol. 13, no. 1, pp. 172–184, 2019.
  * [9] N. Antonello, E. De Sena, M. Moonen, P. A. Naylor, and T. Van Waterschoot, “Room impulse response interpolation using a sparse spatio-temporal representation of the sound field,” IEEE/ACM Trans. Acoust., Speech, Signal Process., vol. 25, no. 10, pp. 1929–1941, 2017.
  * [10] W. Jin and W. B. Kleijn, “Theory and design of multizone soundfield reproduction using sparse methods,” IEEE/ACM Trans. Acoust., Speech, Signal Process., vol. 23, no. 12, pp. 2343–2355, 2015.
  * [11] M. Pezzoli, M. Cobos, F. Antonacci, and A. Sarti, “Sparsity-based sound field separation in the spherical harmonics domain,” in Int. Conf. Acoust. Speech Signal Process, IEEE, 2022.
  * [12] O. Thiergart, G. Del Galdo, M. Taseska, and E. A. P. Habets, “Geometry-based spatial sound acquisition using distributed microphone arrays,” IEEE Trans. Acoust., Speech, Signal Process., vol. 21, no. 12, pp. 2583–2594, 2013\.
  * [13] A. Fahim, P. N. Samarasinghe, and T. D. Abhayapala, “Sound field separation in a mixed acoustic environment using a sparse array of higher order spherical microphones,” in 2017 HSCMA, pp. 151–155, IEEE, 2017.
  * [14] D. L. Donoho, “Compressed sensing,” IEEE Trans. on information theory, vol. 52, no. 4, pp. 1289–1306, 2006.
  * [15] I. Tsunokuni, K. Kurokawa, H. Matsuhashi, Y. Ikeda, and N. Osaka, “Spatial extrapolation of early room impulse responses in local area using sparse equivalent sources and image source method,” Applied Acoustics, vol. 179, p. 108027, 2021.
  * [16] D. Caviedes-Nozal, N. A. Riis, F. M. Heuchel, J. Brunskog, P. Gerstoft, and E. Fernandez-Grande, “Gaussian processes for sound field reconstruction,” The Journal of the Acoustical Society of America, vol. 149, no. 2, pp. 1107–1119, 2021.
  * [17] N. Ueno, S. Koyama, and H. Saruwatari, “Sound field recording using distributed microphones based on harmonic analysis of infinite order,” Signal Process. Letters, vol. 25, no. 1, pp. 135–139, 2017.
  * [18] N. Ueno, S. Koyama, and H. Saruwatari, “Kernel ridge regression with constraint of helmholtz equation for sound field interpolation,” in Int. Workshop Acoust. Signal Enhanc., pp. 1–440, IEEE, 2018.
  * [19] J. G. Ribeiro, N. Ueno, S. Koyama, and H. Saruwatari, “Region-to-region kernel interpolation of acoustic transfer functions constrained by physical properties,” IEEE/ACM Trans. Acoust., Speech, Signal Process., vol. 30, pp. 2944–2954, 2022.
  * [20] J. G. Ribeiro, S. Koyama, and H. Saruwatari, “Kernel interpolation of acoustic transfer functions with adaptive kernel for directed and residual reverberations,” arXiv preprint arXiv:2303.03869, 2023.
  * [21] M. Olivieri, R. Malvermi, M. Pezzoli, M. Zanoni, S. Gonzalez, F. Antonacci, and A. Sarti, “Audio information retrieval and musical acoustics,” IEEE Instrum. Meas. Mag, vol. 24, no. 7, pp. 10–20, 2021.
  * [22] L. Comanducci, F. Borra, P. Bestagini, F. Antonacci, S. Tubaro, and A. Sarti, “Source localization using distributed microphones in reverberant environments based on deep learning and ray space transform,” IEEE/ACM Trans. Acoust., Speech, Signal Process., vol. 28, pp. 2238–2251, 2020.
  * [23] M. Olivieri, M. Pezzoli, F. Antonacci, and A. Sarti, “A physics-informed neural network approach for nearfield acoustic holography,” Sensors, vol. 21, no. 23, 2021.
  * [24] M. J. Bianco, P. Gerstoft, J. Traer, E. Ozanich, M. A. Roch, S. Gannot, and C.-A. Deledalle, “Machine learning in acoustics: Theory and applications,” JASA, vol. 146, no. 5, pp. 3590–3628, 2019.
  * [25] M. Pezzoli, D. Perini, A. Bernardini, F. Borra, F. Antonacci, and A. Sarti, “Deep prior approach for room impulse response reconstruction,” Sensors, vol. 22, no. 7, p. 2710, 2022.
  * [26] E. Fernandez-Grande, X. Karakonstantis, D. Caviedes-Nozal, and P. Gerstoft, “Generative models for sound field reconstruction,” JASA, vol. 153, no. 2, pp. 1179–1190, 2023.
  * [27] M. Pezzoli, F. Antonacci, and A. Sarti, “Implicit neural representation with physics-informed neural networks for the reconstruction of the early part of room impulse responses,” in Forum Acusticum 2023, EAA, 2023.
  * [28] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Deep image prior,” in Proceedings of the IEEE Conf. on computer vision and pattern recognition, pp. 9446–9454, 2018.
  * [29] G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, and B. Catanzaro, “Image inpainting for irregular holes using partial convolutions,” in Proceedings of the European conference on computer vision (ECCV), pp. 85–100, 2018.
  * [30] F. Lluís, P. Martínez-Nuevo, M. Bo Møller, and S. Ewan Shepstone, “Sound field reconstruction in rooms: Inpainting meets super-resolution,” JASA, vol. 148, no. 2, pp. 649–659, 2020.
  * [31] J. A. Barrachina, C. Ren, C. Morisseau, G. Vieillard, and J.-P. Ovarlez, “Complex-valued vs. real-valued neural networks for classification perspectives: An example on non-circular data,” in ICASSP 2021-2021 IEEE Int. Conf. on Acoustic., Speech and Signal Process. (ICASSP), pp. 2990–2994, IEEE, 2021.
  * [32] D. Griffin and J. Lim, “Signal estimation from modified short-time fourier transform,” IEEE Transactions on acoustics, speech, and signal processing, vol. 32(2), pp. 236–243, 1984.
  * [33] C. Trabelsi, O. Bilaniuk, Y. Zhang, D. Serdyuk, S. Subramanian, J. F. Santos, S. Mehri, N. Rostamzadeh, Y. Bengio, and C. J. Pal, “Deep complex networks,” in ICLR, 2018.
  * [34] M. M. Halimeh, T. Haubner, A. Briegleb, A. Schmidt, and W. Kellermann, “Combining adaptive filtering and complex-valued deep postfiltering for acoustic echo cancellation,” in Int. Conf. Acoust. Speech Signal Process, pp. 121–125, IEEE, 2021.
  * [35] S. Zhao, T. H. Nguyen, and B. Ma, “Monaural speech enhancement with complex convolutional block attention module and joint time frequency losses,” in Int. Conf. Acoust. Speech Signal Process., pp. 6648–6652, IEEE, 2021.
  * [36] Y.-S. Lee, C.-Y. Wang, S.-F. Wang, J.-C. Wang, and C.-H. Wu, “Fully complex deep neural network for phase-incorporating monaural source separation,” in Int. Conf. Acoust. Speech Signal Process, pp. 281–285, IEEE, 2017.
  * [37] K. N. Watcharasupat, T. N. T. Nguyen, W.-S. Gan, S. Zhao, and B. Ma, “End-to-end complex-valued multidilated convolutional neural network for joint acoustic echo cancellation and noise suppression,” in Int. Conf. Acoust. Speech Signal Process, pp. 656–660, IEEE, 2022.
  * [38] L. Comanducci, F. Antonacci, and A. Sarti, “Synthesis of soundfields through irregular loudspeaker arrays based on convolutional neural networks,” arXiv preprint arXiv:2205.12872, 2022.
  * [39] Y. Kuroe, M. Yoshid, and T. Mori, “On activation functions for complex-valued neural networks—existence of energy functions—,” in Int.Conf. on Artificial Neural Networks, pp. 985–992, Springer, 2003.
  * [40] A. Pandey and D. Wang, “Exploring deep complex networks for complex spectrogram enhancement,” in ICASSP 2019-2019 IEEE Int.Conf. on Acoust., Speech and Signal Process. (ICASSP), pp. 6885–6889, IEEE, 2019.
  * [41] W. Wirtinger, “Zur formalen theorie der funktionen von mehr komplexen veränderlichen,” Mathematische Annalen, vol. 97, no. 1, pp. 357–375, 1927.
  * [42] H. Leung and S. Haykin, “The complex backpropagation algorithm,” IEEE Trans. on signal process., vol. 39, no. 9, pp. 2101–2104, 1991.
  * [43] M. F. Amin, M. I. Amin, A. Y. H. Al-Nuaimi, and K. Murase, “Wirtinger calculus based gradient descent and levenberg-marquardt learning algorithms in complex-valued neural networks,” in Int.Conf. on Neural Information Processing, pp. 550–559, Springer, 2011.
  * [44] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th Int. Conf., Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234–241, Springer, 2015.
  * [45] D. Kingma and J. Ba, “Adam: A method for stochastic optimization, 3rd int. conf. on learning representations,” arXiv preprint arXiv:1412.6980, 2014\.