In addition, we provide the zero temperature limit of the JC formula (127)

$\displaystyle J(\varphi)$
$\displaystyle=\frac{2\Delta_{0}\sin\frac{\varphi}{2}}{\Phi_{0}}$ (253)
$\displaystyle\times\int_{0}^{\infty}d\bar{\omega}\left(\frac{\bar{k}^{(-)}\cos\frac{\varphi}{2}}{\sqrt{\bar{\omega}^{2}+1}}+\bar{\Delta}_{loc}\right)\frac{R(\bar{\omega})\bar{k}^{(-)}}{\sqrt{\bar{\omega}^{2}+1}}.$

##  Appendix G Majorana junction

In this Appendix, we evaluate bulk Green’s functions and their spatial
derivatives which are needed for constructing the $d$-matrix in the Majorana
junction model (both in the short $W\to 0$ and finite $W$ cases). The derived
expressions are used for plotting energy- and current-phase relations in the
numerical examples presented in the main text.

###  G.1 Short junction

The bulk Green’s function $g(x)$ occurring in (134) is defined by

$\displaystyle g(x)=\int\frac{dk}{2\pi}e^{ikx}g_{k},$ (254)

where

$\displaystyle g_{k}=$ $\displaystyle\begin{pmatrix}a_{k}^{(+)}&-\Delta_{0}\\\
-\Delta_{0}&a_{k}^{(-)}\end{pmatrix}^{-1},$ (255) $\displaystyle
a_{k}^{(\pm)}$ $\displaystyle=z\mp h_{k}^{(0)}\mp\alpha
k\sigma_{z}+B\sigma_{x},$ (256) $\displaystyle h_{k}^{(0)}=$
$\displaystyle\frac{k^{2}}{2m}-\mu.$ (257)

The inverse in (255) may be written as

$\displaystyle
g_{k}=\frac{P_{k}}{Q_{k}}=\frac{1}{Q_{k}}\left(\begin{array}[]{cc}a_{k}^{(-)}A_{k}^{(+)}&\Delta_{0}A_{k}^{(-)}\\\
\Delta_{0}A_{k}^{(+)}&a_{k}^{(+)}A_{k}^{(-)}\end{array}\right),$ (260)

where

$\displaystyle A_{k}^{(+)}$
$\displaystyle=\text{adj}\,[a_{k}^{(+)}a_{k}^{(-)}-\Delta^{2}_{0}]$
$\displaystyle=z^{2}-\Delta_{0}^{2}+B^{2}-(h_{k}^{(0)})^{2}-\alpha^{2}k^{2}$
$\displaystyle-2zB\sigma_{x}+2\alpha kh_{k}^{(0)}\sigma_{z}+2iB\alpha
k\sigma_{y},$ (261) $\displaystyle A_{k}^{(-)}$
$\displaystyle=\text{adj}\,[a_{k}^{(+)}a_{k}^{(-)}-\Delta^{2}_{0}]$
$\displaystyle=z^{2}-\Delta_{0}^{2}+B^{2}-(h_{k}^{(0)})^{2}-\alpha^{2}k^{2}$
$\displaystyle-2zB\sigma_{x}+2\alpha kh_{k}^{(0)}\sigma_{z}-2iB\alpha
k\sigma_{y}$ (262)

and

$\displaystyle Q_{k}$
$\displaystyle=\det\,[a_{k}^{(+)}a_{k}^{(-)}-\Delta^{2}_{0}]$
$\displaystyle=\det\,[a_{k}^{(-)}a_{k}^{(+)}-\Delta^{2}_{0}]$
$\displaystyle=[z^{2}-\Delta_{0}^{2}+B^{2}-(h_{k}^{(0)})^{2}-\alpha^{2}k^{2}]^{2}$
$\displaystyle-4z^{2}B^{2}-4\alpha^{2}k^{2}(h_{k}^{(0)})^{2}+4\alpha^{2}B^{2}k^{2}.$
(263)

Setting $z=\omega+i0^{+}$ we find on the basis of (147)

$\displaystyle g(x)$
$\displaystyle=i\Theta(x)\sum_{s=1}^{4}\frac{(2m)^{4}e^{ik_{s,+}x}}{\prod_{s^{\prime}\neq
s}(k_{s,+}-k_{s^{\prime},+})\prod_{s^{\prime}}(k_{s,+}-k_{s^{\prime},-})}\left(\begin{array}[]{cc}a_{k}^{(-)}A_{k}^{(+)}&\Delta_{0}A_{k}^{(-)}\\\
\Delta_{0}A_{k}^{(+)}&a_{k}^{(+)}A_{k}^{(-)}\end{array}\right)\bigg{|}_{k=k_{s,+}}$
(266)
$\displaystyle-i\Theta(-x)\sum_{s=1}^{4}\frac{(2m)^{4}e^{ik_{s,-}x}}{\prod_{s^{\prime}}(k_{s,-}-k_{s^{\prime},+})\prod_{s^{\prime}\neq
s}(k_{s,-}-k_{s^{\prime},-})}\left(\begin{array}[]{cc}a_{k}^{(-)}A_{k}^{(+)}&\Delta_{0}A_{k}^{(-)}\\\
\Delta_{0}A_{k}^{(+)}&a_{k}^{(+)}A_{k}^{(-)}\end{array}\right)\bigg{|}_{k=k_{s,-}},$
(269)

where $k_{s,+}$ and $k_{s,-}$ are the the roots of the equation $Q_{k}=0$ with
positive and imaginary parts, respectively.

The expression (269) allows one to easily establish

$\displaystyle g^{\prime}(0^{+})$
$\displaystyle=-\sum_{s=1}^{4}\frac{(2m)^{4}k_{s,+}}{\det\prod_{s^{\prime}\neq
s}(k_{s,+}-k_{s^{\prime},+})\prod_{s^{\prime}}(k_{s,+}-k_{s^{\prime},-})}\left(\begin{array}[]{cc}a_{k}^{(-)}A_{k}^{(+)}&\Delta_{0}A_{k}^{(-)}\\\
\Delta_{0}A_{k}^{(+)}&a_{k}^{(+)}A_{k}^{(-)}\end{array}\right)\bigg{|}_{k=k_{s,+}},$
(272) $\displaystyle g^{\prime}(0^{-})$
$\displaystyle=\sum_{s=1}^{4}\frac{(2m)^{4}k_{s,-}}{\prod_{s^{\prime}}(k_{s,-}-k_{s^{\prime},+})\prod_{s^{\prime}\neq
s}(k_{s,-}-k_{s^{\prime},-})}\left(\begin{array}[]{cc}a_{k}^{(-)}A_{k}^{(+)}&\Delta_{0}A_{k}^{(-)}\\\
\Delta_{0}A_{k}^{(+)}&a_{k}^{(+)}A_{k}^{(-)}\end{array}\right)\bigg{|}_{k=k_{s,-}}.$
(275)

###  G.2 Expansion in the spin-orbit dominated regime

To reproduce the results of Ref. [Murthy  _et al._ , 2020] in the spin-orbit
dominated regime, we perform the expansion of (269) to the leading order
$1/\alpha$. This gives

$\displaystyle g(x)\approx\frac{e^{iC_{0}|x|/\alpha}e^{-i2m\alpha
x\sigma_{z}}}{2i\alpha}\left[\frac{z+\Delta_{0}\tau_{x}}{C_{0}}-\text{sgn}\,(x)\tau_{z}\sigma_{z}\right]$
$\displaystyle+\frac{e^{iC_{++}|x|/\alpha}}{2i\alpha}\left[\frac{z-(B-\Delta_{0})\sigma_{x}}{C_{++}}+\text{sgn}\,(x)\tau_{z}\sigma_{z}\right]\frac{1+\tau_{x}\sigma_{x}}{2}$
$\displaystyle+\frac{e^{iC_{+-}|x|/\alpha}}{2i\alpha}\left[\frac{z-(B+\Delta_{0})\sigma_{x}}{C_{+-}}+\text{sgn}\,(x)\tau_{z}\sigma_{z}\right]\frac{1-\tau_{x}\sigma_{x}}{2},$
(276)

where

$\displaystyle C_{0}$ $\displaystyle=z\sqrt{1-\frac{\Delta_{0}^{2}}{z^{2}}},$
(277) $\displaystyle C_{\tau\sigma}$ $\displaystyle=z\sqrt{1-\frac{(\sigma
B-\tau\Delta_{0})^{2}}{z^{2}}},\quad\tau,\sigma=\pm.$ (278)

Next, we establish the quantities relevant for the calculation of the
$d$-matrix:

$\displaystyle g(0^{\pm})$
$\displaystyle=g(0)=\frac{1}{2i\alpha}\sum_{\tau=\pm}\sum_{\sigma=\pm}\bar{g}_{\tau\sigma}\frac{1+\tau\tau_{x}}{2}\frac{1+\sigma\sigma_{x}}{2},$
(279) $\displaystyle\frac{1}{m}g^{\prime}(0^{\pm})$
$\displaystyle=\pm\tau_{z}-\frac{z+\tau_{x}\Delta_{0}}{C_{0}}\sigma_{z},$
(280)

where

$\displaystyle\bar{g}_{\tau\sigma}=\frac{z+\tau\Delta_{0}}{C_{0}}+\frac{z-\sigma
B+\tau\Delta_{0}}{C_{\tau\sigma}}.$ (281)

Inserting these results at $z=\omega+i0^{+}$ into (135), after a simple albeit
tedious calculation we recover the equations for the sub-gap bound states at
the absent contact potential ($V_{0}=0$)

$\displaystyle\cos^{2}\frac{\varphi}{2}$
$\displaystyle=\frac{\omega^{2}+\Delta_{0}^{2}-B^{2}+C_{++}C_{+-}}{\omega^{2}+\Delta_{0}^{2}-B^{2}-C_{++}C_{+-}},$
(282) $\displaystyle\cos^{2}\frac{\varphi}{2}$
$\displaystyle=\frac{\omega^{2}}{\Delta_{0}^{2}},$ (283)

previously reported in Ref. [Murthy  _et al._ , 2020].

We note in passing that for an isolated Majorana wire a boundary Green’s
function of the form discussed in Appendix A.2 suggests the following equation
for the bound states

$\displaystyle\det
g(0)=\frac{1}{(2\alpha)^{4}}\prod_{\tau,\sigma}\bar{g}_{\tau\sigma}=0,$ (284)

which factorizes into the four equations $\bar{g}_{\tau\sigma}=0$ for
$\tau,\sigma=\pm$. The Majorana zero mode $\omega=0$ appears at $B>\Delta_{0}$
as a solution satisfying the two equations $\bar{g}_{++}=\bar{g}_{--}=0$.

###  G.3 Long junction

In the case of the long junction we, in addition, need the bulk position space
Green’s function of the normal central region. Setting $\Delta_{0}=0$ in (255)
we represent it as

$\displaystyle g_{k}^{C}=\sum_{\tau=\pm}\frac{\text{adj}\,a_{k}^{(\tau)}}{\det
a_{k}^{(\tau)}}\frac{1+\tau\tau_{z}}{2}=\sum_{\tau=\pm}g_{k}^{(\tau)}\frac{1+\tau\tau_{z}}{2},$
(285)

where

$\displaystyle\text{adj}\,a_{k}^{(\tau)}$ $\displaystyle=z-\tau
h_{k}^{(0)}+\tau\alpha k\sigma_{z}-B\sigma_{x},$ (286) $\displaystyle\det
a_{k}^{(\tau)}$ $\displaystyle=[z-\tau
h_{k}^{(0)}]^{2}-\alpha^{2}k^{2}-B^{2}.$ (287)

It follows

$\displaystyle
g^{C}(x)=\sum_{\tau=\pm}g^{(\tau)}(x)\frac{1+\tau\tau_{z}}{2},\quad
g^{(\tau)}(x)=\int\frac{dk}{2\pi}e^{ikx}g_{k}^{(\tau)}.$ (288)

By the virtue of (147) we find for $z=\omega+i0^{+}$

$\displaystyle g^{(\tau)}(x)=-\frac{(2m\alpha)^{2}}{k^{2}_{1,+}-k_{2,+}^{2}}$
(289)
$\displaystyle\times\sum_{s=1}^{2}(-1)^{s+1}\frac{e^{ik_{s,+}|x|}}{2i\alpha}\left[\frac{\omega-\tau
h_{k_{s,+}}^{(0)}-B\sigma_{x}}{\alpha
k_{s,+}}+\text{sgn}(x)\tau\sigma_{z}\right],$

where $k_{s,+}=-k_{s,-}$ are the roots of the bi-quadratic equation $\det
a_{k}^{(\tau)}=0$ with $\text{Im}\,k_{s,+}>0$.

We consequently find

$\displaystyle
g^{(\tau)}(0^{\pm})=-\frac{(2m\alpha)^{2}}{k^{2}_{1,+}-k_{2,+}^{2}}$ (290)
$\displaystyle\times\sum_{s=1}^{2}\frac{(-1)^{s+1}}{2i\alpha}\frac{\omega-\tau
h_{k_{s,+}}^{(0)}-B\sigma_{x}}{\alpha k_{s,+}}$

and

$\displaystyle g^{(\tau)\,^{\prime}}(0^{\pm})=m\tau\left[\pm
1-\frac{2m\alpha\sigma_{z}}{k_{1,+}+k_{2,+}}\right],$ (291)

from which the $\mathcal{L}_{R/L\rightarrow C}$ functions in Eqs. (79) and
(80) may be deduced. Hence the $d$ matrix for the model of Section III.3 may
be constructed on the basis of Eqs. (74), (77), and (78).

It is worth noting the useful formula

$\displaystyle\partial_{\varphi}d=\frac{i}{4}\left(\begin{array}[]{cc}\left[\tau_{z},\mathcal{L}_{L}\right]&0\\\
0&\left[\tau_{z},\mathcal{L}_{R}\right]\end{array}\right),$ (294)

which is applicable for the JC calculation by means of Eqs. (95)-(98).

## References

  * Datta (1997) S. Datta,  _Electronic transport in mesoscopic systems_ (Cambridge university press, 1997).
  * Freericks (2016) J. K. Freericks,  _Transport in multilayered nanostructures: the dynamical mean-field theory approach_ (World Scientific, 2016).
  * Groth  _et al._ (2014) C. W. Groth, M. Wimmer, A. R. Akhmerov, and X. Waintal, New J. Phys. 16, 063065 (2014).
  * Ando (1991) T. Ando, Phys. Rev. B 44, 8017 (1991).
  * Cheng and Lutchyn (2012) M. Cheng and R. M. Lutchyn, Phys. Rev. B 86, 134522 (2012).
  * Yokoyama  _et al._ (2014) T. Yokoyama, M. Eto, and Y. V. Nazarov, Phys. Rev. B 89, 195407 (2014).
  * Landauer (1957) R. Landauer, IBM J. Res. Dev. 1, 223 (1957).
  * Landauer (1970) R. Landauer, Philos. Mag. 21, 863 (1970).
  * Caroli  _et al._ (1971) C. Caroli, R. Combescot, D. Lederer, P. Nozieres, and D. Saint-James, J. Phys. C: Solid State Phys. 4, 2598 (1971).
  * Khomyakov  _et al._ (2005) P. A. Khomyakov, G. Brocks, V. Karpan, M. Zwierzycki, and P. J. Kelly, Phys. Rev. B 72, 035450 (2005).
  * Fisher and Lee (1981) D. S. Fisher and P. A. Lee, Phys. Rev. B 23, 6851 (1981).
  * Boumrar  _et al._ (2020) H. Boumrar, M. Hamidi, H. Zenia, and S. Lounis, J. Phys. Condens. Matter 32, 355302 (2020).
  * Blonder  _et al._ (1982) G. E. Blonder, M. Tinkham, and T. M. Klapwijk, Phys. Rev. B 25, 4515 (1982).
  * Beenakker (1991) C. W. J. Beenakker, Phys. Rev. Lett. 67, 3836 (1991).
  * Andreev (1964) A. Andreev, Sov. Phys. JETP 19, 1228 (1964).
  * Andreev (1966) A. Andreev, Sov. Phys. JETP 22, 18 (1966).
  * Arnold (1978) G. B. Arnold, Phys. Rev. B 18, 1076 (1978).
  * Arnold (1985) G. B. Arnold, J. Low Temp. Phys. 59, 143 (1985).
  * Feuchtwang (1974a) T. E. Feuchtwang, Phys. Rev. B 10, 4121 (1974a).
  * Feuchtwang (1974b) T. E. Feuchtwang, Phys. Rev. B 10, 4135 (1974b).
  * Feuchtwang (1975) T. E. Feuchtwang, Phys. Rev. B 12, 3979 (1975).
  * Golubov  _et al._ (2004) A. A. Golubov, M. Y. Kupriyanov, and E. Il’ichev, Rev. Mod. Phys. 76, 411 (2004).
  * Eschrig (2018) M. Eschrig, Philosophical Transactions of the Royal Society A 376, 20150149 (2018).
  * García-Moliner and Rubio (1969) F. García-Moliner and J. Rubio, J. Phys. C: Solid State Phys. 2, 1789 (1969).
  * García-Moliner  _et al._ (1969) F. García-Moliner, V. Heine, and J. Rubio, J. Phys. C: Solid State Phys. 2, 1797 (1969).
  * García-Moliner and Velasco (1991) F. García-Moliner and V. Velasco, Phys. Rep. 200, 83 (1991).
  * García-Moliner and Velasco (1994) F. García-Moliner and V. Velasco, Surf. Sci. 299-300, 332 (1994).
  * Rodriguez-Coppola  _et al._ (1990) H. Rodriguez-Coppola, V. R. Velasco, F. Garcia-Moliner, and R. Pérez-Alvarez, Phys. Scr. 42, 115 (1990).
  * Inglesfield (1971) J. E. Inglesfield, J. Phys. C: Solid State Phys. 4, L14 (1971).
  * Inglesfield (1981) J. E. Inglesfield, J. Phys. C: Solid State Phys. 14, 3795 (1981).
  * Wortmann  _et al._ (2002) D. Wortmann, H. Ishida, and S. Blügel, Phys. Rev. B 65, 165103 (2002).
  * van Heck  _et al._ (2017) B. van Heck, J. I. Väyrynen, and L. I. Glazman, Phys. Rev. B 96, 075404 (2017).
  * Murthy  _et al._ (2020) C. Murthy, V. D. Kurilovich, P. D. Kurilovich, B. van Heck, L. I. Glazman, and C. Nayak, Phys. Rev. B 101, 224501 (2020).
  * Lutchyn  _et al._ (2010) R. M. Lutchyn, J. D. Sau, and S. Das Sarma, Phys. Rev. Lett. 105, 077001 (2010).
  * Oreg  _et al._ (2010) Y. Oreg, G. Refael, and F. von Oppen, Phys. Rev. Lett. 105, 177002 (2010).
  * Akkermans  _et al._ (1991) E. Akkermans, A. Auerbach, J. E. Avron, and B. Shapiro, Phys. Rev. Lett. 66, 76 (1991).
  * Souma and Suzuki (2002) S. Souma and A. Suzuki, Phys. Rev. B 65, 115307 (2002).
  * Note (1) Note that the heterostructural Green’s function $G(x,x^{\prime})$ is a highly non-trivial object, containing information about all possible multi-barrier scattering phenomena and interface-localized bound states.
  * (39) K. Piasotski, M. Pletyukhov, and A. Shnirman, In preparation .
  * Lévy-Leblond (1995) J.-M. Lévy-Leblond, Phys. Rev. A 52, 1845 (1995).
  * Piasotski  _et al._ (2022) K. Piasotski, N. Müller, D. M. Kennes, H. Schoeller, and M. Pletyukhov, Phys. Rev. B 106, 165405 (2022).
  * Note (2) In real experiments, the phase difference is typically varied by closing the system into a circular geometry far away from the tunneling region, and varying the magnetic flux threading the resulting ring.
  * Abrikosov (2017) A. A. Abrikosov,  _Fundamentals of the Theory of Metals_ (Courier Dover Publications, 2017).
  * Chamon  _et al._ (2010) C. Chamon, R. Jackiw, Y. Nishida, S.-Y. Pi, and L. Santos, Phys. Rev. B 81, 224515 (2010).
  * Haberkorn  _et al._ (1978) W. Haberkorn, H. Knauer, and J. Richter, Phys. Status Solidi (a) 47, K161 (1978).
  * Žonda  _et al._ (2015) M. Žonda, V. Pokorný, V. Janiš, and T. Novotný, Sci. Rep. 5, 8821 (2015).
  * Altland and Simons (2010) A. Altland and B. D. Simons,  _Condensed matter field theory_ (Cambridge university press, 2010).
  * Meden (2019) V. Meden, J. Phys. Condens. Matter 31, 163001 (2019).
  * Robinson  _et al._ (2007) J. W. A. Robinson, S. Piano, G. Burnell, C. Bell, and M. G. Blamire, Phys. Rev. B 76, 094522 (2007).
  * Chtchelkatchev  _et al._ (2001) N. M. Chtchelkatchev, W. Belzig, Y. V. Nazarov, and C. Bruder, JETP Letters 74, 323 (2001).
  * Note (3) Note that one can envisage the situation in which the two bands touch one another at exactly zero energy. In this case, however, the states around zero energy are expected to disperse quadratically, as opposed to the linear dispersion found in the topological regime.
  * Matute-Cañadas  _et al._ (2022) F. J. Matute-Cañadas, C. Metzger, S. Park, L. Tosi, P. Krogstrup, J. Nygård, M. F. Goffman, C. Urbina, H. Pothier, and A. L. Yeyati, Phys. Rev. Lett. 128, 197702 (2022).
  * Harris  _et al._ (2020) C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, and T. E. Oliphant, Nature 585, 357 (2020).
  * Wimmer (2009) M. Wimmer,  _Quantum transport in nanostructures: From computational concepts to spintronics in graphene and magnetic tunnel junctions_ , Ph.D. thesis, University of Regensburg (2009).
  * Piasotski (2023) K. Piasotski,  _On the universal properties of excess charges in one dimension_ , Ph.D. thesis, RWTH Aachen University (2023).
  * Zazunov  _et al._ (2018) A. Zazunov, A. Iks, M. Alvarado, A. L. Yeyati, and R. Egger, Beilstein J. Nanotechnol. 9, 1659 (2018).
  * Alvarado  _et al._ (2020) M. Alvarado, A. Iks, A. Zazunov, R. Egger, and A. L. Yeyati, Phys. Rev. B 101, 094511 (2020).
  * Kaladzhyan and Bena (2019) V. Kaladzhyan and C. Bena, Phys. Rev. B 100, 081106 (2019).
  * Pinon  _et al._ (2020) S. Pinon, V. Kaladzhyan, and C. Bena, Phys. Rev. B 101, 115405 (2020).
  * Komnik and Heinze (2017) A. Komnik and S. Heinze, Phys. Rev. B 96, 155103 (2017).
  * Zazunov  _et al._ (2017) A. Zazunov, R. Egger, M. Alvarado, and A. L. Yeyati, Phys. Rev. B 96, 024516 (2017).
  * Peng  _et al._ (2017) Y. Peng, Y. Bao, and F. von Oppen, Phys. Rev. B 95, 235143 (2017).
  * Zazunov  _et al._ (2016) A. Zazunov, R. Egger, and A. Levy Yeyati, Phys. Rev. B 94, 014502 (2016).
  * Arrachea  _et al._ (2009) L. Arrachea, G. S. Lozano, and A. A. Aligia, Phys. Rev. B 80, 014425 (2009).
  * Müller  _et al._ (2021) N. Müller, K. Piasotski, D. M. Kennes, H. Schoeller, and M. Pletyukhov, Phys. Rev. B 104, 125447 (2021).


# From Continuity to Editability: Inverting GANs with Consecutive Images

Yangyang Xu1, Yong Du2, Wenpeng Xiao1, Xuemiao Xu1,3,4,5∗ and Shengfeng He1,5
1South China University of Technology 2Ocean University of China
3Guangdong Provincial Key Lab of Computational Intelligence and Cyberspace
Information
4State Key Laboratory of Subtropical Building Science
5Ministry of Education Key Laboratory of Big Data and Intelligent Robot
Corresponding authors ({xuemx,hesfe}@scut.edu.cn).

###### Abstract

Existing GAN inversion methods are stuck in a paradox that the inverted codes
can either achieve high-fidelity reconstruction, or retain the editing
capability. Having only one of them clearly cannot realize real image editing.
In this paper, we resolve this paradox by introducing consecutive images
(e.g., video frames or the same person with different poses) into the
inversion process. The rationale behind our solution is that the continuity of
consecutive images leads to inherent editable directions. This inborn property
is used for two unique purposes: 1) regularizing the joint inversion process,
such that each of the inverted code is semantically accessible from one of the
other and fastened in a editable domain; 2) enforcing inter-image coherence,
such that the fidelity of each inverted code can be maximized with the
complement of other images. Extensive experiments demonstrate that our
alternative significantly outperforms state-of-the-art methods in terms of
reconstruction fidelity and editability on both the real image dataset and
synthesis dataset. Furthermore, our method provides the first support of
video-based GAN inversion, and an interesting application of unsupervised
semantic transfer from consecutive images. Source code can be found at:
https://github.com/cnnlstm/InvertingGANs_with_ConsecutiveImgs.

##  1 Introduction

Generative Adversarial Networks (GANs) [6, 10, 11] has demonstrated versatile
image editing capability, especially by discovering the spontaneously learned
interpretable directions that can manipulate corresponding image attributes
[18, 9, 20, 5, 23]. Concretely, given a random latent code $w$, image editing
can be achieved by pushing the latent vector along a specific semantic
direction (e.g., age, gender):

$I{{}^{\prime}}=G(w+\alpha\times\mathop{\mathbb{n}}^{\rightarrow}),$ (1)

where $I{{}^{\prime}}$ is the edited image, $G$ is the generator, $\alpha$ is
a scaling factor, and $\mathbb{\mathop{n}\limits^{\rightarrow}}$ is the
interpretable direction.

As a consequence, recent attempts [1, 2, 19, 25] aim to migrate this power to
real image editing by inverting an image to the latent code $w$. There are two
keys for this task, i) whether the inverted code can faithfully reconstruct
the original input, and, ii) whether the pre-acquired semantic directions can
be successfully applied. However, existing methods seem to stuck in a paradox,
as achieving one end will inevitably sacrifice the other. As shown in Fig.
LABEL:fig:teaser, I2S, I2S++, and pSp [1, 2, 19] concentrate only on obtaining
faithful reconstruction, but the inverted codes show limited editability. In
contrast, latent codes obtained from In-domain inversion [25] (Fig.
LABEL:fig:teaserd) are regularized in the semantically meaningful domain at
the expense of fidelity. We argue that balancing these two factors solely
based on a single image is extremely challenging, as there is no indicator to
shed light on the editable domain in the latent space, preventing the
optimization from obtaining a perfect balance between two factors.

In this paper, we resolve the above problem by introducing consecutive images,
which can be either a video segment or the same person with different poses,
to form a joint optimization process. The rational behind our alternative
solution is that the _continuity_ brought by consecutive images can be used as
an indicator to constrain the _editability_. In particular, to ensure each of
the inverted latent codes is editable, we jointly optimize multiple latent
codes by enforcing each of them is semantically accessible from one of the
other code with a simple linear combination. In addition, we further explore
the sequential continuity for _fidelity_ , by injecting multi-source
supervision that common regions of the reconstructed images should be
consistent in all the consecutive images. We establish dense correspondences
between input images, and then apply the obtained correspondences to warp each
of the reconstructions to the neighbors for a consistent and coherence
measurement.

To evaluate the proposed method, we construct a real video-based dataset
RAVDESS-12, and another consecutive images dataset synthesized by manipulating
attributes from the generated images of StyleGAN [11]. Extensive experiments
demonstrate the superior performance of our solution over existing methods in
terms of editability and reconstruction fidelity. Furthermore, our method is
capable to perform various applications, e.g., video-based GAN inversion,
unsupervised semantic transfer, and image morphing.

In summary, our contributions are three-fold:

  * •

We propose an alternative GAN inversion method for consecutive images, and
delve deep into the continuity of consecutive images for GAN inversion.

  * •

We tailor two novel constraints, one is the mutually accessible constraint
that formulates consecutive images inversion as a linear combination process
in the latent space to ensure editability, and the inversion consistency
constraint that works in the RGB space to guarantee reconstruction fidelity,
by measuring reconstruction consistency across images.

  * •

We demonstrate optimal performances in terms of editability and reconstruction
fidelity, and we support various new applications like video-based GAN
inversion and unsupervised semantic transfer.

##  2 Related Work

Image Editing via Latent Space Exploration. Generative models show great
potential in synthesizing versatile images by taking random latent codes as
inputs. Recent works show that the latent space of a pre-trained GAN encodes
rich semantic directions. Varying the latent codes with a specific direction
can edit the image with the target attribute. In particular, Radford et al.
[18] observe that there are directions in the latent space corresponding to
adding smiles or glasses on the faces. Ganalyze et al. [5] explore the
memorability direction in the latent space by a fixed assessor. Jahanian et
al. [9] study the steerability of GANs to fit some image transformations. Shen
et al. [20] explore the semantic boundary in the latent space of the binary
attributes. Voynov et al. [23] discover the semantic directions hidden in the
latent space by an unsupervised model-agnostic procedure. Varying the latent
codes with such directions can manipulate the corresponding attributes of the
output images. It is natural to transferring these techniques on real image
editing. Before that, it is required to invert a real image back to the latent
code.

GAN Inversion. To realize real image editing, GAN inversion methods are
proposed to inference a latent code of an input image based on the pre-trained
GAN [17, 26, 1, 4, 16]. These methods can be categorized into two classes,
optimization-based and encoder-based. The former individually optimizes the
latent code for a specific image, with a concentration on pixel-wise
reconstruction [1, 2, 4, 14]. However, ensuring reconstruction fidelity cannot
guarantee the output latent code is editable by the learned directions. On the
other hand, encoder-based methods train a general encoder that maps real
images to latent codes [25, 19]. Especially, In-Domain GAN inversion [25]
combines the learned encoder with an optimization process to align the encoder
with the semantic knowledge of the generator. However, existing methods do not
resolve the problem of editable domain in the latent space, thus they cannot
achieve a perfect balance between editability and fidelity. We aim to solve
this problem from a novel view of considering multiple consecutive images.

##  3 Method

Figure 1: The pipeline of the proposed consecutive images based GAN inversion.
$G$ is a pretrained generator of StyleGAN and $F$ is a pretrained FlowNet for
calculating the bidirectional optical flow. The upper part shows the Mutually
Accessible Constraint. Given consecutive images as input, we enforce each of
them is semantically accessible from one of the other codes with a simple
linear combination. Both $w_{1}$ and
$\mathbb{\mathop{n}\limits^{\rightarrow}}_{k}$ are the optimization targets.
The bottom shows the Inversion Consistency Constraint in the RGB space. Note
here we only show the forward flow among $I_{b}$ and $I_{b+1}$ when
calculating the $\mathcal{L}_{ICC}$ for simplicity, meanwhile,
$\mathcal{L}_{C}$ and $\mathcal{L}_{P}$ are also omitted.

###  3.1 Overview

The editability of the latent code and the fidelity of the reconstruction are
the two vital factors that affect the performance of GAN inversion. To satisfy
both sides, we exploit the continuity brought by consecutive images which
depict a same subject with different variations. The pipeline of our approach
is illustrated in Fig. 1. Given a sequence of consecutive images as input, the
proposed method aims to seek their optimal latent codes in the latent space,
and then feed them into a pretrained and fixed generator for reconstruction.
In particular, i) we define a linear combination mechanism among consecutive
images, which would facilitate the editability of the latent codes via a joint
optimization with semantic directions, and ii) we establish a consistency
constraint in the RGB space between the warped results of the reconstructed
images and their corresponding originals, promoting the fidelity of the
reconstruction. Note that we choose the generator of StyleGAN [10] as the
pretrained one in our model.

###  3.2 Consecutive Images Based GAN Inversion

Mutually Accessible Constraint. For each image in an input set, it may
gradually change into the others, just as shown in Fig. 1, the mouth opens
progressively. Or it can vary to the others in a drastic way, such as the same
person at very different poses. In either case, given the first image $I_{1}$,
it can be intuitively assumed that the latent codes of the other images are
linear combinations of that of the starting image along with a specific
semantic direction $\mathbb{\mathop{n}\limits^{\rightarrow}}$ (e.g.,
expression, pose). Then, the latent code $w_{1+k}$ of the image $I_{1+k}$ can
be formulated as follows:

$w_{1+k}=w_{1}+\alpha_{k}\times\mathop{\mathbb{n}}^{\rightarrow},\quad
k=1,2,\cdots,T-1,$ (2)

where $T$ denotes the total number of images. However, the assumption is too
strong since consecutive images are likely to vary from one to the others in
different attributes. On the other hand, the specific semantic direction
$\mathbb{\mathop{n}\limits^{\rightarrow}}$ can be predefined [20, 23, 22] ,
but the semantically equivalent input images are required. To cope with images
with arbitrary semantic variations, we regard the direction as one of our
optimization targets. Note that the scaling factors $\alpha_{k}$-s are also
learnable, and therefore, we absorb them into the direction and reformulate
Eq.(2) as the proposed mutually accessible constraint, which is as follows:

$w_{1+K}=w_{1}+\sum_{k=1}^{K}\mathop{\mathbb{n}_{k}}^{\rightarrow}.$ (3)

In this way, we can figure out the latent codes of all the images via a joint
optimization of $w_{1}$ and $\mathbb{\mathop{n}\limits^{\rightarrow}}_{k}$-s.
Such a simple linear combination mechanism can promote the editability of the
latent codes. The main reasons are that, i) each of the inverted latent codes
can be regarded as an edited code with respect to one of the others, and ii)
if the images vary in a specific semantic direction, the scales of variations
could be learned adaptively, and more importantly, iii) it is able to deal
with the variations of different attributes among consecutive images.
Moreover, the learned $\mathbb{\mathop{n}\limits^{\rightarrow}}_{k}$-s have
the potential to be transferred to other latent codes as predefined semantic
directions.

Inversion Consistency Constraint. Once we have the latent codes, we feed them
into the generator $G(\cdot)$ to reconstruct consecutive images. For a certain
base image $I_{b}$, its reconstruction $O_{b}$ with regard to the latent code
$w_{b}$ is calculated by

$O_{b}=G(w_{b}),\quad b=1,2,\cdots,T.$ (4)

In order to ensure a fidelity of the reconstructions, we particularly consider
an inversion consistency between common regions of the reconstructed images
and the input consecutive images. Specifically, we tailor an inversion
consistency constraint loss based on bidirectional flows in the RGB space. As
shown in the bottom of Fig. 1, first, the forward flows $f_{b\Rightarrow b+k}$
between base image $I_{b}$ and the other images $I_{b+k}$ can be calculated by
a pretrained FlowNet2 [8] $F(\cdot)$, which is formulated as follows:

$f_{b\Rightarrow b+k}=F(I_{b},I_{b+k}),$ (5)

where $k\in\mathbb{Z}\cap[1-b,T-b]-\\{0\\}$. Then we warp $I_{b}$ with this
flow to form the warped images $\hat{I}_{b+k}$, which can be presented as
follows:

$\hat{I}_{b+k}=warp(I_{b},f_{b\Rightarrow b+k}).$ (6)

Also, we have the warped results $\hat{O}_{b+k}$ of the recovered images
$O_{b}$, which is presented as follows:

$\hat{O}_{b+k}=warp(O_{b},f_{b\Rightarrow b+k}).$ (7)

Since consecutive images describe a same subject, an inherent relationship
should be existed among the generated warpings $\\{\hat{I}_{b+k}\\}$ for each
base image $I_{b}$. And this relationship should be transferred to the other
warpings $\\{\hat{O}_{b+k}\\}$. In the same way, we can calculate the backward
flow $f_{b+k\Rightarrow b}$ between $I_{b+k}$ and $I_{b}$, and the
corresponding warpings $\\{\hat{I}_{b}^{b+k}\\}$, $\\{\hat{O}_{b}^{b+k}\\}$.
By iterating over all the values of $b$ and $k$, we inject multi-source
supervision from the warpings of input images to confine the reconstructions,
and the proposed inversion consistency constraint loss ${\cal L}_{ICC}$ is
given by

$\mathcal{L}_{ICC}=\sum_{b}\sum_{k}(\|\hat{I}_{b}^{b+k}-\hat{O}_{b}^{b+k}\|_{2}+\|\hat{I}_{b+k}-\hat{O}_{b+k}\|_{2}),$
(8)

where $\|\cdot\|_{2}$ denotes a pixel-wise $L_{2}$ distance.

Moreover, we consider to maintain a pixel-wise consistency between the input
images and its corresponding recoveries. A pixel-wise consistency loss ${\cal
L}_{C}$ is therefore introduced in our objective, that is

$\mathcal{L}_{C}=\sum_{m\in q\cup\\{0\\}}\|O_{1+m}-I_{1+m}\|_{2},$ (9)

To guarantee a fine visual perception of the recontructions, we also utilize a
perceptual loss ${\cal L}_{P}$ which is formulated as follows:

$\displaystyle\mathcal{L}_{P}=\frac{1}{4}\sum_{j=1}^{4}\sum_{m\in
q\cup\\{0\\}}\|\Phi_{V}^{j}(I_{1+m})-\Phi_{V}^{j}(O_{1+m})\|_{2},$ (10)

where $\Phi_{V}^{j}(\cdot)$ denotes the $j$th layer of a pretrained VGG-16
network, and we follow Abdal et al. [1] to select the features produced by the
conv1_1, conv1_2, conv3_2 and conv4_2 layers of VGG-16 for modeling the loss.

Finally, the whole objective function $\cal L$ is defined as follows:

${\cal L}=\lambda_{1}{\cal L}_{ICC}+\lambda_{2}{\cal L}_{C}+\lambda_{3}{\cal
L}_{P},$ (11)

where $\lambda$s denote the balance factors. Then the latent code $w_{1}$ and
the directions $\mathbb{\mathop{n}\limits^{\rightarrow}}_{k}$ can be optimized
by

$\\{w^{*}_{1},\mathbb{\mathop{n}\limits^{\rightarrow}}_{k}^{*}\\}=\mathop{\arg\min}_{\\{w_{1},\mathbb{\mathop{n}\limits^{\rightarrow}}_{k}\\}}\lambda_{1}\mathcal{L}_{ICC}+\lambda_{2}\mathcal{L}_{C}+\lambda_{3}\mathcal{L}_{P}.$
(12)

Note that we follow [1] that initialize $w_{1}$ with the mean latent vector of
$\mathcal{W+}$ space. And the direction
$\mathbb{\mathop{n}\limits^{\rightarrow}}_{k}$-s are initially set to zero and
updated during optimization.

##  4 Experiments

Table 1: Comparisons with existing GAN inversion methods on image reconstruction with four metrics on two datasets. $\downarrow$ denotes the lower the better and the best results are marked in bold. Methods Metrics | RAVDESS-12 Dataset | Synthesized Dataset
---|---|---
| NIQE$\downarrow$ | FID$\downarrow$ | LPIPS$\downarrow$ | MSE$\downarrow$($\times$e-3) | NIQE$\downarrow$ | FID$\downarrow$ | LPIPS$\downarrow$ | MSE$\downarrow$($\times$e-3)
I2S [1] | 3.770 | 16.284 | 0.162 | 8.791 | 3.374 | 48.909 | 0.271 | 35.011
pSp [19] | 3.668 | 29.701 | 0.202 | 22.337 | 3.910 | 84.355 | 0.391 | 46.244
InD [25] | 3.765 | 18.135 | 0.193 | 9.963 | 3.152 | 42.773 | 0.352 | 44.645
Ours | 3.596 | 13.136 | 0.148 | 5.972 | 2.807 | 37.225 | 0.250 | 24.395
I2S++ [2] | 3.358 | 0.320 | 0.003 | 0.174 | 2.644 | 2.967 | 0.014 | 1.458
Ours++ | 3.352 | 0.311 | 0.003 | 0.165 | 2.597 | 2.897 | 0.014 | 1.432

###  4.1 Implementation Details

We implement the proposed method in Pytorch on a PC with an Nvidia GeForce RTX
3090. We utilize the generator of StyleGAN [10] pre-trained on the FFHQ
dataset [10] with the resolution of $1024\times 1024$. The latent codes and
semantic directions are optimized using Adam optimizer [12]. We follow [1]
that use 5000 gradient descent steps with a learning rate of 0.01,
${\beta}_{1}=0.9$, ${\beta}_{2}=0.999$, and ${\epsilon}=1e^{-8}$. We
empirically set the balancing weights in Eq. (12) as $\lambda_{1}=1$,
$\lambda_{2}=1$ and $\lambda_{3}=1$. And we set $T=5$ in Eq. (4), which
indicates 5 consecutive images are contained in each input sequence.

###  4.2 Experimental Settings

Datasets.  We first conduct our experiments on the RAVDESS dataset [13] with
real videos. The original RAVDESS dataset contains 2,452 videos with 24
subjects speaking and singing with various semantic expressions. We select 12
videos of them for evaluation, resulting in 1,454 frames, we name this dataset
as RAVDESS-12 Dataset. Since there are no ground truth latent codes for real
images, we cannot subjectively evaluate the inverted code and its editability
in the latent space. On the other hand, it is demonstrated that the learned
semantic directions work very well in generated images. As a result, we
construct a synthesized dataset, containing 1000 samples that were randomly
generated by StyleGAN. For each sample, we vary its latent code with 5 random
combinations of the $\alpha$ value (ranging from -3 to 3) and semantic
direction (acquired from InterfaceGAN [20]), producing 5000 images. We record
the latent codes of the original samples, the corresponding editing
specifications, and the edited latent codes for evaluating the editability.
GAN inversion methods will invert the original samples and edit them to target
attributes for comparisons.

Competitors. We mainly compare with four GAN inversion methods: Image2StyleGAN
(I2S) [1], Image2StyleGAN++ (I2S++) [2], In-domain Inversion (InD) [25], and
pSp network [19]. All the methods are inverted to the same $\mathcal{W+}$
latent space of StyleGAN, applying the same directions for editing. It is
worth noting that I2S++ introduces the additional noise space $\mathcal{N}$
for small details recovery. A main problem is that, the inverted two latent
codes in the $\mathcal{W+}$ and $\mathcal{N}$ spaces are highly coupled, but
the learned semantic directions are optimized in $\mathcal{W+}$ only. Applying
them changes the $\mathcal{W+}$ space latent code but leaves the noise vector
unchanged, these unpaired vectors yield “ghosting” artifacts after editing
(see Fig. 3). As a result, we mainly use it for reconstruction comparisons,
and we also extend our method to include the noise space, named as Ours++.

(a) Original

(b) I2S

(c) pSp

(d) InD

(e) Ours

\- - - - - - - - - - - - - - - - - - - - - - - -

(f) I2S++

(g) Ours++

Figure 2: Qualitative comparison on image reconstruction. Compared with the
works that optimized in the $\mathcal{W+}$ space (left part), our method can
reconstruct the most faithful appearances. Involving the $\mathcal{N}$ space
largely improves reconstruction (right part), but Ours++ show better color
preservation than I2S++ (second row).

Evaluation Metrics.  For the quantitative comparisons, we use four metrics,
Naturalness Image Quality Evaluator (NIQE) [15], Fréchet Inception Distance
(FID) [7], Learned Perceptual Image Patch Similarity (LPIPS) [24], and pixel-
wise Mean Square Error (MSE), for evaluating the reconstruction fidelity.
Especially, FID computes the Wasserstein-2 distance between the distribution
of input and output images. NIQE evaluates the quality perceived by a human,
which is a completely blind assessment with no request for the GT image. Since
there is no GT for the semantic editing task on the real RAVDESS-12 Dataset,
we use FID and NIQE to evaluate real image editing results.

###  4.3 Evaluation on Image Reconstruction

Quantitative Evaluation. We first evaluate the reconstruction fidelity of the
inverted codes. Quantitative comparisons are shown in Tab. 1. We can see that
our method significantly outperforms three editable GAN inversion methods
(upper part) on both the real dataset and the synthesized one. Especially for
the pixel-wise difference metric MSE, we largely improve the state-of-the-art
by 31%. This indicates that the proposed joint optimization successfully
incorporates complementary information from neighboring images. Besides, by
involving the noise space $\mathcal{N}$, I2S++ and Ours++ achieve the most
faithful reconstruction compared with other methods. Thanks to the inter-image
coherence, we further push the reconstruction record a bit.

Qualitative Evaluation. Qualitative comparisons are shown in Fig. 10. We can
see that the Image2StyleGAN cannot recover the image color correctly.
Meanwhile, the pSp and InD cannot recover the finest facial details of the
original images (see teeth in the first row). Compared with the above three
methods, our method can reconstruct faithful appearance details.
Unsurprisingly, I2S++ recovers the finest details among all competitors. That
mainly because of their optimization performed in the $\mathcal{N}$ space
encodes high-frequency information. We also depict our results optimized in
$~{}\mathcal{N}$ space, and we preserve the original color better than I2S++
(see the second row).

(a) Input

(b) I2S

(c) Ind

(d) pSp

(e) Ours

\- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

(f) I2S++

(g) Ours++

+ Age – + Pose –

Figure 3: Qualitative comparison on semantic editing with Pose and Age attributes on the real RAVDESS-12 Dataset. Images marked by red boxes are the reconstructed targets, and images in the middle row of each example are the inversion results. We can tell that our method can support more favorable semantic editing. Table 2: Quantitative evaluation on real image manipulation with two blind metrics on the RAVDESS-12 Dataset. $\downarrow$ denotes the lower the better and the best results are marked in bold. Metrics | I2S [1] | pSp [19] | InD [25] | Ours
---|---|---|---|---
NIQE $\downarrow$ | 3.776 | 5.242 | 3.693 | 3.254
FID $\downarrow$ | 21.609 | 30.128 | 19.271 | 15.482

+ Age – + Smile –

(a) GT

(b) I2S

(c) Ind

(d) pSp

(e) Ours

Figure 4: Qualitative comparison on semantic editing with Smile and Age
attributes on the Synthesized Dataset. It’s noticed that we force the input
sequence to contain different semantic changes from its corresponding editing
direction. the first sequence contains the semantic changes with “ _gender_ ”
for optimization and “ _smile_ ” for semantic edit, and the second is “ _pose_
” for optimization while “ _age_ ” for the edit testing. And our edited
results are more similar with the ground truths.

###  4.4 Evaluation on Image Editing

In this section, we evaluate our GAN inversion method on real image editing as
well as synthesized images. We conduct two editing tasks based on the inverted
latent codes, the first one is semantic manipulation and the second is image
morphing.

####  4.4.1 Semantic Manipulation

Semantic manipulation aims to edit a real image by varying its inverted codes
along with a specific semantic direction. We use five semantic directions
(i.e., gender, pose, smile, eyeglasses, and age) acquired by [20] in the
experiment.

Table 3: Quantitative evaluation on image manipulation with four metrics on the Synthesized Dataset. $\downarrow$ denotes the lower the better and the best results are marked in bold. Metrics | I2S [1] | pSp [19] | InD [25] | Ours
---|---|---|---|---
NIQE $\downarrow$ | 3.390 | 3.917 | 3.193 | 3.163
FID $\downarrow$ | 35.894 | 58.342 | 48.867 | 33.872
LPIPS $\downarrow$ | 0.399 | 0.452 | 0.424 | 0.347
MSE$\downarrow$($\times$e-3) | 89.671 | 126.642 | 101.563 | 70.224

Qualitative Evaluation. The qualitative comparisons on real data are shown in
Fig. 3. We can see that our manipulated faces have visually more plausible
results than those of the competitors. In particular, the manipulated results
gained by Image2StyleGAN [1] present noisy artifacts with pose changes and
glasses are entangled with the age attributes, revealing that the edited
latent codes are escaped from the editable domain. Similar situations can be
found in I2S++ [2]. The manipulated faces based on pSp [19] are almost
unchanged. This is because pSp focuses on learning a direct mapping from the
input to latent code, ignoring the editability. This problem is addressed by
In-domain inversion [25], but it also sacrifices the reconstruction quality.
In contrast, thanks to the jointly considered inherent editability constraint
between consecutive images, our inverted latent codes are more semantically
editable, leading to more disentangled manipulations. On the other hand, the
noise space optimization methods (right part of Fig. 3) show obvious noise
artifacts than the others with pose changes, this is because the pre-optimized
noise vector is not suitable for the edited latent vector $w$. However, Ours++
can disentangle gender from glasses better than the inverted vector from
I2S++.

To evaluate whether the obtained inversion can be edited by arbitrary
directions, we force the editing direction different from the semantic changes
contained the input sequence on the synthesized dataset. The qualitative
comparisons on the synthesized data are shown in Fig. 11. Similar to the
evaluation on real data, I2S produces obvious artifacts, pSp fails to edit the
results, and InD cannot preserve the original identity. Our manipulated
results are more similar with the GTs, which indicates that our inverted codes
are much closer with the GT latent codes and also inherit their editability.

Quantitative Evaluation. We present the quantitative comparisons in Tab. 2 and
Tab. 3. Our method achieves the best results on both the RAVDESS-12 Dataset
and the Synthesized Dataset. In particular, for the blind metric NIQE, our
edited results achieve 13.8% improvement over the state-of-the-art method,
which indicates that our editing is more visually plausible. Quantitative
results on the Synthesized Dataset can evaluate whether the inverted codes are
close enough with the GT code such that we can reuse their semantic
information. From two non-blind metrics LPIPS and MSE, we can see that our
edited results are very similar to the GTs. Thanks to our semantically
accessible regularization in the latent space, our inverted latent codes show
a strong editability compared with the competitors.

Ours InD pSp I2S

Inverted A$\hskip 8.53581pt\longleftarrow$ Morphing $\longrightarrow\hskip
8.53581pt$Inverted B

Figure 5: Qualitative comparison on image morphing task. We can see that our
result present a continuous process and the morphing faces are realistic.

####  4.4.2 Image Morphing

Image morphing aims to fuse two images semantically by interpolating their
latent codes. It is another way to evaluate whether the inverted codes indeed
lie in the latent space and reuse the semantic knowledge. For the high-quality
inverted codes, their interpolated results should also stay in the editable
domain and the semantic varies continuously. Qualitative comparisons are shown
in Fig. 5. We can see that the morphing results produced by Image2StyleGAN [1]
have noticeable artifacts. Meanwhile, the results produced by pSp [19] are
unrealistic with the unnatural hairs. In contrast, our method presents high-
quality results with a continuous morphing process. We also present the
quantitative evaluation on the morphing task in Tab. 4 and Tab. 5, we can see
that our inversion results outperform the other inversion methods on both the
real dataset and the synthesized one.

Table 4: Quantitative evaluation on image morphing with two blind metrics on the RAVDESS-12 Dataset. $\downarrow$ denotes the lower the better and the best results are marked in bold. Metrics | I2S [1] | pSp [19] | InD [25] | Ours
---|---|---|---|---
NIQE $\downarrow$ | 4.255 | 5.350 | 4.051 | 3.688
FID $\downarrow$ | 40.627 | 38.474 | 38.925 | 37.695
Table 5: Quantitative evaluation on image morphing with four metrics on the Synthesized Dataset. $\downarrow$ denotes the lower the better and the best results are marked in bold. Metrics | I2S [1] | pSp [19] | InD [25] | Ours
---|---|---|---|---
NIQE $\downarrow$ | 3.389 | 3.800 | 3.212 | 3.115
FID $\downarrow$ | 31.776 | 30.192 | 21.901 | 18.621
LPIPS $\downarrow$ | 0.472 | 0.467 | 0.469 | 0.402
MSE$\downarrow$($\times$e-3) | 141.432 | 121.834 | 125.674 | 98.354

Figure 6: Our unsupervisedly acquired direction
$\mathbb{\mathop{n}\limits^{\rightarrow}}$ from consecutive images can be used
for transferring the semantics. The first row is the input set that is
regarded as reference, and the images in red boxes are the target faces. We
can transfer the semantic changes of the reference to the target faces, even
with more than one attribute changed.

###  4.5 Semantic Transfer

As discussed in Sec. 3.2, both the latent code $w$ and the semantic direction
$\mathbb{\mathop{n}\limits^{\rightarrow}}$ can be unsupervisedly obtained
after inversion. Besides the latent code, our acquired direction
$\mathbb{\mathop{n}\limits^{\rightarrow}}$ represents the semantic changes of
the input images. Given the input images as reference, we can transfer its
semantic changes to the target faces.

The transfer results are shown in Fig. 6. We can see that the semantic
attributes of target faces are modified following the reference image set.
Note that there are more than one attribute has been changed in the reference.
For example, in the right example, the mouth and pose varies simultaneously
but we can still capture those changes. This shows that our acquired direction
is disentangled with the referenced and can be applied on other faces. Other
than existing supervised [20, 21] or unsupervised [22, 23] learning of
interpretable directions, this sheds light on a new exemplar-based learning of
semantic direction.

###  4.6 Ablation Study

In this section, we analyze the efficacy of our two components: mutually
accessible constraint (MAC) and inversion consistency constraint (ICC). Note
that without these two components, our method equals to the Image2StyleGAN
inversion and we set it as our baseline. By unplugging one of these two
constraints, we yield two variants of “$w/o$ MAC” and “$w/o$ ICC”. In this
case, $\mathcal{L}_{ICC}$ is removed and all the latent codes are optimized
simultaneously.

Table 6: Ablation study on image reconstruction with four metrics. $\downarrow$ denotes the lower the better and the best results are marked in bold. Variants | NIQE$\downarrow$ | FID$\downarrow$ | LPIPS$\downarrow$ | MSE$\downarrow$($\times$e-3)
---|---|---|---|---
Baseline | 3.770 | 16.284 | 0.162 | 8.791
$w/o$ MAC | 3.685 | 13.375 | 0.151 | 8.065
$w/o$ ICC | 3.765 | 14.791 | 0.160 | 8.508
Ours | 3.596 | 13.136 | 0.148 | 5.972
Table 7: Ablation study on semantic manipulation with two blind metrics. $\downarrow$ denotes the lower the better and the best results are marked in bold. Metrics | Baseline |  $w/o$ MAC |  $w/o$ ICC | Ours
---|---|---|---|---
NIQE$\downarrow$ | 3.776 | 3.659 | 3.398 | 3.254
FID$\downarrow$ | 21.609 | 16.121. | 17.274 | 15.482

We perform ablation study experiment on image reconstruction and semantic
manipulation tasks on the RAVDESS-12 Dataset. Quantitative comparisons of GAN
inversion are shown in Tab. 6. We can see that every variant outperforms the
baseline on all metrics. This indicates both two components contribute to the
GAN inversion performance. Meanwhile, variant ($w/o$ MAC) performs better than
the variant ($w/o$ ICC), this indicates that the inversion consistency brought
by consecutive images contributes more for the GAN inversion task. In Tab. 7
of semantic editing, we observe a different situation. We can see the variant
($w/o$ ICC) performs better than the variant ($w/o$ MAC), this reveals that
mutually accessible constraint confines the inverted latent codes to stay in
the editable domain. The above two evaluations show that our two constraints
work very well following our design principles.

+ Age –

(a) Baseline

(b) $w/o$ MAS

(c) $w/o$ ICS

(d) Ours

Figure 7: Ablation study on semantic editing with two variants and baseline by
editing the “age” attribute.

We show the results of different variants in Fig. 7 by changing the “age”
attribute. We can see that the baseline and variant ($w/o$ MAC) entangles with
glasses, showing that concentrating only on reconstruction fidelity lacks
editability of the inverted codes. In contrast, variant ($w/o$ ICC) and our
final result can successfully modify the “age” attribute, revealing the strong
regularization power of our designed mutually accessible constraint.

##  5 Conclusion

In this paper, we propose an alternative GAN inversion method for consecutive
images, we formulate consecutive images inversion as a linear combination
process in the latent space that ensures editability, and transfer the
reconstruction consistency across inputs in the RGB space to guarantee
reconstruction fidelity. The experiment results demonstrate the effectiveness
in terms of editability and reconstruction fidelity. Besides, we also support
various new applications like video-based GAN inversion and unsupervised
semantic transfer.

Acknowledgment: This project is supported by the Key-Area Research and
Development Program of Guangdong Province, China (2020B010165004,
2020B010166003, 2018B010107003); National Natural Science Foundation of China
(61972162, 61772206, U1611461, 61472145); Guangdong International Science and
Technology Cooperation Project (2021A0505030009); Guangdong Natural Science
Foundation (2021A1515012625); Guangzhou Basic and Applied Research Project
(202102021074); CCF-Tencent Open Research fund (CCF-Tencent RAGR20190112,
RAGR20210114); China Postdoctoral Science Foundation (2020M682240,
2021T140631) and Fundamental Research Funds for the Central Universities
(202113035).

## References

  * [1] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan: How to embed images into the stylegan latent space?  In ICCV, pages 4432–4441, 2019.
  * [2] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan++: How to edit the embedded images?  In CVPR, pages 8296–8305, 2020.
  * [3] Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka.  Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows.  ACM TOG, 40(3):1–21, 2021.
  * [4] Antonia Creswell and Anil Anthony Bharath.  Inverting the generator of a generative adversarial network.  IEEE TNNLS, 30(7):1967–1974, 2018.
  * [5] Lore Goetschalckx, Alex Andonian, Aude Oliva, and Phillip Isola.  Ganalyze: Toward visual definitions of cognitive image properties.  In ICCV, pages 5744–5753, 2019.
  * [6] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In NeurIPS, volume 27, 2014.
  * [7] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In NeurIPS, pages 6626–6637, 2017.
  * [8] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox.  Flownet 2.0: Evolution of optical flow estimation with deep networks.  In CVPR, pages 2462–2470, 2017.
  * [9] Ali Jahanian, Lucy Chai, and Phillip Isola.  On the “steerability” of generative adversarial networks.  2019\.
  * [10] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In CVPR, pages 4401–4410, 2019.
  * [11] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In CVPR, pages 8110–8119, 2020.
  * [12] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [13] Steven R Livingstone and Frank A Russo.  The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english.  PloS one, 13(5):e0196391, 2018.
  * [14] Fangchang Ma, Ulas Ayaz, and Sertac Karaman.  Invertibility of convolutional generative networks from partial measurements.  In NeurIPS, volume 31, 2018.
  * [15] Anish Mittal, Rajiv Soundararajan, and Alan C Bovik.  Making a “completely blind” image quality analyzer.  Signal Process Letter, 20(3):209–212, 2012.
  * [16] Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo.  Exploiting deep generative prior for versatile image restoration and manipulation.  In ECCV, pages 262–277, 2020.
  * [17] Guim Perarnau, Joost Van De Weijer, Bogdan Raducanu, and Jose M Álvarez.  Invertible conditional gans for image editing.  2016\.
  * [18] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  In ICLR, 2015.
  * [19] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or.  Encoding in style: a stylegan encoder for image-to-image translation.  In CVPR, pages 2287–2296, 2021.
  * [20] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou.  Interpreting the latent space of gans for semantic face editing.  In CVPR, pages 9243–9252, 2020.
  * [21] Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou.  Interfacegan: Interpreting the disentangled face representation learned by gans.  IEEE TPAMI, 2020.
  * [22] Yujun Shen and Bolei Zhou.  Closed-form factorization of latent semantics in gans.  In CVPR, pages 1532–1540, 2021.
  * [23] Andrey Voynov and Artem Babenko.  Unsupervised discovery of interpretable directions in the gan latent space.  In ICML, 2020.
  * [24] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, pages 586–595, 2018.
  * [25] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou.  In-domain gan inversion for real image editing.  In ECCV, 2020.
  * [26] Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros.  Generative visual manipulation on the natural image manifold.  In ECCV, pages 597–613, 2016.

##  6 Supplemental Results

Table 8: Quantitative evaluations on image reconstruction and semantic editing tasks on the nonlinear dataset. Metrics Methods | Image Reconstruction | Nonlinear Semantic Edit
---|---|---
| I2S | pSp | InD | Ours | I2S | pSp | InD | Ours
NIQE $\downarrow$ | 3.632 | 3.439 | 3.254 | 2.997 | 3.940 | 3.974 | 3.703 | 3.476
FID $\downarrow$ | 40.098 | 62.932 | 77.692 | 33.692 | 48.032 | 64.224 | 51.607 | 37.039
LPIPS $\downarrow$ | 0.252 | 0.323 | 0.414 | 0.238 | 0.489 | 0.522 | 0.471 | 0.403
MSE$\downarrow$($\times$e-3) | 30.767 | 79.878 | 83.294 | 25.192 | 87.361 | 118.285 | 99.623 | 69.449

To further demonstrate our method is not restricted with the linear-based
editing, we synthesize a new dataset under the nonlinear constraints by
StyleFlow [3]. It consists of 1,000 sequences resulting in 5,000 images, with
different semantic changes, such as pose, illumination, expression,
eyeglasses, gender, and age. We conduct an inversion experiment on it and the
results are shown in the Tab. 8, Fig. 8. Our method can obtain an accurate
reconstruction on this nonlinear dataset. Besides, we also conduct nonlinear
semantic editing task using the nonlinear StyleFlow [3], as shown in the right
parts of Tab. 8 and Fig. 9. Our results are more similar with the GT (see the
hair color of “ _age+_ ” in the right-bottom corner of Fig. 9). These results
prove that our method is not constrained by linear-editing assumption. That
because our method is an _optimization-based_ GAN inversion method that does
not rely on any attribute constraint for the input images, and the
optimization is image-specific _without training_ a general network.

(a) Original

(b) I2S

(c) pSp

(d) InD

(e) Ours

Figure 8: Qualitative comparison on image reconstruction on the nonlinear
dataset.

(a) GT

(b) I2S

(c) pSp

(d) InD

(e) Ours

– Age + + Gender –

Figure 9: Qualitative comparison on semantic edit on the nonlinear dataset.

We also give more qualitative comparison on image reconstitution on RAVDESS-12
Dataset and the linear-based Synthesized Dataset in Fig. 10. We can see that
our method can reconstruct the most faithful appearances by optimization
latent code in the $\mathcal{W+}$ space. Involving the $\mathcal{N}$ space
largely improves reconstruction quality and Ours++ can reconstruct the correct
colors.

The qualitative comparison on image editing task on Synthesized dataset can be
seen in Fig. 11. Our edited results are more similar with the ground truths
and show the pleasure appearances, which indicates that our inverted latent
codes are close enough with the GT codes.

(a) Original

(b) I2S

(c) pSp

(d) InD

(e) Ours

\- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

(f) I2S++

(g) Ours++

Figure 10: Qualitative comparison on image reconstruction.

+ Age – + Gender –

(a) GT

(b) I2S

(c) Ind

(d) pSp

(e) Ours

Figure 11: Qualitative comparison on semantic editing.


# Exploring ISAC Technology for UAV SAR Imaging

Stefano Moro, Francesco Linsalata, Marco Manzoni, Maurizio Magarini and
Stefano Tebaldini . Department of Electronics, Information and Bioengineering,
Politecnico di Milano, Italy

###### Abstract

This paper illustrates the potential of an Integrated Sensing and
Communication (ISAC) system, operating in the sub-6 GHz frequency range, for
Synthetic Aperture Radar (SAR) imaging via an Unmanned Aerial Vehicle (UAV)
employed as an aerial base station. The primary aim is to validate the
system’s ability to generate SAR imagery within the confines of modern
communication standards, including considerations like power limits, carrier
frequency, bandwidth, and other relevant parameters. The paper presents two
methods for processing the signal reflected by the scene. Additionally, we
analyze two key performance indicators for their respective fields, the Noise
Equivalent Sigma Zero (NESZ) and the Bit Error Rate (BER), using the QUAsi
Deterministic RadIo channel GenerAtor (QuaDRiGa), demonstrating the system’s
capability to image buried targets in challenging scenarios. The paper shows
simulated Impulse Response Functions (IRF) as possible pulse compression
techniques under different assumptions. An experimental campaign is conducted
to validate the proposed setup by producing a SAR image of the environment
captured using a UAV flying with a Software-Defined Radio (SDR) as a payload.

###### Index Terms:

UAV, SAR, SDR, Sensing and Communications for Emergencies

##  I Introduction

In recent times, a notable surge in technological progress has greatly
expanded the capabilities of Unmanned Aerial Vehicles (UAVs) across various
applications [1]. These advancements include enhancements in battery life,
augmented reliability, the development of advanced software for autonomous
navigation, and the utilization of lightweight materials [2]. In addition, the
reduced cost of drones has substantially improved the feasibility of deploying
UAV networks [3] and presented possible solutions for continuous operations by
coordinating fly and recharge operations [4].

Search and rescue operations are paramount among the many compelling
applications of UAVs. Their rapid deployment, ease of operation, and
adaptability make them indispensable in saving lives during emergencies [5].
Equipping drones with various sensors, including optical and thermal cameras,
LiDARs, and radars, prepares them for rescue missions of different natures.
Notably, radar stands out for its ability to send Electromagnetic (EM) waves
into the ground, enabling the detection of victims buried under debris or
snow, a critical capability after events like earthquakes or avalanches [6].
Moreover, this same capacity can also provide essential wireless connectivity
to rescuers, often unavailable in disaster-affected or remote areas [7].

The Integrated Sensing and Communication (ISAC) paradigm offers a seamless
solution for both problems. Here, radar sensing operations can share the
spectrum or even the same signal as the communication link. The literature
features numerous studies delving into various ISAC applications. For example,
in [5], several potential ISAC applications are explored from the perspective
of Content Service Providers (CSPs) in scenarios like Smart Home, Sensing as a
Service, Environmental monitoring, and more. The work in [8] delves into UAV
applications and optimizes waveform design for ISAC, while [3] provides a
comprehensive account of three UAV applications within a communications
network. Moreover, in [1], it details a thorough review of single and multi-
UAV ISAC, examining the pros and cons of drone swarms.

(a) Impulse response with a QPSK modulated sub-carriers.

(b) Impulse response with a 256-QAM modulated sub-carriers.

Figure 1: Range compression of one OFDM symbol with QPSK and 256-QAM modulated
sub-carriers. The red-dotted line highlights the side lobe suppression level,
persistent at -13dB

###  I-A Contributions

The main contributions can be summarized as follows:

  * •

Show the potentiality of the ISAC approach by applying it to the context of
UAV-based communication to generate SAR images, staying within the boundaries
of common standards [9, 10].

  * •

Produce simulated SAR images with the QUAsi Deterministic RadIo channel
GenerAtor (QuaDRiGa) [11] 5G-standard-compliant channel model and investigate
performance analyses measuring Bit Error Rate (BER) and Noise Equivalent Sigma
Zero (NESZ) of the system under various conditions, furnishing a numerical
assessment of the system performances. We present a scenario involving the
potential burial of one or more individuals beneath the snow, as might occur
after an avalanche.

  * •

Provide the first ISAC implementation of a communication waveform for
environmental imaging employing a UAV equipped with a standard sub-6 GHz link,
utilizing norm-compliant transmitted powers, a prescribed number of antennas,
and their respective directivity and gains.

  * •

Present the results of an experimental campaign to validate what is proposed
and show the UAV-based setup’s effectiveness while emulating an Orthogonal
Frequency Division Multiplexing (OFDM) transmission by convolving actual radar
chirp measurement onboard a UAV.

###  I-B Paper structure

Section II delineates the waveform characterization and the parameters used.
Subsequently, we present some principles of SAR imaging. In Section III, we
furnish a numerical assessment of the system performances in multiple
scenarios involving the burial of individuals beneath the snow. Section IV
highlights a simulator employing the standard-compliant QuaDRiGa channel
generator to perform SAR focusing and study the system’s Impulse Response
Function (IRF). In Section V, we culminate the paper by showcasing actual data
acquired by a UAV equipped with a Software-Defined Radar (SDR) payload.
Finally, Section VI summarizes the results and provides final remarks.

##  II Signal Modeling and SAR imaging

This section comprehensively explains the chosen communication signal model,
including standardized values for critical parameters. Furthermore, we present
an evaluation of two different pulse compression techniques and the main
algorithm used to generate a SAR image.

###  II-A OFDM signal model and parameters

The foundation of most digital communication protocols is OFDM [12], chosen
for its resilience to wireless channel impairments and versatile design in
both the time and frequency domains. The time domain’s most general OFDM
transmission structure is arranged into frames, subframes, and slots. All the
details of the internal definition of the OFDM frame can be found in [12, 13].

Regarding its physical layer parameters, which encompass aspects like
bandwidth, sub-carrier spacing, symbol duration, receiver noise figure, and
more, detailed information is provided in Sec. IV. These values are sourced
from contemporary literature and established standards [12, 10, 14].

To facilitate the reader, we report the definition of a single OFDM symbol as

$\displaystyle x(t)=\sum^{M-1}_{m=0}s(m)e^{j2\pi m\Delta ft}g(t)$ (1)

where $s(m)$ is the constellation symbol (QAM, QPSK, etc.), $\Delta f=1/T_{s}$
is the sub-carrier spacing defined by the inverse of the symbol interval
$T_{s}$, $M$ is the number of subcarriers, and $g(t)$ is the pulse shaping
filter. The actual transmitted waveform is generated by attaching the Cyclic
Prefix (CP), with a length of 6.57% of the OFDM symbol, as defined by the
standard.

(a)

(b)

Figure 2: In (2(a)), the BER and NESZ vs EIRP for different UAV heights at
fixed snow depth of 1 meter. In (2(b)), the BER and NESZ vs EIRP for different
snow depths. The nominal flying altitude of the UAV is 100 m. The extinction
ratio in the snow can be found in [15]

###  II-B Pulse compression of OFDM signals

The process chain toward obtaining a focused SAR image comprises two main
steps: the range and the azimuth compression. The former is commonly called
matched filtering in communication theory. Several studies are present in the
literature addressing this topic for OFDM signals [16, 17]. We present here
two possible range compression methods. The first employs the conventional
matched filter, while the second is Zero Forcing (ZF), a well-known approach
in digital communication that consists of an inversion of the frequency
response of the pulse.

####  II-B1 Matched filter

For the matched filter case, the received signal is multiplied in the
frequency domain by the complex conjugate of the transmitted one

$X_{\text{RC}}(f)=X^{*}(f)X_{\text{RX}}(f),$ (2)

where $X(f)$ and $X_{\text{RX}}(f)$ represent the frequency domain
representations of the transmitted and received signal, respectively. The
inverse Fourier Transform of $X_{\text{RC}}(f)$ leads to the range compressed
signal. With this procedure, the lack of spectrum equalization can lead to
possible performance degradation. With perfect equalization, we expect to have
an IRF, also defined as the auto-correlation of the pulse, that closely
resembles the ideal cardinal sine function. However, when each subcarrier has
a different channel gain, we have an unbalanced power spectrum. This will
generate higher sidelobes in the IRF that cause interference between targets
close in range. All this translates to a decreased quality in the generated
SAR image.

####  II-B2 Zero Forcing

One of the alternatives commonly used is the ZF receiver. With this approach,
we multiply the received signal by the inverse of the reference waveform

$X_{\text{RC}}(f)=\frac{1}{X(f)}X_{\text{RX}}(f).$ (3)

One possible drawback is the presence of very high gain values in the inverse
filter due to a very small amplitude in the reference signal. To mitigate
this, we can rewrite the inverse procedure as

$X_{\text{RC}}(f)=\frac{1}{X(f)+k}X_{\text{RX}}(f),$ (4)

where $k$ is computed based on the SNR and the channel response [18]. A
simulation has been carried out to compare these two approaches, and the
results using an OFDM waveform composed of 1024 sub-carriers are depicted in
Fig. 1. Figure 1(a) reports the comparison when symbols $s(m)$ are drawn from
a QPSK constellation. We can see that the ZF filter provides a better side-
lobes rejection than the matched filter, even in the case of a quasi-flat
spectrum, leading to an almost perfect sinc-like IRF.

In Fig. 1(b), we present instances of the pulse-compressed signal employing
256-QAM modulated sub-carriers. In this case, the noise floor after matched
filtering is higher than in the previous example due to power imbalances
between subcarriers. The ZF, on the other hand, leads once again to the
desired IRF without sidelobes and a much lower noise floor.

###  II-C SAR Image Formation

The second step of SAR imaging is called SAR focusing. We can employ the Time
Domain Back Projection (TDBP) algorithm, which is suitable even for non-linear
trajectories, like those typical of lightweight UAV systems. In TDBP, every
pulse (range-compressed OFDM symbol) is interpolated on a grid that represents
the area of interest and then re-phased, utilizing as a prior the platform’s
trajectory. All interpolated and re-phased pulses are then coherently summed,
leading to the final high-resolution SAR image.
For a specified pixel at position $x,y$, we can define the TDBP as:

$F(x,y)=\int
x_{\text{RC}}\left(t=\frac{R(\tau:x,y)}{c},\tau\right)e^{j\frac{4\pi}{\lambda}R(\tau:x,y)}d\tau$
(5)

where $F(x,y)$ is the 2D SAR image, $c$ the speed of light, $\lambda$ the
wavelength and $R(\tau:x,y)$ the nominal radar-to-pixel distance at instant
$\tau$ for the $x,y$ pixel. The range-compressed signal is
$x_{\text{RC}}(t,\tau)$. Notice that $t$ is the time instant within a single
pulse (also called fast-time in radar jargon), while $\tau$ is the time axis
sampled every PRI (also called slow-time).
The primary downside of this algorithm lies in its computational complexity.
For each pulse, it is necessary to compute the radar-to-pixel distances and
then make an interpolation followed by a complex multiplication. Assuming a
grid with dimensions of $N\times N$ pixels and an aperture of $N$ positions,
the number of operations is roughly $N^{3}$. More efficient algorithms, such
as the Fast Factorized Back Projection introduced in [19], can produce
equivalent images with fewer computations.

##  III Performance analysis

One of the most used measurements of the sensitivity of a radar system is the
so-called NESZ. It corresponds to the equivalent normalized radar cross
section ($\sigma_{0}$) that yields a unitary Signal-to-Noise Ratio (SNR) of
the resulting image [20]. In any radar system, the received power can be
modeled as:

$P_{\rm{RX}}=\frac{P_{\rm{TX}}G_{\rm{TX}}f(\theta)}{4\pi
R^{2}}\sigma\frac{A_{e}f(\theta)}{4\pi R^{2}}$ (6)

where $P_{\rm{TX}}$ is the transmitted power, $G_{\rm{TX}}$ is the antenna
gain of the transmitter, $f(\theta)$ is the antenna directivity function,
$\sigma$ is the target’s Radar Cross Section (RCS), $A_{e}$ is the equivalent
antenna area of the receiver and $R$ the radar-to-target distance. As known
from signal theory, after range compression (or matched filtering), the signal
power increases by a factor $T_{p}^{2}$, while the noise power is equal to
$P_{n}=N_{0}T_{p}$, leading to a SNR equal to:

$\rm{SNR}_{\text{RC}}=\frac{P_{\rm{RX}}T_{p}}{N_{0}}$ (7)

where $T_{p}$ is the pulse length, and $N_{0}$ is the noise power spectral
density. Similarly, after azimuth compression, we have the final focused image
where the $\rm{SNR}$ is increased once again by the number of processed pulses
$N_{\tau}$

$\rm{SNR}_{\text{foc}}=\rm{SNR}_{\text{RC}}N_{\tau}$ (8)

Finally, we can get the $\rm{NESZ}$ as:

$\rm{NESZ}=\frac{\sigma_{0}}{\rm{SNR_{\text{foc}}}}.$ (9)

where $\sigma_{0}=\sigma/(\rho_{\rm{rg}}\rho_{\rm{az}})$, in which
$\rho_{\rm{rg}}$ and $\rho_{\rm{az}}$ are the range and azimuth resolutions
respectively. We assessed the imaging capability of the aerial base station by
computing the NESZ, utilizing parameters that align with real-world
implementations of communication standards [12, 21, 10]. As reported in Table
I, the Equivalent Isotropic Radiated Power (EIRP) ranges from 0 to the maximum
prescribed limit of 23 dBm.

(a)

(b)

(c)

(d)

Figure 3: Simulated SAR images for different target depths under the snow at a
nominal UAV flying altitude of 150 m.

We used the widely known QuaDRiGa channel generator to accurately replicate
the specific scenario of interest [11]. QuaDRiGa generates channel
coefficients and their corresponding delays, employing a "statistical ray-
tracing model". Through this approach, we achieve three-dimensional
propagation, geometric polarization, continuous temporal evolution, and
spatially correlated slow and fast fading.

Two pivotal parameters for computing the NESZ involve the system’s symbol
duration and Pulse Repetition Frequency (PRF). We must first establish what we
define as a pulse and determine its duration to attain a communication system.
There are two possible approaches to consider. The symbol-based pulse defines
an equivalent radar pulse as a single OFDM symbol lasting
$8\leavevmode\nobreak\ \mu$ s. This translates to a PRF of 125 kHz and a duty
cycle of 100%. The unambiguous range, defined as the longest range to which a
transmitted pulse can travel out to and back again between consecutive pulses,
is calculated as $R_{\text{amb}}=c/(2\leavevmode\nobreak\ \text{PRF})=1.2$ km.
The second approach, the frame-based pulse, takes an entire frame consisting
of $N$ OFDM symbols as the equivalent radar pulse. This increases the pulse
length by a factor of $N$ and consequently extends the unambiguous range.
Additionally, a frame-based pulse offers greater flexibility by allowing the
transmitter to incorporate non-transmission intervals between frames, which is
not viable in a symbol-based pulse. We can easily note that both approaches
yield the same NESZ. In the symbol-based approach, the symbol duration is
shorter, thus resulting in a slight SNR gain after range compression. However,
the PRF increases, leading to more pulses integrated with azimuth compression.
In contrast, the frame-based approach exhibits the opposite: the single pulse
is $N$ times longer, resulting in a much higher gain after pulse compression
but with a lower PRF. Consequently, the SNR and the NESZ remain equivalent for
both approaches.

TABLE I: Communication standard parameters

Parameter | Value
---|---
Bandwidth $B$ | 40 MHz
$M_{\rm FFT}$ | 64
Subcarrier spacing $\Delta f$ |  $120$ KHz
Number of subcarrier $M$ | 52
Data symbol duration $T$ | 8.33 $\mu$s
$M_{\rm CP}$ | 12
Noise Figure | 7 dB
$f_{c}$ | 5.9 GHz
$EIRP_{max}$ | 23 dBm
$G_{\rm tmax}$ | 10 dB
QuaDRiGa NTN scenario | QuaDRiGa_NTN_Rural_NLOS
QuaDRiGa standard scenario | 5G-ALLSTAR_Rural_LOS

(a)

(b)

(c)

Figure 4: In (4(a)), the UAV is equipped with the SDR board enclosed
underneath it. In (4(b)), the optical image of the scene acquired by a camera
onboard the flying UAV with a detail of the S-band Corner Reflector. In
(4(c)), a radar image generated using TDBP processing of the OFDM signal
superimposed on an optical satellite view. The red dotted line indicates the
drone’s trajectory and the black circle highlights a corner reflector.

Our simulation also considered the air-to-ground communication link’s Bit
Error Rate (BER). Figure 2(a) evaluates both metrics (NESZ and BER) for
varying EIRP and UAV nominal heights. The graph’s straight line indicates the
NESZ shows a linear dependency on the EIRP, while the nominal height
introduces a consistent shift to the NESZ curves, resulting in lower received
power for longer distances.

Furthermore, we aimed to assess the system’s performance in more challenging
environments. One such scenario, where a UAV-based ISAC system would prove
invaluable, involves situations where one or more individuals are trapped
beneath the snow, for example, after an avalanche. In this context, the base
station on the UAV can establish a crucial communication link for the search
and rescue team while simultaneously conducting sensing operations beneath the
snow. In this scenario, the BER measurement proves the feasibility of
establishing a communication link with a buried user.

Figure 2(b) illustrates the NESZ and BER evaluation with the drone flying 100
meters above the scene, varying the depth of the trapped individuals from 0.5
to 2 m. The power attenuation experienced by the signal traversing through dry
snow is calculated according to the model outlined in [15], and it is added to
the propagation attenuation provided by the QuaDRiGa channel simulator. All
other parameters remain consistent with those in Fig. 2(a). Our results show
that subtle targets can be detected for shallow snow depths and reasonable
transmitted power. However, the performance degrades rapidly as the snow depth
approaches 2 m.

##  IV Simulation results

This section presents the outcomes of a simulation conducted with the same
parameters utilized in the NESZ calculation detailed in Sec. III and listed in
Table I. To emulate a realistic scenario, we set the transmitted power within
the allowable limit of the standard, and the receiver noise figure to a value
of a Commerical-Off-The-Shelf (COTS) hardware. We employed the QPSK modulation
scheme within this simulator to generate OFDM symbols. Our simulated scenarios
were constructed, featuring a point scatterer set at an off-nadir angle of
45∘. This target emulates the presence of a buried person, and in ideal
conditions, we expect to generate an image representing a perfect
bidimensional cardinal sine. It is characterized by a Radar Cross Section
(RCS) of $\sigma=1\leavevmode\nobreak\ \text{m}^{2}$. Depending on the
specific simulation, we positioned the target on the terrain’s surface or
buried beneath the snow. This variability in target placement allowed us to
explore the system’s responses to diverse environmental conditions, with
target depths spanning from 0.5 to 2 meters.

In Figure 3(a), we can see a target sharply focused, with the IRF of the
system when the target is located out of the snow. As anticipated, the IRF
recalls the bi-dimensional cardinal sine function we would expect by a simple
point scatterer, which is typical of a SAR imaging system. In Figures 3(b),
3(c), and 3(d), the target is submerged beneath 0.5, 1, and 2 m of snow,
respectively. It is evident that the ticker the level of snow present between
the target and the drone, the more the SNR and NESZ degrade. This, of course,
makes target detection a challenging activity.

##  V Experimental activity results

In this section, we present the outcomes of our experimental campaign. We
established a configuration where a stationary transmitter, mounted on a 4 m
pole, illuminated an area, and a receiver was fixed to a flying drone at an
altitude of 40 meters. In radar jargon, this setup is defined as bistatic
because the transmitter and receiver are not co-located. Both transceivers
were equipped with the USRP B210 Software Defined Radio (SDR), each
complemented by a 6 dBi patch antenna. The UAV employed for this survey was
the DJI Matrice 600 Pro, visible in Fig. 4(a), and the scene of interest is
depicted in Fig. 4(b).

During this measurement campaign, the transmitter sends a radar waveform,
employing the standard chirp function defined as

$g^{\text{chirp}}(t)=\mathrm{rect}\left(\frac{t}{T_{p}}\right)e^{j\pi\alpha
t^{2}},$ (10)

where $\alpha=B/T_{p}$, $B$ the total bandwidth, $T_{p}$ the pulse length, and
the receiver collects the scene’s reflections. Even in this setup, we can
emulate an OFDM transmission starting from the chirp data. For each echo, we
must convolve the OFDM pulse with the chirp-based range compressed signal as

$x_{RC}^{\text{OFDM}}(t)=x_{RC}^{\text{chirp}}(t)*g^{\text{OFDM}}(t).$ (11)

With this approach, we are employing the range-compressed data matrix as a
continuous channel definition of our scene. Afterward, we can range-compress
the signal again with a matched or ZF filter as detailed in Sec. II-B. An
important step is to adapt the PRF of the SDR to the communication’s protocol
specifications. Finally, we can generate the image shown in Fig. 4(c) using
the TDBP algorithm. As expected, it appears as a line in the focused SAR image
because all those pixels have the same bistatic range. The encircled target is
the corner reflector installed on the field, as shown in Fig. 4(b).
Additionally, we observe the reflection of a small woodland area nearby.

##  VI Conclusion

This research bridges digital communication and sensing, opening avenues for
innovative applications by delving into the imaging capabilities of a UAV-
based ISAC system. The investigation starts with defining the transmitted OFDM
signal and providing realistic, standard-compliant values for crucial
parameters like transmitted power, bandwidth, PRF, number of sub-carriers, and
antenna gains. Two distinct pulse compression methods for OFDM signals are
introduced: the traditional matched filtering and the zero forcing technique.
The paper also thoroughly analyzes the performances of the setup, studying the
NESZ and BER for a base station operating under challenging conditions. This
assessment is carried out by varying the UAV flying altitudes and the snow
depth. The results yield promising results with NESZ and BER below -20 dB and
$10^{-4}$, respectively, affirming the system’s capability to detect and
communicate with endangered users. Finally, the study provides compelling
evidence from simulated and experimental data affirming the UAV-based ISAC
system’s capability to image its environment while accurately adhering to
established industry standards.

## Acknowledgment

We are glad to acknowledge that the European Union partially supported this
work under the Italian National Recovery and Resilience Plan (NRRP) of
NextGenerationEU, partnership on “Telecommunications of the Future”
(PE00000001 - program “RESTART”) CUP: D43C22003080001, Structural Project S 13
ISaCAGE.

## References

  * [1] K. Meng, Q. Wu, J. Xu, W. Chen, Z. Feng, R. Schober, and A. L. Swindlehurst, “UAV-enabled integrated sensing and communication: Opportunities and challenges,” pp. 1–9, conference Name: IEEE Wireless Communications.
  * [2] E. Petritoli, F. Leccese, and L. Ciani, “Reliability assessment of UAV systems,” in _2017 IEEE International Workshop on Metrology for AeroSpace (MetroAeroSpace)_. Padua, Italy: IEEE, Jun. 2017, pp. 266–270.
  * [3] Y. Zeng, R. Zhang, and T. J. Lim, “Wireless communications with unmanned aerial vehicles: opportunities and challenges,” vol. 54, no. 5, pp. 36–42, conference Name: IEEE Communications Magazine.
  * [4] A. Trotta, M. Di Felice, K. R. Chowdhury, and L. Bononi, “Fly and recharge: Achieving persistent coverage using small unmanned aerial vehicles (suavs),” in _2017 IEEE ICC_. IEEE, 2017, pp. 1–7.
  * [5] Y. Cui, F. Liu, X. Jing, and J. Mu, “Integrating sensing and communications for ubiquitous IoT: Applications, trends, and challenges,” vol. 35, no. 5, pp. 158–167, conference Name: IEEE Network.
  * [6] A. Grathwohl, M. Stelzig, J. Kanz, P. Fenske, A. Benedikter, C. Knill, I. Ullmann, I. Hajnsek, A. Moreira, G. Krieger, M. Vossiek, and C. Waldschmidt, “Taking a look beneath the surface: Multicopter UAV-based ground-penetrating imaging radars,” vol. 23, no. 10, pp. 32–46, conference Name: IEEE Microwave Magazine.
  * [7] F. Linsalata, A. Albanese, V. Sciancalepore, F. Roveda, M. Magarini, and X. Costa-Perez, “Otfs-superimposed prach-aided localization for uav safety applications,” in _2021 IEEE GLOBECOM_ , 2021, pp. 1–6.
  * [8] D. K. Pin Tan, J. He, Y. Li, A. Bayesteh, Y. Chen, P. Zhu, and W. Tong, “Integrated sensing and communication in 6g: Motivations, use cases, requirements, challenges and future directions,” in _2021 1st IEEE International Online Symposium on JC &S_, pp. 1–6.
  * [9] 3rd Generation Partnership Project. (2022) 3GPP Release 18. [Online]. Available: https://www.3gpp.org/release18
  * [10] J. B. Kenney, “Dedicated short-range communications (dsrc) standards in the united states,” _Proceedings of the IEEE_ , vol. 99, no. 7, pp. 1162–1182, 2011.
  * [11] S. Jaeckel, L. Raschkowski, K. Börner, and L. Thiele, “Quadriga: A 3-d multi-cell channel model with time evolution for enabling virtual field trials,” _IEEE transactions on antennas and propagation_ , vol. 62, no. 6, pp. 3242–3256, 2014.
  * [12] M. H. C. Garcia, A. Molina-Galan, M. Boban, J. Gozalvez, B. Coll-Perales, T. Şahin, and A. Kousaridas, “A tutorial on 5G NR V2X communications,” _IEEE Communications Surveys & Tutorials_, vol. 23, no. 3, pp. 1972–2026, 2021.
  * [13] 3GPP, “NR; physical channels and modulation (release 15),” 3rd Generation Partnership Project (3GPP), Technical Specification (TS) 38.211, Jan. 2020, version 15.8.0.
  * [14] M. N. Ahangar, Q. Z. Ahmed, F. A. Khan, and M. Hafeez, “A survey of autonomous vehicles: Enabling communication technologies and challenges,” _Sensors_ , vol. 21, no. 3, p. 706, 2021.
  * [15] M. Tiuri, A. Sihvola, E. Nyfors, and M. Hallikaiken, “The complex dielectric constant of snow at microwave frequencies,” _IEEE Journal of Oceanic Engineering_ , vol. 9, no. 5, pp. 377–382, Dec. 1984.
  * [16] J. T. Rodriguez, F. Colone, and P. Lombardo, “Supervised Reciprocal Filter for OFDM Radar Signal Processing,” _IEEE Transactions on Aerospace and Electronic Systems_ , vol. 59, no. 4, pp. 3871–3889, Aug. 2023.
  * [17] ——, “Experimental evaluation of Supervised Reciprocal Filter Strategies for OFDM-radar signal processing,” in _2023 IEEE RadarConf23_. IEEE, 2023, pp. 1–6.
  * [18] J. Mark and W. Zhuang, “Wireless communications and networking".” Prentice Hall, 2003.
  * [19] L. M. Ulander, H. Hellsten, and G. Stenstrom, “Synthetic-aperture radar processing using fast factorized back-projection,” _IEEE Transactions on Aerospace and electronic systems_ , vol. 39, no. 3, pp. 760–776, 2003.
  * [20] N. Miranda, P. J. Meadows, A. Pilgrim, R. Piantanida, A. Recchia, D. Giudici, D. Small, and A. Schubert, “Sentinel-1b preliminary results obtained during the orbit acquisition phase [work in progress],” _Procedia Computer Science_ , vol. 100, pp. 1313–1318, 2016.
  * [21] 3GPP TS 38.211, “NR; Physical channels and modulation,” 2018.