# High-velocity blue-shifted Fe XXV He$\alpha$ line during a superflare of the
RS CVn-type star IM Peg

Shun Inoue Department of Physics, Kyoto University, Kitashirakawa-Oiwake-cho,
Sakyo-ku, Kyoto, 606-8502, Japan<EMAIL_ADDRESS>Wataru Buz
Iwakiri International Center for Hadron Astrophysics, Chiba University, Inage-
ku, Chiba, 263-8522 Japan  Teruaki Enoto Department of Physics, Kyoto
University, Kitashirakawa-Oiwake-cho, Sakyo-ku, Kyoto, 606-8502, Japan  RIKEN
Cluster for Pioneering Research, 2-1 Hirosawa, Wako, Saitama, 351-0198, Japan
Hiroyuki Uchida Department of Physics, Kyoto University, Kitashirakawa-Oiwake-
cho, Sakyo-ku, Kyoto, 606-8502, Japan  Miki Kurihara Department of Astronomy,
Graduate School of Science, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku,
Tokyo 113-0033, Japan  Japan Aerospace Exploration Agency, Institute of Space
and Astronautical Science, Chuo-ku, Sagamihara, Kanagawa 252-5210, Japan
Masahiro Tsujimoto Japan Aerospace Exploration Agency, Institute of Space and
Astronautical Science, Chuo-ku, Sagamihara, Kanagawa 252-5210, Japan  Yuta
Notsu Laboratory for Atmospheric and Space Physics, University of Colorado
Boulder, 3665 Discovery Drive, Boulder, CO 80303, USA  National Solar
Observatory, 3665 Discovery Drive, Boulder, CO 80303, USA  Kenji Hamaguchi
Astrophysics Science Division, NASA’s Goddard Space Flight Center, Greenbelt,
MD 20771, USA  Department of Physics, University of Maryland, Baltimore
County, Baltimore, MD, USA  Keith Gendreau Astrophysics Science Division,
NASA’s Goddard Space Flight Center, Greenbelt, MD 20771, USA  Zaven
Arzoumanian  Astrophysics Science Division, NASA’s Goddard Space Flight
Center, Greenbelt, MD 20771, USA  Takeshi Go Tsuru Department of Physics,
Kyoto University, Kitashirakawa-Oiwake-cho, Sakyo-ku, Kyoto, 606-8502, Japan

###### Abstract

Monitor of All-sky X-ray Image (MAXI) detected a superflare, releasing
$5\times 10^{37}$ erg in 2$-$10 keV, of the RS CVn-type star IM Peg at 10:41
UT on 2023 July 23 with its Gas Slit Camera (GSC; 2$-$30 keV). We conducted
X-ray follow-up observations of the superflare with Neutron Star Interior
Composition ExploreR (NICER; 0.2$-$12 keV) starting at 16:52 UT on July 23
until 06:00 UT on August 2. NICER X-ray spectra clearly showed emission lines
of the Fe XXV He$\alpha$ and Fe XXVI Ly$\alpha$ for $\sim 1.5$ days since the
MAXI detection. The Fe XXV He$\alpha$ line was blue-shifted with its maximum
Doppler velocity reaching $-2200\pm 600$ $\mathrm{km\>s^{-1}}$, suggesting an
upward-moving plasma during the flare, such as a coronal mass ejection (CME)
and/or chromospheric evaporation. This is the first case that the Fe XXV
He$\alpha$ line is blue-shifted during a stellar flare and its velocity
overwhelmingly exceeds the escape velocity of the star ($-230$
$\mathrm{km\>s^{-1}}$). One hour before the most pronounced blueshift
detection, a signature of reheating the flare plasma was observed. We discuss
the origin of the blueshift, a CME or high-velocity chromospheric evaporation.

X-rays: stars — stars: coronae — stars: activity — stars: flare — stars: mass-
loss

††facilities: NICER (Gendreau et al., 2016)††software: astropy (Astropy
Collaboration et al., 2013, 2018), HEAsoft, Xspec (Arnaud, 1996), PyXspec
(Gordon & Arnaud, 2021), PyAtomDB (Foster & Heuer, 2020)

##  1 Introduction

Stellar magnetic activity, such as flares and coronal mass ejections (CMEs),
and their effects on exoplanets have recently attracted wide attention (e.g.,
Airapetian et al., 2020). One particular observational advance over the past
decade is the increase of measurements of prominence eruptions, the initial
interval of stellar CMEs. Optical spectroscopic observations revealed
“blueshifts” or “blue asymmetries” of chromospheric lines, which are
attributed to prominence eruptions (Vida et al., 2016; Flores Soriano &
Strassmeier, 2017; Fuhrmeister et al., 2018; Honda et al., 2018; Vida et al.,
2019; Muheki et al., 2020a, b; Koller et al., 2021; Maehara et al., 2021; Lu
et al., 2022; Namekata et al., 2021; Inoue et al., 2023; Notsu et al., 2024;
Namekata et al., 2024; Inoue et al., 2024). However, the information of the
chromospheric line is limited in the initial interval of mass ejection (Sinha
et al., 2019). Optical observations alone cannot determine whether the erupted
prominence eventually develops into a CME, escaping from the star. Thus, it is
essential to observe X-ray lines, which have the information of the motion of
CMEs.

Figure 1: (a) 64 s binned $0.3-4$ keV count rate (cps;
$\mathrm{counts\>s^{-1}}$) during all NICER observations of IM Peg. The time
zero of MJD 60148.5 corresponds to 12:00 UT 2023 July 23. The one standard
deviation statistical error bars are smaller than the symbol size. The navy
vertical dashdot line indicates the time when MAXI detected the flare
($\sim-0.05$ day). (b) Enlarged light curve during the first two ObsIDs
6203900101 (blue) and 6203900102 (green). Interval numbers in our spectral
analysis are shown above.  Figure 2: Background-subtracted $0.3-9$ keV NICER
spectra of IM Peg duing the interval a0$-$a4 (panel a, ObsID 6203900101) and
b0$-$b4 (panel b, ObsID 6203900102). The best-fit model using the three-
temperature CIE components (vapec) are shown by the solid black line, composed
of 92 MK (red dashed), 15 MK (green dashdot), and 0.92 MK (blue dotted) for
panel a and 64 MK (red dashed), 13 MK (green dashdot), and 0.92 MK (blue
dotted) for panel b. The navy solid lines show the background spectra
calculated with the 3C50 model. The right upper inset panels show the enlarged
spectra around the iron emission lines. The lower panels are the ratio of the
observed data relative to the best-fit model.

So far, there are only two X-ray studies of detecting the blueshifts caused by
the plasma flow during a stellar flare (Argiroffi et al., 2019; Chen et al.,
2022). Both observations were conducted by High-Energy Transmission Grating
(HETG: Canizares et al., 2005) on Chandra X-ray Observatory (Weisskopf et al.,
2000). The stars of which Argiroffi et al. (2019) and Chen et al. (2022)
reported blueshifts were the G type star HR9024 and M type star EV Lac,
respectively. The reported velocity of blueshifts is $\sim-100$
$\mathrm{km\>s^{-1}}$, below the escape velocity at the star surface. Both
Argiroffi et al. (2019) and Chen et al. (2022) discussed the formation
temperatures of the lines and discussed whether the blueshifts are attributed
to a CME or chromospheric evaporation (Fisher et al., 1985). However, since
these HETG studies investigated only the lines formed at temperatures below
$20$ MK ($\sim 1.7$ keV), it was not known whether high-temperature ($>50$ MK)
lines such as the Fe XXV He$\alpha$ and Fe XXVI Ly$\alpha$ are blueshifted
during flares. On the solar flares such high-temperature ($>50$ MK) plasma is
thought to be formed above the top of a flare loop, and is related to the
particle acceleration (Caspi, 2010).

Large flares with its energy exceeding $10^{35}\>\mathrm{erg}$ from
magnetically active RS CVn-type stars have been observed in both X-ray and
visible light (e.g., Tsuru et al., 1989; Tsuboi et al., 2016; Sasaki et al.,
2021; Inoue et al., 2023). Inoue et al. (2023) reported a prominence eruption
during a flare releasing $7\times 10^{35}$ erg in the RS CVn type star V1355
Orionis, of which outward velocity was measured to be $1600$
$\mathrm{km\>s^{-1}}$ by $\mathrm{H\alpha}$ observations. On the other hand,
no blueshifted X-ray lines have been reported despite many X-ray observations
of RS CVn-type stars (e.g., Testa et al., 2004). The good photon statistics of
large flares from RS CVn-type stars could allow us to statistically
investigate a blueshift of the Fe XXV He$\alpha$ line.

In this letter, we report the X-ray observation of a superflare from the RS
CVn-type star IM Peg with Neutron Star Interior Composition ExploreR (NICER;
Gendreau et al., 2016). The large effective area of NICER (600
$\mathrm{cm}^{2}$ at 6 keV; Gendreau et al., 2012; Arzoumanian et al., 2014)
allows us to conduct time-resolved spectral analysis during flares. We
describe observation and data reduction (Section 2), results and discussion
(Section 3), and conclusion (Section 4). In this paper, the error ranges
indicate 90 % confidence level unless otherwise indicated.

Figure 3:  (a) The interval b1 spectrum fitted with a phenomenological model
(powerlaw+gauss+gauss). The vertical green and blue dashdot lines indicate the
line center of He$\alpha$ and Ly$\alpha$ emission lines at the rest frame in
the CIE model, respectively. The green and blue shaded areas show the $90\%$
error range of the line center of the two gaussians. (b) Theoretically
expected CIE spectrum during the interval b1 at the electron temperature of
$T_{e}^{\mathrm{b1}}\sim 76$ MK calculated with PyAtomDB. (c) Same spectrum as
panel (b) folded by the NICER response function.  Table 1: Best-fit spectral
parameters of the phenomenological model during the interval b1
Phenomenonological model (powerlaw+gauss1+gauss2)
---
powerlaw | $\Gamma$ | $2.21\pm 0.18$
$K^{\mathrm{powerlaw}}$ ($\mathrm{keV^{-1}\>cm^{-2}\>s^{-1}}$) | $0.28\pm 0.07$
gauss1 |  $E_{l}^{\mathrm{gauss1}}$ (keV) | $6.74\pm 0.01$
$\sigma$ (keV) |  $0.00$ (fix)
$K^{\mathrm{gauss1}}$ ($10^{-3}\>\mathrm{cm^{-2}\>s^{-1}}$) | $1.25\pm 0.19$
gauss2 |  $E_{l}^{\mathrm{gauss2}}$ (keV) | $6.96\pm 0.08$
$\sigma$ (keV) |  $0.00$ (fix)
$K^{\mathrm{gauss2}}$ ($10^{-3}\>\mathrm{cm^{-2}\>s^{-1}}$) | $0.29\pm 0.14$
$\chi^{2}$ (d.o.f.) | 49 (72)
Null hyp. prob. | 0.98

##  2 Observation and Data Reduction

IM Peg is a well-known active RS CVn-type star (e.g., Buzasi et al., 1987), of
which basic parameters are summarised in Table 1 of Zellem et al. (2010),
e.g., the stellar mass and radius of this K-type star are $M=1.8M_{\odot}$,
and $R=13.3R_{\odot}$, respectively, where $M_{\odot}$ and $R_{\odot}$ are the
mass and radius of the Sun, respectively. The corresponding escape velocity at
the stellar surface is $v_{e}=\sqrt{2GM/R}=230\>\mathrm{km\>s^{-1}}$. This
long-period RS CVn binary consists of K2 III + G4V stars with orbital period
of 24.6 d. The K2 III star has synchronized rotation and is heavily spotted
and the likely source of the superflare.

Monitor of All-sky X-ray Image (MAXI; Matsuoka et al., 2009) detected a flare
from IM Peg at 10:41 UT on 2023 July 23. This flare was classified as a
superflare by the radiated energy of $5\times 10^{37}$ erg in 2$-$10 keV
(Iwakiri et al., 2023). We carried out follow up observations with NICER from
16:52 UT on July 23 to 06:00 UT on August 2.

We downloaded the NICER data of ObsIDs: 6203900101$-$6203900111 from the
HEASARC archive. We employed the standard analysis procedure. First, we used
nicerl2 in HEASoft ver. 6.32.1 to filter and calibrate raw data using the
calibration database (CALDB) version xti20221001 with two options, which were
(a) overshoot count rate range of 0$-$5; (b) cut off rigidity greater than 1.5
$\mathrm{GeV\>c^{-1}}.$ Then, we extracted light curves from the filtered
events with xselect and generated ObsID-averaged source and background spectra
with nicerl3-spect and the 3C50 model (Remillard et al., 2022). We also
extracted time-resolved spectra of ObsIDs 6203900101 and 6203900102 with
nimaketime, niextract-event and nicerl3-spect. We used Xspec ver. 12.12.1
(Arnaud, 1996) and PyXspec ver. 2.1.0 (Gordon & Arnaud, 2021) for our spectral
analyses.

Figure 4:  (a) 0.3$-$4 keV light curve same as Figure 1b. (b) Time evolution
of the electron temperature calculated from the 5$-$9 keV continuum. (c) Best-
fit normalization ($K^{\mathrm{gauss1}},\>K^{\mathrm{gauss2}}$) of the two
gaussians. (d) Ly$\alpha$/He$\alpha$ ratio calculated by the two gaussian
best-fit normalizations ($K^{\mathrm{gauss2}}/K^{\mathrm{gauss1}}$). (e)
Doppler velocity of the line center of the Fe XXV He$\alpha$ line. The
negative sign corresponds with blue-shift.

##  3 Analysis and Result

Figure 1 shows the $0.3-4$ keV light curve of all NICER observations of IM Peg
covering the decay phase of the flare for $\sim 10$ days. We divided the first
two ObsIDs 6203900101 and 6203900102 into good time intervals (GTIs) with
their typical exposures of 500$-$1500 s. These GTIs are numbered as the
interval a0$-$b4 (Figure 1b). We combined the three GTIs of the interval b3
and b4 to obtain better statistics for the following spectral analysis.

First, we extracted two spectra from the interval a0$-$a4 and b0$-$b4 (Figure
2). We fit the wide-band (0.3$-$9 keV) spectra with a collisionally-ionized
equilibrium (CIE) model composed of three temperature components
(vapec+vapec+vapec) convolved with interstellar absorption (tbabs). We tied
the abundance among the three components. The best-fit temperatures of the
three components of the interval a0$-$a4 and b0$-$b4 were 92 MK, 15 MK, and
0.92 MK and 64 MK, 13 MK, and 0.92 MK, respectively.

The Fe XXV He$\alpha$ and Fe XXVI Ly$\alpha$ lines were clearly detected
during the interval a0$-$a4 and b0$-$b4 (Figure 2), as expected from the
temperatures (92 MK; a0$-$a4 and 64 MK; b0$-$b4) of the hottest plasma
components (Smith et al., 2001). The NICER energy resolution of 137 eV at 6
keV cannot resolve the fine structures of the Fe XXV He$\alpha$ and Fe XXVI
Ly$\alpha$ lines; the former consists of the resonance, intercombination, and
forbidden lines at 6.60$-$6.72 keV, whereas the latter of Ly$\alpha_{1}$ and
Ly$\alpha_{2}$ lines at 6.95$-$6.97 keV (Appendix A). Therefore, both lines
are approximated by a single Gaussian function with their lab-frame line
centers at $\sim 6.69$ keV (He$\alpha$) and $\sim 6.97$ keV (Ly$\alpha$).

The simple CIE model fittings show residuals around the iron line emissions
(6$-$7 keV) in the ObsID-averaged spectra (interval a0$-$a4 and b0$-$b4 in
Figure 2). The highest-temperature component of the CIE model successfully
explain the high-energy continuum above 5 keV, but fails to reproduce the line
profiles (the ratio panels in Figure 2). The enlarged spectra around the iron
emission lines in Figure 2 suggest that they are blue-shifted. To investigate
the time evolution of these two iron emission lines, we further extracted
eight spectra of the individual interval a0$-$b4 (see Appendix B). Then, we
found the most pronounced residuals around the iron emission lines appeared in
the interval b1 spectrum. Neither the CIE nor off-CIE models can explain the
5–9 keV spectrum during the interval b1 without assuming that the iron
emission lines were blue-shifted, which suggests that the plasma is moving
toward us (Appendix C).

In the following analysis, we focus on the spectral fittings around the iron
features in 6$-$7 keV. In Figure 3a, we analyzed the interval b1 spectrum with
a phenomenological model (powerlaw+gauss1+gauss2) in narrow-band (5$-$9 keV).
The best-fit parameters of the line center of the gauss1 and gauss2 were
$E_{l}^{\mathrm{gauss1}}=6.74\pm 0.01$ keV and
$E_{l}^{\mathrm{gauss2}}=6.96\pm 0.08$ keV, respectively (Table 1). To
investigate how far these results deviate from the lab-frame ones, we modeled
the CIE spectrum (Figure 3b) and folded it by NICER response functions (Figure
3c) at the electron temperature of the interval b1 with PyAtomDB (Foster &
Heuer, 2020). In this process, we used RMF and ARF files made for the interval
b1 event file with nicerl3-spect (Section 2). During the interval b1, the
continuum fitting with apec, ignoring the iron line bands (i.e., 5$-$9 keV
except for 6.3$-$7.4 keV), gives its electron temperature at
$T_{e}^{\mathrm{b1}}=76$ MK. We also fit the model spectrum (Figure 3c) with
the phenomenological model (powerlaw+gauss1+gauss2) and get the best-fit
parameters of $E_{l}^{\mathrm{He\alpha}}(T_{e}^{\mathrm{b1}})=6.69$ keV and
$E_{l}^{\mathrm{Ly\alpha}}(T_{e}^{\mathrm{b1}})=6.97$ keV. Given these
results, the energy shifts of the Fe XXV He$\mathrm{\alpha}$ and Fe XXVI
Ly$\mathrm{\alpha}$ lines are calculated as $\Delta
E^{\mathrm{He\alpha}}=E_{l}^{\mathrm{gauss1}}-E_{l}^{\mathrm{He\alpha}}(T_{e}^{\mathrm{b1}})=0.05\pm
0.01\>\mathrm{keV}$ and $\Delta
E^{\mathrm{Ly\alpha}}=E_{l}^{\mathrm{gauss1}}-E_{l}^{\mathrm{He\alpha}}(T_{e}^{\mathrm{b1}})=-0.01\pm
0.08\>\mathrm{keV}$, respectively. The Doppler velocity of
$v^{\mathrm{He\alpha}}=-c\Delta
E^{\mathrm{He\alpha}}/E_{l}^{\mathrm{He\alpha}}(T_{e}^{\mathrm{b1}})=-2200\pm
600\>\mathrm{km\>s^{-1}}$ (90 % confidence level) overwhelmingly exceeded the
escape velocity of IM Peg ($-230\>\mathrm{km\>s^{-1}}$). On the other hand,
the error range of $v^{\mathrm{Ly\alpha}}=-c\Delta
E^{\mathrm{Ly\alpha}}/E_{l}^{\mathrm{Ly\alpha}}(T_{e}^{\mathrm{b1}})=500\pm
3500\>\mathrm{km\>s^{-1}}$ were so large that we could not quantitively
confirm a blueshift of the Fe XXVI Ly$\alpha$ line.

Using the same procedure, we also estimated the electron temperature and the
Doppler velocity of the Fe XXV He$\alpha$ and Fe XXVI Ly$\alpha$ lines for all
intervals. Figure 4a$-$e show the time evolution of physical parameters
obtained by our narrow-band (5$-$9 keV) spectral analysis. The electron
temperature and intensities (normalizations) of two lines decrease with time,
while the line intensity ratio (Fe XXVI Ly$\alpha$ / Fe XXV He$\alpha$) stay
constant within errors. These results are consistent with the superflare of
the RS CVn-type star UX Ari reported in Kurihara et al. (2024). One
interesting point is that there was a sign of reheating only during the
interval b0 (Figure 4b). Figure 4e shows that there is a tendency for the line
center of the Fe XXV He$\alpha$ line to be blue-shifted except for the
interval b4. The velocity has a peak value of $-2200\pm
600\>\mathrm{km\>s^{-1}}$ at the interval b1. The intervals other than b1 when
the blueshift was clearly identified were the intervals b0 and b3, with
velocities of $-1000\pm 500\>\mathrm{km\>s^{-1}}$ and $-1300\pm
650\>\mathrm{km\>s^{-1}}$, respectively.

In order to detect the blueshift at the 50 eV level, it is also essential to
check the energy calibration of the detector. As a reference of a relative
energy shift among the NICER observational data, we selected supernovae
remnant Cas A, which has been frequently observed by NICER. This source has a
similar X-ray intensity as IM Peg during the flare, the prominent iron Fe XXV
He$\alpha$ emission line, and is not time-varying. We extracted 605 spectra
with the same exposure as each GTI of IM Peg, of which spectra are
individually fitted. The 605 Fe XXV He$\alpha$ emission line center follows
the Gaussian distribution with a central value of
$6.602_{-0.00015}^{+0.00047}$ keV, which is a reasonable value for the Fe XXV
He$\alpha$ line and within the range reported by previous satellite
observations ($6.56-6.62$ keV; Appendix D). The width of the Gaussian
distribution of this 605 sample is 8.7 eV (26 eV at 3$\sigma$, equivalent to
$1200\>\mathrm{km\>s^{-1}}$), which is sufficiently smaller than the velocity
discussed in this paper. We also confirmed that the light leak had little
impact on the NICER data since most of our observations (intervals a0$-$a4,
b0$-$b4) were performed during an orbital night with low background
(undershoot values below $\sim 10\>\mathrm{counts\>s^{-1}}$). Furthermore, the
NICER team has been checking time-dependent instrumental gain shifts of NICER
detectors in 2018–2021. They suggest the shift in 2023 is at most 10–15 eV
near the Fe-K line energy band (private communication with Dr. Craig Markwardt
as the lead of NICER calibration activities). This instrumental gain shift is
smaller than the detected $\sim 50$ eV blueshift.

##  4 Discussion and Conclusion

We presented the superflare of the RS CVn-type star IM Peg with NICER from
16:52 UT on 2023 July 23. During the early interval of the observations, NICER
spectra clearly showed Fe XXV He$\alpha$ and Fe XXVI Ly$\alpha$ emission
lines. The Fe XXV He$\alpha$ line was blue-shifted during the flare, which
suggests that the plasma is moving toward us. By phenomenological fitting and
CIE modeling, the maximum Doppler velocity of the Fe XXV He$\alpha$ line was
estimated to be $-2200\pm 600\>\mathrm{km\>s^{-1}}$.

This measurement allows us to estimate the total kinetic energy of the blue-
shifted plasma, which could lead to understanding global energetics of
superflares (cf. Emslie et al., 2012). The flare loop size is estimated to be
$L_{\mathrm{SY}}\sim 10^{-1}R$ using the magnetic reconnection model equations
in Shibata & Yokoyama (2002) with assuming that the temperature and emission
measure of the interval a0 (Figure 5a) to be the peak values. We assumed that
the coronal density is $n_{e}=10^{10-13}\>\mathrm{cm^{-3}}$ (Aschwanden &
Benz, 1997; Güdel, 2004; Reale, 2007; Sasaki et al., 2021) in this
calculation. Then, we calculated the kinetic energy of the blueshift as
$E_{\mathrm{kin}}^{\mathrm{He\alpha}}\sim
L_{\mathrm{SY}}^{3}n_{0}m_{\mathrm{p}}(v^{\mathrm{He\alpha}})^{2}\sim
10^{36}\>\mathrm{erg}$, where $m_{\mathrm{p}}$ is the mass of the proton. The
estimated kinetic energy is less than a few parcent of the X-ray flare energy
of $5\times 10^{37}\>\mathrm{erg}$ (Iwakiri et al., 2023).

There are two possible cases for interpreting the blue-shifted Fe XXV
He$\alpha$ line: a CME or chromospheric evaporation. In the CME
interpretation, the $\sim-2200$ $\mathrm{km\>s^{-1}}$ blueshifts of the Fe XXV
He$\alpha$ line originate from a CME. The typical velocity of chromospheric
evaporation is $\sim-100$ $\mathrm{km\>s^{-1}}$ (e.g., Doschek et al., 1980).
The velocity of solar CMEs ($-20\sim-3000$ $\mathrm{km\>s^{-1}}$: Webb &
Howard, 2012) can explain the observed velocity. The CME, whose velocity
overwhelmingly exceeded the escape velocity ($\sim-230\>\mathrm{km\>s^{-1}}$),
would certainly have flown outward from the star. Furthermore, we see that
there is a trend of acceleration from the interval a4 to b1 (Figure 4g). This
is consistent with Argiroffi et al. (2019), in which they discussed a
blueshift during the post-flare phase as the CME.

It is also possible that the blue-shifted emission from the CME was present
during the interval a0$-$a4 as well as b1, but was not clearly observed due to
the predominance of strong emission from the flare component. In this
interpretation, we can also fit the Fe XXV He$\alpha$ line with two gaussian
components, the flare component and the CME component. In the case, the
velocity of the CME were estimated to be $-6100\pm 900$ $\mathrm{km\>s^{-1}}$
for the interval a0$-$a4 and $-4400\pm 1000$ $\mathrm{km\>s^{-1}}$ for the
interval b0$-$b4, respectively (Appendix E).

The formation temperature ($T_{\mathrm{peak}}$) of the Fe XXV He$\alpha$ line
($T_{\mathrm{peak}}\sim$ 60 MK) is higher than that of other stellar CME
candidates. Argiroffi et al. (2019) discussed a later blueshift of
$\mathrm{O_{VIII}}$ line ($T_{\mathrm{peak}}\sim 3$ MK) on the G type star
HR9024 as a CME candidate. Chen et al. (2022) reported upflows of 3$-$16 MK
plasma accompanied by the decrease of the coronal density on the M type star
EV Lac. The temperature of solar CMEs is also on the order of a few MK
(Sheoran et al., 2023). The discovery of the Fe XXV He$\mathrm{\alpha}$ line
blueshift, which could not be investigated by the Chandra HETG observations
(Argiroffi et al., 2019; Chen et al., 2022), means that CMEs on magnetically
active stars like RS CVn-type stars consist of hot plasma ($\sim$ 60 MK).

In the chromospheric evaporation interpretation, the reheating during the
interval b0 (Figure 4d) one hour before the most pronounced blueshift can be
well understood. From the interval a4 to b0, the electron temperature rose
from 60 MK to 100 MK over the course of an hour. For some reason, the plasma
in the chromosphere may have been rapidly reheated at roughly half a day after
the flare peak, causing the high-velocity chromosphere evaporation.

To conclude, this work reported the blueshifts of the Fe XXV He$\alpha$ line
during a stellar flare for the first time. Given the impact of CMEs on
exoplanets (e.g., Airapetian et al., 2016), it is important to determine
whether such blueshifts are attributed to a CME or chromospheric evaporation.
To know the answer, we need to distinguish whether the line itself is blue-
shifted or non-shifted and blue-shifted components are added together. The
high resolution spectroscopy of X-Ray Imaging and Spectroscopy Mission (XRISM;
Tashiro et al., 2020) with its microcalorimeter Resolve (Ishisaki et al.,
2018) will enable us to observe the line profile of iron emission lines and
distinguish them.

NICER analysis software and data calibration were provided by the NASA NICER
mission and the Astrophysics Explorers Program. We thank K. Shibata (Doshisha
University), H. Maehara (NAOJ) and K. Namekata (Kyoto University) for their
useful comments and discussions. We thank C. Markward (NASA/GSFC) for the
calibration comments. This research is supported by JSPS KAKENHI grant No.
24KJ1483 (S.I.). T.E. was supported by the RIKEN Hakubi project. Y.N.
acknowledge support from NASA ADAP award program No. 80NSSC21K0632.

##  Appendix A Fine structure of the Fe XXV He$\alpha$ and XXVI Ly$\alpha$
lines

Table 2: Line center energy of the Fe XXV He$\alpha$ and Ly$\alpha$ lines Ion | $E_{l}^{NICER}$ | Line Id. | Line Energy
---|---|---|---
| (keV) |  | (keV)
Fe XXV | $\sim 6.69$ | w | 6.700
x | 6.682
y | 6.668
z | 6.637
Fe XXVI | $\sim 6.97$ | Ly$\alpha_{1}$ | 6.973
Ly$\alpha_{2}$ | 6.952

We display the line center energy of all lines of the Fe XXV He$\alpha$ and
Ly$\alpha$ lines in Table 2 (c.f. Table 1 of Bianchi et al., 2005). As
mentioned in Section 3, these fine structures are not resolved in the NICER
spectra

##  Appendix B Wideband spectra

Figure 5:  Same as Figure 2, but during the interval a0, b1, and b4 for panels
(a), (b), and (c), respectively. The temperatures of three components are 101
MK, 15MK, and 1 MK for panel (a), 65 MK, 13 MK, and 1 MK for panel (b), 49 MK,
13 MK, and 1 MK for panel (c).

We provide wideband (0.3$-$9.0 keV) spectra of the interval a0, b1, and b4
fitted with the three temperature collisionally-ionized equilibrium (CIE)
models (vapec) as examples in Figure 5. The interval a0 spectrum with the best
photon statistics showed the slight systematic residuals around iron emission
lines, which were not as much as the interval b1. The interval b1 spectrum
showed the most pronounced residuals around iron emission lines as discussed
in Section 3. The interval b4 spectrum with the relatively poor photon
statistics showed no systematic residuals around iron emission lines. This is
consistent with the fact that the Doppler velocity of the interval b4 spectrum
obtained by our narrow band spectral analysis (Figure 4e) was not blue-
shifted.

##  Appendix C Fitting the interval b1 spectrum with Plasma models

Figure 6: Comparison of spectral fittings by different plasma models around
the iron emission lines during the interval b1: the CIE (apec, panel a),
ionizing (nei, panel b), and recombining (rnei, panel c) plasma models with
the velocity fixed at $v=0$. The lower panels show the fitting residuals.
Panels d, e, and f are the same as panels a, b, and c, respectively, but
letting the velocity of the plasma free.  Table 1: Best-fit spectral
parameters of plasma models during the interval b1 Collisional ionization
equilibrium plasma model (apec)
---
Velocity: fixed at zero | Velocity: free
apec |  $kT$ (keV) / $T$ (MK) |  $6.90\pm 0.69$ / $80.0\pm 8.0$ | apec |  $kT$ (keV) / $T$ (MK) |  $6.16\pm 0.65$ / $71.5\pm 7.54$
$Z$ ($Z_{\odot}$) | $0.25\pm 0.05$ |  $Z$ ($Z_{\odot}$) | $0.28\pm 0.05$
$v\>(10^{3}\>\mathrm{km\>s^{-1}})$ |  $0.00$ (fix) | $v\>(10^{3}\>\mathrm{km\>s^{-1}})$ | $-2.54\pm 0.68$
$K^{\mathrm{apec}}$ | $0.49\pm 0.04$ | $K^{\mathrm{apec}}$ | $0.51\pm 0.05$
$\chi^{2}$ (d.o.f.) | 86 (75) |  $\chi^{2}$ (d.o.f.) | 45 (74)
Null hyp. prob. | 0.17 | Null hyp. prob. | 0.99
Ionizing plasma model (nei)
Velocity: fixed at zero | Velocity: free
nei |  $kT$ (keV) / $T$ (MK) |  $8.01\pm 1.67$ / $92.9\pm 19.4$ | nei |  $kT$ (keV) / $T$ (MK) |  $6.74\pm 1.12$ / $78.2\pm 13.0$
$Z$ ($Z_{\odot}$) | $0.24\pm 0.06$ |  $Z$ ($Z_{\odot}$) | $0.28\pm 0.05$
$n_{e}t$ ($\mathrm{s\>cm^{-3}}$) | $\gtrsim 2\times 10^{12}$ |  $n_{e}t$ ($\mathrm{s\>cm^{-3}}$) | $\gtrsim 2\times 10^{11}$
$v\>(10^{3}\>\mathrm{km\>s^{-1}})$ |  $0.00$ (fix) | $v\>(10^{3}\>\mathrm{km\>s^{-1}})$ | $-2.45\pm 0.46$
$K^{\mathrm{nei}}$ | $0.46\pm 0.06$ | $K^{\mathrm{nei}}$ | $0.49\pm 0.06$
$\chi^{2}$ (d.o.f.) | 86 (74) |  $\chi^{2}$ (d.o.f.) | 44 (73)
Null hyp. prob. | 0.16 | Null hyp. prob. | 0.99
Recombining plasma model (rnei)
Velocity: fixed at zero | Velocity: free
rnei |  $kT$ (keV) / $T$ (MK) |  $7.00\pm 0.76$ / $81.2\pm 8.81$ | rnei |  $kT$ (keV) / $T$ (MK) |  $6.18\pm 0.68$ / $71.7\pm 7.89$
$kT_{\mathrm{init}}$ (keV) / $T_{\mathrm{init}}$ (MK) |  $9.06$ / $105$ |  $kT_{\mathrm{init}}$ (keV) / $T_{\mathrm{init}}$ (MK) | 9.06 / 105
$Z$ ($Z_{\odot}$) | $0.25\pm 0.05$ |  $Z$ ($Z_{\odot}$) | $0.29\pm 0.05$
$n_{e}t$ ($\mathrm{s\>cm^{-3}}$) | $\gtrsim 5\times 10^{11}$ |  $n_{e}t$ ($\mathrm{s\>cm^{-3}}$) | $\gtrsim 6\times 10^{12}$
$v\>(10^{3}\>\mathrm{km\>s^{-1}})$ | 0.00 (fix) | $v\>(10^{3}\>\mathrm{km\>s^{-1}})$ | $-2.63\pm 0.53$
$K^{\mathrm{rnei}}$ | $0.49\pm 0.04$ | $K^{\mathrm{rnei}}$ | $0.51\pm 0.03$
$\chi^{2}$ (d.o.f.) | 87 (74) |  $\chi^{2}$ (d.o.f.) | 44 (73)
Null hyp. prob. | 0.15 | Null hyp. prob. | 0.99

We also tried CIE (apec) and off-CIE plasma models, the ionizing (nei) and
recombining (rnei) models, for the interval b1 spectrum. When we used the
recombinning model, we fixed the initial temperature to the electron
temperature of the interval a0 of $\sim 105$ MK. All models can not eliminate
the residuals at $v=0$ (Figure 6a$-$c). On the other hand, when we leave the
velocity $v$ free and again fit the spectrum with apec, nei, and rnei (Figure
6d$-$f), bulk motion, and thus the blue-shifted line center corresponding with
the negative sign of the velocity, improved the fittings with better
$p$-values, eliminating the residuals (Table 1). The best-fit parameters of
the velocity were $-2540\pm 680\>\mathrm{km\>s^{-1}}$, $-2450\pm
460\>\mathrm{km\>s^{-1}}$, and $-2630\pm 530\>\mathrm{km\>s^{-1}}$ for apec,
nei, and rnei, respectively. These values are consistent with the velocity of
$-2200\pm 600\>\mathrm{km\>s^{-1}}$ obtained with the phenomenological fitting
(Section 3) within error bars. Furthermore, the best-fit parameters of
temperature, abundance, velocity, and normalization became consistent within
errorbars among the three plasma models. This suggests that the observed 5$-$9
keV spectra can be explained either by CIE or off-CIE plasma models, assuming
the plasma moving at a velocity of about $-2500$ $\mathrm{km\>s^{-1}}$. In
other words, no model can reproduce the observed iron emission lines without
the blueshifts.

##  Appendix D Energy determination accuracy of NICER around iron emission
lines

Figure 1: (a)$-$(c) Examples of NICER Cas A spectra fitted with the
phenomenological model of powerlaw+gauss, whose exposure times are (a) $\sim
500$ sec, (b) $\sim 1000$ sec, and (c) $\sim 1500$ sec, respectively. Orange
and green dashed lines indicate powerlaw and gauss components, respectively.
(d) The distribution of the line center of the gaussian function ($\Delta
E=E-E_{\mathrm{med}}$). The upper panel shows the histogram of the
distribution for all data ($N=605$), and the Gaussian fitting (black dashdot
line). The color map shows the distribution at each range of the exposure
time. The right panel shows the histogram of the distribution at each exposure
time.

As discussed in Section 3, the Fe XXV He$\mathrm{\alpha}$ line was seem to be
blueshifted during the flare. However, the accuracy of the NICER response
function in determining the energy of photons should be checked. Therefore, we
need to investigate how uncertainty the NICER response function has and
whether the observed blueshifts are attributed to the astronomical phenomenon.

We analyzed a large number of NICER data (192 ObsIDs) of the supernova remnant
Cas A (e.g., Holt et al., 1994; Maeda et al., 2009), which has a count rate
comparable to that of IM Peg during the flare, the strong iron emission line,
and no time variability. We downloaded all Cas A data observed by NICER from
HEASARC archive and processed them in the same manner as described in Section
2. Then, we extracted GTI-divided and all-GTI spectra from all Obs-ID data. As
shown in Figure 1a$-$c, we fitted the spectra, whose exposure times are
$500<t_{\mathrm{exp}}<1500$ sec, in 5$-$7.4 keV with the phenomenological
model (powerlaw+gauss). The exposure time of most intervals during the flare
on IM Peg is in this range. The total number of Cas A spectra we fitted was
605.

Figure 1d shows the distribution of the line center of the Gaussian. The
median value of the line center was $E_{\mathrm{med}}=6.602$ keV. The central
value and standard deviation of $\Delta E$ $(=E-E_{\mathrm{med}})$ were
$\mu=(1.6\pm 3.1)\times 10^{-4}$ keV and $\sigma=(8.7\pm 0.3)\times 10^{-3}$
keV, respectively. The error ranges of $\mu$ and $\sigma$ are 90 % confidence
level. Holt et al. (1994) reported the difference in the Doppler shift between
the locations in Cas A. According to their observations with ASCA Observatory,
the range of the line center energy of the iron emission line is
$6.5568-6.6080$ keV (see Table 1 in Holt et al., 1994). On the other hand,
Maeda et al. (2009) showed the line center energy is $6.621$ keV from the
spectrum extracted from the entire Cas A observed with Suzaku. NICER 90 %
confidence level of the line center in our analysis
($6.60185\>\mathrm{keV}<E_{\mathrm{med}}+\mu<6.60247\>\mathrm{keV}$) is in the
range of these previous works. Furthermore, our results indicates that the
upper limit of the uncertainty of the NICER response function is limited to
$|\Delta E|\lesssim 3\sigma=2.6\times 10^{-2}$ keV, which corresponds to the
Doppler velocity $v\sim\pm 1200\>\mathrm{km\>s^{-1}}$. This means that the
$-2200\pm 600\>\mathrm{km\>s^{-1}}$ ($(5\pm 1)\times 10^{-2}$ keV) blueshift
detected during the interval b1 ($t_{\mathrm{exp}}=834$ sec) can not be
explained by the uncertainty of the NICER energy determination.

Furthermore, since we were multiplying both modeling and observed spectra by
the response files at each time and taking the difference of the line center
between them (Section 3), the response-derived Doppler shift should cancel out
to some extent. Given these considerations, it is highly possible that the
astronomical phenomenon made the observed blueshifts.

##  Appendix E Fitting the Fe XXV He$\alpha$ line with two gaussian components

Figure 1: Background-subtracted 5$-$9 keV spectra of IM Peg fitted with a
phenomenological model (powerlaw+gauss*3) during the interval a0$-$a4 and
b0$-$b4 for panels (a) and (b), respectively. Green dashdot and blue dashed
lines correspond to the Fe XXV He$\alpha$ and the Fe XXVI Ly$\alpha$ lines,
respectively, whose line centers are fixed to the values in the laboratory
frame. Navy dotted lines indicate the blue-shifted components of the Fe XXV
He$\alpha$ line.  Figure 2: Doppler velocity of the blue-shifted components of
the Fe XXV He$\alpha$ line (see e.g., Figure 1) overlaid on that of the line
center of the Fe XXV He$\alpha$ line shown in Figure 4g. Navy and red dashed
lines indicate the Doppler velocity of the blue-shifted components of the Fe
XXV He$\alpha$ line with the $90\%$ error range during the combined interval
a0$-$a4 and b0$-$b4, respectively, when we fit the spectra with
powerlaw+gauss*3 model. Green and orange dashdot lines indicate the Doppler
velocity of the line center of the Fe XXV He$\alpha$ line with the $90\%$
error range during the combined interval a0$-$a4 and b0$-$b4, respectively,
when we fit the spectra with powerlaw+gauss*2 model.

When we consider the blueshifts of the Fe XXV He$\alpha$ line being attributed
to a CME, it is physically reasonable that there are both non-shifted (flare)
and blue-shifted (CME) components of the Fe XXV He$\alpha$ line as discussed
in the case of the blue-shifted H$\alpha$ line (e.g., Inoue et al., 2023).
Then, we also fitted the interval a0$-$a4 and b0$-$b4 spectra with the
phenomenological model (powerlaw+gauss*3) considering the CME components
(Figure 1). We did not fit the individual interval spectra with this model
because photon statistics are insufficient to determine the CME components. We
fixed the line centers of the flare components of the Fe XXV He$\alpha$ and Fe
XXVI Ly$\alpha$ lines to the lab-frame ones. As a result, the best-fit values
of the line centers of the CME components of the Fe XXV He$\alpha$ line were
$6.83\pm 0.02$ keV and $6.79\pm 0.02$ keV during interval a0$-$a4 and b0$-$b4,
respectively. These values correspond to the Doppler velocities of $-6100\pm
900$ $\mathrm{km\>s^{-1}}$ and $-4400\pm 1000$ $\mathrm{km\>s^{-1}}$,
respectively (Figure 2).

## References

  * Airapetian et al. (2016) Airapetian, V. S., Glocer, A., Gronoff, G., Hébrard, E., & Danchi, W. 2016, Nature Geoscience, 9, 452, doi: 10.1038/ngeo2719
  * Airapetian et al. (2020) Airapetian, V. S., Barnes, R., Cohen, O., et al. 2020, International Journal of Astrobiology, 19, 136–194, doi: 10.1017/S1473550419000132
  * Argiroffi et al. (2019) Argiroffi, C., Reale, F., Drake, J. J., et al. 2019, Nature Astronomy, 3, 742, doi: 10.1038/s41550-019-0781-4
  * Arnaud (1996) Arnaud, K. A. 1996, in Astronomical Society of the Pacific Conference Series, Vol. 101, Astronomical Data Analysis Software and Systems V, ed. G. H. Jacoby & J. Barnes, 17
  * Arzoumanian et al. (2014) Arzoumanian, Z., Gendreau, K. C., Baker, C. L., et al. 2014, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9144, Space Telescopes and Instrumentation 2014: Ultraviolet to Gamma Ray, ed. T. Takahashi, J.-W. A. den Herder, & M. Bautz, 914420, doi: 10.1117/12.2056811
  * Aschwanden & Benz (1997) Aschwanden, M. J., & Benz, A. O. 1997, ApJ, 480, 825, doi: 10.1086/303995
  * Astropy Collaboration et al. (2013) Astropy Collaboration, Robitaille, T. P., Tollerud, E. J., et al. 2013, A&A, 558, A33, doi: 10.1051/0004-6361/201322068
  * Astropy Collaboration et al. (2018) Astropy Collaboration, Price-Whelan, A. M., Sipőcz, B. M., et al. 2018, AJ, 156, 123, doi: 10.3847/1538-3881/aabc4f
  * Bianchi et al. (2005) Bianchi, S., Matt, G., Nicastro, F., Porquet, D., & Dubau, J. 2005, MNRAS, 357, 599, doi: 10.1111/j.1365-2966.2005.08661.x
  * Buzasi et al. (1987) Buzasi, D. L., Ramsey, L. W., & Huenemoerder, D. P. 1987, ApJ, 322, 353, doi: 10.1086/165733
  * Canizares et al. (2005) Canizares, C. R., Davis, J. E., Dewey, D., et al. 2005, PASP, 117, 1144, doi: 10.1086/432898
  * Caspi (2010) Caspi, A. 2010, PhD thesis, University of California, Berkeley
  * Chen et al. (2022) Chen, H., Tian, H., Li, H., et al. 2022, ApJ, 933, 92, doi: 10.3847/1538-4357/ac739b
  * Doschek et al. (1980) Doschek, G. A., Feldman, U., Kreplin, R. W., & Cohen, L. 1980, ApJ, 239, 725, doi: 10.1086/158158
  * Emslie et al. (2012) Emslie, A. G., Dennis, B. R., Shih, A. Y., et al. 2012, ApJ, 759, 71, doi: 10.1088/0004-637X/759/1/71
  * Fisher et al. (1985) Fisher, G. H., Canfield, R. C., & McClymont, A. N. 1985, ApJ, 289, 425, doi: 10.1086/162902
  * Flores Soriano & Strassmeier (2017) Flores Soriano, M., & Strassmeier, K. G. 2017, A&A, 597, A101, doi: 10.1051/0004-6361/201629338
  * Foster & Heuer (2020) Foster, A. R., & Heuer, K. 2020, Atoms, 8, 49, doi: 10.3390/atoms8030049
  * Fuhrmeister et al. (2018) Fuhrmeister, B., Czesla, S., Schmitt, J. H. M. M., et al. 2018, A&A, 615, A14, doi: 10.1051/0004-6361/201732204
  * Gendreau et al. (2012) Gendreau, K. C., Arzoumanian, Z., & Okajima, T. 2012, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 8443, Space Telescopes and Instrumentation 2012: Ultraviolet to Gamma Ray, ed. T. Takahashi, S. S. Murray, & J.-W. A. den Herder, 844313, doi: 10.1117/12.926396
  * Gendreau et al. (2016) Gendreau, K. C., Arzoumanian, Z., Adkins, P. W., et al. 2016, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9905, Space Telescopes and Instrumentation 2016: Ultraviolet to Gamma Ray, ed. J.-W. A. den Herder, T. Takahashi, & M. Bautz, 99051H, doi: 10.1117/12.2231304
  * Gordon & Arnaud (2021) Gordon, C., & Arnaud, K. 2021, PyXspec: Python interface to XSPEC spectral-fitting program, Astrophysics Source Code Library, record ascl:2101.014.  http://ascl.net/2101.014
  * Güdel (2004) Güdel, M. 2004, A&A Rev., 12, 71, doi: 10.1007/s00159-004-0023-2
  * Holt et al. (1994) Holt, S. S., Gotthelf, E. V., Tsunemi, H., & Negoro, H. 1994, PASJ, 46, L151
  * Honda et al. (2018) Honda, S., Notsu, Y., Namekata, K., et al. 2018, PASJ, 70, 62, doi: 10.1093/pasj/psy055
  * Inoue et al. (2023) Inoue, S., Maehara, H., Notsu, Y., et al. 2023, ApJ, 948, 9, doi: 10.3847/1538-4357/acb7e8
  * Inoue et al. (2024) Inoue, S., Enoto, T., Namekata, K., et al. 2024, PASJ, 76, 175, doi: 10.1093/pasj/psae001
  * Ishisaki et al. (2018) Ishisaki, Y., Ezoe, Y., Yamada, S., et al. 2018, Journal of Low Temperature Physics, 193, 991, doi: 10.1007/s10909-018-1913-4
  * Iwakiri et al. (2023) Iwakiri, W., Hamaguchi, K., Gendreau, K., et al. 2023, The Astronomer’s Telegram, 16152, 1
  * Koller et al. (2021) Koller, F., Leitzinger, M., Temmer, M., et al. 2021, A&A, 646, A34, doi: 10.1051/0004-6361/202039003
  * Kurihara et al. (2024) Kurihara, M., Iwakiri, W. B., Tsujimoto, M., et al. 2024, ApJ, 965, 135, doi: 10.3847/1538-4357/ad35c5
  * Lu et al. (2022) Lu, H.-p., Tian, H., Zhang, L.-y., et al. 2022, A&A, 663, A140, doi: 10.1051/0004-6361/202142909
  * Maeda et al. (2009) Maeda, Y., Uchiyama, Y., Bamba, A., et al. 2009, PASJ, 61, 1217–1228, doi: 10.1093/pasj/61.6.1217
  * Maehara et al. (2021) Maehara, H., Notsu, Y., Namekata, K., et al. 2021, PASJ, 73, 44, doi: 10.1093/pasj/psaa098
  * Matsuoka et al. (2009) Matsuoka, M., Kawasaki, K., Ueno, S., et al. 2009, PASJ, 61, 999, doi: 10.1093/pasj/61.5.999
  * Muheki et al. (2020a) Muheki, P., Guenther, E. W., Mutabazi, T., & Jurua, E. 2020a, A&A, 637, A13, doi: 10.1051/0004-6361/201936904
  * Muheki et al. (2020b) —. 2020b, MNRAS, 499, 5047, doi: 10.1093/mnras/staa3152
  * Namekata et al. (2021) Namekata, K., Maehara, H., Honda, S., et al. 2021, Nature Astronomy, 6, 241, doi: 10.1038/s41550-021-01532-8
  * Namekata et al. (2024) Namekata, K., Airapetian, V. S., Petit, P., et al. 2024, ApJ, 961, 23, doi: 10.3847/1538-4357/ad0b7c
  * Notsu et al. (2024) Notsu, Y., Kowalski, A. F., Maehara, H., et al. 2024, ApJ, 961, 189, doi: 10.3847/1538-4357/ad062f
  * Reale (2007) Reale, F. 2007, A&A, 471, 271, doi: 10.1051/0004-6361:20077223
  * Remillard et al. (2022) Remillard, R. A., Loewenstein, M., Steiner, J. F., et al. 2022, The Astronomical Journal, 163, 130, doi: 10.3847/1538-3881/ac4ae6
  * Sasaki et al. (2021) Sasaki, R., Tsuboi, Y., Iwakiri, W., et al. 2021, ApJ, 910, 25, doi: 10.3847/1538-4357/abde38
  * Sheoran et al. (2023) Sheoran, J., Pant, V., Patel, R., & Banerjee, D. 2023, Frontiers in Astronomy and Space Sciences, 10, 27, doi: 10.3389/fspas.2023.1092881
  * Shibata & Yokoyama (2002) Shibata, K., & Yokoyama, T. 2002, ApJ, 577, 422, doi: 10.1086/342141
  * Sinha et al. (2019) Sinha, S., Srivastava, N., & Nandy, D. 2019, ApJ, 880, 84, doi: 10.3847/1538-4357/ab2239
  * Smith et al. (2001) Smith, R. K., Brickhouse, N. S., Liedahl, D. A., & Raymond, J. C. 2001, ApJ, 556, L91, doi: 10.1086/322992
  * Tashiro et al. (2020) Tashiro, M., Maejima, H., Toda, K., et al. 2020, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 11444, Space Telescopes and Instrumentation 2020: Ultraviolet to Gamma Ray, ed. J.-W. A. den Herder, S. Nikzad, & K. Nakazawa, 1144422, doi: 10.1117/12.2565812
  * Testa et al. (2004) Testa, P., Drake, J. J., & Peres, G. 2004, ApJ, 617, 508, doi: 10.1086/422355
  * Tsuboi et al. (2016) Tsuboi, Y., Yamazaki, K., Sugawara, Y., et al. 2016, PASJ, 68, 90, doi: 10.1093/pasj/psw081
  * Tsuru et al. (1989) Tsuru, T., Makishima, K., Ohashi, T., et al. 1989, PASJ, 41, 679
  * Vida et al. (2019) Vida, K., Leitzinger, M., Kriskovics, L., et al. 2019, A&A, 623, A49, doi: 10.1051/0004-6361/201834264
  * Vida et al. (2016) Vida, K., Kriskovics, L., Oláh, K., et al. 2016, A&A, 590, A11, doi: 10.1051/0004-6361/201527925
  * Wargelin et al. (2005) Wargelin, B. J., Beiersdorfer, P., Neill, P. A., Olson, R. E., & Scofield, J. H. 2005, ApJ, 634, 687, doi: 10.1086/496874
  * Webb & Howard (2012) Webb, D. F., & Howard, T. A. 2012, Living Reviews in Solar Physics, 9, 3, doi: 10.12942/lrsp-2012-3
  * Weisskopf et al. (2000) Weisskopf, M. C., Tananbaum, H. D., Van Speybroeck, L. P., & O’Dell, S. L. 2000, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 4012, X-Ray Optics, Instruments, and Missions III, ed. J. E. Truemper & B. Aschenbach, 2–16, doi: 10.1117/12.391545
  * Zellem et al. (2010) Zellem, R., Guinan, E. F., Messina, S., et al. 2010, PASP, 122, 670, doi: 10.1086/653711


# NeRF-Supervised Feature Point Detection and Description

Ali Youssef
Department of Computer Science
University College London
<EMAIL_ADDRESS>
&Francisco Vasconcelos
Department of Computer Science
University College London
<EMAIL_ADDRESS>
Wellcome / EPSRC Centre for Interventional and Surgical Sciences (WEISS).

###### Abstract

Feature point detection and description is the backbone for various computer
vision applications, such as Structure-from-Motion, visual SLAM, and visual
place recognition. While learning-based methods have surpassed traditional
handcrafted techniques, their training often relies on simplistic homography-
based simulations of multi-view perspectives, limiting model generalisability.
This paper introduces a novel approach leveraging neural radiance fields
(NeRFs) for realistic multi-view training data generation. We create a diverse
multi-view dataset using NeRFs, consisting of indoor and outdoor scenes. Our
proposed methodology adapts state-of-the-art feature detectors and descriptors
to train on NeRF-synthesised views supervised by perspective projective
geometry. Our experiments demonstrate that the proposed methods achieve
competitive or superior performance on standard benchmarks for relative pose
estimation, point cloud registration, and homography estimation while
requiring significantly less training data compared to existing approaches.

_K_ eywords Feature detection and description $\cdot$ Neural Radiance Fields
$\cdot$ Datasets

##  1 Introduction

Feature point detection and description under different scene viewpoints is a
common starting point for many multi-view problems including Structure-from-
Motion [1], visual SLAM [2, 3], or visual place recognition [4, 5]. In the
past decade, different learning-based approaches to this problem [6, 7, 8, 9]
have replaced handcrafted techniques in many applications [10, 11, 12].
Crucially, most of these models can be fine-tuned in a self-supervised manner
on any single-view dataset. This is achieved by applying different homography
warpings to the training data, simulating different viewpoints of the same
scene with known point-to-point "ground-truth" mappings. While this training
scheme is simple and flexible, the generated homography warpings are a crude
simplification of multi-view perspective, which can lead to limited model
generalisability.

This paper aims at leveraging image synthesis with neural radiance fields
(NeRFs) as a more realistic way of generating multi-view training data for
feature detection and description models (see Fig. 1). Since NeRFs require
multi-view data to synthesise novel views, we can no longer rely on the single
image datasets typically used for training the above mentioned homography-
based methods. We therefore create our own dataset consisting of both indoor
and outdoor image sequences around static scenes, and reconstruct all of them
with NeRFacto [13]. This enables the generation of arbitrary viewpoints of
each scene consistent with a pinhole projective model with known point-to-
point mappings via point re-projection. We propose a general methodology to
upgrade state-of-the-art homography-based methods to train them on projective
views synthesised from NeRF-type algorithms. Our contributions are as follows:

  * •

We create a new multi-view dataset consisting of images from 10 different
indoor and outdoor scenes, and a total of 10000 NeRF-synthesised views from
these scenes with corresponding depth maps, intrinsic and extrinsic
parameters.

  * •

We propose two general methodologies (end-to-end and projective adaptation) to
train state-of-the-art point detection and description methods using a loss
function based on NeRF re-projection error.

  * •

We re-train adapted versions of SuperPoint [6] and SiLK [14] using our NeRF-
synthesised data and compare them against the original baselines trained on
the much larger MS-COCO dataset. We outperform the original baselines for
relative pose estimation on ScanNet and YFCC100M datasets, similar performance
for pair-wise point cloud registration, while only slightly under-performing
on the HPatches homography estimation benchmark.

(a) Input Image

(b) Image w/ Homography

(c) Image Diff. Viewpoint

Figure 1: Visual representation of multi-view data. Learning-based detectors
and descriptors achieve supervision on single-view datasets by simulating
different viewpoints through homograhpic warpings to the input image $I$ (see
Fig. 1(a)) resulting in $I^{{}^{\prime}}$ (see Fig. 1(b)). While we achieve
supervision by directly sampling a NeRF rendered image from a different
viewpoint (see Fig. 1(c)).

##  2 Related Work

Research on multi-view feature detection and description has a large focus on
creating representations invariant to geometric transformations (e. g. scale,
rotation, affinity) and to illumination conditions, so that the same scene
point can be reliably recognised and matched regardless of its viewpoint.
Classic methods achieved this by handcrafting feature extractors that
incorporate these invariance properties by design. Some of these decade-old
methods stood the test of time and are still widely used today [15, 16]. More
recently, the success of deep feature extraction enabled neural networks to
learn these invariance properties from training data. Large open-source
datasets [17, 18, 19] have been extensively employed to train either learning-
based interest point detectors and descriptors [6, 20, 14, 21], or detector-
free local feature matchers [22, 23, 24].

Some of these methods [6, 14] are trained in a self-supervised manner on
uncalibrated RGB single-view datasets such as MS-COCO [17] by simulating
multiple views as homography warpings. Other methods [20, 21] are trained with
additional supervision using calibrated multi-view RGB-D datasets such as
ScanNet [18]. These two classes of methods have pros and cons. The multi-view
data generated by self-supervised methods is not representative of all
possible viewpoint changes (homographies only model planar scenes or pure
rotations) and thus it does not fully generalise to some application
scenarios. On the positive side, they can be trained on arbitrarily large
number of transformations and are also extremely flexible as they can be fine-
tuned on any uncalibrated single-view dataset. Fully supervised methods are
trained on more realistic multi-view data that covers full projective view
changes with occlusions, however, the required training data is significantly
more complex to obtain. The variety of scenes and viewpoint changes in the
training data is also limited by the practicalities of real data acquisition.
To get the best of both worlds, we propose using NeRF reconstructions obtained
from RGB image sequences, which are both simple to acquire and enable the
synthetic generation of arbitrary projective viewpoints beyond homographies.

Progress on image synthesis with NeRF and its variants [25, 26, 27] has
exploded in the past years, with very fast improvements in computational
efficiency, image synthesis quality, reconstruction accuracy, and input data
requirements. As these techniques enable synthesising arbitrary viewpoints of
a reconstructed scene, they can be used for rich data augmentation [28]. Most
related to our work, they have been utilised to self-supervise multi-view
methods for learning viewpoint invariant object descriptors [29], stereo
disparity [30], and optical flow [31]. However, to the best of our knowledge
NeRF self-supervision has been under-explored for point feature detection and
description.

##  3 Methodology

###  3.1 NeRF Dataset

We build a NeRF dataset for 10 indoor and outdoor scenes, containing in total
10000 synthetic images with their corresponding intrinsic/extrinsic parameters
and depth maps. We note that this dataset is extremely small compared to what
is typically used to train state-of-the-art feature detection (30 times
smaller than MS-COCO, 250 times smaller than ScanNet). Our goal is to show
that leveraging NeRF self-supervision enables state-of-the-art performance on
multiple benchmarks with significantly less training data.

We captured four indoor and one outdoor scenes with an IPhone 10 at 4K
resolution, one indoor and one outdoor scene with a Samsung A52 at 4k
resolution. We also utilise two blender generated indoor scenes with images
sourced from [32] and one outdoor scene publicly available through [33] as
they are scenes that consist of high quality RGB images, while being are rich
with features. While different training sets could be constructed using
existing publicly available data, we focus on a trade-off between scene
diversity and minimal dataset size.

We obtain camera poses for all images using COLMAP [34] and use them to
generate synthetic views. We trained a Nerfacto model for each scene, which is
part of the NeRFStudio framework [13] and merges improvements from a range of
NeRF implementations [35, 33, 36, 37]. For each scene we rendered 1000
synthetic views and corresponding depth maps with $44^{\circ}$ field of view
and $640\times 480$ resolution. We use NeRFStudio’s interactive real-time
viewer to define custom camera trajectories for each scene that ensure
diversity in viewpoints while maintaining global scene overlap. Trajectories
consist of uniformly sampled smooth trajectories around the scene coupled with
small rotations around the camera centre.

(a) Custom Camera trajectory

(b) Camera 2 Viewpoint

Figure 2: Customised Camera Trajectory. A visual representation of a custom
camera trajectory in NeRF Studio’s [13] interactive real-time viewer. Fig.
2(b) demonstrates the viewpoint of Camera 2 in Fig. 2(a).

###  3.2 NeRF Point Re-Projection

Similarly to prior self-supervised methods [6, 14] we consider neural networks
that, at inference time, detect interest point locations $\mathbf{p}$ and
extract their descriptors $\mathbf{d}$ from a single view. Consider two views
$I$, $I^{\prime}$ synthesised by NeRF with known intrinsic
($\mathcal{K},\mathcal{K}^{\prime}$) and extrinsic
($R,R^{\prime},\mathbf{t},\mathbf{t}^{\prime}$) parameters. For any arbitrary
point with homogeneous pixel coordinates $\mathbf{p}$ in image $I$, we know
its NeRF reconstructed depth $d$ and can represent it in 3D world coordinates
as:

$\displaystyle\mathbf{P}$
$\displaystyle=[R|\mathbf{t}]\frac{\mathbf{p}_{c}}{||\mathbf{p}_{c}||}d,\quad\mathbf{p}_{c}=\mathcal{K}^{-1}\mathbf{p}.$
(1)

Similarly, $\mathbf{P}$ can be re-projected into view $I^{\prime}$ as:

$\displaystyle\mathbf{p}=\mathcal{K^{\prime}}\frac{\mathbf{P}{{}^{{}^{\prime}}}}{\mathbf{P}{{}^{{}^{\prime}}_{z}}},\quad\mathbf{P}{{}^{{}^{\prime}}}=[R^{\prime
T}|-R^{\prime T}\mathbf{t^{\prime}}]\mathbf{P}.$ (2)

Assuming that reconstructed depths $d$ are accurate, the above equations
enable self-supervising a point detector, by designing a loss function that
promotes a set of re-projected points $\mathbf{p}^{\prime}$ detected in view
$I$ to align with points independently detected on view $I^{\prime}$. However,
many distinctive interest points lie on scene edges at a depth discontinuity.
In this case any small deviation in $\mathbf{p}$ can result in large $d$
variations corresponding to background and foreground depths. To make re-
projection stable around edges, we focus on matching foreground objects only
(minimising potential view occlusions). This can be achieved in a very simple
manner by computing $d$ as the minimum depth within a $5\times 5$ pixel window
centred in $\mathbf{p}$ (see example in Fig. 3).

(a) Input Image

(b) PrP w/o Depth Window

(c) PrP w/ Depth Window

Figure 3: Depth window estimation. The interest points, depicted in red and
purple situated at the painting’s frame in Fig. 3(a), exhibit misprojection
onto image $I^{{}^{\prime}}$ without the depth window estimation method.
However, misprojection errors are effectively mitigated by utilising the depth
window estimation method as seen in Fig. 3(c).

##  4 Implementations

###  4.1 SiLK-PrP

Self-supervised point detection methods that are trained end-to-end can be
easily adapted to utilise NeRF training data. Using SiLK [14] as a baseline,
we propose an adapted version that, during training, simply replaces
homography warpings with NeRF renderings and point re-projection, while
keeping the rest of the training pipeline and loss functions intact.

In the original SiLK approach, image pairs with ground-truth bi-directional
correspondences are generated on-the-fly during training with homography
warpings. We rely on our pre-rendered NeRF dataset described in section 3.1,
to avoid significant computational overhead caused by on-the-fly image
rendering. During training we randomly sample image pairs $I,I^{{}^{\prime}}$
from the same scene, Eqs. 1 and 2 are then utilised during training to
generate dense ground-truth point correspondences, discretised at pixel level.

###  4.2 SuperPoint-PrP

SuperPoint is a self-supervised method that goes through multiple stages of
training. Firstly, an encoder with a detector head is trained in a fully
supervised manner to detect corners on a synthetic dataset with simple
geometric shapes. The pre-trained detector, called MagicPoint, is used to
generate pseudo-ground truth interest points on a real dataset (_e.g._ MS-
COCO) using a process called homographic adaptation. This involves generating
several homography warped copies of a training image, passing them through the
trained detector, unwarping all predictions back to the original training
image, and finally aggregating all unwarped predictions to generate the pseudo
label. This process can be repeated several times, where after each round of
homographic adaptation, the detector head is further trained, updating the
pseudo-ground truth labels, while improving the detector head
generalisability. After a final round of homographic adaptation, a descriptor
head is added to the model, which is jointly trained with the whole model with
an additional descriptor loss term on the real dataset resulting in the
SuperPoint model.

With the NeRF training data approach, we use the same architecture of the
original SuperPoint model. However, modify the homographic adaptation process
as well as the descriptor loss to handle our NeRF training data and Point Re-
Projection process.

####  4.2.1 Projective Adaptation

Similarly to our SiLK-PrP approach, we consider pre-rendered NeRF images as
training data. In this case, we exploit the fact that training sequences for
each scene are are generated along a continuous trajectory.

We randomly sample from a window of 20 consecutive rendered NeRF images
$I_{i\rightarrow i+19}$. We take pseudo ground-truth points for initial image
$I_{i}$ and random sample 15 of the remaining images as the warped versions of
$I_{i}$.

The probability heatmap $\mathbf{{H}_{i}}$ for image $I_{i}$ is then computed,
followed by randomly extracting 15 images out of the subsequent images. For
each of the randomly selected image $I_{r}$, their probability heatmap
$\mathbf{{H}_{r}}$ is computed, and interest points $\mathbf{{p}_{r}}$ are
extracted from each heatmap, while using non-maximum suppression to filter out
feature points that are too close to each other.

Once interest points $\mathbf{{p}_{r}}$ are obtained, the PrP process is
applied to project the interest points aligning them with the initial frame
${I}_{i}$, for clarity the projected interest points can be denoted as
$\mathbf{{p}^{{}^{\prime}}_{i}}$.

Having obtained the projected interest points
$\mathbf{{p}^{{}^{\prime}}_{i}}$, we generate a mask denoted as
$\mathbf{{B}_{i^{\prime}}}$, for each interest point in
$\mathbf{{p}^{{}^{\prime}}_{i}}$ coordinates on the binary mask, a 3x3 patch
centred around the interest point before projection is extracted from the
probability heatmap $\mathbf{{H}_{r}}$, the patch is then applied on the
binary heatmap $\mathbf{{B}_{i^{\prime}}}$. Thus, mask
$\mathbf{{B}_{i^{\prime}}}$, serves as the probability heatmap of rendered
image $I_{r}$ projected onto the input render frame ${I}_{i}$.

Finally, the input’s rendered image probability heatmap $\mathbf{{H}_{i}}$ and
mask $\mathbf{{B}_{i^{\prime}}}$ for each frame, are aggregated together, non-
maximum suppression is applied once more to the final aggregated heatmap to
obtain pseudo ground-truth interest points for the input rendered image
$I_{i}$.

####  4.2.2 Descriptor Loss

SuperPoint’s [6] loss consists of an interest point detection loss
$\mathcal{L}_{p}$ and the other for the descriptor $\mathcal{L}_{d}$.

The interest point detection loss is a cross entropy loss between the interest
point detector prediction $\mathcal{X}$ on the input image, and the ground
truth interest points $\mathcal{Y}$. We effectively do not apply any
modifications to the interest point detection loss.

SuperPoint’s [6] descriptor loss $\mathcal{L}_{d}$, is a hinge loss between
every pair of descriptors cells, $d_{h_{w}}\in\mathcal{D}$ from $\mathcal{I}$
and $d_{{h^{\prime}}{w^{\prime}}}\in\mathcal{D}^{{}^{\prime}}$ from
$\mathcal{I}^{{}^{\prime}}$. "Cells" are essentially pixels in lower
dimension, where for an input image of size $H\times W$, a cell has a
dimension of $H_{c}=\frac{H}{8}$ $\times$ $W=\frac{W}{8}$. In simpler terms,
each pixel in the encoded image maps an $8\times 8$ region of the input image.
SuperPoint’s descriptor loss can be defined as:

$\displaystyle\mathcal{L}_{d}(D,D^{\prime},S)=\frac{1}{(H_{c}W_{c})^{2}}\sum^{H_{c},W_{c}}_{\begin{subarray}{c}h=1\\\
w=1\end{subarray}}\sum^{H_{c},W_{c}}_{\begin{subarray}{c}h^{\prime}=1\\\
w^{\prime}=1\end{subarray}}l_{d}(d_{h_{w}},d^{\prime}_{{h^{\prime}}{w^{\prime}}};s_{hw{h^{\prime}}{w^{\prime}}})$
(3)

where $S$ is a homography induced correspondence, defined as:

$\displaystyle
s_{hw{h^{\prime}}{w^{\prime}}}=\left\\{\begin{array}[]{ll}1,&\mbox{if
}||\widehat{\mathcal{H}p_{hw}}-p_{{h^{\prime}}{w^{\prime}}}||\leq 8.\\\
0,&\mbox{otherwise}\end{array}\right.$ (6)

where $p_{hw}$ is the centre pixel in the $(h,w)$ cell, and
$\widehat{\mathcal{H}p_{hw}}$ is multiplying the centre pixel cell $p_{hw}$ by
a homography $\mathcal{H}$ which relates the input image and its warped pair.

We reformulate the homography induced correspondence of SuperPoint [6], in
such a way that centre pixel $p_{hw}$ can be transformed using the PrP process
rather than transforming it through homography $\mathcal{H}$. Let
$\mathcal{C}(\cdot)$ represent a function that takes $p_{hw}$ as an input,
along with all parameters utilised in the PrP process (see Eqs. 1 and 2), for
simplicity, these parameters can be combined and denoted as $M$. Therefore,
the PrP induced correspondence can be defined as:

$\displaystyle
s_{hw{h^{\prime}}{w^{\prime}}}=\left\\{\begin{array}[]{ll}1,&\mbox{if
}||\mathcal{C}(M;p_{hw})-p_{{h^{\prime}}{w^{\prime}}}||\leq 8.\\\
0,&\mbox{otherwise}\end{array}\right.$ (9)

###  4.3 Implementation details

For all experiments, we present results using our own training pipeline for
both SuperPoint-PrP and SiLK-PrP. Additionally, we report results for the
baseline SuperPoint[6] model using our own reproduced implementation.

To mitigate misprojection errors during the PrP process, we employ a random
sampling approach for selecting image $I^{{}^{\prime}}$ from a subset of the
training data. This subset is constrained within a range determined by a lower
limit, $\lambda_{l}$, and an upper limit, $\lambda_{u}$, based on the input
image $I$. Specifically, we set $\lambda_{l}$ to 70 and $\lambda_{u}$ to 150,
ensuring a balance between viewpoint changes while avoiding projection errors.
Moreover, the depth rectification process threshold $\epsilon$ was set as
$3e^{-2}$ meters.

Lastly, for SuperPoint-PrP and SiLK-PrP training, we apply the same
photometric augmentation applied on the baseline models [6, 14] such as random
brightness and contrast, Gaussian noise, motion blur, but do not apply
homographic augmentation such as rotation, scaling or translation. Lastly, all
models are trained using PyTorch [38] and a single NVIDIA RTX 3090 Ti GPU.

####  4.3.1 SiLK-PrP training.

SiLK-PrP is trained on the NeRF dataset in an end-to-end manner for a total of
100,000 iterations. We keep the same SiLK hyperparameters when training SiLK-
PrP with ADAM optimiser, a learning rate of $1e^{-4}$ and betas as (0.9,
0.999), while the batch size was set as 1.

####  4.3.2 SuperPoint-PrP training.

Similar to SuperPoint’s baseline training pipeline, we obtain the base
MagicPoint model by conducting training on the Synthetic Shapes dataset for
200,000 iterations. The MagicPoint model is identical to the SuperPoint model
omitting the descriptor head, while the Synthetic Shapes dataset consists of
synthetic 2D geometrical shapes such as lines, triangles and cubes.

To produce the MagicPoint-PrP model, we further train the base MagicPoint
model through two rounds of Projective Adaptation on the NeRF dataset, each
lasting 30,000 iterations. Finally, the SuperPoint-PrP model is obtained by
training both the detector and descriptor heads of MagicPoint-PrP once again
on the NeRF dataset for a total of 300,000 iterations. The training process is
performed using the baseline SuperPoint model hyperparameters with ADAM
optimiser, a learning rate of $1e^{-3}$ and betas set to (0.9, 0.999). The
batch size is set at 32 for the MagicPoint-PrP model and 2 for the SuperPoint-
PrP model.

##  5 Experiments

In this section, we assess the performance of the PrP-trained interest point
detector and descriptor models in comparison to their corresponding baseline
models. The evaluation encompasses accessing the interest point detection
quality and homography estimation on the HPatches dataset [39]. Additionally,
on ScanNet [18], we evaluate the indoor relative pose estimation and point
cloud registration capabilities of each model. Lastly, we evaluate the outdoor
relative pose estimation of each model using the YFCC100M [40] dataset. For
all evaluations, we perform feature matching using mutual nearest neighbour.
Through the aforementioned experiments, we analyse the following:

  * •

Our NeRF dataset is approximately $\frac{1}{30}$ the size of the MS-COCO
dataset [17], and $\frac{1}{250}$ the size of ScanNet’s [18]. Despite the
relatively small scale of our training dataset, we analyse the
generalisability of the NeRF-trained models on real-world data and numerous
conditions across multiple datasets. Surprisingly, the PrP-models trained on
our NeRF datasets consistently demonstrate comparable results in most
evaluations. This poses the question of the necessity for large open-source
datasets in training learning-based detectors and descriptors.

  * •

Unlike [17, 18, 19, 41], our NeRF dataset fully consists of synthetic images
of real scenes. Thus, we assess the interest point detection quality of the
NeRF-trained detectors relative to their corresponding baseline models.
Conducted experiments revealed that the quality of interest points generated
by NeRF-trained models remained consistently high, showing no degradation
compared to the baseline models.

  * •

Lastly, as we explicitly render each scene in our NeRF dataset from various
viewpoints, simulating real camera motion instead of relying on homography to
mimic camera motions, we explore whether models supervised by the PrP process
during training demonstrate improvement in their relative pose estimation
capabilities.

###  5.1 Homography Estimation

Following [23, 6, 22, 14, 7] we evaluate homography estimation on the HPatches
dataset [39]. The HPatches dataset consist of 57 scenes with varying
illumination, and 59 scenes with large viewpoint variation. Each scene
contains 6 images, related by ground truth homographies.

####  5.1.1 Evaluation Protocol.

We follow [23, 14] are resize the shorter image side to a dimension of 480. To
evaluate interest point detection quality and performance, we report the
repeatability metric as in [6, 14]. Moreover, we evaluate the keypoint
descriptors, we report the mean matching accuracy (MMA) _i.e._ the ratio of
matches where reprojection error is below a specified threshold [20]; and
matching score _i.e._ The ratio between correctly matched points and total
number of detected points. Lastly, we use OpenCV with RANSAC algorithim to
estimate the homography between an image pair, and report the homography
estimation accuracy and area under the curve (AUC) error over all image pairs.
We keep the top 1k detected interest points for SuperPoint and 10k for SiLK.

####  5.1.2 Baselines.

We assess SuperPoint-PrP by comparing it to the baseline SuperPoint model [6].
Additionally, we introduce a hybrid variant of SuperPoint-PrP denoted as "SP-
PrP-Hyb". This variant incorporates the MagicPoint model, trained through two
rounds of Homographic Adaptation, followed by training SuperPoint model on the
NeRF dataset using our PrP process and Projective Adaptation. Furthermore, we
present two versions of PrP SiLK models; the first, SiLK-PrP, is trained
without rotation or scaling augmentation as detailed in Sec. 4.1. The second,
SiLK-PrP-Aug, includes rotation and scaling during training.

####  5.1.3 Results.

As seen in Tab. 1, the PrP trained models are outperformed by their baseline
counterparts [14, 6] in all of the HPatches metrics. Nonetheless, this outcome
is expected as both SiLK-PrP and SuperPoint-PrP are not rotation or scale
invariant, in contrast to their corresponding baseline models. On the other
hand, despite SiLK-PrP-Aug’s efforts to incorporate in-place augmentation for
rotation and scale invariance (see Fig. 4), it falls short compared to the
SiLK model.

Tab. 2 further delves into SiLK-PrP-Aug’s homography estimation performance,
revealing its homography estimation aligns with SiLK in scenes with varying
viewpoints. However, a dip in performance is observed in scenes with varying
illumination conditions. This discrepancy contributes to the observed
performance gap between SiLK-PrP-Aug and SiLK [14] in Tab. 1. We opted to use
the same photometric augmentation used for SiLK [14] when training SiLK-PrP
and SiLK-PrP-Aug, however, optimising photometric augmentation parameters
during training SiLK-PrP and SiLK-PrP-Aug on the NeRF dataset might offer
improved results.

Table 1: HPatches Metrics. Both baseline SiLK [14] and SuperPoint [6], surpass SiLK-PrP and SuperPoint-PrP on HPatches metrics. | Rep. $\uparrow$ | Hom. Est. Acc. $\uparrow$ | Hom. Est. AUC $\uparrow$ | MMA $\uparrow$ | MS $\uparrow$
---|---|---|---|---|---
|  $\epsilon$ = 3 |  $\epsilon$ = 5 |  $\epsilon$ = 3 |  $\epsilon$ = 5 |  $\epsilon$ = 3 |  $\epsilon$ = 5 |  $\epsilon$ = 3 |  $\epsilon$ = 5 |
SP | 0.60 | 0.70 | 0.80 | 0.86 | 0.53 | 0.65 | 0.61 | 0.65 | 0.46
SP-PrP-Hyb | 0.56 | 0.66 | 0.76 | 0.85 | 0.50 | 0.62 | 0.60 | 0.66 | 0.45
SP-PrP | 0.54 | 0.67 | 0.73 | 0.82 | 0.47 | 0.60 | 0.58 | 0.66 | 0.42
SiLK | 0.77 | 0.85 | 0.84 | 0.90 | 0.65 | 0.74 | 0.66 | 0.67 | 0.38
SiLK-PrP | 0.72 | 0.80 | 0.75 | 0.81 | 0.54 | 0.64 | 0.62 | 0.63 | 0.31
SiLK-PrP-Aug | 0.74 | 0.82 | 0.78 | 0.85 | 0.56 | 0.67 | 0.60 | 0.61 | 0.32
Table 2: HPatches metrics separated. SiLK-PrP and SiLK are similar in estimating homography for scenes with varying viewpoints, while SiLK-PrP falls behind in varying illumination scenes. | HPatches Viewpoint | HPatches Illumination
---|---|---
| Rep. $\uparrow$ | Hom. Est. Acc. $\uparrow$ | Rep. $\uparrow$ | Hom. Est. Acc. $\uparrow$
|  $\epsilon$ = 3 |  $\epsilon$ = 5 |  $\epsilon$ = 3 |  $\epsilon$ = 5 |  $\epsilon$ = 3 |  $\epsilon$ = 5 |  $\epsilon$ = 3 |  $\epsilon$ = 5
SiLK | 0.79 | 0.86 | 0.73 | 0.84 | 0.75 | 0.84 | 0.95 | 0.96
SiLK-PrP-Aug | 0.79 | 0.87 | 0.71 | 0.81 | 0.69 | 0.77 | 0.87 | 0.89

(a) SiLK [14]

(b) SiLK-PrP

(c) SiLK-PrP-Aug

Figure 4: Rotation and Scale invariance. In Fig. 4(b), it is evident that
SilK-PrP lacks rotation and scale invariance compared to SiLK (see Fig. 4(a))
[14]. Incorporating rotation and scaling SiLK-PrP’s training process, rotation
and scale invariance is achieved (see Fig. 4(c)).

###  5.2 Relative Pose Estimation

For evaluating the relative pose estimation on indoor scenes, we leverage the
ScanNet dataset [18]. We employ the 1500 test image pairs provided by [22].

To assess pose estimation on outdoor scenes, we employ the YFCC100M dataset
[40]. We utilise the YFCC100M test split introduced by [22]. This split
samples 1000 image pairs from four scenes each, thus, the total number of
image pairs for the outdoor pose evaluation is 4000 image pairs.

####  5.2.1 Evaluation Protocol.

For indoor relative pose evaluation, we follow [23, 22] and resize ScanNet’s
images to a size of [640 $\times$ 480]. For SuperPoint, we use the top 1k
predicted points and 20k for SiLK.

Additionally, for outdoor relative pose evaluation, we resize the YFCC100M
images so that the longest dimension is equal to 1200. We use the top 2k
predicted points for SuperPoint and 20k for SiLK.

For both indoor and outdoor pose evaluations, we report the pose error AUC at
thresholds ($5^{\circ}$, $10^{\circ}$, $20^{\circ}$), where the pose error is
the maximum between the angular rotation error and angular translation error.
Furthermore, we compute the essential matrix using the matched interest points
using OpenCV with a RANSAC threshold of 0.5 over the mean of the focal length.

####  5.2.2 Baselines.

Similar to the Homography Estimation evaluation Sec. 5.1, we compare
SuperPoint-PrP and SuperPoint-PrP-Hyb with the baseline SuperPoint model.
Likewise, we assess SiLK-PrP and SiLK-PrP-Aug against the baseline SiLK model.

####  5.2.3 Indoor Pose Estimation Results.

As depicted in Tab. 3, the PrP models consistently surpass their respective
baseline counterparts across all angular pose error thresholds. Although
performance enhancement for SuperPoint-PrP and SuperPoint-PrP-Hyb over
SuperPoint [6] is marginal, a more substantial improvement is evident in the
case of SiLK-PrP and SiLK-PrP-Aug in comparison to SiLK [14], particularly as
the angular threshold increases.

Unfortunately, the relative pose estimation computes the translation error as
the angular translation error between the ground truth and estimated
translation vectors. The computed angular translation error suffers from
instability as it could only be computed up to a certain scale factor [42,
43]. This instability could be seen in Fig. 5, where it is evident that the
angular translation error is unstable when the norm of the ground truth
relative translation vector ($||t_{GT}||$) is approximately 0.2 and below.

To address this issue, we refined the indoor relative pose estimation
assessment. For scenes where $||t_{GT}||$ is below the threshold $\epsilon$
set at 0.15, we exclusively report the pose error AUC based on angular
rotation error. Furthermore, for scenes where the threshold $\epsilon$ exceeds
0.15, we report the pose error AUC as the maximum value between angular
rotation error and angular translation error. As presented in Tab. 4, it is
evident that all models excel in estimating relative rotation in scenes
characterised by minimal translation between viewpoints. Notably, the PrP
models exhibit a slight performance advantage over their baseline counterparts
in this specific scenario. Additionally, in scenes where the ground truth
relative translation exceeds the defined threshold, further improvement can be
observed for the PrP models compared to results reported in Tab. 3.

Figure 5: Instability in Angular Translation Error. Density plot illustrating
how the angular translation error is unstable in situations where the ground
truth relative translation between two camera viewpoints is minimal.

Table 3: Indoor Relative Pose Estimation Error AUC. The models trained using
the PrP process outperform their respective baseline models [14, 6] in
accurately estimating relative poses within indoor scenes for all reported
angular thresholds.

Pose Est. AUC | @$5^{\circ}$ $\uparrow$ | @$10^{\circ}$ $\uparrow$ | @$20^{\circ}$ $\uparrow$
---|---|---|---
SP | 4.95 | 11.90 | 21.67
SP-PrP-Hyb | 5.78 | 13.52 | 24.74
SP-PrP | 5.20 | 12.53 | 22.82
SiLK | 7.18 | 16.14 | 27.84
SiLK-PrP | 9.42 | 20.17 | 32.31
SiLK-PrP-Aug | 9.60 | 20.75 | 34.24

Table 4: Separated Indoor Relative Pose Estimation Error AUC. For image pairs with minimal relative translation in viewpoints, the PrP models exhibit superior performance in accurately estimating the relative rotation, especially the SuperPoint-PrP and SiLK-PrP-Aug models. Pose Est. AUC |  | Pose Est. AUC $||t_{gt}||\leq\epsilon$
---
Rot.,
157 image pairs
| Pose Est. AUC $||t_{gt}||>\epsilon$
---
$\max{(\text{Rot.},\text{Transl.})}$,
1363 image pairs
| @$5^{\circ}\uparrow$ | @$10^{\circ}\uparrow$ | @$20^{\circ}\uparrow$ | @$5^{\circ}\uparrow$ | @$10^{\circ}\uparrow$ | @$20^{\circ}\uparrow$
SP | 50.34 | 60.87 | 82.58 | 5.21 | 12.19 | 22.19
SP-PrP-Hyb | 47.56 | 67.95 | 81.27 | 6.14 | 14.22 | 25.45
SP-PrP | 51.21 | 69.67 | 82.36 | 5.42 | 12.94 | 23.14
SiLK | 54.33 | 73.02 | 85.24 | 7.62 | 17.02 | 28.82
SiLK-PrP | 57.59 | 74.19 | 85.28 | 10.22 | 21.53 | 33.52
SiLK-PrP-Aug | 57.60 | 75.54 | 86.38 | 10.22 | 21.79 | 35.07

####  5.2.4 Outdoor Pose Estimation Results.

Similar to results obtained in Tab. 3, the PrP models surpass their
corresponding baseline models in relative pose estimation for outdoor scenes,
as observed by results shown in Tab. 5. The results reported in Tabs. 3 to 5
collectively highlight that models subjected to PrP supervision during
training demonstrate significant improvement in their relative pose estimation
capability.

Table 5: Outdoor Relative Pose Estimation Error AUC. The PrP trained models exhibit superior performance compared to their baseline counterparts [14, 6] in estimating relative poses for outdoor scenes, similar to observed results in Tab. 3 Pose Est. AUC | @$5^{\circ}$ $\uparrow$ | @$10^{\circ}$ $\uparrow$ | @$20^{\circ}$ $\uparrow$
---|---|---|---
SP | 6.97 | 14.89 | 25.96
SP-PrP-Hyb | 9.58 | 19.62 | 32.32
SP-PrP | 9.02 | 18.27 | 30.33
SiLK | 7.82 | 15.05 | 24.93
SiLK-PrP | 9.44 | 17.31 | 27.04
SiLK-PrP-Aug | 7.82 | 14.33 | 22.52

###  5.3 Pairwise Point Cloud Registration

We follow recent work [44, 14] and evaluate the interest point detectors and
descriptors on a 3D point cloud registration task. Evaluation is performed on
ScanNet’s [18] official test split, where image pairs are sampled 20 frames
apart.

####  5.3.1 Evaluation Protocol.

The point cloud registration evaluation is conducted by resizing the ScanNet
[18] images to dimensions of [128 $\times$ 128]. Given a pair of RGB-D images,
a 6-DOF pose is estimated that best aligns the first image to the second
image. The metrics reported during evaluation are angular rotation and
translation errors in degrees and centimetres respectively. Moreover, the
chamfer distance is also reported between the ground-truth point cloud and the
reconstructed point cloud in centimetres. For all three reported metrics the
accuracy, mean and median errors over the entire dataset are reported at
different thresholds [44].

####  5.3.2 Baselines.

Unlike previous evaluations (Secs. 5.1 and 5.2) where sparse feature matching
was employed, we follow [44, 14] and perform the point cloud registration
using dense feature matching. The ratio-test is utilised to find
correspondences and alignment is seen as Procustes problem solved by a
weighted Kabsch’s algorithim. [45, 46, 14].

####  5.3.3 Results.

As depicted in Tab. 6, both SuperPoint-PrP and SuperPoint-PrP-Hyb consistently
outperform the baseline SuperPoint model across all metrics by a small margin,
except for translation at a 5 cm threshold. Moreover, SilK generally maintains
a 1-3$\%$ advantage over SiLK-PrP and SiLK-PrP-Aug, except at a 5 cm
translation threshold, where it stands at 5$\%$; however, the SiLK PrP trained
models remain competitive with the baseline SiLK model.

It is essential to highlight that SiLK is trained on a relatively small image
size of [164x164], whereas the NeRF dataset comprises larger images [640x480].
Since the point cloud registration is conducted on an image size of [128x128],
it is expected that SiLK outperforms SiLK-PrP at lower image dimensions,
however, results also indicate that the SiLK PrP trained models could detect
high quality interest points at low resolutions.

Table 6: Point Cloud Reg. SuperPoint-PrP and SuperPoint-PrP demonstrate
marginal enhancements over the baseline SuperPoint model. Meanwhile, SiLK-PrP
and SiLK-PrP-Aug maintain competitiveness with the baseline SiLK model.

| Rot. | Transl. | Chamfer
---|---|---|---
| Acc. $\uparrow$ | Err. $\downarrow$ | Acc. $\uparrow$ | Err. $\downarrow$ | Acc. $\uparrow$ | Err. $\downarrow$
| $5^{\circ}$ | $10^{\circ}$ | $45^{\circ}$ | M. | Med. | 5 | 10 | 25 | M. | Med. | 1 | 5 | 10 | M. | Med.
SP | 79.8 | 90.3 | 98.1 | 5.7 | 2.2 | 45.4 | 70.6 | 89.0 | 13.5 | 5.6 | 62.9 | 84.4 | 89.5 | 9.4 | 0.5
SP-PrP-Hyb | 82.0 | 92.0 | 98.5 | 5.2 | 2.2 | 44.0 | 72.5 | 91.6 | 12.0 | 5.7 | 64.4 | 87.1 | 91.9 | 7.0 | 0.5
SP-PrP | 81.4 | 91.9 | 98.4 | 5.2 | 2.3 | 43.3 | 71.6 | 91.5 | 12.1 | 5.8 | 63.9 | 87.1 | 91.9 | 7.9 | 0.5
SiLK | 95.6 | 97.5 | 99.1 | 2.6 | 0.8 | 79.9 | 91.4 | 96.9 | 6.1 | 2.2 | 89.4 | 95.8 | 97.1 | 4.7 | 0.1
SilK-PrP | 93.0 | 96.2 | 98.7 | 3.4 | 1.0 | 71.3 | 87.0 | 95.3 | 8.1 | 2.8 | 84.3 | 93.7 | 95.7 | 5.7 | 0.1
SiLK-PrP-A. | 94.6 | 97.2 | 99.0 | 2.8 | 0.9 | 75.0 | 89.5 | 96.4 | 7.0 | 2.5 | 86.8 | 95.1 | 96.7 | 5.4 | 0.1

##  6 Conclusion

This paper introduces a novel approach to supervise learning-based feature
point detectors and descriptors, leveraging perspective projective geometry on
synthesised NeRF data. Despite the fact that our proposed dataset is
exclusively composed of synthetic images rather than real RGB images, and is
comparatively smaller than large open-source datasets, results indicated there
was no observed decrease in generalisability or feature point detection
quality. As expected our models generally outperform homography-trained
baselines on multi-view benchmarks with highly non-planar scenes, while
slightly underperforming on a homography estimation benchmark. The larger
potential for further advancements lies in improving the quality of training
data with advances in neural rendering that can generate higher quality
synthesised images devoid of artefacts, and most importantly more precise
depth maps to avoid misprojection errors.

## References

  * [1] Onur Ozyesil, Vladislav Voroninski, Ronen Basri, and Amit Singer.  A survey of structure from motion, 2017.
  * [2] Takafumi Taketomi, Hideaki Uchiyama, and Sei Ikeda.  Visual slam algorithms: a survey from 2010 to 2016.  IPSJ Transactions on Computer Vision and Applications, 9(1):16, Jun 2017.
  * [3] Iman Abaspur Kazerouni, Luke Fitzgerald, Gerard Dooly, and Daniel Toal.  A survey of state-of-the-art on visual slam.  Expert Systems with Applications, 205:117734, 2022.
  * [4] Stephanie Lowry, Niko Sünderhauf, Paul Newman, John J. Leonard, David Cox, Peter Corke, and Michael J. Milford.  Visual place recognition: A survey.  IEEE Transactions on Robotics, 32(1):1–19, 2016.
  * [5] Xiwu Zhang, Lei Wang, and Yan Su.  Visual place recognition: A survey from deep learning perspective.  Pattern Recognition, 113:107760, 2021.
  * [6] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superpoint: Self-supervised interest point detection and description.  CoRR, abs/1712.07629, 2017.
  * [7] Jerome Revaud, Philippe Weinzaepfel, César De Souza, Noe Pion, Gabriela Csurka, Yohann Cabon, and Martin Humenberger.  R2d2: Repeatable and reliable detector and descriptor.  arXiv preprint arXiv:1906.06195, 2019.
  * [8] Xiaoming Zhao, Xingming Wu, Jinyu Miao, Weihai Chen, Peter C. Y. Chen, and Zhengguo Li.  Alike: Accurate and lightweight keypoint detection and descriptor extraction.  IEEE Transactions on Multimedia, 25:3101–3112, 2023.
  * [9] Jiexiong Tang, Ludvig Ericson, John Folkesson, and Patric Jensfelt.  Gcnv2: Efficient correspondence prediction for real-time slam.  IEEE Robotics and Automation Letters, 4(4):3505–3512, 2019.
  * [10] Guangqiang Li, Lei Yu, and Shumin Fei.  A deep-learning real-time visual slam system based on multi-task feature extraction network and self-supervised feature points.  Measurement, 168:108403, 2021.
  * [11] Hudson Martins Silva Bruno and Esther Luna Colombini.  Lift-slam: A deep-learning feature-based monocular visual slam method.  Neurocomputing, 455:97–110, September 2021.
  * [12] Aji Resindra Widya, Akihiko Torii, and Masatoshi Okutomi.  Structure-from-motion using dense cnn features with keypoint relocalization, 2018.
  * [13] Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David Mcallister, Justin Kerr, and Angjoo Kanazawa.  Nerfstudio: A modular framework for neural radiance field development.  In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH ’23, New York, NY, USA, 2023. Association for Computing Machinery.
  * [14] Pierre Gleize, Weiyao Wang, and Matt Feiszli.  Silk: Simple learned keypoints.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22499–22508, October 2023.
  * [15] David G. Lowe.  Distinctive Image Features from Scale-Invariant Keypoints.  International Journal of Computer Vision, 60(2):91–110, November 2004.
  * [16] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski.  Orb: An efficient alternative to sift or surf.  In 2011 International Conference on Computer Vision, pages 2564–2571, 2011.
  * [17] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft coco: Common objects in context.  In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, pages 740–755, Cham, 2014. Springer International Publishing.
  * [18] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas A. Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  CoRR, abs/1702.04405, 2017.
  * [19] Zhengqi Li and Noah Snavely.  Megadepth: Learning single-view depth prediction from internet photos.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2041–2050, 2018.
  * [20] Michal J. Tyszkiewicz, Pascal Fua, and Eduard Trulls.  DISK: learning local features with policy gradient.  CoRR, abs/2006.13566, 2020.
  * [21] Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler.  D2-net: A trainable cnn for joint description and detection of local features.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8084–8093, 2019.
  * [22] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4937–4946, 2020.
  * [23] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou.  Loftr: Detector-free local feature matching with transformers.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8918–8927, 2021.
  * [24] Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi.  Cotr: Correspondence transformer for matching across images.  In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6187–6197, 2021.
  * [25] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  ACM Trans. Graph., 41(4):102:1–102:15, July 2022.
  * [26] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  CoRR, abs/2103.13415, 2021.
  * [27] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman.  Zip-nerf: Anti-aliased grid-based neural radiance fields.  In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 19640–19648, Los Alamitos, CA, USA, oct 2023. IEEE Computer Society.
  * [28] Casimir Feldmann, Niall Siegenheim, Nikolas Hars, Lovro Rabuzin, Mert Ertugrul, Luca Wolfart, Marc Pollefeys, Zuria Bauer, and Martin R. Oswald.  Nerfmentation: Nerf-based augmentation for monocular depth estimation, 2024.
  * [29] Lin Yen-Chen, Pete Florence, Jonathan T. Barron, Tsung-Yi Lin, Alberto Rodriguez, and Phillip Isola.  Nerf-supervision: Learning dense object descriptors from neural radiance fields.  In 2022 International Conference on Robotics and Automation (ICRA), page 6496–6503. IEEE Press, 2022.
  * [30] F. Tosi, A. Tonioni, D. De Gregorio, and M. Poggi.  Nerf-supervised deep stereo.  In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 855–866, Los Alamitos, CA, USA, jun 2023. IEEE Computer Society.
  * [31] Han Ling.  Adfactory: An effective framework for generalizing optical flow with nerf, 2023.
  * [32] Benedikt Bitterli.  Rendering resources, 2016.  https://benedikt-bitterli.me/resources/.
  * [33] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5460–5469, 2022.
  * [34] Johannes L. Schonberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [35] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7206–7215, 2021.
  * [36] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  ACM Trans. Graph., 41(4):102:1–102:15, July 2022.
  * [37] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu.  NeRF$--$: Neural radiance fields without known camera parameters.  arXiv preprint arXiv:2102.07064, 2021.
  * [38] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  CoRR, abs/1912.01703, 2019.
  * [39] Vassileios Balntas, Karel Lenc, Andrea Vedaldi, and Krystian Mikolajczyk.  Hpatches: A benchmark and evaluation of handcrafted and learned local descriptors.  In CVPR, 2017.
  * [40] Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  The new data and new challenges in multimedia research.  CoRR, abs/1503.01817, 2015.
  * [41] Torsten Sattler, Will Maddern, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Marc Pollefeys, Josef Sivic, Fredrik Kahl, and Tomas Pajdla.  Benchmarking 6dof outdoor visual localization in changing conditions.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8601–8610, 2018.
  * [42] R. I. Hartley and A. Zisserman.  Multiple View Geometry in Computer Vision.  Cambridge University Press, ISBN: 0521540518, second edition, 2004.
  * [43] Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua.  Learning to find good correspondences.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2666–2674, 2018.
  * [44] Mohamed El Banani, Luya Gao, and Justin Johnson.  UnsupervisedR&R: Unsupervised Point Cloud Registration via Differentiable Rendering.  In CVPR, 2021.
  * [45] W. Kabsch.  A solution for the best rotation to relate two sets of vectors.  Acta Crystallographica Section A, 32(5):922–923, Sep 1976.
  * [46] Christopher Choy, Wei Dong, and Vladlen Koltun.  Deep global registration.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2511–2520, 2020.