# Testing Cluster Membership of Planetary Nebulae with High-Precision Proper
Motions. I.
HST Observations of JaFu 1 Near the Globular Cluster Palomar 6

Howard E. Bond Department of Astronomy & Astrophysics, Pennsylvania State
University, University Park, PA 16802, USA  Space Telescope Science Institute,
3700 San Martin Dr., Baltimore, MD 21218, USA  Andrea Bellini Space Telescope
Science Institute, 3700 San Martin Dr., Baltimore, MD 21218, USA  Kailash C.
Sahu Space Telescope Science Institute, 3700 San Martin Dr., Baltimore, MD
21218, USA  Eureka Scientific Inc., 2542 Delmar Avenue, Suite 100, Oakland, CA
94602, USA Howard E. Bond<EMAIL_ADDRESS>
###### Abstract

If a planetary nebula (PN) is shown to be a member of a star cluster, we
obtain important new constraints on the mass and chemical composition of the
PN’s progenitor star, which cannot be determined for PNe in the field. Cluster
membership can be tested by requiring the projected separation between the PN
and cluster to be within the tidal radius of the cluster, and the objects to
have nearly identical radial velocities (RVs) and interstellar extinctions,
and nearly identical proper motions (PMs). In an earlier study, we used PMs to
confirm that three PNe, which had already passed the other tests, are highly
likely to be members of Galactic globular clusters (GCs). For a fourth object,
the PN JaFu 1, which lies in the Galactic bulge near the GC Palomar 6 on the
sky and has a similar RV, the available PM measurement gave equivocal results.
We have now obtained new high-resolution images of the central star of JaFu 1
with the Hubble Space Telescope (HST) which, combined with archival HST frames
taken 14 and 16 years earlier, provide a high-precision PM. Unfortunately, we
find that the PM of the central star differs from that of the cluster with
high statistical significance, and thus is unlikely to be a member of Palomar
6. Nevertheless, JaFu 1 is of astrophysical interest because its nucleus
appears to be a member of the rare class of “EGB 6-type” central stars, which
are associated with compact emission-line knots.

##  1 Introduction: Planetary Nebulae in Open and Globular Star Clusters

Planetary nebulae (PNe) mark the final, rapid transition of a low- or
intermediate-mass star from the asymptotic giant branch (AGB) to the top of
the white-dwarf cooling sequence. PNe provide information on such topics as
late stages of stellar evolution, stellar nucleosynthesis, mixing processes
that bring processed material to the surface, and the ejection of these
elements into the interstellar medium. For overviews of PNe see, for example,
Kwitter et al. (2014), Weidmann et al. (2020), and Kwitter & Henry (2022).

Studies of PNe were long hampered by uncertainties in the distances to
Galactic PNe. Thus basic information such as the luminosities of the central
stars and the physical sizes and masses of the nebulae was poorly constrained.
This situation has been improved considerably in the last few years by the
availability of trigonometric parallaxes for many planetary-nebula nuclei
(PNNi) from the Gaia mission (e.g., Bucciarelli & Stanghellini, 2023; Chornay
et al., 2023; Hernández-Juárez et al., 2024, and references therein).

However, even if a precise distance is available for a PN, the mass and
composition of its central star’s progenitor remain unknown. This means that
finding and verifying PNe that are members of star clusters is crucially
important. For these objects, the progenitor star’s mass and metallicity can
be determined from the cluster’s main-sequence turnoff and metal content; the
absolute luminosities of the PN and the PNN can be found using the cluster’s
distance via main-sequence fitting; and the mass of the PNN can be deduced
from theoretical core-mass/luminosity relations—thus providing a point on the
important initial-final mass relation (e.g., Cummings et al., 2018; Barnett et
al., 2021; Marigo, 2022). Additionally, the chemical composition of the PN,
along with the progenitor mass, provide information about the dredge-up of
processed material from the stellar interior—theoretically predicted to depend
strongly on the initial mass of the progenitor star (e.g., Karakas & Lugaro,
2016).

Unfortunately, only a very few PNe have been shown to be probable members of
star clusters. Among open clusters (OCs) a convincing case is IPHASX
J055226.2+323724 (PN G177.6+03.0). This PN has recently been confirmed as a
member of the OC M37 (NGC 2099), based on the Gaia proper motion (PM) of its
central star, its nebular radial velocity (RV), and its interstellar
extinction, all of which agree with those of the cluster (Griggio et al.,
2022; Fragkou et al., 2022b; Werner et al., 2023). There are over a dozen
other Galactic PNe that lie near OCs on the sky, but nearly all fail the
membership tests of having the same PM, RV, and extinction as the cluster (see
detailed discussion and references in Frew et al. 2016 and Davis et al. 2019).
Only two Galactic PNe remain at present as strong candidate members of OCs to
add to the case of the M37 PN. These are PHR J1315$-$6555 in the OC ESO
96-SC04 = AL 1 (Parker et al., 2011; Fragkou et al., 2019a), and BMP
J1613$-$5406 in NGC 6067 (Frew et al., 2016; Fragkou et al., 2019b, 2022a). In
the Local Group, a PN that likely belongs to an OC in M31 was identified by
Bond (2015) and analyzed spectroscopically by Davis et al. (2019).

In the case of globular clusters (GCs), there are three and possibly four
known cases of PNe that appear to belong to Galactic GCs (Jacoby et al., 1997;
Bond et al., 2020, and references therein). Their central stars are likely to
be descended from interacting or merged binaries (e.g., Bond, 2015; Jacoby et
al., 2017; Sun et al., 2019), since single low-mass post-AGB stars in these
ancient clusters probably evolve too slowly to ionize their ejecta before they
disperse. Our team (Bond et al., 2020) obtained PMs of the central stars of
these four PNe from Gaia Data Release 2 (DR2), or by measuring archival multi-
epoch Hubble Space Telescope (HST) frames. For three of them—the PNe in M15,
M22, and NGC 6441—the PMs (as well as their RVs) provide strong support for
cluster membership. For the fourth case, JaFu 1 near the GC Palomar 6
(hereafter Pal 6), whose central star is too faint for Gaia, our PM
measurement was less precise due to the relatively low quality and short time
baseline of the available HST frames; nevertheless, our findings raised doubts
about its cluster membership.

In order to investigate with more precision whether JaFu 1 is a member of Pal
6, we have obtained new HST images in order to significantly reduce the
uncertainties in the PM of its central star, and we report the results here.
In the same HST program, we have also imaged the nuclei of PHR J1315$-$6555
and BMP J1613$-$5406, and we will present our findings for these objects in
future papers in this series (although the PM measurement for the latter
target will have to await second-epoch HST observations to be obtained in
2026).

##  2 JaFu 1: Candidate Member of Palomar 6

In the 1990s, G. Jacoby and L. Fullton (see Jacoby et al. 1997) carried out
ground-based narrow-band [O III] 5007 Å imaging of 133 Galactic GCs, with the
aim of finding PNe in the target clusters. Their survey resulted in
discoveries of two new PNe that were candidate cluster members, designated
JaFu 1 (PN G002.1+01.7; in Pal 6), and JaFu 2 (PN G353.5$-$05.0; in NGC 6441).
As noted above, JaFu 2 appears to be a genuine member of its host cluster.

Pal 6 is a faint and heavily reddened GC, discovered by Abell (1955) in the
course of his examination of photographs from the Palomar Observatory Sky
Survey (POSS). Lying in the Milky Way bulge at Galactic coordinates $l=2\fdg
09,b=+1\fdg 78$, Pal 6 is superposed on a rich star field. Figure 1 shows a
ground-based image of the cluster and its surroundings, created using the
PanSTARRS-1 Image Access tool.111https://ps1images.stsci.edu/cgi-
bin/ps1cutouts Here the cluster lies at the lower right of the frame, and JaFu
1 is located at the center of the white rectangle at the upper left.

JaFu 1 lies about $230^{\prime\prime}$ from the center of Pal 6, at a position
where the stellar field is heavily dominated by bulge stars, as shown in
Figure 1. However, this separation is still well within the cluster’s tidal
radius of $\sim$$500^{\prime\prime}$, calculated from data tabulated by Harris
(2010).222The Harris compilation of GC properties, 2010 December version, is
available online at http://physwww.mcmaster.ca/~harris/mwgc.dat

Figure 1:  Color image created from PanSTARRS-1 $g$, $r$, and $i$ frames,
showing the globular cluster Pal 6 (below right) and the location of the
planetary nebula JaFu 1 (faintly visible at the center of the white rectangle
at upper left). North is at the top, east on the left. Height of frame is
$320^{\prime\prime}$.

The distance of Pal 6 is about 7.2 kpc, according to an analysis of near-IR
stellar photometry by Lee & Carney (2002). This has been updated more recently
to $7.05^{+0.46}_{-0.43}$ kpc in a comprehensive compilation of GC distances
by Baumgardt & Vasiliev (2021). An independent estimate of the distance to
JaFu 1 could in principle be obtained using the statistical relation between
H$\alpha$ surface brightness and linear radius for PNe derived by Frew et al.
(2016); however, these authors actually used JaFu 1 as one of their
calibrators for the method, assuming it to be a cluster member and adopting
the Lee & Carney (2002) cluster distance. In any case, since the position of
JaFu 1 in the surface-brightness versus radius relations plotted by Frew et
al. (2016) agrees well with other calibrators, the distances of the PN and the
cluster are statistically similar.

The question of JaFu 1’s cluster membership has been reviewed in detail by
Jacoby et al. (1997, 2017) and Bond et al. (2020). We note here that the
strongest evidence for membership comes from the emission-line RV measurement
for JaFu 1 by Jacoby et al. (1997) of $+176\pm 15\,{\>\rm km\>s^{-1}}$. This
agrees remarkably well with the RV of Pal 6 of $+176.3\pm 1.5\,{\>\rm
km\>s^{-1}}$, determined from the mean of about a dozen individual member
stars (Baumgardt et al., 2019). However, as Jacoby et al. (1997) noted, there
is an appreciable velocity dispersion among Galactic-bulge stars. Thus there
is a non-zero chance of a field PN having an RV similar to that of the
cluster, estimated by Jacoby et al. (1997) to be about 3%. An argument against
membership is the large angular separation of PN and cluster; Jacoby et al.
(1997) used the surface density of field PNe in the Galactic bulge to estimate
a $\sim$15% probability of a bulge PN lying this close to the cluster by
chance. Potential negative evidence is that the interstellar reddening of the
PN appears to be larger than that of Pal 6. Jacoby et al. (1997, 2017) derived
reddening estimates for the PN of $E(B-V)=1.93$ and 1.67 from considerations
of the nebular spectrum and the intrinsic color of the central star. The
reddening of the cluster is lower, at $E(B-V)=1.46$ (Harris, 2010). However,
inspection of POSS and PanSTARRS images of the field shows the extinction to
be patchy and likely higher in the vicinity of the PN than it is for the
cluster, so this membership test is not decisive. Lastly, as noted in the
Introduction, Bond et al. (2020) measured a low-precision PM for the central
star from HST images with a short time baseline; it was discordant with the
mean PM of cluster member stars at about a 2$\sigma$ level.

##  3 New and Archival HST Observations of JaFu 1

Table 1 lists the available HST images of JaFu 1. They were obtained during
two visits in 2008 and 2010, and during our new visit in 2024.333Frames were
downloaded from the Mikulski Archive for Space Telescopes at
https://archive.stsci.edu/. All of the HST frames analyzed in this paper are
available at http://dx.doi.org/10.17909/d691-h991. The 2008 observations were
made with the Wide Field Planetary Camera 2 (WFPC2), using its broad-band
F555W ($V$) and F814W ($I$) filters, and its narrow-band F656N (H$\alpha$)
filter. The WFPC2 frames were aimed at direct imaging of the faint PN and
photometry of the central star, and are not optimal for precision stellar
astrometry since they were obtained in the camera’s low-resolution WF3
channel, and the telescope pointings were not dithered. The 2010 frames were
obtained with the Wide Field Channel (WFC) of the Advanced Camera for Surveys
(ACS), this time for imaging of the PN in the narrow-band F502N ([O III] 5007
Å) filter, and again without dithering.

Table 1: Log of Hubble Space Telescope Observations of JaFu 1  Date | Camera and | Filter | Pixel Scale | Exposures | Program ID
---|---|---|---|---|---
| Channel |  | [mas pixel-1] | [s] | and PI
2008 Mar 14 | WFPC2/WF3 | F555W | 100 | $2\times 160$ | GO-11308 (O. De Marco)
|  | F656N |  | $2\times 500$ |
|  | F814W |  | $2\times 160$ |
2010 Mar 14 | ACS/WFC | F502N | 50 | $3\times 796$ | GO-11558 (O. De Marco)
2024 Feb 14 | WFC3/UVIS | F555W | 40 | $6\times 120$ | GO-17531 (H.E.B.)
|  | F656N |  | $2\times 300$ |

For our new HST imaging in 2024, we used the UVIS (ultraviolet and visible
light) channel of the Wide Field Camera 3 (WFC3) in F555W with a six-point
dither pattern, along with two dithered exposures in F656N. To maximize
observing efficiency, we employed a $2047\times 2050$ pixel
($81^{\prime\prime}\times 81^{\prime\prime}$) subarray.

Figure 2 shows a false-color image of JaFu 1 at HST resolution, prepared by
combining the archival ACS frame in [O III] (green) with our WFC3 frames in
H$\alpha$ (red) and $V$ (blue). Here we used the drizzle-combined images
corrected for geometric distortion and charge-transfer efficiency (CTE),
obtained from the Mikulski Archive pipeline (filename suffix _drc). The field
shown in Figure 2 corresponds to the region enclosed in the white rectangle in
Figure 1. Note that the [O III] image was obtained 14 years prior to the
H$\alpha$ and $V$ data. We aligned the images such that JaFu 1 and its central
star are at the same pixel location; as a consequence, a slight but noticeable
misalignment of some of the field stars due to their intrinsic PMs can be
seen.444Our figure can be compared with a color image of JaFu 1 prepared by
Jacoby et al. (2017, their Figure 1) from the WFPC2 and ACS narrow-band
images.

Figure 2:  False-color HST image of the field centered on the planetary nebula
JaFu 1, outlined by the white rectangle in Figure 1. Created from WFC3 frames
in H$\alpha$ (red) and $V$ (blue), and an ACS frame in [O III] 5007 Å (green).
North is at the top, east on the left. Height of frame is $50^{\prime\prime}$.

The image of JaFu 1 shows an elliptical ring with diffuse edges, with minor
and major axes of about $5\farcs 5\times 7\farcs 3$ (corresponding to physical
dimensions of $0.19\times 0.25$ pc, at a distance of 7 kpc). The nucleus is
conspicuously off-center to the north inside the surrounding ring.

##  4 Data Analysis

We computed absolute PMs of stars in the HST field following the prescriptions
given in Bellini et al. (2014, 2018b), and described in detail by Bond et al.
(2020). In brief, we measured positions and magnitudes of neighbor-subtracted
sources on the pipeline-calibrated, unresampled exposures from the three
epochs listed in Table 1 (filename suffixes _flc for ACS and WFC3 with CTE
corrections, _c0f for WFPC2, which are not CTE-corrected). We used a
combination of first- and second-pass reduction techniques based on the
software tools hst1pass555hst1pass is available at
https://www.stsci.edu/hst/instrumentation/wfc3/software-tools. (Anderson,
2022a, b) and KS2 (see Bellini et al. 2017). These routines take advantage of
high-precision effective point-spread function (PSF) models (Anderson & King
2000; Anderson 2018; Bellini et al. 2018a) and geometric-distortion solutions
(Anderson & King 2000, 2006; Bellini & Bedin 2009; Bellini et al. 2011). We
defined a common reference-frame system based on positions for the brighter
stars in the field contained in the Gaia Data Release
3666https://vizier.cds.unistra.fr/viz-bin/VizieR-3?-source=I/355/gaiadr3 (DR3;
Gaia Collaboration et al. 2016, 2023) catalog. We then adjusted all single-
exposure source positions onto this frame by means of general six-parameter
linear transformations, using a local network of neighboring sources around
each star. Finally, for each source we determined its PM by fitting the slope
of its reference-frame-transformed positions as a function of exposure epoch,
via an iterative central-overlap method, as described in Bellini et al.
(2014).

Figure 3:  Measured positions of the central star of JaFu 1 in our master
reference frame versus observation epoch. Labels indicate the HST cameras used
for the images. The $X$ and $Y$ positions are measured in pixels, with $X$
pointing west and $Y$ pointing north, and the pixel scale is 40 mas pixel-1.
Positions from broad-band filters are plotted with black filled circles, and
measurements from narrow-band emission-line filters with green filled circles.
Points at the same epochs are slightly separated in time for clarity. The red
lines show the proper-motion fits, and the shaded red areas their 1$\sigma$
confidence intervals. See text for further details.

Figure 3 illustrates the PM solution for the central star of JaFu 1. Here we
plot the measured positions of the star in the master reference frame at our
three observation epochs; the $(X,Y)$ positions are in pixels, with $X$ toward
west and $Y$ toward north. For reasons discussed in Section 6, we distinguish
between measurements in broad-band filters (black points) and in narrow-band
emission-line filters (green points), but the agreement between them is good.
The red lines show linear fits to the data, whose slopes we convert to PMs
using the frame scale of 40 mas pixel-1. For our final PM solution, we omitted
the two WFC3 2024 exposures in F656N, due to the lack of a good distortion
model in that filter. However, we did determine and plot the positions in this
filter by applying the distortion model for the F606W filter, and there appear
to be no large discrepancies.

The PMs resulting from our analysis are relative to the average PM of stars in
the HST field. To convert them into absolute PMs, we applied a correction
based on the absolute PMs of stars in common between the HST and Gaia
catalogs. The mean absolute PM components of 122 Gaia stars in the HST field
brighter than $G=19$ are
$(\mu_{\alpha}\,\cos\delta,\mu_{\delta})_{0}=(-2.16\pm 0.13$, $-4.55\pm 0.10)$
mas yr-1.

Table 2 lists our results for the nucleus of JaFu 1. The first two rows list
the central star’s J2000 equatorial coordinates (epoch 2016), and the next two
rows give the corresponding Galactic longitude and latitude. Rows five and six
list the absolute PM components and their internal errors. In addition to
these internal uncertainties there are slightly larger external errors due to
the conversion from relative to absolute PMs, as presented in the previous
paragraph. The final three rows list magnitudes (Vega scale) that we
determined from the ACS and WFC3 frames.777We measured count rates in
10-pixel-radius apertures, corrected the rates to infinite apertures, and
applied Vega-scale zero-points. The aperture corrections and zero-points for
ACS and WFC3 are available from https://acszeropoints.stsci.edu and Calamida
et al. (2021), respectively. Calibrated magnitudes from the earlier WFPC2
frames were published by Jacoby et al. (2017). We note that the central star
is unusually bright in F502N and F656N, indicating that it has strong nebular
emission lines of [O III] and H$\alpha$; we return to this point in Section 6.

Table 2: Parameters for the Central Star of JaFu 1  Parameter | Value
---|---
RA (J2000) | 17 43 57.239
Dec (J2000) |  $-26$ 11 53.83
$l$ [deg] | 2.142
$b$ [deg] | +1.745
$\mu_{\alpha}\cos\delta$ [mas yr-1] | $-5.054\pm 0.075$
$\mu_{\delta}$ [mas yr-1] | $-8.323\pm 0.059$
F502N magnitude | $21.54\pm 0.01$
F555W magnitude | $22.50\pm 0.02$
F656N magnitude | $16.77\pm 0.03$

##  5 Proper-Motion Results

Figure 4 summarizes the results of our astrometric analysis. Panel (a)
presents a map showing the positions of all Gaia DR3 stars brighter than
$G=19$ mag in the field of Pal 6 and JaFu 1. The gold circle at the right is
centered on the cluster and has a radius of $72^{\prime\prime}$, equal to the
cluster’s half-light radius (Harris, 2010). The red polygon northeast of the
cluster outlines the field in which the archival HST data overlap with our new
2024 data. A green cross at its center marks the location of the nucleus of
JaFu 1.

Figure 4:  (a) Positions of Gaia DR3 stars brighter than $G=19$ in the
vicinity of the globular cluster Pal 6 (gold circle). The red border outlines
the field in which archival HST data overlap with our new 2024 images. The
position of the planetary nebula JaFu 1 is marked with a green cross. (b)
Color-magnitude diagram for the Gaia stars. Gold points are likely cluster
members, having positions and PMs within the gold circles in both Panels (a)
and (c). Red points are the Gaia stars that lie within the HST field in Panel
(a). (c) Vector-point PM diagram of Gaia stars shown in Panel (a). Red points
are Gaia stars lying within the HST field of view. The gold circle highlights
likely Pal 6 members according to their PMs. (d) and (e) Absolute PMs in right
ascension and declination for stars measured in the HST field and cataloged in
Gaia DR3. Red lines are the loci of equality. There is good consistency within
the errors. (f) Vector-point diagram for the absolute PMs we measured in the
HST field. The PM of the nucleus of JaFu 1 is marked with a green circle; its
uncertainties are smaller than the plotting symbol. The gold circle for the
cluster motion is the same as in Panel (c). Turquoise arrows show the
directions of PMs in Galactic longitude $l$ and latitude $b$. See text for
details of these diagrams.

Panel (b) in Figure 4 plots the color-magnitude diagram of all the stars from
Panel (a) in the Gaia photometric system of $G$ magnitude versus $G_{\rm
BP}-G_{\rm RP}$ color. Gaia sources lying within the half-light radius of the
cluster and having a PM consistent with cluster membership [i.e., lying within
the gold circle in Panel (c)] are plotted with gold filled circles. Note the
well-defined red-giant branch, and the lack of a blue horizontal branch,
consistent with the cluster’s relatively high metallicity of $\rm[Fe/H]=-0.91$
(Harris, 2010) or slightly higher (e.g., Vásquez et al., 2018). Stars falling
within the HST field are plotted using red filled circles.

Panel (c) shows the vector-point diagram for the Gaia DR3 stars in Panels (a)
and (b). The gold circle is centered on the mean PM for Pal 6 members of
$(\mu_{\alpha}\,\cos\delta,\mu_{\delta})=(-9.17,-5.26)$ mas yr-1, and it has a
radius of 0.95 mas yr-1, equal to 5 times the cluster’s central PM dispersion
of 0.19 mas yr-1. Here the mean PM and dispersion are both taken from
Baumgardt et al. (2019). Red points again correspond to stars lying in the HST
field. It is clear, from Panels (b) and (c), that most if not all of the Gaia
stars located within the HST field are not cluster members: the red-giant
branch of Pal 6 shown by the gold points in Panel (b) is almost entirely
missing among the red points, and there is a significant population of
foreground Galactic-disk main-sequence stars. Only one Gaia star in the HST
field shares the PM of Pal 6.

Comparisons between the absolute PMs of stars as measured from HST and Gaia
data are shown in Panels (d) and (e) for the right-ascension and declination
directions, respectively. PM uncertainties for the HST measurements are
typically smaller than the size of the plotting symbols, but for some of the
Gaia measurements the error bars are larger and are plotted. Most of the large
outliers in Panels (d) and (e) are seen to have close neighbors at HST
resolution. The red diagonal lines are not fits to the data but simply the
bisectors of the plane, and indicate a generally good agreement between the
two sets of PM measurements (after application of the conversion from relative
to absolute for the HST data, as described above).

Finally, Panel (f) plots the vector-point diagram for the stars within the red
field in Panel (a), determined from the HST data. Here we show all measured
stars with a PM error less than 2 mas yr-1. The uncertanties are again usually
smaller than the plotting symbols, except for a few of the fainter objects for
which the error bars are shown. Of the 430 stars measured in the HST frames,
only five had PM uncertainties larger than $\pm$2 mas yr-1. In this panel
turquoise arrows mark the directions of Galactic longitude and latitude. Note
that the PM distribution is elongated in the direction of Galactic longitude,
and that the Sun’s participation in Galactic rotation produces a reflex offset
of the mean PM from zero along this direction. The gold circle in Panel (f),
repeated from Panel (c), shows the locus of PMs for members of Pal 6. Our
measured absolute PM of the nucleus of JaFu 1 (listed in Table 2) is plotted
as a green circle in Panel (f). Its uncertainties are smaller than the
plotting symbol.

Our results show that the PM of the JaFu 1 central star differs from the mean
PM of members of Pal 6 with very high statistical significance. The offset in
total PM between them is 5.13 mas yr-1; this is formally about a 40$\sigma$
difference, given the uncertainty in the total PM dominated by the external
error of $\sim$0.13 mas yr-1 (see Section 4).

The large difference in PMs between the PN and the cluster appears to rule out
cluster membership. This makes the close agreement in RVs between the two
(Section 2) a remarkable coincidence—unless we consider an alternative
possibility that the progenitor star was a cluster member, which has recently
been ejected from the cluster through dynamical interactions or tidal
stripping. However, the relative directions of the PMs make this scenario
unlikely. The PN lies to the northeast of the cluster (see Figure 1). Thus, if
the PN had been ejected, its PM should lie in a northeast direction relative
to that of the cluster; actually, its relative PM is toward the southeast
[Figure 4, Panel (f)].

##  6 JaFu 1 as an EGB 6-type Planetary Nebula

Jacoby et al. (2017) found that the central star of JaFu 1 is unusually bright
in [O III] and H$\alpha$, based on their photometric analyses of the 2008 and
2010 WFPC2 and ACS images.888This feature appears to confirm that the star is
indeed the true nucleus of the PN, and not a chance alignment, in spite of its
remarkably off-center location. Our magnitude measurements in these
bandpasses, derived from ACS and WFC3 images, confirm the star’s enhanced
brightness in the narrow-band filters, as shown in our Table 2. These results
imply that JaFu 1 belongs to the rare class of “EGB 6-type” PNe, whose central
stars are accompanied by compact emission knots (CEKs), unresolved from the
nuclei in ground-based imaging. This property of the prototype, EGB 6 itself,
was discovered by Ellis et al. (1984). Frew & Parker (2010) defined the EGB 6
class, pointing out a handful of other PNNi that are also accompanied by CEKs.
Liebert et al. (2013) and Bond et al. (2016), using HST images, demonstrated
the remarkable fact that in EGB 6 the CEK is resolved from the central star at
a separation of about $0\farcs 16$, and appears to be associated with a dust-
embedded low-luminosity binary companion star. More recently, Bond et al.
(2024) have discovered that the nucleus of the PN Abell 57 is accompanied by a
CEK and shares many of the properties of EGB 6, but in this case high-
resolution imaging is lacking.

The similarity of the central star of JaFu 1 to the nucleus of EGB 6 raises
the possibility that the former may likewise exhibit an offset in the position
of the central star between narrow- and broad-band images. The distance of EGB
6, based on its Gaia trigonometric parallax, is about 750 pc. JaFu 1 is
$\sim$10 times further away; if its nucleus is a binary system similar to that
of EGB 6, it would not appear resolved in our HST images. However, there could
potentially be an astrometric offset of as much as $\sim$16 mas of the
centroid of the emission knot from that of the central star, if JaFu 1 has a
CEK separated from the central star by the same projected linear amount as in
EGB 6. This would introduce systematic errors into a PM measurement based on a
combination of narrow- and broad-band images obtained at different epochs.

We carried out two tests of this possibility. First, we added measurements of
the two WFC3 F656N frames from 2024 to the PM solution based on 15 frames
presented above. This astrometry has relatively large systematic
uncertainties, because we had to apply the distortion model for the broad-band
F606W filter to these frames, given the lack of a suitable model for the
narrow-band filter. The result was a change of the derived PM of the central
star by less than $0.001\,\rm mas\,yr^{-1}$. Second, we measured the offset
between the positions of the central star at the 2024 epoch in the two
filters, F555W and F656N, again using the F606W distortion model for the
H$\alpha$ frames. Here we found a shift of $1.0\pm 0.9$ mas. Thus we do not
find a statistically significant offset of the CEK from the central star in
JaFu 1, and hence no appreciable effect on our determination of its PM. The
projected physical separation of the knot and central star in JaFu 1 appears
to be considerably smaller than in EGB 6.

##  7 Discussion

We have used multi-epoch high-resolution HST imaging to measure a precise PM
for the central star of the PN JaFu 1, a candidate member of the GC Pal 6. Our
results reveal a large discrepancy between the PMs of the nucleus and of the
cluster, decisively ruling out cluster membership. The close agreement in RV
between the PN and cluster, discussed in Section 2, thus becomes a
statistically remarkable chance coincidence.

As Panel (f) in Figure 4 shows, JaFu 1 has a large negative PM in the
direction of Galactic longitude. This property indicates that the object
belongs to the Galactic bulge population, since Galactic-disk stars lying in
the direction of the Galactic center have generally smaller PMs; see, for
example, Figure 8 in Libralato et al. (2021). The high RV of the object also
supports bulge membership.

Although we conclude that JaFu 1 is not a member of Pal 6, we are still left
with three cases of PNe that appear to be genuine members of Galactic GCs, as
described in Section 1. These objects continue to challenge our understanding
of the evolution of low-mass stars (e.g., Jacoby et al., 2017; Bond et al.,
2020; Kwitter & Henry, 2022). And JaFu 1, even if it belongs to the Galactic-
bulge population rather than Pal 6, still confronts us with the puzzles of its
off-center nucleus and of the nature of EGB 6 central stars.

Based on observations with the NASA/ESA Hubble Space Telescope obtained from
the Data Archive at the Space Telescope Science Institute (STScI), which is
operated by the Association of Universities for Research in Astronomy,
Incorporated, under NASA contract NAS5-26555. Support for Program number
GO-17531 was provided through grants from STScI under NASA contract
NAS5-26555. This work has made use of data from the European Space Agency
(ESA) mission Gaia (https://www.cosmos.esa.int/gaia), processed by the Gaia
Data Processing and Analysis Consortium (DPAC,
https://www.cosmos.esa.int/web/gaia/dpac/consortium). Funding for the DPAC has
been provided by national institutions, in particular the institutions
participating in the Gaia Multilateral Agreement. The Pan-STARRS1 Surveys
(PS1) and the PS1 public science archive have been made possible through
contributions by the Institute for Astronomy, the University of Hawaii, the
Pan-STARRS Project Office, the Max-Planck Society and its participating
institutes, the Max Planck Institute for Astronomy, Heidelberg and the Max
Planck Institute for Extraterrestrial Physics, Garching, The Johns Hopkins
University, Durham University, the University of Edinburgh, the Queen’s
University Belfast, the Harvard-Smithsonian Center for Astrophysics, the Las
Cumbres Observatory Global Telescope Network Incorporated, the National
Central University of Taiwan, the Space Telescope Science Institute, the
National Aeronautics and Space Administration under Grant No. NNX08AR22G
issued through the Planetary Science Division of the NASA Science Mission
Directorate, the National Science Foundation Grant No. AST-1238877, the
University of Maryland, Eötvös Loránd University (ELTE), the Los Alamos
National Laboratory, and the Gordon and Betty Moore Foundation.

## References

  * Abell (1955) Abell, G. O. 1955, PASP, 67, 258
  * Anderson (2018) Anderson, J. 2018, Focus-Diverse PSFs for Five Commonly Used WFC3/UVIS Filters, Instrument Science Report WFC3 2018-14, 19 pages, ,
  * Anderson (2022a) —. 2022a, One-Pass HST Photometry with hst1pass, Instrument Science Report WFC3 2022-5, 55 pages, ,
  * Anderson (2022b) —. 2022b, One-Pass HST Photometry with hst1pass, Instrument Science Report ACS 2022-02, ,
  * Anderson & King (2000) Anderson, J., & King, I. R. 2000, PASP, 112, 1360
  * Anderson & King (2006) —. 2006, PSFs, Photometry, and Astronomy for the ACS/WFC, Instrument Science Report ACS 2006-01, 34 pages, ,
  * Barnett et al. (2021) Barnett, J. W., Williams, K. A., Bédard, A., & Bolte, M. 2021, AJ, 162, 162
  * Baumgardt et al. (2019) Baumgardt, H., Hilker, M., Sollima, A., & Bellini, A. 2019, MNRAS, 482, 5138
  * Baumgardt & Vasiliev (2021) Baumgardt, H., & Vasiliev, E. 2021, MNRAS, 505, 5957
  * Bellini et al. (2011) Bellini, A., Anderson, J., & Bedin, L. R. 2011, PASP, 123, 622
  * Bellini et al. (2017) Bellini, A., Anderson, J., Bedin, L. R., et al. 2017, ApJ, 842, 6
  * Bellini et al. (2018a) Bellini, A., Anderson, J., & Grogin, N. A. 2018a, Focus-diverse, empirical PSF models for the ACS/WFC, Instrument Science Report ACS 2018-8, ,
  * Bellini & Bedin (2009) Bellini, A., & Bedin, L. R. 2009, PASP, 121, 1419
  * Bellini et al. (2014) Bellini, A., Anderson, J., van der Marel, R. P., et al. 2014, ApJ, 797, 115
  * Bellini et al. (2018b) Bellini, A., Libralato, M., Bedin, L. R., et al. 2018b, ApJ, 853, 86
  * Bond (2015) Bond, H. E. 2015, AJ, 149, 132
  * Bond et al. (2020) Bond, H. E., Bellini, A., & Sahu, K. C. 2020, AJ, 159, 276
  * Bond et al. (2024) Bond, H. E., Chaturvedi, A. S., Ciardullo, R., et al. 2024, arXiv e-prints, arXiv:2405.11087
  * Bond et al. (2016) Bond, H. E., Ciardullo, R., Esplin, T. L., et al. 2016, ApJ, 826, 139, (B16)
  * Bucciarelli & Stanghellini (2023) Bucciarelli, B., & Stanghellini, L. 2023, A&A, 680, A104
  * Calamida et al. (2021) Calamida, A., Mack, J., Medina, J., et al. 2021, New time-dependent WFC3 UVIS inverse sensitivities, Instrument Science Report WFC3 2021-4, 33 pages, ,
  * Chornay et al. (2023) Chornay, N., Walton, N. A., Jones, D., & Boffin, H. M. J. 2023, arXiv e-prints, arXiv:2311.17820
  * Cummings et al. (2018) Cummings, J. D., Kalirai, J. S., Tremblay, P. E., Ramirez-Ruiz, E., & Choi, J. 2018, ApJ, 866, 21
  * Davis et al. (2019) Davis, B. D., Bond, H. E., Ciardullo, R., & Jacoby, G. H. 2019, ApJ, 884, 115
  * Ellis et al. (1984) Ellis, G. L., Grayson, E. T., & Bond, H. E. 1984, PASP, 96, 283
  * Fragkou et al. (2019a) Fragkou, V., Parker, Q. A., Zijlstra, A., Shaw, R., & Lykou, F. 2019a, MNRAS, 484, 3078
  * Fragkou et al. (2019b) Fragkou, V., Parker, Q. A., Zijlstra, A. A., Crause, L., & Barker, H. 2019b, Nature Astronomy, 3, 851
  * Fragkou et al. (2022a) Fragkou, V., Parker, Q. A., Zijlstra, A. A., et al. 2022a, Galaxies, 10, 44
  * Fragkou et al. (2022b) —. 2022b, ApJ, 935, L35
  * Frew & Parker (2010) Frew, D. J., & Parker, Q. A. 2010, PASA, 27, 129
  * Frew et al. (2016) Frew, D. J., Parker, Q. A., & Bojičić, I. S. 2016, MNRAS, 455, 1459
  * Gaia Collaboration et al. (2016) Gaia Collaboration, Prusti, T., de Bruijne, J. H. J., et al. 2016, A&A, 595, A1
  * Gaia Collaboration et al. (2023) Gaia Collaboration, Vallenari, A., Brown, A. G. A., et al. 2023, A&A, 674, A1
  * Griggio et al. (2022) Griggio, M., Bedin, L. R., Raddi, R., et al. 2022, MNRAS, 515, 1841
  * Harris (2010) Harris, W. E. 2010, arXiv e-prints, arXiv:1012.3224
  * Hernández-Juárez et al. (2024) Hernández-Juárez, D., Rodríguez, M., & Peña, M. 2024, arXiv e-prints, arXiv:2403.04606
  * Jacoby et al. (2017) Jacoby, G. H., De Marco, O., Davies, J., et al. 2017, ApJ, 836, 93
  * Jacoby et al. (1997) Jacoby, G. H., Morse, J. A., Fullton, L. K., Kwitter, K. B., & Henry, R. B. C. 1997, AJ, 114, 2611
  * Karakas & Lugaro (2016) Karakas, A. I., & Lugaro, M. 2016, ApJ, 825, 26
  * Kwitter & Henry (2022) Kwitter, K. B., & Henry, R. B. C. 2022, PASP, 134, 022001
  * Kwitter et al. (2014) Kwitter, K. B., Méndez, R. H., Peña, M., et al. 2014, Rev. Mexicana Astron. Astrofis., 50, 203
  * Lee & Carney (2002) Lee, J.-W., & Carney, B. W. 2002, AJ, 123, 3305
  * Libralato et al. (2021) Libralato, M., Lennon, D. J., Bellini, A., et al. 2021, MNRAS, 500, 3213
  * Liebert et al. (2013) Liebert, J., Bond, H. E., Dufour, P., et al. 2013, ApJ, 769, 32, (L13)
  * Marigo (2022) Marigo, P. 2022, Universe, 8, 243
  * Parker et al. (2011) Parker, Q. A., Frew, D. J., Miszalski, B., et al. 2011, MNRAS, 413, 1835
  * Sun et al. (2019) Sun, W., Peng, E. W., Ko, Y., et al. 2019, ApJ, 885, 145
  * Vásquez et al. (2018) Vásquez, S., Saviane, I., Held, E. V., et al. 2018, A&A, 619, A13
  * Weidmann et al. (2020) Weidmann, W. A., Mari, M. B., Schmidt, E. O., et al. 2020, A&A, 640, A10, (W20)
  * Werner et al. (2023) Werner, K., Reindl, N., Raddi, R., et al. 2023, A&A, 678, A89


II, I II/1

# Automatic Co-Registration of Aerial Imagery and Untextured Model Data
utilizing Average Shading Gradients

S. Schmitzb,a M. Weinmannb B. Rufa,b aFraunhofer IOSB, Video Exploitation
Systems, Karlsruhe, Germany -
{sylvia.schmitz<EMAIL_ADDRESS>
bInstitute of Photogrammetry and Remote Sensing, Karlsruhe Institute of
Technology,
Karlsruhe, Germany - {martin.weinmann<EMAIL_ADDRESS>
###### Abstract

The comparison of current image data with existing 3D model data of a scene
provides an efficient method to keep models up to date. In order to transfer
information between 2D and 3D data, a preliminary co-registration is
necessary. In this paper, we present a concept to automatically co-register
aerial imagery and untextured 3D model data. To refine a given initial camera
pose, our algorithm computes dense correspondence fields using SIFT flow
between gradient representations of the model and camera image, from which
2D-3D correspondences are obtained. These correspondences are then used in an
iterative optimization scheme to refine the initial camera pose by minimizing
the reprojection error. Since it is assumed that the model does not contain
texture information, our algorithm is built up on an existing method based on
Average Shading Gradients (ASG) to generate gradient images based on raw
geometry information only. We apply our algorithm for the co-registering of
aerial photographs to an untextured, noisy mesh model. We have investigated
different magnitudes of input error and show that the proposed approach can
reduce the final reprojection error to a minimum of 1.27 $\pm$0.54 pixels,
which is less than 10 % of its initial value. Furthermore, our evaluation
shows that our approach outperforms the accuracy of a standard Iterative
Closest Point (ICP) implementation.

###### keywords:

Co-registration, Pose estimation, 2D-3D Correspondence, Average Shading
Gradients, Iterative Closest Point

##  1 INTRODUCTION

Due to technological advancements in the field of sensor technology and
algorithm development for the acquisition, generation and processing of 3D
data, the availability and use of 3D models has risen significantly. The
advantages of 3D visualization are applied successfully in applications such
as urban navigation and planning, ecological development or security
surveillance. City administrations, for example, already maintain city models
which are used advantageously in spatial and urban planning. A prerequisite
for a reasonable use is that these models match reality as closely as
possible. However, the acquisition and provision of large-scale 3D models with
a high level of detail is still very expensive and time-consuming.
Consequently, such models are typically only generated every few years. In
order to compensate for the time gap between the acquisition and dissemination
of publicly accessible 3D models, we work on augmenting the model data with
up-to-date aerial imagery and perform a change detection to update the
existing model.

Figure 1: Registration task: Estimation of camera parameters, which describe
the relative pose between image and model.

In order to enable the transfer of information between aerial imagery and a 3D
model, a preliminary co-registration is necessary. This task can be formulated
as the estimation of camera parameters which describe the relative position
and orientation between image and model as illustrated in Figure 1. Given
point correspondences between image and model, this is a well-known task for
which numerous efficient methods have already been developed (for instance
proposed by Lepetit et al.,  (2009), Penate-Sanchez et al.,  (2013) or Gao et
al.,  (2003)). If there is no information available regarding corresponding
points, the main challenge is to identify discriminative points which can be
used to estimate the correspondences between the model and the image. If the
model data is textured, these correspondences can be identified by exploiting
image features (e.g. SIFT, SURF or ORB). However, texture of the model cannot
always be relied on, as it might be outdated, of different modality, or simply
missing. For this reason, we present a concept to automatically co-register
aerial imagery and untextured 2.5D or 3D models, which is working with raw
geometry information only.

Our algorithm is based on the concept for automatic registration of images to
untextured geometry, which has been proposed by Plötz and Roth,  (2017).
Assuming an initial camera pose, a given untextured 2.5D or 3D model and a
camera image, our algorithm estimates the intrinsic and extrinsic camera
parameters based on 2D-3D correspondences between pixels in the input image
and object points in the model. In order to detect reliable correspondences,
image features of the input image are compared with features extracted on
rendered views of the 3D model. A basic component of many proven feature
descriptors consists of intensity gradients, which we also use in our
algorithm to describe and compare image features. Since we deal with an
untextured model, the calculation of texture gradients on rendered images
generated from the model is not possible. To this end, we rely only on
gradients of shadings by applying the so-called Average Shading Gradients
(ASG) proposed by Plötz and Roth,  (2017). This is a rendering technique in
which observable shading gradients are averaged over all possible lighting
directions of the 3D scene under the assumption of a local illumination model.

This paper is organized as follows. After presenting related work that focuses
on image-to-model co-registration, we describe our algorithm in Section 3.
There, we explain the generation of gradient images as well as the
correspondence search and pose estimation. In Section 4, we present and
discuss our dataset and the results we achieved with our approach. Finally, we
provide concluding remarks and suggestions for future work in Section 5.

##  2 RELATED WORK

Using 2D-3D correspondences to estimate the relative camera pose of a single
query image with respect to a given 3D model is a widely applied approach.
Similar to the concept of this paper, Russell et al.,  (2011), Aubry et al.,
(2014), Irschara et al.,  (2009) and Sibbing et al.,  (2013) determine the
necessary correspondences by comparing the query image to rendered views of
the model. The approach of Irschara et al.,  (2009) is based on a mapping of
SIFT features extracted from the query image to SIFT features extracted from
images of a database. The database used consists of already registered images
as well as images of the 3D model, which were generated from different
viewpoints. In a similar manner, Russell et al.,  (2011) and Aubry et al.,
(2014) present approaches that generate images from a model to register
paintings. Russell et al.,  (2011) combine the approach of Irschara et al.,
(2009) with a matching of GIST descriptors (Oliva and Torralba, , 2001) to
find a rough camera pose. The co-registration is then improved by matching the
contours in the painting to the view-dependent contours of the model. Aubry et
al.,  (2014), on the other hand, use rendered images from a model to identify
features that are reliably recognizable in any 2D representation and are used
for matching. The features used in the approaches mentioned are based on
texture information contained in the 3D model. In contrast, this paper assumes
an untextured model. In addition, Russell et al.,  (2011) and Aubry et al.,
(2014) assume that the camera is located near the ground, while in this work
we focus on the co-registration of aerial imagery.

Likewise, the task of co-registering aerial photographs to a 3D model is
addressed by Vasile et al.,  (2006), Frueh et al.,  (2004) and Mastin et al.,
(2009). In (Frueh et al., , 2004), the goal is to transfer texture information
from aerial photographs to a city model. The required registration of the
images is based on an assignment of projected 3D lines of the model to 2D
lines, which are extracted in the aerial photograph. Using a given camera pose
with an accuracy comparable to that of a Global Positioning System (GPS) and
Inertial Navigation System (INS) system, an exhaustive search is performed
using extrinsic and intrinsic camera parameters. Vasile et al.,  (2006)
present a similar approach. Pseudo-intensity images are generated using
shadows of light detection and ranging (LiDAR) data which are compared to the
2D aerial image by correlation. Similar to (Frueh et al., , 2004), the
complete camera pose is then determined by an exhaustive search with GPS
information providing an initial estimate. Both methods lead to accurate
registrations, but are very time-consuming. The goal to transfer information
from images to a model is also pursued by Mastin et al.,  (2009). In their
approach, aerial images are registered with respect to a LiDAR point cloud in
order to generate a photorealistic 3D model. The registration process based on
Mutual Information determines camera parameters that maximize the
transinformation between the distribution of image features and projected
LiDAR features. Used LiDAR features arise from elevations and probability of
detection, whereas image features result from illumination intensities. The
camera parameters that maximize the transinformation are determined using
simplex methods. The use of OpenGL and graphics hardware in the optimization
process leads to significantly shorter registration times compared to the
methods of Frueh et al.,  (2004) and Vasile et al.,  (2006). The three methods
co-register aerial photographs with respect to a LiDAR point cloud. In
contrast, the aim of our work is to register an aerial photograph to a meshed
3D model, independent from the image sensor technology used to generate the
model.

Lately, learning-based approaches are in particular attracting a great deal of
attention. Solutions based on convolutional neural networks (CNNs), proposed
by Brachmann and Rother,  (2018), Kendall et al.,  (2015) and Hou et al.,
(2018), are presented to solve the task of 2D/3D co-registration. In order to
determine the camera pose in a given 3D environment using an RGB image,
Brachmann and Rother,  (2018) propose a CNN that predicts 2D-3D
correspondences. Hypotheses regarding the camera parameters are determined
from four correspondences each. A second CNN is used to determine the most
probable camera pose. Kendall et al.,  (2015) describe the training of a CNN
that directly estimates extrinsic camera parameters based on an image. The
required training data are generated by a Structure-from-Motion (SfM) process
and used for the fine tuning of pre-trained models. Hou et al.,  (2018) also
estimate the pose directly via a CNN, proposing a new loss function based on
Riemann’s geometry. In contrast to learning-based approaches, this paper
presents a method that does not require complex training procedures or a large
amount of training data. Updating and adapting the method to other conditions,
such as prior knowledge of the camera pose, is easily possible without having
to train new models.

##  3 METHODOLOGY

Figure 2: Overview of the processing pipeline of the co-registration
algorithm: Gradient images rendered from the model are matched to a gradient
image of the photograph using SIFT flow. The correspondences determined are
used to estimate the relative pose based on a DLT. The final pose is selected
in the verification step.

The overall procedure to estimate the relative camera pose between an
untextured model and a camera image (referred to as query image) is depicted
in Figure 2. Given an untextured 3D model, an initial camera pose
$\mathrm{P}^{\mathrm{initial}}$ and a depth image generated by SfM from the
camera images, our algorithm utilizes gradient representations of the input
model and query image to extract features which are used in an iterative
optimization scheme to refine the initial camera pose. Using coarse poses
$\mathrm{P}^{\mathrm{coarse}}_{i}$ close to the initial camera pose, we render
the model with ASG, yielding multiple gradient images of the input model. The
query image is also transformed into a gradient representation. Thus, at the
end of the first step, we have a set of gradient images
$\mathrm{I}^{\mathrm{\nabla ASG}}_{i}$ extracted from the 3D model associated
with coarse camera poses $\mathrm{P}^{\mathrm{coarse}}_{i}$ and one gradient
image $\mathrm{I}^{\mathrm{\nabla In}}$ corresponding to the query image. In
the second stage, we compute a dense correspondence field between each
$\mathrm{I}^{\mathrm{\nabla ASG}}_{i}$ and $\mathrm{I}^{\mathrm{\nabla In}}$
using the SIFT flow algorithm (Liu et al., , 2011). With these 2D-3D
correspondences, we employ a Direct Linear Transformation (DLT) (Hartley and
Zisserman, , 2003) within a RANSAC loop to iteratively refine each coarse
camera pose $\mathrm{P}^{\mathrm{coarse}}_{i}$ to receive
$\mathrm{P}^{\mathrm{fine}}_{i}$. Given the set of refined camera poses, a
final verification step selects the pose $\mathrm{P}^{\mathrm{fine}}_{i}$ with
the smallest reprojection error as the output pose
$\mathrm{P}^{\mathrm{out}}$.

###  3.1 Gradient Representations

The first step of the registration process is the computation of a set of
gradient images, which represents the model from different perspectives. For
this purpose, we perpetuate the initial extrinsic camera parameters with
Gaussian noise to generate multiple views on the model distributed around the
provided camera pose $\mathrm{P}^{\mathrm{initial}}$. Gradient images are
generated from the model using the generated coarse camera poses
$\mathrm{P}^{\mathrm{coarse}}_{i}$. Since the model does not contain any color
or gray value information, gradients can result solely from shadings due to
specific lighting of the scene. Hence, we use ASG in the same manner as in
(Plötz and Roth, , 2017).
A gradient image can be described by convolving the image matrix and
derivative filters in the x and y directions:

$\displaystyle||\nabla\mathrm{I}||=\sqrt{\left(\mathrm{h_{x}}*\mathrm{I}\right)^{2}+\left(\mathrm{h_{y}}*\mathrm{I}\right)^{2}},$
(1)

with $\mathrm{I}$ being the image and $\mathrm{h_{x}}$, $\mathrm{h_{y}}$
indicating the derivative filters. Under the assumption of a Lambertian
illumination model and the use of a point light source, the intensities of a
rendered image $\mathrm{I}$ can be described by:

$\displaystyle\mathrm{I}=\max(0,-\mathrm{n}^{\top}\mathrm{l}).$ (2)

In this formulation, the normal direction is expressed by the vector
$\mathrm{n}$ and the direction of light by the vector $\mathrm{l}$. With
predefined light direction $\mathrm{l}$, the combination of the above
equations allows the calculation of a rendered image in gradient
representation. However, the assumption of a fixed light direction has
considerable disadvantages. Discontinuities in the normal map, which cause
shadings under an assumed light direction and thus high values in the gradient
image, do not lead to observable gradients under the assumption of a different
light direction. The approach of ASG counteracts this undesirable behavior.
The observed gradient strengths are averaged over all feasible light
directions $\mathrm{l}$ along the unit sphere $\mathcal{S}$. The corresponding
mathematical description is:

$\displaystyle\overline{||\nabla\mathrm{I}||}=\int_{\mathcal{S}}\Big{[}\left(\mathrm{h_{x}}*\max(0,-\mathrm{n}^{\top}\mathrm{l})\right)^{2}$
$\displaystyle+\,\left(\mathrm{h_{y}}*\max(0,-\mathrm{n}^{\top}\mathrm{l})\right)^{2}\Big{]}^{\frac{1}{2}}\;\text{d}\mathrm{l}.$
(3)

The exact calculation of $\overline{||\nabla\mathrm{I}||}$ is very
computation-intensive due to the complex integrand. Therefore, approximations
are proposed by Plötz and Roth,  (2017) that allow the estimation of
$\overline{||\nabla\mathrm{I}||}$ in closed form:

$\displaystyle\overline{||\nabla\mathrm{I}||}$
$\displaystyle\approx\dfrac{1}{2}\int_{\mathcal{S}}\Big{[}\left(\mathrm{h_{x}}*\mathrm{n}^{\top}\mathrm{l}\right)^{2}\,+\,\left(\mathrm{h_{y}}*\mathrm{n}^{\top}\mathrm{l}\right)^{2}\Big{]}^{\frac{1}{2}}\;\text{d}\mathrm{l}$
$\displaystyle\leq\frac{1}{2}\sqrt{\int_{\mathcal{S}}\left(\mathrm{h_{x}}*(\mathrm{n}^{\top}\mathrm{l})\right)^{2}+\left(\mathrm{h_{y}}*(\mathrm{n}^{\top}\mathrm{l})\right)^{2}\;\text{d}\mathrm{l}}$
$\displaystyle=\frac{1}{2}\sqrt{\int_{\mathcal{S}}\left((\mathrm{h_{x}}*\mathrm{n})^{\top}\mathrm{l}\right)^{2}\;\text{d}\mathrm{l}+\int_{\mathcal{S}}\left((\mathrm{h_{y}}*\mathrm{n})^{\top}\mathrm{l}\right)^{2}\;\text{d}\mathrm{l}}$
$\displaystyle=\sqrt{\frac{\pi}{3}}\sqrt{\sum_{i=1}^{3}(\mathrm{h_{x}}*\mathrm{n}_{i})^{2}+(\mathrm{h_{y}}*\mathrm{n}_{i})^{2}}.$
(4)

The designation $\mathrm{n}_{i}$ indicates the x-, y- and z-component of the
normal map. For the proof of the equality of the expressions in Section 3.1,
we refer to (Plötz and Roth, , 2017). The represented form enables an
efficient calculation of the desired gradient image, which is based
exclusively on the convolution of the normal map with derivation filters.
Figure 3 shows result images, which can be derived by means of ASG.

Figure 3: Gradient images created using Average Shading Gradients. (a) Normal
maps of the model, (b) Resulting gradient images.

We use the presented approximation of ASG to render one gradient image for
each generated coarse pose $\mathrm{P}^{\mathrm{coarse}}_{i}$. Features of
these images are to be matched with features of the query image that is to be
registered. In order to get comparable features, the input image also needs to
be converted to a representation that contains gradients induced by shadings.
Applying discrete derivative operators on the single camera image, as proposed
by Plötz and Roth,  (2017), results in gradients which are related to ASG
computed on the model to a certain degree. To obtain even more comparable
gradients, we include additional camera images captured with a small spatial
offset to the query image. The added images allow to compute depth information
and normal vectors for the query image by means of SfM. Consequently, we can
utilize ASG to transform the input image into a representation
$\mathrm{I}^{\mathrm{\nabla In}}$ equivalent to the gradient representations
$\mathrm{I}^{\mathrm{\nabla ASG}}_{i}$ of the renderings.

###  3.2 Feature Matching

In order to match the rendered images to the query image, we use the SIFT flow
algorithm (Liu et al., , 2011) computing dense flow fields between each
$\mathrm{I}^{\mathrm{\nabla ASG}}_{i}$ and $\mathrm{I}^{\mathrm{\nabla In}}$.
This algorithm works similar to the optical flow method, determining a pixel-
wise shift between two images. Instead of computing correspondences by
individual pixel intensities, the matching in the SIFT flow algorithm is based
on the comparison of SIFT descriptors calculated for each pixel. In order to
prevent the assignment of pixels from regions of homogeneous structure in the
input image to empty regions in the rendering, only pixels located in textured
regions are included in the calculation of the flow.

To compute correspondences, the flow vectors are first determined from the
query gradient image $\mathrm{I}^{\mathrm{\nabla In}}$ to each rendered
gradient image $\mathrm{I}^{\mathrm{\nabla ASG}}_{i}$, then from each rendered
image $\mathrm{I}^{\mathrm{\nabla ASG}}_{i}$ to the query gradient image
$\mathrm{I}^{\mathrm{\nabla In}}$. Pixel pairs connected by two opposite flow
vectors are recorded as corresponding pair. The determined point
correspondences are converted into 2D-3D correspondences between the input
image and the model. The 3D model points are reconstructed from the
appropriate points of the rendered images by considering the associated coarse
camera poses $\mathrm{P}^{\mathrm{coarse}}_{i}$.

###  3.3 Pose Estimation

Given the determined 2D-3D correspondences, we improve successively each
coarse pose $\mathrm{P}^{\mathrm{coarse}}_{i}$. Assuming that enough correct
correspondences have been detected, a RANSAC scheme can be used to reliably
determine the relative camera pose between the query image and the model.
Within the inner RANSAC loop, six 2D-3D point pairs are randomly selected from
the available correspondences. From these, intrinsic and extrinsic camera
parameters are determined by applying the DLT algorithm (Hartley and
Zisserman, , 2003). Subsequently, we check how many of the given
correspondences support the determined camera pose and form a consensus set by
computing the reprojection error for each correspondence. The camera pose
corresponding to the largest consensus set represents the result and thus the
improved camera pose $\mathrm{P}^{\mathrm{fine}}_{i}$. Empirically, we have
found that a good termination criterion is the computation of a consensus set
that contains at least 65 % of all correspondences. If this criterion is not
met, the calculation is aborted after a maximum number of 500 iterations.

###  3.4 Plausibility Check

The final step of our automatic co-registration process selects the most
appropriate camera pose from all the refined poses
$\mathrm{P}^{\mathrm{fine}}_{i}$. Since a correct registration is not possible
in every case, we also use this step to decide on the success of the
registration. For this purpose, the mutual reprojection error is calculated in
pairs between all refined pose estimates. This is defined by the following
equation:

$\displaystyle\delta(\mathcal{P},\mathcal{P}^{\prime})=$
$\displaystyle{}\frac{1}{2}\left(\frac{1}{\left|\mathcal{V}\right|}\sum_{\mathbf{x}\in\mathcal{V}}||\mathcal{P}(\mathbf{x})-\mathcal{P}^{\prime}(\mathbf{x})||_{2}\right.$
$\displaystyle\left.+\,\frac{1}{\left|\mathcal{V}^{\prime}\right|}\sum_{\mathbf{x}\in\mathcal{V}^{\prime}}||\mathcal{P}(\mathbf{x})-\mathcal{P}^{\prime}(\mathbf{x})||_{2}\right).$
(5)

$\mathcal{P}$ and $\mathcal{P}^{\prime}$ denote poses which project the model
coordinates into an image plane; $\mathcal{V}$ and $\mathcal{V}^{\prime}$
denote the set of visible points in the image area. Thus the error measure
describes the mean Euclidean distance between visible pixels. If the error for
two poses is below a threshold value, which is fixed to 5 % of the longest
image dimension, the considered poses are considered compatible. An undirected
graph is defined by the compatibility relationship of all poses to each other.
The nodes of the graph each represent a camera pose. The edges of the graph
represent the compatibility between camera poses. By means of a depth-first
search, the largest connected component of the graph is determined. If this
contains more than three nodes, the registration process is considered
successful. As the final pose $\mathrm{P}^{\mathrm{out}}$ we select the pose
from the largest connected component of the graph, for which the largest
consensus set was reached in the previous RANSAC scheme.

##  4 EXPERIMENTS

The quantitative evaluation of the registration procedure is based on a
dataset comprising 164 aerial photographs captured with a DJI Phantom 3
Professional. These depict a free-standing building from three sides at three
different heights (2 m, 8 m and 15 m). Background objects, such as trees or
people, are also depicted and may differ between the images. Figure 4(a) shows
a selection of the aerial photographs used. Based on all given images, a 3D
point cloud is created using the SfM pipeline COLMAP (Schönberger and Frahm, ,
2016; Schönberger et al., , 2016) (Figure 4(b)). From the relevant points,
i.e. those representing the building, a surface model was generated by a
triangular meshing. The Poisson Surface Reconstruction method (Kazhdan et al.,
, 2006) provided in COLMAP, was used for this purpose. A view of the mesh
model is shown in Figure 4(c). The resulting model shows some errors due to
the limited accuracy of the point cloud.

(a) Input images

(b) Point cloud

(c) Mesh model

Figure 4: (a) Example images of our dataset. The point cloud generated from
the images can be seen in (b).(c) Resulting mesh model.

Aerial photographs are usually taken using unmanned aerial vehicle (UAV)
systems equipped with GPS and Inertial Measurement Unit (IMU) sensors. This
allows to derive a camera pose for each image, which can then be used to
initialize the registration process. Since the dataset used does not contain
any sensor data, we simulate the initial camera poses
$\mathrm{P}^{\mathrm{in}}$ by applying additive white Gaussian noise to the
translation and rotation parameters of the ground truth poses
$\mathrm{P}^{\mathrm{GT}}$. The ground truth poses are derived from the SfM
process on which the 3D reconstruction is based.

For our experiments, we have selected 50 test images of the dataset associated
with initial camera poses and scaled them to a size of 505 $\times$ 275
pixels. When selecting the images, we took care to cover as wide a range of
perspectives as possible. For each registration process of our tests, we
generated and improved 15 coarse poses and automatically choose the best
refined pose.

###  4.1 Accuracy and Success Rate

We evaluated the presented co-registration algorithm (hereinafter referred to
as ASG approach) with respect to the accuracy of the estimated camera poses
and the success rate of the registration. Using the ASG approach, all eleven
degrees of freedom of the desired camera pose can be determined. However, in
many applications, intrinsic camera parameters are given by a preliminary
sensor calibration. Under these conditions, an Efficient Perspective-n-Point
(EPnP) method according to (Lepetit et al., , 2009) can be used instead of the
DLT to estimate the camera pose from 2D-3D correspondences. The performance of
the presented co-registration algorithm was evaluated for both initial
conditions (intrinsic parameters known and unknown) using 50 selected images.
During our tests, we successively increased the mean initial error, which
represents the average displacement of the initial poses
$\mathrm{P}^{\mathrm{in}}$ in relation to the true poses
$\mathrm{P}^{\mathrm{GT}}$. We quantify initial errors by the mutual
reprojection error as defined in Section 3.4.

Figure 5: The rate of success for known and unknown intrinsic camera
parameters.

First we have examined how many images are rated as successfully co-registered
by the automatic registration process. Figure 5 presents the corresponding
results for the assumption of known and unknown intrinsic camera parameters.
The results show that, under the condition of unknown intrinsic parameters,
more images are accepted as successfully registered. This is due to the fact
that the camera pose can be adapted more flexibly to the found correspondences
due to the higher degree of freedom. However, as shown in the following
section, this is at the expense of the accuracy of the estimate. It can also
be seen that, as the initial error increases, the registration task becomes
more demanding and some of the images are not registered. With low initial
errors of about 30 pixels, 90 % and 82 % of the images are accepted within the
automatic verification, whereas a medium error of about 100 pixels only
results in 66 % and 54 %, respectively.

A second aspect that was evaluated is the improvement of the initial errored
poses by the proposed co-registration. Again, we use the mutual reprojection
error (Section 3.4), to evaluate the pose estimation. To summarize the
distribution of the errors, the mean value, the standard deviation and the
range of the reprojection errors are stated. The resulting error statistics
for known and unknown intrinsic camera parameters are shown in Table 1 for
comparison.

Initial error | Intrinsics | Accuracy | Error range
---|---|---|---
$30.19$ | unknown | $3.10\pm 2.01$ | $1.22-11.93$
| known | $1.27\pm 0.54$ | $0.57-2.20$
$56.90$ | unknown | $4.10\pm 3.31$ | $1.32-15.75$
| known | $1.54\pm 1.37$ | $0.28-8.40$
$79.53$ | unknown | $4.39\pm 3.26$ | $1.49-16.85$
| known | $1.81\pm 2.13$ | $0.41-12.86$
$103.83$ | unknown | $3.70\pm 2.50$ | $1.87-12.50$
| known | $2.51\pm 1.71$ | $0.85-6.50$
Table 1: This table holds the accuracy of our co-registration approach for
various mean initial errors under known and unknown intrinsic camera
parameters. The accuracy is given by the mean value and the standard deviation
of the reprojection errors of registered images.

The following observations are made: The maximum reprojection error with
respect to all registered images is 16.85 pixels. This corresponds to about
3.3 % of the largest image dimension. Even with a small initial error of about
30 pixels, this is equal to a reduction of the error by almost 50 %. From
this, it can be concluded that automatic verification only accepts estimates
that are more accurate than the given incorrect pose
$\mathrm{P}^{\mathrm{in}}$. Furthermore, it is shown that the average
reprojection error can be reduced to up to 3.1 pixels when estimating all
camera parameters. If the intrinsic parameters are already given, an average
value of 1.3 pixels can be even achieved. Even with high initial errors of
more than 100 pixels, good pose estimates with an average reprojection error
of 3.7 or 2.5 pixels are achieved by our co-registration process. This
corresponds to an improvement to 2.63 % or 1.07 % of the mean initial error.
The large differences in the reprojection errors between the various test
images, which can be seen in the large error ranges, are noticeable. The high
standard deviations, which are in the same order of magnitude as the mean
value itself, also indicate a strong scatter of the errors. Certain viewing
perspectives of the scene are therefore more difficult to estimate than
others. The examination of individual results shows: If different sides of the
building are depicted on a query image, a more precise registration can
usually be made than for images in which mainly one side of the building is
visible. This is due to the influence of the spatial constellation of the
correspondence points on the accuracy of pose estimation methods. Thus the
equation system of the DLT gets singular or numerically unstable for a planar
point set. The EPnP method explicitly distinguishes the planar point
configuration from the non-planar case, resulting in increasing inaccuracy in
the intermediate near-planar case.

Figure 7 shows sample images for which our registration algorithm fails. It
can be seen that these are images where all visible points of the building
have a similar distance to the camera.

In summary:

  * •

With the automatic verification, on the test data our registration approach
only provides camera poses that are more accurate than the initial camera
poses.

  * •

The average reprojection error of the camera poses is improved by the
registration process up to 1.07 % of the average initial error.

  * •

The achieved accuracies increase only slightly for high initial errors.

  * •

With known intrinsic camera parameters, a more precise estimation of camera
poses is possible.

###  4.2 Comparison to an ICP Implementation

In order to rate our results obtained, we compared the accuracy and success
rate of our approach to a standard Iterative Closest Point (ICP)
implementation. Within the evaluation, we used the implementation of the
software library Point Cloud Library (PCL) (Rusu and Cousins, , 2011). In
order to use the ICP algorithm to estimate the relative pose between an image
and a model, the input image needs to be represented as a point cloud. This is
possible if depth information of the image and the intrinsic camera parameters
are known. These requirements were also met for the ASG approach, which only
estimates extrinsic camera parameters. In contrast to our approach, the ICP
algorithm achieves an alignment of two point clouds. The relative camera pose
between image and model can then be derived from the transformation required
for the alignment.

For the evaluation, we used 10 images which are registered once by the ICP
algorithm and once by our ASG approach. Regarding the ICP method, the
registration of an image is evaluated as successful if the reprojection error
under the estimated camera pose falls below a threshold value of 20 pixels.
Whether a successful registration was achieved by applying the ASG approach is
still decided automatically. Table 2 shows the number of successfully
registered test images for the different approaches. For a small average
initial error (approx. 50 pixels), the ASG approach is superior to the ICP
process. If the average initial error is almost 80 pixels, the success rate is
identical with 6 out of 10 images. If the initial errors are further
increased, the number of registered images remains constant for the ICP
method, while it drops to 3 for the ASG approach.

Initial error |  ASG |  ICP
---|---|---
$30.25$ |  $9$ |  $6$
$55.12$ |  $8$ |  $6$
$80.22$ |  $6$ |  $6$
$101.15$ |  $3$ |  $6$
Table 2: This table shows the number of successfully registered images using ASG and ICP approaches. A total of 10 images were used for the test. Initial error | Test images | ASG | ICP
---|---|---|---
$30.25$ | $6$ | $1.12\pm 1.22$ | $11.16\pm 4.45$
$55.12$ | $6$ | $2.13\pm 2.82$ | $9.70\pm 5.60$
$80.22$ | $4$ | $1.89\pm 1.25$ | $10.41\pm 4.50$
$90.51$ | $3$ | $2.80\pm 1.48$ | $10.60\pm 5.84$
Table 3: This table indicates the accuracies achieved using ICP and ASG
approaches. The mean value and the standard deviation of the achieved
reprojection errors are given. For the error determination, only test images
were used which were registered successfully by both procedures.

Figure 6: Qualitative results showing the input image together with a
projection of the input model given the corresponding camera pose. The input
model of the building structure is projected in transparent colors, which
encode the normal vectors of the model surfaces. In the first row, the input
model is projected given the initial camera pose. These images clearly show
the error in the camera pose with which our algorithm is initialized. The
images in the second and third rows depict the results of our co-registration
algorithm for known and unknown camera intrinsics. The fourth row shows the
results of the ICP algorithm. The results indicate that, in most cases, our
algorithm achieves a good alignment between the input model and the input
image.

Figure 7: Images that cannot be registered by our algorithm.

To compare the accuracy of the two methods, we determined the reprojection
errors for all test images that can be successfully registered by both
methods. Table 3 shows the corresponding error statistics (mean and standard
deviation of reprojection errors) for increasing initial errors. It shows that
the average error resulting from the pose estimates of the ICP method is about
five times higher than that of the ASG method. Therefore, the ASG method
allows to estimate camera poses closer to the true pose than the pose
estimates provided by the ICP method. The poorer performance of the ICP method
can partly be explained by noisy depth values of the aerial photographs. These
have a negative effect on the accuracy of the point cloud, which results from
the backprojection of the depth image. As mentioned before, the used point
cloud of the model is also erroneous. The difficulty to adjust two noisy point
clouds to each other is reflected in the high reprojection errors. ASG
registration is more robust to errors in the input data. This can be explained
by the fact that the geometric error minimized within the RANSAC scheme is
based only on selected 2D-3D correspondences. These correspondences were
previously determined using stable characteristics. In contrast, in the ICP
procedure, all model points are included in the calculation of the error to be
minimized. The observations of the comparison of registration procedures can
be summarized as follows:

  * •

The accuracy of the estimated camera parameters using the ASG approach exceeds
one of the ICP implementation. The reprojection error is on average five times
smaller.

  * •

At given initial poses with a mean reprojection error of less than 60 pixels,
more images can be registered using the ASG approach than using the ICP
implementation.

  * •

For rough pose estimates with a mean reprojection error of over 80 pixels,
more images can be registered by means of the ICP implementation than by means
of the ASG approach.

###  4.3 Qualitative Results

Figure 6 shows a selection of the results of the evaluated co-registration
algorithms. It can be seen that our algorithm (second and third row) can
handle large initial errors in the translation as well as minor errors in
rotation. It can be observed that our proposed algorithm, based on the
assumption of known intrinsic camera parameters (ASG with EPnP), gives the
most accurate results. The estimation of the full camera pose (ASG with DLT)
commonly yields similar results, but in some cases larger errors can be
detected. For example, in the projection of the center image in Figure 6, the
roof of the building disappears. Furthermore, it can be seen that registration
by means of the ICP algorithm also improves the given poses
$\mathrm{P}^{\mathrm{in}}$. However, the deviations from the true poses are
greater than the deviations obtained using our algorithm.

##  5 CONCLUSION & FUTURE WORK

In conclusion, we proposed an algorithm to automatically estimate the relative
camera pose between aerial imagery and untextured 2.5D or 3D model data. To
refine an initial guess of the camera pose, we compute feature-based dense
correspondence fields between an aerial photograph and rendered images
generated from different perspectives on the model. Since textural features
are not present in the model, the compared features are derived from gradients
that are based solely on the object geometry. To obtain such gradients related
to the photograph as well as gradients related to the model, we use ASG, a
method in which observable gradients from shadings are averaged over all
possible light directions. Our evaluation shows that initial error-prone
camera poses are significantly improved by our registration algorithm.
Especially under the condition of calibrated camera sensors, good results are
achieved. The error can thus be reduced to up to 1.07 % of its initial value.
With regard to the accuracy of estimated camera poses, our presented approach
exceeds the results provided by the ICP algorithm. In addition, the automatic
verification of the pose estimation provides a reliable statement about the
success of the registration.

In our future work we want to integrate and evaluate further suitable methods
for the estimation of poses from correspondences into our method instead of
DLT or EPnP. In particular, approaches that include existing depth information
of the input image are of interest. In addition, we want to evaluate the the
difference in performance between methods that use line features, such as
those extracted with the help of ASG, and methods that rely on point features.

## REFERENCES

  * Aubry et al., (2014) Aubry, M., Russell, B. C. and Sivic, J., 2014.  Painting-to-3D model alignment via discriminative visual elements.  ACM Transactions on Graphics 33(2), pp. 14:1–14:14.
  * Brachmann and Rother, (2018) Brachmann, E. and Rother, C., 2018.  Learning less is more - 6D camera localization via 3D surface regression.  In: _Proc. IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 4654–4662.
  * Frueh et al., (2004) Frueh, C., Sammon, R. and Zakhor, A., 2004.  Automated texture mapping of 3D city models with oblique aerial imagery.  In: _Proc. International Symposium on 3D Data Processing, Visualization and Transmission_ , pp. 396–403.
  * Gao et al., (2003) Gao, X.-S., Hou, X.-R., Tang, J. and Cheng, H.-F., 2003.  Complete solution classification for the perspective-three-point problem.  IEEE Transactions on Pattern Analysis and Machine Intelligence 25(8), pp. 930–943.
  * Hartley and Zisserman, (2003) Hartley, R. and Zisserman, A., 2003.  Multiple View Geometry In Computer Vision.  Cambridge University Press.
  * Hou et al., (2018) Hou, B., Miolane, N., Khanal, B., Lee, M., Alansary, A., McDonagh, S., Hajnal, J., Rueckert, D., Glocker, B. and Kainz, B., 2018.  Deep pose estimation for image-based registration.  Medical Imaging with Deep Learning.
  * Irschara et al., (2009) Irschara, A., Zach, C., Frahm, J.-M. and Bischof, H., 2009.  From structure-from-motion point clouds to fast location recognition.  In: _Proc. IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 2599–2606.
  * Kazhdan et al., (2006) Kazhdan, M., Bolitho, M. and Hoppe, H., 2006.  Poisson surface reconstruction.  In: _Proc. Eurographics Symposium on Geometry Processing_.
  * Kendall et al., (2015) Kendall, A., Grimes, M. and Cipolla, R., 2015.  PoseNet: A convolutional network for real-time 6-DOF camera relocalization.  In: _Proc. IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 2938–2946.
  * Lepetit et al., (2009) Lepetit, V., Moreno-Noguer, F. and Fua, P., 2009.  EPnP: An accurate O(n) solution to the pnp problem.  International Journal of Computer Vision 81(2), pp. 155–166.
  * Liu et al., (2011) Liu, C., Yuen, J. and Torralba, A., 2011.  SIFT-Flow: Dense correspondence across scenes and its applications.  IEEE Transactions on Pattern Analysis and Machine Intelligence 33(5), pp. 978–994.
  * Mastin et al., (2009) Mastin, A., Kepner, J. and Fisher, J., 2009.  Automatic registration of LiDAR and optical images of urban scenes.  In: _Proc. IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 2639–2646.
  * Oliva and Torralba, (2001) Oliva, A. and Torralba, A., 2001.  Modeling the shape of the scene: A holistic representation of the spatial envelope.  International Journal of Computer Vision 42(3), pp. 145–175.
  * Penate-Sanchez et al., (2013) Penate-Sanchez, A., Andrade-Cetto, J. and Moreno-Noguer, F., 2013.  Exhaustive linearization for robust camera pose and focal length estimation.  IEEE Transactions on Pattern Analysis and Machine Intelligence 35(10), pp. 2387–2400.
  * Plötz and Roth, (2017) Plötz, T. and Roth, S., 2017.  Automatic registration of images to untextured geometry using average shading gradients.  International Journal of Computer Vision 125(1-3), pp. 65–81.
  * Russell et al., (2011) Russell, B. C., Sivic, J., Ponce, J. and Dessales, H., 2011.  Automatic alignment of paintings and photographs depicting a 3D scene.  In: _Proc. IEEE International Conference on Computer Vision Workshops_ , pp. 545–552.
  * Rusu and Cousins, (2011) Rusu, R. B. and Cousins, S., 2011.  3D is here: Point Cloud Library (PCL).  In: _Proc. IEEE International Conference on Robotics and Automation_ , pp. 1–4.
  * Schönberger and Frahm, (2016) Schönberger, J. L. and Frahm, J.-M., 2016.  Structure-from-motion revisited.  In: _Proc. IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 4104–4113.
  * Schönberger et al., (2016) Schönberger, J. L., Zheng, E., Frahm, J.-M. and Pollefeys, M., 2016.  Pixelwise view selection for unstructured multi-view stereo.  In: _Proc. European Conference on Computer Vision_ , pp. 501–518.
  * Sibbing et al., (2013) Sibbing, D., Sattler, T., Leibe, B. and Kobbelt, L., 2013.  SIFT-realistic rendering.  In: _Proc. IEEE International Conference on 3D Vision_ , pp. 56–63.
  * Vasile et al., (2006) Vasile, A., Waugh, F. R., Greisokh, D. and Heinrichs, R. M., 2006.  Automatic alignment of color imagery onto 3D laser radar data.  In: _Proc. IEEE Conference on Applied Imagery and Pattern Recognition Workshops_ , pp. 6–12.