# A Search for Chemical Anomalies of Seven A-types Stars

Yahya Nasolo,1 and Şeyma Çalışkan1
1Department of Astronomy and Space Sciences, Ankara University, Tandoğan,
Ankara 06100, Türkiye E-mail<EMAIL_ADDRESS>
(Accepted XXX. Received YYY; in original form ZZZ)

###### Abstract

We present a chemical abundance analysis of seven A-type stars with no
detailed chemical abundance measurements in the literature. High-resolution
spectra of the targets – HD 2924, HD 4321, HD 26553, HD 125658, HD 137928, HD
154713, and HD 159834 – were obtained using the Coudé Echelle Spectrograph at
the TÜBİTAK National Observatory. We determined the atmospheric abundances of
the samples and measured the elemental abundances of C, N, O, Na, Mg, Al, Si,
S, K, Ca, Sc, Ti, V, Cr, Mn, Fe, Co, Ni, Cu, Zn, Sr, Y, Zr, Ba, La, Ce, Nd,
Sm, Eu, and Gd. The masses of the stars were estimated based on their
evolutionary tracks, and their ages were calculated using isochrones. We also
calculated the radii of the stars. The abundance patterns of HD 4321, HD
125658, and HD 154713 were found to be in agreement with those of classical Am
stars, with underabundant Ca and Sc, overabundant heavier elements, and
moderate overabundance of iron-peak elements. We found that HD 137928 and HD
159834 have similar abundance characteristics to marginal Am-type stars. The
elemental distributions of HD 2924 and HD 26553 are consistent with the
pattern of normal A-type stars. The iron, nickel, and zinc abundances of HD
125658 and HD 137928 are significantly higher than other Am stars. These
values suggest that they are among the most metal-rich Am stars.

###### keywords:

Stars: chemically peculiar – abundances – HD 2924 – HD 4321 – HD 26553 – HD
125658 – HD 137928 – HD 154723 – HD 159834

††pubyear: 2022††pagerange: A Search for Chemical Anomalies of Seven A-types
Stars–A Search for Chemical Anomalies of Seven A-types Stars

##  1 Introduction

In the course of stellar evolution, stars inherit the chemical composition of
their birthplace; however, the atmospheric chemical abundance distribution
changes over time due to poorly understood internal stellar processes. The
warmer post-ZAMS mid-massive stars, corresponding to mid-B through early-F
spectral type stars, attract particular attention since a substantial fraction
exhibit chemical abundance anomalies, which reflect these internal processes.
Detailed chemical abundance analysis of these stars is essential since it
potentially provides a basis for understanding such phenomena. The chemically
peculiar (CP) metallic-line (Am) stars are the typical targets for these
studies. CP stars are characterized by enhanced absorption lines (Preston,
1974; Monier, 2019). Am stars are A- to early F-type stars sharing the same
region with the normal A-type stars, with effective temperature
(${T}_{\text{eff}}$) between 7000 and 9000 K. They lie in the cooler portion
of the main-sequence CP stars. The Am stars were first recognized as a group
by Titus & Morgan (1940). Preston (1974) classified the CP stars into four
groups, with Am stars being the CP1 group. Am stars are characterized by three
spectral characteristics: the strength of their Balmer lines; their Ca II K
lines, which are weaker than those of normal A-type stars with the same
temperature; and their enhanced metallic lines relative to their normal
counterparts. In typical Am stars spectra, the intensity of the Ca II K line
is weaker than Balmer lines and resembles an earlier spectral type. The
metallic lines are more intense and resemble a later spectral type (Roman et
al., 1948). Abt & Morgan (1976)stated that the Ca II K line in the spectra of
such stars resembles the spectral type at least five subtypes earlier. The Am
stars with Ca II K and metallic lines differing by less than five subtypes are
referred to as marginal Am stars (Trust et al., 2021). Underabundance of Sc II
and/or Ca II is a discovery criterion for this class (Michaud et al., 2015).
Based on observational statistics, the majority of Am stars are slow rotators
(Fossati et al., 2008). This supports the hypothesis of chemical separation
due to radiative diffusion and gravitational settling, since slow rotation is
expected not to destroy the separation phenomenon (Michaud et al., 2015).
Chemical separation requires calm atmospheres, which are expected given the
slow rotation of these stars (Takeda et al., 2008; Fossati et al., 2008; Abt,
2009). In close binaries, slow rotation is associated with tidal
synchronization (Khokhlova, 1981). However, observations of low-rotating
normal A-type stars pose a challenge to this view of the impact of rotation.
Normal A-type stars exhibit nearly solar abundances with the heavier elements
falling within the range of $\pm{0.40}$ dex relative to the Sun (Adelman &
Unsuree, 2007). This paper focuses on determining in detail the chemical
abundances and fundamental parameters (e.g. effective temperatures and surface
gravities) of the sample stars. The paper is organized as follows: Section 2
describes the target selection, observations, and data reduction. We present
the determination of atmospheric stellar parameters and the procedure used to
perform the abundance analysis in Section 3. The evolutionary status of the
stars is shown by employing the HR diagram in Section 4. We discuss the
results in Section 5, and we conclude in Section 6.

##  2 OBSERVATION DATA

###  2.1 Target selection

One of the dominant mixing processes that suppress the effect of chemical
separation in atomic diffusion, is linked to rotational velocity. A
statistical analysis of observations suggests that all main-sequence stars of
the targeted temperature regime with equatorial rotational velocities less
than $80$ km s-1 are Am /Fm stars (Abt, 2000). Abt & Moyd (1973) initially
reported that the equatorial velocity of observed CP1 stars is less than $120$
km s-1. This observation is consistent with chemical separation due to atomic
diffusion occurring when the He II convection zone disappears at equatorial
velocities lower than $70-120$ km s-1 (Michaud, 1982). We therefore selected
targets in this temperature regime with low rotation velocities ($v$ sin $i$ <
$70$ km s-1) and whose chemical abundance has not been previously derived.

###  2.2 Observations and data reduction

For this study, we used the high-resolution R$\sim$40,000 spectroscopic data
obtained by the Coudé Echelle Spectrograph attached to the 1.5 m
Russian–Turkish Telescope (RTT-150) at TÜBİTAK National Observatory (TUG). The
spectra cover a wavelength range of 3900 to 8770 Å. Along with the target
star’s spectra, each observation run began with acquiring calibration frames
and arc spectra: the halogen lamp for flat fielding and the Th–Ar lamp for
wavelength calibration. Table 1 lists the observation log of our targets,
including observation times, exposure times, numbers of co-added spectra, and
signal-to-noise ratios (SNR). The raw spectra were reduced using IRAF’s
standard reduction routines. Initially, image header information was
methodized to allow processing in the IRAF environment. The reduction process
started with bias subtraction followed by flat fielding. The next step was the
removal of scattered light from star spectra. Using the Th-Ar spectra, we
conducted wavelength calibration and lastly performed cosmic ray removal. We
measured the Doppler shift on the spectra using unblended moderately strong
absorption lines. Then, the heliocentric radial velocities
(${V}_{\text{helio}}$) were determined and necessary Doppler corrections were
applied to all spectra. For six of the targets, we acquired two spectral
frames, which were combined to improve the SNR. We could not acquire a second
frame for HD 125658 due to adverse weather. The echelle orders of the spectra
were normalized according to spline-fitted continuum points. Figure 1 shows a
sample of normalized orders of the spectra of the stars at about 5100 Å , as
well as some of the identified unblended lines used for chemical abundance
determination. The lines clearly exhibit the broadening effect of rotational
velocity.

Figure 1: A sample of normalized orders of the target stars and some of the identified unblended lines used for chemical abundance determination. Table 1: The observation log of the target stars. The V-band magnitude were estimated by Oja (1991) Star name | V-mag | Exposure time | Observation time | SNR
---|---|---|---|---
| (mag) | (s) | (JD) | (@5000-7000 Å)
HD 2924 | 6.700 [0.009] | 4500 x 2 | 2457676.37750 | 297
HD 4321 | 6.510 [0.009] | 4500 x 2 | 2457675.50366 | 250
HD 26553 | 6.080 [0.009] | 4500 x 2 | 2455554.33347 | 287
HD 125658 | 6.445 [0.010] | 4500 x 1 | 2458232.38508 | 115
HD 137928 | 6.430 [0.010] | 4500 x 2 | 2458233.33802 | 280
HD 154713 | 6.441 [0.010] | 4500 x 2 | 2458234.41497 | 302
HD 159834 | 6.100 [0.010] | 4500 x 2 | 2458232.51027 | 362

##  3 ATMOSPHERE PARAMETERS AND CHEMICAL ABUNDANCES

Spectroscopy and photometry are the traditional methods used to determine
stellar atmospheric parameters. In this work, we employed both methods. We
used the photometric method to find the initial atmospheric parameters. Then,
we refined the parameters with the spectroscopic method.

###  3.1 Line identification and equivalent width measurement

We used line lists from the Kurucz line database (Kurucz & Bell, 1995) and the
Vienna Atomic Line Database (Piskunov et al. (1995); Kupka et al. (1999);
Ryabchikova et al. (1999)). Line identification and equivalent width
measurements of the atomic species found in the spectra of the target stars
were implemented through an IDL visualization program, BINMAG6 (Kochukhov,
2018). Equivalent width measurements were performed by Gaussian profile
fitting of the lines. Table 2 lists the measured spectral lines.

Table 2:  List of spectral lines used in the abundance analysis of target stars. The full table is available online. A portion of the table is shown here Atomic Sp. | Wavelength | log $gf$ | Equivalent width (pm) |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---
| (nm) |  | HD2924 | HD4321 | HD26553 | HD125658 | HD137928 | HD154713 | HD159834
C I | 477.0027 | -2.722 | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | 01.50
C I | 477.1742 | -2.120 | $\dots$ | 06.46 | $\dots$ | 05.87 | 02.93 | 03.78 | 04.22
C I | 477.5897 | -2.163 | 02.98 | 05.38 | 01.60 | 03.09 | $\dots$ | $\dots$ | $\dots$
C I | 493.2049 | -1.884 | 04.45 | 06.89 | 03.04 | 03.56 | $\dots$ | 02.74 | 02.90
C I | 502.3849 | -2.400 | $\dots$ | 02.25 | 00.71 | $\dots$ | $\dots$ | $\dots$ | 02.02

###  3.2 Stellar parameters

Photometric stellar parameters were determined using the Strömgren $ubvy\beta$
photometric indices of Paunzen (2015). The photometric data were used in the
calibrations performed by Napiwotzki et al. (1993) to determine the
${T}_{\text{eff}}$ and surface gravity (log $g$) of each target. These
calibrations have systematic and statistical errors of 200 K for
${T}_{\text{eff}}$ and 0.10 dex for log $g$ (Napiwotzki et al., 1993). The
computed stellar parameters are given in Table 3.

Table 3: Photometrically determined stellar parameters Star name | ${T}_{\text{eff}}$ | log $g$
---|---|---
| [K] | [dex]
HD 2924 | 8633 | 3.45
HD 4321 | 7659 | 3.27
HD 26553 | 7783 | 2.41
HD 125658 | 7966 | 3.94
HD 137928 | 8382 | 3.81
HD 154713 | 8217 | 3.63
HD 159834 | 7799 | 3.57

Spectroscopic atmospheric parameters determination. One of the classic methods
used to yield the effective temperature spectroscopically is to employ Fe I
lines throughout the stellar spectrum and iterate the effective temperature
until their abundances show no trend with the excitation potential (EP) of
each line. In contrast with photometric methods, no reddening information is
needed for this technique; however, high-resolution spectra with a resolving
power of R$\sim\lambda/\Delta\lambda$ $\geq$ 15,000 are required, preferably
with wide wavelength coverage throughout the optical range (Frebel et al.,
2013). The different methods are known to produce temperatures with systematic
offsets from each other. We used the photometrically determined atmospheric
parameters ((${T}_{\text{eff}}$, log $g$) to construct initial model
atmospheres used in the EP method, where ${T}_{\text{eff}}$ was derived from
the excitation equilibrium of the Fe I lines. The models were constructed
using ATLAS9 (Kurucz, 1993a). Depending on the ${T}_{\text{eff}}$ derived from
the EP method, we derived the stellar surface gravities through Fe I-II
ionization equilibrium. The log $g$ was determined from the balance of
abundances derived from individual Fe I-II while the value of log $g$ was
iterated. The microturbulent velocity ($\xi$) was calculated by adopting the
values which produced no trend of equivalent widths with abundances derived
from individual Fe I lines. This allowed the chemical abundance of iron to be
derived. We measured the $v$ sin $i$ of the target stars by comparing the
observed spectra with the synthetic spectra. The synthetic spectra were
produced using SYNTHE (Kurucz, 2005b, 1993b). All atmospheric parameters
derived from spectroscopy are listed in Table 4.

Table 4: Atmospheric parameters from spectroscopic analysis Star name |  ${T}_{\text{eff}}$ | log $g$ | [Fe/H] |  $\xi$ |  ${V}_{\text{helio}}$ |  $v$ sin $i$
---|---|---|---|---|---|---
| [K] | [dex] | [dex] | [ km s-1] | [ km s-1] | [ km s-1]
HD 2924 | 8650$\pm$150 | 3.70$\pm$0.10 | 0.12$\pm$0.18 | 2.60$\pm$0.5 | 01.90$\pm$0.25 | 30$\pm$5
HD 4321 | 8500$\pm$150 | 4.00$\pm$0.10 | 0.14$\pm$0.15 | 3.00$\pm$0.5 | -07.60$\pm$0.70 | 24$\pm$4
HD 26553 | 8350$\pm$150 | 2.20$\pm$0.10 | 0.08$\pm$0.19 | 1.90$\pm$0.5 | -22.00$\pm$2.60 | 10$\pm$2
HD 125658 | 8800$\pm$150 | 4.60$\pm$0.10 | 0.41$\pm$0.10 | 5.00$\pm$0.5 | 01.70$\pm$0.50 | 26$\pm$5
HD 137928 | 9100$\pm$150 | 3.80±0.10 | 0.42$\pm$0.12 | 4.00$\pm$0.5 | -03.90$\pm$0.40 | 34$\pm$5
HD 154713 | 8350$\pm$150 | 3.50$\pm$0.10 | 0.18$\pm$0.15 | 3.00$\pm$0.5 | -06.30$\pm$0.50 | 39$\pm$5
HD 159834 | 8350$\pm$150 | 3.90$\pm$0.10 | 0.13$\pm$0.20 | 3.20$\pm$0.5 | -16.20$\pm$2.00 | 14$\pm$3

###  3.3 Abundance analysis

We derived the abundance of each atomic species using the measured equivalent
widths in WIDTH9 (Kurucz, 2005a; Sbordone et al., 2004), which uses ATLAS9
model atmospheres to compute a line formation in Local Thermodynamic
Equilibrium. We avoided using lines of equivalent widths greater than 200 mÅ
for the abundance analysis to avoid the use of saturated lines that bias the
abundances. We included hyperfine structure (HFS) splitting for the V, Sc, Mn,
Y, and Ba spectral lines. HFS split lines due to electromagnetic multipole
interaction between the nucleus and electrons, which arises from the energy of
the nuclear magnetic dipole moment interacting with the magnetic field
generated by the electrons, and shifts the lines of different isotopes. HFS
has a small broadening effect, shifting the lines and altering their shape,
which may lead to erroneous interpretations of spectra with the equivalent
width method.

To measure the abundance of HFS elements, we generated synthetic spectra using
SYNTHE (Kurucz, 1993b, 2005b) with HFS line lists from the Kurucz database,
and we fitted them with the observed spectra lines. The spectra generation
includes the broadening effects due to the instrumental profile and turbulent
velocity. The abundance of the best fit between the observed and synthetic
spectrum was adopted. The elemental abundances have random ($\sigma_{r}$) and
systematic errors. The random errors are from the standard deviations of
individual measurements. The systematic error of the chemical abundances due
to atmosphere parameters is

$[\sigma_{sys}\text{ =
}\sqrt{\sigma_{Teff}^{2}+\sigma_{logg}^{2}+\sigma_{\xi}^{2}}]$ (1)

The total error is

$[\sigma_{tot}\text{ = }\sqrt[3]{\sigma_{sys}^{2}+\sigma_{r}^{2}}\text{
/}\sqrt{N}]$ (2)

where N is the number of lines used to derive the chemical abundances. We
present the elemental abundances along with the number of lines and their
total errors in Tables 4 and 5. The solar abundance ((log $\epsilon_{\sun}$) )
values are taken from Grevesse & Sauval (1998). Figs 2, 3, and 4 show the
differential to solar abundances with total error bars. The abundance patterns
of HD 4321, HD 125658, and HD 154713 show an agreement with the abundance
patterns of classical Am stars, with underabundant Ca and Sc, overabundant
heavier elements, and moderate over-abundance of iron-peak elements. We
compared these patterns with the classical Am HD 33266 studied by Çalışkan et
al. (2015). We find that HD 137928 and HD 159834 have similar abundance
characteristics to marginal Am-type stars. We also compared their patterns
with the marginal Am HD 185658 (KIC 9349245) from Trust et al. (2021). The
chemical element distributions of HD 2924 and the giant HD 26553 are
consistent with the pattern of normal A-type stars.

Table 5: Derived elemental abundances with the total error estimates of HD 2924 and HD 26553. Atomic Sp. | HD 2924 | HD 2924 | HD 26553 | HD 26553 | Solar
---|---|---|---|---|---
| log $\epsilon$ | [x/H] | log $\epsilon$ | [x/H] | log $\epsilon_{\sun}$
C I | 8.45 | -0.07$\pm$0.11 (6) | 8.37 | -0.15$\pm$0.12 (19) | 8.52
N I | 8.05 | 0.13$\pm$0.04 (1) | 8.20 | 0.28$\pm$0.09 (2) | 7.92
O I | 8.95 | 0.12$\pm$0.04 (1) | 8.91 | 0.08$\pm$0.04 (8) | 8.83
Na I | $\dots$ | $\dots$ | 6.54 | 0.21$\pm$0.17 (3) | 6.33
Mg I | 7.55 | -0.03$\pm$0.22 (4) | 7.77 | 0.19$\pm$0.19 (4) | 7.58
Mg II | 7.62 | 0.04$\pm$0.14 (1) | 7.80 | 0.22$\pm$0.26 (2) | 7.58
Al II | $\dots$ | $\dots$ | 6.72 | 0.25$\pm$0.05 (1) | 6.47
Si I | $\dots$ | $\dots$ | 7.68 | 0.13$\pm$0.15 (8) | 7.55
Si II | 7.48 | -0.07$\pm$0.14 (2) | 7.50 | -0.05$\pm$0.12 (2) | 7.55
S I | $\dots$ | $\dots$ | 7.55 | 0.22$\pm$0.15(6) | 7.33
K I | $\dots$ | $\dots$ | 5.33 | 0.21$\pm$0.18(1) | 5.12
Ca I | 6.49 | 0.13$\pm$0.19 (7) | 6.40 | 0.04$\pm$0.24 (17) | 6.36
Ca II | $\dots$ | $\dots$ | 6.37 | 0.01$\pm$0.13 (4) | 6.36
Sc IIHFS | 3.09 | -0.08$\pm$0.16 (3) | 3.04 | -0.13$\pm$0.21 (11) | 3.17
Ti I | $\dots$ | $\dots$ | 5.01 | -0.01$\pm$0.20 (4) | 5.02
Ti II | 5.31 | 0.29$\pm$0.26 (1) | 4.99 | -0.03$\pm$0.19 (38) | 5.02
V IIHFS | 4.17 | 0.17$\pm$0.22 (1) | 4.02 | 0.02$\pm$0.15 (9) | 4.00
Cr I | 5.76 | 0.09$\pm$0.26 (2) | 5.72 | 0.05$\pm$0.19(9) | 5.67
Cr II | 5.85 | 0.18$\pm$0.16 (10) | 5.71 | 0.04$\pm$0.17(39) | 5.67
Mn IHFS | 5.30 | -0.09$\pm$0.16 (1) | 5.29 | -0.10$\pm$0.19(8) | 5.39
Mn IIHFS | 5.30 | -0.09$\pm$0.04 (1) | 5.26 | -0.13$\pm$0.07(7) | 5.39
Fe I | 7.62 | 0.12$\pm$0.18 (42) | 7.58 | 0.08$\pm$0.19(202) | 7.50
Fe II | 7.64 | 0.14$\pm$0.18 (46) | 7.58 | 0.08$\pm$0.11(102) | 7.50
Co II | $\dots$ | $\dots$ | 5.15 | 0.23$\pm$0.06(1) | 4.92
Ni I | $\dots$ | $\dots$ | 6.19 | -0.06$\pm$0.17(20) | 6.25
Ni II | $\dots$ | $\dots$ | 6.20 | -0.05$\pm$0.06(3) | 6.25
Cu I | $\dots$ | $\dots$ | 4.53 | 0.32$\pm$0.16(1) | 4.21
Zn I | 4.83 | 0.23$\pm$0.13 (1) | 4.70 | 0.10$\pm$0.17(2) | 4.60
Sr II | 3.31 | 0.34$\pm$0.37 (2) | 3.44 | 0.47$\pm$0.37(3) | 2.97
Y IIHFS | 2.62 | 0.38$\pm$0.15 (1) | 2.37 | 0.13$\pm$0.17(5) | 2.24
Zr II | 3.21 | 0.61$\pm$0.32 (1) | 2.75 | 0.15$\pm$0.15(8) | 2.60
Ba IIHFS | 2.41 | 0.28$\pm$0.24 (1) | 2.43 | 0.30$\pm$0.34(4) | 2.13
La II | $\dots$ | $\dots$ | 1.61 | 0.44$\pm$025(2) | 1.17
Ce II | $\dots$ | $\dots$ | 1.72 | 0.14$\pm$0.18(1) | 1.58
Table 6: Derived elemental abundances with the total error estimates of HD 4321, HD 125658, HD 137928, HD 154713 and HD 159834. Atomic Sp. | HD 4321 | HD 4321 | HD 125658 | HD 125658 | HD 137928 | HD 137928 | HD 154713 | HD 154713 | HD 159834 | HD 159834 | Solar
---|---|---|---|---|---|---|---|---|---|---|---
| log $\epsilon$ | [x/H] | log $\epsilon$ | [x/H] | log $\epsilon$ | [x/H] | log $\epsilon$ | [x/H] | log $\epsilon$ | [x/H] | log $\epsilon_{\sun}$
C I | 8.72 | 0.20$\pm$0.07 (11) | 8.72 | 0.20$\pm$0.09(7) | 8.73 | 0.21$\pm$0.08(2) | 8.42 | -0.10$\pm$0.11(3) | 8.46 | -0.06$\pm$0.08(11) | 8.52
O I | 8.88 | 0.05$\pm$0.25 (1) | 8.58 | -0.25$\pm$0.18(2) | 9.03 | 0.20$\pm$0.01(1) | $\dots$ | $\dots$ | 8.72 | -0.11$\pm$0.05(1) | 8.83
Na I | 6.73 | 0.40$\pm$0.12 (3) | 6.71 | 0.38$\pm$0.10(2) | $\dots$ | $\dots$ | $\dots$ | $\dots$ | 6.40 | 0.07$\pm$0.14(2) | 6.33
Mg I | 7.57 | -0.01$\pm$0.15 (4) | 7.73 | 0.15$\pm$0.14(3) | 7.78 | 0.20$\pm$0.21(4) | 7.25 | -0.19$\pm$0.22(4) | 7.49 | -0.09$\pm$0.13(4) | 7.58
Mg II | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | 7.62 | 0.04$\pm$0.10(2) | 7.58
Al II | $\dots$ | $\dots$ | 6.70 | 0.23$\pm$0.03(1) | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | 6.47
Si I | 7.63 | 0.08$\pm$0.14 (3) | 7.98 | 0.43$\pm$0.12(7) | $\dots$ | $\dots$ | $\dots$ | $\dots$ | 7.75 | 0.20$\pm$0.11(5) | 7.55
Si II | 7.56 | 0.01$\pm$0.18 (1) | $\dots$ | $\dots$ | 7.77 | 0.22$\pm$0.17(3) | 7.43 | -0.12$\pm$0.17(3) | $\dots$ | $\dots$ | 7.55
S I | 7.66 | 0.33$\pm$0.12 (5) | 7.71 | 0.38$\pm$0.11(2) | 7.69 | 0.36$\pm$0.10(1) | 7.64 | 0.31$\pm$0.10(1) | 7.41 | 0.08$\pm$0.09(1) | 7.33
K I | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | 5.13 | 0.01$\pm$0.13(1) | 5.12
Ca I | 6.20 | -0.16$\pm$0.16 (10) | 6.25 | -0.11$\pm$0.15(11) | 6.47 | 0.11$\pm$0.16(3) | 6.05 | -0.31$\pm$0.18(6) | 6.28 | -0.08$\pm$0.17(12) | 6.36
Ca II | 6.31 | -0.05$\pm$0.05 (2) | 6.27 | -0.09$\pm$0.04(2) | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | 6.36
Sc IIHFS | 2.62 | -0.55$\pm$0.14 (5) | 2.76 | -0.41$\pm$0.18(3) | 2.79 | -0.38$\pm$0.19(2) | 2.63 | -0.54$\pm$0.11(2) | 2.86 | -0.27$\pm$0.12(5) | 3.17
Ti II | 5.23 | 0.21$\pm$0.22 (18) | 5.39 | 0.37$\pm$0.10(17) | 5.31 | 0.29$\pm$0.18(11) | 5.21 | 0.19$\pm$0.23(11) | 5.39 | 0.37$\pm$0.22(22) | 5.02
V IIHFS | 4.18 | 0.18$\pm$0.12 (2) | 4.42 | 0.42$\pm$0.08(2) | 4.52 | 0.52$\pm$0.13(1) | 4.21 | 0.21$\pm$0.11(1) | 4.31 | 0.31$\pm$0.12(3) | 4.00
Cr I | 5.84 | 0.17$\pm$0.15 (5) | 6.17 | 0.50$\pm$0.15(3) | $\dots$ | $\dots$ | 5.79 | 0.12$\pm$0.27(2) | $\dots$ | $\dots$ | 5.67
Cr II | 5.87 | 0.20$\pm$0.11 (24) | 6.07 | 0.40$\pm$0.07(19) | 6.15 | 0.48$\pm$0.12(18) | 5.83 | 0.16$\pm$0.25(5) | 5.86 | 0.19$\pm$0.12(21) | 5.67
Mn IHFS | 5.62 | 0.23$\pm$0.18 (2) | 5.91 | 0.32$\pm$0.13(4) | 5.81 | 0.42$\pm$0.15(3) | 5.58 | 0.19$\pm$0.16(1) | 5.60 | 0.21$\pm$0.13(3) | 5.39
Mn IIHFS | 5.61 | 0.22$\pm$0.03 (1) | 5.98 | 0.39$\pm$0.08(2) | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | 5.39
Fe I | 7.64 | 0.14$\pm$0.15 (106) | 7.93 | 0.43$\pm$0.13(129) | 7.92 | 0.42$\pm$0.15(64) | 7.68 | 0.18$\pm$0.20(59) | 7.66 | 0.16$\pm$0.16(108) | 7.50
Fe II | 7.62 | 0.12$\pm$0.15 (49) | 7.89 | 0.39$\pm$0.08(50) | 7.91 | 0.41$\pm$0.14(40) | 7.68 | 0.18$\pm$0.21(36) | 7.65 | 0.15$\pm$0.16(59) | 7.50
Ni I | 6.46 | 0.21$\pm$0.14 (6) | 6.95 | 0.70$\pm$0.12(13) | 6.90 | 0.65$\pm$0.13(4) | 6.47 | 0.22$\pm$0.15(6) | 6.55 | 0.30$\pm$0.13(10) | 6.25
Cu I | $\dots$ | $\dots$ | 4.93 | 0.72±0.13(1) | $\dots$ | $\dots$ | $\dots$ | $\dots$ | 4.53 | 0.32±0.21(2) | 4.21
Zn I | 5.08 | 0.48$\pm$0.14 (2) | 5.31 | 0.71$\pm$0.12(2) | 5.38 | 0.78$\pm$0.13(1) | 4.79 | 0.19$\pm$0.14(1) | 5.05 | 0.45$\pm$0.17(2) | 4.60
Sr II | 3.59 | 0.62$\pm$0.39 (2) | 3.46 | 0.49$\pm$0.47(1) | 3.67 | 0.70$\pm$0.51(1) | 3.82 | 0.85$\pm$0.48(1) | 3.94 | 0.97$\pm$0.51(1) | 2.97
Y IIHFS | 2.56 | 0.32$\pm$0.11 (2) | 3.51 | 1.07$\pm$0.10(5) | 3.26 | 1.02$\pm$0.15(3) | $\dots$ | $\dots$ | 2.81 | 0.57$\pm$0.17(7) | 2.24
Zr II | 3.23 | 0.63$\pm$0.14 (2) | 4.08 | 1.48$\pm$0.13(3) | 3.81 | 1.21$\pm$0.11(3) | 3.28 | 0.68$\pm$0.18(2) | 3.51 | 0.91$\pm$0.10(3) | 2.60
Ba IIHFS | 3.06 | 0.83$\pm$0.27 (3) | 3.61 | 1.18$\pm$0.23(3) | 3.36 | 1.23$\pm$0.22(2) | 2.92 | 0.79$\pm$0.29(3) | 3.02 | 0.89$\pm$0.30(2) | 2.13
La II | 2.45 | 1.28$\pm$0.14 (1) | 2.58 | 1.41$\pm$0.12(1) | $\dots$ | $\dots$ | $\dots$ | $\dots$ | 2.68 | 1.51$\pm$0.19(1) | 1.17
Ce II | 1.94 | 0.36$\pm$0.12 (1) | 2.99 | 1.41$\pm$0.11(1) | $\dots$ | $\dots$ | $\dots$ | $\dots$ | 2.77 | 1.19$\pm$0.15(4) | 1.58
Nd II | 2.55 | 1.05$\pm$0.20 (2) | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | 1.50
Sm II | 2.97 | 1.96$\pm$0.14 (1) | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | 2.59 | 1.58$\pm$0.13(1) | 1.01
Eu II | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | 1.42 | 0.91$\pm$0.14(1) | 0.51
Gd II | 2.90 | 1.78$\pm$0.10 (1) | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | 1.12
Figure 2: The differences between our derived chemical abundances and the
solar abundances found by Grevesse & Sauval (1998) for HD 2924 and HD 26553
Figure 3: The differences between our derived chemical abundances and the
solar abundances found by Grevesse & Sauval (1998) for the classic Am stars HD
4321, HD 125658, and HD 154713. HD 33266 is a classical Am star studied by
Çalışkan et al. (2015). Figure 4: The differences between our derived chemical
abundances and the solar abundances found by Grevesse & Sauval (1998) for the
marginal Am stars HD 137928 and HD 159834. HD 185658 is a marginal Am star
studied by Trust et al. (2021)

##  4 EVOLUTIONARY STATUS

To estimate the evolutionary status of the target stars, we precisely
pinpointed their positions on the HR diagram. This is of particular importance
as it allowed us to investigate the region occupied by the Am stars. With the
help of isochrones and evolutionary tracks with solar metallicity (Bressan et
al., 2012), we estimated the stellar masses and ages. The luminosity
($\text{L/L}_{\sun}$) was calculated using the standard technique. We obtained
E(B-V) from the work of Kervella et al. (2019) and computed extinction with
the standard extinction law

$[{A}_{\text{V}}\text{= 3.1 \texttimes E(B-V)}]$ (3)

The bolometric correction BC was calculated using the temperature-dependent
function of Torres (2010). The absolute magnitudes (MV) were determined using
the Gaia parallaxes (Gaia Collaboration, 2020). The errors in luminosities are
from uncertainties in ${T}_{\text{eff}}$, V-band indices, and Gaia parallaxes.
Using the Stefan–Boltzmann law as in Montambaux (2018), we calculated the
radius from the ${T}_{\text{eff}}$ and log ($\text{L/L}_{\sun}$) values. The
E(B-V) values and calculated parameters are listed in Table 7. The determined
masses, ages and radii are listed in Table 8.

Table 7: The logarithmic effective temperature log ${T}_{\text{eff}}$ (this study), E(B-V) (Kervella et al. (2019)), bolometric correction BC (Torres (2010)), absolute magnitude, MV (this study), parallax, $\pi$, (Gaia Collaboration, 2020) and luminosity, log($\text{L/L}_{\sun}$) (this study). Star name | log ${T}_{\text{eff}}$ |  $\pi$ (mas) |  E(B-V) | BC | MV | log($\text{L/L}_{\sun}$)
---|---|---|---|---|---|---
HD 2924 | 3.9370 [0.0075] | 4.3978 [0.0292] | 0.040 | -0.0156 | -0.2078 | 1.9453 [0.0100]
HD 4321 | 3.9294 [0.0076] | 5.9672 [0.0209] | 0.034 | -0.0001 | 0.2834 | 1.7427 [0.0078]
HD 26553 | 3.9217 [0.0077]] | 1.5391 [0.0370] | 0.170 | 0.0120 | -3.5107 | 3.2555 [0.0215]
HD 125658 | 3.9445 [0.0072] | 11.9629 [0.1441] | 0.007 | -0.0341 | 1.6918 | 1.1929 [0.0147]
HD 137928 | 3.9590 [0.0071] | 8.1997 [0.0243] | 0.014 | -0.0341 | 0.9556 | 1.5051 [0.0111]
HD 154713 | 3.9217 [0.0077] | 6.3445 [0.0227] | 0.021 | 0.0120 | 0.3878 | 1.6961 [0.0070]
HD 159834 | 3.9217 [0.0077] | 6.4269 [0.1041] | 0.023 | 0.0120 | 0.0687 | 1.8237 [0.0153]
Figure 5: The position of target stars on the HR diagram. Evolutionary tracks and isochrones are from Bressan et al. (2012). Table 8: The masses, ages, and radii of the target stars. Star name | Mass [$M_{\sun}$] | Age [Myr] | Radius [$R_{\sun}$]
---|---|---|---
HD 2924 | 2.40 [0.09] | 490 [40] | 04.20 [0.40]
HD 4321 | 2.00 [0.08] | 625 [50] | 03.40 [0.30]
HD 26553 | 5.10 [0.30] | 090 [05] | 20.30 [2.20]
HD 125658 | 1.75 [0.06] | 370 [40] | 01.80 [0.20]
HD 137928 | 2.00 [0.07] | 550 [50] | 02.30 [0.25]
HD 154713 | 2.00 [0.08] | 675 [60] | 03.40 [0.30]
HD 159834 | 2.20 [0.07] | 600 [50] | 03.90 [0.40]

##  5 Results

###  5.1 Normal A-type stars

####  5.1.1 HD 2924

This star was classified as spectral type A2IV (Cowley et al., 1969). Zorec &
Royer (2012) measured ${T}_{\text{eff}}$ as 8730 K (log ${T}_{\text{eff}}$ =
3.941) and reported log ($\text{L/L}_{\sun}$) = 2.3083, mass = 3.06
$M_{\sun}$, and a rotational velocity of 30 km s-1. McDonald et al. (2012)
measured the ${T}_{\text{eff}}$ as 7800 K and reported a luminosity of 134.59
(log($\text{L/L}_{\sun}$) = 2.129). Duflot et al. (1995) measured a radial
velocity of 1.9 km s-1, whereas de Bruijne & Eilers (2012) and Brandt (2021)
reported a radial velocity of 2.1 km s-1. In this study, we measured the
fundamental parameters of HD 2924 as ${T}_{\text{eff}}$ = 8650 K, log $g$ =
3.7 and $\xi$ = 2.60 km s-1. We derived $v$ sin $i$ and ${V}_{\text{helio}}$
as 30 km s-1 and 1.90 km s-1, respectively. The abundance of calcium is 0.13
dex and scandium -0.08 dex. The iron-peak and heavy elements do not show
significant overabundance as we expect for Am stars, hence we classified HD
2924 as a normal A-type star. The elemental abundance distribution of HD 2924
is shown in Fig. 2. We estimated the mass, age, and radius as 2.40$\pm$0.09
$M_{\sun}$, 490$\pm$60 Myr, and 4.20$\pm$0.40 $R_{\sun}$, respectively. The
difference between the calculated luminosity in this study and the luminosity
given in Zorec & Royer (2012) is largely due to the parallax values used in
the calculation of luminosities.

####  5.1.2 HD 26553

HD 26553 was listed as A4III by Cowley et al. (1969). de Bruijne & Eilers
(2012) and Brandt (2021) reported a radial velocity of -24.9 km s-1. Royer et
al. (2002) measured $v$ sin $i$ as 18 km s-1. We determined the atmospheric
parameters of HD 26553 as follows; ${T}_{\text{eff}}$ = 8350 K, log $g$ = 2.2,
and $\xi$ = 1.9 km s-1. We measured $v$ sin $i$ and ${V}_{\text{helio}}$ as 12
km s-1 and 22 km s-1, respectively. With [Ca/H] = 0.04 dex, [Sc/H] = -0.13 dex
and an average abundance of iron-peak and heavy elements, the star exhibits
solar-like abundances. The chemical abundance pattern, shown in Fig. 2, does
not show Am characteristics and is consistent with the pattern of normal
A-type stars. The physical parameters were estimated as mass = 5.1±0.3
$M_{\sun}$, age = 90$\pm$5 Myr and radius = 20.3±2.2 $R_{\sun}$. The HR
diagram reveals that HD 26553 is a post-TAMS star.

###  5.2 Classical Am stars

####  5.2.1 HD 4321

HD 4321 was listed as an A2III star in the HD catalogue of Cannon & Pickering
(1993). The star’s radial velocity was measured as -8.3 km s-1 (de Bruijne &
Eilers, 2012; Brandt, 2021). McDonald et al. (2012) reported an effective
temperature of 7787 K and a luminosity of 50.05 $L_{\sun}$ (log
($\text{L/L}_{\sun}$)=1.70). Royer et al. (2002) measured $v$ sin $i$ as 24 km
s-1. We measured the star’s ${T}_{\text{eff}}$, log $g$, and $\xi$ as 8500 K,
4.00, and 3.00 km s-1, respectively. We measured the $v$ sin $i$ and
${V}_{\text{helio}}$ of HD 4321 as 24 km s-1 and -7.6 km s-1. Its [Ca/H] and
[Sc/H] values were found to be underabundant with respect to the solar
abundances by –0.16 and –0.55 dex. The star exhibits an overabundance (on
average) in iron-peak and rare-earth elements with respect to the solar
abundances. Its chemical abundance pattern, shown in Fig. 3, implies a
classical Am star. The mass of HD 4321 was estimated as 2.00$\pm$0.08
$M_{\sun}$. We estimated the age and the radius as 625$\pm$60 Myr and
3.40$\pm$0.30 $R_{\sun}$, respectively.

####  5.2.2 HD 125658

This target was first noted as an Am-type candidate by Bertaud (1958). Cowley
et al. (1969) classified HD 125658 as A5III. It was listed as an A2 star in
the HD catalogue (Cannon & Pickering, 1993). de Bruijne & Eilers (2012) and
Brandt (2021) measured the radial velocity as 1.3 km s-1. McDonald et al.
(2012) derived the ${T}_{\text{eff}}$ as 8123 K and the luminosity as 17.9
$L_{\sun}$ (log ($\text{L/L}_{\sun}$) = 1.25). David & Hillenbrand (2015)
reported ${T}_{\text{eff}}$ as 8430 K, log $g$ as 4.27, the mean age from 1D
posterior as 581 Myr, and the mass as 1.9 $M_{\sun}$. From 2D linear
interpolation, the age and mass were measured as 363 Myr and 1.78 $M_{\sun}$,
respectively. Royer et al. (2002) measured a $v$ sin $i$ of 26 km s-1. We
measured the atmospheric parameters of HD 125658 as ${T}_{\text{eff}}$ = 8800
K, log $g$ = 4.60, and $\xi$ = 5.07 km s-1. Its $v$ sin $i$ and heliocentric
radial velocity ${V}_{\text{helio}}$ were derived as 26 km s-1 and 1.66 km
s-1. HD 125658 has characteristic abundances consistent with the abundance of
classical Am-type stars. The underabundance of Ca at -0.11 and Sc at -0.41 and
the overabundance pattern in iron-peak and heavy elements with respect to the
solar abundances are shown in Fig. 3. We note that the star has significantly
higher iron, nickel, copper, and zinc abundances than the other two Am-type
samples. The mass, age, and radius were estimated as 1.75$\pm$0.06 $M_{\sun}$,
370$\pm$40 Myr, and 1.80 $\pm$0.20$~{}R_{\sun}$, respectively.

####  5.2.3 HD 154713

HD 154713 was listed as an A0 star in the HD catalogue (Cannon & Pickering,
1993). The spectral type of HD 154713 was classified as A3IV (Cowley et al.,
1969). Zorec & Royer (2012) measured ${T}_{\text{eff}}$ as 8550 K (log
${T}_{\text{eff}}$ = 3.932) and reported log ($\text{L/L}_{\sun}$) = 1.81,
mass = 2.52 $M_{\sun}$, and a rotational velocity of 39 km s-1. McDonald et
al. (2012) measured the ${T}_{\text{eff}}$ as 7954 K and reported a luminosity
of 47.48 $L_{\sun}$ (log ($\text{L/L}_{\sun}$) = 1.679). de Bruijne & Eilers
(2012) and Brandt (2021) reported a radial velocity of -8.7 km s-1. We
determined the atmospheric parameters of HD 154713 as ${T}_{\text{eff}}$ =
8350 K, log $g$ = 3.50, and $\xi$ = 3.00 km s-1. We derived the values of $v$
sin $i$ and ${V}_{\text{helio}}$ as 39 km s-1 and -6.30 km s-1, respectively.
The abundance of calcium is -0.31 dex and scandium -0.54 dex. The iron-peak
and heavy element abundances are slightly overabundant with respect to solar;
hence, HD 154713 is consistent with classical Am-type stars. The abundance
pattern is shown in Fig. 3. We estimated the mass, age, and radius as
2.00$\pm$0.08 $M_{\sun}$, 675$\pm$60 Myr, and 3.40$\pm$0.40 $R_{\sun}$,
respectively.

###  5.3 Marginal Am stars

####  5.3.1 HD 137928

This star was ascribed a spectral type A2 IV (Cowley et al., 1969). Zorec &
Royer (2012) measured ${T}_{\text{eff}}$ as 8649 K (log ${T}_{\text{eff}}$ =
3.937) and reported log ($\text{L/L}_{\sun}$) = 1.744, mass = 2.45 $M_{\sun}$,
and $v$ sin $i$ = 34 km s-1. McDonald et al. (2012) measured the
${T}_{\text{eff}}$ as 8637 K and reported a luminosity of 32.62 $L_{\sun}$
(log ($\text{L/L}_{\sun}$) = 1.513). de Bruijne & Eilers (2012) and Brandt
(2021) reported a radial velocity of -4.90km s-1. Kervella et al. (2019)
measured the radial velocity as -4.985km s-1. We determined the
${T}_{\text{eff}}$ of HD 137928 as 9100 K. The log $g$ and $\xi$ were measured
as 3.8 and 4.03 km s-1, respectively. The $v$ sin $i$ was determined as 34 km
s-1 and ${V}_{\text{helio}}$ as -3.9 km s-1. The abundance of calcium is 0.11
dex and that of scandium is -0.38 dex. The Ca abundance is nearly solar-like,
while Sc is slightly underabundant. Iron-peak and heavy elements have an
overabundance implying marginal Am star abundance characteristics. HD 137928
has high iron, nickel, and zinc abundances. This makes it a metal-rich star.
The elemental abundance pattern of HD 137928 is shown in Fig. 4. Its mass was
determined as 2.00$\pm$0.07 $M_{\sun}$. HD 137928’s radius was measured as
2.30$\pm$0.25 $R_{\sun}$ at the age of 550$\pm$50 Myr.

####  5.3.2 HD 159834

This star was first classified as A7III by Eggen (1962). Cowley et al. (1969)
reported the spectral type as A7IV. HD 159834 was listed as an A2 star in the
HD catalogue (Cannon & Pickering, 1993). Zorec & Royer (2012) measured its
${T}_{\text{eff}}$ as 7925 K (log ${T}_{\text{eff}}$ = 3.899) and reported a
luminosity of log ($\text{L/L}_{\sun}$) = 1.6256, a mass of 2.29 $M_{\sun}$,
and a rotational velocity of km s-1. McDonald et al. (2012) measured the
${T}_{\text{eff}}$ as 7724 K and reported a luminosity of 34.94 $L_{\sun}$
(log ($\text{L/L}_{\sun}$) = 1.54). de Bruijne & Eilers (2012) and Brandt
(2021) reported a radial velocity of -17 km s-1. Burkhart & Coupry (1991)
measured the ${T}_{\text{eff}}$, log $g$ and [Fe/H] as 8030 K, 3.7, and 0.1
dex, respectively. Kervella et al. (2019) measured a radial velocity of –16.82
km s-1. Finally, the atmospheric parameters of HD 159834 were determined as
${T}_{\text{eff}}$ = 8350 K, log $g$ = 3.9, and $\xi$ = 3.2 km s-1. We
measured $v$ sin $i$ and ${V}_{\text{helio}}$ as 14 km s-1 and -16.20 km s-1.
The calcium abundance is –0.08 dex, while that of scandium is –0.27 dex. The
abundances are close to solar, with scandium slightly underabundant. The iron-
peak and heavy elements are moderately overabundant, implying a marginal Am
star. The elemental abundance pattern is shown in Fig. 4. We determined the
mass, age, and radius as 2.20$\pm$0.07 $M_{\sun}$, 600$\pm$50 Myr, and
3.90$\pm$0.4 $R_{\sun}$, respectively.

##  6 Conclusion

In this paper, we present a chemical abundance analysis of seven A-type stars,
for which detailed abundance analyses are absent in the literature, to search
for their chemical anomalies. These stars are slow-rotating A-type stars. Slow
rotators were targeted because rotational velocity is linked to the mixing
process of stellar materials that suppress the effect of chemical separation.
We first obtained fundamental atmospheric parameters and velocities for each
target, including the effective temperature (${T}_{\text{eff}}$), surface
gravity (log $g$), metallicity [Fe/H], microturbulent velocity ($\xi$),
rotational velocity ($v$ sin $i$), and radial velocity (${V}_{\text{helio}}$).
We then performed a detailed abundance analysis for each star, obtained their
luminosity log ($\text{L/L}_{\sun}$), and pinpointed their position in the HR
diagram. Using evolutionary tracks and isochrones, we estimated the masses and
the ages of the targets. Using the Teff and luminosity values, we calculated
the radii of the stars. The present work is the first detailed abundance
analysis of these targets. Six of the targets are main–sequence stars with
some of them near to the Terminal-Age Main Sequence, while HD 26553 was
revealed to be a post-TAMS giant. Based on the spectral analysis, the
resulting chemical abundance pattern of five targets revealed that they are
Am-type stars, with HD 4321, HD 125658, and HD 154713 classified as classical
Am stars, while HD 137928 and HD 159834 were classified as marginal Am stars.
The abundance patterns of HD 2924 and HD 26553 are consistent with those of
normal A-type stars. Two of our samples, HD 125658 (classified as a classical
Am star) and HD 137928 (classified as a marginal Am star), have high iron,
nickel, and zinc abundances. Their carbon, sodium, magnesium, sulphur,
scandium, vanadium, manganese, yttrium, and barium abundances are very
similar. They also have very similar atmospheric and physical parameters.
Considering all these similarities, these two stars are among the most
metal–rich Am stars. The HR diagram shows that Am stars populate the main
sequence. On average based on the chemical separation theory, the main-
sequence slow rotator HD 2924 is expected to exhibit Am characteristics. The
presence of its non-Am characteristics poses a challenge on rotation effect to
the hypothesis chemical separation, which shows that there is a need to study
more of these targets to explore the origin of their peculiarity and their
interior processes.

## Acknowledgements

The authors acknowledge the TÜBİTAK National Observatory (TUG) with the
project ID 14BRTT150–671.

## Data Availability

The data underlying this article are available in the article and in its
online supplementary material. Upon request, further data underlying this
article will be shared on reasonable request to the corresponding author.

## References

  * Abt (2000) Abt H. A., 2000, ApJ, 544, 933
  * Abt (2009) Abt H. A., 2009, PASP, 121, 1291
  * Abt & Morgan (1976) Abt H. A., Morgan W. W., 1976, ApJ, 205, 446
  * Abt & Moyd (1973) Abt H. A., Moyd K. I., 1973, ApJ, 182, 809
  * Adelman & Unsuree (2007) Adelman S. J., Unsuree N., 2007, Baltic Astronomy, 16, 183
  * Bertaud (1958) Bertaud C., 1958, Journal des Observateurs, 42, 45
  * Brandt (2021) Brandt T. D., 2021, ApJS, 254, 42
  * Bressan et al. (2012) Bressan A., Marigo P., Girardi L., Salasnich B., Dal Cero C., Rubele S., Nanni A., 2012, MNRAS, 427, 127
  * Burkhart & Coupry (1991) Burkhart C., Coupry M. F., 1991, A&A, 249, 205
  * Cannon & Pickering (1993) Cannon A. J., Pickering E. C., 1993, VizieR Online Data Catalog, p. III/135A
  * Cowley et al. (1969) Cowley A., Cowley C., Jaschek M., Jaschek C., 1969, AJ, 74, 375
  * David & Hillenbrand (2015) David T. J., Hillenbrand L. A., 2015, ApJ, 804, 146
  * Duflot et al. (1995) Duflot M., Fehrenbach C., Mannone C., Burnage R., Genty V., 1995, A&AS, 110, 177
  * Eggen (1962) Eggen O. J., 1962, Royal Greenwich Observatory Bulletins, 51, 79
  * Fossati et al. (2008) Fossati L., Bagnulo S., Landstreet J., Wade G., Kochukhov O., Monier R., Weiss W., Gebran M., 2008, A&A, 483, 891
  * Frebel et al. (2013) Frebel A., Casey A. R., Jacobson H. R., Yu Q., 2013, ApJ, 769, 57
  * Gaia Collaboration (2020) Gaia Collaboration 2020, VizieR Online Data Catalog, p. I/350
  * Grevesse & Sauval (1998) Grevesse N., Sauval A. J., 1998, Space Sci. Rev., 85, 161
  * Kervella et al. (2019) Kervella P., Arenou F., Mignard F., Thévenin F., 2019, A&A, 623, A72
  * Khokhlova (1981) Khokhlova V. L., 1981, in Liege International Astrophysical Colloquia. pp 457–463
  * Kochukhov (2018) Kochukhov O., 2018, BinMag: Widget for comparing stellar observed with theoretical spectra, Astrophysics Source Code Library, record ascl:1805.015 (ascl:1805.015)
  * Kupka et al. (1999) Kupka F., Piskunov N., Ryabchikova T. A., Stempels H. C., Weiss W. W., 1999, A&AS, 138, 119
  * Kurucz (1993a) Kurucz R., 1993a, ATLAS9 Stellar Atmosphere Programs and 2 km/s grid. Kurucz CD-ROM No. 13. Cambridge, 13
  * Kurucz (1993b) Kurucz R., 1993b, SYNTHE Spectrum Synthesis Programs and Line Data. Kurucz CD-ROM No. 18. Cambridge, 18
  * Kurucz (2005a) Kurucz R. L., 2005a, Memorie della Societa Astronomica Italiana Supplementi, 8, 14
  * Kurucz (2005b) Kurucz R. L., 2005b, Memorie della Societa Astronomica Italiana Supplementi, 8, 189
  * Kurucz & Bell (1995) Kurucz R. L., Bell B., 1995, Atomic line list
  * McDonald et al. (2012) McDonald I., Zijlstra A. A., Boyer M. L., 2012, MNRAS, 427, 343
  * Michaud (1982) Michaud G., 1982, ApJ, 258, 349
  * Michaud et al. (2015) Michaud G., Alecian G., Richer J., 2015, Atomic Diffusion in Stars, doi:10.1007/978-3-319-19854-5.
  * Monier (2019) Monier R., 2019, Research Notes of the American Astronomical Society, 3, 167
  * Montambaux (2018) Montambaux G., 2018, Foundations of Physics, 48, 395
  * Napiwotzki et al. (1993) Napiwotzki R., Schoenberner D., Wenske V., 1993, A&A, 268, 653
  * Oja (1991) Oja T., 1991, A&AS, 89, 415
  * Paunzen (2015) Paunzen E., 2015, A&A, 580, A23
  * Piskunov et al. (1995) Piskunov N. E., Kupka F., Ryabchikova T. A., Weiss W. W., Jeffery C. S., 1995, A&AS, 112, 525
  * Preston (1974) Preston G. W., 1974, ARA&A, 12, 257
  * Roman et al. (1948) Roman N. G., Morgan W. W., Eggen O. J., 1948, ApJ, 107, 107
  * Royer et al. (2002) Royer F., Grenier S., Baylac M. O., Gómez A. E., Zorec J., 2002, A&A, 393, 897
  * Ryabchikova et al. (1999) Ryabchikova T. A., Piskunov N. E., Stempels H. C., Kupka F., Weiss W. W., 1999, Physica Scripta Volume T, 83, 162
  * Sbordone et al. (2004) Sbordone L., Bonifacio P., Castelli F., Kurucz R. L., 2004, Memorie della Societa Astronomica Italiana Supplementi, 5, 93
  * Takeda et al. (2008) Takeda Y., Han I.-W., Kang D.-I., Lee B.-C., Kim K.-M., 2008, Journal of Korean Astronomical Society, 41, 83
  * Titus & Morgan (1940) Titus J., Morgan W. W., 1940, ApJ, 92, 256
  * Torres (2010) Torres G., 2010, AJ, 140, 1158
  * Trust et al. (2021) Trust O., Jurua E., De Cat P., Joshi S., Lampens P., 2021, MNRAS, 504, 5528
  * Zorec & Royer (2012) Zorec J., Royer F., 2012, A&A, 537, A120
  * Çalışkan et al. (2015) Çalışkan Ş., Kılıçoğlu T., Elmaslı A., Nasolo Y., Avcı Z., Albayrak B., 2015, New Astron., 34, 6
  * de Bruijne & Eilers (2012) de Bruijne J. H. J., Eilers A. C., 2012, A&A, 546, A61


#  Planning to Practice: Efficient Online Fine-Tuning
by Composing Goals in Latent Space

Kuan Fang∗, Patrick Yin∗, Ashvin Nair, Sergey Levine
University of California, Berkeley ∗Authors contributed equally to this work.

###### Abstract

General-purpose robots require diverse repertoires of behaviors to complete
challenging tasks in real-world unstructured environments. To address this
issue, goal-conditioned reinforcement learning aims to acquire policies that
can reach configurable goals for a wide range of tasks on command. However,
such goal-conditioned policies are notoriously difficult and time-consuming to
train from scratch. In this paper, we propose Planning to Practice (PTP), a
method that makes it practical to train goal-conditioned policies for long-
horizon tasks that require multiple distinct types of interactions to solve.
Our approach is based on two key ideas. First, we decompose the goal-reaching
problem hierarchically, with a high-level planner that sets intermediate
subgoals using conditional subgoal generators in the latent space for a low-
level model-free policy. Second, we propose a hybrid approach which first pre-
trains both the conditional subgoal generator and the policy on previously
collected data through offline reinforcement learning, and then fine-tunes the
policy via online exploration. This fine-tuning process is itself facilitated
by the planned subgoals, which breaks down the original target task into
short-horizon goal-reaching tasks that are significantly easier to learn. We
conduct experiments in both the simulation and real world, in which the policy
is pre-trained on demonstrations of short primitive behaviors and fine-tuned
for temporally extended tasks that are unseen in the offline data. Our
experimental results show that PTP can generate feasible sequences of subgoals
that enable the policy to efficiently solve the target tasks. 111Supplementary
video: sites.google.com/view/planning-to-practice

##  I Introduction

Developing controllers that can solve a range of tasks and generalize broadly
in real-world settings is a long-standing challenge in robotics. Such
generalization in other domains such as computer vision and natural language
processing has been attributed to training large machine learning models on
massive datasets [1]. Consequently, one promising approach to robustly handle
a wide variety of potential situations that a robot might encounter is to
train on a large dataset of robot behaviors. Prior work in robotics has
demonstrated effectively generalization from training on large datasets for
individual tasks, such as grasping [2, 3]. However, a general-purpose robot
should be able to perform a wide range of skills, and should also be taskable.
That is, it must be able to complete a specific task when specified by a
human, including temporally extended tasks that require sequencing many skills
together to complete. This topic has been studied in prior work on goal-
conditioned reinforcement learning, where a robot aims to perform a task given
a desired end state [4, 5, 6]. What remains to make these methods widely
applicable for real-world robotics?

Figure 1: Our method, Plan to Practice (PTP), solves long-horizon goal-
conditioned tasks by combining planning and fine-tuning. We begin with an
offline dataset containing a variety of behaviors, and train a subgoal
generator and goal-conditioned policy on this data. Then, to learn more
complex multi-stage tasks, we optimize over subgoals using the subgoal
generator, which corresponds to a planning procedure over (visual) subgoals,
and fine-tune the policy with online RL by practicing these subgoals. This
enables the robot to solve multi-stage tasks directly from images.

While goal-conditioned policies can be trained effectively for relatively
short-horizon tasks, temporally extended multi-stage can pose a significant
challenge for current methods. These tasks present a major exploration
challenge during online learning, and a major challenge for credit assignment
during offline learning.In this paper, we aim to address these challenges by
combining two ideas. The first is that long-horizon goal-reaching tasks can be
decomposed into shorter-horizon tasks consisting of subgoals. The second is
that these subgoals can be used to _fine-tune_ a goal-conditioned policy
online, even if its performance from offline data is poor.The first idea
enables us to address the exploration challenge, by automatically generating
intermediate subgoals that can be “practiced” on the way to a longer-horizon
final goal. In the framework of goal-conditioned RL, solving long-horizon
tasks can be reduced to the problem of optimization over a sequence of
subgoals for the goal-conditioned policy, and this optimization over subgoals
can be regarded as a kind of high-level planning, where the optimizer selects
waypoints for achieving a distant goal. The high-level planner itself can use
a learned high-level model.

However, if we rely entirely on offline data, credit assignment challenges
make it difficult to perform longer-horizon tasks even with subgoal planning.
Even if the offline RL policy performs well on each individual skill, there
may be errors from stitching skills together because the initial states of
each stage diverge from the offline data when they are composed together. In
practice, this leads to poor performance when using only offline training.
Therefore, the second key idea in our work is to utilize subgoal planning not
merely to _perform_ a multi-stage task, but also to make it possible to
_practice_ that task to finetune it online. While online training for
temporally extended tasks is ordinarily difficult, by addressing the
exploration challenge with subgoal planning, we make it possible for the robot
to practice a series of relatively short-horizon tasks, which makes this kind
of finetuning feasible. Thus, the planner acts both as a higher level policy
when performing the task, and as a scaffolding curriculum for finetuning the
lower-level goal-conditioned policy By collecting data actively in a specific
environment, we can directly experience the distribution shift and can use
reinforcement learning to improve performance under this shift.

To this end, we propose Planning to Practice (PTP), an approach that
efficiently trains a goal-conditioned policy to solve multi-step tasks by
setting subgoals to exploit the compositional structure of the offline data.
An outline is shown in Fig. 1. Our approach is based around a planner that
composes generated subgoals to guide the goal-conditioned policy during an
online fine-tuning phase. To propose diverse and reachable subgoals to form
the candidate plans, we design a conditional subgoal generator based on
conditional variational autoencoder (CVAE) [7]. Through training on the
offline dataset, the conditional subgoal generator captures the distribution
of reachable subgoals from a given state and generates sequences of subgoals
from the learned latent space in a recursive manner. Our subgoal planning
algorithm hierarchically searches for subgoals in a coarse-to-fine manner
using multiple conditional subgoal generators that are trained to generate
goals at different temporal resolutions. Both the goal-conditioned policy and
the conditional subgoal generators are pre-trained on the offline data, and
the policy is fine-tuned on the novel target task.

Our main contribution is a system for learning to solve long-horizon goal-
reaching tasks by fine-tuning the goal-conditioned policy with subgoal
planning in a learned latent space. We evaluate our approach on multi-stage
robotic manipulation tasks with raw image observations and image goals in both
simulation and the real world. After being pre-trained on short demonstrations
of primitive interactions, our approach is able to find feasible subgoal
sequences as plans for unseen final goals by recursively generating subgoals
with the learned conditional subgoal generators. By comparing our approach
with both model-free methods and prior approaches that optimize over subgoals,
we demonstrate that the produced plans significantly improve the learning
efficiency and the resultant success rates during the online fine-tuning.

##  II Related Work

We propose to use a combination of optimization-based planning and fine-tuning
with goal-conditioned reinforcement learning from prior data in order to allow
robots to learn temporally extended skills. In this section, we cover prior
methods in offline RL, planning, goal-conditioned RL, and how they relate to
our method.

Learning from prior data. Offline reinforcement learning methods learn from
prior data [8, 9, 10, 11, 12, 13, 14], and can also finetune through online
interaction [15, 16, 17, 4, 18, 19]. Such methods have been used in a variety
of robotic settings [20, 21, 6, 17]. Our focus is not on introducing new
offline RL methods. Rather, our work shows that planning over subgoals for a
goal-conditioned policy that is pretrained offline can enable finetuning for
temporally extended skills that would otherwise be very difficult to learn.

Goal-conditioned reinforcement learning. The aim of goal-conditioned
reinforcement learning (GCRL) is to control the agent to efficiently reach
specified goal states [22, 23, 24]. Compared to policies that are trained to
solve a fixed task, the same goal-conditioned policy can perform a variety of
tasks when it is commanded with different goals. Such flexibility allows GCRL
to better share knowledge across different tasks and make use of goal
relabeling techniques to improve the sample efficiency without meticulous
reward engineering [25, 26, 27, 28, 29, 30, 31, 32]. Prior has explored
various strategies for proposing goals for exploration [33, 34, 4, 35], and
studied goal-conditioned RL from offline data [5]. However, such works
generally aim to learn short-horizon behaviors, and learning to reach goals
that require multiple stages (e.g., several manipulation primitives) is very
difficult, as shown in our experiments. Our work aims to extend model-free
goal-conditioned RL methods by incorporating elements of planning to enable
effective finetuning for multi-stage tasks.

Planning. A wide range of methods have been developed for planning in
robotics. At the most abstract level, symbolic task planning searches over
discrete logical formulas to accomplish abstract goals [36]. Motion planning
methods solve the geometric problem of reaching a goal configuration with
dynamics and collision constraints [37, 38, 39, 40, 41]. Prior methods have
also considered task and motion planning as a combined problem [42]. These
methods generally assume high-level structured representations of environments
and tasks, which can be difficult to actualize in real-world environments.
Since in our setting we only have image inputs and not structured scene
representations, we focus on methods that can handle raw images for
observations and task specification.

Combining goal-conditioned RL and planning. A number of recent works have
sought to integrate concepts from planning with goal-conditioned policies in
order to plan sequences of subgoals for longer-horizon tasks [43, 44, 45, 46,
47, 48, 49]. These prior methods either propose subgoals from the set of
previously seen states, or directly optimize over subgoals, often by utilizing
a latent variable model to obtain a concise representation of image-based
states [33, 50, 51, 43, 47, 4, 35]. The method we employ is most closely
related conceptually to the method proposed by Pertsch et al. [47], which also
employs a hierarchical subgoal optimization, and the method proposed by
Nasiriany et al. [43], which also optimizes over sequences of latent vectors
from a generative model. Our approach makes a number of low-level
improvements, including the use of a conditional generative model [34], which
we show leads to significantly better performance. More importantly, our
method differs conceptually from these prior works in that our focus is
specifically on utilizing subgoal optimization as a way to enable finetuning
goal-conditioned policies for longer-horizon tasks. We show that it is in fact
this capacity to enable effective finetuning that enables our method to solve
more complex multi-stage tasks in our experiments.

##  III Problem Statement

In this paper, we consider the problem of learning to complete a long-horizon
task specified by a goal image. The robot learns over a variety of initial
configurations and goal distributions, which cover a range of behaviors such
as opening or closing a drawer, and picking, placing, or pushing an object. As
prior data, the robot has access to an offline dataset of trajectories
$\mathcal{D}_{\text{offline}}=\\{\tau_{1},\tau_{2},\dots,\tau_{N}\\}$ for
offline pre-training. In each trajectory, the robot is controlled by a human
tele-operator or a scripted policy to achieve one of the goals the environment
affords. A goal-conditioned policy is pre-trained on this dataset using
offline RL algorithms.

After offline pre-training, the robot isplaced in a particular environment
that it has online access to interact in. Even though the initial
configuration of this environment may have been included in the set of
training environments, the goal distribution for this environment at test time
requires sequencing multiple skills together, which is not present in the
offline data. For instance, as shown in Fig. 1, the robot would need to first
slide away the can that blocks the drawer, then reaches the handle of the
drawer, and finally opens the drawer.

Naïvely running offline RL may not solve the long-horizon test tasks for two
reasons. First, the robot is given a test goal distribution that is long
horizon but offline dataset consists of individual skills. The method needs to
somehow compose these individual skills autonomously in order to succeed at
goals drawn from the test distribution. Second, offline RL may not solve the
task due to distribution shift. Distribution shift appears in two forms:
distribution shift between transitions in the prior data and transitions
obtained by the actively rolling out the policy, and the distribution shift
introduced when performing tasks sequentially. If the robot may actively
interact in the new environment to improve its policy, how can the robot
further practice and improve its performance?

##  IV Preliminaries

We consider a goal-conditioned Markov Decision Process (MDP) denoted by a
tuple $M=(\mathcal{S},\mathcal{A},\rho,P,\mathcal{G},\gamma)$ with state space
$\mathcal{S}$, action space $\mathcal{A}$, initial state probability $\rho$,
transition probability $P$, a goal space $\mathcal{G}$, and discount factor
$\gamma$. In each episode, a desired goal $s_{g}\in\mathcal{G}$ is sampled for
the robot to reach. At each time step $t$, a goal-conditioned policy
$\pi(a_{t}|s_{t},s_{g})$ selects an action $a_{t}\in\mathcal{A}$ conditioned
on the current state $s_{t}$ and goal $s_{g}$. After each step, the robot
receives the goal-reaching reward $r_{t}(s_{t+1},s_{g})$. The robot aims to
reach the goal by maximizing the average cumulative reward
$\mathbb{E}[\Sigma_{t}\gamma^{t}r_{t}]$. Our approach learns a goal-
conditioned policy $\pi$ for solving the target task specified by a desired
final goal $s_{g}$. The goal-conditioned policy is pre-trained on a previously
collected offline dataset $\mathcal{D}_{\text{offline}}$ and then fine-tuned
to reach $s_{g}$ by accumulating data into an online replay buffer
$\mathcal{D}_{\text{online}}$. $\mathcal{D}_{\text{offline}}$ contains diverse
short-horizon interactions with objects in the environment. During online
fine-tuning, we would like the policy to learn to improve and compose these
short-horizon behaviors for multi-stage tasks specified by $s_{g}$.

Defining informative goal-reaching rewards and extracting useful state
representations from high-dimensional raw observations such as images can be
challenging. Following the practice in prior work [33, 4], we pre-train a
state encoder $h=\phi(s)$ to extract the latent state representation $h$. By
encoding the states and goals to the latent space, we can obtain an
informative goal-reaching reward function $r_{t}=R(h_{t+1},h_{g})$ by
computing $h_{t+1}=\phi(s_{t+1})$ and $h_{g}=\phi(s_{g})$. Specifically,
$R(h_{t+1},h_{g})$ returns 0 when $||h_{t+1},h_{g}||<\epsilon$ and -1
otherwise, where $\epsilon$ is a selected threshold. In addition, we also use
$\phi(s_{t+1})$ as the backbone feature extractor in all of our models that
take $s$ as an input. For simplicity, we directly use $s$ to denote $h$ in the
rest of the paper. The details of the state encoder are explained in Sec.
VI-B.

##  V Planning to Practice

We propose Planning to Practice (PTP), an approach that efficiently fine-tunes
a goal-conditioned policy to solve novel tasks. To enable the robot to
efficiently learn to solve the target task, we propose to use subgoals to
facilitate the online fine-tuning of the goal-conditioned policy. Given the
initial state $s_{0}$ and the goal state $s_{g}$, we search for a sequence of
$K$ subgoals $\hat{s}_{1:K}=\hat{s}_{1},...,\hat{s}_{K}$ to guide the robot to
reach $s_{g}$. Such subgoals will inform the goal-conditioned policy
$\pi(a|s,s_{g})$ what is the immediate next step on the path to $s_{g}$ and
provide the policy more dense reward signals compared to directly using the
final goal. We choose the sequence of subgoals at the beginning of each
episode and feed the first subgoal in the sequence to the goal-conditioned
policy. The policy will switch to the next subgoal in the sequence when the
current subgoal is reached or the time budget assigned for the current subgoal
runs out.

The main challenge is to search for a sequence of subgoals that can lead to
the desired final goals while ensuring each subgoal is a valid state that can
be reached from the previous subgoal. Particularly when the states correspond
to full images, most vectors will not actually represent valid states, and
indeed naïvely optimizing over image pixels may simply result in out-of-
distribution inputs that lead to erroneous results when input into the goal-
conditioned policy.

As outlined in Fig. 1, we devise a method to effectively propose and select
valid subgoal sequences to guide online fine-tuning by means of a generative
model. At the heart of our approach is a conditional subgoal generator
$g(\cdot|s_{0})$ that recursively produces candidate subgoals in a
hierarchical manner conditioned on the initial state $s_{0}$. To find the
optimal sequence of subgoals $\hat{s}_{1:K}^{*}$, we first sample $N$
candidate sequences $\hat{s}_{1:K}^{1},...,\hat{s}_{1:K}^{N}$ from the state
space using the conditional subgoal generator. Then we rank the candidate
sequences using a cost function $c$. The sequence that corresponds to the
lowest cost will be selected as $\hat{s}_{1:K}^{*}$ for the goal-conditioned
policy. Through this sampling-based planning procedure, we choose the subgoal
for guiding the goal-conditioned policy $\pi$ during online fine-tuning. The
overall algorithm is summarized in Algorithm 1. Next we describe the design of
each module in details.

Algorithm 1 Planning To Practice (PTP)

1:set of final goals $\mathcal{G}$, time horizon $T$, offline data
$\mathcal{D}_{\text{offline}}$, number of subgoals $K$.

2:Train $\pi(a|s,s_{g})$ and $g(s,z)$ on $\mathcal{D}_{\text{offline}}$.

3:Initialize the online replay buffer
$\mathcal{D}_{\text{online}}\leftarrow\varnothing$.

4:while not converged do

5: Reset the environment and observe $s_{0}$.

6: Sample $s_{g}$ from $\mathcal{G}$.

7: Plan for the subgoals $\hat{s}_{1:K}$.

8: $k\leftarrow 1$

9: for $t=1,...,T$ do

10: Compute the action $a_{t}\leftarrow\pi(a_{t}|s_{t},\hat{s}_{k})$

11: Observe the state $s_{t+1}$ and the reward $r_{t}$

12:
$\mathcal{D}_{\text{online}}\leftarrow\mathcal{D}_{\text{online}}\cup(s_{t},a_{t},r_{t},s_{t+1})$.

13: if $t\pmod{\Delta t}==0$ or $||s_{t+1}-\hat{s}_{K}||<\epsilon$ then

14: $k\leftarrow\min(k+1,K)$

15: end if

16: end for

17: Train $\pi$ on batches sampled from $\mathcal{D}_{\text{offline}}$ and
$\mathcal{D}_{\text{online}}$.

18:end while

###  V-A Conditional Subgoal Generation

The effectiveness of our planner relies on the generation of diverse and
feasible sequences of subgoals as candidates. Specifically, we would like to
generate the candidates by sampling from the distribution of suitable subgoal
sequences $p(\hat{s}_{1},...,\hat{s}_{K}|s_{0})$ conditioned on the initial
state $s_{0}$. Most existing methods independently sample the subgoal at each
step from a learned prior distribution [47] or a replay buffer [44], which is
unlikely to propose useful plans for tasks with large, combinatorial state
spaces (i.e., with multiple objects).

We propose to break down $p(\hat{s}_{1},...,\hat{s}_{K}|s_{0})$ into
$p(\hat{s}_{1}|s_{0})\Pi_{i=1}^{k}p(\hat{s}_{i}|\hat{s}_{i-1})$ through
modeling the conditional distribution $p(s^{\prime}|s)$ of the reachable next
subgoal $s^{\prime}$. By utilizing temporal compositionality, the conditional
subgoal generation paradigm improves generalization and enables generation of
sequences of arbitrary lengths.

We use a conditional variational encoder (CVAE) [7] to capture the
distribution of reachable goals $p(s^{\prime}|s)$. In the CVAE, we define the
decoder as $g(s,z)$ and the encoder as $q(z|s,s^{\prime})$, where $z$ is the
learned latent representation of the transitions and it is sampled from a
prior probability $p(z)$. To propose a sequence of subgoals, we use $g(s,z)$
as the conditional subgoal generator. Conditioned on the initial state
$s_{0}$, the first subgoal $\hat{s}_{1}$ can be generated as
$\hat{s}_{1}=g(s_{0},z_{1})$ given the sampled $z_{1}\sim p(z)$. Then the
$i_{\text{th}}$ subgoal can be recursively generated by sampling $z_{i}\sim
p(z)$ and computing $\hat{s}_{i}=g(\hat{s}_{i-1},z_{i})$ given the previous
subgoal $\hat{s}_{i-1}$. In this way, we could sample a sequence of i.i.d.
latent representations $z_{1},...,z_{K}$ and recursively generate
$\hat{s}_{1},...,\hat{s}_{K}$ conditioned on the initial state $s_{0}$ using
the conditional subgoal generator.

The CVAE is trained to minimize the evidence lower bound (ELBO) [52] of
$p(s^{\prime}|s)$ given the offline dataset $\mathcal{D}$. During training, we
sample transitions $(s_{t},s_{\tau})$ from the offline dataset to form the
minibatches, where $\tau=t+\Delta t$ is a future step that is $\Delta t$ steps
ahead. Instead of using a fixed $\Delta t$, we sample $\Delta t$ from a range
for each transition to provide richer data. To encourage the trained model to
be robust to compounding errors, we sample sequences composed of multiple
states and use the subgoal reconstructed at the previous step as the context
in the next step. Therefore, the objective for training the conditional
subgoal generator is:

$\mathbb{E}_{q(z|s_{t},s_{\tau})}||s_{\tau}-g(s_{t},z)||^{2}+D_{KL}[q(z|s_{t},s_{\tau})||p(z)]$
(1)

where $D_{KL}[\cdot||\cdot]$ indicates the KL-Divergence.

###  V-B Efficient Planning in the Latent Space

Algorithm 2 $Plan(s_{0},s_{g},L,K,M,N)$

1:the initial state $s_{0}$, the goal state $s_{g}$, number of subgoals $K$,
number of levels $L$, multiplier $M$, number of samples $N$.

2:Sample $N$ latent action sequences $\\{z_{1:K}^{i}\\}_{i=1}^{N}$.

3:Recursively generate subgoals $\\{\hat{s}_{1:K}^{i}\\}_{i=1}^{N}$ using
$g(s,z)$.

4:Select $z_{1:K}^{*}$ and $\hat{s}_{1:K}^{*}$ of the lowest cost.

5:Update $z_{1:K}^{*}$ and $\hat{s}_{1:K}^{*}$ using MPPI.

6:if L = 1 then

7: return $\hat{s}_{1:K}^{*}$

8:else

9: Denote $\hat{s}_{0}^{*}\leftarrow s_{0}$.

10: Initialize the plan $\hat{\mathcal{S}}$ as an empty list

11: for $i=1,...,K$ do

12: Append $Plan(\hat{s}_{i-1}^{*},\hat{s}_{i}^{*},L-1,M,M,N)$ to
$\hat{\mathcal{S}}$

13: end for

14: return $\hat{\mathcal{S}}$

15:end if

We build a planner that efficiently searches for sequences of subgoals in the
latent space as shown in Algorithm 2. To tackle the large search space of
candidate subgoal sequences, we design a hierarchical planning algorithm that
searches for subgoals in a coarse-to-fine manner and re-use the previously
selected subgoals as candidates in new episodes.

The hierarchical planning is conducted at $L$ levels with different temporal
resolutions $\Delta t_{1}$, …, $\Delta t_{L}$. The temporal resolution of each
level is an integral multiple of that of the previous level, i.e., $\Delta
t_{i}=M\Delta t_{i-1}$, where $M$ is a scaling factor and is set to 2 in our
experiments. We first plan for the subgoals
$\hat{s}_{1}^{1},\hat{s}_{2}^{1},...$ on the first level. Then the subgoals
$\hat{s}_{1:K}^{l}$ of finer temporal resolution are planned on each level $l$
to connect the subgoals planned on the previous level $l-1$. Specifically,
given the adjacent subgoals $\hat{s}_{i}^{l-1}$ and $\hat{s}_{i+1}^{l-1}$
produced on the previous level, we plan for a segment of $M$ subgoals
$\hat{s}_{i*M+1}^{l},...,\hat{s}_{(i+1)*M}^{l}$ on the level $l$, by treating
$\hat{s}_{i}^{l-1}$ and $\hat{s}_{i+1}^{l-1}$ as the initial state and final
goal state in Eqn. V-C. The planned segments are returned to the previous
level and concatenated as a more fine-grained plan. For this purpose, we train
$L$ conditional subgoal generators to propose subgoals that are $\Delta
t_{1}$, …, $\Delta t_{L}$ steps away, respectively. In contrast to the prior
work [47], the conditional subgoal generators enable us to plan for unseen
goals that are beyond the temporal horizon of the demonstrations in the
offline dataset by exploiting the compositional structure of the
demonstrations. By recursively generating the subgoals across time at each
level, we only need to enforce that the temporal resolution of the top level
$\Delta t_{L}$ is smaller than the time horizon of each trajectory, since the
conditional subgoal generator $f^{1}(s,z)$ needs to be trained on trajectories
at least $\Delta t_{L}+1$ steps in length.

We maintain a latent plan buffer for each level to further facilitate the
planning with the conditional subgoal generator. After each episode, the
selected latent representations on each level are appended to the
corresponding latent plan buffer. In each target task, the subgoals are
supposed to have the same semantic meaning. In spite of the variations of the
initial and goals state in each episode, the optimal plans in the latent space
can often be similar to each other. Therefore, we sample half of the latent
representations from the prior distribution $p(z)$ and the other half from the
latent plan buffer among the initial samples to enhance the chance of finding
a close initial guess.

We build our planner upon the model predictive path integral (MPPI) [53],
which iteratively optimizes the plan through importance sampling. In each
interaction, we perturb the chosen plan in the latent space with a small
Gaussian noise as new candidates.

Figure 2: Target tasks. Three multi-stage tasks are designed for our
experiments in the simulation and the real world respectively. In each target
task, the robot needs to strategically interacts with the environment (e.g.,
first takes out an object in the drawer then closes the drawer). The initial
state and the desired goal state are shown for each task.  Figure 3:
Quantitative comparison in simulation. The average success rate across 3 runs
is shown with the shaded region indicating the standard deviation. The
negative x-axis indicates the epochs of offline pre-training and positive
x-axis indicates epochs of online fine-tuning. Using offline learning and
planning, our method PTP is able to solve these tasks partially (at 0 epochs).
Then with online finetuning the performance improves further. In contrast,
prior methods have lower offline performance and do not fine-tune successfully
in most cases, as they do not collect coherent online data.

###  V-C Cost Function For Feasible Subgoals

To provide informative guidance to the policy $\pi(a|s,s_{g})$, we would like
that the final goal $s_{g}$ can be reached at the end of the episode while
encouraging the transition between each pair of subgoals to be feasible within
a limited time budget. As explained in Sec. IV, the goal state is considered
to be reached when the Euclidean distance between the last subgoal in the plan
and the desired goal is close in the learned latent space. Inspired by [43],
the feasibility of each transition between adjacent subgoals can be measured
using the goal-conditioned value function $V(s,s^{\prime})$ and the log
likelihood of the latent $\log p(z)$. Therefore, finding subgoals
$\hat{s}_{1:K}^{*}$ can be formulated as the optimization problem:

minimize $\displaystyle||s_{g}-\hat{s}_{K}||$ (2) subject to $\displaystyle
V_{i,i+1}\geq\delta_{1},\log p(z_{i})\geq\delta_{2},\text{for $i=1,...,K$}$

where $\delta_{1}$ and $\delta_{2}$ are thresholds and we define
$\hat{s}_{0}=s_{0}$, $V_{i,i+1}=V(\hat{s}_{i-1},\hat{s}_{i})$ for convenience.
Then the cost function can be written with Lagrangian multipliers $\eta_{1}$
and $\eta_{2}$:

$\displaystyle c(s_{0},s_{g},z_{1:K})$ $\displaystyle=$
$\displaystyle||s_{g}-\hat{s}_{K}||-\sum_{i=1}^{K}\left(\eta_{1}V_{i,i+1}+\eta_{2}\log
p(z_{i})\right)$

The details of our method are explained in Sec. VI-B.

##  VI Experiments

In our experiments, we aim to answer the following questions: 1) Can PTP
propose and select feasible subgoals as plans for real-world robotic
manipulation tasks? 2) Can the subgoals planned by PTP facilitate online fine-
tuning of the goal-conditioned policies to solve target tasks unseen in the
offline dataset? 3) How does each design option affect the performance of PTP?

###  VI-A Experimental Setup

Environment. As shown in Fig. 2, our experiments are conducted in a table-top
manipulation environment with a Sawyer robot. At the beginning of each
episode, a fixed drawer and two movable objects are randomly placed on the
table. The robot can change the state of the environment by opening/closing
the drawer, sliding the objects, and picking and placing objects into
different destinations, etc. At each time step, the robot receives a 48 x 48
RGB image via a Logitech C920 camera as the observation and takes a
5-dimensional continuous action to change the gripper status through position
control. The action dictates the change of the coordinates along the three
axes, the change of the rotation, and the status of the fingers. We use
PyBullet [coumans2016pybullet] for our simulated experiments.

Prior data. The prior data consists of varied demonstrations for different
primitive tasks. In each demonstrated trajectory, we randomly initialize the
environment and perform primitive interactions such as opening the drawer and
poking the object. These trajectories are collected using teleoperation in the
real world, and a scripted policy that uses privileged information of the
environment (e.g., the object pose and the status of the drawer) in
simulation. The trajectories vary in length from 5 to 150 time steps, with
2,344 trajectories in the real world and 4,000 in simulation.

Target tasks. In each target task, a desired goal state is specified by a 48 x
48 RGB image (same dimension with the observation). The robot is tasked to
reach the goal state by interacting with the objects on the table. Task
success for our evaluation is determined based on the object positions at the
end of each episode (this metric is not used for learning). As shown in Fig.
2, we design three target tasks that require multi-stage interactions with the
environment to complete. These target tasks are designed with temporal
dependencies between stages (e.g., the robot needs to first move away a can
that blocks the drawer before opening the drawer). The transitions from the
initial state to the goal state are unseen in the offline data. The episode
length is 400 steps in simulation and 125 steps in the real world, which are
much longer than the time horizon of the demonstrations.

Baselines and ablations. We compare PTP with 3 baselines and 3 ablations.
Model-Free uses a policy directly conditioned on the final goal and conducts
online fine-tuning without using any subgoals. LEAP [43] learns a variational
auto-encoder (VAE) [52] to capture the prior distribution of states and plans
for subgoals without conditioning on any context. GCP [47] learns a goal
predictor that hierarchically generates intermediate subgoals between the
initial state and the goal state. To analyze the design options in PTP, we
also compare with variations of our method by removing the latent plan buffer
(PTP (w/o B)), the hierarchical planning algorithm (PTP (w/o H)), and both of
these two designs (PTP (w/o H and B)). All methods use the same neural network
architecture in the goal-conditioned policy and are pre-trained on the same
offline dataset.

###  VI-B Implementation Details

Following [4], we use a vector quantized variational autoencoder (VQ-VAE) [55]
as the state encoder, which encodes a $48\times 48\times 3$ image to a
720-dimensional encoding. The conditional subgoal generator is implemented
with a U-Net architecture [56] and decodes the subgoal from an 8-dimensional
latent representations conditioned on the encoding of the current state. In
our planner, we use $L=3$, $K=8$, $M=2$, $N=1024$, and we run MPPI for 5
iteration on each level. $g$ is trained to predict subgoals that are 15, 30,
and 60 steps away. Implicit Q-Learning (IQL) [57] is used as the underlying RL
algorithm for offline pre-training and online fine-tuning with default
hyperparameters. We use the same network architectures for the policy and the
value functions from [4] for simulation experiments. For real-world
experiments, we use a convolutional neural network instead. We use Adam
optimizer with a learning rate of $3\cdot 10^{-4}$ and a batch size of 256.
During training, we relabel the goal with future hindsight experience replay
[25] with $70\%$ probability. We use $\epsilon=2$ for the reward function
defined in Sec. IV, $\eta_{1}=0.001,\eta_{2}=0.01$ in Eqn. V-C.

Figure 4: Planned subgoal sequences. Each row shows the sequence of subgoals
produced by each method. The initial state and the final goal are shown at the
two ends.

###  VI-C Quantitative Comparisons

We evaluate PTP and baselines on three unseen target tasks. We use simulated
versions of these tasks for comparisons and ablations, and real-world tasks,
where all pretraining and finetuning uses only real-world data, to evaluate
the practical effectiveness of the method.

Simulation. We first pre-train the goal-conditioned policy on the offline
dataset for 100 epochs and the run online fine-tuning for the target task for
150 epochs. Each epoch takes 2,000 simulation steps (only during fine-tuning)
and 2,000 training iterations. We run online fine-tuning using each method
with 3 different random seeds. After each epoch, we test the policy in the
target task for 5 episodes. We report the average success rate across 3 runs
in Fig. 3 where the negative x-axis indicates the offline pre-training epochs
and positive x-axis indicates the online fine-tuning epochs.

As shown in Fig. 3, our full model consistently outperforms baselines with a
large performance gap. The generated subgoals not only enable the pre-trained
policy to achieve higher success rate by breaking down the hard problems into
easier pieces, but also introduces larger performance improvements during
online fine-tuning. After fine-tuning for 150 epochs, the policy achieves the
success rates of 84.9%, 59.9%, 49.3% in the three target tasks respectively.
Compared to the policy pre-trained on the offline dataset, the performance is
significantly improved (+31.6%, +37.8%, and +13.8%). When directly using the
final goal or subgoals generated by baseline methods, the policy’s performance
plateaus at around 0.0% to 30.0% and does not improve much during online fine-
tuning.

We found that the hierarchical planner and the latent plan buffer are crucial
for PTP’s performance. Without these two design options, the planner often
suffers from the large search space of possible subgoal sequences and the
resultant success rates decrease. The latent plan buffer significantly
improves the performance of non-hierarchical PTP while it has a minor effect
on hierarchical PTP.

Real-world evaluation. We pre-train the policy for 200 epochs and fine-tune it
for 10 epochs. In each epoch, we run 10,000 training iterations and collect
1,000 steps in the real world. We train on three target tasks which are shown
in Figure 2, and report the success rate of the goal-conditioned policy before
and after online fine-tuning in Table I. Planning enables the robot to succeed
partially with just the offline initialized policy, achieving success rates of
$12.5\%,75.0\%$ and $25.0\%$ on the three tasks. (When the offline policy is
conditioned on only the final goal image without planning, the success rate is
$0\%$.) Then in each task, we fine-tune to a significantly higher success
rate.

Qualitatively, at the beginning of fine-tuning, the robot often fails,
deviating from the planned subgoals or colliding with the environment. With
the planned subgoals, the original long-horizon task is broken down to short
snippets that are easier to complete. Even if a subgoal is not reached
successfully at first, the data is useful to collect additional experience and
fine-tune the policy. After fine-tuning for 4-5 epochs, we already observe
that the robot’s performance reaching subgoals during training time
significantly improves, collecting even more coherent and useful data. After
10 epochs, we achieve success rates of $62.5\%,100.0\%$ and $50.0\%$. In
comparison, GCP cannot provide useful guidance to the policy when the
generated goals are noisy.

TABLE I: The real-world success rates before and after online fine-tuning. The tasks are described in Sec. VI-A. Task |  | PTP (Ours)
---
Offline $\rightarrow$ Online
| GCP
---
Offline $\rightarrow$ Online
Task A | $12.5\%\rightarrow\textbf{62.5\%}$ | $12.5\%\rightarrow 0.0\%$
Task B | $75.0\%\rightarrow\textbf{100.0\%}$ | $50.0\%\rightarrow 75.0\%$
Task C | $25.0\%\rightarrow\textbf{50.0\%}$ | $25.0\%\rightarrow 12.5\%$

###  VI-D Generated Subgoals

In Fig. 4, we present qualitative results of the generated subgoals for each
task in the real world. Each row shows a sequence of generated subgoals
produced by the planner in each method. In all the three target tasks, PTP
successfully plans for a sequence of subgoals that can lead to the desired
final goal. The transition between adjacent subgoals are feasible within a
short period of time. By comparison, both of the baseline methods fail to
generate reasonable plans. Without conditioning on the current state, LEAP
[43] can hardly produce any realistic images of the environment. Most of the
generated subgoals are highly noisy images with duplicated robot arms and
objects. The quality of the subgoals produced by GCP [47] is higher than that
of LEAP but still much worse than ours. GCP cannot generalize well for the
initial state and the goal state that are out of the distribution of the
offline dataset, which contains only short snippets of demonstrations.

##  VII Conclusion and Discussion

We presented PTP, a method for real-world learning of temporally extended
skills by utilizing planning and fine-tuning to stitch together skills from
prior data. First, planning is used to convert a long-horizon task into
achievable subgoals for a lower level goal-conditioned policy trained from
prior data. Then, the goal-conditioned policy is further fine-tuned with
active online interaction, mitigating the distribution shift between the
offline data and actual states seen during rollouts. This procedure allows
robots to extend their capabilities autonomously, composing previously seen
data into more complicated and useful skills.

## Acknowledgement

Our research is supported by the Office of Naval Research, ARL DCIST CRA
W911NF-17-2-0181, ARO W911NF-21-1-0097, and AFOSR FA9550-22-1-0273. We thank
Dhruv Shah and Ilya Kostrikov for their constructive feedback.

## References

  * [1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _NeurIPS_ , 2012, pp. 1097–1105.
  * [2] S. Levine, P. Pastor, _et al._ , “Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection,” _International Journal of Robotics Research_ , 2017.
  * [3] T. Yu, A. Kumar, _et al._ , “Conservative data sharing for multi-task offline reinforcement learning,” _NeurIPS_ , vol. 34, 12 2021.
  * [4] A. Khazatsky, A. Nair, _et al._ , “What can i do here? learning new skills by imagining visual affordances,” _ICRA_ , pp. 14 291–14 297, 2021.
  * [5] Y. Chebotar, K. Hausman, _et al._ , “Actionable models: Unsupervised offline reinforcement learning of robotic skills,” 4 2021.
  * [6] D. Kalashnikov, J. Varley, _et al._ , “Mt-opt: Continuous multi-task robotic reinforcement learning at scale,” 2021.
  * [7] K. Sohn, X. Yan, and H. Lee, “Learning Structured Output Representation using Deep Conditional Generative Models,” in _Advances in Neural Information Processing Systems_ , 2015.
  * [8] S. Lange, T. Gabel, and M. Riedmiller, “Batch reinforcement learning,” in _Reinforcement learning_. Springer, 2012, pp. 45–73.
  * [9] S. Fujimoto, D. Meger, and D. Precup, “Off-policy deep reinforcement learning without exploration,” in _ICML_. PMLR, 2019, pp. 2052–2062.
  * [10] A. Kumar, J. Fu, _et al._ , “Stabilizing off-policy q-learning via bootstrapping error reduction,” in _NeurIPS_ , 2019.
  * [11] C. Zhang, S. R. Kuppannagari, and V. Prasanna, “BRAC+: Going deeper with behavior regularized offline reinforcement learning,” 2021.
  * [12] A. Kumar, A. Zhou, _et al._ , “Conservative q-learning for offline reinforcement learning,” in _NeurIPS_ , 2020.
  * [13] S. Fujimoto and S. S. Gu, “A minimalist approach to offline reinforcement learning,” _arXiv preprint arXiv:2106.06860_ , 2021.
  * [14] A. Singh, A. Yu, _et al._ , “COG: Connecting new skills to past experience with offline reinforcement learning,” in _CoRL_ , 2020.
  * [15] A. Nair, M. Dalal, _et al._ , “AWAC: Accelerating online reinforcement learning with offline datasets,” 2020.
  * [16] A. Villaflor, J. Dolan, and J. Schneider, “Fine-tuning offline reinforcement learning with model-based policy optimization,” 2020.
  * [17] Y. Lu, K. Hausman, _et al._ , “Aw-opt: Learning robotic skills with imitation and reinforcement at scale,” 2021.
  * [18] S. Lee, Y. Seo, _et al._ , “Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble,” 2021.
  * [19] L. Meng, M. Wen, _et al._ , “Offline pre-trained multi-agent decision transformer: One big sequence model tackles all smac tasks,” 2021.
  * [20] D. Kalashnikov, A. Irpan, _et al._ , “Scalable deep reinforcement learning for vision-based robotic manipulation,” in _Conference on Robot Learning_. PMLR, 2018, pp. 651–673.
  * [21] S. Cabi, S. G. Colmenarejo, _et al._ , “Scaling data-driven robotics with reward sketching and batch reinforcement learning,” _arXiv preprint arXiv:1909.12200_ , 2019.
  * [22] L. P. Kaelbling, “Learning to achieve goals,” in _IJCAI_ , 1993.
  * [23] T. Schaul, D. Horgan, _et al._ , “Universal value function approximators,” in _ICML_ , 2015.
  * [24] B. Eysenbach, R. Salakhutdinov, and S. Levine, “C-learning: Learning to achieve goals via recursive classification,” _ArXiv_ , vol. abs/2011.08909, 2021.
  * [25] M. Andrychowicz, D. Crow, _et al._ , “Hindsight experience replay,” in _Advances in Neural Information Processing Systems_ , 2017.
  * [26] V. H. Pong, M. Dalal, _et al._ , “Skew-fit: State-covering self-supervised reinforcement learning,” _ArXiv_ , vol. abs/1903.03698, 2020.
  * [27] M. Fang, T. Zhou, _et al._ , “Curriculum-guided hindsight experience replay,” in _NeurIPS_ , 2019.
  * [28] Y. Ding, C. Florensa, _et al._ , “Goal-conditioned imitation learning,” in _NeurIPS_ , 2019.
  * [29] A. Gupta, V. Kumar, _et al._ , “Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning,” in _CoRL_ , 2019.
  * [30] H. Sun, Z. Li, _et al._ , “Policy continuation with hindsight inverse dynamics,” in _NeurIPS_ , 2019.
  * [31] B. Eysenbach, X. Geng, _et al._ , “Rewriting history with inverse rl: Hindsight inference for policy improvement,” _ArXiv_ , vol. abs/2002.11089, 2020.
  * [32] D. Ghosh, A. Gupta, _et al._ , “Learning to reach goals via iterated supervised learning,” _arXiv: Learning_ , 2021.
  * [33] A. Nair, V. Pong, _et al._ , “Visual Reinforcement Learning with Imagined Goals,” in _NeurIPS_ , 2018.
  * [34] A. Nair, S. Bahl, _et al._ , “Contextual imagined goals for self-supervised robotic learning,” in _CoRL_ , 2019.
  * [35] E. Chane-Sane, C. Schmid, and I. Laptev, “Goal-conditioned reinforcement learning with imagined subgoals,” in _ICML_ , 2021.
  * [36] R. E. Fikes and N. J. Nilsson, “Strips: A new approach to the application of theorem proving to problem solving,” _IJCAI_ , 1971.
  * [37] L. Kavraki, P. Svestka, _et al._ , “Probabilistic roadmaps for path planning in high-dimensional configuration spaces,” _IEEE Transactions on Robotics and Automation_ , vol. 12, pp. 566–580, 1996.
  * [38] S. Koenig and M. Likhachev, “D* lite,” _AAAI_ , 2002.
  * [39] S. Karaman and E. Frazzoli, “Sampling-based algorithms for optimal motion planning:,” _IJRR_ , vol. 30, pp. 846–894, 6 2011.
  * [40] M. Zucker, N. Ratliff, _et al._ , “Chomp: Covariant hamiltonian optimization for motion planning,” _IJRR_ , 2013.
  * [41] M. Kalakrishnan, S. Chitta, _et al._ , “Stomp: Stochastic trajectory optimization for motion planning,” _Proceedings - IEEE International Conference on Robotics and Automation_ , pp. 4569–4574, 2011.
  * [42] S. Srivastava, E. Fang, _et al._ , “Combined Task and Motion Planning Through an Extensible Planner-Independent Interface Layer,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2014.
  * [43] S. Nasiriany, V. H. Pong, _et al._ , “Planning with goal-conditioned policies,” in _NeurIPS_ , 2019.
  * [44] B. Eysenbach, R. Salakhutdinov, and S. Levine, “Search on the replay buffer: Bridging planning and reinforcement learning,” in _NeurIPS_ , 2019.
  * [45] K. Fang, Y. Zhu, _et al._ , “Dynamics learning with cascaded variational inference for multi-step manipulation,” _Conference on Robot Learning (CoRL)_ , 2019.
  * [46] H. Charlesworth and G. Montana, “Plangan: Model-based planning with sparse rewards and multiple goals,” _ArXiv_ , vol. abs/2006.00900, 2020.
  * [47] K. Pertsch, O. Rybkin, _et al._ , “Long-horizon visual planning with goal-conditioned hierarchical predictors,” _ArXiv_ , vol. abs/2006.13205, 2020.
  * [48] A. Sharma, A. Gupta, _et al._ , “Autonomous reinforcement learning via subgoal curricula,” 2021.
  * [49] T. Zhang, B. Eysenbach, _et al._ , “C-planning: An automatic curriculum for learning goal-reaching tasks,” _ArXiv_ , vol. abs/2110.12080, 2021.
  * [50] B. Ichter, J. Harrison, and M. Pavone, “Learning sampling distributions for robot motion planning,” _ICRA_ , pp. 7087–7094, 9 2018.
  * [51] S. Nair and C. Finn, “Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation,” _arXiv preprint arXiv:1909.05829_ , 2019.
  * [52] D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” in _International Conference on Learning Representations (ICLR)_ , 2014.
  * [53] M. S. Gandhi, B. Vlahov, _et al._ , “Robust model predictive path integral control: Analysis and performance guarantees,” _IEEE Robotics and Automation Letters_ , vol. 6, pp. 1423–1430, 2021.
  * [54] E. Coumans and Y. Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning,” http://pybullet.org, 2016–2021.
  * [55] A. van den Oord, O. Vinyals, and K. Kavukcuoglu, “Neural discrete representation learning,” in _NeurIPS_ , 2017.
  * [56] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _MICCAI_ , 2015.
  * [57] I. Kostrikov, A. Nair, and S. Levine, “Offline reinforcement learning with implicit q-learning,” _arXiv preprint arXiv:2110.06169_ , 2021.