# Peculiar Near-Nucleus Outgassing of Comet 17P/Holmes During Its 2007
Outburst

Chunhua Qi1, Michiel R. Hogerheijde2, David Jewitt3, Mark A. Gurwell1, David
J. Wilner1 1Harvard–Smithsonian Center for Astrophysics, 60 Garden Street, MS
42, Cambridge, MA 02138, USA  2Leiden Observatory, Leiden University, PO Box
9513, 2300 RA Leiden, The Netherlands  3Department of Earth, Planetary and
Space Sciences and Department of Physics and Astronomy, University of
California at Los Angeles, 595 Charles Young Drive East, Los Angeles, CA
90095, USA

###### Abstract

We present high angular resolution Submillimeter Array observations of the
outbursting Jupiter family comet 17P/Holmes on 2007 October 26–29, achieving a
spatial resolution of 2.5′′, or $\sim$3000 km at the comet distance. The
observations resulted in detections of the rotational lines CO 3–2, HCN 4–3,
H13CN 4–3, CS 7–6, H2CO 31,2-21,1, H2S 22,0–21,1, and multiple CH3OH lines,
along with the associated dust continuum at 221 and 349 GHz. The continuum has
a spectral index of 2.7$\pm$0.3, slightly steeper than blackbody emission from
large dust particles. From the imaging data, we identify two components in the
molecular emission. One component is characterized by a relatively broad line
width ($\sim$1 km s-1 FWHM) exhibiting a symmetric outgassing pattern with
respect to the nucleus position. The second component has a narrower line
width ($<$0.5 km s-1 FWHM) with the line center red-shifted by 0.1–0.2 km s-1
(cometocentric frame), and shows a velocity shift across the nucleus position
with the position angle gradually changing from $66\arcdeg$ to $30\arcdeg$
within the four days of observations. We determine distinctly different CO/HCN
ratios for each of the components. For the broad-line component we find CO/HCN
$<$ 7, while in the narrow-line component, CO/HCN = 40 $\pm$ 5\. We
hypothesize that the narrow-line component originates from the ice grain halo
found in near-nucleus photometry, believed to be created by sublimating
recently released ice grains around the nucleus during the outburst. In this
interpretation, the high CO/HCN ratio of this component reflects the more
pristine volatile composition of nucleus material released in the outburst.

###### Subject headings:

Comets: general —comets:individual (17P/Holmes) —radio lines:planetary systems
—submillimeter:planetary systems —techniques:interferometric

##  1\. Introduction

Comets are thought to contain the most primitive material left over from the
planetesimal building stage of the Sun’s protoplanetary disk. Their physical
and chemical composition provides an important link between nebular and
interstellar processes (Ehrenfreund & Charnley 2000; Mumma & Charnley 2011). A
fundamental goal of cometary research is to understand the origin and nature
of cometary nuclei, in order to explore how they link back to the
protoplanetary disk. The most abundant constituent of a cometary nucleus,
water, is difficult to detect in ground-based observations due to interference
from the Earth’s atmospheric water vapor. However, many other molecules in
cometary atmospheres can be detected, and these can be used to trace the
distribution and kinematics of gas production from a comet. Of particular
interest are parent molecules (those sequestered in the cometary nucleus),
which sublime directly from the nucleus. Measurements of parent molecule
production rates can reveal relative abundances in the nucleus itself and
therefore provide information about the conditions under which the comet
formed.

There are at least two distinct long-term reservoirs for cometary nuclei, the
Oort Cloud and the Kuiper Belt. Measuring the composition from each of the
reservoirs may shed light on their origins in the protoplanetary disk from
which they formed. Radio interferometers possessing broad bandwidth coverage
coupled with high spectral and spatial resolution can be used to examine
spatial differences among parent molecules to probe the heterogeneity in the
coma and nucleus (e.g., Blake et al. 1999). Simultaneous sampling of multiple
parent molecules can test variations of abundances and abundance ratios
associated with changes in the active area (Mumma & Charnley 2011).

However, it is not straightforward to determine the composition of a comet
without directly sampling the nucleus. Instead, we typically measure the coma
emission of parent and daughter species released from the nucleus. This
technique, while powerful, presents at least two problems. The first is that
it is usually difficult to obtain spatial information on multiple species
simultaneously, which is critical for the determination of abundance ratios
due to the variable nature of comets (e.g., outflow asymmetries and jet
activity). The second is that the chemical composition in the coma is not
necessarily representative of the composition of the nucleus. For example,
chemical reactions in the coma can affect the coma composition (e.g., HNC is
likely to be formed in situ via chemistry in the inner coma (Mumma & Charnley
2011)). More seriously, thermal processing of the comet surface is expected to
lead to preferential volatile loss, resulting in a gradient in composition
with depth beneath the physical surface. Sublimation then proceeds from a
range of depths, depending on the volatility of the ices, their permeability
to gas flow, and the progression of thermally conducted heat moving into the
interior. The gases escaping from the heated surface may not be representative
of the bulk composition of the ices measured deep in the nucleus.

Jupiter family comets likely originated in the Kuiper Belt reservoir, although
the precise source region within the Kuiper belt remains unclear (Volk &
Malhotra 2008). They have short orbital periods and average residence times in
the inner solar system of $\sim$0.4 Myr (Levison & Duncan 1994). Volatile loss
in previous orbits makes it especially difficult to determine their initial
composition from observations of their comae; they can be very hard to observe
because of their usually weak emission, and repeated passes near the Sun
deplete volatiles from the surface and near-surface of the nucleus. However, a
cometary outburst potentially provides the opportunity to release fresh,
minimally altered material from the interior of the nucleus. The unexpected
outburst of Jupiter Family comet 17P/Holmes in 2007 provided just such an
opportunity to detect and study fresh material from the comet’s deeper layers.

In this paper, we present Submillimeter Array (SMA) observations of the
outburst of 17P/Holmes, in which the brightness increased from apparent
magnitude $V\sim$17 on UT 2007 October 23.3 (Hsieh et al. 2010) to $V\sim$ 2.0
on UT 2007 October 25.0 (Li et al. 2011), corresponding to the sudden ejection
of (2 to 90)$\times$1010 kg of material (e.g. Ishiguro et al. 2010; Li et al.
2011). We use simultaneous imaging of HCN and CO from the outburst to identify
two different components in the molecular emission. These components exhibit
drastic differences in volatile composition as measured by CO/HCN, suggesting
that one component samples pristine material from deep in the nucleus.

##  2\. Observations

Figure 1.— Interferometric maps of the (sub)mm continuum emission of comet
17P/Holmes obtained from 26 to 29 October 2007 UT with the SMA. The contours
are 2.5 mJy Beam-1 (1$\sigma$) $\times[2,4,6,8,10,12,14]$ (for October 26);
0.55 mJy Beam-1 (1$\sigma$) $\times[3,6,12,24]$ (for October 27); 2.0
(1$\sigma$) $\times[2,4,6,8,10]$ (for October 28); 1.8 (1$\sigma$)
$\times[2,4,6,8,10]$ (for October 29).

Observations of 17P/Holmes were made on four consecutive days between 2007
October 26th and 29th using the SMA eight antenna interferometer located atop
Mauna Kea, Hawaii. The SMA operates in the 1.3 and 0.86 mm atmospheric
transmission windows which cover rotational transitions from a variety of
important cometary volatile species, such as HCN, CO and CS. The SMA was used
in the compact configuration, resulting in a spatial resolution of 2.5′′ at
349 GHz and 4′′ at 221 GHz, corresponding to 2950 and 4730 km in linear scale
at the Earth-comet distance of 1.63 AU. The primary beams are 31.2′′ and
49.8′′ FWHM, respectively. The SMA operated in a double-sideband (DSB) mode
with an intermediate frequency band of 4–6 GHz which was sent over fiber optic
transmission lines to 24 overlapping digital correlator bands (or “chunks”)
covering the 2 GHz spectral window. Phase-switching techniques were used to
separate and simultaneously recover both sidebands, resulting in 4 GHz of
total coverage (2 GHz per sideband). Doppler-tracking of 17P/Holmes was
performed so that the rapidly changing velocity of the comet, as defined by
available ephemerides at the time of observations (specifically, JPL orbital
solution #K077/6, HORIZONS System), was compensated for in real time, so the
velocity reported here is under the cometocentric velocity frame. At all
frequencies, simultaneous dust continuum measurements were made using the
line-free chunks. Tables 1 and 2 summarize the observational parameters for
the continuum and spectral line setups used for each date of observation. In
particular, observations on October 26, 28, and 29 focused on HCN and either
CO or CS near 349 GHz, while those from October 27 aimed to detect H2CO, H2S,
CH3OH and the deuterated species HDO and DCN near 221 GHz.

Calibration of visibility phases and amplitudes was achieved with periodic
snapshots of the quasar 0359+509 at 20 minute intervals. Measurements of
Uranus provided an absolute scale for calibration of the flux densities. The
derived mean flux densities of 0359+509 at the time of the observations were
2.05 Jy at 349 GHz and 3.25 Jy at 221 GHz. All data were phase- and amplitude-
calibrated using the MIR software package
111http://www.cfa.harvard.edu/$\sim$cqi/mircook.html. Continuum and spectral
line maps were then generated and CLEANed using the MIRIAD software package.

##  3\. Results

Figure 2.— Spatially integrated spectra at the peak continuum position of
17P/Holmes. The fluxes are averaged over the emission areas. All spectra are
plotted in a cometocentric velocity frame. The vertical dashed line mark the
cometocentric velocity of 0 km s-1.

Continuum emission at 221 and 349 GHz is readily detected by the SMA. Figure 1
shows (sub)mm continuum emission maps of comet 17P/Holmes obtained from 26 to
29 October 2007 and integrated flux densities are reported in Table 1. The
(sub)mm continuum emission probes the thermal radiation of mm-sized dust
particles in the coma and the peak of the emission is taken to reveal the
location of the nucleus (e.g. Jewitt & Luu 1992; Blake et al. 1999; Qi 2001).

Table 1Observational parameters for continuum observations  Date(UT) |  $\lambda$(mm) | Frequency(GHz) | Beam | PA | Flux (mJy) | Offsets (′′)
---|---|---|---|---|---|---
2007 October 26.3-26.7 | 0.86 | 349 | $2\farcs 7\times 2\farcs 1$ | 11.2∘ | 96.4$\pm$2.3 | (0.92,-0.26)
2007 October 27.3-27.7 | 1.35 | 221 | $3\farcs 9\times 3\farcs 6$ | 40.8∘ | 30.3$\pm$0.6 | (0.59,-0.20)
2007 October 28.3-28.7 | 0.86 | 349 | $2\farcs 4\times 2\farcs 1$ | 16.9∘ | 40.9$\pm$1.8 | (0.85,-0.26)
2007 October 29.3-29.7 | 0.86 | 348 | $2\farcs 4\times 2\farcs 2$ | 13.7∘ | 48.8$\pm$1.4 | (0.83,-0.17)

Table 2 lists the observational parameters and resulting detections for the
spectral line observations, while in Figure 2 the spectra from several
molecular species are displayed for each day. These spectra are all extracted
from a 10′′ square box centered on the nucleus, corresponding to the maximum
area with significant emission. We note that the decrease of the continuum
flux from October 26 to 29 is not smooth as the continuum flux on the 29th is
slightly higher than that on the 28th. This is also accompanied by a small but
significant increase in the velocity-integrated intensity level of HCN 4–3 as
reported in Table 2 from the 28th to 29th. This is probably caused by the
intermittent release of both big dust grains and volatiles sublimating from
the small icy grains near the nucleus by the outburst.

Table 2Observational parameters for line observations  Lines | Rest Frequency | Beam | PA | Channel spacing | FWHM aa Fitted with a Gaussian. Might not reflect the real line width. | Int. Intensity. bbIntegrated over 10′′ box.
---|---|---|---|---|---|---
| (GHz) | (′′) | (∘) | (km s-1) | (km s-1) | (Jy km s-1)
UT: 2007 October 26.3-26.7
HCN 4–3 | 354.505 | 2.7$\times$2.1 | 12.9 | 0.17 | 0.89 | 52.09[0.22]
H13CN 4–3 | 345.340 | 2.6$\times$2.0 | 9.6 | 0.35 | 0.92 | 1.32[0.22]
CO 3–2 | 345.796 | 2.6$\times$2.0 | 9.6 | 0.18 | 0.42 | 2.27[0.21]
UT: 2007 October 27.3-27.7
H2CO 31,2–21,1 | 225.698 | 3.8$\times$3.2 | 49.6 | 0.27 | 0.72 | 3.03[0.14]
HDO 31,2–22,1 | 225.897 | 3.8$\times$3.2 | 49.6 | 0.54 | – | [0.14]
DCN 3–2 | 217.239 | 4.2$\times$3.9 | -39.1 | 0.28 | – | [0.14]
H2S 22,0–21,1 | 216.710 | 4.2$\times$3.9 | -39.1 | 0.56 | – | 1.36[0.14]
CH3OH 51,0–42,0 | 216.946 | 4.2$\times$3.9 | -39.1 | 0.56 | – | 1.34[0.14]
UT: 2007 October 28.3-28.7
HCN 4–3 | 354.505 | 2.5$\times$2.1 | 19.4 | 0.086 | 0.69 | 48.79[0.16]
H13CN 4–3 | 345.340 | 2.4$\times$2.1 | 14.2 | 0.35 | 0.68 | 1.07[0.16]
CO 3–2 | 345.796 | 2.4$\times$2.1 | 14.2 | 0.088 | 0.50 | 1.75[0.15]
CH3OH 130,0–121,0 | 355.603 | 2.5$\times$2.1 | 19.4 | 0.34 | – | 2.40[0.16]
UT: 2007 October 29.3-29.7
HCN 4–3 | 354.505 | 2.4$\times$2.1 | 15.6 | 0.17 | 0.65 | 50.16[0.21]
CS 7–6 | 342.883 | 2.4$\times$2.2 | 11.1 | 0.18 | 0.46 | 4.45[0.21]

For each day, we were able to cover several species within the 4 GHz bandwidth
(both upper and lower sidebands). This provides the opportunity to compare the
extent of the emission from different species observed simultaneously in the
coma during the outburst. On October 27, the SMA was tuned near 221 GHz
(1.4mm) in order to detect H2CO, H2S and CH3OH, and search for the deuterated
species HDO and DCN (see Table 2). Observations near 349 GHz (0.86mm) of HCN
with either CO or CS were obtained on October 26, 28 and 29. While several
species at 1.4mm were detected and the integrated intensities are reported in
Table 2, the velocity resolution was insufficient for a detailed kinematic
study, so in this paper we focus on the HCN and CO/CS imaging and defer the
analysis of H2CO, CH3OH and H2S to future work.

To better compare the emission of the different species, we regenerated
channel images of the emission lines on a common velocity grid. Figures 3, 4,
and 5 present the resampled HCN 4–3, CO 3–2, and CS 7–6 emission. The line
width of HCN 4–3 is much broader than those of CO 3–2 or CS 7–6 for each day,
and the emission of HCN is also more extended spatially than that of CO and
CS. Most of the HCN emission appears symmetric, consistent with expectations
for isotropic outgassing. However, there are some deviations from this
symmetric pattern. Some HCN emission (for example near +0.4 km s-1) extends
toward the northeast. In contrast, the CO 3–2 and CS 7–6 emission is
significantly narrower in velocity width, and for both species the emission
shifts position with velocity, showing a gradient across the continuum
(nucleus) position. The peak emission from both lines is around the nucleus
position, but about 0.1–0.2 km s-1 red-shifted (cometocentric frame).

##  4\. Analysis

Figure 3.—  SMA observations of molecular emission from Comet 17P/Holmes on
October 26th, 2007, 2 days after the onset of outburst. The residual of the
HCN 4–3 emission (third row), obtained by subtracting the best-fit symmetric
outgassing model (second row) from the data (first row), is likely to created
by sublimating recently released ice grains around the nucleus during the
outburst. The extent of the CO 3–2 emission (fourth row) is consistent with
the residual of HCN emission, indicating a higher volatile content in this
component. For HCN emission, the contours start with 2$\sigma$ and each
following contour step is 3$\sigma$ (1$\sigma$=0.3 Jy Beam-1). For CO
emission, the contours start with 2$\sigma$ and step in 1$\sigma$. The cross
marks the position of the nucleus and arrows show the direction of the Sun.
North is the positive Y-axis and East is the positive X axis.

###  4.1. Continuum

Continuum measurements at 0.86 mm (Table 1) from UT October 26, 28 and 29 show
fading by a factor $\sim$2, roughly corresponding to an exponential decay with
e-folding time of $\sim$4 days. By interpolation, the flux density at
wavelength $\lambda_{0.86}$ = 0.86 mm on UT October 27 is estimated to be
$S_{0.86}$ = 68$\pm$7 mJy, which compares with the flux density at
$\lambda_{1.35}$ = 1.35 mm on this date of $S_{1.35}$ = 30.3$\pm$0.6 mJy. The
different beam sizes at 0.86 mm and 1.35 mm sample different volumes of the
near-nucleus dust coma, requiring a correction. We assume that the dust
density varies with the inverse square of the distance from the nucleus, in
which case the quantity of dust intercepted by an aperture varies in
proportion to $\phi$, the angular radius of the aperture. The effective
aperture size is given by the projected beam which, for our data, is
elliptical. We take the effective beam radius as
$\phi=(\phi_{max}\phi_{min})^{1/2}$, where $\phi_{min}$ and $\phi_{max}$ are
the beam ellipse maximum and minimum radii from Table 1. We find $\phi_{0.86}$
= 2.3″ and $\phi_{1.35}$ = 3.7″. Then, the spectral index of the continuum of
17P in the wavelength range from 0.86 to 1.35 mm is given by

$\alpha=\frac{log\left[(S_{1.35}/S_{0.86})(\phi_{0.86}/\phi_{1.35})\right]}{log(\lambda_{1.35}/\lambda_{0.86})}$
(1)

By substitution, we obtain $\alpha$ = 2.7$\pm$0.3, where the quoted
uncertainty mainly reflects the error in $S_{0.86}$ on UT October 27
introduced by the interpolation from adjacent dates. Evidently, $\alpha$ is
slightly larger than the expected blackbody spectral index in the Rayleigh-
Jeans regime, namely $\alpha$ = 2. However, it is consistent with earlier
measurements of dust in comet C/Hale-Bopp, which gave $\alpha$ = 2.60$\pm$0.13
(Jewitt and Matthews 1999). Values of $\alpha$ near 2 indicate that the
continuum cross-section is dominated by optically large dust particles ($2\pi
a/\lambda>1$, corresponding to particle radii $a\gtrsim$ 200 $\mu$m).

We used Equation (6) of Jewitt and Matthews (1999) to estimate the mass loss
rate from $S_{1.35}$ (Table 1), obtaining $dM/dt$ = 2.6$\times$106 kg s-1 on
UT October 27. This value, about 1000 times larger than typical of Jupiter
family comets at the same heliocentric distance, may be compared with the peak
mass loss rate three days earlier on UT 2007 October 24.5, estimated from
optical continuum data as being up to 1.4$\times$107 kg s-1 (Li et al. 2011).
On the other hand, the (gas) production rate five days later, on UT 2007
November 01, had fallen to 2$\times$104 kg s-1 (Schleicher 2009). Even
accounting for the decay of the brightness of 17P between October 27 and
November 01, it appears that 17P was an extremely dust-rich comet.

###  4.2. Symmetric outgassing model

Figure 4.—  Same as figure 3, but for October 28th, 2007.

Figure 5.—  Same as figure 3, but for October 29th, 2007\. For CS emission,
the contours start with 2$\sigma$ and step in 1$\sigma$ (1$\sigma$=0.3 Jy
Beam-1).

Figures 3, 4, and 5 suggest that much of the HCN emission toward 17P/Holmes
originated from symmetric outgassing. This emission typically has a relatively
large linewidth (FWHM $\sim$ 1 km s-1). There is also clear evidence that some
HCN emission does not follow this symmetric pattern. The CO 3–2 and CS 7–6
emission is dominated by emission asymmetric with respect to the systemic
velocity of the comet (0 km s-1 in the cometocentric frame), and shows smaller
line widths of FWHM $<$ 0.5 km s-1. Is there a relation between the
asymmetries in the HCN line and the narrow CO and CS emission? To answer this
question requires detailed modeling of the HCN emission. The modeling analysis
consists of the following steps. First, we construct a symmetric outgassing
model for HCN. Second, we subtract the symmetric outgassing model from the
data, and compare the subtracted HCN residual with the CO 3–2 and CS 7–6
emission to determine if they share the same origin. Finally, we investigate
any compositional differences between the symmetric and asymmetric parts of
the coma.

For a first-order analysis of the distribution of HCN in the coma, we assume
that the density in the coma drops with distance from the nucleus, which is
the only source for HCN. The distribution of density as a function of radius
can be approximated using the Haser model with a constant expansion velocity
vexp. The modeling procedure is similar to that presented in Hogerheijde et
al. (2009), adapted to the geometrical circumstances of 17P/Holmes. We
calculate line formation and molecular excitation in the coma using an adapted
version of the Accelerated Monte Carlo code by Hogerheijde & van der Tak
(2000), in which we consider excitation via collisions with water and
electrons, spontaneous and induced emission of line photons, and absorption of
line and continuum photons, including those of the solar infrared spectrum
(see Section 4.2 of Hogerheijde et al. (2009) for details of molecular
excitation calculation). For the main collisional partner, we adopt a water
production rate Q(H2O) = 7, 4, and 3$\times$1029 mol s-1, appropriate for
October 26th, 28th, and 29th (Dello Russo et al. 2008). For the gas
temperature, we adopt a constant kinetic temperature of 45 K, based on the
results of Bockelée-Morvan et al. (2008).

Table 3Spectroscopic values for HCN 4–3 hyperfine components  J F | J’ F’ | Frequency (GHz) |  Rel. Intensity
---|---|---|---
4 4 | 3 4 | 354.5038689 | 0.021
4 3 | 3 2 | 354.5053670 | 0.24
4 4 | 3 3 | 354.5054778 aaReference frequency | 0.31
4 5 | 3 4 | 354.5055234 | 0.41
4 3 | 3 4 | 354.5058468 | 5$\times$10-4
4 3 | 3 3 | 354.5074558 | 0.021

HCN has a hyperfine structure due to the nuclear quadruple moment of 14N. We
adopt the six HCN hyperfine components of the 4–3 transition and their
relative intensities from the JPL Molecular Spectroscopy database listed in
the Table 3. Relative populations between the hyperfine levels are assumed to
be in LTE. The HCN emission is not very optically thick. For example on
October 28th, the integrated intensity of the central four hyperfine
components is 46.30 Jy km s-1, and the outer hyperfine component 43–33 (well
separated from the others) is 1.25 Jy km s-1. So the measured ratio of this
component over the central four combined is 0.026$\pm$0.003, close to the
expected ratio of 0.022 for optically thin emission. After construction of the
emission model, we multiply it by the appropriate primary beams and resample
with the $uv$ data using the actual SMA antenna positions to generate model
visibilities. To find the best-fitting description of the symmetric emission,
we only consider emission outside the cometocentric velocity range of $-0.2$
to $+0.4$ km s-1, corresponding to the range outside where the narrow CO and
CS emission dominates. Using the comet model and the excitation calculation
described above, we compute a grid of synthetic HCN visibility datasets over a
range of QHCN and Vexp values and compare with the observations. The best-fit
model is obtained by minimizing the $\chi^{2}$, the weighted difference
between the data and the model with the real and imaginary part of the complex
visibility measured in the $(u,v)$-plane sampled by the HCN observations. Our
analysis is performed in the visibility domain rather than the image domain,
to avoid the non-linear effects of the imaging and deconvolution process.

Table 4HCN symmetric outgassing model fitting result aa Symmetric outgassing model fitting on velocity range outside of $-$0.2 to $+$0.4 km -1, i.e. not including the asymmetric narrow-line component of the HCN emission. Date | vexp(km s-1) | QHCN(1026 mol s-1)
---|---|---
2007 October 26 | 0.46[0.01] | 12.5[1.0]
2007 October 28 | 0.44[0.01] | 8.0[0.3]
2007 October 29 | 0.38[0.01] | 6.3[0.2]

Figure 6.— Iso-$\chi^{2}$ surfaces of $Q_{\rm HCN}$ versus $v_{\rm exp}$.
Contours correspond to the 1–6 $\sigma$ errors.

The best-fit HCN production rates and expansion velocities are derived and
listed in Table 4. Figures 3, 4, and 5 present comparisons between the
observed channel maps and the best-fit model. The model reproduces the main
features of symmetric pattern of the HCN emission. The line optical depth at
center is around 0.5–0.6, consistent with the optical depth assessed above
using the hyperfine components ratio. Figure 6 shows the $\chi^{2}$ surfaces
for the QHCN versus Vexp values, which enables us to quantify the
uncertainties associated with the model production rates and the expansion
velocities fit for the three dates. The best-fit HCN production rates decrease
from the 26th to the 29th and the outflow expansion velocity drops quickly.
However there are significant residuals between the data and model. Figure 7
shows the spectra of the symmetric HCN line and the residual, compared with
the full spectra of HCN line taken on October 28, demonstrating the narrow
line width of the residual compared to the symmetric model.

Figure 7.—  The broad-line component (dashed line) and narrow-line component
(dotted line), compared with the full HCN data spectra (solid line) from
17P/Holmes on 2007 October 28 UT. The positions and relative intensities of
the hyperfine components of HCN 4–3 transition are shown at the bottom. The
broad-line component is retrieved from the best-fit symmetric outgassing model
with vexp=0.44 km s-1 and QHCN=8$\times$1026 s-1. The narrow-line component is
derived from the residual of the HCN emission subtracted from the best-fit
symmetric outgassing model. The vertical dashed line marks the cometocentric
velocity of 0 km s-1.

Using this symmetric model, we need QHCN = (1.20$\pm$0.01)$\times$1027 and
Q${}_{\rm H^{13}CN}$ = (1.5$\pm$0.3) $\times$1025 s-1 to match the October 28
HCN and H13CN 4–3 integrated intensities (see Table 2). Therefore we derive
H12CN/H13CN = 80$\pm$16, which is consistent with the nominal terrestrial
value of 89 and with the measurement (H12CN/H13CN=114$\pm$26) by Bockelée-
Morvan et al. (2008). We note that the production rates reported in Table 4
are derived to fit only the part of the HCN lines contributed from the
symmetric outgassing. It is not possible to fit the full HCN and H13CN
emission using only the symmetric model. However, as long as we can generate
similar integrated intensities and treat both lines in the same way, this
should give reasonable estimates of the abundance ratio. With the same method,
we obtain QHCN = 1.5$\times$1027 for 2007 October 27 by interpolating the HCN
production rates between October 26 and October 28, and the upper limit of
QDCN = 1.2$\times$1025 to achieve 0.42 Jy km s-1 flux density (3$\sigma$ upper
limit) of DCN 3–2 emission obtained on 2007 October 27, and derive DCN/HCN $<$
8$\times$10-3, consistent with the bulk D/H ratio in HCN measured from comet
Hale-Bopp (2.3$\pm$0.4 $\times$10-3, Meier et al. 1998). However, we cannot
yet separate the contribution from the freshly released material in the
nucleus as in Blake et al. (1999) due to the limited sensitivity.

The derived bulk HCN production rate of 1.2 $\times$ 1027 s-1 on October 28
and an interpolated value of 1.5 $\times$ 1027 s-1 on October 27 are generally
consistent with the values derived from single dish observations (2.0$\times$
1027, IRAM 30m, Bockelée-Morvan et al. 2008) and infrared observations
(2.4$\times$ 1027, NIRSPEC, Dello Russo et al. 2008) derived on October 27,
considering the uncertainties involved in the measurements and the use of
different aperture sizes.

###  4.3. Narrow-line component

Figure 8.— Integrated intensity images red-shifted (integrated from 0.3 to 0.6
km s-1) vs blue-shifted (integrated from $-$0.3 to 0 km s-1) for HCN 4–3
residuals, CO 3–2 and CS 7–6 on 2007 October 26–29. The contours are at
50%,70%, and 95% of the peak value. The cross marks the peak of the continuum
emission, which locates the position of the nucleus. P.A. is measured from the
peak of the blue-shifted image to that of the red-shifted image. The Sun
direction is shown for 2007 October 26 and the solar direction only changed
4.1 degrees clockwise from October 26 to 29.

As shown in Figure 7, the linewidth of the residual HCN emission is much
smaller than those in the symmetric outgassing models. The spatio-kinematic
characteristics of the residual HCN emission are very similar to those of the
CO 3–2 and CS 7–6 observed simultaneously on those days. We will refer to this
component as the narrow-line component. The main characteristics that separate
it from the broad-line emission are:

  1. 1.

The narrow-line component occupies a velocity range from $-0.2$ to +0.4 km
s-1, much narrower than the broad-line component associated with symmetric
outgassing that emits from $-0.8$ to +0.8 km s-1.

  2. 2.

The line center (where the image peak coincides with the peak of the mm
continuum emission or nucleus position) for the HCN 4–3 residuals, CO 3–2, and
CS 7–6 are all around 0.1–0.2 km s-1 red-shifted, while the line center
velocity of the broad-line component is 0 km s-1.

  3. 3.

The narrow-line component shows emission that is red-shifted toward the
northeast and blue-shifted toward the southwest relative to the mm continuum
peak. Figure 8 shows the red-shifted (integrated from $+$0.3 to $+$0.6 km s-1)
and the blue-shifted (integrated from $-$0.3 to 0 km s-1) emission of the HCN
4–3 residuals, compared with the CO 3–2 and CS 7–6 emission on 2007 October
26–29. Especially on the 29th, the position angle (P.A.) of the HCN residual
is the same as that of CS 7–6. The velocity shift of CO 3–2 is not as obvious
on 26th but it does show a similar extension as the HCN residual on 28th.

The volatiles in the narrow-line component likely share the same origin in the
outbursting coma of 17P/Holmes. The P.A. of the red/blue gradient of the
narrow emission changed gradually from 66 to 30∘ during the period of
observations. Although in the same sense, this is a much larger shift than
that of the P.A. of the line toward the Sun, which only changed from 41.0 to
36.9∘. We propose that the (residual) HCN, CO, and CS emission of the narrow
component may come from volatiles released from very small grains of pure ice
freshly released from the nucleus. Without absorptive dust, the ice warms up
at a much slower rate. CO might be released faster than HCN and CS (presumably
a product of CS2 in the ice), accounting for the morphological difference
between CO and CS (and the HCN residuals). We hypothesize that the narrow-line
component originates from the ice grain halo found in the near-nucleus
photometry which is believed to be created by sublimating ice grains around
the nucleus due to the outburst (Stevenson & Jewitt 2012). Based on time-
series photometry of 17P/Holmes before the outburst, Snodgrass et al. (2006)
estimated a rotation period for 17P/Holmes of $7.2<P_{rot}<12.8$ h. If the
change of the P.A. of the narrow-line component was caused by the rotation of
the nucleus, the gradual change from 66 to 30 degrees from 26th to 29th
implies a nucleus rotational period of 11.8 h (clockwise in the plane
perpendicular to the line of the sight) or 12.2 h (anti-clockwise). We discuss
the nature of this narrow-line component further in the discussion section.

###  4.4. CO/HCN ratios

Figure 9.—  left: The integrated intensity map of CO 3–2 toward 17P/Holmes on
2007 October 28 UT, integrated between ($-$0.8,$-$0.2) and (0.4,0.8) km s-1;
right: the simulated map with vexp=0.44 km s-1 and QCO=5.0$\times$1027 s-1,
integrated over the same velocity range. The contour level is 0.08 Jy Beam-1
km s-1. The peak of the simulated map matches the rms (0.09 Jy Beam-1 km s-1)
of the left image, providing the upper limit of the contribution to the CO
emission from the broad-line component.  Figure 10.— Broad-line components of
HCN 4–3 and CO 3–2 emission (upper limit) from 17P/Holmes on 2007 October 28
UT. The solid-line HCN spectra, retrieved from the best-fit symmetric
outgassing model with vexp=0.44 km s-1 and QHCN=8$\times$1026 s-1, is overlaid
with the data spectra in gray. The solid-line CO spectra is the simulated
spectra from the model with vexp=0.44 km s-1 and QCO=5$\times$1027 s-1. The
simulated CO emission model generates the intensity image integrated between
(-0.8,-0.2) and (0.4,0.8) km s-1 as shown in Figure 9, giving the upper limit
of the contribution to the CO emission from the broad-line component. The
vertical dashed line marks the cometocentric velocity of 0 km s-1.  Figure
11.— Narrow-line components of HCN 4–3 and CO 3–2 emission from 17P/Holmes on
2007 October 28 UT. The HCN spectra (in gray) is retrieved from the residual
of HCN emission subtracted by the best-fit symmetric outgassing model with
vexp=0.44 km s-1 and QHCN=8$\times$1026 s-1. The solid-line HCN spectra is the
simulated spectra from the model with vexp=0.15 km s-1 and QHCN=1$\times$1026
s-1 which generates the same flux as the spectra in gray. The solid-line CO
spectra is simulated from the model with vexp=0.15 km s-1 and
QCO=4.5$\times$1027 s-1 which matches the CO data flux in the gray spectra,
1.75 Jy km s-1, while the dotted-line CO spectra is simulated with vexp=0.15
km s-1 and QCO=3.5$\times$1027 s-1 which generates the integrated flux 1.35 Jy
km s-1. The vertical dashed line marks the cometocentric velocity of $+$0.1 km
s-1.

Here we investigate the CO/HCN ratios within the two components based on the
data obtained on 2007 October 28, which had the finest spectral resolution for
both HCN 4–3 and CO 3–2. For HCN, the parameters of the broad-line component
come from the best-fit symmetric outgassing model as described above, by
fitting the data avoiding the velocity range from $-$0.2 to +0.4 km s-1. The
best-fit production rate is 8.0$\times$ 1026 s-1 and the expansion velocity
0.44 km s-1. The narrow-line component is retrieved from the residual HCN
emission after subtraction of the best-fit symmetric outgassing model. Figure
4 and 7 show the simulated HCN images and spectra of the two components
compared with the data. As shown in the figures the narrow-line component
mainly emits over the velocity range of $-$0.2 to +0.4 km/s, which is
consistent with the CO emission obtained simultaneously. This indicates that
most of the CO emission, if not all, must originate from the narrow-line
component.

In order to estimate how much the broad-line component contributes to the CO
emission, we make use of the emission avoiding the velocity range from $-$0.2
to +0.4 km s-1, similar to the HCN analysis. However, since the signal is too
weak for the CO emission, we integrate the emission with velocity between
$-$0.8 to +0.8 km s-1, again avoiding the velocity range from $-$0.2 to +0.4
km s-1, which is clearly dominated by the narrow-line component. Figure 9
shows the integrated intensity image, which does not show any significant
emission left with the noise level at 0.09 Jy Beam-1 km s-1. The upper limit
of the broad-line component contribution can be obtained by simulating CO
emission with the same broad-line expansion velocity as determined in HCN
emission, i.e., 0.44 km s-1, and matching the 1$\sigma$ integrated flux level
within the same velocity range. This is achieved with QCO = 5.0$\times$1027
s-1. Figure 9 compares the integrated intensity of the simulated image with
that of CO data and Figure 10 demonstrates the broad-line components of HCN
4–3 and CO 3–2, compared with the data. The CO/HCN ratio in the broad-line
component is therefore $<$ 7\. Note that the upper limit of the CO 3–2 broad-
line component contributes 0.3 Jy km s-1 in the velocity range from $-$0.2 to
0.4 km s-1, which is about 1/6 of the total integrated intensity of CO 3–2
emission.

In order to obtain the CO/HCN ratio in the narrow-line component, we first
simulate HCN emission based on the HCN residual, using the same symmetric
outgassing model described above, only with much smaller expansion velocity.
However we only fit on the integrated flux based on the spectra, as we cannot
reproduce the spatial shift across the nucleus position. Although not ideal,
this approach should give a reasonable estimate of the abundance ratio, as
long as CO and HCN are treated in the same way. Fitting the residual HCN
spectra yields a production rate of QHCN = 1.0$\times$1026 s-1 and expansion
velocity 0.15 km s-1. Assuming there is no contribution of the broad-line
component to the CO line, we find QCO = 4.5$\times$1027 s-1 to match the
integrated intensity of 1.75 Jy km s-1 for CO 3-2 with an expansion velocity
of 0.15 km s-1. This results in a CO/HCN ratio of 45. If we consider a maximum
contribution of the wide component to the CO line equal to the upper limit
derived above, i.e., 0.3 Jy km s-1, we find QCO = 3.5$\times$1027 s-1 to
reproduce the integrated intensity of 1.45 Jy km s-1, and a CO/HCN ratio =
35\. Figure 11 shows the narrow-line components of HCN 4–3 and CO 3–2,
compared with the CO data and residual HCN spectra. Since the unknown
contribution from the broad-line component dominates the uncertainty of the
calculation, we conclude that the ratio of CO/HCN in the narrow-line component
can be constrained to 40$\pm$5\. This is significantly higher than the
CO/HCN$<7$ found for the broad-line component. Using the same analysis, we
determine the ratio CO/HCN $<$ 9 for the broad-line component and 45$\pm$10
for the narrow-line component on October 26. This is consistent with the
result on October 28, despite larger uncertainties due to the fact that the
spectral resolution of HCN and CO lines carried out on October 26 is a factor
of two worse than on October 28, which creates large uncertainties in how much
the broad-line component contributes to the CO emission on October 26.

HCN is the molecule which exhibits the lowest abundance variation from comet
to comet (Biver et al. 2002; Mumma & Charnley 2011). Therefore, production
rates relative to that of HCN can be used for a comparative study of molecular
abundances. From a study of a sample of 24 comets, Biver et al. (2002) found
that CO/HCN varies from $<$24 to 180. In particular, 4 long period comets show
CO/HCN$>$85 while 4 other comets, including one short period comet, show
CO/HCN$<$40\. However, there is still no obvious correlation between the
dynamical class and the chemical composition of comets (Biver et al. 2002;
Crovisier, 2007). We find in the near-nucleus outgassing of comet 17P/Holmes,
the CO/HCN ratio is much higher in the narrow component than the broad one.
The difference of CO/HCN ratios in the two components demonstrates the
heterogeneity of the nucleus of this Jupiter Family Comet.

The decrease in the HCN production rates indicates an $e-$folding time $\sim$4
days in the October 26 to 29 period (Table 4). Schleicher (2009) measured the
decline of the production rates of optical species. From their Table 3 we find
an $e-$folding time $\sim$14 days in the 2007 November to 2007 December
period, with an indication that the fading timescale increases as time passes
up to their last observation in 2008 March. Interestingly, their CN production
rate, $\log_{10}(Q_{\rm CN})$ = 27.4 on November 1, is higher than the HCN
production rate $\log_{10}(Q_{\rm HCN})$ = 27.1 we derived four days earlier
(October 28) when matching the whole flux of HCN 4–3 emission. This inequality
could be an artifact of the use of the Haser model (which applies to a coma
expanding in steady state and which may not accurately describe the outburst
coma of 17P) or it could be an indication that CN has a parent in addition to
HCN. Furthermore, the photometer entrance apertures used in the optical
measurements, ranging from 24 to 204 arcsec, are different from the primary
beams (31 arcsec) used in our observations. This could also contribute to the
difference in production rates derived for the CN and HCN molecules.

##  5\. Discussion

What are the two components in the near-nucleus coma of the outbursting
17P/Holmes? As shown from the above model fitting, the HCN emission from the
broad-line component can be fit with symmetric outgassing with an expanding
velocity about 0.5 km s-1. Drahus et al. (2007) and Snodgrass et al. (2007)
found that an almost-spherical dust coma expanded around the nucleus of
17P/Holmes at 0.55 km s-1. Both the expansion velocity and symmetric
outgassing pattern of the broad-line component appear to be related to this
spherical dust coma releasing dust from the surface of the nucleus.

However, within the dust coma, bright streaks emerged from the nucleus, likely
produced by fragments that moved away from the nucleus with velocities of up
to 0.1 km s-1 (Trigo-Rodriguez et al. 2007; Stevenson et al. 2010). Gaillard
et al. (2007) also postulated that the streaks originated from disintegrating
cometary fragments. The images from Hubble telescope’s Wide Field Planetary
Camera 2 (WFPC2) also showed a “bow tie”
appearance222http://www.spacetelescope.org/news/heic0718/, about twice as much
dust lies along the east-west direction as along the north-south direction,
similar to the morphology we have detected in the narrow-line component. We
find the production rate from the broad-line component of HCN 4–3 decreased
but the whole integrated intensity of the line actually increased from October
28th to 29th, which implies that the narrow-line component should have
increased. Coupled with the increase of the mm continuum flux level between
the two days, this provides strong evidence for disintegrating cometary
fragments continuously supplying the dust and gas near the nucleus. So we
believe the narrow-line component detected in the SMA spectral line imaging
shown here should associate with volatiles sublimating from the fragments near
the nucleus which was the most recently ejected and most pristine in nature.
The main problem with this interpretation is how long these streaks/fragments
persisted. Stevenson & Jewitt (2012) presented the near-nucleus photometry
over a period of three months following the outburst of comet 17P/Holmes and
they found a halo consisting of freshly released icy grains persisted around
the nucleus and extended to radii of several thousand kilometers, encompassing
the emission area we detected. In our interpretation, the narrow-line
component likely originated from this halo, with volatiles continuously
supplied from the freshly released icy grains. Yang et al. (2009) suggested
that only clean ice grains, with $<$10% impurities, could survive long enough
to travel 3500 km from the nucleus before sublimating, which could explain the
offsets from the nucleus seen in the narrow-line component as in emission of
CO, CS and HCN residuals. The line width of the narrow-line component does not
change much with time over the four days of observations, compared to the
rapid narrowing of the expansion velocity of the symmetric outgassing
component. This is consistent with the long lifetime of the halo feature found
by Stevenson & Jewitt (2012).

##  6\. Summary

The Jupiter family comet 17P/Holmes underwent a dramatic outburst in 2007,
providing us with a rare opportunity to access fresh material released by the
powerful outburst from beneath the thermally-processed surface. We have
observed 17P/Holmes with the SMA from October 26 through October 29 2007 at a
spatial resolution of $\sim$3000 km at the comet distance and found peculiar
outgassing in the near-nucleus coma. Our key results include the following.

  1. 1.

We find two components in the molecular emission from the near-nucleus coma:
one has relatively broad linewidth ($\sim$1 km s-1 FWHM), showing a symmetric
outgassing pattern with respect to the nucleus position. The other has a
narrow linewidth ($<$0.5 km s-1 FWHM). HCN 4–3 emission has both components,
while the emission of CS 7–6 and CO 3–2 is dominated by the narrow-line
component.

  2. 2.

The morphologies of the two components are different: The peaks of the broad-
line component in the HCN 4–3 emission are on the nucleus position and do not
change with velocity; the line center of the narrow-line component is red-
shifted by 0.1–0.2 km s-1 (cometocentric frame). The peaks of the emission
shift with velocity across the nucleus position and the position angle of the
red/blue gradient changed from $66\arcdeg$ to $30\arcdeg$ during the four days
of observations.

  3. 3.

We determine distinctly different CO/HCN ratios for each of the components:
CO/HCN $<$7 in the broad-line component and CO/HCN=40$\pm$5 in the narrow-line
one. This different line ratio reflects the heterogeneity of the nucleus of
this Jupiter family comet.

  4. 4.

We determine the ${}^{12}\rm C$/${}^{13}\rm C$ ratio=80$\pm$16 based on the
simultaneous observations of HCN and H13CN 4–3 emission, which is consistent
with the terrestrial value.

These observations demonstrate the unique power of radio interferometers with
high spatial and spectral resolution to map small scale spatio-kinematic
differences in emission of primary volatiles in the near-nucleus cometary
comae to probe the heterogeneity in the nucleus. Simultaneous observations of
multiple primary volatiles are key to understanding and interpreting the
outgassing properties of the nucleus. The newly constructed Atacama Large
Millimeter/Submillimeter Array, or ALMA, will allow to assess the composition
of a large sample of comets to examine its relationship with cometary
formation region.

Facilities: SMA

The SMA is a joint project between the Smithsonian Astrophysical Observatory
and the Academia Sinica Institute of Astronomy and Astrophysics and is funded
by the Smithsonian Institution and the Academia Sinica. We thank SMA Director
Raymond Blundell for quick allocation of the DDT time for the above
observations and the SMA staff, especially Ken (Taco) Young and Nimesh Patel,
for observational supports. DJ acknowledges support from NASA’s Origins
program.

## References

  * Biver et al. (2002) Biver, N., Bockelée-Morvan, D., Crovisier, J., et al. 2002, Earth Moon and Planets, 90, 323
  * Blake et al. (1999) Blake, G. A., Qi, C., Hogerheijde, M. R., Gurwell, M. A., & Muhleman, D. O. 1999, Nature, 398, 213
  * Bockelée-Morvan et al. (2008) Bockelée-Morvan, D., Biver, N., Jehin, E., et al. 2008, ApJ, 679, L49
  * Crovisier (2007) Crovisier, J. 2007, arXiv:astro-ph/0703785
  * Dello Russo et al. (2008) Dello Russo, N., Vervack, R. J., Jr., Weaver, H. A., et al. 2008, ApJ, 680, 793
  * Drahus et al. (2007) Drahus, M., Paganini, L., Ziurys, L., et al. 2007, IAU Circ., 8891, 1
  * Ehrenfreund & Charnley (2000) Ehrenfreund, P., & Charnley, S. B. 2000, ARA&A, 38, 427
  * Gaillard et al. (2007) Gaillard, B., Lecacheux, J., & Colas, F. 2007, Central Bureau Electronic Telegrams, 1123, 1
  * Hogerheijde et al. (2009) Hogerheijde, M. R., Qi, C., de Pater, I., et al. 2009, AJ, 137, 4837
  * Hogerheijde & van der Tak (2000) Hogerheijde, M. R., & van der Tak, F. F. S. 2000, A&A, 362, 697
  * Hsieh et al. (2010) Hsieh, H. H., Fitzsimmons, A., Joshi, Y., Christian, D., & Pollacco, D. L. 2010, MNRAS, 407, 1784
  * Ishiguro et al. (2010) Ishiguro, M., Watanabe, J.-i., Sarugaku, Y., et al. 2010, ApJ, 714, 1324
  * Jewitt & Luu (1992) Jewitt, D., & Luu, J. 1992, Icarus, 100, 187
  * Jewitt & Matthews (1999) Jewitt, D., & Matthews, H. 1999, AJ, 117, 1056
  * Levison & Duncan (1994) Levison, H. F., & Duncan, M. J. 1994, Icarus, 108, 18
  * Li et al. (2011) Li, J., Jewitt, D., Clover, J. M., & Jackson, B. V. 2011, ApJ, 728, 31
  * Meier et al. (1998) Meier, R., Owen, T. C., Jewitt, D. C., et al. 1998, Science, 279, 1707
  * Mumma & Charnley (2011) Mumma, M. J., & Charnley, S. B. 2011, ARA&A, 49, 471
  * Qi (2001) Qi, C. 2001, Ph.D. Thesis, California Institute of Technology
  * Schleicher (2009) Schleicher, D. G. 2009, AJ, 138, 1062
  * Sekanina (2009) Sekanina, Z. 2009, International Comet Quarterly, 31, 5
  * Snodgrass et al. (2006) Snodgrass, C., Lowry, S. C., & Fitzsimmons, A. 2006, MNRAS, 373, 1590
  * Snodgrass et al. (2007) Snodgrass, C., Fitzsimmons, A., Boehnhardt, H., et al. 2007, Central Bureau Electronic Telegrams, 1111, 1
  * Stevenson et al. (2010) Stevenson, R., Kleyna, J., & Jewitt, D. 2010, AJ, 139, 2230
  * Stevenson & Jewitt (2012) Stevenson, R., & Jewitt, D. 2012, AJ, 144, 138
  * Trigo-Rodriguez et al. (2007) Trigo-Rodriguez, J. M., Abraham, P., Konkoly, A., et al. 2007, Central Bureau Electronic Telegrams, 1118, 1
  * Volk & Malhotra (2008) Volk, K., & Malhotra, R. 2008, ApJ, 687, 714
  * Yang et al. (2009) Yang, B., Jewitt, D., & Bus, S. J. 2009, AJ, 137, 4538


# Vision-based DRL Autonomous Driving Agent with Sim2Real Transfer

Dianzhao Li and Ostap Okhrin Authors are with the Chair of Econometrics and
Statistics, esp. in the Transport Sector, Technische Universität Dresden,
Dresden, 01187, Germany and Center for Scalable Data Analytics and Artificial
Intelligence (ScaDS.AI) Dresden/Leipzig, Germany. (E-mails: dianzhao.li@tu-
dresden.de<EMAIL_ADDRESS>
###### Abstract

To achieve fully autonomous driving, vehicles must be capable of continuously
performing various driving tasks, including lane keeping and car following,
both of which are fundamental and well-studied driving ones. However, previous
studies have mainly focused on individual tasks, and car following tasks have
typically relied on complete leader-follower information to attain optimal
performance. To address this limitation, we propose a vision-based deep
reinforcement learning (DRL) agent that can simultaneously perform lane
keeping and car following maneuvers. To evaluate the performance of our DRL
agent, we compare it with a baseline controller and use various performance
metrics for quantitative analysis. Furthermore, we conduct a real-world
evaluation to demonstrate the Sim2Real transfer capability of the trained DRL
agent. To the best of our knowledge, our vision-based car following and lane
keeping agent with Sim2Real transfer capability is the first of its kind. We
have made the codes and the videos of the simulation and real-world evaluation
accessible online111Code and videos are available on:
https://github.com/DailyL/Sim2Real_autonomous_vehicle.

##  I INTRODUCTION

In recent years, DRL has garnered significant attention from researchers as a
powerful method to solve complex control problems across various fields,
including Robotics [kober2013reinforcement, andrychowicz2020learning,
gu2017deep, haarnoja2017reinforcement], Game playing
[silver2017mastering,mnih2015human,silver2016mastering,hessel2018rainbow,wurman2022outracing],
Finance [10, 11], and Natural language processing [12, 13]. In the field of
autonomous driving, DRL has emerged as a natural tool to tackle the intricate
decision-making required in complex situations.

The car following task is a basic driving task that can be addressed by rule-
based control strategies, which are developed based on empirical observations
of driver behavior and usually employ a set of fixed parameters to describe
vehicle behavior. The Wiedemann car following model [14] is one such model
that describes the psycho-physiological aspects of driving behavior through
four distinct driving regimes: free flow, approaching slower vehicles, car-
following near the steady-state equilibrium, and critical situations requiring
stronger braking actions. In addition to the Wiedemann car following model,
intelligent driver model (IDM) [15] is another widely used time-continuous car
following model. It can generate realistic acceleration profiles and plausible
behaviors in all single-lane traffic scenarios. IDM takes into account the
desired speed, desired headway distance, and the difference between the actual
and desired speed and headway. It can also model different driving behaviors
such as aggressive and conservative driving. However, these models may not
accurately capture the complex and dynamic behavior of drivers in real-world
scenarios. To overcome this limitation, several studies have explored the use
of DRL for the car following task. Considering the safety, efficiency, and
comfort, the DRL car-following agent can provide improved performance in
dynamic driving conditions[lin2019longitudinal, yen2020proactive,
naing2022dynamic]. Furthermore, the real-world driving data can be used to
train DRL agents for human-like behaviors [19] or improvement of driving
policies [20]. Besides utilizing various input features for vehicle control,
vision-based car-following models exist that mimic the behavior of human
drivers. These models rely on visual cues and processing to accurately track
and follow other vehicles on the road, similar to how human drivers operate
[21, 22]. Lane keeping is another essential component of autonomous driving
that help to improve road safety and reduce driver fatigue. It uses a
combination of sensors, cameras, and algorithms to detect lane markings on the
road and provide feedback to the steering system and keep the vehicle centered
within the lane. Lane keeping has been extensively researched with various
solutions, for instance, PID controller, fuzzy logic controller, and DRL
algorithm [amditis2010situation, sallab2016end, basjaruddin2015lane], etc.

In addition to training DRL agents in simulated environments, it is also a
challenge to transfer the trained agents to real-world traffic environments
(Sim2Real). Even though the simulators are getting realistic, there are still
significant discrepancies between two environments (Sim2Real gap). To address
this issue, several methods have been proposed, including domain
randomization, domain adaptation, and knowledge distillation [tobin2017domain,
traore2019continual, hinton2015distilling], to narrow the gap between the
simulation and reality. With the Sim2Real techniques, researchers are able to
transfer the trained DRL agent to real-world to perform lane keeping,
objective detection, overtaking, etc [hu2022sim,
reiher2020sim2real,muller2018driving,li2023platform].

However, the aforementioned works are mainly focused on a single driving task
such as lane keeping or car following, and the car following task mostly
depends on multiple sensors to detect the gap between two vehicles and the
velocity difference. To the best of our knowledge, there are no related works
for vision-based multi-task driving agents with the abilities for Sim2Real
transfer. This research gap motivated us to propose a robust DRL agent
performing multi-task driving behavior that only depends on visual input.

[scale=0.75]system.pdf

Figure 1: The proposed DRL framework for vision-based multi-task autonomous
driving agents. The _perception module_ leverages camera images to produce
impact attributes regarding the environment, then the DRL _control module_
utilizes the information to control the agent with enhanced generalization.

Summarizing, the main contributions of this work are as follows:

  * •

The training framework depicted in Fig. 1 with the separation of the
_perception module_ and DRL _control module_ is proposed to narrow the
Sim2Real gap. The inclusion of impact affordances derived from the _perception
module_ has been demonstrated to enhance the stability of training and the
overall generalization of the model.

  * •

Considering safety, efficiency, and lane keeping capabilities, the trained DRL
agent can perform stable car-following behavior, while no vehicle driving in
front, the DRL agent shows excellent lane keeping ability.

  * •

Compare with the baseline controller and evaluate the performance with
multiple metrics, where the DRL agent achieves similar performance, even
though it only depends on visual input.

  * •

The Sim2Real transfer is performed for trained DRL agent in the real-world
environment, and the generalization of the agent under the noise of the real
world is evaluated.

##  II Background

###  II-A Deep Reinforcement Learning

Reinforcement Learning (RL) involves an agent interacting with an environment
to maximize the reward it receives. This is done by treating the process as a
Markov Decision Process (MDP), in which the agent observes a state $s_{t}$ of
the environment at time step $t$, and then chooses an action $a_{t}$ according
to the current policy $\pi(a_{t}|s_{t})$. The environment transitions to a new
state $s_{t+1}$ with probability $P(s_{t+1}|s_{t},a_{t})$, and provides a
reward $r_{t+1}$ to the agent as feedback. The objective is to determine an
optimal policy $\pi^{*}$ that maximizes the expected discounted cumulative
rewards
$\mathbb{E}_{\pi^{*}}\left\\{\sum_{k=0}^{\infty}{\gamma}^{k}r_{t+k+1}\right\\}$,
where $\gamma\in\left[0,1\right]$ is a discount factor that determines the
tradeoff between immediate and future rewards.

To measure the quality of a state $s_{t}$ for a given policy $\pi$, we use a
state-value function $V^{\pi}$($s_{t}$) =
$\mathbb{E}_{\pi}\left\\{R_{t}|s_{t}\right\\}$, which represents the expected
cumulative reward starting from state $s_{t}$ and following policy $\pi$.
Similarly, the action-value function $Q^{\pi}$($s_{t},a_{t}$) =
$\mathbb{E}_{\pi}\left\\{R_{t}|s_{t},a_{t}\right\\}$ measures the expected
cumulative reward starting from state $s_{t}$, taking action $a_{t}$, and then
following policy $\pi$.

However, in practice, the state and action space can be vast and complex,
making it computationally infeasible to use a tabular format to find optimal
policies. To overcome this issue, DRL employs Deep Neural Networks to
approximate the optimal Q-values and has shown remarkable success in solving
complex tasks and achieving human-level performance.

###  II-B Proximal Policy Optimization (PPO)

Proximal Policy Optimization (PPO) is a policy-based method, which means that
it directly learns a policy, mapping from states to actions, without computing
a value function [33]. It uses a trust region optimization approach that
restricts the size of the updates to the policy. This allows the algorithm to
maintain the stability of the policy while still making progress toward the
optimal policy. PPO uses a clipping parameter to limit the size of the policy
updates, which prevents the new policy from diverging too far from the old
policy and helps to ensure that the policy remains stable during training.
Therefore, PPO is computationally efficient and scalable to handle large-scale
problems. It is highly effective in a wide range of applications, including
game playing, robotics, and natural language processing.

PPO-Clip [1] Input: initial policy parameter $\theta_{0}$ and value function
parameter $\phi_{0}$. k = 0, 1, 2, … Collect set of trajectories
$\mathcal{D}_{k}=\\{\tau_{i}\\}$ by running policy $\pi_{k}=\pi(\theta_{k})$
in the environment. Compare rewards-to-go $\hat{R}_{t}$. Compute advantage
estimates, $\hat{A}_{t}$ based on the current value function $V_{\phi_{k}}$.
Update the policy by maximizing the PPO-Clip objective:
0.65$\theta_{k+1}=\underset{\theta}{\mathrm{argmax}}\
\frac{1}{|\mathcal{D}_{k}|T}\sum\limits_{{\scriptscriptstyle\tau\in\mathcal{D}_{k}}}\sum\limits_{\scriptscriptstyle
t=0}^{\scriptscriptstyle
T}\min\left[\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{k}}(a_{t}|s_{t})}A^{\pi_{\theta_{k}}}(s_{t},a_{t}),g\left\\{\epsilon,A^{\pi_{\theta_{k}}}(s_{t},a_{t})\right\\}\right]$,
via stochastic gradient ascent. Fit value function by regression on MSE:

0.66
$\phi_{k+1}=\underset{\phi}{\mathrm{argmin}}\frac{1}{|\mathcal{D}_{k}|T}\sum\limits_{{\scriptscriptstyle\tau\in\mathcal{D}_{k}}}\sum\limits_{\scriptscriptstyle
t=0}^{\scriptscriptstyle
T}{\left\\{V_{\phi}(s_{t})-\hat{R}_{t}\right\\}}^{2},$

via gradient descent algorithm.

###  II-C Simulation and real-world platform

In this study, we employ the Duckietown platform to train and assess the
performance of our DRL agent. The platform as shown in Fig. 1 offers both
simulation and real-world environments that enable researchers to train agents
to execute diverse driving behaviors[34]. The vehicle on this platform is
equipped with a wide-angle monocular camera facing forward, which captures
environmental information. To train and test autonomous driving agents using
different methods, the Gym-Duckietown simulation is provided [35]. It is
developed using Python/OpenGL (Pyglet) and offers various customization
options through different wrappers, such as one for the Robot Operating System
(ROS), which facilitate the transition to the real robot. In the simulator,
the agent is placed inside a Duckietown, which consists of a loop of roads
with turns, intersections, obstacles, traffic lights, and other Duckiebots.

To evaluate the performance of the trained agent in the real world, we use the
real-world Duckiebot shown in Fig. 3, which is equipped with a Jetson Nano 2GB
computation unit responsible for image acquisition, processing, and vehicle
control. Notably, the images obtained from the camera in the real-world
environment may have visible differences compared to those obtained from the
simulator, owing to variations in lighting conditions and camera angles. This
presents a significant challenge when translating the trained agent into a
real robot, which we have addressed in this study.

[width=.45]duckiebot_front.pdf |  [width=.45]duckiebot_back.pdf
---|---
(a) | (b)
Figure 2: A figure caption

[scale=0.45]duckiebot.pdf

Figure 3: Robot car used during the real-world evaluation. (a) Side view of
the robot car which equipped with front-view camera and Jetson Nano 2GB. (b)
Back view of the car with a pattern of circles.

##  III Multi-task driving model with DRL

###  III-A Problem formulation

Achieving fully autonomous driving requires addressing fundamental driving
tasks such as car following, lane keeping, and overtaking. In this study, we
focus specifically on lane keeping and car following tasks. Our autonomous
agent is designed to use only camera images to detect lanes and sense the
vehicle in front. When there is no vehicle ahead on the road, the agent must
execute robust lane keeping behavior. When a slower-moving vehicle blocks its
path, the agent must safely follow the vehicle while maintaining a safe
distance and driving efficiency. To achieve a successful transfer of learned
behavior from the simulated environment to the evaluation simulator and the
real-world environment, the agent must be capable of accounting for any
discrepancies between the simulation and the real-world scenarios. Moreover,
the trained agent must consider the latency of image transfer and processing
and control the vehicle drive properly with adequate tolerance.

###  III-B Perception module for image processing

Since the DRL agent only depends on images from the camera, the perception
module is responsible for extracting task-related affordances. As shown in
Fig. 1, for our tasks, the perception module needs to detect the lanes on the
road and the pattern on the back of the leading vehicle.

To enable lane detection, we follow a multi-step image processing pipeline
[34]. This pipeline estimates two important parameters: the lateral
displacement $d_{\delta}$ to the right lane center and the angle offset
$\theta_{\delta}$ relative to the center axis. These estimates are later used
in the control module to facilitate lane keeping. For the vision-based car
following task, we attach a pattern of circles to the back of the leading
vehicle as shown in Fig. 3. To detect the circles, we use the
_findCirclesGrid_ function from OpenCV [36], which identifies the centers of
the circle grid. Next, we compute the Euclidean distance $d_{s}$ between two
circles to determine the distance between the two vehicles. As $d_{s}$ changes
according to the distance between two vehicles, the change of $d_{s}$ is used
to indicate the time-to-collision between the two vehicles [21].

###  III-C Action and observation space

This section covers the fundamental setup of our DRL agent. The action space
of our autonomous vehicle is defined by the speed command $v_{c,t}$ and
steering angle $\phi_{t}$, which follows the normal practice for such setups.
To enable our DRL agent to perform lane keeping and car following maneuvers,
we need to provide observations that support these tasks. As discussed before,
the agent is divided into two distinct modules: the perception module and the
control module to enhance the generalization. The perception module discussed
in Section III-B is responsible for processing images captured by the front-
view camera of the vehicle and extracting relevant environmental attributes
required for driving.

For lane keeping task, the lateral deviation $d_{\delta}$ and orientation
deviation $\theta_{\delta}$ processed from perception module are used. For
vision-based car following task, to sense the velocity difference and the
distance to leading vehicle, the distances between two points on the back of
leading vehicle of current and last timestep are used. The overall input state
$s_{t}$ for the DRL agent at time step $t$ is thus defined as

$s_{t}={\left(\frac{d_{\delta,t}}{d_{w}},\frac{\theta_{\delta,t}}{\pi},v_{t},d_{s,t},d_{s,t-1},a_{v,t-1},a_{\phi,t-1}\right)}^{\top},\notag$
(1)

where $d_{w}$ indicates the lane width, $v_{t}$ denotes the velocity of the
ego vehicle, $d_{s,t}$ and $d_{s,t-1}$ are the distances between two points on
the back of leading vehicle of current and last timestep. Additionally, action
output from control module $a_{v,t-1}$ and $a_{\phi,t-1}$ of last timestep are
used to assist the agent to perform more smooth control. Here we use the work
from [32] as a source of reference for lane keeping task, but this work is
mainly focused on the combination of car following and lane keeping task, as
well as its evaluation.

###  III-D Reward function

The reward function is the cornerstone of any RL algorithm, as it guides the
behavior of the trained agent. To achieve the desired outcome, it is crucial
to consider the lane keeping and car following capabilities of the agent. To
address this, we design a reward function with two primary sub-rewards: one
for lane keeping and another for car following. To ensure the ability of the
agent to accurately follow the lane, we utilize two key metrics: the lateral
deviation and orientation deviation relative to the right lane center. These
measures allow us to closely monitor the position and orientation of the
vehicle, and ensure that it stays within the designated lane with only a small
margin of error. To achieve optimal car following behavior, we take into
account two key factors: safety and driving efficiency. By carefully balancing
these elements, we can ensure that the agent maintains a safe following
distance while also driving efficiently.

####  III-D1 Safety

Ensuring safety is the top priority, the agent needs to maintain a safe
distance from the vehicle in front to avoid any potential collisions. To
support this, we use a metric known as time-to-collision (TTC), which allows
us to penalize any unsafe driving behavior. TTC is a widely used safety factor
that indicates the amount of time left before two vehicles collide, based on
their current velocities:

$\text{TTC}(t)=\frac{d_{n-1,n}(t)}{\Delta v_{n-1,n}(t)},\notag$ (2)

where $d_{n-1,n}(t)$ is the bumper-to-bumper diastance between leader $n-1$
and follower $n$, $\Delta v_{n-1,n}(t)$ represent the speed difference between
two vehicles. To incorporate TTC as safety subreward, a lower bound value of
TTC should be determined. Considering that the vehicle used in our simulation
and real-world is miniature model, thus we choose the TTC lower bound as 1.5s.
Then we construct the safety subreward $R_{safe}$ as:

$R_{safe}(\text{TTC})=\cases{\log}(\frac{\text{TTC}}{1.5}),&0\leq\text{TTC}\leq
1.5s,\\\ 0,\text{otherwise}.\notag$ (3)

In this way, if the TTC is less than 1.5 seconds, the agent will receive a
penalty, and as the TTC approaches zero, the reward for safety $R_{safe}$ will
become increasingly negative, potentially approaching negative infinity.

####  III-D2 Driving efficiency

To quantity measure the driving efficiency, we employ time headway, which
represents the time for the ego vehicle to reach the back of the leading
vehicle.

$h(t)=\frac{d_{n-1,n}(t)+l_{n-1}}{v_{n}(t)},\notag$ (4)

where $l_{n-1}$ is the length of leading vehicle $n-1$ and $v_{n}(t)$ is the
speed of ego vehicle. To utilize time headway in the reward function, we
construct it with log-normal distribution
$H\sim\text{Lognormal}(\mu,\delta^{2})$. Next, the driving efficiency reward
component $R_{eff}$ is showed with probability density function (pdf) for time
headway:

$R_{eff}(h)=\frac{1}{\sqrt{2\pi}h\delta}\exp\left\\{-\frac{(\ln(h)-\mu)^{2}}{2\delta^{2}}\right\\}.\notag$
(5)

We choose $\mu=0.4226$ and $\delta=0.4365$ based on empirical data [37]. The
reward for time headway is maximized when it is close to 1.26s, and a smaller
or larger headway results in a smaller reward value.

####  III-D3 Lateral and orientation deviation

For Lane keeping ability of the agent, we should consider the lateral and
orientation deviation relative to the right lane center. Therefore, we
incorporate a penalty for the lateral deviation by measuring the cross-track
error ($e_{y}(t)$) towards the target path with the following equation:

$R_{l,\delta}(t)=a^{k_{r_{c}}e_{y}(t)}-1,\notag$ (6)

where $a$ and $k_{r_{c}}$ are tunable parameters that controls sensitivity. By
utilizing this equation, the agent will not be punished when the cross-track
error is zero. As the value of $e_{y}(t)$ increases, the reward will decrease,
approaching infinity.

For orientation deviation penalty, the agent uses the angle offset
$\chi_{yaw,t}$ between the current heading and the desired heading:

$R_{\phi,\delta}(t)=\chi_{yaw,t}.\notag$ (7)

Therefore, the overall reward function is defined as the sum of these reward
components. As the weights measure the trade-off of different components, we
conduct extensive experiments to test different weight combinations. Through
this rigorous testing process, we identify and select the set of weights that
produces the best overall performance.

$R=w_{s}R_{safe}+w_{e}R_{eff}+w_{l,\delta}R_{l,\delta}+w_{\phi,\delta}R_{\phi,\delta},\notag$
(8)

where $w_{s}$ and $w_{e}$ are set as 0.3 and 0.2 respectively. Since the agent
only incurs penalties for lane keeping tasks, we choose smaller values for
these weights to avoid large negative rewards. In contrast, car following
weights $w_{l,\delta}$ and $w_{\phi,\delta}$ are set to 0.8 and 0.5,
respectively.

##  IV Experiments and results

###  IV-A Training process

To train multi-task agents for autonomous driving, we need to create a leader-
follower pair. Therefore, we use the Ornstein-Uhlenbeck process to generate
random leader trajectories [38], which helps the agent learn to follow a
variety of driving patterns and improves their ability to generalize to new
situations. Fig. 4 shows an example leader velocity trajectory generated using
the Ornstein-Uhlenbeck (OU) process. Before each episode of the training
process, random velocity trajectories will be generated and control the
leading robot to run with the desired velocity. For steering angle control,
the leader uses the proportional–derivative (PD) controller baseline provided
by Gym-Duckietown [35]. Therefore, the leader can perform lane keeping
behavior with desired random velocity. The DRL agent must effectively follow
the leading vehicle by maintaining a safe distance and suitable velocity, all
while executing proper lane keeping behavior. The initial bumper-to-bumper gap
between two vehicles is set randomly in the range [0m, 2.0m].

[width=]leader_speed.pdf

Figure 4: An example velocity trajectory for the leading vehicle generated by
the Ornstein-Uhlenbeck process for the training process.

The training process utilizes an NVIDIA GeForce RTX 3080 GPU and typically
requires approximately seven hours to complete one million steps of
interaction with the simulated environment. The training is initiated with ten
varying seeds and executed for one million timesteps on multiple occasions.
Fig. 5 depicts the successful convergence of the PPO agent during the training
process, leading to a stable and consistent performance at the end of the
training.

[width=]reward_plot_2.pdf

Figure 5: Training results of PPO agent with ten independent seeds over one
million steps.

###  IV-B Evaluation in simulation

Next, we evaluate our model in a simulation using the Gym-Duckietown
environment. Since there are no existing benchmarks for vision-based lane
keeping and car following models, we compare our approach to a combination of
IDM and PD controllers. The IDM controller uses information such as speed
difference to the leading vehicle and bumper-to-bumper gap to regulate the
speed of the ego vehicle and achieve car following behavior. Meanwhile, the PD
controller provided by Gym-Duckietown controls the steering angle to perform
lane keeping maneuvers. Several experiments are conducted to determine the
best parameters for the baseline agent, which are presented in Table I.

TABLE I: Parameters used for baseline controller in the evaluation. Symbol | Description | Value
---|---|---
$v_{des}$ | Desired velocity |  1.0m/s
$T$ | Safe time gap |  1s
$a$ | Maximum acceleration |  1m/s^2
$b_{comf}$ | Comfortable deceleration |  1m/s^2
$g_{min}$ | Minimum gap |  0.2m
$K_{p}$ | Proportional gain | 2.0
$K_{d}$ | Derivative gain | 5.0
|  |

First, we assess the car following performance for both the DRL agent and the
baseline agent using distinct sets of random leader velocity trajectories,
which are generated via the OU process explained during the training phase.
Fig. 6 depicts the results of the evaluation, where both agents follow the
leader with randomly generated velocities. As shown in Fig. 6, the DRL agent
and IDM controller are following close to the leading vehicle, with IDM
showing a more aggressive driving behavior and maintaining smaller distances.
For a more detailed analysis, we compare the TTC and time headway of both
followers, where the TTC is less than 10 seconds and the time headway is less
than 5 seconds. As depicted in Fig. 7 and Table IV-B, despite the baseline
controller has access to all the necessary information regarding the car
following and lane keeping maneuvers, the vision-based DRL agent still achieve
similar performance. We consider a TTC below 0.5 seconds as a safety-critical
situation for the miniature vehicle. DRL agent has a minimal value of 0.56
seconds and for baseline controller is 0.53 seconds. Two followers are both
performing safe car following behaviors. From Fig. 7, we also observe that the
time headway of the DRL agent is mainly distributed around 1.2 seconds, which
is the optimal value according to the setup of the reward function discussed
in Section III-D2.

During the validation process, the lane keeping performance for both agents is
also assessed using four metrics: survival time ($T_{s}$), lateral deviation
($\delta_{l}$) from the center of the right lane, orientation deviation
($\delta_{\phi}$) from the tangent of the right lane center, and major
infractions ($i_{m}$) which measure the amount of time the vehicle spends
outside of the drivable zones. The survival time ($T_{s}$) metric indicates
the length of time until the simulation is terminated, which occurs when the
vehicle drives outside of the road. The lateral deviation ($\delta_{l}$) and
orientation deviation ($\delta_{\phi}$) are both integrated over time to
assess the overall performance. The major infractions ($i_{m}$) metric
measures the amount of time the vehicle spends outside of the drivable zones.
All four metrics are computed as the median value of 10 episodes, each lasting
for 50 seconds. Table IV-B shows that the DRL agent achieved a similar level
of performance compared to the baseline agent, which was controlled by a PD
controller and had full access to the true information of the environment.
Remarkably, the DRL agent performed well despite not having access to the true
information of the environment. Neither controller had any major infractions,
indicating that both were able to maintain good lane-keeping ability and avoid
crossing the middle line while following the leading vehicle. Moreover, the
DRL agent achieved a smaller lateral deviation during evaluation.

[scale=0.48]comparision_random.pdf

Figure 6: Example trajectories of DRL agent (red) and baseline controller
(green) following random leader trajectory (orange) during the evaluation.

[scale=0.41]ttc_and_headway_random.pdf

Figure 7: Distribution of TTC and time headway for DRL agent (red) and
baseline controller (green). TABLE II: Evaluation results of two agents
following random generated leading vehicle trajectories.

!  Performance metrics DRL agent Baseline controller 4*TTC [s] Min. 0.56 0.53
Mean 5.13 5.90 Med. 4.97 6.23 Std. 2.30 2.58 4[0pt][l]Time headway [s] Min.
0.58 0.49 Mean 1.23 1.61 Med. 1.21 1.42 Std. 0.23 0.72 Survival Time ($T_{s}$)
[s] 50 50 Lateral deviation ($\delta_{l}$) [m$\cdot$s] 1.00 1.18 Orient.
deviation ($\delta_{\phi}$) [rad$\cdot$s] 3.03 1.43 Major infractions
($i_{m}$) [s] 0.0 0.0

For a more comprehensive evaluation, a leading vehicle speed profile is
designed that takes into account safety-critical driving situations, which do
not appear in normal driving scenarios. Fig. 8 illustrates the vehicle
trajectories of the DRL agent and baseline agent following the leading vehicle
with the self-defined speed profile. As the leader starts with normal
acceleration and reaches the velocity of 0.7m/s, both the DRL agent and
baseline controller follow this change. The IDM controller is more aggressive
and follows the leader closer. However, the DRL agent, which obtained images
to follow the leader, exhibits a more passive driving style and maintained a
greater distance from the leader. From $T=\SI{13}{s}$, the leader performs
non-safety-critical braking until it stops. The DRL agent and baseline
controller both decelerate properly and keep suitable distances from the
leader. Later, from $T=\SI{33}{s}$, the leader performs several safety-
critical accelerating and braking maneuvers. We observe that both controllers
handle these situations well and maintained safe distances from the leader.

[scale=0.42]comparision_self_defined.pdf

Figure 8: Example trajectories of DRL agent (red) and baseline controller
(green) following a self defined external leader trajectory (orange) during
the evaluation.

The TTC and time headway of both followers at each time step are displayed in
Fig. 9. According to Table IV-B, the DRL agent has a lower TTC bound of 0.48
seconds, which is still within the tolerance level, despite the unusual
braking behavior of the leading vehicle during the evaluation. In contrast,
the IDM agent has a minimal value of 0.38 seconds, which falls under the
category of serious conditions. Moreover, the TTC distributions of both
followers are mainly concentrated around 3s with similar variance considering
TTC under 10s. Regarding the time headway, both followers exhibit similar
performance, with the values of DRL mainly distributed around 1.2 seconds.

Considering the lane keeping performance under this safety-critical
circumstance, Table IV-B indicates that the DRL agent and baseline controller
are still able to handle the lane keeping maneuver while maintaining suitable
car following capabilities. The lateral deviation of the DRL agent is smaller
than the baseline controller and both agents have zero major infractions.

[scale=0.41]ttc_and_headway_self_defined.pdf

Figure 9: Distribution of TTC and time headway for DRL agent (red) and
baseline controller (green).

TABLE III: Evaluation results of two agents following self-defined leading
vehicle trajectories.

!  Performance metrics DRL agent Baseline controller 4*TTC [s] Min. 0.48 0.38
Mean 3.25 3.39 Med. 2.54 2.32 Std. 2.28 2.26 4[0pt][l]Time headway [s] Min.
0.62 0.54 Mean 1.40 1.12 Med. 1.27 0.91 Std. 0.55 0.63 Survival Time ($T_{s}$)
[s] 50 50 Lateral deviation ($\delta_{l}$) [m$\cdot$s] 1.09 1.12 Orient.
deviation ($\delta_{\phi}$) [rad$\cdot$s] 3.51 0.96 Major infractions
($i_{m}$) [s] 0.0 0.0

###  IV-C Evaluation in real-world environment

For the Sim2Real transformation, the real-world scenario is used to assess the
lane keeping and car following abilities of the trained DRL agent in the real
world. However, due to the unavailability of accurate information about other
vehicles, IDM cannot be applied in the real world. Consequently, our
evaluation focuses solely on demonstrating the car following ability of the
DRL agent. During the evaluation process, we utilize a PID baseline to control
one leading vehicle, which demonstrates lane keeping behavior along the track.
Simultaneously, the ego vehicle, controlled by the DRL agent, aims to perform
both car following and lane keeping tasks. The DRL agent effectively guides
the ego vehicle to follow the leader at an appropriate speed. When the leading
vehicle comes to a stop, the ego vehicle also halts after it, maintaining a
suitable distance. The evaluation videos are available on the project website.

##  V CONCLUSIONS

This study proposes a vision-based DRL agent that can simultaneously perform
lane keeping and car following tasks. The overall system is divided into two
modules: the perception module and the control module. The perception module
extracts task-relevant attributes of the surroundings, while the control
module is a DRL agent that takes these attributes as input. To evaluate the
performance of the DRL agent, we compare it with a baseline algorithm in both
simulation and real-world environments. In the simulation, we compare the car
following and lane keeping capabilities of the DRL agent and baseline
controller using various performance metrics. In the real-world environment,
we demonstrate that the DRL agent can follow the leading vehicle while
maintaining lane keeping ability. In future work, we plan to enhance our DRL
agent by incorporating a comfort factor to address unstable driving behavior.
Additionally, we aim to deploy more advanced algorithms for improved
generalization.

## ACKNOWLEDGMENT

This work was funded by Center for Scalable Data Analytics and Artificial
Intelligence (ScaDS.AI) Dresden/Leipzig, Germany. The authors would like to
thank Fabian Hart, Martin Waltz, Niklas Paulig, and Ankit Chaudhari for
constructive criticism of the manuscript, and also thank Martin Treiber, for
providing the IDM parameters for evaluation.

## References

  * [1] J. Kober, J. A. Bagnell, and J. Peters, “Reinforcement learning in robotics: A survey,” _The International Journal of Robotics Research_ , vol. 32, no. 11, pp. 1238–1274, 2013.
  * [2] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray _et al._ , “Learning dexterous in-hand manipulation,” _The International Journal of Robotics Research_ , vol. 39, no. 1, pp. 3–20, 2020.
  * [3] S. Gu, E. Holly, T. Lillicrap, and S. Levine, “Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates,” in _2017 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2017, pp. 3389–3396.
  * [4] T. Haarnoja, H. Tang, P. Abbeel, and S. Levine, “Reinforcement learning with deep energy-based policies,” in _International conference on machine learning_. PMLR, 2017, pp. 1352–1361.
  * [5] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton _et al._ , “Mastering the game of go without human knowledge,” _nature_ , vol. 550, no. 7676, pp. 354–359, 2017.
  * [6] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-level control through deep reinforcement learning,” _nature_ , vol. 518, no. 7540, pp. 529–533, 2015.
  * [7] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot _et al._ , “Mastering the game of go with deep neural networks and tree search,” _nature_ , vol. 529, no. 7587, pp. 484–489, 2016.
  * [8] M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver, “Rainbow: Combining improvements in deep reinforcement learning,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 32, no. 1, 2018.
  * [9] P. R. Wurman, S. Barrett, K. Kawamoto, J. MacGlashan, K. Subramanian, T. J. Walsh, R. Capobianco, A. Devlic, F. Eckert, F. Fuchs _et al._ , “Outracing champion gran turismo drivers with deep reinforcement learning,” _Nature_ , vol. 602, no. 7896, pp. 223–228, 2022.
  * [10] Z. Jiang, D. Xu, and J. Liang, “A deep reinforcement learning framework for the financial portfolio management problem,” _arXiv preprint arXiv:1706.10059_ , 2017.
  * [11] H. Yang, X.-Y. Liu, S. Zhong, and A. Walid, “Deep reinforcement learning for automated stock trading: An ensemble strategy,” in _Proceedings of the first ACM international conference on AI in finance_ , 2020, pp. 1–8.
  * [12] I. V. Serban, C. Sankar, M. Germain, S. Zhang, Z. Lin, S. Subramanian, T. Kim, M. Pieper, S. Chandar, N. R. Ke _et al._ , “A deep reinforcement learning chatbot,” _arXiv preprint arXiv:1709.02349_ , 2017.
  * [13] W. Zhang, Y. Feng, F. Meng, D. You, and Q. Liu, “Bridging the gap between training and inference for neural machine translation,” _arXiv preprint arXiv:1906.02448_ , 2019.
  * [14] R. Wiedemann, _Simulation des Straßenverkehrsflusses_. Vol. 8 of Schriftenreihe des IfV: Institut für Verkehrswesen, University of Karlsruhe, Germany, 1974.
  * [15] M. Treiber, A. Hennecke, and D. Helbing, “Congested traffic states in empirical observations and microscopic simulations,” _Physical review E_ , vol. 62, no. 2, p. 1805, 2000.
  * [16] Y. Lin, J. McPhee, and N. L. Azad, “Longitudinal dynamic versus kinematic models for car-following control using deep reinforcement learning,” in _2019 IEEE Intelligent Transportation Systems Conference (ITSC)_. IEEE, 2019, pp. 1504–1510.
  * [17] Y.-T. Yen, J.-J. Chou, C.-S. Shih, C.-W. Chen, and P.-K. Tsung, “Proactive car-following using deep-reinforcement learning,” in _2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2020, pp. 1–6.
  * [18] H. Naing, W. Cai, T. Wu, and L. Yu, “Dynamic car-following model calibration with deep reinforcement learning,” in _2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2022, pp. 959–966.
  * [19] M. Zhu, X. Wang, and Y. Wang, “Human-like autonomous car-following model with deep reinforcement learning,” _Transportation research part C: emerging technologies_ , vol. 97, pp. 348–368, 2018.
  * [20] D. Li and O. Okhrin, “Modified ddpg car-following model with a real-world human driving experience with carla simulator,” _Transportation Research Part C: Emerging Technologies_ , vol. 147, p. 103987, 2023.
  * [21] M. Schwarzinger, T. Zielke, D. Noll, M. Brauckmann, and W. Von Seelen, “Vision-based car-following: detection, tracking, and identification,” in _Proceedings of the Intelligent Vehicles92 Symposium_. IEEE, 1992, pp. 24–29.
  * [22] M. Zayed, J. Boonaert, and M. Bayart, “License plate tracking for car following with a single camera,” in _Proceedings. The 7th International IEEE Conference on Intelligent Transportation Systems (IEEE Cat. No. 04TH8749)_. IEEE, 2004, pp. 719–724.
  * [23] A. Amditis, M. Bimpas, G. Thomaidis, M. Tsogas, M. Netto, S. Mammar, A. Beutner, N. Möhler, T. Wirthgen, S. Zipser _et al._ , “A situation-adaptive lane-keeping support system: Overview of the safelane approach,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 11, no. 3, pp. 617–629, 2010.
  * [24] A. E. Sallab, M. Abdou, E. Perot, and S. Yogamani, “End-to-end deep reinforcement learning for lane keeping assist,” _arXiv preprint arXiv:1612.04340_ , 2016.
  * [25] N. C. Basjaruddin, D. Saefudin, S. A. Aryani _et al._ , “Lane keeping assist system based on fuzzy logic,” in _2015 International Electronics Symposium (IES)_. IEEE, 2015, pp. 110–113.
  * [26] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in _2017 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2017, pp. 23–30.
  * [27] R. Traoré, H. Caselles-Dupré, T. Lesort, T. Sun, N. Díaz-Rodríguez, and D. Filliat, “Continual reinforcement learning deployed in real-life using policy distillation and sim2real transfer,” _arXiv preprint arXiv:1906.04452_ , 2019.
  * [28] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _arXiv preprint arXiv:1503.02531_ , 2015.
  * [29] C. Hu, S. Hudson, M. Ethier, M. Al-Sharman, D. Rayside, and W. Melek, “Sim-to-real domain adaptation for lane detection and classification in autonomous driving,” in _2022 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2022, pp. 457–463.
  * [30] L. Reiher, B. Lampe, and L. Eckstein, “A sim2real deep learning approach for the transformation of images from multiple vehicle-mounted cameras to a semantically segmented image in bird’s eye view,” in _2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2020, pp. 1–7.
  * [31] M. Müller, A. Dosovitskiy, B. Ghanem, and V. Koltun, “Driving policy transfer via modularity and abstraction,” _arXiv preprint arXiv:1804.09364_ , 2018.
  * [32] D. Li and O. Okhrin, “A platform-agnostic deep reinforcement learning framework for effective sim2real transfer in autonomous driving,” _arXiv preprint arXiv:2304.08235_ , 2023.
  * [33] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [34] L. Paull, J. Tani, H. Ahn, J. Alonso-Mora, L. Carlone, M. Cap, Y. F. Chen, C. Choi, J. Dusek, Y. Fang _et al._ , “Duckietown: an open, inexpensive and flexible platform for autonomy education and research,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2017, pp. 1497–1504.
  * [35] M. Chevalier-Boisvert, F. Golemo, Y. Cao, B. Mehta, and L. Paull, “Duckietown environments for openai gym,” https://github.com/duckietown/gym-duckietown, 2018.
  * [36] G. Bradski, “The OpenCV Library,” _Dr. Dobb’s Journal of Software Tools_ , 2000.
  * [37] K. Vogel, “A comparison of headway and time to collision as safety indicators,” _Accident analysis & prevention_, vol. 35, no. 3, pp. 427–433, 2003.
  * [38] L. S. Ornstein and G. E. Uhlenbeck, “On the theory of the brownian motion,” _Physical review_ , vol. 36, no. 5, p. 823, 1930.