# Crystallographic orientation and induced potential effects in photoelectron
emission from metal surfaces by ultrashort laser pulses

C. A. Ríos Rubiano and R. Della Picca  CONICET and Centro Atómico Bariloche
(CNEA) Bariloche, Argentina.  D. M. Mitnik  Instituto de Astronomía y Física
del Espacio (IAFE, CONICET-UBA), Casilla de correo 67, sucursal 28, C1428EGA
Buenos Aires, Argentina.  Fac. de Ciencias Exactas y Naturales, Universidad de
Buenos Aires.  V. M. Silkin  Donostia International Physics Center (DIPC),
20018 San Sebastián, Spain, and Depto. de Física de Materiales, Facultad de
Ciencias Químicas, Universidad del País Vasco, Apdo. 1072, 20080 San
Sebastián, Spain  IKERBASQUE, Basque Foundation for Science, 48011 Bilbao,
Spain  M. S. Gravielle  Instituto de Astronomía y Física del Espacio (IAFE,
CONICET-UBA), Casilla de correo 67, sucursal 28, C1428EGA Buenos Aires,
Argentina.

###### Abstract

The influence of the crystallographic orientation of a typical metal surface,
like aluminum, on electron emission spectra produced by grazing incidence of
ultrashort laser pulses is investigated by using the band-structure-based-
Volkov (BSB-V) approximation. The present version of the BSB-V approach
includes not only a realistic description of the surface interaction,
accounting for band structure effects, but also effects due to the induced
potential that originates from the collective response of valence-band
electrons to the external electromagnetic field. The model is applied to
evaluate differential electron emission probabilities from the valence band of
Al(100) and Al(111). For both crystallographic orientations, the contribution
of partially occupied surface electronic states and the influence of the
induced potential are separately analyzed as a function of the laser carrier
frequency. We found that the induced potential strongly affects photoelectron
emission distributions, opening a window to scrutinize band structure effects.

##  I Introduction

In the last decade photoelectron emission (PE) from metal surfaces has
received renewed attention as a result of the technological achievement of
lasers with pulse durations of the order of attoseconds, which make it
possible to study the behavior of electrons in condensed matter at their
natural temporal orders Cavalieri07 ; Miaja06 ; Miaja08 ; Saathoff08 ; Miaja09
; Mathias09 ; Neppl12 ; Neppl15 . Such a remarkable experimental progress
needs to be accompanied by intensive theoretical research since the underlying
quantum processes involve complex many-body mechanisms, whose complete
understanding is still far from being achieved Faraggi07 ; Krasovskii11 ;
Lemell15 ; Borisov13 ; Liao14 ; PazourekRew2015 ; Baggesen08 ; Jouin14 . This
paper aims to contribute to the study of PE from metal surfaces due to the
interaction of ultrashort laser pulses with valence-band electrons. In
particular, the article focuses on the influence of the crystallographic
orientation of the surface on electron emission spectra, investigating the
contributions of the surface-band structure and the induced surface potential
for different crystal faces.

To describe the PE process we make use of a time-dependent distorted-wave
method named band-structure-based-Volkov (BSB-V) approximation BSB-V . The
BSB-V approach includes an accurate description of the electron-surface
interaction, given by the band-structure-based (BSB) model chsiss99 , while
the action of the laser field on the emitted electron is represented by means
of the Volkov phase Volkov35 . The BSB model is based on the one-dimensional
pseudopotential by Chulkov et al. chsiss99 ; Chulkov97 , which takes into
account the electronic structure of the surface, replicating the width and
position of the projected bulk energy gap and the surface and first image
electronic states ecpejpc78 ; fibev90 ; justss92 ; neinepl92 ; Bran4 . In this
version of the BSB-V approximation we also incorporate the contribution of the
induced surface potential, which is generated by the dynamic response of the
metal surface to the laser field Faraggi09 . The induced potential is derived
in a consistent way from the unperturbed BSB electronic states by using a
linear response theory Alducin2003 .

The BSB-V approximation, including the dynamic induced contribution, is
applied to evaluate double-differential (energy- and angle- resolved) PE
distributions for two different orientations of aluminum: Al(100) and Al(111).
For both crystal faces, the influence of partially occupied surface electronic
states (SESs) and the induced potential are examined by considering different
parameters of the laser pulse. Our results indicate that the induced potential
plays an important role in PE spectra. It makes visible band structures
signatures in PE spectra for the resonant case wherein the laser frequency
coincides with the surface plasmon frequency, as well as for laser pulses with
high carrier frequencies.

The article is organized as follows. In Sec. II we introduce the extended
version of the BSB-V approximation, which takes into account the effect of the
induced surface potential through a Volkov-type phase. In Sec. III results are
shown and discussed, while our conclusions are summarized in Sec. IV. Atomic
units are used unless otherwise stated.

##  II Theoretical method

Let us consider a finite laser pulse, characterized by a time-dependent
electric field $\mathbf{F}_{L}(t)$, grazingly impinging on a metal surface
$S$. As a consequence of the interaction, a valence-band electron, initially
in the state $\Phi_{i}$, is ejected above the vacuum level, reaching a final
state $\Phi_{f}$. Within the framework of the time-dependent distorted wave
formalism Dewanga1994 , the BSB-V transition amplitude for the electronic
transition $\Phi_{i}\rightarrow\Phi_{f}$ reads BSB-V :

$\mathcal{A}_{if}=-i\int_{-\infty}^{+\infty}\mathrm{d}t\,\left\langle\chi_{f}^{(BSBV)}(\mathbf{r},t)|\mathcal{V}(\mathbf{r},t)|\Phi_{i}(\mathbf{r},t)\right\rangle,$
(1)

where

$\mathcal{V}(\mathbf{r},t)=\mathbf{r}\cdot\mathbf{F}_{L}(t)+V_{I}(\mathbf{r},t)$
(2)

is the perturbative potential at the time $t$ and
$\chi_{f}^{(BSBV)}(\mathbf{r},t)$ is the final BSB-V distorted wave function,
with $\mathbf{r}$ the position vector of the active electron. The first term
of Eq. (2) represents the interaction potential with the laser, expressed in
the length gauge, while the second term, $V_{I}$, denotes the induced surface
potential that is produced by electronic density fluctuations caused by the
external field. The frame of reference is placed at the position of the
crystal border, which is shifted outward with respect to the position of the
topmost atomic layer by half of the interplanar distance, with the
$\mathbf{\hat{z}}$ axis being oriented normal to the surface, pointing towards
the vacuum region.

Within the BSB-V approach, the unperturbed states $\Phi_{i}$ and $\Phi_{f}$
are solutions of the Schrödinger equation associated with the one-dimensional
electron-surface potential $V_{S}(z)$ given by Ref. Chulkov97 , which depends
on $z$, the component of $\mathbf{r}$ perpendicular to the surface plane.
Hence, the states $\Phi_{i}\equiv\Phi_{\mathbf{k}_{is},n_{i}}(\mathbf{r},t)$
and $\Phi_{f}\equiv\Phi_{\mathbf{k}_{fs},n_{f}}(\mathbf{r},t)$ can be
expressed as

$\Phi_{\mathbf{k}_{s},n}(\mathbf{r},t)=\frac{1}{2\pi}\exp\left(i\mathbf{k}_{s}\cdot\mathbf{r}_{s}\right)\phi_{n}(z)e^{-iEt},$
(3)

where $\mathbf{k}_{s}$ ($\mathbf{r_{s}}$) is the component of the electron
momentum (position vector) parallel to the surface plane, $\phi_{n}(z)$ is the
one-dimensional eigenfunction with eigenenergy $\varepsilon_{n}$ derived from
the potential $V_{S}(z)$, and $E=k_{s}^{2}$ $/2+\varepsilon_{n}$ is the total
electron energy.

According to the grazing incidence condition and the translational invariance
of the problem in the plane parallel to the surface, the laser field is
linearly polarized perpendicularly to the surface, that is,
$\mathbf{F}_{L}(t)=F_{L}(t)\mathbf{\hat{z}}$, where the temporal profile of
the pulse reads:

$F_{L}(t)=F_{0}\,\sin{(\omega t+\varphi)}\;\sin^{2}{(\pi\,t/\tau),}$ (4)

for $0<t<\tau$, and vanishes at all other times. In Eq. (4) $F_{0}$ represents
the maximum field strength, $\omega$ is the carrier frequency, $\tau$ is the
pulse duration, and $\varphi$ is the carrier envelope phase, which is defined
as $\varphi=(\pi-\omega\tau)/2$ for symmetric pulses. In this work we consider
laser pulses with a fixed number $N$ of full cycles inside the envelope; then,
the pulse duration is defined as $\tau=NT$, with $T=2\pi/\omega$ the laser
oscillation period.

The induced potential $V_{I}$ is evaluated from a linear response theory based
on the BSB wave functions of Eq. (3) Silkin2010 . Making use of a slab
geometry to derive the one-dimensional wave functions $\phi_{n}(z)$, the
induced field $\mathbf{F}_{I}=-\nabla_{\mathbf{r}}V_{I}(\mathbf{r},t)$ can be
nearly expressed as

$\mathbf{F}_{I}(z,t)=\left\\{\begin{array}[]{cc}F_{I}(t)\
\mathbf{\hat{z}\qquad}&\text{for }-d<z<0,\\\
0&\text{outside,}\end{array}\right.$ (5)

where $d$ is the width of the slab, formed by a sufficiently large number of
atomic layers of the metallic crystal. The function

$F_{I}(t)=-\frac{1}{2\pi}\int_{-\infty}^{\infty}\mathrm{d}\nu\widetilde{F}_{L}(\nu)\,\mathrm{f}_{I}(\nu)\,e^{-i\nu
t}$ (6)

is the induced field inside the metal at the time $t$, with
$\widetilde{F}_{L}(\nu)$ denoting the Fourier transform of $F_{L}(t)$ and
$\mathrm{f}_{I}(\nu)$ being the dynamic response induced by a unitary and
monochromatic electric field of frequency $\nu$.

From Eqs. (2) and (5) it is possible to build
$\chi_{f}^{(BSBV)}(\mathbf{r},t)$ by introducing the distortions of both the
external and the induced fields in the momentum distribution of the final
state $\Phi_{\mathbf{k}_{fs},n_{f}}$, by means of a Volkov-type phase Macri03
; Faraggi09 ; BSB-V . It reads:

$\displaystyle\chi_{f}^{(BSBV)}\left(\mathbf{r},t\right)$ $\displaystyle=$
$\displaystyle\Phi_{\mathbf{k}_{fs},n_{f}}\left(\mathbf{r}-\mathbf{\hat{z}}\
\alpha_{L}\left(t\right),t\right)$ (7) $\displaystyle\times\exp\left[iz\
A_{\text{tot}}\left(z,t\right)-i\beta_{L}(t)\right],$

where the function

$A_{\text{tot}}\left(z,t\right)=\left\\{\begin{array}[]{lc}\,A_{L}\left(t\right)+\,A_{I}\left(t\right)&\text{for
}-d<z<0,\\\ A_{L}\left(t\right)&\text{outside,}\end{array}\right.$

represents the position-dependent total vector potential at the time $t$, with

$A_{\mu}(t)=-\int\nolimits_{+\infty}^{t}\mathrm{d}t^{\prime}\,F_{\mu}(t^{\prime}),\quad\mu=L,I,$
(8)

being the vector potentials, with incoming asymptotic conditions, associated
with the laser ($\mu=L$) and induced ($\mu=I$) fields, respectively. In turn,
the functions

$\alpha_{L}(t)=\int\nolimits_{+\infty}^{t}\mathrm{d}t^{\prime}\,A_{L}(t^{\prime}),$
(9)

and

$\beta_{L}(t)=\frac{1}{2}\int\nolimits_{+\infty}^{t}\mathrm{d}t^{\prime}\,[A_{L}(t^{\prime})]^{2},$
(10)

involved in Eq. (7), are respectively related to the quiver amplitude and the
ponderomotive energy of the laser.

Finally, by replacing Eqs. (2), (3), and (7) in Eq. (1), the BSB-V transition
amplitude, including the induced contribution, reduces to
$\mathcal{A}_{if}=\,\delta(\mathbf{k}_{fs}-\mathbf{k}_{is})\,a_{if}$, where
the Dirac delta function imposes the momentum conservation in the plane
parallel to the surface and

$a_{if}=-i\,\int_{0}^{+\infty}\mathrm{d}t\,R_{if}(t)\,e^{i\left[\Delta\varepsilon
t+\beta_{L}(t)\right]}$ (11)

represents the one-dimensional transition amplitude, with
$\Delta\varepsilon=\varepsilon_{n_{f}}-\varepsilon_{n_{i}}$ being the energy
gained by the electron during the process. The function $R_{if}$ denotes the
form factor given by

$\displaystyle R_{if}(t)$ $\displaystyle=$
$\displaystyle\int\limits_{-\infty}^{+\infty}\mathrm{d}z\,\phi_{n_{f}}^{\ast}\displaystyle\big{(}z-\alpha_{L}(t)\big{)}\,\phi_{n_{i}}(z)\,g_{f}(z)\,\mathcal{V}(z,t)$
(12) $\displaystyle\times\exp\left[-i\,z\,A_{\text{tot}}(z,t)\right],$

where $g_{f}(z)=e^{z\Theta(-z)/\lambda_{f}}$ accounts for the stopping of the
ionized electron inside the material BSB-V , with $\Theta$ being the unitary
Heaviside function and $\lambda_{f}=\lambda\left(E_{f}\right)$ being the
electron-mean-free path as a function of the final electron energy
$E_{f}=k_{fs}^{2}$ $/2+\varepsilon_{n_{f}}$.

Analogous to Ref. BSB-V , the BSB-V differential probability of PE from the
surface valence band can be expressed in terms of the one-dimensional
transition amplitude of Eq. (11) as:

$\frac{\mathrm{d}^{2}P}{\mathrm{d}E_{f}\mathrm{d}\Omega_{f}}=2k_{f}\
\rho(k_{fz})\sum\limits_{n_{i}}\left|a_{if}\right|^{2}\Theta(\widetilde{k}_{n_{i}}-k_{fs}),$
(13)

where $\Omega_{f}$ is the solid angle determined by the final electron
momentum $\mathbf{k}_{f}=\mathbf{k}_{fs}+k_{fz}\mathbf{\hat{z}}$, with
$k_{fz}=\sqrt{2\varepsilon_{n_{f}}}$. The angle $\Omega_{f}$ is defined as $\
\ \Omega_{f}=(\theta_{f},\varphi_{f})$, where $\theta_{f}$ and $\varphi_{f}$
are respectively the polar and azimuthal angles, with $\theta_{f}$ measured
with respect to the surface plane. In Eq. (13), the sum indicates the addition
over all the $\phi_{n_{i}}$ states with energies $\varepsilon_{n_{i}}$ $\leq-
E_{W}$ ($E_{W}$ the function work), $\rho(k_{fz})$ is the density of final
states $\phi_{n_{f}}$ with perpendicular momentum $k_{fz}$, and the factor $2$
takes into account the spin states. The Heaviside function
$\Theta(\widetilde{k}_{n_{i}}-k_{fs})$ comes from the momentum conservation in
the direction parallel to the surface plane, with
$\widetilde{k}_{n_{i}}=\sqrt{-2\left(\varepsilon_{n_{i}}+E_{W}\right)}$.

##  III Results

We apply the BSB-V approximation to simulate PE distributions from the valence
band of Al(100) and Al(111). Since the ejection parallel to the polarization
vector of the laser field is expected to provide the major contribution to the
PE rate BSB-V , in this work we only consider electron emission normal to the
surface plane, i.e., $\theta_{f}=90\ \deg$. The maximum field strength was
chosen as $F_{0}=10^{-3}$ a.u. (intensity $I_{L}=3.52\times 10^{-10}$ W/cm2),
which belongs to the perturbative range, far from the damage threshold of the
material Lemell03  .

The BSB-V differential probability was evaluated from Eq. (13) by varying the
carrier frequency and the duration of the laser pulse. In the calculation, the
BSB wave functions $\phi_{n}(z)$ were numerically derived by expanding them
onto a basis of plane waves, defined as

$\left\\{\exp\left[i2\pi j(z+d/2)/D\right],\ j=-n_{0},..,n_{0}\right\\},$

where $2n_{0}+1$ is the number of basis functions and $D$ is the unit cell
width, which acts as a normalization length. By using such an expansion in Eq.
(12), the form factor $R_{if}(t)$ was reduced to a closed form in terms of the
laser and induced fields, while the numerical integration on time involved in
Eq. (11) was done with a relative error lower than $1\%$. Moreover, taking
into account that the functions $\phi_{n_{f}}$ do not allow to distinguish
electrons emitted inside the solid from those ejected towards the vacuum
region, to evaluate the emission probability we averaged the contributions
from the two different wave functions associated with the same positive energy
$\varepsilon_{n_{f}}$ by considering that ionized electrons emitted to the
vacuum region represent approximately a $50\%$ of the total ionized electrons
from the valence band Faraggi04 ; BSB-V .

The parameters associated with the different orientations of aluminum are the
followings. The Al(100) surface presents a work function $E_{W}=0.161$ a.u.
and an interplanar distance of $3.80$ a.u., while the corresponding BSB wave
functions $\phi_{n}(z)$ were obtained by using a basis of plane waves with
$n_{0}=220$, a unit cell width $D=342.04$ a.u., and a slab width $d=266.00$
a.u. (i.e., $71$ atomic layers). The Al(111) surface is characterized by a
work function $E_{W}=0.156$ a.u. and an interplanar distance of $4.39$ a.u.,
and the $\phi_{n}(z)$ wave functions were evaluated using a plane wave basis
with $n_{0}=170$, $D=394.92$ a.u., and $d=307.16$ a.u. (i.e., $71$ atomic
layers). Both faces of aluminum display the same Fermi energy, $E_{F}=0.41$
a.u., and therefore, the same surface plasmon frequency $\omega_{s}=0.40$
a.u., which characterizes the collective motion of valence-band electrons. The
energy-dependent electron mean free-path $\lambda(E_{f})$ was interpolated
from data corresponding to the aluminum bulk, extracted from Ref. MFPAl .

Figure 1: (Color online) Double differential PE probabilities in the normal
direction (i.e., $\theta_{f}=90\deg$), as a function of the final electron
energy $E_{f}$. Each column corresponds to a different carrier frequency of
the laser pulse: $\omega=0.057$, $0.4$, $0.7$ and $1.5$ a.u. from left to
right columns, respectively. Each row corresponds to a laser pulse with a
different duration, that is, with $N=2$ (upper row), $N=4$ (middle row), and
$N=6$ (bottom row) cycles inside the envelope, respectively. In all panels,
BSB-V results for two aluminium faces are displayed: Al(100), with blue thick
lines, and Al(111), with red thin lines.

First, in order to provide an overall scenery of the influence of the aluminum
crystal face, in Fig. 1 we compare PE distributions from Al(100) and Al(111)
by considering a different number of cycles (rows) \- $N=$ $2$, $4$, and $6$
\- as well as different carrier frequencies (columns) - $\omega=0.057$, $0.4$,
$0.7$, and $1.5$ a.u. For the lowest frequency (i.e., $\omega=0.057$ a.u.),
which corresponds to the experimental value for the Ti:sapphire laser system,
PE spectra from the Al(100) surface are more than one order of magnitude
higher than the ones corresponding to the Al(111) face. But when $\omega$
increases, emission probabilities from both aluminum faces become comparable
in magnitude, departing appreciably each other only in the high energy region
for high carrier frequencies, as observed for $\omega=1.5$ a.u. Noticeably,
for the intermediate frequency $\ \omega=0.7$ a.u. the electron emission
distributions from Al(100) and Al(111) are similar in the whole electron
energy range. For this frequency, the small differences between the spectra
corresponding to the different orientations for $N=2$ disappear almost
completely as $N$ increases.

Concerning the influence of the pulse duration, it is more appreciable for
high and intermediate frequencies. On the one hand, for short pulses, with
only two cycles inside the envelope, PE spectra (upper row of Fig. 1) present
a maximal emission at low electron energies, with smoothly decreasing
intensity as the velocity of ejected electrons augments. Noteworthy, for
carrier frequencies in the range $\omega$ $\gtrsim$ $\omega_{s}$ this low-
energy maximum presents a different structure depending on the crystal face.
While for Al(111) it shows a single peak structure, for the Al(100) face the
maximum displays double-hump features, which are particularly visible for the
frequency $\omega=0.4$ a.u. resonant with the surface plasmon frequency. In
this particular case, the double structure is clearly observed even for long
pulse durations, being related to the contribution of partially occupied SESs,
as it will discussed in Sec. III. A.

On the other hand, when the duration of the pulse increases to include several
cycles inside the envelope, the carrier frequency approximates to the photon
energy and consequently, electron distributions become governed by the
multiphoton mechanism associated with a Keldysh parameter $\gamma=$
$\omega\sqrt{E_{W}}/F_{0}$ greater than the unity. For the higher frequencies
- $\omega=$ $0.4$, $0.7$ and $1.5$ a.u. - the spectra of Fig. 1 corresponding
to $4$\- and $6$-cycle laser pulses display a broad maximum due to the
absorption of one-photon of energy $\omega$, which corresponds to the first of
the above-threshold-ionization (ATI) peaks. This ATI maximum is roughly placed
at $E_{f}\simeq$ $\left\langle E_{i}\right\rangle-U_{p}+\omega$, where
$\left\langle E_{i}\right\rangle$ is the initial energy averaged over all
initial states, with $\left\langle E_{i}\right\rangle\simeq$ $-0.44$ and
$-0.43$ a.u. for Al(100) and Al(111) respectively, and $U_{p}=$
$F_{0}^{2}/(4\omega^{2})$ is the ponderomotive energy, which results
negligible in the present cases. The width of these ATI peaks depends on the
pulse duration, decreasing as $\tau$ increases Faraggi09 , as also observed in
PE from atoms Rodriguez04 . However, in contrast to the atomic case, the
electron emission from metal surfaces presents a lower limit in the width of
the ATI peaks, which is produced by the energy spread of the metal valence
band, characterized by the Fermi energy. This fact causes the absence of ATI
structures in the multiphotonic spectra for $\omega=$ $0.057$ a.u. (left lower
panels of Fig. 1) because the energy difference between consecutive ATI peaks
is much lower than the width of each peak.

From Fig. 1 it is also observed that the multiphotonic spectra for $\omega=$
$1.5$ a.u. (right lower panels) show different structures in the low energy
region, depending on the crystallographic orientation of the surface. The
origin of such crystal face effects can be attributed to two different
contributions: the emission from partially occupied SESs and the induced
potential. These contributions will be separately analyzed in the following
subsections.

###  III.1 Photoelectron emission from SESs

Figure 2: (Color online) Potential $V_{S}(z)$ (lower graph), together with the
square modulus of the corresponding SESs,
$\left|\phi_{{}_{SES}}(z)\right|^{2}$ (upper graph), for Al(100), with blue
thick lines, and for Al(111), with red thin lines.

One of the most remarkable effects of the crystal band structure is the
presence of partially occupied SESs, which display a highly localized electron
density at the edge of the crystal surface, favoring the release of electrons
from the material. Even though for high carrier frequencies, SESs were found
to play a minor role in PE spectra from Al(111) BSB-V , the relative
importance of the SES contribution varies with the crystallographic
orientation and the parameters of the laser pulse. In Fig. 2, we plot the
surface potential $V_{S}(z)$ for Al(100) and Al(111), together with the square
modulus of the corresponding SESs, $\left|\phi_{{}_{SES}}(z)\right|^{2}$, with
eigenenergies $\varepsilon_{{}_{SES}}=-0.263$ a.u. and
$\varepsilon_{{}_{SES}}=-0.32$ a.u., respectively. The average depth of the
potential well, defined as $V_{S0}=E_{F}+E_{W}$, is similar for both
orientations. However, the corrugation of $V_{S}(z)$ for the (100) face is
almost a factor $6$ larger than the one corresponding to the (111)
orientation, producing a stronger localization of the SESs of Al(100) close to
the surface border, as observed in Fig. 2.

Figure 3: (Color online) PE distribution in the normal direction, as a
function of the final electron energy, for a 6-cycle laser pulse with a
carrier frequency $\omega=0.4$ a.u. BSB-V results for (a) Al(100) and (b)
Al(111) are displayed in the upper and lower panels, respectively. In both
panels, the thick solid line corresponds to the emission probability from the
valence band, and the black thin dashed and solid lines, to partial
contributions from the top of the valence band and partially occupied SESs,
respectively.

The marked difference between the SES densities of both aluminum faces at the
crystal border can be traced from the superimposed structures of the PE
spectra corresponding to the resonant frequency $\omega\simeq\omega_{s}=$
$0.4$ a.u. For a $6$\- cycle laser pulse with $\omega=$ $0.4$ a.u., in Fig. 3
we compare differential emission probabilities from Al(100) and Al(111) with
partial values due to emission from SESs as well as from states at the top of
the valence band. Despite the general shape of the spectra of Fig. 3 is not
affected by the crystallographic orientation, a closer examination of the
first ATI peak reveals the presence of different superimposed structures for
each face, as also observed in Fig. 1. For Al(100) the first ATI peak displays
a double-bump structure, with two bulges peaking at $E_{f}\approx 0.14$ a.u.
and $E_{f}\approx 0.24$ a.u. respectively, whereas for Al(111) only one
maximum at $E_{f}\approx 0.24$ a.u. exists. This latter maximum, present for
the two faces, is placed at $E_{f}\simeq$ $-E_{W}+\omega$, corresponding to
one-photon absorption from initial states at the top of the valence band,
whose partial contribution is also displayed in the figure. Instead, the peak
at $E_{f}\sim 0.14$ a.u., only visible in the Al(100) spectrum, is produced by
the absorption of one-photon from partially occupied SESs, being placed at
$E_{f}\simeq$ $\varepsilon_{{}_{SES}}+\omega$. From the comparison of Figs. 3
(a) and (b) we conclude that the presence or absence of SES signatures in the
PE distribution, depending on the crystal face, is associated with the mayor
or minor localization of the electronic density of SESs at the crystal border.
For Al(100) such a localization is more than three times higher than the one
corresponding to Al(111), making its contribution clearly discernible in the
PE spectrum. While for Al(111), the SES contribution results completed
concealed by emission from other initial states, which washes out SES
footprints from the electron distribution. Therefore, PE spectra under
resonant conditions might offer an attractive window to obtain information
about the surface band structure. Besides, in the resonant case the
contribution of the plasmon decay mechanism should be also included, producing
an additional structure just at the electronic energy $E_{f}\simeq$
$\omega_{s}$.

Figure 4: (Color online) Analogous to Fig. 3 for $\omega=1.5$ a.u.

Previous band-structure effects disappear as the carrier frequency departs
from the resonant one. But noteworthy they become again visible for high
frequencies, as shown in Fig. 4. In this figure we consider electron emission
from Al(100) and Al(111) for a laser pulse with $\omega=$ $1.5$ a.u. and $6$
cycles. In this case, different low-energy structures are present in the PE
spectra for the two faces. The low-energy hump due to ejection of slow
electrons from the top of the valence band is again visible for both faces,
while the one associated with emission from SESs is appreciable only for the
(100) face. As discussed above, this is a consequence of the different
corrugation and SES electronic density of the two crystallographic
orientations. The observation of such effects significantly depends on the
action of the induced potential. Furthermore, even though for high frequencies
these low-energy structures are less noticeable than in the resonant case,
they might still provide information about the relative importance of SES
contributions.

###  III.2 Induced potential effects

Figure 5: (Color online) Normalized laser and induced fields, as a function of
the normalized time $t/T$, for a 6-cycle laser pulse impinging on Al(111). The
carrier frequency of the laser pulse is: a) $\omega=0.057$ a.u., b)
$\omega=0.4$ a.u. , c) $\omega=0.7$ a.u., and d) $\omega=1.5$ a.u. In all
panels, black thin line, normalized laser field $F_{L}(t)/F_{0}$; violet thick
line, normalized induced field $F_{I}(t)/F_{0}$.  Figure 6: (Color online) PE
distribution from the valence band of both aluminum faces - Al(100) (upper
row) and Al(111) (lower row) - as a function of the final electron energy. The
impinging 6-cycle laser pulse has a carrier frequency $\omega=0.057$, $0.4$,
$0.7$ and $1.5$ a.u. from left to right columns, respectively. Thick (thin)
solid lines, BSB-V results obtained including (without including) the induced
potential.  Figure 7: (Color online) Analysis of the frequency domain
distribution of partial contributions to PE from the valence band of Al(100)
by a 6-cycle laser pulse with $\omega=1.5$ a.u. (a) Frequency profiles of the
laser and induced fields, as a function of the frequency. The Fourier
transform of the laser field $\tilde{F}_{L}$ (induced field $\tilde{F}_{I}$)
is displayed with black thin (violet thick) solid line. (b) The BSB-V PE
distribution, as a function of the electron energy, is shown with blue thin
line, and the SES contribution with black dot-dashed line. The Fourier
transform contribution, $|\tilde{F}_{tot}(E_{f}+\varepsilon_{{}_{SES}})|^{2}$
(in arb. units), is plotted with violet solid line.

With the aim of understanding how the collective response of valence-band
electrons to the laser field affects the distribution of emitted electrons, in
this subsection we examine the contribution of the induced field to PE
spectra.

For laser pulses with several oscillations, the electronic rearrangement
induced in the metal by the external perturbation strongly depends on the
carrier frequency. To illustrate such a variation, in Fig. 5 we plot the laser
and induced fields for $6$-cycle laser pulses, with different frequencies,
impinging on the Al(111) surface. For frequencies much lower than
$\omega_{s}$, like $\omega=0.057$ a.u., the induced and external fields are
similar in strength but in counterphase, tending to the static limit in which
the total electric field vanishes inside the metal. But when the frequency
augments to reach the resonant value $\omega\simeq\omega_{s}=$ $0.4$ a.u., the
maximum strength of $F_{I}(t)$ is four times larger than $F_{0}$; therefore,
the total field inside the metal becomes dominated by the induced response of
the metal surface. In turn, for $\omega>\omega_{s}$ surface electrons try to
follow the external field, in such a way that the laser and induced fields
oscillate almost with the same phase, but the intensity of the induced field
decreases steeply as the carrier frequency augments.

The meaningful differences among the induced responses for different
frequencies of the external field, observed in Fig. 5, are directly reflected
in their contributions to the electron emission process. In Fig. 6 we compare
BSB-V differential emission probabilities, derived from Eq. (13), with BSB-V
values obtained without including the induced surface interaction, that is, by
fixing $F_{I}(t)=A_{I}(t)=0$, considering the same laser parameters as in Fig.
5. Results for both crystallographic orientations -Al(100) (upper panels) and
Al(111) (lower panels)- are displayed in the figure. From Fig. 5 (a), for
$\omega=0.057$ a.u. the induced response screens the laser field inside the
metal almost completely. Then, in this case the inclusion of the induced
potential causes a marked reduction of the PE yield, this effect being
stronger for Al(111) than for Al(100). Instead, for $\omega=0.4$ a.u. the
laser pulse becomes resonant with $\omega_{s}$, originating a large increase
of the emission probability, greater than two order of magnitude. Notice that
for this resonant frequency, the induced field is not only higher than the
external field but also it persists twice the original pulse duration (see
Fig. 5 (b)), contributing to augment the emission probability after the
external field turned off. For a carrier frequency slightly higher than the
surface plasmon frequency, like $\omega=0.7$ a.u., the induced field goes on
reinforcing the action of the laser pulse, although its intensity is four
times lower than that corresponding to the resonant case, as shown in Fig. 5
(c). Hence, the inclusion of induced potential within the BSB-V approach
produces a moderate increase of the PE probability, being again higher for the
(111) face.

The case of $\omega=1.5$ a.u., shown in the right column of Fig. 6, deserves
further discussion. For such a high frequency, electrons are not able to
follow the quick variation of the external perturbation and consequently, the
maximum strength of $F_{I}(t)$ is more than one order of magnitude lower than
the one of the laser field, as displayed in Fig. 5 (d). As it was expected,
this small induced response does not affect appreciably the main electron
emission, which occurs around the first ATI peak. However, we remarkably found
that the induced potential introduces a pronounced growth of the probability
at low electron energies, just in the region where the double-hump low-energy
structure associated with SES emission appears in the Al(100) case. To
investigate in detail this unforeseen contribution, in Fig. 7 (a) we plot the
decomposition in frequencies of both the laser ($\tilde{F_{L}}(\nu)$) and the
induced field ($\tilde{F_{I}}(\nu)$) for this case. The utility of analyzing
the frequency domain of the fields lies on the fact that the PE spectrum can
be roughly estimated as proportional to the square modulus of the Fourier
transform of the total electric field, that is, $|\tilde{F}_{tot}(\nu)|^{2}$,
evaluated at $\nu=E_{f}+E_{i}$, with
$\tilde{F}_{tot}(\nu)=\tilde{F}_{L}(\nu)+\tilde{F}_{I}(\nu)$ and $E_{i}$
covering the energy range of all initially occupied states. In Fig. 7 (a),
although the Fourier transform of the laser field is several orders of
magnitude higher than the one of the induced field around the carrier
frequency, $\tilde{F_{I}}(\nu)$ retains a peak associated with the resonance
$\nu\cong\omega_{s}$, which largely overpasses the value of
$\tilde{F}_{L}(\omega_{s})$. This resonant peak is found to be the origin of
the low-energy SES contribution of the spectrum for Al(100), as shown in Fig.
7 (b), where we observe that the curve corresponding to
$|\tilde{F}_{tot}(E_{f}+\varepsilon_{{}_{SES}})|^{2}$ (multiplied by an
arbitrary factor) almost coincides with the SES contribution.

##  IV Conclusions

We have studied PE spectra produced by the interaction of ultrashort laser
pulses with the valence band of two different faces of aluminium - Al(100) \-
Al(111) - by using of the BSB-V approximation. In the present version of the
BSB-V approach we have incorporated the contribution of the induced field,
originated from the collective response of surface electrons to the external
perturbation. We found that the induced response of the metal surface strongly
affects electron emission distributions for a wide range of laser frequencies,
including high $\omega$\- values. In the resonant case with the surface
plasmon frequency, the effect of the induced field contributes to make visible
signatures coming from partially occupied SESs of the Al(100) surface, while
for Al(111) these SES structures are completely washed out by emission from
other initially occupied states. Similar features are also observed in the
low-energy region of the spectra for high carrier frequencies of the laser
pulse, for which the influence of the induced potential was expected
negligible. These findings open the way to investigate band structure effects
by varying the parameters of the laser pulse, the crystal orientation, and the
observation region.

###### Acknowledgements.

Financial support from CONICET, UBA, and ANPCyT of Argentina is acknowledged.

## References

  * [1] A. L. Cavalieri, N. Muller, Th Uphues, V. S. Yakovlev, A. Baltuska, B. Horvath, B. Schmidt, L. Blumel, R. Holzwarth, S. Hendel, M. Drescher, U. Kleineberg, P. M. Echenique, R. Kienberger, F. Krausz, and U. Heinzmann.  Attosecond spectroscopy in condensed matter.  Nature, 449(7165):1029–1032, Oct 2007.
  * [2] L. Miaja-Avila, C. Lei, M. Aeschlimann, J. L. Gland, M. M. Murnane, H. C. Kapteyn, and G. Saathoff.  Laser-Assisted Photoelectric Effect from Surfaces.  Phys. Rev. Lett., 97:113604, Sep 2006.
  * [3] L. Miaja-Avila, G. Saathoff, S. Mathias, J. Yin, C. La-o vorakiat, M. Bauer, M. Aeschlimann, M. M. Murnane, and H. C. Kapteyn.  Direct Measurement of Core-Level Relaxation Dynamics on a Surface-Adsorbate System.  Phys. Rev. Lett., 101:046101, Jul 2008.
  * [4] G. Saathoff, L. Miaja-Avila, M. Aeschlimann, M. M. Murnane, and H. C. Kapteyn.  Laser-assisted photoemission from surfaces.  Phys. Rev. A, 77:022903, Feb 2008.
  * [5] L. Miaja-Avila, J. Yin, S. Backus, G. Saathoff, M. Aeschlimann, M. M. Murnane, and H. C. Kapteyn.  Ultrafast studies of electronic processes at surfaces using the laser-assisted photoelectric effect with long-wavelength dressing light.  Phys. Rev. A, 79:030901, Mar 2009.
  * [6] S Mathias, M Wiesenmayer, F Deicke, A Ruffing, L Miaja-Avila, M M Murnane, H C Kapteyn, M Bauer, and M Aeschlimann.  Time and angle resolved photoemission spectroscopy using femtosecond visible and high-harmonic light.  Journal of Physics: Conference Series, 148(1):012042, 2009.
  * [7] S. Neppl, R. Ernstorfer, E. M. Bothschafter, A. L. Cavalieri, D. Menzel, J. V. Barth, F. Krausz, R. Kienberger, and P. Feulner.  Attosecond Time-Resolved Photoemission from Core and Valence States of Magnesium.  Phys. Rev. Lett., 109:087401, Aug 2012.
  * [8] S. Neppl, R. Ernstorfer, A. L. Cavalieri, C. Lemell, G. Wachter, E. Magerl, E. M. Bothschafter, M. Jobst, M. Hofstetter, U. Kleineberg, J. V. Barth, D. Menzel, J. Burgdorfer, P. Feulner, F. Krausz, and R. Kienberger.  Direct observation of electron propagation and dielectric screening on the atomic length scale.  Nature, 517(7534):342–346, Jan 2015.  Letter.
  * [9] M. N. Faraggi, M. S. Gravielle, and D. M. Mitnik.  Interaction of ultrashort laser pulses with metal surfaces: Impulsive jellium-Volkov approximation versus the solution of the time-dependent Schrödinger equation.  Phys. Rev. A, 76:012903, Jul 2007.
  * [10] E. E. Krasovskii.  Attosecond spectroscopy of solids: Streaking phase shift due to lattice scattering.  Phys. Rev. B, 84:195106, Nov 2011.
  * [11] C. Lemell, S. Neppl, G. Wachter, K. Tőkési, R. Ernstorfer, P. Feulner, R. Kienberger, and J. Burgdörfer.  Real-time observation of collective excitations in photoemission.  Phys. Rev. B, 91:241101, Jun 2015.
  * [12] A. G. Borisov, D. Sánchez-Portal, A. K. Kazansky, and P. M. Echenique.  Resonant and nonresonant processes in attosecond streaking from metals.  Phys. Rev. B, 87:121110, Mar 2013.
  * [13] Q. Liao and U. Thumm.  Attosecond Time-Resolved Photoelectron Dispersion and Photoemission Time Delays.  Phys. Rev. Lett., 112:023602, Jan 2014.
  * [14] Renate Pazourek, Stefan Nagele, and Joachim Burgdörfer.  Attosecond chronoscopy of photoemission.  Rev. Mod. Phys., 87:765–802, Aug 2015.
  * [15] J. C. Baggesen and L. B. Madsen.  Theory for time-resolved measurements of laser-induced electron emission from metal surfaces.  Phys. Rev. A, 78:032903, Sep 2008.
  * [16] H. Jouin, M. Raynaud, G. Duchateau, G. Geoffroy, A. Sadou, and P. Martin.  Quantum-classical model for the surface plasmon enhanced photoemission process at metal surfaces.  Phys. Rev. B, 89:195136, May 2014.
  * [17] C. A. Rios Rubiano, M. S. Gravielle, D. M. Mitnik, and V. M. Silkin.  Band-structure effects in photoelectron-emission spectra from metal surfaces.  Phys. Rev. A, 85:043422, Apr 2012.
  * [18] E.V. Chulkov, V.M. Silkin, and P.M. Echenique.  Image potential states on metal surfaces: binding energies and wave functions.  Surface Science, 437(3):330 – 352, 1999.
  * [19] D.M. Wolkow.  über eine klasse von lösungen der diracschen gleichung.  Zeitschrift für Physik, 94(3-4):250–260, 1935.
  * [20] E.V. Chulkov, V.M. Silkin, and P.M. Echenique.  Image potential states on lithium, copper and silver surfaces.  Surface Science, 391(1):L1217 – L1223, 1997.
  * [21] P M Echenique and J B Pendry.  The existence and detection of Rydberg states at surfaces.  Journal of Physics C: Solid State Physics, 11(10):2065, 1978.
  * [22] F Finocchi, C.M. Bertoni, and S Ossicini.  Surface Science Section Simple metal surfaces and image potential states.  Vacuum, 41(1):535 – 537, 1990.
  * [23] Leszek Jurczyszyn and Maria Stȩślicka.  Barrier-resonance states in an external electric field.  Surface Science, 266(1):141 – 144, 1992.
  * [24] M. Nekovee and J. E. Inglesfield.  Threshold Behaviour of Surface Density of States at the Vacuum Level.  EPL (Europhysics Letters), 19(6):535, 1992.
  * [25] P.M. Echenique, R. Berndt, E.V. Chulkov, Th. Fauster, A. Goldmann, and U. Höfer.  Decay of electronic excitations at metal surfaces.  Surface Science Reports, 52(7–8):219 – 317, 2004.
  * [26] Marisa Faraggi, Iñigo Aldazabal, Maria Silvia Gravielle, Andres Arnau, and Vyacheslav M. Silkin.  Study of the induced potential produced by ultrashort pulses on metal surfaces.  J. Opt. Soc. Am. B, 26(12):2331–2336, Dec 2009.
  * [27] M. Alducin, V. M. Silkin, J. I. Juaristi, and E. V. Chulkov.  Energy loss of ions at metal surfaces: Band-structure effects.  Phys. Rev. A, 67:032903, Mar 2003.
  * [28] D.P. Dewangan and J. Eichler.  Charge exchange in energetic ion-atom collisions.  Physics Reports, 247(2–4):59 – 219, 1994.
  * [29] V M Silkin, A K Kazansky, E V Chulkov, and P M Echenique.  Time-dependent screening of a point charge at a metal surface.  Journal of Physics: Condensed Matter, 22(30):304013, 2010.
  * [30] P. A. Macri, J. E. Miraglia, and M. S. Gravielle.  Ionization of hydrogen targets by short laser pulses.  J. Opt. Soc. Am. B, 20(9):1801–1806, Sep 2003.
  * [31] C. Lemell, X.-M. Tong, F. Krausz, and J. Burgdörfer.  Electron Emission from Metal Surfaces by Ultrashort Pulses: Determination of the Carrier-Envelope Phase.  Phys. Rev. Lett., 90:076403, Feb 2003.
  * [32] M. N. Faraggi, M. S. Gravielle, and V. M. Silkin.  Quantum-mechanical model for valence-electron emission from metal surfaces.  Phys. Rev. A, 69:042901, Apr 2004.
  * [33] David R. Penn.  Electron mean-free-path calculations using a model dielectric function.  Phys. Rev. B, 35:482–486, Jan 1987.
  * [34] V. D. Rodríguez, E. Cormier, and R. Gayet.  Ionization by short uv laser pulses: Secondary above-threshold-ionization peaks of the electron spectrum investigated through a modified Coulomb-Volkov approach.  Phys. Rev. A, 69:053402, May 2004.


Institut Polytechnique de Paris<EMAIL_ADDRESS>D. Akola 1
[100]

# Comparative study of multi-person tracking methods

Denis Mbey Akola

###### Abstract

This paper presents a study of two tracking algorithms (SORT and Tracktor++ )
that were ranked first positions on the MOT Challenge leaderboard 111The
MOTChallenge web page: https://motchallenge.net. The purpose of this study is
to discover the techniques used and to provide useful insights about these
algorithms in the tracking pipeline that could improve the performance of MOT
tracking algorithms. To this end, we adopted the popular tracking-by-detection
approach. We trained our own Pedestrian Detection model using the MOT17Det
dataset. We also used a re-identification model trained on the MOT17 dataset
for Tracktor++ to reduce the false re-identification alarms. We then present
experimental results that show that Tracktor++ is a better multi-person
tracking algorithm than SORT. We also performed ablation studies to discover
the contribution of the re-identification(RE-ID) network and motion to the
results of Tracktor++. We finally conclude by providing some recommendations
for future research.

###### keywords:

Object detection, object tracking, data association

###### category:

##  1 Introduction

A large amount of video data is being produced today. This is largely due to
the recent improvement in video technology that makes it much easier for
people to record high-quality videos. Videos contain rich information which
could be very useful when we are able to understand these pieces of
information using various analytical tools. Thus, being able to analyze videos
has a wide range of applications in surveillance, autonomous driving, and
pedestrian tracking. Person tracking in video scenes is still an active
challenge, especially in crowded environments. This is a result of the
occlusions and fast detection that often occur in such settings.

The person-tracking task is divided into two main aspects. First, we have to
be able to detect people accurately in a video frame, and we must be able to
establish some relationships that connect people in one video frame to
another. This involves being able to identify each person and retain their
identities in future video frames. In the detection step, object detection
models are used to detect the objects of interest. In our case, the people in
the videos. The data association is the matching step where persons in video
frames are linked to form a trajectory over time [2].

With the recent advances in object detection, we have robust object detectors
and these perform fairly well, but they still struggle to make good
predictions in crowded, partial, and fully occluded scenes. Data association
or tracking is still a challenge because of false alarms and missing data,
especially in crowded environments. Moreover, data association is a challenge
because of the complex and dynamic movement activity of human beings.

In this paper, we present a comparative study of two tracking algorithms and
provide some recommendations for future research in multiple-object tracking

##  2 Related work

Multiple-object tracking finds applications in many areas ranging from
autonomous driving to surveillance [18]. Despite the wide application domain
of tracking, it is still a challenge that is actively being investigated by
the research community. From a broader point of view, there are basically two
approaches to the multi-object tracking problem. These two are detection-based
tracking and detection-free tracking [15].

In the detection-based tracking approach, objects of interest are first
detected using classical deep-learning object detection models, and the tracks
are subsequently associated with forming trajectories. This approach is
promising as it is best suitable for online-tracking applications like
autonomous driving [12]. The second kind of approach is detection-free
tracking which requires manual initialisation of a fixed number of objects in
the first frame. Detection-free tracking methods then try to localize these
objects in subsequent frames. This method might not work well because the
objects of interest appearances may change drastically over time making it
difficult for the target object to be tracked. Also, little information about
the target object is always available at the beginning of the tracking
process. Mostly the information available as input to the tracking algorithm
is the initial bounding box which distinguishes the object of interest from
the background [25]. Since we are interested in tracking people in videos, for
the sake of terminology, we will proceed to call such a task multi-person
tracking. In the remainder of this section, we provide a brief review of
related work on object detectors for multi-person tracking and some of the
state-of-the-art object tracking methods.

###  2.1 Object detectors for multi-person tracking

As identified in the detection-based tracking approach, object detection
models are used for the object detection task. Object detection involves
scanning and searching for objects of certain classes (e.g. humans, cars, and
buildings) in an image/video frame [18]. Object detection is at the heart of
scene understanding [21]. Thus, the quality of detections from object
detection models affects the results we can obtain in a typical tracking task.
The most common object detection models used in the detection-based tracking
are R-CNN [8], Fast R-CNN [7], Faster R-CNN [19], and Mask R-CNN [9]. These
algorithms are inspired by convolutional neural networks which are well
adapted to working with images.
R-CNN network has three modules [8]. The first module embeds the region
proposal network. This module generates category-independent region proposals.
The second module contains a large convolutional neural network that generates
a fixed-length feature vector from the regions proposed by the region proposal
network. The third module contains a set of class-specific linear Support
Vector Machines (SVMs) that are used to classify input images into classes.
The region proposal network uses a selective search algorithm to propose
approximate object regions in the input image [23]. R-CNN network is quite
computationally expensive because both the region proposal and classification
tasks are performed separately without computation sharing.
Fast R-CNN [7] was engineered to overcome the challenges of R-CNN notable for
the run-time problem of R-CNN. Fast R-CNN takes an image as input and then
processes the whole image with several convolutional (conv) and max pooling
layers to produce a convolutional feature map corresponding to the image. Fast
R-CNN has the region of interest (IOU) pooling layer which extracts fixed-
length feature vectors from the feature map. Each of the feature vectors
generated from the ROI layer are fed into a sequence of fully connected layers
that finally branch out into sibling output layers(the softmax layer and a
layer that contains the bounding box for the predicted class in the softmax
layer). However, Fast R-CNN uses a selective search method to generate the
pooling map from the convolution feature map which slows down its operation
[7] [18].
Faster R-CNN [19] has two modules. The first module is a deep fully
convolutional network layer that proposes regions and the second module is the
Fast R-CNN detector. Faster R-CNN networks adopt the fast R-CNN network and
make improvements to the way the region proposals are generated in Fast R-CNN
to reduce the overall run-time of the Faster R-CNN network. The region
proposal network (RPN) in Faster R-CNN takes an image as input and outputs
rectangular object proposals each with an objectness score. The intuition
behind Faster R-CNN is sharing computations between the Region proposal
network and the R-CNN detector. As a result, a formal assumption is that both
the RPN and R-CNN networks share the same convolutional layers. Anchors are
placed at each convolution. Anchors (spatial windows of different sizes and
different aspect ratios) are placed at different locations in the input
feature map. Faster R-CNN as the name suggests has a high detection speed than
its counterparts [19].
Mask R-CNN [9] is yet another extension of Faster R-CNN for both object
detection and segmentation. The work of [18] provides an extensive review of
object detectors. For this paper, we trained a Faster R-CNN with ResNet-50
[20] and Feature Pyramid Networks (FPN) [14] on the MOT17Det 222MOT17Det:
https://motchallenge.net/data/MOT17Det/ pedestrian detection dataset.

###  2.2 Data association methods

Data association involves computing the similarities between tracklets and
detection boxes and then matching them using various similarity criteria.
Different metrics have been employed to compute these similarities. Likewise,
different strategies have been used to match tracklets and detection boxes. In
this section, we will offer a brief review of some of the commonly used
methods to compute the similarity as well as the various matching strategies
in multi-object tracking. The common similarity metrics used include motion,
appearance, and distance/location.
For instance, in the SORT tracking algorithm [4], the authors used both motion
and location similarity metrics in a simple way. First, they used Kalman
Filter [11] to predict the location of tracks in new frames and used
Intersection over Union (IOU) to compute the similarity score between
detection boxes and predicted boxes. The Hungarian algorithm is used to assign
detections to existing targets. Also, a minimum IOU was imposed to reject
assignments where the detection to target overlaps is less than a certain
threshold.
The authors of CenterTrack [26] designed their tracktor to learn object motion
and achieve robust results in case of large camera motions or low frame rate.
In their approach, their tracking network was able to learn and match
prediction boxes anywhere in the receptive field even if there existed no
overlap between the boxes. For the motion model, they used a sparse optical
flow estimation network which was learned together with the detection network
and does not require dense supervision. A greedy assignment strategy was used
in their matching step. In TransTrack [22], the authors used a joint-detection
and tracking pipeline to achieve detection and tracking in a single stage.
They applied object features from previous frames as queries of current frames
and as a result, introduced a set of object queries for detecting new incoming
objects. Kalman filter was used in their motion step. Also, they included a
Re-identification (Re-ID) model to help them re-identify objects after they go
occluded for a long time using the appearance similarity. For the matching
step, they used the classic Hungarian algorithm and Non-Maximum
Suppression(NMS) merging method inspired by [16].
In Tracktor++ [2], the authors used the bounding box regression of an object
detector to predict the position of an object in the next frame. In this work,
they adopted two motion models to handle bounding box positions in future
frames. In the case of sequences with moving cameras, they applied the camera
motion compensation(CMC) technique by aligning frames through registration
using an enhanced Correlation Coefficient as introduced in [5]. For low frame
sequences, they applied the constant velocity assumptions (CVA) [1] for all
object frames. They also used a Re-ID model based on appearance vectors
generated using a Siamese neural network. For the matching strategy, they used
the Hungarian algorithm.
In DeepSort [4], the authors improved the SORT tracking algorithm by adopting
a Re-ID network to improve the tracking performance of SORT. To this end, they
trained a re-identification CNN network to extract appearance features from
detection boxes. Likewise, they adopted a new matching strategy which first
matches detection boxes to the most recent tracklets, and then to the lost
tracks.

Following this review, we now offer comparative studies on the performance of
SORT and Tracktor++ tracking algorithms in the MOT17 dataset [17] and provide
some insights for the development of future multi-tracking algorithms.

##  3 Object Detection Model

###  3.1 Pedestrain Detection Model

In this comparative work, we adopted Faster R-CNN network with ResNet-50 [20]
and Feature Pyramid Networks(FPN) [14] trained on the MOT17Det dataset
333MOT17Det: https://motchallenge.net/data/MOT17Det/ for pedestrian detection.
The Faster R-CNN model has a three-stage architecture. In the first stage, the
Region Proposal Network (RPN) was used to generate region proposals for each
image. However, the RPN network was adapted by replacing the single-scale
feature map with FPN as discussed in [13] and implemented in [2]. The second
stage is the Fast R-CNN network was used to extract features from the region
proposals generated by the RPN stage. The last layer contains classification
and regression heads. The classification head assigns an objectness score to
each box proposal by evaluating the likelihood of a proposal region showing a
pedestrian. The regression head has the responsibility of refining the
bounding box location tightly around an object. The final set of object
detections was obtained by applying non-maximum suppression to the refined
bounding box proposals.

###  3.2 Model implementation and training

The Pedestrian Detection model was implemented in Pytorch using Faster R-CNN
multi-object detector with Feature Pyramid Network (FPN) with ResNet-50 [20]
as the feature extractor. Moreover, we replaced the Region of Interest (ROI)
with crop and resize pooling layer as in the tracktor++ paper [2]. The model
was trained on a single NVIDIA GeForce GTXTITAN X GPU with 12 GB of memory for
12 hours. The object detector layers parameters were updated with an initial
learning rate of 0.001, batch size of 8, momentum of 0.9, and weight decay of
0.0005. We used the Stochastic Gradient Descent(SGD) [ruder2016overview]
optimizer to update the learning rate if it remained constant for 10 training
epochs. The training loss curve of the model is shown in figure 1. The model
achieved an average precision (AP) score of 0.815 and an average recall(AR) of
0.852.

Figure 1: Training loss curve

Figure 2 shows the bounding boxes of an image that was processed by the object
detection model.

Figure 2: Object detector output

##  4 SORT Tracking

The SORT tracking algorithm was introduced by Alex Bewley et al in 2017 [4].
SORT makes the assumption that the presence of short-term and long-term
occlusions occur less frequently and thus, ignores these edge cases. The SORT
algorithm tried to alleviate the problem of computation complexity and
therefore it does not use a re-identification model in the tracking pipeline
as well. Kalman Filter and Hungarian algorithms were employed in SORT to
handle motion prediction and data-matching respectively. SORT propagates
target identities into next frames by using the inter-frame displacements of
each object with a linear constant velocity model which was independent of
other objects and camera motion. SORT created new tracks only if the
Intersection Over Union(IOU) of new detections was less than a certain
threshold ($IOU_{min}$). When this condition is met, SORT creates new
tracklets based on the geometry of detection bounding boxes. SORT then
initialized the velocity of new tracks to zero because the velocity of tracks
was unobserved initially. SORT associates new tracklets with detections to
accumulate enough evidence to prevent tracking of false positives. SORT
deletes tracklets if they were not detected for a number of frames
($T_{Lost}$). The value of $T_{Lost}$ was set to 1.

##  5 Tracktor++ tracking

The Tracktor++ tracking algorithm proposed by the authors of [2] is an online
tracking algorithm that exploits the regression head of object detection
models to regress and classify bounding boxes for the purpose of multi-object
tracking. In fact, the authors argued that all we need to perform multi-object
tracking is an object detection model. The challenge of multi-object tracking
is to form a trajectory using frames from a video sequence. A typical
trajectory of frames is defined as a list of ordered object bounding boxes
$T_{k}=\\{b_{t_{1}}^{k},b_{t_{2}}^{k},...\\}$ where
$b_{t_{1}}^{k},b_{t_{2}}^{k},...$ denotes bounding boxes with coordinates
$(x,y,w,h)$ and $t$ represents a frame in the video. In the initialization
step of Tracktor++, tracks are formed using the initial set of detections from
the object detection model. The regression head of the pedestrian detection
model was then leveraged to regress active trajectories to the current frame,
$t$. To this end, the bounding boxes of objects in the frame, $t-1$ are
regressed to the new position of the objects in the frame,$t$. For newer
detections following the initial detections, a new trajectory was started only
if the IOU with any of the existing trajectories was less than a threshold,
${\lambda_{new}}$. In some cases, the two frames might have different tracked
objects. In such situations, Tracktor++ utilized two well-known motion models
to improve the bounding box positions in future frames. Camera Motion
Compensation (CMC) was used for sequences with a moving camera to align the
frame using the Enhanced Correlation Coefficient(ECC). Otherwise, for low
frame sequences, a constant velocity assumption(CVA) was applied. Tracker also
used a short-term re-identification (reID) based on appearance vectors
generated by a Siamese neural network. This network helped in re-matching
deactivated tracklets with new bounding boxes based on the IOU threshold.

##  6 Tracking metrics

In this section, we will explain the MOT tracking performance metrics that
would be used in our comparative study for the tracking results of SORT and
Tracktor++. First and foremost, there is the need for standard MOT performance
metrics so that the performance of various tracking algorithms can be
verified. Using such benchmark performance metrics would help the research
community to learn insights from models that have better performance and this
would positively influence future research in this domain. Thus, we adopt the
MOT metrics defined in [3] and [10]. The evaluation metrics for tracking are:
global min-cost F1 score (IDF1, $(\uparrow)$) [10], Multi-object tracking
accuracy (MOTA, $(\uparrow)$) [3], Recall (Rcll, $(\uparrow)$) [10], number of
mostly tracked trajectories (MT,$(\uparrow)$) [10], number of mostly lost
trajectories(ML, $(\downarrow)$) [10], number of false detections
(FP,($\downarrow$)) [10], number of missed detections (FN, ($\downarrow$))
[10], and ID switches( ID Sw., $(\downarrow)$) [10] which is number of times
an ID switches to a different previously tracked object. Evaluation metrics
with ($\uparrow$), mean higher scores denote better performance; while for
metrics with ($\downarrow$), mean lower scores denote better performance. We
performed tracking experiments using the MOT17 dataset. We used the training
data for this experiment because the training set had ground truth detection
which we could use to compute the motmetrics for both SORT and Tracktor++. The
MOT challenge datasets consist of several challenging pedestrian tracking
sequences with crowded scenes and occlusions.

###  6.1 Experiments with SORT algorithm

In this section, we present the tracking results of using the SORT algorithm
on the MOT17 training dataset. We used private detections to evaluate the
performance of SORT. By private detections, we mean we used our object
detection model to predict the bounding boxes for each frame in the training
set. The tracking results for SORT are shown in Table 1.

Sequence MOTA$\uparrow$ (%) IDF1$\uparrow$ Rcll$\uparrow$ Prcn$\uparrow$
MT$\uparrow$ ML$\downarrow$ FP$\downarrow$ FN$\downarrow$ ID Sw.$\downarrow$
Time $\downarrow$ MOT17-02-DPM 11.60 0.50 1.30 59.40 0 62 159 18348 183 161.18
MOT17-02-FRCNN 40.90 43.60 51.40 96.80 12 14 313 9024 149 159.69 MOT17-02-SDP
10.60 0.50 1.30 59.40 0 62 159 18348 183 159.68 MOT17-04-DPM 20.00 0.20 0.20
61.20 0 83 59 47464 56 291.48 MOT17-04-FRCNN 76.00 74.70 77.60 98.20 43 10 667
10676 71 291.56 MOT17-04-SDP 31.00 0.20 0.20 61.20 0 83 59 47464 56 291.56
MOT17-05-DPM 16.20 2.30 2.70 72.60 0 133 71 6729 105 161.84 MOT17-05-FRCNN
47.80 50.70 51.90 96.10 9 44 145 3325 139 161.26 MOT17-05-SDP 17.20 2.30 2.70
72.60 0 133 71 6729 105 161.87 MOT17-09-DPM 19.20 0.70 1.00 84.40 0 26 10 5271
35 173.74 MOT17-09-FRCNN 64.10 55.60 66.10 98.40 12 1 58 1805 47 138.78
MOT17-09-SDP 19.60 0.70 1.00 84.40 0 26 10 5271 35 138.65 MOT17-10-DPM 18.70
0.80 3.90 65.30 0 57 267 12337 448 173.74 MOT17-10-FRCNN 54.00 26.50 60.30
96.30 9 5 294 5096 518 174.24 MOT17-10-SDP 22.70 0.80 3.90 65.30 0 57 267
12337 448 173.51 MOT17-11-DPM 24.30 1.00 1.40 65.00 0 75 71 9304 86 235.41
MOT17-11-FRCNN 69.30 58.90 72.40 97.40 23 7 179 2609 110 234.89 MOT17-11-SDP
19.30 1.00 1.40 65.00 0 75 71 9304 86 235.23 MOT17-13-DPM 23.00 1.40 3.00
52.70 0 110 318 11288 267 201.77 MOT17-13-FRCNN 50.10 47.10 54.70 96.10 18 26
262 5271 273 201.74 MOT17-13-SDP 17.00 1.40 3.00 52.70 0 110 318 11288 267
201.69 OVERALL 32.03 26.70 23.00 95.30 126 1199 3828 259288 667 4087.72

Table 1: SORT tracking results for MOT17 for private detections

###  6.2 Experiment with Tracktor++ algorithm

Tracktor++ utilizes the Re-ID model to recover tracks after they have been
occluded for some time. To this end, we trained the Re-ID model with ResNet-50
[20] as the feature extractor using the MOT17 Dataset. The Re-ID model had a
batch size of 32 and a learning rate of 0.0001. AdaGrad optimization algorithm
[ruder2016overview] was used as our optimizer. The model was trained on a
single NVIDIA GeForce GTXTITAN X GPU with 12 GB of memory for about 6 hours.
The tables 2, 3 and 4 shows tracking results of Tracktor++ for all private
detections on the MOT17 Dataset 444MOT17Dataset :
https://motchallenge.net/data/MOT17/. The MOT17 Challenge 555The MOTChallenge
web page: https://motchallenge.net also provides detections for the dataset
that was generated using classical object detectors like Faster RCNN [7],
Deformable Parts Model [6], and Scaled Dependent Pooling(SDP) [24] detector.

Sequence MOTA$\uparrow$ (%) IDF1$\uparrow$ Rcll$\uparrow$ Prcn$\uparrow$
MT$\uparrow$ ML$\downarrow$ FP$\downarrow$ FN$\downarrow$ ID Sw.$\downarrow$
Time $\downarrow$ MOT17-02-DPM 46.70 42.30 50.40 95.50 11 13 439 9215 245
219.04 MOT17-02-FRCNN 46.70 42.30 50.40 95.50 11 13 439 9215 245 220.21
MOT17-02-SDP 46.70 42.30 50.40 95.50 11 13 439 9215 245 220.34 MOT17-04-DPM
75.30 69.80 76.10 99.50 44 16 196 11350 200 407.58 MOT17-04-FRCNN 75.30 69.80
76.10 99.50 44 16 196 11350 200 407.57 MOT17-04-SDP 75.30 69.80 76.10 99.50 44
16 196 11350 200 406.08 MOT17-05-DPM 58.50 55.20 65.60 96.60 48 14 162 2376
330 225.54 MOT17-05-FRCNN 58.50 55.20 65.60 96.60 48 14 162 2376 330 227.41
MOT17-05-SDP 58.50 55.20 65.60 96.60 48 14 162 2376 330 226.99 MOT17-09-DPM
66.10 55.10 67.80 99.30 12 2 27 1715 63 186.42 MOT17-09-FRCNN 66.10 55.10
67.80 99.30 12 2 27 1715 63 187.13 MOT17-09-SDP 66.10 55.10 67.80 99.30 12 2
27 1715 63 186.69 MOT17-10-DPM 61.40 36.00 78.90 89.00 34 1 1251 2707 1001
246.05 MOT17-10-FRCNN 61.40 36.00 78.90 89.00 34 1 1251 2707 1001 246.99
MOT17-10-SDP 61.40 36.00 78.90 89.00 34 1 1251 2707 1001 247.21 MOT17-11-DPM
74.10 62.60 77.90 97.80 38 7 168 2081 194 321.95 MOT17-11-FRCNN 74.10 62.60
77.90 97.80 38 7 168 2081 194 322.29 MOT17-11-SDP 74.10 62.60 77.90 97.80 38 7
168 2081 194 320.05 MOT17-13-DPM 51.00 37.10 83.20 87.40 73 5 1399 1956 2353
292.01 MOT17-13-FRCNN 51.00 37.10 83.20 87.40 73 5 1399 1956 2353 292.37
MOT17-13-SDP 51.00 37.10 83.20 87.40 73 5 1399 1956 2353 292.69 OVERALL 64.90
55.70 72.00 95.70 780 174 10926 94200 13158 5702.61

Table 2: Tracktor++ with no REID network, and no motion model tracking results
for MOT17 for private detection without motion model

Sequence MOTA$\uparrow$ (%) IDF1$\uparrow$ Rcll$\uparrow$ Prcn$\uparrow$
MT$\uparrow$ ML$\downarrow$ FP$\downarrow$ FN$\downarrow$ ID Sw.$\downarrow$
Time $\downarrow$ MOT17-02-DPM 48.40 43.90 53.40 94.90 14 13 500 661 210
284.64 MOT17-02-FRCNN 47.70 43.60 53.40 95.50 14 13 602 653 213 282.70
MOT17-02-SDP 49.20 44.80 53.50 95.50 12 11 450 637 223 283.75 MOT17-04-DPM
76.90 70.20 77.10 96.10 44 15 479 10871 77 598.75 MOT17-04-FRCNN 75.10 70.30
77.20 96.40 47 14 300 10841 73 602.42 MOT17-04-SDP 76.70 70.10 77.20 96.60 44
15 290 10834 73 600.13 MOT17-05-DPM 58.70 55.60 72.90 94.60 59 15 718 1876 91
282.24 MOT17-05-FRCNN 58.79 56.40 72.80 94.90 58 13 684 1879 85 269.82
MOT17-05-SDP 59.40 55.70 72.60 94.50 56 15 216 1894 96 265.59 MOT17-09-DPM
66.50 55.25 73.80 90.30 15 2 300 1395 46 222.55 MOT17-09-FRCNN 66.80 55.20
73.40 92.70 14 2 401 1416 47 221.47 MOT17-09-SDP 67.70 55.80 74.20 91.70 14 2
200 1372 44 221.79 MOT17-10-DPM 62.25 49.80 82.70 94.40 37 1 162 2219 253
315.17 MOT17-10-FRCNN 61.70 49.30 82.40 89.90 37 1 741 2264 267 319.58
MOT17-10-SDP 67.28 49.20 83.00 94.30 39 1 679 2181 273 329.58 MOT17-11-DPM
75.65 65.50 80.90 88.60 44 6 391 1805 47 319.78 MOT17-11-FRCNN 75.95 63.00
81.20 87.60 44 6 399 1772 45 390.09 MOT17-11-SDP 75.76 62.78 80.90 84.60 43 6
385 1799 48 396.82 MOT17-13-DPM 52.52 38.70 86.70 95.90 80 4 2768 1553 372
348.63 MOT17-13-FRCNN 53.81 38.50 86.70 86.70 81 3 2386 1552 376 346.40
MOT17-13-SDP 51.65 38.20 86.40 87.00 82 3 1517 1589 362 347.23 OVERALL 65.77
56.23 94.8 84.7 878 161 14568 85063 3321 7291.05

Table 3: Tracktor++ with REID network and no motion model tracking results for
MOT17 for private detection without motion model

Sequence MOTA$\uparrow$ (%) IDF1$\uparrow(\%)$ Rcll$\uparrow$ Prcn$\uparrow$
MT$\uparrow$ ML$\downarrow$ FP$\downarrow$ FN$\downarrow$ ID Sw.$\downarrow$
Time $\downarrow$ MOT17-02-DPM 50.00 44.90 54.30 85.40 14 13 490 630 225
522.83 MOT17-02-FRCNN 52.40 43.80 54.90 85.20 14 13 590 558 238 520.16
MOT17-02-SDP 52.60 45.20 53.20 85.90 14 12 420 432 214 524.97 MOT17-04-DPM
77.80 70.90 77.60 96.10 44 15 471 450 81 1114.41 MOT17-04-FRCNN 77.90 69.90
77.50 96.40 44 15 286 360 79 1110.55 MOT17-04-SDP 77.00 73.00 77.70 96.30 44
15 280 250 76 1118.18 MOT17-05-DPM 60.20 54.40 74.10 77.80 58 13 710 433 92
368.38 MOT17-05-FRCNN 59.60 56.80 78.60 77.52 56 13 256 1861 88 368.96
MOT17-05-SDP 60.70 56.60 74.00 78.49 56 13 207 1739 93 369.63 MOT17-09-DPM
66.20 52.10 73.20 90.98 15 2 384 1380 45 458.14 MOT17-09-FRCNN 66.70 53.10
73.40 91.55 14 2 365 1399 40 459.20 MOT17-09-SDP 65.80 59.60 73.80 91.65 14 2
355 151 45 460.08 MOT17-10-DPM 64.60 50.70 83.00 77.96 36 1 165 2015 278
752.58 MOT17-10-FRCNN 63.90 50.50 83.50 78.59 36 1 673 2209 286 757.78
MOT17-10-SDP 64.70 50.20 84.00 78.90 36 1 695 2010 293 752.84 MOT17-11-DPM
77.10 66.20 87.60 86.70 42 6 365 1045 45 987.77 MOT17-11-FRCNN 77.20 61.30
81.40 85.90 43 6 397 1701 46 989.27 MOT17-11-SDP 76.00 61.40 83.95 85.90 43 6
269 1712 43 992.01 MOT17-13-DPM 53.50 41.30 83.77 72.70 78 5 2731 1495 330
1054.97 MOT17-13-FRCNN 54.90 43.00 86.70 73.60 80 5 2174 1520 333 1055.24
MOT17-13-SDP 52.00 39.30 86.80 73.70 78 5 1480 1528 327 1057.83 OVERALL 67.20
58.50 94.30 86.80 859 164 13763 24878 3297 15795.79

Table 4: Tracktor tracking results for MOT17 for private detection with reid,
and motion

###  6.3 Discussion of results

Following the experimental studies, this section provides a discussion on the
performance of the two tracking algorithms (SORT and Tracktor++). The
performance evaluation criteria were based on the metrics that were identified
in Section 6. Both algorithms were evaluated using the MOT17 training dataset.
Table 1 presents the tracking results of SORT on the MOT17 dataset. Likewise
Tables 2, 3, and 4 present the tracking results of Tracktor++ on the same
dataset. Comparing the tracking performance of Tracktor++ without additives
(Motion, ReID) with SORT shows that Tracktor++ has superior performance over
SORT. The multiple object tracking accuracy(MOTA), recall(Rcll), precision
(Prcn), number of mostly tracked trajectories( MT) for Tracktor++ were 64.90%,
55.70%, 72.00%, 95.70%, and 780 respectively as against 32.03%, 23%, 95.30%,
and 126 for SORT. Also, comparing the bare-bone Tracktor++ algorithm with
Tracktor++ combined with the re-identification model, we saw as in table 2 and
table 3 that there was a slight improvement in the MOTA, recall, and precision
scores of Tracktor++ when it was augmented with the Re-ID model. Also, the
number of tracklet ID switches is reduced when using Re-ID because it helps to
recover tracklets that get missing temporarily due to occlusions. Furthermore,
when, we activated both the motion model and re-identification network in
Tracktor++, we saw as from the results in table 4 that, there was an
improvement in the performance of Tracktor++. Thus, tradeoffs have to be made
in multi-people tracking methods about which models to adopt in the tracking
pipelines. The inclusion of motion and re-id models in the tracking pipeline
improves the performance of Tracktor++ but, that comes at an increased cost in
computation time as shown in Table 3, and 4.

##  7 Recommendation for future work

Multi-object tracking is still an active research topic and the computer
vision community has been making some significant progress in it. From the
results of these two tracking algorithms, we can see that they are far from
being the state of art algorithms. Specifically for the MOT Challenge, a lot
of new algorithms have been proposed and they have better results than these
two algorithms. However, from the analysis of these two algorithms, we can
gain some insights about what could be done and possible techniques we could
apply to improve the performance of multi-object tracking models. First, we
have identified that the quality of detections from the object detector has a
great impact on the performance of the tracking algorithm. This is because if
we have good detection results which we provide as input to the tracking
pipeline, we can reduce the number of false alarms and also reduce the false
negative scores of tracking algorithms. Second, one of the major reasons for
the poor performance of tracking algorithms is the presence of occlusions and
crowded scenes. While recent research has pushed forward the quality of object
detectors and in our case pedestrian object detectors to be precise, these
detectors are still not immune to the effects of occlusions. As a result, the
state-of-the-art object detectors, still do not perform well when predicting
bounding boxes in crowded scenes. One of the ways of mitigating this problem
is to provide more datasets containing crowded scenes which could be used to
train detectors to offset this current problem. Also, improving upon the
techniques used to recover tracklets that go missing for some time due to
occlusions in the tracking pipeline would lead to some performance gains.
Moreover, the context contains useful information that could be exploited to
improve the performance of multi-object tracking models. For instance, if we
are able to capture the knowledge of human motion in tracking algorithms we
could be able to improve the number of tracked trajectories (MT) of our
tracking algorithms. However, many of the approaches so far have modeled human
motion based on constant velocity assumptions. Adopting a motion model where
we capture the dynamic motion of people could help us to recover lost
tracklets. For instance, adopting a Long Short-Term Memory (LSTM) recurrent
neural network to model the motion activities of humans in a tracking scenario
would help us reduce the number of tracklets we lose due to occlusions.
Furthermore, Re-ID models are for recovering lost or missing tracks that occur
due to occlusion. But, the performance of the Re-ID network has a great impact
on the number of lost tracklets we can recover when they go missing due to
noise. Thus, using robust and high-performing Re-ID models could push forward
the performance of tracking algorithms. Also, the RE-ID models adopted must be
trained using data that is significantly related to the tracking problem
otherwise, we could have a high number of false re-identifications which could
reduce the performance of tracking algorithms.

##  8 Conclusion

This paper presented a comparative study of two multi-person tracking
algorithms. We have shown that the SORT algorithm is simple, but does not
perform so well on the tracking task as the Tracktor++ algorithm does. We have
also considered some trade-offs researchers have to make when adopting motion
models. Also, tracking-by-detection is the common approach to the task of
multiple-object tracking. We thus recommend that time should be spent on
training and tuning object detectors to have good performance so that quality
bounding boxes can be fed as input into the tracking algorithms we design. The
usual assumption of modeling human motion using constant velocity heuristics
is not always true. Thus, we argue that adopting models that try to handle the
dynamic nature of human motion could improve the number of mostly tracked
trajectories (MT) and minimize the number of ID switches (ID switches) in
tracking algorithms. Finally, the matching strategies based on object
appearance similarity scores should use good object appearance models that are
trained using tracking relevant datasets to reduce the false recovery of
tracklets as this hurts the performance of tracking algorithms.

## References

  * [1] Anton Andriyenko and Konrad Schindler.  Multi-target tracking by continuous energy minimization.  CVPR 2011, pages 1265–1272, 2011.
  * [2] Philipp Bergmann, Tim Meinhardt, and Laura Leal-Taixe.  Tracking without bells and whistles.  2019 IEEE/CVF International Conference on Computer Vision (ICCV), Oct 2019.  URL: http://dx.doi.org/10.1109/ICCV.2019.00103, doi:10.1109/iccv.2019.00103.
  * [3] Keni Bernardin and Rainer Stiefelhagen.  Evaluating multiple object tracking performance: The clear mot metrics.  J. Image Video Process., 2008, jan 2008.  doi:10.1155/2008/246309.
  * [4] Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft.  Simple online and realtime tracking.  In 2016 IEEE International Conference on Image Processing (ICIP). IEEE, sep 2016.  URL: https://doi.org/10.1109%2Ficip.2016.7533003, doi:10.1109/icip.2016.7533003.
  * [5] Georgios Evangelidis and Emmanouil Psarakis.  Parametric Image Alignment Using Enhanced Correlation Coefficient Maximization.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(10):1858–1865, October 2008.  URL: https://hal.inria.fr/hal-00864385.
  * [6] Pedro F Felzenszwalb, Ross B Girshick, David McAllester, and Deva Ramanan.  Object detection with discriminatively trained part-based models.  IEEE transactions on pattern analysis and machine intelligence, 32(9):1627–1645, 2010.
  * [7] Ross B. Girshick.  Fast R-CNN.  CoRR, abs/1504.08083, 2015.  URL: http://arxiv.org/abs/1504.08083, arXiv:1504.08083.
  * [8] Ross B. Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  CoRR, abs/1311.2524, 2013.  URL: http://arxiv.org/abs/1311.2524, arXiv:1311.2524.
  * [9] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn, 2017.  URL: https://arxiv.org/abs/1703.06870, doi:10.48550/ARXIV.1703.06870.
  * [10] Jaeyong Ju, Daehun Kim, Bonhwa Ku, David K. Han, and Hanseok Ko.  Online multi-object tracking with efficient track drift and fragmentation handling.  J. Opt. Soc. Am. A, 34(2):280–293, Feb 2017.  URL: http://opg.optica.org/josaa/abstract.cfm?URI=josaa-34-2-280, doi:10.1364/JOSAA.34.000280.
  * [11] Rudolf E. Kálmán.  A new approach to linear filtering and prediction problems" transaction of the asme journal of basic.  1960\.
  * [12] Laura Leal-Taixé.  Multiple object tracking with context awareness, 2014.  URL: https://arxiv.org/abs/1411.7935, doi:10.48550/ARXIV.1411.7935.
  * [13] Tsung-Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie.  Feature pyramid networks for object detection.  CoRR, abs/1612.03144, 2016.  URL: http://arxiv.org/abs/1612.03144, arXiv:1612.03144.
  * [14] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection, 2016.  URL: https://arxiv.org/abs/1612.03144, doi:10.48550/ARXIV.1612.03144.
  * [15] Wenhan Luo, Xiaowei Zhao, and Tae-Kyun Kim.  Multiple object tracking: A review.  ArXiv, abs/1409.7618, 2014.
  * [16] Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixé, and Christoph Feichtenhofer.  Trackformer: Multi-object tracking with transformers.  CoRR, abs/2101.02702, 2021.  URL: https://arxiv.org/abs/2101.02702, arXiv:2101.02702.
  * [17] Anton Milan, Laura Leal-Taixé, Ian Reid, and Stefan Roth.  Mot16: A benchmark for multi-object tracking.  03 2016.
  * [18] Sankar Pal, Anima Pramanik, Jhareswar Maiti, and Pabitra Mitra.  Deep learning in multi-object detection and tracking: state of the art.  Applied Intelligence, 51, 09 2021.  doi:10.1007/s10489-021-02293-7.
  * [19] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun.  Faster R-CNN: towards real-time object detection with region proposal networks.  CoRR, abs/1506.01497, 2015.  URL: http://arxiv.org/abs/1506.01497, arXiv:1506.01497.
  * [20] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [21] Daniel Stadler and Jürgen Beyerer.  Improving multiple pedestrian tracking by track management and occlusion handling.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10953–10962, 2021.  doi:10.1109/CVPR46437.2021.01081.
  * [22] Peize Sun, Yi Jiang, Rufeng Zhang, Enze Xie, Jinkun Cao, Xinting Hu, Tao Kong, Zehuan Yuan, Changhu Wang, and Ping Luo.  Transtrack: Multiple-object tracking with transformer.  CoRR, abs/2012.15460, 2020.  URL: https://arxiv.org/abs/2012.15460, arXiv:2012.15460.
  * [23] Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders.  Selective search for object recognition.  International journal of computer vision, 104(2):154–171, 2013\.
  * [24] Fan Yang, Wongun Choi, and Yuanqing Lin.  Exploit all the layers: Fast and accurate cnn object detector with scale dependent pooling and cascaded rejection classifiers.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2129–2137, 2016.  doi:10.1109/CVPR.2016.234.
  * [25] Lu Zhang and Laurens van der Maaten.  Structure preserving object tracking.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2013.
  * [26] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl.  Tracking objects as points.  CoRR, abs/2004.01177, 2020.  URL: https://arxiv.org/abs/2004.01177, arXiv:2004.01177.