# Stellar Flares Are Far-Ultraviolet Luminous

Vera L. Berger Churchill Scholar  Cavendish Laboratory, Department of Physics,
University of Cambridge, JJ Thomson Avenue, Cambridge, CB3 0HE, UK  Institute
for Astronomy, University of Hawai‘i, 2680 Woodlawn Drive, Honolulu, HI 96822,
USA  Department of Physics and Astronomy, Pomona College, 333 N. College Way,
Claremont, CA 91711, USA  Jason T. Hinkle FINESST FI  Institute for Astronomy,
University of Hawai‘i, 2680 Woodlawn Drive, Honolulu, HI 96822, USA  Michael
A. Tucker CCAPP Fellow  Department of Astronomy, The Ohio State University,
140 West 18th Avenue, Columbus, OH 43210, USA  Center for Cosmology and
Astroparticle Physics, The Ohio State University, 191 W. Woodruff Avenue,
Columbus, OH 43210, USA  Benjamin J. Shappee Institute for Astronomy,
University of Hawai‘i, 2680 Woodlawn Drive, Honolulu, HI 96822, USA  Jennifer
L. van Saders Institute for Astronomy, University of Hawai‘i, 2680 Woodlawn
Drive, Honolulu, HI 96822, USA  Daniel Huber Institute for Astronomy,
University of Hawai‘i, 2680 Woodlawn Drive, Honolulu, HI 96822, USA  Sydney
Institute for Astronomy (SIfA), School of Physics, University of Sydney, NSW
2006, Australia  Jeffrey W. Reep Institute for Astronomy, University of
Hawai‘i, 34 Ohia Ku St., Pukalani, HI 96768, USA  Xudong Sun (孙旭东) Institute
for Astronomy, University of Hawai‘i, 34 Ohia Ku St., Pukalani, HI 96768, USA
Kai E. Yang (杨凯) Institute for Astronomy, University of Hawai‘i, 34 Ohia Ku
St., Pukalani, HI 96768, USA

###### Abstract

We identify 182 flares on 158 stars within 100 pc of the Sun in both the near-
ultraviolet (NUV: $1750-2750$ Å) and far-ultraviolet (FUV: $1350-1750$ Å)
using high-cadence light curves from the Galaxy Evolution Explorer (_GALEX_).
Ultraviolet (UV) emission from stellar flares plays a crucial role in
determining the habitability of exoplanetary systems. However, whether such UV
emission promotes or threatens such life depends strongly on the energetics of
these flares. Most studies assessing the effect of flares on planetary
atmospheres assume a $9\,000$ K blackbody spectral energy distribution that
produces more NUV flux than FUV flux ($\mathcal{R}\equiv F_{\rm FUV}/F_{\rm
NUV}\approx\frac{1}{6}$). Instead, we observe the opposite with the excess FUV
reaching $\mathcal{R}\approx\frac{1}{2}\\--2$, roughly $3\\--12$ times the
expectation of a $9\,000$ K blackbody. The ratio of FUV to NUV time-integrated
flare energies is 3.0 times higher on average than would be predicted by a
constant $9\,000$ K blackbody during the flare. Finally, we find that the
FUV/NUV ratio at peak tentatively correlates ($\sim 2\sigma$ significance)
both with total UV flare energy and with the $G-RP$ color of the host star. On
average, we observe higher FUV/NUV ratios at peak in $E_{\text{UV}}>10^{32}$
erg flares and in flares on fully convective stars.

stars: flare – stars: low-mass – ultraviolet: stars – astrobiology

††journal: MNRAS

##  1 Introduction

Stellar flares are energetic bursts of electromagnetic radiation driven by
magnetic reconnection (e.g., litvinenko99; benz10; shibata16). Flares are more
frequently observed on low-mass stars ($M<1.5\hbox{M${}_{\odot}$}$), which
have surface convection zones (e.g., petterson89; balona15; davenport16;
doorsselaere17). Flares on M-dwarf stars are of particular interest because of
these stars’ heightened magnetic activity and ideal candidacy to search for
exoplanets in the habitable zone (e.g., endl03; kaltenegger09; reiners12).

Ultraviolet emission from flares impacts the habitability of exoplanets
orbiting flaring stars. rimmer18 delineated “abiogenesis zones” in which flare
rates and energies could deliver sufficient UV photons to drive prebiotic
chemistry. On the other hand, tilley19 characterized “ozone depletion zones”
in which M-dwarf flares of sufficient rates and energies deplete the ozone
column of habitable exoplanets. Flares with energies above $10^{34}$ erg may
contribute to either effect. Thus, for sufficiently high flare rates, its
unclear exactly where flares help versus hinder the development of complex
molecules.

(a)

(b)

Figure 1: _Left panel:_ Normalized blackbody curves (solid lines) spanning a
range of observed temperatures plotted against filter response functions
(dashed lines) for telescopes commonly used for flare studies. The TESS
(brown), _Kepler_ (black), and ASAS-SN V band (gray) response functions fall
on the Rayleigh-Jeans tail for temperatures above $\sim 7\,000$ K. _GALEX_ NUV
(red) and FUV (teal) filter response functions fall around the peaks of
blackbody SEDs for temperatures $\sim 10\,000\\--30\,000$ K. _Right panel:_
Relation between blackbody temperature and FUV/NUV $\lambda L_{\lambda}$
ratio, computed using synthetic photometry. The dashed line at 4.33 denotes
the numerically-computed asymptote once the _GALEX_ filters are both in the
Rayleigh-Jeans limit.

Recent optical studies have found few stars that display superflare
($E>10^{33}$ erg) rates and energies sufficient to affect their habitability
(schmidt14; schmidt16; schmidt19; rodriguez18; rodriguez20; gunther20;
feinstein20; zeldes21; bogner21). These studies typically assume a constant-
temperature blackbody spectrum for flares ($\hbox{$T_{\rm eff}$}\approx
9\,000-10\,000$ K) to estimate the incident UV flux. Given that a $9\,000$ K
blackbody produces 84% less FUV emission than NUV emission, small temperature
deviations from the adopted spectral energy distribution (SED) can produce
large variations in the incident UV flux.

While large-scale observational studies of stellar flares have primarily been
conducted in the optical wavelengths (e.g., walkowicz11; shibayama13;
hawley14; davenport16; schmidt19; rodriguez20; gunther20; feinstein20), these
wavelengths can comprise a small fraction of a flare’s total emission
(maehara12; namekata17). As displayed in Figure 1(a), the filter responses for
the _Kepler_ space telescope (kepler), the Transiting Exoplanet Survey
Satellite (_TESS,_ tess) and the All-Sky Automated Survey for Supernovae
(ASAS-SN, asassn_shappee; asassn_kochanek; asassn_hart) capture only the
Rayleigh-Jeans tail of blackbody emission for $\hbox{$T_{\rm eff}$}>7\,000$ K.
In addition, NUV and optical flare emission may arise from M-dwarf stars’
chromospheres and upper photospheres (joshi21), whereas far-ultraviolet
emission stems from the upper chromosphere and may correspond to the flare’s
impulsive phase of heating and compression of plasma (neupert68; dennis93;
hawley03; benz10).

While NUV and optical emission may be well-represented by a blackbody, some
superflares have exhibited significant line emission in the FUV, sometimes
surpassing quiescent flux levels by a factor of 100 (france16). loyd18a
computed a blackbody temperature of $15\,500$ K for the “Hazflare” in FUV
spectra from the HST Cosmic Origins Spectrograph (COS, $1170-1430$ Å). This
flare exhibited substantial enhancements in C III, Si IV, S III, and N V
emission lines; however, as with the 1985 Great Flare on AD Leo (hawley91), a
hot blackbody continuum dominated the total FUV emission. Furthermore,
froning19 obtained a far/extreme-UV spectrum of the M dwarf GJ 674 with HST
COS ($1065-1365$ Å) during a flare, revealing a blackbody-dominated spectrum
with $\hbox{$T_{\rm eff}$}\approx 40\,000$ K and numerous superimposed C, Si,
N, and Fe emission lines.

(a)

(b)

(c)

(d)

Figure 2: Example NUV and FUV light curves for two flares in our sample that
are representative of the range of FUV/NUV flux and SNR we observe. _Upper
panel:_ Two high SNR flares spanning the range of FUV/NUV fluxes present in
our sample. On the left is Gaia 3537511712697046656, a K6V star 56 pc from
Earth. On the right is Gaia 1461125613285603840, an M4V star 13 pc from Earth.
_Lower panel:_ Flares chosen randomly from our sample. On the left is Gaia
365548347350541952, an M1.5V star 22 pc from Earth. On the right is Gaia
3858896247076028288, an M3.5V star 15 pc from Earth.

In contrast, FUV spectra of the Sun show flare emission is dominated by line
emission instead of a thermal continuum (brekke96; simoes19). Given the
intrinsic differences between M-dwarf and solar magnetic fields (and thus
flare activity; gunther20), it remains unclear when line emission dominates or
continuum dominates for a given flare.

A more direct method for studying the impact of flares on habitability is
time-resolved UV photometry of stars during flares. The NASA Galactic
Evolution Explorer (GALEX) space mission (galex05; galex07_morrissey) provides
a unique opportunity to study ultraviolet emission from flares. _GALEX_
simultaneously observed in the far-ultraviolet (FUV: $1350-1750$ Å) and near-
ultraviolet (NUV: $1750-2750$ Å) bands (galex05). In addition, the micro-
channel plate detectors aboard _GALEX_ recorded the timing of individual
photon events to $\approx 5~{}\rm{ms}$ precision. We can therefore construct
high-cadence light curves with gPhoton (million16) to process GALEX
observations at arbitrary time resolution.

Early flare studies utilizing the time-resolved UV photometry of GALEX have
been limited to small sample sizes. Studies of a few flares using simultaneous
_GALEX_ FUV and NUV observations have characterized increases in FUV/NUV flux
during superflares reaching a maximum ratio of 13 in the most extreme case, a
factor of $>50$ higher than would be predicted by a $9\,000$ K blackbody
(robinson05; welsh06). FUV emission from host stars can photolyze CO2 in
planetary atmospheres, producing atomic oxygen which can then recombine to
form O2 (tian14). Therefore, high ratios of FUV/NUV radiation such as that
observed by these studies may produce sufficient abiotic atmospheric oxygen to
constitute a false biosignature (tian14; harman15). In addition, high FUV/NUV
ratios suggest that that the typical SED assumptions for flare studies may be
underestimating the true energy and ionizing flux of superflares.

Gaia EDR3 ID |  RA [deg] |  Dec [deg] |  Distance [pc] |  $M_{\text{G}}$ [mag] |  $RP$ [mag] |  Spectral Type
---|---|---|---|---|---|---
1013481650428107648 |  132.6026 |  46.3209 |  $81.95\pm 0.15$ |  14.22 |  13.09 |  M3V
1027636694403121664 |  130.6758 |  51.4823 |  $60.10\pm 0.07$ |  13.92 |  12.73 |  M3V
1047687216648750976 |  159.5389 |  59.4848 |  $93.44\pm 0.56$ |  16.77 |  15.41 |  M5V
1058559634300870400 |  162.0490 |  64.7243 |  $96.62\pm 0.21$ |  15.21 |  13.96 |  M4V
1059924128230565888 |  161.0179 |  67.0126 |  $47.40\pm 0.03$ |  13.66 |  12.50 |  M3V
1070172916631499520 |  150.0283 |  68.1115 |  $76.99\pm 0.14$ |  15.44 |  14.22 |  M3.5V
1073027695493625856 |  157.6594 |  68.5490 |  $88.37\pm 0.14$ |  14.39 |  13.19 |  M3.5V
1076599905693041280 |  156.8567 |  71.0655 |  $95.87\pm 0.26$ |  15.39 |  14.17 |  M3.5V
1113525010848246784 |  96.6996 |  71.1834 |  $90.77\pm 0.42$ |  16.80 |  15.50 |  M4.5V
1119235182751380736 |  140.6038 |  70.1558 |  $91.59\pm 1.32$ |  18.56 |  17.11 |  M6V
$\cdots$ |  $\cdots$ |  $\cdots$ |  $\cdots$ |  $\cdots$ |  $\cdots$ |  $\cdots$
Table 1: Star properties; full table available in the online version of the
manuscript.

In this work, we investigate all stars within 100 pc that were observed by
GALEX simultaneously in the NUV and FUV. We use a sample of 182 flares on 158
stars and show that this FUV excess emission is not restricted to superflares
but a general characteristic of flares on low-mass stars.

In Section 2 we provide an overview of our sample and flare detection method.
Section 3 characterizes FUV and NUV flare emission in comparison to blackbody
temperatures. Section 4 discusses the implications of our findings for
characterizing the impact of stellar flares on the habitability of orbiting
exoplanets.

##  2 Sample Selection and Flare Detection

We select our targets from the Gaia Catalogue of Nearby Stars (GCNS), which is
complete down to spectral type M8 within 100 pc (gaiaedr3). We exclude sources
within one FWHM ($\approx 5.6$ arcsec) of another star in the GCNS to avoid
contamination. We then use the gPhoton database and processing software
(million16) to search for _GALEX_ coverage of stars in the GCNS and perform
aperture photometry. We exclude data with flags from gPhoton corresponding to
aperture or annulus events in pixels that are contiguous to a masked hotspot
or the detector edge. We bin photon events at cadences scaled by the physical
distance of the corresponding sources. Shorter bins are employed for nearby
sources. For white-light flares, amplitude and duration are correlated
(maehara15), and so we expect shorter flares to be fainter and thus only be
visible in more nearby stars. More distant objects are binned with longer
cadences to enhance signal-to-noise ratios. We use 10, 20, and 30 s bins for
stars within 15 pc, between 15 - 50 pc, and between $50-100$ pc respectively.

We find that bright sources exhibit significant systematics. We remove any
light curves with median flux brighter than 15.4 mag NUV, a limit established
by examining 2000 randomly chosen light curves by eye. Visual examination of
stellar images and their background light curves suggests that contaminated
sources either have a high cross-correlation between background and source
flux over time, or a high standard deviation in background flux. We minimize
this contamination by also removing light curves with background flux with a
median above 40 counts or standard deviation above 6 counts.

Table 2: Flare properties; full table available in the online version of the manuscript.  Gaia EDR3 ID | t${}_{\text{start}}$ | t${}_{\text{end}}$ | Exp. Time | $\mathcal{R}_{E}$ | $\mathcal{R}_{\lambda L\lambda}$ | UV Energy
---|---|---|---|---|---|---
| [_GALEX_ s] | [_GALEX_ s] | [s] |  |  | [erg]
1013481650428107648 | 794114866 | 794115011 | 145 | 0.57 ±0.06 | 1.16 ±0.09 | 1.96${}^{+0.10}_{-0.10}$e+32
1027636694403121664 | 823888055 | 823888475 | 420 | 0.39 ±0.03 | 0.75 ±0.04 | 2.93${}^{+0.09}_{-0.09}$e+32
1047687216648750976 | 890596076 | 890596136 | 60 | 0.84 ±0.21 | 1.16 ±0.21 | 4.70${}^{+0.60}_{-0.60}$e+31
1058559634300870400 | 758796977 | 758797577 | 600 | 0.78 ±0.04 | 0.65 ±0.05 | 4.00${}^{+0.17}_{-0.17}$e+32
1059924128230565888 | 820629087 | 820629182 | 100 | 0.81 ±0.23 | 1.20 ±0.39 | 9.03${}^{+1.44}_{-1.37}$e+30
1069666346712783488 | 882307565 | 882308342 | 777 | 0.50 ±0.07 | 0.46 ±0.14 | 2.28e${}^{+0.16}_{-0.16}$+32
1070172916631499520 | 820545790 | 820545880 | 90 | 0.30 ±0.16 | 0.52 ±0.21 | 2.09e${}^{+0.35}_{-0.35}$+31
1073027695493625856 | 882461389 | 882462136 | 748 | 0.48 ±0.02 | 0.92 ±0.03 | 3.06e${}^{+0.04}_{-0.04}$+33
1076599905693041280 | 859394503 | 859394653 | 150 | 0.48 ±0.08 | 0.59 ±0.09 | 1.31e${}^{+0.10}_{-0.10}$+32
1113525010848246784 | 912762216 | 912762336 | 120 | 0.74 ±0.15 | 1.54 ±0.23 | 7.87${}^{+0.82}_{-0.84}$e+31
$\cdots$ | $\cdots$ | $\cdots$ | $\cdots$ | $\cdots$ | $\cdots$ | $\cdots$

We use light curves in the NUV for flare detection because the _GALEX_ NUV
detector has better throughput and efficiency (bianchi14). The flare detection
approach relies on finding data points that are inconsistent with noise in the
light curve and representative of true source variability (e.g., chang15). We
estimate a baseline flux by iteratively sigma-clipping the light curve
2$\sigma$ above the median flux. Given a time period $[t_{a},t_{b}]$ of a
light curve, we compute the mean flux $\bar{f}_{ab}$ and standard deviation
$\sigma_{ab}$. For the $i$th flux measurement with calculated uncertainty
$\sigma_{i}$ within the time period, we evaluate:

$\frac{f_{i}-\bar{f}_{ab}}{\sqrt{\sigma_{i}^{2}+\sigma_{ab}^{2}}}\geq n,$ (1)

where a flux satisfying Equation 1 corresponds to an $n\sigma$ detection. We
first search for an individual $3\sigma$ detection in the light curve and then
look to the neighboring points in the range $[t_{i-2},t_{i+2}]$, excluding
$t_{i}$. If there are two adjacent points in this range corresponding to
$2\sigma$ detections we then flag the visit as containing a flare. Finally, we
require that the flux does not peak at the first or last photometric point in
the visit since some objects show decreasing or increasing trends only at the
very beginning or end of a visit.

For this study we only include flares with observations in both FUV and NUV to
enable the calculation of color temperatures. A full analysis of all light
curves, including NUV-only flares and injection-recovery tests, is reserved
for a forthcoming paper (Berger et al. in prep). As a final cut, we require
that the FUV light curve has fewer than 3 missing observations within 5 points
of peak NUV flare brightness. This method produces 193 candidates. Even with
our quality cuts, six additional sources display variability that we classify
as a false-positive upon examining the light curves and 2D images. These
candidates appear within crowded fields.

Our final sample is composed of 182 flares with simultaneous observations in
the NUV and FUV. We present their sources in Table 1. Spectral classification
of our sample is discussed in Section 3.3 but the sample is composed of 79%
fully convective stars ($\geq$ M3, $n=143$), 16% partially convective M stars
($n=30$), and 5% K stars ($n=9$). Figure 2 displays the range of light curves
we observe; the top panel shows two high signal-to-noise flares that span the
range of FUV/NUV flux we observe, and the lower panel shows two representative
randomly selected flares in our sample.

##  3 FUV Flare Emission

In this section we calculate FUV/NUV emission ratios, inferred temperatures
and UV energies for the 182 flares in our sample, summarized in Table 2, then
compare the inferred blackbody color temperatures to the typically-assumed
$9\,000$ K blackbody SED for flares.

###  3.1 Flare temperatures assuming blackbody emission

Under the assumption that the spectral energy distribution of a flare can be
well-represented as a blackbody, we can construct a one-to-one relation
between blackbody temperature and FUV/NUV $\lambda L_{\lambda}$ ratio. We do
this by integrating the transmission function and blackbody SED over the range
of wavelengths covered by each filter (koornneef86). We assume interstellar
extinction is negligible since our sources are within the Local Bubble (e.g.,
cox87; fossati17), characterized by a low-density interstellar medium. Figure
1(b) shows the FUV/NUV $\lambda L_{\lambda}$ ratio for a range of blackbody
temperatures computed using synthetic photometry. Note that the color-
temperature curve flattens considerably around $50\,000$ K, where both _GALEX_
filters land on the Rayleigh-Jeans tail of the blackbody and thus the FUV-NUV
color can no longer distinguish between hotter temperatures. The color-
temperature curve asymptotes to FUV/NUV $=4.33$ for arbitrarily high $T_{\rm
eff}$.

We estimate the inferred blackbody temperature for each observation during a
flare by directly measuring the FUV/NUV $\lambda L_{\lambda}$ ratio.
Uncertainties on the color temperature are estimated by propagating the
uncertainties on observed fluxes.

###  3.2 Excess FUV emission

We estimate the FUV excess flux for each observation during a flare by scaling
a $9\,000$ K blackbody to the observed NUV flux. Then, we impute the expected
FUV emission for comparison to our observations. We denote the ratio of time-
integrated energies by $\mathcal{R}_{E}$$=E_{\rm FUV}/E_{\rm NUV}$.

A $9\,000$ K blackbody produces a time-integrated energy ratio of 0.17 whereas
we find a median $\mathcal{R}_{E}$ = 0.50 ($T_{\rm eff}\approx 13\,500$ K) and
a maximum $\mathcal{R}_{E}$ = 1.06 ($T_{\rm eff}\approx 21\,000$ K). Thus, a
$9\,000$ K blackbody underestimates the median observed $\mathcal{R}_{E}$ by a
factor of 3.0 and underestimates the maximum ratio by a factor of 6.3. Among
the flares in our sample, 98% ($n=178$) display $\mathcal{R}_{E}$ ratios that
exceed expectations for a $9\,000$ K blackbody. This suggests that a constant
$9\,000$ K blackbody SED is insufficient to account for the levels of FUV
emission we observe.

On the other hand, FUV/NUV ratios at individual epochs in a flare can
significantly exceed the time-integrated values. To quantify this, we compute
the observed ratios of FUV/NUV $\lambda L_{\lambda}$ at peak NUV brightness
for each flare, which we refer to as $\mathcal{R}_{\lambda L\lambda}$. Figures
3(b) and 4(b) show the $\mathcal{R}_{\lambda L\lambda}$ values with respect to
spectral type and flare UV energy. Note that relying on the NUV time of peak
introduces a slight bias against NUV-faint flares. Thus, true FUV/NUV ratios
are likely higher around flare peak. This behavior is demonstrated in Figure
2(c), where the observed FUV flux both exceeds the NUV flux and peaks before
the NUV.

We find a median $\mathcal{R}_{\lambda L\lambda}$= 0.57 ($T_{\rm eff}\approx
14\,300$ K) and a maximum $\mathcal{R}_{\lambda L\lambda}$= 2.12 ($T_{\rm
eff}\approx 49\,600$ K). Thus, a $9\,000$ K blackbody underestimates the
median observed $\mathcal{R}_{\lambda L\lambda}$ by a factor of 3.4 and
underestimates the maximum ratio by a factor of 12.6.

(a) Energy ratio by spectral type

(b) $\lambda L_{\lambda}$ ratio at peak NUV by spectral type

(c) Bootstrapped energy ratio by spectral type

(d) Bootstrapped $\lambda L_{\lambda}$ ratio by spectral type

(e) Energy ratio vs. G-RP color

(f) $\lambda L_{\lambda}$ ratio vs. G-RP color

Figure 3: Histograms and scaled kernel density estimates (top), bootstrapped
mean values (middle), and scatter plots with lines of best fit (bottom) for
FUV/NUV energy ratios (left) and $\lambda L_{\lambda}$ at peak NUV flux
(right), with respect to spectral type. The ratios corresponding to
$9\,000-10\,000$ K blackbody SEDs are denoted by vertical and horizontal black
dashed lines in the top and bottom panels, respectively. Blackbody
temperatures corresponding to their respective FUV/NUV ratios are marked on
the top axis. X-axes are not consistent between panels. Lines of best fit on
the bottom panel are computed using least-squares optimization (orange) and
Theil-Sen regression (turquoise). Color darkens with spectral type; ratios for
K stars are plotted in light pink, partially convective M stars in red, and
fully convective stars in brown.

(a) Energy ratio

(b) $\lambda L_{\lambda}$ ratio at peak NUV

(c) Bootstrapped energy ratio

(d) Bootstrapped $\lambda L_{\lambda}$ ratio

(e) Energy ratio vs. log(flare energy)

(f) $\lambda L_{\lambda}$ ratio vs. log(flare energy)

Figure 4:  Same as Figure 3, but with respect to UV flare energy. Color
lightens with increase in UV flare energy; ratios corresponding to energies
below $10^{31}$ erg in dark green, to UV energies between $10^{31}-10^{32}$
erg in medium green, and to energies of $10^{32}$ erg or higher in light
green.

###  3.3 Results by spectral type

Stellar magnetic fields, which are the energy sources for stellar flares,
originate from dynamo-related convective motions and differential rotation,
(e.g., petterson89; balona15; doorsselaere17) and so the depth of the
convection zone may correlate with flare properties. To test this idea, we
derive spectral types for the flare stars in our sample using Gaia G-RP color
(gaiaedr3) and the spectral type-color sequence from pecaut13spt, estimating a
fully convective boundary of M3V (jao18). We use this system because we do not
have rotation periods for every star in the sample to compute a Rossby number.
We divide our sample into three spectral classification bins: K stars,
partially convective M dwarfs (M0V-M2V), and fully convective stars (M3$+$).
Figures 3(a) and 3(b) illustrate the distributions of time-integrated energy
ratios $\mathcal{R}_{E}$ and $\mathcal{R}_{\lambda L\lambda}$ by spectral bin.
Table 3 displays the medians and standard deviations for each kernel density
estimate (KDE).

There appears to be a trend with an increasing median $\mathcal{R}_{\lambda
L\lambda}$ with later spectral types and thus deepening convective envelopes.
The distribution of $\mathcal{R}_{\lambda L\lambda}$ for K stars has a median
of 0.443 ($12\,700$ K); M0V-M2V stars a median $\mathcal{R}_{\lambda
L\lambda}$ of 0.541 ($14\,000$ K); and fully convective stars a median
$\mathcal{R}_{\lambda L\lambda}$ of 0.577 ($14\,400$ K). The $\mathcal{R}_{E}$
distributions peak at similar values but skew towards higher ratios, a similar
trend to $\mathcal{R}_{\lambda L\lambda}$. Furthermore, fully convective stars
exhibit the largest deviations from the assumed $9\,000$ K blackbody.

We explore the significance of these findings in three ways:

1) As a first attempt, we pairwise compared the spectral type categories using
Kolmogorov-Smirnov (K-S) and Anderson-Darling tests. Neither of these tests
were able to differentiate either the $\mathcal{R}_{E}$ or
$\mathcal{R}_{\lambda L\lambda}$ distributions at 95% confidence.

2) As a next attempt, we bootstrapped the mean ratios for each spectral type
category. We resampled each category with replacement $10\,000$ times and
calculated the mean of the distribution. We compute $\mathcal{R}_{E}$ means
and quantile intervals for K, early M, and fully convective stars of $0.49\pm
0.05$, $0.49\pm 0.03$, and $0.52\pm 0.02$. We compute $\mathcal{R}_{\lambda
L\lambda}$ means and quantile intervals for K, early M, and fully convective
stars of $0.45\pm 0.08$, $0.53\pm 0.04$, and $0.59^{+0.02}_{-0.03}$,
respectively. The appears to be a statistically significant correlation
between $\mathcal{R}_{\lambda L\lambda}$ and spectral type.

3) Finally, we explore a potential linear relationship between the FUV/NUV
ratios and $G$-$RP$ color as a proxy for spectral type (shown in Figure 3e and
3f). As a first step, we cut obvious outliers, five with negative
$\mathcal{R}_{\lambda L\lambda}$ and two with large but very uncertain
$\mathcal{R}_{\lambda L\lambda}$. The Pearson correlation coefficient is 0.18
for $\mathcal{R}_{\lambda L\lambda}$ vs. $G$-$RP$ and 0.06 for
$\mathcal{R}_{E}$ vs. $G$-$RP$. These indicate a tentative weak correlation
but the significance is difficult to determine because the Pearson correlation
coefficient does not consider uncertainties. Instead, we will perform a linear
fit and bootstrap the sample to determine if there is a statistically
significant positive slope. First, to reduce the covariance between the slope
and intercept we move the intercept to the median color of the sample as given
by:

$\mathcal{R}=m\times({\rm log_{10}(E_{\text{UV}}/{\rm erg}})-1.2)+b,$ (2)

for slope $m$, intercept $b$, and ratio $\mathcal{R}$ (either
$\mathcal{R}_{E}$ or $\mathcal{R}_{\lambda L\lambda}$). Second, to account
both for underestimated errors and for the intrinsic scatter in the flare
population, we iteratively fit a best-fit line adding increasing uncertainty
in quadrature with the $\mathcal{R}_{E}$ or $\mathcal{R}_{\lambda L\lambda}$
uncertainties until a reduced $\chi^{2}=1$ was found for each fit. We report
the best-fit values in Table 4 and show the best-fits in Figures 3(e) and
3(f). Next we determine the uncertainty on the linear fit by bootstrapping the
sample with replacement $10\,000$ times. We show 100 randomly selected
bootstrapped fits in Figures 3(e) and 3(f) to indicate the fit uncertainty. In
Table 4 we report the 16th and 84th percentile along with the percentage of
the resamples where the slope of the best fit is $\leq 0$. Finally, to verify
the least-square fits are not being driven by outliers, we redo the linear
fits using Theil-Sen Regression, bootstrap the fits, and also report those
fits in Table 4. We then take the fraction of bootstrap trials with slopes
$\leq$ 0 as our p-value for the detection of a positive relation. When
reporting the significance, we take the more conservative value between the
least-square and Theil-Sen fits. We find that the $\mathcal{R}_{\lambda
L\lambda}$ vs. color slope is consistent with being positively correlated at
nearly 2 sigma and the $\mathcal{R}_{E}$ vs. color slope is consistent with
being positively correlated at nearly 1.5 sigma.

Thus we conclude there is a tentative trend with increasing average
$\mathcal{R}_{\lambda L\lambda}$ and $\mathcal{R}_{E}$ with later spectral
types, especially for $\mathcal{R}_{\lambda L\lambda}$ and for fully
convective stars. However, we note that the uncertainties of the
$\mathcal{R}_{\lambda L\lambda}$ and $\mathcal{R}_{E}$ ratios are large
relative to the difference in the means between spectral type categories.

###  3.4 Results by flare UV energy

We also investigate for trends between the total flare UV energy and FUV/NUV
ratio. First, we estimate the total UV flux at each epoch by fitting a linear
SED between ($\lambda_{\text{eff}}^{\text{FUV}},f_{\text{FUV}}$) and
($\lambda_{\text{eff}}^{\text{NUV}},f_{\text{NUV}}$) for an observed FUV flux
$f_{\text{FUV}}$ and corresponding NUV flux $f_{\text{NUV}}$ and effective
wavelengths $\lambda_{\text{eff}}^{\text{FUV}}=1528$ Å and
$\lambda_{\text{eff}}^{\text{NUV}}=2271$ Å (morrissey05). We integrate the
linear SED over the wavelength range covered by 10% of peak filter response
for the _GALEX_ filters, $1343\\--2831$ Å, as shown in Figure 1(a). We then
use the distance from Gaia as presented in Table 1 to compute total flare UV
energies ($E_{\text{UV}}$). We find that our flare sample has
$E_{\text{UV}}=2\times 10^{29}\\--3\times 10^{33}$ erg, with a median UV
energy of $3.8\times 10^{31}$ erg.

We divide the flare sample into 3 bins of $E_{\text{UV}}$, flares with
$E_{\text{UV}}<10^{31}$ erg, $10^{31}<E_{\text{UV}}<10^{32}$ erg, and
$E_{\text{UV}}>10^{32}$ erg. Figures 4(a) and 4(b) display $\mathcal{R}_{E}$
and $\mathcal{R}_{\lambda L\lambda}$ as a function of $E_{\text{UV}}$. The
corresponding values of the kernel density estimates by UV energy are noted in
Table 3. The distribution of $\mathcal{R}_{\lambda L\lambda}$ for flare UV
energies below $10^{31}$ erg has a median of 0.530 ($13\,800$ K); energies
between $10^{31}-10^{32}$ erg a median of 0.535 ($13\,900$ K); and energies
$10^{32}$ erg and above a median of 0.628 ($15\,000$ K).

We then perform similar statistical tests as in Section 3.3.

1) K-S and Anderson Darling tests again can mostly not distinguish the
$\mathcal{R}_{E}$ or $\mathcal{R}_{\lambda L\lambda}$ distributions for the
$E_{\text{UV}}$ groups. However, the K-S test between the
$10^{31}<E_{\text{UV}}<10^{32}$ erg and $E_{\text{UV}}>10^{32}$ erg
$\mathcal{R}_{\lambda L\lambda}$ samples has a marginal, p-value of 0.03,
detection that they are drawn from different distributions.

2) We compute $\mathcal{R}_{E}$ means and quantile intervals for flare
energies of $E_{\text{UV}}<10^{31}$ erg, $10^{31}<E_{\text{UV}}<10^{32}$ erg,
and $E_{\text{UV}}>10^{32}$ erg of $0.50\pm 0.03$, $0.51\pm 0.02$, and
$0.52\pm 0.02$. We compute $\mathcal{R}_{\lambda L\lambda}$ means and quantile
intervals of $0.53\pm 0.04$, $0.55\pm 0.03$, and $0.64\pm 0.04$, respectively.
There appears to be a statistically significant trend where the most energetic
flares have a higher $\mathcal{R}_{\lambda L\lambda}$ but it does not appear
such a relation exists for $\mathcal{R}_{E}$.

3) We follow the same procedure as in Section 3.3 investigating if there is a
linear relationship between FUV/NUV ratios and $E_{\text{UV}}$. First, the
Pearson correlation coefficient is 0.11 for $\mathcal{R}_{\lambda L\lambda}$
vs. $E_{\text{UV}}$ and 0.02 for $\mathcal{R}_{E}$ vs. $E_{\text{UV}}$. These
indicate a tentative weak correlation but we bootstrap least-squares and
Theil-Sen linear fits to test their statistical significance. Again to reduce
covariance between the fit parameters we choose a pivot near the median
$E_{\text{UV}}$ as shown by:

$\mathcal{R}=m\times({\rm log_{10}(E_{\text{UV}}/{\rm erg}})-31.6)+b,$ (3)

for slope $m$, intercept $b$, and ratio $\mathcal{R}$. The best-fits are shown
in Figures 4(e) and 4(f). In Table 4 we report the fit statistics. We find
that the $\mathcal{R}_{\lambda L\lambda}$ vs. $E_{\text{UV}}$ slope is
consistent with being positively correlated at $\gtrsim 2$ sigma. However, the
outlier-resistant Theil-Sen fit is consistent with no relationship between
$\mathcal{R}_{E}$ and $E_{\text{UV}}$.

Thus we conclude there is a trend with increasing $\mathcal{R}_{\lambda
L\lambda}$ for more UV luminous flares, especially for the largest
($E_{\text{UV}}>10^{32}$ erg) flares. However, we do not find evidence for a
similar trend with $\mathcal{R}_{E}$.

Table 3: Kernel Density Estimates |  $\mathcal{R}_{E}$ median |  16th percentile |  84th percentile |  $\mathcal{R}_{\lambda L\lambda}$ median |  16th percentile |  84th percentile
---|---|---|---|---|---|---
$\log(E_{UV}<31)$ |  0.501 |  0.277 |  0.734 |  0.530 |  0.219 |  0.833
$31\leq\log(E_{UV})<32$ |  0.497 |  0.289 |  0.739 |  0.535 |  0.242 |  0.829
$\log(E_{UV})\geq 32$ |  0.518 |  0.382 |  0.662 |  0.628 |  0.395 |  0.865
K |  0.494 |  0.320 |  0.666 |  0.443 |  0.178 |  0.739
Early M |  0.493 |  0.318 |  0.652 |  0.541 |  0.301 |  0.743
Fully Convective |  0.509 |  0.313 |  0.725 |  0.577 |  0.284 |  0.863
Table 4: Fits to FUV/NUV trends with respect to G-RP color and flare UV energy.  |  Best-fit $m$ |  Best-fit $b$ |  16th percentile $m$ |  84th percentile $m$ |  16th percentile $b$ |  84th percentile $b$ |  % $m$ $\leq$ 0
---|---|---|---|---|---|---|---
Least-squares fit
$\mathcal{R}_{\lambda L\lambda}$ vs. G-RP |  0.14 |  0.56 |  0.05 |  0.23 |  0.55 |  0.58 |  16.80
$\mathcal{R}_{\lambda L\lambda}$ vs. UV energy |  0.08 |  0.54 |  0.05 |  0.10 |  0.52 |  0.56 |  10.04
$\mathcal{R}_{E}$ vs. G-RP |  0.06 |  0.49 |  0.00 |  0.13 |  0.48 |  0.51 |  16.25
$\mathcal{R}_{E}$ vs. UV energy |  0.03 |  0.49 |  0.01 |  0.05 |  0.47 |  0.50 |  14.65
Theil-Sen fit |  |  |  |  |  |  |
$\mathcal{R}_{\lambda L\lambda}$ vs. G-RP |  0.24 |  0.56 |  0.13 |  0.30 |  0.55 |  0.58 |  10.30
$\mathcal{R}_{\lambda L\lambda}$ vs. UV energy |  0.04 |  0.56 |  0.02 |  0.07 |  0.54 |  0.58 |  13.80
$\mathcal{R}_{E}$ vs. G-RP |  0.09 |  0.50 |  0.00 |  0.16 |  0.49 |  0.52 |  18.95
$\mathcal{R}_{E}$ vs. UV energy |  0.00 |  0.50 |  0.00 |  0.03 |  0.48 |  0.52 |  31.15

##  4 Potential Impacts on Habitability

We have demonstrated that uniformly selected field stars exhibit flares that
are FUV luminous and not well-represented by a constant $9\,000-10\,000$ K
blackbody. This finding holds for observed fluxes, integrated energy and peak
luminosity. In this section, we discuss some implications of this finding.

Ultraviolet-C radiation (UVC, $\sim$2000-2800 Å) is expected to drive
prebiotic chemistry (todd18; rimmer18) and ozone depletion (segura10;
tilley19). If blackbody temperatures during flares indeed exceed $9\,000$ K,
then flares deliver higher levels of NUV flare radiation than assumed when
characterizing the impact of flaring on abiogenesis and atmospheric ozone
depletion.

Optical studies of flares and their potential impact on exoplanet habitability
typically assume a constant $9\,000$ K blackbody for flares, which impacts
their calculation of flare energy (schmidt19; rodriguez18; gunther20;
zeldes21; bogner21). zeldes21 selected a sample of five superflaring stars and
found that none fell into either zone.

In a sample of 1228 flaring stars from TESS, gunther20 found eight percent of
sources to display sufficient flaring rates and energies to fall into the
ozone depletion zone, and only one percent to fall in the abiogenesis zone. If
instead these flares are well-represented by $13\,500$ K blackbody SED, the
number of stars with sufficient flaring activity to fall in the ozone
depletion zone would increase by 60%. In particular, the number of early M
stars (M0-M4) in the ozone depletion zone would increase by 270%.

Characterizing the impact of higher flare temperatures on prebiotic chemistry
is less straightforward. rimmer18 assumes an Earth-like atmosphere which will
absorb wavelengths shorter than 2100 Å, and focuses on NUV radiation
($2000-2800$ Å) as a driver of prebiotic chemistry. rimmer18 estimates the
total number of photons between $2000-2800$ Å reaching a planet’s surface by
assuming the AD Leo flare spectrum in their model is flat at wavelengths
shorter than the U band. We found this is likely not the case in the FUV
($1350-1750$ Å) and NUV ($1750-2750$ Å). This calls into question their
conclusion that the number of NUV photons reaching a planet’s surface can be
inferred from the U-band flare energy.

In future work (Berger et al. 2024, in prep) we will directly compute
population-averaged NUV flare rate distributions, directly probing the
wavelength ranges thought to be important for the formation of complex
molecules. Naively scaling a blackbody in TESS from 9000 K to $13\,500$ K will
result in an increase in NUV photons by a factor 3.86.

UV radiation from stars impacts the atmospheric composition of orbiting
planets. Large ratios of FUV/NUV radiation as seen on M dwarfs may produce
sufficient abiotic atmospheric oxygen to produce a false biosignature (tian14;
harman15). These studies characterize UV radiation in steady state and do not
account for flaring events. Additional modeling work should be undertaken to
investigate if the stronger FUV flux during flares that we observe in this
work can significantly alter atmospheric oxygen abundances and thus affecting
interpretation of exo-atmospheric signatures. A few FUV spectra of flares
exist (hawley91; france16; loyd18a; froning19) but the short timescales of
flares make obtaining these observations difficult.

We identified no observations with FUV/NUV flux ratios above the 4.33
$\mathcal{R}_{E}$ asymptote in our color-temperature curve (Figure 1) above
which the overall FUV emission could not be attributed solely to blackbody
emission. Thus for the stars in our sample, we cannot determine the extent to
which line emission contributes to the total FUV emission we observe. To
characterize the respective contributions of line and continuum emission to
overall FUV flare emission will require obtaining FUV spectra of a population
of stars in quiescence and flare. Spectral observations of a large solar flare
suggest that line emission can indeed dominate the overall UV flare emission
(simoes19). Our results highlight the importance of simultaneous observations
of flares across the electromagnetic spectrum in order to fully capture the
emission processes involved (e.g., paudel21).

Looking forward, two missions planned for launch in 2026 may advance our
understanding of stellar flares in the ultraviolet. The Ultraviolet Transient
Astronomy Satellite (ULTRASAT; ultrasat23) plans to observe $>10^{5}$ flaring
and variable stars in the NUV ($2300-2900$ Å), sensitive to $<23.5$ AB mag.
ULTRASAT will provide insight into the levels of NUV radiation from flares
that may drive prebiotic chemistry or atmospheric ozone depletion. However,
the single-filter design of ULTRASAT will not allow any temperature estimates.
Additionally, the Monitoring Activity of Nearby sTars with uv Imaging and
Spectroscopy (MANTIS; mantisprop22) mission will work alongside JWST and may
be a better probe of flare energetics. MANTIS will observe cool stars in the
extreme-ultraviolet ($100-900$ Å), far-ultraviolet ($900-2000$ Å), near-
ultraviolet ($2000-3200$ Å), and visible ($3200-10\,000$ Å) wavelengths,
providing a detailed estimate of the color temperature. MANTIS will provide
the first glimpse at many stars in the EUV, and may shed light on the
proportion of flare radiation originating from continuum versus UV line
emission.

##  5 Conclusions

In this work, we have identified 182 flares on 158 stars with simultaneous
observations in the NUV and FUV from the _GALEX_ space telescope. We select
our targets from the Gaia Catalogue of Nearby Stars which is complete down to
spectral type M8 within our survey volume of 100 pc. Thus this study suffers
from minimal selection basis. We have computed color temperatures at each
epoch of the flares in our sample using a derived relation between FUV/NUV
flux and blackbody temperature. Our main results are as follows.

  1. 1.

A constant $9\,000$ K blackbody underpredicts the FUV emission for 98% of
flares in our sample.

  2. 2.

We find that the FUV/NUV ratio at peak appears to positively correlate ($\sim
2\sigma$ significance) both with total UV flare energy and with increasing
$G-RP$ color of the host star.

Future work (Berger in prep.) will use this _GALEX_ flare sample and an
injection and recovery analysis to compute the UV flare rate distribution and
produce a UV flare model.

## Acknowledgements

VLB acknowledges support from the Research Experience for Undergraduates
program at the Institute for Astronomy, University of Hawaii-Manoa funded
through NSF grant #2050710. VLB thanks the Institute of Astronomy for its
hospitality over the course of this project. VLB was also supported by a
Churchill Scholarship. JTH was supported by NASA grant 80NSSC21K0136. BJS is
supported by NSF grants AST-1907570, AST-1920392 and AST-1911074. JvS
acknowledges support from NASA grant 80NSSC22K0293. D.H. acknowledges support
from the Alfred P. Sloan Foundation and the Australian Research Council
(FT200100871). We thank Dr. Michael Liu and Dr. Allison Youngblood for useful
discussions.

Software: matplotlib (hunter07), numpy (harris20), pandas (mckinney10), scipy
(scipy), astropy (astropy:2013; astropy:2018; astropy:2022).

## Data Availability

This manuscript only uses publicly available data from _GALEX_ and _Gaia_. All
derived data products will be made publicly available upon publication.


(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext:  Department of Electrical & Computer Engineering
University of Washington, Seattle WA, USA
11email: {hwhuang, cycyang, wchai, zyjiang<EMAIL_ADDRESS>

# Exploring Learning-based Motion Models in Multi-Object Tracking

Hsiang-Wei Huang  0009-0009-2474-8869 Cheng-Yen Yang  0009-0004-2631-6756
Wenhao Chai  0000-0003-2611-0008
Zhongyu Jiang  0000-1234-1234-5555 Jenq-Neng Hwang  0000-0002-8877-2421

###### Abstract

In the field of multi-object tracking (MOT), traditional methods often rely on
the Kalman Filter for motion prediction, leveraging its strengths in linear
motion scenarios. However, the inherent limitations of these methods become
evident when confronted with complex, nonlinear motions and occlusions
prevalent in dynamic environments like sports and dance. This paper explores
the possibilities of replacing the Kalman Filter with various learning-based
motion model that effectively enhances tracking accuracy and adaptability
beyond the constraints of Kalman Filter-based systems. In this paper, we
proposed MambaTrack, an online motion-based tracker that outperforms all
existing motion-based trackers on the challenging DanceTrack and SportsMOT
datasets. Moreover, we further exploit the potential of the state-space-model
in trajectory feature extraction to boost the tracking performance and
proposed MambaTrack+, which achieves the state-of-the-art performance on
DanceTrack dataset with 56.1 HOTA and 54.9 IDF1.

###### Keywords:

Multi-Object Tracking State-space Model

##  1 Introduction

Figure 1: We demonstrate the challenge wherein the Kalman filter can fail to
accurately predict the target’s location due to fast and irregular movement in
the DanceTrack dataset. The left visualized results compare the predicted
bounding boxes of our proposed MambaTrack and the state-of-the-art method
ByteTrack on the dancetrack0005 sequence from the validation split, with the
blue and green bounding boxes denoting the predicted bounding box of the
target, and the red bounding box is the ground truth. The right figure
compares the IoU between the prediction and ground truth bounding boxes; our
Mamba model consistently demonstrates better location prediction accuracy
compared to the Kalman filter.

Multi-object tracking, a fundamental task in computer vision, aims at tracking
multiple target objects in a video stream. Among all the different trackers,
motion-based trackers [58, 5, 3] are especially popular and can be easily
applied to many real-world scenarios due to their simplicity and lower
computational costs. Motion-based trackers, like most of the state-of-the-art
trackers [1, 58, 5, 12, 47, 34, 3], heavily rely on the prediction of a motion
model during tracking. Most of these trackers adopt the Kalman filter [26] as
their motion model, which demonstrates effectiveness when tracked objects have
regular and linear movements, such as pedestrians [29, 11]. However, these
trackers struggle to perform well on datasets [38, 10] with objects having
diverse and irregular motions. Several studies [5, 23] show that the linear
motion assumption of the Kalman filter breaks down when tracking objects in
dancing or sports scenarios, where the movement is nonlinear. For this reason,
we aim to explore stronger motion models that can provide more robust and
adaptive motion predictions to improve the performance of Kalman filter-based
trackers.

In this work, we seek to explore learning-based motion models as a replacement
for the Kalman filter. Unlike the Kalman filter, which predicts an object’s
location using a linear motion assumption, the learning-based motion model
adaptively predicts the tracklet’s location in the next timestamp by
leveraging tracklet information from previous frames. This enables the motion
model to be more adaptive to each dataset’s motion pattern, thus making more
reliable predictions even when the target’s motion is irregular or nonlinear.
Among different types of learning-based models, sequence models [19, 17] are
particularly suitable as motion models in object-tracking tasks. Sequence
models are designed to handle sequential information, which can intuitively be
used as a motion model for predicting a tracklet’s location based on past
tracklet information. Several different learning-based sequence models are
implemented and used as the motion models in our experiments, including
traditional Recurrent Neural Networks (RNN) [33] and Long Short-Term Memory
(LSTM) networks [20]. More advanced architectures, including attention-based
models like the Transformer [43] and structured state-space model like Mamba
[17] are also explored in our paper.

In this paper, we first propose MambaTrack, a pioneering online MOT approach
that leverages exceptional context reasoning capabilities. By utilizing
bounding box predictions within sequentially tracked targets and enhancing
trajectory representations, MambaTrack significantly improves tracking
outcomes, achieving state-of-the-art performance across challenging datasets
such as DanceTrack [38] and SportsMOT [10]. It also maintains comparable
performance on more generalized datasets like MOT17 [29], relative to other
motion-based methods. Notably, our work represents the first successful
integration of such a state-space model into the Multi-Object Tracking (MOT)
task, distinguished by rapid training and inference speeds.

Building upon MambaTrack, we further propose MambaTrack+, which exploits the
power of the state-space model to extract trajectory features to boost the
tracking performance. The proposed MambaTrack+ further improves the
performance of MambaTrack and achieves 56.1 HOTA on DanceTrack [38] and 71.3
HOTA on SportsMOT [10], significantly outperforming previous Kalman filter-
based and learning-based methods.

##  2 Related Works

###  2.1 Tracking by Detection

Multi-object tracking aims to detect and track every target object in a video
stream [8, 31, 53, 36]. With the rapid development of object detectors [13,
16, 44, 6], most state-of-the-art trackers [1, 58, 5, 34, 12] adopt the
tracking-by-detection paradigm. This approach involves applying an object
detector to image frames, with the obtained detections then utilized for
tracking. Centernet [13] is widely used by several tracking-by-detection
methods [48, 46, 45, 41] due to its efficiency. Recently, YOLOX [16] has been
adopted by most state-of-the-art tracking-by-detection methods [1, 58, 5, 34,
12] for its superior detection performance.

Many tracking-by-detection methods [3, 47, 58, 5] use the Kalman filter [26]
as a motion model to predict the location of tracklets in the next frame. The
Intersection over Union (IoU) between the predicted boxes and detection boxes
is used as the similarity metric for association. However, previous works [5]
have identified several limitations of the Kalman filter, including
sensitivity to state noise and temporal error magnification. To address these
limitations, OC-SORT proposes an Observation-centric Re-Update (ORU) strategy
to reduce the accumulated error of the Kalman filter by backchecking and re-
updating its parameters when a tracklet is re-activated. However, ORU is only
effective when a tracklet is correctly re-identified after a short period of
being untracked. In scenarios of diverse and large motion, the Kalman filter’s
sensitivity to state noise can lead to a significant shift in final position
estimation after just a few frames of being untracked. Thus, a fundamental
solution to this problem is to incorporate a learning-based motion model that
can adaptively predict an object’s location based on the target’s historical
tracklet information.

###  2.2 Motion Model for Multi-Object Tracking

Most of the tracking by detection methods [3, 47, 58, 5, 7] incorporate motion
model for association.

However, even with the recent improved performance on the public MOT datasets
[29, 11], several challenges including the tracking of objects with similar
appearance and the motion modeling of objects with diverse motion [38, 10]
still remain unsolved. The recent emergence of end-to-end transformer-based
trackers [55, 60] introduce the motion and appearance joint learning tracking,
which demonstrates better performance on challenging datasets like DanceTrack
[38], but these methods share a common drawback of higher computational cost
and complex training procedure. However, even the recent benchmarks and
challenges focusing on targets with diverse motion, most of the tracking by
detection method [58, 1, 59, 47, 3, 12] still in favor of incorporating the
Kalman filter [26] as motion model. With the linear motion assumption of the
Kalman filter, these methods inevitably fail on datasets like DanceTrack [38]
and SportsMOT [10] with objects having diverse and irregular motion. For this
reason, we aim to explore different learning-based motion models and offer
more robust alternatives to the tracking-by-detection paradigm under
challenging tracking scenarios.

###  2.3 Sequence Model

####  2.3.1 Recurrent Neural Network.

Recurrent neural networks (RNNs) work sequentially, given an input sequence,
and propagate the hidden states through the sequence to produce a prediction.
Previous work [30] utilizes an RNN to control the motion prediction and state
update of the tracklet. Long short-term memory (LSTM) [20] is a type of
recurrent neural network architecture designed to capture long-term
dependencies and mitigate the vanishing gradient problem of RNN. They are
particularly effective in modeling sequential data where long-range
dependencies are important. In the context of tracklet location prediction,
LSTM networks can be used to capture complex patterns and temporal
dependencies of the tracklets, allowing for accurate prediction of future
locations based on past tracklets.

####  2.3.2 Transformer.

The transformer architecture [43] revolutionizes sequence modeling tasks by
introducing the self-attention mechanism. Unlike RNNs and LSTMs, transformers
do not rely on recurrent connections, instead, it process the entire input
sequence in parallel using self-attention mechanisms, which enable a more
efficient way to capture global information. Transformers have achieved state-
of-the-art performance in various natural language processing and computer
vision tasks and have also been successfully applied to sequence modeling
tasks in other domains. In our work, we explore the use of transformer
architecture for tracklet location prediction, leveraging their ability to
model long-range dependencies to capture complex motion patterns in sequential
data.

####  2.3.3 Mamba.

Building upon the foundations laid by the state-space model (SSM) [19], a
novel sequence model architecture, named Mamba [17], has been introduced.
Mamba represents an input-dependent and hardware-efficient SSM, which achieves
superior performance and efficiency compared to the transformer. Different
from previous SSM [19], Mamba possesses the ability to efficiently select data
in an input-dependent manner with the selection mechanism. Moreover, Mamba
incorporates a hardware-aware algorithm that optimizes computational
efficiency. It demonstrates linear scaling in sequence length during both
training and inference, facilitating fast training and inference processes,
its ability for long range sequence modeling also makes it a potential choice
for motion model in the object tracking task.

Figure 2: (Left) The MambTrack+ architecture involves feeding a sequence of
bounding boxes from the same track into a linear projection layer sequentially
for motion modeling. Predictions and embeddings are generated by the model,
with the hidden state $h_{t}$ updated at each time frame. These predictions
are utilized for detecting and matching existing tracks, while trajectory
embeddings facilitate tracklet merging. (Right) The detailed structure of the
Mamba block used in our framework.

##  3 Method

###  3.1 Task Definition

Given a tracklet’s of the past $n$ frames
$\\{B_{t-n},...,B_{t-1}\\}\in\mathbb{R}^{n\times[x,y,w,h]}$, the motion model
aims at predicting the tracklet’s location in the next time stamp
${B_{T}}\in\mathbb{R}^{1\times[x,y,w,h]}$ based on the information from the
past $n$ frames bounding boxes. The predicted bounding box will further be
used for association with the detections $X_{t}$ from the next timestamp $t$.

###  3.2 Motion Models

We propose to implement the motion model using the original Mamba [17] block,
with an extra linear projection layer, which transforms the bounding box to a
higher dimension and a local multi-layer perception for predicting the
tracklet’s next location based on a continuous system with parameters $A,B,C$.

$\displaystyle y(t)=C\cdot h(t)$ (1) $\displaystyle h(t)=A\cdot h(t-1)+B\cdot
x(t).$ (2)

The Selective SSM (state-space model) is essentially the discrete
representation $\bar{A}$, $\bar{B}$ of the continuous system $A$, $B$ by
applying ZOH (zero-order hold) discretization formulation with a timescale
parameter $\Delta$, i.e.,

$\displaystyle\bar{A_{t}}$ $\displaystyle=\exp(\Delta
A)=1-\sigma(\text{Linear}(x_{t}))$ (3) $\displaystyle\bar{B}_{t}=(\bar{A}$
$\displaystyle)^{-1}(\exp(\Delta A)-I)\cdot\Delta
B=\sigma(\text{Linear}(x_{t})),$ (4)

where the discrete output $y_{t}$ and hidden state $h_{t}$ from Eq. 1 and 2
as:

$\displaystyle y_{t}=C\cdot h_{t}$ (5) $\displaystyle h_{t}=\bar{A_{t}}\cdot
h_{t-1}+\bar{B_{t}}\cdot x_{t},$ (6)

then, by incorporating selective State Space Models (SSMs) into a streamlined
end-to-end neural network architecture that foregoes traditional attention
mechanisms as described in the original work, we can significantly increase
the inference speed while offering linear scalability with sequence length.

Similar to the implementation of the most recurrent models, we compute and
propagate the hidden states and finally produce the tracklet’s next location
in the final output by feeding the output to the prediction head, which is a
simple multilayer perceptron $\textbf{MLP}_{pred}$:

$\mathbf{Y}_{t}=\textbf{MLP}_{pred}(y_{t}).$ (7)

To our knowledge, this is the first method that incorporates a state-space
model for motion modeling in an MOT task. We incorporate this motion model
with the association method BYTE [58] and named this proposed architecture
MambaTrack.

###  3.3 Motion Prediction

The prediction head described in Eq. 7 aims to predict the tracklet’s location
in the next frame. To enable accurate bounding box prediction, we apply
Generalized IoU loss (GIoU), $\mathcal{L}_{\text{giou}}$, [32] and MSE loss,
$\mathcal{L}_{\text{mse}}$, between the predicted bounding box and the ground-
truth bounding box. $\mathcal{L}_{\text{giou}}$ is first used in object
detection tasks. It enhances the traditional IoU metric by considering the
spatial layout of bounding boxes by penalizing the discrepancy between the
area of the union and the intersection of predicted and ground truth bounding
boxes. Furthermore, it improves the precision of bounding box predictions and
better aligns the objective of the motion modeling task. Meanwhile,
$\mathcal{L}_{\text{mse}}$ calculates the average squared difference between
the corresponding coordinates of predicted and ground truth bounding boxes. By
minimizing these losses, the model learns to generate bounding box predictions
closer to the ground truth in terms of spatial coordinates. The final motion
prediction loss is as follows:

$\mathcal{L}_{\text{pred}}=\mathcal{L}_{\text{giou}}+\mathcal{L}_{\text{mse}}.$
(8)

###  3.4 Association

To show learning-based motion model can conduct better motion modeling than
the Kalman filter, we incorporate the learning-based motion model, Mamba, with
the data-association method BYTE proposed in ByteTrack [58]. Unlike our
MambaTrack, which uses Mamba as the motion model, ByteTrack incorporates the
Kalman filter to predict tracklet location in the next frame. The association
is conducted between the predicted location and the detection using the
Hungarian algorithm. Same as ByteTrack, we use YOLO-X [16] as our object
detector to evaluate public benchmarks [29, 38].

###  3.5 MambaTrack+: Unleashing the Power of Mamba

####  3.5.1 Motion Pattern in Hidden State.

A strong property of the structured state-space model is its ability for long-
context reasoning [17, 18, 19], which has demonstrated superior performance
over transformer model on datasets like long-range arena [40]. The reason for
Mamba and other structured state-space models to achieve superior long-context
reasoning is their ability to preserve previous input signals in the hidden
state and propagate over time. This property enables Mamba to capture the
motion patterns and predict the next location based on the object’s historical
trajectory, resulting in significant performance gains in online tracking
scenarios. During the process of motion modeling, the hidden states of Mamba
contain rich motion patterns of the tracklet. To further exploit the ability
of Mamba to preserve the tracklet’s motion pattern in the hidden state, we
also propose MambaTrack+, which aims at unleashing the power of Mamba by
extracting the trajectory feature in the hidden state and further connecting
tracklets with similar trajectory features to boost the tracking performance.

####  3.5.2 Trajectory Representation.

MambaTrack+ incorporates an extra trajectory embedding head in the model which
takes the result from the mamba block $y_{t}$ as input into another trainable
multilayer perceptron $\textbf{MLP}_{emb}$ to get the trajectory
representation $\mathbf{f}_{t}$. The trajectory head captures global
representations of the tracklet from the trajectory information from the
hidden states, and thus contains rich trajectory motion patterns and position
information that can be further used for tracklet level association as a post-
processing approach to connect those fragment tracklets during the tracking
process.

$\mathbf{f}_{t}=\textbf{MLP}_{emb}(y_{t})$ (9)

The training objective of trajectory representations is to minimize the
embedding features’ cosine distance between the same tracklet while maximizing
between different ones. We applied the cosine embedding loss
$\mathcal{L}_{\text{cos}}$ to achieve this objective. A pair of trajectories
from tracklet $i$ and tracklet $j$ will be sampled during each loss backward,
and the cosine embedding loss $\mathcal{L}_{\text{cos}}$ will be calculated
following:

$\mathcal{L}_{\text{cos}}(i,j)=\begin{cases}1-\text{cos}(\mathbf{f}_{i},\mathbf{f}_{j}),&\text{if
i $=$ j}\\\ \max(0,\text{cos}(\mathbf{f}_{i},\mathbf{f}_{j})),&\text{if i
$\neq$ j}\end{cases}$ (10)

where $\mathbf{f}_{i}$ represents the global head’s predicted features by
forwarding the trajectory sampled from tracklet $i$ through the mamba blocks.

We jointly train the bounding box prediction head and global features head by
adding the loss together. The end-to-end loss is thus defined as:

$\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{pred}}+\mathcal{L}_{\text{cos}}.$
(11)

####  3.5.3 Tracklet Merging.

Many previous works proposed using a deep learning-based model [12, 56, 37,
54, 57, 22, 52] to conduct tracklet merging based on the appearance or motion
pattern similarity of tracklet pair. A prevalent approach among these works
involves employing Siamese networks for this task. While these tracklet
merging networks can boost the tracking performance by connecting the fragment
trajectories from the same identity, the stand-alone Siamese network needs to
compute the similarity between each tracklet pair and further introduces extra
$O(N^{2})$ computational cost, with $N$ denoting the number of tracklets.
Different from these works, our proposed MambaTrack+ extracts the motion
pattern of tracklet in the tracking process using the same model for motion
prediction, which prevents the need for training another separate tracklet
merging network. Furthermore, the Siamese network requires one forward pass
for each tracklet pair, while MambaTrack+ extracts the trajctory feature
during tracking, which largely reduces the extra computational cost during the
merging process. After the tracking is finished, the trajectory feature of
each tracklet will be compared using cosine similarity, we further use
hierarchical clustering to merge the tracklet with similar trajectory features
and boost the tracking performance.

##  4 Experiments

###  4.1 Datasets

####  4.1.1 MOT17 [29].

The MOT17 dataset is a widely used benchmark for multiple object tracking in
video sequences. It consists of 14 real-world video data captured from various
scenes, such as crowded pedestrian areas and urban traffic from static and
moving cameras. The dataset is split into training and testing following the
original benchmark’s setting. As in the half-val setting, training involves
utilizing the first half for training data and the second half for validation
data.

####  4.1.2 DanceTrack [38].

DanceTrack is a challenging dataset that contains targets with similar
appearance and diverse motion from 100 group dancing videos, presenting a more
complex dataset compared to MOT17, which primarily features pedestrians with
linear movement patterns. The non-linear motion can serve as a significant
testing benchmark for the robustness of the motion model.

####  4.1.3 SportsMOT [10].

SportsMOT is a recent dataset that focuses on MOT in sports scenarios [62], it
is challenging due to the fast-paced and erratic movements exhibited by
athletes, often involving abrupt changes in direction and velocity. It
contains 240 videos collected from multiple sports including basketball,
volleyball, and football where only the players on the field are being
annotated.

###  4.2 Implementation Details

####  4.2.1 Detector.

For fair comparison on MOT17 and DanceTrack datasets, we use the same
detections obtained from ByteTrack’s YOLOX model, which is trained on multiple
datasets, including MOT17 dataset, Crowdhuman [35], and ETHZ [14]. For the
SportsMOT [10] dataset, we train a YOLOX detector [16] on the SportsMOT
training set following the same training recipe of ByteTrack [58].

####  4.2.2 Trajectory Data.

To train a learning-based motion model, the training data is collected from
the trajectories in different multi-object tracking datasets, including MOT17,
DanceTrack and SportsMOT. In the training process, the trajectories are
randomly selected with the length between $(2,n)$, where $n$ is the max input
length. We use padding to fill the tracklets length to $n$ when the sampled
tracklets have a length smaller than $n$. The next bounding box location of
these sampled tracklets is the ground-truth bounding box.

####  4.2.3 Training/Settings.

We trained our MambaTrack and MambaTrack+ using Adam optimizer with $0.0001$
initial learning rate for 500 epochs with a batch size of $32$ on all of our
experiments. We implemented the motion models using $2$ mamba blocks, with a
hidden dimension of $64$, and expansion factor of $2$. For tracklet merging in
MambaTrack +, a temporal distance threshold of 50 frames and a spatial
distance threshold of 50 pixels are employed to avoid merging unreasonable
tracklet pairs. All the experiments are conducted on a single NVIDIA RTX 4080
GPU.

###  4.3 Evaluation Metrics

We utilize commonly used tracking metrics, including HOTA (Higher Order
Tracking Accuracy) [28] for its comprehensive evaluation of both detection
(DetA) and association accuracy (AssA); and CLEAR metrics [2], where IDF1 and
MOTA serve as the standard benchmark for tracking performance across various
scenarios. Together, these metrics offer a robust assessment of our tracking
algorithms’ performance.

###  4.4 Benchmark Evaluation

Table 1: Performance on MOT17. We compared our performance with different
Kalman filter-based and learning-based methods using the official YOLO-X
detections.

Methods | HOTA | IDF1 | MOTA
---|---|---|---
Kalman filter-based: |  |  |
SORT [3] | 34.0 | 39.8 | 43.1
FairMOT [59] | 59.3 | 72.3 | 73.7
ByteTrack [58] | 63.1 | 77.3 | 80.3
OCSORT [5] | 63.2 | 77.5 | 78.0
Learning-based: |  |  |
CenterTrack [63] | 52.2 | 64.7 | 67.8
TraDes [49] | 52.7 | 63.9 | 69.1
QDTrack [15] | 53.9 | 66.3 | 68.7
PermaTrack [42] | 55.5 | 68.9 | 73.8
TransTrack [39] | 54.1 | 63.9 | 74.5
TransCenter [51] | 54.5 | 62.2 | 73.2
MeMOT [4] | 56.9 | 69.0 | 72.5
MOTR [55] | 57.2 | 68.4 | 71.9
GTR [64] | 59.1 | 71.5 | 75.3
MotionTrack [50] | 60.9 | 73.5 | 76.5
MambaTrack | 60.5 | 72.3 | 78.2
MambaTrack+ | 61.1 | 73.9 | 78.1

####  4.4.1 MOT17.

Table 1 compares our proposed method with other state-of-the-art methods on
the MOT17 test set. We mainly compare our proposed MambaTrack with several
learning-based methods, including Global Tracking Transformers [64] and MOTR
[55]. With a much lower computational cost compared with other learning-based
methods [55, 64], MambaTrack surpasses other learning-based methods and
achieves comparable results with those state-of-the-art Kalman filter-based
methods on the MOT17 dataset. Furthermore, the proposed MambaTrack+ further
boosts the performance in HOTA by 0.6% and IDF1 by 1.6%, demonstrating the
effectiveness of the proposed tracklet merging method. However, we contend
that the MOT17 dataset is insufficient to adequately showcase the tracking
capabilities of learning-based methods. The motion patterns of objects within
MOT17 are overly simplistic, potentially resulting in Kalman filter
implementations outperforming learning-based motion models. To fully broaden
our investigation, we further conducted experiments on two much more
challenging datasets: DanceTrack [38] and SportsMOT [10].

Table 2: Performance on DanceTrack. We compared our performance with different Kalman filter-based and learning-based trackers, note that most of the learning-based methods [15, 49, 55, 39, 64] also incorporate appearance during tracking, while our proposed MambaTrack is a pure motion-based tracker.  Methods | HOTA | DetA | AssA | MOTA | IDF1
---|---|---|---|---|---
Kalman filter-based |  |  |  |  |
FairMOT [59] | 39.7 | 66.7 | 23.8 | 82.2 | 40.8
SORT [3] | 47.9 | 72.0 | 31.2 | 91.8 | 50.8
DeepSORT [3] | 45.6 | 71.0 | 29.7 | 87.8 | 47.9
ByteTrack [58] | 47.3 | 71.6 | 31.4 | 89.5 | 52.5
OC-SORT [5] | 54.6 | 80.4 | 40.2 | 89.6 | 54.6
Learning-based |  |  |  |  |
CenterTrack [63] | 41.8 | 78.1 | 22.6 | 86.8 | 35.7
TraDes [49] | 43.3 | 74.5 | 25.4 | 86.2 | 41.2
QDTrack [15] | 45.7 | 72.1 | 29.2 | 83.0 | 44.8
TransTrack [39] | 45.5 | 75.9 | 27.5 | 88.4 | 45.2
MOTR [55] | 54.2 | 73.5 | 40.2 | 79.7 | 51.5
GTR [64] | 48.0 | 72.5 | 31.9 | 84.7 | 50.3
MotionTrack [50] | 52.9 | 80.9 | 34.7 | 91.3 | 53.8
MambaTrack | 55.5 | 80.8 | 38.3 | 90.1 | 53.9
MambaTrack+ | 56.1 | 80.8 | 39.0 | 90.3 | 54.9

####  4.4.2 DanceTrack.

We compare MambaTrack with several different state-of-the-art methods on the
DanceTrack test set in Table 2. MambaTrack outperforms ByteTrack [58] by a
large margin of $8.2\%$ in HOTA when using the same detections, demonstrating
the power of learning-based motion model over Kalman filter under a more
challenging tracking scenario. Furthermore, MambaTrack+ also demonstrates the
ability to boost the performance, improving HOTA by $0.6\%$ and IDF1 by
$1.0\%$.

Table 3: Performance on SportsMOT. Star symbol ∗ denotes using the same detections generated by YOLOX. All the methods were trained using the SportsMOT training set.  Methods | HOTA | DetA | AssA | IDF1 | MOTA
---|---|---|---|---|---
Kalman filter-based |  |  |  |  |
FairMOT [59] | 49.3 | 70.2 | 48.0 | 62.3 | 86.4
ByteTrack∗ [58] | 62.0 | 77.0 | 49.9 | 68.7 | 93.9
OC-SORT∗ [5] | 70.2 | 85.9 | 57.4 | 70.4 | 93.9
Learning-based |  |  |  |  |
GTR [64] | 54.5 | 64.8 | 45.9 | 55.8 | 67.9
CenterTrack [63] | 62.7 | 82.1 | 48.9 | 60.0 | 90.8
QDTrack [15] | 60.4 | 77.5 | 47.2 | 62.3 | 90.1
TransTrack [39] | 68.9 | 82.7 | 57.5 | 71.5 | 92.6
MambaTrack∗ | 70.4 | 86.7 | 57.2 | 69.5 | 94.7
MambaTrack+∗ | 71.3 | 86.7 | 58.6 | 71.1 | 94.9

####  4.4.3 SportsMOT.

We trained MambaTrack using the SportsMOT training set and compared it with
other state-of-the-art methods on the SportsMOT test set. MambaTrack achieves
the highest HOTA performance among all the learning-based methods and
outperforms ByteTrack by 8.2% HOTA and 7.3% AssA under the same association
method and detector setting with the works we compared in Table 3.

###  4.5 Ablation Studies

####  4.5.1 Different Motion Models.

To evaluate the performance of our method in complex, non-linear motion
scenarios relative to existing motion models, we conduct a comparative
analysis that includes both linear and non-linear prediction techniques. The
analysis in Table 4 covers linear models, such as Intersection over Union
(IoU) and the Kalman Filter, as well as non-linear approaches, including
learning-based Multilayer Perceptrons (MLP), Recurrent Neural Networks (RNN
and LSTM), and a basic Transformer model. To ensure a fair and insightful
comparison, all training and tracking parameters are standardized across the
models. This methodical approach allows us to clearly understand the impact of
different motion modeling techniques on the effectiveness of our method. The
learning-based motion models, evaluated on DanceTrack [38], demonstrate
superior performance compared to traditional methods. MLP exhibits a 0.9%
improvement in HOTA over Kalman filter, while Mamba achieves a 4.3% higher
HOTA. This suggests the efficacy of deep learning approaches in motion
modeling tasks, with the potential for further improvements in predictive
accuracy and efficiency.

Table 4: Performance comparison of different motion models on DanceTrack validation set. We implemented most of the motion models (besides MLP) with similar model size for fair comparison in motion modeling task. KF stands for Kalman filter.  Module |  $\\#$ Param |  HOTA | DetA | AssA | MOTA | IDF1
---|---|---|---|---|---|---
IoU | - | 44.7 | 79.6 | 25.3 | 87.3 | 36.8
KF | - | 47.8 | 74.0 | 31.0 | 88.2 | 48.3
MLP | 6K | 48.7 | 77.0 | 31.0 | 87.3 | 46.7
RNN | 0.2M | 50.0 | 77.4 | 32.5 | 87.4 | 47.8
LSTM | 0.2M | 50.4 | 77.2 | 33.0 | 87.3 | 48.7
Transformer | 0.2M | 51.8 | 77.6 | 34.8 | 87.6 | 50.5
Mamba | 0.2M | 52.1 | 77.6 | 35.1 | 87.4 | 49.2

Max Input Length. We conduct experiments to investigate the impact of varying
input lengths on the training and tracking of MambaTrack, while keeping all
other parameters constant on the MOT17, DanceTrack, and SportsMOT datasets.
The results as in Fig 3 indicate that our proposed method achieves better
performance with shorter input sequences, such as 5 frames ($\approx 0.25$
seconds), particularly on challenging datasets like DanceTrack, where the
subjects exhibit rapid and irregular movements. Conversely, longer input
sequences appear to be more effective in scenarios involving pedestrian and
sports player tracking. We believe this is attributed to two factors: 1.) The
frequent positional switch in DanceTrack results in frequent ID switch, where
extending the input length causes bounding boxes of a tracklet to be mixed
with bounding boxes from different tracklets, thereby degrading performance.
2.) The rapid movements and lower FPS in DanceTrack pose challenges for
location prediction, as locations from too far back may not contribute
effectively to accurate localization, and might even introduce noise to the
model. In MOT17 and SportsMOT, given the easier tracking scenarios, we
hypothesize that MambaTrack is capable of encapsulating motion information
more effectively in these contexts, thereby enhancing performance with input
lengths of up to 25 frames ($\approx 1$ second).

Figure 3: Tracking Results of MambaTrack with different Max Input Length on MOT17, DanceTrack, and SportsMOT. Table 5: Ablation study on the effect of Different Matching Cost on MOT17, DanceTrack and SportsMOT, we conduct different matching cost including the $IoU$ along with $L1$ and $L2$.  Dataset |  $IoU$ |  $L1$ |  $L2$ | HOTA | IDF1 | AssA | MOTA | DetA
---|---|---|---|---|---|---|---|---
MOT17 | $\checkmark$ |  |  | 65.5 | 75.3 | 66.3 | 76.9 | 65.2
$\checkmark$ | $\checkmark$ |  | 66.9 | 77.2 | 66.4 | 76.9 | 67.9
$\checkmark$ |  | $\checkmark$ | 65.4 | 73.3 | 66.3 | 76.8 | 65.2
DanceTrack | $\checkmark$ |  |  | 55.5 | 53.9 | 38.3 | 90.1 | 80.8
$\checkmark$ | $\checkmark$ |  | 54.3 | 53.8 | 36.8 | 90.1 | 80.6
$\checkmark$ |  | $\checkmark$ | 54.7 | 53.3 | 37.5 | 90.1 | 80.5
SportsMOT | $\checkmark$ |  |  | 70.4 | 86.7 | 57.2 | 69.5 | 94.7
$\checkmark$ | $\checkmark$ |  | 70.3 | 86.9 | 57.1 | 69.3 | 94.8
$\checkmark$ |  | $\checkmark$ | 70.3 | 86.8 | 57.0 | 69.3 | 94.7

Matching Cost. We further investigate the impact and effect of leveraging
different matching costs in the association stage of MambaTrack on MOT17,
DanceTrack, and SportsMOT. We tested three different settings of cost
combinations: IoU only, IoU+L1, and IoU+L2. Note that we apply a weighted sum
for the two later combination cases. In Table 5, we present the best result
from a set of experiments that tested different weights for better comparison
purposes.

Our evaluation indicates a distinct performance improvement when using IoU+L1
as the matching cost on MOT17. Specifically, the combination of IoU and L1
metrics yields the highest HOTA score of $66.9\%$ and IDF1 of $77.2\%$.
However, on DanceTrack and SportsMOT, the integration of neither L1 nor L2
does not lead to the same level of performance improvement as observed in the
MOT17 dataset. This suggests that the characteristics of the DanceTrack and
SportsMOT dataset—which may include factors such as more complex object
motions, different object densities, or diverse scene dynamics—potentially
diminish the efficacy of combining these metrics.

####  4.5.2 Inference Speed.

Most of the previous learning-based methods [4, 55, 60, 15, 64] suffer from a
common drawback: their significant computational overhead and slower inference
speeds (typically lower than 10 FPS). However, MambaTrack, despite being a
learning-based approach, mitigates this issue by incorporating a lightweight
motion model. It achieves real-time inference speed (28.8 FPS on NVIDIA RTX
4080 GPU), akin to Kalman filter-based trackers.

##  5 Conclusions

Figure 4: More visualizations showcase a comparison between the bounding box
shapes, with the green boxes representing the Kalman filter from ByteTrack’s
prediction and the yellow boxes depicting the motion model’s prediction from
MambaTrack, the red boxes denotes the detection. Notably, MambaTrack
demonstrates superior accuracy in location prediction, particularly evident
when tracking objects with substantial movements, as observed in DanceTrack.

We present a comprehensive examination of the limitations inherent in
traditional Kalman Filter-based multi-object tracking (MOT) methodologies,
particularly under the challenging conditions posed by rapid and irregular
motion. By exploring a spectrum of learning-based motion models, we have
demonstrated a significant leap in tracking performance, adaptability, and
accuracy. Our proposed novel trackers, MambaTrack and MambaTrack+, leverage
the advanced capabilities of state-space model to effectively capture and
predict complex motion patterns, thereby addressing the critical shortcomings
of previous MOT systems. The proposed method achieves state-of-the-art
performance on various challenging public benchmarks, including DanceTrack and
SportsMOT datasets. We will further explore involving Re-ID and human pose [9,
25, 27, 61, 24, 21] feature in motion model as our future work.

## References

  * [1] Aharon, N., Orfaig, R., Bobrovsky, B.Z.: Bot-sort: Robust associations multi-pedestrian tracking. arXiv preprint arXiv:2206.14651 (2022)
  * [2] Bernardin, K., Stiefelhagen, R.: Evaluating multiple object tracking performance: The clear mot metrics. J. Image Video Process. 2008 (2008)
  * [3] Bewley, A., Ge, Z., Ott, L., Ramos, F., Upcroft, B.: Simple online and realtime tracking. In: 2016 IEEE international conference on image processing (ICIP). pp. 3464–3468. IEEE (2016)
  * [4] Cai, J., Xu, M., Li, W., Xiong, Y., Xia, W., Tu, Z., Soatto, S.: Memot: Multi-object tracking with memory. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8090–8100 (2022)
  * [5] Cao, J., Pang, J., Weng, X., Khirodkar, R., Kitani, K.: Observation-centric sort: Rethinking sort for robust multi-object tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9686–9696 (2023)
  * [6] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: European conference on computer vision. pp. 213–229. Springer (2020)
  * [7] Chaabane, M., Zhang, P., Beveridge, J.R., O’Hara, S.: Deft: Detection embeddings for tracking. arXiv preprint arXiv:2102.02267 (2021)
  * [8] Chai, W., Guo, X., Wang, G., Lu, Y.: Stablevideo: Text-driven consistency-aware diffusion video editing. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 23040–23050 (2023)
  * [9] Chai, W., Jiang, Z., Hwang, J.N., Wang, G.: Global adaptation meets local generalization: Unsupervised domain adaptation for 3d human pose estimation. arXiv preprint arXiv:2303.16456 (2023)
  * [10] Cui, Y., Zeng, C., Zhao, X., Yang, Y., Wu, G., Wang, L.: Sportsmot: A large multi-object tracking dataset in multiple sports scenes. arXiv preprint arXiv:2304.05170 (2023)
  * [11] Dendorfer, P., Rezatofighi, H., Milan, A., Shi, J., Cremers, D., Reid, I., Roth, S., Schindler, K., Leal-Taixé, L.: Mot20: A benchmark for multi object tracking in crowded scenes. arXiv preprint arXiv:2003.09003 (2020)
  * [12] Du, Y., Zhao, Z., Song, Y., Zhao, Y., Su, F., Gong, T., Meng, H.: Strongsort: Make deepsort great again. IEEE Transactions on Multimedia (2023)
  * [13] Duan, K., Bai, S., Xie, L., Qi, H., Huang, Q., Tian, Q.: Centernet: Keypoint triplets for object detection. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 6569–6578 (2019)
  * [14] Ess, A., Leibe, B., Schindler, K., Van Gool, L.: A mobile vision system for robust multi-person tracking. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition. pp. 1–8. IEEE (2008)
  * [15] Fischer, T., Huang, T.E., Pang, J., Qiu, L., Chen, H., Darrell, T., Yu, F.: Qdtrack: Quasi-dense similarity learning for appearance-only multiple object tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023)
  * [16] Ge, Z., Liu, S., Wang, F., Li, Z., Sun, J.: Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430 (2021)
  * [17] Gu, A., Dao, T.: Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 (2023)
  * [18] Gu, A., Dao, T., Ermon, S., Rudra, A., Ré, C.: Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems 33, 1474–1487 (2020)
  * [19] Gu, A., Goel, K., Ré, C.: Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396 (2021)
  * [20] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9(8), 1735–1780 (1997)
  * [21] Huang, H.W., Yang, C.Y., Jiang, Z., Kim, P.K., Lee, K., Kim, K., Ramkumar, S., Mullapudi, C., Jang, I.S., Huang, C.I., et al.: Enhancing multi-camera people tracking with anchor-guided clustering and spatio-temporal consistency id re-assignment. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5238–5248 (2023)
  * [22] Huang, H.W., Yang, C.Y., Ramkumar, S., Huang, C.I., Hwang, J.N., Kim, P.K., Lee, K., Kim, K.: Observation centric and central distance recovery for athlete tracking. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 454–460 (2023)
  * [23] Huang, H.W., Yang, C.Y., Sun, J., Kim, P.K., Kim, K.J., Lee, K., Huang, C.I., Hwang, J.N.: Iterative scale-up expansioniou and deep features association for multi-object tracking in sports. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 163–172 (2024)
  * [24] Jiang, Z., Chai, W., Li, L., Zhou, Z., Yang, C.Y., Hwang, J.N.: Unihpe: Towards unified human pose estimation via contrastive learning. arXiv preprint arXiv:2311.16477 (2023)
  * [25] Jiang, Z., Zhou, Z., Li, L., Chai, W., Yang, C.Y., Hwang, J.N.: Back to optimization: Diffusion-based zero-shot 3d human pose estimation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 6142–6152 (2024)
  * [26] Kalman, R.E.: A new approach to linear filtering and prediction problems (1960)
  * [27] Liu, H., He, J.Y., Cheng, Z.Q., Xiang, W., Yang, Q., Chai, W., Wang, G., Bao, X., Luo, B., Geng, Y., et al.: Posynda: Multi-hypothesis pose synthesis domain adaptation for robust 3d human pose estimation. In: Proceedings of the 31st ACM International Conference on Multimedia. pp. 5542–5551 (2023)
  * [28] Luiten, J., Osep, A., Dendorfer, P., Torr, P., Geiger, A., Leal-Taixé, L., Leibe, B.: Hota: A higher order metric for evaluating multi-object tracking. International journal of computer vision 129, 548–578 (2021)
  * [29] Milan, A., Leal-Taixé, L., Reid, I., Roth, S., Schindler, K.: Mot16: A benchmark for multi-object tracking. arXiv preprint arXiv:1603.00831 (2016)
  * [30] Milan, A., Rezatofighi, S.H., Dick, A., Reid, I., Schindler, K.: Online multi-target tracking using recurrent neural networks. In: Proceedings of the AAAI conference on Artificial Intelligence. vol. 31 (2017)
  * [31] Qi, Z., Zhu, R., Fu, Z., Chai, W., Kindratenko, V.: Weakly supervised two-stage training scheme for deep video fight detection model. In: 2022 IEEE 34th International Conference on Tools with Artificial Intelligence (ICTAI). pp. 677–685. IEEE (2022)
  * [32] Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., Savarese, S.: Generalized intersection over union: A metric and a loss for bounding box regression. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 658–666 (2019)
  * [33] Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. nature 323(6088), 533–536 (1986)
  * [34] Seidenschwarz, J., Brasó, G., Serrano, V.C., Elezi, I., Leal-Taixé, L.: Simple cues lead to a strong multi-object tracker. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13813–13823 (2023)
  * [35] Shao, S., Zhao, Z., Li, B., Xiao, T., Yu, G., Zhang, X., Sun, J.: Crowdhuman: A benchmark for detecting human in a crowd. arXiv preprint arXiv:1805.00123 (2018)
  * [36] Song, E., Chai, W., Wang, G., Zhang, Y., Zhou, H., Wu, F., Guo, X., Ye, T., Lu, Y., Hwang, J.N., et al.: Moviechat: From dense token to sparse memory for long video understanding. arXiv preprint arXiv:2307.16449 (2023)
  * [37] Suljagic, H., Bayraktar, E., Celebi, N.: Similarity based person re-identification for multi-object tracking using deep siamese network. Neural Computing and Applications 34(20), 18171–18182 (2022)
  * [38] Sun, P., Cao, J., Jiang, Y., Yuan, Z., Bai, S., Kitani, K., Luo, P.: Dancetrack: Multi-object tracking in uniform appearance and diverse motion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20993–21002 (2022)
  * [39] Sun, P., Cao, J., Jiang, Y., Zhang, R., Xie, E., Yuan, Z., Wang, C., Luo, P.: Transtrack: Multiple object tracking with transformer. arXiv preprint arXiv:2012.15460 (2020)
  * [40] Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., Metzler, D.: Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006 (2020)
  * [41] Tokmakov, P., Li, J., Burgard, W., Gaidon, A.: Learning to track with object permanence. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10860–10869 (2021)
  * [42] Tokmakov, P., Li, J., Burgard, W., Gaidon, A.: Learning to track with object permanence. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10860–10869 (2021)
  * [43] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [44] Wang, C.Y., Bochkovskiy, A., Liao, H.Y.M.: Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7464–7475 (2023)
  * [45] Wang, Q., Zheng, Y., Pan, P., Xu, Y.: Multiple object tracking with correlation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3876–3886 (2021)
  * [46] Wang, Y., Kitani, K., Weng, X.: Joint object detection and multi-object tracking with graph neural networks. In: 2021 IEEE International Conference on Robotics and Automation (ICRA). pp. 13708–13715. IEEE (2021)
  * [47] Wojke, N., Bewley, A., Paulus, D.: Simple online and realtime tracking with a deep association metric. In: 2017 IEEE international conference on image processing (ICIP). pp. 3645–3649. IEEE (2017)
  * [48] Wu, J., Cao, J., Song, L., Wang, Y., Yang, M., Yuan, J.: Track to detect and segment: An online multi-object tracker. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12352–12361 (2021)
  * [49] Wu, J., Cao, J., Song, L., Wang, Y., Yang, M., Yuan, J.: Track to detect and segment: An online multi-object tracker. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12352–12361 (2021)
  * [50] Xiao, C., Cao, Q., Zhong, Y., Lan, L., Zhang, X., Cai, H., Luo, Z., Tao, D.: Motiontrack: Learning motion predictor for multiple object tracking. arXiv preprint arXiv:2306.02585 (2023)
  * [51] Xu, Y., Ban, Y., Delorme, G., Gan, C., Rus, D., Alameda-Pineda, X.: Transcenter: Transformers with dense queries for multiple-object tracking (2021)
  * [52] Yang, C.Y., Huang, H.W., Jiang, Z., Kuo, H.C., Mei, J., Huang, C.I., Hwang, J.N.: Sea you later: Metadata-guided long-term re-identification for uav-based multi-object tracking. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 805–812 (2024)
  * [53] Ye, T., Chen, S., Liu, Y., Chai, W., Bai, J., Zou, W., Zhang, Y., Jiang, M., Chen, E., Xue, C.: Sequential affinity learning for video restoration. In: Proceedings of the 31st ACM International Conference on Multimedia. pp. 4147–4156 (2023)
  * [54] You, S., Yao, H., Bao, B.K., Xu, C.: Multi-object tracking with spatial-temporal tracklet association. ACM Transactions on Multimedia Computing, Communications and Applications 20(5), 1–21 (2024)
  * [55] Zeng, F., Dong, B., Zhang, Y., Wang, T., Zhang, X., Wei, Y.: Motr: End-to-end multiple-object tracking with transformer. In: European Conference on Computer Vision. pp. 659–675. Springer (2022)
  * [56] Zhang, Y., Wang, S., Fan, Y., Wang, G., Yan, C.: Translink: Transformer-based embedding for tracklets’ global link. In: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1–5. IEEE (2023)
  * [57] Zhang, Y., Wang, S., Wang, Q., Yan, C., Fan, R.: Multi-moving camera pedestrian tracking with a new dataset and global link model. arXiv preprint arXiv:2312.11035 (2023)
  * [58] Zhang, Y., Sun, P., Jiang, Y., Yu, D., Weng, F., Yuan, Z., Luo, P., Liu, W., Wang, X.: Bytetrack: Multi-object tracking by associating every detection box. In: European Conference on Computer Vision. pp. 1–21. Springer (2022)
  * [59] Zhang, Y., Wang, C., Wang, X., Zeng, W., Liu, W.: Fairmot: On the fairness of detection and re-identification in multiple object tracking. International Journal of Computer Vision 129, 3069–3087 (2021)
  * [60] Zhang, Y., Wang, T., Zhang, X.: Motrv2: Bootstrapping end-to-end multi-object tracking by pretrained object detectors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22056–22065 (2023)
  * [61] Zhang, Z., Chai, W., Jiang, Z., Ye, T., Song, M., Hwang, J.N., Wang, G.: Mpm: A unified 2d-3d human pose representation via masked pose modeling. arXiv preprint arXiv:2306.17201 (2023)
  * [62] Zhao, Z., Chai, W., Hao, S., Hu, W., Wang, G., Cao, S., Song, M., Hwang, J.N., Wang, G.: A survey of deep learning in sports applications: Perception, comprehension, and decision. arXiv preprint arXiv:2307.03353 (2023)
  * [63] Zhou, X., Koltun, V., Krähenbühl, P.: Tracking objects as points. In: European conference on computer vision. pp. 474–490. Springer (2020)
  * [64] Zhou, X., Yin, T., Koltun, V., Krähenbühl, P.: Global tracking transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8771–8780 (2022)