# The TYPHOON stellar population synthesis survey:
I. The young stellar population of the Great Barred Spiral NGC 1365

Eva Sextl  Universitäts-Sternwarte, Fakultät für Physik, Ludwig-Maximilians
Universität München, Scheinerstr. 1, 81679 München, Germany  Rolf-Peter
Kudritzki  Universitäts-Sternwarte, Fakultät für Physik, Ludwig-Maximilians
Universität München, Scheinerstr. 1, 81679 München, Germany  Institute for
Astronomy, University of Hawaii at Manoa, 2680 Woodlawn Drive, Honolulu, HI
96822, USA  Andreas Burkert  Universitäts-Sternwarte, Fakultät für Physik,
Ludwig-Maximilians Universität München, Scheinerstr. 1, 81679 München, Germany
I-Ting Ho  Max-Planck-Institute for Astronomy, Königstuhl 17, D-69117
Heidelberg, Germany  H. Jabran Zahid  Microsoft Research, 14820 NE 36th St,
Redmond, WA 98052, USA  Mark Seibert  The Observatories, Carnegie Institution
for Science, 813 Santa Barbara Street, Pasadena, CA 91106, USA  Andrew J.
Battisti  Research School of Astronomy and Astrophysics, Australian National
University, Canberra, ACT 2611, Australia  ARC Centre of Excellence for All
Sky Astrophysics in 3 Dimensions (ASTRO 3D), Australia  Barry F. Madore  The
Observatories, Carnegie Institution for Science, 813 Santa Barbara Street,
Pasadena, CA 91106, USA  Jeffrey A. Rich  The Observatories, Carnegie
Institution for Science, 813 Santa Barbara Street, Pasadena, CA 91106, USA

(Accepted 31 October, 2023)

###### Abstract

We analyze TYPHOON long slit absorption line spectra of the starburst barred
spiral galaxy NGC 1365 obtained with the Progressive Integral Step Method
covering an area of 15 square kpc. Applying a population synthesis technique,
we determine the spatial distribution of ages and metallicity of the young and
old stellar population together with star formation rates, reddening,
extinction and the ratio RV of extinction to reddening. We detect a clear
indication of inside-out growth of the stellar disk beyond 3 kpc characterized
by an outward increasing luminosity fraction of the young stellar population,
a decreasing average age and a history of mass growth, which was finished 2
Gyrs later in the outermost disk. The metallicity of the young stellar
population is clearly super solar but decreases towards larger galactocentric
radii with a gradient of -0.02 dex/kpc. On the other hand, the metal content
of the old population does not show a gradient and stays constant at a level
roughly 0.4 dex lower than that of the young population. In the center of NGC
1365 we find a confined region where the metallicity of the young population
drops dramatically and becomes lower than that of the old population. We
attribute this to infall of metal poor gas and, additionally, to interrupted
chemical evolution where star formation is stopped by AGN and supernova
feedback and then after several Gyrs resumes with gas ejected by stellar winds
from earlier generations of stars. We provide a simple model calculation as
support for the latter.

Barred spiral galaxies(136) — Starburst galaxies(1570) — Stellar
populations(1622) — Galaxy chemical evolution(580)

##  1 Introduction

Spectroscopic studies of galaxies with integral field units (IFU) have become
an important tool to investigate the evolution of galaxies. Spatially resolved
maps of their chemical composition, stellar ages, star formation rates and gas
properties provide unique information about the complex physical processes
affecting galaxy evolution (see, for instance, Bittner et al. (2020), Carrillo
et al. (2020), Parikh et al. (2021), Emsellem et al. (2022), Pessa et al.
(2023), Westmoquette et al. (2011), Sánchez-Blázquez et al. (2014),Thainá-
Batista et al. (2023)). Usually, because of the relatively small field of view
of the available IFUs (of the order of one arcminute) this work has been
mostly concentrated on galaxies with small angular size or only central
regions. Therefore, given the enormous potential of these spectroscopic
stellar population studies, we have started a project with the goal to extend
this work to cover large parts of galactic disks together with their central
regions. The TYPHOON survey which uses stepwise combined long slit spectra of
galaxies seems ideal for this purpose. For the population synthesis analysis
of the integrated stellar population absorption line spectra we will use the
technique developed and described in Sextl et al. (2023).

We start the project with the population synthesis analysis with the
starbursting type 2 Seyfert barred-spiral NGC 1365 in the Fornax cluster. With
an isophotal radius of R25 = 5.61 arcmin or 29.55 kpc at a distance of 18.1
Mpc (Ho et al., 2017) NGC 1365 is a galaxy of huge dimensions and ideally
suited for a long-slit IFU-like investigation. It is well studied with respect
to the dynamics of its gas and stellar content (Lindblad (1999), Zánmar
Sánchez et al. (2008), Jałocha et al. (2010)). It is an almost face-on galaxy,
which avoids line of sight confusion complications and minimizes effects of
interstellar extinction. The central region shows extensive star formation and
the presence of a low luminosity AGN with two conical outflows (see Venturi et
al. (2018) and references therein). In addition, the chemical composition of
its ionized ISM has been carefully investigated (Ho et al., 2017; Chen et al.,
2023) and first long slit star formation studies have already been carried out
(Sánchez-Blázquez et al., 2011). Since the galaxy is characterized by strong
star formation activity, the blue sensitivity of TYPHOON is an advantage and
allows for a good characterization of the properties of its young stellar
population. This will provide important information extending the work and the
results obtained recently within the comprehensive ESO VLT PHANGS-MUSE survey
(Emsellem et al., 2022; Pessa et al., 2023).

We describe the observations in section 2 and the population synthesis
analysis technique in section 3. The results are presented in sections 4
followed by a discussion in 5.

Figure 1: TYPHOON BVI color composite image of NGC 1365. The position of the
central AGN is indicated by a small green cross. The bar extends roughly 200”
on the sky (Lindblad, 1999) corresponding to $\sim$17.6 kpc projected length.
The two dashed ellipses indicate galactocentric distances of 3 and 15 kpc,
respectively. East is towards the left and north towards the top.

##  2 Observations

The TYPHOON survey (P.I. B. Madore) uses the Las Campanas du Pont 2.5m
telescope Wide Field CCD imaging spectrograph with a custom long-slit of 18
arcmin length and 1.65 arcsec width which progressively scans across the
galaxies (Progressive Integral Step Method, PrISM) to construct 3D data cubes
of 1.65 times 1.65 arcsec2 spaxels. At a distance of 18.1 Mpc (Jang et al.,
2018) 1.65 arcsec are equivalent to 145 pc. The spectral resolution
corresponds to a FWHM of 8.2 Å. In the case of NGC 1365 the slit was placed
along the north-south direction. More details of the observations are
described in Ho et al. (2017) and Chen et al. (2023). Figure 1 provides a BVR
color composite image constructed from the TYPHOON data cube. The figure has
already been shown in Ho et al. (2017) but is repeated here for illustration.
NGC 1365 is a massive galaxy with a stellar mass of log M∗ = 10.95 (in solar
units, Muñoz-Mateos et al. 2013, Leroy et al. 2019), an isophotal radius of
5.61 arcmin (de Vaucouleurs, 1991), inclination angle of 35.7 degrees and a
position angle of 49.5 degrees based on 2MASS photometry (Jarrett et al. 2003,
see Ho et al. 2017 Table 1).

For our population synthesis analysis we use the spectral range from 4000 to
7070 Å where the flux calibrated spectra have the best signal. Compared to the
range from 4800 to 7000 Å used in the PHANGS-MUSE study by Pessa et al. (2023)
this is an important blueward extension which enables a more accurate
characterization of the young stellar population.

If needed, we combine the spectra of neighboring individual spaxels by Voronoi
binning using Cappellari & Copin (2003) to obtain a minimum signal-to-noise
ratio of 30 in the stellar continuum at 5000 Å. However, in order to avoid
averaging over too large spatial dimensions we exclude bins consisting of more
than 400 TYPHOON spaxel. We also remove bins containing the contribution of
bright foreground stars. This leaves us with spectra of 359 bins distributed
over the galaxy. As an example, Figure 2 shows the spectra of two bins and the
corresponding stellar population fits. Figures 3 and 5 give an impression of
the spatial distribution of the bins analyzed.

Figure 2: TYPHOON spectrum (orange) and the corresponding stellar population
fit (dark blue). Note that for the fit ISM emission and absorption lines and
broad stellar Balmer lines are masked out. The resulting gaps are shown as
straight lines. The top spectrum corresponds to bin 16 at
$\Delta\alpha=-1.073$ arcmin and $\Delta\delta=1.292$ arcmin (northern spiral
arm) and the bottom spectrum to bin 153 at $\Delta\alpha=-0.192$ arcmin and
$\Delta\delta=-0.11$ arcmin (on the southern dust lane in the center). The
different physical properties of the stellar populations are discussed in
section 4.

Figure 3: Reddening (top) and extinction (bottom) maps of NGC 1365 obtained
with our population synthesis fit. The central region is indicated by the
green cross. The ellipses indicate galactocentric distances of 3, 10, and 15
kpc.

Figure 4: Radial distribution as a function of deprojected galactocentric
distance of reddening E(B-V) (top), extinction AV (middle) and RV (bottom).
Figure 5: Map of average population age. The dashed ellipses indicate
galactocentric distances of 3, 10, and 15 kpc, respectively.

Figure 6: Radial distribution of average population age (top), the ratio
byoung/bold (middle), which is ratio of the contribution of the young and old
population to the total observed population spectrum, and the ratio of average
star formation rates of the young to the old population. Figure 7: Radial
distribution of the logarithm of metallicity relative to the sun. The young
and old population are plotted in blue and red, respectively. HII-region
metallicities based on oxygen abundances (Ho et al., 2017) are shown in
yellow. We also show regressions for the young stellar population (solid) and
the HII-regions (dashed) calculated for galactocentric distances larger than
3.5 kpc.

##  3 Analysis Method

Our population synthesis method is described in detail in Sextl et al. (2023).
We fit the SED and absorption line spectra of the integrated stellar
population with a linear combination of the spectra of single stellar
populations (SSP) of different ages and metallicities. This allows us to
constrain the average metallicity and age of the population together with
reddening E(B-V), extinction AV and the ratio RV = AV/E(B-V), which
characterizes the steepness of the reddening law. The observed and SSP
template spectra are normalized in the range between 5500 to 5550 Å.
Therefore, the metallicities and ages obtained in this way are V-band
luminosity weighted averages (see Sextl et al. 2023).

The model spectrum of the integrated stellar population Mλ combines the
spectra of single stellar populations (SSPs) fλ,i(ti, [Z]i) with age ti and
logarithmic metallicity [Z]i = log Z/Z☉

$M_{\lambda}=D_{\lambda}(R_{V},E(B-V))\left[\sum_{i=1}^{n_{SSP}}b_{i}f_{\lambda,i}(t_{i},[Z]_{i})+b_{a}f_{\lambda}^{a}\right]$
(1)

where the coefficients bi describe the contribution of burst i with age ti and
metallicity [Z]i and D${}_{\lambda}(R_{V},E(B-V))$ accounts for the absorption
by interstellar dust. ba accounts for the contribution of a featureless AGN
continuum f${}_{\lambda}^{a}$ with wavelength slope $\lambda^{-0.5}$ (as in
Cardoso et al. (2017)). This additional template is only utilized in the
fitting process of bins near the center which show broad-line region (BLR)
features in their spectra.

We apply the Flexible Stellar Population Synthesis code (FSPS, version 3.2)
(Conroy et al., 2009; Conroy & Gunn, 2010) for the calculation of the
individual SSP spectra together with the MILES library (Sánchez-Blázquez et
al., 2006) and MESA stellar evolution isochrones (Choi et al., 2016; Dotter,
2016) and a Chabrier (2003) initial mass function are adopted. It is important
to note that FSPS utilizes not only one main stellar library option, but adds
supplementary stellar spectra from complementary evolutionary phases. Thus,
the limited number of stars with T $>9000$ K in the empirical MILES library
(as can be seen in Martins & Coelho (2007)), is enlarged by hot star spectra
from Eldridge et al. (2017) and Wolf-Rayet spectra from Smith et al. (2002).
Spectra of AGB-, post-AGB- and carbon stars are added (Lançon & Wood, 2000;
Rauch, 2003; Aringer et al., 2009). The final set of our SSP spectra is then
adjusted to the TYPHOON spectral resolution. Because of the low TYPHOON
spectral resolution we do not consider the line broadening effects of stellar
velocity dispersion, which has been measured, for instance, by Bittner et al.
(2020) or Pessa et al. (2023).

As in Sextl et al. (2023) we account for finite time lengths of the stellar
bursts of 0.1, 1.0 and 10.0 Myr but we find that SSP spectra with the shortest
burst length result in the best spectral fits with lowest residual $\chi^{2}$
value. To account for the effects of interstellar dust, we apply the
attenuation law by Calzetti et al. (2000) with variable RV (deviating from the
Calzetti standard value RV=4.05), derived empirically in local starburst
galaxies.

For the choice of SSPs we use the high resolution age described in Sextl et
al. (2023), which starts at 0.1 Myr with a step to 1.0 Myr and then continues
with logarithmic steps $\Delta$log t (in Gyr) alternating between 0.05 and 0.1
dex until 12.59 Gyr. The metallicities start at [Z] = -1.5 and increases in
steps of 0.25 dex until [Z] = 0.5 as the highest value of the grid. We have,
thus, a grid with na = 52 ages and nz = 9 metallicities and a total number of
nSSP = 468 SSPs.

In our fitting procedure we correct for the radial velocity shifts of the
observed spectra. We also measure the equivalent widths of the ISM nebular
emission lines of hydrogen Hβ to account for nebular emission continuum if
needed (see Sextl et al. 2023). The equivalent width is used to estimate the
contribution of the nebular continuum at the wavelength of Hβ and the
wavelength dependence is calculated by accounting for nebular hydrogen and
helium bound-free, free-free and 2-photon emission. The (generally weak)
nebular continuum contribution is then subtracted from the observed spectra.
In addition, as in Sextl et al. (2023) spectral regions contaminated by ISM
emission or absorption lines are not included in the spectral fit of the
stellar population.

After these steps, the coefficients bi and ba are then determined by adopting
a grid of RV and E(B-V) values. For each pair of these quantities we calculate
D${}_{\lambda}(R_{V},E(B-V))$, use a least square algorithm to directly solve
for the coefficients b, calculate the model spectrum and a $\chi^{2}$ value by
comparing with the observed spectrum. The minimum of $\chi^{2}$ defines the
best fit. Errors are estimated by fitting the observed spectra modified by
adding Monte Carlo Gaussian noise with zero mean and a standard deviation
corresponding to the flux error at each wavelength point. The uncertainties of
the stellar population parameters are then calculated as the standard
deviation of their distributions produced by 20 such Monte Carlo realizations.

Following the arguments in Sextl et al. (2023) we use a characteristic age
limit of 1.6 Gyr to distinguish between the young and old population and
determine average ages and metallicities of these two populations separately.
Through t${}_{i}\leq$ 1.6 Gyr and t${}_{i}\geq$ 1.6 Gyr, respectively, we
introduce the young and old population and define the corresponding
metallicities [Z]y, [Z]o and ages log(ty), log(to) via

$b_{y}=\sum_{i_{y}}b_{i},b_{o}=\sum_{i_{o}}b_{i}$ (2)

and

$[Z]_{y}={\frac{1}{b_{y}}}\sum_{i_{y}}b_{i}[Z]_{i}$ (3)
$[Z]_{o}={\frac{1}{b_{o}}}\sum_{i_{o}}b_{i}[Z]_{i}$ (4)
$log(t_{y})={\frac{1}{b_{y}}}\sum_{i_{young}}b_{i}log(t_{i})$ (5)
$log(t_{o})={\frac{1}{b_{o}}}\sum_{i_{old}}b_{i}log(t_{i}).$ (6)

The average values of the total population, young and old, are then calculated
as

$[Z]=b_{y}[Z]_{y}+b_{o}[Z]_{o},log(t)=b_{y}log(t_{y})+b_{o}log(t_{o})$ (7)

As explained in Sextl et al. (2023) metallicities and ages obtained in this
way are luminosity weighted averages. The results of our analysis are
presented in the next section.

Figure 8: The central region of NGC 1365. Maps of the color excess E(B-V)
(top), $R_{V}$ (middle), and the ratio of the average star formation rate of
the young and old population (bottom). Isocontours of interstellar reddening
E(B-V) are overplotted for E(B-V) = 0.2 and 0.3 mag.

Figure 9: Central maps of stellar metallicity [Z]y and [Z]o of the young (top)
and old (middle) population with the same axis range. The bottom figure shows
the difference $\Delta$ [Z] = [Z]y \- [Z]o . Negative differences are marked
in pink tones. Isocontours of interstellar reddening E(B-V) are overplotted as
in Figure 8. The mean errors of [Z]y and [Z]o in the FoV are $\sim$0.12 dex
and $\sim$0.10 dex, respectively.

Figure 10: Central maps of the ratio by/bo (top), the ages ty of the young
stellar population (middle), and the stellar mass ratio of the young and old
population (bottom). Isocontours of interstellar reddening E(B-V) are
overplotted as in Figure 8.

##  4 Results

In the following subsections we present the results of our stellar population
fits. We start with the maps of interstellar extinction. We then discuss the
distribution of ages and metallicity of the young and old stellar population
in the outer disk and the central region of the galaxy.

###  4.1 Reddening and extinction

The spatial distribution of reddening E(B-V) and extinction AV by interstellar
dust is shown in Figure 3. We find very strong effects in the center and bar
and a moderate enhancement along the spiral arms. For the outer disk outside
the spiral arms reddening is low and close to the Milky Way foreground
reddening of E(B-V)MW = 0.018 mag (Schlafly & Finkbeiner, 2011). We provide
additional information in Figure 4, which shows the radial distributions as a
function of galactocentric distance together with the fit uncertainties. We
note again that in our spectral fit of interstellar dust we have included the
determination of RV which is allowed to deviate from the ”standard” values
such as 3.1 for the diffuse Milky Way ISM or 4.05 for the Calzetti-law in
starburst galaxies. Such deviations are common in star forming galaxies (see
discussion and references in Sextl et al. 2023). The bottom of Figure 4 shows
the radial distribution of the RV values which we encounter together with
their errors. Bins with E(B-V) values $<0.005$ have been removed since the
slope of the attenuation curve cannot be quantified in this case. We find a
mean value around 4.0 but with a wide scatter between 1.5 and 6.7. As
comprehensively discussed in Salim & Narayanan (2020), extinction is not the
same as attenuation due to the additional geometric and scattering components
in the latter case. This also implies that one should be cautious here with
the common interpretation of large values in $R_{V}$ implying larger grain
sizes (Battisti et al., 2017; Calzetti et al., 2000). It can nevertheless be
used as an indicator for obscuration (Calzetti, 2001).

A more detailed discussion of the central region will be given below.

###  4.2 Population ages and inside-out growth

Figure 5 displays the spatial map of the average population ages across the
surface of NGC 1365 and Figure 6 (top) provides the corresponding radial
distribution. We find a maximum of populations ages around a galactocentric
distance of 5 kpc and lower ages towards the center and the outer edge. The
decrease of age beyond 5 kpc is a clear indication that the outer disk is more
and more dominated by the young stellar population. This is confirmed by the
radial dependence of the ratio by/bo, which represents the ratio of the
luminosity contribution of the young and old population to the total observed
population spectrum and which is also given in Figure 6. The contribution of
the young population gradually increases when going beyond 3 kpc towards
larger galactocentric distances.

As shown in Sextl et al. (2023) (equations 11 and 12 and text at the end of
section 4) the SSP fit coefficients bi can also be used to estimate the ratio
of star formation rates of the young and old population. This ratio is also
shown at the bottom of Figure 6. The outer disk shows a gradual increase of
this ratio from 3 kpc outwards. We note that the average age of the young
population in the different spatial bins of this region is between 0.1 and 1.0
Gyr with a very few exceptions of a very young population only 5 Myr old. The
average ages of the old population are between 10 and 12.6 Gyr.

The decrease of average ages and the increased contribution to luminosity and
star formation by the young population towards the outer radii is a clear
indication of inside-out growth of the stellar disk of NGC 1365. This will be
further discussed in Section 5.

From 3 kpc inwards the trend with galactocentric distance reverses. Average
ages decrease and the contribution of the young population becomes stronger
again. We will discuss this in the subsections below.

Figure 11: Age of the stellar population in each spectral bin at the time when
80 % of the stellar population has formed as a function of galactocentric
radius in kpc. A linear regression curve is also shown. For details see text.

###  4.3 The metallicity of the young and old population of the outer disk

The radial distribution of metallicities (defined as [Z] = log Z/Z⊙) for the
young and old stellar population is plotted in Figure 7. We also add the
metallicities of HII-regions obtained by Ho et al. (2017). They are based on
oxygen abundances as a proxy for metallicity and the Baysian strong line
calibration developed by Ho et al. The Asplund et al. 2009 value of log
N(O)/N(H) + 12 = 8.69 has been adopted for the solar oxygen abundance.

In the outer disk beyond 3.5 kpc we find a negative metallicity gradient of
the young population of $-0.0207\pm 0.0028$ dex/kpc in line with the
expectations for a stellar disk which has formed inside-out. The stellar
gradient is slightly steeper than the one obtained from HII-region emission
lines and the value of metallicity is somewhat lower but these differences can
be attributed to uncertainties of the HII-region strong line calibrations (see
discussion in Ho et al. 2017; Bresolin et al. 2016). For the outer old
population we note that the low metallicity bins with $[Z]\leq-0.35$ at a
distance of 8-11 kpc are all located within the leading arm of the southern
spiral right outside the bar. Apart from this anomaly there is no clear
indication of an outer radial gradient of the old population. This is a
striking result and will be discussed in section 5. The global difference of
$\approx$ 0.4 dex in [Z] between the young and old population agrees very well
with the difference found for the massive star forming SDSS galaxies studied
by Sextl et al. (2023) and with the prediction of chemical evolution models
(see, for instance, Kudritzki et al. 2021a, b).

In addition to the outer negative gradient of [Z]y Figure 7 reveals a steep
drop to low stellar metallicities of the young population towards the center
of the galaxy. The metallicity of the old population, on the other hand,
increases on average. This will addressed in the next subsection where we
discuss the central region of the galaxy.

###  4.4 Central region

As already indicated in Figures 4 and 6 the central region of NGC 1365 is
characterized by high interstellar extinction and a strong contribution of the
young stellar population to the observed luminosity. This is demonstrated in
more detail by Figures 8 and 10 which provide zoomed maps of the central
region of reddening E(B-V), RV, average star formation rate ratio, the
luminosity contributions ratio of the young and old population, mass ratios,
and the age of the young population. However, there is an obvious asymmetry in
the sense that the increased luminosity contribution and star formation
activity of the young population is confined mostly to the northern area with
E(B-V) $\sim$ 0.2 to 0.3 mag and less is found in the corresponding south-east
region. This has been discussed most recently by Schinnerer et al. (2023) in
their analysis of JWST mid-IR imaging and ALMA CO observations. They concluded
that the asymmetric gas infall along the bar has already initiated the intense
star formation in the northern bar lane whereas in the dense molecular clouds
in the south star formation has yet to come but will start very soon. This is
in accordance with the rather similar gas properties in both regions.

We note that while luminosity contribution of the young population is very
high in the central region, the ratio of masses of the populations is My/Mo is
small with a mean value of 0.03 (see Figure 10 bottom). The reason for this
significant difference to by/bo is the higher luminosity of younger (and more
massive) stars. Details of the computation of the mass ratio are described in
Section 5.

The central map of RV (Figure 8 middle) does not indicate any kind of spatial
correlation. It appears that the RV values are randomly distributed depending
on how different gas clouds contribute to the attenuation (see Salim &
Narayanan 2020; Battisti et al. 2017).

Most strikingly, as shown in Figure 9, the metallicity [Z]y of the young
population formed in this central region is very low, sometimes even lower
than the metallicity of the old population. Obviously, the surface bins with
low metallicity in Figure 7 are all confined to a coherent area in the very
center. This population has been formed most recently, as can be seen from
Figure 10. This is in accordance with Whitmore et al. (2023) who identified
several young star clusters with ages $<10$ Myr in the central regions of NGC
1365. Since our spectral analysis contemplates all stars in a spaxel, not only
the massive clusters, we expect slightly higher ages in our analysis. In
general, it seems to be of prime importance to provide SSP templates with
sufficient young ages for the fit to capture bins with such young clusters
accordingly.

We note that central stellar metallicities of NGC 1365 have already been
studied within two surveys using the ESO VLT integral field unit MUSE, the
TIMER (Gadotti et al., 2019, 2020; Bittner et al., 2020) and the PHANGS
(Emsellem et al., 2022) survey. Both surveys do not distinguish between
metallicities of the young and old population and provide only an average over
the total population but also indicate a drop of total metallicity towards the
center (Pessa et al., 2023; Bittner, 2021). From our result we learn that this
drop is caused by a strong burst of the formation of a very young population
of low metallicity.

It is important to check whether the low metallicities encountered are an
artefact of the fitting procedure or are caused by numerical uncertainty. As a
first step, we have compared the minimum fit $\chi^{2}$ values as a function
of the metallicity obtained. We did not find any hint of a systematic
difference. Fits at low metallicity have the same quality as high metallicity
ones. In addition, we have applied an independent population synthesis
algorithm and repeated the analysis for all TYPHOON bins by using the
STARLIGHT package (Cid Fernandes et al., 2005; Asari et al., 2007) for all
TYPHOON bins. Very similar results were found including the low metallicity
bins in the center. We also note that keeping $R_{V}$ fixed to the Calzetti
standard value of 4.05 or assuming a Milky Way reddening law with constant
$R_{V}=3.1$ (Cardelli et al., 1989) produces a similar peculiar region of low
metallicity.

We also experimented with differential interstellar extinction between the
young and old population. Following Lo Faro et al. (2017) and Yuan et al.
(2018) we introduced a factor fatt, which increases the reddening of the very
young stars (t $\leq$ 10 Myr) relative to the older stars by adopting
E(B-V)(t$\leq$ 10Myr) = E(B-V)/fatt. Applying a value of fatt = 0.5 we did not
encounter substantial changes in the derived stellar properties larger than
the fit uncertainties in the outer parts of the galaxy. However, in the
central region, where the extinction is highest, the effect of inversion of
metallicity between the young and old stellar population is increased. Fits
with differential extinction tend to lower the metallicity for the young
population even further by on average $\sim$ 0.15 dex whereas [Z]o remains the
same. Since fatt as an additional free parameter is poorly constrained, we
refrain from a detailed follow up at this point but note that differential
extinction tends to make the situation in the center more extreme.

Finally, we have checked how well the metal line absorption features are
reproduced by our low metallity SSP fits. In Figure 13 we repeat the fit of
bin 153 (already displayed in Figure 2) but now show the detailed fit of the
metal absorption line features in two selected spectral regions. According to
our fit the metallicity of the young population in this bin is [Z]y = -0.36.
We conclude that given the quality of the spectra the metal lines are on
average reproduced well. We take all this as a confirmation of our results.

An obvious question is whether there are HII regions in this confined region
of low stellar metallicity and what their metallicities are. Very
unfortunately, the most recent PHANGS-MUSE (Groves et al., 2023) and TYPHOON
(Chen et al. 2023) HII-region and ISM emission line studies do not provide
sufficient information allowing for conclusions about the ISM metallicity in
this specific region. This is mostly caused by the strong contribution of the
central AGN to the ISM ionizing radiation field.

One should also keep in mind that in the TYPHOON observation of NGC 1365 one
spaxel corresponds to 145 pc, so the whole low-metallicity region covers
approx. a circle of 2 kpc in diameter. In kpc-resolution surveys at higher
redshifts such relatively small regions would have been overlooked easily or
would be dismissed as faulty spaxels. This makes the local TYPHOON sample
especially valuable.

The physical nature of the low metallicity of the central young stellar
population is discussed in the next section.

Figure 12: The mass metallicity relationship of star forming galaxies. Results
from spectroscopy of individual red and blue supergiant stars are shown as
stars (Bresolin et al. 2022, Urbaneja et al., 2023). These [Z] values refer to
galactocentric distances of 0.4 R25 for galaxies with a metallicity gradient
and to mean values for low mass irregular galaxies without a gradient. The
value for NGC 1365 is given by the pink hexagon (see text). The population
synthesis results by Sextl et al. (2023) for the average metallicity of the
young stellar population of SDSS galaxies are plotted as khaki lines

. The green curve is a prediction from galaxy evolution lookback models by
Kudritzki et al. (2021a, b).

Figure 13: Spectral fit (black) of the absorption line features of bin 153.
The metallicity of the young population is [Z]y = -0.36. Note that the TYPHOON
spectrum has a gap from 5524 to 5561Å. The spectral regions at 5007 and 6300 Å
are not included in the $\chi^{2}$ spectral fit due to the contamination by
ISM emission.

##  5 Discussion

As concluded in section 4.2 and Figures 5 and 6, the decrease of ages averaged
over the total population and the increase of the ratio by/bo is indicative of
inside-out growth of the outer stellar disk of NGC 1365. An alternative way to
investigate this is to look at the radial dependence of stellar mass growth.
As described in Sextl et al. (2023) the fit coefficients bi in Equation (1) of
the spectral fit of an observed spectral bin can be related to the relative
number contribution Ni of stars of isochrone i, for which the SSP model
spectrum is calculated, and the V-band luminosity Li(V) of the isochrone via

$b_{i}=N_{i}L_{i}.$ (8)

With Ni = bi/Li and the mass Mi of each isochrone the function

$g(\log(t_{i}))=\dfrac{\sum_{i_{1}=1}^{i}\sum_{i_{2}=1}^{n_{z}}N_{i_{1},i_{2}}M_{i_{1},i_{2}}}{\sum_{i_{1}=1}^{n_{a}}\sum_{i_{2}=1}^{n_{z}}N_{i_{1},i_{2}}M_{i_{1},i_{2}}}$
(9)

describes the accumulative mass growth in the surface area of each spectral
bin as a function of time. Note that in Equation (9) the inner sums in the
nominator and denominator add up the contributions by different metallicities
at the same age, whereas the outer sums accumulate the contribution of
different ages.

Applying Equation (9) we can calculate the age (or lookback time) for each
spectral bin, when 80 % of the stellar mass has formed. The corresponding
radial distribution of ages is shown in Figure 11. We see a clear indication
of inside-out growth beyond 2.5 kpc. At the outermost parts of the disk the
mass growth was accomplished about 2 Gyr later than in the inner parts. We
note that a similar result has been obtained by Pessa et al. (2023).

According to standard chemical evolution models including galactic winds and
gas infall (see, for instance, Hou et al. 2000; Chiappini et al. 2001;
Kudritzki et al. 2015; Weinberg et al. 2017; Kang et al. 2021) the inside-out
growth of the disk is the physical reason for the negative metallicity
gradient of the young stellar population and the ISM in star forming disk
spiral galaxies, which we also encounter in NGC 1365. The value of -0.02
dex/kpc determined in our work seems very small but NGC 1365 is a huge galaxy
and renormalizing with respect to the isophotal radius leads to -0.59 dex/R25.
This value is in good agreement with gradients found from the quantitative
spectroscopy of individual supergiant stars in nearby galaxies (see, for
example, Kudritzki et al. 2012; Bresolin et al. 2022; Liu et al. 2022 and
references therein) and comprehensive studies of metallicity gradients
obtained from HII-regions such as Ho et al. (2015).

The average outer metallicity of the young population represented by the value
[Z]y = 0.15 at 0.4 R25 = 11.82 kpc is in good agreement with the mass-
metallicity relationship (MZR) of star forming galaxies. Figure 12 shows a MZR
comparison with the results obtained by the analysis of individual supergiant
stars (Bresolin et al. 2022, Urbaneja et al., submitted to ApJ) and the SDSS
population synthesis study of the young population by Sextl et al. (2023).

As already mentioned above, the average difference between the metallicity of
the old and the young population of about 0.4 dex agrees with the result
obtained by Sextl et al. (2023) in their investigation of the integrated
spectra of 250000 SDSS galaxies. As for the metallicity gradients, the fact
that we do not encounter a gradient for the outer old population is very
interesting. When comparing gradients as a function of population age galaxy
chemical evolution models yield different results depending on the assumptions
made. Hou et al. (2000) find that gradients of the older population are
steeper, whereas Chiappini et al. (2001) find the opposite. Roškar et al.
(2008) point out that stellar migration will tend to flatten abundance
gradients with time and predict younger populations to have a steeper
gradient. It seems that population synthesis analysis might provide a powerful
tool to investigate this issue and to provide additional constraints on galaxy
evolution in this way. This will be an important part of our continued TYPHOON
population synthesis survey.

A striking result is the low metallicity of the young population in a very
confined region in the center (Figure 9). A straightforward first explanation
is strong inflow of metal poor gas which has influenced the most recent star
formation. This would be in line with the most recent work by Schinnerer et
al. (2023) and the results by Zánmar Sánchez et al. (2008) who detected gas
infall along the bar. Tabatabaei et al. (2013) gauged a relatively short
timescale of only 300 Myrs in their submilimeter analysis on which gas from
the bar looses angular momentum and flows into the center. However, it is not
clear whether this infalling gas is metal poor and whether the amount of gas
accumulated has been sufficient to form a new metal poor population. In
principle, galaxy evolution models with infall and outflow can produce a
younger population with a metallicity lower than the older population
(Spitoni, 2015) but the differences are not as extreme as we encounter in NGC
1365. A crucial test would be a metallicity measurement of the infalling gas.

We propose an additional scenario as an explanation for the low metallicity of
the central young stellar population, namely interrupted chemical evolution.
In this scenario chemical evolution in the central region started in the usual
way with heavy star formation and building up metals. Then, at a time when ISM
metallicity had already reached high values of above solar, feedback from the
central AGN together with stellar feedback from supernovae and winds created a
strong central galactic wind and ejected the central gas so that star
formation stopped for a long time. For a massive galaxy such as NGC 1365 an
AGN can stay intermittently active for 1-5 Gyr and inhibit star formation on a
scale of 1 Gyr (Stasińska et al., 2015).

In the meantime, the most massive stars which have already formed, continue to
recycle metal enriched gas to the ISM through supernovae and winds but due to
the energetic AGN and stellar activity this matter continues to be expelled.
Then, after several Gyr this activity drops and a new reservoir of metal
poorer ISM gas is built up again from stellar winds and mass-loss of stars of
low mass of the older population, which formed earlier. This central reservoir
eventually becomes dense enough to start formation again and the newly formed
stars show the low metallicity composition of the older population.

Figure 14: A simple chemical evolution model as example for the central region
of NGC 1365. The metallicity of the ISM present in the center before the end
of star formation at 4 Gyr is given in blue and the V-band luminosity averaged
metallicity of the stars in orange. The abscissa is the evolution time in Gyr.
The gray shaded area marks the time when the central ISM gas is ejected and
star formation is quenched. The green symbols show the gas metallicity
provided by stellar winds and supernovae of the old population after the
central activity ejecting gas has stopped. For more details see text.

Figure 14 shows the result of a simple Monte Carlo simulation, where we apply
a closed-box chemical evolution model. We assume that stars form out of
pristine gas with zero metallicity in an intense star formation process
following a standard initial mass function (Kroupa, 2001). Metal enriched gas
is recycled to the ISM through supernovae and stellar winds. We distinguish
between stars with masses higher or lower than 9M⊙. For the former, all matter
except a remnant of 3M⊙ is returned to the ISM and the mass of the convective
core (Maeder, 1987; Ekström et al., 2012) is converted into metals by core-
collapse supernovae. For the latter, we adopt a white dwarf mass using an
initial-final mass relationship (Cummings et al., 2018) and recycle the
remaining matter. For the metals produced by these objects we follow Kobayashi
& Nomoto (2009) and assume that 7 percent of the stars in the mass range
between 3 to 9 M⊙ increase the mass of the final WD through binary accretion
until the Chandrasekhar limit. A subsequent Supernova type Ia explosion
returns the white dwarf mass as metals. For the stellar life times of both
groups we use the main sequence fit values given by Ekström et al. (2012). In
the case of SNIa explosions we do not account for the accretion time, which is
short compared with the stellar life time (Kobayashi & Nomoto, 2009).

As a result of our calculation, ISM metallicity (shown in blue) is increasing
quickly. Young, newly formed stars will resemble the ISM metallicity but the
metallicity of older stars will be lower. We show the V-band luminosity
averaged metallicities of the stars in orange. (As mentioned above the
metallicities and ages we derive in our population synthesis approach are
luminosity weighted quantities).

After 4 Gyr star formation comes to a hold because of the energetic processes
in the center, which eject all star forming gas, and the evolution stops.
After all the ISM gas is quickly expelled from the center, the existing
massive stars still recycle gas to the ISM. However, because of the AGN
activity the new ISM gas still continues to be expelled and the central region
remains gas free. Then, 8 Gyr after the whole chemical evolution started, the
central activity stops and a new reservoir of ISM gas forms because of stellar
winds and mass-loss from the existing low mass stars of different ages and low
metallicity. The ISM metallicity of this new reservoir of gas is shown in
green. It is slightly lower than the luminosity weighted metallicities of the
old stars, which agrees qualitatively with Figure 9. Young stars forming out
of this gas will resemble this metallicity. This explains how the formation of
a lower metallicity young stellar population in the center is possible without
invoking the infall of metal poor gas from outside.

An obvious question is whether the dying stars of the old population can
provide enough gas for a new young population. As a very simple estimate we
consider stars born at 3 Gyr in Figure 14. In our scenario the gas provided by
stars with a main sequence lifetime shorter than 5 Gyr (or more massive than
1.18 $M_{\odot}$) is expelled by the central AGN activity. However, the gas
recycled by stars less massive than 1.18 $M_{\odot}$ but with a higher mass
than 0.958 $M_{\odot}$ (corresponding to a main sequence lifetime of 9 Gyr)
can settle into the central region, because AGN activity has now stopped (we
adopt $\tau_{MS}=8(M/M_{\odot})^{-2.8}$ Gyr for the low mass main sequence
lifetime). Using the IMF we calculate the mass of these dying stars in
relation to the mass of all remaining stars with masses lower than 0.958
$M_{\odot}$ and obtain a ratio of 0.124. Assuming that half of this mass
fraction is recycled as gas we obtain a ratio of gas mass to the mass of old
stars of 0.06, which is a factor of two higher than the observed average mass
ratio of the young to the old population (see Figure 10 bottom). A starburst
of very high star formation efficiency (Fisher et al., 2022) could, thus,
explain the observations within the framework of our model.

The goal of this simplified calculation is only to demonstrate that such an
additional scenario can work. A better match of the observations could
certainly be obtained by modifying parameters such as star formation rates,
evolution and yields etc. and, most importantly, by including infall of metal
poor gas, but that is beyond the scope of this paper. We note that the idea of
recycled gas accumulating from an aging stellar population is not new, see for
instance in Ciotti & Ostriker (2007).

The situation in the center of NGC 1365 is not unique as it seems. The Milky
Way shows a metallicity of the young stellar population which increases from
[Z] $\sim$ -0.35 at 15 kpc galactocentric distance to $\sim$ +0.3 at 4 kpc
(Genovali et al., 2014; da Silva et al., 2022) but suddenly drops to solar or
even subsolar towards the galactic center (Najarro et al., 2004, 2009; Davies
et al., 2009a, b; Martins et al., 2008; Origlia et al., 2013; Do et al.,
2015). Figure 5 in Genovali et al. (2014) gives an excellent impression of the
very similar situation in the Milky Way. Additionally, recent ESO VLT
observations revealed an intense star formation period in the galactic center
$\sim$ 0.6-1 Gyr ago after a long quiescent phase of approximately 6 Gyrs
(Nogueras-Lara et al., 2020).

In summary, we conclude that detailed spatially resolved population synthesis
studies of young and old populations in star forming galaxies are an extremely
powerful tool to investigate the evolution of galaxies. We will continue this
work within the framework of our TYPHOON survey.

Acknowledgements. This work was initiated and supported by the Munich
Excellence Cluster Origins funded by the Deutsche Forschungsgemeinschaft (DFG,
German Research Foundation) under Germany’s Excellence Strategy EXC-2094
390783311.

## References

  * Aringer et al. (2009) Aringer, B., Girardi, L., Nowotny, W., Marigo, P., & Lederer, M. T. 2009, A&A, 503, 913, doi: 10.1051/0004-6361/200911703
  * Asari et al. (2007) Asari, N. V., Cid Fernandes, R., Stasińska, G., et al. 2007, MNRAS, 381, 263, doi: 10.1111/j.1365-2966.2007.12255.x
  * Asplund et al. (2009) Asplund, M., Grevesse, N., Sauval, A. J., & Scott, P. 2009, ARA&A, 47, 481, doi: 10.1146/annurev.astro.46.060407.145222
  * Battisti et al. (2017) Battisti, A. J., Calzetti, D., & Chary, R. R. 2017, ApJ, 840, 109, doi: 10.3847/1538-4357/aa6fb2
  * Bittner (2021) Bittner, A. 2021, PhD thesis, Ludwig-Maximilians University of Munich, Germany
  * Bittner et al. (2020) Bittner, A., Sánchez-Blázquez, P., Gadotti, D. A., et al. 2020, A&A, 643, A65, doi: 10.1051/0004-6361/202038450
  * Bresolin et al. (2022) Bresolin, F., Kudritzki, R.-P., & Urbaneja, M. A. 2022, ApJ, 940, 32, doi: 10.3847/1538-4357/ac9584
  * Bresolin et al. (2016) Bresolin, F., Kudritzki, R.-P., Urbaneja, M. A., et al. 2016, ApJ, 830, 64, doi: 10.3847/0004-637X/830/2/64
  * Calzetti (2001) Calzetti, D. 2001, PASP, 113, 1449, doi: 10.1086/324269
  * Calzetti et al. (2000) Calzetti, D., Armus, L., Bohlin, R. C., et al. 2000, ApJ, 533, 682, doi: 10.1086/308692
  * Cappellari & Copin (2003) Cappellari, M., & Copin, Y. 2003, MNRAS, 342, 345, doi: 10.1046/j.1365-8711.2003.06541.x
  * Cardelli et al. (1989) Cardelli, J. A., Clayton, G. C., & Mathis, J. S. 1989, ApJ, 345, 245, doi: 10.1086/167900
  * Cardoso et al. (2017) Cardoso, L. S. M., Gomes, J. M., & Papaderos, P. 2017, A&A, 604, A99, doi: 10.1051/0004-6361/201630378
  * Carrillo et al. (2020) Carrillo, A., Jogee, S., Drory, N., et al. 2020, MNRAS, 493, 4094, doi: 10.1093/mnras/staa397
  * Chabrier (2003) Chabrier, G. 2003, PASP, 115, 763, doi: 10.1086/376392
  * Chen et al. (2023) Chen, Q.-H., Grasha, K., Battisti, A. J., et al. 2023, MNRAS, 519, 4801, doi: 10.1093/mnras/stac3790
  * Chiappini et al. (2001) Chiappini, C., Matteucci, F., & Romano, D. 2001, ApJ, 554, 1044, doi: 10.1086/321427
  * Choi et al. (2016) Choi, J., Dotter, A., Conroy, C., et al. 2016, ApJ, 823, 102, doi: 10.3847/0004-637X/823/2/102
  * Cid Fernandes et al. (2005) Cid Fernandes, R., Mateus, A., Sodré, L., Stasińska, G., & Gomes, J. M. 2005, MNRAS, 358, 363, doi: 10.1111/j.1365-2966.2005.08752.x
  * Ciotti & Ostriker (2007) Ciotti, L., & Ostriker, J. P. 2007, ApJ, 665, 1038, doi: 10.1086/519833
  * Conroy & Gunn (2010) Conroy, C., & Gunn, J. E. 2010, ApJ, 712, 833, doi: 10.1088/0004-637X/712/2/833
  * Conroy et al. (2009) Conroy, C., Gunn, J. E., & White, M. 2009, ApJ, 699, 486, doi: 10.1088/0004-637X/699/1/486
  * Cummings et al. (2018) Cummings, J. D., Kalirai, J. S., Tremblay, P. E., Ramirez-Ruiz, E., & Choi, J. 2018, ApJ, 866, 21, doi: 10.3847/1538-4357/aadfd6
  * da Silva et al. (2022) da Silva, R., Crestani, J., Bono, G., et al. 2022, A&A, 661, A104, doi: 10.1051/0004-6361/202142957
  * Davies et al. (2009a) Davies, B., Origlia, L., Kudritzki, R.-P., et al. 2009a, ApJ, 694, 46, doi: 10.1088/0004-637X/694/1/46
  * Davies et al. (2009b) —. 2009b, ApJ, 696, 2014, doi: 10.1088/0004-637X/696/2/2014
  * de Vaucouleurs (1991) de Vaucouleurs, G. 1991, Science, 254, 1667
  * Do et al. (2015) Do, T., Kerzendorf, W., Winsor, N., et al. 2015, ApJ, 809, 143, doi: 10.1088/0004-637X/809/2/143
  * Dotter (2016) Dotter, A. 2016, ApJS, 222, 8, doi: 10.3847/0067-0049/222/1/8
  * Ekström et al. (2012) Ekström, S., Georgy, C., Eggenberger, P., et al. 2012, A&A, 537, A146, doi: 10.1051/0004-6361/201117751
  * Eldridge et al. (2017) Eldridge, J. J., Stanway, E. R., Xiao, L., et al. 2017, PASA, 34, e058, doi: 10.1017/pasa.2017.51
  * Emsellem et al. (2022) Emsellem, E., Schinnerer, E., Santoro, F., et al. 2022, A&A, 659, A191, doi: 10.1051/0004-6361/202141727
  * Fisher et al. (2022) Fisher, D. B., Bolatto, A. D., Glazebrook, K., et al. 2022, ApJ, 928, 169, doi: 10.3847/1538-4357/ac51c8
  * Gadotti et al. (2019) Gadotti, D. A., Sánchez-Blázquez, P., Falcón-Barroso, J., et al. 2019, MNRAS, 482, 506, doi: 10.1093/mnras/sty2666
  * Gadotti et al. (2020) Gadotti, D. A., Bittner, A., Falcón-Barroso, J., et al. 2020, A&A, 643, A14, doi: 10.1051/0004-6361/202038448
  * Genovali et al. (2014) Genovali, K., Lemasle, B., Bono, G., et al. 2014, A&A, 566, A37, doi: 10.1051/0004-6361/201323198
  * Groves et al. (2023) Groves, B., Kreckel, K., Santoro, F., et al. 2023, MNRAS, 520, 4902, doi: 10.1093/mnras/stad114
  * Ho et al. (2015) Ho, I. T., Kudritzki, R.-P., Kewley, L. J., et al. 2015, MNRAS, 448, 2030, doi: 10.1093/mnras/stv067
  * Ho et al. (2017) Ho, I. T., Seibert, M., Meidt, S. E., et al. 2017, ApJ, 846, 39, doi: 10.3847/1538-4357/aa8460
  * Hou et al. (2000) Hou, J. L., Prantzos, N., & Boissier, S. 2000, A&A, 362, 921, doi: 10.48550/arXiv.astro-ph/0007164
  * Jałocha et al. (2010) Jałocha, J., Bratek, Ł., Kutschera, M., & Skindzier, P. 2010, MNRAS, 406, 2805, doi: 10.1111/j.1365-2966.2010.16887.x
  * Jang et al. (2018) Jang, I. S., Hatt, D., Beaton, R. L., et al. 2018, ApJ, 852, 60, doi: 10.3847/1538-4357/aa9d92
  * Jarrett et al. (2003) Jarrett, T. H., Chester, T., Cutri, R., Schneider, S. E., & Huchra, J. P. 2003, AJ, 125, 525, doi: 10.1086/345794
  * Kang et al. (2021) Kang, X., Chang, R., Kudritzki, R.-P., Gong, X., & Zhang, F. 2021, MNRAS, 502, 1967, doi: 10.1093/mnras/stab147
  * Kobayashi & Nomoto (2009) Kobayashi, C., & Nomoto, K. 2009, ApJ, 707, 1466, doi: 10.1088/0004-637X/707/2/1466
  * Kroupa (2001) Kroupa, P. 2001, MNRAS, 322, 231, doi: 10.1046/j.1365-8711.2001.04022.x
  * Kudritzki et al. (2015) Kudritzki, R.-P., Ho, I. T., Schruba, A., et al. 2015, MNRAS, 450, 342, doi: 10.1093/mnras/stv522
  * Kudritzki et al. (2021a) Kudritzki, R.-P., Teklu, A. F., Schulze, F., et al. 2021a, ApJ, 922, 274, doi: 10.3847/1538-4357/ac32cf
  * Kudritzki et al. (2021b) —. 2021b, ApJ, 922, 274, doi: 10.3847/1538-4357/ac32cf
  * Kudritzki et al. (2012) Kudritzki, R.-P., Urbaneja, M. A., Gazak, Z., et al. 2012, ApJ, 747, 15, doi: 10.1088/0004-637X/747/1/15
  * Lançon & Wood (2000) Lançon, A., & Wood, P. R. 2000, A&AS, 146, 217, doi: 10.1051/aas:2000269
  * Leroy et al. (2019) Leroy, A. K., Sandstrom, K. M., Lang, D., et al. 2019, ApJS, 244, 24, doi: 10.3847/1538-4365/ab3925
  * Lindblad (1999) Lindblad, P. O. 1999, A&A Rev., 9, 221, doi: 10.1007/s001590050018
  * Liu et al. (2022) Liu, C., Kudritzki, R.-P., Zhao, G., et al. 2022, ApJ, 932, 29, doi: 10.3847/1538-4357/ac69cc
  * Lo Faro et al. (2017) Lo Faro, B., Buat, V., Roehlly, Y., et al. 2017, MNRAS, 472, 1372, doi: 10.1093/mnras/stx1901
  * Maeder (1987) Maeder, A. 1987, A&A, 173, 247
  * Martins et al. (2008) Martins, F., Hillier, D. J., Paumard, T., et al. 2008, A&A, 478, 219, doi: 10.1051/0004-6361:20078469
  * Martins & Coelho (2007) Martins, L. P., & Coelho, P. 2007, MNRAS, 381, 1329, doi: 10.1111/j.1365-2966.2007.11954.x
  * Muñoz-Mateos et al. (2013) Muñoz-Mateos, J. C., Sheth, K., Gil de Paz, A., et al. 2013, ApJ, 771, 59, doi: 10.1088/0004-637X/771/1/59
  * Najarro et al. (2009) Najarro, F., Figer, D. F., Hillier, D. J., Geballe, T. R., & Kudritzki, R. P. 2009, ApJ, 691, 1816, doi: 10.1088/0004-637X/691/2/1816
  * Najarro et al. (2004) Najarro, F., Figer, D. F., Hillier, D. J., & Kudritzki, R. P. 2004, ApJ, 611, L105, doi: 10.1086/423955
  * Nogueras-Lara et al. (2020) Nogueras-Lara, F., Schödel, R., Gallego-Calvente, A. T., et al. 2020, Nature Astronomy, 4, 377, doi: 10.1038/s41550-019-0967-9
  * Origlia et al. (2013) Origlia, L., Oliva, E., Maiolino, R., et al. 2013, A&A, 560, A46, doi: 10.1051/0004-6361/201322586
  * Parikh et al. (2021) Parikh, T., Thomas, D., Maraston, C., et al. 2021, MNRAS, 502, 5508, doi: 10.1093/mnras/stab449
  * Pessa et al. (2023) Pessa, I., Schinnerer, E., Sanchez-Blazquez, P., et al. 2023, A&A, 673, A147, doi: 10.1051/0004-6361/202245673
  * Rauch (2003) Rauch, T. 2003, A&A, 403, 709, doi: 10.1051/0004-6361:20030412
  * Roškar et al. (2008) Roškar, R., Debattista, V. P., Quinn, T. R., Stinson, G. S., & Wadsley, J. 2008, ApJ, 684, L79, doi: 10.1086/592231
  * Salim & Narayanan (2020) Salim, S., & Narayanan, D. 2020, ARA&A, 58, 529, doi: 10.1146/annurev-astro-032620-021933
  * Sánchez-Blázquez et al. (2006) Sánchez-Blázquez, P., Peletier, R. F., Jiménez-Vicente, J., et al. 2006, MNRAS, 371, 703, doi: 10.1111/j.1365-2966.2006.10699.x
  * Sánchez-Blázquez et al. (2014) Sánchez-Blázquez, P., Rosales-Ortega, F., Méndez-Abreu, J., et al. 2014, Astronomy & Astrophysics, 570, A6
  * Schinnerer et al. (2023) Schinnerer, E., Emsellem, E., Henshaw, J. D., et al. 2023, ApJ, 944, L15, doi: 10.3847/2041-8213/acac9e
  * Schlafly & Finkbeiner (2011) Schlafly, E. F., & Finkbeiner, D. P. 2011, ApJ, 737, 103, doi: 10.1088/0004-637X/737/2/103
  * Sextl et al. (2023) Sextl, E., Kudritzki, R.-P., Zahid, H. J., & Ho, I. T. 2023, ApJ, 949, 60, doi: 10.3847/1538-4357/acc579
  * Smith et al. (2002) Smith, L. J., Norris, R. P. F., & Crowther, P. A. 2002, MNRAS, 337, 1309, doi: 10.1046/j.1365-8711.2002.06042.x
  * Spitoni (2015) Spitoni, E. 2015, MNRAS, 451, 1090, doi: 10.1093/mnras/stv1008
  * Stasińska et al. (2015) Stasińska, G., Costa-Duarte, M. V., Vale Asari, N., Cid Fernandes, R., & Sodré, L. 2015, MNRAS, 449, 559, doi: 10.1093/mnras/stv078
  * Sánchez-Blázquez et al. (2011) Sánchez-Blázquez, P., Ocvirk, P., Gibson, B. K., Pérez, I., & Peletier, R. F. 2011, MNRAS, 415, 709, doi: 10.1111/j.1365-2966.2011.18749.x
  * Tabatabaei et al. (2013) Tabatabaei, F. S., Weiß, A., Combes, F., et al. 2013, A&A, 555, A128, doi: 10.1051/0004-6361/201321487
  * Thainá-Batista et al. (2023) Thainá-Batista, J., Cid Fernandes, R., Herpich, F. R., et al. 2023, MNRAS, 526, 1874, doi: 10.1093/mnras/stad2698
  * Venturi et al. (2018) Venturi, G., Nardini, E., Marconi, A., et al. 2018, A&A, 619, A74, doi: 10.1051/0004-6361/201833668
  * Weinberg et al. (2017) Weinberg, D. H., Andrews, B. H., & Freudenburg, J. 2017, ApJ, 837, 183, doi: 10.3847/1538-4357/837/2/183
  * Westmoquette et al. (2011) Westmoquette, M., Smith, L., & Iii, J. 2011, MNRAS, 414, 3719, doi: 10.1111/j.1365-2966.2011.18675.x
  * Whitmore et al. (2023) Whitmore, B. C., Chandar, R., Rodríguez, M. J., et al. 2023, ApJ, 944, L14, doi: 10.3847/2041-8213/acae94
  * Yuan et al. (2018) Yuan, F.-T., Argudo-Fernández, M., Shen, S., et al. 2018, A&A, 613, A13, doi: 10.1051/0004-6361/201731865
  * Zánmar Sánchez et al. (2008) Zánmar Sánchez, R., Sellwood, J. A., Weiner, B. J., & Williams, T. B. 2008, ApJ, 674, 797, doi: 10.1086/524940


# Video Instance Shadow Detection

Zhenghao Xing1,∗, Tianyu Wang1, , Xiaowei Hu2, , Haoran Wu1, Chi-Wing Fu1, and
Pheng-Ann Heng1
1 The Chinese University of Hong Kong
2 Shanghai Artificial Intelligence Laboratory Joint first authorsCorresponding
author<EMAIL_ADDRESS>
###### Abstract

Video instance shadow detection aims to simultaneously detect, segment,
associate, and track paired shadow-object associations in videos. This work
has three key contributions to the task. First, we design SSIS-Track, a new
framework to extract shadow-object associations in videos with paired tracking
and without category specification; especially, we strive to maintain paired
tracking even the objects/shadows are temporarily occluded for several frames.
Second, we leverage both labeled images and unlabeled videos, and explore
temporal coherence by augmenting the tracking ability via an association cycle
consistency loss to optimize SSIS-Track’s performance. Last, we build SOBA-
VID, a new dataset with 232 unlabeled videos of 5,863 frames for training and
60 labeled videos of 1,182 frames for testing. Experimental results show that
SSIS-Track surpasses baselines built from SOTA video tracking and instance-
shadow-detection methods by a large margin. In the end, we showcase several
video-level applications.

##  1 Introduction

Instance shadow detection [29, 27, 28] suggests finding the shadows together
with the associated objects in images to support photo-editing applications
and light direction estimation. With the explosive growth of videos on the
Internet, the demand for video editing is enormous. However, existing
approaches [29, 27, 28] provide mainly image-level editing, making it hard to
allow easy-to-use and good-quality video editing due to the lack of
considering the temporal dimension and also the shadow-object tracking.

Figure 1:  Visual comparison of results produced by SSIS-Track (ours) and by
SSIS+Mask2former. Each row shows three frames from a video clip. Detected
shadows and objects that are associated are marked in the same color. Please
zoom in to see details.

Video instance shadow detection (VISD) is a new task, aiming not only to find
shadows together with their associated objects in video frames but also to
track each shadow, each object, and each shadow-object association in the
video. VISD has great potential to benefit many applications,  _e.g_., (i)
instance copy-paste, in which we copy a shadow-object pair extracted from a
video and paste it together into another (or the same) video; Fig.
LABEL:fig:teaser shows an example. (ii) the vision system of autonomous
driving, for which we can enhance its object awareness by considering the
object shadows to help avoid blind-spot accidents when the object is occluded;
and (iii) video in-painting [15, 18], which obliterates the entire object by
simultaneously accessing the object mask with the associated shadow mask.

To achieve VISD is challenging, as we have to detect and track individual
shadows and objects, as well as their associations, over time. Intuitively, we
may combine methods for video instance segmentation and instance shadow
detection to achieve the task. However, doing so has several critical
drawbacks. First, existing approaches for video instance segmentation are
applicable only to instances of specific categories. Yet, VISD aims to detect
any foreground object with shadows. Second, methods for video instance
segmentation require rich features from the objects for tracking. Yet, shadow
instances have limited features. Last, existing methods for instance shadow
detection only focus on paired shadow-object associations, thus failing to
track unpaired shadow/object instances. Yet, shadows and objects in a video
may not be visible all the time together. If one of them is occluded or moves
out of view for a while, existing methods would lose track of them.

To address the drawbacks, we design SSIS-Track, a new end-to-end framework for
VISD that (i) detects object instances of any categories with their associated
shadow instances, (ii) tracks object/shadow instances and their associations,
and (iii) retrieves unpaired instances by using the detected instances from
adjacent video frames.

To achieve instance paired tracking, we build SSIS-Track by designing a
tracking branch over SSIS [27, 28]. First, we leverage the labeled images in
SOBA [29] to train the tracking branch by contrastive learning [5, 7], such
that it can learn to generate distinguishable tracking embeddings of
shadow/object instances. Specifically, we encourage the distance between the
tracking embeddings to be small when they are from the same instance. Also, we
collect unlabeled videos in SOBA-VID to refine the performance of SISS-Track
by exploring the temporal information in a self-supervised manner [10, 14,
30]. Further, we adopt a tracking-by-detection approach to match instances
frame by frame to enable instance paired tracking. Last, we design a new
retrieving mechanism to handle unpaired instances that typically happen when
an object/shadow is temporarily occluded or out of view in the video. Fig. 1
shows examples, where our SSIS-Track successfully detects various paired
shadow-object associations and retrieves an unpaired shadow instance in
videos.

To train SSIS-Track with less reliance on video instance annotations, we
compile SOBA-VID, a new dataset with a set of unlabeled videos for training
and another set of labeled videos for testing. Besides, for quantitative
performance evaluation on the VISD task, we formulate the SOAP-VID metric
based on SOAP [29] by considering Intersection-over-Union (IoU) computation in
a spatio-temporal manner[34]. In the end, we perform a series of experiments,
manifesting SSIS-Track’s effectiveness and applicability in video editing.

##  2 Related Work

#### Instance shadow detection

is a variant of instance segmentation, additionally considering the detection
of shadows and their associations with objects. LISA [29] is the first method
for the task, with a two-stage end-to-end framework to predict bounding boxes
and masks of shadow/object instances, as well as their associations, in
images. It also includes a strategy to pair shadow and object instances.
Recently, SSIS [27, 28] approaches the task with a single-stage fully
convolutional network architecture and a bidirectional relation learning
module to learn the relation between the shadow and object instances. Video
instance shadow detection (VISD) extends the instance shadow detection task to
the video domain. Its goal is to achieve instance shadow detection and also
instance paired tracking in videos. In this work, SSIS serves as a baseline of
our SSIS-Track.

#### Video instance segmentation.

The major difference between video instance segmentation (VIS) and video
instance shadow detection (VISD) is that VIS cannot effectively track shadow
instances with the associated objects. Mainstream approaches for the VIS task
are mainly online or offline.

Online methods, also known as tracking-by-detection, detect instances in each
frame and then match the detected instances in a frame-by-frame manner.
MaskTrack R-CNN [34] is the first online method for VIS, with a tracking head
added upon Mask R-CNN [8] to predict tracking embeddings of all the detected
instances and then use them for instance matching in videos. Similar methods
include VPSNet [12] and TrackR-CNN [26]. Other methods,  _e.g_., CrossVIS [35]
and STC [11], adopt a similar tracking head, but are built upon CondInst [24]
instead of Mask R-CNN.

Offline methods directly segment instances in videos by considering each
instance as a sub-volume in the 3D spatio-temporal volume of the input video.
Inspired by DETR [3], a transformer-based method VisTR [31] first considers
VIS as a sequence decoding problem. This idea is also adopted by IFC [9],
SeqFormer [32], and Mask2Former [4]. Recently, the first semi-supervised VIS
approach [5], which does not require video annotations, is proposed. So far,
existing approaches for VIS can only handle classification, detection,
segmentation, and tracking of instances for a maximum of 40 object categories
by training their deep models on the current datasets [34, 20]. Besides, the
mainstream approaches rely heavily on annotated videos. In comparison, SSIS-
Track has no reliance on video instance annotations in the training process
and detects foreground objects and their associated shadows without
identifying their categories.

#### Self-supervised learning for videos.

To obtain labeled video data is one of the most critical bottlenecks of
exercising deep learning on videos. This motivates the development of self-
supervised learning to take out the burden of annotating the videos.
Mainstream self-supervised learning approaches [10, 14, 30, 13, 33] learn
pixel or instance representation for downstream video tasks,  _e.g_., object
tracking, by exploiting temporal correspondence in unlabeled videos. As a
self-supervision signal, cycle consistency loss is often used to train the
tracking embedding. By doing so, we can finetune SSIS-Track on unlabeled
videos and further enhance SSIS-Track’s performance in paired tracking. Beyond
the existing methods, the tracking embedding for the correspondence module in
SSIS-Track is formed by, paired shadow and object embeddings where both space
and time correspondences are formulated during the training to promote paired
shadow-object tracking.

##  3 SOBA-VID Dataset

Figure 2: Example frames in our SOBA-VID dataset contain (a) shadow instances,
(b) object instances, and (c) their associations. More examples are presented
in (d).  Figure 3: Statistical properties of the SOBA-VID test set.

We build SOBA-VID (Shadow-OBject Association of VIDeos), a new dataset for
video instance shadow detection. When compiling it, we consider the following
criteria: (i) the collected videos must contain common shadow-object pairs;
(ii) the shadows in videos should be clear without ambiguity; (iii) the
instance categories and scenarios in the collected videos should be diverse,
for both the occluded shadows and objects; and (iv) we try to include videos
with motion backgrounds and diverse light directions.

Overall, SOBA-VID has 292 videos with 7045 frames, collected from (i) existing
datasets, DAVIS [2], YouTube-VIS [34], and OVIS [20], (ii) self-collected
videos, and (iii) keyword search on the Internet by “shadow plus common
instances.” We randomly split the videos into a training set (232 videos, 5863
frames) and a testing set (60 videos, 1182 frames). For the videos in test
set, we manually annotate the masks of each shadow/object instance frame by
frame using Apple Pencil and associate them to form shadow-object pairs (134
pairs in total); see Fig. 2 for some examples. Further, we analyze the
statistics of SOBA-VID. From Fig. 3, we can see that (i) SOBA-VID contains a
diverse number of paired shadow-object associations per video, with around
2.23 pairs on average; and (ii) as shown in the right sub-figure, the length
of both videos and instances are diverse and the average lengths of videos and
instances are $19.7$ and $17.2$, respectively. Note that Fig. 2 (d) shows
various example frames in SOBA-VID; please refer to the supplementary material
for more examples and results.

##  4 Methodology

Figure 4: The schematic illustration of our two-stage SSIS-Track framework. In
the first stage (top), the framework first learns from images with labels in a
supervised manner. Then, in the second stage (bottom), it learns from videos
without labels in a self-supervised manner.

###  4.1 Overall Network Architecture

Fig. 4 shows the overall architecture of SSIS-Track, which has two stages. (i)
Given labeled images from SOBA, we adopt SSIS [27] to learn to detect and
segment the shadow instances with its associated object instances by using
image-level supervision. Then, we leverage the tracking tower to dynamically
generate the network parameters, which will be assigned to the tracking head
to predict the tracking embeddings. (ii) Given the unlabeled videos from SOBA-
VID, we use the temporal correspondence to further improve the ability of the
tracking branch to perform paired tracking, which is trained in a self-
supervised manner. In the following, we elaborate on how to learn a tracking
embedding for instance paired tracking (Section 4.2) and introduce the
bidirectional retrieving mechanism to further enhance the tracking ability
(Section 4.3).

###  4.2 Instance Paired Tracking

Except training our framework on the SOBA dataset to teach it to perform
instance shadow detection on single images, we develop and incorporate various
new components,  _i.e_., tracking tower, tracking head, and tracking
controller, together forming the tracking branch, such that it can effectively
learn to track paired shadows and objects in videos. Procedure-wise, we first
train the newly-added tracking components on single images in a supervised
manner, and then adopt a self-supervised learning approach by training SSIS-
Track on unlabeled videos, such that we can avoid the burden of labeling masks
of paired shadows and objects frame-by-frame on videos.

#### Learning from labeled images.

First, we leverage labeled image data to train SSIS-Track. As shown in Fig. 4,
given a single image, we adopt the feature pyramid network [16] to extract
features. Beyond the box and class towers in SSIS [27, 28] to predict shadow
and object instances on single images, we add a tracking tower and a tracking
controller to generate parameters of the convolutional filters for each
shadow/object instance. The generated filters are then used in the tracking
head to predict the tracking embedding of each instance. The inputs to the
tracking head are the concatenation of the mask features and relative
coordinates of each instance; see [27, 28] for details.

To learn to track, we propose to make the tracking embeddings from the same
instance to be closer in the latent space, or more distant, otherwise. Here,
we adopt contrastive learning [5, 7]. Specifically, for the $i$-th instance
$\Omega_{i}$, we compute the center embedding $C_{i}$ as the average value of
all the embeddings belonging to this instance:

$C_{i}\ =\ \frac{1}{N_{i}}{\textstyle\sum_{e\in\Omega_{i}}}f_{e}\ ,$ (1)

where $f_{e}$ denotes the instance-level tracking embedding of $\Omega_{i}$
and $N_{i}$ denotes the number of locations on the feature maps belong to
$\Omega_{i}$ with classification score $\textgreater$ 0.05 [25]. To encourage
closer embeddings from the same instance, we adopt the center loss to minimize
the L1 distance:

$L_{i}^{center}\ =\ \sum_{e\in\Omega_{i}}\parallel C_{i}-f_{e}\parallel\ .$
(2)

To make the embeddings distinctive from embeddings of other instances, we push
the center embeddings of all the other instances further apart. To do so, we
compute a dense similarity matrix $Sim(i,j)$ that represents the similarity
between any two center embeddings:

$Sim(i,j)=\frac{exp(C_{i}^{T}\cdot
C_{j})}{{\textstyle\sum_{k=0}^{K}}exp(C_{i}^{T}\cdot C_{k})}\ ,$ (3)

where $K$ is the total number of instances. Note that we calculate the
similarity matrix for both shadows and objects, respectively, based on the
predicted classification results from the class tower, denoted as $Sim_{S}$
and $Sim_{O}$. Then, we introduce the contrast loss in terms of the cross
entropy $C_{en}$ to maximize the self-matching likelihood and push different
instances as far away as possible:

$L^{contra}\ =\ C_{en}(Sim_{S},I)+C_{en}(Sim_{O},I)\ ,$ (4)

where $I$ is an identity matrix.

By combining all $L_{i}^{center}$ and $L^{contra}$ together, we can jointly
optimize the framework to learn the tracking ability from the input images.

#### Learning from unlabeled videos.

Unlike the image data that provides labels for each instance, we adopt videos
without labels to train the framework through self-supervised learning. Given
two adjacent frames from in video, we first compute the tracking embeddings
for the pixels with classification score $\textgreater$ 0.05 on each instance.
Then, we adopt non-maximum suppression (NMS) to obtain a paired shadow-object
tracking embedding for each instance.

Afterward, we generate a transition matrix $A$ by computing the embedding
similarity of each pair of instances at different time frames, where each
element $A_{t}^{t+1}(i,j)$ stands for the transition probability of the
$i^{th}$ instance at time $t$ to the $j^{th}$ instance at time $t+1$. By using
the idea of cycle consistency loss [10, 14, 30], if the $i^{th}$ instance at
time $t$ is able to transit (similar) to the $j^{th}$ instance at time $t+1$,
the inverse transition should also hold. Thus, we encourage the diagonal
elements in the multiplication results of these two transition matrices
$A_{t}^{t+1}$ and $A_{t+1}^{t}$ to have large values by comparing them with an
identity matrix ($I$):

$L^{cyc}\ =\ C_{en}(A_{t}^{t+1}A_{t+1}^{t},I)\ .$ (5)

Note that cycle consistency loss in our approach adopted paired shadow-object
tracking embeddings and it is conducted in both directions. Fig. 4 shows the
detailed structure of performing self-supervised learning on unannotated
videos in our SSIS-Track framework.

#### Tracking paired instances across frames.

To perform instance paired tracking, we design a matching strategy to track
each paired shadow-object association across time frames following [34].
First, we assume $N$ instances in the tracking queue with their tracking
embeddings represented by the features of each instance. Then, we assume $M$
instances in the next video frame and compute their tracking embeddings.
Moreover, we obtain the similarity $Score(f_{i},f_{j})$ between the instances
in the frame and the instances in the tracking queue by computing the cosine
similarity of their tracking embeddings and normalizing via a bidirectional
softmax function:

$\begin{split}Score(f_{i},f_{j})=&[\frac{exp(cos(f_{i},f_{j}))}{{\textstyle\sum_{k=1}^{M}}exp(cos(f_{k},f_{j}))}\\\
&+\frac{exp(cos(f_{i},f_{j}))}{{\textstyle\sum_{k=1}^{N}}exp(cos(f_{i},f_{k}))}]/2\
,\end{split}$ (6)

where $f_{i}$ denotes the detected instance tracking embedding in the current
frame and $f_{j}$ denotes the latest instance tracking embedding in the
tracking queue. If the score is lower than a predefined matching threshold, we
consider the instance as a new identity object and put its embedding into the
tracking queue. Otherwise, we consider it as a tracked instance and update the
corresponding embedding in the tracking queue. Note that we concatenate the
tracking embeddings of the paired shadow-object associations as the overall
tracking embedding, which is used in this strategy.

Other cues including the IoU score and confidence score are also utilized in
the testing stage for tracking paired instances across frames, following [34].
For each newly-detected instance $i$, let $f_{i}$, $b_{i}$, and $c_{i}$ denote
its tracking embedding, bounding box, and confidence score, respectively. For
instance $j$ in the tracking queue with tracking embedding $f_{j}$ and
bounding box $b_{j}$, we can obtain the similarity score of the instances $i$
and $j$ by

$\begin{split}FinalScore(i,j)\ =\ \alpha Score(f_{i},f_{j})+\beta
IoU(b_{i},b_{j})+\gamma c_{i}\ ,\end{split}$ (7)

where $\alpha$, $\beta$, and $\gamma$ are hyperparameters to balance different
cues. Note that we only use Eq. (7) in testing.

###  4.3 Bidirectional Retrieving Mechanism for Unpaired Shadow/Object
Instances

Figure 5: The schematic illustration of the bidirectional retrieving mechanism
in our method.

SSIS [27, 28] is able to perform instance shadow detection only on individual
images and cannot detect shadow/object instances without paired object/shadow
instances. For video instance shadow detection, some shadows/objects may be
missing for a short while, due to occlusion or moved out of the view. Hence,
we should try to track individual shadow/object instances and find their
associated object/shadow instances when they re-appear in the video.

To achieve this, we formulate a bidirectional retrieving mechanism, which is
used only in the inference phase. In the matching procedure, we compute the
similarity matrix using Eq. (6) by taking the tracking embeddings of
individual shadow/object instances in the current frame and in the tracking
queue. If we can successfully match their tracking embeddings, we will update
the tracking embedding of the individual shadow/object instance in the queue
and keep the corresponding tracking embedding of the paired object/shadow
instance unchanged. Hence, we can track the paired shadow-object association
after it re-appears in the video. If we cannot match the tracking embeddings
in both the current frame and the tracking queue, we will ignore the
shadow/object instance, since we can only handle paired shadow-object
associations that appeared for at least one video frame. To further tackle the
situation, where the paired shadow-object associations are in later frames, we
will adopt this mechanism again but in a reverse order.

Fig. 5 shows two examples: (i) object $n$ in (a) disappears in time frame
$t-1$, but his shadow remains. We can successfully track the shadow instance
and re-match it with its object in time frame $t$; (ii) shadow $m$ appears
only in time frames $t-1$ and $t$. By using our retrieving mechanism in a
reverse order, we are able to match object $m$ at the previous time frame
$t-2$. Hence, by leveraging the tracking embeddings, we can retrieve unpaired
instances back to improve the performance, thus promoting better visual
effects on the applications of video instance shadow detection.

##  5 Experimental Results

###  5.1 Evaluation Metrics

To evaluate video instance shadow detection, we extend the evaluation metric
SOAP[29] for single-image instance shadow detection to videos. A sample is
considered a true positive when the Intersection-over-Union (IoU) between the
predicted and ground-truth shadow instances, object instances, and shadow-
object associations are all no less than a threshold $\tau$. Specifically, we
replace the IoU in SOAP with the spatio-temporal IoU in [34] and name the
updated metric as SOAP-VID. We report the average over multiple $\tau$
[0.5:0.05:0.95] in SOAP-VID. Using the spatio-temporal IoU, we also examine
the average precision for shadow/object instance and paired shadow-object
associations over the thresholds [0.5:0.05:0.95], which are denoted as
Instance AP and Association AP, respectively.

###  5.2 Implementation Details

#### Training parameters.

We train SSIS-Track by following the training strategies of CondInst [24] and
AdelaiDet [23]. For images, we adopt the weights of CondInst trained on COCO
[17] to initialize the weights of the backbone network, and then use the same
parameters in SSIS [27, 28] to train SSIS-Track on the SOBA dataset [29]. For
videos, we adopt the weights of SSIS-Track trained on images to initialize the
the model, set the mini-batch size as four, and optimize SSIS-Track on an
NVIDIA RTX 3090 GPU for 5,000 iterations. We set the initial learning rate as
$1e-5$ and reduce it to $1e-6$ at 4,500 iterations. Also, we re-scale the
input images without changing the aspect ratio, such that the shorter side is
no less than 672 pixels.

#### Inference.

We process input videos in an online manner. The tracking branch of SSIS-Track
predicts the tracking embeddings for both paired and unpaired shadow/object
instances frame by frame and matches the instances across frames using the
strategy in Section 4.2.

Table 1: Comparison with two baseline methods for video instance shadow
detection. Our method outperforms all the baseline methods by a large margin,
where the improvements over “SSIS + Mask2Former” on SOAP-VID, Association AP,
and Instance AP are 24.5%, 20.4%, and 20.6%, respectively.

Method | SOAP-VID | Association AP | Instance AP
---|---|---|---
SSIS + IoU Tracker[1] | 21.5 | 31.7 | 25.7
SSIS + Mask2Former[4] | 31.8 | 51.1 | 42.2
SSIS-Track | 39.6 | 61.5 | 50.9

Table 2: Ablation: major components on SOBA-VID test set.

Method With Tracking Branch Contrastive Learning Temporal Correspondence
Retrieving Mechanisim SOAP-VID Association AP Instance AP SSIS 29.7 40.0 36.5
SSIS-Track $\checkmark$ $\checkmark$ 36.8 55.4 45.2 $\checkmark$ $\checkmark$
$\checkmark$ 38.0 55.9 46.8 $\checkmark$ $\checkmark$ $\checkmark$
$\checkmark$ 39.6 61.5 50.9

###  5.3 Comparison with Baseline Methods

We consider two baselines in our comparison: (i) adopt SSIS [27] to detect
shadow instances, object instances, and shadow-object associations on each
individual frame and use the IoU Tracker [1] to match the instances across
different frames based on the IoU values between the detected bounding boxes;
and (ii) leverage the tracking algorithm Mask2former[4] to detect and track
the object instances across all video frames, use SSIS to detect the
shadow/object instances and shadow-object associations on each frame, and
merge the detection results with the tracking results based on the mask IoU of
each object instance.

Table 1 reports the comparison results. Our method significantly surpasses
both baselines under all the evaluation metrics on the SOBA-VID test set. The
IoU Tracker relies heavily on the quality of the detection results, whereas
Mask2Former can only recognize objects of the same category as those in the
training data [34]. In contrast, our approach benefits from the end-to-end
training and knowledge learned from the unlabeled videos; see Fig. 1 for
visual comparisons between SSIS-Track and “SSIS + Mask2Former”.

Table 3: Ablation: instance paired tracking on SOBA-VID test set.

Method Object Embedding Shadow Embedding SOAP-VID Asso. AP Inst. AP SSIS-Track
$\checkmark$ 38.4 59.7 49.3 $\checkmark$ 38.3 59.1 48.6 $\checkmark$
$\checkmark$ 39.6 61.5 50.9

Figure 6: (a) and (b) show visual comparisons between SSIS-Track and Omnimatte
[19]. (c) shows that SSIS-Track is able to find both static and dynamic
instance pairs. The yellow boxes show Omnimatte’s results, which are
contaminated with background noises. The red boxes show that the mask quality
of SSIS-Track is far better than Omnimatte.

###  5.4 Comparison with Omnimatte

Lu  _et al_. [19] present a new task that predicts a subject together with its
associated visual effects,  _e.g_., reflection, dust, and shadow. They design
a framework named Omnimatte that takes a video as the input and approaches
this task through self-supervised learning. However, this method requires
other forms of inputs, such as the instance masks, optical flows, and
homographies, which are provided by other methods [22, 4, 6]. Fig. 6 shows
visual comparisons between SSIS-Track and Omnimatte, where (a) and (b) show
that the masks predicted by Omnimatte contain regions of the background, and
also do not include the complete shadow regions. (c) indicates that our SSIS-
Track is applicable to static instances while Omnimate fails as also mentioned
in [19]. Note that Omnimatte takes about three and a half hours to generate
the associated masks for this 21-frames video, while our method takes only
around 18 seconds in a single RTX 3090 GPU.

###  5.5 Ablation Study

#### Component analysis.

We perform an ablation study by adding the major components,  _i.e_., tracking
branch with contrastive learning, temporal correspondence, and retrieving
mechanism, step by step, starting from the baseline of using SSIS, which
performs instance tracking by taking the output of the mask controller as the
tracking embedding. Table 2 reports the results, showing that each component
contributes to improving the overall performance. Note that contrastive
learning and temporal correspondence encourage the framework to generate
tracking embeddings for achieving paired tracking of instances while the
retrieving mechanism helps to find the missed unpaired instances.

#### Instance paired tracking analysis.

Next, we explore the effectiveness of instance paired tracking. As Table 3
shows, we achieve paired tracking by using tracking embeddings of individual
shadow/object instances. Yet, when using the concatenation of both the shadow
and object embeddings, the framework obtains the best performance for all the
evaluation metrics.

##  6 Applications

We showcase video-level applications of video instance shadow detection to
demonstrate the performance and applicability of SSIS-Track. As this paper can
only show static images, please watch the demo videos in the supplementary
material.

#### Video inpainting.

Video instance shadow detection can generate shadow-object association masks
for paired instances across frames, which can be used not only to remove
objects together with their associated shadows but also to replace the video
background and retain the objects with their shadows. Fig. 7 (a) shows some
snapshots of this video inpainting application using the extracted shadow-
object association masks provided by SSIS-Track. With the help of SSIS-Track,
the SOTA video inpainting method [15] can successfully remove two motorcycles
with their associated shadows simultaneously. Also, Fig. 7 (b) shows that our
shadow-object association masks can cooperate with Stable Diffusion [21] to
replace the video background and put the dancer on Mars with her associated
shadow.

Figure 7: Example sequences demonstrate the application of video instance
shadow detection in video inpainting.

#### Instance cloning.

Fig. 8 presents the application of video instance cloning based on video
instance shadow detection, where we can successfully generate many cinematic
video effects,  _e.g_., a man with a suitcase walks in a crowd flow. To
achieve this, we duplicate the paired shadow-object associations across frames
from one sequence to another with the adjusted frame rate and motion blur, the
final result is presented in Fig. 8 (c). In contrast, Fig. 8 (b) presents the
result supported by Mask2Former [4], where it (i) fails to detect the instance
in frames and also (ii) produces unrealistic effects without the associated
shadows.

Figure 8: A man walks in a crowd created using our results.

#### Shadow editing.

We further demonstrate the application of instance paired tracking in Fig. 9,
where the shadow of Usain Bolt is replaced with the shadow of a Cheetah. With
the help of the paired masks predicted by SSIS-Track across frames, we can
first remove the shadow of Usain Bolt via video inpainting [15]; see Fig. 9
(b). Then, we can acquire a sequence of shadows of a Cheetah (see Fig. 9 (c))
and adjust it to the object instance of Usain Bolt; see the final result in
Fig. 9 (d).

Figure 9: Example sequences demonstrate the application of video instance
shadow detection in shadow editing.

##  7 Conclusion

This paper presents the video instance shadow detection task, which requires
us to simultaneously detect, segment, associate, and track paired shadow-
object associations in videos. To approach this task, we formulate the SSIS-
Track framework, which is optimized to learn paired tracking of shadow and
object instances from labeled images in a supervised manner and then from
unlabeled videos in a self-supervised manner. We further design a
bidirectional retrieving mechanism to maintain paired tracking even the
objects/shadows are temporarily missed for several frames. Besides, we build
SOBA-VID, a new dataset with unlabeled videos for training and labeled videos
for testing. The experimental results show the superiority of our method in
multiple aspects. In the end, we showcase the applicability of video instance
shadow detection and its potential to support various video-level editing
applications.

#### Limitations.

Our SSIS-Track takes around 5000MB memory to process a 21-frame video of
resolution 1280 $\times$ 720\. In the future, we hope to develop a more
efficient model to perform this task. Also, we plan to explore video-editing
effects through generative models.

## References

  * [1] Erik Bochinski, Volker Eiselein, and Thomas Sikora.  High-speed tracking-by-detection without using image information.  In AVSS, pages 1–6, 2017.
  * [2] Sergi Caelles, Alberto Montes, Kevis-Kokitsi Maninis, Yuhua Chen, Luc Van Gool, Federico Perazzi, and Jordi Pont-Tuset.  The 2018 DAVIS challenge on video object segmentation.  arXiv:1803.00557, 2018.
  * [3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In ECCV, pages 213–229, 2020.
  * [4] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar.  Masked-attention mask transformer for universal image segmentation.  In CVPR, 2022.
  * [5] Yang Fu, Sifei Liu, Umar Iqbal, Shalini De Mello, Humphrey Shi, and Jan Kautz.  Learning to track instances without video annotations.  In CVPR, pages 8680–8689, 2021.
  * [6] Matthias Grundmann, Vivek Kwatra, and Irfan Essa.  Auto-directed video stabilization with robust l1 optimal camera paths.  In CVPR, pages 225–232, 2011.
  * [7] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In CVPR, pages 9729–9738, 2020.
  * [8] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask R-CNN.  In ICCV, pages 2961–2969, 2017.
  * [9] Sukjun Hwang, Miran Heo, Seoung Wug Oh, and Seon Joo Kim.  Video instance segmentation using inter-frame communication transformers.  In NeurIPS, pages 13352–13363, 2021.
  * [10] Allan Jabri, Andrew Owens, and Alexei Efros.  Space-time correspondence as a contrastive random walk.  In NeurIPS, pages 19545–19560, 2020.
  * [11] Zhengkai Jiang, Zhangxuan Gu, Jinlong Peng, Hang Zhou, Liang Liu, Yabiao Wang, Ying Tai, Chengjie Wang, and Liqing Zhang.  STC: Spatio-temporal contrastive learning for video instance segmentation.  ECCVW, 2022.
  * [12] Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon.  Video panoptic segmentation.  In CVPR, pages 9859–9868, 2020.
  * [13] Zihang Lai and Weidi Xie.  Self-supervised learning for video correspondence flow.  In BMVC, 2019.
  * [14] Xueting Li, Sifei Liu, Shalini De Mello, Xiaolong Wang, Jan Kautz, and Ming-Hsuan Yang.  Joint-task self-supervised learning for temporal correspondence.  In NeurIPS, 2019.
  * [15] Zhen Li, Cheng-Ze Lu, Jianhua Qin, Chun-Le Guo, and Ming-Ming Cheng.  Towards an end-to-end framework for flow-guided video inpainting.  In CVPR, pages 17562–17571, 2022.
  * [16] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In CVPR, pages 2117–2125, 2017.
  * [17] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: Common objects in context.  In ECCV, pages 740–755, 2014.
  * [18] Rui Liu, Hanming Deng, Yangyi Huang, Xiaoyu Shi, Lewei Lu, Wenxiu Sun, Xiaogang Wang, Jifeng Dai, and Hongsheng Li.  FuseFormer: Fusing fine-grained information in transformers for video inpainting.  In ICCV, pages 14040–14049, 2021.
  * [19] Erika Lu, Forrester Cole, Tali Dekel, Andrew Zisserman, William T. Freeman, and Michael Rubinstein.  Omnimatte: Associating objects and their effects in video.  In CVPR, pages 4507–4515, 2021.
  * [20] Jiyang Qi, Yan Gao, Yao Hu, Xinggang Wang, Xiaoyu Liu, Xiang Bai, Serge Belongie, Alan Yuille, Philip Torr, and Song Bai.  Occluded video instance segmentation: A benchmark.  IJCV, 2022.
  * [21] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-Resolution image synthesis with latent diffusion models.  In CVPR, pages 10684–10695, 2022.
  * [22] Zachary Teed and Jia Deng.  RAFT: Recurrent all-pairs field transforms for optical flow.  In ECCV, pages 402–419, 2020.
  * [23] Zhi Tian, Hao Chen, Xinlong Wang, Yuliang Liu, and Chunhua Shen.  AdelaiDet: A toolbox for instance-level recognition tasks.  https://git.io/adelaidet, 2019.
  * [24] Zhi Tian, Chunhua Shen, and Hao Chen.  Conditional convolutions for instance segmentation.  In ECCV, pages 282–298, 2020.
  * [25] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  FCOS: Fully convolutional one-stage object detection.  In ICCV, pages 9627–9636, 2019.
  * [26] Paul Voigtlaender, Michael Krause, Aljosa Osep, Jonathon Luiten, Berin Balachandar Gnana Sekar, Andreas Geiger, and Bastian Leibe.  MOTS: Multi-object tracking and segmentation.  In CVPR, pages 7942–7951, 2019.
  * [27] Tianyu Wang*, Xiaowei Hu*, Chi-Wing Fu, and Pheng-Ann Heng.  Single-stage instance shadow detection with bidirectional relation learning.  In CVPR, pages 1–11, 2021.
  * [28] Tianyu Wang*, Xiaowei Hu*, Pheng-Ann Heng, and Chi-Wing Fu.  Instance shadow detection with a single-stage detector.  IEEE TPAMI, pages 1–14, 2022.
  * [29] Tianyu Wang*, Xiaowei Hu*, Qiong Wang, Pheng-Ann Heng, and Chi-Wing Fu.  Instance shadow detection.  In CVPR, pages 1880–1889, 2020.
  * [30] Xiaolong Wang, Allan Jabri, and Alexei A. Efros.  Learning correspondence from the cycle-consistency of time.  In CVPR, pages 2566–2576, 2019.
  * [31] Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia.  End-to-end video instance segmentation with transformers.  In CVPR, pages 8741–8750, 2021.
  * [32] Junfeng Wu, Yi Jiang, Song Bai, Wenqing Zhang, and Xiang Bai.  SeqFormer: Sequential transformer for video instance segmentation.  In ECCV, pages 553–569, 2022.
  * [33] Jiarui Xu and Xiaolong Wang.  Rethinking self-supervised correspondence learning: A video frame-level similarity perspective.  In ICCV, pages 10075–10085, 2021.
  * [34] Linjie Yang, Yuchen Fan, and Ning Xu.  Video instance segmentation.  In ICCV, pages 5188–5197, 2019.
  * [35] Shusheng Yang, Yuxin Fang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu.  Crossover learning for fast online video instance segmentation.  In ICCV, pages 8043–8052, 2021.