# Fresh Insights on the Kinematics of M49’s Globular Cluster System with
MMT/Hectospec Spectroscopy

Matthew A. Taylor National Research Council of Canada Postdoctoral Fellow
National Research Council of Canada, Herzberg Astronomy and Astrophysics
Research Centre, 5701 West Saanich Road, Victoria, BC V9E 2E7 Canada  Youkyung
Ko Korea Astronomy and Space Science Institute, 776 Daedeok-daero, Yuseong-Gu,
Daejeon 34055, Korea  Patrick Côté National Research Council of Canada,
Herzberg Astronomy and Astrophysics Research Centre, 5701 West Saanich Road,
Victoria, BC V9E 2E7 Canada  Laura Ferrarese National Research Council of
Canada, Herzberg Astronomy and Astrophysics Research Centre, 5701 West Saanich
Road, Victoria, BC V9E 2E7 Canada  Eric W. Peng Kavli Institute for Astronomy
and Astrophysics, Peking University, Beijing 100871, China  Ann Zabludoff
Steward Observatory, University of Arizona, 933 N. Cherry Ave., Tucson, AZ
85721, USA  Joel Roediger National Research Council of Canada, Herzberg
Astronomy and Astrophysics Research Centre, 5701 West Saanich Road, Victoria,
BC V9E 2E7 Canada  Rubén Sánchez-Janssen UK Astronomy Technology Centre, Royal
Observatory, Blackford Hill, Edinburgh, EH9 3HJ, UK  David Hendel Department
of Astronomy and Astrophysics, University of Toronto, 50 St. George Street,
Toronto, ON M5S 3H4 Canada  Igor Chilingarian Center for Astrophysics —
Harvard and Smithsonian, 60 Garden St. MS09, Cambridge, MA 02138, USA
Sternberg Astronomical Institute, M.V. Lomonosov Moscow State University,
Universitetsky prospect 13, Moscow, 119234, Russia  Chengze Liu  Department of
Astronomy, School of Physics and Astronomy, and Shanghai Key Laboratory for
Particle Physics and Cosmology, Shanghai Jiao Tong University, Shanghai
200240, China  Chelsea Spengler Institute of Astrophysics, Pontificia
Universidad Católica de Chile, Av. Vicuña Mackenna 4860, 780436 Macul,
Santiago, Chile  Hongxin Zhang CAS Key Laboratory for Research in Galaxies and
Cosmology, Department of Astronomy, University of Science and Technology of
China, Hefei, Anhui 230026, China  School of Astronomy and Space Science,
University of Science and Technology of China, Hefei 230026, China

###### Abstract

We present the first results of an MMT/Hectospec campaign to measure the
kinematics of globular clusters (GCs) around M49—the brightest galaxy in the
Virgo galaxy cluster, which dominates the Virgo B subcluster. The data include
kinematic tracers beyond 95 kpc ($\sim\\!5.2$ effective radii) for M49 for the
first time, enabling us to achieve three key insights reported here. First,
beyond $\sim 20\arcmin-30\arcmin$ ($\sim 100-150$ kpc), the GC kinematics
sampled along the minor photometric axis of M49 become increasingly hotter,
indicating a transition from GCs related to M49 to those representing the
Virgo B intra-cluster medium. Second, there is an anomaly in the line-of-sight
radial velocity dispersion ($\sigma_{r,los}$) profile in an annulus $\sim
10-15\arcmin$ ($\sim 50-90$ kpc) from M49 in which the kinematics cool by
$\Delta\sigma_{r,los}\approx 150\,{\rm km}\,{\rm s}^{-1}$ relative to those
in- or outward. The kinematic fingerprint of a previous accretion event is
hinted at in projected phase-space, and we isolate GCs that both give rise to
this feature, and are spatially co-located with two prominent stellar shells
in the halo of M49. Third, we find a subsample of GCs with velocities
representative of the dwarf galaxy VCC 1249 that is currently interacting with
M49. The spatial distribution of these GCs closely resembles the morphology of
VCC 1249’s isophotes, indicating that several of these GCs are likely in the
act of being stripped from the dwarf during its passage through M49’s halo.
Taken together, these results point toward the opportunity of witnessing on-
going giant halo assembly in the depths of a cluster environment.

galaxies: clusters: individual (Virgo) — galaxies: formation — galaxies:
individual (M49) — galaxies: kinematics and dynamics — galaxies: star
clusters: general

††journal: ApJ††facilities: MMT(Hectospec), CFHT††software: astropy (Astropy
Collaboration et al., 2013), matplotlib (Hunter, 2007), numpy (Van Der Walt et
al., 2011) scikit-learn (Pedregosa et al., 2011), scipy (Virtanen et al.,
2020), topcat (Taylor, 2005)

##  1 Introduction

Low-surface brightness (LSB; $\mu_{V}\approx 26-29\,{\rm mag}\,{\rm
arcsec}^{-2}$) tidal debris like streams and shell systems are common around
giant elliptical galaxies (gE’s) in the nearby universe (e.g., Malin & Carter,
1980). Shell systems arise via phase mixing following the merger of one or
more galaxies, typically at a high mass ratio, along radial orbits (e.g.,
Quinn, 1984). During the merger, stars of the perturbed galaxies are jostled
into new orbits by differential tidal effects and temporarily align in phase
space leading to interleaved shells of stars apparent in deep imaging. While
such systems can be relatively long-lived after mergers in the field, the
loosely bound stellar debris can be destroyed rapidly through tidal
interactions in a cluster environment (Mihos, 2004).

While mergers with fainter ($M_{V}\gtrsim-15$ mag) satellites produce shell
structures generally dominated by individual stars and readily traced
kinematically via planetary nebulae (e.g., Longobardi et al., 2015a, b), they
can only be expected to contribute a handful of globular clusters (GCs)
despite hosting larger numbers per unit luminosity compared to their brighter
counterparts (e.g., Harris & van den Bergh, 1981; Georgiev et al., 2010). On
the other hand, more massive progenitors potentially contribute richer systems
of GCs to any individual event. Given enough observable GC tracers, such
shells are detectable through the phase-aligned kinematics that make shells
visible in the first place (e.g., Romanowsky et al., 2012), but can also trace
fine structure even in the absence of velocity information (Lim et al., 2017).

Kinematics derived from integrated diffuse stellar light is a useful tool to
investigate the formation of gE’s. A foundational result in this regard was
provided by (Dressler, 1979), who presented stellar line-of-sight radial
velocity dispersions ($\sigma_{r,los}$) for the brightest galaxy in the Abell
2029 cluster, IC 1101, out to several dozens of kpc. The average
$<\\!\sigma_{r,los}\\!>$-profile was seen to increase with radius, but fell
short of the $<\\!\sigma_{r,los}\\!>$ of the cluster as a whole. Positive
$<\\!\sigma_{r,los}\\!>$-profile gradients have since been found to be common
features of early-type galaxies in the local universe (often reaching the host
cluster dispersions) and can be interpreted as sampling the transition from
dynamics dominated by the host giant, to that more influenced by the intra-
cluster medium (ICM) (e.g., Loubser et al., 2008; Coccato et al., 2009;
Richtler et al., 2011; Murphy et al., 2014; Bender et al., 2015; Veale et al.,
2017; Barbosa et al., 2018).

The challenge of studying line-of-sight velocity distributions (LOSVDs) via
integrated diffuse stellar light, is that the surface-brightness of such
galaxies rapidly drops beyond the core regions, so that at distant radii,
spectral signal-to-noise drops to levels where the LOSVD cannot be reliably
recovered. A potential solution to this is to use other kinematic tracers like
GCs and PNe (e.g., Coccato et al., 2009; Longobardi et al., 2015a, b; Hartke
et al., 2018), which are relatively accessible kinematic tracers that reach
far out into the halos of their giant hosts, and richly populate the ICM of
the clusters in which their hosts reside.

M49 is the brightest galaxy in the Virgo galaxy cluster, and dominant member
of the Virgo B sub-cluster. M49 has a smooth, blue halo that was likely built
up early through the accretion of low-mass (${\cal M}_{\star}\lesssim
10^{8}\,M_{\odot}$; Hartke et al., 2018), low-metallicity satellites (Mihos et
al., 2013). It also exhibits multiple LSB features, including a complex,
interleaved shell system reaching out to 50-100 kpc (Janowiecki et al., 2010;
Ferrarese et al., 2012), with $(B-V)_{0}\approx 0.85$ mag colors that are
redder than the underlying halo (Mihos et al., 2013). Given the mean
$(B-V)_{0}\approx 0.77$ mag color of the surrounding dwarf galaxy reservoir
(van Zee et al., 2004), Mihos et al. (2013) argue that these shells are
consistent with the recent accretion of a single dwarf.

M49 is currently interacting with the dwarf galaxy VCC 1249 (e.g., Arrigoni
Battaia et al., 2012), lying at a projected distance of 5.4′ (26.2 kpc). VCC
1249 was imaged in $B$, $I$, and H I by Patterson & Thuan (1992), who found
evidence for ongoing interaction, including significant ram-pressure
stripping. Meanwhile, a tidal “plume” was found by Janowiecki et al. (2010) to
connect VCC 1249 and M49, further bolstering the notion that these two
galaxies are caught in the act of a high mass-ratio merger.

Here we present the first results from a wide spectroscopic survey of GCs
around M49, with a focus on those likely to be associated with recent and
ongoing interactions. Throughout this contribution we assume a distance to M49
of $16.7\pm 0.6$ Mpc ($(m-M)_{0}=31.12\pm 0.08$; Blakeslee et al., 2009),
corresponding to 4.9 kpc per arcminute and an effective radius ($r_{\rm eff}$)
of $226\pm 34$″(18.3 kpc) (Ferrarese et al., 2020).

##  2 Observations and Data Processing

Moderate resolution ($R=2000$) spectra were obtained between 2014 March and
June, using the fiber-fed multi-object spectrometer Hectospec (Fabricant et
al., 2005) mounted on the 6.5 m MMT, with 300 $1.5\arcsec$ diameter fibers
available across a $1^{\circ}$ diameter field of view. Given the spatial
extent of the intra-cluster medium (ICM) surrounding M49, we obtained spectra
from ten unique pointings (Fig. 1; orange shading). We chose a 270 mm-1
grating, giving a dispersion of 1.2 Å pixel-1 across the 3650-9200 Å
wavelength range. The integrated exposure time of each configuration ranges
from 4500 s to 7200 s, covering a total of $N_{\rm obj}$ = 3015 targets.

We reduced the spectra using the HSRED
pipeline111http://archive.mmto.org/node/536. The pipeline includes bias
subtraction, dark correction, flat-fielding, aperture extraction, and
wavelength calibration. We adopted the cross-correlation method (Tonry &
Davis, 1979) to measure heliocentric radial velocities of the targets, using
the xcsao task in the IRAF RVSAO package (Kurtz & Mink, 1998).

Figure 1: The spatial distribution of GCs in the vicinity of M49 with measured
$v_{r,los}$, overlayed on a DSS image. Dots denote GCs with colors indicating
galaxy associations as determined by our classification strategy described in
§ 3.1, as mapped to the legend at the top of the figure. The gold shaded
regions represent the ten MMT/Hectospec pointings.

##  3 Analysis

Here we describe a few aspects of our analysis, including the definition of
our GC sample, and an overview of the kinematics that leads to some of the key
aspects of the M49 system that we discuss in § 4.

###  3.1 GC Sample Definition

We identify GCs among the spectroscopic targets based on line-of-sight radial
velocities ($v_{r,los}$) and concentration indices ($c_{ap}$). $c_{ap}$ is
useful to discriminate between point-like and extended sources (e.g., Durrell
et al., 2014), and we define it as the difference between four and eight pixel
aperture-corrected magnitudes measured from Next Generation Virgo Survey
(NGVS; Ferrarese et al., 2012) $g^{\prime}$-band imaging. As a first cut, we
consider targets showing $v_{r,los}<3000\,{\rm km}\,{\rm s}^{-1}$ (while also
meeting our $c_{ap}$ criteria) to be GCs associated with Virgo, yielding 525
GCs in the survey region. While these criteria—in particular the inclusion of
morphometric information—should be very effective at filtering out foreground
stellar contaminants, we take a final step to cull potential stellar sources
by querying the Gaia Early Data Release 3 database (Gaia Collaboration et al.,
2016, 2020). Specifically we filter those objects with proper motions ($\mu$)
that imply unreasonable physical velocities at the distance of Virgo. We find
141 GCs with matched Gaia coordinates, of which 55 have $\mu$ measurements. Of
these, 13 show $\mu\gtrsim 1\arcsec\,{\rm yr}^{-1}$ (translating to physical
velocities $\gtrsim 10^{4}\,{\rm km}\,{\rm s}^{-1}$) and/or listed
uncertainties inconsistent with $\mu\approx 0\arcsec\,{\rm yr}^{-1}$. Despite
confidence in our above selection criteria, we err on the side of caution and
remove them from the subsequent analysis. We note that if all are true
foreground contaminants, this suggests an upper limit of $\sim 2.5$ percent
contamination to our sample prior to their omission.

We desire to isolate the kinematics of M49 and the Virgo B ICM, and attempt to
do so by using spatial and kinematic information to clean the sample of GCs
that may be members of other galaxies in the survey footprint. Using galaxy
information compiled by the NGVS, we consider a GC to belong to a particular
galaxy, including M49 itself, if it is both projected within ten $r_{\rm eff}$
of a given galaxy, and the velocity difference between a GC and the galaxy is
smaller than three times the galaxy’s central line-of-sight velocity
dispersion ($\sigma_{0,gal}$; McElroy, 1995), i.e.,
$|v_{r,los}-v_{r,gal}|<3\sigma_{0,gal}$. If $r_{\rm eff}$ or $\sigma_{0,gal}$
are unknown (e.g., faint dwarf galaxies), we assume $30\arcsec$ and $50\,{\rm
km}\,{\rm s}^{-1}$, respectively. Finally, if a GC satisfies the criteria to
be associated with M49 and another galaxy, we assign that GC to the other
galaxy (so that only those exclusively satisfying our criteria for M49 are
assigned to it), and those that do not satisfy the criteria for any galaxy are
assumed to belong to the ICM.

We show the results of these efforts in Fig. 1, where circles denote
$10$$r_{\rm eff}$ of galaxies in the region, and dots indicate GCs with
measured $v_{r,los}$. Colors are matched between GCs and galaxies according to
the legend shown above the figure, and grey circles correspond to galaxies for
which no GCs are identified. Several prominent galaxies in the region are
indicated, whose GC kinematics will be analyzed in a future work. Following
this procedure, we classify 58 and 415 GCs to the ICM and M49 respectively,
and do not consider the remaining 52 GCs likely to be associated with other
galaxies.

These observations build upon a previous spectral survey of 263 GCs sampling
the innermost $\sim 500\arcsec$ ($R_{\rm M49}$$\approx 40$ kpc) around M49
(Côté et al., 2003). To increase sampling fidelity in this region, we
incorporate their measurements in this work. We find 76 GCs to be in both
catalogues, such that our total MMT sample includes 449 GCs with $v_{r,los}$
measured for the very first time. This represents a $\sim 170$ percent
increase over previous catalogues, sampling a ten-fold increase in M49-centric
spatial extent. Focussing only on those GCs classified as either belonging to
M49 or the ICM, we continue with 660 unique spectroscopic GCs.

Finally, given the $997\,{\rm km}\,{\rm s}^{-1}$ recessional velocity and
$\sim 300\,{\rm km}\,{\rm s}^{-1}$ dispersion of M49, there remains the
potential for stellar contaminants to be falsely identified as GCs without
careful vetting. This is particularly true for targets at the lower range of
the GC LOSVD, and of larger concern for objects at large $R_{\rm M49}$, where
GCs tend toward bluer colors (see below) that more easily blend with the
optical colors of foreground stars. On top of the initial vetting of sources
using $c_{ap}$ as a discriminant, we pay extra scrutiny to those in the M49
outskirts ($R_{\rm M49}$$\gtrsim 100$ kpc) that also show $v_{r}\lesssim
100\,{\rm km}\,{\rm s}^{-1}$. We find eight GCs that satisfy these criteria
and find that, after careful consideration of visible source morphology,
concentration, multi-band color information, and existence (or lack) of Gaia-
based proper motions, we find only two that are likely of stellar origin, and
omit them from the analysis moving forward.

Figure 2: Our GMM color selection technique fit to our photometric sample of
M49 GC candidates, and applied to our spectroscopic sample. The upper panel
shows the color classification fit to the sample of Côté et al. (2003)
(triangles), using the metallicity sensitive $(C-T1)_{0}$ index, while below
we fit to the $(g^{\prime}-i^{\prime})_{0}$ colors of 4343 GC candidates
(small dots), and apply it to our spectroscopic GC sample (triangles). In each
panel we show the mean red/blue colors in each radial bin by the larger
colored dots, and the intersection between the two GMM components in purple.
Finally, in the lower panel we show the decreasing fraction of red GCs by the
black relation, corresponding to the right-hand ordinate.

####  3.1.1 GC Color Classification

We further refine the sample by classifying GCs into red (presumably metal-
rich) or blue (metal-poor) subsamples. GC systems around giant galaxies show a
well-known color/metallicity bimodality that reflects galactic formation
histories (e.g., Brodie & Strader, 2006). In one scenario, the blue/metal-poor
GCs form in-situ from pristine gas alongside their host, whereas gas-rich
mergers induce later starbursts that give rise to the red/metal-rich
population (e.g., Ashman & Zepf, 1992; Kissler-Patig, 1997; Beasley et al.,
2002). Alternatively, in a hierarchical assembly scenario giant galaxies
develop GC systems through a two-stage process in which red/metal-rich GCs
form early, deep in the potential wells and are rapidly enriched during the
early, intense star formation episodes of their giant hosts. Conversely, many
blue/metal-poor GCs are accreted over time through mergers with lower-mass
satellites and thus generally populate the outer regions of giant halos (e.g.,
Côté et al., 1998, 2000). Either way, years of observational evidence shows
that while red GCs tend to have distributions that closely reflect a giant’s
spheroidal light, blue GCs become more numerically dominant in the outer
galaxy halos (e.g., Geisler et al., 1996; Ashman & Zepf, 1998; Brodie &
Strader, 2006; Forbes et al., 2012).

Given the general shift toward bluer GCs in galaxy outskirts, we classify GCs
by color in six radial bins as a function of projected M49-centric distance
($R_{\rm M49}$) using a two-component Gaussian mixture model (GMM). To
minimize the risk of unknown color selection biases that may have arisen
during the MMT spectroscopic target selection, we incorporate 4343 GC
candidates photometrically selected from the NGVS that represents the parent
sample from which the spectroscopic targets were selected. In each of the six
radial bins, we use the two-component GMM to classify the total
sample—including the spectroscopic subsample—as either blue or red, where the
classification is defined based on the intersection of the two Gaussians in
each bin. Fig. 2 shows the results, where the distinction between GC colors is
indicated by the respective symbols, and the purple relations show
intersections between GMM components in each bin. The upper panel shows
results for the Côté et al. (2003) sample where for many GCs only their
metallicity-sensitive $(C-T1)_{0}$ color is available, and we apply the
technique separately but in the same fashion as the sample shown in the lower
panel.

In both panels of Fig. 2 the current dataset(s) are shown by triangles, while
in the lower panel the larger photometric GC candidate sample is shown by
fainter dots. Meanwhile, the mean colors of each bin are indicated by the
larger dots with $1\sigma$ uncertainties. For either sample, there is very
little variation with $R_{\rm M49}$, with peak-to-valley color variations
$\Delta(g^{\prime}-i^{\prime})_{0}\lesssim 0.1$ mag. We find 177 red and 470
blue GCs, with the latter increasingly dominant at large $R_{\rm M49}$. We
note the rapid drop-off of red GCs in our spectroscopic sample beyond $R_{\rm
M49}$$\approx 20\arcmin$ ($\approx 100$ kpc), and total lack of such beyond
$R_{\rm M49}$$\approx 50\arcmin$ ($\approx 240$ kpc), the implications of
which are discussed below.

###  3.2 One-dimensional Kinematic Profiles

Our results are based on the first two moments (i.e., $v_{r,los}$ and
$\sigma_{r.los}$) of the GC LOSVD, as profiled along the projected $R_{\rm
M49}$. To account for the ellipsoidal morphology of the M49 surface-brightness
profile, we recast the projected circular-based angular distance from M49’s
photometric center to its ellipsoidal major axis-equivalent distance based on
morphological properties measured by the NGVS, namely $r_{\rm eff,M49}=226\pm
34\arcsec$ (based on a curve-of-growth analysis applied to $g^{\prime}$-band
NGVS imaging) and ellipticity $\epsilon=0.15\pm 0.04$ with a major axis
position angle at $PA=-22\pm 2^{\circ}$ (East of North), with the latter
parameters calculated from M49’s isophotal center averaged between 1″ and one
$r_{\rm eff,M49}$.

Fig. 3 shows the overall one dimensional kinematic profile of M49, where the
running mean $<\\!v_{r,los}\\!>$\- and $<\\!\sigma_{r,los}\\!>$-profiles are
shown in the left- and right-hand columns, respectively. The profiles shown in
purple reflect the combined M49+ICM GC samples (see § 3.1) with no
consideration for GC color, below which we show the same for the red and blue
subsamples. Furthermore, we show in the lower quartet of panels a zoom-in to
the innermost $20\arcmin$ to highlight the dynamics in the immediate vicinity
of M49. For each profile, we radially bin the data centered on each GC in
turn, such that each bin contains the 35 nearest (in terms of 1D $R_{\rm
M49}$) GCs—15 for the red sub-sample, on account of their lower numbers—from
which the bi-weight mean $<\\!v_{r,los}\\!>$ and $<\\!\sigma_{r,los}\\!>$ are
calculated and plotted on Fig. 3 as a function of average $R_{\rm M49}$.
Similarly, and to place these data in the context of the overall Virgo B
subcluster, we show by the gold points/shading the running mean profiles of
galaxies strewn through the sampled region. In all cases, shading that
brackets the data represents the $2\sigma$ uncertainties calculated via 1000
bootstrap realizations.

Figure 3: One-dimensional radial $<\\!v_{r,los}\\!>$ (left) and
$<\\!\sigma_{r,los}\\!>$ (right) profiles, with the bottom quartet of panels
representing a zoom in of the top. Running mean kinematic profiles are shown
for the M49+ICM GC sample (purple; top panels in each quartet), and red/blue
GC subsamples (colored accordingly; bottom panels in each quartet). To place
the GC kinematics in the context of the overall Virgo B subcluster, gold
points/shading shows the running mean $<\\!v_{r,los}\\!>$ and
$<\\!\sigma_{r,los}\\!>$profiles for Virgo B galaxies, with data taken from
the NGVS. For comparison, we point out the locations of VCC 1249 and
associated stellar shells, while the orange relation shows the
$<\\!\sigma_{r,los}\\!>$ profile for PNe in the region from Hartke et al.
(2018). Dashed black lines indicate the M49’s systemic $v_{r,los}$. Overall,
we find quite complex GC kinematics interior to $\sim 20\arcmin$, outside of
which the profiles become smoother and show hints of merging with the Virgo B
ICM.

##  4 Results & Discussion

A complete analysis and discussion of the GC kinematics across the entire
field of view probed by the MMT/Hectospec data is reserved for a future work,
but here we comment on three noteworthy features in Fig. 3 that are described
and further explored below.

###  4.1 Transition to the Intra-cluster Medium

We first note the marked decline in the fraction of red GCs ($f_{R}$) out to
$R_{\rm M49}$$\approx 20-30\arcmin$ ($\approx 100-150$ kpc) that can be seen
in Figs. 2 and 3, which is illustrated by the black relation in Fig. 2 (lower
panel). Such a feature is observed to be common in the GC systems of giant
galaxies (e.g., Geisler et al., 1996; Ashman & Zepf, 1998; Brodie & Strader,
2006; Forbes et al., 2012), whether built up via the merger (e.g., Ashman &
Zepf, 1992; Kissler-Patig, 1997; Beasley et al., 2002) or hierarchical (e.g.,
Côté et al., 1998, 2000) scenarios. In the two-stage hierarchical model of
galaxy mass assembly a radial decline in $f_{R}$ is to be expected, as these
GCs are expected to form early, deep in M49’s potential well. Likewise, the
transition to bluer GCs is consistent with later accretion of GCs from
interactions with low-mass satellites and/or those thrown into the ICM by the
dynamically violent environment. We note that this latter interpretation is
not conclusive, as in at least some giant GC systems, the metallicities of
metal-poor GCs differs from that of GCs associated with dwarf galaxies (e.g.,
Strader et al., 2004), and we continue with this point in mind.

Figure 4: The $<\\!\sigma_{r,los}\\!>$ profile computed in radial bins for
direct comparison to the PNe kinematics from Hartke et al. (2018). The top
panel shows the $<\\!v_{r,los}\\!>$ profile for the present GC sample (solid
blue, red, and black relations), while the bottom shows
$<\\!\sigma_{r,los}\\!>$ with the dashed lines indicating the profile from
Hartke et al. (2018), split into their subsamples following the figure legend.
In both panels we show the binned profiles of Virgo B subcluster galaxies by
the gold relations.

We are interested in whether the kinematic profiles in this region hint at a
transition from GCs intrinsic to M49 to those dominated by the dynamics of the
ICM of the Virgo B subcluster. Fig. 4 shows a direct comparison to the results
of Hartke et al. (2018), who recently reported the transition to the ICM
through a study of M49 planetary nebulae (PNe), finding that their dynamics
begin to merge with that of the ICM near $R_{\rm M49}$$\approx 100$ kpc. This
result is consistent with large-scale positive $<\\!\sigma_{r,los}\\!>$
gradients seen in the PNe kinematics of M49 (Hartke et al., 2018; Pulsoni et
al., 2018), but also in the outer halos of other giant galaxies such that the
tracer kinematics merge with that of the surrounding ICM and indicates the
increasing influence of the local ICM (e.g., Dressler, 1979; Bender et al.,
2015; Barbosa et al., 2018). We find a similar signature in the transition
from the complex dynamics at $R_{\rm M49}$$\lesssim 100$ kpc, to the smoother
$<\\!\sigma_{r,los}\\!>$ profile beyond, where Fig. 3 shows the dispersion to
increase almost monotonically with $R_{\rm M49}$ from a relatively flat
$<\\!\sigma_{r,los}\\!>$$\approx 300\,{\rm km}\,{\rm s}^{-1}$ to the
$<\\!\sigma_{r,los}\\!>$$\approx 400-500\,{\rm km}\,{\rm s}^{-1}$ level more
consistent with the gold points/shading that indicate the
$<\\!\sigma_{r,los}\\!>$-profile arising from Virgo B subcluster giants. Based
on this, these results bolster those of Hartke et al. (2018), in that beyond
$R_{\rm M49}$$\,\approx\\!100$ kpc, GC dynamics become increasingly
representative of the Virgo B subcluster ICM.

We further investigate the LOSVD in Fig. 5, where—following similar previous
strategies (e.g., Longobardi et al., 2018)—we decompose the
$<\\!v_{r,los}\\!>$ (top panel) and $<\\!\sigma_{r,los}\\!>$ (bottom) profiles
along the major (dots/solid lines; $\Phi=-21^{\circ}$ East of North) and minor
(triangles/dashed lines; $\Phi=69^{\circ}$) axes. $R_{\rm M49}$ increases
along the SSE-NNW and WSW-ENE directions, and we build the 1D profiles by
sampling along each axis using a 30′-wide strip (i.e., selecting GCs within
$\pm 15\arcmin$ of each axis). The blue and red subsamples are colored
accordingly, with the red sample truncated at $40\arcmin$ ($\sim 200$ kpc)
owing to a dearth of red GCs with measured $v_{r,los}$ beyond this radius. For
all samples, $<\\!v_{r,los}\\!>$ are consistent with the systemic M49
$v_{r,los}=997\,{\rm km}\,{\rm s}^{-1}$ in the inner $\sim 25\arcmin$ ($\sim
100$ kpc), but each show a level of asymmetry at larger radii.

Looking at the major axis $<\\!v_{r,los}\\!>$ profile, a mild rotational
signature is evident in both blue and red samples, with a peak-to-valley
$\Delta v_{r,los}\approx 200\,{\rm km}\,{\rm s}^{-1}$. A more detailed
analysis of potential rotation at these radii will appear in a future work
alongside those of other giant galaxies in the region. Along the major axis,
both red and blue $<\\!v_{r,los}\\!>$ profiles behave similarly to the extent
of the sampled red GCs, but beyond $\sim 40\arcmin$ that of the blue GCs
becomes asymmetric with the SSE sample showing a continued increase out to
$\sim 50\arcmin$ before plunging by $\sim 200\,{\rm km}\,{\rm s}^{-1}$ beyond,
albeit with large uncertainties due to sparse GC sampling. Meanwhile, on the
other side of M49, the blue GC $<\\!v_{r,los}\\!>$ profile shows a steady
positive gradient back to the systemic $v_{r,los}$ of M49.

Switching attention to the $<\\!v_{r,los}\\!>$ profile along M49’s minor
photometric axis, we again report very similar behavior between red and blue
GCs to the extent of the red sample. We again see $<\\!v_{r,los}\\!>$
consistent with M49’s systemic $v_{r,los}$ out to $\sim 25\arcmin$, but beyond
this there is a bifurcation at negative $R_{\rm M49}$ with those toward the
WSW showing lower $<\\!v_{r,los}\\!>$ by $\gtrsim 100\,{\rm km}\,{\rm
s}^{-1}$. However, red GCs sampled toward the ENE closely follow the major
axis profile out to the limit of the sample such that the overall minor axis
profile is generally flat in the sampled region. For the minor axis blue GCs,
the profile is flat out to $\pm 50\arcmin$, and are in excellent agreement
with the $v_{r,los}$ of M49. Beyond $\sim 50\arcmin$, the profile plummets in
the NNW quadrant, although the sparesness of GCs and corresponding increase in
$<\\!v_{r,los}\\!>$ uncertainties preclude firm conclusions to be drawn here.

Figure 5: One-dimensional $<\\!v_{r,los}\\!>$ (upper panel) and
$<\\!\sigma_{r,los}\\!>$ (lower panel) profiles (e.g. Longobardi et al., 2018)
sampled along the major (solid/circles) and minor (dashed/triangles) M49
photometric axes. Values for $R_{\rm M49}$ increase from SSE-NNW along the
major axis, and from WSW-ENE along the minor axis. Blue and red GC profiles
are colored accordingly, while gold relations/shading shows the same kinematic
profiles derived in Fig. 3 but decomposed along each axis. Horizontal dashed
black relations represent the $v_{r,los}=997\,{\rm km}\,{\rm s}^{-1}$ and
$\sigma_{r,los}=303\,{\rm km}\,{\rm s}^{-1}$ of M49 itself. The decomposed
axial profiles show hints of major axis rotation inside of $\sim 50\arcmin$,
with divergent large-scale $<\\!\sigma_{r,los}\\!>$ gradients that indicate a
merging of GC kinematics with the surrounding Virgo B ICM.

The bottom panel of Fig. 5 shows the same as the top, but for the
$<\\!\sigma_{r,los}\\!>$ profiles. In all cases, the profiles exhibit very
flat behavior in the inner $\sim 25\arcmin$, with the blue GCs closely
matching M49’s $\sigma_{0}=303\,{\rm km}\,{\rm s}^{-1}$ (dashed black
relation), while the red GCs show a similarly flat profile, but with
kinematics cooler by $\sim 50\,{\rm km}\,{\rm s}^{-1}$. Beyond $\sim
25\arcmin$ the profiles begin to diverge. Similar to the $<\\!v_{r,los}\\!>$
profiles, the red GCs hint at a bifurcation, with those sampled along the
major axis showing a mildly positive gradient in contrast with those sampled
along the minor axis; however, we hesitate to make any strong inferences given
the dearth of data points and associated uncertainties.

Like the red GCs beyond $\sim 25\arcmin$, the blue sample shows divergent
behavior between the two axes; however, both axes show large scale gradients
that are also reflected in the PNe kinematics reported by Pulsoni et al.
(2018). Here we see an overall symmetric cooling of the
$<\\!\sigma_{r,los}\\!>$ profile along the major axis indicated by negative
gradients to the extent of the sampled region. In contrast, we see that the
positive gradient shown in Fig. 3 is largely due to the GCs that align with
M49’s minor photometric axis. Here the $<\\!\sigma_{r,los}\\!>$ profile rises
symmetrically, with a peak-to-valley heating by $\sim 200\,{\rm km}\,{\rm
s}^{-1}$. As in Fig. 3, we show the dynamics of the Virgo B subcluster
galaxies as a rolling average, but decomposed along the overall direction of
each axis. While there is a clear asymmetry in the Virgo B galaxy dynamics
(likely due to the overall unrelaxed state of Virgo (e.g., Binggeli et al.,
1985, 1987; Binggeli & Cameron, 1993)), the GC $<\\!\sigma_{r,los}\\!>$
profile rises to the $\sim 400-500\,{\rm km}\,{\rm s}^{-1}$ level consistent
with the overall signal shown by the galaxies. If we consider GCs sampled
along the minor axis beyond $50\,\arcmin$, we find an overall
$\sigma_{ICM}=425\pm 60\,{\rm km}\,{\rm s}^{-1}$ that compares well with the
ICM PNe sampled by Hartke et al. (2018), who estimate
$\sigma_{ICL}=397^{+38}_{-36}\,{\rm km}\,{\rm s}^{-1}$.

Figure 6: Projected phase space in $10\arcmin<$$R_{\rm M49}$$<50\arcmin$
($50\lesssim$$R_{\rm M49}$$/{\rm kpc}\lesssim 150$) where the ICM has
increasing influence on GC dynamics. Brown and orange data indicate the
distributions of GCs more likely to be dynamically influenced by M49 and the
ICM, respectively, with Gaussian distribution parameters indicated. Dashed and
solid black curves indicate robust $1\sigma$ and $2\sigma$ estimates used to
classify the two populations (e.g., Longobardi et al., 2015b).

To further investigate the transition from GCs dominated by M49 to those
increasingly influenced by the ICM, we pay particular attention to the region
$10\arcmin<$$R_{\rm M49}$$<50\arcmin$ ($50\lesssim$$R_{\rm M49}$$/{\rm
kpc}\lesssim 150$). Fig. 6 shows the projected phase-space for this region
sampled in four equal 10′-wide bins. As an attempt to isolate the two
populations, we adopt a “robust sigma” approach (McNeil et al., 2010;
Longobardi et al., 2015b). Briefly, in each bin we calculate the dispersion of
the total GC sample, and mask out $2\sigma$ outliers with respect to M49’s
systemic $v_{r,sys}=997\,{\rm km}\,{\rm s}^{-1}$. From the survivors, we
recalculate $\sigma$ and iterate using the Monte Carlo procedure outlined in
McNeil et al. (2010) to rescale it to that of a complete Gaussian
distribution. We apply this procedure in each of the four bins, and fit a
fourth-order polynomial to the robust $\sigma$ estimates to correct for radial
gradients. The dashed and solid black relations on Fig. 6 show the $1\sigma$
and $2\sigma$ thresholds from the polynomial fit, respectively, where we use
the latter to identify individual GCs as either belonging to the population
more likely to be under the influence of M49 itself (brown points/histogram)
or that of the ICM (orange).

Continuing to follow the approach of Longobardi et al. (2015b), we also show
on Fig. 6 representative Gaussians for each population, with the observed
$\mu$ and $\sigma$ listed in each bin/panel. We find 49 ICM GCs identified in
this range, and integrating across each radial bin suggests that 8 more ICM
GCs are indistinguishable from the main M49 sample, which implies 57 GCs in
this transition range associated with the Virgo B subcluster ICM. If we
consider the 30 GCs with projected $R_{\rm M49}$ beyond $50\arcmin$ to all be
part of the ICM population (as suggested by Figs. 3-5), this brings our total
number of implied ICM GCs to 87.

###  4.2 GCs Associated with Shells

Figure 7: A $g^{\prime}$-band image from the NGVS, centered on M49 and unsharp
masked to highlight LSB features, including VCC 1249 and the two visible
stellar shells to the Southeast. GCs in the elliptical annulus covering the
$R_{\rm M49}$ range encompassing the kinematic “dip” anomaly in Fig. 3 are
shown, with $v_{r,los}$ indicated by their colors as mapped to the upper color
bar. For reference, we indicate by the yellow dashed-line box the zoom-in
region centered on VCC 1249 and displayed in Fig. 9, inside of which are GCs
potentially associated with the dwarf galaxy with colors parametrizing
$v_{r,los}$ as mapped to the right-hand color bar. In each case, blue and red
GCs are denoted by circular and square symbols, respectively. As indicated by
the two colorbars, GCs inside the dashed box show $v_{r,los}$ are more
representative of that shown by the nearby dwarf, while those outside are
consistent with M49 and/or the visible shells to the SE.

The next feature in Fig. 3 is the rapid drop in the $<\\!\sigma_{r,los}\\!>$
profile shown by the cluster of points in $9\arcmin\\!\lesssim$$R_{\rm
M49}$$\\!\lesssim\\!15\arcmin$, which exhibit the kinematically coolest
signature of the entire sample. To investigate possible sources that might
give rise to this feature, we limit GCs to those within this annulus and
consider their distribution in $\Phi_{\rm M49}$ to search for evidence of
clustering caused by a background galaxy projected near to M49. A K-S test
provides no evidence to reject the notion that they are drawn from the same
distribution as the overall sample, which would not be the case if these GCs
were localized to a single host.

Fig. 7 shows two isolated subsamples—one corresponding to the nearby dwarf
galaxy VCC 1249 (see § 4.3), and the other to the current region of
interest—plotted over a $g^{\prime}$-band image of M49. The image has been
unsharp-masked to highlight sharp LSB features with a particular focus on two
shells to the Southeast of M49 (Janowiecki et al., 2010). All symbols outside
of the dashed box (denoting the region shown in Fig. 9, see § 4.3) represent
GCs that fall within the $9\arcmin\\!\lesssim$$R_{\rm
M49}$$\\!\lesssim\\!15\arcmin$ annulus that straddles the “dip” in the
$<\\!\sigma_{r,los}\\!>$-profile, with color parameterizing $v_{r,los}$ as
mapped to the upper color bar. Furthermore, we identify those likely to be
associated with the stellar shells (see below) by the circles (blue GCs) and
squares (red GCs).

We are interested in whether a portion of the GCs comprising the cold
kinematic signature might be associated with the two stellar shells, and as an
initial approach we isolate those that project on top of the visible shell
arcs. Isolating those GCs in the range $90^{\circ}\leq\Phi_{\rm M49}\leq
180^{\circ}$ results in 27 GCs (21 blue and six red) that collectively show
$<\\!\sigma_{r,los}\\!>$$=151\pm 27\,{\rm km}\,{\rm s}^{-1}$. One outlying GC
($v_{r,los}\lesssim 400\,{\rm km}\,{\rm s}^{-1}$) is unlikely to be associated
with the shells and its omission results in a somewhat lower
$<\\!\sigma_{r,los}\\!>$$=141\pm 18\,{\rm km}\,{\rm s}^{-1}$. These GCs
collectively make up the coolest kinematic feature seen in the data, and given
their projection alongside the two visible shells makes a physical association
plausible.

An expected signature of a GC accretion event is a chevron-like structure in
the projected phase-space distribution (e.g., Romanowsky et al., 2012). A
robust search and statistical analysis of such a feature requires detailed
simulations of M49’s GC system, which is beyond the scope of this
observational work. Despite this, we perform a search for the signature using
a friends-of-friends group-finding approach (e.g., Perrett et al., 2003;
Rudick et al., 2009), as successfully implemented for the M87 system
(Romanowsky et al., 2012). Briefly, this algorithm relies on a parametric
distance $\Delta s$ defined by the parameters $w_{R}$ and $w_{v}$, weighted by
$R_{\rm M49}$ and $v_{r,los}$, respectively. A third parameter is the linking
length $\Lambda$ so that groups of objects in the projected phase-space can be
identified by collections of objects that show $|\Delta s|<\Lambda$.

The group-finding results are sensitive to the adopted $w_{r}$, $w_{v}$, and
$\Lambda$ parameters, which need to be chosen to highlight real features in
the data while minimizing spurious groupings. This fine-tuning requires
detailed modelling and simulations that are beyond the scope of this work, and
we instead lean on Romanowsky et al. (2012), who tuned these parameters using
simulations for the M87 system finding $w_{r}=0.035$ dex, $w_{v}=50\,{\rm
km}\,{\rm s}^{-1}$, and $\Lambda=0.07$. Given the similarities between M49 and
M87 in terms of environment (i.e., dominant gEs of their respective Virgo
subclusters) and spatial scales of interest (e.g., $50\lesssim R/{\rm
kpc}\lesssim 100$), we adopt these parameters in our investigation.

Figure 8: Results of our group-finding algorithm and simulated phase space
distribution. The upper panel shows the group-finding output assuming $w_{R}$,
$w_{v}$, and $\lambda$ taken from the optimum parameters determined by
Romanowsky et al. (2012) for the M87 GC-shell system. Individual datapoints
are indicated by dots, where we highlight a handful of groups that make up a
chevron-like feature. In the bottom panel, we show the results of 1000
realizations of phase-space populated by simulated GCs. Color denotes the
total number of GCs populating each cell in the diagram, and we note the lack
of any obvious substructure in the region where the stellar shells are
visible.

The upper panel of Fig. 8 shows the outcome of the group-finding using our
adopted $w_{r}$, $w_{v}$, and $\lambda$. The top panel shows the projected
phase-space, where dots show individual GCs as linked by the algorithm output.
We find a tentative detection of a handful of groups (highlighted red) that
comprise a chevron-like feature with edges corresponding to the coolest
kinematic feature discussed above, and terminating near the systemic velocity
of M49 (horizontal dashed line). A formal estimate of the statistical
significance of this feature requires detailed simulation work, but we show a
qualitative assessment in the bottom panel of Fig. 8. Here we simulate the GC
population by drawing from the observed spatial distribution of GCs and a
$N(\mu_{obs},\sigma_{obs})$ distribution in $v_{r,los}$ where $\mu_{obs}$ and
$\sigma_{obs}$ are the observed mean and dispersion of our sample $v_{r,los}$.
The colormap in the bottom panel of Fig. 8 shows the number density of GCs
that results from 1000 Monte Carlo realizations, with the groups making up the
tentative chevron feature overplotted in red. While the observed shape mimics
the overall shape of the phase-space envelope, if these particular groupings
were common, they would form a recognizable structure in this region of phase-
space. We conclude that, while far from a “smoking gun” detection of a clear
kinematic fingerprint, this feature is at least consistent with these GCs
being associated with a shell arising from a recent merger with M49 and/or the
ongoing interaction between it and VCC 1249.

We find a total of 38 GCs (seven of which are red) to make up this kinematic
feature, which are a subset of the overall sample falling in the
$9\arcmin\\!\lesssim$$R_{\rm M49}$$\\!\lesssim\\!15\arcmin$ annulus. We
differentiate them by the squares/circles mapped to the upper color bar in
Fig. 7. When we consider only this sample, we find the combined red/blue GCs
to show a very cold $<\\!\sigma_{r,los}\\!>$$=84\pm 8\,{\rm km}\,{\rm
s}^{-1}$, while those that project directly on the visible shells arcs (i.e.
$90^{\circ}<\Phi_{\rm M49}<180^{\circ}$) exhibit a remarkably low
$<\\!\sigma_{r,los}\\!>$$=61\pm 10\,{\rm km}\,{\rm s}^{-1}$. Given that these
GCs make up a tentative kinematic fingerprint of a recent accretion event, and
that they represent the coldest kinematic signature driving the dip in the
$<\\!\sigma_{r,los}\\!>$-profile seen in Fig. 3, we consider it plausible or
even likely that at least a significant fraction of these 38 GCs are caught in
the act of being accreted onto M49’s outer halo.

###  4.3 GCs Associated with VCC 1249

Figure 9: A cutout image corresponding to the yellow dashed box displayed on
Fig. 7, centered on VCC 1249 and with isophotal contours oveprlotted in red.
Circles and squares indicate five blue and two red GCs, respectively, with
$v_{r,los}$ consistent with VCC 1249 itself.

The ongoing interaction (e.g. Arrigoni Battaia et al., 2012) between M49 and
the nearby (5.4′; 26 kpc) dwarf galaxy VCC 1249, presents an excellent
opportunity to identify GCs in the act of being accreted onto the giant halo.
Hartke et al. (2018) report a kinematic feature marked by a cluster of PNe
likely to be associated with the dwarf that are spatially coincident, share a
systemic $v_{r,los}$ indistinguishable from VCC 1249 itself, and show a
collective $\sigma_{\rm PNe,VCC1249}=40^{+28}_{-19}\,{\rm km}\,{\rm s}^{-1}$.

While Hartke et al. (2018) were able to isolate a kinematically cooler subset
of PNe corresponding to VCC 1249’s $\sigma_{\rm VCC\,1249}\approx 40\,{\rm
km}\,{\rm s}^{-1}$, we find no clear group of GCs that correspond directly in
phase-space with the dwarf. However, recognizing that GCs may be in the
process of being stripped from VCC 1249 (i.e. may not exactly coincide
spatially), we search for candidates in a similar fashion to that described in
§ 3.1. Specifically, we isolate GCs lying within five $r_{\rm eff}$ of the
dwarf ($r_{\rm eff,VCC\,1249}=48\arcsec$; 3.65 kpc; Ferrarese et al., 2020)
and with $|v_{r,los}-v_{r,{\rm VCC\,1249}}|<3\sigma_{0,{\rm VCC\,1249}}$. We
find seven GCs meeting these criteria, including two red and five blue, which
are indicated on Fig. 7 by circles (blue GCs) and squares (red) nearest the
dwarf galaxy. Color parameterizes GC $v_{r,los}$ as mapped to the right-hand
color gradient. These GCs are not uniformly distributed around VCC 1249,
showing a stream-like distribution toward M49, and no candidates found
Eastward of the dwarf.

Fig. 9 shows a zoom-in corresponding to the dashed box shown on Fig. 7,
centered on VCC 1249, with a field of view of ten $r_{\rm eff,VCC\,1249}$ per
side. Overplotted are the seven VCC 1249 GC candidates, as well as isophotal
contours (red curves). Noteworthy in Fig. 9 is the clear banana-like shape of
the VCC 1249 isophotes, closely approximating the overall boomerang shape of
the GC distribution. A stream-like feature can be expected from GCs being
stripped in such an interaction and based on the similar morphologies shown by
the GCs and VCC 1249 isophotes, we posit that we are witnessing these GCs
actively being accreted into M49’s halo.

In the crowded environment surrounding M49, there may be a non-negligible
chance that we select GCs that happen to spatially coincide with VCC 1249,
while also sharing similar kinematics. To quantify this chance, we carry out
10 000 Monte Carlo realizations mimicking the selection of the VCC 1249 GC
candidates and consider the absolute number of GCs selected in each
realization, as well as their overall spatial distributions. For each
realization, we generate a random set of $(\alpha,\delta)$ coordinates within
five $r_{\rm eff,M49}$ of M49 itself (to only sample regions of similar GC
density as VCC 1249) and (as done for the VCC 1249 GC candidate selection)
isolate those within five $r_{\rm eff,VCC1249}$ of that point with
$|v_{r}-v_{r,{\rm VCC1249}}|\leq 3\sigma_{r,{\rm VCC1249}}$. We further
consider only those realizations that result in $N_{gc}=7\pm 2$ GCs, resulting
in 537 (5.37 percent) samples that meet this criteria. The elongated
distribution of the VCC 1249 GC candidates provides another constraint, and to
account for this we find the best fitting ellipse whose edge follows the GC
distribution and consider its ellipticity, $e_{\rm samp}$, in comparison to
that describing a fit to the VCC 1249 candidates, $e_{\rm VCC1249}$. Cases
where $e_{\rm samp}>e_{\rm VCC1249}$ imply more elongated distributions and
taking this into account alongside $N_{gc}$, results in 82 (0.82 percent)
samples with similarly elongated shapes. We conclude that at least some of
these GCs are very likely to be members of the VCC 1249 GC system currently
being accreted by M49 itself.

##  5 Summary and Conclusions

We have presented the first results from a spectroscopic survey of GCs around
the brightest galaxy in the Virgo galaxy cluster, M49. We use spatial and
velocity information to clean the sample of GCs belonging to the myriad
galaxies in the field, resulting in a sample consisting of GCs associated with
M49 itself, or which have no clear association with any galaxy in the Virgo B
subcluster. We further classified GCs as either blue (metal-poor) or red
(metal-rich), and analyzed their kinematic profiles alongside the overall
sample.

Beyond $\sim 100$ kpc of M49, the overall mean $<\\!v_{r,los}\\!>$ profile
declines and the kinematics become hotter, as evidenced by a positive
$<\\!\sigma_{r,los}\\!>$ gradient. We further decompose the kinematic profiles
along the major and minor photometric axes of M49, and find both symmetric and
asymmetric M49-centric profiles depending on both GC color and axis
subsamples. We find that the hotter kinematics in the extreme halo of M49 is
dominated by blue GCs sampled along M49’s minor axis, with a positive
$<\\!\sigma_{r,los}\\!>$ gradient that smoothly merges with the kinematics of
the Virgo B subcluster. We argue that this implies a transition from GCs
dominated by the potential well of M49 to that defined by the Virgo B ICM. We
apply a radially dependent robust sigma approach to further investigate this
transition region, and estimate the number of GCs in our sample that are most
likely to be kinematically ruled by the larger subcluster dynamics.

A notable dip in the overall $<\\!\sigma_{r,los}\\!>$ profile reveals a
population of GCs with particularly cold kinematics, a subsample of which are
spatially co-located with the two stellar shells in projection. From these
GCs, we find a hint of a kinematic feature in projected phase-space, and
identify the most likely GCs to be genuinely associated with the visible
shells. The blue colors of these GCs, combined with their coherent kinematics
and spatial alignment suggests that at least some of these GCs have been
caught in the act of being accreted onto M49’s halo.

Finally, we report the spatial and kinematic alignment of a group of seven GCs
that coincide with the nearby dwarf galaxy VCC 1249, which is currently
interacting with M49 itself. Isolating GCs with $v_{r,los}$ consistent with
the dwarf, and projecting nearby reveals a stream stretching toward M49, with
a distribution that mimics the isophotal contours of VCC 1249. We attempt to
quantify the statistical significance of the kinematic/spatial alignment with
the dwarf, finding them to be significant at the $3\sigma$ level, suggesting
that msot of these GCs are currently being stripped from VCC 1249 as it passes
through M49’s halo.

Taken together, these data appear to straddle the transition from the GC
system intrinsic to the dominant galaxy of the Virgo B subcluster to those
ruled by Virgo ICM dynamics. Additionally, we seem to have come across the
rare opportunity to study a giant galaxy caught in the act of accreting GCs
into its outer halo. Further exploitation of these data will shed valuable
light on the process of halo mass assembly of a giant galaxy in a cluster
environment.

We extend our gratitude to the anonymous referee, whose detailed comments
served to greatly improve the original work. We also thank J. Hartke, who
kindly provided data allowing a direct comparison to their previous work.
M.A.T., P.C., L.F., and J.R. are supported by the National Research Council of
Canada (NRC). I.C.’s research is supported by the Smithsonian Astrophysical
Observatory Telescope Data Center, the Russian Science Foundation grant
17-72-20119 and the Program of the Interdisciplinary Scientific and
Educational School of MV. Lomonosov Moscow State University “Fundamental and
Applied Space Research”. C.L. acknowledges support from the NSFC, Grant No.
11673017, 11833005, 11933003. This research uses data obtained through the
Telescope Access Program (TAP), which has been funded by the National
Astronomical Observatories, Chinese Academy of Sciences, and the Special Fund
for Astronomy from the Ministry of Finance. Observations reported here were
obtained at the MMT Observatory, a joint facility of the University of Arizona
and the Smithsonian Institution and on observations obtained with
MegaPrime/MegaCam, a joint project of the Canada-France-Hawaii Telescope
(CFHT) and CEA/DAPNIA, at the CFHT which is operated by the NRC, the Institut
National des Sciences de l’Univers of the Centre National de la Recherche
Scientifique of France, and the University of Hawaii. We thank Perry Berlind,
Erin Martin, and the MMT staff for their professional support of these
observations. This research has made use of NASA’s Astrophysics Data System
Bibliographic Services, and NASA’s Extragalactic Data System.

## References

  * Arrigoni Battaia et al. (2012) Arrigoni Battaia, F., Gavazzi, G., Fumagalli, M., et al. 2012, A&A, 543, A112
  * Ashman & Zepf (1992) Ashman, K. M. & Zepf, S. E. 1992, ApJ, 384, 50
  * Ashman & Zepf (1998) Ashman, K. M. & Zepf, S. E. 1998, Globular cluster systems / Keith M. Ashman, Stephen E. Zepf. Cambridge, U. K. ; New York : Cambridge University Press, 1998
  * Astropy Collaboration et al. (2013) Astropy Collaboration, Robitaille, T. P., Tollerud, E. J., et al. 2013, A&A, 558, A33
  * Barbosa et al. (2018) Barbosa, C. E., Arnaboldi, M., Coccato, L., et al. 2018, A&A, 609, A78
  * Beasley et al. (2002) Beasley, M. A., Baugh, C. M., Forbes, D. A., et al. 2002, MNRAS, 333, 383
  * Bender et al. (2015) Bender, R., Kormendy, J., Cornell, M. E., et al. 2015, ApJ, 807, 56
  * Binggeli et al. (1985) Binggeli, B., Sandage, A., & Tammann, G. A. 1985, AJ, 90, 1681
  * Binggeli et al. (1987) Binggeli, B., Tammann, G. A., & Sandage, A. 1987, AJ, 94, 251
  * Binggeli & Cameron (1993) Binggeli, B. & Cameron, L. M. 1993, A&AS, 98, 297
  * Blakeslee et al. (2009) Blakeslee, J. P., Jordán, A., Mei, S., et al. 2009, ApJ, 694, 556
  * Brodie & Strader (2006) Brodie, J. P., & Strader, J. 2006, ARA&A, 44, 193
  * Coccato et al. (2009) Coccato, L., Gerhard, O., Arnaboldi, M., et al. 2009, MNRAS, 394, 1249
  * Côté et al. (1998) Côté, P., Marzke, R. O., & West, M. J. 1998, ApJ, 501, 554
  * Côté et al. (2000) Côté, P., Marzke, R. O., West, M. J., et al. 2000, ApJ, 533, 869
  * Côté et al. (2003) Côté, P., McLaughlin, D. E., Cohen, J. G., et al. 2003, ApJ, 591, 850
  * Dressler (1979) Dressler, A. 1979, ApJ, 231, 659
  * Durrell et al. (2014) Durrell, P. R., Côté, P., Peng, E. W., et al. 2014, ApJ, 794, 103
  * Fabricant et al. (2005) Fabricant, D., Fata, R., Roll, J., et al. 2005, PASP, 117, 1411
  * Fabricant et al. (2008) Fabricant, D. G., Kurtz, M. J., Geller, M. J., et al. 2008, PASP, 120, 1222
  * Ferrarese et al. (2012) Ferrarese, L., Côté, P., Cuillandre, J.-C., et al. 2012, ApJS, 200, 4
  * Ferrarese et al. (2020) Ferrarese, L., Côté, P., MacArthur, L. A., et al. 2020, ApJ, 890, 128
  * Forbes et al. (2012) Forbes, D. A., Ponman, T., & O’Sullivan, E. 2012, MNRAS, 425, 66
  * Gaia Collaboration et al. (2016) Gaia Collaboration, Prusti, T., de Bruijne, J. H. J., et al. 2016, A&A, 595, A1. doi:10.1051/0004-6361/201629272
  * Gaia Collaboration et al. (2020) Gaia Collaboration, Brown, A. G. A., Vallenari, A., et al. 2020, arXiv:2012.01533
  * Gavazzi et al. (2003) Gavazzi, G., Boselli, A., Donati, A., et al. 2003, A&A, 400, 451
  * Geisler et al. (1996) Geisler, D., Lee, M. G., & Kim, E. 1996, AJ, 111, 1529
  * Georgiev et al. (2010) Georgiev, I. Y., Puzia, T. H., Goudfrooij, P., et al. 2010, MNRAS, 406, 1967
  * Harris & van den Bergh (1981) Harris, W. E. & van den Bergh, S. 1981, AJ, 86, 1627
  * Hartke et al. (2018) Hartke, J., Arnaboldi, M., Gerhard, O., et al. 2018, A&A, 616, A123
  * Hunter (2007) Hunter, J. D. 2007, IEEE Computer Soc., 9, 90
  * Janowiecki et al. (2010) Janowiecki, S., Mihos, J. C., Harding, P., et al. 2010, ApJ, 715, 972
  * Kim et al. (2000) Kim, E., Lee, M. G., & Geisler, D. 2000, MNRAS, 314, 307
  * Kissler-Patig (1997) Kissler-Patig, M. 1997, A&A, 319, 83
  * Kormendy et al. (2009) Kormendy, J., Fisher, D. B., Cornell, M. E., et al. 2009, ApJS, 182, 216
  * Kurtz & Mink (1998) Kurtz, M. J., & Mink, D. J. 1998, PASP, 110, 934
  * Longobardi et al. (2015a) Longobardi, A., Arnaboldi, M., Gerhard, O., et al. 2015, A&A, 579, L3
  * Longobardi et al. (2015b) Longobardi, A., Arnaboldi, M., Gerhard, O., et al. 2015, A&A, 579, A135
  * Longobardi et al. (2018) Longobardi, A., Arnaboldi, M., Gerhard, O., et al. 2018, A&A, 620, A111
  * Loubser et al. (2008) Loubser, S. I., Sansom, A. E., Sánchez-Blázquez, P., et al. 2008, MNRAS, 391, 1009
  * Lim et al. (2017) Lim, S., Peng, E. W., Duc, P.-A., et al. 2017, ApJ, 835, 123
  * Malin & Carter (1980) Malin, D. F., & Carter, D. 1980, Nature, 285, 643
  * McElroy (1995) McElroy, D. B. 1995, ApJS, 100, 105
  * McNeil et al. (2010) McNeil, E. K., Arnaboldi, M., Freeman, K. C., et al. 2010, A&A, 518, A44
  * Mihos (2004) Mihos, J. C. 2004, Clusters of Galaxies: Probes of Cosmological Structure and Galaxy Evolution, 277
  * Mihos et al. (2013) Mihos, J. C., Harding, P., Rudick, C. S., et al. 2013, ApJ, 764, L20
  * Murphy et al. (2014) Murphy, J. D., Gebhardt, K., & Cradit, M. 2014, ApJ, 785, 143
  * Patterson & Thuan (1992) Patterson, R. J., & Thuan, T. X. 1992, ApJ, 400, L55
  * Pedregosa et al. (2011) Pedregosa, F., Varoquaux, G., Gramfort, A., et al. 2011, Journal of Machine Learning Research, 12, 2825
  * Peng et al. (2006) Peng, E. W., Côté, P., Jordán, A., et al. 2006, ApJ, 639, 838
  * Perrett et al. (2003) Perrett, K. M., Stiff, D. A., Hanes, D. A., et al. 2003, ApJ, 589, 790
  * Pulsoni et al. (2018) Pulsoni, C., Gerhard, O., Arnaboldi, M., et al. 2018, A&A, 618, A94
  * Quinn (1984) Quinn, P. J. 1984, ApJ, 279, 596
  * Richtler et al. (2011) Richtler, T., Salinas, R., Misgeld, I., et al. 2011, A&A, 531, A119
  * Romanowsky et al. (2012) Romanowsky, A. J., Strader, J., Brodie, J. P., et al. 2012, ApJ, 748, 29
  * Rudick et al. (2009) Rudick, C. S., Mihos, J. C., Frey, L. H., et al. 2009, ApJ, 699, 1518. doi:10.1088/0004-637X/699/2/1518
  * Strader et al. (2004) Strader, J., Brodie, J. P., & Forbes, D. A. 2004, AJ, 127, 3431
  * Taylor (2005) Taylor, M. B. 2005, Astronomical Data Analysis Software and Systems XIV, 347, 29
  * Tonry & Davis (1979) Tonry, J., & Davis, M. 1979, AJ, 84, 1511
  * Van Der Walt et al. (2011) Van Der Walt, S., Colbert, S. C., & Varoquaux, G. 2011, Comput. Sci. Eng., 13, 22
  * van Zee et al. (2004) van Zee, L., Barton, E. J., & Skillman, E. D. 2004, AJ, 128, 2797
  * Veale et al. (2017) Veale, M., Ma, C.-P., Thomas, J., et al. 2017, MNRAS, 464, 356
  * Virtanen et al. (2020) Virtanen, P., Gommers, R., Oliphant, T. E., et al. 2020, Nature Methods, 17, 261


# Multi-layer Feature Fusion Convolutional Network for Audio-visual Speech
Enhancement

Xinmeng Xu, Jianjun Hao (Corresponding author: Jianjun Hao) Xinmeng Xu is with
the Electronic $\&$ Elect. Engineering, School of Engineering, Trinity College
Dublin, Dublin, Ireland (e-mail: xux3@tcd.ie).Jianjun Hao is with School of
Foreign languages, HBUCM, Wuhan, China (e-mail: loyalcolin@163.com).

###### Abstract

Speech enhancement can potentially benefit from the visual information from
the target speaker, such as lip movement and facial expressions, because the
visual aspect of speech is essentially unaffected by acoustic environment. In
this paper, we address the problem of enhancing corrupted speech signal from
videos by using audio-visual (AV) neural processing. Most of recent AV speech
enhancement approaches separately process the acoustic and visual features and
fuse them via a simple concatenation operation. Although this strategy is
convenient and easy to implement, it comes with two major drawbacks: 1)
evidence in speech perception suggests that in humans the AV integration
occurs at a very early stage, in contrast to previous models that process the
two modalities separately at early stage and combine them only at a later
stage, thus making the system less robust, and 2) a simple concatenation does
not allow to control how the information from the acoustic and the visual
modalities is treated. To overcome these drawbacks, we propose a multi-layer
feature fusion convolution network (MFFCN), which separately process acoustic
and visual modalities for preserving each modality features while fusing both
modalities features layer by layer in encoding phase for enjoying the human AV
speech perception. In addition, considering the balance between the two
modalities, we design channel and spectral attention mechanisms to provide
additional flexibility in dealing with different types of information
expanding the representational ability of the convolution neural network.
Experimental results show that the proposed MFFCN demonstrates the performance
of the network superior to the state-of-the-art models.

###### Index Terms:

Speech enhancement, audio-visual, multi-layer feature fusion convolution
network (MFFCN), channel attention, spectral attention

##  I Introduction

Background noises greatly reduce the quality and intelligibility of the speech
signal, limiting the performance of speech-related applications, such as
speech recognition [1, 2], speaker verification [3, 4], and speech conversion
[5, 6], in real-world conditions. Consequently, there is the need for the
development of speech enhancement (SE) to generate enhanced speech with better
speech quality and clarity by suppressing background noise components in noisy
speech.

Figure 1: The general deep-learning-based audio-visual speech enhancement
system.

Conventional SE approaches, such as spectral subtraction [7], minimum mean
squared error (MMSE) estimation [8], Wiener filtering [9], Kalman filter [10],
and subspace methods [11], have been extensively studied in the past.
Recently, deep learning technologies have been successfully used for SE owing
to their superior capability to model complex non-linearity [12]. Although
these deep learning-based approaches are effective compared to conventional SE
approaches, they are prone to a label permutation (or ambiguity) error due to
their frame-by-frame or short segment-based processing paradigm [13, 14]. To
address this issue, the permutation invariant training using permutation loss
criterion is proposed [15, 16], but the label ambiguity problem still appears
in the inference stage, especially for unseen speakers.

Leveraging the visual streams of target speech signals is one of the effective
alternatives. Many researchers [17, 18, 19] have proved that visual cues such
as facial/lip movements can supplement acoustic information of the
corresponding speaker, helping speech perception, especially in noisy
environments. In audio-visual speech enhancement (AVSE) systems, acoustic and
visual features are integrated to derive unique characteristics [20, 21, 22].
In recent AVSE systems, the actual data processing is obtained with deep-
learning-based techniques. Generally, acoustic and visual features are
processed separately through using two neural network models. After that, the
output vectors of these models are fused, often by concatenation, which then
is used as input to another deep learning model (as shown in Fig 1). This
strategy is convenient and easy to be implemented.

Figure 2: Schematic diagram of the proposed multi-layer feature fusion
convolutional network (MFFCN).

In real-world scenarios, however, this strategy comes with two significant
drawbacks, (1) evidence in speech perception suggests that in humans the AV
integration occurs at a very early stage [23], the one that processes the two
modalities separately and combines them only at a later stage is failed to
exploit the the correlation between audio and video at a very early stage, but
the disadvantage of early fusion in AVSE systems is that usually the features
between the two modalities are inherently different, (2) a simple
concatenation does not allow to control how the information from the acoustic
and the visual modalities is treated, in consequence, the two modalities may
dominate each other, determining a decrease in the system’s performance.

In this letter, we highlight those limitations and tackle the appropriate
fusion strategy designing and unbalance between acoustic and visual modalities
problems, in AVSE processing. We propose a novel multi-layer feature fusion
convolutional network for robust speech enhancement, dubbed as MFFCN, by
designing a multi-layer fusion strategy to integrate early fusion and later
fusion with a single model, and by designing the channel and spectral
attention mechanisms to provide additional flexibility in dealing with
different types of information. In particular, the MFFCN separately processes
acoustic and visual modalities and fuses audio-visual modalities at each
encoder layer to extract AV features, which enjoys the benefit of human
auditory perception while avoiding the problem of inherent difference of
acoustic and visual modalities during early fusion. The channel and spectral
attention mechanisms are inserted in concatenated operation to give the weight
to the channels of concatenated features while paying more attention to
informative regions of each concatenated feature map. We demonstrate the
effectiveness of the proposed network with extensive experiments, achieving
large improvements in unconditioned scenarios on GRID [24] and TCD-TIMIT [25]
datasets.

##  II Model Architecture

###  II-A Overview

The diagram of proposed MFFCN is illustrated in Fig 2, where its inputs are
noisy speech Mel-spectrogram $Y$, and video frames $V$, and it’s output is the
enhanced speech Mel-spectrogram $S$. The MFFCN follows an encoder-decoder
architecture, and consists of encoder part, fusion part, embedding part, and
decoder part.

The encoder part of MFFCN is comprised of audio encoder and video encoder. The
audio encoder is designed as a CNN taking spectrogram as input, and each layer
of an audio encoder is followed by strided convolutional layer, batch
normalization, and Leaky-ReLU for non-linearity. The video encoder is used to
process the input face embedding through a number of max-pooling convolutional
layers followed by batch normalization, and Leaky-ReLU. Note that the
dimension of visual feature vector after convolutional layer has to be the
same as the corresponding audio feature vector. In addition, the audio decoder
is reversed in the audio encoder part by deconvolutions, followed again by
batch normalization and Leaky-ReLU.

Fusion part designates a merged dimension to implement fusion, and the audio
and video streams take the concatenation operation by channel attention for
giving the weight to each channel of audio and video feature vectors and are
through spectral attention to underscore the informative region of each
concatenated feature map. The embedding part is the bottleneck of the encoder-
decoder architecture, in which the channel and spectral attention mechanisms
are used to process the encoded features and its output is then fed into two
LSTM blocks for aggregating temporal contexts. The MFFCN fuse acoustic and
visual modalities at early and later stages while separately processing both
of modalities, which not only enjoys the benefits of human auditory system but
also avoid the the features inherent difference of two modalities.

###  II-B Channel and Spectral Attention Mechanisms

(a) Channel attention mechanism

(b) Spectral attention mechanism

Figure 3: Schematic diagram of channel and spectral attention mechanisms. TABLE I: Detailed architecture of the MFFCN encoder. Conv1 denotes the first convolution layer of the MFFCN encoder part. | Conv1 | Conv2 | Conv3 | Conv4 | Conv5 | Conv6 | Conv7 | Conv8 | Conv9 | Conv10
---|---|---|---|---|---|---|---|---|---|---
Num Filters | 64 | 64 | 128 | 128 | 256 | 256 | 512 | 512 | 1024 | 1024
Filter Size | (5, 5) | (4, 4) | (4, 4) | (4, 4) | (2, 2) | (2, 2) | (2, 2) | (2, 2) | (2, 2) | (2, 2)
Stride(audio) | (2, 2) | (1, 1) | (2, 2) | (1, 1) | (2, 1) | (1, 1) | (2, 1) | (1, 1) | (1, 5) | (1, 1)
MaxPool(video) | (2, 4) | (1, 2) | (2, 2) | (1, 1) | (2, 1) | (1, 1) | (2, 1) | (1, 1) | (1, 5) | (1, 1)

Convolutional operation treats the channel-wise and spectral unit-wise feature
equally, therefore, extracting AV features by the concatenation operation with
convolutional layers is unable to control how the information from the
acoustic and the visual modalities is treated. To address this issue, we
propose the channel and spectral attention to provide additional flexibility
in dealing with different types of information and expand the representational
ability of convolution operation.

####  II-B1 Channel Attention Mechanism

The channel attention mainly concerns that different channel feature have
totally different weighted information. Inspired by [26, 27], we employ the
channel-wise selective operation to design channel attention mechanism to
perform weighting for different channels with different type of features.

As illustrated in Fig 3(a), given acoustic feature A and visual feature V the
output of channel attention N is obtained by fusing sub-module in the
following four steps:

M $\displaystyle=\text{Conv}(\text{Concat}([\textbf{V},\textbf{A}])),$ (1) g
$\displaystyle=\text{GlobalPooling}(\textbf{M}),$ (2)
$\displaystyle\textbf{w}_{V},\textbf{w}_{A}$
$\displaystyle=\text{softmax}([F_{FC}^{1}(\textbf{g}),F_{FC}^{2}(\textbf{g})]),$
(3) $\displaystyle\textbf{N}=$
$\displaystyle\text{Conv}(\text{Concat}([\textbf{V}\cdot\textbf{w}_{V},\textbf{A}\cdot\textbf{w}_{A}])),$
(4)

where $F_{FC}^{1}(\cdot)$ and $F_{FC}^{2}(\cdot)$ refer to two independent
fully-connected layers, “Conv” and “Concate” represent convolution and
concatenation operation, respectively.

To be concrete, we first concatenate the encoded acoustic and visual features
from two branches of MFFCN encoder part and apply a convolution block to make
the size of concatenation the same as the acoustic or visual features. Then,
we adopt a global pooling to produce a global vector g which is used as the
guidance of adaptive and accurate channel weighting operation between two
modalities. Next, we apply two fully connected layers $F_{FC}^{1}(\cdot)$ and
$F_{FC}^{2}(\cdot)$ to generate two weight vectors ($\textbf{w}_{V}$ and
$\textbf{w}_{A}$) to assign the weight for each channel. In this way, acoustic
and visual features can be processed selectively based on their
characteristics.

####  II-B2 Spectral Attention Mechanism

Considering the informative feature distribution is uneven on different
feature maps of different modalities, we propose a spectral attention
mechanism to make the network pay more attention to informative features, such
as lip shape on video modality and speech presence regions on audio modality.

As illustrated in Fig 3(b), we directly feed the input N (the output of the
channel attention mechanism) into two convolution layers with ReLU and sigmoid
activation function to generate the weighting map:

$\textbf{w}_{N}=\sigma(\text{Conv}(\delta(\text{Conv}(\textbf{N})))),$ (5)

where $\sigma$ is sigmoid function, and $\delta$ is the ReLU function.

Finally, the input N is element-wise multiplied with the weighting map
$\textbf{w}_{N}$ to obtain the output of spectral attention O.

TABLE II: Performance of trained networks Evaluation metrics | STOI(%) | PESQ
---|---|---
Test SNR | -5 dB | 0 dB | -5 dB | 0 dB
Interference | Speech | Ambient | Speech | Ambient | Speech | Ambient | Speech | Ambient
Unprocessed | 57.82 | 51.44 | 64.74 | 62.59 | 1.59 | 1.03 | 1.66 | 1.24
Audio-only CRN [31] | 78.26 | 78.67 | 80.82 | 83.35 | 2.01 | 2.19 | 2.47 | 2.58
VSE [32] | 79.94 | 83.33 | 84.61 | 87.90 | 2.36 | 2.41 | 2.77 | 2.94
AV(SE)2 [33] | 81.06 | 85.41 | 86.17 | 89.44 | 2.42 | 2.49 | 2.81 | 2.99
Deep U-Net Early Fusion [34] | 81.23 | 85.34 | 87.86 | 90.23 | 2.47 | 2.53 | 2.79 | 3.01
MFFCN (Proposed) | 83.24 | 87.82 | 90.21 | 91.46 | 2.71 | 2.80 | 2.97 | 3.09
MFFCN-w/o channel attention | 81.36 | 86.23 | 88.69 | 89.31 | 2.63 | 2.75 | 2.91 | 3.04
MFFCN-w/o spectral attention | 80.59 | 85.14 | 86.36 | 89.23 | 2.56 | 2.47 | 2.76 | 3.02
MFFCN-w/o attention mechanisms | 80.47 | 82.71 | 85.02 | 88.94 | 2.39 | 2.47 | 2.83 | 2.96

##  III Experimental Setup

###  III-A Dataset

We use two public AV datasets to train our model: the first is GRID [24] that
consist of video recording where 18 male speakers and 16 female speakers
pronounce 1000 sentences each, the second is TCD-TIMIT [25], which comprise 32
male speakers and 30 female speakers around 200 videos each.

We shuffle and split the datasets to training set, validation set, and test
set to 24300, 4400, and 1200 utterances, respectively. The noise dataset is
self-collected set, containing 25.3 hours ambient noise categorized into 12
types: room, car, instrument, engine, train, human chatting, air-brake, water,
street, mic-noise, ring-bell, and music. The speech-noise mixtures in training
and validation are generated by randomly selecting utterances from speech
dataset and noise dataset and then mixing them up at random SNR between -10dB
and 10dB. The evaluation set is generated SNR at 0dB and -5 dB.

###  III-B Training and Network Parameters

The audio representation is extracted from raw audio waveforms by using Short
Time Fourier Transform (STFT) with Hanning window function after resampling
the audio signal to 16 kHz. Each frame contains a window of 40 milliseconds,
and the frame shift is 160 samples (10 milliseconds). For each speech frame, a
log Mel-scale spectrogram is extracted by multiplying the spectrogram via a
Mel-scale filter bank, resulting in spectrograms of size 80$\times$20\. Visual
feature is extracted from the input videos that is re-sampled to 25 frames per
second. The video segment processed as input is the sizeof
128$\times$128$\times$5 using the 20 mouth landmarks from 68 facial landmarks
suggested by Kazemi et al.[28]. For convenience, the processed video segment
is zoomed to 80$\times$80$\times$5 by bilinear interpolation algorithm[29].
The proposed MFFCN has 10 convolutional layers for each encoder, as shown in
Table I and 10 deconvolutional layers for each decoder. The models are trained
with the Adam optimizer [30] with the learning rate of 0.0002 , and batch size
of 8. The mean squared error (MSE) serves as the objective function.

TABLE III: Comparison with different fusion strategies

Fusion Strategy | -5 dB | 0 dB
---|---|---
STOI (%) | PESQ | STOI (%) | PESQ
Early Fusion | 75.23 | 2.24 | 81.41 | 2.66
Late Fusion | 75.01 | 2.21 | 81.54 | 2.59
Intermediate Fusion (VSE) | 76.68 | 2.28 | 83.54 | 2.71
Intermediate Fusion (AV(SE)2) | 79.31 | 2.37 | 84.88 | 2.79
Multi-layer Feature Fusion | 80.47 | 2.39 | 85.02 | 2.83

##  IV Results and Analysis

###  IV-A Overall Performance

The performance of MFFCN is evaluated with the following metrics: Perceptual
evaluation of speech quality (PESQ) (from -0.5 to 4.5) [35], and short-time
objective intelligibility measure (from 0 to 100($\%$)) [36]. Table II
presents comprehensive evaluations for four baseline models on untrained
noises and untrained speakers, in which “Speech” denotes the noise from human
speech and “Ambient” denotes the ambient noise. In addition, the four elected
baseline models are: Audio-only CRN: an audio-only speech enhancement model
based on convolutional recurrent network [31].VSE: a middle fusion audio-
visual neural network for visual speech enhancement [32]. AV(SE)2: an
intermediate fusion audio-visual speech enhancement model with several cross-
modal squeeze-excitation [33]. Deep U-Net: a early fusion audio-visual speech
enhancement model with RNN attention blocks [34].

According to Table II, the proposed MFFCN yields the best results in all
conditions. Firstly, we compare the audio-only CRN with other 4 models. We
observe that even audio-only CRN achieves about 20.44$\%$ to 27.33$\%$ STOI
improvement and 0.42 to 1.64 PESQ improvement over unprocessed mixtures, the
audio-only CRN is unable to outperform audio-visual SE methods. Secondly, the
AV(SE)2 and Deep U-net improves the network performance by adopting different
attention mechanisms, and these two models performs better than VSE that is
not applying attention operation. This demonstrates the necessity of balancing
between acoustic and visual modalities. Finally, our proposed MFFCN
consistently outperforms VSE, AV(SE)2, and Deep U-net in all conditions.

To verify the effect of proposed multi-layer feature fusion strategy, we show
the comparison in Table III to investigate the effect of different fusion
strategies. Note that (1) early fusion which fuses acoustic and visual
modalities at first network layer, late fusion fuse the two modalities at last
network layer, (2) intermediate fusion (VSE) that is used in VSE approach and
fuse the two modalities at bottleneck, (3) intermediate fusion (AV(SE)2) that
is used in AV(SE)2 approach and fuse the two modalities at every decoder
layers, (4) multi-layer feature fusion is the proposed fusion strategy.
According to Table III, we observe that multi-layer feature fusion
significantly performs better than the other 4 fusion strategies, which fully
demonstrates the superiority of our proposed fusion strategy.

###  IV-B Ablation Study

In this study, we evaluate variants of proposed MFFCN when removing channel
attention mechanism, spectral attention mechanism, and two attention
mechanisms, respectively. From Table II, we can conclude that the existence of
the channel and spectral attention mechanisms does promote the network
performance. The channel and spectral attention mechanisms improve 5.11$\%$
and 0.33 on STOI and PESQ, respectively, under the condition of ambient noise
at -5 dB. In addition, a single spectral attention mechanism performs better
than a single channel attention mechanism. Fig 4 shows the visualization of
noisy speech, MFFCN without channel and spectral attention mechanisms enhanced
speech, MFFCN without spectral attention enhanced speech, MFFCN without
channel attention enhanced speech, complete MFFCN enhanced speech, and clean
speech, respectively.

(a) Noisy

(b) w/o C $\&$ S

(c) w/o S

(d) w/o C

(e) Complete

(f) Clean

Figure 4: Example of input and enhanced spectra from an example speech
utterance. (a) Noisy speech input from test data under the condition of
ambient noise at -5 dB. (b) Speech enhanced by MFFCN-w/o channel and spectral
attention. (c) Speech enhanced by MFFCN-w/o channel attention. (d) Speech
enhanced by proposed MFFCN-w/o spectral attention. (e) Speech enhanced by
MFFCN. (f) Clean speech.

##  V Conclusion

This letter proposes a MFFCN model for AVSE. In particular, the MFFCN adopts
multi-layer fusion strategy to fuse AV features layer by layer at encoder
phase while separately processing them. In addition, the channel and spectral
attention mechanisms are introduced to assign the weight to the channels while
paying more attention to informative regions of the fused AV feature maps to
avoid the unbalancing between the two modalities. Results show that the
proposed model has better performance than recent models on the GRID and TCD-
TIMIT datasets.

## References

  * [1] Hinton, Geoffrey, et al., “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,” IEEE Signal processing magazine, vol. 29, no. 6, pp. 82-97, 2012.
  * [2] Baevski A., Hsu W.N., Conneau A. and Auli M, “Unsupervised speech recognition,” Advances in Neural Information Processing Systems, vol. 34, 2021.
  * [3] Wu, Haibin, et al., “Improving the adversarial robustness for speaker verification by self-supervised learning,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 202-217, 2021.
  * [4] Liu T., Das R. K., Lee K. A. and Li H. “Neural acoustic-phonetic approach for speaker verification with phonetic attention mask,” IEEE Signal Processing Letters, vol. 29, pp. 782-786, 2022.
  * [5] Zhou K., Sisman B., Liu R. and Li H., “Emotional voice conversion: Theory, databases and esd,” Speech Communication, vol. 137, pp. 1-18, 2022.
  * [6] Zhang M., Zhou Y., Zhao L. and Li H., “Transfer learning from speech synthesis to voice conversion with non-parallel training data,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 1290-1302, 2021.
  * [7] S. Boll, “Suppression of acoustic noise in speech using spectral subtraction,” IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 27, no. 2, pp. 113–120, 1979.
  * [8] Y. Ephraim and D. Malah, “Speech enhancement using a minimum-mean square error short-time spectral amplitude estimator,” IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 32, no. 6, pp. 1109–1121, 1984.
  * [9] Jae Lim and A. Oppenheim, “All-pole modeling of degraded speech,”IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 26, no. 3, pp. 197–210, 1978.
  * [10] K. Paliwal and A. Basu, “A speech enhancement method based on Kalman filtering,” ICASSP ’87. IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 12, pp. 177–180, 1987.
  * [11] Y. Ephraim and H. L. Van Trees, “A signal subspace approach for speech enhancement,” IEEE Transactions on Speech and Audio Processing, vol. 3, no. 4, pp. 251–266, 1995.
  * [12] D. Wang and J. Chen, “Supervised speech separation based on deep learning: An overview,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 10, pp. 1702– 1726, 2018.
  * [13] Hershey J R, Chen Z, Le Roux J and Watanabe S, “Deep clustering: Discriminative embeddings for segmentation and separation”, in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 31-35, 2016.
  * [14] Zhuo Chen, Yi Luo, and Nima Mesgarani, “Deep attractor network for single-microphone speaker separation”, in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 246-250, 2017.
  * [15] Yu D, Kolbæk M, Tan ZH, Jensen J., “Permutation invariant training of deep models for speaker-independent multi-talker speech separation”, in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 241-245, 2017.
  * [16] Kolbæk M, Yu D, Tan ZH, Jensen J., “Multitalker speech separation with utterance-level permutation invariant training of deep recurrent neural networks,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 10, pp. 1901-1913, 2017.
  * [17] McGurk H and MacDonald J, “Hearing lips and seeing voices,” Nature, vol.264, no. 5588, pp. 746-748, 1976.
  * [18] Massaro DW and Simpson JA, “Speech perception by ear and eye: A paradigm for psychological inquiry,” Psychology Press, 2014.
  * [19] Rosenblum LD, “Speech perception as a multimodal phenomenon”, Current Directions in Psychological Science, vol. 17, no. 6, pp. 405-409, 2008.
  * [20] Afouras T, Chung JS, Zisserman A, “The Conversation: Deep Audio-Visual Speech Enhancement”, Proc. Interspeech, pp. 3244-3248, 2018.
  * [21] Xu X, Wang Y, Xu D, Peng Y, Zhang C, Jia J, Chen B. “VSEGAN: Visual speech enhancement generative adversarial network”, in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) pp. 7308-7311, 2022.
  * [22] Adeel A, Gogate M, Hussain A, “Contextual deep learning-based audio-visual switching for speech enhancement in real-world environments,” Information Fusion, vol 29, pp. 163-170, 2022.
  * [23] Schwartz JL, Berthommier F, Savariaux C, “Audio-visual scene analysis: evidence for a “very-early” integration process in audio-visual speech perception,” In Seventh International Conference on Spoken Language Processing, 2002.
  * [24] M. Cooke, J. Barker, S. Cunningham, and X. Shao, “An audio-visual corpus for speech perception and automatic speech recognition,” The Journal of the Acoustical Society of America, vol. 120, no. 5, pp. 2421–2424, 2006.
  * [25] N. Harte and E. Gillen, “TCD-TIMIT: An audio-visual corpus of continuous speech,” IEEE Transactions on Multimedia, vol. 17, no. 5, pp. 603–615, 2015.
  * [26] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7132–7141, 2018.
  * [27] X. Li, W. Wang, X. Hu, and J. Yang, “Selective kernel networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 510–519, 2019.
  * [28] Kazemi V, Sullivan J, “One millisecond face alignment with an ensemble of regression trees,” In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 1867-1874, 2014.
  * [29] K. T. Gribbon and D. G. Bailey, “A novel approach to real-time bi-linear interpolation,” in Proceedings. DELTA 2004. Second IEEE International Workshop on Electronic Design, Test and Applications. IEEE, pp. 126–131, 2004.
  * [30] Kingma D P. and Jimmy B. “Adam: A Method for Stochastic Optimization,” In The International Conference on Learning Representations (ICLR), 2015.
  * [31] K. Tan and D. Wang, “A convolutional recurrent neural network for real-time speech enhancement,” in Proc. Interspeech, pp. 3229–3233, 2018.
  * [32] A. Gabbay, A. Shamir, and S. Peleg, “Visual speech enhancement,” in Proc. Interspeech, pp. 1170–1174, 2018.
  * [33] M. L. Iuzzolino and K. Koishida, “AV(SE)2: Audio-visual squeeze-excite speech enhancement,” in ICASSP 2020-2020 IEEE international conference on acoustics, speech and signal processing (ICASSP), IEEE, pp. 7539–7543, 2020.
  * [34] Hwang, J.W., Park, R.H. and Park, H.M., “Efficient Audio-Visual Speech Enhancement Using Deep U-Net With Early Fusion of Audio and Video Information and RNN Attention Blocks,” IEEE Access, vol. 9, pp.137584-137598, 2021.
  * [35] A . W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra, “Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,” in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), IEEE, pp. 749–752, 2001.
  * [36] Taal CH, Hendriks RC, Heusdens R, Jensen J, “An algorithm for intelligibility prediction of time–frequency weighted noisy speech”, IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, no.7, pp. 2125-2136, 2011.